Scale Space and Variational Methods in Computer Vision: Second International Conference, SSVM 2009, Voss, Norway, June 1-5, 2009. Proceedings (Lecture Notes in Computer Science)

Lecture Notes in Computer Science Commenced Publication in 1973 Founding and Former Series Editors: Gerhard Goos, Juris Hartmanis, and Jan van Leeuwen

Editorial Board David Hutchison Lancaster University, UK Takeo Kanade Carnegie Mellon University, Pittsburgh, PA, USA Josef Kittler University of Surrey, Guildford, UK Jon M. Kleinberg Cornell University, Ithaca, NY, USA Alfred Kobsa University of California, Irvine, CA, USA Friedemann Mattern ETH Zurich, Switzerland John C. Mitchell Stanford University, CA, USA Moni Naor Weizmann Institute of Science, Rehovot, Israel Oscar Nierstrasz University of Bern, Switzerland C. Pandu Rangan Indian Institute of Technology, Madras, India Bernhard Steffen University of Dortmund, Germany Madhu Sudan Massachusetts Institute of Technology, MA, USA Demetri Terzopoulos University of California, Los Angeles, CA, USA Doug Tygar University of California, Berkeley, CA, USA Gerhard Weikum Max-Planck Institute of Computer Science, Saarbruecken, Germany

5567

Xue-Cheng Tai Knut Mørken Marius Lysaker Knut-Andreas Lie (Eds.)

Scale Space and Variational Methods in Computer Vision Second International Conference, SSVM 2009 Voss, Norway, June 1-5, 2009 Proceedings

13

Volume Editors Xue-Cheng Tai Department of Mathematics University of Bergen, Norway and Division of Mathematical Science Nanyang Technological University, Singapore E-mail: [email protected] Knut Mørken Department of Informatics and Centre of Mathematics for Applications University of Oslo, Norway E-mail: [email protected] Marius Lysaker Simula Research Laboratory Lysaker, Norway E-mail: [email protected] Knut-Andreas Lie Centre of Mathematics for Applications University of Oslo, Norway and SINTEF ICT, Oslo, Norway E-mail: [email protected]

Library of Congress Control Number: Applied for CR Subject Classification (1998): I.4, I.5, I.3.5, I.2.10, G.1.2, F.2.2 LNCS Sublibrary: SL 6 – Image Processing, Computer Vision, Pattern Recognition, and Graphics ISSN ISBN-10 ISBN-13

0302-9743 3-642-02255-3 Springer Berlin Heidelberg New York 978-3-642-02255-5 Springer Berlin Heidelberg New York

This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. springer.com © Springer-Verlag Berlin Heidelberg 2009 Printed in Germany Typesetting: Camera-ready by author, data conversion by Scientific Publishing Services, Chennai, India Printed on acid-free paper SPIN: 12689675 06/3180 543210

Preface

This book contains 71 original, scientific articles that address state-of-the-art research related to scale space and variational methods for image processing and computer vision. Topics covered in the book range from mathematical analysis of both established and new models, fast numerical methods, image analysis, segmentation, registration, surface and shape construction and processing, to real applications in medical imaging and computer vision. The ideas of scale space and variational methods related to partial differential equations are central concepts. The papers reflect the newest developments in these fields and also point to the latest literature. All the papers were submitted to the Second International Conference on Scale Space and Variational Methods in Computer Vision, which took place in Voss, Norway, during June 1–5, 2009. The papers underwent a peer review process similar to that of high-level journals in the field. We thank the authors, the Scientific Committee, the Program Committee and the reviewers for their hard work and helpful collaboration. Their contribution has been crucial for the efficient processing of this book, and for the success of the conference. Finally, we wish to thank those who have supported and helped to organize the conference. First and foremost it is a pleasure to acknowledge the generous financial support from the Centre of Mathematics for Applications (CMA) at the University of Oslo and the Research Council of Norway. In addition, partial support was given by the Centre of Integrated Petroleum Research (CIPR) at the University of Bergen and the Simula Research Laboratory (SRL). Moreover, we would like to thank Tiril P. Gurholt and Andrew McMurry for their support, both with technical and administrative matters. Members and students from the Mathematical Imaging and Vision Group at the Nanyang Technological University of Singapore and the University of Bergen, Norway deserve special thanks for their kind help. March 2009

Xue-Cheng Tai Knut Mørken Marius Lysaker Knut-Andreas Lie

Organization

Organizing Commitee and Editors Xue-Cheng Tai

University of Bergen, Norway, and Nanyang Technology University Singapore, Conference Chair Sintef, Norway Simula Research Laboratory, Norway University of Oslo, Norway

Knut-Andreas Lie Marius Lysaker Knut Mørken

Scientific Committee Alfred M.Bruckstein Tony F. Chan Mads Nielsen Stanley Osher Nikos Paragios Bart M. ter Haar Romeny Christoph Schnoerr Fiorella Sgallari Joachim Weickert

Technion IIT, Israel University of California at LA, USA University of Copenhagen, Denmark University of California at LA, USA Ecole Centrale de Paris, France Eindhoven University of Technology, The Netherlands University of Heidelberg, Germany University of Bologna, Italy Saarland University, Germany

Program Commitee Luis Alvarez Noura Azzabou Thomas Brox Bernhard Burgeth Vicent Caselles Raymond Chan Yeowmeng Chee Yunmei Chen Daniel Cremers Francoise Dibos Michael Felsberg Luc Florack

Lewis Griffin Anders Heyden Charles Kervrann Ron Kimmel Arjan Kuijper Georg Langs Antonio Leitao Riccardo March Antonio Marquina Etienne Memin Karol Mikula Jan Modersitzki

Michael Ng Mila Nikolova Martin Rumpf Otmar Scherzer Nir Sochen Gabriele Steidl Demetri Terzopoulos David Tschumperle Baba C. Vemuri Hongkai Zhao Haomin Zhou

VIII

Organization

Invited Speakers Antonin Chambolle, CMAP - Ecole Polytechnique, France Raymond Chan, The Chinese University of Hong Kong, China Amiram Grinvald, Weizmann Institute of Science, Israel

Sponsoring Institutions Centre of Mathematics for Applications, University of Oslo Research Council of Norway Centre of Integrated Petroleum Research, University of Bergen Simula Research Laboratory

Table of Contents

Segmentation and Detection Graph Cut Optimization for the Piecewise Constant Level Set Method Applied to Multiphase Image Segmentation . . . . . . . . . . . . . . . . . . . . . . . . . Egil Bae and Xue-Cheng Tai Tubular Anisotropy Segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fethallah Benmansour and Laurent D. Cohen

1 14

An Unconstrained Multiphase Thresholding Approach for Image Segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Benjamin Berkels

26

Extraction of the Intercellular Skeleton from 2D Images of Embryogenesis Using Eikonal Equation and Advective Subjective Surface Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Paul Bourgine, Peter Frolkoviˇc, Karol Mikula, Nadine Peyri´eras, and Mariana Remeˇs´ıkov´ a

38

On Level-Set Type Methods for Recovering Piecewise Constant Solutions of Ill-Posed Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Adriano DeCezaro, Antonio Leit˜ ao, and Xue-Cheng Tai

50

The Nonlinear Tensor Diffusion in Segmentation of Meaningful Biological Structures from Image Sequences of Zebrafish Embryogenesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Olga Drbl´ıkov´ a, Karol Mikula, and Nadine Peyri´eras

63

Composed Segmentation of Tubular Structures by an Anisotropic PDE Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Elena Franchini, Serena Morigi, and Fiorella Sgallari

75

Extrapolation of Vector Fields Using the Infinity Laplacian and with Applications to Image Segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Laurence Guillot and Carole Le Guyader

87

A Schr¨ odinger Equation for the Fast Computation of Approximate Euclidean Distance Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Karthik S. Gurumoorthy and Anand Rangarajan

100

Semi-supervised Segmentation Based on Non-local Continuous Min-Cut . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Nawal Houhou, Xavier Bresson, Arthur Szlam, Tony F. Chan, and Jean-Philippe Thiran

112

X

Table of Contents

Momentum Based Optimization Methods for Level Set Segmentation . . . Gunnar L¨ ath´en, Thord Andersson, Reiner Lenz, and Magnus Borga Optimization of Divergences within the Exponential Family for Image Segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Francois Lecellier, Stephanie Jehan-Besson, Jalal Fadili, Gilles Aubert, and Marinette Revenu Convex Multi-class Image Labeling by Simplex-Constrained Total Variation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jan Lellmann, J¨ org Kappes, Jing Yuan, Florian Becker, and Christoph Schn¨ orr Geodesically Linked Active Contours: Evolution Strategy Based on Minimal Paths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Julien Mille and Laurent D. Cohen

124

137

150

163

Validation of Watershed Regions by Scale-Space Statistics . . . . . . . . . . . . . Tomoya Sakai and Atsushi Imiya

175

Adaptation of Eikonal Equation over Weighted Graph . . . . . . . . . . . . . . . . Vinh-Thong Ta, Abderrahim Elmoataz, and Olivier L´ezoray

187

A Variational Model for Interactive Shape Prior Segmentation and Real-Time Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Manuel Werlberger, Thomas Pock, Markus Unger, and Horst Bischof

200

Image Enhancement and Reconstruction A Nonlinear Probabilistic Curvature Motion Filter for Positron Emission Tomography Images . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Musa Alrefaya, Hichem Sahli, Iris Vanhamel, and Dinh Nho Hao

212

Finsler Geometry on Higher Order Tensor Fields and Applications to High Angular Resolution Diffusion Imaging . . . . . . . . . . . . . . . . . . . . . . . . . Laura Astola and Luc Florack

224

Bregman-EM-TV Methods with Application to Optical Nanoscopy . . . . . Christoph Brune, Alex Sawatzky, and Martin Burger

235

PDE-Driven Adaptive Morphology for Matrix Fields . . . . . . . . . . . . . . . . . Bernhard Burgeth, Michael Breuß, Luis Pizarro, and Joachim Weickert

247

On Semi-implicit Splitting Schemes for the Beltrami Color Flow . . . . . . . Lorina Dascal, Guy Rosman, Xue-Cheng Tai, and Ron Kimmel

259

Multi-scale Total Variation with Automated Regularization Parameter Selection for Color Image Restoration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yiqiu Dong and Michael Hinterm¨ uller

271

Table of Contents

Multiplicative Noise Cleaning via a Variational Method Involving Curvelet Coefficients . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Sylvain Durand, Jalal Fadili, and Mila Nikolova Projected Gradient Based Color Image Decomposition . . . . . . . . . . . . . . . . Vincent Duval, Jean-Fran¸cois Aujol, and Luminita Vese

XI

282 295

A Dual Formulation of the TV-Stokes Algorithm for Image Denoising . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Christoffer A. Elo, Alexander Malyshev, and Talal Rahman

307

Anisotropic Regularization for Inverse Problems with Application to the Wiener Filter with Gaussian and Impulse Noise . . . . . . . . . . . . . . . . . . Micha Feigin and Nir Sochen

319

Locally Adaptive Total Variation Regularization . . . . . . . . . . . . . . . . . . . . . Markus Grasmair Basic Image Features (BIFs) Arising from Approximate Symmetry Type . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Lewis D. Griffin, Martin Lillholm, Mike Crosier, and Justus van Sande

331

343

An Anisotropic Fourth-Order Partial Differential Equation for Noise Removal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mohammad Reza Hajiaboli

356

Enhancement of Blurred and Noisy Images Based on an Original Variant of the Total Variation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Khalid Jalalzai and Antonin Chambolle

368

Coarse-to-Fine Image Reconstruction Based on Weighted Differential Features and Background Gauge Fields . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Bart Janssen, Remco Duits, and Luc Florack

377

Edge-Enhanced Image Reconstruction Using (TV) Total Variation and Bregman Refinement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Shantanu H. Joshi, Antonio Marquina, Stanley J. Osher, Ivo Dinov, John D. Van Horn, and Arthur W. Toga Nonlocal Variational Image Deblurring Models in the Presence of Gaussian or Impulse Noise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Miyoun Jung and Luminita A. Vese A Geometric PDE for Interpolation of M -Channel Data . . . . . . . . . . . . . . Frank Lenzen and Otmar Scherzer An Edge-Preserving Multilevel Method for Deblurring, Denoising, and Segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Serena Morigi, Lothar Reichel, and Fiorella Sgallari

389

401 413

426

XII

Table of Contents

Fast Dejittering for Digital Video Frames . . . . . . . . . . . . . . . . . . . . . . . . . . . Mila Nikolova

439

Sparsity Regularization for Radon Measures . . . . . . . . . . . . . . . . . . . . . . . . . Otmar Scherzer and Birgit Walch

452

Split Bregman Algorithm, Douglas-Rachford Splitting and Frame Shrinkage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Simon Setzer Anisotropic Smoothing Using Double Orientations . . . . . . . . . . . . . . . . . . . Gabriele Steidl and Tanja Teuber

464 477

Image Denoising Using TV-Stokes Equation with an Orientation-Matching Minimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Xue-Cheng Tai, Sofia Borok, and Jooyoung Hahn

490

Augmented Lagrangian Method, Dual Methods and Split Bregman Iteration for ROF Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Xue-Cheng Tai and Chunlin Wu

502

The Convergence of a Central-Difference Discretization of Rudin-Osher-Fatemi Model for Image Denoising . . . . . . . . . . . . . . . . . . . . . Ming-Jun Lai, Bradley Lucier, and Jingyue Wang

514

Theoretical Foundations for Discrete Forward-and-Backward Diffusion Filtering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Martin Welk, Guy Gilboa, and Joachim Weickert

527

L0 -Norm and Total Variation for Wavelet Inpainting . . . . . . . . . . . . . . . . . Andy C. Yau, Xue-Cheng Tai, and Michael K. Ng

539

Total-Variation Based Piecewise Affine Regularization . . . . . . . . . . . . . . . . Jing Yuan, Christoph Schn¨ orr, and Gabriele Steidl

552

Image Denoising by Harmonic Mean Curvature Flow . . . . . . . . . . . . . . . . . Mourad Z´era¨ı

565

Motion Analysis, Optical Flow, Registration and Tracking Tracking Closed Curves with Non-linear Stochastic Filters . . . . . . . . . . . . Christophe Avenel, Etienne M´emin, and Patrick P´erez A Multi-scale Feature Based Optic Flow Method for 3D Cardiac Motion Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Alessandro Becciu, Hans van Assen, Luc Florack, Sebastian Kozerke, Vivian Roode, and Bart M. ter Haar Romeny

576

588

Table of Contents

XIII

A Combined Segmentation and Registration Framework with a Nonlinear Elasticity Smoother . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Carole Le Guyader and Luminita A. Vese

600

A Scale-Space Approach to Landmark Constrained Image Registration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Eldad Haber, Stefan Heldmann, and Jan Modersitzki

612

A Variational Approach for Volume-to-Slice Registration . . . . . . . . . . . . . . Stefan Heldmann and Nils Papenberg Hyperbolic Numerics for Variational Approaches to Correspondence Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Henning Zimmer, Michael Breuß, Joachim Weickert, and Hans-Peter Seidel

624

636

Surfaces and Shapes From a Single Point to a Surface Patch by Growing Minimal Paths . . . . . Fethallah Benmansour and Laurent D. Cohen

648

Optimization of Convex Shapes: An Approach to Crystal Shape Identification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Timo Eirola and Toni Lassila

660

An Implicit Method for Interpolating Two Digital Closed Curves on Parallel Planes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Nikolaos Gabrielides and Laurent Cohen

672

Pose Invariant Shape Prior Segmentation Using Continuous Cuts and Gradient Descent on Lie Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Niels Chr. Overgaard, Ketut Fundana, and Anders Heyden

684

A Non-local Approach to Shape from Ambient Shading . . . . . . . . . . . . . . . Emmanuel Prados, Nitin Jindal, and Stefano Soatto

696

An Elasticity Approach to Principal Modes of Shape Variation . . . . . . . . Martin Rumpf and Benedikt Wirth

709

Pre-image as Karcher Mean Using Diffusion Maps: Application to Shape and Image Denoising . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Nicolas Thorstensen, Florent Segonne, and Renaud Keriven

721

Fast Shape from Shading for Phong-Type Surfaces . . . . . . . . . . . . . . . . . . . Oliver Vogel, Michael Breuß, Thomas Leichtweis, and Joachim Weickert

733

Generic Scene Recovery Using Multiple Images . . . . . . . . . . . . . . . . . . . . . . Kuk-Jin Yoon, Emmanuel Prados, and Peter Sturm

745

XIV

Table of Contents

Scale Space and Feature Extraction Highly Accurate PDE-Based Morphology for General Structuring Elements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Michael Breuß and Joachim Weickert Computational Geometry-Based Scale-Space and Modal Image Decomposition: Application to Light Video-Microscopy Imaging . . . . . . . Anatole Chessel, Bertrand Cinquin, Sabine Bardin, Jean Salamero, and Charles Kervrann

758

770

Highlight on a Feature Extracted at Fine Scales: The Pointwise Lipschitz Regularity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Christophe Damerval and Sylvain Meignen

782

Line Enhancement and Completion via Linear Left Invariant Scale Spaces on SE(2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Remco Duits and Erik Franken

795

Spatio-Featural Scale-Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Michael Felsberg Scale Spaces on the 3D Euclidean Motion Group for Enhancement of HARDI Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Erik Franken and Remco Duits

808

820

On the Rate of Structural Change in Scale Spaces . . . . . . . . . . . . . . . . . . . . David Gustavsson, Kim S. Pedersen, Francois Lauze, and Mads Nielsen

832

Transitions of a Multi-scale Image Hierarchy Tree . . . . . . . . . . . . . . . . . . . . Arjan Kuijper

844

Local Scale Measure for Remote Sensing Images . . . . . . . . . . . . . . . . . . . . . Bin Luo, Jean-Fran¸cois Aujol, and Yann Gousseau

856

Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

869

Graph Cut Optimization for the Piecewise Constant Level Set Method Applied to Multiphase Image Segmentation Egil Bae1 and Xue-Cheng Tai2 1

Department of Mathematics, University of Bergen, Norway [email protected] 2 Department of Mathematics, University of Bergen, Norway and Division of Mathematical Sciences, School of Physical and Mathematical Sciences, Nanyang Technological University, Singapore [email protected]

Abstract. The piecewise constant level set method (PCLSM) has recently emerged as a variant of the level set method for variational interphase problems. Traditionally, the Euler-Lagrange equations are solved by some iterative numerical method for PDEs. Normally the speed is slow. In this work, we focus on the piecewise constant level set method (PCLSM) applied to the multiphase Mumford-Shah model for image segmentation. Instead of solving the Euler-Lagrange equations of the resulting minimization problem, we propose an efficient combinatorial optimization technique, based on graph cuts. Because of a simplification of the length term in the energy induced by the PCLSM, the minimization problem is not NP hard. Numerical experiments on image segmentation demonstrate that the new approach is very superior in terms of efficiency, while maintaining the same quality.

1

Introduction

The level set method [1, 2] is a powerful tool for interphase problems. It has numerous applications in computer vision, fluid dynamics and inverse problems. The interphase is implicitly represented by a higher dimensional level set function. Originally, the signed distance functions were used as level set functions. Later the work of [3, 4, 5] introduced piecewise constant level set functions, representing the interphases by discontinuities. This has certain advantages, such as ability to represent several interphases by one single level set function. This method will be referred to as the piecewise constant level set method (PCLSM) In computer vision, the level set method has been applied with great success to image segmentation. Of particular importance is the Mumford-Shah model [6], 

Support from the Norwegian Research Council (eVita project 166075), National Science Foundation of Singapore (NRF2007IDM-IDM002-010) and Ministry of Education of Singapore (Moe Tier 2 T207B2202) are gratefully acknowledged.

X.-C. Tai et al. (Eds.): SSVM 2009, LNCS 5567, pp. 1–13, 2009. c Springer-Verlag Berlin Heidelberg 2009 

2

E. Bae and X.-C. Tai

which is an established image segmentation model. In [7, 8], Chan and Vese proposed a numerical realization of this model based on traditional level set functions. In [3, 4, 5], piecewise constant level set functions were proposed. Both approaches lead to a system of nonlinear PDEs that needs to be solved numerically. They both have the drawback of expensive computation. This work aims to significantly reduce the computational cost of the piecewise constant level set method for the multiphase Mumford-Shah model. The length term is often simplified in the energy induced when representing this model by piecewise constant level set functions. We will show that this simplification makes it possible to efficiently compute global minimizers via graph cuts, when the mean image intensity value in each phase is known. Graph cuts is a wellknown technique in image analysis and computer vision [9, 10, 11, 12]. Usually, NP-hard multilabeling problems are approached by constructing algorithms for finding approximate suboptimal solutions, such as alpha expansion [12]. We instead do the approximation in the model, and then compute the exact solution of the approximate model. The graph used for optimization is constructed as in [13, 14, 15], except for some small modifications. Finally, for unknown mean intensity values, an iterative algorithm is presented, which we believe will have large practical value because of the strong efficiency. In case of two phases, some work on graph cut optimization for the MumfordShah model has been made in [16, 17]. Also a multiphase approach based on graph cuts has recently been presented in [18]. The process is started by splitting the image into two regions, by solving the two-phase Mumford-Shah model to optimality. In the next step, each new region is splitted in two by solving the twophase Mumford-Shah model within each region. The process is repeated until the intensity variation within each region falls below a predefined threshold. The limitation of this approach is that the possibility of a region to evolve has been ignored. For instance, the optimal interphase for two regions may not be a subset of the optimal interphases for three regions. An experiment in Section 4 will clarify this. The paper is organized as follows: Section 2 gives a brief overview of the piecewise constant level set method and the Mumford-Shah model. Section 3 presents the new integer optimization approach, while numerical experiments are presented in Section 4.

2 2.1

Image Segmentation and the PCLSM The Mumford-Shah Model

The Mumford-Shah model [6] is an established image segmentation model with a wide range of applications. Let u0 be the input image. In the most common variant with closed contours, one seeks a partition {Ωi }ni=1 of the image domain Ω, and an approximation image u which minimizes the functional  E(u, Γi ) = Ω

(u − u0 )2 dx + μ

 Ω\∪i Γi

|∇u|2 dx +

 n  ν i=1

Γi

ds,

(1)

Graph Cut Optimization for the Piecewise Constant Level Set Method

3

where {Γi }ni=1 denotes the interphases between the regions {Ωi }ni=1 . Often u is assumed to be constant within each phase, in which case the second term disappears and one ends up with the simpler version 

(u − u0 )2 dx +

E({ci }, Γi ) = Ω

 n  ν

ds,

(2)

Γi

i=1

n where u = i=1 ci Ψi , and Ψi is the characteristic function of Ωi . As a numerical realization, Chan and Vese [7,8] proposed to represent the above functional with level set functions, and solve the resulting gradient descent equations numerically. In order to represent n phases, log2 (n) level set functions were required. For any n > 2 the length term had to be simplified. 2.2

Piecewise Constant Level Set Functions

In [3, 4, 19], instead the piecewise constant level set method was proposed, and applied to the Mumford-Shah model. This approach has certain benefits, such as the ability to represent any number of phases with one single level set function. Let {Ωi }ni=1 be a partition of the domain Ω into n regions. Any such partition can be described by a piecewise constant level set function φ as follows φ=i

in Ωi

for i = 1, 2, ..., n.

(3)

Note that all interphases are represented by discontinuities in φ. The MumfordShah functional can now be written in terms of φ  E(c, φ) =

(u − u0 )2 dx +

Ω

n  ν |∇ψi |dx. 2 i=1 Ω

(4)

n where u = i=1 ci ψi , and ψi is the characteristic function of Ωi . It can be derived from the level set function by ψi =

1 αi





(φ − j) with αi =

j=1j =i

(i − k).

(5)

k=1k =i

The length term can be approximated by the total variation of the level set function itself, especially when the number of phases is not too large E(c, φ) =

n   i=1

0 2



(u − u ) dx + ν Ω

|∇φ|dx,

(6)

Ω

see for instance [5] for a justification. Most often this approximation is preferred, since it is computationally easier. Such a simplification of the length term has also been made in [20, 21] among others for multiphase image segmentation. In this work we will consider (6).

4

E. Bae and X.-C. Tai

There are some variants of the total variation regularization term. The com monly used version is the isotropic total variation: T V2 (φ) = Ω |∇φ|2 dx =   |φx1 |2 + |φx2 |2 dx. In computation, often a simpler verΩ  order to simplify  sion is used: T V1 (φ) = Ω |∇φ|1 dx = Ω |φx1 | + |φx2 | dx. However, since T V1 is not isotropic, regularization will be stronger in certain directions. A more isotropic version based on the 1-norm can be obtained by splitting T V1 using the original gradient counterclockwise π/4 radians:   operator, and one rotated T V1, π4 (φ) = 12 Ω |∇φ(x)|1 + |R π4 ∇φ(x)|1 dx, where R π4 ∇ is the gradient in the rotated coordinate system. It is also possible to create even more isotropic versions by considering more such rotations. Previous attempts to minimize (6) have been made by continuous optimization. In order to force a solution taking only integer values, the following constraint was imposed n  K(φ) = (φ − i) = 0 (7) i=1

The constrained optimization problem (6) and (7) could be solved by the augmented lagrangian method as in [3, 4, 19]. Some attempts to speed up the computation can be found in [5]. In the next section we propose to solve the minimization problem by graph cuts. We start by discretizing the variational problem (6) on a grid P of mesh size δ = 1. For each p = (i, j) ∈ P, define the neighborhood systems N4 (p) = {(i ± 1, j), (i, j ± 1)}, and N8 (p) = {(i ± 1, j), (i, j ± 1), (i ± 1, j ± 1)}. The modification of the definition for boundary points is clear. The discrete energy function can now be written compactly Ed (c, φ) =

 p∈P

δ 2 (up − u0p )2 + ν





p∈P q∈Nk (p)

1 wpq |φp − φq |, 2

(8)

where k = 4 for T V1 and k = 8 for T V1, π4 . The weights wpq are given by 2

4δ . In case of two phases, similar weights can also be derived by using wpq = k||p−q|| 2 the Cauchy-Crofton formula [22]. Note that each term is being counted twice in the last summation. This is compensated by multiplication by the factor 12 .

3

Integer Optimization for PCLSM and Mumford-Shah

Instead of imposing constraints to force an integer solution by continuous optimization, we instead propose the much more natural approach of using integer optimization to minimize (6). We will show that the discretized functional (8) can be minimized by graph cuts in case the values c are known in Section 3.2. Finally, in Section 3.3 an algorithm is designed to minimize with respect to both c and φ.

Graph Cut Optimization for the Piecewise Constant Level Set Method

3.1

5

Background on Graph Cuts and Terminology

Min-cut is a well known optimization problem. Due to a duality theorem by Ford and Fulkerson [23], there are several fast algorithms for this problem. Graph cuts is a reference to such algorithms, and was introduced as a computer vision tool by Greig et. al. [9] in connection with markov random fields [24]. A graph G = (V, E) is a set of vertices V and a set of directed edges E. We let (a, b) denote the directed edge going from vertex a to vertex b, and let c(a, b) denote the cost (weight) on this edge. In the graph cut scenario there are two distinguished vertices in V, called the source {s} and the sink {t}. A cut on G is a partitioning of the vertices V into two disjoint an connected (through edges) sets (Vs , Vt ) such that s ∈ Vs and t ∈ Vt . For each cut, the set of severed edges C is uniquely defined as C = {(a, b) | a ∈ Vs , b ∈ Vt and (a, b) ∈ E}.

(9)

We say that the cut severs the edge e if e is contained in C. From now on, we refer to the cut as the set of severed edges C. The cost of the cut is defined as  |C| = c(e). (10) e∈C

We are interested in finding the cut of minimum cost on G, from now on called the minimum cut. The duality theorem by Ford and Fulkerson [23] states this is equivalent to finding the maximum flow from {s} to {t}, where the edge weights are bounds on the maximum amount of flow that can be pushed through the edges. Cuts of minimum cost can thus be computed very efficiently by max-flow algorithms such as Ford-Fulkerson [23]. See [10] for a detailed discussion about implementation. 3.2

Graph Cuts for the Multiphase PCLSM

For fixed values c, we will show that the minimizer of (8) can be obtained by finding the minimum cut over an appropriate graph, i.e. we will construct a graph G such that min |C| = min Ed (c, φ) + σ, (11) C cut on G

φ

where σ is a constant that will be specified later. Note that the minimizer φ is not influenced by this constant. Some work on graph cuts for the two phase Mumford-Shah model can be found in [17,16]. Unfortunately, the extension to more than two phases is NP hard [25]. The usual graph cut approach to optimization problems of several labels, is to use some sort of approximation method, such as the alpha expansion [12]. Since we have already made an approximation in the model (6), we will show that graph cuts can be used to find the exact minimum. The idea is to introduce an extra dimension to take care of several phases. We construct the graph in a similar way as Ishikawa [13, 14], except for some small technical differences:

6

E. Bae and X.-C. Tai

(a)

(b)

Fig. 1. (a) The graph corresponding to a 1D signal of 6 grid points used for 4 phase segmentation. Edges in ED are depicted as vertical arrows and edges in ER are depicted as horizontal arrows. The gray curve is used to visualize the cut, vertices in the interior to the curve belongs to Vs , vertices in the exterior to the curve belongs to Vt . Edges in C are depicted as dotted arrows. Figure (b) shows the values of φ at each grid point corresponding to the cut in (a), they are determined from definition 1.

our graph consists of one less layer of vertices and edges, and is a generalization from the binary construction of Greig et. al. [9]. We also avoid edges of infinite capacity. When the number of phases is small, this will have a little effect on the efficiency. For each grid point p ∈ P, we associate (n−1) vertices in the graph G, denoted vp, , = 1, ..., n − 1. The set of vertices V is formally defined V = {vp, | p ∈ P, ∈ {1, ..., n − 1}} ∪ {s} ∪ {t}.

(12)

An illustration in case of a 1D image where P = {1, 2, ..., 6}, is shown in Figure 1. For ease of visualization, no 2D cases are shown. The edges are arranged in two groups, ED and ER . The first group ED corresponds to the data term in (8). It is defined as ED = ∪p∈P Ep ,

(13)

where for each p ∈ P the edge set Ep is defined as Ep = (s, vp,1 ) ∪n−2 =1 (vp, , vp,+1 ) ∪ (vp,n−1 , t).

(14)

The edges in ED are illustrated as the vertical arrows in Figure 1. The second group of edges ER corresponds to the regularization term in (8). These are illustrated as the horizontal arrows in Figure 1, i.e. ER = {(vp, , vq, ) | p ∈ P, q ∈ Nk (p), ∈ {1, ..., n − 1}}.

(15)

We say that a cut is admissible if it severs exactly one edge in Ep for each p ∈ P. We can now establish the relationship between a cut on G and a level set function φ.

Graph Cut Optimization for the Piecewise Constant Level Set Method

7

Definition 1. Let C ⊂ E be an admissible cut on G. For any grid point p ∈ P, the corresponding level set function φ is defined as ⎧ if (s, vp,1 ) ∈ C, ⎨1 φp = + 1 if (vp, , vp,+1 ) ∈ C, (16) ⎩ n if (vp,n−1 , t) ∈ C. Note that φ is single valued by the admissible cut requirement. We can now define the edge costs (weights) such that the relationship (11) is satisfied. We start by edges in ED , i.e. the data edges σ c (s, vp,1 ) = δ 2 |u0p − c1 |2 + |P| ∀p ∈ P, σ 2 0 2 c (vp, , vp,+1 ) = δ |up − c+1 | + |P| ∀p ∈ P, σ c (vp,n , t) = δ 2 |u0p − cn |2 + |P| ∀p ∈ P.

∀ ∈ {1, ..., n − 2},

(17)

The costs (weights) for the regularization edges ER are defined by c (vp, , vq, ) = νwpq , ∀p ∈ P, ∀q ∈ Nk (p), ∀ ∈ {1, ..., n − 1}.

(18)

By choosing σ as any positive value, the cut of minimum cost will be admissible, which implies that its corresponding level set function is single valued. Theorem 1. Let C be a minimum cut on G, then C is admissible if σ > 0 Proof. Suppose C is a minimum cut on G and for some p ∈ P several edges in Ep belongs to C. That is, there exists a set of indices Lp such that (vp, , vp,+1 ) ∈ C ∀ ∈ Lp . Define the cut C ∗ s.t. for each p ∈ P, vp, ∈ Vs∗ if ≤ max Lp , else vp, ∈ Vt∗ . Then C ∗ ∩ ED ⊂ C ∩ ED . Since σ > 0, no edges have zero weight, therefore |C ∗ ∩ ED | < |C ∩ ED |. Furthermore, |C ∗ ∩ ER |cardinality ≤ |C ∩ ER |cardinality . For T V1 , the weights on all edges in ER are equal. Therefore |C ∗ | < |C|, which is a contradiction. The same contradiction can also be derived for T V1, π4 . To summarize, for any piecewise constant level set function φ taking values in {1, 2, · · · n}, there exists a unique admissible cut on G. Moreover, the function φ and its corresponding cut satisfies |C| = Ed (c, φ).

(19)

Thus, a function φ corresponding to a minimum cut, is a minimizer of the functional (8), i.e. it solves the segmentation problem. Note that in case n = 2, the extra dimension breaks down, and the graph becomes identical to that of Greig et. al. [9] for binary problems. It is also possible to exactly minimize (8) as in [26], by solving a sequence of binary optimization problems via graph cuts. This approach is likely to be faster when n is very large, and is a power of 2. However, for image segmentation n is relatively small, and we expect the presented approach to be faster.

8

3.3

E. Bae and X.-C. Tai

Algorithm for Minimizing the Mumford-Shah Functional

The algorithm presented in the last section minimizes Ed (c, φ) with respect to φ for a fixed c. Vice versa, for a fixed φ the values c minimizing Ed (c, φ) are given by the average intensity in each region  0 u (x)ψi (x) dx i = 1, 2, ..., n, (20) ci = Ω  ψ (x) dx Ω i or in discrete form

 p∈P ci = 

u0p ψi,p

p∈P

ψi,p

i = 1, 2, ..., n.

(21)

We want an algorithm to minimize both with respect to φ and c. This is achieved by combining the two above results in the following iterative descent algorithm Algorithm 1. Estimate initial values c0 , set l = 0 while( ||cl − cl−1 || > tol ) 1. Use graph cuts to estimate φ from ˜ φ = arg minφ˜ Ed (cl , φ).

(22)

2. Update cl+1 according to equation (21). 3. Update l ← l + 1.

Note that no initialization of the level set function is required. Only the values c0 need to be initialized, which can be achieved very efficiently by the isodata algorithm [27]. Note that algorithm 1 has an exact termination criterion, as tol can be set to zero. In all our experiments, convergence was reached in 4-12 iterations. It must be noted that this algorithm is no longer guaranteed to find the global minima. Theoretically it may get trapped in a local minima close to the initial values c0 . However, in practice it is usually rather insensitive to initialization.

4

Numerical Experiments

In this section we validate our new optimization method by numerical experiments. The results are compared with the original gradient descent approach [3] for minimizing (4) (note: not the variant with simplified length term). The implementation of both these methods is made in C++. Comparisons are made both with respect to quality and computation time on an intel 2.19 GHz laptop. The list of computation times is shown in Table 1. The test images are shown in Figure 2. In all results, the estimated phases are depicted as a bright region. The results of experiment 1 and 2 are depicted in Figure 3 (a) - (d). We observe that graph cut

Graph Cut Optimization for the Piecewise Constant Level Set Method

9

Fig. 2. Test images Table 1. Computation times in seconds for gradient descent vs graph cut optimization

Experiment1 Experiment2 Experiment3 Brain

Size 100x100 100x100 92x98 933x736

Number of Phases Gradient descent Graph Cut 4 50.3 0.120 5 70.0 0.179 5 55.4 0.165 4 5401 25.22

optimization solve the multiphase problem with at least as good quality as gradient descent. In experiment 3 Figure 3(e)(f), the number of regions is assumed to be unknown. The optimal number of regions can be estimated by using more phases than necessary in the minimization problem. As we can see, this results in some empty phases, while the remaining phases capture the correct regions. We have also tested the method on a synthetic brain MRI image. The noise level is 7%, and non-uniformity of the RF-puls is of 20 % (see http://www.bic.mni.mcgill.ca/brainweb/ for details). We want to extract four tissue classes from the image: region 1; background, region 2; cerebrospinal fluid, region 3; gray matter and region 4; white matter. This is achieved by minimizing the Mumford-Shah model with 4 phases. In Figure 4 we compare the results of graph cuts, gradient descent and the exact results. The background phase is not shown. Again, we observe that graph cut results are very good, while the computation time is dramatically reduced compared to gradient descent(c.f. Table 1, brain). Finally, in Figure 5 we show an example which demonstrates the limitation of the multiphase approach presented in [18], described in the related work section. For the chosen parameter ν, the global minimum consists of three phases, which we are able to detect by applying our multiphase algorithm with 4 phases, Figure 5(b) top. The result of the first step of the algorithm presented in [18] is shown in Figure 5(b) buttom, which is the global minimum of the two phase Mumford-Shah functional. Clearly, no further splitting of these regions can result in the correct three phases, since the interphase from the first step is not allowed to evolve.

10

E. Bae and X.-C. Tai

(a) Experiment 1: graph cut

(b) Experiment 1: gradient descent

(c) Experiment 2: graph cut

(d) Experiment 2: gradient descent

(e) Experiment 3: graph cut

(f) Experiment 3: gradient descent Fig. 3. (a) and (b) Experiment 1, from left to right: phase 1 - phase 4. (c) and (d) Experiment 2, from left to right: phase 1 - phase 5. (e) and (f) Experiment 3, from left to right: phase 1 - phase 5.

Graph Cut Optimization for the Piecewise Constant Level Set Method

11

(a) graph cut

(b) gradient descent

(c) exact phases Fig. 4. From left to right: phase 1 - phase 3. (a) Graph cut, (b) gradient descent, (c) exact phases.

(a)

(b)

Fig. 5. (a) Input image. (b) Top: Our approach, from left to right phase 1 - phase 4. Buttom: First step of approach reported in [18], from left to right phase 1 - phase 2.

5

Summary

We have presented an algorithm for efficiently minimizing the energy induced by the piecewise constant level set representation of the multiphase MumfordShah functional. This minimization method is based on graph cuts. Numerical

12

E. Bae and X.-C. Tai

experiments demonstrated the method is very superior in efficiency compared to the previous PDE based approach, while maintaining the same quality of results.

References 1. Dervieux, A., Thomasset, F.: A finite element method for the simulation of a Rayleigh-Taylor instability. In: Approximation methods for Navier-Stokes problems, Proc. Sympos., Univ. Paderborn, Paderborn, 1979. Lecture Notes in Math., vol. 771, pp. 145–158. Springer, Berlin (1980) 2. Osher, S., Sethian, J.: Fronts propagating with curvature dependent speed: algorithms based on hamilton-jacobi formulations. J. Comput. Phys. 79(1), 12–49 (1988) 3. Lie, J., Lysaker, M., Tai, X.: A variant of the level set method and applications to image segmentation. Math. Comp. 75(255), 1155–1174 (2006) (electronic) 4. Lie, J., Lysaker, M., Tai, X.: A binary level set model and some applications to mumford-shah image segmentation. IEEE Transactions on Image Processing 15(5), 1171–1181 (2006) 5. Tai, X., Christiansen, O., Lin, P., Skjaelaaen, I.: Image segmentation using some piecewise constant level set methods with mbo type of project. International Journal of Computer Vision 73, 61–76 (2007) 6. Mumford, D., Shah, J.: Optimal approximation by piecewise smooth functions and associated variational problems. Comm. Pure Appl. Math. 42, 577–685 (1989) 7. Chan, T., Vese, L.: Active contours without edges. IEEE Image Proc. 10, 266–277 (2001) 8. Vese, L.A., Chan, T.F.: A new multiphase level set framework for image segmentation via the mumford and shah model. International Journal of Computer Vision 50, 271–293 (2002) 9. Greig, D.M., Porteous, B.T., Seheult, A.H.: Exact maximum a posteriori estimation for binary images. Journal of the Royal Statistical Society, Series B, 271–279 (1989) 10. Boykov, Y., Kolmogorov, V.: An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision. In: Figueiredo, M., Zerubia, J., Jain, A.K. (eds.) EMMCVPR 2001. LNCS, vol. 2134, pp. 359–374. Springer, Heidelberg (2001) 11. Kolmogorov, V., Zabih, R.: What energy functions can be minimized via graph cuts? IEEE Transactions on Pattern Analysis and Machine Intelligence 26(2), 147– 159 (2004) 12. Boykov, Y., Veksler, O., Zabih, R.: Fast approximate energy minimization via graph cuts. In: ICCV, vol. (1), pp. 377–384 (1999) 13. Ishikawa, H.: Exact optimization for markov random fields with convex priors. IEEE Transactions on Pattern Analysis and Machine Intelligence 25(10), 1333– 1336 (2003) 14. Ishikawa, H., Geiger, D.: Segmentation by grouping junctions. In: CVPR 1998: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Washington, DC, USA, pp. 125–131. IEEE Computer Society, Los Alamitos (1998) 15. Darbon, J., Sigelle, M.: Image restoration with discrete constrained total variation part ii: Levelable functions, convex priors and non-convex cases. J. Math. Imaging Vis. 26(3), 277–291 (2006)

Graph Cut Optimization for the Piecewise Constant Level Set Method

13

16. Darbon, J.: A note on the discrete binary mumford-shah model. In: Gagalowicz, A., Philips, W. (eds.) MIRAGE 2007. LNCS, vol. 4418, pp. 283–294. Springer, Heidelberg (2007) 17. Zehiry, N.E., Xu, S., Sahoo, P., Elmaghraby, A.: Graph cut optimization for the mumford-shah model. In: Proceedings of the Seventh IASTED International Conference visualization, imaging and image processing, pp. 182–187. Springer, Heidelberg (2007) 18. El-Zehiry, N.Y., Elmaghraby, A.: A graph cut based active contour for multiphase image segmentation. In: IEEE International Conference on Image Processing, pp. 3188–3191 (2008) 19. Lie, J., Lysaker, M., Tai, X.: Piecewise constant level set methods and image segmentation. In: Kimmel, R., Sochen, N.A., Weickert, J. (eds.) Scale-Space 2005. LNCS, vol. 3459, pp. 573–584. Springer, Heidelberg (2005) 20. Chung, G., Vese, L.A.: Energy minimization based segmentation and denoising using a multilayer level set approach. In: Rangarajan, A., Vemuri, B.C., Yuille, A.L. (eds.) EMMCVPR 2005. LNCS, vol. 3757, pp. 439–455. Springer, Heidelberg (2005) 21. Jung, Y.M., Kang, S.H., Shen, J.: Multiphase image segmentation via modicamortola phase transition. SIAM J. Appl. Math. 67, 1213–1232 (2007) 22. Boykov, Y., Kolmogorov, V.: Computing geodesics and minimal surfaces via graph cuts. In: ICCV 2003: Proceedings of the Ninth IEEE International Conference on Computer Vision, Washington, DC, USA, pp. 26–33. IEEE Computer Society, Los Alamitos (2003) 23. Ford, L., Fulkerson, D.: Flows in networks. Princeton University Press, Princeton (1962) 24. Geman, S., Geman, D.: Stochastic relaxation, gibbs distributions, and the bayesian restoration of images. In: Readings in uncertain reasoning, pp. 452–472. Morgan Kaufmann Publishers Inc., San Francisco (1990) 25. Dahlhaus, E., Johnson, D.S., Papadimitriou, C.H., Seymour, P.D., Yannakakis, M.: The complexity of multiway cuts (extended abstract). In: STOC 1992: Proceedings of the twenty-fourth annual ACM symposium on Theory of computing, pp. 241– 251. ACM, New York (1992) 26. Darbon, J., Sigelle, M.: Image restoration with discrete constrained total variation part i: Fast and exact optimization. J. Math. Imaging Vis. 26(3), 261–276 (2006) 27. Velasco, F.R.D.: Thresholding using the ISODATA clustering algorithm. IEEE Trans. Systems Man Cybernet. 10(11), 771–774 (1980)

Tubular Anisotropy Segmentation Fethallah Benmansour and Laurent D. Cohen CEREMADE, UMR CNRS 7534, Université Paris Dauphine, Place du Maréchal De Lattre De Tassigny, 75775 PARIS CEDEX 16, France {benmansour,cohen}@ceremade.dauphine.fr

Abstract. In this paper we present a new interactive method for tubular structure extraction. The main application and motivation for this work is vessel tracking in 2D and 3D images. The basic tools are minimal paths solved using the fast marching algorithm. This allows interactive tools for the physician by clicking on a small number of points in order to obtain a minimal path between two points or a set of paths in the case of a tree structure. Our method is based on a variant of the minimal path method that models the vessel as a centerline and surface. This is done by adding one dimension for the local radius around the centerline. The crucial step of our method is the definition of the local metrics to minimize. We have chosen to exploit the tubular structure of the vessels one wants to extract to built an anisotropic metric giving higher speed on the center of the vessels and also when the minimal path tangent is coherent with the vessel’s direction. This measure is required to be robust against the disturbance introduced by noise or adjacent structures with intensity similar to the target vessel. We obtain promising results on noisy synthetic and real 2D and 3D images.

1 Introduction In this paper we deal with the problem of finding a complete segmentation of tubular structures like vessels. The main objective is to extract at the same time the centerline of the tubular structure and its boundary. During the last two decades, the extraction of vascular objects such as the blood vessel, coronary arteries, or other tube-like structures has attracted the attention of more and more researchers. Various methods such as vascular image enhancement methods [1, 2, 3], or others were proposed, see [4] for a complete survey. Some of these methods extract the vessel boundary directly, and then use thinning methods to find its centerline. Other methods extract only the centerline and then estimate the vessel width to extract its boundary. Deschamps and Cohen [5] proposed to use the minimal path method to find the centerline. The minimal path technique introduced by Cohen and Kimmel [6] captures the global minimum curve between two points given by the user. This leads to the global minimum of an active contour energy. Since then, the minimal path method has been improved by many researchers, and adapted to anisotropic media as done by Jbabdi et al for tractography [7]. Unfortunately, despite their numerous advantages, classical minimal path techniques exhibit some disadvantages. First, vessel boundary extraction can be very difficult, even in 2D X.-C. Tai et al. (Eds.): SSVM 2009, LNCS 5567, pp. 14–25, 2009. c Springer-Verlag Berlin Heidelberg 2009 

Tubular Anisotropy Segmentation

15

where the vessel’s boundary can be completely described by two curves. Second, the path given by the minimal path technique does not always yield to the centerline of the vessel. A readjustment step is required to obtain a central trajectory. Third, the minimal path technique provides only a trajectory and does not give information about the vessel boundary and local width. Li and Yezzi [8] proposed a new variant of the classical, purely spatial, minimal path technique by incorporating an extra non-spatial dimension into the search space. Each point of the 4D path (after adding the extra dimension for the 3D image) consists of three spatial coordinates plus a fourth coordinate which describes the vessel thickness at that corresponding 3D point. Thus, each 4D point represents a sphere in 3D space, and the vessel is obtained by taking the envelope of these spheres as we move along the 4D curve. A crucial step of this method is to build an adequate potential that drives the propagation. Li and Yezzi [8] proposed different isotropic potentials. As they said in the conclusion of their paper, the proposed potentials are very parameter dependent and they hoped to find more appropriate choice of potential. In particular, one can see in their paper, that the potential used does not yield to a correct detection of the radius when it is not constant (see figure 6 in [8]). An other drawback of Li and Yezzi method is that they did not take into account the vessel orientation. Our first contribution is to take into account the vessel orientation by defining a suited anisotropic metric that makes the propagation faster along the centerlines and for the adequate radius. Law et al. [9] proposed a new scalar descriptor called Optimally Oriented Flux (OOF) for the detection of curvilinear structures. But they did not exploit the orientation given by their descriptor. The major advantage of the OOF technique is that it does not consider the regions in the vicinity of target objects, where background noise or adjacent structures with intensity similar to the target vessels are possibly present. Therefore, the disturbance introduced by the closely located nearby structures is avoided. The second contribution of this paper is to build an anisotropic metric based on the OOF descriptor, its scalar function as well as its orientation. That makes the propagation faster along the vesselÕs center line and for exact associated scale. This means that the path location, orientation and scale (radius) have to be coherent with the local geometry of the image extracted by the OOF. In section 2, we give some background on minimal path method and Anisotro-pic Fast Marching. In section 3 the Optimally Oriented Flux descriptor is presented as well as the metric construction. In section 4, results on synthetic and real data are shown. Finally, conclusions and perspectives follow in section 5.

2 Background on Minimal Path Method A minimal path, first introduced in the isotropic (P does not depend on the orientation of the path) case [6], is a pathway minimizing the energy functional,    E(γ) = P γ(s), γ  (s) ds (1) 

γ

γ  (.)T M(γ(.))γ  (.) describes an infinitesimal distance where, P(γ(.), γ  (.)) = along a pathway γ relative to a metric tensor M (symmetric definite positive). Thus,

16

F. Benmansour and L.D. Cohen

we are considering only the case of an elliptic medium. In the isotropic case M(.) = P 2 (.)I, where I is the identity matrix. A curve connecting p1 to p2 that globally minimizes the above energy (1) is a minimal path between p1 and p2 , noted Cp1 ,p2 . The solution of this minimization problem is obtained through the computation of the minimal action map U : Ω → R+ associated to p1 on the domain Ω which can be a 2D, 3D or 4D domain. The minimal action is the minimal energy integrated along a path between p1 and any point x of the domain Ω :      ∀ x ∈ Ω, U(x) = min P γ(s), γ (s) ds , (2) γ∈Ap1 ,x

γ

where Ap1 ,x is the set of paths linking x to p1 . The values of U may be regarded as the arrival times of a front propagating from the source p1 with oriented velocity related to the metric tensor M−1 . U satisfies the Eikonal equation ∇U(x)M−1 (x) = 1 for x ∈ Ω, and U(p1 ) = 0,

(3)

√ where vM = vT M v. The map U has only one local minimum, the point p1 , and its flow lines satisfy the Euler-Lagrange equation of functional (1). Thus, the minimal path Cp1 ,p2 can be retrieved with a simple gradient descent on U from p2 to p1 (see Fig. 1), solving the following ordinary differential equation with standard numerical methods like Heun’s or Runge-Kutta’s :   dCp1 ,p2 (s) ∝ −M−1 (Cp1 ,p2 (s))∇U Cp1 ,p2 (s) , with Cp1 ,p2 (0) = p2 . ds

(4)

Proof of (3) and (4) can be found in [10,7]. On figure 1, we show some examples of the minimal path method on an isotropic case and an anisotropic one. On the first image of figure 1 the metric is isotropic and the potential P in the grey region is twice as low as the white one. Isolevel sets of the minimal action map associated to the source point p1 are displayed and so the minimal path Cp1 ,p2 . The second image represents a metric M. We took two constant metrics in each half side of the image with different orientations. On the last image, the minimal action map U associated to the metric M and to the source point p1 is shown. The minimal path Cp1 ,p2 is found by solving equation (4).

p1 p1

p2

p2 0

50

100

150

200

250

300

350

Fig. 1. Minimal path examples on an isotropic case on the left image. On the middle, visualization by small ellipses of eigenvalues of a metric constant on each half side of the image. On the right, the minimal action map associated to the source point p1 with the minimal path Cp1 ,p2 .

Tubular Anisotropy Segmentation

17

The Fast Marching Method (FMM) is a numerical method introduced by Sethian in [11] and Tsitsiklis in [12] for efficiently solving the isotropic Eikonal equation on a cartesian grid. The central idea behind the FMM is to visit grid points in an order consistent with the way wavefronts of constant action propagate. It leads to a single-pass algorithm for solving equation (3) and computing the minimal action map U. Tsitsiklis’s method relies on minimizing directly the energy functional of equation (1) while Sethian’s method uses the Eikonal equation. Both methods are suitable for isotropic metric, but they fail for anisotropic metric [13]. To deal with anisotropy, Sethian and Vladimirsky [10] proposed an update scheme that converges to the viscosity solution of the anisotropic Eikonal equation. A simplified scheme, based on the original Tsitsiklis’s method [12], was proposed by Lin in [14] to approximate the solution of the anisotropic Eikonal equation. Contrary to Sethian and Vladimirsky’s ordered upwind method (OUM) [10], Lin’s algorithm does not converge to the viscosity solution of the Eikonal equation. In this paper we used Lin’s scheme to solve the anisotropic Eikonal equation, since it is much faster (than OUM) and the introduced errors do not affect much the extracted geodesics. The FMM is a front propagation approach that computes the values of U in increasing order, and the structure of the algorithm is almost identical to Dijkstra’s algorithm for computing shortest paths on graphs [15]. In the course of the algorithm, each grid point is tagged as either Alive (point for which U has been computed and frozen), Trial (point for which U has been estimated but not frozen) or Far (point for which U is unknown). The set of Trial points forms an interface between the set of grid points for which U has been frozen (the Alive points) and the set of other grid points (the Far points). This interface may be regarded as a front expanding from the source until every grid point has been reached. Let us denote by NM (x) the set of M neighbors of a grid point x, where M = 2 × d if the dimension of Ω is equal to d. Initially, all grid points are tagged as Far, except the source point p1 that is tagged as Trial. At each iteration of the FMM one chooses the Trial point with the smallest U value, denoted by xmin . Then, xmin is tagged as Alive and the value of U is updated for each point of the set NM (xmin ) which is either Trial or Far. In order to satisfy a causality condition, the way U is updated in the vicinity of xmin requires special care. The iteration ends by tagging every Far point of the set NM (xmin ) as Trial. The algorithm automatically stops when all grid points are Alive. The key to the speed of the FMM is the use of a priority queue to quickly find the Trial point with the smallest U value. If Trial points are ordered in a min-heap data structure, the computational complexity of the FMM is O(N logN ), where N is the total number of grid points. A crucial step of the Fast Marching algorithm is the computation of the weighted distance between the front and the neighbouring voxels in the Trial set. Here, we present a way to estimate this weighted distance in the anisotropic case and only in 3D. It is straightforward to extend it to 4D. Since the distance is anisotropic, we cannot use the standard methods, because they rely on the fact that the geodesics are perpendicular to the level sets of U. To take into account the anisotropy Jbabdi et al [7] and Lin [14] considered a set of simplexes that cover the whole neighbourhood around a voxel of the narrow band. The definition of a simplex neighbouring a point x is simply a set of three points (x1 , x2 , x3 ) that are among the 26 neighbours of x, defining a triangle that we

18

F. Benmansour and L.D. Cohen x3 xm x

x2

x1

Fig. 2. On the left Position of the optimal point on a simplex such as to minimize the geodesic distance to x. On the right the considered simplexes.

denote x1 x2 x3 . There are 48 such triangles around x for the 26 connexity. To make the update procedure faster, we propose to consider only the simplexes defined by a t-uple of three points of the 6-neighbors of x. There are 8 such triangles (see Fig. 2), and by making this modification, the precision of the algorithm is lower but the algorithm is six times faster. To estimate U(xm ), where xm is a neighbor of the last trial point xmin , we make two approximations. If the geodesic passing by xm comes from a triangle x1 x2 x3 then the time of arrival is given by:  xm  U(x) + U(xm ) = min P (γ, γ  ) (5) x∈x1 x2 x3

x

The term one wants to minimize is approximated by : f (α) =

3

i=1

3  

  αi U(xi ) + x − αi xi  i=1

M(x)

,

(6)

3 where α = (α1 , α2 , α3 ), with i=1 αi = 1 since the point x is in the triangle (see figure 2). This equation follows Tsitsiklis’s approximation [12]. The first term approximates the value of the minimal action map at the point x = 3i=1 αi xi by a simple linear interpolation. And the second term approximates the remaining distance by considering the metric constant along the segment [x, xm ] equal to its value at point xm . 3 The function f is convex and the constraints on α, i.e i=1 αi = 1 and αi ≥ 0, define a convex subset. Thus the minimization of f can be done using classical optimization tools. See [7] for more details. For each of the eight triangles, we get a value u. Finally, we choose the triangle giving the smallest value of u. Note that in order to approximate ∇U, computing the derivatives of U in the triangle using the estimate U(xn ) gives a consistent approximation of ∇U(xn ) by the following: ∇U(xn ) = (U(xn ) − U(x ))

xn − x , xn − x 

where x is the minimizer of function f , see figure 2 left, and . is the Euclidean norm. The computation of the gradient is very useful since it is used to solve the gradient descent described by equation (4).

Tubular Anisotropy Segmentation

19

3 Optimally Oriented Flux: An Anisotropy Descriptor We are interested in the construction of a metric that extracts from the image the geometric information leading to reconstruction of vessels. This means that we wish to find an estimate for the local orientation and scale and a criterion on the local geometry to distinguish the presence of vessels from the background. At the position x on an image I, the amount of the image gradient projected along the axis v flowing out from a 3D sphere (or a 2D circle) Sr is measured as in [9], 

 h da, (7) (∇(G ∗ I(x + h)) · v)v · f (x, v; r) = |h| ∂Sr where G is a Gaussian function with a scale factor of 1 pixel, r is the sphere (or circle) radius, h is the position vector along ∂Sr and da is the infinitesimal area (or length) on ∂Sr . To detect vessels having higher intensity than the background region, one would be interested in finding the vessel direction which minimizes f (x, v; r), i.e. we are looking for: arg min f (x, v; r). Using the divergence theorem, it can be shown that f (x, v; r) v can be calculated using a simple convolution, f (x, v; r) = vT {(∂i,j G) ∗ I ∗ 1Sr } v,

(8)

where (∂i,j G) is the Hessian matrix of function G and 1Sr is the indicator function inside the sphere (or circle) Sr . By differentiating the above equation with respect to v, minimization of function f is in turn acquired as solving a generalized eigenvalue decomposition problem. Solving the aforementioned generalized eigen decomposition problem gives d eigenvalues (where d = 2 or 3 is the dimension of the image), λ1 (·) ≤ · · · ≤ λd (·) and d eigenvectors vi (·), i.e. λi (x; r) = f (x, vi (x; r); r) for i = 1, . . . , d. To handle the vessels having various radii, a multi-scale approach should be used along with the OOF method. In [9], Law and Chung have proposed to normalize the OOF’s eigenvalues by the sphere surface area when the OOF method is incorporated in a multiscale approach for 3D image volumes. In the 2D case the eigenvalues are normalized by the circle perimeter 2πr. In the 3D case the eigenvalues are normalized by the sphere area 4πr2 . In the 2D case (see figure 3), for a point on the centerline and if r is equal to radius of the vessel, the first eigenvector v1 represents the direction orthogonal to the vessel. v2 reprensents the direction along the vessel. In the 3D case, if the point is on the centerline, the two eigenvectors associated to the first eigenvalues (λ1 , λ2 ) represent the directions orthogonal the vessel. v3 represents the direction along the vessel, see figure 4. On the same figure, one can see that if the point x is on the centerline, the minimal response of the function f is obtained when the radius r is equal to the exact radius of the tube. If the point is inside the tube but not on the centerline, v3 is parallel to the tube orientation, and the other eigenvectors depends on the scale r. If the point is outside the tube (last line), then the vector v3 , corresponding to the red area, is oriented toward the centerline. Li and Yezzi [8] proposed a new variant of the classical, purely spatial, minimal path technique by incorporating an extra non-spatial dimension into the search space. The crucial step of this method is to build an adequate metric that drives the propagation.

20

F. Benmansour and L.D. Cohen

Fig. 3. The plots of the values of f (x, v; r) obtained from the synthetic image shown in the left, at four different positions with various radii and projection axes. (a) Four interested positions, denoted as x1 , x2 , x3 and x4 are shown along with the original synthetic image. (b) An illustration regarding the polar coordinate system used in (c)-(f). (c)-(f) The plots of the values of f (·) and the corresponding eigenvectors, computed at the four different positions shown in (a), using various values of r and different projection axes (cos θ sin θ)T .

Fig. 4. Plot of f (x, v; r) superimposed on the original 3D synthetic image for three different points(on each line) and different values of the radius : r = 3, . . . , 7 from left to right. The radius of the tube on the top half side image is equal to 4, and equal to 6 on the bottom half side. Similarly to figure 3, the visualization of the normalized flux function is done using a spherical coordinate system (instead of the polar one used in 2D). The first point is on the centerline of the tube. The second point is inside the tube but not on the centerline. The third point is outside the tube. The reader should zoom on each image. Notice that the colormaps are different.

Li and Yezzi [8] proposed different isotropic potentials. The main drawback, as they mention, is that these potentials are very parameter dependent and they do not exploit the vessel orientation. Our main contribution is to improve Li and Yezzi method by adding to it an anisotropic formulation, and the anisotropic metric is constructed by extension of the OOF descriptor presented by Law et al. [9].

Tubular Anisotropy Segmentation

21

Fig. 5. The constructed metric for different scales r = 1, 5, 10, 15, 20 from left to right. The original image is shown in figure 3 (a), the radius of the structure is equal to 10. We used the same color range for all images, so one can see that the optimal anisotropy is obtained along the centerline of the tubular structure when the scale r is equal to the exact radius of the tube. On ˜ the top, we show a display of M(x, r)−1 . On the bottom, responses of Pradii are shown.

The (d + 1)D minimal path is found by minimizing the following energy:   γ  (s)T M(γ(s))γ  (s) ds, where M is the (d + 1)D anisotropic metric we want γ to construct. It is not natural to consider orientations on the (d + 1) dimension, i.e the radii dimension. Thus one candecompose by block the metric M as follows:  ˜ M(x, r) 0 ˜ where M(x, r) is a d × d symmetric definite M(x, r) = 0 Pradii (x, r) positive matrix giving the spatial anisotropy and Pradii (x, r) is the radii potential (also strictly positive). Since the result given by the anisotropic minimal path method is very dependent on the metric, results inherit advantages and drawbacks of the constructed metric, thus we should be very carful with its construction. First, let us fix conditions on the desired ˜ has to be well oriented along the vessel centerline. And metric. The spatial metric M the radii potential Pradii has to be small for the adequate scale for any point of the image. ˜ is symmetric Pradii corresponds to the inverse speed for the radii dimension. Since M d T ˜ m definite positive, we can decompose it as follows: M(.) = i (.)ui (.)ui (.) , i=1 where 0 < m1 ≤ · · · ≤ md are the eigenvalues and ui are the associated eigenvectors. √ The velocity of the propagating front along direction ui is equal to 1/ mi . We used the OOF descriptor to construct the metric as follows:     d λj (.) λ (.) i j = i T i=1 ˜ M(.) = vi (.)vi (.) , Pradii (.) = β exp α . exp α d−1 d i=1 (9) The constant α is controlled by  an intuitive parameter, which is the maximal  exp(αλ2 (x, r)) in the 2D case and μ = spatial anisotropy ratio: μ = max x,r exp(αλ1 (x, r)) d

22

F. Benmansour and L.D. Cohen

⎫

⎧ 3 (x,r) ⎨ exp α λ2 (x,r))+λ ⎬ 2  in the 3D case. By choosing the maximal spatial

max x,r ⎩ 2 (x,r) ⎭ exp α λ1 (x,r))+λ 2 anisotropy ratio μ, the constant α is fixed. And by doing so, the anisotropy descriptor M becomes contrast invariant because the OOF is linear on the image. The parameter β controls the radii speed. In 2D (it is very similar in 3D), if Pradii ≤ exp(αλ1 ) then the Fast Marching propagation is faster for the radii than the spatial dimensions. If Pradii ≥ exp(αλ2 ) then the propagation is slower. One can tune parameter β depending on the tubular structure one wants to extract. If its radius changes a lot then β should be chosen such that the propagation on the radii dimension is faster. If not β is chosen such that the propagation is less sensitive on the radii dimension. On figure 5 the constructed metric of image 3 at some different scales is shown. Since we chose the same color range for the visualization, we can see that the directions are well detected, and that the optimal values are obtained along the centerline of the tube

Fig. 6. The red cross points are source points given by the user, and the blue ones are end points. On each case the segmented centerlines are displayed as well as the envelope of the moving discs. In the middle, the associated minimal action map U as well as the 3D minimal path between the two selected points are shown (transparent visualization).

Tubular Anisotropy Segmentation

23

when the scale is equal to the  tube radius. For our experiments, we took μ = 10 and 1) β such that max exp(αλ = 5, this means that in the worst case, the speed along Pradii the radii dimension is 5 times faster than the spatial dimensions. We did so, because we wanted our algorithm to be sensitive to the radii dimension.

4 Experimental Results Our method is minimally interactive. First, the user has to precise if the desired vessels are darker or brighter than the background. So, we can consider different criteria on the signs of the eigenvalues. Then the scale range [rmin , rmax ], which corresponds to the range of radii of the vessel one wants to extract, is given by the user. Finally few points are required as source points or end points of the Fast Marching algorithm. We used the metric described in the previous section to find the minimal anisotropic path (as described in section 2) between two or more selected points (see figures 6 and 7). For any selected point, the associated radius is equal to the minimal radius rmin given by the user. On figure 6, segmentation results on synthetic and real noisy 2D images are shown. On the first synthetic image, the source point and destination are selected on the centerline. The obtained tube is perfectly detected as well as the centerline. On the second

Fig. 7. First line : RCA segmentation using the tubular anisotropy approach shown on the whole image and on the selected sub-volume. Second line : LAD segmentation shown on the whole image and on the selected sub-volume. Only few points are required (the extremities of the paths). The tubular anisotropy method provides the centerline as well as vessels boundaries.

24

F. Benmansour and L.D. Cohen

image, the initial points are not centered. But the centerline given by our algorithm goes back fast to the real centerline. This makes our algorithm robust to initialization. The third synthetic image shows that our approach is robust to scale changing. On the last line of figure 6, segmentation results are shown on real noisy images. In figure 7, segmentation results are shown on real medical images. First, right coronary arteries (RCA) are segmented. Second, left anterior descending (LAD) arteries are segmented. One can see that the obtained radii on the principal coronary branches are larger than those of the secondary. Thus, our approach is robust to scale changing and bifurcations. Nevertheless, our current implementation requires huge memory allocations due to the 4D and anisotropic aspects. To overcome this issue, we added a preprocessing interactive tool to select a sub-volume containing the desired vessels (see figure 7). Moreover, we are working on a new implementation of the tubular anisotropy approach to make the memory allocation dynamic and hence to benefit from the front propagation aspect of the fast marching algorithm. Besides the reduction of the computation time (which has been actually achieved), we will save on memory allocation and will have a new version of our algorithm that extract the whole coronary arteries using a regular PC.

5 Conclusion In this paper we have proposed a new general method for tubular structure extraction in 2D and 3D images. Our method exploit the orientation of the vessels by using the optimally oriented flux to construct a multi-resolution anisotropic metric that extracts from the image the local geometry and describes the vessels orientation and scales. Combining this metric with anisotropic minimal path technique, we were able to find a complete description of the tubular structure, i.e the centerline as well as the boundary. To summarize, our method is minimally interactive, robust to initialization, scale variations and bifurcations.

Acknowledgements We would like to thank Professor Anthony J. Yezzi and Max Wai-Kong Law for interesting discussions. Also Eduardo Davila for his precious help for the implementation of the interface. This work was partially supported by ANR grant SURF -NT05-2_45825.

References 1. Sato, Y., Nakajima, S., Shiraga, N., Atsumi, H., Yoshida, S., Koller, T., Gerig, G., Kikinis, R.: Three-dimensional multi-scale line filter for segmentation and visualization of curvilinear structures in medical images. Med. Image Anal. 2(2), 143–168 (1998) 2. Krissian, K.: Flux-based anisotropic diffusion applied to enhancement of 3D angiogram. TMI 21(11), 1440–1442 (2002) 3. Frangi, A., Niessen, W.J., Vincken, K.L., Viergever, M.A.: Multiscale vessel enhancement filtering. In: Wells, W.M., Colchester, A.C.F., Delp, S.L. (eds.) MICCAI 1998. LNCS, vol. 1496, pp. 130–137. Springer, Heidelberg (1998)

Tubular Anisotropy Segmentation

25

4. Kirbas, C., Quek, F.K.H.: A review of vessel extraction techniques and algorithms. ACM Computing Surveys 36, 81–121 (2004) 5. Deschamps, T., Cohen, L.: Fast extraction of minimal paths in 3D images and applications to virtual endoscopy. MIA 5(4) (December 2001) 6. Cohen, L.D., Kimmel, R.: Global minimum for active contour models: a minimal path approach. International Journal of Computer Vision 24, 57–78 (1997) 7. Jbabdi, S., Bellec, P., Toro, R., Daunizeau, J., Pélégrini-Issac, M., Benali, H.: Accurate anisotropic fast marching for diffusion-based geodesic tractography. Journal of Biomedical Imaging 2008(1), 1–12 (2008) 8. Li, H., Yezzi, A.: Vessels as 4D curves: Global minimal 4D paths to extract 3D tubular surfaces and centerlines. IEEE Transactions on Medical Imaging 26(9), 1213–1223 (2007) 9. Law, M.W.K., Chung, A.C.S.: Three dimensional curvilinear structure detection using optimally oriented flux. In: ECCV, vol. 4, pp. 368–382 (2008) 10. Sethian, J.A., Vladimirsky, A.: Fast methods for the eikonal and related hamilton- jacobi equations on unstructured meshes. Proceedings of the National Academy of Sciences 97(11), 5699–5703 (2000) 11. Sethian, J.A.: A fast marching level set for monotonically advancing fronts. Proceedings of the National Academy of Sciences 93, 1591–1595 (1996) 12. Tsitsiklis, J.N.: Efficient algorithms for globally optimal trajectories. IEEE Transactions on Automatic Control 40, 1528–1538 (1995) 13. Chopp, D.L.: Replacing iterative algorithms with single-pass algorithms. Proc. Nat. Acad. Sc. USA 98(20), 10992–10993 (2001) 14. Lin, Q.: Enhancement, extraction, and visualization of 3D volume data. PhD thesis, Linkopings Universitet (2003) 15. Dijkstra, E.W.: A note on two problems in connection with graphs. Numerische Mathematic 1, 269–271 (1959)

An Unconstrained Multiphase Thresholding Approach for Image Segmentation Benjamin Berkels Institut für Numerische Simulation, Rheinische Friedrich-Wilhelms-Universität Bonn, Nussallee 15, 53115 Bonn, Germany [email protected] http://numod.ins.uni-bonn.de/ Abstract. In this paper we provide a method to find global minimizers of certain non-convex 2-phase image segmentation problems. This is achieved by formulating a convex minimization problem whose minimizers are also minimizers of the initial non-convex segmentation problem, similar to the approach proposed by Nikolova, Esedo¯ glu and Chan. The key difference to the latter model is that the new model does not involve any constraint in the convex formulation that needs to be respected when minimizing the convex functional, neither explicitly nor by an artificial penalty term. This approach is related to recent results by Chambolle. Eliminating the constraint considerably simplifies the computational difficulties, and even a straightforward gradient descent scheme leads to a reliable computation of the global minimizer. Furthermore, the model is extended to multiphase segmentation along the lines of Vese and Chan. Numerical results of the model applied to the classical piecewise constant Mumford-Shah functional for two, four and eight phase segmentation are shown.

1

Introduction

Image segmentation is one of the fundamental research topics in the field of image processing. In particular, the Mumford-Shah model [1] is widely used in this context. One of the difficulties of this and many other variational image processing models is that the underlying energy functional has local, non-global minima. This is not only a theoretical problem, since the commonly used numerical minimization techniques often get stuck in local minima that differ considerably from a global minimum, hence possibly producing useless results. The goal of this paper is to introduce a method to obtain a global minimizer of the Mumford-Shah functional for 2-phase segmentation that only involves solving an unconstrained convex minimization problem. This method can be extended to multiphase segmentation by the ideas of Vese and Chan [2] in a canonical way. 1.1

Related Work

The problem of minimizing the Mumford-Shah segmentation functional has been extensively studied in the last decade leading to a wide range of existing methods, X.-C. Tai et al. (Eds.): SSVM 2009, LNCS 5567, pp. 26–37, 2009. c Springer-Verlag Berlin Heidelberg 2009 

Image Segmentation by Unconstrained Multiphase Thresholding

27

each with its own shortcomings. One of the first numerical feasible methods to obtain (local) minimizers of the functional was proposed by Chan and Vese [3]. They build on the levelset methods of Osher and Sethian [4] and parameterize the unknown set by a levelset function. Shen [5] developed a Γ -convergence formulation along with a simple implementation by the iterated integration of a linear Poisson equation. The unknown set is represented in a diffuse way by a phase field. In [6], Esedo¯glu and Tsai tackle the minimization problem based on the threshold dynamics of Merriman, Bence and Osher [7] for evolving an interface by its mean curvature. Here the minimization is achieved by alternating the solution of a linear parabolic partial differential equation and simple thresholding. Alvino and Yezzi [8] approximate Mumford-Shah segmentation using reduced image bases. According to them, the majority of the robustness of Mumford-Shah segmentation can be obtained without allowing each pixel to vary independently. Their approximative model has comparable performance to Mumford-Shah segmentations where each pixel is allowed to vary freely. A way to obtain global minimizers was introduced by Nikolova, Esedo¯ glu and Chan [9]. Here, a convex constrained minimization problem has to be solved followed by a simple thresholding of the latter minimizer. This method is closely related to the method we propose in this paper, the key difference is that [9] requires a constraint in the convex minimization while the model proposed in this paper does not involve any constraint in the convex formulation. On the other hand there are methods to solve a certain class of minimal surface problems by unconstrained convex optimization, cf. the work of Chambolle and Darbon [10,11]. The 2-phase Mumford-Shah functional belongs to this class, yet due to the best of our knowledge nobody seems to have tapped the potential offered by these general insights for Mumford-Shah based image segmentation so far.

2

Constrained Global 2-Phase Minimization

First let us describe the general framework and revise the work of Nikolova et al. [9], the starting point for our model. In the following, Ω denotes our computational domain, an arbitrary but fixed subset of Rn . For given indicator functions f1 , f2 ∈ L1 (Ω) such that f1 , f2 ≥ 0 a.e. we consider the prototype Mumford-Shah energy   EMS [Σ] := f1 dx + f2 dx + ν Per(Σ), (1) Σ

Ω\Σ

where Per(Σ) denotes the perimeter of the set Σ ⊂ Ω in Ω. If u0 is an image, c1 , c2 ∈ R are two grey values and fi (x) := (u0 (x) − ci )2 , this is the well known piecewise constant Mumford-Shah functional for 2-phase segmentation, i.e.   E[Σ, c1 , c2 ] = (u0 − c1 )2 dx + (u0 − c2 )2 dx + ν Per(Σ). (2) Σ

Ω\Σ

28

B. Berkels

Remark 1. Because of





EMS [Σ] = Σ

(f1 − f2 )dx + ν Per(Σ) +  

f2 dx, Ω

ˆMS [Σ] =:E

ˆMS share the same minimizers. EMS and E 2

Remark 2. For h(x) := e−|x| , we have    (f1 + h) dx + (f2 + h) dx + ν Per(Σ) − hdx, EMS [Σ] = Σ Ω\Σ  Ω  =C<∞

i.e. replacing f1 and f2 by f1 + h and f2 + h does not affect the minimizers of EMS . This, combined with f1 , f2 ≥ 0 a.e., means that we can assume f1 , f2 > 0 a.e. in Ω without loss of generality. To obtain (local) minimizers of the functional above, Chan and Vese [3] proposed to parametrize the unknown set Σ by a levelset function φ and get the energy  ECV [φ] := H(φ)f1 + (1 − H(φ))f2 + ν|∇(H(φ))|dx. Ω

Here, H(·) denotes the Heaviside function, i.e. H(s) = 1 for s > 0 and H(s) = 0 else. A gradient descent will be used for minimization, therefore H is replaced by a smeared out Heaviside function, e.g. Hδ (x) := 12 + π1 arctan xδ , where δ > 0. While the specific choice is not important, it is important to use a function whose derivative does not have compact support (cf. [3]). This gives the regularized energy  ECV,δ [φ] := Hδ (φ)f1 + (1 − Hδ (φ)) f2 + ν |∇(Hδ (φ))| dx (3) Ω

and yields the gradient descent 

∇φ . ∂t φ = Hδ (φ) (f2 − f1 ) + ν div |∇φ|

(4)

One of the major drawbacks of the energy (3) is its non-convexity in φ. In [9], Nikolova et al. noted that the gradient descent (4) and 

∇φ ∂t φ = (f2 − f1 ) + ν div |∇φ| have the same stationary points, because Hδ (φ) > 0. Obviously the latter is the gradient descent of the energy  ECE [φ] := (f1 − f2 )φ + ν|∇φ|dx. Ω

Image Segmentation by Unconstrained Multiphase Thresholding

29

In general, f1 − f2 takes positive and negative values, therefore the energy is not bounded (neither from below nor from above). In other words, it does not necessarily have a minimizer. However, this is easily fixed by restricting the minimization to 0 ≤ φ(x) ≤ 1 for all x ∈ Ω. Based on this, the following theorem holds: Theorem 1. For given indicator functions f1 , f2 ∈ L1 (Ω) such that f1 , f2 ≥ 0 a.e., let  u := argmin (f1 − f2 )˜ u + ν|∇˜ u|dx = argmin ECE [˜ u] 0≤˜ u≤1

0≤˜ u≤1

Ω

and Σc := {x ∈ Ω|u(x) > c}. Then Σc is a minimizer of the Mumford-Shah energy (1) for all c ∈ [0, 1). Proof. Nikolova et al. proved this theorem in [9] for a.e. c ∈ [0, 1], we extend it here to hold not only for almost every, but for every c ∈ [0, 1). First, we briefly sketch the prove given by Nikolova et al. for a.e. c ∈ [0, 1]. Using 0 ≤ u ≤ 1 and the coarea formula, one can show  1 ECE [u] = EMS [Σc ]dc − C, 0

where C is a constant independent of u. Let Σ∗ ⊂ Ω be a minimizer of EMS (the existence of such minimizers using convergence in measure follows from standard arguments) and let M := {c ∈ [0, 1]|EMS [Σc ] > EMS [Σ∗ ]}. Assuming μ(M ) > 0 leads to the contradiction ECE [χΣ∗ ] < ECE [u] therefore μ(M ) = 0 holds and the statement is proven for a.e. c ∈ [0, 1]. Here, χA denotes the characteristic function of the set A. Now we extend the statement to all c ∈ [0, 1), inspired by the proof of Lemma 4 (iii) in [12]: Again let u be a minimizer of ECE under the constraint 0 ≤ u ≤ 1 and denote its superlevelsets by Σc . Choose an arbitrary but fixed cˆ ∈ [0, 1). The statement holds for a.e. c ∈ [0, 1], so by Remark 1, there exists a sequence (cn ) ∈ [0, 1]N with cn ↓ cˆ such that ˆMS [Σ]. Σcn ∈ argmin E Σ⊂Ω

Since the superlevelsets of a function are contained in each other, we have χΣcn =

∞ χ nk=1 Σck → χΣ ∪ pointwise a.e., where Σ ∪ := n=1 Σcn . Setting g := f1 − f2 and using Lebesgue’s dominated convergence theorem, we obtain     gdx = gχΣ ∪ dx = lim gχΣcn dx = lim gdx. Σ∪

Ω

n→∞

n→∞

Ω

Σcn

  Here we used gχΣcn  ≤ |g| ≤ |f1 | + |f2 | to provide the integrable upper bound. For each n and Σ ⊂ Ω, we have   gdx + ν Per(Σcn ) ≤ gdx + ν Per(Σ). Σ cn

Σ

30

B. Berkels

Using the continuity argument from above and the lower semicontinuity of the perimiter (cf. [13]), we get   gdx + ν Per(Σ ∪ ) ≤ gdx + ν Per(Σ), Σ∪

Σ

i.e. Σ ∪ is a minimizer of EMS [·, cn ]. Combining this with Σc = {x ∈ Ω|u(x) > c} =

∞ 

{x ∈ Ω|u(x) > cn } =

n=1

concludes the proof.

∞ 

Σc n

n=1



Knowing that Theorem 1 holds true for all c ∈ [0, 1) also remedies the last bit of “uncertainty” left in [9]. Remark 3. For any function u that fulfills the constraint, obviously {u > 1} = ∅. Therefore we cannot expect Theorem 1 to hold for c = 1. To solve the constrained optimization problem, Nikolova et al. show that the constrained problem has the  same minimizers as the unconstrained problem if a penalty term of the form α p(u(x)) is added with asufficiently large coefficient  α (cf. [9], Claim 1). Here p denotes p(s) = max{0, 2 s − 12  − 1} . While this result already gives a method to find global minimizers of EMS by solving a convex, unconstrained minimization problem, its practical relevance is limited. Most numerical minimizations methods rely on the gradient of the functional, but the proposed penalty term is not differentiable, making a regularization necessary. But any smooth regularization of the penalty term will stop the minimizers of the convex, constrained functional to coincide with those of the convex functional with penalty term. The stronger the regularization, the more the minimizers deviate. Furthermore, the regularization imposes numerical difficulties. If an explicit gradient descent is used for the minimization (as proposed in [9]), a suitable timestep size control is needed to ensure convergence. The step sizes allowed by such methods, e.g. the Armijo rule [14], typically correspond to the size of the region in which the linearization of the functional properly approximates the functional. Due to the nature of the penalty term p, the linearization at 0 and 1 of a regularized version of it only approximates the regularization properly in a region that is of the size of the regularization parameter. So, as soon as the current iterate of the gradient descent takes values near 0 or 1, the timestep control only allows timestep sizes of the order of the regularization parameter, which, as mentioned above, cannot be chosen too big. Instead of using a penalty term one could of course also approach the constrained convex optimization problem directly. This is done for example by Bresson et al. [15]. Their approach does not need a penalty term and gives an efficient algorithm to minimize ECE , but has to introduce an additional unknown v and a regularization parameter θ and needs to minimize for u and v alternatingly. Furthermore, the key idea to apply Chambolle’s TV minimization algorithm [16] can

Image Segmentation by Unconstrained Multiphase Thresholding

31

also be directly applied to our model to obtain a simpler and faster minimization algorithm: There is no need to introduce v, θ and the alternating minimization. Therefore it is worth to investigate whether it is possible to simplify the problem by getting rid of the constraint altogether.

3

Unconstrained Global 2-Phase Minimization

Another alternative to Chan Vese is a phase field approach [6, 5] with a typical double well term: 

 1 2 2 EPH, [u] := u (1 − u)2 + |∇u| dx. u2 f1 + (1 − u)2 f2 + ν Ω A minimizer u of this energy is a diffuse representation of the segmentation, i.e. {u = 0} and {u = 1} represent the two segments respectively with a smooth transition in between. EPH, [u] is known to Γ -converge to EMS [5], but unfortunately not convex and does not permit jumps in u for > 0. Knowing both ECE and EPH, , the question arises whether it is possible to combine the advantages of both models while eliminating some of the disadvantages. Heuristically looking at both energies served as motivation to investigate the following energy:  E[u] := u2 f1 + (1 − u)2 f2 + ν|∇u|dx. (5) Ω

This energy is convex because it does not involve the non-convex double well term of EPH, , and can be minimized without imposing constraints because it does not have the indicator term from ECE that is not bounded from below. Furthermore, it permits jumps in u. Remark 4. Given a function u, obviously we have E[min{max{0, u}, 1}] ≤ E[u]. Therefore, a minimizer umin fulfills 0 ≤ umin ≤ 1. While the proposed functional has some nice obvious properties, it is far from obvious whether there is a relation between its minimizer and minimizers of EMS . Before we tackle this question, let us remark a link between ECE and E: Remark 5. There is a direct relationship between ECE and E: A straightforward calculation shows u2 f1 + (1 − u)2 f2 = (f1 − f2 ) u + (u − 12 )2 (f1 + f2 ) − 14 (f1 + f2 ) + f2 . Therefore



(f1 − f2 ) u + (u − 12 )2 (f1 + f2 ) − 14 (f1 + f2 ) + f2 + ν|∇u|dx  =ECE [u] + (f1 + f2 )(u − 12 )2 dx + C.

E[u] =

Ω

Ω

In other words, E essentially equals ECE plus an additional quadratic penalty energy. The constant C is clearly irrelevant for the minimizers.

32

B. Berkels

To investigate the relation between the minimizers of E and minimizers of EMS we can make use of the theory derived in the context of the connection between minimal surface problems and total variation minimization. The following general statement has been made by Chambolle [17], Chambolle and Darbon [11], in the continuous setting, its discrete counterpart is well known: Theorem 2. Let Ψ : Ω × R → R, (x, s) → Ψ (x, s) such that Ψ (x, ·) is C 1 and uniformly convex for all x ∈ Ω and  u := argmin Ψ (x, u ˜(x)) + ν|∇˜ u|dx. u ˜

Ω

Then Σc := {x ∈ Ω|u(x) > c} for all c ∈ R is a minimizer of  ∂s Ψ (x, c)dx + ν Per(Σ). Σ

Note that this general statement cannot be directly applied to the model of Nikolova et al. discussed in Section 2 because the integrand is neither uniformly (not even strictly) convex nor does the general statement incorporate the constraint. As remarked in [11], the proof for a more specific statement given in [10] still applies to Theorem 2. Theorem 3. If u is a minimizer of (5), then {u > 12 } minimizes   EMS [Σ] = f1 dx + f2 dx + ν Per(Σ). Σ

Ω\Σ

Proof. Let Ψ (x, s) := s2 f1 (x)+(1−s)2 f2 (x). Obviously Ψ (x, ·) is C 2 for all x ∈ Ω and we have ∂s Ψ (x, s) = 2sf1 (x)+2(s−1)f2 (x) and ∂s2 Ψ (x, s) = 2(f1 (x)+f2 (x)). From Remark 2, we know that f1 , f2 > 0 a.e., therefore Ψ (x, ·) is uniformly convex for a.e. x ∈ Ω. Now just apply Theorem 2, noting ∂s Ψ (x, 12 ) = f1 (x) − f2 (x) and Remark 1.

In this sense, our theorem is a corollary of Theorem 2. The preceding theorem finally tells us how to find a global minimizer of EMS [·] given in (1): Minimize the convex energy (5) and threshold the minimizer to 1 2 . In case of the piecewise constant Mumford-Shah functional for 2-phase segmentation, we obtain a global minimizer of the Mumford-Shah energy (2) with respect to Σ for fixed gray values c1 , c2 . We do not necessarily find a global minimizer with respect to Σ, c1 and c2 . Another link between ECE and EPH, is the so-called piecewise constant levelset method [18] for 2-phase segmentation that constrains the levelset function to be piecewise constant. If this constraint is approximated with a penalty energy, the method equals the phase field approach. If the constraint is relaxed to a certain boundedness constraint, the method equals [9]. In both cases the fidelity term has to be altered accordingly, making use of the fact that this term is the same in ECE and EPH, if u only takes the values 0 and 1.

Image Segmentation by Unconstrained Multiphase Thresholding

33

Since (5) is similar to the Rudin-Osher-Fatemi energy [19], there is a wide variety of established minimization schemes to choose from, ranging from a straightforward gradient descent scheme with a differentiable approximation of the BV term over primal thresholding methods [20] to sophisticated methods based on the dual formulation of the BV norm, e.g. [16, 11]. 2 With Ψ (x, s) = 12 (s − (f2 (x) − f1 (x))) , another immediate consequence of Theorem 2 is that the zero superlevelset of a minimizer of the ROF energy  1 2 (u − (f2 − f1 )) + ν|∇u|dx EROF [u] := (6) 2 Ω ˆMS and therefore of EMS . This is another way to obtain is a global minimizer of E a global minimizer of EMS by unconstrained convex optimization, but compared to (5) this method has a few shortcomings, cf. Sections 4 and 5. Furthermore, the boundedness mentioned in Remark 4 does not hold for minimizers of the ROF energy. Perhaps this is one of the reasons why nobody seems to have used the classical ROF function for Mumford-Shah based image segmentation so far.

4

Multiphase Segmentation

Our functional can be extended to multiphase segmentation by the using the idea of Vese and Chan [2] in a straightforward manner. To keep notation at bay, we restrict the discussion to segmentation in 4 phases. The segmentation in 2n phases works analogously. Let f1 , f2 , f3 , f4 ∈ L1 (Ω) such that fi ≥ 0 a.e., then the multiphase functional is given by  E[u1 , u2 ] := u21 u22 f1 + (1 − u1 )2 u22 f2 Ω

+ u21 (1 − u2 )2 f3 + (1 − u1 )2 (1 − u2 )2 f4

(7)

+ ν (|∇u1 | + |∇u2 |) dx. If we fix u2 , the reduced functional E[·, u2 ] is the same as the 2-phase functional (5) with the indicator functions f˜1 = u22 f1 + (1 − u2 )2 f3 and f˜2 = u22 f2 + (1 − u2 )2 f4 . As in the 2-phase case, we can assume fi > 0 a.e. without loss of generality and because either u22 > 0 or (1 − u2 )2 holds, we have f˜1 , f˜2 > 0. Therefore, all statements proven for the 2-phase functional can be applied to E[·, u2 ], i.e. we can compute the global minimum (for fixed u2 ). The same applies for fixed u1 , so as an optimization strategy, we propose to minimize with respect to u1 and u2 alternatingly. Even though it is easy to extend (5) to multiphase segmentation, the same does not apply to the ROF energy (6). There is no apparent extension in the sense of [2] to formulate the multiphase segmentation in a single functional.

5

Indicator Parameters

In typical segmentation tasks, the indicator functions depend on unknown parameters, e.g. the grey values for each segment in case of the piecewise constant

34

B. Berkels

Mumford-Shah model. For the sake of simplicity, we discuss the latter model in its 2-phase formulation here, i.e. fi (x) := (u0 (x) − ci )2 , i = 1, 2, but this discussion applies to other indicator functions and multiphase segmentation as well. During the minimization of (5) we have to minimize for c1 and c2 as well. This is typically done in an alternating fashion, but there are two apparent possibilities to update the grey values: Minimize (5) with respect to c1 and c2 or do so for the energy in the set formulation (1). The two possible updating formulae for c1 two arising are     u2 u0 dx u2 dx or c1 = u0 dx dx. c1 = Ω

Ω

{u> 12 }

{u> 12 }

The two possibilities only coincide if u is binary. The first formula not only averages u0 in {u > 12 }, instead it takes into account the values of u0 everywhere, but weights the values according to u2 . To a certain degree this is similar to the effect of the regularization of the Heaviside function in the model of Chan and Vese. From our experiments, this reduces the chance of getting stuck in local minima that can still occur when minimizing over u and the indicator parameters. Particulary in the case of multiphase segmentation it turned out to be beneficial. Due to the different way f1 and f2 are used in the ROF energy (6), it is not quadratic in c1 and c2 . So this functional does not give a natural formula to update the grey values.

6

Numerical Examples

To conclude, we show the practical usability of the proposed model by applying it to the classical piecewise constant Mumford-Shah functional, see equation (2). As minimization method we use an explicit gradient descent scheme with the Armijo√rule [14] as timestep size control. The absolute value is regularized by |z| = z 2 + 2 (in all examples presented here, = 0.1 is used). For the spatial discretization, we use bilinear finite elements on a regular quadrilateral grid, i.e. each pixel of the input image u0 corresponds to a node of the finite element mesh. The grey values c1 and c2 are initialized with 0 and 1 respectively and updated occasionally during the gradient descent. Figure 1 shows results of our method and of the one proposed by Nikolova et al. [9] on one artificial image and one digital photo. In both examples, the minimizer u from our model is far from being binary, but this is nothing to be expected from the theory presented in this paper. The 0.5-superlevelset gives an accurate segmentation that is not influenced by the presence of heavy noise (top row) and works on non-binary input images (bottom row). The minimizers u of the Nikolova et al. model look very different, but the segmentation obtained from the 0.5-superlevelsets is almost identical. Upon closer inspection, the minimizer u of our model from the top row of Figure 1 looks very much like as obtained by minimizing the ROF energy with

Image Segmentation by Unconstrained Multiphase Thresholding

35

Fig. 1. Segmentation of an artificial noisy structure (ν = 2 · 10−3 , top row) and the well-known Matlab cameraman image (ν = 4·10−3 , bottom row): Input image u0 (left), segmentation function u and 0.5-superlevelset of u colored with the average grey values c1 , c2 obtained by our model (middle) and by using ECE (right). The slight difference of the grey values is attributed to the employed update formula, cf. Section 5.

Fig. 2. 4-phase segmentation of an artificial noisy image (top row) and a MRI image (bottom row) (ν = 6 · 10−4 ): Input image u0 (left), segmentation functions u1 and u2 (middle), segmentation colored with the average grey values c1 , ..., c4 (right)

u0 as input image. This is not surprising due to the following observation: If u0 is binary, i.e. u0 = χA for a set A ⊂ Ω and c1 = 0, c2 = 1 we have f1 = (χA − 0)2 = χA and f2 = (χA − 1)2 = χΩ\A and therefore  E[u] = Ω

(u − χΩ\A )2 + ν|∇u|dx,

36

B. Berkels

Fig. 3. Segmentation of a digital photo (ν = 2 · 10−5 ). Input image u0 (left), segmentation in four (middle) and eight (right) segments colored with the average grey values c bigmama / PIXELIO. of the segments. Original image 

Fig. 4. Intermediate results of the segmentation in eight segments shown in Figure 3 after 50 (left), 250 (middle) and 700 (right) gradient descent steps

i.e. E equals the ROF energy in this special case. This is not the case if u0 is non-binary which can be seen from the bottom row of Figure 1. Figure 2 shows 4-phase segmentation results. Those indicate the tendency of the segmentation functions to become binary for small values of ν. Finally, Figure 3 illustrates the behavior of the method for different numbers of segments and Figure 4 shows three timesteps of the 8-phase segmentation.

References 1. Mumford, D., Shah, J.: Optimal approximation by piecewise smooth functions and associated variational problems. Communications on Pure Applied Mathematics 42, 577–685 (1989) 2. Vese, L.A., Chan, T.F.: A multiphase level set framework for image segmentation using the Mumford and Shah model. International Journal of Computer Vision 50(3), 271–293 (2002) 3. Chan, T.F., Vese, L.A.: Active contours without edges. IEEE Transactions on Image Processing 10(2), 266–277 (2001) 4. Osher, S.J., Sethian, J.A.: Fronts propagating with curvature dependent speed: Algorithms based on Hamilton–Jacobi formulations. Journal of Computational Physics 79, 12–49 (1988) 5. Shen, J.: Γ -convergence approximation to piecewise constant Mumford-Shah segmentation. In: Blanc-Talon, J., Philips, W., Popescu, D.C., Scheunders, P. (eds.) ACIVS 2005. LNCS, vol. 3708, pp. 499–506. Springer, Heidelberg (2005)

Image Segmentation by Unconstrained Multiphase Thresholding

37

6. Esedo¯ glu, S., Tsai, Y.H.R.: Threshold dynamics for the piecewise constant Mumford-Shah functional. Journal of Computational Physics 211(1), 367–384 (2006) 7. Merriman, B., Bence, J.K., Osher, S.J.: Diffusion generated motion by mean curvature. CAM Report 92-18, UCLA (1992) 8. Alvino, C.V., Yezzi, A.J.: Fast Mumford-Shah segmentation using image scale space bases. In: Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, vol. 6498 (2007) 9. Nikolova, M., Esedo¯ glu, S., Chan, T.F.: Algorithms for finding global minimizers of image segmentation and denoising models. SIAM Journal on Applied Mathematics 66(5), 1632–1648 (2006) 10. Chambolle, A.: An algorithm for mean curvature motion. Interfaces and free Boundaries 6, 195–218 (2004) 11. Chambolle, A., Darbon, J.: On total variation minimization and surface evolution using parametric maximum flows. CAM Report 08-19, UCLA (2008) 12. Alter, F., Caselles, V., Chambolle, A.: A characterization of convex calibrable sets in RN . Mathematische Annalen 332(2), 329–366 (2005) 13. Ambrosio, L., Fusco, N., Pallara, D.: Functions of bounded variation and free discontinuity problems. Oxford Mathematical Monographs. Oxford University Press, New York (2000) 14. Kosmol, P.: Methoden zur numerischen Behandlung nichtlinearer Gleichungen und Optimierungsaufgaben, 2nd edn. Teubner, Stuttgart (1993) 15. Bresson, X., Esedo¯ glu, S., Vandergheynst, P., Thiran, J., Osher, S.: Fast global minimization of the active contour/snake model. Journal of Mathematical Imaging and Vision 28(2), 151–167 (2007) 16. Chambolle, A.: An algorithm for total variation minimization and applications. Journal of Mathematical Imaging and Vision 20(1-2), 89–97 (2004) 17. Chambolle, A.: Total variation minimization and a class of binary MRF models. In: Rangarajan, A., Vemuri, B.C., Yuille, A.L. (eds.) EMMCVPR 2005. LNCS, vol. 3757, pp. 136–152. Springer, Heidelberg (2005) 18. Lie, J., Lysaker, M., Tai, X.C.: A binary level set model and some applications to Mumford-Shah image segmentation. IEEE Transactions on Image Processing 15(5), 1171–1181 (2006) 19. Rudin, L., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Physica D 60, 259–268 (1992) 20. Daubechies, I., Defrise, M., de Mol, C.: An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Communications on Pure and Applied Mathematics 57(11), 1413–1457 (2004)

Extraction of the Intercellular Skeleton from 2D Images of Embryogenesis Using Eikonal Equation and Advective Subjective Surface Method Paul Bourgine1 , Peter Frolkovič2, Karol Mikula2 , Nadine Peyriéras3, and Mariana Remešíková2 1

2

CREA, Ecole Polytechnique-CNRS, 1 rue Descartes, 75005, Paris, France [email protected] Department of Mathematics, Slovak University of Technology, Radlinského 11, 81368 Bratislava [email protected], [email protected], [email protected] 3 CNRS-DEPSN, Avenue de la Terasse, 91198, Gif-sur-Yvette, France [email protected]

Abstract. We suggest an efficient method for automatic detection of the intercellular skeleton in microscope images of early embryogenesis. The method is based on the solution of two advective PDEs. First, we solve numerically the time relaxed eikonal equation in order to obtain the signed distance function to a given set – a set of points representing cell centers or a set of closed curves representing segmented inner borders of cells. The second step is a segmentation process driven by the advective version of subjective surface equation where the velocity field is given by the gradient of the computed distance function. The first equation is discretized by Rouy-Tourin scheme and we suggest a fixing strategy that significantly improves the speed of the computation. The second equation is solved using a classical upwind strategy. We present several test examples and we show a practical application - the intercellular skeleton extracted from a 2D image of a zebrafish embryo.

1

Introduction

The measure of the cell contact surface (intercellular skeleton) is an important quantitative characteristic of a living organism, especially during its embryonic development [6]. Together with other characteristics, e.g. the volume of the embryo, the global and local density of cells, the density of cell divisions etc., it provides an insight into the process of the evolution of the organism and allows to detect abnormalities or to compare individuals evolving in different conditions. The intercellular skeleton can be extracted from the miscroscope images of the evolving embryo. Fig. 1 shows an example of suitable image data. These images display significant cell structures (cell nuclei and cell membranes) of a zebrafish embryo at an early stage of its development and they were obtained by a two-photon confocal microscope. X.-C. Tai et al. (Eds.): SSVM 2009, LNCS 5567, pp. 38–49, 2009. c Springer-Verlag Berlin Heidelberg 2009 

Extraction of the Intercellular Skeleton

39

The main goal of our paper is to introduce an efficient and easily implementable method for detecting the intercellular skeleton. Our technique is based on numerical solution of a pair of advective partial differential equations. The first step is the solution of the time relaxed eikonal equation with a special Dirichlet type condition. The solution of such an equation is the distance function to a given set. This can be a set of points representing cell centers or a set of closed curves representing inner borders of cells obtained by segmentation. In case we deal with the curves, we construct the signed distance function with negative values in the interior part. We discretize the problem using the explicit Rouy-Tourin scheme and we suggest to extend the original scheme by a fixing technique. The idea of fixing is based on the fact that the Rouy-Tourin scheme applied to the time relaxed eikonal equation produces in every point monotonically increasing values approaching the value of the distance function. At some moment, the value will reach some steady state and the point can be excluded from the calculations. This strategy provides a significant improvement of the efficiency of the method and it brings a natural stopping criterion for the computation. We compare the performance of our algorithm with the computation of the exact distance function and we provide some examples of situations when the numerical solution can be obtained faster. The second step of our procedure is the segmentation using the advective version of the subjective surface equation. For each cell, we construct an initial segmentation function. Afterwards, all level sets of this function are evolving according to the velocity field given by the gradient of the computed signed distance function. By taking one of the level sets of the final form of the evolving function, we obtain the part of the required intercellular skeleton corresponding to one particular cell. The complete skeleton is constructed as the union of the results corresponding to individual cells. Using the distance function corresponding to cell centers, we get a Voronoi type cell skeleton that is already a good approximation of the real one as the cell formations are naturally similar to Voronoi tiling. A very realistic skeleton localization can be obtained if we consider the distance function to the segmented inner boundaries of the cells, assuming that we have a good quality cell segmentation. For pratical purposes, it is even sufficient to perform only a few time steps of the Rouy-Tourin scheme and use a rough estimate of the distance function in order to obtain a correctly oriented velocity field for the segmentation. This makes the method very efficient without loss of the quality of the resulting skeleton. The advective subjective surface equation is discretized by an explicit upwind approach. The paper is organized as follows. In Sec. 2, we describe the mathematical models for the two substeps of the procedure. Sec. 3 presents the numerical schemes and it explains the idea of the fixing algorithm intended to reduce the CPU time needed to compute the numerical solution of the eikonal euqation. In Sec. 4, we provide a series of numerical experiments as well as an example of a skeleton extracted from a 2D microscope image of a zebrafish embryo. Let us note that we present a method for solving two-dimensional problems, but the extension to three dimensions is rather straightforward.

40

P. Bourgine et al.

Fig. 1. 2D slices of 3D image of a zebrafish embryo. Left, the cell nuclei. Right, the cell membranes.

2

Mathematical Models

The first equation involved in our skeleton extraction strategy is the eikonal equation with time relaxation dt + |∇d| = 1

(1)

solved in the domain Ω × [0, TD ] where Ω is the image domain and coupled with a Dirichlet type condition d(x, t) = 0,

x ∈ Ω0 ⊂ Ω.

(2)

By the problem formulated in this way the solution d approaches, as time is evolving, the distance function to the set Ω0 . In our case, as we have already mentioned, Ω0 can be a set of points corresponding to approximate cell centers or a set of closed curves representing the segmented inner boundaries of the cells. The signed distance function can be constructed straightforwardly. The result of the cell shape segmentation is a level set function. Choosing one of the level sets to represent the inner boundary of the cell, we are able to recognize the inner and outer parts of the cell [1, 2] and assign the corresponding sign to the distance function. The distance function corresponding to a set of points is always positive. In the second step, we use the computed signed distance function in the advective part of the subjective surface model [4, 2] and we solve the equation ut + ∇g · ∇u = 0

(3)

where (x, t) ∈ Ω × [0, TS ] and g(x) = d (x, TD ) according to [8] or g(x) = −1/(1 + Kdp (x, TD )) with K > 0, p > 0 as in [4, 2]. The unknown function u is initialized by a piecewise constant profile localized around the approximate cell center. Then it is evolved by (3). The intercellular borders are represented by a p

Extraction of the Intercellular Skeleton

41

chosen level set of the function u(x, TS ). Due to the properties of the function d (see Fig. 3 and 6), the border lines of the neighboring cells correspond to the ridges of the distance function and are attached to each other and thus form the intercellular skeleton.

3 3.1

Numerical Schemes Time Relaxation Method with Fixing for Computing the Distance Function

In order to solve the equation (1) with the condition (2) numerically, we use an explicit time discretization with time step τD . Afterwards, the equation is discretized in space by the Rouy-Tourin scheme [3], cf. also [5,7]. As it is natural for image processing applications, the space grid elements correspond to the pixels of the image. Let us consider a rectangular space domain with dimensions Lx × Ly . The space grid is then uniform and consist of square elements Vij , i = 1 . . . nx , j = 1 . . . ny , nx = Lx /hD , ny = Ly /hD where hD is the length of the side of the pixel. For each volume Vij , let dij represent the approximate value of the solution d in the center of Vij in time step n. Let us define Mijpq , p, q ∈ {−1, 0, 1}, |p| + |q| = 1 as   2 Mijpq = min dni+p,j+q − dnij , 0 The Rouy-Tourin scheme for problem (1) then reads as follows      τD 1,0 0,−1 n+1 n + max Mij max Mij−1,0 , Mij , Mij0,1 dij = dij + τD − hD

(4)

This scheme is stable for τD ≤ hD /2 and we take advantage of the fact, that it produces monotonically increasing updates that are gradually approaching a steady state. This property allows us to implement (4) in a computationally efficient way. Let us consider the index set F n that contains the indices (i, j) of the volumes where the steady state has been already reached, i.e. there exists such n0 ∈ N , n0 ≤ n, that dnij0 = dnij0 −1 . The set F 0 is given as follows. At the beginning, we compute exact distances to the set Ω0 (which is a set of points corresponding to cell centers or a set of curves representing the inner boundaries of cells) in a local (one pixel) neighborhood. Then F 0 consists of the indices of all volumes with these exact values including the set Ω0 . Then the method is given by Algorithm 1. 3.2

Advective Subjective Surface Method for Detecting the Intercellular Skeleton

Now we discretize equation (3). Again, we consider explicit time discretization with time step τS and the space grid elements are indentified with the pixels of the image. The space discretization is based on the upwind principle. If the

42

P. Bourgine et al.

Algorithm 1. Fixing method for distance function • if (i, j) ∈ F n then continue • else   −1,0   0,−1  1,0 0,1 τD = dn max Mij , Mij , Mij + max Mij • dn+1 ij + τD − hD ij •

n+1 if dn+1 = dn = F n ∪ {(i, j)} ij then F ij

lentgh of the side of the pixel is denoted by hS and we define the central differy x ences Dij g = (gi+1,j − gi−1,j )/(2hS ), Dij g = (gi,j+1 − gi,j−1 )/(2hS ), we get the following approximation of (3)  x   x  τS  un+1 max Dij = unij − g, 0 (uij − ui−1,j ) + min Dij g, 0 (ui+1,j − uij ) ij hS   y    y g, 0 (uij − ui,j−1 ) + min Dij g, 0 (ui,j+1 − uij ) (5) + max Dij As the initial condition we take a shock-like profile localized around the cell center. Due to the properties of the signed distance function computed by the method described in Sec. 3.1, we can see that the advective velocity ∇g drives all level lines of the initial segmentation function to the ridges of the distance function, cf. Fig. 6. These ridges represent the intercellular skeleton.

4 4.1

Numerical Experiments Computation of the Signed Distance Function

Now let us present some computational results obtained by Algorithm 1. We inspected the experimental order of convergence of the suggested method, the CPU time needed for the computation and the effect of the fixing strategy. The results were also compared with the distance function computed analytically. First, let us make a note about the stopping criteria for the methods. If the fixing technique is not applied, the computation is stopped either when ||dn+1 − dn ||L1 (Ω) ≤ ε1 << 1 or when a prescribed number of time steps is performed. If we use the fixing strategy, we stop when (i, j) ∈ F n for all i = 1 . . . nx , j = 1 . . . ny , i.e. when all values are already fixed. In practice, the condition dn+1 = dnij is replaced by |dn+1 − dnij | ≤ ε2 << 1. If we want to ij ij compare the two methods, we first run the computation with fixing until all points are fixed and then the method with no fixing is prescribed to stop at exactly the same time. Now let us assume that the error of the numerical method in L2 (Ω)-norm is of the form E(h) = Chα , where h is the space discretization step. Obtaining experimentally E(h) and E(h/2) (see e.g. Table 1, column 4), we can express α = log2 (E(h)/(E(h/2)), which is called the experimental order or convergence (EOC). In the first experiment, we computed the distance function to seven given points situated in a square domain Ω = [−1, 1] × [−1, 1] and we measured the

Extraction of the Intercellular Skeleton

43

L2 (Ω) error with respect to the exact distance function and the EOC. The results and some details of the computations are displayed in Table 1, 2 and 3. Table 1 shows the comparison of the method with fixing with the original scheme without any fixing. The value in the seven points was set to 0, and the values in a one pixel neighborhood of the points were set to the values of the exact solution. In the stopping criterion for the fixing method, we set ε2 = 10−5 . Fig. 2 displays the distance function computed by the fizing method using nx = ny = 320. Comparing the results in the table, we can see the effect of the fixing strategy on the CPU time as well as on the L2 error and EOC. We can observe that in this case the computation is approximately two times faster and also we get smaller error when we fix the solution by our algorithm. The fact that the L2 error apparently depends on the value of ε2 led us to perform the experiment presented in Table 2. We were looking for the optimal choice of ε2 , i.e. the value that would provide the smallest L2 error. We can see that this value is different for different discretization parameters. Finally, in Table 3, we present the EOC for a slightly different implementation of the fixing method. In this case, the solution was initialized with the exact values not only in a one pixel neighborhood, but in a neighborhood whose size was independent of the space step. Again, we were looking for the optimal value of ε2 . We can observe that the EOC is approximately equal to 1. Table 1. Comparison of results of computational tests for the method without fixing and the method with fixing. Computation of the distance function to seven given points. nx 40 80 160 320 640

τD time steps L2 (Ω)-error CP U EOC L2 (Ω)-error fixing 0.025 52 3.447525e-2 0.01 3.440187e-2 0.0125 90 2.373985e-2 0.05 0.53825 2.359906e-2 0.00625 163 1.533921e-2 0.4 0.63009 1.508116e-2 0.003125 305 9.498908e-3 3.1 0.69139 9.016883e-3 0.0015625 583 5.704456e-3 23.18 0.73567 4.844598e-3

CP U fixing 0.0 0.03 0.21 1.73 13.08

EOC fixing 0.54376 0.64598 0.74205 0.89625

Table 2. Results of computational tests for the fixing method. Computation of the distance function to seven given points with determination of optimal ε2 . nx τD time steps 40 0.025 37 80 0.0125 73 160 0.00625 145 320 0.003125 287 640 0.0015625 569 1280 0.00078125 1132

L2 (Ω) - error 1.976552e-2 1.373675e-2 9.133363e-3 5.857149e-3 3.645805e-3 2.164545e-3

ε2 3.4e-3 1.3e-3 4.0e-4 1.3e-4 4.0e-5 1.1e-5

CP U 0.0 1.02 0.19 1.62 12.77 100.09

EOC 0.52494 0.58882 0.64095 0.68396 0.75217

The next experiment is similar. Instead of seven points, we computed the distance function to four polygons. Table 4 and Fig. 3 show the results.

44

P. Bourgine et al.

Table 3. Experimental order of convergence for the fixing method with exact values prescribed in a fixed neighborhood of seven given points and with determination of optimal ε2 nx τD time steps 40 0.025 35 80 0.0125 71 160 0.00625 140 320 0.003125 276 640 0.0015625 545

L2 (Ω) - error 2.648673e-2 1.352430e-2 6.928000e-3 3.598868e-3 1.795268e-3

ε2 6.2e-3 1.5e-3 3.5e-4 9.0e-5 2.2e-5

1.0

CP U 0.0 0.02 0.19 1.58 12.29

EOC 0.96972 0.96504 0.94490 1.00033

80

60 0.8

0.5

0.6

40

0.4 0.2 0.0 20

0.0 0

0.5 0 20 40 60

1.0 1.0

0.5

0.0

0.5

1.0

80

Fig. 2. Distance function to seven given points computed by the fast fixing method. Left, the contours of the function. Right, the 3D plot.

Summarizing the results presented in Table 1, 2 and 3, we can determine the experimental complexity of our algorithm. By careful checking of the CPU times needed for the computations, we can see that the complexity is not higher than O(N 3/2 ), where N is the number of unknowns. The next two tests were realized in order to compare the fixing method with computation of the exact distance function. Again, we used ε2 = 10−5 for the fixing mehotd. In the first case, a certain number of points was randomly generated in a given 2D domain. After, the distance function to this set of points was computed both numerically by our fixing algorithm and analytically by finding the nearest point to every pixel. The CPU time was measured for different numbers of points and plotted in graphs presented in Fig. 4. We can observe that the computational cost of the analytical computation is increasing with increasing number of points while the cost of the numerical computation is decreasing. For a certain number of points, depending on the size of the image domain, the numerical method becomes more efficient. We can see that in all cases displayed in the figure, this number is at most 1% of the number of image pixels so it is practically meaningful. Another experiment was performed in order to test the numerical method on data qualitatively similar to segmented cell structures. The result of the cell segmentation is a level set function and the inner borders of the cells are

Extraction of the Intercellular Skeleton

45

Table 4. Comparison of computational tests for the method without fixing and the method with fixing. Computation of the distance function to four polygons. nx 40 80 160 320 640

τD time steps L2 (Ω) - error CP U EOC L2 (Ω) - error fixing 0.025 36 1.122800e-2 0.01 1.121228e-2 0.0125 61 8.266972e-3 0.03 0.44167 8.231755e-3 0.00625 108 5.606750e-3 0.26 0.56020 5.539007e-3 0.003125 203 3.599271e-3 2.04 0.63946 3.475747e-3 0.0015625 390 2.228162e-3 15.46 0.69185 2.041495e-3

1.0

CP U EOC fixing fixing 0.01 0.01 0.44581 0.11 0.57157 0.96 0.67230 7.24 0.76770

80

60

0.4

0.5

40

0.2 0.0 0.2

20

0.0 0

0.5 0 20 40 60

1.0 1.0

0.5

0.0

0.5

1.0

80

Fig. 3. Distance function to four polygons computed by the RMF. Left, the contours of the function with the shapes indicated. Right, the 3D plot.

represented by a chosen isoline of this function. For our test purposes, we used images with isolines in form of either randomly placed circles or randomly placed rectangles of a random size (Fig. 5). The maximum image intensity (255) was in the centers of these shapes and then it gradually decreased to 0 with increasing distance from the center. In order to compute the distance function to a certain level set numerically, the location of the level set was detected and the solution was initialized by exact values in its one pixel neighborhood. After, the fixing method was applied. For the exact computation, we used a procedure that constructs set of points corresponding to the chosen isoline, finding its crossections with the pixel structure. Then, the nearest of these points was found for each pixel. The CPU time for such a procedure is documented in Table 5, column 5. By construction, the points of the isoline are sorted by their coordinates and therefore the computation of the exact distance function can be optimized – we do not have to go through the whole list of points but the nearest point to a pixel can be always found in a certain neighborhood of the nearest point to the previous pixel. The CPU times for this optimized approach are listed in Table 5, column 6. According to Table 5, we can observe that in all cases considered here, the numerical solution was faster than the optimized analytical computation. We considered isoline for value I = 128, hD = 1.0, τD = 0.5, ε2 = 10−5 and the dimensions of the image domain were 512 × 512 pixels.

46

P. Bourgine et al.

2.5

10

2.0

8

1.5

6

1.0

4

0.5

2

35 30 25 20 15 10 5 0

0

0

1000

2000

3000

4000

5000

6000

0

1000

2000

3000

4000

5000

6000

0

1000

2000

3000

4000

5000

6000

Fig. 4. Comparison of the fixing method (solid line) with computation of exact solution (dashed line)–plot of CPU time depending on the number of randomly distributed points. Left, image domain with 256 × 256 pixels. Middle, 512 × 512 pixels. Right, 1024 × 1024 pixels.

Fig. 5. Isolines (I = 128) of test level set functions simulating cell structures Table 5. Comparison of the numerical and analytical computation of the distance function to the cell-like structures type of level set time steps L2 (Ω) - rel. error CPU num. CPU exact CPU exact opt. 256 rectangles 52 3.509519e-2 0.91 35.56 2.87 1024 rectangles 33 5.813329e-2 0.63 75.82 2.88 256 circles 54 2.441320e-2 0.91 34.55 3.00 1024 circles 34 4.153412e-2 0.64 68.41 3.07

In the last experiment, we computed numerically the distance function to a set of segmented cells of a zebrafish embryo. In Fig. 6, we show the computed distance function as well as the vector field given by the gradient of this function. We used hD = 1.0, τD = 0.5 and ε2 = 10−5 . 4.2

Extraction of the Intercellular Skeleton

In the first experiment, we computed the skeleton using the distance function corresponding to approximate cell centers. The result is a Voronoi type skeleton.

Extraction of the Intercellular Skeleton

47

1.0

0.5

0.0

0.5

1.0 1.0

0.5

0.0

0.5

1.0

40 10

30

20 30

20

40 5

10

0 5

Fig. 6. Signed distance function to segmented cells. Top left, segmentation of inner boundaries of cells. Top right, contour plot of the signed distance function. Bottom left, detail of the 3D plot of this distance function. Bottom right, detail of the vector field given by ∇g, g(x) = d(x), with recognizable position of the intercellular skeleton.

As we can see in Fig. 7, this skeleton represents a very good approximation of the cell structure in the sense that it can provide reliable estimate of some quantities, like the area of the cell contact surface. It can be used in the cases when no or bad quality membrane images are provided and therefore the correct membrane segmentation is not possible. If we have a good quality segmentation of the inner boundaries of the cells, we can detect the intercellular skeleton more precisely using the signed distance function to the segmented objects. An example is shown in Fig. 8. In both experiments, we set hD = 1.0, τD = 0.5, ε2 = 10−5 , hS = 1.0, τS = 0.1. The automatic segmentation of inner cell borders displayed in Fig. 8 was obtained by the generalized subjective surface method [2] applied to a microscope image of zebrafish cell membranes, similar to Fig. 1.

48

P. Bourgine et al.

Fig. 7. Voronoi type skeleton. On the left, the approximate cell centers. On the right, the corresponding Voronoi type skeleton.

Fig. 8. Real skeleton detection. On the left, the cell segmentation. On the right, the corresponding intercellular skeleton.

1.0

0.5

0.0

0.5

1.0 1.0

0.5

0.0

0.5

1.0

Fig. 9. Left, the result after 10 steps of computation of the distance function. Right, the skeleton obtained using the corresponding vector field.

Extraction of the Intercellular Skeleton

49

Remark 1. In practice, the main determining factor of the quality of the detected skeleton is the correct orientation of the vector field generated by the gradient of the distance function. As it follows from the character of the problem and also from the numerical procedure, the correct orientation of the vector field can be already obtained as soon as the values in all image pixels are nontrivially updated by the numerical scheme (4), i.e. before they are definitely fixed. This allows a significant reduction of the computational time without loss of the quality of the final result. In Fig. 9 left, the result obtained after 10 steps of the numerical computation of the distance function is shown. At this moment, the values were nontrivially updated in a sufficiently large neighborhood of the segmented cells. We can construct the vector field for equation (3) and find the corresponding intercellular skeleton (Fig. 9 right). Comparing Fig. 8 and 9 we can see that the skeleton found in this way is of the same quality as in the case of complete fixing of the values in the whole domain. Let us note that the complete fixing would require 123 time steps.

Acknowledgments The work was supported by the European projects Embryomics and BioEmergences, the grants APVV-RPEU-0004-06, APVV-0351-07, APVV-LPP-0020-07 and the grant of VEGA 1/0269/09.

References 1. Frolkovič, P., Mikula, K., Peyriéras, N., Sarti, A.: A counting number of cells and cell segmentation using advection-diffusion equations. Kybernetika 43(6), 817–829 (2007) 2. Mikula, K., Peyriéras, N., Remešíková, M., Sarti, A.: 3D embryogenesis image segmentation by the generalized subjective surface method using the finite volume technique. In: Proceedings of FVCA5 – 5th International Symposium on Finite Volumes for Complex Applications. Hermes Publ., Paris (2008) 3. Rouy, E., Tourin, A.: Viscosity solutions approach to shape-from-shading. SIAM Journal on Numerical Analysis 29(3), 867–884 (1992) 4. Sarti, A., Malladi, R., Sethian, J.A.: Subjective Surfaces: A Method for Completing Missing Boundaries. Proc. Nat. Acad. Sci. 12(97), 6258–6263 (2000) 5. Sethian, J.A.: Level Set Methods and Fast Marching Methods: Evolving Interfaces in Computational Geometry. In: Fluid Mechanics, Computer Vision, and Material Science. Cambridge University Press, New York (1999) 6. Tassy, O., Daian, F., Hudson, C., Bertrandt, V., Lemaire, P.: A quantitative approach to the study of the cell shapes and interactions during early chordate embryogenesis. Currrent Biology 16, 345–358 (2006) 7. Zhao, H.-K.: Fast sweeping method for eikonal equations. Mathematics of Computation 74, 603–627 (2005) 8. Zhao, H.-K., Osher, S., Fedkiw, R.: Fast surface reconstruction using the level set method. In: Proc. IEEE workshop on variational and level set methods - VLSM 2001, Vancouer, pp. 194–201 (2001)

On Level-Set Type Methods for Recovering Piecewise Constant Solutions of Ill-Posed Problems Adriano DeCezaro1, Antonio Leitão2 , and Xue-Cheng Tai3,4 1

Institute of Mathematics Statistics and Physics, Federal University of Rio Grande, Av. Italia km 8, 96201-900 Rio Grande, Brazil [email protected] 2 Department of Mathematics, Federal University of St. Catarina, P.O. Box 476, 88040-900 Florianópolis, Brazil [email protected] 3 Division of Mathematical Science, School of Physical and Mathematical Sciences, Nanyang Technological University, Singapore [email protected] 4 Department of Mathematics, University of Bergen, Johannes Brunsgate 12, N-5008 Bergen, Norway

Abstract. We propose a regularization method for solving ill-posed problems, under the assumption that the solutions are piecewise constant functions with unknown level sets and unknown level values. A level set framework is established for the inverse problem and a Tikhonov regularization approach is proposed. Existence of generalized minimizers for the Tikhonov functional is proven. Moreover, we establish convergence and stability results, characterizing our Tikhonov approach as a regularization method. Based on the necessary conditions of optimality for the Tikhonov functional, a level-set type method is derived and implemented numerically for solving an inverse source problem. This allow us to test the quality of the proposed algorithm.

1

Introduction

Several inverse problems of interest consist of identifying an unknown physical quantity u ∈ X, that can be represented by a piecewise constant function, over a bounded given domain Ω, from the set of data y ∈ Y , where X, Y are Hilbert spaces. This process being described by the model F (u) = y ,

(1)

where F : D(F ) ⊂ X → Y and the set of data is obtained by indirect measurements of the parameter. Because of this, in practical applications the exact data y ∈ Y is, in general, not known. Given is only approximate measured data y δ ∈ Y , corrupted by noise of level δ > 0 and satisfying y δ − yY ≤ δ . X.-C. Tai et al. (Eds.): SSVM 2009, LNCS 5567, pp. 50–62, 2009. c Springer-Verlag Berlin Heidelberg 2009 

(2)

On Level-Set Type Methods

51

In the case where the unknown function u is a piecewise constant function distinguishing between two given values (i.e., up to an affine transformation, u is a characteristic function), level set approaches were considered in [1, 2, 3, 4, 5, 6]. In this case, since the level values of u are known, one needs only to identify the level sets of u, i.e. the inverse problem reduces to a shape identification problem. In the case where the unknown function u is a piecewise constant function distinguishing between several given values, multiple level set approaches were considered in [6,7,8]. If the level values of u are also unknown the inverse problem becomes harder, since one has to identify both the level sets as well as the level values of the unknown parameter u. In this case, the dimension of the parameter space increases by the number of unknown level values. Our starting point in this article is the assumption that the parameter u in (1) is a piecewise constant function assuming two unknown values, i.e. u(x) ∈ {c1 , c2 } a.e. in Ω ⊂ Rd , d = 2, 3. In this case one can assume the existence of an open mensurable set D ⊂⊂ Ω s.t. u(x) = c1 , x ∈ D and u(x) = c2 x ∈ Ω/D. We propose a level set approach to represent the unknown parameter u. First we introduce the H 1 -function φ, which act as a regularization on the parameter space. Then, using the Heaviside projector H : H 1 (Ω) → L∞ (Ω), a solution of (1) can be represented in the form u = c1 H(φ) + c2 (1 − H(φ)) =: P (φ, c1 , c2 ) .

(3)

With this notation we have D = {x ∈ Ω ; φ(x) > 0} and Ω/D = {x ∈ Ω ; φ(x) < 0}. The level values c1 , c2 ∈ R are unknown and have to be determined as well. As already observed in [3], the Heaviside operator H maps H 1 (Ω) into the set V := {χD ; D ⊂ Ω measurable, Hn−1 (∂D) < ∞}, where Hn−1 (S) denotes the (n-1)-dimensional Hausdorff-measure of the set S. Therefore, the operator P in (3) maps H 1 (Ω) × R2 into the admissible parameter set U := {u = q(v, c1 , c2 ); v ∈ V and c1 , c2 ∈ R}, where q : V × R2  (v, c1 , c2 ) → c1 v + c2 (1 − v) ∈ L∞ (Ω). Using the level set framework introduced above, the inverse problem in (1), with data given as in (2), can be written in the form of the operator equation F (P (φ, c1 , c2 )) = y δ .

(4)

Once an approximate solution (φ, c1 , c2 ) of (4) is obtained, a corresponding solution of (1) can be computed using equation (3). In this article, approximate solutions to (4) are obtained by minimizing the Tikhonov functional  Gα (φ, c1 , c2 ) := F (P (φ, c1 , c2 )) − y δ 2Y + α β1 |H(φ)|BV + β2 φ − φ0 2H 1  2 + β3 j=1 |cj − cj0 |2 , (5) based on T V -H 1 penalization. Here φ0 and cj0 are known reference parameters. This Tikhonov functional extends the ones proposed in [5, 6, 9] (based on T V penalization) and [3, 8] (based on T V -H 1 penalization). To motivate the regularization terms in (5), notice that they effect: i) the boundedness of the level lines of φ as well as it’s H 1 -norm; ii) the boundedness of cj . These two facts allow us to guarantee existence of (generalized) minimizers of Gα in L∞ ∩ BV.

52

A. DeCezaro, A. Leitão, and X.-C. Tai

This article is outlined as follows: In Section 2 we introduce the concept of generalized minimizers for the functional Gα in (5). In Section 3 we derive a convergence analysis for this Tikhonov approach. In Section 4 we introduce stabilized functionals and prove that the corresponding minimizers approximate a minimizer of Gα . Section 5 is devoted to numerical experiments. A level set type method is implemented for solving a two-dimensional inverse potential problem.

2

The Concept of Generalized Minimizers

We shall consider the model problem described as in the introduction under the following general assumptions: (A1) Ω ⊆ Rn is bounded, connected, with piecewise C 1 boundary ∂Ω. (A2) The operator F : D(F ) ⊂ L1 (Ω) → Y is continuous and Fréchetdifferentiable on D(F ) with respect to the L1 (Ω)-topology. (A3) ε, α and βj , j = 1, 2, 3 denote positive parameters. (A4) Equation (1) has a solution, i.e. there exists u ∈ U satisfying F (u) = y and a function φ ∈ H 1 (Ω) satisfying |∇φ| = 0, in a neighborhood of {φ = 0} such that H(φ) = z, for some z ∈ V. Moreover, there exist constants values c1 , c2 ∈ R such that q(z, c1 , c2 ) = u. For each ε > 0, we define the operator Pε (φ, c1 , c2 ) := c1 Hε (φ) + c2 (1 − Hε (φ)) ,

(6)

where Hε is the continuous approximation to H given by:  1 + t/ε for t ∈ [−ε, 0] Hε (t) := . H(t) for t ∈ R/ [−ε, 0] In order to guarantee existence of a minimizer of Gα in (5), we adapt to the level-set framework described above, the concept of generalized minimizers formulated in [3]. Definition 1. Let the operators H, P , Hε and Pε be defined as above. a) A vector (z, φ, c1 , c2 ) ∈ L∞ (Ω) × H 1 (Ω) × R2 is called admissible when there exists a sequence {φk } of H 1 (Ω)-functions satisfying limk φk − φL2 = 0, and also there exists a sequence {εk } ∈ R+ converging to zero such that limk Hεk (φk ) − zL1 = 0. b) A minimizer of Gα is considered to be any admissible vector (z, φ, c1 , c2 ) minimizing 2  Gα (z, φ, c1 , c2 ) := F (q(z, c1 , c2 )) − y δ  Y + αR(z, φ, c1 , c2 ) (7) over the set of admissible vectors, where

 R(z, φ, c1 , c2 ) = ρ(z, φ) + β3 2j=1 |cj − cj0 |2 ,    ρ(z, φ) := inf lim inf β1 |Hεk (φk )|BV + β2 φk − φ0 2H 1 . k→∞

(8) (9)

The infimum in (9) is taken over all sequences {εk } and {φk } characterizing (z, φ, c1 , c2 ) as an admissible vector.

On Level-Set Type Methods

53

c) A generalized minimizer of Gα (φ, c1 , c2 ) is an admissible vector (z, φ, c1 , c2 ) minimizing the functional Gα in (7) on the set of admissible vectors. 2.1

Relevant Properties of Admissible Vectors

First we verify some basic properties of the operators Pε , Hε and q that will be necessary in the subsequent analysis. Lemma 1. Let Ω be given as above and j = 1, 2. (i) Let {zk }k∈N be a bounded sequence in L∞ (Ω) converging to some element z in L1 (Ω) and {cjk }k∈N be a sequence of real numbers converging to cj . Then q(zk , c1k , c2k ) converges to q(z, c1 , c2 ) in L1 (Ω). (ii) Let (z, φ) ∈ L1 (Ω) × H 1 (Ω), be such that Hε (φ) → z in L1 (Ω) as ε → 0 and let c1 , c2 ∈ R. Then Pε (φ, c1 , c2 ) → q(z, c1 , c2 ) in L1 (Ω) as ε → 0. (iii) Given ε > 0, let {φk }k∈N be a sequence in H 1 (Ω) converging to φ ∈ H 1 (Ω) in the L2 -norm. Then Hε (φk ) → Hε (φ) in L1 (Ω), as k → ∞. Moreover, if {cjk }k∈N are sequences of real numbers converging to some cj , then q(Hε (φk ), c1k , c2k ) → q(Hε (φ), c1 , c2 ) in L1 (Ω), as k → ∞. Proof. Since Ω is assumed to be bounded, we have L∞ (Ω) ⊂ L1 (Ω). To prove (i), notice that 1 2 1 2 q(zk , ck , ck )−q(z, c , c )L1 = |c1k zk + c2k (1 − zk ) − c1 z − c2 (1 − z)| dx Ω

1  (|c | + |c2 |)|zk − z| + |c2k − c2 | dx ≤ |zk | (|c1k − c1 | + |c2k − c2 |)dx + Ω

Ω

≤|Ω| zk  L∞ (|c1k − c1 | + |c2k − c2 |) + (|c1 | + |c2 |)zk − zL1 + |Ω| |c2k − c2 | , which converges to zero as k → ∞. Assertion (ii) follows with similar arguments. The first part of assertion (iii) is a direct consequence of the inequality Hε (φk )− Hε (φ)L1 (Ω) ≤ ε−1 meas(Ω)φk − φL2 (Ω) . The second part of assertion (iii) follows by a combination of the inequality above and assertion (i).  Lemma 2. Let (zk , φk , c1k , c2k ) be a sequence of admissible vectors converging in L1 (Ω)× L2 (Ω)× R2 to some (z, φ, c1 , c2 ). Then (z, φ, c1 , c2 ) is also an admissible vector. Sketch of the proof. In order to prove that (z, φ, c1 , c2 ) is also an admissible vector, one uses an argument of extraction of diagonal subsequences, analogously as in [8, Lemma 2].  2.2

Relevant Properties of the Penalization Functional

In the next lemmas we verify two properties of the functional R which are fundamental for the convergence analysis in Section 3. Lemma 3. The functional R in (8) is coercive on the set of admissible vectors.

54

A. DeCezaro, A. Leitão, and X.-C. Tai

Sketch of the proof. Let (z, φ, c1 , c2 ) be an admissible vector. From [8, Lemma 4] follows

 ρ(z, φ) ≥ β1 |z|BV + β2 φ − φ0 2H 1 . (10) Now, from (10) and the definition of R in (8) follows β1 |z|BV + β2 φ − φ0 2H 1 + β3

2

|cj − cj0 |2 ≤ 2 ρ(z, φ) + β3 j=1 |cj − cj0 |2 = R(z, φ, c1 , c2 ) , j=1



concluding the proof.

Lemma 4. The functional R in (8) is weak lower semi-continuous on the set of admissible vectors, i.e. given a sequence {(zk , φk , c1k , c2k )} of admissible vectors such that zk → z in L1 (Ω), φk  φ in H 1 (Ω), cjk → cj in R, for some admissible vector (z, φ, c1 , c2 ), then it follows R(z, φ, c1 , c2 ) ≤ lim inf R(zk , φk , c1k , c2k ) . k∈N

Proof. The functional ρ(z, φ) is weak lower semi-continuous cf. [8, Lemma 5]. Moreover, the Euclidean norm in R2 is also lower semi-continuous. The lemma follows from the fact that the functional R in (8) is a linear combination of lower semi-continuous functionals. 

3

Convergence Analysis

First we prove that for any positive parameters α, β the functional Gα in (5) is well posed. Theorem 1 (Well-Posedness). The functional Gα in (5) attains minimizers on the set of admissible vectors. Proof. Notice that the set of admissible vectors is not empty, since (0, 0, 0, 0) is admissible. Let {(zk , φk , c1k , c2k )} be a minimizing sequence for Gα , i.e. a sequence of admissible vectors satisfying Gα (zk , φk , c1k , c2k ) → inf Gα ≤ Gα (0, 0, 0, 0) < ∞. Then, {Gα (zk , φk , c1k , c2k )} is a bounded sequence of real numbers. Therefore, {(zk , φk , c1k , c2 , k)} is uniformly bounded in BV × H 1 (Ω) × R2 . Thus, the Sobolev compact embedding theorem [10] and the Bolzano-Weierstrass theorem guarantees the existence of a subsequence (denoted again by {(zk , φk , c1k , c2k )}) and the existence of (z, φ, c1 , c2 ) ∈ L1 (Ω) × H 1 (Ω) × R2 such that φk → φ in L2 (Ω), φk  φ in H 1 (Ω), zk → z in L1 (Ω) and cjk → cj in R. From Lemma 2 we conclude that (z, φ, c1 , c2 ) is an admissible vector. Moreover, from the weak lower semi-continuity of R together with the continuity of F and q we obtain   lim Gα (zk , φk , c1k , c2k ) = lim F (q(zk , c1k , c2k )) − y δ 2Y + αR(zk , φk , c1k , c2k ) k→∞

k→∞

≥ F (q(z, c1 , c2 )) − y δ 2Y + αR(z, φ, c1 , c2 ) = Gα (z, φ, c1 , c2 ) , (11) proving that (z, φ, c1 , c2 ) minimizes Gα .



On Level-Set Type Methods

55

In the next theorems we present the main convergence and stability results. The proofs use classical techniques from the analysis of Tikhonov type regularization methods (see, e.g., [11, 12]). Theorem 2 (Convergence for exact data). Assume that we have exact data, i.e. y δ = y and βj > 0 , j = 1, 2, 3. For every α > 0 let (zα , φα , c1α , c2α ) denote a minimizer of Gα on the set of admissible vectors. Then, for every sequence of positive numbers {αk }k∈N converging to zero there exists a subsequence, denoted again by {αk }l∈N , such that (zαk , φαk , c1αk , c2αk ) is strongly convergent in L1 (Ω)× L2 (Ω) × R2 . Moreover, the limit is a solution of (1). Proof. Let (z † , φ† , c1,† , c2,† ) be a solution of (1) – its existence is guaranteed by assumption (A4). Let {αk }k∈N be a sequence of positive numbers converging to zero. For each k ∈ N, let (zk , φk , c1k , c2k ) := (zαk , φαk , c1αk , c2αk ) be a minimizer of Gαk . Then, for each k ∈ N we have  2 Gαk (zk , φk , c1k , c2k ) ≤ F (q(z † , c1,† , c2,† )) − y Y + αk R(z † , φ† , c1,† , c2,† )

= αk R(z † , φ† , c1,† , c2,† ) . (12)

Since αk R(zk , , φk , c1k , c2k ) ≤ Gαk (zk , φk , c1k , c2k ), it follows from (12) that R(zk , φk , c1k , c2k ) ≤ R(z † , φ† , c1,† , c2,† ) < ∞ .

(13)

Moreover, from the assumption on the sequence {αk }, it follows that lim αk R(z † , φ† , c1,† , c1,† ) = 0 .

k→∞

(14)

From (13) and Lemma 3 we conclude that the sequences {φk }, {zk } and {cjk }j=1,2 are bounded in H 1 (Ω) , BV and R2 respectively. Using an argument of extraction of diagonal subsequences (see proof of Lemma 2) we can guarantee the existence ˜ c˜1 , c˜2 ) such that of an admissible vector (˜ z , φ, ˜ c˜1 , c˜2 ) in L1 (Ω) × L2 (Ω) × R2 . (zk , φk , c1k , c2k ) → (˜ z , φ, From Lemma 1 (i) follows that q(˜ z , c˜1 , c˜2 ) = lim q(zk , c1k , c2k ) on L1 (Ω). Using k→∞

the continuity of the operator F together with (12) and (14) we conclude that y = lim F (q(zk , c1k , c2k )) = F (q(˜ z , c˜1 , c˜2 )) . k→∞

On the other hand, from the lower semi-continuity of R and (13) it follows that ˜ c˜1 , c˜2 ) ≤ lim inf R(zk , φk , c1 , c2 ) R(˜ z , φ, k k k→∞

≤ lim sup R(zk , φk , c1k , c2k )) ≤ R(z †, φ† , c˜1 , c˜2 ) , k→∞

concluding the proof.



56

A. DeCezaro, A. Leitão, and X.-C. Tai

Theorem 3 (Convergence for noisy data). Let α = α(δ) be a function satisfying lim α(δ) = 0 and lim δ 2 α(δ)−1 = 0. Moreover, let {δk }k∈N be a seδ→0

δ→0

quence of positive numbers converging to zero and y δk ∈ Y be corresponding noisy data satisfying (2). Then, there exist a subsequence, denoted again by {δk }, and a sequence {αk := α(δk )} such that (zαk , φαk , c1αk , c2αk ) converges in L1 (Ω) × L2 (Ω) × R2 to solution of (1). Proof. Let (z † , φ† , c1,† , c1,† ) be a solution of (1).1 For each k ∈ N, denote by (zk , φk , c1k , c2k ) := (zα(δk ) , φα(δk ) , c1α(δk ) , c2α(δk ) ) a minimizer of Gα(δk ) . Then, for each k ∈ N we have 2  Gαk (zk , φk , c1k , c2k ) ≤ F (q(z † , c1,† , c1,† )) − y δk Y + α(δk )R(z † , φ† , c1,† , c2,† ) ≤ δk2 + α(δk )R(z † , φ† , c1,† , c2,† ) .

(15)

Taking the limit k → ∞ in (15), it follows from the theorem assumptions  2 that lim F (q(zk , c1k , c2k )) − y δk  ≤ lim Gαk (zk , φk , c1k , c2k ) = 0. Therefore, k→∞

k→∞

lim F (q(zk , c1k , c2k )) = y. Moreover, from (15) and the definition of Gαk , it fol-

k→∞

lows that R(zk , φk , c1k , c2k ) ≤ δk2 α(δk )−1 + R(z † , φ† , c1,† , c2,† ). Thus, from the assumptions on the function α(δk ), we conclude that lim sup R(zk , φk , c1k , c2k ) ≤ k→∞

R(z † , φ† , c1,† , c2,† ). The proof follows arguing as in the proof of Lemma 2.

4



Numerical Solution

In the sequel we introduce a functional which can be handled numerically, and whose minimizers are ’close’ to the minimizers of Gα . Let Gε,α be the stabilized functional defined by  Gε,α (φ, c1 , c2 ) := F (Pε (φ, c1 , c2 )) − y δ 2Y + α β1 |Hε (φ)|BV +  2 + β2 φ − φ0 2H 1 + β3 j=1 |cj − cj0 |2 , (16) where Pε (φ, c1 , c2 ) := q(Hε (φ), c1 , c2 ) is the functional defined in (6). The functional Gε,α is well-posed as the following lemma shows: Lemma 5. Given positive constants α, ε, βj , j = 1, 2, 3 as above, a function φ0 ∈ H 1 (Ω) and cj0 ∈ R, j = 1, 2, the functional Gε,α in (16) attains a minimizer on H 1 (Ω) × R2 . Proof. Since inf{Gε,α (φ, c1 , c2 ) : (φ, c1 , c2 ) ∈ H 1 (Ω) × R2 } ≤ Gε,α (0, 0, 0) < ∞, there exists a minimizing sequence {(φk , c1k , c2k )} in H 1 (Ω) × R2 satisfying lim Gε,α (φk , c1k , c2k ) = inf{Gε,α (φ, c1 , c2 ) : (φ, c1 , c2 ) ∈ H 1 (Ω) × R2 } .

k→∞ 1

The existence of solutions is guaranteed by (A4).

On Level-Set Type Methods

57

Then, for fixed α > 0, the sequences {φk } and {cjk }j=1,2 are bounded in H 1 (Ω) and R2 respectively. Therefore, φk  φ in H 1 (Ω) and cjk → cj in R, j = 1, 2. Moreover, by the weak lower semi-continuity of the H 1 –norm and the continuity of the Euclidean norm in R, it follows that φ − φ0 2H 1 ≤ lim inf φk − φ0 2H 1 , k→∞

and |cj − cj0 | ≤ lim inf |cjk − cj0 |. k→∞

From the Sobolev compact embedding theorem [13] we have φk → φ in L2 (Ω). Therefore, Lemma 1 implies √ Hε (φjk ) − Hε (φj )L1 ≤ ε−1 meas(Ω)φk − φL2 → 0,

Pε (φk , c1k , c2k ) − Pε (φ, c1 , c2 )L1 = q(Hε (φk ), c1k , c2k ) − q(Hε (φ), c1 , c2 )L1 → 0. Thus, it follows from [10, Theorem 1, pg 172] that |Hε (φ)|BV ≤ lim inf |Hε (φk )|BV . k→∞

Now, from the continuity of F and q, together with the estimates above we obtain  Gε,α (φ, c1 , c2 ) ≤ lim F (Pε (φk , c1k , c2k )) − y δ 2Y + α β1 lim inf |Hε (φk )|BV + k→∞ k→∞  2 2 + β2 lim inf φk − φ0 H 1 + β3 lim inf j=1 |cjk − cj0 |2 k→∞

≤ lim inf k→∞

concluding the proof.

Gε,α (φk , c1k , c2k )

k→∞

= inf Gε,α , 

In the sequel we prove that, when ε → 0, the minimizers of Gε,α approximate a minimizer of the functional Gα . Theorem 4. Let α and βj be given as above. For each ε > 0, denote by (φε,α , c1ε,α , c2ε,α ) a minimizer of Gε,α . There exists a sequence of positive numbers εk → 0 such that (Hεk (φεk ,α ), φεk ,α , c1εk ,α , c2εk ,α ) converges strongly in L1 (Ω)×L2 (Ω)× R2 and the limit minimizes Gα on the set of admissible vectors. Proof. The functional Gα attains a generalized minimizer (zα , φα , c1α , c2α ) on the set of admissible vectors (cf. Theorem 1). From Definition 1, there exists a sequence {εk } of positive numbers converging to zero and corresponding sequences {φk } in H 1 (Ω) satisfying φk → φα in L2 (Ω), Hεk (φk ) → zα in L1 (Ω). Moreover, we can further assume [8, Lemma 3] that R(zα , φα , c1α , c1α ) = lim

k→∞

  2 β1 |Hεk (φk )|BV + β2 φk − φ0 2H 1 + β3 j=1 |cjk − cj0 |2 .

Let (φεk , c1εk , c2εk ) be a minimizer of Gεk ,α . The sequences {φεk }, {Hεk (φεk )} and {cjk }j=1,2 are uniformly bounded in H 1 (Ω), BV(Ω) and R2 respectively. By the compact Sobolev embedding theorem [13], the compact embedding of BV into L1 [10] and the Bolzano-Weierstrass theorem, there exist convergent subsequences ˜ z˜ and c˜j . Summarizing, we have φε → φ˜ in whose limits are denoted by φ, k 2 1 ˜ c˜1 , c˜2 ) ∈ L (Ω), Hεk (φεk ) → z˜ in L (Ω), and cjk → c˜j in R, j = 1, 2. Thus, (˜ z , φ, L1 (Ω) × H 1 (Ω) × R2 is an admissible vector (cf. Lemma 2).

58

A. DeCezaro, A. Leitão, and X.-C. Tai

From the definition of R, Lemma 1 and the continuity of F , it follows that F (q(˜ z , c˜1 , c˜2 )) − y δ 2Y = lim F (Pεk (φεk , c1εk , c2εk )) − y δ 2Y , 

k→∞

2  ˜ c˜1 , c˜2 ) ≤ lim inf β1 |Hε (φε )|BV + β2 φε − φ0 2 1 + β3  |cj − cj |2 . R(˜ z , φ, εk k k k 0 H k→∞

j=1

Therefore, ˜ c˜1 , c˜2 ) = F (q(˜ ˜ c˜1 , c˜2 ) z , φ, z , c˜1 , c˜2 )) − y δ 2Y + αR(˜ z , φ, Gα (˜ ≤ lim inf Gεk ,α (φεk , c1εk , c2εk ) ≤ lim inf Gεk ,α (φk , c1k , c2k ) k→∞

≤

lim sup F (Pεk (φk , c1k , c2k )) k→∞

−

k→∞ δ 2 y Y



+ α lim sup β1 |Hεk (φk )|BV + β2 φk − φ0 2H 1 + β3

=

k→∞ F (q(zα , c1α , c2α ))

2 j=1

|cjk − cj0 |2

− y δ 2Y + αR(zα , φα , c1α , c2α ) = Gα (zα , φ1α , c1α , c2α ) ,

˜ c1 , c2 ) as a minimizer of Gα . characterizing (˜ z , φ, α α 4.1





Optimality Conditions for the Stabilized Functional

For numerical purposes it is convenient to derive first order optimality conditions for minimizers of the stabilized functionals Gε,α . Therefore, we consider Gε,α in (16) with Y = L2 (Ω) and we look for the Gâteaux directional derivatives with respect to φ and the unknown constants cj for j = 1, 2. Since Hε (φ) is self-adjoint, we can write the optimality conditions for the functional Gε,α in the form of the system α(Δ − I)(φ − φ0 ) = Lε,α,β (φ, c1 , c2 ), in Ω ; (φ − φ0 ) · ν = 0, at ∂Ω (17a) α (cj − cj0 ) = Ljε,α,β (φ, c1 , c2 ), j = 1, 2 .

(17b)

Here ν(x) is the external unit normal vector at x ∈ ∂Ω, β¯ := (2β3 )−1 , and Lε,α,β (φ, c1 , c2 ) = (c1 − c2 )β2−1 Hε (φ)∗ F  (Pε (φ, c1 , c2 ))∗(F (Pε (φ, c1 , c2 )) − y δ )   −β1 (2β2 )−1 Hε (φ) ∇· ∇Hε (φ)/|∇Hε (φ)| , (18a)

∗ L1ε,α,β (φ, c1 , c2 ) = β¯ F  (Pε (φ, c1 , c2 ))Hε (φ) (F (Pε (φ, c1 , c2 )) − y δ ), (18b)

 ∗ L2ε,α,β (φ, c1 , c2 ) = β¯ F  (Pε (φ, c1 , c2 ))(1 − Hε (φ)) (F (Pε (φ, c1 , c2 )) − y δ ).(18c)

5

Numerical Results

In this section a level-set type method based on the system of optimality conditions (17) is used for solving an inverse potential problem of recovering a piecewise constant function u : Ω → {c1 , c2 }, from measurements of the Cauchy data of its corresponding potential on the boundary of the domain Ω = (0, 1) × (0, 1). Notice that no knowledge of the image of u (values c1 , c2 ∈ R) is assumed.

On Level-Set Type Methods

5.1

59

The Inverse Potential Problem

To describe the direct problem, we define the operator F : L2 (Ω) → L2 (∂Ω) by F : u(x) → F (u) := wν |∂Ω , where u is a piecewise constant function in L2 (Ω) with u(x) ∈ {c1 , c2 } a.e. in Ω, and w ∈ H 1 (Ω) solves the elliptic boundary value problem Δw = u , in Ω ; w = 0 , at ∂Ω . (19) Since u ∈ L2 (Ω), the Dirichlet boundary value problem in (19) has a unique solution, namely the potential w ∈ H 2 (Ω) ∩ H01 (Ω). The inverse problem we are concerned with, consists in determining the piecewise constant source function u from measurements of the Neumann trace of w at ∂Ω, i.e. from wν |∂Ω . Using the above notation, the inverse potential problem can be written in the abbreviated form F (u) = y δ , where the data y δ has the same meaning as in (2). Other inverse problems for the operator F were considered in [3, 8]. In [3] a level set method was used for recovering the indicator function u = χD of a star-shaped domain D ⊂ R2 . In [8] a multiple level set method was used for recovering a simple function u : Ω → {c1 , . . . , c4 }. In both cases, knowledge of the (finite) image of u was assumed. 5.2

A Level-Set Algorithm for the Inverse Potential Problem

In the sequel we describe the level set regularization algorithm. This method compares to the level set method as proposed in [8]. The complexity of our algorithm is as follows: at each iteration of the level set method, four elliptic boundary value problems (BVP) are solved (two of Dirichlet type and two of Neumann type). In Table 1 an explicit fixed point procedure for solving the the optimality condition (18) is outlined. In the first step the residual rk ∈ L2 (∂Ω) of the iterate (φk , c1k , c2k ) is evaluated. This corresponds to solving one elliptic BVP of Dirichlet type. In the second step the solution hk ∈ H 1 (Ω) of the adjoint problem for the residual is evaluated. This corresponds to solving one elliptic BVP of Dirichlet type. In the fourth step, the velocity function vk ∈ H 1 (Ω) for the level-set function is evaluated. This corresponds to solving an elliptic BVPs of Neumann type. In the subsequent numerical experiments this algorithm was implemented using a finite element method for the solution of partial differential equations. 5.3

Numerical Experiment

In our experiment we consider the inverse problem of reconstructing the right hand side u in (19) from the knowledge of a single pair of Cauchy data (0, y δ ) at ∂Ω. We further assume that the level value c2 = 0 is given, and that we have to identify only the support of u and the level value c1 ∈ R+ . The data y δ = y = F (u) for solving the inverse problem is known exactly, i.e. δ = 0, and is obtained by solving numerically the elliptic boundary value problem

60

A. DeCezaro, A. Leitão, and X.-C. Tai Table 1. Level set algorithm for the inverse potential problem 1. Evaluate the residual rk := F (Pε (φk , c1k , c2k )) − y δ = (wk )ν |∂Ω − y δ , where wk solves wk = 0 , at ∂Ω . Δwk = Pε (φk , c1k , c2k ) , in Ω ; 2. Evaluate hk := F  (Pε (φk , c1k , c2k ))∗ (rk ) ∈ L2 (Ω), solving Δhk = 0 , in Ω ; hk = rk , at ∂Ω . 3. Calculate Lε,α,β (φk , c1k , c2k ) and Ljε,α,β (φk , c1k , c2k ), j = 1, 2 as in (18). 4. Evaluate the velocity vk ∈ H 1 (Ω), solving (Δ − I)vk = Lε,α,β (φk , c1k , c2k ) , in Ω ; (vk )ν = 0 , at ∂Ω . 5. Update the level set function φk and the level values cjk , j = 1, 2: φk+1 = φk +

1 α

vk ,

cjk+1 = cjk +

1 α

Ljε,α,β (φk , c1k , c2k ) .

in (19) (the word ’exactly’ here means: up to the precision of the numerical method used for solving the direct problem). For the direct problem we use the values: c1 = 1, c2 = 0 to compute the exact solution. In the computation of the inverse problem, the exact solution is known a priori to assume the values {c1 , 0} (with unknown c1 ). Moreover, when the data are given exactly, the iterative level-set method is implemented without the additional regularization term |Hε (φ)|BV , i.e. β1 = 0. The solution u of the inverse problem as well as the initial guess Pε (φ0 , c10 ) for the level-set method are shown in Figure 1. Notice that the support of u corresponds to a non-connected proper subset of Ω, The initial guess c10 = 1.5 is used for the unknown level value. In Figure 2 the evolution of the level set method for the first 1500 iterative steps is presented. As one can see in this figure, the shapes of both inclusions are reasonably reconstructed, and the level value c1 is accurately reconstructed as well. The iteration is stopped when the residual drops below the predefined precision F (Pε (φk , c1k )) − yL2 < 10−2 .

Fig. 1. Numerical experiment: The picture on the left hand side shows the coefficient to be reconstructed. On the right hand side, the initial condition for the level-set method.

On Level-Set Type Methods

61

Fig. 2. Numerical experiment: On the left hand side a plot of P (φk , c1k ) for k = 1500. The picture on the right hand side shows the corresponding iteration error.

We performed other numerical simulations with different choice of initial guess (φ0 , c10 ), and observed that the number of iterative steps required in order to obtain a reasonable approximation (up to the predefined precision of 10−2 in the L2 -norm) strongly depends on the choice of the initial guess c10 . On the other hand, the final result is not sensitive with respect to the choice of the initial guess φ0 .

Acknowledgments A.DC acknowledges the support from CNPq, grant 474593/2007-0. The work of A.L. is supported by the Brazilian National Research Council CNPq, grants 306020/2006-8, 474593/2007-0, and by the Alexander von Humbolt Foundation AvH. This article was written during a visit of the author to NTU (Singapore). X.-C.T. acknowledges the support from NTU SUG 20/07 and MOE Tier II project T207B2202 (ARC 29/07).

References 1. Santosa, F.: A level-set approach for inverse problems involving obstacles. ESAIM Contrôle Optim. Calc. Var. 1, 17–33 (1995/1996) 2. Leitão, A., Scherzer, O.: On the relation between constraint regularization, level sets, and shape optimization. Inverse Problems 19, L1–L11 (2003) 3. Frühauf, F., Scherzer, O., Leitão, A.: Analysis of regularization methods for the solution of ill-posed problems involving discontinuous operators. SIAM J. Numer. Anal. 43, 767–786 (2005) 4. Chung, E., Chan, T., Tai, X.C.: Electrical impedance tomography using level set representation and total variational regularization. J. Comput. Phys. 205(1), 357– 372 (2005) 5. Chan, T., Tai, X.C.: Identification of discontinuous coefficients in elliptic problems using total variation regularization. SIAM J. Sci. Comput. 25(3), 881–904 (2003) 6. Chan, T., Tai, X.C.: Level set and total variation regularization for elliptic inverse problems with discontinuous coefficients. J. Comput. Phys. 193(1), 40–66 (2004)

62

A. DeCezaro, A. Leitão, and X.-C. Tai

7. Chung, J., Vese, L.: Image segmantation using a multilayer level-sets apprach. UCLA C.A.M. Report 193(03-53), 1–28 (2003) 8. DeCezaro, A., Leitão, A., Tai, X.C.: On multiple level-set regularization methods for inverse problems. Inverse Problems 25 (to appear, 2009) 9. Tai, X.C., Chan, T.: A survey on multiple level set methods with applications for identifying piecewise constant functions. Int. J. Num. Anal. Model 1(1), 25–47 (2004) 10. Evans, L., Gariepy, R.: Measure theory and fine properties of functions. Studies in Advanced Mathematics. CRC Press, Boca Raton (1992) 11. Engl, H., Kunisch, K., Neubauer, A.: Convergence rates for Tikhonov regularisation of nonlinear ill-posed problems. Inverse Problems 5(4), 523–540 (1989) 12. Engl, H.W., Hanke, M., Neubauer, A.: Regularization of inverse problems. Mathematics and its Applications, vol. 375. Kluwer Academic Publishers Group, Dordrecht (1996) 13. Adams, R.: Sobolev Spaces. Academic Press, New York (1975)

The Nonlinear Tensor Diffusion in Segmentation of Meaningful Biological Structures from Image Sequences of Zebrafish Embryogenesis Olga Drblíková1 , Karol Mikula1 , and Nadine Peyriéras2 1

Slovak University of Technology, Radlinského 11, 813 68 Bratislava, Slovakia [email protected], [email protected] http://www.math.sk/drblikov, http://www.math.sk/mikula 2 CNRS-DEPSN, Institut de Neurobiologie Alfred Fessard, Batiment 32-33, Avenue de la Terrasse, 91198 Gif sur Yvette, France [email protected]

Abstract. In this contribution we develop a strategy for segmentation of evolving biological structures in image sequences. Our approach is based on combination of nonlinear tensor diffusion image smoothing and subjective surface based image segmentation. Since the fine cell structure would restrain the evolving segmentation function to achieve a shape of meaningful biological structures, we have to smooth properly the images in the sequence. To that goal we apply the nonlinear tensor diffusion which enhances the connectivity of bordering structure lines and smoothes their inner parts. For the numerical implementations we use semi-implicit diamond-cell finite volume methods both for filtering and segmentation. We show application of the method in image segmentation of early stages of zebrafish embryogenesis.

1

Introduction

The subjective surface based segmentation is an efficient tool for the extraction of 2D or 3D image objects, cf. [10,9,1]. It is also the case when dealing with twophoton laser scanning microscopy images in detecting and segmenting structures at cellular and subcellular level, cf. [6, 8]. However, the use of such algorithms when segmenting the supercellular structures is not straightforward. Using an original (not filtered) image leads to entirely useless results due to the presence of small cell structures. Then a useful tool is filtering by the nonlinear tensor diffusion enhancing the coherence of structure boundaries and smoothing the inner cell structures and noise. The model, cf. [11, 7, 4], has the following form ∂t u − ∇ · (D∇u) = 0, u(x, 0) = u0 (x), (D∇u) · n = 0,

in QT ≡ I × Ω,

(1)

in Ω, on I × ∂Ω,

(2) (3)

where u represents a greylevel 3D image intensity, u0 ∈ L2 (Ω), I = [0, T ] denotes a time interval, Ω is an image domain, D = D(u(x, t)) is a diffusion tensor and X.-C. Tai et al. (Eds.): SSVM 2009, LNCS 5567, pp. 63–74, 2009. c Springer-Verlag Berlin Heidelberg 2009 

64

O. Drblíková, K. Mikula, and N. Peyriéras

n is the outer normal unit vector to ∂Ω. The model is useful when a strong filtering is desirable in a preferred direction, e.g. along 2D edge surfaces in 3D images and a low smoothing is expected in the perpendicular direction.

2

Design of the Diffusion Tensor

The matrix D depends on a smoothed intensity gradient, which is given as ∇ut˜ = (ux1 , ux2 , ux3 )T , where ut˜(x, t) = (Gt˜ ∗ u(·, t))(x),

(t˜ > 0)

(4)

and Gt˜ is a Gaussian kernel. Provided μ = ||∇ut˜||2 > 0 we choose a triplet of vectors (v1 , v2 , v3 ) as follows v1  ∇ut˜,

v2 ⊥ ∇ut˜,

v3 ⊥ ∇ut˜,

v2 ⊥ v3 .

(5)

The direction of vector v1 corresponds to the direction of the largest intensity change. The other two vectors give a tangential plane to a level set of image intensity which may represent a 2D surface edge in a 3D image, provided that μ is large. It is called a coherence plane P, cf. [4, 7], and corresponds to an eigenspace corresponding to the eigenvalue 0 of the outer product ∇ut˜ ⊗ ∇ut˜. In order to improve the coherence, the diffusion tensor D must steer a filtering process such that the diffusion is strong and increasing with the level of μ along the coherence plane and is small in the perpendicular direction. We achieve it choosing the eigenvalues of the diffusion tensor, which determine the diffusivities in the directions v1 , v2 and v3 as κ1 = α,  κ2 =

α ∈ (0, 1), α 1, α, if μ = 0,   α + (1 − α) exp

−C μ

, C>0

(6) otherwise.

Further, we apply other convolution with a smoothing kernel ρ to get the diffusion matrix D in the form ⎛ ⎞  κ1 0 0 B, if μ = 0, B = ⎝ 0 κ2 0 ⎠ D = Gρ ∗ D0 , where D0 = (7) P BP −1 otherwise, 0 0 κ2 and P represents a transition matrix from the basis (v1 , v2 , v3 ) to (e1 , e2 , e3 ). The exponential function in (6) is used to ensure that κ2 does not exceed 1. The process never stops owing to the positive parameter α. Even if μ tends to zero, a small linear diffusion with a diffusivity α > 0 still remains there. C has the role of a threshold parameter. If μ C then κ2 ≈ 1, and, conversely if μ C then κ2 ≈ α. After some manipulations we get that at any point where μ > 0, the matrix D0 has the following form ⎞ ⎛ 2 ux1 ux2 (κ1 − κ2 ) ux1 ux3 (κ1 − κ2 ) ux1 κ1 + (u2x2 + u2x3 )κ2 1⎝ ux1 ux2 (κ1 − κ2 ) u2x2 κ1 + (u2x1 + u2x3 )κ2 ux2 ux3 (κ1 − κ2 ) ⎠ (8) μ ux1 ux3 (κ1 − κ2 ) ux2 ux3 (κ1 − κ2 ) u2x3 κ1 + (u2x1 + u2x2 )κ2

The Nonlinear Tensor Diffusion in Structure Segmentation

65

in the standard basis (e1 , e2 , e3 ). Such choice of the matrix D0 was given in [4], it is independ on a concrete choice of v2 and v3 and can be directly and fast evaluated using the diamond-cell finite volume technique (see also next section). Then the matrices are spatially averaged using the Gaussian smoothing with a variance ρ to get the final matrix D elements. The diffusion tensor possesses the smoothness, symmetry and positive definiteness properties, cf. [4].

3

The Finite Volume Scheme for 3D Nonlinear Tensor Diffusion

Let the image u(x) be represented by a bounded mapping u : Ω → R and given by n1 × n2 × n3 voxels (finite volumes) such that it looks like a mesh with n1 rows, n2 columns and n3 layers. Let us consider an image domain Ω = (0, n1 h) × (0, n2 h) × (0, n3 h), with a voxel size h. We consider the diffusion process in a time interval I = [0, T ]. Let the time discretization is given by 0 = t0 < t1 < ... < tNmax = T with tn = tn−1 + k, where k is a length of a discrete time step. We will look for an approximation of solution at time tn for every n = 1, ..., Nmax . We start the scheme derivation integrating the equation (1) over a finite volume K, then provide a semi-implicit time discretization and use the divergence theorem to have 

unK − un−1 K m(K) − (Dn−1 ∇un ) · nK,σ ds = 0, (9) k σ σ∈EK ∩Eint

where unK , K ∈ Th , denotes the mean value of un on K and Th is a cubic finite volume mesh. Further quantities and notations are given as follows: m(K) is the 3D measure of finite volume K with the boundary ∂K, σKL = K ∩ L is a side of the finite volume K, where L ∈ Th is a neighboring finite volume to K for which holds that the volumes K and L share a 2D surface element with a nonzero area. At several places we will replace σKL by σ only due to a notation simplification. EK represents the set of sides such that ∂K = σ∈EK σ and E = K∈Th EK . The set of boundary sides is denoted by Eext , that is Eext = {σ ∈ E, σ ⊂ ∂Ω} and Eint = E \ Eext . Υ is the set of pairs of neighboring finite volumes defined by Υ = {(K, L) ∈ Th2 , K  = L, m(K ∩ L)  = 0} and nK,σ is the normal unit vector to σ outward to K. Our discrete approximation solution is defined as uh,k (x, t) =

N

max



unK χ{x ∈ K}χ{tn−1 < t ≤ tn },

n=0 K∈Th

where the function χ(A) is given by  1, χ{A} = 0,

if A is true, elsewhere.

(10)

66

O. Drblíková, K. Mikula, and N. Peyriéras

The finite volume approximation at the n-th time step is given by unh,k (x) = 

n 1 uK χ{x ∈ K} and initial values as u0K = m(K) u0 (x)dx, K ∈ Th . We K∈Th

K

can define an auxiliary unknown φnσ (unh,k ) representing an approximation of the  1 n−1 exact averaged flux m(σ) ∇un ) · nK,σ ds for any K and σ ∈ EK in order σ (D to rewrite (9) in the form unK − un−1 1 K − k m(K)



φnσ (unh,k )m(σ) = 0,

σ∈EK ∩Eint

where m(σ) is the measure of side σ.

Fig. 1. The co-volumes associated with the side σ = σWE (left) and σ = σEW (right)

φnσ (unh,k ) is built with the help of a co-volume mesh, cf. e.g. [2, 3], for the 2D case. We create a co-volume χσ associated with σ around each finite volume side by joining four vertices of this side and midpoints of the finite volumes which are common to this side, cf. Fig. 1. The co-volume boundary consists of triangles σ ¯ ⊂ ∂χσ (their vertices are denoted by N1 (¯ σ ), N2 (¯ σ ) and N3 (¯ σ )) and nχσ ,¯σ is the normal unit vector to σ ¯ outward to χσ . First, we approximate the gradient  averaged  on χnσ . Applying the divergence theorem we obtain 1 1 n ∇u dx = u nχσ ,¯σ ds which can be approximated as follows m(χσ ) χσ m(χ σ ) ∂χσ 

1 1 n n n u m(¯ σ )nχσ ,¯σ . The values at xE + u + u pnσ (u) = m(χ (¯ σ ) (¯ σ ) (¯ σ ) N N N ) 3 1 2 3 σ σ ¯ ∈∂χσ

and xW are denoted as uE and uW . Further, we evaluate the values uTN , uTS , uBN , and uBS at the vertices xTN , xTS , xBN , and xBS , cf. Fig. 1, as the arithmetic mean of uK , where K represents the finite volumes which are common to the vertex. Since the mesh is uniform and squared, we can simplify our discrete √ h3 2 2 scheme applying the following relations: m(χσ ) = 3 , m(¯ σ ) = 4 h . After a short calculation we can state pnσ (u) =

uEn − unW un + unBN − unTS − unBS nK,σ + TN t1K,σ h 2h un + unTS − unBN − unBS t2K,σ , + TN 2h

(11)

The Nonlinear Tensor Diffusion in Structure Segmentation

67

6

φσT

φσN 

σT

φσW

σE σW

φ σE

σN

-

σS σB

φσS

φσB

?

Fig. 2. A finite volume K, its boundaries σi , i = E, W, N, S, T, B and the fluxes outward to the finite volume K

where t1K,σ is a unit vector parallel to xTN −xTS such that (xTN −xTS )·t1K,σ > 0 and t2K,σ is a unit vector parallel to xTN −xBN such that (xTN −xBN )·t2K,σ > 0. We replace the exact gradient ∇un by the discrete gradient pnσ (u) to get the numerical flux in the form φnσ (unh,k ) = (Dσ pnσ (u)) · nK,σ . ⎛

(12)

⎞

¯σ D ¯σ D ¯σ D 11 12 13 ¯σ D ¯σ D ¯ σ ⎠ denotes an approximation of the mean value Dσ = Dσn−1 = ⎝ D 12 22 23 σ ¯σ ¯σ ¯ 13 D D23 D33 of the matrix D along σ which was evaluated at the previous time step using ∞ functions due to the convolutions in un−1 h,k . The elements of matrix Dσ are C (4) and (7). Let us emphasize that in (12) we always consider the matrix Dσ written in the basis (nK,σ , t1K,σ , t2K,σ ), cf. [2,3] for an analogy with the 2D model. In practice it means cf. Fig. ⎛ σ that, ⎞ 2, the matrix D given in the standard basis on a side σ σ σ D11 D12 D13 σ σ σ ⎠ D22 D23 by ⎝ D12 is the same in the new basis on two sides σW and σE . It σ σ σ D13 D23⎛D33 ⎞ σ σ σ D22 D12 D23 σ σ σ ⎠ D11 D13 has the form ⎝ D12 in the new basis for two other sides σS and σN σ σ σ D D D ⎛23 σ 13 σ 33 σ ⎞ D33 D23 D13 σ σ σ ⎠ D22 D12 and it becomes ⎝ D23 for the last two sides σB and σT . Using such σ σ σ D13 D12 D11 matrix representations, the definition (12) can be written in the form ⎡⎛ ⎞⎤ ⎛ ⎞ n ⎞⎛ un E −uW ¯σ D ¯σ D ¯σ D 1 h 11 12 13 ⎢ ¯ σ ¯ σ ¯ σ ⎠ ⎜ unTN +unBN −unTS −unBS ⎟⎥ ⎝ ⎠ φnσ (unh,k ) = ⎣⎝ D · 0 = ⎠ ⎦ ⎝ 12 D22 D23 n 2h n n ¯σ D ¯σ D ¯σ un TN +uTS −uBN −uBS 0 D 13 23 33 n

n

n

n

n

n

2h

n

n

n

n

σ uE −uW σ uTN +uBN −uTS −uBS σ uTN +uTS −uBN −uBS ¯ 12 ¯ 13 ¯ 11 +D +D . =D h 2h 2h

68

O. Drblíková, K. Mikula, and N. Peyriéras

Finally, let us summarize our semi-implicit finite volume scheme: unK − un−1 1 K − k m(K) where

φnσ (unh,k ) =



φnσ (unh,k )m(σ) = 0,

(13)

σ∈EK ∩Eint n n n u ¯ σ uTN ¯ σ E − uW + D D 11 12

h n n n u σ TN + uTS − uBN ¯ 13 +D 2h

+ unBN − unTS − unBS 2h − unBS .

(14)

Due to the computation of the values uT N , uT S , uBN and uBS in (14) as the arithmetic mean of neighboring voxel values, we end up with the 27 point finite volume scheme.

4

Segmentation

Our segmentation approach is based on the subjective surface method [10] and its finite volume implementation from [9]. The mathematical model has the following form    ∇u 0 2 2 , in QT ≡ I × Ω,(15) ∂t u = ε + |∇u| ∇. g(|∇Gσ ∗ I |)  ε2 + |∇u|2 u(x, 0) = u0 (x), u = 0,

in Ω,

(16)

on I × ∂Ω,

(17)

where I 0 is the image which is segmented and ε is the regularization parameter. The solution u represents here the evolving segmentation function. The function g = g(|∇Gσ ∗ I 0 |) has the role of the edge detector, which requires a suitable 1 choice of g in practice, e.g. g(s) = 1+Ks 2 , K > 0. In the subjective surface method we start the segmentation constructing the initial segmentation function located in an approximate object center. The segmentation function is driven by equation (15) and evolves to a numerical steady state. Its shock profile gives the segmentation result and shape of the object. To that goal, we choose a suitable isoline of the shock profile which represents the boundary of the segmented object. This isoline is most naturally taken as the average of maximal and minimal value of the final segmentation function.

5

Numerical Experiments

The goal of this section is to discuss our computational results and the influence of nonlinear tensor diffusion filtering on the time evolving biological structure segmentation. We perform our experiments on the 3D image sequences of cell nuclei, cf. Fig. 3, and cell membranes, cf. Figs. 4-6. The images represent early stages of the zebrafish embryogenesis and were created by the two-photon laser scanning microscope. We apply the 3D numerical scheme to filter the images,

The Nonlinear Tensor Diffusion in Structure Segmentation

69

Fig. 3. 2D slices of a 3D zebrafish embryo image. Left: the original image. Right: the filtered image after 50 time steps.

then the segmentation is performed on 2D image slices (512×512 pixels) in order to firstly test the performance and capabilities of the method. First experiment illustrates the behaviour of nonlinear tensor diffusion in filtering of this type of images, cf. Fig. 3. One can observe that this type of diffusion improves the connectivity of structure bordering lines while it smoothes the structure interiors. One can compare the original image showing separate nuclei but with observable structure borders with the filtered one, where the structure border lines are connected. Our next experiments are devoted to the segmentation of eye retina structure in the several subsequent image slices. First, the initial segmentation function is given by two cones which are inside the structure such that their partially overlapping bases sufficiently cover the eye structure area. Then we evolved it in the original as well as filtered images. Using the original image we obtained the final state of segmentation function represented by a variety of different level lines, cf. Fig. 4 (top, right). The question is, which isoline would represent the most precisely the structure shape. The natural choice is a medium isoline which is depicted in the original image Fig. 4 (top, left). One can clearly see the large difference between the segmented and real structure shape due to the restraints of evolving segmentation function caused by inner cell structures. In order to compare our method with other filtering techniques we performed several tests. The segmentation results obtained on the images filtered by the geodesic mean curvature flow (GMCF) filtering, the mean curvature flow (MCF) filtering and the Perona-Malik (PM) filtering are shown in Fig. 5. In Fig. 5 (right), we can see that after filtering the profiles of final segmentation functions are not well suited for our purposes although the MCF results is rather close to the real one. They are again given by several different isolines and medium one, cf. Fig. 5 (left), represents the segmented structure only partially. This is a consequence of edge

70

O. Drblíková, K. Mikula, and N. Peyriéras

Fig. 4. The eye retina segmentation using the 2D original image (top) and image filtered by 20 time steps of the nonlinear tensor diffusion (middle). Left: the averaged isoline of the final state of segmentation function is superimposed to the original and filtered image, respectively. Right: the graphs of the final state of segmentation function is plotted after 2000 time steps using the original image and after 200 time steps using the filtered image. At the bottom we display the original (left) and the filtered image (right).

The Nonlinear Tensor Diffusion in Structure Segmentation

71

Fig. 5. The eye retina segmentation using the filtered image by 100 steps of the GMCF filtering (top), 25 steps of the MCF filtering (middle) and 20 steps of the PM filtering (bottom). Left: the averaged isoline of the final state of segmentation function is superimposed to the filtered image. Right: the graphs of the final state of segmentation function is plotted after 3000 segmentation steps using the GMCF filtering, after 500 segmentation steps using the MCF filtering and 5000 segmentation steps using the PM filtering.

72

O. Drblíková, K. Mikula, and N. Peyriéras

Fig. 6. The segmentation results for the image sequence which are superimposed to the original slices

preserving smoothing by GMCF and PM which can not remove inner cell structures. On the contrary, the final steady state of segmentation function evolving in the image filtered by nonlinear tensor diffusion consists of isolines accumulated

The Nonlinear Tensor Diffusion in Structure Segmentation

73

along the real structure boundary, cf. Fig. 4 (middle, right). The formation of correct shock profile was enabled due to the smoothing of cell structure barriers and noise removal and the emphasizing of structure boundaries. Embedding the medium isoline into the image, cf. Fig. 4 (middle, left) we achieved the precise structure shape. Then the segmentation procedure was successively applied in the image sequence part consisting of 11 images, cf. Fig. 6. We use the backward in time strategy starting from the last image of the sequence segmented as explained above. The initial segmentation function for other slices is taken as the final result of the segmentation of the previous image. Fig. 6 shows the segmentation results displayed on the original membrane images from last 150th image slice (top), to the 145th slice (middle) up to the 140th slice of the processed image sequence (bottom). In experiments dealing with the nonlinear tensor anisotropic diffusion we used the spatial step h = 0.01, time step k = 0.0001, C = 1, α = 0.001, t˜ = 10−5 , ρ = 0.002, 20 time steps for the filtering of membranes images and t˜ = 10−10 , ρ = 0.1, 50 time steps for the filtering of nuclei images. The arising linear systems were solved by the Gauss-Seidel iterative method. For the segmentation experiments we use the following parameters: ε = 10−4 , the spatial step h = 0.01, time step k = 0.01, δ = 10−6 for a stopping criterion and K = 100 (a constant of the 1 function g(s) = 1+Ks 2 ), cf. [9]. The resulting linear systems were solved by the SOR method.

6

Conclusions

In the article we concern with the technique for embryo structure segmentation in image sequences. Since a noise and cell structures restrain the correct segmentation evolution, as the first step, we smooth the image sequence. We choose the nonlinear tensor diffusion due to the fact that this filtering not only smoothes image objects but emphasizes connections of their boundaries as well. Then, the segmentation process starts using an artificial initial function centered inside the biological structure of the first image in the sequence. The segmentation result given by the subjective surface method obtained for this image is used as the initial condition for the next image of processed sequence, etc. Our experiments confirm the usefulness of the nonlinear tensor diffusion for this type of segmentation.

Acknowledgment The work was supported by the European projects Embryomics and BioEmergences, the grants APVV-RPEU-0004-06, APVV-0351-07, APVV-LPP-0020-07 and the grant of VEGA 1/0269/09.

74

O. Drblíková, K. Mikula, and N. Peyriéras

References 1. Corsaro, S., Mikula, K., Sarti, A., Sgallari, F.: Semi-implicit co-volume method in 3D image segmentation. SIAM J. Sci. Comput. 28(6), 2248–2265 (2006) 2. Coudiere, Y., Vila, J.P., Villedieu, P.: Convergence rate of a finite volume scheme for a two-dimensional convection-diffusion problem. M2AN Math. Model. Numer. Anal. 33, 493–516 (1999) 3. Drblíková, O., Mikula, K.: Convergence Analysis of Finite Volume Scheme for Nonlinear Tensor Anisotropic Diffusion in Image Processing. SIAM J. Numer. Anal. 46(1), 37–60 (2007) 4. Drblíková, O., Mikula, K.: Semi-implicit Diamond-cell Finite Volume Scheme for 3D Nonlinear Tensor Diffusion in Coherence Enhancing Image Filtering. In: Eymard, R., Herard, J.M. (eds.) Finite Volumes for Complex Applications V: Problems and Perspectives, ISTE and WILEY, London, pp. 343–350 (2008) 5. Eymard, R., Gallouët, T., Herbin, R.: Finite Volume Methods. In: Ciarlet, P., Lions, J.L. (eds.) Handbook for Numerical Analysis, vol. 7. Elsevier, Amsterdam (2000) 6. Frolkovič, P., Mikula, K., Peyriéras, N., Sarti, A.: A counting number of cells and cell segmentation using advection-diffusion equations. Kybernetika 43(6), 817–829 (2007) 7. Meijering, E., Niessen, W., Weickert, J., Viergever, M.: Diffusion-Enhanced Visualization and Quantification of Vascular Anomalies in Three-Dimensional Rotational Angiography: Results of an In-Vitro Evaluation. Medical Image Analysis 6(3), 217– 235 (2002) 8. Mikula, K., Peyriéras, N., Remešíková, M., Sarti, A.: 3D embryogenesis image segmentation by the generalized subjective surface method using the finite volume technique. In: Eymard, R., Herard, J.M. (eds.) Finite Volumes for Complex Applications V: Problems and Perspectives, ISTE and WILEY, London, pp. 585–592 (2008) 9. Mikula, K., Sarti, A., Sgallari, F.: Co-volume level set method in subjective surface based medical image segmentation. In: Suri, J., et al. (eds.) Handbook of Medical Image Analysis: Segmentation and Registration Models, pp. 583–626. Springer, New York (2005) 10. Sarti, A., Malladi, R., Sethian, J.A.: Subjective Surfaces: A Method for Completing Missing Boundaries. Proceedings of the National Academy of Sciences of the United States of America 12(97), 6258–6263 (2000) 11. Weickert, J.: Coherence-enhancing diffusion filtering. Int. J. Comput. Vision 31, 111–127 (1999)

Composed Segmentation of Tubular Structures by an Anisotropic PDE Model Elena Franchini, Serena Morigi, and Fiorella Sgallari Department of Mathematics-CIRAM, University of Bologna, Bologna, Italy {franchini,morigi,sgallari}@dm.unibo.it

Abstract. In this work we introduce the composed segmentation (Csegmentation), that is a priori composition of sources to obtain a single one segmentation result according to specific logic combinations. The approach and the segmentation model are general but we apply the C-segmentation technique to the challenging problem of segmenting tubular-like structures. The reconstruction is obtained by continuously deforming an initial distance function following the Partial Differential Equation (PDE)-based diffusion model derived from a minimal volumelike variational formulation. The gradient flow for this functional leads to a nonlinear curvature motion model. An anisotropic variant is provided which includes a diffusion tensor aimed to follow the tube geometry. Numerical examples demonstrate the ability of the proposed method to produce high quality 2D/3D segmentations of complex and eventually incomplete synthetic and real data.

1

Introduction

Segmentation of three-dimensional (3D) images can be a very useful computer aided diagnosis tool for clinical routines or surgical planning. We use the term composed segmentation for systems that extract structures from several images, by combining them according to specific Boolean operations. Traditionally, the segmentation process independently performed on single images have to be combined by cumbersome algorithms. The goal of C-segmentation is to combine complementary multispatial, multisensor, multitemporal and/or multiview information into one new domain containing only the information to be segmented. The term composed means by Boolean operations which depends on the application requirements. The individual images entering the C-segmentation process need to be registered to a common frame of reference, this is a nontrivial task which could affect the robustness of the segmentation approach, but it is not addressed in this work. We assume the input images have been preliminary registered. Let us illustrate the role of C-segmentation in different applications. Multimodal fusion deals with images that capture different physical properties of the original scene. In this case, C-segmentation identifies and segments the union of regions of interest. Multispatial fusion is related to several images which cover a single one scene, for example several aerial photographs to represent an entire territorial region, or multiple CT scans to reconstruct a human organ. The X.-C. Tai et al. (Eds.): SSVM 2009, LNCS 5567, pp. 75–86, 2009. c Springer-Verlag Berlin Heidelberg 2009 

76

E. Franchini, S. Morigi, and F. Sgallari

C-segmentation unifies all the information, eventually replicated in the multiple sources, into a single one segmented structure. Multitemporal composition requires the comparison between images representing the same structures acquired at different timing. For example, in medical analysis, a tumor-region growing is monitored by subsequently images of the region of interest. The C-segmentation can identify and segment the difference between structures in two images, reconstructing the grown area. While our segmentation methodology is quite general, we focus our attention on the most challenging problem of tubular-like segmentation which is particularly difficult in case of multiple sources due to the huge amount of connected structures that should be reconstructed. In particular, we will consider applications in medical image analysis which are interested in the extraction of anatomical surfaces of tubular structures like blood vessels. Indeed, problems like aneurysm or stenosis can occur in a vessel, and the clinicians need tools to help them in interpreting and quantifying the images for evaluating the pathology, for proposing a therapy or a surgical operation, for planning minimally invasive treatment. A number of deformable model-based approaches for vessel segmentation or, generally tube-like structures, have received considerable attention and success. We refer the reader to [4] for an extended review on vessel segmentation algorithms. Since explicit deformable model representation is usually impractical, level set techniques to evolve a deformable model have been recently introduced, which provide implicit representation of a deformable model. A curve in 2D or a surface in 3D evolves in such a way as to cover a complex shape or structure. Its initialization can be either manual or automatic and it needs not to be close to the desired solution. A disadvantage of level sets segmentation approach is the computational effort required to cover the entire domain of interest which is, in general, one dimension higher than the original one. Interested readers are referred to recent literature on the level set segmentation strategy for tubular structures [5], [6], [7], [8], [10]. A generalization of the single-channel active contour without edges model is proposed in [9] for object detection using logic operations. This logic framework suffers from the active contour model limits, and is not suitable for detecting tubular structures. In this work, we modify a geometric deformable model segmentation procedure based on level sets [2], to obtain a fast and accurate method for solving the C-segmentation problem to extract tubular structures from multiple 2D/3D images, and we apply the proposed segmentation method for segmenting blood vessels, neurovascular structures and similar characteristics medical images. The main contributions of this work concern the design of a strategy to deal with directionality in the vessels based on a diffusion tensor, and the capability to compose segmentation of multiple images according to Boolean operations. The former makes the segmentation algorithm able to follow tubular structures and connect eventually disconnected parts, while the latter let simultaneously combine different information into a robust segmentation method. The proposed method is able to segment twisted, convoluted, and occluded structures without

Composed Segmentation of Tubular Structures

77

the user interactivity, following branching of different layers, from thinner to larger structures. One of the major disadvantages of the geometric deformable models, that is the computational cost, is strongly reduced by the proposed numerical approach which limits the dimensions of the linear systems involved in the solution. The paper is organized as follows. The non-linear PDE model for tubular structures segmentation is introduced in Section 2, and numerical aspects related to the discretization of the PDE model are discussed in Section 3. The Csegmentation algorithm is discussed in details in Section 4, and the anisotropic variant of the segmentation model is introduced in Section 5. Synthetic as well as real tests are provided in Section 6. Some selected 3D examples are also presented in Section 6 to demonstrate the effectiveness of this technique for automatic segmentation of bloods vessels in volumetric MRA/CTA images. Section 7 contains concluding remarks.

2

A Segmentation Model for Tubular Structures

Several recently proposed 3D segmentation methods are based on deformable models, which can naturally capture the physics and geometry of shapes varying in space and time. In this section we formulate the segmentation problem as a special deformation of a 3D manifold driven by the structures we want to recover. Classical segmentation problems show oversmoothed structures and eventually uncomplete boundaries and the surface evolution usually flows over the boundaries of longer and thinner objects when propagating. A common choice to detect structure boundaries and to drive diffusion or segmentation process is the Perona-Malik diffusivity g(s) = 1/(1 + (s/ρ)2 ),

(1)

where ρ > 0 is a small positive constant. For the implicit representation of the segmented surface, we consider a special 3D manifold which is the graph of a trivariate function φ mapping an open set Ω ⊂ R3 into R. The problem of determining the surface that best fits the object boundary represented in a 3D image I, can be posed as a volume minimization problem with objective function   Vg := g(∇I)dV dV = 1 + ∇φ2 dxdydz, (2) Ω

where the metric g is defined by (1) in Ω and Vg represents the weighted volume of a 3D manifold on Ω. The volume functional (2) can be minimized and according to the steepest descent, reading ε = 1, we have   ∇φ ∂φ  2 2 = ε + ∇φ ∇. g(∇I)  , (3) ∂t ε2 + ∇φ2

78

E. Franchini, S. Morigi, and F. Sgallari

or, equivalently, in advection-diffusion form   ∂φ  2 ∇φ 2 = ε + ∇φ g(∇I)∇.  + ∇g · ∇φ. ∂t ε2 + ∇φ2

(4)

The PDE model (4), ε = 1, represents the mean curvature motion of the 3D manifold in 4D space with metric g. The metric g in (4) is the edge function appropriately chosen so that the object boundaries act as attractors under a particular flow. This term allows us to extract sharp features, such as edges, corners, spikes, and to accelerate the deformation of the initial function. In the evolution of φ according (4) the 3D manifold assumes constant values for most regions far from the boundaries. The first term in (4) corresponds to a minimal volume regularization weighted by the function g, while the second term corresponds to the attraction to the image edges. The advection term in equation (4) introduces a driving force which moves the level surfaces towards the object boundaries. Equation (3) in case ε ∈ (0, 1] is proposed in [2] for dealing with the boundary completion problem. The variability in the parameter ε, ε ∈ (0, 1], provides both a regularization effect and a hole filling strategy. The effect of the parameter ε is to segment boundaries which are eventually uncompleted due to, for example, noise or corruptions in the acquisition phase. However, this does not help in the reconstruction of slightly disconnected tubular structures. The latter problem is solved by the introduction of a suitable diffusion tensor, which is discussed in Section 5. The starting initial function φ0 is usually a problem, since it involves user interaction for locating some starting points at one particular recognizable part of the structure to be segmented inside the 3D image. This is overcome by our method which automatically initialize the surface evolution using a suitably designed distance function, as described in Section 4. We can adapt the PDE model (3) to compute Boolean operations between implicit surfaces M1 and M2 . This can be carried out quite easily, using the min, max tools on the related signed distance functions dist1 (x) and dist2 (x). In fact the union, intersection and differences between two surfaces can be obtained applying the evolving PDE (3) initialized by (9) in Section 4, where dist(x) is defined respectively by dist(x) = min{dist1 (x), dist2 (x)}, union dist(x) = max{dist1 (x), dist2 (x)} intersection dist(x) = max{dist1 (x), −dist2 (x)} difference(M1 − M2 ) dist(x) = max{−dist1 (x), dist2 (x)} difference(M2 − M1 ).

3

(5)

Solving the PDE Model

The computational method for solving (3) is based on an efficient semi-implicit co-volume scheme as suggested in [2]. The semi-implicit in time discretization is obtained by treating the nonlinear terms of the equation from the previous

Composed Segmentation of Tubular Structures

79

time step while the linear ones are considered at the current time level. Timediscretization of (3) by Euler’s method yields the following semi-discrete scheme Let τ be a uniform discrete time step, φ0 be a given initial function. Then, for every discrete time step tn = nτ , n = 1, . . . N , we look for a function φn , solution of the equation   1 φn − φn−1 ∇φn 0  = ∇. g  , (6) τ ε2 + ∇φn−1 2 ε2 + ∇φn−1 2 where g 0 := g(∇I). The computational domain is obtained through Ω-decomposition into cubic cells and the construction of a co-volume mesh using a complementary 3D tetrahedral grid. Following the classical finite volume methodology we integrate (6) over every co-volume p, p = 1, . . . , M and according to the details explained in [2] we get at the time step n, a system of linear equations which can be written in matrix-vector form as AΦn = b,

(7)

where A ∈ RM×M is the coefficient matrix, which is symmetric and diagonally dominant M-matrix, and Φn = (φ1 , . . . , φM ) is the vector solution. Since the unknown function φ(x, t) evolves only on nodes sufficiently close to the structure boundary, we speed up the computation by determining the updated values for φ(x, t) only for the nodes identified by initial function φ0 (x) > η, for a given small positive threshold η. In case of vessel structures, for example, this means a significant reduction of the computational effort required since the number of nodes representing the vessels is small compared with the dimension of the entire 3D image which contains them. In practice, at each time step, the number of unknowns of the linear system (7) is significatively reduced, and thus both the storage and the computational cost are much lower. Since at each row of A corresponds a node in Ω, if we consider a limited number of nodes M1 << M , we get a linear system with a sparse coefficient matrix of rank M1 , which contains M − M1 zero rows and columns. It is easy to verify that if we first apply a suitable permutation of rows and columns of A , and corresponding elements of b, we get a linear  ∈ RM1 ×M1 with full rank which has the same system with a coefficient matrix A properties as A, that is, it’s symmetric and positive definite. In a similar way, the same permutation applied to the components of the right-hand side vector b, leads to a vector b ∈ RM1 . Therefore, instead of solving the linear system (7) which involves a M × M coefficient matrix, we can apply the preconditioned conjugate gradient iterative method, using diagonal scaling as preconditioner,  of the linear system for computing the solution Φ  = b, AΦ with a negligible approximation error on the solution.

(8)

80

E. Franchini, S. Morigi, and F. Sgallari

4

The C-Segmentation Algorithm

The C-segmentation procedure consists of two steps: the partitioning phase driven by the Boolean operations to obtain an initial function, and the segmentation phase. The case of segmentation of tubular structures from a single input image is trivially a simplification of the proposed C-segmentation method. In this section, we discuss the implementation of the PDE model (3) for segmentation of multiple 3D images. Let Ω be a common support. Each gray-level 3D image i containing tubular structures of interest is thresholded to obtain a rough segmentation estimate which is used to generate an initial distance function. The thresholding process consists of making a binary image Ii from the input i image based on a user threshold. Since the images can be represented by a real-valued function defined on a region Ω ⊂ R3 or by the discretization of such a function, the threshold process let us define in a natural way the sub-domain Ωi ⊂ Ω, Ωi := {x ∈ Ω : Ii (x) = 1}. The choice of the threshold value is not an issue due to the fact that in practical cases the structures to be segmented are characterized by a particular intensity value represented by a given gray level. For example, vascular system containing a contrast fluid, can be identified as the brightest formations in the volume. As it will be better illustrated by the examples in Section 6 the 3D mask Ii resulting from the pre-computed binarization, in general, preserves the largest tubular structures while breaks up into small pieces the thinnest ones. These structures will be perfectly recover and reconnected by the surface evolution process. Each image i has an associated set (region) Ωi . Depending on the type of the fusion task {Ωi }i=1,··· ,S , where S is the number of input images, need not be exactly disjoint and/or cover the whole region Ω. The composed signed distance function dist(x) is then obtained by the signed distance functions disti (x), i = 1, .., S, with respect to Ωi , following the rules (5). We define the initial function φ0 as follows  1 if dist(x) < 0 . (9) φ0 (x) = dist(x) (1 − max{dist(x)} ) else The implicit function φ0 represents the initial surface which continuously evolves, by following equation (3), towards the boundaries of the tubular structures. The evolution is stopped when a change of solution in time (in L2 -norm) is less than a prescribed tolerance. A critical issue is the definition of the composed diffusion function g(·) in (3) when the C-segmentation considers several input images. For example, in case of difference between two images, which are characterized by g 1 and g 2 , the composed diffusion function g 0 used in (3) is given by if ((g 1 ≤ )and(g 2 ≤ ) or (g 1 ≥ )and(g 2 ≥ )) g 0 = 1 else g 0 = min(g 1 , g 2 ),

(10)

with > 0, we used = 1 · 10−3 in the computational examples. We can proceed similarly for other Boolean operations.

Composed Segmentation of Tubular Structures

81

Finally, the reconstructed surface is obtained from the implicit surface φ as the zero level set of the function φ(x) − s, that is the s-level set of φ: {x ∈ Ω : φ(x) = s}, where s = (max(Φ) + min(Φ))/2. This is motivated by the fact that the flow driven by (3) forms a sharp step in the proximity of the object boundaries, while it approaches at constant values inside/outside the object.

5

An Anisotropic Variant of the Segmentation Algorithm

In this section we provide a variant of the isotropic model (3) designed to improve significantly the connectivity of the coherent structures in the segmentation. The idea is to incorporate local orientation of the tubular structures into the dynamic segmentation process in such a way that at each time step the surface evolves by isotropic mean curvature motion in homogeneous regions, while it is driven by the directional field representing the orientation of the tube in presence of tubular structures. We aim to capture the vessel’s structure and the vessels directions locally by a local spatial coherence descriptor. Coherence enhancing image smoothing has been introduced by [3] and successfully applied in image filtering by anisotropic diffusion. This type of nonlinear diffusion includes the construction of a diffusion tensor which is built as follows. Given an image I, and its Gaussian-smoothed version ∇Iσ , a regularized shape descriptor is provided by Jδ (∇Iσ ) := (Kδ ∗ (∇Iσ ⊗ ∇Iσ )) (11) where Kδ is a Gaussian kernel with δ ≥ 0. The matrix Jδ is symmetric positive semi-definite and its eigenvalues μ1 ≥ μ2 integrate the variation of the gray values within a neighborhood of size O(δ). They describe the average contrast in the corresponding eigendirections v1 and v2 . The orientation of the eigenvector v2 , corresponding to the smaller eigenvalue, represents the direction of lowest fluctuations, the so-called coherence orientation. In this way, constant areas are characterized by μ1 = μ2 = 0, while straight edges give μ1 μ2 = 0. The normalized coherence value which measures the anisotropic structures within a window of scale δ is thus defined as c=

(μ1 − μ2 )2 , max{(μ1 − μ2 )2 }

c ∈ [0, 1].

(12)

Thus c approaches to 1, for anisotropic structures and tends to zero for isotropic structures. The diffusion tensor D is a matrix with the same eigenvectors as the (regularized) structure tensor Jδ and its eigenvalues are given by λ1 = g(∇I)  g(∇I) λ2 = g(∇I) + (1 − g(∇I))e−κ/c

if μ1 = μ2 κ > 0, else

(13)

where g(·) is the composed diffusion function, defined for example, by (10) which suitably adapts its values to the anisotropy. The parameter κ has the role of a

82

E. Franchini, S. Morigi, and F. Sgallari

threshold, and c is the coherence defined in (12). Therefore, the matrix D has the following form

T

v1 λ 0 . (14) D = [ v1 v2 ] 1 0 λ2 v2T In local homogeneous areas of an image the diffusion is reduced to be the isotropic mean curvature motion driven by (3), in fact we have D = g(∇I). Areas nearby elongated structures are characterized by values of g(·) approaching to zero, which gives λ2 >> λ1 . The effect on the diffusion of the segmentation function is thus stronger along the coherence directions. In all the experiments reported in Section 6 we set κ = 1 · 10−5 . Since if c >> κ then λ2 ≈ 1, while if c << κ then λ2 ≈ g(∇I), the segmentation function flows along the coherence direction when it approaches to the edges and stops when the object boundary is reached. The incorporation of the diffusion tensor D defined by (14) in the segmentation model (3) leads to the following nonlinear anisotropic segmentation   ∂φ  2 ∇φ = ε + ∇φ2 ∇. D  , φ(x, 0) = φ0 (x) x ∈ Ω. (15) ∂t ε2 + ∇φ2 We will refer to the models (3) and (15) by isotropic and anisotropic segmentation models, respectively. The PDE model (15) can be easily extended to 3D or 4D segmentation problems; we refer the reader to [1] for the numerical aspects involved in the discretization of the diffusion tensor.

6

Results and Experiments

We tested the performance of the proposed C-segmentation method on 2D/3D synthetic and real examples. Example 1 and 2 demonstrate the performance of the anisotropic C-segmentation model compared with the isotropic one to verify the important role of the diffusion tensor in the segmentation of tubular structures. Examples 3, 4 and 5 illustrate results from the application of composed segmentation of 2D/3D images. For all the experiments, we apply C-segmentation algorithm as illustrated in Section 4, with or without diffusion tensor, and we set the time step parameters involved to τ = 1 · 10−3 , for 3D images and τ = 1 · 10−2 for the 2D images, and ε = 1 · 10−3 in (3). We stop the iterations when the change of solution in time is less than 1 · 10−4. In our experiments, the segmentation of images with high contrast (see Examples 2 and 3) provides good results when an automatic choice for parameter s is chosen such as s = (max(Φ) + min(Φ))/2, while in case of images like in Examples 1, 4 and 5, the results are more sensitive to the choice of s, and visual inspection could be required to tune the automatic choice of s. Example 1. In the first 2D example we consider the segmentation of a carotid vascular system represented in a Magnetic Resonance Angiography (MRA) image of 182 × 182 pixels shown in Fig. 1 (left). MRA is based on detection of signals from flowing blood and suppression of signals from other, static, tissues.

Composed Segmentation of Tubular Structures

83

Fig. 1. Carotid vascular system segmentation; the original image (left); results of the isotropic model (center), and anisotropic model (right)

Fig. 2. First row: original synthetic image (left); results obtained by the isotropic (left) and anisotropic segmentation model (right). Second row: associated segmentation function φ is shown at the final time step of isotropic (left) and anisotropic segmentation (right).

The blood vessels appear as high intensity regions in the image. The structures to be segmented represent several vessels of variable diameters which are close to each other, partial occlusions and intersections make the segmentation very challenging. In Fig.1 the segmented structures obtained by 10 iterations of the isotropic (center) and anisotropic (right) segmentation models are displayed. The boundary curves in pink color are extracted using contour values s = 0.90 and s = 0.98, respectively. Visual comparison shows the anisotropic segmentation method to give the most accurate restoration. Example 2. The second example illustrates the ability to reconstruct structures which present small occlusions along the coherence direction. The synthetic image to be segmented of dimension 200 × 200 pixels is shown in Fig. 2 (first row, left). Applying 10 time steps of the anisotropic segmentation model, the structure is well reconstructed while maintaining the narrowing, as shown in Fig.2

84

E. Franchini, S. Morigi, and F. Sgallari

(a)

(b)

(c)

(d)

Fig. 3. C-segmentation of images (a) and (b). Result in (c) of (b)\(a) and in (d) of (a)\(b).

(first row, right). The propagation driven by the isotropic segmentation model leads to enhance the disconnections, as shown in Fig.2 (first row, center). The boundary curves determined as iso-contours of the segmentation functions shown in Fig.2 (second row), using s = 0.95 are over-imposed on the original image and shown in Fig.2 (first row). Example 3. Verification of the proposed C-segmentation method is carried out on a real echo image show in Fig.3 (a) where we generated one synthetic bump and two synthetic holes, simulating aneurysm and stenosis effects, as illustrated in Fig.3 (b). Applying C-segmentation to the Boolean difference (b) \ (a) we are able to enhance the aneurysm as shown in Fig.3 (c), while to detect the stenosis effects we apply C-segmentation to the Boolean difference (a) \ (b), obtaining the results illustrated in Fig.3 (d). Example 4. Volumetric segmentation is applied to a Computed Tomographic Angiography (CTA) data set shown in Fig.4(left). The volumetric data set kidney of dimension 201 × 201 × 201 has been extracted from a 436 × 436 × 540 CTA image of the kidney vasculature system and present vessel patterns with different curvatures, diameters and bifurcations. The final segmentation obtained after 10 time steps of the segmentation algorithm is shown in Fig.4(right). Example 5. The last experiment demonstrates the performance of the proposed technique to segment a complex structure extracted from two overlapping 3D

Composed Segmentation of Tubular Structures

85

Fig. 4. Segmentation of the kidney volume data set: (left) original CTA image (right) segmentation result

Fig. 5. Segmentation of the lung volume data set: (left) original MRA image, (right) C-segmentation result

MRA images representing the airway tree of a human lung. The original MRA image is illustrated in Fig.5(left) using volume rendering. In particular the ellipsoidal area outlined in the image includes the region of interest for segmentation inside the lung image. The data set lung is DICOM format and sample images available on the web site (http://www.osirix-viewer.com/Downloads.html). We simulated the acquisition of two partially overlapping 3D MRA images covering the area of interest consisting of 156 × 156 × 156 voxels each, and we applied the C-segmentation algorithm using union Boolean composition. Fig.5(right) shows the obtained single one segmentation of the human airway tree, where several branching generations were detected. These preliminary results demonstrate the applicability of the developed method for the C-segmentation of quite different and topological complex tubular structures.

86

7

E. Franchini, S. Morigi, and F. Sgallari

Conclusions

In this paper we introduced the C-segmentation method based on the distance function which is successfully applied to 2D/3D images containing tubular structures. The algorithm is automatic, accurate and fast. The latter is due to a speed up strategy in the iterative method for linear systems. The algorithm is provided with a diffusion tensor to move the evolving surface toward elongated tubular structures, connecting gaps in the underlying raw data, while keeping the structures distinguished along the coherence direction. This model defines a general segmentation framework which combines object information from different images into any logical combination, rather than following the difficult a posteriori process to compose segmentation results obtained separately.

Acknowledgments This work has been supported by PRIN-MIUR-Cofin 2006, project and by University of Bologna "Funds for selected research topics".

References 1. Drblikova, O., Mikula, K.: Semi-implicit Diamond-cell Finite volume Scheme for 3D Nonlinear Tensor Diffusion in Coherence Enhancing Image Filtering. In: Eymard, R., Herard, J.M. (eds.) Finite Volumes for Complex Applications V: Problems and Perspectives, ISTE and WILEY, London, pp. 343–350 (2008) 2. Corsaro, S., Mikula, K., Sarti, A., Sgallari, F.: Semi-implicit covolume method in 3D image segmentation. SIAM J. Sci. Comput. 28(6), 2248–2265 (2006) 3. Weickert, J., Scharr, H.: A scheme for coherence enhancing diffusion filtering with optimized rotation invariance. Journal of Visual Communication and Image Representation 13(1/2), 103–118 (2002) 4. Kirbas, C., Quek, F.: A review of vessel extraction techniques and algorithms. ACM Computing Surveys 36(2), 81–121 (2004) 5. Hassan, H., Farag, A.A.: Cerebrovascular segmentation for MRA data using levels set. International Congress Series, vol. 1256, pp. 246–252 (2003) 6. Scherl, H., et al.: Semi automatic level set segmentation and stenosis quatification of internal carotid artery in 3D CTA data sets. Medical Image Analysis 11, 21–34 (2007) 7. Cohen, L.D., Deschamps, T.: Segmentation of 3D tubular objects with adaptive front propagation and minimal tree extraction for 3D medical imaging. Computer Methods in Biomechanics and Biomedical Engineering 10(4), 289–305 (2007) 8. Gooya, A., Liao, H., et al.: A variational method for geometric regularization of vascular segmentation in medical images. IEEE Transaction on image processing 17, 1295–1312 (2008) 9. Sandberg, B., Chan, T.F.: A logic framework for active contours on multi-channel images. J. Vis. Commun. Image R. 16, 333–358 (2005) 10. Westin, C.-F., Lorigo, L.M., Faugeras, O.D., Grimson, W.E.L., Dawson, S., Norbash, A., Kikinis, R.: Segmentation by Adaptive Geodesic Active Contours. In: Delp, S.L., DiGoia, A.M., Jaramaz, B. (eds.) MICCAI 2000. LNCS, vol. 1935, pp. 266–275. Springer, Heidelberg (2000)

Extrapolation of Vector Fields Using the Infinity Laplacian and with Applications to Image Segmentation Laurence Guillot1 and Carole Le Guyader2 1 Laboratoire Jean-Alexandre Dieudonné, UMR 6621, Université de Nice Sophia Antipolis – CNRS, Faculté des Sciences Parc Valrose, 06108 – Nice Cedex 02, France [email protected] 2 IRMAR, UMR CNRS 6625, Institut des Sciences Appliquées de Rennes, 20, Avenue des Buttes de Coësmes, CS 14315, 35043 Rennes Cedex, France [email protected]

Abstract. In this paper, we investigate a new Gradient-Vector-Flow (GVF)( [38])-inspired static external force field for active contour models, deriving from the edge map of a given image and allowing to increase the capture range. Contrary to prior related works, we reduce the number of unknowns to a single one v by assuming that the expected vector field is the gradient field of a scalar function. The model is phrased in terms of a functional minimization problem comprising a data fidelity term and a regularizer based on the super norm of Dv. The minimization is achieved by solving a second order singular degenerate parabolic equation. A comparison principle as well as the existence/uniqueness of a viscosity solution together with regularity results are established. Experimental results for image segmentation with details of the algorithm are also presented.

1 1.1

Introduction Motivations

Many of the well-known variational segmentation methods require a careful choice of the initial condition. One of the most famous variational methods to process this partition of the image is the active contour model introduced by Kass, Witkin, and Terzopoulos ([30]). It consists in evolving a parameterized curve so that it matches the object boundary. The shape taken by the curve through the process is related to an energy minimization, this energy comprising a data fitting term and a regularizer, and being non-convex. Thereof, we can only expect local minimizers, which, in practice, means that the contour to be deformed must be initialized near the object boundary. Cohen ([18]) has proposed a way to alleviate this constraint by adding an inflating/deflating force in the modelling, defined by kn, n denoting the unit inward normal to the curve and k, a constant. According to the sign of the constant k, the curve inflates X.-C. Tai et al. (Eds.): SSVM 2009, LNCS 5567, pp. 87–99, 2009. c Springer-Verlag Berlin Heidelberg 2009 

88

L. Guillot and C. Le Guyader

or deflates. Thereby, in practice, the contour to be deformed is either initialized inside the object, or it encloses the object of interest. In [38], Xu and Prince address both the problems of initialization and slow and/or poor convergence near boundaries with strong concavities by introducing a new static external force called Gradient Vector Flow (GVF). The initialization constraint is removed, that is, initialization can be made inside, outside or across the object boundaries, and the front evolution is easily handled even in boundary concavities. The main idea behind this model is to increase the capture range of the external edge-map-related force field, and to make the contour evolve toward the desired boundaries, here where classical methods would fail to. Unlike classical active contours, the introduced external force does not derive from a potential function and cannot be computed straightforwardly from the image edge map. More precisely, the model cannot be phrased in terms of a unique functional minimization problem but is defined in two steps. In a first step, the external force (GVF) w = (u, v)T is obtained by minimizing an energy functional in a variational framework. The corresponding Euler-Lagrange equations are computed and lead to solve a linear partial differential equation decoupled system by a gradient descent method. The second step consists then in replacing, in the dynamic snake equation, the classical potential force by the newly computed external force w. This method motivated the following works. In [34], Paragios et al. propose to integrate this boundary spatial diffusion technique to the geodesic active contours ( [16]). In [29], Jifeng et al. propose to improve the diffusion properties of the GVF force field. They obtain a new force by replacing the Laplacian operator used in the GVF model by its diffusion term in the normal direction that is, the normalized infinity Laplacian operator. Unlike the GVF model, their new field (called NGVF for GVF in the normal direction) is anisotropic. Furthermore, the NGVF is stable for bigger time steps, improves slightly segmentation results, and allows to detect more quickly long and thin concavities. Our work is much motivated by [38] and [29]. We wanted to provide, in a rigorous mathematical framework, a new method to generate this external force field. Contrary to these prior works, we propose to reduce the number of unknowns to a single one, by assuming that the sought vector field is the gradient field of a scalar function. Also, the introduced minimization problem contains a data-fitting term related to the original GVF model and a regularizer that penalizes the super norm of the unknown gradient. Thus the problem becomes related to the absolutely minimizing Lipschitz extensions and to the infinity Laplacian. The absolute minimal Lipschitz extension model was introduced by Aronsson in [2] (see also [1, 3, 4]) in the following way. Given Ω ⊂ IRn a bounded, open and connected domain with sufficiently smooth boundary, and b ∈ C(∂Ω), solve inf

u∈W 1,∞ (Ω), u=b

on ∂Ω

DuL∞ (Ω) .

(1)

A minimizer of (1) is called an absolutely minimizing Lipschitz interpolant of b|∂Ω inside Ω. Aronsson proved the existence of an absolute minimal Lipschitz

Extrapolation of Vector Fields Using the Infinity Laplacian

89

extension and Jensen proved the uniqueness. Aronsson also derived the EulerLagrange equation governing the absolute minimizer in the sense of viscosity solutions: Δ∞ u = D2 u(Du, Du) = 0 in Ω. (2) We refer to [5, 10, 20] for more details. The operator Δ∞ is called the infinity Laplacian and solutions of (2) are said to be ∞-harmonic. Jensen proved a comparison principle and an existence/uniqueness result of (2) for Lipschitz continuous boundary data (see [28]). Before depicting our model, we briefly make a non-exhaustive review of some prior works related to AMLE and infinity Laplacian in the field of image processing. 1.2

Prior Related Works

As stressed by Caselles et al., the equation Δ∞ u = D2 u(Du, Du) = 0 was introduced in the field of computer vision as edge detector (see [37], [39]). It earlier appeared in the domain of edge enhancement (see [35]) and served as the basis of Canny edge detection [12]. In [17], Caselles et al. investigate the AMLE and the infinity Laplacian in the field of image processing with applications to the restoration of images. Motivated by prior applications devoted to coding ( [13], [14]), they address the issue of interpolating data given on a set of points and/or curves in the plane. Another application, dedicated to shape metamorphism (the process which consists in evolving a source shape into a target shape by intermediate steps) is proposed by Cong et al. in [19] and makes use of the infinity Laplacian. Also, in [31], Mémoli et al. propose a new framework for brain warping using Minimizing Lipschitz Extensions. To finish, in [24], Elion and Vese aim at solving the (BV, G) decomposition model introduced by Meyer in [32]. In that purpose, Elion and Vese focus on an isotropic decomposition of the image f ≈ u + v with v = ΔP = div(DP ) and DP ∈ (L∞ (Ω))2 . The outline of the paper is as follows: Sect. 2 is devoted to the depiction of the model and the derivation of the associated evolution problem. Section 3 is dedicated to the theoretical study of the obtained parabolic problem. We first prove a comparison principle, then prove existence and uniqueness of a viscosity solution. Regularity results of this solution are also given. We conclude the paper with experimental results and integrate this new external force field in a segmentation problem. Details of the algorithm are also provided.

2

Depiction of the Model

Let Ω be a bounded open subset of IRn , ∂Ω its boundary and let I be a given ¯ → IR. For the purpose of illustration we bounded image function defined by I : Ω consider n = 2. Let g be an edge-detector map. The function g is applied to the norm of the image gradient, and satisfies the following properties: g : [0, ∞[→ [0, ∞[, g(0) = 1, g strictly decreasing, and limr→+∞ g(r) = 0. An example of 1 such a function is g : r  → 1+r 2 . We denote by W = (w1 , w2 ) = −Dg(||DI||) the

90

L. Guillot and C. Le Guyader

associated gradient vector field. In homogeneous regions, ||DI|| 0 so g(||DI||) is almost equal to 1. On boundaries, ||DI|| is large so g(||DI||) is almost zero. Also, in homogeneous regions, W is almost the null vector. Along the boundaries, the vector field W points toward the middle of the edges (see such an example in Fig. 1). We plan to extrapolate the vector field on the whole image domain in a variational framework. A majority of existing regularization functionals aims at minimizing the global variation of the unknown and thus provides little local control. In this work, we propose to minimize the super norm of the unknown gradient. This choice is also motivated by the fact that the Laplacian operator (that naturally appears in the GVF model) can be decomposed into the sum of the second derivative in the normal direction, and the second derivative in the tangent direction. The former component that is kept in the NGVF model weighs heavily in the extrapolation process and has good properties unlike the later component which proves to be parasitic particularly when dealing with thin and long concavities. Also, unlike prior related works, we reduce the number of unknowns to a single one by assuming that the expected vector field is the gradient vector field of a scalar function. We thus propose to minimize the following functional:  inf ||Dv − W ||2 ||W ||2 dx + μ ||Dv||L∞ (Ω) , (3) 1,∞ v∈W

(Ω)

Ω

where μ > 0 is a tuning parameter. Remark 1. Functional (3) is defined on W 1,∞ (Ω). The domain Ω being bounded, the inclusion L∞ (Ω) ⊂ L2 (Ω) holds so Dv ∈ L2 (Ω). Remark 2. If v is a minimizer of (3), so is v + C where C denotes any real constant. This is not a problem since we are interested in the associated gradient vector field. If v ∈ W 1,∞ (Ω), v is Lipschitz continuous and thus, by Rademacher’s theorem, differentiable almost everywhere. To minimize the above energy, we make use of the absolutely minimizing Lipschitz extensions. Following the results on AMLE recalled in Sect. 1, we obtain the Euler-Lagrange equation satisfied by v if it minimizes (3) and solve it by gradient descent. More precisely, classically, in image processing, the equation is defined on a domain R of IR2 (e.g., on the square [0, 1] × [0, 1]). In this case, boundary conditions must be defined: Neumann boundary conditions on ∂R are well-suited to the image processing framework since it corresponds to the reflection of the data through the edges. Thus it is no longer necessary to define boundary values. Following [6] and [15], we propose to simplify the problem by working with periodic solutions. The function v, primarily defined on [0, 1] × [0, 1], is extended to IR2 . First, by symmetry, we extend it to [−1, 1] × [−1, 1] and then in all of IR2 by periodicity (see Sect. 3.3.1 from [6]). We thus obtain that ∀h ∈ ZZ 2 , ∀x ∈ IR2 ,

Extrapolation of Vector Fields Using the Infinity Laplacian

91

v(x + 2h) = v(x). Also, we assume that the initial condition v0 and the functions x → wk (x), k = 1, 2 are extended to IR2 with the same periodicity. Given T > 0, we then obtain the following problem: ⎧   ∂v ⎪ ⎪ = 2 ||W ||2 Δv + 2  D||W ||2 , Dv − 2 div ||W ||2 W ⎪ ⎪ ∂t

⎪ ⎪ Dv Dv ⎪ 2 ⎨ , on IR2 × (0, T ), + μD v |Dv| |Dv| (4)

⎪ Dv Dv ⎪ ⎪ ⎪ , , = b(x)Δv −  d(x), Dv − h(x) + μ D 2 v ⎪ ⎪ |Dv| |Dv| ⎪ ⎩ 2 v(x, t = 0) = v0 (x) in IR ,   with b : x  → 2||W (x)||2 , d : x  → −2 D||W ||2 (x), h : x  → 2 div ||W ||2 W (x), and with the assumptions v0 ∈ C(IR2 ) ∩ W 1,∞ (IR2 ), b ∈ C(IR2 ) and bounded by ξb , d ∈ C(IR2 ) ∩ W 1,∞ (IR2 ), bounded by ξd and with Lipschitz constant κd , h ∈ C(IR2 ) ∩ W 1,∞ (IR2 ), bounded by ξh and with Lipschitz constant κh , and with ·, · denoting the euclidean scalar product in IR2 . We also assume that the mapping IR2  x  → b1/2 (x) is Lipschitz continuous on IR2 with Lipschitz constant κb1/2 .

3

Theoretical Results

This problem falls within the framework of the theory of viscosity solutions. Indeed, we obtain a second order singular degenerate parabolic equation. The concept of viscosity solutions has been introduced in 1981 by Crandall and Lions ( [22]). This theory was developed to study first-order partial differential equations of nondivergence form, typically, Hamilton-Jacobi equations. Later, the study of viscosity solutions was extended to second-order elliptic and parabolic equations (for a good introduction to the theory of viscosity solutions, we refer to Barles [8, 7], the article of Crandall, Ishii and Lions [21], Crandall, Lions [23], Ishii [26], and Ishii, Lions [27]). We also refer to the related work [9]. In our problem, the evolution equation in (4) can be rewritten in the form:   ∂v + G x, Dv, D2 v = 0, ∂t with G : IR2 × IR2 − {0IR2 } × S 2 (S 2 being the set of symmetric 2 × 2 matrices equipped with its natural partial order) defined by: p pT X , |p| |p| 

p p = d(x), p + h(x) − b(x) trace (X) − μ trace X , |p|2

G(x, p, X) = d(x), p + h(x) − b(x) trace (X) − μ

= c(x, p) + E (x, X) + F (p, X) , with the following properties:

92

L. Guillot and C. Le Guyader

 p p X 2 |p|

and E : (x, X)  →

if X ≤ Y then F (p, X) ≥ F (p, Y ).

(5)

1. The operators G, F : (p, X)  → −μ trace



−b(x) trace (X) are independent of v and are elliptic, i.e., ∀X, Y ∈ S 2 , ∀p ∈ IR2 ,

The operators G, E, and F are therefore proper. 2. F is locally bounded on IR2 × S 2 , continuous on IR2 \ {0IR2 } × S 2 , and F ∗ (0, 0) = F∗ (0, 0) = 0,

(6)

where F ∗ (resp. F∗ ) is the upper semicontinuous (usc) envelope (resp. lower semicontinuous (lsc) envelope) of F . 3. c : IR2 × IR2  (x, p)  → d(x), p + h(x) is locally Lipschitz continuous in space and ∀x, y ∈ IR2 × IR2 , |c(x, p) − c(y, p)| ≤ (κd |p| + κh ) |x − y|.

(7)

We start by proving a comparison principle that will be useful to prove the uniqueness of the viscosity solution of the considered problem. Theorem 1 (Comparison principle). Let u ∈ U SC(IR2 × [0, T )), bounded, periodic (with the same periodicity as the initial condition of (4)), be a subsolution and v ∈ LSC(IR2 × [0, T )), bounded, periodic (with the same periodicity as the initial condition of (4)), be a supersolution of (4). Assume that u0 (x) = u(x, 0) ≤ v0 (x) = v(x, 0) in IR2 , then u ≤ v in IR2 × [0, T ). Proof. This proof is rather classical. We follow the arguments of [21]. We first observe that for λ > 0, u ˜ = u − T λ−t is also a subsolution of (4) and u ˜t + G∗ (x, D˜ u, D 2 u ˜) ≤ −

λ λ ≤ − 2. 2 (T − t) T

Since u ≤ v follows from u ˜ ≤ v in the limit λ → 0, it will simply suffice to prove the comparison under the additional assumptions: ⎧ ⎨ (i) u + G x, Du, D2 u ≤ − λ . t ∗ T2 (8) ⎩ (ii) lim u(x, t) = −∞. t→T

Let us set M = supIR2 ×[0,T ) u(x, t) − v(x, t). We aim to show that M ≤ 0. In this purpose, we argue by contradiction and assume that M > 0. We introduce the duplication function f (x, − (4ε)−1|x − y|4 and consider  y, t) = u(x, t) − v(y, t) −1 M0 = supIR2 ×IR2 ×[0,T ) u(x, t) − v(y, t) − (4ε) |x − y|4 , ε > 0. Obviously, M0 ≥ M > 0. Moreover, this supremum is reached owing to the bound above of u and −v, the fact that f is such that ∀h ∈ ZZ 2 , f (x + 2h, y + 2h, t) = f (x, y, t), and (8)(ii). We denote by (x0 , y0 , t0 ) ∈ IR2 × IR2 × [0, T ) a point of maximum. We first prove that t0 > 0 for ε small enough and then rise a contradiction using Th. 8.3 from [21], which allows to conclude that M ≤ 0. Consequently, u ≤ v in IR2 × [0, T ).  

Extrapolation of Vector Fields Using the Infinity Laplacian

93

We now give an existence result using the classic Perron’s method (see Sect. 4 from [21]). We start by constructing a subsolution U − . Let us set U − = inf IR2 (v0 ) − Ct with C = ξh . U − is twice differentiable in space, once differentiable in time, bounded, and periodic with the same periodicity as v0 and U − is a subsolution of (4). Similarly, U + = supIR2 (v0 ) + Ct is a supersolution of (4). Obviously, U − (x, 0) ≤ U + (x, 0). We can define: v = sup {w; w periodic with the same periodicity as v0 , subsolution such that U − ≤ w ≤ U + } . In that case, Perron’s method states that v is a periodic discontinuous solution of (4) with the same periodicity as v0 . Clearly, the solution is bounded since U + is bounded. Also as v is a solution, v ∗ is a subsolution and v∗ a supersolution so from the comparison principle v ∗ ≤ v∗ . But v∗ ≤ v ∗ so v ∗ = v∗ = v, which gives that v is continuous on IR2 × [0, T ). Conclusion 1. We have proved the existence and uniqueness of a bounded, periodic, continuous on IR2 × [0, T ) viscosity solution of (4). We now prove that a solution of (4) is Lipschitz continuous in space, and uniformly continuous in time. Theorem 2 (Regularity results). Let us assume that ||Dv0 ||L∞ (IR2 ) ≤ B0 with B0 > 0. Then the solution of (4) satisfies: ||Dv(·, t)||L∞ (IR2 ) ≤ B(t), αt

with B(t) = κh e

−1 α

+ B0 eαt , and with α = 8κ2b1/2 + κd .

Proof. The function v is bounded, continuous on IR2 × [0, T ), and periodic with 1  the same periodicity as v0 . We set Φε (x, y, t) = B(t) |x − y|2 + ε2 2 and aim at proving that v(x, t) − v(y, t) ≤ Φε (x, y, t). Let us set M = sup(x,y)∈IR2 ×IR2 , t∈[0,T ) (v(x, t) − v(y, t) − Φε (x, y, t)). We thus aim to show that M ≤ 0. Once again, we argue by contradiction and assume that M > 0. So we conclude that v(x, t) − v(y, t) ≤ Φε (x, y, t) and letting ε tend to 0, one obtains: v(x, t) − v(y, t) ≤ B(t)|x − y|. Exchanging x and y yields: |v(x, t) − v(y, t)| ≤ B(t)|x − y|.

 

Theorem 3 (Regularity results). The solution v is uniformly continuous in time.

94

L. Guillot and C. Le Guyader

Proof. We proceed like in [25]. In a first time, we assume that v0 is bounded, periodic, C 2 , and such that there exists C, ||Dv0 ||L∞ (IR2 ) , ||D2 v0 ||L∞ (IR2 ) ≤ C. Let us set   C1 = sup ζ + E(x, D2 v0 ) + F∗ (Dv0 , D2 v0 ), ζ − E(x, D2 v0 ) − F ∗ (Dv0 , D2 v0 ) x∈IR2

with ζ = ξd ||Dv0 ||L∞ (IR2 ) + ξh . Let us also set v − = v0 − C1 t and v + = v0 + C1 t. It can be checked that v − is a subsolution of (4) and v + is a supersolution. Then, there exists a unique solution v of (4) and, by the comparison principle, it yields: ∀x ∈ IR2 , ∀t ∈ [0, T ), |v(x, t) − v0 (x)| ≤ C1 t. Letting u(x, t) = v(x, t + h), we obtain that u is the solution of  ∂u + G(x, Du, D2 u) = 0 . ∂t u(x, t = 0) = v(x, h) Classical arguments (comparison principle) allow to conclude that |u(x, t) − v(x, t)| ≤ C1 h, that is |v(x, t + h) − v(x, t)| ≤ C1 h. So v is uniformly continuous in time. Then we assume that v0 is only bounded, periodic and Lipschitz continuous, and use mollification (see Chap. IV from [11] and Sect. 2.5 from [6]). Using the first step of the proof, we obtain the result and the modulus of continuity of v which depends on B0 .   Conclusion 2. We have proved the existence and uniqueness of a viscosity solution of problem (4), bounded, periodic, continuous on IR2 × [0, T ), Lipschitz continuous in space so differentiable almost everywhere, and uniformly continuous in time. We now discretize the evolution equation. In the sequel, we set Ω  x = (x1 , x2 ).

4

Experimental Results

Let Δx1 and Δx2 be the spatial steps, Δt be the time step and (x1i , x2j ) = (iΔx1 , jΔx2 ) be the grid points, 1 ≤ i ≤ M and 1 ≤ j ≤ N . For a function Ψ : Ω → IR, let Ψijn = Ψ (iΔx1 , jΔx2 , nΔt). To discretize (4), we use an explicit finite difference scheme as follows. Also, the problem is complemented by Neumann boundary conditions. For the discretization of the convection component, we refer to [36] (we have used the usual notations for the finite difference operators and the notation d = (d1 , d2 )).   n+1 n n n = vi,j + Δt bi,j Dx1 x1 vi,j + Dx2 x2 vi,j vi,j  x1 n   x1 n  vi,j + min (d1 )i,j , 0 D+ vi,j (9) −Δt max (d1 )i,j , 0 D−

  x2 n   x2 n +max (d2 )i,j , 0 D− vi,j + min (d2 )i,j , 0 D+ vi,j − Δt hi,j +Δt μ

n n 2 n n n n n 2 Dx1 x1 vi,j (Dx1 vi,j ) +2Dx1 vi,j Dx2 vi,j Dx1 ,x2 vi,j +Dx2 x2 vi,j (Dx2 vi,j ) n 2 n 2 (Dx1 vi,j ) +(Dx2 vi,j ) +ε

.

Extrapolation of Vector Fields Using the Infinity Laplacian

95

Fig. 1. On the left, depiction of the initial gradient vector field W = −Dg(||DI||), on the right, the obtained vector field with our proposed approach (μ = 0.05, Δt = 0.1)

Fig. 2. On the left, depiction of the initial gradient vector field W = −Dg(||DI||), on the right, the obtained vector field with our proposed approach (μ = 0.1, Δt = 0.1)

4.1

Numerical Experimentations of Extrapolation

The experiments have been performed on a 2.21 GHz Athlon with 1.00 GB of RAM. In all our experiments, Δx1 = Δx2 = 1. We apply our model to real data and for each test, we provide a view of the initial gradient vector field −Dg(||DI||) and a view of the extrapolated vector field. The initialization was made either by setting v0 ≡ 0, or by setting v0 ≡ −g(||DI||). In all the tests we performed, it does not seem to influence the obtained result. The number of iterations as well as the computational time (order of the second) are similar for the three methods (GVF, NGVF and our proposed approach). Our method qualitatively performs in a way similar to the GVF and the NGVF: we increase the capture range of the vector field and we obtain downward components within the boundary concavity. Nevertheless, contrary to the the GVF and NGVF models, the method requires only one unknown. We start with an image taken from the Image Toolbox of Matlab (Fig. 1), and with an image showing a slice of Tuffeau

96

L. Guillot and C. Le Guyader

Fig. 3. Steps of the segmentation of the synthetic image taken from [34]

Fig. 4. Steps of the segmentation of the image of the brain

Extrapolation of Vector Fields Using the Infinity Laplacian

97

(Fig. 2, Courtesy of ISTO/ESRF). Our proposed approach performs well but seems to be sensitive to the textures of the objects contained in the image. 4.2

Application to Segmentation

This part is dedicated to segmentation and more precisely to the integration of this extrapolated vector field in the geodesic active contour model, in order to alleviate the constraint on the choice of the initial condition. The geodesic active contour model, introduced by Caselles et al. in [16], is cast in the level set setting developed by Osher and Sethian in [33]. We propose, as done in [34], to replace W = −Dg(||DI||) of the geodesic active contour model by the extrapolated vector field obtained with our proposed approach. To illustrate this, we propose an example taken from [34]. It demonstrates that the initial condition can be made of several contours selected inside, outside or across the boundaries of interest, provided the initial curves contain part of the skeleton of the extrapolated vector field. The classical geodesic active contour model does not authorize this flexibility in the initialization step and therefore the method alone would fail to detect all the shapes. Of course, the proposed method cannot detect automatically interior contours but this drawback is overcome, still with the flexibility in the initialization step. We illustrate this remark with Fig. 4 that represents a slice of the brain (Courtesy of the Laboratory Of Neuro Imaging, UCLA).

5

Conclusion

This paper was devoted to the theoretical study of a new method to extrapolate vector fields using the infinity Laplacian and with applications to image processing. Contrary to prior related works, the number of unknowns is reduced to a single one. The problem is phrased in a variational framework and the EulerLagrange equation is then derived. It is solved using a gradient descent method, which leads to a parabolic problem that falls within the viscosity solution theory framework. The existence and uniqueness of a viscosity solution continuous in space and time, Lipschitz continuous in space and uniformly continuous in time is established. The theoretical study is complemented by several numerical experimentations, first dedicated to the extrapolation problem, and then extended to the segmentation problem. The experimentations show that the proposed approach performs well, even if in strong concavities the results are slightly less accurate than with the NGVF. The model is sensitive to the geometry of the boundaries and to the textures present in the images. In the segmentation framework, the introduction of this new force field allows to widen the choice of the initial condition.

References 1. Aronsson, G.: Minimization problems for the functional supx F (x, f (x), f  (x)). Arkiv für Mate. 6, 33–53 (1965) 2. Aronsson, G.: Minimization problems for the functional supx F (x, f (x), f  (x)). II. Arkiv für Mate. 6, 409–431 (1966)

98

L. Guillot and C. Le Guyader

3. Aronsson, G.: Extension of functions satisfying Lipschitz conditions. Arkiv für Mate. 6(6), 551–561 (1967) 4. Aronsson, G.: On the partial differential equation u2x uxx + 2ux uy uxy + u2y uyy = 0. Arkiv für Mate. 7, 395–425 (1968) 5. Aronsson, G., Crandall, M., Juutinen, P.: A tour of the theory of absolutely minimizing functions. Bull. Amer. Math. Soc. (N.S.) 41, 439–505 (2004) 6. Aubert, G., Kornprobst, P.: Mathematical Problems in Image Processing: Partial Differential Equations and the calculus of Variations. Springer, Heidelberg (2002) 7. Barles, G.: Solutions de viscosité des équations de Hamilton-Jacobi. Springer, Heidelberg (1994) 8. Barles, G.: Solutions de viscosité et équations elliptiques du deuxième ordre. Cours de DEA (1997) 9. Barles, G., Busca, J.: Existence and comparaison results for fully nonlinear degenerate elliptic equations without zeroth-order term. Comm. Partial Differential Equations 26, 2323–2337 (2001) 10. Barron, E.N., Evans, L.C., Jensen, R.: The infinity Laplacian, Aronsson’s equation and their generalizations. Trans. Amer. Math. Soc. 360(1), 77–101 (2008) 11. Brézis, H.: Analyse fonctionnelle. Dunod (1999) 12. Canny, J.: A computational approach to edge detection. IEEE Trans. Pattern Anal. Mach. Intell. 8, 679–698 (1986) 13. Carlsson, S.: Sketch based coding of grey level images. Signal Process. 15, 57–83 (1988) 14. Casas, J.R.: Image compression based on perceptual coding techniques. Ph.D. dissertation, Dept. Signal Theory Commun., UPC, Barcelona, Spain (1996) 15. Caselles, V., Catté, F., Coll, C., Dibos, F.: A geometric model for active contours in image processing. Numer. Math. 66, 1–31 (1993) 16. Caselles, V., Kimmel, R., Sapiro, G.: Geodesic Active Contours. Int. J. Comput. Vision 22(1), 61–87 (1997) 17. Caselles, V., Morel, J.M., Sbert, C.: An Axiomatic Approach to Image Interpolation. IEEE Trans. Image Process. 7(3), 376–386 (1998) 18. Cohen, L.D.: On Active Contour Models and Balloons. CVGIP: Image Understanding 53(2), 211–218 (1989) 19. Cong, G., Esser, M., Parvin, B., Bebis, G.: Shape Metamorphism Using p-Laplacian Equation. In: ICPR, vol. 4, pp. 15–18 (2004) 20. Crandall, M.G.: A visit with the ∞-Laplace equation. Preprint, Notes from a CIME course (2005) 21. Crandall, M.G., Ishii, H., Lions, P.L.: User’s guide to viscosity solutions of second order partial differential equations. Bull. Amer. Math. Soc. 27, 1–67 (1992) 22. Crandall, M.G., Lions, P.L.: Viscosity solutions of Hamilton-Jacobi Equations. Trans. Amer. Math. Soc. 277, 1–42 (1983) 23. Crandall, M.G., Lions, P.L.: On existence and uniqueness of solutions of HamiltonJacobi equations. Non-Linear Anal. 10, 353–370 (1986) 24. Elion, C., Vese, L.A.: An image decomposition model using the total variation and the infinity Laplacian. In: Proceedings SPIE, vol. 6498, pp. 64980W-1–64980W-10 (2007) 25. Forcadel, N.: Dislocations dynamics with a mean curvature term: short time existence and uniqueness. Differential and Integral Equations 21(3-4), 285–304 (2008) 26. Ishii, H.: Existence and uniqueness of solutions of Hamilton-Jacobi equations. Funkcial. Ekvac. 29, 167–188 (1986) 27. Ishii, H., Lions, P.L.: Viscosity solutions of fully nonlinear second-order elliptic partial differential equations. J. Differ. Equations 83, 26–78 (1990)

Extrapolation of Vector Fields Using the Infinity Laplacian

99

28. Jensen, R.: Uniqueness of Lipschitz extensions minimizing the sup-norm of the gradient. Arch. Rat. Mech. Anal. 123(1), 51–74 (1993) 29. Jifeng, N., Chengke, W., Shigang, L., Shuqin, Y.: NGVF: An improved external force field for active contour model. Pattern Recogn. Lett. 28, 58–63 (2007) 30. Kass, M., Terzopoulos, D., Witkin, A.: Snakes: Active contour models. Int. J. Comput. Vision 1, 321–331 (1988) 31. Mémoli, F., Sapiro, G., Thompson, P.: Brain and surface warping via minimizing Lipschitz extensions. In: MFCA, International Workshop on Mathematical Foundations of Computational Anatomy (2006) 32. Meyer, Y.: Oscillating Patterns in Image Processing and Nonlinear Evolution Equations. AMS 22 (2001) 33. Osher, S., Sethian, J.A.: Fronts propagation with curvature dependent speed: Algorithms based on Hamilton-Jacobi formulations. J. Comput. Phys. 79, 12–49 (1988) 34. Paragios, N., Mellina-Gottardo, O., Ramesh, V.: Gradient Vector Flow Fast Geodesic Active Contours. In: Proc. IEEE Intl. Conf. Computer Vision, vol. 1, pp. 67–73 (2001) 35. Prewitt, J.M.S.: Object enhancement and extraction. In: Lipkin, B., Rosenfeld, A. (eds.) Picture Processing and Psychopictorics, pp. 75–149. Academic Press, New York (1970) 36. Sethian, J.A.: Level Set Methods and Fast Marching Methods: Evolving interfaces in Computational Geometry. In: Fluid Mechanics, Computer Vision and Material Science. Cambridge University Press, Londres (1999) 37. Torre, V., Poggio, T.A.: On edge detection. IEEE Trans. Pattern Anal. Mach. Intell. 8, 147–163 (1986) 38. Xu, C., Prince, J.L.: Snakes, shapes, and gradient vector flow. IEEE Trans. Image Process. 7(3), 359–369 (1998) 39. Yuille, A.L., Poggio, T.A.: Scaling theorems for zero-crossings. IEEE Trans. Pattern Anal. Mach. Intell. 8, 15–25 (1986)

A Schrödinger Equation for the Fast Computation of Approximate Euclidean Distance Functions Karthik S. Gurumoorthy and Anand Rangarajan Dept. of CISE, University of Florida, Gainesville, FL, USA

Abstract. Computational techniques adapted from classical mechanics and used in image analysis run the gamut from Lagrangian action principles to HamiltonJacobi field equations: witness the popularity of the fast marching and fast sweeping methods which are essentially fast Hamilton-Jacobi solvers. In sharp contrast, there are very few applications of quantum mechanics inspired computational methods. Given the fact that most of classical mechanics can be obtained as a limiting case of quantum mechanics (as Planck’s constant h tends to zero), this paucity of quantum mechanics inspired methods is surprising. In this work, we derive relationships between nonlinear Hamilton-Jacobi and linear Schrödinger equations for the Euclidean distance function problem (in 1D, 2D and 3D). We then solve the Schrödinger wave equation instead of the corresponding HamiltonJacobi equation. We show that the Schrödinger equation has a closed form solution and that this solution can be efficiently computed in O(N log N ), N being the number of grid points. The Euclidean distance can then be recovered from the wave function. Since the wave function is computed for a small but non-zero h, the obtained Euclidean distance function is an approximation. We derive analytic bounds for the error of the approximation and experimentally compare the results of our approach with the exact Euclidean distance function on real and synthetic data.

1 Introduction Image analysis [1,2] has a tradition of importing and adapting a host of classical physics based approaches including Lagrangian based variational principles and their associated Euler-Lagrange equations [3], Hamiltonian dynamics [4] and more recently HamiltonJacobi based methods [5]. Approaches in image analysis do not strictly adhere to the classical mechanics sequence [6] of i) first specifying a Lagrangian action principle, ii) deriving the corresponding Euler-Lagrange equation, iii) employing a Legendre transformation to convert the Lagrangian dynamics to a first-order Hamiltonian dynamics, and finally, iv) employing a canonical transformation to derive the Hamilton-Jacobi equation whose solution also yields a solution to the original variational problem. Instead, most research in image analysis uses a combination of one or more of these four approaches depending on the problem at hand. For example, in surface reconstruction [3], a popular approach consists of writing a variational form and then finding a solution using preconditioned conjugate gradient or quasi-Newton type iterative methods. While we notice a plethora of classical mechanics inspired techniques in image analysis, the same cannot be said about quantum mechanics. Despite the well known fact X.-C. Tai et al. (Eds.): SSVM 2009, LNCS 5567, pp. 100–111, 2009. c Springer-Verlag Berlin Heidelberg 2009 

A Schrödinger Equation for the Fast Computation

101

that most of classical mechanics is a limiting case of quantum mechanics as Planck’s constant  → 0 [7], there is very little application of quantum mechanical principles in image analysis problems. Rather than speculate on the reasons for this dearth of applications, we wish to point out that in this paper, we are primarily interested in exploiting a concrete relationship between the classical, non-linear Hamilton-Jacobi equation and the quantum, linear Schrödinger equation. We feel that focusing more narrowly on this relationship (which will become more obvious as we proceed) is more productive than dwelling on the more mysterious and specifically quantum mechanical issues of i) interpretation of the wave function, ii) role of probabilities and, iii) the problem of measurement. While these issues are certainly important, they do not play any role in this paper. In summary, we are mainly interested in exploiting the relationship between the Schrödinger and Hamilton-Jacobi equations in order to derive computationally efficient algorithms which are applicable in image analysis problems where Hamilton-Jacobi theory is used. In the theoretical physics literature, a Schrödinger wave equation at the energy state h E has the form ψ(X, t) = φ(X) exp( iEt  ), ( ≡ 2π ) [8], where φ(X)—the stationary state wave function—is the eigenstate of the Hamiltonian operator H corresponding to the eigenvalue E. When the Hamilton-Jacobi scalar field S ∗ appears as the exponent ∗ of the stationary state wave function, specifically φ(X) = exp( −S (X) ), and if φ(X) satisfies the linear Schrödinger equation, namely Hφ = Eφ, we show that as  → 0, S ∗ satisfies the Hamilton-Jacobi equation for a carefully chosen problem. The novel aspect is that a nonlinear Hamilton-Jacobi equation is obtained in the limit as  → 0 of a linear Schrödinger equation. Consequently, instead of solving the Hamilton-Jacobi equation, one can solve the Schrödinger wave equation (taking advantage of its linearity), and then compute an approximate S ∗ for small values of . This computational procedure would be approximately equivalent to solving the original Hamilton-Jacobi equation. With the basic setup in place, we now turn our attention to an actual application. Our goal is to apply the Schrödinger formalism to a well known problem that has been successfully attacked by Hamilton-Jacobi theory. To this end, we choose the problem of computing Euclidean distance functions on a grid for a given set of points where the task is to assign at each grid point a value corresponding to the Euclidean distance to its nearest neighbor from the given point-set. The literature is replete with elegant and pioneering works which have successfully solved this problem. To name a few, the well known fast marching [9] and fast sweeping [10] methods are essentially O(N log N ) Hamilton-Jacobi based algorithms where N is the number of grid points. [The fast sweeping method has an added advantage in that the algorithm appears (empirically)1 to be O(N ).] These techniques focus on directly solving the non-linear eikonal equation [4], whereas our approach shows that one can instead solve a linear equation and obtain the solution to the non-linear eikonal equation in the limit as  → 0. The important connection between the Schrödinger wave equation and the Hamilton-Jacobi equation [7] is illuminated during the process. In the more traditional (computer science) algorithms literature, computational geometry inspired techniques like Voronoi diagrams and KD-Trees [11] have also solved this problem. However, computing Voronoi 1

After a careful reading of [10], it is still unclear to us if the fast sweeping method is formally rather than empirically O(N ).

102

K.S. Gurumoorthy and A. Rangarajan

diagrams or building data structures for KD-Trees in 3D and higher dimensions is expensive and the O(N log N ) complexity is not retained at higher dimensions [11]. Our technique (which as we shall see is based on application of the fast Fourier transform (FFT) [12]) is very simple and elegant and remains O(N log N ) irrespective of the spatial dimension. The basic question asked and answered in this paper is: Can we design a Schrödinger equation that computes an exponentiated Euclidean distance function on a grid? The distance function is obtained from the exponent of the Schrödinger wave function. A naïve approach to solve this Euclidean distance problem would be to visit every grid point and compute the Euclidean distance to all members of the point-set and pick the smallest distance. The complexity of this naïve approach is obviously related to the product of the number of grid points and the cardinality of the point-set. The fast sweeping and fast marching methods avoid this naïve complexity and as we shall see, so does the Schrödinger wave function approach.

2 Euclidean Distance Functions We now describe the Euclidean distance function problem. Given a point-set Y = {Yk ∈ RD , k ∈ {1, . . . , K}} where D is the dimensionality of the point-set and a set of equally spaced Cartesian grid points X, the Euclidean distance function problem requires us to assign S ∗ (X) = min X − Yk  (1) k

with the Euclidean norm used in (1). If efficient computation is a non-issue, this is a simple problem. We visit each grid point in the set X and compute the distance to every point Yk , ∀k ∈ {1, . . . , K} and assign S ∗ (X) the minimum distance. If the number of grid points is N , the naïve complexity is O(N KD). 2.1 Hamilton-Jacobi Formulation The Hamilton-Jacobi equation approach to the Euclidean distance function problem stems from considering the following variational problem which in 2D is  I[q] =

t1

L(q1 , q2 , q˙1 , q˙2 , t)dt

(2)

to

where the Lagrangian L is defined as L(q1 , q2 , q˙1 , q2˙, t) ≡

1 2 (q˙1 + q˙2 2 ). 2

Defining pi ≡ ∂∂L q˙i and applying a Legendre transformation [6] [by inverting pi = to get the function q˙i (q1 , q2 , p1 , p2 , t)], we define the Hamiltonian of the system as

(3) ∂L ∂ q˙i

H(q1 , q2 , p1 , p2 , t) ≡ p1 q˙1 + p2 q˙2 − L(q1 , q2 , q˙1 (q1 , q2 , p1 , p2 , t), q˙2 (q1 , q2 , p1 , p2 , t)) (4)

A Schrödinger Equation for the Fast Computation

103

which for the Euclidean distance function problem turns out to be H(q1 , q2 , p1 , p2 , t) =

1 2 (p + p22 ). 2 1

(5)

The Hamilton-Jacobi equation is obtained via a canonical transformation [6] of the Hamiltonian. In the 2D case, it is ∂S ∂S ∂S + H(q1 , q2 , , , t) = 0 ∂t ∂q1 ∂q2

(6)

where we have replaced the generalized momentum variables pi with ∂S ∂L = pi = . ∂qi ∂ q˙i

(7)

Since the Hamiltonian in (5) is a constant independent of time, equation (6) can be simplified to the static Hamilton-Jacobi equation. By separation of variables, we get S(q1 , q2 , t) = S ∗ (q1 , q2 ) − Et

(8)

∗

where E is the total energy of the system and S (q1 , q2 ) is called the Hamilton’s characteristic function [13]. Using the definition of pi from (7) in (5) and observing that ∂S ∂S ∗ ∂qi = ∂qi , we get  2  ∗ 2  1 ∂S ∗ ∂S = E. (9) + 2 ∂q1 ∂q2 Choosing the energy E to be 12 , we obtain   S∗ = 1

(10) ∗

which is the well known eikonal equation [4] where the forcing term is 1. S is the required Hamilton-Jacobi scalar field which is efficiently obtained by the fast sweeping and fast marching methods. 2.2 Schrödinger Wave Equation for Euclidean Distance Functions We now derive and solve a Schrödinger wave equation for this Euclidean distance function problem instead of solving the non-linear eikonal equation. The Schrödinger wave equation is written as [8] i

∂ψ = Hψ ∂t

(11)

where ψ(X, t) is the wave function and H is the Hamiltonian operator obtained by first ∂ quantization2—where the momentum variables pi are replaced with the operator i ∂x . i Using the definition of our Hamiltonian from (5), ψ satisfies (in 2D)   2 ∂ 2 ψ ∂ 2ψ  ∂ψ = . (12) + i ∂t 2 ∂x21 ∂x22 2

First quantization is still mysterious. For an informal but illuminating treatment, please see http://math.ucr.edu/home/baez/categories.html

104

K.S. Gurumoorthy and A. Rangarajan

Using separation of variables ψ(X, t) = φ(X)f (t), we get 2 2 φ  f˙ = =E (13) if 2 φ where E is the energy state of the system. By choosing the energy of the system to be 1 2 as before, we get   it (14) f (t) = exp 2 and hence   it (15) ψ(X, t) = φ(X) exp 2 where φ(X) satisfies the equation 2 2 φ = φ.

(16)

2.3 Eikonal Equation for Euclidean Distance Functions ∗

∗ We now show that when φ = exp( −S  ) and satisfies (16), S asymptotically satisfies the eikonal equation (10) as  → 0. We show this for the 2D case but the generalization to higher dimensions is straightforward. ∗ 1 ,x2 ) When φ(x1 , x2 ) = exp( −S (x ), the first partials of φ are      ∗ 1 1 ∂φ −S ∂S ∗ ∂φ −S ∗ ∂S ∗ = − exp , = − exp . (17) ∂x1   ∂x1 ∂x2   ∂x2

The second partials needed for the Laplacian are   ∗ 2    1 1 ∂2φ ∂S −S ∗ ∂ 2 S ∗ −S ∗ exp = exp − , ∂x21 2  ∂x1   ∂x21   ∗ 2    1 1 ∂S −S ∗ ∂ 2 S ∗ ∂2φ −S ∗ = 2 exp − exp . ∂x22   ∂x2   ∂x22 From this, equation (16) can be rewritten as  ∗ 2  ∗ 2   2 ∗ ∂2S ∗ ∂S ∂S ∂ S =1 + − + ∂x1 ∂x2 ∂x21 ∂x22

(18)

(19)

which in simplified form is   S ∗ 2 −  2 S ∗ = 1.

(20)

The additional  2 S ∗ term [relative to (10)] is referred to as the viscosity term [9]. (Note that this term emerges naturally from the Schrödinger equation derivation—an intriguing result.) Since | 2 S ∗ | is bounded, as  → 0, (20) tends to   S∗ = 1

(21)

which is the original eikonal equation (10) for the Euclidean distance function. This relationship motivates us to solve the linear Schrödinger equation (16) instead of the non-linear eikonal equation and then compute the distance function via S ∗ (X) = − log φ(X).

(22)

A Schrödinger Equation for the Fast Computation

105

3 Closed Form Solutions for the Approximate Euclidean Distance Function and Proofs of Convergence We now derive the closed form solution for φ(X) (in 1D, 2D and 3D) satisfying equation (16) and hence for S ∗ (X) by (22) and observe that we get the actual Euclidean distance function in the limit as  → 0. In order to satisfy the condition that S ∗ (Yk ) = 0,∀Yk , k ∈ {1, . . . , K}, we consider the forced version of the equation (16) which is −2 2 φ + φ =

K 

δ(X − Yk ).

(23)

k=1

Using a Green’s function approach [14] (where the form of the solution depends on the number of spatial dimensions), we can write expressions for the solution φ. Below, let r = mink X − Yk —the actual Euclidean distance function at the grid point X. 1D: In 1D, the solution [14] for φ is 1  exp 2 K

φ(X) =



k=1

−|X − Yk | 

 .

(24)

Using the relationship in (22), we get ∗

S (X) = − log

K 

 exp

k=1

Observe that

−|X − Yk |  

−r S (X) ≤ − log exp  = r +  log(2). ∗

Also,

 +  log (2) .

(25)

 +  log(2)

   −r +  log(2) S ∗ (x) ≥ − log K exp  = − log K + r +  log(2).

(26)

(27)

As  → 0,  log K → 0 and  log  → 0. Furthermore, we see from (26) and (27) that lim S ∗ (X) = r.

(28)

  K 1  X − Yk  φ(X) = K0 2π2 

(29)

→0

2D: In 2D, the solution [14] for φ is

k=1

106

K.S. Gurumoorthy and A. Rangarajan

where K0 is the modified Bessel function of the second kind. Using (22), we get S ∗ (X) = − log

K 

 K0

k=1

X − Yk  

Then,

r

S ∗ (X) ≤ − log K0 Using the relation K0 ( hr ) ≥

r exp(− h )

√r

when

h

 ∗

S (X) ≤ − log = − log Moreover

 r h

 exp r



+  log(2π2 ).

+  log(2π2 ).

(30)

(31)

≥ 0.5, we get 

−r 



+  log(2π2 )

 + r +  log(2π2 ). r

   −r +  log(2π2 ). S ∗ (X) ≥ − log KK0 

(32)

(33)

r Using the relation K0 ( r ) ≤ exp( −r  ) when h ≥ 1.5, we get    −r +  log(2π2 ) S ∗ (X) ≥ − log K exp 

= − log K + r +  log(2π2 ).

(34)

As  → 0,  log K → 0,  log r → 0 and  log  → 0. Furthermore, we see from (32) and (34) that lim S ∗ (X) = r. (35) →0

3D: In 3D, the solution [14] for φ is based on the modified spherical Bessel function of the second kind: 

K exp −X−Yk    1 φ(X) = . (36) 4π2 X − Yk  k=1

Using (22), S ∗ (X) = − log

K exp  k=1



−X−Yk  

X − Yk 



 +  log 4π2 .

(37)

Then, ∗

exp

−r 

 +  log(4π2 ) r

 = r +  log r +  log 4π2 .

S (X) ≤ − log

(38)

A Schrödinger Equation for the Fast Computation

Also,

 S ∗ (X) ≥ − log K

exp

−r   

r

107

+  log(4π2 )

= − log K + r +  log r +  log(4π2 ).

(39)

As  → 0,  log K → 0,  log r → 0 and  log  → 0. Furthermore, we see from (38) and (39) that lim S ∗ (X) = r. (40) →0

Hence, we have shown that (in 1D, 2D and 3D), the closed form solution for φ guarantees that S ∗ approaches the true Euclidean distance function in the limit  → 0.

4 Error Bound between the Obtained and True Euclidean Distance Function The solution for φ in 1D, 2D and 3D motivates us to compute the function   K  −X − Yk  ˜ exp φ(X) = 

(41)

k=1

(instead of computing φ) and then to compute the approximate distance function ˜ S˜∗ (X) = − log φ(X)

(42)

(instead of computing S ∗ ). The reasons are two-fold. Firstly, φ˜ can be computed efficiently in O(N log N ) time using the fast Fourier transform (FFT) [12] as explained in the subsequent section. Secondly, lim→0 S˜∗ (X) = r, as shown below, where r is the true Euclidean distance function value at the grid point X (r = mink X − Yk ), and this is the lim→0 S ∗ (X) as seen from the previous section. Hence, for small values of ∗ , S˜∗ (X) is a very good

approximation  to S (X).

 K −X−Yk  ˜∗ can be approximated as exp −r As  → 0, k=1 exp   . Hence S (X)

−r  ≈ − log exp  = r. The bound derived below between S˜∗ (X) and r also unveils the proximity between the computed and the actual Euclidean distance function. Note from (41) that   −r ∗ ˜ S (X) ≤ − log exp = r. (43)  Also, observe that

and hence,

   −r S˜∗ (X) ≥ − log K exp  = − log K + r

(44)

r − S˜∗ (X) ≤  log K.

(45)

108

K.S. Gurumoorthy and A. Rangarajan

From (43) and (45),

|r − S˜∗ (X)| ≤  log K. (46) ∗ ˜ Equation (46) shows that as  → 0, S (X) → r. It is worth commenting that the bound  log K is actually very tight as (i) it scales only as the logarithm of the cardinality of the point-set (K) and (ii) it can be made arbitrarily small by choosing a small but non-zero value of .

5 Efficient Computation of the Approximate Euclidean Distance Function The motivation for computing φ˜ instead of φ is the fact that the direct computation of φ at the N grid locations is O(N K) which is O(N 2 ) when the cardinality of the point-set is O(N ), whereas computing φ˜ at the N grid locations can be done in O(N log N ) using an FFT implementation [12]. The realization that FFT can be employed 

to compute K k φ˜ stems from the insight that the summation term, namely, k=1 exp −X−Y is 

 actually the discrete convolution between the functions f (X) = exp −X com puted at the grid locations, with the function g(X) which takes the value 1 at the pointset locations and 0 at other grid locations. By the convolution theorem [15], a discrete convolution can be obtained as the inverse Fourier transform of the product of two individual transforms, which for two O(N ) sequences can be computed in O(N log N ) time and hence φ˜ can be determined efficiently at the N grid locations. One just needs to compute the discrete Fourier transform (DFT) of the sampled version of the functions f (X) and g(X), compute their point-wise product and then compute the inverse discrete Fourier transform. Taking the logarithm of the inverse discrete Fourier transform and multiplying it by (−), gives the approximate Euclidean distance function. The algorithm is adumbrated in Table 1. Table 1. Approximate Euclidean distance function algorithm

 1. Compute the function f (X) = exp −X at the grid locations.  2. Define the function g(X) which takes the value 1 at the point-set locations and 0 at other grid locations. 3. Compute the FFT of f and g, namely F (u) and G(u) respectively. 4. Compute the function H(u) = F (u) ∗ G(u). ˜ 5. Compute the inverse FFT of H which gives φ(X) at the grid locations. ˜ 6. Take the logarithm of φ(X) and multiply it by (−) to get the approximate Euclidean distance function at the grid locations.

5.1 Computation of the Approximate Euclidean Distance Function in Higher Dimensions Our technique has a straightforward generalization to higher dimensions. Regardless of the spatial dimension, the approximate Euclidean distance function, S˜∗ can be computed by exactly following the steps delineated in the table above. It is worthwhile mentioning that computing the discrete Fourier transform using FFT is always O(N log N )

A Schrödinger Equation for the Fast Computation

109

irrespective of the spatial dimension. Hence, for all dimensions, S˜∗ can be computed at the given N grid points, in O(N log N ). This speaks for the scalability of our technique, which is generally not the case with other methods, for example KD-Trees [11].

6 Experiments In this section, we show the efficacy of our technique by computing the approximate Euclidean distance function S˜∗ and comparing it to the actual Euclidean distance function S, first on randomly generated 2D point-sets and then on a set of bounded 3D grid points. We began with a 2D grid consisting of points between (−30, −30) and (30, 30). Hence, the total number of grid points is N = 61 × 61 = 3721. We randomly chose around 1000 grid locations as data points (point-set). Then 50, 000 experiments were run for values of  ranging from 0.1 to 0.5 in steps of 0.01. The errorbar plot in Figure 1 shows the mean and standard deviation of the percentage error at each value of . The error is less than 0.5% at  = 0.1 demonstrating the algorithm’s ability to compute accurate Euclidean distances. Next, we took the Stanford bunny dataset3 and used the coordinates of the data points on the model as our point-set locations. Since the input data locations need not be at integer locations, we scaled the space uniformly in all dimensions and rounded off the data so that the data lies at integer locations. The input data was also shifted so that it was approximately symmetrically located with respect to the x, y and z axis. We should comment that shifting the data doesn’t affect the Euclidean distance function value and uniform scaling of all dimensions is also not an issue, as the distances can be rescaled once they are computed. After these basic data manipulations, the cardinality of the point set was K = 3019 with the data confined to the cubic region −16 ≤ x ≤ 16, −15 ≤ y ≤ 15 and −12 ≤ z ≤ 12. Our grid consisted of the set of all integer locations within this cubic region. The number of grid locations was N = 25575. We computed the Euclidean distance function value at each of these grid locations using our technique for different values of  and compared it to the true  Euclidean distance function value. −X At small values of , exp drops off very quickly and hence for grid locations  which are far away from the point-set, the convolution done using FFT needs to be precise (without round-off error) for the computed distance to be meaningful. Such high precision support may not be available and hence our technique may produce erroneous results at these grid locations for very small values of . But at those grid locations which are close to the point-set, the accuracy of the computed distance improves as  is decreased. Hence, to circumvent this problem of choosing , we ran our technique for different values of  and chose the distance function values obtained at large values of  at those grid locations whose average computed distance is larger and vice versa.

3

Go to http://www.cc.gatech.edu/projects/large_models/bunny.html to obtain this dataset.

110

K.S. Gurumoorthy and A. Rangarajan

percentage error

20

15

10

5

0

0.1

0.2

0.3 hbar values

0.4

0.5

Fig. 1. Percentage error versus  in 50,000 2D experiments

Fig. 2. Isosurfaces: (i) Left: Actual Euclidean distance function and (ii) Right: Our approach

When we ran our technique for the set of  ∈ {0.1, 0.2, 0.3, 0.4} and used {3, 6, 10} respectively as the threshold of the average computed distance for choosing the appropriate distance function, it gave the following set of results. The maximum absolute difference between the actual and the computed Euclidean distance value over all the grid locations is 0.9066 and the average absolute difference is 0.1322. The accuracy is fairly high since the furthest grid point from the point-set is at a distance of 15.5242 and the average overall distance computed at the grid locations from the point-set is 3.5449. The average error is 0.1322 3.5449 ∗ 100 = 3.72%. This error can be lowered by using higher precision numerical methods for convolution [16]. We plotted the isosurface obtained by connecting the grid points which are at a distance of 0.5 from the point set, determined both by the true Euclidean distance function and our technique. Figure 2 shows the two surfaces. Notice the similarity between the two plots. It provides anecdotal visual evidence for the usefulness of our approach.

7 Discussion In this paper, we have introduced a new approach to solving the non-linear eikonal equation (with a constant forcing term equal to 1). We have proved that the solution of the eikonal equation can be obtained as a limiting case of the solution to the corresponding linear Schrödinger wave equation. The key here is the embedding of the nonlinear Hamilton-Jacobi equation in a linear Schrödinger equation. Our Schrödinger wave equation formalism for solving the Euclidean distance function problem (which has been successfully attacked by pioneering Hamilton-Jacobi solvers such as the fast sweeping [10] and fast marching [9] methods) leverages this deep relationship between the two regimes of modern physics. In the future, we would like to solve the more

A Schrödinger Equation for the Fast Computation

111

general, static Hamilton Jacobi equation using techniques inspired from quantum mechanics as a counterpart to classical mechanics based techniques. In all likelihood, this will involve direct discretization of the Schrödinger wave equation which was not required for the Euclidean distance function problem. We expect that the linearity of the Schrödinger equation will result in fast algorithms even in this more general setting.

References 1. Horn, B.K.P.: Robot vision. MIT Press, Cambridge (1986) 2. Kimmel, R.: Numerical geometry of images: Theory, algorithms, and applications. Springer, Heidelberg (2003) 3. Grimson, W.E.L.: An implementation of a computational theory of visual surface interpolation. Computer Vision, Graphics, and Image Processing 22(1), 39–69 (1983) 4. Siddiqi, K., Tannenbaum, A., Zucker, S.W.: A Hamiltonian approach to the eikonal equation. In: Hancock, E.R., Pelillo, M. (eds.) EMMCVPR 1999. LNCS, vol. 1654, pp. 1–13. Springer, Heidelberg (1999) 5. Kao, C.Y., Osher, S.J., Tsai, Y.H.: Fast sweeping methods for static Hamilton-Jacobi equations. SIAM Journal on Numerical Analysis 42(6), 2612–2632 (2004) 6. Goldstein, H., Poole, C.P., Safko, J.L.: Classical mechanics. Addison-Wesley, Reading (2002) 7. Butterfield, J.: On Hamilton-Jacobi theory as a classical root of quantum theory. In: Elitzur, A., Dolev, S., Kolenda, N. (eds.) Quo-Vadis Quantum Mechanics, pp. 239–274. Springer, Heidelberg (2005) 8. Griffiths, D.J.: Introduction to quantum mechanics. Addison-Wesley, Reading (2004) 9. Osher, S.J., Sethian, J.A.: Fronts propagating with curvature dependent speed: algorithms based on Hamilton-Jacobi formulations. Journal of Computational Physics 79(1), 12–49 (1988) 10. Zhao, H.K.: A fast sweeping method for eikonal equations. Mathematics of Computation 74, 603–627 (2005) 11. de Berg, M., Cheong, O., van Kreveld, M., Overmars, M.: Computational geometry: Algorithms and applications. Springer, Heidelberg (2008) 12. Cooley, J.W., Tukey, J.W.: An algorithm for the machine calculation of complex Fourier series. Mathematics of Computation 19(90), 297–301 (1965) 13. Arnold, V.I.: Mathematical methods of classical mechanics. Springer, Heidelberg (1989) 14. Abramowitz, M., Stegun, I.A.: Handbook of mathematical functions with formulas, graphs, and mathematical tables. Government Printing Office, USA (1964) 15. Bracewell, R.N.: The Fourier transform and its applications. McGraw-Hill Science and Engineering, New York (1999) 16. Hida, Y., Li, H.S., Bailey, D.H.: Quad-double arithmetic: Algorithms, implementation, and application. Technical Report LBNL-46996, Lawrence Berkeley National Laboratory, Berkeley, CA 94720 (2000)

Semi-supervised Segmentation Based on Non-local Continuous Min-Cut Nawal Houhou1 , Xavier Bresson2 , Arthur Szlam2 , Tony F. Chan2 , and Jean-Philippe Thiran1 1

Signal Processing Laboratory 5, Swiss Federal Institute of Technology (EPFL), Lausanne, Switzerland 2 Department of Mathematics, University of California Los Angeles, CA 90095-1555, USA

Abstract. We propose a semi-supervised image segmentation method that relies on a non-local continuous version of the min-cut algorithm and labels or seeds provided by a user. The segmentation process is performed via energy minimization. The proposed energy is composed of three terms. The first term defines labels or seed points assigned to objects that the user wants to identify and the background. The second term carries out the diffusion of object and background labels and stops the diffusion when the interface between the object and the background is reached. The diffusion process is performed on a graph defined from image intensity patches. The graph of intensity patches is known to better deal with textures because this graph uses semi-local and non-local image information. The last term is the standard TV term that regularizes the geometry of the interface. We introduce an iterative scheme that provides a unique minimizer. Promising results are presented on synthetic textures a nd real-world images.

1

Introduction

Image segmentation is an important problem in image processing. The objective of segmentation algorithms is to partition an image into a finite number of semantically important regions such as anatomical or functional structures in medical images or objects in natural images. Well-posed approaches to solve the image segmentation problem are energy minimization methods. This paper introduces an energy minimization algorithm to solve the semi-supervised segmentation problem based on the continuous min cut/max flow model originally defined by Strang in [1]. Semi-supervised segmentation models defined in a continuous setting have already been proposed in the literature. Among them, Protiere and Sapiro proposed in [2] an interactive algorithm for segmentation. Cremers et al. introduced in [3] an algorithm based on the level set method to perform interactive image segmentat ion. Appleton and Talbot introduced in [4] a semi-supervised segmentation model based on the continuous min-cut of Strang. Unger et al. defined in [5] a segmentation method also based on the min-cut model of Strang in [1]. The semi-supervised segmentation models using the continuous min-cut are based on local image information. These models X.-C. Tai et al. (Eds.): SSVM 2009, LNCS 5567, pp. 112–123, 2009. c Springer-Verlag Berlin Heidelberg 2009 

Semi-supervised Segmentation Based on Non-local Continuous Min-Cut

113

s.a. [4,5] perform very well for the segmentation of smooth regions, but they are less efficient with textures. In this paper, we extend the continuous min cut to a non-local formulation along the same line as non-local means defined by Buades, Coll and Morel in [6] and the variational non-local means model of Gilboa and Osher [7]. This non-local extension of the continuous min cut can be obtained in different ways. We used the original discrete min-cut model [8] to define the non-local continuous min-cut, which turns to be the H 1 norm defined on graph and a term that constraints the labels. The H 1 norm carries out the diffusion of object and background labels on the graph of image patches [9, 6], which holds semi-local and non-local image information that can better segment textures and real-world objects. Besides, the continuous formulation of the min-cut algorithm allows us introducing other regularization processes such as the TV energy. The TV energy is indeed useful to regularize the boundary between the object and the background. Besides, the TV energy can smooth out the segmentation of small sets favored by the min-cut algorithm as noticed by Shi and Malik in [10] (see Figure 1).

2

Graph, Min-Cut and Diffusion

Graph representation. Let G = (V, E) be a weighted undirected graph, where the sets V are the graph nodes and E the edges connection nodes. In this paper, each node Vi represents a pixel i in an image I with support Ω ⊂ Rn where typically n = {2, 3}. The similarity between two pixels/nodes i and j in Ω is measured by the edge function on the graph, namely wij . In the case of image segmentation, two pixels i and j that belong to the same object/class are said to be connected and define a measure wij close to unity. Inversely, two pixels i and j that do not belong to the same class are said to be not connected and define a measure wij close to zero. A standard construction approach for the weight matrix wij is as follows. Let h(i, j) be some general non-negative distance measure between nodes i and j, then the weight wij is computed with a Gaussian kernel of 0-mean and variance σ such that: h(i, j) 1 exp(− ), Z σ2 where σ is the scaling parameter and Z is the normalization factor. wij =

Image Feature. The distance h(i, j) depends on image feature. The choice of features is difficult and critical to get an optimal segmentation result. For piecewise smooth and constant images, the gray-level value can be enough. For texture images, a feature vector at each pixel computing from a filter bank (as suggested in e.g. [11]) can be efficient. A recent promising image feature to represent and process textures is the image intensity patch around the current pixel. The patch idea as feature vector was first introduced for texture synthesis [9, 12, 13] then for image denoising. Buades et al. in [6] proposed to compute the weight matrix with patch differences and denoise the image with a non-local averaging. Gilboa and Osher in [7] proposed a variational model for non-local denoising based on

114

N. Houhou et al.

patch differences. Fina lly, Bresson and Chan in [14] proposed a variational unsupervised segmentation method also based on patch differences. In this paper, we will use the graph of image patches of Buades et al. Min-Cut. By definition a binary cut partitions a graph into two subsets. This partition process for graphs can be used for image segmentation when we want to find an object and the background. In optimization theory about maximum flows in flow networks [8], the optimal partition of the graph V into two sets A and B such that A ∪ B = V and A ∩ B = ∅ can be computed by finding the minimal cut (min-cut), i.e. the minimization of the inter-similarity between two sets A and B of V . In other words, given two particular nodes s ∈ A and t ∈ B in the graph, the min-cut partition can be written as:  min − cut(A, B) = min −wij xi xj , (1) x

xi >0,xj <0

where x is a N -dimensional indicator vector, with N = card(V ), such that xi = 1 if node i ∈ A, and xi = −1 otherwise. The min-cut partition gives the minimal capacity defined as the total weight between the nodes for each subset A and B. The min-cut approach has been applied to several computer vision problems, see [15, 16] for image restoration, [17, 18, 4, 5] for image segmentation, [19] for stereo and motion, and [20] for texture synthesis. Diffusion. In the case of binary partition of a graph, min-cut partition and diffusion are equivalent. Indeed, let us denote W the symmetric matrix such that W (i, j) = w(i, j). Then, the graph partition energy defined in (1) is equivalent to a Graph Laplacian operator: 1 (2) wij (xi − xj )2 , 8  where D is a N × N diagonal matrix with di = j w(i, j) on its diagonal, the matrix D − W is called the Graph-Laplacian. cut(A, B) =

Proof. See e.g. [10]. Let x+1 (resp. x−1 2 2 ) be the indicator function for xi > 0 (resp. xi < 0). Then (1) can be written in the matrix form as follows: cut(A, B) = which implies: xT (D − W )x =

1 T x (D − W )x, 4 1 wij (xi − xj )2 . 2 i,j

The weighted Graph Laplacian corresponds to a finite difference approximation of the continuous Lapacian operator. The graph Laplacian can also be a non-local operator. Semi-supervised segmentation. We observe that min-cut partitioning algorithms are defined as semi-supervised segmentation techniques. The min-cut seeks for

Semi-supervised Segmentation Based on Non-local Continuous Min-Cut

115

the optimal partition of the graph given particular nodes called the source "s" and the sink "t". Hence, it is easy to assign some pixels as source and some as sink if the pixel belongs to the object or the background. Several graphbased partitioning methods have been proposed in the literature s.a. [17, 21, 22, 23, 18]. The previous papers are based on discrete minimization methods to compute the min-cut given the labels. In this paper, we propose a continuous minimization method to solve the min-cut problem with labels and non-local image information.

3 3.1

Proposed Segmentation Method Continuous Min-Cut

Energy minimization problem. In this section a new non-local semi-supervised segmentation algorithm is introduced. The algorithm relies on the continuous formulation of the discrete min-cut problem defined as: min − cut(A, B) = min x

s.t.



wij (xi − xj )2

(3)

i,j

xk = +1, ∀k ∈ S xk = −1, ∀k ∈ T,

where S are the labels selected for the object and T are the labels assigned to the background. We propose the continuous min-cut (CMC) problem as follows (which is a constrained minimization problem w.r.t. a real-valued function u):  1 CM C(u) = min w(x, y)(u(x) − u(y))2 dxdy u 2 Ω×Ω s.t. u(x) = 1, ∀x ∈ S u(x) = 0, ∀x ∈ T, which is equivalent to this unconstrained minimization problem for u:   1 2 w(x, y)(u(x) − u(y)) dxdy + λ(x)(u − u0 )2 dx, (4) CM C(u) = min u 2 Ω×Ω Ω   1 if x ∈ S ∞ if x ∈ S ∪ T where u0 (x) = and λ(x) = , 0 if x ∈ T 0 otherwise where function λ provides the degree of confidence with respect to the labels. Non-local H 1 energy. The first term of (4) is deduced from (3) using the change of variable ui = xi2+1 ∈ {0, 1} then relaxing ui to [0, 1]. This term is also known as the non-local H 1 energy ( [24, 7]) defined as:   1 1 1 2 HG (u) = w(x, y)(u(x) − u(y)) dxdy = |∇G u|2 dx = ||u||HG1 , (5) 2 Ω×Ω 2 Ω

116

N. Houhou et al.

 where |∇G u|2 := Ω w(x, y)(u(x)−u(y))2 dy is the square norm of the continuous graph gradient of u. The optimality condition for (5) is:  w(x, y)(u(x) − u(y))dy = ΔG u = 0, Ω

where ΔG u is the continuous graph Laplacian of u. Labels. The second term of (4) introduces the hard constraint of labels in the energy minimization approach. This term comes from Unger et al. in [25], which incorporates seed points (assigned either to the object or to the background) in the geodesic active contour/snake model [26]. This term constrains function u to be equal to u0 for x ∈ S ∪ T and being equal to anything else for x ∈ S ∪ T . Function λ is highly discontinuous, which requires some regularization process to handle it. Unger et al. proposed a splitting operation to solve this problem. A new function v is introduced s.t.:  1 min λ(x)(v − u0 )2 dx + ||u − v||22 , u 2θ Ω where the term ||u − v||22 forces v ≈ u as θ → 0. The optimality solution w.r.t. v leads to:  2λ(x)θu0 − u u if λ → 0 = v= . u if λ → ∞ 2λ(x)θ − 1 0 3.2

Proposed Semi-supervised Segmentation Algorithm: Continuous Min-Cut + TV

Final model. The previous section introduced the continuous formulation of the min cut problem. In this section, we proposed to merge the continuous min-cut with the Total Variation (TV) energy. The TV term offers two advantages. First, TV regularizes the geometry of the contour between classes (object and background). Experiments showed that the continuous min-cut can provide irregularities along the contour. Second, Shi and Malik in [10] observed that the min-cut algorithm tends to favor misclassification of small sets, which are smoothed out with the TV regularization process. Finally, we propose the following energy minimization model for semisupervised segmentation:  E(u) = ||u||HG1 + λ(x)(u − u0 )2 dx + β||u||T V , (6) where ||u||T V =



Ω

Ω

|∇u|dx.

Minimization process. A direct use of the calculus of variation to (6) will produce a very slow minimization process. We propose to use a splitting operation to minimize E more efficiently. We introduce two new functions v, s s.t.:  λ(x)(v − u0 )2 dx + β||s||T V E(u, v, s) = ||u||HG1 + Ω

1 1 + ||u − v||22 + ||s − v||22 . 2θv 2θs

(7)

Semi-supervised Segmentation Based on Non-local Continuous Min-Cut

117

1 2θv ||u − w(x,y)u(y)dy+v(x) Ω  . Funcθv Ω w(x,y)dy+1

Then, v, s being fixed, we search for u as the solution of minu ||u||HG1 + v||22 , which is given by a fixed point method as u =

θv



 tions u, s being fixed, we search for v as the solution of minv Ω λ(x)(v − vs u0 )2 dx + 2θ1v ||u − v||22 + 2θ1s ||s − v||22 , which is given by v = θsθv+θ if λ = 0 s +θv and v = u0 if λ = ∞. Functions u, v being fixed, we search for s as the solution of mins β||s||T V + 2θ1s ||s − v||22 , which solution is given e.g. the Projection algorithm of Chambolle [27]. We propose the following iterative scheme for minimizing energy (7): ⎧ ⎪ un+1 ⎪ ⎪ ⎪ ⎪ ⎪ ⎨ n+1 v ⎪ ⎪ ⎪ ⎪ sn+1 ⎪ ⎪ ⎩ n+1 p

3.3

= =

θv





w(x,y)un (y)dy+v n (x)  θv Ω w(x,y)dy+1

Ω

θs un+1 +θv sn θs +θv

u0

if λ = 0 if λ = ∞ ,

n≥0

(8)

=v − θdivp β(pn +1/8∇(divpn −v n+1 /θs )) = β+1/8|∇(divpn −vn+1 /θs )| n+1

n

Some Properties of the Models (6) and (7)

Convexity. Both energy minimization models (6) and (7) are strictly convex 1 (since the HG term is strictly convex), which implies the existence of a unique minimizing solution independently of the initial condition. Hence, even using gradient descent approaches, the algorithm does not get stuck in a local minimum. Thus, as far as the labels are correctly defined, the results will be independent of the initialization. Relation with the original min-cut problem (1) or equivalently (3). The continuous min-cut problem (4) has the same solution as the discrete min-cut problem when considering characteristic/indicator functions of sets, i.e.: min {ECMC (u = 1A )} = min {cut(A, Ω\A)}. 1A

A

(9)

We remind that we relax function u to be between [0, 1] to define a continuous version of the min-cut algorithm, which can be minimized with continuous minimization tools. Then, the segmentation result is given by thresholding the minimizer u of (6) with any value in (0, 1). Non-trivial steady state solution of (6). The final steady state solution of (6) is not the mean value of the initial function. Call ut=0 = ut=0 the mean value function. It is easy to show by contraction that ut=0 is not solution to (6). If ut=∞ = ut=0 then E(ut=∞ ) = Ω λ(x)(ut=0 − u0 )2 dx > 0, and the minimizer is thus given by ut=0 = u0 . However, u0 (x) = 1 ∀x ∈ S, 0 ∀x ∈ T . Thus, ut=0 = u0 . We notice that Gilboa and Osher in [7] also use the energy ||u||HG1 to perform semi-supervised segmentation. However, they did not use a term to constraint the labels as in this work. They minimized energy ||u||HG1 starting with a trinary initial function ut=0 = {−1, 0, 1} (labelled pixels for the object are assigned to

118

N. Houhou et al.

the value 1 and those for the background to the value −1). However, the minimizing solution is the mean value function ut=∞ = ut=0 . Hence, this algorithm requires to stop the diffusion process.

4

Results

This section presents some results of the proposed semi-supervised segmentation algorithm. The graph is defined from local and non-local image information:  |i−j|2 (j)|2 + |F (i)−F if i, j ∈ Na×a (i) σ12 σ22 w(i, j) = , (10) 0 otherwise where Na×a (i) is a square window of size a × a around i. The computational cost of the similarity between pixel on the whole image is very expensive, however we chose to simply select points in a close neighborhood. This implies the supposition that if two points are far away, they are not connected. From a2 neighbors only the cl = 8 closest points are selected. The feature vector F is a square patch of size f ×f centered on each pixel. The segmentation is driven by (8). The initial condition for u, v and s are given by the label S, i.e. u = v = s = 1 if x ∈ S and u = v = s = 0 otherwise. With an unoptimized Matlab implementation, the graph computation lasts approximatively 15 seconds and the segmentation is performed in approximatively 1 minute. The image size is 128 × 128. TV Regularization Effect. The importance of the TV-Regularization effect is emphasized in this paragraph. A salt-and-pepper noise is added on a two-phase image with different means 1(a). The inside and outside labels are presented on Figure 1(a). The results show that if the TV regularization is not performed then the segmentation fails (Fig 1(b)). When the TV regularization is used, then the segmentation succeeds.

(a)

(b)

(c)

Fig. 1. Application of our algorithm on a image with a salt-and-pepper noise. (a)Initialization (b)The segmentation result without TV-regularization.(c)The segmentation result with TV-regularization.

Semi-supervised Segmentation Based on Non-local Continuous Min-Cut

119

Texture Images. We apply our algorithm to a synthetic texture image composed of five different patterns. Figures 2(a) and 2(c) show the initializations and Figures 2(b) and 2(d) the corresponded results. The patch size is chosen to be 9 × 9 which correspond to the pattern size for the two selected textures.

(a)

(b)

(c)

(d)

Fig. 2. Results on synthetic textures. (a) and (c) Initializations. (b) and (d) results.

(a)

(b)

(c)

(d)

(e)

(f)

Fig. 3. Results on real-world images from the Berkeley dataset. Left column: Initial labels. Right column: Segmentation Result.

Natural Images. We apply now our algorithm to a set of natural images taken from the Berkeley segmentation dataset [28]. In the fist column of Figure 3, the inside and outside labels are shown and in the second column the segmentation results.

120

N. Houhou et al.

(a)

(b)

(c)

(d)

(e)

(f)

Fig. 4. Results on real world color images from the Berkeley dataset. Left column: Initial labels. Right column: Segmentation Result.

Fig. 5. First row, Segmentation of the liver. (a) Initial labels. (b) Segmentation Result. Second row, Segmentation of the lateral muscles on the neck. (c) Initial labels. (d) Segmentation Result. (e) Zoom on the segmentation of the muscles.

Color Images. We consider the simple case of Red-Green-Blue (RGB) channels. The first step consists of computing the graph by taking into account each channel, i.e. F = (Fr , Fg , Fb ), where respectively the red, green and blue feature channel. Images are also taken from the Berkeley segmentation dataset [28]. In the fist column of Figure 3, the inside and outside labels are shown and in the second column the segmentation results. Medical Images. We apply our segmentation algorithm on 2-D medical images of CT scans of the abdomen and the head and neck. Figures 5(a) and 5(c) present

Semi-supervised Segmentation Based on Non-local Continuous Min-Cut

121

the inside and outside initial labels. Figures 5(b) and 5(d) show the segmentation results. For the liver segmentation, the label on the background (black) prevents the diffusion from capturing as well the heart. The segmentation of the structures in the neck are challenging and the result that we obtain are promising.

5

Discussion and Conclusion

In this paper, a non-local semi-supervised segmentation method has been proposed. The success of graph partitioning algorithms for image segmentation has motivated this work. Our objective was to translate the discrete min-cut algorithm into a non-local continuous min-cut algorithm. The addition of hard constraints with the source and sink labels has been done naturally in the proposed continuous framework. Besides, it has also been easy to introduce new terms such as the TV term that regularizes the geometry of the boundary between the object and the background. The non-local continuous min-cut is also equivalent to a diffusion process. The diffusion is done on the graph of image intensity patches, which holds semi-local and non-local image information useful to segment textures and complex patterns. Our semi-supervised segmentation has provided promising segmentation results for textures and real-world objects. Future work will focus on comparing the efficiency of our segmentation algorithm with other related semi-supervised segmentation algorithms. We would like also to extend our method to 3-D medical images.

Acknowledgements Nawal Houhou was supported by Swiss National Science Foundation #205320101621, Xavier Bresson was supported by ONR N00014-03-1-0071 and ONR MURI subcontract from Stanford University and Arthur Szlam was supported by NSF DMS-0811203. The authors would like also to thank the referees for their constructive comments.

References 1. Strang, G.: Maximal Flow Through A Domain. Mathematical Programming 26(2), 123–143 (1983) 2. Protiere, A., Sapiro, G.: Interactive image segmentation via adaptive weighted distances. IEEE Transactions on Image Processing 16(4), 1046–1057 (2007) 3. Cremers, D., Fluck, O., Rousson, M., Aharon, S.: A Probabilistic Level Set Formulation for Interactive Organ Segmentation. In: SPIE (2007) 4. Appleton, B., Talbot, H.: Globally minimal surfaces by continuous maximal flows. IEEE Transactions on Pattern Analysis and Machine Intelligence 28(1), 106–118 (2006) 5. Unger, M., Pock, T., Cremers, D., Bischof, H.: Tvseg - interactive total variation based image segmentation. In: British Machine Vision Conference (BMVC), Leeds, UK (September 2008)

122

N. Houhou et al.

6. Buades, A., Coll, B., Morel, J.: A review of image denoising algorithms, with a new one. Multiscale Modeling & Simulation 4(2), 490–530 (2005) 7. Gilboa, G., Osher, S.: Nonlocal linear image regularization and supervised segmentation. Multiscale Modeling and Simulation 6(2), 595–630 (2007) 8. Elias, P., Feinstein, A., Shannon, C.E.: Note on Maximum Flow Through a Network. IRE Transactions on Information Theory 2, 117–119 (1956) 9. Efros, A., Leung, T.: Texture Synthesis by Non-Parametric Sampling. In: IEEE International Conference on Computer Vision, vol. 2, pp. 10–33 (1999) 10. Shi, J., Malik, J.: Normalized cuts and image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence 22, 888–905 (2000) 11. Malik, J., Belongie, S., Leung, T., Shi, J.: Contour and texture analysis for image segmentation. International Journal of Computer Vision 43(1), 7–27 (2001) 12. Efros, A., Freeman, W.: Image quilting for texture synthesis and transfer. In: Proceedings of the Conference on Computer graphics and interactive techniques, SIGGRAPH, pp. 341–346. ACM, New York (2001) 13. Liang, L., Liu, C., Xu, Y., Guo, B., Shum, H.: Real-time texture synthesis by patch-based sampling. ACM Trans. Graph. 20(3), 127–150 (2001) 14. Bresson, X., Chan, T.: Non-local Unsupervised Variational Image Segmentation Models, UCLA CAM Report 08-67 (2008) 15. Wu, Z., Leahy, R.: An Optimal graph theoretic approach to data clustering: Theory and its application to image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence 15(11), 1101–1113 (1993) 16. Ishikawa, H., Geiger, D.: Segmentation by grouping junctions. In: Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 125–131 (1998) 17. Boykov, Y., Jolly, M.P.: Interactive graph cuts for optimal boundary and region segmentation of objects in n-d images. In: Proceedings of Eighth IEEE International Conference on Computer Vision, vol. 1, pp. 105–112 (2001) 18. Boykov, Y., Funka-Lea, G.: Graph cuts and efficient n-d image segmentation. Int. J. Comput. Vision 70(2), 109–131 (2006) 19. Boykov, Y., Veksler, O., Zabih, R.: Fast approximate energy minimization via graph cuts. IEEE Transactions on Pattern Analysis and Machine Intelligence 23, 2001 (2001) 20. Kwatra, V., Schödl, A., Essa, I., Turk, G., Bobick, A.: Graphcut textures: Image and video synthesis using graph cuts. In: Proceedings of the Conference on Computer graphics and interactive techniques, SIGGRAPH, vol. 22(3), pp. 277–286 (July 2003) 21. Blum, A., Chawla, S.: Learning from labeled and unlabeled data using graph mincuts. In: Proceedings of the Eighteenth International Conference on Machine Learning, pp. 19–26. Morgan Kaufmann Publishers Inc., San Francisco (2001) 22. Yu, S., Shi, J.: Segmentation given partial grouping constraints. IEEE Transactions on Pattern Analysis and Machine Intelligence 26(2), 173–183 (2004) 23. Grady, L., Funka-lea, G.: Multi-label image segmentation for medical applications based on graph-theoretic electrical potentials. In: Proceedings of the European Conference on Computer Vision, pp. 230–245. Springer, Heidelberg (2004) 24. Zhou, D., Scholkopf, B.: A Regularization Framework for Learning from Graph Data. In: Workshop on Statistical Relational Learning and Its Connections to Other Fields (2004) 25. Unger, M., Pock, T., Bischof, H.: Continuous globally optimal image segmentation with local constraints. In: Computer Vision Winter Workshop 2008 (2008)

Semi-supervised Segmentation Based on Non-local Continuous Min-Cut

123

26. Caselles, V., Kimmel, R., Sapiro, G.: Geodesic Active Contours. International Journal of Computer Vision 22(1), 61–79 (1997) 27. Chambolle, A.: An Algorithm for Total Variation Minimization and Applications. Journal of Mathematical Imaging and Vision 20(1–2), 89–97 (2004) 28. Martin, D., Fowlkes, C., Tal, D., Malik, J.: A Database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, July 2001, vol. 2, pp. 416–423 (2001)

Momentum Based Optimization Methods for Level Set Segmentation Gunnar Läthén1,3 , Thord Andersson2,3 , Reiner Lenz1,3 , and Magnus Borga2,3 1

Department of Science and Technology, Linköping University Department of Biomedical Engineering, Linköping University Center for Medical Image Science and Visualization, Linköping University 2

3

Abstract. Segmentation of images is often posed as a variational problem. As such, it is solved by formulating an energy functional depending on a contour and other image derived terms. The solution of the segmentation problem is the contour which extremizes this functional. The standard way of solving this optimization problem is by gradient descent search in the solution space, which typically suffers from many unwanted local optima and poor convergence. Classically, these problems have been circumvented by modifying the energy functional. In contrast, the focus of this paper is on alternative methods for optimization. Inspired by ideas from the machine learning community, we propose segmentation based on gradient descent with momentum. Our results show that typical models hampered by local optima solutions can be further improved by this approach. We illustrate the performance improvements using the level set framework.

1

Introduction

A very popular and powerful approach for solving image segmentation problems is through the calculus of variations. In this setting the solution is represented by a contour, which parameterizes an energy functional depending on various image based quantities such as intensities or gradients. In general, the set of possible contours constitutes the solution space, where the goal is to find the contour which extremizes the energy in this space. As an optimization problem, there are many possible strategies to find this solution. One approach is to use the method of graph cuts to find a global optimum [1]. However, this can only be applied to a small class of energy functionals. For more general problems, the standard method has been to deform an initial contour in the steepest (gradient) descent of the energy. Equations of motion for the contour is derived using the Euler-Lagrange equation and the condition that the first variation of the energy functional should vanish at a (local) optimum. Then, the contour is evolved to steady-state given the resulting equations. A standard implementation of this strategy is usually hampered by two common problems. The first problem is sensitivity to local optima, which are manifested due to noisy data. To avoid this, the usual approach has been to modify the energy functional by adding regularizing terms. The second common problem is poor convergence due X.-C. Tai et al. (Eds.): SSVM 2009, LNCS 5567, pp. 124–136, 2009. c Springer-Verlag Berlin Heidelberg 2009 

Momentum Based Optimization Methods for Level Set Segmentation

125

to difficulties in choosing good initial conditions. To improve convergence, very effective solvers based on multi-grid [2, 3] and AOS schemes [4, 5, 6] have been developed. However, these methods all search for a solution in the gradient descent direction and little focus has been given to the underlying optimization problem. This has been identified in recent work [7, 8], where the metric defining the notion of steepest descent (gradient) has been studied. By changing the metric in the solution space, local optima due to noise are avoided in the search path. Along the same direction, this paper presents an alternative search strategy for the optimization solver. Our idea stems from the machine learning community, where an optimization problem is solved to update the system to adapt to a given stimulus. A simple, but effective, modification to gradient descent was proposed in [9], which basically adds a momentum to the motion in solution space. This simulates the physical properties of inertia and momentum and effectively allows the search to avoid local optima and accelerate in favorable directions. In this paper, we show how this idea can be used for image segmentation in a variational framework using level set methods. The results show faster convergence and less sensitivity to local optima. The paper will proceed as follows. In Section 2, we describe the idea of gradient descent with momentum in a general setting and give examples highlighting the benefits. Then, Section 3 presents how this idea can be used to solve segmentation problems in a level set framework. This is exemplified in Section 4 and Section 5 where we give implementation details and compute segmentations given a common energy functional. Finally, Section 6 concludes the paper and presents ideas for future work.

2

Gradient Descent with Momentum

Considering general optimization problems, gradient descent is a very simple approach which can handle many types of cost functions. It is intuitive, since it always moves in the direction of steepest descent, which locally gives the largest amount of decrease in the cost function. In addition, it only requires first order derivatives of the function, providing simple and fast computations. On the other hand, it is well known that gradient descent suffers from poor convergence and high sensitivity to local optima for many practical problems. Therefore, other descent directions (Newton, Quasi-Newton, etc.) have been studied and proved superior, see e.g. [10] for a rigorous reference. A simple alternative to these, more theoretically sophisticated methods, is often applied in the machine learning community. A typical problem here is the construction of adaptive systems that can classify unknown inputs. This can be formulated as an optimization problem and one of the goals of machine learning is to construct fast learning or adaptation rules that can be implemented in very simple hardware or software devices. To improve the convergence and robustness of a simple gradient descent solution, while avoiding the complexity of more sophisticated optimization methods, gradient descent with momentum

126

G. Läthén et al.

was proposed [9]. The starting point of our derivation of the proposed method is the following description of a standard line search optimization method: xk+1 = xk + sk

(1)

sk = αk pk

(2)

where xk is the current iterate, sk is the next step consisting of length αk in direction pk . To guarantee convergence, it is often required that pk be a descent direction while αk gives a sufficient decrease in the cost function. A simple realization of this is gradient descent which moves in the steepest descent direction according to pk = −∇fk , where f is the cost function, while αk satisfies the Wolfe conditions [10]. Turning to gradient descent with momentum, we will adopt some terminology from the machine learning community and choose a search direction according to: sk = −η(1 − ω)∇fk + ωsk−1 (3) where η is the learning rate and ω ∈ [0, 1] is the momentum. Note that ω = 0 gives standard gradient descent sk = −η∇fk , while ω = 1 gives “infinite inertia” sk = sk−1 . The intuition behind this strategy is that the current iterate has an inertia, which prohibits sudden changes in the velocity. This will effectively filter out high frequency changes in the cost function and allow for greater steps in favourable directions. Selecting appropriate parameters, our hope is that the rate of convergence is increased while eventual local optima will be overstepped. The effect of the momentum term is illustrated in Figure 1. The iterates with momentum ω = 0 show the behaviour of standard gradient descent when varying the learning rate (step length) η. In comparison, for an appropriate choice of momentum ω = 0.1, the solution approaches the optimum more rapidly. It can be seen however, that too high momentum of ω = 0.4 leeds to oscillations. 1

10

100 90

0

10

80 −1

10

70 60

−2

10

50 −3

10

40

−4

30

10

20

−5

10

10 −6

20

40

60

80

(a) Iterates and cost function

100

10

0

η = 0.04, ω = 0 η = 0.4, ω = 0 η = 0.4, ω = 0.1 η = 0.4, ω = 0.4 2

4

6

8

10

(b) Convergence rate

Fig. 1. Gradient descent search with/without momentum on a quadratic cost function

Momentum Based Optimization Methods for Level Set Segmentation

3

127

Minimizing Level Set Flows

As was previously outlined, segmentation problems in image analysis are often described as optimization problems with solutions derived using the calculus of variations. The standard procedure is to formulate an energy functional by means of a contour and various image derived terms. Extremals of this functional are then identified by an Euler-Lagrange equation, which is used to derive equations of motion for the contour [11]. This typical procedure yields a gradient descent search in a high dimensional solution space, in which each possible contour is represented by a point. For example [11] presents, among others, the derivation of weighted region described by the following functional:  E(C) = f (x, y)dxdy (4) ΩC

where C is a 1D curve embedded in a 2D domain, ΩC is the inside region of C, and f (x, y) is a scalar function. This functional is used to maximize some quantity given by f (x, y) inside C. A simple example is f (x, y) = 1 which measures, and maximizes, the area. Calculating the first variation of Eq. (4) yields the evolution equation: ∂C = −f (x, y)n (5) ∂t where n is the curve normal. Again, setting f (x, y) = 1 gives a constant flow in the normal direction, typically referred to as the “balloon force”. The representation, or parameterization, of the contour C can in general be chosen arbitrarily. However, it is often convenient to use the implicit level set method by Osher and Sethian [12], since this allows for arbitrary topological changes. To summarize the basic ideas, a contour is represented implicitly as a zero level set of a time dependent scalar function (referred to as the level set function). Formally, a contour C is described by C = {x : φ(x, t) = 0}. To deform C, the level set function is evolved in time according to a set of partial differential equations (PDEs). The transition from the equations of motion for a parametrized curve (Eq. (5)) to a level set PDE is accomplished by a simple procedure. In general, the motion ∂C ∂t = γn translates to the level set equation ∂φ ∂t = γ |∇φ| [11]. Thus, Eq. (5) gives the familiar level set equation: ∂φ = −f (x, y) |∇φ| ∂t

(6)

The remainder of this section will describe how we modify the typical level set method update scheme to incorporate a momentum term as presented in Section 2. 3.1

Momentum for Minimizing Level Set Flows

We have noted that the contour evolving according to the Euler-Lagrange equation yields a gradient descent search. Recall that each contour can be represented as a point in the solution space (the structure of the space will depend on

128

G. Läthén et al.

parameterization). Thus, we can approximate the direction of the gradient by computing the vector between two subsequent points. In the level set framework we achieve this by taking the difference between two subsequent time instances of the level set function, representing the entire level set function as one vector: φ(tn ) − φ(tn−1 ) (7) Δt where f is a cost function in compliance with the terminology used in Section 2. Note that this is indeed an approximation, depending on the time difference Δt = tn − tn−1 . Following the ideas from Section 2, we update the level set function to incorporate a momentum term: ∇f (tn ) ≈

 n ) − φ(tn−1 ) φ(t + ωs(tn−1 ) Δt φ(tn ) = φ(tn−1 ) + Δts(tn ) s(tn ) = −η(1 − ω)

(8) (9)

The complete procedure works as follows: Procedure UpdateLevelset 1

Given the level set function φ(tn−1 ), compute the next (intermediate)  n ). This is performed by evolving φ according to a PDE time step φ(t (such as Eq. (6)) using standard techniques (e.g. Euler integration).

2

Compute the approximate gradient by Eq. (7).

3

Compute a step s(tn ) according to Eq. (8). This step effectively modifies the gradient direction by incorporating the momentum term as a fraction of the previous step s(tn−1 ).

4

Compute the next time step φ(tn ) by Eq. (9). Note that this replaces the intermediate level set function computed in Step 1.

The procedure is very simple and is directly compatible with any type of level set implementation.

4

Experiments

We now describe some details of the implementation and illustrate properties of the suggested method using two examples. Here we study 1D curves embedded in a 2D domain, but the approach readily generalizes to 2D surfaces in 3D given the level set framework. 4.1

Implementation Details

We have implemented the proposed ideas in Matlab using standard level set techniques based on [13, 14]. Reference code can be found online at the site http://dmforge.itn.liu.se/ssvm09/. Some details of our implementation are the following:

Momentum Based Optimization Methods for Level Set Segmentation

129

– The level set function is reinitialized (reset to a signed distance function) after Step 1 and Step 4. This is typically performed using the fast marching [15] or fast sweeping algorithms [16]. There are two reasons for this: Firstly it is required for stable evolution in time due to the use of explicit Euler integration. Secondly we want a momentum induced by the zero level set of φ (the contour), rather than all level sets of φ. Reinitialization could be omitted, with the effect of introducing a momentum on all individual level sets. Interpreting each sample of φ as a parameter of the contour, this is equivalent to applying momentum on each parameter. While feasible, we have not experimented with momentum without incorporating reinitialization. – We avoid instabilities by dampening s(tn ) in Step 3 using a sigmoidal function: sˆ(s(tn ), smax ) =

1+

2smax −2s(t n )/smax e

− smax

(10)

where smax is the maximum step length allowed. – Any explicit or implicit time integration scheme can be used in Step 1. Due to its simplicity, we have used explicit Euler integration which might require several inner iterations in Step 1 to advance the level set function by Δt time units. 4.2

Weighted Region Based Flow

To verify our idea, we have used a simple energy functional based on a weighted region term (Eq. (4)) combined with a penalty on curve length for regularization. The goal is to maximize:   E(C) = f (x, y)dxdy − α ds (11) ΩC

C

where α is a regularization weight parameter. The target function f (x, y) is image based, computed using the approach in [17]. This method uses quadrature filters [18] across multiple scales to detect line structures. Taking the real part of the complex filter response, f (x, y) gives positive values on the inside of linear structures, negative on the outside, and zero on the edges. Translating Eq. (11) to a level set PDE following [11] gives: ∂φ = −f (x, y) |∇φ| + ακ |∇φ| ∂t

(12)

where κ is the curvature of the contour. First we illustrate some properties of the method with a synthetic test image depicted in Figure 2(a), which mimics the common problem of intensity variation in medical imaging. The intensity of the object ranges from 0.3 to 1, while the noise level is 0.1. This image yields the target function f (x, y) in Figure 2(b) where bright and dark colors indicate positive and negative values respectively. As exemplified in our first experiment (Figure 3) the dip in contrast results in a local optimum in the solution space.

130

G. Läthén et al.

(b) Target function f (x, y)

(a) Input image

Fig. 2. Synthetic test image illustrating the presence of a local optima in the solution space

(a) time = 0

(b) time = 40

(c) time = 100

(d) time = 170

(e) time = 300

(f) time = 870

Fig. 3. Iterations without momentum (conventional gradient descent)

Figure 3 shows the results after evolving the level set function by Eq. (12) until convergence without momentum, using conventional methods. We define convergence as |∇f |∞ < 0.03 (using the infinity/maximum norm), with ∇f given in Eq. (7). For this experiment we used parameters α = 0.7 and we reinitialized the level set function every fifth time unit. For comparison, Figure 4 shows the results after running our method using parameters α = 0.7, ω = 0.8, η = 10, smax = 100, Δt = 5. Plots of the energy functional for both experiments are shown in Figure 5. Here, we plot the weighted area term and the length penalty term separately, to illustrate the balance between the two. Note that the functional without momentum in Figure 5(a) is monotonically increasing, due to the nature of gradient descent, while the functional with momentum visits a number of local maxima during the search. To further exemplify the behaviour of our method, we created a slightly modified version of Figure 2(a), shown in Figure 6(a). In contrast to Figure 2(a), the shape in Figure 6(a) is disconnected, so the global optimum is expected to contain two separated regions. Not surprisingly, conventional gradient descent captures only a local minimum as displayed in Figure 7, while gradient descent with momentum succeeds in capturing the global solution as two separated

Momentum Based Optimization Methods for Level Set Segmentation

(a) time = 0

(b) time = 20

(c) time = 40

(d) time = 60

(e) time = 150

(f) time = 200

(g) time = 245

(h) time = 320

(i) time = 460

131

Fig. 4. Iterations using momentum 1800

1800 Energy functional Length penalty integral Target function integral

1600

1600

1400

1400

1200

1200

1000

1000

800

800

600

600

400

400

200

200

0 0

100

200

300

400 500 time

600

700

(a) Without momentum

800

0 0

Energy functional Length penalty integral Target function integral 100

200

300

400

time

(b) With momentum

Fig. 5. Plots of energy functionals for synthetic test image in Figure 2(a)

regions (Figure 8). For this experiment, we used the same parameters as in Figure 3 and Figure 4. As a third test image we used a 458 × 265 retinal image from the DRIVE database [19], shown in Figure 9(a). The target function f (x, y) is illustrated in Figure 9(b). As in the previous experiment, bright and dark colors indicate positive and negative values for f (x, y). The convergent result without momentum using parameters α = 0.07 and reinitialization every tenth time unit is shown in Figure 10, given the initial condition in Figure 10(a). Applying

132

G. Läthén et al.

(b) Target function f (x, y)

(a) Input image

Fig. 6. Synthetic test image illustrating the presence of a local optima in the solution space

(a) time = 0

(b) time = 200

(c) time = 515

Fig. 7. Iterations without momentum (conventional gradient descent)

(a) time = 0

(b) time = 40

(c) time = 70

(d) time = 180

(e) time = 240

(f) time = 485

Fig. 8. Iterations using momentum

the idea of momentum yields the result in Figure 11, using the parameters α = 0.07, ω = 0.5, η = 1.3, smax = 40, Δt = 10. The energy functionals are plotted in Figure 12 to display the convergence of both methods.

5

Results

The synthetic test image in Figure 2(a) illustrates a local optimum in the solution space when applying the parameters in our first experiment. As expected,

Momentum Based Optimization Methods for Level Set Segmentation

(a) Input image

133

(b) Target f (x, y)

Fig. 9. Retinal image

(a) time = 0

(b) time = 20

(c) time = 40

(d) time = 100

(e) time = 200

(f) time = 400

(g) time = 600

(h) time = 1210

Fig. 10. Iterations without momentum (conventional gradient descent)

134

G. Läthén et al.

(a) time = 0

(b) time = 20

(c) time = 40

(d) time = 100

(e) time = 200

(f) time = 400

(g) time = 600

(h) time = 820

Fig. 11. Iterations using momentum 9000

9000 8000

8000

7000

7000

6000

6000

5000

Energy functional Length penalty integral Target function integral

4000

5000

3000

3000

2000

2000

1000

1000

0 0

200

400

600 time

800

1000

(a) Without momentum

1200

Energy functional Length penalty integral Target function integral

4000

0 0

100

200

300

400 time

500

600

700

(b) With momentum

Fig. 12. Plots of energy functionals for the retinal image in Figure 9(a)

800

Momentum Based Optimization Methods for Level Set Segmentation

135

the conventional gradient descent approach converges to this local optimum as depicted in Figure 3. In contrast, our proposed method gains enough momentum in order to overstep the optimum, while at the same time the global solution is reached more rapidly. The process (illustrated in Figure 4) intuitively expands the curve beyond a local optimum, followed by a retraction if the search does not provide any increase in that direction. Using a slightly modified input image, our second example shows that our method is capable of capturing global optima, even when the solution consists of separated regions (Figure 8). Our third example illustrates our method on real data using a retinal image. In Figure 10 we see that conventional gradient descent fails to capture many weak signal blood vessels. This is a typical case of local optima solutions introduced by noise and poor image contrast. Under the same conditions, gradient descent with momentum captures practically all visible vessels as shown in Figure 11. Note that this example does not include any verification of the accuracy of the segmented vessels. The primary purpose is to illustrate that our method reaches a stronger optimum value for the energy functional, as shown in Figure 12.

6

Conclusions and Future Work

In this paper we have presented the idea of gradient descent with momentum in the context of segmentation using the level set method. We have illustrated the drawbacks of conventional gradient descent and showed examples on how the solution is improved by adding momentum. In contrast to much of the previous work, we have improved the solution by changing the method of solving the optimization problem rather than changing the parameters of the energy functional. In the future, we will further study the general optimization problem of image segmentation to propose more efficient solutions. Regarding the particular idea of momentum, we will apply this on real applications and verify the quality of the results.

References 1. Boykov, Y., Kolmogorov, V.: Computing geodesics and minimal surfaces via graph cuts. In: Proc. ICCV 2003, October 2003, vol. 1, pp. 26–33 (2003) 2. Papandreou, G., Maragos, P.: Multigrid geometric active contour models. IEEE Transactions on Image Processing 16(1), 229–240 (2007) 3. Kenigsberg, A., Kimmel, R., Yavneh, I.: A multigrid approach for fast geodesic active contours. Technical Report CIS-2004-06, Technion–Israel Inst. Technol., Haifa (2004) 4. Paragios, N., Mellina-Gottardo, O., Ramesh, V.: Gradient vector flow fast geometric active contours. IEEE Transactions on Pattern Analysis and Machine Intelligence 26(3), 402–407 (2004) 5. Goldenberg, R., Kimmel, R., Rivlin, E., Rudzsky, M.: Fast geodesic active contours. IEEE Transactions on Image Processing 10(10), 1467–1475 (2001)

136

G. Läthén et al.

6. Weickert, J., Kühne, G.: Fast methods for implicit active contour models. In: Geometric Level Set Methods in Imaging, Vision and Graphics. Springer, Heidelberg (2003) 7. Charpiat, G., Keriven, R., Pons, J.P., Faugeras, O.: Designing spatially coherent minimizing flows for variational problems based on active contours. In: Proc. ICCV 2005, October 2005, vol. 2, pp. 1403–1408 (2005) 8. Sundaramoorthi, G., Yezzi, A., Mennucci, A.: Sobolev active contours. International Journal of Computer Vision 73(3), 345–366 (2007) 9. Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning internal representations by error propagation, pp. 318–362. MIT Press, Cambridge (1986) 10. Nocedal, J., Wright, S.J.: Numerical Optimization, 2nd edn. Springer, Heidelberg (2006) 11. Kimmel, R.: Fast edge integration. In: Geometric Level Set Methods in Imaging, Vision and Graphics. Springer, Heidelberg (2003) 12. Osher, S., Sethian, J.A.: Fronts propagating with curvature-dependent speed: Algorithms based on Hamilton-Jacobi formulations. Journal of Computational Physics 79, 12–49 (1988) 13. Osher, S., Fedkiw, R.: Level Set and Dynamic Implicit Surfaces. Springer, New York (2003) 14. Peng, D., Merriman, B., Osher, S., Zhao, H.K., Kang, M.: A pde-based fast local level set method. Journal of Computational Physics 155(2), 410–438 (1999) 15. Sethian, J.: A fast marching level set method for monotonically advancing fronts. Proceedings of the National Academy of Science 93, 1591–1595 (1996) 16. Zhao, H.K.: A fast sweeping method for eikonal equations. Mathematics of Computation (74), 603–627 (2005) 17. Läthén, G., Jonasson, J., Borga, M.: Phase based level set segmentation of blood vessels. In: Proc. ICPR 2008, Tampa, FL, USA, IAPR (December 2008) 18. Granlund, G.H., Knutsson, H.: Signal Processing for Computer Vision. Kluwer Academic Publishers, Netherlands (1995) 19. Staal, J., Abramoff, M., Niemeijer, M., Viergever, M., van Ginneken, B.: Ridge based vessel segmentation in color images of the retina. IEEE Transactions on Medical Imaging 23(4), 501–509 (2004)

Optimization of Divergences within the Exponential Family for Image Segmentation Francois Lecellier1 , Stephanie Jehan-Besson2, Jalal Fadili1 , Gilles Aubert3 , and Marinette Revenu1 1

Laboratoire GREYC, University of Caen, France Laboratoire LIMOS, University of Clermont-Ferrand, France Laboratoire J.A. Dieudonné, University of Nice Sophia-Antipolis, France 2

3

Abstract. In this work, we propose novel results for the optimization of divergences within the framework of region-based active contours. We focus on parametric statistical models where the region descriptor is chosen as the probability density function (pdf) of an image feature (e.g. intensity) inside the region and the pdf belongs to the exponential family. The optimization of divergences appears as a flexible tool for segmentation with and without intensity prior. As far as segmentation without reference is concerned, we aim at maximizing the discrepancy between the pdf of the inside region and the pdf of the outside region. Moreover, since the optimization framework is performed within the exponential family, we can cope with difficult segmentation problems including various noise models (Gaussian, Rayleigh, Poisson, Bernoulli ...). We also experimentally show that the maximisation of the KL divergence offers interesting properties compare to some other data terms (e.g. minimization of the anti-log-likelihood). Experimental results on medical images (brain MRI, contrast echocardiography) confirm the applicability of this general setting.

1 Introduction We propose here to focus on the segmentation of homogeneous regions in noisy images using statistical region-based active contour models (RBAC). In RBAC, region-based terms can be advantageously combined with boundary-based ones [1, 2]. The evolution equation is generally deduced from a general criterion to minimize that includes both region integrals and boundary integrals. The combination of those two terms in the energy functional allows the use of photometric image properties, such as texture [3] and noise [4], as well as geometric properties such as the shape prior of the object to be segmented. In statistical region-based active contours, see [5] for a review, image features (e.g. intensity) are considered as random variables whose distribution may be parametric (e.g. Gaussian) or non parametric [6]. Classically, the authors consider the minimization of the anti-log-likelihood for segmentation [7,8,4]. In this paper, we rather focus on the optimization of distance between pdfs. Such distances or more generally divergences can be used in two different manner. On the one hand, they can be used for segmentation with distribution intensity prior and in this case, we aim at minimizing the distance between the pdf of the evolving region and a reference one. On the other hand, they can be used for segmentation without reference and in this second case, we aim X.-C. Tai et al. (Eds.): SSVM 2009, LNCS 5567, pp. 137–149, 2009. c Springer-Verlag Berlin Heidelberg 2009 

138

F. Lecellier et al.

at maximizing the distance between the pdf of the inside region and the pdf of the outside region. In the literature, the minimization of divergences between non parametric pdfs has first been proposed in [6] for video sequences. It has then been developed for cardiac structures tracking in perfusion MRI (p-MRI) sequences in [9]. As far as segmentation using the maximization of divergences is concerned, some authors [10] have also proposed to take benefit of the maximization of the Bhattacharya distance of non parametric pdfs for segmentation. On the other hand, divergences between Gaussian distributions have been developed for DTI segmentation in [11]. In this paper, we propose to set a general framework for the optimization of divergences between parametric pdfs within the exponential family. To the best of our knowledge, such a framework has never been studied for region-based active contour segmentation. The rationale behind using the exponential family is that it includes, among others, Gaussian, Rayleigh, Poisson and Bernoulli distributions that have proven to be useful to model the noise structure [4] in many real image acquisition devices (e.g. Poisson for photon counting devices such as X-ray or CCD cameras, Rayleigh for ultrasound images, etc). Using shape derivative tools as in [12, 6], our effort focuses on constructing a general expression for the derivative of the energy (with respect to a domain), and on deriving the corresponding evolution speed. Our general framework is also specialized to some particular cases, such as the optimization of the KullbackLeibler (KL) divergence [13], which gives a simple expression of the derivative. This theoretical framework is then more explicitly detailed and illustrated for the case of the segmentation without reference. In this case, we aim at maximizing the dissimilarity between the pdf of the intensity within the region inside the evolving contour and the pdf of the intensity within the region outside the contour. In other words, we perform a competition between the pdfs of these two regions through the maximisation of divergences. Experimental results are given for the particular case of the KL divergence. We experimentally compare this data term to the classical minimization of the anti-log-likelihood [7, 14] for the segmentation of the White Matter in brain MRI and we show that KL maximisation is able to extract a single Gaussian from a mixture of Gaussian. We also show the applicability of our data term for the segmentation of the left ventricle in contrast echocardiography where the noise is modelled using Rayleigh. In this paper, we first set our general setting and introduce shape gradients in section 2. In section 3, we propose to give some general results for the exponential family and then for the shape derivative of divergences between pdfs. These results are then specialized for the KL divergence using the Maximum Likelihood Estimation (MLE) for the parameters. Experimental results for the maximization of KL divergence are given in section 4.

2 Optimization of Divergences between Pdfs: General Setting In this section, we set our general setting for segmentation through the optimization of distances between pdfs or more generally divergences. 2.1 General Setting Consider a function y : Rn → χ ⊂ R which describes the feature of interest. The term y(x) then represents the value of the feature y at location x where x ∈ Rn . Let q(y, Ω)

Optimization of Divergences within the Exponential Family for Image Segmentation

139

be the probability density function (pdf) of the feature y within the image region of interest. We now assume that we have a function Ψ : R+ × R+ → R+ which allows us to compare two pdfs. This function is small if the pdfs are similar and large otherwise. It allows us to introduce the following functional which represents the distance or more generally the divergence between the current pdf estimate q(y, Ω) and another one p(y) which may also depend on another domain:  D(Ω) = Ψ (q(y, Ω), p(y)) dy. (1) χ

The distance can be for example the symmetrized Kullback-Leibler divergence when p(y) Ψ (q, p) = 12 (p(y) log q(y,Ω) + q(y, Ω) log q(y,Ω) p(y) ). Such divergences represent a general setting for both segmentation with and without reference. Indeed, in segmentation problems, we generally search for homogeneous regions regarding with a given feature. We may then modelize the segmentation problem as the maximization of the distance between the pdf of the feature within the inside region and the pdf of the feature within the outside region. In order to fix ideas, let us consider a partition of an image in two regions where Ω is the inside region and Ω c the complementary outside region. The segmentation may then be formulated as the maximization of the following criterion:  D(Ω, Ω c ) = Ψ (q(y, Ω), p(y, Ω c )) dy. (2) χ

On the other hand, we can also consider that we have a reference histogram pref and that we search for the domain that minimizes the divergence between q and pref . This last framework may be applied to tracking or to supervised segmentation where a reference pdf is learned on the region of interest. The theoretical results given in this paper can be used for both applications. 2.2 Shape Gradient Descent In order to find an optimum, we perform a shape gradient descent using region-based active contours. We then have to compute the derivative of the criterion according to the domain using shape derivation tools [15]. Shape derivative tools applied to region-based active contours are described in [12, 6] and we won’t remind all the definitions in this paper. Let us just remind that, from the shape derivative, we can derive the evolution equation that will drive the active contour towards a (local) minimum of the criterion. Let us suppose that the shape derivative of the criterion D(Ω) in the direction V may be written as follows: < D (Ω), V >= −



speed(x, Ω)(V(x) · N(x))da(x) ,

(3)

∂Ω

where N is the unit inward normal to ∂Ω and da its area element. When minimizing the distance D(Ω), interpreting equation (3) as the L2 inner product on the space of velocities, the straightforward choice is to take V = speed(x, Ω)N. When minimizing the distance D(Ω), we can then deduce the following evolution equation: ∂Γ = speed(x, Ω) N(x) , ∂τ

(4)

140

F. Lecellier et al.

On the contrary, when maximizing the criterion, we take the opposite sign for the velocity.

3 General Results for Shape Derivative of Divergences within the Exponential Family In this paper, we consider that pdfs belong to the exponential family. In this case, the current pdf estimate q(y, Ω) is now indexed by a set of parameters θ ∈ Θ ⊂ Rκ (e.g. we have κ = 2 and θ = (μ, σ)T where μ is the mean and σ the variance for the Normal family). When using the exponential family, we rather index the pdf by η which is the natural parameter as explained below. In order to derive the criterion, we must take into account the dependence of the natural parameter with the domain. We then restrict our study to the full rank κ-parameter canonical exponential family [16]. For this family, we can establish a 1-1 correspondence between η and Ω and so compute directly the shape derivative of D(Ω). In the sequel, let us first introduce the exponential family and some properties and then explain the computation of the shape derivative. We then specialize our result when parameters are estimated using the Maximum Likelihood Estimation (MLE) method. We also give some results for the optimization of the Kullback-Leibler (KL) divergence. In this case, the shape derivative reduces to a very simple general expression. 3.1 The Exponential Family: Definition and Properties The multi-parameter exponential family [17] is naturally indexed by a κ-dimensional real parameter vector η and a κ-dimensional natural statistic vector T(Y ). We draw the reader’s attention to the fact that η is a function of θ ∈ Θ which is the parameter of interest in most applications (for the Gaussian distribution, we have θ = (μ, σ)T ). Definition 1. The family of distributions of a Random Variable (RV) Y {qθ : θ ∈ Θ ⊆ Rκ }, is said a κ-parameter canonical exponential family, if there exists real-valued functions: • • • •

η(θ) = [η 1 , ..., η κ ]T with η i : Θ ⊆ Rκ → R h:R→R B:Θ→R T = [T1 , ..., Tκ ]T : Rκ → R

such that the pdf qθ (y) may be written as: qθ (y) = h(y) exp[η(θ), T(y) − B(θ)]

with

y∈χ⊂R.

(5)

The term T is called the natural sufficient statistic and η the natural parameter vector. The term η, T denotes the scalar product. Letting the model be indexed by the natural parameter η rather that θ, the canonical κ−parameter exponential family generated by T and h is defined as follows: qη (y) = h(y) exp[η(θ), T(y) − A(η)] , (6)  +∞ with A(η) = log −∞ h(y) exp[η(θ), T(y)]dy. The natural parameter space is defined as E = {η ∈ Rκ ; −∞ < A(η) < +∞}.

Optimization of Divergences within the Exponential Family for Image Segmentation

141

Some Common Distributions. Table 1 provides a synthetic description of some common distributions of the exponential family: Table 1. Some common canonical exponential families. B(α, β) is the Euler Beta function. Distribution

θT

η(θ)T

T(y)T

Normal

(μ, σ 2 )

−1 ( σμ2 , 2σ 2)

(y, y 2 )

1 2



A(η) η2

− 2η12 − log

−η2 π



Gamma (λ, p) (−λ, p − 1) (y, log y) −(η2 + 1) log −η1 + log Γ (η2 + 1) Beta (r, s) (r − 1, s − 1) (log y, log(1 − y)) − log B(η1 + 1, η2 + 1) Poisson μ log μ y eη Exponential λ −λ y − log −η Rayleigh θ2 −1/2θ2 y2 − log −2η

Properties. The following results will be useful for our RBAC scheme based on the exponential family. Their proofs may be found in [16]. These properties give us a relation between the parameters η and the domain Ω through the use of the expectation of the natural statistics T(Y ). The first theorem provides general relations between the gradient of A and the expectation of T(Y ) while the second theorem allows us to establish a 1 − 1 correspondence between η and E[T(Y )] (for the full rank exponential family). Such a relation may then be used to express the parameter η and derive it according to the domain. Theorem 1. Let {qη : η ∈ E} a κ-parameter canonical exponential family with natural sufficient statistic T(Y ) and open natural parameter space E, we then have the following properties: 1. 2. 3. 4.

E is convex. A:E → S ⊆ R is convex. E[T(Y )] = ∇A(η). ¨ Cov[T (Y )] = A(η).

∂A ∂A ∂A T where ∇A = ( ∂η , , .., ∂η ) represents the gradient of A, and A¨ is the Hessian 1 ∂η2 κ 2 matrix of A with A¨ij = ∂ A . ∂ηi ∂ηj

The following theorem establishes the conditions of strict convexity of A, and then those for ∇A to be 1-1 on E. This is a very useful result for optimization (derivation) purposes: Theorem 2. Let {qη : η ∈ E} a full rank (i.e. Cov[T (Y )] is a positive-definite matrix) κ-parameter canonical exponential family with natural sufficient statistic T(Y ) and open natural parameter space E, we have [16]: 1. η → ∇A(η) is 1-1 on E. 2. The family may be uniquely parameterized by μ(η) ≡ E[T(Y )] = ∇A(η). 3. The anti-log-likelihood function is a strictly convex function of η on E. These results establish a 1-1 correspondence between η and E[T(Y )] such that: μ = ∇A(η) = E[(T(Y )] ⇔ E η = φ (E[T(Y )]) , holds uniquely with ∇A and φ continuous.

(7)

142

F. Lecellier et al.

Estimation of the Hyperparameters. The relation 7 allows us to express the parameter η as a function of E[(T(Y )]. In order to estimate the parameters, we replace E[T(Y )] by the empirical estimate of the mean T(Y ). This corresponds to the MLE of the parameter. Indeed, the MLE of η corresponds to minimizing the anti-log-likelihood score (for independent and identically distributed (iid) data). By differentiation of the anti-log-likelihood according to η, we find ∇A(η MLE ) = T (Y ). Note however that in this case, this is the discrete sample mean. The following example illustrates this stating: −1 Example 1. When dealing with the Rayleigh distribution, we have η = 2θ 2 , A(η) = 2  − log(−2η) and T (y) = y . By computing A (η) = T(Y ), we find that − η1 =  1 2 y(x)2 dx, which corresponds to the MLE of the parameter θ2 given by θML |Ω| Ω  1 = 2|Ω| y(x)2 dx. Ω

3.2 Shape Derivative of the Criterion In this section, we propose to derive according to the domain the functional (1). The dependence of the functional with the domain is due to the estimation of the parameter η detailed above. In the sequel, for the sake of simplicity, we will invariably denote η for the natural parameter and its finite sample estimate over the domain (without a ˆ ). We are now ready to state our main result: slight abuse of notation, this should be η Theorem 3. The Gâteaux derivative, in the direction of V, of the functional (1), is: < D (Ω), V >= ∇V η, C ,

(8)

where ∇V η = [< ∇η1 (Ω), V >, ..., < ∇ηκ (Ω), V >] is the Gâteaux derivative of η in the direction of V, ., . is the usual scalar product of two vectors and: C = E[∂1 Ψ (q(Y, η(Ω)), p(Y ))(T(Y ) − E[T(Y )])]. The term ∂1 Ψ denotes the partial derivative of Ψ according to the first variable. The proof is detailed in Appendix A.2. We then have to compute the shape derivative ∇V η. Such a computation requires an estimation of the expectation E[T(Y )] as explained in the next section. 3.3 Computing the Shape Derivative for the MLE Estimator As mentioned in section (3.1.3), the expectation E[T(Y )] can be replaced with the empirical estimate of the mean T(Y ) which is computed over the considered domain Ω. Using such an estimation for the hyperparameter, we can state the following proposition: Lemma 1. Within the full rank exponential family, and using the MLE estimator for the hyperparameters, the shape derivative ∇V η can be expressed as: ¨ −1 ∇V (T) . ∇V η = A(η)

(9)

Optimization of Divergences within the Exponential Family for Image Segmentation

143

¨ −1 = I(η)−1 is the inverse of the Hessian matrix of A¨ which is also the where A(η) fisher information matrix I. The derivative ∇V (T) is given by:    1 T(y) − T(y(a)) (V · N)da(x) , (10) ∇V (T) = |Ω| ∂Ω The proof is given in Appendix A.3. We can then replace the shape derivative of the natural parameters given in Lemma 1 in the general Theorem 3. The corollary that gives the shape derivative then follows: Corollary 1. The Gâteaux derivative, in the direction of V, of the functional (1), is: 1 < D (Ω), V >= |Ω| 

  κ

∂Ω

i=1

Ci

κ 

−1 ¨ [A(η)]ij (Tj (y) − Tj (y(a))) (V · N)da ,

j=1

where the κ components of the vector C are defined as follows: Ci = E[∂1 Ψ (q(Y, η(Ω)), p(Y ))(Ti (Y ) − Ti (Y )] i ∈ [1, κ]. The term ∂1 Ψ denotes the partial derivative of Ψ according to the first variable. In order to fix ideas, the functional D(Ω) can be chosen as the Kullback-Leibler divergence, in this case ∂1 Ψ (q, p) = log q + 1 − log p − pq . In order to compute the vector C in Corollary 1, we can assume that the pdf p belongs to the exponential family and to the same parametric law as the pdf q. Let us denote by η 1 the parameter of the pdf p. This parameter is supposed to be already computed or dependent of another domain and so does not depend on the domain Ω. We then state the following proposition: Lemma 2. When p(y, η 1 ) and q(y, η(Ω)) are two members of the exponential family that belong to the same parametric law with respective parameters η 1 and η, and when the functional D(Ω) is chosen as the KL divergence, we find for the vector C defined in Theorem 1: ¨ C = A(η)(η − η 1 ) + ∇A(η) − ∇A(η 1 ) . A proof is given in appendix C. This expression demonstrates that the derivative can be very simply computed using the natural parameters and the sufficient statistics of the law. Let us give two examples of computation for both the Rayleigh and the Gaussian law. Example 2. When dealing with the Rayleigh following example 2, with  distribution,  θ12 1 2 2 2 θ2 θ = 2 y , the term C is equal to C = 2θ θ2 − θ2 . we then find for the derivative 1 of KL divergence:  1 y(a)2 C < KL (Ω), V >= (1 − )(V · N)da(x) . (11) 2 |Ω| ∂Ω 2θ 2θ2 Example 3. When dealing with the Gaussian distribution, the term C is equal to

2 C=

( σσ 2 + 1)(μ − μ1 ) r 2  μ2 + σ 2 − μ21 − σ12 + 2 σσ 2 μ2 − μμ1 + σ 4 ( σ12 − 1

1

1 ) σ2

.

(12)

144

F. Lecellier et al.

We then find for the derivative of KL divergence: < KL (Ω), V >=    2μ 1 μ −(y − μ) C (1 + ) − C 1 2 2 σ 2 |Ω| ∂Ω σ2 σ   C2 μ +(y 2 − σ 2 − μ2 ) C1 2 − 2 (V · N)da(x) . σ 2σ

4 Maximisation of Divergences In this section, we propose to concentrate on the segmentation of an image into two regions (namely Ω and its complement Ω c ) by maximizing the criterion 2. 4.1 Evolution Equation When using the MLE estimator for the parameters, and noting that Ω and Ω c shares 1 the same boundary with opposite normals, we take T(y) = |Ω| Ω T(y(x))dx and  c 1 T(y) = |Ω c | Ω c T(y(x))dx. Using Corollary 1 and the fact that < D (Ω, Ω c ), V >= ∇V η, C + ∇V η c , Cc , we find for the evolution equation: κ κ  1   ∂Γ ¨ −1 (Tj (y) − Tj (y(x)) = Ci (Ω) A(η) ij ∂τ |Ω| i=1 j=1

−

κ κ   1  c ¨ c )−1 (Tj (y)c − Tj (y(x)) N. C (Ω ) A(η i ij |Ω c | i=1 j=1

For the KL divergence, the term C is evaluated as explained in section 3.3. A classical regularization term λκ is added where λ is a positive constant and κ the curvature. As far as the numerical implementation is concerned, we use the level set method approach first proposed by Osher and Sethian [18]. 4.2 Comparison with Other Methods in the Gaussian Case In this section, we propose to compare the behavior of our data term based on the maximization of the symmetrized Kullback-Leibler divergence between parametric pdfs to two other well-known region-based methods [7, 14]. The first method is the famous Chan & Vese method [14]. Such a criterion implies a Gaussian distribution for the feature y with a fixed variance. The corresponding evolution equation can be found in [14]. The second method has been first proposed by [7] and aims at minimizing the anti-loglikelihood for a Gaussian distribution. The evolution equation can be found in [7]. In order to compare these terms, let us express the non symmetrized KL divergence using the expectation under the pdf q, denoted by Eq , as follows: D(qp) = Eq [log(q(Y, η Ω ))] − Eq [log(p(Y, η Ω c ))]

(13)

Optimization of Divergences within the Exponential Family for Image Segmentation

145

To get the gist of using KLD as a criterion in an RBAC functional, consider the data yi = {y(x)|x ∈ Ω} as an iid sequence from the statistical model q(y, η Ω ). Using the weak law of large number for a very large domain Ω, the first term (which corresponds  1 to the entropy) can then be expressed as |Ω| log(q(y(x), η(Ω))dx. Maximizing the Ω first term in KL divergence can then be seen as equivalent to minimizing the anti-loglikelihood score [19] divided by the size of the sample (which corresponds to the entropy under the law of large number). Using the same assumptions, the second term of KL divergence can be seen as the minimization of the plausibility of the data provided by Ω c in the inside region Ω. When using the symmetrized version, we act both on Ω and Ω c . Let us now compare experimentally the behavior of these criterions for the extraction of an homogeneous region corrupted by a Gaussian noise in an image. We propose to take the example of the segmentation of the White Matter (WM) in T1-weighted brain MRI images. We perform the three evolution equations using the Gaussian assumption for the pdf of the feature y within each region. The feature y is chosen as the Intensity of the image. The initial contour is given in Figure 1.(a) and we also show the two initial pdfs (b), namely qη (I, Ω) which corresponds to the distribution of the intensity I inside the region Ω and qηc (I, Ω c ) which corresponds to the distribution of I inside

0.025 hist_in hist_out

0.02

0.015

0.01

0.005

0 0

50

100

150

200

Intensity

(a) initial contour

0.03

(b) associated pdfs

0.03

0.06

hist_in hist_out

hist_in hist_out

0.025

hist_in hist_out

0.025

0.05

0.02

0.02

0.04

0.015

0.015

0.03

0.01

0.01

0.02

0.005

0.005

0

0.01

0 0

50

100

150

Intensity

(c) Chan & Vese

200

0 0

50

100

150

Intensity

(d) log likelikood

200

0

50

100

150

200

Intensity

(e) KL maximization

Fig. 1. T1-weighted brain MRI segmentation results (extraction of the White Matter). The pdf of the intensity inside the contour is in solid line, the pdf of the intensity outside the contour is in dotted lines. (a): initial contour and (b) : associated pdfs, column (c): final contour and pdfs for the Chan & Vese method [14], column (d): for the log-likelihood method [7], column (e): for the maximization of the KL divergence.

146

F. Lecellier et al.

the region Ω c (i.e. outside the region ). In Figure 1, we can observe the final active contour obtained using our criterion (22) and the two other criterions mentioned above. We can remark that our criterion acts as an extractor of the most important Gaussian in the initial mixture of Gaussian (see Figure 1.e). The two other criterions separate the mixture without extracting a single Gaussian. So, with our method, we can directly obtain the White Matter of the brain without a multiphase scheme. 4.3 Examples of Applications In this part, we consider two examples of application (brain MRI images and contrast echocardiogaphy) using two different noise models (Gaussian and Rayleigh). Concerning 3D T1-weighted MRI images of the brain, the noise model is assumed to be represented by a Rician distribution [20]. For large signal intensities the noise distribution can be considered as a Gaussian distribution (this is the case for the White Matter (WM) or the Gray Matter (GM)). We propose in Figure 2 an example of WM segmentation by maximizing the KL divergence between Gaussian distributions. When evaluating quantitatively our results of WM segmentation on the simulated brain T1weighted MRI images provided by the Montreal Neurological Institute Brain Web URL, we find a dice coefficient of 0.91, a very law False Positive Fraction (FPF) of 0.8% and a True Positive Fraction (TPF) of 84%.

(a) 3D rendering of the WM

(b) slice 72

(c) slice 75

(d) slice 84

Fig. 2. 3D Segmentation of WM in a T1 brain MRI using KL maximization

As the Rayleigh distribution is well suited to model noise in echography [20], this noise model was applied for segmentation of the left ventricle in contrast echocardiography. Final contours for several images of the sequence are shown in Figure 3. The segmentation is accurate all along the sequence. Note that experimental results reported in [21, 4] prove that when using the appropriate noise model, segmentation results are more accurate and less sensitive to the choice of the regularization parameters.

frame 1

frame 31

frame 40

Fig. 3. Segmentation of the LV in a contrast echocardiographic sequence

Optimization of Divergences within the Exponential Family for Image Segmentation

147

References 1. Kass, M., Witkin, A., Terzopoulos, D.: Snakes: Active contour models. International Journal of Computer Vision 1, 321–332 (1988) 2. Caselles, V., Kimmel, R., Sapiro, G.: Geodesic active contours. International Journal of Computer Vision 22(1), 61–79 (1997) 3. Aujol, J.F., Aubert, G., Blanc-Féraud, L.: Wavelet-based level set evolution for classification of textured images. IEEE Transactions on Image Processing 12(12), 1634–1641 (2003) 4. Martin, P., Réfrégier, P., Goudail, F., Guérault, F.: Influence of the noise model on level set active contour segmentation. IEEE PAMI 26, 799–803 (2004) 5. Cremers, D., Rousson, M., Deriche, R.: A review of statistical approaches to level set segmentation: integrating color, texture, motion and shape. International Journal of Computer Vision 72(2), 195–215 (2007) 6. Aubert, G., Barlaud, M., Faugeras, O., Jehan-Besson, S.: Image segmentation using active contours: Calculus of variations or shape gradients? SIAM Applied Mathematics 63(6), 2128–2154 (2003) 7. Zhu, S., Yuille, A.: Region competition: unifying snakes, region growing, and bayes/MDL for multiband image segmentation. IEEE PAMI 18, 884–900 (1996) 8. Paragios, N., Deriche, R.: Geodesic active regions: A new paradigm to deal with frame partition problems in computer vision. JVCIR 13, 249–268 (2002) 9. Rougon, N., Discher, A., Prêteux, F.: Region-based statistical segmentation using informational active contours. In: SPIE Conf. on Mathematics of Data/Image Pattern Recognition, San Diego, CA (August 2006) 10. Michailovich, O., Rathi, Y., Tannenbaum, A.: Image segmentation using active contours driven by the bhattacharyya gradient flow. IEEE Transactions on Image Processing 16, 2787– 2801 (2007) 11. Wang, Z., Vemuri, B.: DTI segmentation using an information theoretic tensor dissimilarity measure. IEEE Transactions on Medical Imaging 24(10), 1267–1277 (2005) 12. Jehan-Besson, S., Barlaud, M., Aubert, G.: DREAM2 S: Deformable regions driven by an eulerian accurate minimization method for image and video segmentation. International Journal of Computer Vision (53) , 45–70 (2003) 13. Kullback, S.: Information Theory and Statistics. Wiley, New York (1959) 14. Chan, T.F., Vese, L.A.: Active contour without edges. IEEE Transactions on Image Processing 10, 266–277 (2001) 15. Delfour, M., Zolésio, J.: Shape and geometries. Advances in Design and Control. SIAM, Philadelphia (2001) 16. Bickel, P., Docksum, K.: Mathematical statistics: basic ideas and selected topics, 2nd edn., vol. I. Prentice-Hall, London (2001) 17. Koopman, P.: On distributions admitting a sufficient statistic. Trans. Am. Math. Soc. 39, 399–409 (1936) 18. Osher, S., Sethian, J.: Fronts propagating with curvature-dependent speed: Algorithms based on hamilton-jacobi formulation. Journal of Computational Physics 79, 12–49 (1988) 19. Cover, T.M., Thomas, J.A.: Elements of Information Theory. Wiley, New York (1991) 20. Goodman, J.: Some fundamental properties of speckle. J. of Optical Society of America 66, 1145–1150 (1976) 21. Lecellier, F., Jehan-Besson, S., Fadili, J., Aubert, G., Revenu, M.: Statistical region-based active contours with exponential family observations. In: ICASSP, vol. 2, pp. 113–116 (2006)

148

F. Lecellier et al.

A Appendix A.1 Shape Derivation Tools Let us remind this useful theorem [15] that will be used in the following proofs.  Theorem 4. The Gâteaux derivative of the functional J(Ω) = f (x, Ω) dx in the Ω   direction of V is: < J  (Ω), V >= fs (x, Ω, V)dx− f (x, Ω)(V·N )da(x) where Ω

∂Ω

N is the unit inward normal to ∂Ω, da its area element and fs the shape derivative of f [15]. A.2 Proof of Theorem 3 To compute < D (Ω), V >, we must first get the derivative of q(y(x), η) with respect to the domain, and apply the chain rule to Ψ (q(y(x), η), p(y)). To simplify the notation we write the Eulerian derivative of η as < η  (Ω), V >= ∇V η = [< η1 (Ω), V >, .., < ηκ (Ω), V >]T . Using the definition of q(y, η) given in (6) and the chain rule applied to A (η(Ω)), we obtain: = h(y) (∇V η, T(y)−∇V η, ∇A(η)) eη(Ω),T(y)−A(η(Ω)) (14) = q(y, η)∇V η, T(y) − ∇A(η) . By the chain rule applied to Ψ (q(y(x), η), p(y)), we get < Ψ  (q(y, η), p(y)), V >=< q  (y, η), V > ∂1 Ψ (q, p),  which gives < D (Ω), V >= χ q(y, η)∂1 Ψ (q, p)∇V η, T(y) − ∇A(η)dy. We introduce  C= q(y, η)∂1 Ψ (q, p) (T(y) − ∇A(η)) dy = E[∂1 Ψ (q, p) (T(Y ) − E[T(Y )])] χ

which completes the proof. A.3 Proof of Lemma1 When using the MLE, the term E[T(Y )] can be empirically estimated with T(Y ) and so derived easily with respect to the domain Ω. We propose to directly derive the expression ∇A(η) = T(Y ) which gives: κ  j=1

< ηj , V >

 ∂2A (η) =< Ti (Y ) , V > ∂ηi ∂ηj

∀i ∈ [1, κ] ,

(15)

¨ which can be written in the compact form ∇V (T) = A(η)∇ V η. ¨ Restricting our study to the full rank exponential family, where A(η) is a symmetric positive-definite, hence invertible, matrix (Theorem 2), the domain derivative of the pa¨ −1 ∇V (T) = ∇V η where ∇V (T) is given rameters η is uniquely determined by A(η)    1 by: ∇V (T) = |Ω| ∂Ω T(y) − T(y(a)) (V · N)da(x) (taking benefit of theorem 4) and the lemma follows.

Optimization of Divergences within the Exponential Family for Image Segmentation

149

A.4 Proof of Lemma 2 Since p and q belongs to the same parametric law, they share the same value for h(y), T(y) and A(η) and then log(q) − log(p) = η − η 1 , T(y) − A(η) + A(η 1 ). The value of C is then C = s1 − s2 , with: s1 = E[(η − η 1 , T(y) − A(η) + A(η 1 ) + 1)(Ti (Y ) − E[Ti (Y )]] p s2 = E[ (Ti (Y ) − E[Ti (Y )]]Ep [(Ti (Y ) − E[Ti (Y )]] q Developing the expression of the expectation of the second term,we find s2 = Ep [(Ti (Y ) − E[Ti (Y )]] = ∇A(η 1 ) − ∇A(η). Using the linearity of the expectation and the fact that E[Tj (Y )(Ti (Y )] − E[Ti (Y )]E[Tj (Y )] designates the co¨ ij = variance matrix of the sufficient statistics T and can then be replaced by A(η) κ ¨ ¨ ¨ Cov[T(Y )]ij = A(η)ji , we find: s1 = j=1 (ηj − η1j )A(η)ij , and then C = A(η) (η − η1 ) + ∇A(η) − ∇A(η 1 ).

Convex Multi-class Image Labeling by Simplex-Constrained Total Variation Jan Lellmann, Jörg Kappes, Jing Yuan, Florian Becker, and Christoph Schnörr Image and Pattern Analysis Group (IPA) HCI, Dept. of Mathematics and Computer Science, University of Heidelberg {lellmann,kappes,yuanjing,becker,schnoerr}@math.uni-heidelberg.de

Abstract. Multi-class labeling is one of the core problems in image analysis. We show how this combinatorial problem can be approximately solved using tools from convex optimization. We suggest a novel functional based on a multidimensional total variation formulation, allowing for a broad range of data terms. Optimization is carried out in the operator splitting framework using Douglas-Rachford Splitting. In this connection, we compare two methods to solve the Rudin-Osher-Fatemi type subproblems and demonstrate the performance of our approach on single- and multichannel images.

1

Introduction

In this paper, we study the variational approach  inf f (u) , f (u) = − u(x), s(x)dx + λ TV(u) , u∈C

λ>0,

(1)

Ω

for determining a labeling u : Ω → RL , that is a contextual classification of each pixel x ∈ Ω into one out of L classes, based on an arbitrary vector-valued similarity function s(x) ∈ RL as input data that has been computed from image data beforehand. The objective function (1) comprises the common form of a data term plus a regularization term. The data term is given by the L2 inner product of the assignment variables u and the similarity function s, and the regularizer is a total variation (TV) formulation for vector-valued data,   TV(u) = ∇u1 2 + · · · + ∇uL 2 dx . (2) Ω

Furthermore, the constraint u ∈ C restricts the vector field u(x) at each location x ∈ Ω to lie in the standard probability simplex, that is u(x) ∈ RL + and  L  i=1 u(x) i = 1 for all x ∈ Ω. Our work is motivated by the following observation. Suppose that at each pixel x ∈ Ω, there is an unambiguous assignment (labeling) of the data s(x) to some class l ∈ {1, . . . , L} represented by the corresponding l-th unit vector, X.-C. Tai et al. (Eds.): SSVM 2009, LNCS 5567, pp. 150–162, 2009. c Springer-Verlag Berlin Heidelberg 2009 

Convex Multi-class Image Labeling by Simplex-Constrained Total Variation

151

Fig. 1. Left: Noisy input image. Right: The labeled image based on the non-binary assignment u as global minimizer of the convex approach (1). The discrete problem is accurately solved by a continuous approach.

u(x) = el . Then, an interface with√area A between two image regions labeled with l and l , respectively, adds A 2 to the regularization term iff l = l , as all but two gradients under the square root vanish. As a result, under these √ assumptions and up to the immaterial constant 2, the TV term corresponds to the well-known Potts model that assigns constant penalties to local changes of the labeling. A significant difference between the Potts model and our approach (1), however, is that the former amounts to solve a discrete combinatorial problem, whereas the latter is a continuous convex optimization problem. Experiments show that our approach (1) approximates discrete decisions fairly well (Fig. 1 and 2) by computing a global optimum to a single convex optimization problem. By contrast, the state-of-the-art discrete approach [1] approximates the combinatorial solution by solving a non-uniquely defined sequence of binary problems via graph cuts. This fact, along with the potential of continuous convex optimization for parallel implementations and their more robust dependency on (hyper-) parameters, motivated to investigate the approach (1) as a promising model for a general “labeling submodule” within computer vision systems. To this end, – We have a closer look at the data and regularization terms (section 2). – We apply an operator splitting approach to (1) in order to decompose the computation of a globally optimal labeling into two independent computational steps: TV denoising for vector-valued data, and projection of the labeling vectors u(x) on the canonical simplex (section 3). – We evaluate two different algorithms for the TV denoising subroutine (section 4) and compare the performance of our convex method to a range of established graph cut-based approaches (section 5). Related work. In contrast to the binary case with anisotropic discretization [2], multi-class energies are generally not submodular and thus cannot be optimized globally using graph cuts [3]. Some extensions exist, which find a local minimum by solving a sequence of binary graph cuts [1]. The continuous formulation – optimization on the set of characteristic functions – is known as continuous cut [5]. Chan et al. [6] showed that this problem can be relaxed and solved on

152

J. Lellmann et al.

Fig. 2. Output of the standard TV approach [4] for scalar-valued images applied to the noisy input image depicted in Fig. 1, for different values of the regularization parameter λ. Irrespective of this value, the performance is worse than with the approach (1) (cf. Fig. 1, right), because the latter approximates the Potts model that does not depend on the size (contrast) of discontinuities. Consequently, the former approach cannot remove noise without degrading weak discontinuities, as is apparent above for the horizontal discontinuities.

a convex set, without losing global optimality. In contrast, our work is aimed at the multi-class case. In [7], a comparable approach based on [8] was presented, which relies on a natural ordering of the labels, as given in e.g. stereo reconstruction. An approach very similar to ours was recently presented in [9], where the authors use a different formulation of the total variation on vector fields, and an alternating optimization method. The (discrete) Potts model was studied in [10], where approximate solutions were computed by an LP relaxation with explicit constraints. In contrast, our approach considers the general TV term and a problem decomposition into efficiently solvable subproblems, without the need to introduce additional variables. Notation. We consider the discretized version of our approach (1). Let Ω = {1, . . . , n1 } × · · · × {1, . . . , nd } ⊆ Rd , d ∈ N, denote a regular image grid of n := |Ω| pixels. The (multidimensional) image space X := Rn×L is equipped with the Euclidean inner product ·, ·Ω over the vectorized elements. We naturally identify v = (v 1 , . . . , v L ) ∈ Rn×L with ((v 1 ) · · · (v L ) ) ∈ RnL . Superscripts v i denote a collection of vectors, while subscripts vk denote vector components. Using the notation e = (1, 1, . . . , 1) , the standard simplex on RL  n×L L and its extension C on R are given by ΔL := v ∈ R v ≥ 0 , e, v = 1

and C := x∈Ω ΔL . Define δC (x) to be 0 iff x ∈ C, and +∞ otherwise.   Let grad := (grad 1 , . . . , gradd ) be the d-dimensional forward difference gradient operator for Neumann boundary conditions. Accordingly, div := −grad is the backward difference divergence operator for Dirichlet boundary conditions. These operators extend to Rn×L via Grad := (IL ⊗ grad), Div := (IL ⊗ div), where IL is the L × L identity matrix. We will also need the convex sets L   12    Bλ := (p1 , . . . , pL ) ∈ Rd×L  pi 22 λ , i=1

(3)

Convex Multi-class Image Labeling by Simplex-Constrained Total Variation



Dλ :=

Bλ ⊆ Rn×d×L , Eλ := {u ∈ Rn×L |u = Div p , p ∈ Dλ } .

153

(4)

x∈Ω

The discrete total variation on vector-valued data is then defined as

TV(u) := σE1 (u) = Gx u2 ,

(5)

x∈Ω

where σM (u) := supp∈M u, p is the support function from convex analysis, and Gx is an (Ld)×n matrix composed of rows of (Grad) s.t. Gx u gives the gradients of all ui in x stacked one above the other.

2

Variational Approach

Based on the introduced notation, our novel approach (1) reads inf f (u) ,

u∈C

λ TV(u) , f (u) = −u, sΩ +       data term regularization term

λ>0,

(6)

As the objective function f and the constraint set C are convex, the overall problem is convex as well. We will now define and motivate each term. Data Term. The data term in (6) is fairly general.   Any vector-valued similarity function s can be used, whose components s(x) i indicate the affinity of some data point at x with class i. As an example, suppose we have image features g(x), x ∈ Ω, prototypical feature vectors G = (G1 , . . . , GL ) as well as a distance measure d on the features. We might think of g as a grayscale image, of G as some prototypical gray values, and of d as a quadratic distance measure, possibly derived from a statistical noise model. The hard assignment of the pixel x ∈ Ω to a label (or class) l(x) ∈ {1, . . . , L} should then be penalized by the distance d(g(x), Gl(x) ) of the corresponding feature to the prototype of the assigned class. Denoting the negative distance by s, and summing up over the image domain, we see that 



 s(x), u(x) for u(x) = el(x) . d g(x), Gl(x) = − (7) x∈Ω

x∈Ω

Thus, instead of looking for l ∈ {1, . . . , L}n , we may equivalently look for u ∈ {e1 , . . . , eL }n . However, the right hand side formulation has the advantage that it extends naturally to the soft assignment u ∈ C: We may now solve the easier problem of optimizing for u on the convex set C. In our experiments, we chose d(x, y) = ||x − y||1 , as the 1 -norm is still convex but known to be more robust against noise and outliers. However, s is not restricted to representing distances. In fact, it may be arbitrarily nonlinear and nonconvex in x and g, and involve nonlocal operations on g. The complexity is completely hidden within the precomputed vector s. Regularization Term. Recall that the regularizer of (6) is defined (5) as

TV(u) = sup u, Div p = Gx u2 . (8) p∈D1

x∈Ω

154

J. Lellmann et al.

This definition for vector-valued u parallels the definition of the “isotropic” total variation measure in the scalar-valued case [11, 4, 12]. It is also known as MTV [13, 14, 15], and was recently studied in [16] in its continuous formulation. Contrary to the anisotropic discretization, where one would substitute the sum of 1-norms in (3), it is less biased towards edges parallel to the axes. See also [17] for an overview of TV-based research and applications. Optimality. After solving the relaxed problem, it remains to show that a binary solution can be recovered. For the continuous, binary case, Chan et al. [6] showed that an exact solution can be obtained by thresholding at almost any threshold. However, their results do not immediately transfer to the discrete multi-class case. In particular, the crucial “layer cake” formula holds for 1 -, but not 2 discretizations of the TV. Contrary to the binary case, it is not clear which rounding scheme to use for vector-valued u. For our experiments, we chose the final class label for each pixel x as the index l of the maximal u∗l (x) of the global optimum u∗ of (6). This defines a suboptimal discrete solution u∗t . Bounding the error f (u∗t ) − f (u∗d) with respect to the unknown discrete optimum u∗d will be subject of our future work.

3

Optimization

Two basic problems arise concerning the optimization of (6): Nondifferentiability of the objective function due to the TV term, and handling of the simplex constraint u ∈ C. We cope with the latter using the tight Douglas-Rachford splitting method as presented in the following section. We refer to [18] for the full derivations. Douglas-Rachford Splitting. Minimization of a proper, convex, lower-semicontinuous (lsc) function f : X → R can be regarded as finding a zero of its (necessarily maximal monotone [19, Chap. 12]) subgradient operator T := ∂f : X ⇒ X. In the operator splitting framework, ∂f is assumed to be decomposable into the sum of two “simple” operators, T = A + B, of which forward and backward steps can practically be computed. Here, we consider the (tight) Douglas-Rachford-Splitting iteration [20, 21], z k+1 ∈ (Jτ A (2Jτ B − I) + (I − Jτ B ))(z k ) ,

(9)

where Jτ T := (I + τ T )−1 is the resolvent of T . Under the very general constraint that A and B are maximal monotone and A + B has at least one zero, the sequence (z k ) will converge to a point z, with the additional property that x := Jτ B (z) is a zero of T ( [22, Thm. 3.15], [22, Prop. 3.20], [22, Prop. 3.19], [23]). In particular, for f = f1 +f2 , fi proper, convex, lsc with ri(dom f1 )∩ri(dom f2 ) = ∅ (ri(S) denoting the relative interior of a set S), it can be shown [19, Cor. 10.9] that ∂f = ∂f1 + ∂f2 , and the ∂fi are maximal monotone. As x ∈ Jτ ∂fi (y) ⇔ x = argmin(2τ )−1 x − y22 + fi (x), the computation of the resolvents reduces to proximal point optimization problems involving only the fi .

Convex Mult