Scale Space and Variational Methods in Computer Vision: Second International Conference, SSVM 2009, Voss, Norway, June 1-5, 2009. Proceedings (Lecture Notes in Computer Science)

Lecture Notes in Computer Science Commenced Publication in 1973 Founding and Former Series Editors: Gerhard Goos, Juris Hartmanis, and Jan van Leeuwen

Editorial Board David Hutchison Lancaster University, UK Takeo Kanade Carnegie Mellon University, Pittsburgh, PA, USA Josef Kittler University of Surrey, Guildford, UK Jon M. Kleinberg Cornell University, Ithaca, NY, USA Alfred Kobsa University of California, Irvine, CA, USA Friedemann Mattern ETH Zurich, Switzerland John C. Mitchell Stanford University, CA, USA Moni Naor Weizmann Institute of Science, Rehovot, Israel Oscar Nierstrasz University of Bern, Switzerland C. Pandu Rangan Indian Institute of Technology, Madras, India Bernhard Steffen University of Dortmund, Germany Madhu Sudan Massachusetts Institute of Technology, MA, USA Demetri Terzopoulos University of California, Los Angeles, CA, USA Doug Tygar University of California, Berkeley, CA, USA Gerhard Weikum Max-Planck Institute of Computer Science, Saarbruecken, Germany

5567

Xue-Cheng Tai Knut Mørken Marius Lysaker Knut-Andreas Lie (Eds.)

Scale Space and Variational Methods in Computer Vision Second International Conference, SSVM 2009 Voss, Norway, June 1-5, 2009 Proceedings

13

Volume Editors Xue-Cheng Tai Department of Mathematics University of Bergen, Norway and Division of Mathematical Science Nanyang Technological University, Singapore E-mail: [email protected] Knut Mørken Department of Informatics and Centre of Mathematics for Applications University of Oslo, Norway E-mail: [email protected] Marius Lysaker Simula Research Laboratory Lysaker, Norway E-mail: [email protected] Knut-Andreas Lie Centre of Mathematics for Applications University of Oslo, Norway and SINTEF ICT, Oslo, Norway E-mail: [email protected]

Library of Congress Control Number: Applied for CR Subject Classification (1998): I.4, I.5, I.3.5, I.2.10, G.1.2, F.2.2 LNCS Sublibrary: SL 6 – Image Processing, Computer Vision, Pattern Recognition, and Graphics ISSN ISBN-10 ISBN-13

0302-9743 3-642-02255-3 Springer Berlin Heidelberg New York 978-3-642-02255-5 Springer Berlin Heidelberg New York

This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. springer.com © Springer-Verlag Berlin Heidelberg 2009 Printed in Germany Typesetting: Camera-ready by author, data conversion by Scientific Publishing Services, Chennai, India Printed on acid-free paper SPIN: 12689675 06/3180 543210

Preface

This book contains 71 original, scientific articles that address state-of-the-art research related to scale space and variational methods for image processing and computer vision. Topics covered in the book range from mathematical analysis of both established and new models, fast numerical methods, image analysis, segmentation, registration, surface and shape construction and processing, to real applications in medical imaging and computer vision. The ideas of scale space and variational methods related to partial differential equations are central concepts. The papers reflect the newest developments in these fields and also point to the latest literature. All the papers were submitted to the Second International Conference on Scale Space and Variational Methods in Computer Vision, which took place in Voss, Norway, during June 1–5, 2009. The papers underwent a peer review process similar to that of high-level journals in the field. We thank the authors, the Scientific Committee, the Program Committee and the reviewers for their hard work and helpful collaboration. Their contribution has been crucial for the efficient processing of this book, and for the success of the conference. Finally, we wish to thank those who have supported and helped to organize the conference. First and foremost it is a pleasure to acknowledge the generous financial support from the Centre of Mathematics for Applications (CMA) at the University of Oslo and the Research Council of Norway. In addition, partial support was given by the Centre of Integrated Petroleum Research (CIPR) at the University of Bergen and the Simula Research Laboratory (SRL). Moreover, we would like to thank Tiril P. Gurholt and Andrew McMurry for their support, both with technical and administrative matters. Members and students from the Mathematical Imaging and Vision Group at the Nanyang Technological University of Singapore and the University of Bergen, Norway deserve special thanks for their kind help. March 2009

Xue-Cheng Tai Knut Mørken Marius Lysaker Knut-Andreas Lie

Organization

Organizing Commitee and Editors Xue-Cheng Tai

University of Bergen, Norway, and Nanyang Technology University Singapore, Conference Chair Sintef, Norway Simula Research Laboratory, Norway University of Oslo, Norway

Knut-Andreas Lie Marius Lysaker Knut Mørken

Scientific Committee Alfred M.Bruckstein Tony F. Chan Mads Nielsen Stanley Osher Nikos Paragios Bart M. ter Haar Romeny Christoph Schnoerr Fiorella Sgallari Joachim Weickert

Technion IIT, Israel University of California at LA, USA University of Copenhagen, Denmark University of California at LA, USA Ecole Centrale de Paris, France Eindhoven University of Technology, The Netherlands University of Heidelberg, Germany University of Bologna, Italy Saarland University, Germany

Program Commitee Luis Alvarez Noura Azzabou Thomas Brox Bernhard Burgeth Vicent Caselles Raymond Chan Yeowmeng Chee Yunmei Chen Daniel Cremers Francoise Dibos Michael Felsberg Luc Florack

Lewis Griffin Anders Heyden Charles Kervrann Ron Kimmel Arjan Kuijper Georg Langs Antonio Leitao Riccardo March Antonio Marquina Etienne Memin Karol Mikula Jan Modersitzki

Michael Ng Mila Nikolova Martin Rumpf Otmar Scherzer Nir Sochen Gabriele Steidl Demetri Terzopoulos David Tschumperle Baba C. Vemuri Hongkai Zhao Haomin Zhou

VIII

Organization

Invited Speakers Antonin Chambolle, CMAP - Ecole Polytechnique, France Raymond Chan, The Chinese University of Hong Kong, China Amiram Grinvald, Weizmann Institute of Science, Israel

Sponsoring Institutions Centre of Mathematics for Applications, University of Oslo Research Council of Norway Centre of Integrated Petroleum Research, University of Bergen Simula Research Laboratory

Table of Contents

Segmentation and Detection Graph Cut Optimization for the Piecewise Constant Level Set Method Applied to Multiphase Image Segmentation . . . . . . . . . . . . . . . . . . . . . . . . . Egil Bae and Xue-Cheng Tai Tubular Anisotropy Segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fethallah Benmansour and Laurent D. Cohen

1 14

An Unconstrained Multiphase Thresholding Approach for Image Segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Benjamin Berkels

26

Extraction of the Intercellular Skeleton from 2D Images of Embryogenesis Using Eikonal Equation and Advective Subjective Surface Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Paul Bourgine, Peter Frolkoviˇc, Karol Mikula, Nadine Peyri´eras, and Mariana Remeˇs´ıkov´ a

38

On Level-Set Type Methods for Recovering Piecewise Constant Solutions of Ill-Posed Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Adriano DeCezaro, Antonio Leit˜ ao, and Xue-Cheng Tai

50

The Nonlinear Tensor Diffusion in Segmentation of Meaningful Biological Structures from Image Sequences of Zebrafish Embryogenesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Olga Drbl´ıkov´ a, Karol Mikula, and Nadine Peyri´eras

63

Composed Segmentation of Tubular Structures by an Anisotropic PDE Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Elena Franchini, Serena Morigi, and Fiorella Sgallari

75

Extrapolation of Vector Fields Using the Infinity Laplacian and with Applications to Image Segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Laurence Guillot and Carole Le Guyader

87

A Schr¨ odinger Equation for the Fast Computation of Approximate Euclidean Distance Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Karthik S. Gurumoorthy and Anand Rangarajan

100

Semi-supervised Segmentation Based on Non-local Continuous Min-Cut . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Nawal Houhou, Xavier Bresson, Arthur Szlam, Tony F. Chan, and Jean-Philippe Thiran

112

X

Table of Contents

Momentum Based Optimization Methods for Level Set Segmentation . . . Gunnar L¨ ath´en, Thord Andersson, Reiner Lenz, and Magnus Borga Optimization of Divergences within the Exponential Family for Image Segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Francois Lecellier, Stephanie Jehan-Besson, Jalal Fadili, Gilles Aubert, and Marinette Revenu Convex Multi-class Image Labeling by Simplex-Constrained Total Variation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jan Lellmann, J¨ org Kappes, Jing Yuan, Florian Becker, and Christoph Schn¨ orr Geodesically Linked Active Contours: Evolution Strategy Based on Minimal Paths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Julien Mille and Laurent D. Cohen

124

137

150

163

Validation of Watershed Regions by Scale-Space Statistics . . . . . . . . . . . . . Tomoya Sakai and Atsushi Imiya

175

Adaptation of Eikonal Equation over Weighted Graph . . . . . . . . . . . . . . . . Vinh-Thong Ta, Abderrahim Elmoataz, and Olivier L´ezoray

187

A Variational Model for Interactive Shape Prior Segmentation and Real-Time Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Manuel Werlberger, Thomas Pock, Markus Unger, and Horst Bischof

200

Image Enhancement and Reconstruction A Nonlinear Probabilistic Curvature Motion Filter for Positron Emission Tomography Images . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Musa Alrefaya, Hichem Sahli, Iris Vanhamel, and Dinh Nho Hao

212

Finsler Geometry on Higher Order Tensor Fields and Applications to High Angular Resolution Diffusion Imaging . . . . . . . . . . . . . . . . . . . . . . . . . Laura Astola and Luc Florack

224

Bregman-EM-TV Methods with Application to Optical Nanoscopy . . . . . Christoph Brune, Alex Sawatzky, and Martin Burger

235

PDE-Driven Adaptive Morphology for Matrix Fields . . . . . . . . . . . . . . . . . Bernhard Burgeth, Michael Breuß, Luis Pizarro, and Joachim Weickert

247

On Semi-implicit Splitting Schemes for the Beltrami Color Flow . . . . . . . Lorina Dascal, Guy Rosman, Xue-Cheng Tai, and Ron Kimmel

259

Multi-scale Total Variation with Automated Regularization Parameter Selection for Color Image Restoration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yiqiu Dong and Michael Hinterm¨ uller

271

Table of Contents

Multiplicative Noise Cleaning via a Variational Method Involving Curvelet Coefficients . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Sylvain Durand, Jalal Fadili, and Mila Nikolova Projected Gradient Based Color Image Decomposition . . . . . . . . . . . . . . . . Vincent Duval, Jean-Fran¸cois Aujol, and Luminita Vese

XI

282 295

A Dual Formulation of the TV-Stokes Algorithm for Image Denoising . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Christoffer A. Elo, Alexander Malyshev, and Talal Rahman

307

Anisotropic Regularization for Inverse Problems with Application to the Wiener Filter with Gaussian and Impulse Noise . . . . . . . . . . . . . . . . . . Micha Feigin and Nir Sochen

319

Locally Adaptive Total Variation Regularization . . . . . . . . . . . . . . . . . . . . . Markus Grasmair Basic Image Features (BIFs) Arising from Approximate Symmetry Type . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Lewis D. Griffin, Martin Lillholm, Mike Crosier, and Justus van Sande

331

343

An Anisotropic Fourth-Order Partial Differential Equation for Noise Removal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mohammad Reza Hajiaboli

356

Enhancement of Blurred and Noisy Images Based on an Original Variant of the Total Variation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Khalid Jalalzai and Antonin Chambolle

368

Coarse-to-Fine Image Reconstruction Based on Weighted Differential Features and Background Gauge Fields . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Bart Janssen, Remco Duits, and Luc Florack

377

Edge-Enhanced Image Reconstruction Using (TV) Total Variation and Bregman Refinement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Shantanu H. Joshi, Antonio Marquina, Stanley J. Osher, Ivo Dinov, John D. Van Horn, and Arthur W. Toga Nonlocal Variational Image Deblurring Models in the Presence of Gaussian or Impulse Noise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Miyoun Jung and Luminita A. Vese A Geometric PDE for Interpolation of M -Channel Data . . . . . . . . . . . . . . Frank Lenzen and Otmar Scherzer An Edge-Preserving Multilevel Method for Deblurring, Denoising, and Segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Serena Morigi, Lothar Reichel, and Fiorella Sgallari

389

401 413

426

XII

Table of Contents

Fast Dejittering for Digital Video Frames . . . . . . . . . . . . . . . . . . . . . . . . . . . Mila Nikolova

439

Sparsity Regularization for Radon Measures . . . . . . . . . . . . . . . . . . . . . . . . . Otmar Scherzer and Birgit Walch

452

Split Bregman Algorithm, Douglas-Rachford Splitting and Frame Shrinkage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Simon Setzer Anisotropic Smoothing Using Double Orientations . . . . . . . . . . . . . . . . . . . Gabriele Steidl and Tanja Teuber

464 477

Image Denoising Using TV-Stokes Equation with an Orientation-Matching Minimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Xue-Cheng Tai, Sofia Borok, and Jooyoung Hahn

490

Augmented Lagrangian Method, Dual Methods and Split Bregman Iteration for ROF Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Xue-Cheng Tai and Chunlin Wu

502

The Convergence of a Central-Difference Discretization of Rudin-Osher-Fatemi Model for Image Denoising . . . . . . . . . . . . . . . . . . . . . Ming-Jun Lai, Bradley Lucier, and Jingyue Wang

514

Theoretical Foundations for Discrete Forward-and-Backward Diffusion Filtering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Martin Welk, Guy Gilboa, and Joachim Weickert

527

L0 -Norm and Total Variation for Wavelet Inpainting . . . . . . . . . . . . . . . . . Andy C. Yau, Xue-Cheng Tai, and Michael K. Ng

539

Total-Variation Based Piecewise Affine Regularization . . . . . . . . . . . . . . . . Jing Yuan, Christoph Schn¨ orr, and Gabriele Steidl

552

Image Denoising by Harmonic Mean Curvature Flow . . . . . . . . . . . . . . . . . Mourad Z´era¨ı

565

Motion Analysis, Optical Flow, Registration and Tracking Tracking Closed Curves with Non-linear Stochastic Filters . . . . . . . . . . . . Christophe Avenel, Etienne M´emin, and Patrick P´erez A Multi-scale Feature Based Optic Flow Method for 3D Cardiac Motion Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Alessandro Becciu, Hans van Assen, Luc Florack, Sebastian Kozerke, Vivian Roode, and Bart M. ter Haar Romeny

576

588

Table of Contents

XIII

A Combined Segmentation and Registration Framework with a Nonlinear Elasticity Smoother . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Carole Le Guyader and Luminita A. Vese

600

A Scale-Space Approach to Landmark Constrained Image Registration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Eldad Haber, Stefan Heldmann, and Jan Modersitzki

612

A Variational Approach for Volume-to-Slice Registration . . . . . . . . . . . . . . Stefan Heldmann and Nils Papenberg Hyperbolic Numerics for Variational Approaches to Correspondence Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Henning Zimmer, Michael Breuß, Joachim Weickert, and Hans-Peter Seidel

624

636

Surfaces and Shapes From a Single Point to a Surface Patch by Growing Minimal Paths . . . . . Fethallah Benmansour and Laurent D. Cohen

648

Optimization of Convex Shapes: An Approach to Crystal Shape Identification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Timo Eirola and Toni Lassila

660

An Implicit Method for Interpolating Two Digital Closed Curves on Parallel Planes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Nikolaos Gabrielides and Laurent Cohen

672

Pose Invariant Shape Prior Segmentation Using Continuous Cuts and Gradient Descent on Lie Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Niels Chr. Overgaard, Ketut Fundana, and Anders Heyden

684

A Non-local Approach to Shape from Ambient Shading . . . . . . . . . . . . . . . Emmanuel Prados, Nitin Jindal, and Stefano Soatto

696

An Elasticity Approach to Principal Modes of Shape Variation . . . . . . . . Martin Rumpf and Benedikt Wirth

709

Pre-image as Karcher Mean Using Diffusion Maps: Application to Shape and Image Denoising . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Nicolas Thorstensen, Florent Segonne, and Renaud Keriven

721

Fast Shape from Shading for Phong-Type Surfaces . . . . . . . . . . . . . . . . . . . Oliver Vogel, Michael Breuß, Thomas Leichtweis, and Joachim Weickert

733

Generic Scene Recovery Using Multiple Images . . . . . . . . . . . . . . . . . . . . . . Kuk-Jin Yoon, Emmanuel Prados, and Peter Sturm

745

XIV

Table of Contents

Scale Space and Feature Extraction Highly Accurate PDE-Based Morphology for General Structuring Elements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Michael Breuß and Joachim Weickert Computational Geometry-Based Scale-Space and Modal Image Decomposition: Application to Light Video-Microscopy Imaging . . . . . . . Anatole Chessel, Bertrand Cinquin, Sabine Bardin, Jean Salamero, and Charles Kervrann

758

770

Highlight on a Feature Extracted at Fine Scales: The Pointwise Lipschitz Regularity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Christophe Damerval and Sylvain Meignen

782

Line Enhancement and Completion via Linear Left Invariant Scale Spaces on SE(2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Remco Duits and Erik Franken

795

Spatio-Featural Scale-Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Michael Felsberg Scale Spaces on the 3D Euclidean Motion Group for Enhancement of HARDI Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Erik Franken and Remco Duits

808

820

On the Rate of Structural Change in Scale Spaces . . . . . . . . . . . . . . . . . . . . David Gustavsson, Kim S. Pedersen, Francois Lauze, and Mads Nielsen

832

Transitions of a Multi-scale Image Hierarchy Tree . . . . . . . . . . . . . . . . . . . . Arjan Kuijper

844

Local Scale Measure for Remote Sensing Images . . . . . . . . . . . . . . . . . . . . . Bin Luo, Jean-Fran¸cois Aujol, and Yann Gousseau

856

Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

869

Graph Cut Optimization for the Piecewise Constant Level Set Method Applied to Multiphase Image Segmentation Egil Bae1 and Xue-Cheng Tai2 1

Department of Mathematics, University of Bergen, Norway [email protected] 2 Department of Mathematics, University of Bergen, Norway and Division of Mathematical Sciences, School of Physical and Mathematical Sciences, Nanyang Technological University, Singapore [email protected]

Abstract. The piecewise constant level set method (PCLSM) has recently emerged as a variant of the level set method for variational interphase problems. Traditionally, the Euler-Lagrange equations are solved by some iterative numerical method for PDEs. Normally the speed is slow. In this work, we focus on the piecewise constant level set method (PCLSM) applied to the multiphase Mumford-Shah model for image segmentation. Instead of solving the Euler-Lagrange equations of the resulting minimization problem, we propose an efficient combinatorial optimization technique, based on graph cuts. Because of a simplification of the length term in the energy induced by the PCLSM, the minimization problem is not NP hard. Numerical experiments on image segmentation demonstrate that the new approach is very superior in terms of efficiency, while maintaining the same quality.

1

Introduction

The level set method [1, 2] is a powerful tool for interphase problems. It has numerous applications in computer vision, fluid dynamics and inverse problems. The interphase is implicitly represented by a higher dimensional level set function. Originally, the signed distance functions were used as level set functions. Later the work of [3, 4, 5] introduced piecewise constant level set functions, representing the interphases by discontinuities. This has certain advantages, such as ability to represent several interphases by one single level set function. This method will be referred to as the piecewise constant level set method (PCLSM) In computer vision, the level set method has been applied with great success to image segmentation. Of particular importance is the Mumford-Shah model [6], 

Support from the Norwegian Research Council (eVita project 166075), National Science Foundation of Singapore (NRF2007IDM-IDM002-010) and Ministry of Education of Singapore (Moe Tier 2 T207B2202) are gratefully acknowledged.

X.-C. Tai et al. (Eds.): SSVM 2009, LNCS 5567, pp. 1–13, 2009. c Springer-Verlag Berlin Heidelberg 2009 

2

E. Bae and X.-C. Tai

which is an established image segmentation model. In [7, 8], Chan and Vese proposed a numerical realization of this model based on traditional level set functions. In [3, 4, 5], piecewise constant level set functions were proposed. Both approaches lead to a system of nonlinear PDEs that needs to be solved numerically. They both have the drawback of expensive computation. This work aims to significantly reduce the computational cost of the piecewise constant level set method for the multiphase Mumford-Shah model. The length term is often simplified in the energy induced when representing this model by piecewise constant level set functions. We will show that this simplification makes it possible to efficiently compute global minimizers via graph cuts, when the mean image intensity value in each phase is known. Graph cuts is a wellknown technique in image analysis and computer vision [9, 10, 11, 12]. Usually, NP-hard multilabeling problems are approached by constructing algorithms for finding approximate suboptimal solutions, such as alpha expansion [12]. We instead do the approximation in the model, and then compute the exact solution of the approximate model. The graph used for optimization is constructed as in [13, 14, 15], except for some small modifications. Finally, for unknown mean intensity values, an iterative algorithm is presented, which we believe will have large practical value because of the strong efficiency. In case of two phases, some work on graph cut optimization for the MumfordShah model has been made in [16, 17]. Also a multiphase approach based on graph cuts has recently been presented in [18]. The process is started by splitting the image into two regions, by solving the two-phase Mumford-Shah model to optimality. In the next step, each new region is splitted in two by solving the twophase Mumford-Shah model within each region. The process is repeated until the intensity variation within each region falls below a predefined threshold. The limitation of this approach is that the possibility of a region to evolve has been ignored. For instance, the optimal interphase for two regions may not be a subset of the optimal interphases for three regions. An experiment in Section 4 will clarify this. The paper is organized as follows: Section 2 gives a brief overview of the piecewise constant level set method and the Mumford-Shah model. Section 3 presents the new integer optimization approach, while numerical experiments are presented in Section 4.

2 2.1

Image Segmentation and the PCLSM The Mumford-Shah Model

The Mumford-Shah model [6] is an established image segmentation model with a wide range of applications. Let u0 be the input image. In the most common variant with closed contours, one seeks a partition {Ωi }ni=1 of the image domain Ω, and an approximation image u which minimizes the functional  E(u, Γi ) = Ω

(u − u0 )2 dx + μ

 Ω\∪i Γi

|∇u|2 dx +

 n  ν i=1

Γi

ds,

(1)

Graph Cut Optimization for the Piecewise Constant Level Set Method

3

where {Γi }ni=1 denotes the interphases between the regions {Ωi }ni=1 . Often u is assumed to be constant within each phase, in which case the second term disappears and one ends up with the simpler version 

(u − u0 )2 dx +

E({ci }, Γi ) = Ω

 n  ν

ds,

(2)

Γi

i=1

n where u = i=1 ci Ψi , and Ψi is the characteristic function of Ωi . As a numerical realization, Chan and Vese [7,8] proposed to represent the above functional with level set functions, and solve the resulting gradient descent equations numerically. In order to represent n phases, log2 (n) level set functions were required. For any n > 2 the length term had to be simplified. 2.2

Piecewise Constant Level Set Functions

In [3, 4, 19], instead the piecewise constant level set method was proposed, and applied to the Mumford-Shah model. This approach has certain benefits, such as the ability to represent any number of phases with one single level set function. Let {Ωi }ni=1 be a partition of the domain Ω into n regions. Any such partition can be described by a piecewise constant level set function φ as follows φ=i

in Ωi

for i = 1, 2, ..., n.

(3)

Note that all interphases are represented by discontinuities in φ. The MumfordShah functional can now be written in terms of φ  E(c, φ) =

(u − u0 )2 dx +

Ω

n  ν |∇ψi |dx. 2 i=1 Ω

(4)

n where u = i=1 ci ψi , and ψi is the characteristic function of Ωi . It can be derived from the level set function by ψi =

1 αi





(φ − j) with αi =

j=1j =i

(i − k).

(5)

k=1k =i

The length term can be approximated by the total variation of the level set function itself, especially when the number of phases is not too large E(c, φ) =

n   i=1

0 2



(u − u ) dx + ν Ω

|∇φ|dx,

(6)

Ω

see for instance [5] for a justification. Most often this approximation is preferred, since it is computationally easier. Such a simplification of the length term has also been made in [20, 21] among others for multiphase image segmentation. In this work we will consider (6).

4

E. Bae and X.-C. Tai

There are some variants of the total variation regularization term. The com monly used version is the isotropic total variation: T V2 (φ) = Ω |∇φ|2 dx =   |φx1 |2 + |φx2 |2 dx. In computation, often a simpler verΩ  order to simplify  sion is used: T V1 (φ) = Ω |∇φ|1 dx = Ω |φx1 | + |φx2 | dx. However, since T V1 is not isotropic, regularization will be stronger in certain directions. A more isotropic version based on the 1-norm can be obtained by splitting T V1 using the original gradient counterclockwise π/4 radians:   operator, and one rotated T V1, π4 (φ) = 12 Ω |∇φ(x)|1 + |R π4 ∇φ(x)|1 dx, where R π4 ∇ is the gradient in the rotated coordinate system. It is also possible to create even more isotropic versions by considering more such rotations. Previous attempts to minimize (6) have been made by continuous optimization. In order to force a solution taking only integer values, the following constraint was imposed n  K(φ) = (φ − i) = 0 (7) i=1

The constrained optimization problem (6) and (7) could be solved by the augmented lagrangian method as in [3, 4, 19]. Some attempts to speed up the computation can be found in [5]. In the next section we propose to solve the minimization problem by graph cuts. We start by discretizing the variational problem (6) on a grid P of mesh size δ = 1. For each p = (i, j) ∈ P, define the neighborhood systems N4 (p) = {(i ± 1, j), (i, j ± 1)}, and N8 (p) = {(i ± 1, j), (i, j ± 1), (i ± 1, j ± 1)}. The modification of the definition for boundary points is clear. The discrete energy function can now be written compactly Ed (c, φ) =

 p∈P

δ 2 (up − u0p )2 + ν





p∈P q∈Nk (p)

1 wpq |φp − φq |, 2

(8)

where k = 4 for T V1 and k = 8 for T V1, π4 . The weights wpq are given by 2

4δ . In case of two phases, similar weights can also be derived by using wpq = k||p−q|| 2 the Cauchy-Crofton formula [22]. Note that each term is being counted twice in the last summation. This is compensated by multiplication by the factor 12 .

3

Integer Optimization for PCLSM and Mumford-Shah

Instead of imposing constraints to force an integer solution by continuous optimization, we instead propose the much more natural approach of using integer optimization to minimize (6). We will show that the discretized functional (8) can be minimized by graph cuts in case the values c are known in Section 3.2. Finally, in Section 3.3 an algorithm is designed to minimize with respect to both c and φ.

Graph Cut Optimization for the Piecewise Constant Level Set Method

3.1

5

Background on Graph Cuts and Terminology

Min-cut is a well known optimization problem. Due to a duality theorem by Ford and Fulkerson [23], there are several fast algorithms for this problem. Graph cuts is a reference to such algorithms, and was introduced as a computer vision tool by Greig et. al. [9] in connection with markov random fields [24]. A graph G = (V, E) is a set of vertices V and a set of directed edges E. We let (a, b) denote the directed edge going from vertex a to vertex b, and let c(a, b) denote the cost (weight) on this edge. In the graph cut scenario there are two distinguished vertices in V, called the source {s} and the sink {t}. A cut on G is a partitioning of the vertices V into two disjoint an connected (through edges) sets (Vs , Vt ) such that s ∈ Vs and t ∈ Vt . For each cut, the set of severed edges C is uniquely defined as C = {(a, b) | a ∈ Vs , b ∈ Vt and (a, b) ∈ E}.

(9)

We say that the cut severs the edge e if e is contained in C. From now on, we refer to the cut as the set of severed edges C. The cost of the cut is defined as  |C| = c(e). (10) e∈C

We are interested in finding the cut of minimum cost on G, from now on called the minimum cut. The duality theorem by Ford and Fulkerson [23] states this is equivalent to finding the maximum flow from {s} to {t}, where the edge weights are bounds on the maximum amount of flow that can be pushed through the edges. Cuts of minimum cost can thus be computed very efficiently by max-flow algorithms such as Ford-Fulkerson [23]. See [10] for a detailed discussion about implementation. 3.2

Graph Cuts for the Multiphase PCLSM

For fixed values c, we will show that the minimizer of (8) can be obtained by finding the minimum cut over an appropriate graph, i.e. we will construct a graph G such that min |C| = min Ed (c, φ) + σ, (11) C cut on G

φ

where σ is a constant that will be specified later. Note that the minimizer φ is not influenced by this constant. Some work on graph cuts for the two phase Mumford-Shah model can be found in [17,16]. Unfortunately, the extension to more than two phases is NP hard [25]. The usual graph cut approach to optimization problems of several labels, is to use some sort of approximation method, such as the alpha expansion [12]. Since we have already made an approximation in the model (6), we will show that graph cuts can be used to find the exact minimum. The idea is to introduce an extra dimension to take care of several phases. We construct the graph in a similar way as Ishikawa [13, 14], except for some small technical differences:

6

E. Bae and X.-C. Tai

(a)

(b)

Fig. 1. (a) The graph corresponding to a 1D signal of 6 grid points used for 4 phase segmentation. Edges in ED are depicted as vertical arrows and edges in ER are depicted as horizontal arrows. The gray curve is used to visualize the cut, vertices in the interior to the curve belongs to Vs , vertices in the exterior to the curve belongs to Vt . Edges in C are depicted as dotted arrows. Figure (b) shows the values of φ at each grid point corresponding to the cut in (a), they are determined from definition 1.

our graph consists of one less layer of vertices and edges, and is a generalization from the binary construction of Greig et. al. [9]. We also avoid edges of infinite capacity. When the number of phases is small, this will have a little effect on the efficiency. For each grid point p ∈ P, we associate (n−1) vertices in the graph G, denoted vp, , = 1, ..., n − 1. The set of vertices V is formally defined V = {vp, | p ∈ P, ∈ {1, ..., n − 1}} ∪ {s} ∪ {t}.

(12)

An illustration in case of a 1D image where P = {1, 2, ..., 6}, is shown in Figure 1. For ease of visualization, no 2D cases are shown. The edges are arranged in two groups, ED and ER . The first group ED corresponds to the data term in (8). It is defined as ED = ∪p∈P Ep ,

(13)

where for each p ∈ P the edge set Ep is defined as Ep = (s, vp,1 ) ∪n−2 =1 (vp, , vp,+1 ) ∪ (vp,n−1 , t).

(14)

The edges in ED are illustrated as the vertical arrows in Figure 1. The second group of edges ER corresponds to the regularization term in (8). These are illustrated as the horizontal arrows in Figure 1, i.e. ER = {(vp, , vq, ) | p ∈ P, q ∈ Nk (p), ∈ {1, ..., n − 1}}.

(15)

We say that a cut is admissible if it severs exactly one edge in Ep for each p ∈ P. We can now establish the relationship between a cut on G and a level set function φ.

Graph Cut Optimization for the Piecewise Constant Level Set Method

7

Definition 1. Let C ⊂ E be an admissible cut on G. For any grid point p ∈ P, the corresponding level set function φ is defined as ⎧ if (s, vp,1 ) ∈ C, ⎨1 φp = + 1 if (vp, , vp,+1 ) ∈ C, (16) ⎩ n if (vp,n−1 , t) ∈ C. Note that φ is single valued by the admissible cut requirement. We can now define the edge costs (weights) such that the relationship (11) is satisfied. We start by edges in ED , i.e. the data edges σ c (s, vp,1 ) = δ 2 |u0p − c1 |2 + |P| ∀p ∈ P, σ 2 0 2 c (vp, , vp,+1 ) = δ |up − c+1 | + |P| ∀p ∈ P, σ c (vp,n , t) = δ 2 |u0p − cn |2 + |P| ∀p ∈ P.

∀ ∈ {1, ..., n − 2},

(17)

The costs (weights) for the regularization edges ER are defined by c (vp, , vq, ) = νwpq , ∀p ∈ P, ∀q ∈ Nk (p), ∀ ∈ {1, ..., n − 1}.

(18)

By choosing σ as any positive value, the cut of minimum cost will be admissible, which implies that its corresponding level set function is single valued. Theorem 1. Let C be a minimum cut on G, then C is admissible if σ > 0 Proof. Suppose C is a minimum cut on G and for some p ∈ P several edges in Ep belongs to C. That is, there exists a set of indices Lp such that (vp, , vp,+1 ) ∈ C ∀ ∈ Lp . Define the cut C ∗ s.t. for each p ∈ P, vp, ∈ Vs∗ if ≤ max Lp , else vp, ∈ Vt∗ . Then C ∗ ∩ ED ⊂ C ∩ ED . Since σ > 0, no edges have zero weight, therefore |C ∗ ∩ ED | < |C ∩ ED |. Furthermore, |C ∗ ∩ ER |cardinality ≤ |C ∩ ER |cardinality . For T V1 , the weights on all edges in ER are equal. Therefore |C ∗ | < |C|, which is a contradiction. The same contradiction can also be derived for T V1, π4 . To summarize, for any piecewise constant level set function φ taking values in {1, 2, · · · n}, there exists a unique admissible cut on G. Moreover, the function φ and its corresponding cut satisfies |C| = Ed (c, φ).

(19)

Thus, a function φ corresponding to a minimum cut, is a minimizer of the functional (8), i.e. it solves the segmentation problem. Note that in case n = 2, the extra dimension breaks down, and the graph becomes identical to that of Greig et. al. [9] for binary problems. It is also possible to exactly minimize (8) as in [26], by solving a sequence of binary optimization problems via graph cuts. This approach is likely to be faster when n is very large, and is a power of 2. However, for image segmentation n is relatively small, and we expect the presented approach to be faster.

8

3.3

E. Bae and X.-C. Tai

Algorithm for Minimizing the Mumford-Shah Functional

The algorithm presented in the last section minimizes Ed (c, φ) with respect to φ for a fixed c. Vice versa, for a fixed φ the values c minimizing Ed (c, φ) are given by the average intensity in each region  0 u (x)ψi (x) dx i = 1, 2, ..., n, (20) ci = Ω  ψ (x) dx Ω i or in discrete form

 p∈P ci = 

u0p ψi,p

p∈P

ψi,p

i = 1, 2, ..., n.

(21)

We want an algorithm to minimize both with respect to φ and c. This is achieved by combining the two above results in the following iterative descent algorithm Algorithm 1. Estimate initial values c0 , set l = 0 while( ||cl − cl−1 || > tol ) 1. Use graph cuts to estimate φ from ˜ φ = arg minφ˜ Ed (cl , φ).

(22)

2. Update cl+1 according to equation (21). 3. Update l ← l + 1.

Note that no initialization of the level set function is required. Only the values c0 need to be initialized, which can be achieved very efficiently by the isodata algorithm [27]. Note that algorithm 1 has an exact termination criterion, as tol can be set to zero. In all our experiments, convergence was reached in 4-12 iterations. It must be noted that this algorithm is no longer guaranteed to find the global minima. Theoretically it may get trapped in a local minima close to the initial values c0 . However, in practice it is usually rather insensitive to initialization.

4

Numerical Experiments

In this section we validate our new optimization method by numerical experiments. The results are compared with the original gradient descent approach [3] for minimizing (4) (note: not the variant with simplified length term). The implementation of both these methods is made in C++. Comparisons are made both with respect to quality and computation time on an intel 2.19 GHz laptop. The list of computation times is shown in Table 1. The test images are shown in Figure 2. In all results, the estimated phases are depicted as a bright region. The results of experiment 1 and 2 are depicted in Figure 3 (a) - (d). We observe that graph cut

Graph Cut Optimization for the Piecewise Constant Level Set Method

9

Fig. 2. Test images Table 1. Computation times in seconds for gradient descent vs graph cut optimization

Experiment1 Experiment2 Experiment3 Brain

Size 100x100 100x100 92x98 933x736

Number of Phases Gradient descent Graph Cut 4 50.3 0.120 5 70.0 0.179 5 55.4 0.165 4 5401 25.22

optimization solve the multiphase problem with at least as good quality as gradient descent. In experiment 3 Figure 3(e)(f), the number of regions is assumed to be unknown. The optimal number of regions can be estimated by using more phases than necessary in the minimization problem. As we can see, this results in some empty phases, while the remaining phases capture the correct regions. We have also tested the method on a synthetic brain MRI image. The noise level is 7%, and non-uniformity of the RF-puls is of 20 % (see http://www.bic.mni.mcgill.ca/brainweb/ for details). We want to extract four tissue classes from the image: region 1; background, region 2; cerebrospinal fluid, region 3; gray matter and region 4; white matter. This is achieved by minimizing the Mumford-Shah model with 4 phases. In Figure 4 we compare the results of graph cuts, gradient descent and the exact results. The background phase is not shown. Again, we observe that graph cut results are very good, while the computation time is dramatically reduced compared to gradient descent(c.f. Table 1, brain). Finally, in Figure 5 we show an example which demonstrates the limitation of the multiphase approach presented in [18], described in the related work section. For the chosen parameter ν, the global minimum consists of three phases, which we are able to detect by applying our multiphase algorithm with 4 phases, Figure 5(b) top. The result of the first step of the algorithm presented in [18] is shown in Figure 5(b) buttom, which is the global minimum of the two phase Mumford-Shah functional. Clearly, no further splitting of these regions can result in the correct three phases, since the interphase from the first step is not allowed to evolve.

10

E. Bae and X.-C. Tai

(a) Experiment 1: graph cut

(b) Experiment 1: gradient descent

(c) Experiment 2: graph cut

(d) Experiment 2: gradient descent

(e) Experiment 3: graph cut

(f) Experiment 3: gradient descent Fig. 3. (a) and (b) Experiment 1, from left to right: phase 1 - phase 4. (c) and (d) Experiment 2, from left to right: phase 1 - phase 5. (e) and (f) Experiment 3, from left to right: phase 1 - phase 5.

Graph Cut Optimization for the Piecewise Constant Level Set Method

11

(a) graph cut

(b) gradient descent

(c) exact phases Fig. 4. From left to right: phase 1 - phase 3. (a) Graph cut, (b) gradient descent, (c) exact phases.

(a)

(b)

Fig. 5. (a) Input image. (b) Top: Our approach, from left to right phase 1 - phase 4. Buttom: First step of approach reported in [18], from left to right phase 1 - phase 2.

5

Summary

We have presented an algorithm for efficiently minimizing the energy induced by the piecewise constant level set representation of the multiphase MumfordShah functional. This minimization method is based on graph cuts. Numerical

12

E. Bae and X.-C. Tai

experiments demonstrated the method is very superior in efficiency compared to the previous PDE based approach, while maintaining the same quality of results.

References 1. Dervieux, A., Thomasset, F.: A finite element method for the simulation of a Rayleigh-Taylor instability. In: Approximation methods for Navier-Stokes problems, Proc. Sympos., Univ. Paderborn, Paderborn, 1979. Lecture Notes in Math., vol. 771, pp. 145–158. Springer, Berlin (1980) 2. Osher, S., Sethian, J.: Fronts propagating with curvature dependent speed: algorithms based on hamilton-jacobi formulations. J. Comput. Phys. 79(1), 12–49 (1988) 3. Lie, J., Lysaker, M., Tai, X.: A variant of the level set method and applications to image segmentation. Math. Comp. 75(255), 1155–1174 (2006) (electronic) 4. Lie, J., Lysaker, M., Tai, X.: A binary level set model and some applications to mumford-shah image segmentation. IEEE Transactions on Image Processing 15(5), 1171–1181 (2006) 5. Tai, X., Christiansen, O., Lin, P., Skjaelaaen, I.: Image segmentation using some piecewise constant level set methods with mbo type of project. International Journal of Computer Vision 73, 61–76 (2007) 6. Mumford, D., Shah, J.: Optimal approximation by piecewise smooth functions and associated variational problems. Comm. Pure Appl. Math. 42, 577–685 (1989) 7. Chan, T., Vese, L.: Active contours without edges. IEEE Image Proc. 10, 266–277 (2001) 8. Vese, L.A., Chan, T.F.: A new multiphase level set framework for image segmentation via the mumford and shah model. International Journal of Computer Vision 50, 271–293 (2002) 9. Greig, D.M., Porteous, B.T., Seheult, A.H.: Exact maximum a posteriori estimation for binary images. Journal of the Royal Statistical Society, Series B, 271–279 (1989) 10. Boykov, Y., Kolmogorov, V.: An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision. In: Figueiredo, M., Zerubia, J., Jain, A.K. (eds.) EMMCVPR 2001. LNCS, vol. 2134, pp. 359–374. Springer, Heidelberg (2001) 11. Kolmogorov, V., Zabih, R.: What energy functions can be minimized via graph cuts? IEEE Transactions on Pattern Analysis and Machine Intelligence 26(2), 147– 159 (2004) 12. Boykov, Y., Veksler, O., Zabih, R.: Fast approximate energy minimization via graph cuts. In: ICCV, vol. (1), pp. 377–384 (1999) 13. Ishikawa, H.: Exact optimization for markov random fields with convex priors. IEEE Transactions on Pattern Analysis and Machine Intelligence 25(10), 1333– 1336 (2003) 14. Ishikawa, H., Geiger, D.: Segmentation by grouping junctions. In: CVPR 1998: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Washington, DC, USA, pp. 125–131. IEEE Computer Society, Los Alamitos (1998) 15. Darbon, J., Sigelle, M.: Image restoration with discrete constrained total variation part ii: Levelable functions, convex priors and non-convex cases. J. Math. Imaging Vis. 26(3), 277–291 (2006)

Graph Cut Optimization for the Piecewise Constant Level Set Method

13

16. Darbon, J.: A note on the discrete binary mumford-shah model. In: Gagalowicz, A., Philips, W. (eds.) MIRAGE 2007. LNCS, vol. 4418, pp. 283–294. Springer, Heidelberg (2007) 17. Zehiry, N.E., Xu, S., Sahoo, P., Elmaghraby, A.: Graph cut optimization for the mumford-shah model. In: Proceedings of the Seventh IASTED International Conference visualization, imaging and image processing, pp. 182–187. Springer, Heidelberg (2007) 18. El-Zehiry, N.Y., Elmaghraby, A.: A graph cut based active contour for multiphase image segmentation. In: IEEE International Conference on Image Processing, pp. 3188–3191 (2008) 19. Lie, J., Lysaker, M., Tai, X.: Piecewise constant level set methods and image segmentation. In: Kimmel, R., Sochen, N.A., Weickert, J. (eds.) Scale-Space 2005. LNCS, vol. 3459, pp. 573–584. Springer, Heidelberg (2005) 20. Chung, G., Vese, L.A.: Energy minimization based segmentation and denoising using a multilayer level set approach. In: Rangarajan, A., Vemuri, B.C., Yuille, A.L. (eds.) EMMCVPR 2005. LNCS, vol. 3757, pp. 439–455. Springer, Heidelberg (2005) 21. Jung, Y.M., Kang, S.H., Shen, J.: Multiphase image segmentation via modicamortola phase transition. SIAM J. Appl. Math. 67, 1213–1232 (2007) 22. Boykov, Y., Kolmogorov, V.: Computing geodesics and minimal surfaces via graph cuts. In: ICCV 2003: Proceedings of the Ninth IEEE International Conference on Computer Vision, Washington, DC, USA, pp. 26–33. IEEE Computer Society, Los Alamitos (2003) 23. Ford, L., Fulkerson, D.: Flows in networks. Princeton University Press, Princeton (1962) 24. Geman, S., Geman, D.: Stochastic relaxation, gibbs distributions, and the bayesian restoration of images. In: Readings in uncertain reasoning, pp. 452–472. Morgan Kaufmann Publishers Inc., San Francisco (1990) 25. Dahlhaus, E., Johnson, D.S., Papadimitriou, C.H., Seymour, P.D., Yannakakis, M.: The complexity of multiway cuts (extended abstract). In: STOC 1992: Proceedings of the twenty-fourth annual ACM symposium on Theory of computing, pp. 241– 251. ACM, New York (1992) 26. Darbon, J., Sigelle, M.: Image restoration with discrete constrained total variation part i: Fast and exact optimization. J. Math. Imaging Vis. 26(3), 261–276 (2006) 27. Velasco, F.R.D.: Thresholding using the ISODATA clustering algorithm. IEEE Trans. Systems Man Cybernet. 10(11), 771–774 (1980)

Tubular Anisotropy Segmentation Fethallah Benmansour and Laurent D. Cohen CEREMADE, UMR CNRS 7534, Université Paris Dauphine, Place du Maréchal De Lattre De Tassigny, 75775 PARIS CEDEX 16, France {benmansour,cohen}@ceremade.dauphine.fr

Abstract. In this paper we present a new interactive method for tubular structure extraction. The main application and motivation for this work is vessel tracking in 2D and 3D images. The basic tools are minimal paths solved using the fast marching algorithm. This allows interactive tools for the physician by clicking on a small number of points in order to obtain a minimal path between two points or a set of paths in the case of a tree structure. Our method is based on a variant of the minimal path method that models the vessel as a centerline and surface. This is done by adding one dimension for the local radius around the centerline. The crucial step of our method is the definition of the local metrics to minimize. We have chosen to exploit the tubular structure of the vessels one wants to extract to built an anisotropic metric giving higher speed on the center of the vessels and also when the minimal path tangent is coherent with the vessel’s direction. This measure is required to be robust against the disturbance introduced by noise or adjacent structures with intensity similar to the target vessel. We obtain promising results on noisy synthetic and real 2D and 3D images.

1 Introduction In this paper we deal with the problem of finding a complete segmentation of tubular structures like vessels. The main objective is to extract at the same time the centerline of the tubular structure and its boundary. During the last two decades, the extraction of vascular objects such as the blood vessel, coronary arteries, or other tube-like structures has attracted the attention of more and more researchers. Various methods such as vascular image enhancement methods [1, 2, 3], or others were proposed, see [4] for a complete survey. Some of these methods extract the vessel boundary directly, and then use thinning methods to find its centerline. Other methods extract only the centerline and then estimate the vessel width to extract its boundary. Deschamps and Cohen [5] proposed to use the minimal path method to find the centerline. The minimal path technique introduced by Cohen and Kimmel [6] captures the global minimum curve between two points given by the user. This leads to the global minimum of an active contour energy. Since then, the minimal path method has been improved by many researchers, and adapted to anisotropic media as done by Jbabdi et al for tractography [7]. Unfortunately, despite their numerous advantages, classical minimal path techniques exhibit some disadvantages. First, vessel boundary extraction can be very difficult, even in 2D X.-C. Tai et al. (Eds.): SSVM 2009, LNCS 5567, pp. 14–25, 2009. c Springer-Verlag Berlin Heidelberg 2009 

Tubular Anisotropy Segmentation

15

where the vessel’s boundary can be completely described by two curves. Second, the path given by the minimal path technique does not always yield to the centerline of the vessel. A readjustment step is required to obtain a central trajectory. Third, the minimal path technique provides only a trajectory and does not give information about the vessel boundary and local width. Li and Yezzi [8] proposed a new variant of the classical, purely spatial, minimal path technique by incorporating an extra non-spatial dimension into the search space. Each point of the 4D path (after adding the extra dimension for the 3D image) consists of three spatial coordinates plus a fourth coordinate which describes the vessel thickness at that corresponding 3D point. Thus, each 4D point represents a sphere in 3D space, and the vessel is obtained by taking the envelope of these spheres as we move along the 4D curve. A crucial step of this method is to build an adequate potential that drives the propagation. Li and Yezzi [8] proposed different isotropic potentials. As they said in the conclusion of their paper, the proposed potentials are very parameter dependent and they hoped to find more appropriate choice of potential. In particular, one can see in their paper, that the potential used does not yield to a correct detection of the radius when it is not constant (see figure 6 in [8]). An other drawback of Li and Yezzi method is that they did not take into account the vessel orientation. Our first contribution is to take into account the vessel orientation by defining a suited anisotropic metric that makes the propagation faster along the centerlines and for the adequate radius. Law et al. [9] proposed a new scalar descriptor called Optimally Oriented Flux (OOF) for the detection of curvilinear structures. But they did not exploit the orientation given by their descriptor. The major advantage of the OOF technique is that it does not consider the regions in the vicinity of target objects, where background noise or adjacent structures with intensity similar to the target vessels are possibly present. Therefore, the disturbance introduced by the closely located nearby structures is avoided. The second contribution of this paper is to build an anisotropic metric based on the OOF descriptor, its scalar function as well as its orientation. That makes the propagation faster along the vesselÕs center line and for exact associated scale. This means that the path location, orientation and scale (radius) have to be coherent with the local geometry of the image extracted by the OOF. In section 2, we give some background on minimal path method and Anisotro-pic Fast Marching. In section 3 the Optimally Oriented Flux descriptor is presented as well as the metric construction. In section 4, results on synthetic and real data are shown. Finally, conclusions and perspectives follow in section 5.

2 Background on Minimal Path Method A minimal path, first introduced in the isotropic (P does not depend on the orientation of the path) case [6], is a pathway minimizing the energy functional,    E(γ) = P γ(s), γ  (s) ds (1) 

γ

γ  (.)T M(γ(.))γ  (.) describes an infinitesimal distance where, P(γ(.), γ  (.)) = along a pathway γ relative to a metric tensor M (symmetric definite positive). Thus,

16

F. Benmansour and L.D. Cohen

we are considering only the case of an elliptic medium. In the isotropic case M(.) = P 2 (.)I, where I is the identity matrix. A curve connecting p1 to p2 that globally minimizes the above energy (1) is a minimal path between p1 and p2 , noted Cp1 ,p2 . The solution of this minimization problem is obtained through the computation of the minimal action map U : Ω → R+ associated to p1 on the domain Ω which can be a 2D, 3D or 4D domain. The minimal action is the minimal energy integrated along a path between p1 and any point x of the domain Ω :      ∀ x ∈ Ω, U(x) = min P γ(s), γ (s) ds , (2) γ∈Ap1 ,x

γ

where Ap1 ,x is the set of paths linking x to p1 . The values of U may be regarded as the arrival times of a front propagating from the source p1 with oriented velocity related to the metric tensor M−1 . U satisfies the Eikonal equation ∇U(x)M−1 (x) = 1 for x ∈ Ω, and U(p1 ) = 0,

(3)

√ where vM = vT M v. The map U has only one local minimum, the point p1 , and its flow lines satisfy the Euler-Lagrange equation of functional (1). Thus, the minimal path Cp1 ,p2 can be retrieved with a simple gradient descent on U from p2 to p1 (see Fig. 1), solving the following ordinary differential equation with standard numerical methods like Heun’s or Runge-Kutta’s :   dCp1 ,p2 (s) ∝ −M−1 (Cp1 ,p2 (s))∇U Cp1 ,p2 (s) , with Cp1 ,p2 (0) = p2 . ds

(4)

Proof of (3) and (4) can be found in [10,7]. On figure 1, we show some examples of the minimal path method on an isotropic case and an anisotropic one. On the first image of figure 1 the metric is isotropic and the potential P in the grey region is twice as low as the white one. Isolevel sets of the minimal action map associated to the source point p1 are displayed and so the minimal path Cp1 ,p2 . The second image represents a metric M. We took two constant metrics in each half side of the image with different orientations. On the last image, the minimal action map U associated to the metric M and to the source point p1 is shown. The minimal path Cp1 ,p2 is found by solving equation (4).

p1 p1

p2

p2 0

50

100

150

200

250

300

350

Fig. 1. Minimal path examples on an isotropic case on the left image. On the middle, visualization by small ellipses of eigenvalues of a metric constant on each half side of the image. On the right, the minimal action map associated to the source point p1 with the minimal path Cp1 ,p2 .

Tubular Anisotropy Segmentation

17

The Fast Marching Method (FMM) is a numerical method introduced by Sethian in [11] and Tsitsiklis in [12] for efficiently solving the isotropic Eikonal equation on a cartesian grid. The central idea behind the FMM is to visit grid points in an order consistent with the way wavefronts of constant action propagate. It leads to a single-pass algorithm for solving equation (3) and computing the minimal action map U. Tsitsiklis’s method relies on minimizing directly the energy functional of equation (1) while Sethian’s method uses the Eikonal equation. Both methods are suitable for isotropic metric, but they fail for anisotropic metric [13]. To deal with anisotropy, Sethian and Vladimirsky [10] proposed an update scheme that converges to the viscosity solution of the anisotropic Eikonal equation. A simplified scheme, based on the original Tsitsiklis’s method [12], was proposed by Lin in [14] to approximate the solution of the anisotropic Eikonal equation. Contrary to Sethian and Vladimirsky’s ordered upwind method (OUM) [10], Lin’s algorithm does not converge to the viscosity solution of the Eikonal equation. In this paper we used Lin’s scheme to solve the anisotropic Eikonal equation, since it is much faster (than OUM) and the introduced errors do not affect much the extracted geodesics. The FMM is a front propagation approach that computes the values of U in increasing order, and the structure of the algorithm is almost identical to Dijkstra’s algorithm for computing shortest paths on graphs [15]. In the course of the algorithm, each grid point is tagged as either Alive (point for which U has been computed and frozen), Trial (point for which U has been estimated but not frozen) or Far (point for which U is unknown). The set of Trial points forms an interface between the set of grid points for which U has been frozen (the Alive points) and the set of other grid points (the Far points). This interface may be regarded as a front expanding from the source until every grid point has been reached. Let us denote by NM (x) the set of M neighbors of a grid point x, where M = 2 × d if the dimension of Ω is equal to d. Initially, all grid points are tagged as Far, except the source point p1 that is tagged as Trial. At each iteration of the FMM one chooses the Trial point with the smallest U value, denoted by xmin . Then, xmin is tagged as Alive and the value of U is updated for each point of the set NM (xmin ) which is either Trial or Far. In order to satisfy a causality condition, the way U is updated in the vicinity of xmin requires special care. The iteration ends by tagging every Far point of the set NM (xmin ) as Trial. The algorithm automatically stops when all grid points are Alive. The key to the speed of the FMM is the use of a priority queue to quickly find the Trial point with the smallest U value. If Trial points are ordered in a min-heap data structure, the computational complexity of the FMM is O(N logN ), where N is the total number of grid points. A crucial step of the Fast Marching algorithm is the computation of the weighted distance between the front and the neighbouring voxels in the Trial set. Here, we present a way to estimate this weighted distance in the anisotropic case and only in 3D. It is straightforward to extend it to 4D. Since the distance is anisotropic, we cannot use the standard methods, because they rely on the fact that the geodesics are perpendicular to the level sets of U. To take into account the anisotropy Jbabdi et al [7] and Lin [14] considered a set of simplexes that cover the whole neighbourhood around a voxel of the narrow band. The definition of a simplex neighbouring a point x is simply a set of three points (x1 , x2 , x3 ) that are among the 26 neighbours of x, defining a triangle that we

18

F. Benmansour and L.D. Cohen x3 xm x

x2

x1

Fig. 2. On the left Position of the optimal point on a simplex such as to minimize the geodesic distance to x. On the right the considered simplexes.

denote x1 x2 x3 . There are 48 such triangles around x for the 26 connexity. To make the update procedure faster, we propose to consider only the simplexes defined by a t-uple of three points of the 6-neighbors of x. There are 8 such triangles (see Fig. 2), and by making this modification, the precision of the algorithm is lower but the algorithm is six times faster. To estimate U(xm ), where xm is a neighbor of the last trial point xmin , we make two approximations. If the geodesic passing by xm comes from a triangle x1 x2 x3 then the time of arrival is given by:  xm  U(x) + U(xm ) = min P (γ, γ  ) (5) x∈x1 x2 x3

x

The term one wants to minimize is approximated by : f (α) =

3

i=1

3  

  αi U(xi ) + x − αi xi  i=1

M(x)

,

(6)

3 where α = (α1 , α2 , α3 ), with i=1 αi = 1 since the point x is in the triangle (see figure 2). This equation follows Tsitsiklis’s approximation [12]. The first term approximates the value of the minimal action map at the point x = 3i=1 αi xi by a simple linear interpolation. And the second term approximates the remaining distance by considering the metric constant along the segment [x, xm ] equal to its value at point xm . 3 The function f is convex and the constraints on α, i.e i=1 αi = 1 and αi ≥ 0, define a convex subset. Thus the minimization of f can be done using classical optimization tools. See [7] for more details. For each of the eight triangles, we get a value u. Finally, we choose the triangle giving the smallest value of u. Note that in order to approximate ∇U, computing the derivatives of U in the triangle using the estimate U(xn ) gives a consistent approximation of ∇U(xn ) by the following: ∇U(xn ) = (U(xn ) − U(x ))

xn − x , xn − x 

where x is the minimizer of function f , see figure 2 left, and . is the Euclidean norm. The computation of the gradient is very useful since it is used to solve the gradient descent described by equation (4).

Tubular Anisotropy Segmentation

19

3 Optimally Oriented Flux: An Anisotropy Descriptor We are interested in the construction of a metric that extracts from the image the geometric information leading to reconstruction of vessels. This means that we wish to find an estimate for the local orientation and scale and a criterion on the local geometry to distinguish the presence of vessels from the background. At the position x on an image I, the amount of the image gradient projected along the axis v flowing out from a 3D sphere (or a 2D circle) Sr is measured as in [9], 

 h da, (7) (∇(G ∗ I(x + h)) · v)v · f (x, v; r) = |h| ∂Sr where G is a Gaussian function with a scale factor of 1 pixel, r is the sphere (or circle) radius, h is the position vector along ∂Sr and da is the infinitesimal area (or length) on ∂Sr . To detect vessels having higher intensity than the background region, one would be interested in finding the vessel direction which minimizes f (x, v; r), i.e. we are looking for: arg min f (x, v; r). Using the divergence theorem, it can be shown that f (x, v; r) v can be calculated using a simple convolution, f (x, v; r) = vT {(∂i,j G) ∗ I ∗ 1Sr } v,

(8)

where (∂i,j G) is the Hessian matrix of function G and 1Sr is the indicator function inside the sphere (or circle) Sr . By differentiating the above equation with respect to v, minimization of function f is in turn acquired as solving a generalized eigenvalue decomposition problem. Solving the aforementioned generalized eigen decomposition problem gives d eigenvalues (where d = 2 or 3 is the dimension of the image), λ1 (·) ≤ · · · ≤ λd (·) and d eigenvectors vi (·), i.e. λi (x; r) = f (x, vi (x; r); r) for i = 1, . . . , d. To handle the vessels having various radii, a multi-scale approach should be used along with the OOF method. In [9], Law and Chung have proposed to normalize the OOF’s eigenvalues by the sphere surface area when the OOF method is incorporated in a multiscale approach for 3D image volumes. In the 2D case the eigenvalues are normalized by the circle perimeter 2πr. In the 3D case the eigenvalues are normalized by the sphere area 4πr2 . In the 2D case (see figure 3), for a point on the centerline and if r is equal to radius of the vessel, the first eigenvector v1 represents the direction orthogonal to the vessel. v2 reprensents the direction along the vessel. In the 3D case, if the point is on the centerline, the two eigenvectors associated to the first eigenvalues (λ1 , λ2 ) represent the directions orthogonal the vessel. v3 represents the direction along the vessel, see figure 4. On the same figure, one can see that if the point x is on the centerline, the minimal response of the function f is obtained when the radius r is equal to the exact radius of the tube. If the point is inside the tube but not on the centerline, v3 is parallel to the tube orientation, and the other eigenvectors depends on the scale r. If the point is outside the tube (last line), then the vector v3 , corresponding to the red area, is oriented toward the centerline. Li and Yezzi [8] proposed a new variant of the classical, purely spatial, minimal path technique by incorporating an extra non-spatial dimension into the search space. The crucial step of this method is to build an adequate metric that drives the propagation.

20

F. Benmansour and L.D. Cohen

Fig. 3. The plots of the values of f (x, v; r) obtained from the synthetic image shown in the left, at four different positions with various radii and projection axes. (a) Four interested positions, denoted as x1 , x2 , x3 and x4 are shown along with the original synthetic image. (b) An illustration regarding the polar coordinate system used in (c)-(f). (c)-(f) The plots of the values of f (·) and the corresponding eigenvectors, computed at the four different positions shown in (a), using various values of r and different projection axes (cos θ sin θ)T .

Fig. 4. Plot of f (x, v; r) superimposed on the original 3D synthetic image for three different points(on each line) and different values of the radius : r = 3, . . . , 7 from left to right. The radius of the tube on the top half side image is equal to 4, and equal to 6 on the bottom half side. Similarly to figure 3, the visualization of the normalized flux function is done using a spherical coordinate system (instead of the polar one used in 2D). The first point is on the centerline of the tube. The second point is inside the tube but not on the centerline. The third point is outside the tube. The reader should zoom on each image. Notice that the colormaps are different.

Li and Yezzi [8] proposed different isotropic potentials. The main drawback, as they mention, is that these potentials are very parameter dependent and they do not exploit the vessel orientation. Our main contribution is to improve Li and Yezzi method by adding to it an anisotropic formulation, and the anisotropic metric is constructed by extension of the OOF descriptor presented by Law et al. [9].

Tubular Anisotropy Segmentation

21

Fig. 5. The constructed metric for different scales r = 1, 5, 10, 15, 20 from left to right. The original image is shown in figure 3 (a), the radius of the structure is equal to 10. We used the same color range for all images, so one can see that the optimal anisotropy is obtained along the centerline of the tubular structure when the scale r is equal to the exact radius of the tube. On ˜ the top, we show a display of M(x, r)−1 . On the bottom, responses of Pradii are shown.

The (d + 1)D minimal path is found by minimizing the following energy:   γ  (s)T M(γ(s))γ  (s) ds, where M is the (d + 1)D anisotropic metric we want γ to construct. It is not natural to consider orientations on the (d + 1) dimension, i.e the radii dimension. Thus one candecompose by block the metric M as follows:  ˜ M(x, r) 0 ˜ where M(x, r) is a d × d symmetric definite M(x, r) = 0 Pradii (x, r) positive matrix giving the spatial anisotropy and Pradii (x, r) is the radii potential (also strictly positive). Since the result given by the anisotropic minimal path method is very dependent on the metric, results inherit advantages and drawbacks of the constructed metric, thus we should be very carful with its construction. First, let us fix conditions on the desired ˜ has to be well oriented along the vessel centerline. And metric. The spatial metric M the radii potential Pradii has to be small for the adequate scale for any point of the image. ˜ is symmetric Pradii corresponds to the inverse speed for the radii dimension. Since M d T ˜ m definite positive, we can decompose it as follows: M(.) = i (.)ui (.)ui (.) , i=1 where 0 < m1 ≤ · · · ≤ md are the eigenvalues and ui are the associated eigenvectors. √ The velocity of the propagating front along direction ui is equal to 1/ mi . We used the OOF descriptor to construct the metric as follows:     d λj (.) λ (.) i j = i T i=1 ˜ M(.) = vi (.)vi (.) , Pradii (.) = β exp α . exp α d−1 d i=1 (9) The constant α is controlled by  an intuitive parameter, which is the maximal  exp(αλ2 (x, r)) in the 2D case and μ = spatial anisotropy ratio: μ = max x,r exp(αλ1 (x, r)) d

22

F. Benmansour and L.D. Cohen

⎫

⎧ 3 (x,r) ⎨ exp α λ2 (x,r))+λ ⎬ 2  in the 3D case. By choosing the maximal spatial

max x,r ⎩ 2 (x,r) ⎭ exp α λ1 (x,r))+λ 2 anisotropy ratio μ, the constant α is fixed. And by doing so, the anisotropy descriptor M becomes contrast invariant because the OOF is linear on the image. The parameter β controls the radii speed. In 2D (it is very similar in 3D), if Pradii ≤ exp(αλ1 ) then the Fast Marching propagation is faster for the radii than the spatial dimensions. If Pradii ≥ exp(αλ2 ) then the propagation is slower. One can tune parameter β depending on the tubular structure one wants to extract. If its radius changes a lot then β should be chosen such that the propagation on the radii dimension is faster. If not β is chosen such that the propagation is less sensitive on the radii dimension. On figure 5 the constructed metric of image 3 at some different scales is shown. Since we chose the same color range for the visualization, we can see that the directions are well detected, and that the optimal values are obtained along the centerline of the tube

Fig. 6. The red cross points are source points given by the user, and the blue ones are end points. On each case the segmented centerlines are displayed as well as the envelope of the moving discs. In the middle, the associated minimal action map U as well as the 3D minimal path between the two selected points are shown (transparent visualization).

Tubular Anisotropy Segmentation

23

when the scale is equal to the  tube radius. For our experiments, we took μ = 10 and 1) β such that max exp(αλ = 5, this means that in the worst case, the speed along Pradii the radii dimension is 5 times faster than the spatial dimensions. We did so, because we wanted our algorithm to be sensitive to the radii dimension.

4 Experimental Results Our method is minimally interactive. First, the user has to precise if the desired vessels are darker or brighter than the background. So, we can consider different criteria on the signs of the eigenvalues. Then the scale range [rmin , rmax ], which corresponds to the range of radii of the vessel one wants to extract, is given by the user. Finally few points are required as source points or end points of the Fast Marching algorithm. We used the metric described in the previous section to find the minimal anisotropic path (as described in section 2) between two or more selected points (see figures 6 and 7). For any selected point, the associated radius is equal to the minimal radius rmin given by the user. On figure 6, segmentation results on synthetic and real noisy 2D images are shown. On the first synthetic image, the source point and destination are selected on the centerline. The obtained tube is perfectly detected as well as the centerline. On the second

Fig. 7. First line : RCA segmentation using the tubular anisotropy approach shown on the whole image and on the selected sub-volume. Second line : LAD segmentation shown on the whole image and on the selected sub-volume. Only few points are required (the extremities of the paths). The tubular anisotropy method provides the centerline as well as vessels boundaries.

24

F. Benmansour and L.D. Cohen

image, the initial points are not centered. But the centerline given by our algorithm goes back fast to the real centerline. This makes our algorithm robust to initialization. The third synthetic image shows that our approach is robust to scale changing. On the last line of figure 6, segmentation results are shown on real noisy images. In figure 7, segmentation results are shown on real medical images. First, right coronary arteries (RCA) are segmented. Second, left anterior descending (LAD) arteries are segmented. One can see that the obtained radii on the principal coronary branches are larger than those of the secondary. Thus, our approach is robust to scale changing and bifurcations. Nevertheless, our current implementation requires huge memory allocations due to the 4D and anisotropic aspects. To overcome this issue, we added a preprocessing interactive tool to select a sub-volume containing the desired vessels (see figure 7). Moreover, we are working on a new implementation of the tubular anisotropy approach to make the memory allocation dynamic and hence to benefit from the front propagation aspect of the fast marching algorithm. Besides the reduction of the computation time (which has been actually achieved), we will save on memory allocation and will have a new version of our algorithm that extract the whole coronary arteries using a regular PC.

5 Conclusion In this paper we have proposed a new general method for tubular structure extraction in 2D and 3D images. Our method exploit the orientation of the vessels by using the optimally oriented flux to construct a multi-resolution anisotropic metric that extracts from the image the local geometry and describes the vessels orientation and scales. Combining this metric with anisotropic minimal path technique, we were able to find a complete description of the tubular structure, i.e the centerline as well as the boundary. To summarize, our method is minimally interactive, robust to initialization, scale variations and bifurcations.

Acknowledgements We would like to thank Professor Anthony J. Yezzi and Max Wai-Kong Law for interesting discussions. Also Eduardo Davila for his precious help for the implementation of the interface. This work was partially supported by ANR grant SURF -NT05-2_45825.

References 1. Sato, Y., Nakajima, S., Shiraga, N., Atsumi, H., Yoshida, S., Koller, T., Gerig, G., Kikinis, R.: Three-dimensional multi-scale line filter for segmentation and visualization of curvilinear structures in medical images. Med. Image Anal. 2(2), 143–168 (1998) 2. Krissian, K.: Flux-based anisotropic diffusion applied to enhancement of 3D angiogram. TMI 21(11), 1440–1442 (2002) 3. Frangi, A., Niessen, W.J., Vincken, K.L., Viergever, M.A.: Multiscale vessel enhancement filtering. In: Wells, W.M., Colchester, A.C.F., Delp, S.L. (eds.) MICCAI 1998. LNCS, vol. 1496, pp. 130–137. Springer, Heidelberg (1998)

Tubular Anisotropy Segmentation

25

4. Kirbas, C., Quek, F.K.H.: A review of vessel extraction techniques and algorithms. ACM Computing Surveys 36, 81–121 (2004) 5. Deschamps, T., Cohen, L.: Fast extraction of minimal paths in 3D images and applications to virtual endoscopy. MIA 5(4) (December 2001) 6. Cohen, L.D., Kimmel, R.: Global minimum for active contour models: a minimal path approach. International Journal of Computer Vision 24, 57–78 (1997) 7. Jbabdi, S., Bellec, P., Toro, R., Daunizeau, J., Pélégrini-Issac, M., Benali, H.: Accurate anisotropic fast marching for diffusion-based geodesic tractography. Journal of Biomedical Imaging 2008(1), 1–12 (2008) 8. Li, H., Yezzi, A.: Vessels as 4D curves: Global minimal 4D paths to extract 3D tubular surfaces and centerlines. IEEE Transactions on Medical Imaging 26(9), 1213–1223 (2007) 9. Law, M.W.K., Chung, A.C.S.: Three dimensional curvilinear structure detection using optimally oriented flux. In: ECCV, vol. 4, pp. 368–382 (2008) 10. Sethian, J.A., Vladimirsky, A.: Fast methods for the eikonal and related hamilton- jacobi equations on unstructured meshes. Proceedings of the National Academy of Sciences 97(11), 5699–5703 (2000) 11. Sethian, J.A.: A fast marching level set for monotonically advancing fronts. Proceedings of the National Academy of Sciences 93, 1591–1595 (1996) 12. Tsitsiklis, J.N.: Efficient algorithms for globally optimal trajectories. IEEE Transactions on Automatic Control 40, 1528–1538 (1995) 13. Chopp, D.L.: Replacing iterative algorithms with single-pass algorithms. Proc. Nat. Acad. Sc. USA 98(20), 10992–10993 (2001) 14. Lin, Q.: Enhancement, extraction, and visualization of 3D volume data. PhD thesis, Linkopings Universitet (2003) 15. Dijkstra, E.W.: A note on two problems in connection with graphs. Numerische Mathematic 1, 269–271 (1959)

An Unconstrained Multiphase Thresholding Approach for Image Segmentation Benjamin Berkels Institut für Numerische Simulation, Rheinische Friedrich-Wilhelms-Universität Bonn, Nussallee 15, 53115 Bonn, Germany [email protected] http://numod.ins.uni-bonn.de/ Abstract. In this paper we provide a method to find global minimizers of certain non-convex 2-phase image segmentation problems. This is achieved by formulating a convex minimization problem whose minimizers are also minimizers of the initial non-convex segmentation problem, similar to the approach proposed by Nikolova, Esedo¯ glu and Chan. The key difference to the latter model is that the new model does not involve any constraint in the convex formulation that needs to be respected when minimizing the convex functional, neither explicitly nor by an artificial penalty term. This approach is related to recent results by Chambolle. Eliminating the constraint considerably simplifies the computational difficulties, and even a straightforward gradient descent scheme leads to a reliable computation of the global minimizer. Furthermore, the model is extended to multiphase segmentation along the lines of Vese and Chan. Numerical results of the model applied to the classical piecewise constant Mumford-Shah functional for two, four and eight phase segmentation are shown.

1

Introduction

Image segmentation is one of the fundamental research topics in the field of image processing. In particular, the Mumford-Shah model [1] is widely used in this context. One of the difficulties of this and many other variational image processing models is that the underlying energy functional has local, non-global minima. This is not only a theoretical problem, since the commonly used numerical minimization techniques often get stuck in local minima that differ considerably from a global minimum, hence possibly producing useless results. The goal of this paper is to introduce a method to obtain a global minimizer of the Mumford-Shah functional for 2-phase segmentation that only involves solving an unconstrained convex minimization problem. This method can be extended to multiphase segmentation by the ideas of Vese and Chan [2] in a canonical way. 1.1

Related Work

The problem of minimizing the Mumford-Shah segmentation functional has been extensively studied in the last decade leading to a wide range of existing methods, X.-C. Tai et al. (Eds.): SSVM 2009, LNCS 5567, pp. 26–37, 2009. c Springer-Verlag Berlin Heidelberg 2009 

Image Segmentation by Unconstrained Multiphase Thresholding

27

each with its own shortcomings. One of the first numerical feasible methods to obtain (local) minimizers of the functional was proposed by Chan and Vese [3]. They build on the levelset methods of Osher and Sethian [4] and parameterize the unknown set by a levelset function. Shen [5] developed a Γ -convergence formulation along with a simple implementation by the iterated integration of a linear Poisson equation. The unknown set is represented in a diffuse way by a phase field. In [6], Esedo¯glu and Tsai tackle the minimization problem based on the threshold dynamics of Merriman, Bence and Osher [7] for evolving an interface by its mean curvature. Here the minimization is achieved by alternating the solution of a linear parabolic partial differential equation and simple thresholding. Alvino and Yezzi [8] approximate Mumford-Shah segmentation using reduced image bases. According to them, the majority of the robustness of Mumford-Shah segmentation can be obtained without allowing each pixel to vary independently. Their approximative model has comparable performance to Mumford-Shah segmentations where each pixel is allowed to vary freely. A way to obtain global minimizers was introduced by Nikolova, Esedo¯ glu and Chan [9]. Here, a convex constrained minimization problem has to be solved followed by a simple thresholding of the latter minimizer. This method is closely related to the method we propose in this paper, the key difference is that [9] requires a constraint in the convex minimization while the model proposed in this paper does not involve any constraint in the convex formulation. On the other hand there are methods to solve a certain class of minimal surface problems by unconstrained convex optimization, cf. the work of Chambolle and Darbon [10,11]. The 2-phase Mumford-Shah functional belongs to this class, yet due to the best of our knowledge nobody seems to have tapped the potential offered by these general insights for Mumford-Shah based image segmentation so far.

2

Constrained Global 2-Phase Minimization

First let us describe the general framework and revise the work of Nikolova et al. [9], the starting point for our model. In the following, Ω denotes our computational domain, an arbitrary but fixed subset of Rn . For given indicator functions f1 , f2 ∈ L1 (Ω) such that f1 , f2 ≥ 0 a.e. we consider the prototype Mumford-Shah energy   EMS [Σ] := f1 dx + f2 dx + ν Per(Σ), (1) Σ

Ω\Σ

where Per(Σ) denotes the perimeter of the set Σ ⊂ Ω in Ω. If u0 is an image, c1 , c2 ∈ R are two grey values and fi (x) := (u0 (x) − ci )2 , this is the well known piecewise constant Mumford-Shah functional for 2-phase segmentation, i.e.   E[Σ, c1 , c2 ] = (u0 − c1 )2 dx + (u0 − c2 )2 dx + ν Per(Σ). (2) Σ

Ω\Σ

28

B. Berkels

Remark 1. Because of





EMS [Σ] = Σ

(f1 − f2 )dx + ν Per(Σ) +  

f2 dx, Ω

ˆMS [Σ] =:E

ˆMS share the same minimizers. EMS and E 2

Remark 2. For h(x) := e−|x| , we have    (f1 + h) dx + (f2 + h) dx + ν Per(Σ) − hdx, EMS [Σ] = Σ Ω\Σ  Ω  =C<∞

i.e. replacing f1 and f2 by f1 + h and f2 + h does not affect the minimizers of EMS . This, combined with f1 , f2 ≥ 0 a.e., means that we can assume f1 , f2 > 0 a.e. in Ω without loss of generality. To obtain (local) minimizers of the functional above, Chan and Vese [3] proposed to parametrize the unknown set Σ by a levelset function φ and get the energy  ECV [φ] := H(φ)f1 + (1 − H(φ))f2 + ν|∇(H(φ))|dx. Ω

Here, H(·) denotes the Heaviside function, i.e. H(s) = 1 for s > 0 and H(s) = 0 else. A gradient descent will be used for minimization, therefore H is replaced by a smeared out Heaviside function, e.g. Hδ (x) := 12 + π1 arctan xδ , where δ > 0. While the specific choice is not important, it is important to use a function whose derivative does not have compact support (cf. [3]). This gives the regularized energy  ECV,δ [φ] := Hδ (φ)f1 + (1 − Hδ (φ)) f2 + ν |∇(Hδ (φ))| dx (3) Ω

and yields the gradient descent 

∇φ . ∂t φ = Hδ (φ) (f2 − f1 ) + ν div |∇φ|

(4)

One of the major drawbacks of the energy (3) is its non-convexity in φ. In [9], Nikolova et al. noted that the gradient descent (4) and 

∇φ ∂t φ = (f2 − f1 ) + ν div |∇φ| have the same stationary points, because Hδ (φ) > 0. Obviously the latter is the gradient descent of the energy  ECE [φ] := (f1 − f2 )φ + ν|∇φ|dx. Ω

Image Segmentation by Unconstrained Multiphase Thresholding

29

In general, f1 − f2 takes positive and negative values, therefore the energy is not bounded (neither from below nor from above). In other words, it does not necessarily have a minimizer. However, this is easily fixed by restricting the minimization to 0 ≤ φ(x) ≤ 1 for all x ∈ Ω. Based on this, the following theorem holds: Theorem 1. For given indicator functions f1 , f2 ∈ L1 (Ω) such that f1 , f2 ≥ 0 a.e., let  u := argmin (f1 − f2 )˜ u + ν|∇˜ u|dx = argmin ECE [˜ u] 0≤˜ u≤1

0≤˜ u≤1

Ω

and Σc := {x ∈ Ω|u(x) > c}. Then Σc is a minimizer of the Mumford-Shah energy (1) for all c ∈ [0, 1). Proof. Nikolova et al. proved this theorem in [9] for a.e. c ∈ [0, 1], we extend it here to hold not only for almost every, but for every c ∈ [0, 1). First, we briefly sketch the prove given by Nikolova et al. for a.e. c ∈ [0, 1]. Using 0 ≤ u ≤ 1 and the coarea formula, one can show  1 ECE [u] = EMS [Σc ]dc − C, 0

where C is a constant independent of u. Let Σ∗ ⊂ Ω be a minimizer of EMS (the existence of such minimizers using convergence in measure follows from standard arguments) and let M := {c ∈ [0, 1]|EMS [Σc ] > EMS [Σ∗ ]}. Assuming μ(M ) > 0 leads to the contradiction ECE [χΣ∗ ] < ECE [u] therefore μ(M ) = 0 holds and the statement is proven for a.e. c ∈ [0, 1]. Here, χA denotes the characteristic function of the set A. Now we extend the statement to all c ∈ [0, 1), inspired by the proof of Lemma 4 (iii) in [12]: Again let u be a minimizer of ECE under the constraint 0 ≤ u ≤ 1 and denote its superlevelsets by Σc . Choose an arbitrary but fixed cˆ ∈ [0, 1). The statement holds for a.e. c ∈ [0, 1], so by Remark 1, there exists a sequence (cn ) ∈ [0, 1]N with cn ↓ cˆ such that ˆMS [Σ]. Σcn ∈ argmin E Σ⊂Ω

Since the superlevelsets of a function are contained in each other, we have χΣcn =

∞ χ nk=1 Σck → χΣ ∪ pointwise a.e., where Σ ∪ := n=1 Σcn . Setting g := f1 − f2 and using Lebesgue’s dominated convergence theorem, we obtain     gdx = gχΣ ∪ dx = lim gχΣcn dx = lim gdx. Σ∪

Ω

n→∞

n→∞

Ω

Σcn

  Here we used gχΣcn  ≤ |g| ≤ |f1 | + |f2 | to provide the integrable upper bound. For each n and Σ ⊂ Ω, we have   gdx + ν Per(Σcn ) ≤ gdx + ν Per(Σ). Σ cn

Σ

30

B. Berkels

Using the continuity argument from above and the lower semicontinuity of the perimiter (cf. [13]), we get   gdx + ν Per(Σ ∪ ) ≤ gdx + ν Per(Σ), Σ∪

Σ

i.e. Σ ∪ is a minimizer of EMS [·, cn ]. Combining this with Σc = {x ∈ Ω|u(x) > c} =

∞ 

{x ∈ Ω|u(x) > cn } =

n=1

concludes the proof.

∞ 

Σc n

n=1



Knowing that Theorem 1 holds true for all c ∈ [0, 1) also remedies the last bit of “uncertainty” left in [9]. Remark 3. For any function u that fulfills the constraint, obviously {u > 1} = ∅. Therefore we cannot expect Theorem 1 to hold for c = 1. To solve the constrained optimization problem, Nikolova et al. show that the constrained problem has the  same minimizers as the unconstrained problem if a penalty term of the form α p(u(x)) is added with asufficiently large coefficient  α (cf. [9], Claim 1). Here p denotes p(s) = max{0, 2 s − 12  − 1} . While this result already gives a method to find global minimizers of EMS by solving a convex, unconstrained minimization problem, its practical relevance is limited. Most numerical minimizations methods rely on the gradient of the functional, but the proposed penalty term is not differentiable, making a regularization necessary. But any smooth regularization of the penalty term will stop the minimizers of the convex, constrained functional to coincide with those of the convex functional with penalty term. The stronger the regularization, the more the minimizers deviate. Furthermore, the regularization imposes numerical difficulties. If an explicit gradient descent is used for the minimization (as proposed in [9]), a suitable timestep size control is needed to ensure convergence. The step sizes allowed by such methods, e.g. the Armijo rule [14], typically correspond to the size of the region in which the linearization of the functional properly approximates the functional. Due to the nature of the penalty term p, the linearization at 0 and 1 of a regularized version of it only approximates the regularization properly in a region that is of the size of the regularization parameter. So, as soon as the current iterate of the gradient descent takes values near 0 or 1, the timestep control only allows timestep sizes of the order of the regularization parameter, which, as mentioned above, cannot be chosen too big. Instead of using a penalty term one could of course also approach the constrained convex optimization problem directly. This is done for example by Bresson et al. [15]. Their approach does not need a penalty term and gives an efficient algorithm to minimize ECE , but has to introduce an additional unknown v and a regularization parameter θ and needs to minimize for u and v alternatingly. Furthermore, the key idea to apply Chambolle’s TV minimization algorithm [16] can

Image Segmentation by Unconstrained Multiphase Thresholding

31

also be directly applied to our model to obtain a simpler and faster minimization algorithm: There is no need to introduce v, θ and the alternating minimization. Therefore it is worth to investigate whether it is possible to simplify the problem by getting rid of the constraint altogether.

3

Unconstrained Global 2-Phase Minimization

Another alternative to Chan Vese is a phase field approach [6, 5] with a typical double well term: 

 1 2 2 EPH, [u] := u (1 − u)2 + |∇u| dx. u2 f1 + (1 − u)2 f2 + ν Ω A minimizer u of this energy is a diffuse representation of the segmentation, i.e. {u = 0} and {u = 1} represent the two segments respectively with a smooth transition in between. EPH, [u] is known to Γ -converge to EMS [5], but unfortunately not convex and does not permit jumps in u for > 0. Knowing both ECE and EPH, , the question arises whether it is possible to combine the advantages of both models while eliminating some of the disadvantages. Heuristically looking at both energies served as motivation to investigate the following energy:  E[u] := u2 f1 + (1 − u)2 f2 + ν|∇u|dx. (5) Ω

This energy is convex because it does not involve the non-convex double well term of EPH, , and can be minimized without imposing constraints because it does not have the indicator term from ECE that is not bounded from below. Furthermore, it permits jumps in u. Remark 4. Given a function u, obviously we have E[min{max{0, u}, 1}] ≤ E[u]. Therefore, a minimizer umin fulfills 0 ≤ umin ≤ 1. While the proposed functional has some nice obvious properties, it is far from obvious whether there is a relation between its minimizer and minimizers of EMS . Before we tackle this question, let us remark a link between ECE and E: Remark 5. There is a direct relationship between ECE and E: A straightforward calculation shows u2 f1 + (1 − u)2 f2 = (f1 − f2 ) u + (u − 12 )2 (f1 + f2 ) − 14 (f1 + f2 ) + f2 . Therefore



(f1 − f2 ) u + (u − 12 )2 (f1 + f2 ) − 14 (f1 + f2 ) + f2 + ν|∇u|dx  =ECE [u] + (f1 + f2 )(u − 12 )2 dx + C.

E[u] =

Ω

Ω

In other words, E essentially equals ECE plus an additional quadratic penalty energy. The constant C is clearly irrelevant for the minimizers.

32

B. Berkels

To investigate the relation between the minimizers of E and minimizers of EMS we can make use of the theory derived in the context of the connection between minimal surface problems and total variation minimization. The following general statement has been made by Chambolle [17], Chambolle and Darbon [11], in the continuous setting, its discrete counterpart is well known: Theorem 2. Let Ψ : Ω × R → R, (x, s) → Ψ (x, s) such that Ψ (x, ·) is C 1 and uniformly convex for all x ∈ Ω and  u := argmin Ψ (x, u ˜(x)) + ν|∇˜ u|dx. u ˜

Ω

Then Σc := {x ∈ Ω|u(x) > c} for all c ∈ R is a minimizer of  ∂s Ψ (x, c)dx + ν Per(Σ). Σ

Note that this general statement cannot be directly applied to the model of Nikolova et al. discussed in Section 2 because the integrand is neither uniformly (not even strictly) convex nor does the general statement incorporate the constraint. As remarked in [11], the proof for a more specific statement given in [10] still applies to Theorem 2. Theorem 3. If u is a minimizer of (5), then {u > 12 } minimizes   EMS [Σ] = f1 dx + f2 dx + ν Per(Σ). Σ

Ω\Σ

Proof. Let Ψ (x, s) := s2 f1 (x)+(1−s)2 f2 (x). Obviously Ψ (x, ·) is C 2 for all x ∈ Ω and we have ∂s Ψ (x, s) = 2sf1 (x)+2(s−1)f2 (x) and ∂s2 Ψ (x, s) = 2(f1 (x)+f2 (x)). From Remark 2, we know that f1 , f2 > 0 a.e., therefore Ψ (x, ·) is uniformly convex for a.e. x ∈ Ω. Now just apply Theorem 2, noting ∂s Ψ (x, 12 ) = f1 (x) − f2 (x) and Remark 1.

In this sense, our theorem is a corollary of Theorem 2. The preceding theorem finally tells us how to find a global minimizer of EMS [·] given in (1): Minimize the convex energy (5) and threshold the minimizer to 1 2 . In case of the piecewise constant Mumford-Shah functional for 2-phase segmentation, we obtain a global minimizer of the Mumford-Shah energy (2) with respect to Σ for fixed gray values c1 , c2 . We do not necessarily find a global minimizer with respect to Σ, c1 and c2 . Another link between ECE and EPH, is the so-called piecewise constant levelset method [18] for 2-phase segmentation that constrains the levelset function to be piecewise constant. If this constraint is approximated with a penalty energy, the method equals the phase field approach. If the constraint is relaxed to a certain boundedness constraint, the method equals [9]. In both cases the fidelity term has to be altered accordingly, making use of the fact that this term is the same in ECE and EPH, if u only takes the values 0 and 1.

Image Segmentation by Unconstrained Multiphase Thresholding

33

Since (5) is similar to the Rudin-Osher-Fatemi energy [19], there is a wide variety of established minimization schemes to choose from, ranging from a straightforward gradient descent scheme with a differentiable approximation of the BV term over primal thresholding methods [20] to sophisticated methods based on the dual formulation of the BV norm, e.g. [16, 11]. 2 With Ψ (x, s) = 12 (s − (f2 (x) − f1 (x))) , another immediate consequence of Theorem 2 is that the zero superlevelset of a minimizer of the ROF energy  1 2 (u − (f2 − f1 )) + ν|∇u|dx EROF [u] := (6) 2 Ω ˆMS and therefore of EMS . This is another way to obtain is a global minimizer of E a global minimizer of EMS by unconstrained convex optimization, but compared to (5) this method has a few shortcomings, cf. Sections 4 and 5. Furthermore, the boundedness mentioned in Remark 4 does not hold for minimizers of the ROF energy. Perhaps this is one of the reasons why nobody seems to have used the classical ROF function for Mumford-Shah based image segmentation so far.

4

Multiphase Segmentation

Our functional can be extended to multiphase segmentation by the using the idea of Vese and Chan [2] in a straightforward manner. To keep notation at bay, we restrict the discussion to segmentation in 4 phases. The segmentation in 2n phases works analogously. Let f1 , f2 , f3 , f4 ∈ L1 (Ω) such that fi ≥ 0 a.e., then the multiphase functional is given by  E[u1 , u2 ] := u21 u22 f1 + (1 − u1 )2 u22 f2 Ω

+ u21 (1 − u2 )2 f3 + (1 − u1 )2 (1 − u2 )2 f4

(7)

+ ν (|∇u1 | + |∇u2 |) dx. If we fix u2 , the reduced functional E[·, u2 ] is the same as the 2-phase functional (5) with the indicator functions f˜1 = u22 f1 + (1 − u2 )2 f3 and f˜2 = u22 f2 + (1 − u2 )2 f4 . As in the 2-phase case, we can assume fi > 0 a.e. without loss of generality and because either u22 > 0 or (1 − u2 )2 holds, we have f˜1 , f˜2 > 0. Therefore, all statements proven for the 2-phase functional can be applied to E[·, u2 ], i.e. we can compute the global minimum (for fixed u2 ). The same applies for fixed u1 , so as an optimization strategy, we propose to minimize with respect to u1 and u2 alternatingly. Even though it is easy to extend (5) to multiphase segmentation, the same does not apply to the ROF energy (6). There is no apparent extension in the sense of [2] to formulate the multiphase segmentation in a single functional.

5

Indicator Parameters

In typical segmentation tasks, the indicator functions depend on unknown parameters, e.g. the grey values for each segment in case of the piecewise constant

34

B. Berkels

Mumford-Shah model. For the sake of simplicity, we discuss the latter model in its 2-phase formulation here, i.e. fi (x) := (u0 (x) − ci )2 , i = 1, 2, but this discussion applies to other indicator functions and multiphase segmentation as well. During the minimization of (5) we have to minimize for c1 and c2 as well. This is typically done in an alternating fashion, but there are two apparent possibilities to update the grey values: Minimize (5) with respect to c1 and c2 or do so for the energy in the set formulation (1). The two possible updating formulae for c1 two arising are     u2 u0 dx u2 dx or c1 = u0 dx dx. c1 = Ω

Ω

{u> 12 }

{u> 12 }

The two possibilities only coincide if u is binary. The first formula not only averages u0 in {u > 12 }, instead it takes into account the values of u0 everywhere, but weights the values according to u2 . To a certain degree this is similar to the effect of the regularization of the Heaviside function in the model of Chan and Vese. From our experiments, this reduces the chance of getting stuck in local minima that can still occur when minimizing over u and the indicator parameters. Particulary in the case of multiphase segmentation it turned out to be beneficial. Due to the different way f1 and f2 are used in the ROF energy (6), it is not quadratic in c1 and c2 . So this functional does not give a natural formula to update the grey values.

6

Numerical Examples

To conclude, we show the practical usability of the proposed model by applying it to the classical piecewise constant Mumford-Shah functional, see equation (2). As minimization method we use an explicit gradient descent scheme with the Armijo√rule [14] as timestep size control. The absolute value is regularized by |z| = z 2 + 2 (in all examples presented here, = 0.1 is used). For the spatial discretization, we use bilinear finite elements on a regular quadrilateral grid, i.e. each pixel of the input image u0 corresponds to a node of the finite element mesh. The grey values c1 and c2 are initialized with 0 and 1 respectively and updated occasionally during the gradient descent. Figure 1 shows results of our method and of the one proposed by Nikolova et al. [9] on one artificial image and one digital photo. In both examples, the minimizer u from our model is far from being binary, but this is nothing to be expected from the theory presented in this paper. The 0.5-superlevelset gives an accurate segmentation that is not influenced by the presence of heavy noise (top row) and works on non-binary input images (bottom row). The minimizers u of the Nikolova et al. model look very different, but the segmentation obtained from the 0.5-superlevelsets is almost identical. Upon closer inspection, the minimizer u of our model from the top row of Figure 1 looks very much like as obtained by minimizing the ROF energy with

Image Segmentation by Unconstrained Multiphase Thresholding

35

Fig. 1. Segmentation of an artificial noisy structure (ν = 2 · 10−3 , top row) and the well-known Matlab cameraman image (ν = 4·10−3 , bottom row): Input image u0 (left), segmentation function u and 0.5-superlevelset of u colored with the average grey values c1 , c2 obtained by our model (middle) and by using ECE (right). The slight difference of the grey values is attributed to the employed update formula, cf. Section 5.

Fig. 2. 4-phase segmentation of an artificial noisy image (top row) and a MRI image (bottom row) (ν = 6 · 10−4 ): Input image u0 (left), segmentation functions u1 and u2 (middle), segmentation colored with the average grey values c1 , ..., c4 (right)

u0 as input image. This is not surprising due to the following observation: If u0 is binary, i.e. u0 = χA for a set A ⊂ Ω and c1 = 0, c2 = 1 we have f1 = (χA − 0)2 = χA and f2 = (χA − 1)2 = χΩ\A and therefore  E[u] = Ω

(u − χΩ\A )2 + ν|∇u|dx,

36

B. Berkels

Fig. 3. Segmentation of a digital photo (ν = 2 · 10−5 ). Input image u0 (left), segmentation in four (middle) and eight (right) segments colored with the average grey values c bigmama / PIXELIO. of the segments. Original image 

Fig. 4. Intermediate results of the segmentation in eight segments shown in Figure 3 after 50 (left), 250 (middle) and 700 (right) gradient descent steps

i.e. E equals the ROF energy in this special case. This is not the case if u0 is non-binary which can be seen from the bottom row of Figure 1. Figure 2 shows 4-phase segmentation results. Those indicate the tendency of the segmentation functions to become binary for small values of ν. Finally, Figure 3 illustrates the behavior of the method for different numbers of segments and Figure 4 shows three timesteps of the 8-phase segmentation.

References 1. Mumford, D., Shah, J.: Optimal approximation by piecewise smooth functions and associated variational problems. Communications on Pure Applied Mathematics 42, 577–685 (1989) 2. Vese, L.A., Chan, T.F.: A multiphase level set framework for image segmentation using the Mumford and Shah model. International Journal of Computer Vision 50(3), 271–293 (2002) 3. Chan, T.F., Vese, L.A.: Active contours without edges. IEEE Transactions on Image Processing 10(2), 266–277 (2001) 4. Osher, S.J., Sethian, J.A.: Fronts propagating with curvature dependent speed: Algorithms based on Hamilton–Jacobi formulations. Journal of Computational Physics 79, 12–49 (1988) 5. Shen, J.: Γ -convergence approximation to piecewise constant Mumford-Shah segmentation. In: Blanc-Talon, J., Philips, W., Popescu, D.C., Scheunders, P. (eds.) ACIVS 2005. LNCS, vol. 3708, pp. 499–506. Springer, Heidelberg (2005)

Image Segmentation by Unconstrained Multiphase Thresholding

37

6. Esedo¯ glu, S., Tsai, Y.H.R.: Threshold dynamics for the piecewise constant Mumford-Shah functional. Journal of Computational Physics 211(1), 367–384 (2006) 7. Merriman, B., Bence, J.K., Osher, S.J.: Diffusion generated motion by mean curvature. CAM Report 92-18, UCLA (1992) 8. Alvino, C.V., Yezzi, A.J.: Fast Mumford-Shah segmentation using image scale space bases. In: Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, vol. 6498 (2007) 9. Nikolova, M., Esedo¯ glu, S., Chan, T.F.: Algorithms for finding global minimizers of image segmentation and denoising models. SIAM Journal on Applied Mathematics 66(5), 1632–1648 (2006) 10. Chambolle, A.: An algorithm for mean curvature motion. Interfaces and free Boundaries 6, 195–218 (2004) 11. Chambolle, A., Darbon, J.: On total variation minimization and surface evolution using parametric maximum flows. CAM Report 08-19, UCLA (2008) 12. Alter, F., Caselles, V., Chambolle, A.: A characterization of convex calibrable sets in RN . Mathematische Annalen 332(2), 329–366 (2005) 13. Ambrosio, L., Fusco, N., Pallara, D.: Functions of bounded variation and free discontinuity problems. Oxford Mathematical Monographs. Oxford University Press, New York (2000) 14. Kosmol, P.: Methoden zur numerischen Behandlung nichtlinearer Gleichungen und Optimierungsaufgaben, 2nd edn. Teubner, Stuttgart (1993) 15. Bresson, X., Esedo¯ glu, S., Vandergheynst, P., Thiran, J., Osher, S.: Fast global minimization of the active contour/snake model. Journal of Mathematical Imaging and Vision 28(2), 151–167 (2007) 16. Chambolle, A.: An algorithm for total variation minimization and applications. Journal of Mathematical Imaging and Vision 20(1-2), 89–97 (2004) 17. Chambolle, A.: Total variation minimization and a class of binary MRF models. In: Rangarajan, A., Vemuri, B.C., Yuille, A.L. (eds.) EMMCVPR 2005. LNCS, vol. 3757, pp. 136–152. Springer, Heidelberg (2005) 18. Lie, J., Lysaker, M., Tai, X.C.: A binary level set model and some applications to Mumford-Shah image segmentation. IEEE Transactions on Image Processing 15(5), 1171–1181 (2006) 19. Rudin, L., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Physica D 60, 259–268 (1992) 20. Daubechies, I., Defrise, M., de Mol, C.: An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Communications on Pure and Applied Mathematics 57(11), 1413–1457 (2004)

Extraction of the Intercellular Skeleton from 2D Images of Embryogenesis Using Eikonal Equation and Advective Subjective Surface Method Paul Bourgine1 , Peter Frolkovič2, Karol Mikula2 , Nadine Peyriéras3, and Mariana Remešíková2 1

2

CREA, Ecole Polytechnique-CNRS, 1 rue Descartes, 75005, Paris, France [email protected] Department of Mathematics, Slovak University of Technology, Radlinského 11, 81368 Bratislava [email protected], [email protected], [email protected] 3 CNRS-DEPSN, Avenue de la Terasse, 91198, Gif-sur-Yvette, France [email protected]

Abstract. We suggest an efficient method for automatic detection of the intercellular skeleton in microscope images of early embryogenesis. The method is based on the solution of two advective PDEs. First, we solve numerically the time relaxed eikonal equation in order to obtain the signed distance function to a given set – a set of points representing cell centers or a set of closed curves representing segmented inner borders of cells. The second step is a segmentation process driven by the advective version of subjective surface equation where the velocity field is given by the gradient of the computed distance function. The first equation is discretized by Rouy-Tourin scheme and we suggest a fixing strategy that significantly improves the speed of the computation. The second equation is solved using a classical upwind strategy. We present several test examples and we show a practical application - the intercellular skeleton extracted from a 2D image of a zebrafish embryo.

1

Introduction

The measure of the cell contact surface (intercellular skeleton) is an important quantitative characteristic of a living organism, especially during its embryonic development [6]. Together with other characteristics, e.g. the volume of the embryo, the global and local density of cells, the density of cell divisions etc., it provides an insight into the process of the evolution of the organism and allows to detect abnormalities or to compare individuals evolving in different conditions. The intercellular skeleton can be extracted from the miscroscope images of the evolving embryo. Fig. 1 shows an example of suitable image data. These images display significant cell structures (cell nuclei and cell membranes) of a zebrafish embryo at an early stage of its development and they were obtained by a two-photon confocal microscope. X.-C. Tai et al. (Eds.): SSVM 2009, LNCS 5567, pp. 38–49, 2009. c Springer-Verlag Berlin Heidelberg 2009 

Extraction of the Intercellular Skeleton

39

The main goal of our paper is to introduce an efficient and easily implementable method for detecting the intercellular skeleton. Our technique is based on numerical solution of a pair of advective partial differential equations. The first step is the solution of the time relaxed eikonal equation with a special Dirichlet type condition. The solution of such an equation is the distance function to a given set. This can be a set of points representing cell centers or a set of closed curves representing inner borders of cells obtained by segmentation. In case we deal with the curves, we construct the signed distance function with negative values in the interior part. We discretize the problem using the explicit Rouy-Tourin scheme and we suggest to extend the original scheme by a fixing technique. The idea of fixing is based on the fact that the Rouy-Tourin scheme applied to the time relaxed eikonal equation produces in every point monotonically increasing values approaching the value of the distance function. At some moment, the value will reach some steady state and the point can be excluded from the calculations. This strategy provides a significant improvement of the efficiency of the method and it brings a natural stopping criterion for the computation. We compare the performance of our algorithm with the computation of the exact distance function and we provide some examples of situations when the numerical solution can be obtained faster. The second step of our procedure is the segmentation using the advective version of the subjective surface equation. For each cell, we construct an initial segmentation function. Afterwards, all level sets of this function are evolving according to the velocity field given by the gradient of the computed signed distance function. By taking one of the level sets of the final form of the evolving function, we obtain the part of the required intercellular skeleton corresponding to one particular cell. The complete skeleton is constructed as the union of the results corresponding to individual cells. Using the distance function corresponding to cell centers, we get a Voronoi type cell skeleton that is already a good approximation of the real one as the cell formations are naturally similar to Voronoi tiling. A very realistic skeleton localization can be obtained if we consider the distance function to the segmented inner boundaries of the cells, assuming that we have a good quality cell segmentation. For pratical purposes, it is even sufficient to perform only a few time steps of the Rouy-Tourin scheme and use a rough estimate of the distance function in order to obtain a correctly oriented velocity field for the segmentation. This makes the method very efficient without loss of the quality of the resulting skeleton. The advective subjective surface equation is discretized by an explicit upwind approach. The paper is organized as follows. In Sec. 2, we describe the mathematical models for the two substeps of the procedure. Sec. 3 presents the numerical schemes and it explains the idea of the fixing algorithm intended to reduce the CPU time needed to compute the numerical solution of the eikonal euqation. In Sec. 4, we provide a series of numerical experiments as well as an example of a skeleton extracted from a 2D microscope image of a zebrafish embryo. Let us note that we present a method for solving two-dimensional problems, but the extension to three dimensions is rather straightforward.

40

P. Bourgine et al.

Fig. 1. 2D slices of 3D image of a zebrafish embryo. Left, the cell nuclei. Right, the cell membranes.

2

Mathematical Models

The first equation involved in our skeleton extraction strategy is the eikonal equation with time relaxation dt + |∇d| = 1

(1)

solved in the domain Ω × [0, TD ] where Ω is the image domain and coupled with a Dirichlet type condition d(x, t) = 0,

x ∈ Ω0 ⊂ Ω.

(2)

By the problem formulated in this way the solution d approaches, as time is evolving, the distance function to the set Ω0 . In our case, as we have already mentioned, Ω0 can be a set of points corresponding to approximate cell centers or a set of closed curves representing the segmented inner boundaries of the cells. The signed distance function can be constructed straightforwardly. The result of the cell shape segmentation is a level set function. Choosing one of the level sets to represent the inner boundary of the cell, we are able to recognize the inner and outer parts of the cell [1, 2] and assign the corresponding sign to the distance function. The distance function corresponding to a set of points is always positive. In the second step, we use the computed signed distance function in the advective part of the subjective surface model [4, 2] and we solve the equation ut + ∇g · ∇u = 0

(3)

where (x, t) ∈ Ω × [0, TS ] and g(x) = d (x, TD ) according to [8] or g(x) = −1/(1 + Kdp (x, TD )) with K > 0, p > 0 as in [4, 2]. The unknown function u is initialized by a piecewise constant profile localized around the approximate cell center. Then it is evolved by (3). The intercellular borders are represented by a p

Extraction of the Intercellular Skeleton

41

chosen level set of the function u(x, TS ). Due to the properties of the function d (see Fig. 3 and 6), the border lines of the neighboring cells correspond to the ridges of the distance function and are attached to each other and thus form the intercellular skeleton.

3 3.1

Numerical Schemes Time Relaxation Method with Fixing for Computing the Distance Function

In order to solve the equation (1) with the condition (2) numerically, we use an explicit time discretization with time step τD . Afterwards, the equation is discretized in space by the Rouy-Tourin scheme [3], cf. also [5,7]. As it is natural for image processing applications, the space grid elements correspond to the pixels of the image. Let us consider a rectangular space domain with dimensions Lx × Ly . The space grid is then uniform and consist of square elements Vij , i = 1 . . . nx , j = 1 . . . ny , nx = Lx /hD , ny = Ly /hD where hD is the length of the side of the pixel. For each volume Vij , let dij represent the approximate value of the solution d in the center of Vij in time step n. Let us define Mijpq , p, q ∈ {−1, 0, 1}, |p| + |q| = 1 as   2 Mijpq = min dni+p,j+q − dnij , 0 The Rouy-Tourin scheme for problem (1) then reads as follows      τD 1,0 0,−1 n+1 n + max Mij max Mij−1,0 , Mij , Mij0,1 dij = dij + τD − hD

(4)

This scheme is stable for τD ≤ hD /2 and we take advantage of the fact, that it produces monotonically increasing updates that are gradually approaching a steady state. This property allows us to implement (4) in a computationally efficient way. Let us consider the index set F n that contains the indices (i, j) of the volumes where the steady state has been already reached, i.e. there exists such n0 ∈ N , n0 ≤ n, that dnij0 = dnij0 −1 . The set F 0 is given as follows. At the beginning, we compute exact distances to the set Ω0 (which is a set of points corresponding to cell centers or a set of curves representing the inner boundaries of cells) in a local (one pixel) neighborhood. Then F 0 consists of the indices of all volumes with these exact values including the set Ω0 . Then the method is given by Algorithm 1. 3.2

Advective Subjective Surface Method for Detecting the Intercellular Skeleton

Now we discretize equation (3). Again, we consider explicit time discretization with time step τS and the space grid elements are indentified with the pixels of the image. The space discretization is based on the upwind principle. If the

42

P. Bourgine et al.

Algorithm 1. Fixing method for distance function • if (i, j) ∈ F n then continue • else   −1,0   0,−1  1,0 0,1 τD = dn max Mij , Mij , Mij + max Mij • dn+1 ij + τD − hD ij •

n+1 if dn+1 = dn = F n ∪ {(i, j)} ij then F ij

lentgh of the side of the pixel is denoted by hS and we define the central differy x ences Dij g = (gi+1,j − gi−1,j )/(2hS ), Dij g = (gi,j+1 − gi,j−1 )/(2hS ), we get the following approximation of (3)  x   x  τS  un+1 max Dij = unij − g, 0 (uij − ui−1,j ) + min Dij g, 0 (ui+1,j − uij ) ij hS   y    y g, 0 (uij − ui,j−1 ) + min Dij g, 0 (ui,j+1 − uij ) (5) + max Dij As the initial condition we take a shock-like profile localized around the cell center. Due to the properties of the signed distance function computed by the method described in Sec. 3.1, we can see that the advective velocity ∇g drives all level lines of the initial segmentation function to the ridges of the distance function, cf. Fig. 6. These ridges represent the intercellular skeleton.

4 4.1

Numerical Experiments Computation of the Signed Distance Function

Now let us present some computational results obtained by Algorithm 1. We inspected the experimental order of convergence of the suggested method, the CPU time needed for the computation and the effect of the fixing strategy. The results were also compared with the distance function computed analytically. First, let us make a note about the stopping criteria for the methods. If the fixing technique is not applied, the computation is stopped either when ||dn+1 − dn ||L1 (Ω) ≤ ε1 << 1 or when a prescribed number of time steps is performed. If we use the fixing strategy, we stop when (i, j) ∈ F n for all i = 1 . . . nx , j = 1 . . . ny , i.e. when all values are already fixed. In practice, the condition dn+1 = dnij is replaced by |dn+1 − dnij | ≤ ε2 << 1. If we want to ij ij compare the two methods, we first run the computation with fixing until all points are fixed and then the method with no fixing is prescribed to stop at exactly the same time. Now let us assume that the error of the numerical method in L2 (Ω)-norm is of the form E(h) = Chα , where h is the space discretization step. Obtaining experimentally E(h) and E(h/2) (see e.g. Table 1, column 4), we can express α = log2 (E(h)/(E(h/2)), which is called the experimental order or convergence (EOC). In the first experiment, we computed the distance function to seven given points situated in a square domain Ω = [−1, 1] × [−1, 1] and we measured the

Extraction of the Intercellular Skeleton

43

L2 (Ω) error with respect to the exact distance function and the EOC. The results and some details of the computations are displayed in Table 1, 2 and 3. Table 1 shows the comparison of the method with fixing with the original scheme without any fixing. The value in the seven points was set to 0, and the values in a one pixel neighborhood of the points were set to the values of the exact solution. In the stopping criterion for the fixing method, we set ε2 = 10−5 . Fig. 2 displays the distance function computed by the fizing method using nx = ny = 320. Comparing the results in the table, we can see the effect of the fixing strategy on the CPU time as well as on the L2 error and EOC. We can observe that in this case the computation is approximately two times faster and also we get smaller error when we fix the solution by our algorithm. The fact that the L2 error apparently depends on the value of ε2 led us to perform the experiment presented in Table 2. We were looking for the optimal choice of ε2 , i.e. the value that would provide the smallest L2 error. We can see that this value is different for different discretization parameters. Finally, in Table 3, we present the EOC for a slightly different implementation of the fixing method. In this case, the solution was initialized with the exact values not only in a one pixel neighborhood, but in a neighborhood whose size was independent of the space step. Again, we were looking for the optimal value of ε2 . We can observe that the EOC is approximately equal to 1. Table 1. Comparison of results of computational tests for the method without fixing and the method with fixing. Computation of the distance function to seven given points. nx 40 80 160 320 640

τD time steps L2 (Ω)-error CP U EOC L2 (Ω)-error fixing 0.025 52 3.447525e-2 0.01 3.440187e-2 0.0125 90 2.373985e-2 0.05 0.53825 2.359906e-2 0.00625 163 1.533921e-2 0.4 0.63009 1.508116e-2 0.003125 305 9.498908e-3 3.1 0.69139 9.016883e-3 0.0015625 583 5.704456e-3 23.18 0.73567 4.844598e-3

CP U fixing 0.0 0.03 0.21 1.73 13.08

EOC fixing 0.54376 0.64598 0.74205 0.89625

Table 2. Results of computational tests for the fixing method. Computation of the distance function to seven given points with determination of optimal ε2 . nx τD time steps 40 0.025 37 80 0.0125 73 160 0.00625 145 320 0.003125 287 640 0.0015625 569 1280 0.00078125 1132

L2 (Ω) - error 1.976552e-2 1.373675e-2 9.133363e-3 5.857149e-3 3.645805e-3 2.164545e-3

ε2 3.4e-3 1.3e-3 4.0e-4 1.3e-4 4.0e-5 1.1e-5

CP U 0.0 1.02 0.19 1.62 12.77 100.09

EOC 0.52494 0.58882 0.64095 0.68396 0.75217

The next experiment is similar. Instead of seven points, we computed the distance function to four polygons. Table 4 and Fig. 3 show the results.

44

P. Bourgine et al.

Table 3. Experimental order of convergence for the fixing method with exact values prescribed in a fixed neighborhood of seven given points and with determination of optimal ε2 nx τD time steps 40 0.025 35 80 0.0125 71 160 0.00625 140 320 0.003125 276 640 0.0015625 545

L2 (Ω) - error 2.648673e-2 1.352430e-2 6.928000e-3 3.598868e-3 1.795268e-3

ε2 6.2e-3 1.5e-3 3.5e-4 9.0e-5 2.2e-5

1.0

CP U 0.0 0.02 0.19 1.58 12.29

EOC 0.96972 0.96504 0.94490 1.00033

80

60 0.8

0.5

0.6

40

0.4 0.2 0.0 20

0.0 0

0.5 0 20 40 60

1.0 1.0

0.5

0.0

0.5

1.0

80

Fig. 2. Distance function to seven given points computed by the fast fixing method. Left, the contours of the function. Right, the 3D plot.

Summarizing the results presented in Table 1, 2 and 3, we can determine the experimental complexity of our algorithm. By careful checking of the CPU times needed for the computations, we can see that the complexity is not higher than O(N 3/2 ), where N is the number of unknowns. The next two tests were realized in order to compare the fixing method with computation of the exact distance function. Again, we used ε2 = 10−5 for the fixing mehotd. In the first case, a certain number of points was randomly generated in a given 2D domain. After, the distance function to this set of points was computed both numerically by our fixing algorithm and analytically by finding the nearest point to every pixel. The CPU time was measured for different numbers of points and plotted in graphs presented in Fig. 4. We can observe that the computational cost of the analytical computation is increasing with increasing number of points while the cost of the numerical computation is decreasing. For a certain number of points, depending on the size of the image domain, the numerical method becomes more efficient. We can see that in all cases displayed in the figure, this number is at most 1% of the number of image pixels so it is practically meaningful. Another experiment was performed in order to test the numerical method on data qualitatively similar to segmented cell structures. The result of the cell segmentation is a level set function and the inner borders of the cells are

Extraction of the Intercellular Skeleton

45

Table 4. Comparison of computational tests for the method without fixing and the method with fixing. Computation of the distance function to four polygons. nx 40 80 160 320 640

τD time steps L2 (Ω) - error CP U EOC L2 (Ω) - error fixing 0.025 36 1.122800e-2 0.01 1.121228e-2 0.0125 61 8.266972e-3 0.03 0.44167 8.231755e-3 0.00625 108 5.606750e-3 0.26 0.56020 5.539007e-3 0.003125 203 3.599271e-3 2.04 0.63946 3.475747e-3 0.0015625 390 2.228162e-3 15.46 0.69185 2.041495e-3

1.0

CP U EOC fixing fixing 0.01 0.01 0.44581 0.11 0.57157 0.96 0.67230 7.24 0.76770

80

60

0.4

0.5

40

0.2 0.0 0.2

20

0.0 0

0.5 0 20 40 60

1.0 1.0

0.5

0.0

0.5

1.0

80

Fig. 3. Distance function to four polygons computed by the RMF. Left, the contours of the function with the shapes indicated. Right, the 3D plot.

represented by a chosen isoline of this function. For our test purposes, we used images with isolines in form of either randomly placed circles or randomly placed rectangles of a random size (Fig. 5). The maximum image intensity (255) was in the centers of these shapes and then it gradually decreased to 0 with increasing distance from the center. In order to compute the distance function to a certain level set numerically, the location of the level set was detected and the solution was initialized by exact values in its one pixel neighborhood. After, the fixing method was applied. For the exact computation, we used a procedure that constructs set of points corresponding to the chosen isoline, finding its crossections with the pixel structure. Then, the nearest of these points was found for each pixel. The CPU time for such a procedure is documented in Table 5, column 5. By construction, the points of the isoline are sorted by their coordinates and therefore the computation of the exact distance function can be optimized – we do not have to go through the whole list of points but the nearest point to a pixel can be always found in a certain neighborhood of the nearest point to the previous pixel. The CPU times for this optimized approach are listed in Table 5, column 6. According to Table 5, we can observe that in all cases considered here, the numerical solution was faster than the optimized analytical computation. We considered isoline for value I = 128, hD = 1.0, τD = 0.5, ε2 = 10−5 and the dimensions of the image domain were 512 × 512 pixels.

46

P. Bourgine et al.

2.5

10

2.0

8

1.5

6

1.0

4

0.5

2

35 30 25 20 15 10 5 0

0

0

1000

2000

3000

4000

5000

6000

0

1000

2000

3000

4000

5000

6000

0

1000

2000

3000

4000

5000

6000

Fig. 4. Comparison of the fixing method (solid line) with computation of exact solution (dashed line)–plot of CPU time depending on the number of randomly distributed points. Left, image domain with 256 × 256 pixels. Middle, 512 × 512 pixels. Right, 1024 × 1024 pixels.

Fig. 5. Isolines (I = 128) of test level set functions simulating cell structures Table 5. Comparison of the numerical and analytical computation of the distance function to the cell-like structures type of level set time steps L2 (Ω) - rel. error CPU num. CPU exact CPU exact opt. 256 rectangles 52 3.509519e-2 0.91 35.56 2.87 1024 rectangles 33 5.813329e-2 0.63 75.82 2.88 256 circles 54 2.441320e-2 0.91 34.55 3.00 1024 circles 34 4.153412e-2 0.64 68.41 3.07

In the last experiment, we computed numerically the distance function to a set of segmented cells of a zebrafish embryo. In Fig. 6, we show the computed distance function as well as the vector field given by the gradient of this function. We used hD = 1.0, τD = 0.5 and ε2 = 10−5 . 4.2

Extraction of the Intercellular Skeleton

In the first experiment, we computed the skeleton using the distance function corresponding to approximate cell centers. The result is a Voronoi type skeleton.

Extraction of the Intercellular Skeleton

47

1.0

0.5

0.0

0.5

1.0 1.0

0.5

0.0

0.5

1.0

40 10

30

20 30

20

40 5

10

0 5

Fig. 6. Signed distance function to segmented cells. Top left, segmentation of inner boundaries of cells. Top right, contour plot of the signed distance function. Bottom left, detail of the 3D plot of this distance function. Bottom right, detail of the vector field given by ∇g, g(x) = d(x), with recognizable position of the intercellular skeleton.

As we can see in Fig. 7, this skeleton represents a very good approximation of the cell structure in the sense that it can provide reliable estimate of some quantities, like the area of the cell contact surface. It can be used in the cases when no or bad quality membrane images are provided and therefore the correct membrane segmentation is not possible. If we have a good quality segmentation of the inner boundaries of the cells, we can detect the intercellular skeleton more precisely using the signed distance function to the segmented objects. An example is shown in Fig. 8. In both experiments, we set hD = 1.0, τD = 0.5, ε2 = 10−5 , hS = 1.0, τS = 0.1. The automatic segmentation of inner cell borders displayed in Fig. 8 was obtained by the generalized subjective surface method [2] applied to a microscope image of zebrafish cell membranes, similar to Fig. 1.

48

P. Bourgine et al.

Fig. 7. Voronoi type skeleton. On the left, the approximate cell centers. On the right, the corresponding Voronoi type skeleton.

Fig. 8. Real skeleton detection. On the left, the cell segmentation. On the right, the corresponding intercellular skeleton.

1.0

0.5

0.0

0.5

1.0 1.0

0.5

0.0

0.5

1.0

Fig. 9. Left, the result after 10 steps of computation of the distance function. Right, the skeleton obtained using the corresponding vector field.

Extraction of the Intercellular Skeleton

49

Remark 1. In practice, the main determining factor of the quality of the detected skeleton is the correct orientation of the vector field generated by the gradient of the distance function. As it follows from the character of the problem and also from the numerical procedure, the correct orientation of the vector field can be already obtained as soon as the values in all image pixels are nontrivially updated by the numerical scheme (4), i.e. before they are definitely fixed. This allows a significant reduction of the computational time without loss of the quality of the final result. In Fig. 9 left, the result obtained after 10 steps of the numerical computation of the distance function is shown. At this moment, the values were nontrivially updated in a sufficiently large neighborhood of the segmented cells. We can construct the vector field for equation (3) and find the corresponding intercellular skeleton (Fig. 9 right). Comparing Fig. 8 and 9 we can see that the skeleton found in this way is of the same quality as in the case of complete fixing of the values in the whole domain. Let us note that the complete fixing would require 123 time steps.

Acknowledgments The work was supported by the European projects Embryomics and BioEmergences, the grants APVV-RPEU-0004-06, APVV-0351-07, APVV-LPP-0020-07 and the grant of VEGA 1/0269/09.

References 1. Frolkovič, P., Mikula, K., Peyriéras, N., Sarti, A.: A counting number of cells and cell segmentation using advection-diffusion equations. Kybernetika 43(6), 817–829 (2007) 2. Mikula, K., Peyriéras, N., Remešíková, M., Sarti, A.: 3D embryogenesis image segmentation by the generalized subjective surface method using the finite volume technique. In: Proceedings of FVCA5 – 5th International Symposium on Finite Volumes for Complex Applications. Hermes Publ., Paris (2008) 3. Rouy, E., Tourin, A.: Viscosity solutions approach to shape-from-shading. SIAM Journal on Numerical Analysis 29(3), 867–884 (1992) 4. Sarti, A., Malladi, R., Sethian, J.A.: Subjective Surfaces: A Method for Completing Missing Boundaries. Proc. Nat. Acad. Sci. 12(97), 6258–6263 (2000) 5. Sethian, J.A.: Level Set Methods and Fast Marching Methods: Evolving Interfaces in Computational Geometry. In: Fluid Mechanics, Computer Vision, and Material Science. Cambridge University Press, New York (1999) 6. Tassy, O., Daian, F., Hudson, C., Bertrandt, V., Lemaire, P.: A quantitative approach to the study of the cell shapes and interactions during early chordate embryogenesis. Currrent Biology 16, 345–358 (2006) 7. Zhao, H.-K.: Fast sweeping method for eikonal equations. Mathematics of Computation 74, 603–627 (2005) 8. Zhao, H.-K., Osher, S., Fedkiw, R.: Fast surface reconstruction using the level set method. In: Proc. IEEE workshop on variational and level set methods - VLSM 2001, Vancouer, pp. 194–201 (2001)

On Level-Set Type Methods for Recovering Piecewise Constant Solutions of Ill-Posed Problems Adriano DeCezaro1, Antonio Leitão2 , and Xue-Cheng Tai3,4 1

Institute of Mathematics Statistics and Physics, Federal University of Rio Grande, Av. Italia km 8, 96201-900 Rio Grande, Brazil [email protected] 2 Department of Mathematics, Federal University of St. Catarina, P.O. Box 476, 88040-900 Florianópolis, Brazil [email protected] 3 Division of Mathematical Science, School of Physical and Mathematical Sciences, Nanyang Technological University, Singapore [email protected] 4 Department of Mathematics, University of Bergen, Johannes Brunsgate 12, N-5008 Bergen, Norway

Abstract. We propose a regularization method for solving ill-posed problems, under the assumption that the solutions are piecewise constant functions with unknown level sets and unknown level values. A level set framework is established for the inverse problem and a Tikhonov regularization approach is proposed. Existence of generalized minimizers for the Tikhonov functional is proven. Moreover, we establish convergence and stability results, characterizing our Tikhonov approach as a regularization method. Based on the necessary conditions of optimality for the Tikhonov functional, a level-set type method is derived and implemented numerically for solving an inverse source problem. This allow us to test the quality of the proposed algorithm.

1

Introduction

Several inverse problems of interest consist of identifying an unknown physical quantity u ∈ X, that can be represented by a piecewise constant function, over a bounded given domain Ω, from the set of data y ∈ Y , where X, Y are Hilbert spaces. This process being described by the model F (u) = y ,

(1)

where F : D(F ) ⊂ X → Y and the set of data is obtained by indirect measurements of the parameter. Because of this, in practical applications the exact data y ∈ Y is, in general, not known. Given is only approximate measured data y δ ∈ Y , corrupted by noise of level δ > 0 and satisfying y δ − yY ≤ δ . X.-C. Tai et al. (Eds.): SSVM 2009, LNCS 5567, pp. 50–62, 2009. c Springer-Verlag Berlin Heidelberg 2009 

(2)

On Level-Set Type Methods

51

In the case where the unknown function u is a piecewise constant function distinguishing between two given values (i.e., up to an affine transformation, u is a characteristic function), level set approaches were considered in [1, 2, 3, 4, 5, 6]. In this case, since the level values of u are known, one needs only to identify the level sets of u, i.e. the inverse problem reduces to a shape identification problem. In the case where the unknown function u is a piecewise constant function distinguishing between several given values, multiple level set approaches were considered in [6,7,8]. If the level values of u are also unknown the inverse problem becomes harder, since one has to identify both the level sets as well as the level values of the unknown parameter u. In this case, the dimension of the parameter space increases by the number of unknown level values. Our starting point in this article is the assumption that the parameter u in (1) is a piecewise constant function assuming two unknown values, i.e. u(x) ∈ {c1 , c2 } a.e. in Ω ⊂ Rd , d = 2, 3. In this case one can assume the existence of an open mensurable set D ⊂⊂ Ω s.t. u(x) = c1 , x ∈ D and u(x) = c2 x ∈ Ω/D. We propose a level set approach to represent the unknown parameter u. First we introduce the H 1 -function φ, which act as a regularization on the parameter space. Then, using the Heaviside projector H : H 1 (Ω) → L∞ (Ω), a solution of (1) can be represented in the form u = c1 H(φ) + c2 (1 − H(φ)) =: P (φ, c1 , c2 ) .

(3)

With this notation we have D = {x ∈ Ω ; φ(x) > 0} and Ω/D = {x ∈ Ω ; φ(x) < 0}. The level values c1 , c2 ∈ R are unknown and have to be determined as well. As already observed in [3], the Heaviside operator H maps H 1 (Ω) into the set V := {χD ; D ⊂ Ω measurable, Hn−1 (∂D) < ∞}, where Hn−1 (S) denotes the (n-1)-dimensional Hausdorff-measure of the set S. Therefore, the operator P in (3) maps H 1 (Ω) × R2 into the admissible parameter set U := {u = q(v, c1 , c2 ); v ∈ V and c1 , c2 ∈ R}, where q : V × R2  (v, c1 , c2 ) → c1 v + c2 (1 − v) ∈ L∞ (Ω). Using the level set framework introduced above, the inverse problem in (1), with data given as in (2), can be written in the form of the operator equation F (P (φ, c1 , c2 )) = y δ .

(4)

Once an approximate solution (φ, c1 , c2 ) of (4) is obtained, a corresponding solution of (1) can be computed using equation (3). In this article, approximate solutions to (4) are obtained by minimizing the Tikhonov functional  Gα (φ, c1 , c2 ) := F (P (φ, c1 , c2 )) − y δ 2Y + α β1 |H(φ)|BV + β2 φ − φ0 2H 1  2 + β3 j=1 |cj − cj0 |2 , (5) based on T V -H 1 penalization. Here φ0 and cj0 are known reference parameters. This Tikhonov functional extends the ones proposed in [5, 6, 9] (based on T V penalization) and [3, 8] (based on T V -H 1 penalization). To motivate the regularization terms in (5), notice that they effect: i) the boundedness of the level lines of φ as well as it’s H 1 -norm; ii) the boundedness of cj . These two facts allow us to guarantee existence of (generalized) minimizers of Gα in L∞ ∩ BV.

52

A. DeCezaro, A. Leitão, and X.-C. Tai

This article is outlined as follows: In Section 2 we introduce the concept of generalized minimizers for the functional Gα in (5). In Section 3 we derive a convergence analysis for this Tikhonov approach. In Section 4 we introduce stabilized functionals and prove that the corresponding minimizers approximate a minimizer of Gα . Section 5 is devoted to numerical experiments. A level set type method is implemented for solving a two-dimensional inverse potential problem.

2

The Concept of Generalized Minimizers

We shall consider the model problem described as in the introduction under the following general assumptions: (A1) Ω ⊆ Rn is bounded, connected, with piecewise C 1 boundary ∂Ω. (A2) The operator F : D(F ) ⊂ L1 (Ω) → Y is continuous and Fréchetdifferentiable on D(F ) with respect to the L1 (Ω)-topology. (A3) ε, α and βj , j = 1, 2, 3 denote positive parameters. (A4) Equation (1) has a solution, i.e. there exists u ∈ U satisfying F (u) = y and a function φ ∈ H 1 (Ω) satisfying |∇φ| = 0, in a neighborhood of {φ = 0} such that H(φ) = z, for some z ∈ V. Moreover, there exist constants values c1 , c2 ∈ R such that q(z, c1 , c2 ) = u. For each ε > 0, we define the operator Pε (φ, c1 , c2 ) := c1 Hε (φ) + c2 (1 − Hε (φ)) ,

(6)

where Hε is the continuous approximation to H given by:  1 + t/ε for t ∈ [−ε, 0] Hε (t) := . H(t) for t ∈ R/ [−ε, 0] In order to guarantee existence of a minimizer of Gα in (5), we adapt to the level-set framework described above, the concept of generalized minimizers formulated in [3]. Definition 1. Let the operators H, P , Hε and Pε be defined as above. a) A vector (z, φ, c1 , c2 ) ∈ L∞ (Ω) × H 1 (Ω) × R2 is called admissible when there exists a sequence {φk } of H 1 (Ω)-functions satisfying limk φk − φL2 = 0, and also there exists a sequence {εk } ∈ R+ converging to zero such that limk Hεk (φk ) − zL1 = 0. b) A minimizer of Gα is considered to be any admissible vector (z, φ, c1 , c2 ) minimizing 2  Gα (z, φ, c1 , c2 ) := F (q(z, c1 , c2 )) − y δ  Y + αR(z, φ, c1 , c2 ) (7) over the set of admissible vectors, where

 R(z, φ, c1 , c2 ) = ρ(z, φ) + β3 2j=1 |cj − cj0 |2 ,    ρ(z, φ) := inf lim inf β1 |Hεk (φk )|BV + β2 φk − φ0 2H 1 . k→∞

(8) (9)

The infimum in (9) is taken over all sequences {εk } and {φk } characterizing (z, φ, c1 , c2 ) as an admissible vector.

On Level-Set Type Methods

53

c) A generalized minimizer of Gα (φ, c1 , c2 ) is an admissible vector (z, φ, c1 , c2 ) minimizing the functional Gα in (7) on the set of admissible vectors. 2.1

Relevant Properties of Admissible Vectors

First we verify some basic properties of the operators Pε , Hε and q that will be necessary in the subsequent analysis. Lemma 1. Let Ω be given as above and j = 1, 2. (i) Let {zk }k∈N be a bounded sequence in L∞ (Ω) converging to some element z in L1 (Ω) and {cjk }k∈N be a sequence of real numbers converging to cj . Then q(zk , c1k , c2k ) converges to q(z, c1 , c2 ) in L1 (Ω). (ii) Let (z, φ) ∈ L1 (Ω) × H 1 (Ω), be such that Hε (φ) → z in L1 (Ω) as ε → 0 and let c1 , c2 ∈ R. Then Pε (φ, c1 , c2 ) → q(z, c1 , c2 ) in L1 (Ω) as ε → 0. (iii) Given ε > 0, let {φk }k∈N be a sequence in H 1 (Ω) converging to φ ∈ H 1 (Ω) in the L2 -norm. Then Hε (φk ) → Hε (φ) in L1 (Ω), as k → ∞. Moreover, if {cjk }k∈N are sequences of real numbers converging to some cj , then q(Hε (φk ), c1k , c2k ) → q(Hε (φ), c1 , c2 ) in L1 (Ω), as k → ∞. Proof. Since Ω is assumed to be bounded, we have L∞ (Ω) ⊂ L1 (Ω). To prove (i), notice that 1 2 1 2 q(zk , ck , ck )−q(z, c , c )L1 = |c1k zk + c2k (1 − zk ) − c1 z − c2 (1 − z)| dx Ω

1  (|c | + |c2 |)|zk − z| + |c2k − c2 | dx ≤ |zk | (|c1k − c1 | + |c2k − c2 |)dx + Ω

Ω

≤|Ω| zk  L∞ (|c1k − c1 | + |c2k − c2 |) + (|c1 | + |c2 |)zk − zL1 + |Ω| |c2k − c2 | , which converges to zero as k → ∞. Assertion (ii) follows with similar arguments. The first part of assertion (iii) is a direct consequence of the inequality Hε (φk )− Hε (φ)L1 (Ω) ≤ ε−1 meas(Ω)φk − φL2 (Ω) . The second part of assertion (iii) follows by a combination of the inequality above and assertion (i).  Lemma 2. Let (zk , φk , c1k , c2k ) be a sequence of admissible vectors converging in L1 (Ω)× L2 (Ω)× R2 to some (z, φ, c1 , c2 ). Then (z, φ, c1 , c2 ) is also an admissible vector. Sketch of the proof. In order to prove that (z, φ, c1 , c2 ) is also an admissible vector, one uses an argument of extraction of diagonal subsequences, analogously as in [8, Lemma 2].  2.2

Relevant Properties of the Penalization Functional

In the next lemmas we verify two properties of the functional R which are fundamental for the convergence analysis in Section 3. Lemma 3. The functional R in (8) is coercive on the set of admissible vectors.

54

A. DeCezaro, A. Leitão, and X.-C. Tai

Sketch of the proof. Let (z, φ, c1 , c2 ) be an admissible vector. From [8, Lemma 4] follows

 ρ(z, φ) ≥ β1 |z|BV + β2 φ − φ0 2H 1 . (10) Now, from (10) and the definition of R in (8) follows β1 |z|BV + β2 φ − φ0 2H 1 + β3

2

|cj − cj0 |2 ≤ 2 ρ(z, φ) + β3 j=1 |cj − cj0 |2 = R(z, φ, c1 , c2 ) , j=1



concluding the proof.

Lemma 4. The functional R in (8) is weak lower semi-continuous on the set of admissible vectors, i.e. given a sequence {(zk , φk , c1k , c2k )} of admissible vectors such that zk → z in L1 (Ω), φk  φ in H 1 (Ω), cjk → cj in R, for some admissible vector (z, φ, c1 , c2 ), then it follows R(z, φ, c1 , c2 ) ≤ lim inf R(zk , φk , c1k , c2k ) . k∈N

Proof. The functional ρ(z, φ) is weak lower semi-continuous cf. [8, Lemma 5]. Moreover, the Euclidean norm in R2 is also lower semi-continuous. The lemma follows from the fact that the functional R in (8) is a linear combination of lower semi-continuous functionals. 

3

Convergence Analysis

First we prove that for any positive parameters α, β the functional Gα in (5) is well posed. Theorem 1 (Well-Posedness). The functional Gα in (5) attains minimizers on the set of admissible vectors. Proof. Notice that the set of admissible vectors is not empty, since (0, 0, 0, 0) is admissible. Let {(zk , φk , c1k , c2k )} be a minimizing sequence for Gα , i.e. a sequence of admissible vectors satisfying Gα (zk , φk , c1k , c2k ) → inf Gα ≤ Gα (0, 0, 0, 0) < ∞. Then, {Gα (zk , φk , c1k , c2k )} is a bounded sequence of real numbers. Therefore, {(zk , φk , c1k , c2 , k)} is uniformly bounded in BV × H 1 (Ω) × R2 . Thus, the Sobolev compact embedding theorem [10] and the Bolzano-Weierstrass theorem guarantees the existence of a subsequence (denoted again by {(zk , φk , c1k , c2k )}) and the existence of (z, φ, c1 , c2 ) ∈ L1 (Ω) × H 1 (Ω) × R2 such that φk → φ in L2 (Ω), φk  φ in H 1 (Ω), zk → z in L1 (Ω) and cjk → cj in R. From Lemma 2 we conclude that (z, φ, c1 , c2 ) is an admissible vector. Moreover, from the weak lower semi-continuity of R together with the continuity of F and q we obtain   lim Gα (zk , φk , c1k , c2k ) = lim F (q(zk , c1k , c2k )) − y δ 2Y + αR(zk , φk , c1k , c2k ) k→∞

k→∞

≥ F (q(z, c1 , c2 )) − y δ 2Y + αR(z, φ, c1 , c2 ) = Gα (z, φ, c1 , c2 ) , (11) proving that (z, φ, c1 , c2 ) minimizes Gα .



On Level-Set Type Methods

55

In the next theorems we present the main convergence and stability results. The proofs use classical techniques from the analysis of Tikhonov type regularization methods (see, e.g., [11, 12]). Theorem 2 (Convergence for exact data). Assume that we have exact data, i.e. y δ = y and βj > 0 , j = 1, 2, 3. For every α > 0 let (zα , φα , c1α , c2α ) denote a minimizer of Gα on the set of admissible vectors. Then, for every sequence of positive numbers {αk }k∈N converging to zero there exists a subsequence, denoted again by {αk }l∈N , such that (zαk , φαk , c1αk , c2αk ) is strongly convergent in L1 (Ω)× L2 (Ω) × R2 . Moreover, the limit is a solution of (1). Proof. Let (z † , φ† , c1,† , c2,† ) be a solution of (1) – its existence is guaranteed by assumption (A4). Let {αk }k∈N be a sequence of positive numbers converging to zero. For each k ∈ N, let (zk , φk , c1k , c2k ) := (zαk , φαk , c1αk , c2αk ) be a minimizer of Gαk . Then, for each k ∈ N we have  2 Gαk (zk , φk , c1k , c2k ) ≤ F (q(z † , c1,† , c2,† )) − y Y + αk R(z † , φ† , c1,† , c2,† )

= αk R(z † , φ† , c1,† , c2,† ) . (12)

Since αk R(zk , , φk , c1k , c2k ) ≤ Gαk (zk , φk , c1k , c2k ), it follows from (12) that R(zk , φk , c1k , c2k ) ≤ R(z † , φ† , c1,† , c2,† ) < ∞ .

(13)

Moreover, from the assumption on the sequence {αk }, it follows that lim αk R(z † , φ† , c1,† , c1,† ) = 0 .

k→∞

(14)

From (13) and Lemma 3 we conclude that the sequences {φk }, {zk } and {cjk }j=1,2 are bounded in H 1 (Ω) , BV and R2 respectively. Using an argument of extraction of diagonal subsequences (see proof of Lemma 2) we can guarantee the existence ˜ c˜1 , c˜2 ) such that of an admissible vector (˜ z , φ, ˜ c˜1 , c˜2 ) in L1 (Ω) × L2 (Ω) × R2 . (zk , φk , c1k , c2k ) → (˜ z , φ, From Lemma 1 (i) follows that q(˜ z , c˜1 , c˜2 ) = lim q(zk , c1k , c2k ) on L1 (Ω). Using k→∞

the continuity of the operator F together with (12) and (14) we conclude that y = lim F (q(zk , c1k , c2k )) = F (q(˜ z , c˜1 , c˜2 )) . k→∞

On the other hand, from the lower semi-continuity of R and (13) it follows that ˜ c˜1 , c˜2 ) ≤ lim inf R(zk , φk , c1 , c2 ) R(˜ z , φ, k k k→∞

≤ lim sup R(zk , φk , c1k , c2k )) ≤ R(z †, φ† , c˜1 , c˜2 ) , k→∞

concluding the proof.



56

A. DeCezaro, A. Leitão, and X.-C. Tai

Theorem 3 (Convergence for noisy data). Let α = α(δ) be a function satisfying lim α(δ) = 0 and lim δ 2 α(δ)−1 = 0. Moreover, let {δk }k∈N be a seδ→0

δ→0

quence of positive numbers converging to zero and y δk ∈ Y be corresponding noisy data satisfying (2). Then, there exist a subsequence, denoted again by {δk }, and a sequence {αk := α(δk )} such that (zαk , φαk , c1αk , c2αk ) converges in L1 (Ω) × L2 (Ω) × R2 to solution of (1). Proof. Let (z † , φ† , c1,† , c1,† ) be a solution of (1).1 For each k ∈ N, denote by (zk , φk , c1k , c2k ) := (zα(δk ) , φα(δk ) , c1α(δk ) , c2α(δk ) ) a minimizer of Gα(δk ) . Then, for each k ∈ N we have 2  Gαk (zk , φk , c1k , c2k ) ≤ F (q(z † , c1,† , c1,† )) − y δk Y + α(δk )R(z † , φ† , c1,† , c2,† ) ≤ δk2 + α(δk )R(z † , φ† , c1,† , c2,† ) .

(15)

Taking the limit k → ∞ in (15), it follows from the theorem assumptions  2 that lim F (q(zk , c1k , c2k )) − y δk  ≤ lim Gαk (zk , φk , c1k , c2k ) = 0. Therefore, k→∞

k→∞

lim F (q(zk , c1k , c2k )) = y. Moreover, from (15) and the definition of Gαk , it fol-

k→∞

lows that R(zk , φk , c1k , c2k ) ≤ δk2 α(δk )−1 + R(z † , φ† , c1,† , c2,† ). Thus, from the assumptions on the function α(δk ), we conclude that lim sup R(zk , φk , c1k , c2k ) ≤ k→∞

R(z † , φ† , c1,† , c2,† ). The proof follows arguing as in the proof of Lemma 2.

4



Numerical Solution

In the sequel we introduce a functional which can be handled numerically, and whose minimizers are ’close’ to the minimizers of Gα . Let Gε,α be the stabilized functional defined by  Gε,α (φ, c1 , c2 ) := F (Pε (φ, c1 , c2 )) − y δ 2Y + α β1 |Hε (φ)|BV +  2 + β2 φ − φ0 2H 1 + β3 j=1 |cj − cj0 |2 , (16) where Pε (φ, c1 , c2 ) := q(Hε (φ), c1 , c2 ) is the functional defined in (6). The functional Gε,α is well-posed as the following lemma shows: Lemma 5. Given positive constants α, ε, βj , j = 1, 2, 3 as above, a function φ0 ∈ H 1 (Ω) and cj0 ∈ R, j = 1, 2, the functional Gε,α in (16) attains a minimizer on H 1 (Ω) × R2 . Proof. Since inf{Gε,α (φ, c1 , c2 ) : (φ, c1 , c2 ) ∈ H 1 (Ω) × R2 } ≤ Gε,α (0, 0, 0) < ∞, there exists a minimizing sequence {(φk , c1k , c2k )} in H 1 (Ω) × R2 satisfying lim Gε,α (φk , c1k , c2k ) = inf{Gε,α (φ, c1 , c2 ) : (φ, c1 , c2 ) ∈ H 1 (Ω) × R2 } .

k→∞ 1

The existence of solutions is guaranteed by (A4).

On Level-Set Type Methods

57

Then, for fixed α > 0, the sequences {φk } and {cjk }j=1,2 are bounded in H 1 (Ω) and R2 respectively. Therefore, φk  φ in H 1 (Ω) and cjk → cj in R, j = 1, 2. Moreover, by the weak lower semi-continuity of the H 1 –norm and the continuity of the Euclidean norm in R, it follows that φ − φ0 2H 1 ≤ lim inf φk − φ0 2H 1 , k→∞

and |cj − cj0 | ≤ lim inf |cjk − cj0 |. k→∞

From the Sobolev compact embedding theorem [13] we have φk → φ in L2 (Ω). Therefore, Lemma 1 implies √ Hε (φjk ) − Hε (φj )L1 ≤ ε−1 meas(Ω)φk − φL2 → 0,

Pε (φk , c1k , c2k ) − Pε (φ, c1 , c2 )L1 = q(Hε (φk ), c1k , c2k ) − q(Hε (φ), c1 , c2 )L1 → 0. Thus, it follows from [10, Theorem 1, pg 172] that |Hε (φ)|BV ≤ lim inf |Hε (φk )|BV . k→∞

Now, from the continuity of F and q, together with the estimates above we obtain  Gε,α (φ, c1 , c2 ) ≤ lim F (Pε (φk , c1k , c2k )) − y δ 2Y + α β1 lim inf |Hε (φk )|BV + k→∞ k→∞  2 2 + β2 lim inf φk − φ0 H 1 + β3 lim inf j=1 |cjk − cj0 |2 k→∞

≤ lim inf k→∞

concluding the proof.

Gε,α (φk , c1k , c2k )

k→∞

= inf Gε,α , 

In the sequel we prove that, when ε → 0, the minimizers of Gε,α approximate a minimizer of the functional Gα . Theorem 4. Let α and βj be given as above. For each ε > 0, denote by (φε,α , c1ε,α , c2ε,α ) a minimizer of Gε,α . There exists a sequence of positive numbers εk → 0 such that (Hεk (φεk ,α ), φεk ,α , c1εk ,α , c2εk ,α ) converges strongly in L1 (Ω)×L2 (Ω)× R2 and the limit minimizes Gα on the set of admissible vectors. Proof. The functional Gα attains a generalized minimizer (zα , φα , c1α , c2α ) on the set of admissible vectors (cf. Theorem 1). From Definition 1, there exists a sequence {εk } of positive numbers converging to zero and corresponding sequences {φk } in H 1 (Ω) satisfying φk → φα in L2 (Ω), Hεk (φk ) → zα in L1 (Ω). Moreover, we can further assume [8, Lemma 3] that R(zα , φα , c1α , c1α ) = lim

k→∞

  2 β1 |Hεk (φk )|BV + β2 φk − φ0 2H 1 + β3 j=1 |cjk − cj0 |2 .

Let (φεk , c1εk , c2εk ) be a minimizer of Gεk ,α . The sequences {φεk }, {Hεk (φεk )} and {cjk }j=1,2 are uniformly bounded in H 1 (Ω), BV(Ω) and R2 respectively. By the compact Sobolev embedding theorem [13], the compact embedding of BV into L1 [10] and the Bolzano-Weierstrass theorem, there exist convergent subsequences ˜ z˜ and c˜j . Summarizing, we have φε → φ˜ in whose limits are denoted by φ, k 2 1 ˜ c˜1 , c˜2 ) ∈ L (Ω), Hεk (φεk ) → z˜ in L (Ω), and cjk → c˜j in R, j = 1, 2. Thus, (˜ z , φ, L1 (Ω) × H 1 (Ω) × R2 is an admissible vector (cf. Lemma 2).

58

A. DeCezaro, A. Leitão, and X.-C. Tai

From the definition of R, Lemma 1 and the continuity of F , it follows that F (q(˜ z , c˜1 , c˜2 )) − y δ 2Y = lim F (Pεk (φεk , c1εk , c2εk )) − y δ 2Y , 

k→∞

2  ˜ c˜1 , c˜2 ) ≤ lim inf β1 |Hε (φε )|BV + β2 φε − φ0 2 1 + β3  |cj − cj |2 . R(˜ z , φ, εk k k k 0 H k→∞

j=1

Therefore, ˜ c˜1 , c˜2 ) = F (q(˜ ˜ c˜1 , c˜2 ) z , φ, z , c˜1 , c˜2 )) − y δ 2Y + αR(˜ z , φ, Gα (˜ ≤ lim inf Gεk ,α (φεk , c1εk , c2εk ) ≤ lim inf Gεk ,α (φk , c1k , c2k ) k→∞

≤

lim sup F (Pεk (φk , c1k , c2k )) k→∞

−

k→∞ δ 2 y Y



+ α lim sup β1 |Hεk (φk )|BV + β2 φk − φ0 2H 1 + β3

=

k→∞ F (q(zα , c1α , c2α ))

2 j=1

|cjk − cj0 |2

− y δ 2Y + αR(zα , φα , c1α , c2α ) = Gα (zα , φ1α , c1α , c2α ) ,

˜ c1 , c2 ) as a minimizer of Gα . characterizing (˜ z , φ, α α 4.1





Optimality Conditions for the Stabilized Functional

For numerical purposes it is convenient to derive first order optimality conditions for minimizers of the stabilized functionals Gε,α . Therefore, we consider Gε,α in (16) with Y = L2 (Ω) and we look for the Gâteaux directional derivatives with respect to φ and the unknown constants cj for j = 1, 2. Since Hε (φ) is self-adjoint, we can write the optimality conditions for the functional Gε,α in the form of the system α(Δ − I)(φ − φ0 ) = Lε,α,β (φ, c1 , c2 ), in Ω ; (φ − φ0 ) · ν = 0, at ∂Ω (17a) α (cj − cj0 ) = Ljε,α,β (φ, c1 , c2 ), j = 1, 2 .

(17b)

Here ν(x) is the external unit normal vector at x ∈ ∂Ω, β¯ := (2β3 )−1 , and Lε,α,β (φ, c1 , c2 ) = (c1 − c2 )β2−1 Hε (φ)∗ F  (Pε (φ, c1 , c2 ))∗(F (Pε (φ, c1 , c2 )) − y δ )   −β1 (2β2 )−1 Hε (φ) ∇· ∇Hε (φ)/|∇Hε (φ)| , (18a)

∗ L1ε,α,β (φ, c1 , c2 ) = β¯ F  (Pε (φ, c1 , c2 ))Hε (φ) (F (Pε (φ, c1 , c2 )) − y δ ), (18b)

 ∗ L2ε,α,β (φ, c1 , c2 ) = β¯ F  (Pε (φ, c1 , c2 ))(1 − Hε (φ)) (F (Pε (φ, c1 , c2 )) − y δ ).(18c)

5

Numerical Results

In this section a level-set type method based on the system of optimality conditions (17) is used for solving an inverse potential problem of recovering a piecewise constant function u : Ω → {c1 , c2 }, from measurements of the Cauchy data of its corresponding potential on the boundary of the domain Ω = (0, 1) × (0, 1). Notice that no knowledge of the image of u (values c1 , c2 ∈ R) is assumed.

On Level-Set Type Methods

5.1

59

The Inverse Potential Problem

To describe the direct problem, we define the operator F : L2 (Ω) → L2 (∂Ω) by F : u(x) → F (u) := wν |∂Ω , where u is a piecewise constant function in L2 (Ω) with u(x) ∈ {c1 , c2 } a.e. in Ω, and w ∈ H 1 (Ω) solves the elliptic boundary value problem Δw = u , in Ω ; w = 0 , at ∂Ω . (19) Since u ∈ L2 (Ω), the Dirichlet boundary value problem in (19) has a unique solution, namely the potential w ∈ H 2 (Ω) ∩ H01 (Ω). The inverse problem we are concerned with, consists in determining the piecewise constant source function u from measurements of the Neumann trace of w at ∂Ω, i.e. from wν |∂Ω . Using the above notation, the inverse potential problem can be written in the abbreviated form F (u) = y δ , where the data y δ has the same meaning as in (2). Other inverse problems for the operator F were considered in [3, 8]. In [3] a level set method was used for recovering the indicator function u = χD of a star-shaped domain D ⊂ R2 . In [8] a multiple level set method was used for recovering a simple function u : Ω → {c1 , . . . , c4 }. In both cases, knowledge of the (finite) image of u was assumed. 5.2

A Level-Set Algorithm for the Inverse Potential Problem

In the sequel we describe the level set regularization algorithm. This method compares to the level set method as proposed in [8]. The complexity of our algorithm is as follows: at each iteration of the level set method, four elliptic boundary value problems (BVP) are solved (two of Dirichlet type and two of Neumann type). In Table 1 an explicit fixed point procedure for solving the the optimality condition (18) is outlined. In the first step the residual rk ∈ L2 (∂Ω) of the iterate (φk , c1k , c2k ) is evaluated. This corresponds to solving one elliptic BVP of Dirichlet type. In the second step the solution hk ∈ H 1 (Ω) of the adjoint problem for the residual is evaluated. This corresponds to solving one elliptic BVP of Dirichlet type. In the fourth step, the velocity function vk ∈ H 1 (Ω) for the level-set function is evaluated. This corresponds to solving an elliptic BVPs of Neumann type. In the subsequent numerical experiments this algorithm was implemented using a finite element method for the solution of partial differential equations. 5.3

Numerical Experiment

In our experiment we consider the inverse problem of reconstructing the right hand side u in (19) from the knowledge of a single pair of Cauchy data (0, y δ ) at ∂Ω. We further assume that the level value c2 = 0 is given, and that we have to identify only the support of u and the level value c1 ∈ R+ . The data y δ = y = F (u) for solving the inverse problem is known exactly, i.e. δ = 0, and is obtained by solving numerically the elliptic boundary value problem

60

A. DeCezaro, A. Leitão, and X.-C. Tai Table 1. Level set algorithm for the inverse potential problem 1. Evaluate the residual rk := F (Pε (φk , c1k , c2k )) − y δ = (wk )ν |∂Ω − y δ , where wk solves wk = 0 , at ∂Ω . Δwk = Pε (φk , c1k , c2k ) , in Ω ; 2. Evaluate hk := F  (Pε (φk , c1k , c2k ))∗ (rk ) ∈ L2 (Ω), solving Δhk = 0 , in Ω ; hk = rk , at ∂Ω . 3. Calculate Lε,α,β (φk , c1k , c2k ) and Ljε,α,β (φk , c1k , c2k ), j = 1, 2 as in (18). 4. Evaluate the velocity vk ∈ H 1 (Ω), solving (Δ − I)vk = Lε,α,β (φk , c1k , c2k ) , in Ω ; (vk )ν = 0 , at ∂Ω . 5. Update the level set function φk and the level values cjk , j = 1, 2: φk+1 = φk +

1 α

vk ,

cjk+1 = cjk +

1 α

Ljε,α,β (φk , c1k , c2k ) .

in (19) (the word ’exactly’ here means: up to the precision of the numerical method used for solving the direct problem). For the direct problem we use the values: c1 = 1, c2 = 0 to compute the exact solution. In the computation of the inverse problem, the exact solution is known a priori to assume the values {c1 , 0} (with unknown c1 ). Moreover, when the data are given exactly, the iterative level-set method is implemented without the additional regularization term |Hε (φ)|BV , i.e. β1 = 0. The solution u of the inverse problem as well as the initial guess Pε (φ0 , c10 ) for the level-set method are shown in Figure 1. Notice that the support of u corresponds to a non-connected proper subset of Ω, The initial guess c10 = 1.5 is used for the unknown level value. In Figure 2 the evolution of the level set method for the first 1500 iterative steps is presented. As one can see in this figure, the shapes of both inclusions are reasonably reconstructed, and the level value c1 is accurately reconstructed as well. The iteration is stopped when the residual drops below the predefined precision F (Pε (φk , c1k )) − yL2 < 10−2 .

Fig. 1. Numerical experiment: The picture on the left hand side shows the coefficient to be reconstructed. On the right hand side, the initial condition for the level-set method.

On Level-Set Type Methods

61

Fig. 2. Numerical experiment: On the left hand side a plot of P (φk , c1k ) for k = 1500. The picture on the right hand side shows the corresponding iteration error.

We performed other numerical simulations with different choice of initial guess (φ0 , c10 ), and observed that the number of iterative steps required in order to obtain a reasonable approximation (up to the predefined precision of 10−2 in the L2 -norm) strongly depends on the choice of the initial guess c10 . On the other hand, the final result is not sensitive with respect to the choice of the initial guess φ0 .

Acknowledgments A.DC acknowledges the support from CNPq, grant 474593/2007-0. The work of A.L. is supported by the Brazilian National Research Council CNPq, grants 306020/2006-8, 474593/2007-0, and by the Alexander von Humbolt Foundation AvH. This article was written during a visit of the author to NTU (Singapore). X.-C.T. acknowledges the support from NTU SUG 20/07 and MOE Tier II project T207B2202 (ARC 29/07).

References 1. Santosa, F.: A level-set approach for inverse problems involving obstacles. ESAIM Contrôle Optim. Calc. Var. 1, 17–33 (1995/1996) 2. Leitão, A., Scherzer, O.: On the relation between constraint regularization, level sets, and shape optimization. Inverse Problems 19, L1–L11 (2003) 3. Frühauf, F., Scherzer, O., Leitão, A.: Analysis of regularization methods for the solution of ill-posed problems involving discontinuous operators. SIAM J. Numer. Anal. 43, 767–786 (2005) 4. Chung, E., Chan, T., Tai, X.C.: Electrical impedance tomography using level set representation and total variational regularization. J. Comput. Phys. 205(1), 357– 372 (2005) 5. Chan, T., Tai, X.C.: Identification of discontinuous coefficients in elliptic problems using total variation regularization. SIAM J. Sci. Comput. 25(3), 881–904 (2003) 6. Chan, T., Tai, X.C.: Level set and total variation regularization for elliptic inverse problems with discontinuous coefficients. J. Comput. Phys. 193(1), 40–66 (2004)

62

A. DeCezaro, A. Leitão, and X.-C. Tai

7. Chung, J., Vese, L.: Image segmantation using a multilayer level-sets apprach. UCLA C.A.M. Report 193(03-53), 1–28 (2003) 8. DeCezaro, A., Leitão, A., Tai, X.C.: On multiple level-set regularization methods for inverse problems. Inverse Problems 25 (to appear, 2009) 9. Tai, X.C., Chan, T.: A survey on multiple level set methods with applications for identifying piecewise constant functions. Int. J. Num. Anal. Model 1(1), 25–47 (2004) 10. Evans, L., Gariepy, R.: Measure theory and fine properties of functions. Studies in Advanced Mathematics. CRC Press, Boca Raton (1992) 11. Engl, H., Kunisch, K., Neubauer, A.: Convergence rates for Tikhonov regularisation of nonlinear ill-posed problems. Inverse Problems 5(4), 523–540 (1989) 12. Engl, H.W., Hanke, M., Neubauer, A.: Regularization of inverse problems. Mathematics and its Applications, vol. 375. Kluwer Academic Publishers Group, Dordrecht (1996) 13. Adams, R.: Sobolev Spaces. Academic Press, New York (1975)

The Nonlinear Tensor Diffusion in Segmentation of Meaningful Biological Structures from Image Sequences of Zebrafish Embryogenesis Olga Drblíková1 , Karol Mikula1 , and Nadine Peyriéras2 1

Slovak University of Technology, Radlinského 11, 813 68 Bratislava, Slovakia [email protected], [email protected] http://www.math.sk/drblikov, http://www.math.sk/mikula 2 CNRS-DEPSN, Institut de Neurobiologie Alfred Fessard, Batiment 32-33, Avenue de la Terrasse, 91198 Gif sur Yvette, France [email protected]

Abstract. In this contribution we develop a strategy for segmentation of evolving biological structures in image sequences. Our approach is based on combination of nonlinear tensor diffusion image smoothing and subjective surface based image segmentation. Since the fine cell structure would restrain the evolving segmentation function to achieve a shape of meaningful biological structures, we have to smooth properly the images in the sequence. To that goal we apply the nonlinear tensor diffusion which enhances the connectivity of bordering structure lines and smoothes their inner parts. For the numerical implementations we use semi-implicit diamond-cell finite volume methods both for filtering and segmentation. We show application of the method in image segmentation of early stages of zebrafish embryogenesis.

1

Introduction

The subjective surface based segmentation is an efficient tool for the extraction of 2D or 3D image objects, cf. [10,9,1]. It is also the case when dealing with twophoton laser scanning microscopy images in detecting and segmenting structures at cellular and subcellular level, cf. [6, 8]. However, the use of such algorithms when segmenting the supercellular structures is not straightforward. Using an original (not filtered) image leads to entirely useless results due to the presence of small cell structures. Then a useful tool is filtering by the nonlinear tensor diffusion enhancing the coherence of structure boundaries and smoothing the inner cell structures and noise. The model, cf. [11, 7, 4], has the following form ∂t u − ∇ · (D∇u) = 0, u(x, 0) = u0 (x), (D∇u) · n = 0,

in QT ≡ I × Ω,

(1)

in Ω, on I × ∂Ω,

(2) (3)

where u represents a greylevel 3D image intensity, u0 ∈ L2 (Ω), I = [0, T ] denotes a time interval, Ω is an image domain, D = D(u(x, t)) is a diffusion tensor and X.-C. Tai et al. (Eds.): SSVM 2009, LNCS 5567, pp. 63–74, 2009. c Springer-Verlag Berlin Heidelberg 2009 

64

O. Drblíková, K. Mikula, and N. Peyriéras

n is the outer normal unit vector to ∂Ω. The model is useful when a strong filtering is desirable in a preferred direction, e.g. along 2D edge surfaces in 3D images and a low smoothing is expected in the perpendicular direction.

2

Design of the Diffusion Tensor

The matrix D depends on a smoothed intensity gradient, which is given as ∇ut˜ = (ux1 , ux2 , ux3 )T , where ut˜(x, t) = (Gt˜ ∗ u(·, t))(x),

(t˜ > 0)

(4)

and Gt˜ is a Gaussian kernel. Provided μ = ||∇ut˜||2 > 0 we choose a triplet of vectors (v1 , v2 , v3 ) as follows v1  ∇ut˜,

v2 ⊥ ∇ut˜,

v3 ⊥ ∇ut˜,

v2 ⊥ v3 .

(5)

The direction of vector v1 corresponds to the direction of the largest intensity change. The other two vectors give a tangential plane to a level set of image intensity which may represent a 2D surface edge in a 3D image, provided that μ is large. It is called a coherence plane P, cf. [4, 7], and corresponds to an eigenspace corresponding to the eigenvalue 0 of the outer product ∇ut˜ ⊗ ∇ut˜. In order to improve the coherence, the diffusion tensor D must steer a filtering process such that the diffusion is strong and increasing with the level of μ along the coherence plane and is small in the perpendicular direction. We achieve it choosing the eigenvalues of the diffusion tensor, which determine the diffusivities in the directions v1 , v2 and v3 as κ1 = α,  κ2 =

α ∈ (0, 1), α 1, α, if μ = 0,   α + (1 − α) exp

−C μ

, C>0

(6) otherwise.

Further, we apply other convolution with a smoothing kernel ρ to get the diffusion matrix D in the form ⎛ ⎞  κ1 0 0 B, if μ = 0, B = ⎝ 0 κ2 0 ⎠ D = Gρ ∗ D0 , where D0 = (7) P BP −1 otherwise, 0 0 κ2 and P represents a transition matrix from the basis (v1 , v2 , v3 ) to (e1 , e2 , e3 ). The exponential function in (6) is used to ensure that κ2 does not exceed 1. The process never stops owing to the positive parameter α. Even if μ tends to zero, a small linear diffusion with a diffusivity α > 0 still remains there. C has the role of a threshold parameter. If μ C then κ2 ≈ 1, and, conversely if μ C then κ2 ≈ α. After some manipulations we get that at any point where μ > 0, the matrix D0 has the following form ⎞ ⎛ 2 ux1 ux2 (κ1 − κ2 ) ux1 ux3 (κ1 − κ2 ) ux1 κ1 + (u2x2 + u2x3 )κ2 1⎝ ux1 ux2 (κ1 − κ2 ) u2x2 κ1 + (u2x1 + u2x3 )κ2 ux2 ux3 (κ1 − κ2 ) ⎠ (8) μ ux1 ux3 (κ1 − κ2 ) ux2 ux3 (κ1 − κ2 ) u2x3 κ1 + (u2x1 + u2x2 )κ2

The Nonlinear Tensor Diffusion in Structure Segmentation

65

in the standard basis (e1 , e2 , e3 ). Such choice of the matrix D0 was given in [4], it is independ on a concrete choice of v2 and v3 and can be directly and fast evaluated using the diamond-cell finite volume technique (see also next section). Then the matrices are spatially averaged using the Gaussian smoothing with a variance ρ to get the final matrix D elements. The diffusion tensor possesses the smoothness, symmetry and positive definiteness properties, cf. [4].

3

The Finite Volume Scheme for 3D Nonlinear Tensor Diffusion

Let the image u(x) be represented by a bounded mapping u : Ω → R and given by n1 × n2 × n3 voxels (finite volumes) such that it looks like a mesh with n1 rows, n2 columns and n3 layers. Let us consider an image domain Ω = (0, n1 h) × (0, n2 h) × (0, n3 h), with a voxel size h. We consider the diffusion process in a time interval I = [0, T ]. Let the time discretization is given by 0 = t0 < t1 < ... < tNmax = T with tn = tn−1 + k, where k is a length of a discrete time step. We will look for an approximation of solution at time tn for every n = 1, ..., Nmax . We start the scheme derivation integrating the equation (1) over a finite volume K, then provide a semi-implicit time discretization and use the divergence theorem to have 

unK − un−1 K m(K) − (Dn−1 ∇un ) · nK,σ ds = 0, (9) k σ σ∈EK ∩Eint

where unK , K ∈ Th , denotes the mean value of un on K and Th is a cubic finite volume mesh. Further quantities and notations are given as follows: m(K) is the 3D measure of finite volume K with the boundary ∂K, σKL = K ∩ L is a side of the finite volume K, where L ∈ Th is a neighboring finite volume to K for which holds that the volumes K and L share a 2D surface element with a nonzero area. At several places we will replace σKL by σ only due to a notation simplification. EK represents the set of sides such that ∂K = σ∈EK σ and E = K∈Th EK . The set of boundary sides is denoted by Eext , that is Eext = {σ ∈ E, σ ⊂ ∂Ω} and Eint = E \ Eext . Υ is the set of pairs of neighboring finite volumes defined by Υ = {(K, L) ∈ Th2 , K  = L, m(K ∩ L)  = 0} and nK,σ is the normal unit vector to σ outward to K. Our discrete approximation solution is defined as uh,k (x, t) =

N

max



unK χ{x ∈ K}χ{tn−1 < t ≤ tn },

n=0 K∈Th

where the function χ(A) is given by  1, χ{A} = 0,

if A is true, elsewhere.

(10)

66

O. Drblíková, K. Mikula, and N. Peyriéras

The finite volume approximation at the n-th time step is given by unh,k (x) = 

n 1 uK χ{x ∈ K} and initial values as u0K = m(K) u0 (x)dx, K ∈ Th . We K∈Th

K

can define an auxiliary unknown φnσ (unh,k ) representing an approximation of the  1 n−1 exact averaged flux m(σ) ∇un ) · nK,σ ds for any K and σ ∈ EK in order σ (D to rewrite (9) in the form unK − un−1 1 K − k m(K)



φnσ (unh,k )m(σ) = 0,

σ∈EK ∩Eint

where m(σ) is the measure of side σ.

Fig. 1. The co-volumes associated with the side σ = σWE (left) and σ = σEW (right)

φnσ (unh,k ) is built with the help of a co-volume mesh, cf. e.g. [2, 3], for the 2D case. We create a co-volume χσ associated with σ around each finite volume side by joining four vertices of this side and midpoints of the finite volumes which are common to this side, cf. Fig. 1. The co-volume boundary consists of triangles σ ¯ ⊂ ∂χσ (their vertices are denoted by N1 (¯ σ ), N2 (¯ σ ) and N3 (¯ σ )) and nχσ ,¯σ is the normal unit vector to σ ¯ outward to χσ . First, we approximate the gradient  averaged  on χnσ . Applying the divergence theorem we obtain 1 1 n ∇u dx = u nχσ ,¯σ ds which can be approximated as follows m(χσ ) χσ m(χ σ ) ∂χσ 

1 1 n n n u m(¯ σ )nχσ ,¯σ . The values at xE + u + u pnσ (u) = m(χ (¯ σ ) (¯ σ ) (¯ σ ) N N N ) 3 1 2 3 σ σ ¯ ∈∂χσ

and xW are denoted as uE and uW . Further, we evaluate the values uTN , uTS , uBN , and uBS at the vertices xTN , xTS , xBN , and xBS , cf. Fig. 1, as the arithmetic mean of uK , where K represents the finite volumes which are common to the vertex. Since the mesh is uniform and squared, we can simplify our discrete √ h3 2 2 scheme applying the following relations: m(χσ ) = 3 , m(¯ σ ) = 4 h . After a short calculation we can state pnσ (u) =

uEn − unW un + unBN − unTS − unBS nK,σ + TN t1K,σ h 2h un + unTS − unBN − unBS t2K,σ , + TN 2h

(11)

The Nonlinear Tensor Diffusion in Structure Segmentation

67

6

φσT

φσN 

σT

φσW

σE σW

φ σE

σN

-

σS σB

φσS

φσB

?

Fig. 2. A finite volume K, its boundaries σi , i = E, W, N, S, T, B and the fluxes outward to the finite volume K

where t1K,σ is a unit vector parallel to xTN −xTS such that (xTN −xTS )·t1K,σ > 0 and t2K,σ is a unit vector parallel to xTN −xBN such that (xTN −xBN )·t2K,σ > 0. We replace the exact gradient ∇un by the discrete gradient pnσ (u) to get the numerical flux in the form φnσ (unh,k ) = (Dσ pnσ (u)) · nK,σ . ⎛

(12)

⎞

¯σ D ¯σ D ¯σ D 11 12 13 ¯σ D ¯σ D ¯ σ ⎠ denotes an approximation of the mean value Dσ = Dσn−1 = ⎝ D 12 22 23 σ ¯σ ¯σ ¯ 13 D D23 D33 of the matrix D along σ which was evaluated at the previous time step using ∞ functions due to the convolutions in un−1 h,k . The elements of matrix Dσ are C (4) and (7). Let us emphasize that in (12) we always consider the matrix Dσ written in the basis (nK,σ , t1K,σ , t2K,σ ), cf. [2,3] for an analogy with the 2D model. In practice it means cf. Fig. ⎛ σ that, ⎞ 2, the matrix D given in the standard basis on a side σ σ σ D11 D12 D13 σ σ σ ⎠ D22 D23 by ⎝ D12 is the same in the new basis on two sides σW and σE . It σ σ σ D13 D23⎛D33 ⎞ σ σ σ D22 D12 D23 σ σ σ ⎠ D11 D13 has the form ⎝ D12 in the new basis for two other sides σS and σN σ σ σ D D D ⎛23 σ 13 σ 33 σ ⎞ D33 D23 D13 σ σ σ ⎠ D22 D12 and it becomes ⎝ D23 for the last two sides σB and σT . Using such σ σ σ D13 D12 D11 matrix representations, the definition (12) can be written in the form ⎡⎛ ⎞⎤ ⎛ ⎞ n ⎞⎛ un E −uW ¯σ D ¯σ D ¯σ D 1 h 11 12 13 ⎢ ¯ σ ¯ σ ¯ σ ⎠ ⎜ unTN +unBN −unTS −unBS ⎟⎥ ⎝ ⎠ φnσ (unh,k ) = ⎣⎝ D · 0 = ⎠ ⎦ ⎝ 12 D22 D23 n 2h n n ¯σ D ¯σ D ¯σ un TN +uTS −uBN −uBS 0 D 13 23 33 n

n

n

n

n

n

2h

n

n

n

n

σ uE −uW σ uTN +uBN −uTS −uBS σ uTN +uTS −uBN −uBS ¯ 12 ¯ 13 ¯ 11 +D +D . =D h 2h 2h

68

O. Drblíková, K. Mikula, and N. Peyriéras

Finally, let us summarize our semi-implicit finite volume scheme: unK − un−1 1 K − k m(K) where

φnσ (unh,k ) =



φnσ (unh,k )m(σ) = 0,

(13)

σ∈EK ∩Eint n n n u ¯ σ uTN ¯ σ E − uW + D D 11 12

h n n n u σ TN + uTS − uBN ¯ 13 +D 2h

+ unBN − unTS − unBS 2h − unBS .

(14)

Due to the computation of the values uT N , uT S , uBN and uBS in (14) as the arithmetic mean of neighboring voxel values, we end up with the 27 point finite volume scheme.

4

Segmentation

Our segmentation approach is based on the subjective surface method [10] and its finite volume implementation from [9]. The mathematical model has the following form    ∇u 0 2 2 , in QT ≡ I × Ω,(15) ∂t u = ε + |∇u| ∇. g(|∇Gσ ∗ I |)  ε2 + |∇u|2 u(x, 0) = u0 (x), u = 0,

in Ω,

(16)

on I × ∂Ω,

(17)

where I 0 is the image which is segmented and ε is the regularization parameter. The solution u represents here the evolving segmentation function. The function g = g(|∇Gσ ∗ I 0 |) has the role of the edge detector, which requires a suitable 1 choice of g in practice, e.g. g(s) = 1+Ks 2 , K > 0. In the subjective surface method we start the segmentation constructing the initial segmentation function located in an approximate object center. The segmentation function is driven by equation (15) and evolves to a numerical steady state. Its shock profile gives the segmentation result and shape of the object. To that goal, we choose a suitable isoline of the shock profile which represents the boundary of the segmented object. This isoline is most naturally taken as the average of maximal and minimal value of the final segmentation function.

5

Numerical Experiments

The goal of this section is to discuss our computational results and the influence of nonlinear tensor diffusion filtering on the time evolving biological structure segmentation. We perform our experiments on the 3D image sequences of cell nuclei, cf. Fig. 3, and cell membranes, cf. Figs. 4-6. The images represent early stages of the zebrafish embryogenesis and were created by the two-photon laser scanning microscope. We apply the 3D numerical scheme to filter the images,

The Nonlinear Tensor Diffusion in Structure Segmentation

69

Fig. 3. 2D slices of a 3D zebrafish embryo image. Left: the original image. Right: the filtered image after 50 time steps.

then the segmentation is performed on 2D image slices (512×512 pixels) in order to firstly test the performance and capabilities of the method. First experiment illustrates the behaviour of nonlinear tensor diffusion in filtering of this type of images, cf. Fig. 3. One can observe that this type of diffusion improves the connectivity of structure bordering lines while it smoothes the structure interiors. One can compare the original image showing separate nuclei but with observable structure borders with the filtered one, where the structure border lines are connected. Our next experiments are devoted to the segmentation of eye retina structure in the several subsequent image slices. First, the initial segmentation function is given by two cones which are inside the structure such that their partially overlapping bases sufficiently cover the eye structure area. Then we evolved it in the original as well as filtered images. Using the original image we obtained the final state of segmentation function represented by a variety of different level lines, cf. Fig. 4 (top, right). The question is, which isoline would represent the most precisely the structure shape. The natural choice is a medium isoline which is depicted in the original image Fig. 4 (top, left). One can clearly see the large difference between the segmented and real structure shape due to the restraints of evolving segmentation function caused by inner cell structures. In order to compare our method with other filtering techniques we performed several tests. The segmentation results obtained on the images filtered by the geodesic mean curvature flow (GMCF) filtering, the mean curvature flow (MCF) filtering and the Perona-Malik (PM) filtering are shown in Fig. 5. In Fig. 5 (right), we can see that after filtering the profiles of final segmentation functions are not well suited for our purposes although the MCF results is rather close to the real one. They are again given by several different isolines and medium one, cf. Fig. 5 (left), represents the segmented structure only partially. This is a consequence of edge

70

O. Drblíková, K. Mikula, and N. Peyriéras

Fig. 4. The eye retina segmentation using the 2D original image (top) and image filtered by 20 time steps of the nonlinear tensor diffusion (middle). Left: the averaged isoline of the final state of segmentation function is superimposed to the original and filtered image, respectively. Right: the graphs of the final state of segmentation function is plotted after 2000 time steps using the original image and after 200 time steps using the filtered image. At the bottom we display the original (left) and the filtered image (right).

The Nonlinear Tensor Diffusion in Structure Segmentation

71

Fig. 5. The eye retina segmentation using the filtered image by 100 steps of the GMCF filtering (top), 25 steps of the MCF filtering (middle) and 20 steps of the PM filtering (bottom). Left: the averaged isoline of the final state of segmentation function is superimposed to the filtered image. Right: the graphs of the final state of segmentation function is plotted after 3000 segmentation steps using the GMCF filtering, after 500 segmentation steps using the MCF filtering and 5000 segmentation steps using the PM filtering.

72

O. Drblíková, K. Mikula, and N. Peyriéras

Fig. 6. The segmentation results for the image sequence which are superimposed to the original slices

preserving smoothing by GMCF and PM which can not remove inner cell structures. On the contrary, the final steady state of segmentation function evolving in the image filtered by nonlinear tensor diffusion consists of isolines accumulated

The Nonlinear Tensor Diffusion in Structure Segmentation

73

along the real structure boundary, cf. Fig. 4 (middle, right). The formation of correct shock profile was enabled due to the smoothing of cell structure barriers and noise removal and the emphasizing of structure boundaries. Embedding the medium isoline into the image, cf. Fig. 4 (middle, left) we achieved the precise structure shape. Then the segmentation procedure was successively applied in the image sequence part consisting of 11 images, cf. Fig. 6. We use the backward in time strategy starting from the last image of the sequence segmented as explained above. The initial segmentation function for other slices is taken as the final result of the segmentation of the previous image. Fig. 6 shows the segmentation results displayed on the original membrane images from last 150th image slice (top), to the 145th slice (middle) up to the 140th slice of the processed image sequence (bottom). In experiments dealing with the nonlinear tensor anisotropic diffusion we used the spatial step h = 0.01, time step k = 0.0001, C = 1, α = 0.001, t˜ = 10−5 , ρ = 0.002, 20 time steps for the filtering of membranes images and t˜ = 10−10 , ρ = 0.1, 50 time steps for the filtering of nuclei images. The arising linear systems were solved by the Gauss-Seidel iterative method. For the segmentation experiments we use the following parameters: ε = 10−4 , the spatial step h = 0.01, time step k = 0.01, δ = 10−6 for a stopping criterion and K = 100 (a constant of the 1 function g(s) = 1+Ks 2 ), cf. [9]. The resulting linear systems were solved by the SOR method.

6

Conclusions

In the article we concern with the technique for embryo structure segmentation in image sequences. Since a noise and cell structures restrain the correct segmentation evolution, as the first step, we smooth the image sequence. We choose the nonlinear tensor diffusion due to the fact that this filtering not only smoothes image objects but emphasizes connections of their boundaries as well. Then, the segmentation process starts using an artificial initial function centered inside the biological structure of the first image in the sequence. The segmentation result given by the subjective surface method obtained for this image is used as the initial condition for the next image of processed sequence, etc. Our experiments confirm the usefulness of the nonlinear tensor diffusion for this type of segmentation.

Acknowledgment The work was supported by the European projects Embryomics and BioEmergences, the grants APVV-RPEU-0004-06, APVV-0351-07, APVV-LPP-0020-07 and the grant of VEGA 1/0269/09.

74

O. Drblíková, K. Mikula, and N. Peyriéras

References 1. Corsaro, S., Mikula, K., Sarti, A., Sgallari, F.: Semi-implicit co-volume method in 3D image segmentation. SIAM J. Sci. Comput. 28(6), 2248–2265 (2006) 2. Coudiere, Y., Vila, J.P., Villedieu, P.: Convergence rate of a finite volume scheme for a two-dimensional convection-diffusion problem. M2AN Math. Model. Numer. Anal. 33, 493–516 (1999) 3. Drblíková, O., Mikula, K.: Convergence Analysis of Finite Volume Scheme for Nonlinear Tensor Anisotropic Diffusion in Image Processing. SIAM J. Numer. Anal. 46(1), 37–60 (2007) 4. Drblíková, O., Mikula, K.: Semi-implicit Diamond-cell Finite Volume Scheme for 3D Nonlinear Tensor Diffusion in Coherence Enhancing Image Filtering. In: Eymard, R., Herard, J.M. (eds.) Finite Volumes for Complex Applications V: Problems and Perspectives, ISTE and WILEY, London, pp. 343–350 (2008) 5. Eymard, R., Gallouët, T., Herbin, R.: Finite Volume Methods. In: Ciarlet, P., Lions, J.L. (eds.) Handbook for Numerical Analysis, vol. 7. Elsevier, Amsterdam (2000) 6. Frolkovič, P., Mikula, K., Peyriéras, N., Sarti, A.: A counting number of cells and cell segmentation using advection-diffusion equations. Kybernetika 43(6), 817–829 (2007) 7. Meijering, E., Niessen, W., Weickert, J., Viergever, M.: Diffusion-Enhanced Visualization and Quantification of Vascular Anomalies in Three-Dimensional Rotational Angiography: Results of an In-Vitro Evaluation. Medical Image Analysis 6(3), 217– 235 (2002) 8. Mikula, K., Peyriéras, N., Remešíková, M., Sarti, A.: 3D embryogenesis image segmentation by the generalized subjective surface method using the finite volume technique. In: Eymard, R., Herard, J.M. (eds.) Finite Volumes for Complex Applications V: Problems and Perspectives, ISTE and WILEY, London, pp. 585–592 (2008) 9. Mikula, K., Sarti, A., Sgallari, F.: Co-volume level set method in subjective surface based medical image segmentation. In: Suri, J., et al. (eds.) Handbook of Medical Image Analysis: Segmentation and Registration Models, pp. 583–626. Springer, New York (2005) 10. Sarti, A., Malladi, R., Sethian, J.A.: Subjective Surfaces: A Method for Completing Missing Boundaries. Proceedings of the National Academy of Sciences of the United States of America 12(97), 6258–6263 (2000) 11. Weickert, J.: Coherence-enhancing diffusion filtering. Int. J. Comput. Vision 31, 111–127 (1999)

Composed Segmentation of Tubular Structures by an Anisotropic PDE Model Elena Franchini, Serena Morigi, and Fiorella Sgallari Department of Mathematics-CIRAM, University of Bologna, Bologna, Italy {franchini,morigi,sgallari}@dm.unibo.it

Abstract. In this work we introduce the composed segmentation (Csegmentation), that is a priori composition of sources to obtain a single one segmentation result according to specific logic combinations. The approach and the segmentation model are general but we apply the C-segmentation technique to the challenging problem of segmenting tubular-like structures. The reconstruction is obtained by continuously deforming an initial distance function following the Partial Differential Equation (PDE)-based diffusion model derived from a minimal volumelike variational formulation. The gradient flow for this functional leads to a nonlinear curvature motion model. An anisotropic variant is provided which includes a diffusion tensor aimed to follow the tube geometry. Numerical examples demonstrate the ability of the proposed method to produce high quality 2D/3D segmentations of complex and eventually incomplete synthetic and real data.

1

Introduction

Segmentation of three-dimensional (3D) images can be a very useful computer aided diagnosis tool for clinical routines or surgical planning. We use the term composed segmentation for systems that extract structures from several images, by combining them according to specific Boolean operations. Traditionally, the segmentation process independently performed on single images have to be combined by cumbersome algorithms. The goal of C-segmentation is to combine complementary multispatial, multisensor, multitemporal and/or multiview information into one new domain containing only the information to be segmented. The term composed means by Boolean operations which depends on the application requirements. The individual images entering the C-segmentation process need to be registered to a common frame of reference, this is a nontrivial task which could affect the robustness of the segmentation approach, but it is not addressed in this work. We assume the input images have been preliminary registered. Let us illustrate the role of C-segmentation in different applications. Multimodal fusion deals with images that capture different physical properties of the original scene. In this case, C-segmentation identifies and segments the union of regions of interest. Multispatial fusion is related to several images which cover a single one scene, for example several aerial photographs to represent an entire territorial region, or multiple CT scans to reconstruct a human organ. The X.-C. Tai et al. (Eds.): SSVM 2009, LNCS 5567, pp. 75–86, 2009. c Springer-Verlag Berlin Heidelberg 2009 

76

E. Franchini, S. Morigi, and F. Sgallari

C-segmentation unifies all the information, eventually replicated in the multiple sources, into a single one segmented structure. Multitemporal composition requires the comparison between images representing the same structures acquired at different timing. For example, in medical analysis, a tumor-region growing is monitored by subsequently images of the region of interest. The C-segmentation can identify and segment the difference between structures in two images, reconstructing the grown area. While our segmentation methodology is quite general, we focus our attention on the most challenging problem of tubular-like segmentation which is particularly difficult in case of multiple sources due to the huge amount of connected structures that should be reconstructed. In particular, we will consider applications in medical image analysis which are interested in the extraction of anatomical surfaces of tubular structures like blood vessels. Indeed, problems like aneurysm or stenosis can occur in a vessel, and the clinicians need tools to help them in interpreting and quantifying the images for evaluating the pathology, for proposing a therapy or a surgical operation, for planning minimally invasive treatment. A number of deformable model-based approaches for vessel segmentation or, generally tube-like structures, have received considerable attention and success. We refer the reader to [4] for an extended review on vessel segmentation algorithms. Since explicit deformable model representation is usually impractical, level set techniques to evolve a deformable model have been recently introduced, which provide implicit representation of a deformable model. A curve in 2D or a surface in 3D evolves in such a way as to cover a complex shape or structure. Its initialization can be either manual or automatic and it needs not to be close to the desired solution. A disadvantage of level sets segmentation approach is the computational effort required to cover the entire domain of interest which is, in general, one dimension higher than the original one. Interested readers are referred to recent literature on the level set segmentation strategy for tubular structures [5], [6], [7], [8], [10]. A generalization of the single-channel active contour without edges model is proposed in [9] for object detection using logic operations. This logic framework suffers from the active contour model limits, and is not suitable for detecting tubular structures. In this work, we modify a geometric deformable model segmentation procedure based on level sets [2], to obtain a fast and accurate method for solving the C-segmentation problem to extract tubular structures from multiple 2D/3D images, and we apply the proposed segmentation method for segmenting blood vessels, neurovascular structures and similar characteristics medical images. The main contributions of this work concern the design of a strategy to deal with directionality in the vessels based on a diffusion tensor, and the capability to compose segmentation of multiple images according to Boolean operations. The former makes the segmentation algorithm able to follow tubular structures and connect eventually disconnected parts, while the latter let simultaneously combine different information into a robust segmentation method. The proposed method is able to segment twisted, convoluted, and occluded structures without

Composed Segmentation of Tubular Structures

77

the user interactivity, following branching of different layers, from thinner to larger structures. One of the major disadvantages of the geometric deformable models, that is the computational cost, is strongly reduced by the proposed numerical approach which limits the dimensions of the linear systems involved in the solution. The paper is organized as follows. The non-linear PDE model for tubular structures segmentation is introduced in Section 2, and numerical aspects related to the discretization of the PDE model are discussed in Section 3. The Csegmentation algorithm is discussed in details in Section 4, and the anisotropic variant of the segmentation model is introduced in Section 5. Synthetic as well as real tests are provided in Section 6. Some selected 3D examples are also presented in Section 6 to demonstrate the effectiveness of this technique for automatic segmentation of bloods vessels in volumetric MRA/CTA images. Section 7 contains concluding remarks.

2

A Segmentation Model for Tubular Structures

Several recently proposed 3D segmentation methods are based on deformable models, which can naturally capture the physics and geometry of shapes varying in space and time. In this section we formulate the segmentation problem as a special deformation of a 3D manifold driven by the structures we want to recover. Classical segmentation problems show oversmoothed structures and eventually uncomplete boundaries and the surface evolution usually flows over the boundaries of longer and thinner objects when propagating. A common choice to detect structure boundaries and to drive diffusion or segmentation process is the Perona-Malik diffusivity g(s) = 1/(1 + (s/ρ)2 ),

(1)

where ρ > 0 is a small positive constant. For the implicit representation of the segmented surface, we consider a special 3D manifold which is the graph of a trivariate function φ mapping an open set Ω ⊂ R3 into R. The problem of determining the surface that best fits the object boundary represented in a 3D image I, can be posed as a volume minimization problem with objective function   Vg := g(∇I)dV dV = 1 + ∇φ2 dxdydz, (2) Ω

where the metric g is defined by (1) in Ω and Vg represents the weighted volume of a 3D manifold on Ω. The volume functional (2) can be minimized and according to the steepest descent, reading ε = 1, we have   ∇φ ∂φ  2 2 = ε + ∇φ ∇. g(∇I)  , (3) ∂t ε2 + ∇φ2

78

E. Franchini, S. Morigi, and F. Sgallari

or, equivalently, in advection-diffusion form   ∂φ  2 ∇φ 2 = ε + ∇φ g(∇I)∇.  + ∇g · ∇φ. ∂t ε2 + ∇φ2

(4)

The PDE model (4), ε = 1, represents the mean curvature motion of the 3D manifold in 4D space with metric g. The metric g in (4) is the edge function appropriately chosen so that the object boundaries act as attractors under a particular flow. This term allows us to extract sharp features, such as edges, corners, spikes, and to accelerate the deformation of the initial function. In the evolution of φ according (4) the 3D manifold assumes constant values for most regions far from the boundaries. The first term in (4) corresponds to a minimal volume regularization weighted by the function g, while the second term corresponds to the attraction to the image edges. The advection term in equation (4) introduces a driving force which moves the level surfaces towards the object boundaries. Equation (3) in case ε ∈ (0, 1] is proposed in [2] for dealing with the boundary completion problem. The variability in the parameter ε, ε ∈ (0, 1], provides both a regularization effect and a hole filling strategy. The effect of the parameter ε is to segment boundaries which are eventually uncompleted due to, for example, noise or corruptions in the acquisition phase. However, this does not help in the reconstruction of slightly disconnected tubular structures. The latter problem is solved by the introduction of a suitable diffusion tensor, which is discussed in Section 5. The starting initial function φ0 is usually a problem, since it involves user interaction for locating some starting points at one particular recognizable part of the structure to be segmented inside the 3D image. This is overcome by our method which automatically initialize the surface evolution using a suitably designed distance function, as described in Section 4. We can adapt the PDE model (3) to compute Boolean operations between implicit surfaces M1 and M2 . This can be carried out quite easily, using the min, max tools on the related signed distance functions dist1 (x) and dist2 (x). In fact the union, intersection and differences between two surfaces can be obtained applying the evolving PDE (3) initialized by (9) in Section 4, where dist(x) is defined respectively by dist(x) = min{dist1 (x), dist2 (x)}, union dist(x) = max{dist1 (x), dist2 (x)} intersection dist(x) = max{dist1 (x), −dist2 (x)} difference(M1 − M2 ) dist(x) = max{−dist1 (x), dist2 (x)} difference(M2 − M1 ).

3

(5)

Solving the PDE Model

The computational method for solving (3) is based on an efficient semi-implicit co-volume scheme as suggested in [2]. The semi-implicit in time discretization is obtained by treating the nonlinear terms of the equation from the previous

Composed Segmentation of Tubular Structures

79

time step while the linear ones are considered at the current time level. Timediscretization of (3) by Euler’s method yields the following semi-discrete scheme Let τ be a uniform discrete time step, φ0 be a given initial function. Then, for every discrete time step tn = nτ , n = 1, . . . N , we look for a function φn , solution of the equation   1 φn − φn−1 ∇φn 0  = ∇. g  , (6) τ ε2 + ∇φn−1 2 ε2 + ∇φn−1 2 where g 0 := g(∇I). The computational domain is obtained through Ω-decomposition into cubic cells and the construction of a co-volume mesh using a complementary 3D tetrahedral grid. Following the classical finite volume methodology we integrate (6) over every co-volume p, p = 1, . . . , M and according to the details explained in [2] we get at the time step n, a system of linear equations which can be written in matrix-vector form as AΦn = b,

(7)

where A ∈ RM×M is the coefficient matrix, which is symmetric and diagonally dominant M-matrix, and Φn = (φ1 , . . . , φM ) is the vector solution. Since the unknown function φ(x, t) evolves only on nodes sufficiently close to the structure boundary, we speed up the computation by determining the updated values for φ(x, t) only for the nodes identified by initial function φ0 (x) > η, for a given small positive threshold η. In case of vessel structures, for example, this means a significant reduction of the computational effort required since the number of nodes representing the vessels is small compared with the dimension of the entire 3D image which contains them. In practice, at each time step, the number of unknowns of the linear system (7) is significatively reduced, and thus both the storage and the computational cost are much lower. Since at each row of A corresponds a node in Ω, if we consider a limited number of nodes M1 << M , we get a linear system with a sparse coefficient matrix of rank M1 , which contains M − M1 zero rows and columns. It is easy to verify that if we first apply a suitable permutation of rows and columns of A , and corresponding elements of b, we get a linear  ∈ RM1 ×M1 with full rank which has the same system with a coefficient matrix A properties as A, that is, it’s symmetric and positive definite. In a similar way, the same permutation applied to the components of the right-hand side vector b, leads to a vector b ∈ RM1 . Therefore, instead of solving the linear system (7) which involves a M × M coefficient matrix, we can apply the preconditioned conjugate gradient iterative method, using diagonal scaling as preconditioner,  of the linear system for computing the solution Φ  = b, AΦ with a negligible approximation error on the solution.

(8)

80

E. Franchini, S. Morigi, and F. Sgallari

4

The C-Segmentation Algorithm

The C-segmentation procedure consists of two steps: the partitioning phase driven by the Boolean operations to obtain an initial function, and the segmentation phase. The case of segmentation of tubular structures from a single input image is trivially a simplification of the proposed C-segmentation method. In this section, we discuss the implementation of the PDE model (3) for segmentation of multiple 3D images. Let Ω be a common support. Each gray-level 3D image i containing tubular structures of interest is thresholded to obtain a rough segmentation estimate which is used to generate an initial distance function. The thresholding process consists of making a binary image Ii from the input i image based on a user threshold. Since the images can be represented by a real-valued function defined on a region Ω ⊂ R3 or by the discretization of such a function, the threshold process let us define in a natural way the sub-domain Ωi ⊂ Ω, Ωi := {x ∈ Ω : Ii (x) = 1}. The choice of the threshold value is not an issue due to the fact that in practical cases the structures to be segmented are characterized by a particular intensity value represented by a given gray level. For example, vascular system containing a contrast fluid, can be identified as the brightest formations in the volume. As it will be better illustrated by the examples in Section 6 the 3D mask Ii resulting from the pre-computed binarization, in general, preserves the largest tubular structures while breaks up into small pieces the thinnest ones. These structures will be perfectly recover and reconnected by the surface evolution process. Each image i has an associated set (region) Ωi . Depending on the type of the fusion task {Ωi }i=1,··· ,S , where S is the number of input images, need not be exactly disjoint and/or cover the whole region Ω. The composed signed distance function dist(x) is then obtained by the signed distance functions disti (x), i = 1, .., S, with respect to Ωi , following the rules (5). We define the initial function φ0 as follows  1 if dist(x) < 0 . (9) φ0 (x) = dist(x) (1 − max{dist(x)} ) else The implicit function φ0 represents the initial surface which continuously evolves, by following equation (3), towards the boundaries of the tubular structures. The evolution is stopped when a change of solution in time (in L2 -norm) is less than a prescribed tolerance. A critical issue is the definition of the composed diffusion function g(·) in (3) when the C-segmentation considers several input images. For example, in case of difference between two images, which are characterized by g 1 and g 2 , the composed diffusion function g 0 used in (3) is given by if ((g 1 ≤ )and(g 2 ≤ ) or (g 1 ≥ )and(g 2 ≥ )) g 0 = 1 else g 0 = min(g 1 , g 2 ),

(10)

with > 0, we used = 1 · 10−3 in the computational examples. We can proceed similarly for other Boolean operations.

Composed Segmentation of Tubular Structures

81

Finally, the reconstructed surface is obtained from the implicit surface φ as the zero level set of the function φ(x) − s, that is the s-level set of φ: {x ∈ Ω : φ(x) = s}, where s = (max(Φ) + min(Φ))/2. This is motivated by the fact that the flow driven by (3) forms a sharp step in the proximity of the object boundaries, while it approaches at constant values inside/outside the object.

5

An Anisotropic Variant of the Segmentation Algorithm

In this section we provide a variant of the isotropic model (3) designed to improve significantly the connectivity of the coherent structures in the segmentation. The idea is to incorporate local orientation of the tubular structures into the dynamic segmentation process in such a way that at each time step the surface evolves by isotropic mean curvature motion in homogeneous regions, while it is driven by the directional field representing the orientation of the tube in presence of tubular structures. We aim to capture the vessel’s structure and the vessels directions locally by a local spatial coherence descriptor. Coherence enhancing image smoothing has been introduced by [3] and successfully applied in image filtering by anisotropic diffusion. This type of nonlinear diffusion includes the construction of a diffusion tensor which is built as follows. Given an image I, and its Gaussian-smoothed version ∇Iσ , a regularized shape descriptor is provided by Jδ (∇Iσ ) := (Kδ ∗ (∇Iσ ⊗ ∇Iσ )) (11) where Kδ is a Gaussian kernel with δ ≥ 0. The matrix Jδ is symmetric positive semi-definite and its eigenvalues μ1 ≥ μ2 integrate the variation of the gray values within a neighborhood of size O(δ). They describe the average contrast in the corresponding eigendirections v1 and v2 . The orientation of the eigenvector v2 , corresponding to the smaller eigenvalue, represents the direction of lowest fluctuations, the so-called coherence orientation. In this way, constant areas are characterized by μ1 = μ2 = 0, while straight edges give μ1 μ2 = 0. The normalized coherence value which measures the anisotropic structures within a window of scale δ is thus defined as c=

(μ1 − μ2 )2 , max{(μ1 − μ2 )2 }

c ∈ [0, 1].

(12)

Thus c approaches to 1, for anisotropic structures and tends to zero for isotropic structures. The diffusion tensor D is a matrix with the same eigenvectors as the (regularized) structure tensor Jδ and its eigenvalues are given by λ1 = g(∇I)  g(∇I) λ2 = g(∇I) + (1 − g(∇I))e−κ/c

if μ1 = μ2 κ > 0, else

(13)

where g(·) is the composed diffusion function, defined for example, by (10) which suitably adapts its values to the anisotropy. The parameter κ has the role of a

82

E. Franchini, S. Morigi, and F. Sgallari

threshold, and c is the coherence defined in (12). Therefore, the matrix D has the following form

T

v1 λ 0 . (14) D = [ v1 v2 ] 1 0 λ2 v2T In local homogeneous areas of an image the diffusion is reduced to be the isotropic mean curvature motion driven by (3), in fact we have D = g(∇I). Areas nearby elongated structures are characterized by values of g(·) approaching to zero, which gives λ2 >> λ1 . The effect on the diffusion of the segmentation function is thus stronger along the coherence directions. In all the experiments reported in Section 6 we set κ = 1 · 10−5 . Since if c >> κ then λ2 ≈ 1, while if c << κ then λ2 ≈ g(∇I), the segmentation function flows along the coherence direction when it approaches to the edges and stops when the object boundary is reached. The incorporation of the diffusion tensor D defined by (14) in the segmentation model (3) leads to the following nonlinear anisotropic segmentation   ∂φ  2 ∇φ = ε + ∇φ2 ∇. D  , φ(x, 0) = φ0 (x) x ∈ Ω. (15) ∂t ε2 + ∇φ2 We will refer to the models (3) and (15) by isotropic and anisotropic segmentation models, respectively. The PDE model (15) can be easily extended to 3D or 4D segmentation problems; we refer the reader to [1] for the numerical aspects involved in the discretization of the diffusion tensor.

6

Results and Experiments

We tested the performance of the proposed C-segmentation method on 2D/3D synthetic and real examples. Example 1 and 2 demonstrate the performance of the anisotropic C-segmentation model compared with the isotropic one to verify the important role of the diffusion tensor in the segmentation of tubular structures. Examples 3, 4 and 5 illustrate results from the application of composed segmentation of 2D/3D images. For all the experiments, we apply C-segmentation algorithm as illustrated in Section 4, with or without diffusion tensor, and we set the time step parameters involved to τ = 1 · 10−3 , for 3D images and τ = 1 · 10−2 for the 2D images, and ε = 1 · 10−3 in (3). We stop the iterations when the change of solution in time is less than 1 · 10−4. In our experiments, the segmentation of images with high contrast (see Examples 2 and 3) provides good results when an automatic choice for parameter s is chosen such as s = (max(Φ) + min(Φ))/2, while in case of images like in Examples 1, 4 and 5, the results are more sensitive to the choice of s, and visual inspection could be required to tune the automatic choice of s. Example 1. In the first 2D example we consider the segmentation of a carotid vascular system represented in a Magnetic Resonance Angiography (MRA) image of 182 × 182 pixels shown in Fig. 1 (left). MRA is based on detection of signals from flowing blood and suppression of signals from other, static, tissues.

Composed Segmentation of Tubular Structures

83

Fig. 1. Carotid vascular system segmentation; the original image (left); results of the isotropic model (center), and anisotropic model (right)

Fig. 2. First row: original synthetic image (left); results obtained by the isotropic (left) and anisotropic segmentation model (right). Second row: associated segmentation function φ is shown at the final time step of isotropic (left) and anisotropic segmentation (right).

The blood vessels appear as high intensity regions in the image. The structures to be segmented represent several vessels of variable diameters which are close to each other, partial occlusions and intersections make the segmentation very challenging. In Fig.1 the segmented structures obtained by 10 iterations of the isotropic (center) and anisotropic (right) segmentation models are displayed. The boundary curves in pink color are extracted using contour values s = 0.90 and s = 0.98, respectively. Visual comparison shows the anisotropic segmentation method to give the most accurate restoration. Example 2. The second example illustrates the ability to reconstruct structures which present small occlusions along the coherence direction. The synthetic image to be segmented of dimension 200 × 200 pixels is shown in Fig. 2 (first row, left). Applying 10 time steps of the anisotropic segmentation model, the structure is well reconstructed while maintaining the narrowing, as shown in Fig.2

84

E. Franchini, S. Morigi, and F. Sgallari

(a)

(b)

(c)

(d)

Fig. 3. C-segmentation of images (a) and (b). Result in (c) of (b)\(a) and in (d) of (a)\(b).

(first row, right). The propagation driven by the isotropic segmentation model leads to enhance the disconnections, as shown in Fig.2 (first row, center). The boundary curves determined as iso-contours of the segmentation functions shown in Fig.2 (second row), using s = 0.95 are over-imposed on the original image and shown in Fig.2 (first row). Example 3. Verification of the proposed C-segmentation method is carried out on a real echo image show in Fig.3 (a) where we generated one synthetic bump and two synthetic holes, simulating aneurysm and stenosis effects, as illustrated in Fig.3 (b). Applying C-segmentation to the Boolean difference (b) \ (a) we are able to enhance the aneurysm as shown in Fig.3 (c), while to detect the stenosis effects we apply C-segmentation to the Boolean difference (a) \ (b), obtaining the results illustrated in Fig.3 (d). Example 4. Volumetric segmentation is applied to a Computed Tomographic Angiography (CTA) data set shown in Fig.4(left). The volumetric data set kidney of dimension 201 × 201 × 201 has been extracted from a 436 × 436 × 540 CTA image of the kidney vasculature system and present vessel patterns with different curvatures, diameters and bifurcations. The final segmentation obtained after 10 time steps of the segmentation algorithm is shown in Fig.4(right). Example 5. The last experiment demonstrates the performance of the proposed technique to segment a complex structure extracted from two overlapping 3D

Composed Segmentation of Tubular Structures

85

Fig. 4. Segmentation of the kidney volume data set: (left) original CTA image (right) segmentation result

Fig. 5. Segmentation of the lung volume data set: (left) original MRA image, (right) C-segmentation result

MRA images representing the airway tree of a human lung. The original MRA image is illustrated in Fig.5(left) using volume rendering. In particular the ellipsoidal area outlined in the image includes the region of interest for segmentation inside the lung image. The data set lung is DICOM format and sample images available on the web site (http://www.osirix-viewer.com/Downloads.html). We simulated the acquisition of two partially overlapping 3D MRA images covering the area of interest consisting of 156 × 156 × 156 voxels each, and we applied the C-segmentation algorithm using union Boolean composition. Fig.5(right) shows the obtained single one segmentation of the human airway tree, where several branching generations were detected. These preliminary results demonstrate the applicability of the developed method for the C-segmentation of quite different and topological complex tubular structures.

86

7

E. Franchini, S. Morigi, and F. Sgallari

Conclusions

In this paper we introduced the C-segmentation method based on the distance function which is successfully applied to 2D/3D images containing tubular structures. The algorithm is automatic, accurate and fast. The latter is due to a speed up strategy in the iterative method for linear systems. The algorithm is provided with a diffusion tensor to move the evolving surface toward elongated tubular structures, connecting gaps in the underlying raw data, while keeping the structures distinguished along the coherence direction. This model defines a general segmentation framework which combines object information from different images into any logical combination, rather than following the difficult a posteriori process to compose segmentation results obtained separately.

Acknowledgments This work has been supported by PRIN-MIUR-Cofin 2006, project and by University of Bologna "Funds for selected research topics".

References 1. Drblikova, O., Mikula, K.: Semi-implicit Diamond-cell Finite volume Scheme for 3D Nonlinear Tensor Diffusion in Coherence Enhancing Image Filtering. In: Eymard, R., Herard, J.M. (eds.) Finite Volumes for Complex Applications V: Problems and Perspectives, ISTE and WILEY, London, pp. 343–350 (2008) 2. Corsaro, S., Mikula, K., Sarti, A., Sgallari, F.: Semi-implicit covolume method in 3D image segmentation. SIAM J. Sci. Comput. 28(6), 2248–2265 (2006) 3. Weickert, J., Scharr, H.: A scheme for coherence enhancing diffusion filtering with optimized rotation invariance. Journal of Visual Communication and Image Representation 13(1/2), 103–118 (2002) 4. Kirbas, C., Quek, F.: A review of vessel extraction techniques and algorithms. ACM Computing Surveys 36(2), 81–121 (2004) 5. Hassan, H., Farag, A.A.: Cerebrovascular segmentation for MRA data using levels set. International Congress Series, vol. 1256, pp. 246–252 (2003) 6. Scherl, H., et al.: Semi automatic level set segmentation and stenosis quatification of internal carotid artery in 3D CTA data sets. Medical Image Analysis 11, 21–34 (2007) 7. Cohen, L.D., Deschamps, T.: Segmentation of 3D tubular objects with adaptive front propagation and minimal tree extraction for 3D medical imaging. Computer Methods in Biomechanics and Biomedical Engineering 10(4), 289–305 (2007) 8. Gooya, A., Liao, H., et al.: A variational method for geometric regularization of vascular segmentation in medical images. IEEE Transaction on image processing 17, 1295–1312 (2008) 9. Sandberg, B., Chan, T.F.: A logic framework for active contours on multi-channel images. J. Vis. Commun. Image R. 16, 333–358 (2005) 10. Westin, C.-F., Lorigo, L.M., Faugeras, O.D., Grimson, W.E.L., Dawson, S., Norbash, A., Kikinis, R.: Segmentation by Adaptive Geodesic Active Contours. In: Delp, S.L., DiGoia, A.M., Jaramaz, B. (eds.) MICCAI 2000. LNCS, vol. 1935, pp. 266–275. Springer, Heidelberg (2000)

Extrapolation of Vector Fields Using the Infinity Laplacian and with Applications to Image Segmentation Laurence Guillot1 and Carole Le Guyader2 1 Laboratoire Jean-Alexandre Dieudonné, UMR 6621, Université de Nice Sophia Antipolis – CNRS, Faculté des Sciences Parc Valrose, 06108 – Nice Cedex 02, France [email protected] 2 IRMAR, UMR CNRS 6625, Institut des Sciences Appliquées de Rennes, 20, Avenue des Buttes de Coësmes, CS 14315, 35043 Rennes Cedex, France [email protected]

Abstract. In this paper, we investigate a new Gradient-Vector-Flow (GVF)( [38])-inspired static external force field for active contour models, deriving from the edge map of a given image and allowing to increase the capture range. Contrary to prior related works, we reduce the number of unknowns to a single one v by assuming that the expected vector field is the gradient field of a scalar function. The model is phrased in terms of a functional minimization problem comprising a data fidelity term and a regularizer based on the super norm of Dv. The minimization is achieved by solving a second order singular degenerate parabolic equation. A comparison principle as well as the existence/uniqueness of a viscosity solution together with regularity results are established. Experimental results for image segmentation with details of the algorithm are also presented.

1 1.1

Introduction Motivations

Many of the well-known variational segmentation methods require a careful choice of the initial condition. One of the most famous variational methods to process this partition of the image is the active contour model introduced by Kass, Witkin, and Terzopoulos ([30]). It consists in evolving a parameterized curve so that it matches the object boundary. The shape taken by the curve through the process is related to an energy minimization, this energy comprising a data fitting term and a regularizer, and being non-convex. Thereof, we can only expect local minimizers, which, in practice, means that the contour to be deformed must be initialized near the object boundary. Cohen ([18]) has proposed a way to alleviate this constraint by adding an inflating/deflating force in the modelling, defined by kn, n denoting the unit inward normal to the curve and k, a constant. According to the sign of the constant k, the curve inflates X.-C. Tai et al. (Eds.): SSVM 2009, LNCS 5567, pp. 87–99, 2009. c Springer-Verlag Berlin Heidelberg 2009 

88

L. Guillot and C. Le Guyader

or deflates. Thereby, in practice, the contour to be deformed is either initialized inside the object, or it encloses the object of interest. In [38], Xu and Prince address both the problems of initialization and slow and/or poor convergence near boundaries with strong concavities by introducing a new static external force called Gradient Vector Flow (GVF). The initialization constraint is removed, that is, initialization can be made inside, outside or across the object boundaries, and the front evolution is easily handled even in boundary concavities. The main idea behind this model is to increase the capture range of the external edge-map-related force field, and to make the contour evolve toward the desired boundaries, here where classical methods would fail to. Unlike classical active contours, the introduced external force does not derive from a potential function and cannot be computed straightforwardly from the image edge map. More precisely, the model cannot be phrased in terms of a unique functional minimization problem but is defined in two steps. In a first step, the external force (GVF) w = (u, v)T is obtained by minimizing an energy functional in a variational framework. The corresponding Euler-Lagrange equations are computed and lead to solve a linear partial differential equation decoupled system by a gradient descent method. The second step consists then in replacing, in the dynamic snake equation, the classical potential force by the newly computed external force w. This method motivated the following works. In [34], Paragios et al. propose to integrate this boundary spatial diffusion technique to the geodesic active contours ( [16]). In [29], Jifeng et al. propose to improve the diffusion properties of the GVF force field. They obtain a new force by replacing the Laplacian operator used in the GVF model by its diffusion term in the normal direction that is, the normalized infinity Laplacian operator. Unlike the GVF model, their new field (called NGVF for GVF in the normal direction) is anisotropic. Furthermore, the NGVF is stable for bigger time steps, improves slightly segmentation results, and allows to detect more quickly long and thin concavities. Our work is much motivated by [38] and [29]. We wanted to provide, in a rigorous mathematical framework, a new method to generate this external force field. Contrary to these prior works, we propose to reduce the number of unknowns to a single one, by assuming that the sought vector field is the gradient field of a scalar function. Also, the introduced minimization problem contains a data-fitting term related to the original GVF model and a regularizer that penalizes the super norm of the unknown gradient. Thus the problem becomes related to the absolutely minimizing Lipschitz extensions and to the infinity Laplacian. The absolute minimal Lipschitz extension model was introduced by Aronsson in [2] (see also [1, 3, 4]) in the following way. Given Ω ⊂ IRn a bounded, open and connected domain with sufficiently smooth boundary, and b ∈ C(∂Ω), solve inf

u∈W 1,∞ (Ω), u=b

on ∂Ω

DuL∞ (Ω) .

(1)

A minimizer of (1) is called an absolutely minimizing Lipschitz interpolant of b|∂Ω inside Ω. Aronsson proved the existence of an absolute minimal Lipschitz

Extrapolation of Vector Fields Using the Infinity Laplacian

89

extension and Jensen proved the uniqueness. Aronsson also derived the EulerLagrange equation governing the absolute minimizer in the sense of viscosity solutions: Δ∞ u = D2 u(Du, Du) = 0 in Ω. (2) We refer to [5, 10, 20] for more details. The operator Δ∞ is called the infinity Laplacian and solutions of (2) are said to be ∞-harmonic. Jensen proved a comparison principle and an existence/uniqueness result of (2) for Lipschitz continuous boundary data (see [28]). Before depicting our model, we briefly make a non-exhaustive review of some prior works related to AMLE and infinity Laplacian in the field of image processing. 1.2

Prior Related Works

As stressed by Caselles et al., the equation Δ∞ u = D2 u(Du, Du) = 0 was introduced in the field of computer vision as edge detector (see [37], [39]). It earlier appeared in the domain of edge enhancement (see [35]) and served as the basis of Canny edge detection [12]. In [17], Caselles et al. investigate the AMLE and the infinity Laplacian in the field of image processing with applications to the restoration of images. Motivated by prior applications devoted to coding ( [13], [14]), they address the issue of interpolating data given on a set of points and/or curves in the plane. Another application, dedicated to shape metamorphism (the process which consists in evolving a source shape into a target shape by intermediate steps) is proposed by Cong et al. in [19] and makes use of the infinity Laplacian. Also, in [31], Mémoli et al. propose a new framework for brain warping using Minimizing Lipschitz Extensions. To finish, in [24], Elion and Vese aim at solving the (BV, G) decomposition model introduced by Meyer in [32]. In that purpose, Elion and Vese focus on an isotropic decomposition of the image f ≈ u + v with v = ΔP = div(DP ) and DP ∈ (L∞ (Ω))2 . The outline of the paper is as follows: Sect. 2 is devoted to the depiction of the model and the derivation of the associated evolution problem. Section 3 is dedicated to the theoretical study of the obtained parabolic problem. We first prove a comparison principle, then prove existence and uniqueness of a viscosity solution. Regularity results of this solution are also given. We conclude the paper with experimental results and integrate this new external force field in a segmentation problem. Details of the algorithm are also provided.

2

Depiction of the Model

Let Ω be a bounded open subset of IRn , ∂Ω its boundary and let I be a given ¯ → IR. For the purpose of illustration we bounded image function defined by I : Ω consider n = 2. Let g be an edge-detector map. The function g is applied to the norm of the image gradient, and satisfies the following properties: g : [0, ∞[→ [0, ∞[, g(0) = 1, g strictly decreasing, and limr→+∞ g(r) = 0. An example of 1 such a function is g : r  → 1+r 2 . We denote by W = (w1 , w2 ) = −Dg(||DI||) the

90

L. Guillot and C. Le Guyader

associated gradient vector field. In homogeneous regions, ||DI|| 0 so g(||DI||) is almost equal to 1. On boundaries, ||DI|| is large so g(||DI||) is almost zero. Also, in homogeneous regions, W is almost the null vector. Along the boundaries, the vector field W points toward the middle of the edges (see such an example in Fig. 1). We plan to extrapolate the vector field on the whole image domain in a variational framework. A majority of existing regularization functionals aims at minimizing the global variation of the unknown and thus provides little local control. In this work, we propose to minimize the super norm of the unknown gradient. This choice is also motivated by the fact that the Laplacian operator (that naturally appears in the GVF model) can be decomposed into the sum of the second derivative in the normal direction, and the second derivative in the tangent direction. The former component that is kept in the NGVF model weighs heavily in the extrapolation process and has good properties unlike the later component which proves to be parasitic particularly when dealing with thin and long concavities. Also, unlike prior related works, we reduce the number of unknowns to a single one by assuming that the expected vector field is the gradient vector field of a scalar function. We thus propose to minimize the following functional:  inf ||Dv − W ||2 ||W ||2 dx + μ ||Dv||L∞ (Ω) , (3) 1,∞ v∈W

(Ω)

Ω

where μ > 0 is a tuning parameter. Remark 1. Functional (3) is defined on W 1,∞ (Ω). The domain Ω being bounded, the inclusion L∞ (Ω) ⊂ L2 (Ω) holds so Dv ∈ L2 (Ω). Remark 2. If v is a minimizer of (3), so is v + C where C denotes any real constant. This is not a problem since we are interested in the associated gradient vector field. If v ∈ W 1,∞ (Ω), v is Lipschitz continuous and thus, by Rademacher’s theorem, differentiable almost everywhere. To minimize the above energy, we make use of the absolutely minimizing Lipschitz extensions. Following the results on AMLE recalled in Sect. 1, we obtain the Euler-Lagrange equation satisfied by v if it minimizes (3) and solve it by gradient descent. More precisely, classically, in image processing, the equation is defined on a domain R of IR2 (e.g., on the square [0, 1] × [0, 1]). In this case, boundary conditions must be defined: Neumann boundary conditions on ∂R are well-suited to the image processing framework since it corresponds to the reflection of the data through the edges. Thus it is no longer necessary to define boundary values. Following [6] and [15], we propose to simplify the problem by working with periodic solutions. The function v, primarily defined on [0, 1] × [0, 1], is extended to IR2 . First, by symmetry, we extend it to [−1, 1] × [−1, 1] and then in all of IR2 by periodicity (see Sect. 3.3.1 from [6]). We thus obtain that ∀h ∈ ZZ 2 , ∀x ∈ IR2 ,

Extrapolation of Vector Fields Using the Infinity Laplacian

91

v(x + 2h) = v(x). Also, we assume that the initial condition v0 and the functions x → wk (x), k = 1, 2 are extended to IR2 with the same periodicity. Given T > 0, we then obtain the following problem: ⎧   ∂v ⎪ ⎪ = 2 ||W ||2 Δv + 2  D||W ||2 , Dv − 2 div ||W ||2 W ⎪ ⎪ ∂t

⎪ ⎪ Dv Dv ⎪ 2 ⎨ , on IR2 × (0, T ), + μD v |Dv| |Dv| (4)

⎪ Dv Dv ⎪ ⎪ ⎪ , , = b(x)Δv −  d(x), Dv − h(x) + μ D 2 v ⎪ ⎪ |Dv| |Dv| ⎪ ⎩ 2 v(x, t = 0) = v0 (x) in IR ,   with b : x  → 2||W (x)||2 , d : x  → −2 D||W ||2 (x), h : x  → 2 div ||W ||2 W (x), and with the assumptions v0 ∈ C(IR2 ) ∩ W 1,∞ (IR2 ), b ∈ C(IR2 ) and bounded by ξb , d ∈ C(IR2 ) ∩ W 1,∞ (IR2 ), bounded by ξd and with Lipschitz constant κd , h ∈ C(IR2 ) ∩ W 1,∞ (IR2 ), bounded by ξh and with Lipschitz constant κh , and with ·, · denoting the euclidean scalar product in IR2 . We also assume that the mapping IR2  x  → b1/2 (x) is Lipschitz continuous on IR2 with Lipschitz constant κb1/2 .

3

Theoretical Results

This problem falls within the framework of the theory of viscosity solutions. Indeed, we obtain a second order singular degenerate parabolic equation. The concept of viscosity solutions has been introduced in 1981 by Crandall and Lions ( [22]). This theory was developed to study first-order partial differential equations of nondivergence form, typically, Hamilton-Jacobi equations. Later, the study of viscosity solutions was extended to second-order elliptic and parabolic equations (for a good introduction to the theory of viscosity solutions, we refer to Barles [8, 7], the article of Crandall, Ishii and Lions [21], Crandall, Lions [23], Ishii [26], and Ishii, Lions [27]). We also refer to the related work [9]. In our problem, the evolution equation in (4) can be rewritten in the form:   ∂v + G x, Dv, D2 v = 0, ∂t with G : IR2 × IR2 − {0IR2 } × S 2 (S 2 being the set of symmetric 2 × 2 matrices equipped with its natural partial order) defined by: p pT X , |p| |p| 

p p = d(x), p + h(x) − b(x) trace (X) − μ trace X , |p|2

G(x, p, X) = d(x), p + h(x) − b(x) trace (X) − μ

= c(x, p) + E (x, X) + F (p, X) , with the following properties:

92

L. Guillot and C. Le Guyader

 p p X 2 |p|

and E : (x, X)  →

if X ≤ Y then F (p, X) ≥ F (p, Y ).

(5)

1. The operators G, F : (p, X)  → −μ trace



−b(x) trace (X) are independent of v and are elliptic, i.e., ∀X, Y ∈ S 2 , ∀p ∈ IR2 ,

The operators G, E, and F are therefore proper. 2. F is locally bounded on IR2 × S 2 , continuous on IR2 \ {0IR2 } × S 2 , and F ∗ (0, 0) = F∗ (0, 0) = 0,

(6)

where F ∗ (resp. F∗ ) is the upper semicontinuous (usc) envelope (resp. lower semicontinuous (lsc) envelope) of F . 3. c : IR2 × IR2  (x, p)  → d(x), p + h(x) is locally Lipschitz continuous in space and ∀x, y ∈ IR2 × IR2 , |c(x, p) − c(y, p)| ≤ (κd |p| + κh ) |x − y|.

(7)

We start by proving a comparison principle that will be useful to prove the uniqueness of the viscosity solution of the considered problem. Theorem 1 (Comparison principle). Let u ∈ U SC(IR2 × [0, T )), bounded, periodic (with the same periodicity as the initial condition of (4)), be a subsolution and v ∈ LSC(IR2 × [0, T )), bounded, periodic (with the same periodicity as the initial condition of (4)), be a supersolution of (4). Assume that u0 (x) = u(x, 0) ≤ v0 (x) = v(x, 0) in IR2 , then u ≤ v in IR2 × [0, T ). Proof. This proof is rather classical. We follow the arguments of [21]. We first observe that for λ > 0, u ˜ = u − T λ−t is also a subsolution of (4) and u ˜t + G∗ (x, D˜ u, D 2 u ˜) ≤ −

λ λ ≤ − 2. 2 (T − t) T

Since u ≤ v follows from u ˜ ≤ v in the limit λ → 0, it will simply suffice to prove the comparison under the additional assumptions: ⎧ ⎨ (i) u + G x, Du, D2 u ≤ − λ . t ∗ T2 (8) ⎩ (ii) lim u(x, t) = −∞. t→T

Let us set M = supIR2 ×[0,T ) u(x, t) − v(x, t). We aim to show that M ≤ 0. In this purpose, we argue by contradiction and assume that M > 0. We introduce the duplication function f (x, − (4ε)−1|x − y|4 and consider  y, t) = u(x, t) − v(y, t) −1 M0 = supIR2 ×IR2 ×[0,T ) u(x, t) − v(y, t) − (4ε) |x − y|4 , ε > 0. Obviously, M0 ≥ M > 0. Moreover, this supremum is reached owing to the bound above of u and −v, the fact that f is such that ∀h ∈ ZZ 2 , f (x + 2h, y + 2h, t) = f (x, y, t), and (8)(ii). We denote by (x0 , y0 , t0 ) ∈ IR2 × IR2 × [0, T ) a point of maximum. We first prove that t0 > 0 for ε small enough and then rise a contradiction using Th. 8.3 from [21], which allows to conclude that M ≤ 0. Consequently, u ≤ v in IR2 × [0, T ).  

Extrapolation of Vector Fields Using the Infinity Laplacian

93

We now give an existence result using the classic Perron’s method (see Sect. 4 from [21]). We start by constructing a subsolution U − . Let us set U − = inf IR2 (v0 ) − Ct with C = ξh . U − is twice differentiable in space, once differentiable in time, bounded, and periodic with the same periodicity as v0 and U − is a subsolution of (4). Similarly, U + = supIR2 (v0 ) + Ct is a supersolution of (4). Obviously, U − (x, 0) ≤ U + (x, 0). We can define: v = sup {w; w periodic with the same periodicity as v0 , subsolution such that U − ≤ w ≤ U + } . In that case, Perron’s method states that v is a periodic discontinuous solution of (4) with the same periodicity as v0 . Clearly, the solution is bounded since U + is bounded. Also as v is a solution, v ∗ is a subsolution and v∗ a supersolution so from the comparison principle v ∗ ≤ v∗ . But v∗ ≤ v ∗ so v ∗ = v∗ = v, which gives that v is continuous on IR2 × [0, T ). Conclusion 1. We have proved the existence and uniqueness of a bounded, periodic, continuous on IR2 × [0, T ) viscosity solution of (4). We now prove that a solution of (4) is Lipschitz continuous in space, and uniformly continuous in time. Theorem 2 (Regularity results). Let us assume that ||Dv0 ||L∞ (IR2 ) ≤ B0 with B0 > 0. Then the solution of (4) satisfies: ||Dv(·, t)||L∞ (IR2 ) ≤ B(t), αt

with B(t) = κh e

−1 α

+ B0 eαt , and with α = 8κ2b1/2 + κd .

Proof. The function v is bounded, continuous on IR2 × [0, T ), and periodic with 1  the same periodicity as v0 . We set Φε (x, y, t) = B(t) |x − y|2 + ε2 2 and aim at proving that v(x, t) − v(y, t) ≤ Φε (x, y, t). Let us set M = sup(x,y)∈IR2 ×IR2 , t∈[0,T ) (v(x, t) − v(y, t) − Φε (x, y, t)). We thus aim to show that M ≤ 0. Once again, we argue by contradiction and assume that M > 0. So we conclude that v(x, t) − v(y, t) ≤ Φε (x, y, t) and letting ε tend to 0, one obtains: v(x, t) − v(y, t) ≤ B(t)|x − y|. Exchanging x and y yields: |v(x, t) − v(y, t)| ≤ B(t)|x − y|.

 

Theorem 3 (Regularity results). The solution v is uniformly continuous in time.

94

L. Guillot and C. Le Guyader

Proof. We proceed like in [25]. In a first time, we assume that v0 is bounded, periodic, C 2 , and such that there exists C, ||Dv0 ||L∞ (IR2 ) , ||D2 v0 ||L∞ (IR2 ) ≤ C. Let us set   C1 = sup ζ + E(x, D2 v0 ) + F∗ (Dv0 , D2 v0 ), ζ − E(x, D2 v0 ) − F ∗ (Dv0 , D2 v0 ) x∈IR2

with ζ = ξd ||Dv0 ||L∞ (IR2 ) + ξh . Let us also set v − = v0 − C1 t and v + = v0 + C1 t. It can be checked that v − is a subsolution of (4) and v + is a supersolution. Then, there exists a unique solution v of (4) and, by the comparison principle, it yields: ∀x ∈ IR2 , ∀t ∈ [0, T ), |v(x, t) − v0 (x)| ≤ C1 t. Letting u(x, t) = v(x, t + h), we obtain that u is the solution of  ∂u + G(x, Du, D2 u) = 0 . ∂t u(x, t = 0) = v(x, h) Classical arguments (comparison principle) allow to conclude that |u(x, t) − v(x, t)| ≤ C1 h, that is |v(x, t + h) − v(x, t)| ≤ C1 h. So v is uniformly continuous in time. Then we assume that v0 is only bounded, periodic and Lipschitz continuous, and use mollification (see Chap. IV from [11] and Sect. 2.5 from [6]). Using the first step of the proof, we obtain the result and the modulus of continuity of v which depends on B0 .   Conclusion 2. We have proved the existence and uniqueness of a viscosity solution of problem (4), bounded, periodic, continuous on IR2 × [0, T ), Lipschitz continuous in space so differentiable almost everywhere, and uniformly continuous in time. We now discretize the evolution equation. In the sequel, we set Ω  x = (x1 , x2 ).

4

Experimental Results

Let Δx1 and Δx2 be the spatial steps, Δt be the time step and (x1i , x2j ) = (iΔx1 , jΔx2 ) be the grid points, 1 ≤ i ≤ M and 1 ≤ j ≤ N . For a function Ψ : Ω → IR, let Ψijn = Ψ (iΔx1 , jΔx2 , nΔt). To discretize (4), we use an explicit finite difference scheme as follows. Also, the problem is complemented by Neumann boundary conditions. For the discretization of the convection component, we refer to [36] (we have used the usual notations for the finite difference operators and the notation d = (d1 , d2 )).   n+1 n n n = vi,j + Δt bi,j Dx1 x1 vi,j + Dx2 x2 vi,j vi,j  x1 n   x1 n  vi,j + min (d1 )i,j , 0 D+ vi,j (9) −Δt max (d1 )i,j , 0 D−

  x2 n   x2 n +max (d2 )i,j , 0 D− vi,j + min (d2 )i,j , 0 D+ vi,j − Δt hi,j +Δt μ

n n 2 n n n n n 2 Dx1 x1 vi,j (Dx1 vi,j ) +2Dx1 vi,j Dx2 vi,j Dx1 ,x2 vi,j +Dx2 x2 vi,j (Dx2 vi,j ) n 2 n 2 (Dx1 vi,j ) +(Dx2 vi,j ) +ε

.

Extrapolation of Vector Fields Using the Infinity Laplacian

95

Fig. 1. On the left, depiction of the initial gradient vector field W = −Dg(||DI||), on the right, the obtained vector field with our proposed approach (μ = 0.05, Δt = 0.1)

Fig. 2. On the left, depiction of the initial gradient vector field W = −Dg(||DI||), on the right, the obtained vector field with our proposed approach (μ = 0.1, Δt = 0.1)

4.1

Numerical Experimentations of Extrapolation

The experiments have been performed on a 2.21 GHz Athlon with 1.00 GB of RAM. In all our experiments, Δx1 = Δx2 = 1. We apply our model to real data and for each test, we provide a view of the initial gradient vector field −Dg(||DI||) and a view of the extrapolated vector field. The initialization was made either by setting v0 ≡ 0, or by setting v0 ≡ −g(||DI||). In all the tests we performed, it does not seem to influence the obtained result. The number of iterations as well as the computational time (order of the second) are similar for the three methods (GVF, NGVF and our proposed approach). Our method qualitatively performs in a way similar to the GVF and the NGVF: we increase the capture range of the vector field and we obtain downward components within the boundary concavity. Nevertheless, contrary to the the GVF and NGVF models, the method requires only one unknown. We start with an image taken from the Image Toolbox of Matlab (Fig. 1), and with an image showing a slice of Tuffeau

96

L. Guillot and C. Le Guyader

Fig. 3. Steps of the segmentation of the synthetic image taken from [34]

Fig. 4. Steps of the segmentation of the image of the brain

Extrapolation of Vector Fields Using the Infinity Laplacian

97

(Fig. 2, Courtesy of ISTO/ESRF). Our proposed approach performs well but seems to be sensitive to the textures of the objects contained in the image. 4.2

Application to Segmentation

This part is dedicated to segmentation and more precisely to the integration of this extrapolated vector field in the geodesic active contour model, in order to alleviate the constraint on the choice of the initial condition. The geodesic active contour model, introduced by Caselles et al. in [16], is cast in the level set setting developed by Osher and Sethian in [33]. We propose, as done in [34], to replace W = −Dg(||DI||) of the geodesic active contour model by the extrapolated vector field obtained with our proposed approach. To illustrate this, we propose an example taken from [34]. It demonstrates that the initial condition can be made of several contours selected inside, outside or across the boundaries of interest, provided the initial curves contain part of the skeleton of the extrapolated vector field. The classical geodesic active contour model does not authorize this flexibility in the initialization step and therefore the method alone would fail to detect all the shapes. Of course, the proposed method cannot detect automatically interior contours but this drawback is overcome, still with the flexibility in the initialization step. We illustrate this remark with Fig. 4 that represents a slice of the brain (Courtesy of the Laboratory Of Neuro Imaging, UCLA).

5

Conclusion

This paper was devoted to the theoretical study of a new method to extrapolate vector fields using the infinity Laplacian and with applications to image processing. Contrary to prior related works, the number of unknowns is reduced to a single one. The problem is phrased in a variational framework and the EulerLagrange equation is then derived. It is solved using a gradient descent method, which leads to a parabolic problem that falls within the viscosity solution theory framework. The existence and uniqueness of a viscosity solution continuous in space and time, Lipschitz continuous in space and uniformly continuous in time is established. The theoretical study is complemented by several numerical experimentations, first dedicated to the extrapolation problem, and then extended to the segmentation problem. The experimentations show that the proposed approach performs well, even if in strong concavities the results are slightly less accurate than with the NGVF. The model is sensitive to the geometry of the boundaries and to the textures present in the images. In the segmentation framework, the introduction of this new force field allows to widen the choice of the initial condition.

References 1. Aronsson, G.: Minimization problems for the functional supx F (x, f (x), f  (x)). Arkiv für Mate. 6, 33–53 (1965) 2. Aronsson, G.: Minimization problems for the functional supx F (x, f (x), f  (x)). II. Arkiv für Mate. 6, 409–431 (1966)

98

L. Guillot and C. Le Guyader

3. Aronsson, G.: Extension of functions satisfying Lipschitz conditions. Arkiv für Mate. 6(6), 551–561 (1967) 4. Aronsson, G.: On the partial differential equation u2x uxx + 2ux uy uxy + u2y uyy = 0. Arkiv für Mate. 7, 395–425 (1968) 5. Aronsson, G., Crandall, M., Juutinen, P.: A tour of the theory of absolutely minimizing functions. Bull. Amer. Math. Soc. (N.S.) 41, 439–505 (2004) 6. Aubert, G., Kornprobst, P.: Mathematical Problems in Image Processing: Partial Differential Equations and the calculus of Variations. Springer, Heidelberg (2002) 7. Barles, G.: Solutions de viscosité des équations de Hamilton-Jacobi. Springer, Heidelberg (1994) 8. Barles, G.: Solutions de viscosité et équations elliptiques du deuxième ordre. Cours de DEA (1997) 9. Barles, G., Busca, J.: Existence and comparaison results for fully nonlinear degenerate elliptic equations without zeroth-order term. Comm. Partial Differential Equations 26, 2323–2337 (2001) 10. Barron, E.N., Evans, L.C., Jensen, R.: The infinity Laplacian, Aronsson’s equation and their generalizations. Trans. Amer. Math. Soc. 360(1), 77–101 (2008) 11. Brézis, H.: Analyse fonctionnelle. Dunod (1999) 12. Canny, J.: A computational approach to edge detection. IEEE Trans. Pattern Anal. Mach. Intell. 8, 679–698 (1986) 13. Carlsson, S.: Sketch based coding of grey level images. Signal Process. 15, 57–83 (1988) 14. Casas, J.R.: Image compression based on perceptual coding techniques. Ph.D. dissertation, Dept. Signal Theory Commun., UPC, Barcelona, Spain (1996) 15. Caselles, V., Catté, F., Coll, C., Dibos, F.: A geometric model for active contours in image processing. Numer. Math. 66, 1–31 (1993) 16. Caselles, V., Kimmel, R., Sapiro, G.: Geodesic Active Contours. Int. J. Comput. Vision 22(1), 61–87 (1997) 17. Caselles, V., Morel, J.M., Sbert, C.: An Axiomatic Approach to Image Interpolation. IEEE Trans. Image Process. 7(3), 376–386 (1998) 18. Cohen, L.D.: On Active Contour Models and Balloons. CVGIP: Image Understanding 53(2), 211–218 (1989) 19. Cong, G., Esser, M., Parvin, B., Bebis, G.: Shape Metamorphism Using p-Laplacian Equation. In: ICPR, vol. 4, pp. 15–18 (2004) 20. Crandall, M.G.: A visit with the ∞-Laplace equation. Preprint, Notes from a CIME course (2005) 21. Crandall, M.G., Ishii, H., Lions, P.L.: User’s guide to viscosity solutions of second order partial differential equations. Bull. Amer. Math. Soc. 27, 1–67 (1992) 22. Crandall, M.G., Lions, P.L.: Viscosity solutions of Hamilton-Jacobi Equations. Trans. Amer. Math. Soc. 277, 1–42 (1983) 23. Crandall, M.G., Lions, P.L.: On existence and uniqueness of solutions of HamiltonJacobi equations. Non-Linear Anal. 10, 353–370 (1986) 24. Elion, C., Vese, L.A.: An image decomposition model using the total variation and the infinity Laplacian. In: Proceedings SPIE, vol. 6498, pp. 64980W-1–64980W-10 (2007) 25. Forcadel, N.: Dislocations dynamics with a mean curvature term: short time existence and uniqueness. Differential and Integral Equations 21(3-4), 285–304 (2008) 26. Ishii, H.: Existence and uniqueness of solutions of Hamilton-Jacobi equations. Funkcial. Ekvac. 29, 167–188 (1986) 27. Ishii, H., Lions, P.L.: Viscosity solutions of fully nonlinear second-order elliptic partial differential equations. J. Differ. Equations 83, 26–78 (1990)

Extrapolation of Vector Fields Using the Infinity Laplacian

99

28. Jensen, R.: Uniqueness of Lipschitz extensions minimizing the sup-norm of the gradient. Arch. Rat. Mech. Anal. 123(1), 51–74 (1993) 29. Jifeng, N., Chengke, W., Shigang, L., Shuqin, Y.: NGVF: An improved external force field for active contour model. Pattern Recogn. Lett. 28, 58–63 (2007) 30. Kass, M., Terzopoulos, D., Witkin, A.: Snakes: Active contour models. Int. J. Comput. Vision 1, 321–331 (1988) 31. Mémoli, F., Sapiro, G., Thompson, P.: Brain and surface warping via minimizing Lipschitz extensions. In: MFCA, International Workshop on Mathematical Foundations of Computational Anatomy (2006) 32. Meyer, Y.: Oscillating Patterns in Image Processing and Nonlinear Evolution Equations. AMS 22 (2001) 33. Osher, S., Sethian, J.A.: Fronts propagation with curvature dependent speed: Algorithms based on Hamilton-Jacobi formulations. J. Comput. Phys. 79, 12–49 (1988) 34. Paragios, N., Mellina-Gottardo, O., Ramesh, V.: Gradient Vector Flow Fast Geodesic Active Contours. In: Proc. IEEE Intl. Conf. Computer Vision, vol. 1, pp. 67–73 (2001) 35. Prewitt, J.M.S.: Object enhancement and extraction. In: Lipkin, B., Rosenfeld, A. (eds.) Picture Processing and Psychopictorics, pp. 75–149. Academic Press, New York (1970) 36. Sethian, J.A.: Level Set Methods and Fast Marching Methods: Evolving interfaces in Computational Geometry. In: Fluid Mechanics, Computer Vision and Material Science. Cambridge University Press, Londres (1999) 37. Torre, V., Poggio, T.A.: On edge detection. IEEE Trans. Pattern Anal. Mach. Intell. 8, 147–163 (1986) 38. Xu, C., Prince, J.L.: Snakes, shapes, and gradient vector flow. IEEE Trans. Image Process. 7(3), 359–369 (1998) 39. Yuille, A.L., Poggio, T.A.: Scaling theorems for zero-crossings. IEEE Trans. Pattern Anal. Mach. Intell. 8, 15–25 (1986)

A Schrödinger Equation for the Fast Computation of Approximate Euclidean Distance Functions Karthik S. Gurumoorthy and Anand Rangarajan Dept. of CISE, University of Florida, Gainesville, FL, USA

Abstract. Computational techniques adapted from classical mechanics and used in image analysis run the gamut from Lagrangian action principles to HamiltonJacobi field equations: witness the popularity of the fast marching and fast sweeping methods which are essentially fast Hamilton-Jacobi solvers. In sharp contrast, there are very few applications of quantum mechanics inspired computational methods. Given the fact that most of classical mechanics can be obtained as a limiting case of quantum mechanics (as Planck’s constant h tends to zero), this paucity of quantum mechanics inspired methods is surprising. In this work, we derive relationships between nonlinear Hamilton-Jacobi and linear Schrödinger equations for the Euclidean distance function problem (in 1D, 2D and 3D). We then solve the Schrödinger wave equation instead of the corresponding HamiltonJacobi equation. We show that the Schrödinger equation has a closed form solution and that this solution can be efficiently computed in O(N log N ), N being the number of grid points. The Euclidean distance can then be recovered from the wave function. Since the wave function is computed for a small but non-zero h, the obtained Euclidean distance function is an approximation. We derive analytic bounds for the error of the approximation and experimentally compare the results of our approach with the exact Euclidean distance function on real and synthetic data.

1 Introduction Image analysis [1,2] has a tradition of importing and adapting a host of classical physics based approaches including Lagrangian based variational principles and their associated Euler-Lagrange equations [3], Hamiltonian dynamics [4] and more recently HamiltonJacobi based methods [5]. Approaches in image analysis do not strictly adhere to the classical mechanics sequence [6] of i) first specifying a Lagrangian action principle, ii) deriving the corresponding Euler-Lagrange equation, iii) employing a Legendre transformation to convert the Lagrangian dynamics to a first-order Hamiltonian dynamics, and finally, iv) employing a canonical transformation to derive the Hamilton-Jacobi equation whose solution also yields a solution to the original variational problem. Instead, most research in image analysis uses a combination of one or more of these four approaches depending on the problem at hand. For example, in surface reconstruction [3], a popular approach consists of writing a variational form and then finding a solution using preconditioned conjugate gradient or quasi-Newton type iterative methods. While we notice a plethora of classical mechanics inspired techniques in image analysis, the same cannot be said about quantum mechanics. Despite the well known fact X.-C. Tai et al. (Eds.): SSVM 2009, LNCS 5567, pp. 100–111, 2009. c Springer-Verlag Berlin Heidelberg 2009 

A Schrödinger Equation for the Fast Computation

101

that most of classical mechanics is a limiting case of quantum mechanics as Planck’s constant  → 0 [7], there is very little application of quantum mechanical principles in image analysis problems. Rather than speculate on the reasons for this dearth of applications, we wish to point out that in this paper, we are primarily interested in exploiting a concrete relationship between the classical, non-linear Hamilton-Jacobi equation and the quantum, linear Schrödinger equation. We feel that focusing more narrowly on this relationship (which will become more obvious as we proceed) is more productive than dwelling on the more mysterious and specifically quantum mechanical issues of i) interpretation of the wave function, ii) role of probabilities and, iii) the problem of measurement. While these issues are certainly important, they do not play any role in this paper. In summary, we are mainly interested in exploiting the relationship between the Schrödinger and Hamilton-Jacobi equations in order to derive computationally efficient algorithms which are applicable in image analysis problems where Hamilton-Jacobi theory is used. In the theoretical physics literature, a Schrödinger wave equation at the energy state h E has the form ψ(X, t) = φ(X) exp( iEt  ), ( ≡ 2π ) [8], where φ(X)—the stationary state wave function—is the eigenstate of the Hamiltonian operator H corresponding to the eigenvalue E. When the Hamilton-Jacobi scalar field S ∗ appears as the exponent ∗ of the stationary state wave function, specifically φ(X) = exp( −S (X) ), and if φ(X) satisfies the linear Schrödinger equation, namely Hφ = Eφ, we show that as  → 0, S ∗ satisfies the Hamilton-Jacobi equation for a carefully chosen problem. The novel aspect is that a nonlinear Hamilton-Jacobi equation is obtained in the limit as  → 0 of a linear Schrödinger equation. Consequently, instead of solving the Hamilton-Jacobi equation, one can solve the Schrödinger wave equation (taking advantage of its linearity), and then compute an approximate S ∗ for small values of . This computational procedure would be approximately equivalent to solving the original Hamilton-Jacobi equation. With the basic setup in place, we now turn our attention to an actual application. Our goal is to apply the Schrödinger formalism to a well known problem that has been successfully attacked by Hamilton-Jacobi theory. To this end, we choose the problem of computing Euclidean distance functions on a grid for a given set of points where the task is to assign at each grid point a value corresponding to the Euclidean distance to its nearest neighbor from the given point-set. The literature is replete with elegant and pioneering works which have successfully solved this problem. To name a few, the well known fast marching [9] and fast sweeping [10] methods are essentially O(N log N ) Hamilton-Jacobi based algorithms where N is the number of grid points. [The fast sweeping method has an added advantage in that the algorithm appears (empirically)1 to be O(N ).] These techniques focus on directly solving the non-linear eikonal equation [4], whereas our approach shows that one can instead solve a linear equation and obtain the solution to the non-linear eikonal equation in the limit as  → 0. The important connection between the Schrödinger wave equation and the Hamilton-Jacobi equation [7] is illuminated during the process. In the more traditional (computer science) algorithms literature, computational geometry inspired techniques like Voronoi diagrams and KD-Trees [11] have also solved this problem. However, computing Voronoi 1

After a careful reading of [10], it is still unclear to us if the fast sweeping method is formally rather than empirically O(N ).

102

K.S. Gurumoorthy and A. Rangarajan

diagrams or building data structures for KD-Trees in 3D and higher dimensions is expensive and the O(N log N ) complexity is not retained at higher dimensions [11]. Our technique (which as we shall see is based on application of the fast Fourier transform (FFT) [12]) is very simple and elegant and remains O(N log N ) irrespective of the spatial dimension. The basic question asked and answered in this paper is: Can we design a Schrödinger equation that computes an exponentiated Euclidean distance function on a grid? The distance function is obtained from the exponent of the Schrödinger wave function. A naïve approach to solve this Euclidean distance problem would be to visit every grid point and compute the Euclidean distance to all members of the point-set and pick the smallest distance. The complexity of this naïve approach is obviously related to the product of the number of grid points and the cardinality of the point-set. The fast sweeping and fast marching methods avoid this naïve complexity and as we shall see, so does the Schrödinger wave function approach.

2 Euclidean Distance Functions We now describe the Euclidean distance function problem. Given a point-set Y = {Yk ∈ RD , k ∈ {1, . . . , K}} where D is the dimensionality of the point-set and a set of equally spaced Cartesian grid points X, the Euclidean distance function problem requires us to assign S ∗ (X) = min X − Yk  (1) k

with the Euclidean norm used in (1). If efficient computation is a non-issue, this is a simple problem. We visit each grid point in the set X and compute the distance to every point Yk , ∀k ∈ {1, . . . , K} and assign S ∗ (X) the minimum distance. If the number of grid points is N , the naïve complexity is O(N KD). 2.1 Hamilton-Jacobi Formulation The Hamilton-Jacobi equation approach to the Euclidean distance function problem stems from considering the following variational problem which in 2D is  I[q] =

t1

L(q1 , q2 , q˙1 , q˙2 , t)dt

(2)

to

where the Lagrangian L is defined as L(q1 , q2 , q˙1 , q2˙, t) ≡

1 2 (q˙1 + q˙2 2 ). 2

Defining pi ≡ ∂∂L q˙i and applying a Legendre transformation [6] [by inverting pi = to get the function q˙i (q1 , q2 , p1 , p2 , t)], we define the Hamiltonian of the system as

(3) ∂L ∂ q˙i

H(q1 , q2 , p1 , p2 , t) ≡ p1 q˙1 + p2 q˙2 − L(q1 , q2 , q˙1 (q1 , q2 , p1 , p2 , t), q˙2 (q1 , q2 , p1 , p2 , t)) (4)

A Schrödinger Equation for the Fast Computation

103

which for the Euclidean distance function problem turns out to be H(q1 , q2 , p1 , p2 , t) =

1 2 (p + p22 ). 2 1

(5)

The Hamilton-Jacobi equation is obtained via a canonical transformation [6] of the Hamiltonian. In the 2D case, it is ∂S ∂S ∂S + H(q1 , q2 , , , t) = 0 ∂t ∂q1 ∂q2

(6)

where we have replaced the generalized momentum variables pi with ∂S ∂L = pi = . ∂qi ∂ q˙i

(7)

Since the Hamiltonian in (5) is a constant independent of time, equation (6) can be simplified to the static Hamilton-Jacobi equation. By separation of variables, we get S(q1 , q2 , t) = S ∗ (q1 , q2 ) − Et

(8)

∗

where E is the total energy of the system and S (q1 , q2 ) is called the Hamilton’s characteristic function [13]. Using the definition of pi from (7) in (5) and observing that ∂S ∂S ∗ ∂qi = ∂qi , we get  2  ∗ 2  1 ∂S ∗ ∂S = E. (9) + 2 ∂q1 ∂q2 Choosing the energy E to be 12 , we obtain   S∗ = 1

(10) ∗

which is the well known eikonal equation [4] where the forcing term is 1. S is the required Hamilton-Jacobi scalar field which is efficiently obtained by the fast sweeping and fast marching methods. 2.2 Schrödinger Wave Equation for Euclidean Distance Functions We now derive and solve a Schrödinger wave equation for this Euclidean distance function problem instead of solving the non-linear eikonal equation. The Schrödinger wave equation is written as [8] i

∂ψ = Hψ ∂t

(11)

where ψ(X, t) is the wave function and H is the Hamiltonian operator obtained by first ∂ quantization2—where the momentum variables pi are replaced with the operator i ∂x . i Using the definition of our Hamiltonian from (5), ψ satisfies (in 2D)   2 ∂ 2 ψ ∂ 2ψ  ∂ψ = . (12) + i ∂t 2 ∂x21 ∂x22 2

First quantization is still mysterious. For an informal but illuminating treatment, please see http://math.ucr.edu/home/baez/categories.html

104

K.S. Gurumoorthy and A. Rangarajan

Using separation of variables ψ(X, t) = φ(X)f (t), we get 2 2 φ  f˙ = =E (13) if 2 φ where E is the energy state of the system. By choosing the energy of the system to be 1 2 as before, we get   it (14) f (t) = exp 2 and hence   it (15) ψ(X, t) = φ(X) exp 2 where φ(X) satisfies the equation 2 2 φ = φ.

(16)

2.3 Eikonal Equation for Euclidean Distance Functions ∗

∗ We now show that when φ = exp( −S  ) and satisfies (16), S asymptotically satisfies the eikonal equation (10) as  → 0. We show this for the 2D case but the generalization to higher dimensions is straightforward. ∗ 1 ,x2 ) When φ(x1 , x2 ) = exp( −S (x ), the first partials of φ are      ∗ 1 1 ∂φ −S ∂S ∗ ∂φ −S ∗ ∂S ∗ = − exp , = − exp . (17) ∂x1   ∂x1 ∂x2   ∂x2

The second partials needed for the Laplacian are   ∗ 2    1 1 ∂2φ ∂S −S ∗ ∂ 2 S ∗ −S ∗ exp = exp − , ∂x21 2  ∂x1   ∂x21   ∗ 2    1 1 ∂S −S ∗ ∂ 2 S ∗ ∂2φ −S ∗ = 2 exp − exp . ∂x22   ∂x2   ∂x22 From this, equation (16) can be rewritten as  ∗ 2  ∗ 2   2 ∗ ∂2S ∗ ∂S ∂S ∂ S =1 + − + ∂x1 ∂x2 ∂x21 ∂x22

(18)

(19)

which in simplified form is   S ∗ 2 −  2 S ∗ = 1.

(20)

The additional  2 S ∗ term [relative to (10)] is referred to as the viscosity term [9]. (Note that this term emerges naturally from the Schrödinger equation derivation—an intriguing result.) Since | 2 S ∗ | is bounded, as  → 0, (20) tends to   S∗ = 1

(21)

which is the original eikonal equation (10) for the Euclidean distance function. This relationship motivates us to solve the linear Schrödinger equation (16) instead of the non-linear eikonal equation and then compute the distance function via S ∗ (X) = − log φ(X).

(22)

A Schrödinger Equation for the Fast Computation

105

3 Closed Form Solutions for the Approximate Euclidean Distance Function and Proofs of Convergence We now derive the closed form solution for φ(X) (in 1D, 2D and 3D) satisfying equation (16) and hence for S ∗ (X) by (22) and observe that we get the actual Euclidean distance function in the limit as  → 0. In order to satisfy the condition that S ∗ (Yk ) = 0,∀Yk , k ∈ {1, . . . , K}, we consider the forced version of the equation (16) which is −2 2 φ + φ =

K 

δ(X − Yk ).

(23)

k=1

Using a Green’s function approach [14] (where the form of the solution depends on the number of spatial dimensions), we can write expressions for the solution φ. Below, let r = mink X − Yk —the actual Euclidean distance function at the grid point X. 1D: In 1D, the solution [14] for φ is 1  exp 2 K

φ(X) =



k=1

−|X − Yk | 

 .

(24)

Using the relationship in (22), we get ∗

S (X) = − log

K 

 exp

k=1

Observe that

−|X − Yk |  

−r S (X) ≤ − log exp  = r +  log(2). ∗

Also,

 +  log (2) .

(25)

 +  log(2)

   −r +  log(2) S ∗ (x) ≥ − log K exp  = − log K + r +  log(2).

(26)

(27)

As  → 0,  log K → 0 and  log  → 0. Furthermore, we see from (26) and (27) that lim S ∗ (X) = r.

(28)

  K 1  X − Yk  φ(X) = K0 2π2 

(29)

→0

2D: In 2D, the solution [14] for φ is

k=1

106

K.S. Gurumoorthy and A. Rangarajan

where K0 is the modified Bessel function of the second kind. Using (22), we get S ∗ (X) = − log

K 

 K0

k=1

X − Yk  

Then,

r

S ∗ (X) ≤ − log K0 Using the relation K0 ( hr ) ≥

r exp(− h )

√r

when

h

 ∗

S (X) ≤ − log = − log Moreover

 r h

 exp r



+  log(2π2 ).

+  log(2π2 ).

(30)

(31)

≥ 0.5, we get 

−r 



+  log(2π2 )

 + r +  log(2π2 ). r

   −r +  log(2π2 ). S ∗ (X) ≥ − log KK0 

(32)

(33)

r Using the relation K0 ( r ) ≤ exp( −r  ) when h ≥ 1.5, we get    −r +  log(2π2 ) S ∗ (X) ≥ − log K exp 

= − log K + r +  log(2π2 ).

(34)

As  → 0,  log K → 0,  log r → 0 and  log  → 0. Furthermore, we see from (32) and (34) that lim S ∗ (X) = r. (35) →0

3D: In 3D, the solution [14] for φ is based on the modified spherical Bessel function of the second kind: 

K exp −X−Yk    1 φ(X) = . (36) 4π2 X − Yk  k=1

Using (22), S ∗ (X) = − log

K exp  k=1



−X−Yk  

X − Yk 



 +  log 4π2 .

(37)

Then, ∗

exp

−r 

 +  log(4π2 ) r

 = r +  log r +  log 4π2 .

S (X) ≤ − log

(38)

A Schrödinger Equation for the Fast Computation

Also,

 S ∗ (X) ≥ − log K

exp

−r   

r

107

+  log(4π2 )

= − log K + r +  log r +  log(4π2 ).

(39)

As  → 0,  log K → 0,  log r → 0 and  log  → 0. Furthermore, we see from (38) and (39) that lim S ∗ (X) = r. (40) →0

Hence, we have shown that (in 1D, 2D and 3D), the closed form solution for φ guarantees that S ∗ approaches the true Euclidean distance function in the limit  → 0.

4 Error Bound between the Obtained and True Euclidean Distance Function The solution for φ in 1D, 2D and 3D motivates us to compute the function   K  −X − Yk  ˜ exp φ(X) = 

(41)

k=1

(instead of computing φ) and then to compute the approximate distance function ˜ S˜∗ (X) = − log φ(X)

(42)

(instead of computing S ∗ ). The reasons are two-fold. Firstly, φ˜ can be computed efficiently in O(N log N ) time using the fast Fourier transform (FFT) [12] as explained in the subsequent section. Secondly, lim→0 S˜∗ (X) = r, as shown below, where r is the true Euclidean distance function value at the grid point X (r = mink X − Yk ), and this is the lim→0 S ∗ (X) as seen from the previous section. Hence, for small values of ∗ , S˜∗ (X) is a very good

approximation  to S (X).

 K −X−Yk  ˜∗ can be approximated as exp −r As  → 0, k=1 exp   . Hence S (X)

−r  ≈ − log exp  = r. The bound derived below between S˜∗ (X) and r also unveils the proximity between the computed and the actual Euclidean distance function. Note from (41) that   −r ∗ ˜ S (X) ≤ − log exp = r. (43)  Also, observe that

and hence,

   −r S˜∗ (X) ≥ − log K exp  = − log K + r

(44)

r − S˜∗ (X) ≤  log K.

(45)

108

K.S. Gurumoorthy and A. Rangarajan

From (43) and (45),

|r − S˜∗ (X)| ≤  log K. (46) ∗ ˜ Equation (46) shows that as  → 0, S (X) → r. It is worth commenting that the bound  log K is actually very tight as (i) it scales only as the logarithm of the cardinality of the point-set (K) and (ii) it can be made arbitrarily small by choosing a small but non-zero value of .

5 Efficient Computation of the Approximate Euclidean Distance Function The motivation for computing φ˜ instead of φ is the fact that the direct computation of φ at the N grid locations is O(N K) which is O(N 2 ) when the cardinality of the point-set is O(N ), whereas computing φ˜ at the N grid locations can be done in O(N log N ) using an FFT implementation [12]. The realization that FFT can be employed 

to compute K k φ˜ stems from the insight that the summation term, namely, k=1 exp −X−Y is 

 actually the discrete convolution between the functions f (X) = exp −X com puted at the grid locations, with the function g(X) which takes the value 1 at the pointset locations and 0 at other grid locations. By the convolution theorem [15], a discrete convolution can be obtained as the inverse Fourier transform of the product of two individual transforms, which for two O(N ) sequences can be computed in O(N log N ) time and hence φ˜ can be determined efficiently at the N grid locations. One just needs to compute the discrete Fourier transform (DFT) of the sampled version of the functions f (X) and g(X), compute their point-wise product and then compute the inverse discrete Fourier transform. Taking the logarithm of the inverse discrete Fourier transform and multiplying it by (−), gives the approximate Euclidean distance function. The algorithm is adumbrated in Table 1. Table 1. Approximate Euclidean distance function algorithm

 1. Compute the function f (X) = exp −X at the grid locations.  2. Define the function g(X) which takes the value 1 at the point-set locations and 0 at other grid locations. 3. Compute the FFT of f and g, namely F (u) and G(u) respectively. 4. Compute the function H(u) = F (u) ∗ G(u). ˜ 5. Compute the inverse FFT of H which gives φ(X) at the grid locations. ˜ 6. Take the logarithm of φ(X) and multiply it by (−) to get the approximate Euclidean distance function at the grid locations.

5.1 Computation of the Approximate Euclidean Distance Function in Higher Dimensions Our technique has a straightforward generalization to higher dimensions. Regardless of the spatial dimension, the approximate Euclidean distance function, S˜∗ can be computed by exactly following the steps delineated in the table above. It is worthwhile mentioning that computing the discrete Fourier transform using FFT is always O(N log N )

A Schrödinger Equation for the Fast Computation

109

irrespective of the spatial dimension. Hence, for all dimensions, S˜∗ can be computed at the given N grid points, in O(N log N ). This speaks for the scalability of our technique, which is generally not the case with other methods, for example KD-Trees [11].

6 Experiments In this section, we show the efficacy of our technique by computing the approximate Euclidean distance function S˜∗ and comparing it to the actual Euclidean distance function S, first on randomly generated 2D point-sets and then on a set of bounded 3D grid points. We began with a 2D grid consisting of points between (−30, −30) and (30, 30). Hence, the total number of grid points is N = 61 × 61 = 3721. We randomly chose around 1000 grid locations as data points (point-set). Then 50, 000 experiments were run for values of  ranging from 0.1 to 0.5 in steps of 0.01. The errorbar plot in Figure 1 shows the mean and standard deviation of the percentage error at each value of . The error is less than 0.5% at  = 0.1 demonstrating the algorithm’s ability to compute accurate Euclidean distances. Next, we took the Stanford bunny dataset3 and used the coordinates of the data points on the model as our point-set locations. Since the input data locations need not be at integer locations, we scaled the space uniformly in all dimensions and rounded off the data so that the data lies at integer locations. The input data was also shifted so that it was approximately symmetrically located with respect to the x, y and z axis. We should comment that shifting the data doesn’t affect the Euclidean distance function value and uniform scaling of all dimensions is also not an issue, as the distances can be rescaled once they are computed. After these basic data manipulations, the cardinality of the point set was K = 3019 with the data confined to the cubic region −16 ≤ x ≤ 16, −15 ≤ y ≤ 15 and −12 ≤ z ≤ 12. Our grid consisted of the set of all integer locations within this cubic region. The number of grid locations was N = 25575. We computed the Euclidean distance function value at each of these grid locations using our technique for different values of  and compared it to the true  Euclidean distance function value. −X At small values of , exp drops off very quickly and hence for grid locations  which are far away from the point-set, the convolution done using FFT needs to be precise (without round-off error) for the computed distance to be meaningful. Such high precision support may not be available and hence our technique may produce erroneous results at these grid locations for very small values of . But at those grid locations which are close to the point-set, the accuracy of the computed distance improves as  is decreased. Hence, to circumvent this problem of choosing , we ran our technique for different values of  and chose the distance function values obtained at large values of  at those grid locations whose average computed distance is larger and vice versa.

3

Go to http://www.cc.gatech.edu/projects/large_models/bunny.html to obtain this dataset.

110

K.S. Gurumoorthy and A. Rangarajan

percentage error

20

15

10

5

0

0.1

0.2

0.3 hbar values

0.4

0.5

Fig. 1. Percentage error versus  in 50,000 2D experiments

Fig. 2. Isosurfaces: (i) Left: Actual Euclidean distance function and (ii) Right: Our approach

When we ran our technique for the set of  ∈ {0.1, 0.2, 0.3, 0.4} and used {3, 6, 10} respectively as the threshold of the average computed distance for choosing the appropriate distance function, it gave the following set of results. The maximum absolute difference between the actual and the computed Euclidean distance value over all the grid locations is 0.9066 and the average absolute difference is 0.1322. The accuracy is fairly high since the furthest grid point from the point-set is at a distance of 15.5242 and the average overall distance computed at the grid locations from the point-set is 3.5449. The average error is 0.1322 3.5449 ∗ 100 = 3.72%. This error can be lowered by using higher precision numerical methods for convolution [16]. We plotted the isosurface obtained by connecting the grid points which are at a distance of 0.5 from the point set, determined both by the true Euclidean distance function and our technique. Figure 2 shows the two surfaces. Notice the similarity between the two plots. It provides anecdotal visual evidence for the usefulness of our approach.

7 Discussion In this paper, we have introduced a new approach to solving the non-linear eikonal equation (with a constant forcing term equal to 1). We have proved that the solution of the eikonal equation can be obtained as a limiting case of the solution to the corresponding linear Schrödinger wave equation. The key here is the embedding of the nonlinear Hamilton-Jacobi equation in a linear Schrödinger equation. Our Schrödinger wave equation formalism for solving the Euclidean distance function problem (which has been successfully attacked by pioneering Hamilton-Jacobi solvers such as the fast sweeping [10] and fast marching [9] methods) leverages this deep relationship between the two regimes of modern physics. In the future, we would like to solve the more

A Schrödinger Equation for the Fast Computation

111

general, static Hamilton Jacobi equation using techniques inspired from quantum mechanics as a counterpart to classical mechanics based techniques. In all likelihood, this will involve direct discretization of the Schrödinger wave equation which was not required for the Euclidean distance function problem. We expect that the linearity of the Schrödinger equation will result in fast algorithms even in this more general setting.

References 1. Horn, B.K.P.: Robot vision. MIT Press, Cambridge (1986) 2. Kimmel, R.: Numerical geometry of images: Theory, algorithms, and applications. Springer, Heidelberg (2003) 3. Grimson, W.E.L.: An implementation of a computational theory of visual surface interpolation. Computer Vision, Graphics, and Image Processing 22(1), 39–69 (1983) 4. Siddiqi, K., Tannenbaum, A., Zucker, S.W.: A Hamiltonian approach to the eikonal equation. In: Hancock, E.R., Pelillo, M. (eds.) EMMCVPR 1999. LNCS, vol. 1654, pp. 1–13. Springer, Heidelberg (1999) 5. Kao, C.Y., Osher, S.J., Tsai, Y.H.: Fast sweeping methods for static Hamilton-Jacobi equations. SIAM Journal on Numerical Analysis 42(6), 2612–2632 (2004) 6. Goldstein, H., Poole, C.P., Safko, J.L.: Classical mechanics. Addison-Wesley, Reading (2002) 7. Butterfield, J.: On Hamilton-Jacobi theory as a classical root of quantum theory. In: Elitzur, A., Dolev, S., Kolenda, N. (eds.) Quo-Vadis Quantum Mechanics, pp. 239–274. Springer, Heidelberg (2005) 8. Griffiths, D.J.: Introduction to quantum mechanics. Addison-Wesley, Reading (2004) 9. Osher, S.J., Sethian, J.A.: Fronts propagating with curvature dependent speed: algorithms based on Hamilton-Jacobi formulations. Journal of Computational Physics 79(1), 12–49 (1988) 10. Zhao, H.K.: A fast sweeping method for eikonal equations. Mathematics of Computation 74, 603–627 (2005) 11. de Berg, M., Cheong, O., van Kreveld, M., Overmars, M.: Computational geometry: Algorithms and applications. Springer, Heidelberg (2008) 12. Cooley, J.W., Tukey, J.W.: An algorithm for the machine calculation of complex Fourier series. Mathematics of Computation 19(90), 297–301 (1965) 13. Arnold, V.I.: Mathematical methods of classical mechanics. Springer, Heidelberg (1989) 14. Abramowitz, M., Stegun, I.A.: Handbook of mathematical functions with formulas, graphs, and mathematical tables. Government Printing Office, USA (1964) 15. Bracewell, R.N.: The Fourier transform and its applications. McGraw-Hill Science and Engineering, New York (1999) 16. Hida, Y., Li, H.S., Bailey, D.H.: Quad-double arithmetic: Algorithms, implementation, and application. Technical Report LBNL-46996, Lawrence Berkeley National Laboratory, Berkeley, CA 94720 (2000)

Semi-supervised Segmentation Based on Non-local Continuous Min-Cut Nawal Houhou1 , Xavier Bresson2 , Arthur Szlam2 , Tony F. Chan2 , and Jean-Philippe Thiran1 1

Signal Processing Laboratory 5, Swiss Federal Institute of Technology (EPFL), Lausanne, Switzerland 2 Department of Mathematics, University of California Los Angeles, CA 90095-1555, USA

Abstract. We propose a semi-supervised image segmentation method that relies on a non-local continuous version of the min-cut algorithm and labels or seeds provided by a user. The segmentation process is performed via energy minimization. The proposed energy is composed of three terms. The first term defines labels or seed points assigned to objects that the user wants to identify and the background. The second term carries out the diffusion of object and background labels and stops the diffusion when the interface between the object and the background is reached. The diffusion process is performed on a graph defined from image intensity patches. The graph of intensity patches is known to better deal with textures because this graph uses semi-local and non-local image information. The last term is the standard TV term that regularizes the geometry of the interface. We introduce an iterative scheme that provides a unique minimizer. Promising results are presented on synthetic textures a nd real-world images.

1

Introduction

Image segmentation is an important problem in image processing. The objective of segmentation algorithms is to partition an image into a finite number of semantically important regions such as anatomical or functional structures in medical images or objects in natural images. Well-posed approaches to solve the image segmentation problem are energy minimization methods. This paper introduces an energy minimization algorithm to solve the semi-supervised segmentation problem based on the continuous min cut/max flow model originally defined by Strang in [1]. Semi-supervised segmentation models defined in a continuous setting have already been proposed in the literature. Among them, Protiere and Sapiro proposed in [2] an interactive algorithm for segmentation. Cremers et al. introduced in [3] an algorithm based on the level set method to perform interactive image segmentat ion. Appleton and Talbot introduced in [4] a semi-supervised segmentation model based on the continuous min-cut of Strang. Unger et al. defined in [5] a segmentation method also based on the min-cut model of Strang in [1]. The semi-supervised segmentation models using the continuous min-cut are based on local image information. These models X.-C. Tai et al. (Eds.): SSVM 2009, LNCS 5567, pp. 112–123, 2009. c Springer-Verlag Berlin Heidelberg 2009 

Semi-supervised Segmentation Based on Non-local Continuous Min-Cut

113

s.a. [4,5] perform very well for the segmentation of smooth regions, but they are less efficient with textures. In this paper, we extend the continuous min cut to a non-local formulation along the same line as non-local means defined by Buades, Coll and Morel in [6] and the variational non-local means model of Gilboa and Osher [7]. This non-local extension of the continuous min cut can be obtained in different ways. We used the original discrete min-cut model [8] to define the non-local continuous min-cut, which turns to be the H 1 norm defined on graph and a term that constraints the labels. The H 1 norm carries out the diffusion of object and background labels on the graph of image patches [9, 6], which holds semi-local and non-local image information that can better segment textures and real-world objects. Besides, the continuous formulation of the min-cut algorithm allows us introducing other regularization processes such as the TV energy. The TV energy is indeed useful to regularize the boundary between the object and the background. Besides, the TV energy can smooth out the segmentation of small sets favored by the min-cut algorithm as noticed by Shi and Malik in [10] (see Figure 1).

2

Graph, Min-Cut and Diffusion

Graph representation. Let G = (V, E) be a weighted undirected graph, where the sets V are the graph nodes and E the edges connection nodes. In this paper, each node Vi represents a pixel i in an image I with support Ω ⊂ Rn where typically n = {2, 3}. The similarity between two pixels/nodes i and j in Ω is measured by the edge function on the graph, namely wij . In the case of image segmentation, two pixels i and j that belong to the same object/class are said to be connected and define a measure wij close to unity. Inversely, two pixels i and j that do not belong to the same class are said to be not connected and define a measure wij close to zero. A standard construction approach for the weight matrix wij is as follows. Let h(i, j) be some general non-negative distance measure between nodes i and j, then the weight wij is computed with a Gaussian kernel of 0-mean and variance σ such that: h(i, j) 1 exp(− ), Z σ2 where σ is the scaling parameter and Z is the normalization factor. wij =

Image Feature. The distance h(i, j) depends on image feature. The choice of features is difficult and critical to get an optimal segmentation result. For piecewise smooth and constant images, the gray-level value can be enough. For texture images, a feature vector at each pixel computing from a filter bank (as suggested in e.g. [11]) can be efficient. A recent promising image feature to represent and process textures is the image intensity patch around the current pixel. The patch idea as feature vector was first introduced for texture synthesis [9, 12, 13] then for image denoising. Buades et al. in [6] proposed to compute the weight matrix with patch differences and denoise the image with a non-local averaging. Gilboa and Osher in [7] proposed a variational model for non-local denoising based on

114

N. Houhou et al.

patch differences. Fina lly, Bresson and Chan in [14] proposed a variational unsupervised segmentation method also based on patch differences. In this paper, we will use the graph of image patches of Buades et al. Min-Cut. By definition a binary cut partitions a graph into two subsets. This partition process for graphs can be used for image segmentation when we want to find an object and the background. In optimization theory about maximum flows in flow networks [8], the optimal partition of the graph V into two sets A and B such that A ∪ B = V and A ∩ B = ∅ can be computed by finding the minimal cut (min-cut), i.e. the minimization of the inter-similarity between two sets A and B of V . In other words, given two particular nodes s ∈ A and t ∈ B in the graph, the min-cut partition can be written as:  min − cut(A, B) = min −wij xi xj , (1) x

xi >0,xj <0

where x is a N -dimensional indicator vector, with N = card(V ), such that xi = 1 if node i ∈ A, and xi = −1 otherwise. The min-cut partition gives the minimal capacity defined as the total weight between the nodes for each subset A and B. The min-cut approach has been applied to several computer vision problems, see [15, 16] for image restoration, [17, 18, 4, 5] for image segmentation, [19] for stereo and motion, and [20] for texture synthesis. Diffusion. In the case of binary partition of a graph, min-cut partition and diffusion are equivalent. Indeed, let us denote W the symmetric matrix such that W (i, j) = w(i, j). Then, the graph partition energy defined in (1) is equivalent to a Graph Laplacian operator: 1 (2) wij (xi − xj )2 , 8  where D is a N × N diagonal matrix with di = j w(i, j) on its diagonal, the matrix D − W is called the Graph-Laplacian. cut(A, B) =

Proof. See e.g. [10]. Let x+1 (resp. x−1 2 2 ) be the indicator function for xi > 0 (resp. xi < 0). Then (1) can be written in the matrix form as follows: cut(A, B) = which implies: xT (D − W )x =

1 T x (D − W )x, 4 1 wij (xi − xj )2 . 2 i,j

The weighted Graph Laplacian corresponds to a finite difference approximation of the continuous Lapacian operator. The graph Laplacian can also be a non-local operator. Semi-supervised segmentation. We observe that min-cut partitioning algorithms are defined as semi-supervised segmentation techniques. The min-cut seeks for

Semi-supervised Segmentation Based on Non-local Continuous Min-Cut

115

the optimal partition of the graph given particular nodes called the source "s" and the sink "t". Hence, it is easy to assign some pixels as source and some as sink if the pixel belongs to the object or the background. Several graphbased partitioning methods have been proposed in the literature s.a. [17, 21, 22, 23, 18]. The previous papers are based on discrete minimization methods to compute the min-cut given the labels. In this paper, we propose a continuous minimization method to solve the min-cut problem with labels and non-local image information.

3 3.1

Proposed Segmentation Method Continuous Min-Cut

Energy minimization problem. In this section a new non-local semi-supervised segmentation algorithm is introduced. The algorithm relies on the continuous formulation of the discrete min-cut problem defined as: min − cut(A, B) = min x

s.t.



wij (xi − xj )2

(3)

i,j

xk = +1, ∀k ∈ S xk = −1, ∀k ∈ T,

where S are the labels selected for the object and T are the labels assigned to the background. We propose the continuous min-cut (CMC) problem as follows (which is a constrained minimization problem w.r.t. a real-valued function u):  1 CM C(u) = min w(x, y)(u(x) − u(y))2 dxdy u 2 Ω×Ω s.t. u(x) = 1, ∀x ∈ S u(x) = 0, ∀x ∈ T, which is equivalent to this unconstrained minimization problem for u:   1 2 w(x, y)(u(x) − u(y)) dxdy + λ(x)(u − u0 )2 dx, (4) CM C(u) = min u 2 Ω×Ω Ω   1 if x ∈ S ∞ if x ∈ S ∪ T where u0 (x) = and λ(x) = , 0 if x ∈ T 0 otherwise where function λ provides the degree of confidence with respect to the labels. Non-local H 1 energy. The first term of (4) is deduced from (3) using the change of variable ui = xi2+1 ∈ {0, 1} then relaxing ui to [0, 1]. This term is also known as the non-local H 1 energy ( [24, 7]) defined as:   1 1 1 2 HG (u) = w(x, y)(u(x) − u(y)) dxdy = |∇G u|2 dx = ||u||HG1 , (5) 2 Ω×Ω 2 Ω

116

N. Houhou et al.

 where |∇G u|2 := Ω w(x, y)(u(x)−u(y))2 dy is the square norm of the continuous graph gradient of u. The optimality condition for (5) is:  w(x, y)(u(x) − u(y))dy = ΔG u = 0, Ω

where ΔG u is the continuous graph Laplacian of u. Labels. The second term of (4) introduces the hard constraint of labels in the energy minimization approach. This term comes from Unger et al. in [25], which incorporates seed points (assigned either to the object or to the background) in the geodesic active contour/snake model [26]. This term constrains function u to be equal to u0 for x ∈ S ∪ T and being equal to anything else for x ∈ S ∪ T . Function λ is highly discontinuous, which requires some regularization process to handle it. Unger et al. proposed a splitting operation to solve this problem. A new function v is introduced s.t.:  1 min λ(x)(v − u0 )2 dx + ||u − v||22 , u 2θ Ω where the term ||u − v||22 forces v ≈ u as θ → 0. The optimality solution w.r.t. v leads to:  2λ(x)θu0 − u u if λ → 0 = v= . u if λ → ∞ 2λ(x)θ − 1 0 3.2

Proposed Semi-supervised Segmentation Algorithm: Continuous Min-Cut + TV

Final model. The previous section introduced the continuous formulation of the min cut problem. In this section, we proposed to merge the continuous min-cut with the Total Variation (TV) energy. The TV term offers two advantages. First, TV regularizes the geometry of the contour between classes (object and background). Experiments showed that the continuous min-cut can provide irregularities along the contour. Second, Shi and Malik in [10] observed that the min-cut algorithm tends to favor misclassification of small sets, which are smoothed out with the TV regularization process. Finally, we propose the following energy minimization model for semisupervised segmentation:  E(u) = ||u||HG1 + λ(x)(u − u0 )2 dx + β||u||T V , (6) where ||u||T V =



Ω

Ω

|∇u|dx.

Minimization process. A direct use of the calculus of variation to (6) will produce a very slow minimization process. We propose to use a splitting operation to minimize E more efficiently. We introduce two new functions v, s s.t.:  λ(x)(v − u0 )2 dx + β||s||T V E(u, v, s) = ||u||HG1 + Ω

1 1 + ||u − v||22 + ||s − v||22 . 2θv 2θs

(7)

Semi-supervised Segmentation Based on Non-local Continuous Min-Cut

117

1 2θv ||u − w(x,y)u(y)dy+v(x) Ω  . Funcθv Ω w(x,y)dy+1

Then, v, s being fixed, we search for u as the solution of minu ||u||HG1 + v||22 , which is given by a fixed point method as u =

θv



 tions u, s being fixed, we search for v as the solution of minv Ω λ(x)(v − vs u0 )2 dx + 2θ1v ||u − v||22 + 2θ1s ||s − v||22 , which is given by v = θsθv+θ if λ = 0 s +θv and v = u0 if λ = ∞. Functions u, v being fixed, we search for s as the solution of mins β||s||T V + 2θ1s ||s − v||22 , which solution is given e.g. the Projection algorithm of Chambolle [27]. We propose the following iterative scheme for minimizing energy (7): ⎧ ⎪ un+1 ⎪ ⎪ ⎪ ⎪ ⎪ ⎨ n+1 v ⎪ ⎪ ⎪ ⎪ sn+1 ⎪ ⎪ ⎩ n+1 p

3.3

= =

θv





w(x,y)un (y)dy+v n (x)  θv Ω w(x,y)dy+1

Ω

θs un+1 +θv sn θs +θv

u0

if λ = 0 if λ = ∞ ,

n≥0

(8)

=v − θdivp β(pn +1/8∇(divpn −v n+1 /θs )) = β+1/8|∇(divpn −vn+1 /θs )| n+1

n

Some Properties of the Models (6) and (7)

Convexity. Both energy minimization models (6) and (7) are strictly convex 1 (since the HG term is strictly convex), which implies the existence of a unique minimizing solution independently of the initial condition. Hence, even using gradient descent approaches, the algorithm does not get stuck in a local minimum. Thus, as far as the labels are correctly defined, the results will be independent of the initialization. Relation with the original min-cut problem (1) or equivalently (3). The continuous min-cut problem (4) has the same solution as the discrete min-cut problem when considering characteristic/indicator functions of sets, i.e.: min {ECMC (u = 1A )} = min {cut(A, Ω\A)}. 1A

A

(9)

We remind that we relax function u to be between [0, 1] to define a continuous version of the min-cut algorithm, which can be minimized with continuous minimization tools. Then, the segmentation result is given by thresholding the minimizer u of (6) with any value in (0, 1). Non-trivial steady state solution of (6). The final steady state solution of (6) is not the mean value of the initial function. Call ut=0 = ut=0 the mean value function. It is easy to show by contraction that ut=0 is not solution to (6). If ut=∞ = ut=0 then E(ut=∞ ) = Ω λ(x)(ut=0 − u0 )2 dx > 0, and the minimizer is thus given by ut=0 = u0 . However, u0 (x) = 1 ∀x ∈ S, 0 ∀x ∈ T . Thus, ut=0 = u0 . We notice that Gilboa and Osher in [7] also use the energy ||u||HG1 to perform semi-supervised segmentation. However, they did not use a term to constraint the labels as in this work. They minimized energy ||u||HG1 starting with a trinary initial function ut=0 = {−1, 0, 1} (labelled pixels for the object are assigned to

118

N. Houhou et al.

the value 1 and those for the background to the value −1). However, the minimizing solution is the mean value function ut=∞ = ut=0 . Hence, this algorithm requires to stop the diffusion process.

4

Results

This section presents some results of the proposed semi-supervised segmentation algorithm. The graph is defined from local and non-local image information:  |i−j|2 (j)|2 + |F (i)−F if i, j ∈ Na×a (i) σ12 σ22 w(i, j) = , (10) 0 otherwise where Na×a (i) is a square window of size a × a around i. The computational cost of the similarity between pixel on the whole image is very expensive, however we chose to simply select points in a close neighborhood. This implies the supposition that if two points are far away, they are not connected. From a2 neighbors only the cl = 8 closest points are selected. The feature vector F is a square patch of size f ×f centered on each pixel. The segmentation is driven by (8). The initial condition for u, v and s are given by the label S, i.e. u = v = s = 1 if x ∈ S and u = v = s = 0 otherwise. With an unoptimized Matlab implementation, the graph computation lasts approximatively 15 seconds and the segmentation is performed in approximatively 1 minute. The image size is 128 × 128. TV Regularization Effect. The importance of the TV-Regularization effect is emphasized in this paragraph. A salt-and-pepper noise is added on a two-phase image with different means 1(a). The inside and outside labels are presented on Figure 1(a). The results show that if the TV regularization is not performed then the segmentation fails (Fig 1(b)). When the TV regularization is used, then the segmentation succeeds.

(a)

(b)

(c)

Fig. 1. Application of our algorithm on a image with a salt-and-pepper noise. (a)Initialization (b)The segmentation result without TV-regularization.(c)The segmentation result with TV-regularization.

Semi-supervised Segmentation Based on Non-local Continuous Min-Cut

119

Texture Images. We apply our algorithm to a synthetic texture image composed of five different patterns. Figures 2(a) and 2(c) show the initializations and Figures 2(b) and 2(d) the corresponded results. The patch size is chosen to be 9 × 9 which correspond to the pattern size for the two selected textures.

(a)

(b)

(c)

(d)

Fig. 2. Results on synthetic textures. (a) and (c) Initializations. (b) and (d) results.

(a)

(b)

(c)

(d)

(e)

(f)

Fig. 3. Results on real-world images from the Berkeley dataset. Left column: Initial labels. Right column: Segmentation Result.

Natural Images. We apply now our algorithm to a set of natural images taken from the Berkeley segmentation dataset [28]. In the fist column of Figure 3, the inside and outside labels are shown and in the second column the segmentation results.

120

N. Houhou et al.

(a)

(b)

(c)

(d)

(e)

(f)

Fig. 4. Results on real world color images from the Berkeley dataset. Left column: Initial labels. Right column: Segmentation Result.

Fig. 5. First row, Segmentation of the liver. (a) Initial labels. (b) Segmentation Result. Second row, Segmentation of the lateral muscles on the neck. (c) Initial labels. (d) Segmentation Result. (e) Zoom on the segmentation of the muscles.

Color Images. We consider the simple case of Red-Green-Blue (RGB) channels. The first step consists of computing the graph by taking into account each channel, i.e. F = (Fr , Fg , Fb ), where respectively the red, green and blue feature channel. Images are also taken from the Berkeley segmentation dataset [28]. In the fist column of Figure 3, the inside and outside labels are shown and in the second column the segmentation results. Medical Images. We apply our segmentation algorithm on 2-D medical images of CT scans of the abdomen and the head and neck. Figures 5(a) and 5(c) present

Semi-supervised Segmentation Based on Non-local Continuous Min-Cut

121

the inside and outside initial labels. Figures 5(b) and 5(d) show the segmentation results. For the liver segmentation, the label on the background (black) prevents the diffusion from capturing as well the heart. The segmentation of the structures in the neck are challenging and the result that we obtain are promising.

5

Discussion and Conclusion

In this paper, a non-local semi-supervised segmentation method has been proposed. The success of graph partitioning algorithms for image segmentation has motivated this work. Our objective was to translate the discrete min-cut algorithm into a non-local continuous min-cut algorithm. The addition of hard constraints with the source and sink labels has been done naturally in the proposed continuous framework. Besides, it has also been easy to introduce new terms such as the TV term that regularizes the geometry of the boundary between the object and the background. The non-local continuous min-cut is also equivalent to a diffusion process. The diffusion is done on the graph of image intensity patches, which holds semi-local and non-local image information useful to segment textures and complex patterns. Our semi-supervised segmentation has provided promising segmentation results for textures and real-world objects. Future work will focus on comparing the efficiency of our segmentation algorithm with other related semi-supervised segmentation algorithms. We would like also to extend our method to 3-D medical images.

Acknowledgements Nawal Houhou was supported by Swiss National Science Foundation #205320101621, Xavier Bresson was supported by ONR N00014-03-1-0071 and ONR MURI subcontract from Stanford University and Arthur Szlam was supported by NSF DMS-0811203. The authors would like also to thank the referees for their constructive comments.

References 1. Strang, G.: Maximal Flow Through A Domain. Mathematical Programming 26(2), 123–143 (1983) 2. Protiere, A., Sapiro, G.: Interactive image segmentation via adaptive weighted distances. IEEE Transactions on Image Processing 16(4), 1046–1057 (2007) 3. Cremers, D., Fluck, O., Rousson, M., Aharon, S.: A Probabilistic Level Set Formulation for Interactive Organ Segmentation. In: SPIE (2007) 4. Appleton, B., Talbot, H.: Globally minimal surfaces by continuous maximal flows. IEEE Transactions on Pattern Analysis and Machine Intelligence 28(1), 106–118 (2006) 5. Unger, M., Pock, T., Cremers, D., Bischof, H.: Tvseg - interactive total variation based image segmentation. In: British Machine Vision Conference (BMVC), Leeds, UK (September 2008)

122

N. Houhou et al.

6. Buades, A., Coll, B., Morel, J.: A review of image denoising algorithms, with a new one. Multiscale Modeling & Simulation 4(2), 490–530 (2005) 7. Gilboa, G., Osher, S.: Nonlocal linear image regularization and supervised segmentation. Multiscale Modeling and Simulation 6(2), 595–630 (2007) 8. Elias, P., Feinstein, A., Shannon, C.E.: Note on Maximum Flow Through a Network. IRE Transactions on Information Theory 2, 117–119 (1956) 9. Efros, A., Leung, T.: Texture Synthesis by Non-Parametric Sampling. In: IEEE International Conference on Computer Vision, vol. 2, pp. 10–33 (1999) 10. Shi, J., Malik, J.: Normalized cuts and image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence 22, 888–905 (2000) 11. Malik, J., Belongie, S., Leung, T., Shi, J.: Contour and texture analysis for image segmentation. International Journal of Computer Vision 43(1), 7–27 (2001) 12. Efros, A., Freeman, W.: Image quilting for texture synthesis and transfer. In: Proceedings of the Conference on Computer graphics and interactive techniques, SIGGRAPH, pp. 341–346. ACM, New York (2001) 13. Liang, L., Liu, C., Xu, Y., Guo, B., Shum, H.: Real-time texture synthesis by patch-based sampling. ACM Trans. Graph. 20(3), 127–150 (2001) 14. Bresson, X., Chan, T.: Non-local Unsupervised Variational Image Segmentation Models, UCLA CAM Report 08-67 (2008) 15. Wu, Z., Leahy, R.: An Optimal graph theoretic approach to data clustering: Theory and its application to image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence 15(11), 1101–1113 (1993) 16. Ishikawa, H., Geiger, D.: Segmentation by grouping junctions. In: Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 125–131 (1998) 17. Boykov, Y., Jolly, M.P.: Interactive graph cuts for optimal boundary and region segmentation of objects in n-d images. In: Proceedings of Eighth IEEE International Conference on Computer Vision, vol. 1, pp. 105–112 (2001) 18. Boykov, Y., Funka-Lea, G.: Graph cuts and efficient n-d image segmentation. Int. J. Comput. Vision 70(2), 109–131 (2006) 19. Boykov, Y., Veksler, O., Zabih, R.: Fast approximate energy minimization via graph cuts. IEEE Transactions on Pattern Analysis and Machine Intelligence 23, 2001 (2001) 20. Kwatra, V., Schödl, A., Essa, I., Turk, G., Bobick, A.: Graphcut textures: Image and video synthesis using graph cuts. In: Proceedings of the Conference on Computer graphics and interactive techniques, SIGGRAPH, vol. 22(3), pp. 277–286 (July 2003) 21. Blum, A., Chawla, S.: Learning from labeled and unlabeled data using graph mincuts. In: Proceedings of the Eighteenth International Conference on Machine Learning, pp. 19–26. Morgan Kaufmann Publishers Inc., San Francisco (2001) 22. Yu, S., Shi, J.: Segmentation given partial grouping constraints. IEEE Transactions on Pattern Analysis and Machine Intelligence 26(2), 173–183 (2004) 23. Grady, L., Funka-lea, G.: Multi-label image segmentation for medical applications based on graph-theoretic electrical potentials. In: Proceedings of the European Conference on Computer Vision, pp. 230–245. Springer, Heidelberg (2004) 24. Zhou, D., Scholkopf, B.: A Regularization Framework for Learning from Graph Data. In: Workshop on Statistical Relational Learning and Its Connections to Other Fields (2004) 25. Unger, M., Pock, T., Bischof, H.: Continuous globally optimal image segmentation with local constraints. In: Computer Vision Winter Workshop 2008 (2008)

Semi-supervised Segmentation Based on Non-local Continuous Min-Cut

123

26. Caselles, V., Kimmel, R., Sapiro, G.: Geodesic Active Contours. International Journal of Computer Vision 22(1), 61–79 (1997) 27. Chambolle, A.: An Algorithm for Total Variation Minimization and Applications. Journal of Mathematical Imaging and Vision 20(1–2), 89–97 (2004) 28. Martin, D., Fowlkes, C., Tal, D., Malik, J.: A Database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, July 2001, vol. 2, pp. 416–423 (2001)

Momentum Based Optimization Methods for Level Set Segmentation Gunnar Läthén1,3 , Thord Andersson2,3 , Reiner Lenz1,3 , and Magnus Borga2,3 1

Department of Science and Technology, Linköping University Department of Biomedical Engineering, Linköping University Center for Medical Image Science and Visualization, Linköping University 2

3

Abstract. Segmentation of images is often posed as a variational problem. As such, it is solved by formulating an energy functional depending on a contour and other image derived terms. The solution of the segmentation problem is the contour which extremizes this functional. The standard way of solving this optimization problem is by gradient descent search in the solution space, which typically suffers from many unwanted local optima and poor convergence. Classically, these problems have been circumvented by modifying the energy functional. In contrast, the focus of this paper is on alternative methods for optimization. Inspired by ideas from the machine learning community, we propose segmentation based on gradient descent with momentum. Our results show that typical models hampered by local optima solutions can be further improved by this approach. We illustrate the performance improvements using the level set framework.

1

Introduction

A very popular and powerful approach for solving image segmentation problems is through the calculus of variations. In this setting the solution is represented by a contour, which parameterizes an energy functional depending on various image based quantities such as intensities or gradients. In general, the set of possible contours constitutes the solution space, where the goal is to find the contour which extremizes the energy in this space. As an optimization problem, there are many possible strategies to find this solution. One approach is to use the method of graph cuts to find a global optimum [1]. However, this can only be applied to a small class of energy functionals. For more general problems, the standard method has been to deform an initial contour in the steepest (gradient) descent of the energy. Equations of motion for the contour is derived using the Euler-Lagrange equation and the condition that the first variation of the energy functional should vanish at a (local) optimum. Then, the contour is evolved to steady-state given the resulting equations. A standard implementation of this strategy is usually hampered by two common problems. The first problem is sensitivity to local optima, which are manifested due to noisy data. To avoid this, the usual approach has been to modify the energy functional by adding regularizing terms. The second common problem is poor convergence due X.-C. Tai et al. (Eds.): SSVM 2009, LNCS 5567, pp. 124–136, 2009. c Springer-Verlag Berlin Heidelberg 2009 

Momentum Based Optimization Methods for Level Set Segmentation

125

to difficulties in choosing good initial conditions. To improve convergence, very effective solvers based on multi-grid [2, 3] and AOS schemes [4, 5, 6] have been developed. However, these methods all search for a solution in the gradient descent direction and little focus has been given to the underlying optimization problem. This has been identified in recent work [7, 8], where the metric defining the notion of steepest descent (gradient) has been studied. By changing the metric in the solution space, local optima due to noise are avoided in the search path. Along the same direction, this paper presents an alternative search strategy for the optimization solver. Our idea stems from the machine learning community, where an optimization problem is solved to update the system to adapt to a given stimulus. A simple, but effective, modification to gradient descent was proposed in [9], which basically adds a momentum to the motion in solution space. This simulates the physical properties of inertia and momentum and effectively allows the search to avoid local optima and accelerate in favorable directions. In this paper, we show how this idea can be used for image segmentation in a variational framework using level set methods. The results show faster convergence and less sensitivity to local optima. The paper will proceed as follows. In Section 2, we describe the idea of gradient descent with momentum in a general setting and give examples highlighting the benefits. Then, Section 3 presents how this idea can be used to solve segmentation problems in a level set framework. This is exemplified in Section 4 and Section 5 where we give implementation details and compute segmentations given a common energy functional. Finally, Section 6 concludes the paper and presents ideas for future work.

2

Gradient Descent with Momentum

Considering general optimization problems, gradient descent is a very simple approach which can handle many types of cost functions. It is intuitive, since it always moves in the direction of steepest descent, which locally gives the largest amount of decrease in the cost function. In addition, it only requires first order derivatives of the function, providing simple and fast computations. On the other hand, it is well known that gradient descent suffers from poor convergence and high sensitivity to local optima for many practical problems. Therefore, other descent directions (Newton, Quasi-Newton, etc.) have been studied and proved superior, see e.g. [10] for a rigorous reference. A simple alternative to these, more theoretically sophisticated methods, is often applied in the machine learning community. A typical problem here is the construction of adaptive systems that can classify unknown inputs. This can be formulated as an optimization problem and one of the goals of machine learning is to construct fast learning or adaptation rules that can be implemented in very simple hardware or software devices. To improve the convergence and robustness of a simple gradient descent solution, while avoiding the complexity of more sophisticated optimization methods, gradient descent with momentum

126

G. Läthén et al.

was proposed [9]. The starting point of our derivation of the proposed method is the following description of a standard line search optimization method: xk+1 = xk + sk

(1)

sk = αk pk

(2)

where xk is the current iterate, sk is the next step consisting of length αk in direction pk . To guarantee convergence, it is often required that pk be a descent direction while αk gives a sufficient decrease in the cost function. A simple realization of this is gradient descent which moves in the steepest descent direction according to pk = −∇fk , where f is the cost function, while αk satisfies the Wolfe conditions [10]. Turning to gradient descent with momentum, we will adopt some terminology from the machine learning community and choose a search direction according to: sk = −η(1 − ω)∇fk + ωsk−1 (3) where η is the learning rate and ω ∈ [0, 1] is the momentum. Note that ω = 0 gives standard gradient descent sk = −η∇fk , while ω = 1 gives “infinite inertia” sk = sk−1 . The intuition behind this strategy is that the current iterate has an inertia, which prohibits sudden changes in the velocity. This will effectively filter out high frequency changes in the cost function and allow for greater steps in favourable directions. Selecting appropriate parameters, our hope is that the rate of convergence is increased while eventual local optima will be overstepped. The effect of the momentum term is illustrated in Figure 1. The iterates with momentum ω = 0 show the behaviour of standard gradient descent when varying the learning rate (step length) η. In comparison, for an appropriate choice of momentum ω = 0.1, the solution approaches the optimum more rapidly. It can be seen however, that too high momentum of ω = 0.4 leeds to oscillations. 1

10

100 90

0

10

80 −1

10

70 60

−2

10

50 −3

10

40

−4

30

10

20

−5

10

10 −6

20

40

60

80

(a) Iterates and cost function

100

10

0

η = 0.04, ω = 0 η = 0.4, ω = 0 η = 0.4, ω = 0.1 η = 0.4, ω = 0.4 2

4

6

8

10

(b) Convergence rate

Fig. 1. Gradient descent search with/without momentum on a quadratic cost function

Momentum Based Optimization Methods for Level Set Segmentation

3

127

Minimizing Level Set Flows

As was previously outlined, segmentation problems in image analysis are often described as optimization problems with solutions derived using the calculus of variations. The standard procedure is to formulate an energy functional by means of a contour and various image derived terms. Extremals of this functional are then identified by an Euler-Lagrange equation, which is used to derive equations of motion for the contour [11]. This typical procedure yields a gradient descent search in a high dimensional solution space, in which each possible contour is represented by a point. For example [11] presents, among others, the derivation of weighted region described by the following functional:  E(C) = f (x, y)dxdy (4) ΩC

where C is a 1D curve embedded in a 2D domain, ΩC is the inside region of C, and f (x, y) is a scalar function. This functional is used to maximize some quantity given by f (x, y) inside C. A simple example is f (x, y) = 1 which measures, and maximizes, the area. Calculating the first variation of Eq. (4) yields the evolution equation: ∂C = −f (x, y)n (5) ∂t where n is the curve normal. Again, setting f (x, y) = 1 gives a constant flow in the normal direction, typically referred to as the “balloon force”. The representation, or parameterization, of the contour C can in general be chosen arbitrarily. However, it is often convenient to use the implicit level set method by Osher and Sethian [12], since this allows for arbitrary topological changes. To summarize the basic ideas, a contour is represented implicitly as a zero level set of a time dependent scalar function (referred to as the level set function). Formally, a contour C is described by C = {x : φ(x, t) = 0}. To deform C, the level set function is evolved in time according to a set of partial differential equations (PDEs). The transition from the equations of motion for a parametrized curve (Eq. (5)) to a level set PDE is accomplished by a simple procedure. In general, the motion ∂C ∂t = γn translates to the level set equation ∂φ ∂t = γ |∇φ| [11]. Thus, Eq. (5) gives the familiar level set equation: ∂φ = −f (x, y) |∇φ| ∂t

(6)

The remainder of this section will describe how we modify the typical level set method update scheme to incorporate a momentum term as presented in Section 2. 3.1

Momentum for Minimizing Level Set Flows

We have noted that the contour evolving according to the Euler-Lagrange equation yields a gradient descent search. Recall that each contour can be represented as a point in the solution space (the structure of the space will depend on

128

G. Läthén et al.

parameterization). Thus, we can approximate the direction of the gradient by computing the vector between two subsequent points. In the level set framework we achieve this by taking the difference between two subsequent time instances of the level set function, representing the entire level set function as one vector: φ(tn ) − φ(tn−1 ) (7) Δt where f is a cost function in compliance with the terminology used in Section 2. Note that this is indeed an approximation, depending on the time difference Δt = tn − tn−1 . Following the ideas from Section 2, we update the level set function to incorporate a momentum term: ∇f (tn ) ≈

 n ) − φ(tn−1 ) φ(t + ωs(tn−1 ) Δt φ(tn ) = φ(tn−1 ) + Δts(tn ) s(tn ) = −η(1 − ω)

(8) (9)

The complete procedure works as follows: Procedure UpdateLevelset 1

Given the level set function φ(tn−1 ), compute the next (intermediate)  n ). This is performed by evolving φ according to a PDE time step φ(t (such as Eq. (6)) using standard techniques (e.g. Euler integration).

2

Compute the approximate gradient by Eq. (7).

3

Compute a step s(tn ) according to Eq. (8). This step effectively modifies the gradient direction by incorporating the momentum term as a fraction of the previous step s(tn−1 ).

4

Compute the next time step φ(tn ) by Eq. (9). Note that this replaces the intermediate level set function computed in Step 1.

The procedure is very simple and is directly compatible with any type of level set implementation.

4

Experiments

We now describe some details of the implementation and illustrate properties of the suggested method using two examples. Here we study 1D curves embedded in a 2D domain, but the approach readily generalizes to 2D surfaces in 3D given the level set framework. 4.1

Implementation Details

We have implemented the proposed ideas in Matlab using standard level set techniques based on [13, 14]. Reference code can be found online at the site http://dmforge.itn.liu.se/ssvm09/. Some details of our implementation are the following:

Momentum Based Optimization Methods for Level Set Segmentation

129

– The level set function is reinitialized (reset to a signed distance function) after Step 1 and Step 4. This is typically performed using the fast marching [15] or fast sweeping algorithms [16]. There are two reasons for this: Firstly it is required for stable evolution in time due to the use of explicit Euler integration. Secondly we want a momentum induced by the zero level set of φ (the contour), rather than all level sets of φ. Reinitialization could be omitted, with the effect of introducing a momentum on all individual level sets. Interpreting each sample of φ as a parameter of the contour, this is equivalent to applying momentum on each parameter. While feasible, we have not experimented with momentum without incorporating reinitialization. – We avoid instabilities by dampening s(tn ) in Step 3 using a sigmoidal function: sˆ(s(tn ), smax ) =

1+

2smax −2s(t n )/smax e

− smax

(10)

where smax is the maximum step length allowed. – Any explicit or implicit time integration scheme can be used in Step 1. Due to its simplicity, we have used explicit Euler integration which might require several inner iterations in Step 1 to advance the level set function by Δt time units. 4.2

Weighted Region Based Flow

To verify our idea, we have used a simple energy functional based on a weighted region term (Eq. (4)) combined with a penalty on curve length for regularization. The goal is to maximize:   E(C) = f (x, y)dxdy − α ds (11) ΩC

C

where α is a regularization weight parameter. The target function f (x, y) is image based, computed using the approach in [17]. This method uses quadrature filters [18] across multiple scales to detect line structures. Taking the real part of the complex filter response, f (x, y) gives positive values on the inside of linear structures, negative on the outside, and zero on the edges. Translating Eq. (11) to a level set PDE following [11] gives: ∂φ = −f (x, y) |∇φ| + ακ |∇φ| ∂t

(12)

where κ is the curvature of the contour. First we illustrate some properties of the method with a synthetic test image depicted in Figure 2(a), which mimics the common problem of intensity variation in medical imaging. The intensity of the object ranges from 0.3 to 1, while the noise level is 0.1. This image yields the target function f (x, y) in Figure 2(b) where bright and dark colors indicate positive and negative values respectively. As exemplified in our first experiment (Figure 3) the dip in contrast results in a local optimum in the solution space.

130

G. Läthén et al.

(b) Target function f (x, y)

(a) Input image

Fig. 2. Synthetic test image illustrating the presence of a local optima in the solution space

(a) time = 0

(b) time = 40

(c) time = 100

(d) time = 170

(e) time = 300

(f) time = 870

Fig. 3. Iterations without momentum (conventional gradient descent)

Figure 3 shows the results after evolving the level set function by Eq. (12) until convergence without momentum, using conventional methods. We define convergence as |∇f |∞ < 0.03 (using the infinity/maximum norm), with ∇f given in Eq. (7). For this experiment we used parameters α = 0.7 and we reinitialized the level set function every fifth time unit. For comparison, Figure 4 shows the results after running our method using parameters α = 0.7, ω = 0.8, η = 10, smax = 100, Δt = 5. Plots of the energy functional for both experiments are shown in Figure 5. Here, we plot the weighted area term and the length penalty term separately, to illustrate the balance between the two. Note that the functional without momentum in Figure 5(a) is monotonically increasing, due to the nature of gradient descent, while the functional with momentum visits a number of local maxima during the search. To further exemplify the behaviour of our method, we created a slightly modified version of Figure 2(a), shown in Figure 6(a). In contrast to Figure 2(a), the shape in Figure 6(a) is disconnected, so the global optimum is expected to contain two separated regions. Not surprisingly, conventional gradient descent captures only a local minimum as displayed in Figure 7, while gradient descent with momentum succeeds in capturing the global solution as two separated

Momentum Based Optimization Methods for Level Set Segmentation

(a) time = 0

(b) time = 20

(c) time = 40

(d) time = 60

(e) time = 150

(f) time = 200

(g) time = 245

(h) time = 320

(i) time = 460

131

Fig. 4. Iterations using momentum 1800

1800 Energy functional Length penalty integral Target function integral

1600

1600

1400

1400

1200

1200

1000

1000

800

800

600

600

400

400

200

200

0 0

100

200

300

400 500 time

600

700

(a) Without momentum

800

0 0

Energy functional Length penalty integral Target function integral 100

200

300

400

time

(b) With momentum

Fig. 5. Plots of energy functionals for synthetic test image in Figure 2(a)

regions (Figure 8). For this experiment, we used the same parameters as in Figure 3 and Figure 4. As a third test image we used a 458 × 265 retinal image from the DRIVE database [19], shown in Figure 9(a). The target function f (x, y) is illustrated in Figure 9(b). As in the previous experiment, bright and dark colors indicate positive and negative values for f (x, y). The convergent result without momentum using parameters α = 0.07 and reinitialization every tenth time unit is shown in Figure 10, given the initial condition in Figure 10(a). Applying

132

G. Läthén et al.

(b) Target function f (x, y)

(a) Input image

Fig. 6. Synthetic test image illustrating the presence of a local optima in the solution space

(a) time = 0

(b) time = 200

(c) time = 515

Fig. 7. Iterations without momentum (conventional gradient descent)

(a) time = 0

(b) time = 40

(c) time = 70

(d) time = 180

(e) time = 240

(f) time = 485

Fig. 8. Iterations using momentum

the idea of momentum yields the result in Figure 11, using the parameters α = 0.07, ω = 0.5, η = 1.3, smax = 40, Δt = 10. The energy functionals are plotted in Figure 12 to display the convergence of both methods.

5

Results

The synthetic test image in Figure 2(a) illustrates a local optimum in the solution space when applying the parameters in our first experiment. As expected,

Momentum Based Optimization Methods for Level Set Segmentation

(a) Input image

133

(b) Target f (x, y)

Fig. 9. Retinal image

(a) time = 0

(b) time = 20

(c) time = 40

(d) time = 100

(e) time = 200

(f) time = 400

(g) time = 600

(h) time = 1210

Fig. 10. Iterations without momentum (conventional gradient descent)

134

G. Läthén et al.

(a) time = 0

(b) time = 20

(c) time = 40

(d) time = 100

(e) time = 200

(f) time = 400

(g) time = 600

(h) time = 820

Fig. 11. Iterations using momentum 9000

9000 8000

8000

7000

7000

6000

6000

5000

Energy functional Length penalty integral Target function integral

4000

5000

3000

3000

2000

2000

1000

1000

0 0

200

400

600 time

800

1000

(a) Without momentum

1200

Energy functional Length penalty integral Target function integral

4000

0 0

100

200

300

400 time

500

600

700

(b) With momentum

Fig. 12. Plots of energy functionals for the retinal image in Figure 9(a)

800

Momentum Based Optimization Methods for Level Set Segmentation

135

the conventional gradient descent approach converges to this local optimum as depicted in Figure 3. In contrast, our proposed method gains enough momentum in order to overstep the optimum, while at the same time the global solution is reached more rapidly. The process (illustrated in Figure 4) intuitively expands the curve beyond a local optimum, followed by a retraction if the search does not provide any increase in that direction. Using a slightly modified input image, our second example shows that our method is capable of capturing global optima, even when the solution consists of separated regions (Figure 8). Our third example illustrates our method on real data using a retinal image. In Figure 10 we see that conventional gradient descent fails to capture many weak signal blood vessels. This is a typical case of local optima solutions introduced by noise and poor image contrast. Under the same conditions, gradient descent with momentum captures practically all visible vessels as shown in Figure 11. Note that this example does not include any verification of the accuracy of the segmented vessels. The primary purpose is to illustrate that our method reaches a stronger optimum value for the energy functional, as shown in Figure 12.

6

Conclusions and Future Work

In this paper we have presented the idea of gradient descent with momentum in the context of segmentation using the level set method. We have illustrated the drawbacks of conventional gradient descent and showed examples on how the solution is improved by adding momentum. In contrast to much of the previous work, we have improved the solution by changing the method of solving the optimization problem rather than changing the parameters of the energy functional. In the future, we will further study the general optimization problem of image segmentation to propose more efficient solutions. Regarding the particular idea of momentum, we will apply this on real applications and verify the quality of the results.

References 1. Boykov, Y., Kolmogorov, V.: Computing geodesics and minimal surfaces via graph cuts. In: Proc. ICCV 2003, October 2003, vol. 1, pp. 26–33 (2003) 2. Papandreou, G., Maragos, P.: Multigrid geometric active contour models. IEEE Transactions on Image Processing 16(1), 229–240 (2007) 3. Kenigsberg, A., Kimmel, R., Yavneh, I.: A multigrid approach for fast geodesic active contours. Technical Report CIS-2004-06, Technion–Israel Inst. Technol., Haifa (2004) 4. Paragios, N., Mellina-Gottardo, O., Ramesh, V.: Gradient vector flow fast geometric active contours. IEEE Transactions on Pattern Analysis and Machine Intelligence 26(3), 402–407 (2004) 5. Goldenberg, R., Kimmel, R., Rivlin, E., Rudzsky, M.: Fast geodesic active contours. IEEE Transactions on Image Processing 10(10), 1467–1475 (2001)

136

G. Läthén et al.

6. Weickert, J., Kühne, G.: Fast methods for implicit active contour models. In: Geometric Level Set Methods in Imaging, Vision and Graphics. Springer, Heidelberg (2003) 7. Charpiat, G., Keriven, R., Pons, J.P., Faugeras, O.: Designing spatially coherent minimizing flows for variational problems based on active contours. In: Proc. ICCV 2005, October 2005, vol. 2, pp. 1403–1408 (2005) 8. Sundaramoorthi, G., Yezzi, A., Mennucci, A.: Sobolev active contours. International Journal of Computer Vision 73(3), 345–366 (2007) 9. Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning internal representations by error propagation, pp. 318–362. MIT Press, Cambridge (1986) 10. Nocedal, J., Wright, S.J.: Numerical Optimization, 2nd edn. Springer, Heidelberg (2006) 11. Kimmel, R.: Fast edge integration. In: Geometric Level Set Methods in Imaging, Vision and Graphics. Springer, Heidelberg (2003) 12. Osher, S., Sethian, J.A.: Fronts propagating with curvature-dependent speed: Algorithms based on Hamilton-Jacobi formulations. Journal of Computational Physics 79, 12–49 (1988) 13. Osher, S., Fedkiw, R.: Level Set and Dynamic Implicit Surfaces. Springer, New York (2003) 14. Peng, D., Merriman, B., Osher, S., Zhao, H.K., Kang, M.: A pde-based fast local level set method. Journal of Computational Physics 155(2), 410–438 (1999) 15. Sethian, J.: A fast marching level set method for monotonically advancing fronts. Proceedings of the National Academy of Science 93, 1591–1595 (1996) 16. Zhao, H.K.: A fast sweeping method for eikonal equations. Mathematics of Computation (74), 603–627 (2005) 17. Läthén, G., Jonasson, J., Borga, M.: Phase based level set segmentation of blood vessels. In: Proc. ICPR 2008, Tampa, FL, USA, IAPR (December 2008) 18. Granlund, G.H., Knutsson, H.: Signal Processing for Computer Vision. Kluwer Academic Publishers, Netherlands (1995) 19. Staal, J., Abramoff, M., Niemeijer, M., Viergever, M., van Ginneken, B.: Ridge based vessel segmentation in color images of the retina. IEEE Transactions on Medical Imaging 23(4), 501–509 (2004)

Optimization of Divergences within the Exponential Family for Image Segmentation Francois Lecellier1 , Stephanie Jehan-Besson2, Jalal Fadili1 , Gilles Aubert3 , and Marinette Revenu1 1

Laboratoire GREYC, University of Caen, France Laboratoire LIMOS, University of Clermont-Ferrand, France Laboratoire J.A. Dieudonné, University of Nice Sophia-Antipolis, France 2

3

Abstract. In this work, we propose novel results for the optimization of divergences within the framework of region-based active contours. We focus on parametric statistical models where the region descriptor is chosen as the probability density function (pdf) of an image feature (e.g. intensity) inside the region and the pdf belongs to the exponential family. The optimization of divergences appears as a flexible tool for segmentation with and without intensity prior. As far as segmentation without reference is concerned, we aim at maximizing the discrepancy between the pdf of the inside region and the pdf of the outside region. Moreover, since the optimization framework is performed within the exponential family, we can cope with difficult segmentation problems including various noise models (Gaussian, Rayleigh, Poisson, Bernoulli ...). We also experimentally show that the maximisation of the KL divergence offers interesting properties compare to some other data terms (e.g. minimization of the anti-log-likelihood). Experimental results on medical images (brain MRI, contrast echocardiography) confirm the applicability of this general setting.

1 Introduction We propose here to focus on the segmentation of homogeneous regions in noisy images using statistical region-based active contour models (RBAC). In RBAC, region-based terms can be advantageously combined with boundary-based ones [1, 2]. The evolution equation is generally deduced from a general criterion to minimize that includes both region integrals and boundary integrals. The combination of those two terms in the energy functional allows the use of photometric image properties, such as texture [3] and noise [4], as well as geometric properties such as the shape prior of the object to be segmented. In statistical region-based active contours, see [5] for a review, image features (e.g. intensity) are considered as random variables whose distribution may be parametric (e.g. Gaussian) or non parametric [6]. Classically, the authors consider the minimization of the anti-log-likelihood for segmentation [7,8,4]. In this paper, we rather focus on the optimization of distance between pdfs. Such distances or more generally divergences can be used in two different manner. On the one hand, they can be used for segmentation with distribution intensity prior and in this case, we aim at minimizing the distance between the pdf of the evolving region and a reference one. On the other hand, they can be used for segmentation without reference and in this second case, we aim X.-C. Tai et al. (Eds.): SSVM 2009, LNCS 5567, pp. 137–149, 2009. c Springer-Verlag Berlin Heidelberg 2009 

138

F. Lecellier et al.

at maximizing the distance between the pdf of the inside region and the pdf of the outside region. In the literature, the minimization of divergences between non parametric pdfs has first been proposed in [6] for video sequences. It has then been developed for cardiac structures tracking in perfusion MRI (p-MRI) sequences in [9]. As far as segmentation using the maximization of divergences is concerned, some authors [10] have also proposed to take benefit of the maximization of the Bhattacharya distance of non parametric pdfs for segmentation. On the other hand, divergences between Gaussian distributions have been developed for DTI segmentation in [11]. In this paper, we propose to set a general framework for the optimization of divergences between parametric pdfs within the exponential family. To the best of our knowledge, such a framework has never been studied for region-based active contour segmentation. The rationale behind using the exponential family is that it includes, among others, Gaussian, Rayleigh, Poisson and Bernoulli distributions that have proven to be useful to model the noise structure [4] in many real image acquisition devices (e.g. Poisson for photon counting devices such as X-ray or CCD cameras, Rayleigh for ultrasound images, etc). Using shape derivative tools as in [12, 6], our effort focuses on constructing a general expression for the derivative of the energy (with respect to a domain), and on deriving the corresponding evolution speed. Our general framework is also specialized to some particular cases, such as the optimization of the KullbackLeibler (KL) divergence [13], which gives a simple expression of the derivative. This theoretical framework is then more explicitly detailed and illustrated for the case of the segmentation without reference. In this case, we aim at maximizing the dissimilarity between the pdf of the intensity within the region inside the evolving contour and the pdf of the intensity within the region outside the contour. In other words, we perform a competition between the pdfs of these two regions through the maximisation of divergences. Experimental results are given for the particular case of the KL divergence. We experimentally compare this data term to the classical minimization of the anti-log-likelihood [7, 14] for the segmentation of the White Matter in brain MRI and we show that KL maximisation is able to extract a single Gaussian from a mixture of Gaussian. We also show the applicability of our data term for the segmentation of the left ventricle in contrast echocardiography where the noise is modelled using Rayleigh. In this paper, we first set our general setting and introduce shape gradients in section 2. In section 3, we propose to give some general results for the exponential family and then for the shape derivative of divergences between pdfs. These results are then specialized for the KL divergence using the Maximum Likelihood Estimation (MLE) for the parameters. Experimental results for the maximization of KL divergence are given in section 4.

2 Optimization of Divergences between Pdfs: General Setting In this section, we set our general setting for segmentation through the optimization of distances between pdfs or more generally divergences. 2.1 General Setting Consider a function y : Rn → χ ⊂ R which describes the feature of interest. The term y(x) then represents the value of the feature y at location x where x ∈ Rn . Let q(y, Ω)

Optimization of Divergences within the Exponential Family for Image Segmentation

139

be the probability density function (pdf) of the feature y within the image region of interest. We now assume that we have a function Ψ : R+ × R+ → R+ which allows us to compare two pdfs. This function is small if the pdfs are similar and large otherwise. It allows us to introduce the following functional which represents the distance or more generally the divergence between the current pdf estimate q(y, Ω) and another one p(y) which may also depend on another domain:  D(Ω) = Ψ (q(y, Ω), p(y)) dy. (1) χ

The distance can be for example the symmetrized Kullback-Leibler divergence when p(y) Ψ (q, p) = 12 (p(y) log q(y,Ω) + q(y, Ω) log q(y,Ω) p(y) ). Such divergences represent a general setting for both segmentation with and without reference. Indeed, in segmentation problems, we generally search for homogeneous regions regarding with a given feature. We may then modelize the segmentation problem as the maximization of the distance between the pdf of the feature within the inside region and the pdf of the feature within the outside region. In order to fix ideas, let us consider a partition of an image in two regions where Ω is the inside region and Ω c the complementary outside region. The segmentation may then be formulated as the maximization of the following criterion:  D(Ω, Ω c ) = Ψ (q(y, Ω), p(y, Ω c )) dy. (2) χ

On the other hand, we can also consider that we have a reference histogram pref and that we search for the domain that minimizes the divergence between q and pref . This last framework may be applied to tracking or to supervised segmentation where a reference pdf is learned on the region of interest. The theoretical results given in this paper can be used for both applications. 2.2 Shape Gradient Descent In order to find an optimum, we perform a shape gradient descent using region-based active contours. We then have to compute the derivative of the criterion according to the domain using shape derivation tools [15]. Shape derivative tools applied to region-based active contours are described in [12, 6] and we won’t remind all the definitions in this paper. Let us just remind that, from the shape derivative, we can derive the evolution equation that will drive the active contour towards a (local) minimum of the criterion. Let us suppose that the shape derivative of the criterion D(Ω) in the direction V may be written as follows: < D (Ω), V >= −



speed(x, Ω)(V(x) · N(x))da(x) ,

(3)

∂Ω

where N is the unit inward normal to ∂Ω and da its area element. When minimizing the distance D(Ω), interpreting equation (3) as the L2 inner product on the space of velocities, the straightforward choice is to take V = speed(x, Ω)N. When minimizing the distance D(Ω), we can then deduce the following evolution equation: ∂Γ = speed(x, Ω) N(x) , ∂τ

(4)

140

F. Lecellier et al.

On the contrary, when maximizing the criterion, we take the opposite sign for the velocity.

3 General Results for Shape Derivative of Divergences within the Exponential Family In this paper, we consider that pdfs belong to the exponential family. In this case, the current pdf estimate q(y, Ω) is now indexed by a set of parameters θ ∈ Θ ⊂ Rκ (e.g. we have κ = 2 and θ = (μ, σ)T where μ is the mean and σ the variance for the Normal family). When using the exponential family, we rather index the pdf by η which is the natural parameter as explained below. In order to derive the criterion, we must take into account the dependence of the natural parameter with the domain. We then restrict our study to the full rank κ-parameter canonical exponential family [16]. For this family, we can establish a 1-1 correspondence between η and Ω and so compute directly the shape derivative of D(Ω). In the sequel, let us first introduce the exponential family and some properties and then explain the computation of the shape derivative. We then specialize our result when parameters are estimated using the Maximum Likelihood Estimation (MLE) method. We also give some results for the optimization of the Kullback-Leibler (KL) divergence. In this case, the shape derivative reduces to a very simple general expression. 3.1 The Exponential Family: Definition and Properties The multi-parameter exponential family [17] is naturally indexed by a κ-dimensional real parameter vector η and a κ-dimensional natural statistic vector T(Y ). We draw the reader’s attention to the fact that η is a function of θ ∈ Θ which is the parameter of interest in most applications (for the Gaussian distribution, we have θ = (μ, σ)T ). Definition 1. The family of distributions of a Random Variable (RV) Y {qθ : θ ∈ Θ ⊆ Rκ }, is said a κ-parameter canonical exponential family, if there exists real-valued functions: • • • •

η(θ) = [η 1 , ..., η κ ]T with η i : Θ ⊆ Rκ → R h:R→R B:Θ→R T = [T1 , ..., Tκ ]T : Rκ → R

such that the pdf qθ (y) may be written as: qθ (y) = h(y) exp[η(θ), T(y) − B(θ)]

with

y∈χ⊂R.

(5)

The term T is called the natural sufficient statistic and η the natural parameter vector. The term η, T denotes the scalar product. Letting the model be indexed by the natural parameter η rather that θ, the canonical κ−parameter exponential family generated by T and h is defined as follows: qη (y) = h(y) exp[η(θ), T(y) − A(η)] , (6)  +∞ with A(η) = log −∞ h(y) exp[η(θ), T(y)]dy. The natural parameter space is defined as E = {η ∈ Rκ ; −∞ < A(η) < +∞}.

Optimization of Divergences within the Exponential Family for Image Segmentation

141

Some Common Distributions. Table 1 provides a synthetic description of some common distributions of the exponential family: Table 1. Some common canonical exponential families. B(α, β) is the Euler Beta function. Distribution

θT

η(θ)T

T(y)T

Normal

(μ, σ 2 )

−1 ( σμ2 , 2σ 2)

(y, y 2 )

1 2



A(η) η2

− 2η12 − log

−η2 π



Gamma (λ, p) (−λ, p − 1) (y, log y) −(η2 + 1) log −η1 + log Γ (η2 + 1) Beta (r, s) (r − 1, s − 1) (log y, log(1 − y)) − log B(η1 + 1, η2 + 1) Poisson μ log μ y eη Exponential λ −λ y − log −η Rayleigh θ2 −1/2θ2 y2 − log −2η

Properties. The following results will be useful for our RBAC scheme based on the exponential family. Their proofs may be found in [16]. These properties give us a relation between the parameters η and the domain Ω through the use of the expectation of the natural statistics T(Y ). The first theorem provides general relations between the gradient of A and the expectation of T(Y ) while the second theorem allows us to establish a 1 − 1 correspondence between η and E[T(Y )] (for the full rank exponential family). Such a relation may then be used to express the parameter η and derive it according to the domain. Theorem 1. Let {qη : η ∈ E} a κ-parameter canonical exponential family with natural sufficient statistic T(Y ) and open natural parameter space E, we then have the following properties: 1. 2. 3. 4.

E is convex. A:E → S ⊆ R is convex. E[T(Y )] = ∇A(η). ¨ Cov[T (Y )] = A(η).

∂A ∂A ∂A T where ∇A = ( ∂η , , .., ∂η ) represents the gradient of A, and A¨ is the Hessian 1 ∂η2 κ 2 matrix of A with A¨ij = ∂ A . ∂ηi ∂ηj

The following theorem establishes the conditions of strict convexity of A, and then those for ∇A to be 1-1 on E. This is a very useful result for optimization (derivation) purposes: Theorem 2. Let {qη : η ∈ E} a full rank (i.e. Cov[T (Y )] is a positive-definite matrix) κ-parameter canonical exponential family with natural sufficient statistic T(Y ) and open natural parameter space E, we have [16]: 1. η → ∇A(η) is 1-1 on E. 2. The family may be uniquely parameterized by μ(η) ≡ E[T(Y )] = ∇A(η). 3. The anti-log-likelihood function is a strictly convex function of η on E. These results establish a 1-1 correspondence between η and E[T(Y )] such that: μ = ∇A(η) = E[(T(Y )] ⇔ E η = φ (E[T(Y )]) , holds uniquely with ∇A and φ continuous.

(7)

142

F. Lecellier et al.

Estimation of the Hyperparameters. The relation 7 allows us to express the parameter η as a function of E[(T(Y )]. In order to estimate the parameters, we replace E[T(Y )] by the empirical estimate of the mean T(Y ). This corresponds to the MLE of the parameter. Indeed, the MLE of η corresponds to minimizing the anti-log-likelihood score (for independent and identically distributed (iid) data). By differentiation of the anti-log-likelihood according to η, we find ∇A(η MLE ) = T (Y ). Note however that in this case, this is the discrete sample mean. The following example illustrates this stating: −1 Example 1. When dealing with the Rayleigh distribution, we have η = 2θ 2 , A(η) = 2  − log(−2η) and T (y) = y . By computing A (η) = T(Y ), we find that − η1 =  1 2 y(x)2 dx, which corresponds to the MLE of the parameter θ2 given by θML |Ω| Ω  1 = 2|Ω| y(x)2 dx. Ω

3.2 Shape Derivative of the Criterion In this section, we propose to derive according to the domain the functional (1). The dependence of the functional with the domain is due to the estimation of the parameter η detailed above. In the sequel, for the sake of simplicity, we will invariably denote η for the natural parameter and its finite sample estimate over the domain (without a ˆ ). We are now ready to state our main result: slight abuse of notation, this should be η Theorem 3. The Gâteaux derivative, in the direction of V, of the functional (1), is: < D (Ω), V >= ∇V η, C ,

(8)

where ∇V η = [< ∇η1 (Ω), V >, ..., < ∇ηκ (Ω), V >] is the Gâteaux derivative of η in the direction of V, ., . is the usual scalar product of two vectors and: C = E[∂1 Ψ (q(Y, η(Ω)), p(Y ))(T(Y ) − E[T(Y )])]. The term ∂1 Ψ denotes the partial derivative of Ψ according to the first variable. The proof is detailed in Appendix A.2. We then have to compute the shape derivative ∇V η. Such a computation requires an estimation of the expectation E[T(Y )] as explained in the next section. 3.3 Computing the Shape Derivative for the MLE Estimator As mentioned in section (3.1.3), the expectation E[T(Y )] can be replaced with the empirical estimate of the mean T(Y ) which is computed over the considered domain Ω. Using such an estimation for the hyperparameter, we can state the following proposition: Lemma 1. Within the full rank exponential family, and using the MLE estimator for the hyperparameters, the shape derivative ∇V η can be expressed as: ¨ −1 ∇V (T) . ∇V η = A(η)

(9)

Optimization of Divergences within the Exponential Family for Image Segmentation

143

¨ −1 = I(η)−1 is the inverse of the Hessian matrix of A¨ which is also the where A(η) fisher information matrix I. The derivative ∇V (T) is given by:    1 T(y) − T(y(a)) (V · N)da(x) , (10) ∇V (T) = |Ω| ∂Ω The proof is given in Appendix A.3. We can then replace the shape derivative of the natural parameters given in Lemma 1 in the general Theorem 3. The corollary that gives the shape derivative then follows: Corollary 1. The Gâteaux derivative, in the direction of V, of the functional (1), is: 1 < D (Ω), V >= |Ω| 

  κ

∂Ω

i=1

Ci

κ 

−1 ¨ [A(η)]ij (Tj (y) − Tj (y(a))) (V · N)da ,

j=1

where the κ components of the vector C are defined as follows: Ci = E[∂1 Ψ (q(Y, η(Ω)), p(Y ))(Ti (Y ) − Ti (Y )] i ∈ [1, κ]. The term ∂1 Ψ denotes the partial derivative of Ψ according to the first variable. In order to fix ideas, the functional D(Ω) can be chosen as the Kullback-Leibler divergence, in this case ∂1 Ψ (q, p) = log q + 1 − log p − pq . In order to compute the vector C in Corollary 1, we can assume that the pdf p belongs to the exponential family and to the same parametric law as the pdf q. Let us denote by η 1 the parameter of the pdf p. This parameter is supposed to be already computed or dependent of another domain and so does not depend on the domain Ω. We then state the following proposition: Lemma 2. When p(y, η 1 ) and q(y, η(Ω)) are two members of the exponential family that belong to the same parametric law with respective parameters η 1 and η, and when the functional D(Ω) is chosen as the KL divergence, we find for the vector C defined in Theorem 1: ¨ C = A(η)(η − η 1 ) + ∇A(η) − ∇A(η 1 ) . A proof is given in appendix C. This expression demonstrates that the derivative can be very simply computed using the natural parameters and the sufficient statistics of the law. Let us give two examples of computation for both the Rayleigh and the Gaussian law. Example 2. When dealing with the Rayleigh following example 2, with  distribution,  θ12 1 2 2 2 θ2 θ = 2 y , the term C is equal to C = 2θ θ2 − θ2 . we then find for the derivative 1 of KL divergence:  1 y(a)2 C < KL (Ω), V >= (1 − )(V · N)da(x) . (11) 2 |Ω| ∂Ω 2θ 2θ2 Example 3. When dealing with the Gaussian distribution, the term C is equal to

2 C=

( σσ 2 + 1)(μ − μ1 ) r 2  μ2 + σ 2 − μ21 − σ12 + 2 σσ 2 μ2 − μμ1 + σ 4 ( σ12 − 1

1

1 ) σ2

.

(12)

144

F. Lecellier et al.

We then find for the derivative of KL divergence: < KL (Ω), V >=    2μ 1 μ −(y − μ) C (1 + ) − C 1 2 2 σ 2 |Ω| ∂Ω σ2 σ   C2 μ +(y 2 − σ 2 − μ2 ) C1 2 − 2 (V · N)da(x) . σ 2σ

4 Maximisation of Divergences In this section, we propose to concentrate on the segmentation of an image into two regions (namely Ω and its complement Ω c ) by maximizing the criterion 2. 4.1 Evolution Equation When using the MLE estimator for the parameters, and noting that Ω and Ω c shares 1 the same boundary with opposite normals, we take T(y) = |Ω| Ω T(y(x))dx and  c 1 T(y) = |Ω c | Ω c T(y(x))dx. Using Corollary 1 and the fact that < D (Ω, Ω c ), V >= ∇V η, C + ∇V η c , Cc , we find for the evolution equation: κ κ  1   ∂Γ ¨ −1 (Tj (y) − Tj (y(x)) = Ci (Ω) A(η) ij ∂τ |Ω| i=1 j=1

−

κ κ   1  c ¨ c )−1 (Tj (y)c − Tj (y(x)) N. C (Ω ) A(η i ij |Ω c | i=1 j=1

For the KL divergence, the term C is evaluated as explained in section 3.3. A classical regularization term λκ is added where λ is a positive constant and κ the curvature. As far as the numerical implementation is concerned, we use the level set method approach first proposed by Osher and Sethian [18]. 4.2 Comparison with Other Methods in the Gaussian Case In this section, we propose to compare the behavior of our data term based on the maximization of the symmetrized Kullback-Leibler divergence between parametric pdfs to two other well-known region-based methods [7, 14]. The first method is the famous Chan & Vese method [14]. Such a criterion implies a Gaussian distribution for the feature y with a fixed variance. The corresponding evolution equation can be found in [14]. The second method has been first proposed by [7] and aims at minimizing the anti-loglikelihood for a Gaussian distribution. The evolution equation can be found in [7]. In order to compare these terms, let us express the non symmetrized KL divergence using the expectation under the pdf q, denoted by Eq , as follows: D(qp) = Eq [log(q(Y, η Ω ))] − Eq [log(p(Y, η Ω c ))]

(13)

Optimization of Divergences within the Exponential Family for Image Segmentation

145

To get the gist of using KLD as a criterion in an RBAC functional, consider the data yi = {y(x)|x ∈ Ω} as an iid sequence from the statistical model q(y, η Ω ). Using the weak law of large number for a very large domain Ω, the first term (which corresponds  1 to the entropy) can then be expressed as |Ω| log(q(y(x), η(Ω))dx. Maximizing the Ω first term in KL divergence can then be seen as equivalent to minimizing the anti-loglikelihood score [19] divided by the size of the sample (which corresponds to the entropy under the law of large number). Using the same assumptions, the second term of KL divergence can be seen as the minimization of the plausibility of the data provided by Ω c in the inside region Ω. When using the symmetrized version, we act both on Ω and Ω c . Let us now compare experimentally the behavior of these criterions for the extraction of an homogeneous region corrupted by a Gaussian noise in an image. We propose to take the example of the segmentation of the White Matter (WM) in T1-weighted brain MRI images. We perform the three evolution equations using the Gaussian assumption for the pdf of the feature y within each region. The feature y is chosen as the Intensity of the image. The initial contour is given in Figure 1.(a) and we also show the two initial pdfs (b), namely qη (I, Ω) which corresponds to the distribution of the intensity I inside the region Ω and qηc (I, Ω c ) which corresponds to the distribution of I inside

0.025 hist_in hist_out

0.02

0.015

0.01

0.005

0 0

50

100

150

200

Intensity

(a) initial contour

0.03

(b) associated pdfs

0.03

0.06

hist_in hist_out

hist_in hist_out

0.025

hist_in hist_out

0.025

0.05

0.02

0.02

0.04

0.015

0.015

0.03

0.01

0.01

0.02

0.005

0.005

0

0.01

0 0

50

100

150

Intensity

(c) Chan & Vese

200

0 0

50

100

150

Intensity

(d) log likelikood

200

0

50

100

150

200

Intensity

(e) KL maximization

Fig. 1. T1-weighted brain MRI segmentation results (extraction of the White Matter). The pdf of the intensity inside the contour is in solid line, the pdf of the intensity outside the contour is in dotted lines. (a): initial contour and (b) : associated pdfs, column (c): final contour and pdfs for the Chan & Vese method [14], column (d): for the log-likelihood method [7], column (e): for the maximization of the KL divergence.

146

F. Lecellier et al.

the region Ω c (i.e. outside the region ). In Figure 1, we can observe the final active contour obtained using our criterion (22) and the two other criterions mentioned above. We can remark that our criterion acts as an extractor of the most important Gaussian in the initial mixture of Gaussian (see Figure 1.e). The two other criterions separate the mixture without extracting a single Gaussian. So, with our method, we can directly obtain the White Matter of the brain without a multiphase scheme. 4.3 Examples of Applications In this part, we consider two examples of application (brain MRI images and contrast echocardiogaphy) using two different noise models (Gaussian and Rayleigh). Concerning 3D T1-weighted MRI images of the brain, the noise model is assumed to be represented by a Rician distribution [20]. For large signal intensities the noise distribution can be considered as a Gaussian distribution (this is the case for the White Matter (WM) or the Gray Matter (GM)). We propose in Figure 2 an example of WM segmentation by maximizing the KL divergence between Gaussian distributions. When evaluating quantitatively our results of WM segmentation on the simulated brain T1weighted MRI images provided by the Montreal Neurological Institute Brain Web URL, we find a dice coefficient of 0.91, a very law False Positive Fraction (FPF) of 0.8% and a True Positive Fraction (TPF) of 84%.

(a) 3D rendering of the WM

(b) slice 72

(c) slice 75

(d) slice 84

Fig. 2. 3D Segmentation of WM in a T1 brain MRI using KL maximization

As the Rayleigh distribution is well suited to model noise in echography [20], this noise model was applied for segmentation of the left ventricle in contrast echocardiography. Final contours for several images of the sequence are shown in Figure 3. The segmentation is accurate all along the sequence. Note that experimental results reported in [21, 4] prove that when using the appropriate noise model, segmentation results are more accurate and less sensitive to the choice of the regularization parameters.

frame 1

frame 31

frame 40

Fig. 3. Segmentation of the LV in a contrast echocardiographic sequence

Optimization of Divergences within the Exponential Family for Image Segmentation

147

References 1. Kass, M., Witkin, A., Terzopoulos, D.: Snakes: Active contour models. International Journal of Computer Vision 1, 321–332 (1988) 2. Caselles, V., Kimmel, R., Sapiro, G.: Geodesic active contours. International Journal of Computer Vision 22(1), 61–79 (1997) 3. Aujol, J.F., Aubert, G., Blanc-Féraud, L.: Wavelet-based level set evolution for classification of textured images. IEEE Transactions on Image Processing 12(12), 1634–1641 (2003) 4. Martin, P., Réfrégier, P., Goudail, F., Guérault, F.: Influence of the noise model on level set active contour segmentation. IEEE PAMI 26, 799–803 (2004) 5. Cremers, D., Rousson, M., Deriche, R.: A review of statistical approaches to level set segmentation: integrating color, texture, motion and shape. International Journal of Computer Vision 72(2), 195–215 (2007) 6. Aubert, G., Barlaud, M., Faugeras, O., Jehan-Besson, S.: Image segmentation using active contours: Calculus of variations or shape gradients? SIAM Applied Mathematics 63(6), 2128–2154 (2003) 7. Zhu, S., Yuille, A.: Region competition: unifying snakes, region growing, and bayes/MDL for multiband image segmentation. IEEE PAMI 18, 884–900 (1996) 8. Paragios, N., Deriche, R.: Geodesic active regions: A new paradigm to deal with frame partition problems in computer vision. JVCIR 13, 249–268 (2002) 9. Rougon, N., Discher, A., Prêteux, F.: Region-based statistical segmentation using informational active contours. In: SPIE Conf. on Mathematics of Data/Image Pattern Recognition, San Diego, CA (August 2006) 10. Michailovich, O., Rathi, Y., Tannenbaum, A.: Image segmentation using active contours driven by the bhattacharyya gradient flow. IEEE Transactions on Image Processing 16, 2787– 2801 (2007) 11. Wang, Z., Vemuri, B.: DTI segmentation using an information theoretic tensor dissimilarity measure. IEEE Transactions on Medical Imaging 24(10), 1267–1277 (2005) 12. Jehan-Besson, S., Barlaud, M., Aubert, G.: DREAM2 S: Deformable regions driven by an eulerian accurate minimization method for image and video segmentation. International Journal of Computer Vision (53) , 45–70 (2003) 13. Kullback, S.: Information Theory and Statistics. Wiley, New York (1959) 14. Chan, T.F., Vese, L.A.: Active contour without edges. IEEE Transactions on Image Processing 10, 266–277 (2001) 15. Delfour, M., Zolésio, J.: Shape and geometries. Advances in Design and Control. SIAM, Philadelphia (2001) 16. Bickel, P., Docksum, K.: Mathematical statistics: basic ideas and selected topics, 2nd edn., vol. I. Prentice-Hall, London (2001) 17. Koopman, P.: On distributions admitting a sufficient statistic. Trans. Am. Math. Soc. 39, 399–409 (1936) 18. Osher, S., Sethian, J.: Fronts propagating with curvature-dependent speed: Algorithms based on hamilton-jacobi formulation. Journal of Computational Physics 79, 12–49 (1988) 19. Cover, T.M., Thomas, J.A.: Elements of Information Theory. Wiley, New York (1991) 20. Goodman, J.: Some fundamental properties of speckle. J. of Optical Society of America 66, 1145–1150 (1976) 21. Lecellier, F., Jehan-Besson, S., Fadili, J., Aubert, G., Revenu, M.: Statistical region-based active contours with exponential family observations. In: ICASSP, vol. 2, pp. 113–116 (2006)

148

F. Lecellier et al.

A Appendix A.1 Shape Derivation Tools Let us remind this useful theorem [15] that will be used in the following proofs.  Theorem 4. The Gâteaux derivative of the functional J(Ω) = f (x, Ω) dx in the Ω   direction of V is: < J  (Ω), V >= fs (x, Ω, V)dx− f (x, Ω)(V·N )da(x) where Ω

∂Ω

N is the unit inward normal to ∂Ω, da its area element and fs the shape derivative of f [15]. A.2 Proof of Theorem 3 To compute < D (Ω), V >, we must first get the derivative of q(y(x), η) with respect to the domain, and apply the chain rule to Ψ (q(y(x), η), p(y)). To simplify the notation we write the Eulerian derivative of η as < η  (Ω), V >= ∇V η = [< η1 (Ω), V >, .., < ηκ (Ω), V >]T . Using the definition of q(y, η) given in (6) and the chain rule applied to A (η(Ω)), we obtain: = h(y) (∇V η, T(y)−∇V η, ∇A(η)) eη(Ω),T(y)−A(η(Ω)) (14) = q(y, η)∇V η, T(y) − ∇A(η) . By the chain rule applied to Ψ (q(y(x), η), p(y)), we get < Ψ  (q(y, η), p(y)), V >=< q  (y, η), V > ∂1 Ψ (q, p),  which gives < D (Ω), V >= χ q(y, η)∂1 Ψ (q, p)∇V η, T(y) − ∇A(η)dy. We introduce  C= q(y, η)∂1 Ψ (q, p) (T(y) − ∇A(η)) dy = E[∂1 Ψ (q, p) (T(Y ) − E[T(Y )])] χ

which completes the proof. A.3 Proof of Lemma1 When using the MLE, the term E[T(Y )] can be empirically estimated with T(Y ) and so derived easily with respect to the domain Ω. We propose to directly derive the expression ∇A(η) = T(Y ) which gives: κ  j=1

< ηj , V >

 ∂2A (η) =< Ti (Y ) , V > ∂ηi ∂ηj

∀i ∈ [1, κ] ,

(15)

¨ which can be written in the compact form ∇V (T) = A(η)∇ V η. ¨ Restricting our study to the full rank exponential family, where A(η) is a symmetric positive-definite, hence invertible, matrix (Theorem 2), the domain derivative of the pa¨ −1 ∇V (T) = ∇V η where ∇V (T) is given rameters η is uniquely determined by A(η)    1 by: ∇V (T) = |Ω| ∂Ω T(y) − T(y(a)) (V · N)da(x) (taking benefit of theorem 4) and the lemma follows.

Optimization of Divergences within the Exponential Family for Image Segmentation

149

A.4 Proof of Lemma 2 Since p and q belongs to the same parametric law, they share the same value for h(y), T(y) and A(η) and then log(q) − log(p) = η − η 1 , T(y) − A(η) + A(η 1 ). The value of C is then C = s1 − s2 , with: s1 = E[(η − η 1 , T(y) − A(η) + A(η 1 ) + 1)(Ti (Y ) − E[Ti (Y )]] p s2 = E[ (Ti (Y ) − E[Ti (Y )]]Ep [(Ti (Y ) − E[Ti (Y )]] q Developing the expression of the expectation of the second term,we find s2 = Ep [(Ti (Y ) − E[Ti (Y )]] = ∇A(η 1 ) − ∇A(η). Using the linearity of the expectation and the fact that E[Tj (Y )(Ti (Y )] − E[Ti (Y )]E[Tj (Y )] designates the co¨ ij = variance matrix of the sufficient statistics T and can then be replaced by A(η) κ ¨ ¨ ¨ Cov[T(Y )]ij = A(η)ji , we find: s1 = j=1 (ηj − η1j )A(η)ij , and then C = A(η) (η − η1 ) + ∇A(η) − ∇A(η 1 ).

Convex Multi-class Image Labeling by Simplex-Constrained Total Variation Jan Lellmann, Jörg Kappes, Jing Yuan, Florian Becker, and Christoph Schnörr Image and Pattern Analysis Group (IPA) HCI, Dept. of Mathematics and Computer Science, University of Heidelberg {lellmann,kappes,yuanjing,becker,schnoerr}@math.uni-heidelberg.de

Abstract. Multi-class labeling is one of the core problems in image analysis. We show how this combinatorial problem can be approximately solved using tools from convex optimization. We suggest a novel functional based on a multidimensional total variation formulation, allowing for a broad range of data terms. Optimization is carried out in the operator splitting framework using Douglas-Rachford Splitting. In this connection, we compare two methods to solve the Rudin-Osher-Fatemi type subproblems and demonstrate the performance of our approach on single- and multichannel images.

1

Introduction

In this paper, we study the variational approach  inf f (u) , f (u) = − u(x), s(x)dx + λ TV(u) , u∈C

λ>0,

(1)

Ω

for determining a labeling u : Ω → RL , that is a contextual classification of each pixel x ∈ Ω into one out of L classes, based on an arbitrary vector-valued similarity function s(x) ∈ RL as input data that has been computed from image data beforehand. The objective function (1) comprises the common form of a data term plus a regularization term. The data term is given by the L2 inner product of the assignment variables u and the similarity function s, and the regularizer is a total variation (TV) formulation for vector-valued data,   TV(u) = ∇u1 2 + · · · + ∇uL 2 dx . (2) Ω

Furthermore, the constraint u ∈ C restricts the vector field u(x) at each location x ∈ Ω to lie in the standard probability simplex, that is u(x) ∈ RL + and  L  i=1 u(x) i = 1 for all x ∈ Ω. Our work is motivated by the following observation. Suppose that at each pixel x ∈ Ω, there is an unambiguous assignment (labeling) of the data s(x) to some class l ∈ {1, . . . , L} represented by the corresponding l-th unit vector, X.-C. Tai et al. (Eds.): SSVM 2009, LNCS 5567, pp. 150–162, 2009. c Springer-Verlag Berlin Heidelberg 2009 

Convex Multi-class Image Labeling by Simplex-Constrained Total Variation

151

Fig. 1. Left: Noisy input image. Right: The labeled image based on the non-binary assignment u as global minimizer of the convex approach (1). The discrete problem is accurately solved by a continuous approach.

u(x) = el . Then, an interface with√area A between two image regions labeled with l and l , respectively, adds A 2 to the regularization term iff l = l , as all but two gradients under the square root vanish. As a result, under these √ assumptions and up to the immaterial constant 2, the TV term corresponds to the well-known Potts model that assigns constant penalties to local changes of the labeling. A significant difference between the Potts model and our approach (1), however, is that the former amounts to solve a discrete combinatorial problem, whereas the latter is a continuous convex optimization problem. Experiments show that our approach (1) approximates discrete decisions fairly well (Fig. 1 and 2) by computing a global optimum to a single convex optimization problem. By contrast, the state-of-the-art discrete approach [1] approximates the combinatorial solution by solving a non-uniquely defined sequence of binary problems via graph cuts. This fact, along with the potential of continuous convex optimization for parallel implementations and their more robust dependency on (hyper-) parameters, motivated to investigate the approach (1) as a promising model for a general “labeling submodule” within computer vision systems. To this end, – We have a closer look at the data and regularization terms (section 2). – We apply an operator splitting approach to (1) in order to decompose the computation of a globally optimal labeling into two independent computational steps: TV denoising for vector-valued data, and projection of the labeling vectors u(x) on the canonical simplex (section 3). – We evaluate two different algorithms for the TV denoising subroutine (section 4) and compare the performance of our convex method to a range of established graph cut-based approaches (section 5). Related work. In contrast to the binary case with anisotropic discretization [2], multi-class energies are generally not submodular and thus cannot be optimized globally using graph cuts [3]. Some extensions exist, which find a local minimum by solving a sequence of binary graph cuts [1]. The continuous formulation – optimization on the set of characteristic functions – is known as continuous cut [5]. Chan et al. [6] showed that this problem can be relaxed and solved on

152

J. Lellmann et al.

Fig. 2. Output of the standard TV approach [4] for scalar-valued images applied to the noisy input image depicted in Fig. 1, for different values of the regularization parameter λ. Irrespective of this value, the performance is worse than with the approach (1) (cf. Fig. 1, right), because the latter approximates the Potts model that does not depend on the size (contrast) of discontinuities. Consequently, the former approach cannot remove noise without degrading weak discontinuities, as is apparent above for the horizontal discontinuities.

a convex set, without losing global optimality. In contrast, our work is aimed at the multi-class case. In [7], a comparable approach based on [8] was presented, which relies on a natural ordering of the labels, as given in e.g. stereo reconstruction. An approach very similar to ours was recently presented in [9], where the authors use a different formulation of the total variation on vector fields, and an alternating optimization method. The (discrete) Potts model was studied in [10], where approximate solutions were computed by an LP relaxation with explicit constraints. In contrast, our approach considers the general TV term and a problem decomposition into efficiently solvable subproblems, without the need to introduce additional variables. Notation. We consider the discretized version of our approach (1). Let Ω = {1, . . . , n1 } × · · · × {1, . . . , nd } ⊆ Rd , d ∈ N, denote a regular image grid of n := |Ω| pixels. The (multidimensional) image space X := Rn×L is equipped with the Euclidean inner product ·, ·Ω over the vectorized elements. We naturally identify v = (v 1 , . . . , v L ) ∈ Rn×L with ((v 1 ) · · · (v L ) ) ∈ RnL . Superscripts v i denote a collection of vectors, while subscripts vk denote vector components. Using the notation e = (1, 1, . . . , 1) , the standard simplex on RL  n×L L and its extension C on R are given by ΔL := v ∈ R v ≥ 0 , e, v = 1

and C := x∈Ω ΔL . Define δC (x) to be 0 iff x ∈ C, and +∞ otherwise.   Let grad := (grad 1 , . . . , gradd ) be the d-dimensional forward difference gradient operator for Neumann boundary conditions. Accordingly, div := −grad is the backward difference divergence operator for Dirichlet boundary conditions. These operators extend to Rn×L via Grad := (IL ⊗ grad), Div := (IL ⊗ div), where IL is the L × L identity matrix. We will also need the convex sets L   12    Bλ := (p1 , . . . , pL ) ∈ Rd×L  pi 22 λ , i=1

(3)

Convex Multi-class Image Labeling by Simplex-Constrained Total Variation



Dλ :=

Bλ ⊆ Rn×d×L , Eλ := {u ∈ Rn×L |u = Div p , p ∈ Dλ } .

153

(4)

x∈Ω

The discrete total variation on vector-valued data is then defined as

TV(u) := σE1 (u) = Gx u2 ,

(5)

x∈Ω

where σM (u) := supp∈M u, p is the support function from convex analysis, and Gx is an (Ld)×n matrix composed of rows of (Grad) s.t. Gx u gives the gradients of all ui in x stacked one above the other.

2

Variational Approach

Based on the introduced notation, our novel approach (1) reads inf f (u) ,

u∈C

λ TV(u) , f (u) = −u, sΩ +       data term regularization term

λ>0,

(6)

As the objective function f and the constraint set C are convex, the overall problem is convex as well. We will now define and motivate each term. Data Term. The data term in (6) is fairly general.   Any vector-valued similarity function s can be used, whose components s(x) i indicate the affinity of some data point at x with class i. As an example, suppose we have image features g(x), x ∈ Ω, prototypical feature vectors G = (G1 , . . . , GL ) as well as a distance measure d on the features. We might think of g as a grayscale image, of G as some prototypical gray values, and of d as a quadratic distance measure, possibly derived from a statistical noise model. The hard assignment of the pixel x ∈ Ω to a label (or class) l(x) ∈ {1, . . . , L} should then be penalized by the distance d(g(x), Gl(x) ) of the corresponding feature to the prototype of the assigned class. Denoting the negative distance by s, and summing up over the image domain, we see that 



 s(x), u(x) for u(x) = el(x) . d g(x), Gl(x) = − (7) x∈Ω

x∈Ω

Thus, instead of looking for l ∈ {1, . . . , L}n , we may equivalently look for u ∈ {e1 , . . . , eL }n . However, the right hand side formulation has the advantage that it extends naturally to the soft assignment u ∈ C: We may now solve the easier problem of optimizing for u on the convex set C. In our experiments, we chose d(x, y) = ||x − y||1 , as the 1 -norm is still convex but known to be more robust against noise and outliers. However, s is not restricted to representing distances. In fact, it may be arbitrarily nonlinear and nonconvex in x and g, and involve nonlocal operations on g. The complexity is completely hidden within the precomputed vector s. Regularization Term. Recall that the regularizer of (6) is defined (5) as

TV(u) = sup u, Div p = Gx u2 . (8) p∈D1

x∈Ω

154

J. Lellmann et al.

This definition for vector-valued u parallels the definition of the “isotropic” total variation measure in the scalar-valued case [11, 4, 12]. It is also known as MTV [13, 14, 15], and was recently studied in [16] in its continuous formulation. Contrary to the anisotropic discretization, where one would substitute the sum of 1-norms in (3), it is less biased towards edges parallel to the axes. See also [17] for an overview of TV-based research and applications. Optimality. After solving the relaxed problem, it remains to show that a binary solution can be recovered. For the continuous, binary case, Chan et al. [6] showed that an exact solution can be obtained by thresholding at almost any threshold. However, their results do not immediately transfer to the discrete multi-class case. In particular, the crucial “layer cake” formula holds for 1 -, but not 2 discretizations of the TV. Contrary to the binary case, it is not clear which rounding scheme to use for vector-valued u. For our experiments, we chose the final class label for each pixel x as the index l of the maximal u∗l (x) of the global optimum u∗ of (6). This defines a suboptimal discrete solution u∗t . Bounding the error f (u∗t ) − f (u∗d) with respect to the unknown discrete optimum u∗d will be subject of our future work.

3

Optimization

Two basic problems arise concerning the optimization of (6): Nondifferentiability of the objective function due to the TV term, and handling of the simplex constraint u ∈ C. We cope with the latter using the tight Douglas-Rachford splitting method as presented in the following section. We refer to [18] for the full derivations. Douglas-Rachford Splitting. Minimization of a proper, convex, lower-semicontinuous (lsc) function f : X → R can be regarded as finding a zero of its (necessarily maximal monotone [19, Chap. 12]) subgradient operator T := ∂f : X ⇒ X. In the operator splitting framework, ∂f is assumed to be decomposable into the sum of two “simple” operators, T = A + B, of which forward and backward steps can practically be computed. Here, we consider the (tight) Douglas-Rachford-Splitting iteration [20, 21], z k+1 ∈ (Jτ A (2Jτ B − I) + (I − Jτ B ))(z k ) ,

(9)

where Jτ T := (I + τ T )−1 is the resolvent of T . Under the very general constraint that A and B are maximal monotone and A + B has at least one zero, the sequence (z k ) will converge to a point z, with the additional property that x := Jτ B (z) is a zero of T ( [22, Thm. 3.15], [22, Prop. 3.20], [22, Prop. 3.19], [23]). In particular, for f = f1 +f2 , fi proper, convex, lsc with ri(dom f1 )∩ri(dom f2 ) = ∅ (ri(S) denoting the relative interior of a set S), it can be shown [19, Cor. 10.9] that ∂f = ∂f1 + ∂f2 , and the ∂fi are maximal monotone. As x ∈ Jτ ∂fi (y) ⇔ x = argmin(2τ )−1 x − y22 + fi (x), the computation of the resolvents reduces to proximal point optimization problems involving only the fi .

Convex Multi-class Image Labeling by Simplex-Constrained Total Variation

155

Application. For our specific problem, we split inf (f1 (u) + f2 (u)) , f1 (u) = −u, sΩ + λTV(u) , f2 (u) = δC (u) .

u∈C

(10)

and get the following Douglas-Rachford scheme: Algorithm 1. Outer loop (Douglas-Rachford) 1: choose some u0 and a fixed step size τ > 0 2: repeat 1 3: solve uk ← argminu { 2τ u − z k 2 − u, s + σEλ (u)} 1 k 4: solve w ← argminw { 2τ w − (2uk − z k )2 + δC (w)} 5: z k+1 ← z k + wk − uk 6: until uk − uk−1 ∞  δouter .

As f is bounded from below on the compact set C and thus attains its minimum. From the remarks in the last section, we get convergence of the scheme for the discrete case: δC (w) and σEλ are both proper, convex, lsc with dom σEλ = Rn and ri(C) = ∅. In practice, one has to deal with solutions of the subproblems with limited accuracy. While there are extensions of the convergence result that take these inexact solutions into account [22, Prop. 4.50], they require the subproblems to be solved with increasing accuracy. However, we found that the method generally converged even though these requirements were not met.

4

Inner Loop Optimization

The second subproblem (Alg. 1, step 4) is a projection on the constraint set, wk = ΠC (2uk − z k ), which requires one projection on the low-dimensional unit simplex ΔL per x ∈ Ω. These projections can be computed in a finite number of steps [24]. The first subproblem (step 3) is equivalent to 1 uk = argminu u − (z k + τ s)2 + (τ λ)T V (u), (11) 2 i.e. an extension to vector vector-valued u of the classical Rudin-Osher-Fatemi (ROF, TV-L2 ) problem with regularization parameter τ λ. Many methods have been suggested to solve the ROF problem, e.g. PDE, fixpoint, or interior point methods for primal [4, 25], dual [26, 27, 28], or mixed [29] formulations. We evaluate two approaches: First, we will formulate a particularly simple gradient projection method in the operator splitting framework, cf. [30]. This scheme was introduced in [27] and extended to the multidimensional case in [31] (see also [16]). The second approach is based on the fast half-quadratic method of Yang et al. [15]. −1 k Forward-backward approach. The optimality of step (z −  k 3, τ   k condition  u) + s ∈ ∂σEλ (u), can be rewritten as u = τ z /τ + s − ΠEλ z /τ + s . To compute the projection ΠEλ , we use the dual representation,   1 1 2 2 ΠEλ (x) = argmin q − xΩ = Div argmin Div p − xΩ + δDλ (p) . (12) 2 p q∈Eλ 2

156

J. Lellmann et al.

Using a simple forward-backward splitting for the inner problem results in the (gradient projection) update rule  pj+1 = ΠDλ p − νDiv (Div p − x) . The projection ΠDλ can be computed explicitly and is separable in x, while the inner part can be computed for all models independently. This opens up the method to parallelization. Convergence is guaranteed for ν < 2/Div Div (see e.g. [22,√Thm. 3.12]). Extending the argument in [26, Thm. 3.1], we find that div  4d. Accord1 ingly, we may set ν < 2d . In our experiments, we set ν = 0.95 2d to avoid numerical problems close to the theoretical maximum. Wrapping up, we have Algorithm 2. Inner loop, forward-backward approach 1: 2: 3: 4: 5:

k

x ← zτ + s, choose arbitrary p0 ∈ Rn×d×L repeat pj+1 = ΠDλ (pj − νDiv (Div p − x)) until pj+1 − pj ∞  δinner uk ← τ (x − Div pj+1 ).

Half-quadratic approach. While the forward-backward method is simple and easy to implement, its convergence speed is in practice not satisfactory. As an alternative, we tested an ROF specialization of the general multichannel image restoration method by Yang et al. [15]. Starting from (11), the problem is to find μ uk = argminu g(u) , g(u) := u − f 2 + T V (u) , (13) 2 where μ := τ1λ and f := z k + τ s. Using a half-quadratic approach [32, 33], Yang et al. derive the splitting/penalty formulation 

 μ β 2 (u, y) = argmin yx  + yx − Gx u + u − f 2Ω . (14) 2 2 Ld nL yx ∈R ,x∈Ω,u∈R x∈Ω

The parameter β controls smoothing of the total variation; setting β  n/(2ε) guarantees ε-suboptimality of the solution of the smoothed problem with respect to the original problem (for a derivation see [18]). Equation (14) can be solved using alternating minimization w.r.t. u and the auxiliary variables yx . The latter is highly parallelizable, as it boils down to n separate explicit operations:  yxj+1 = max Gx u − β −1 , 0 (Gx u/Gx u) . (15) On the other hand, minimizing (14) for u amounts to solving  μ Grad Grad + (μ/β)I(nL) uj+1 = Grad y j+1 + f, β for uj+1 , where y j+1 is a proper rearrangement of the yx .

(16)

Convex Multi-class Image Labeling by Simplex-Constrained Total Variation

157

Fig. 3. Results of the speed comparison between forward-backward (FB) and halfquadratic method (HQ) for the inner problem, applied to data from the first iteration of the outer problem (cf. Table 1). Left to right: Original input, FB with τ λ = 5, HQ with τ λ = 5, FB with τ λ = 20, HQ with τ λ = 20. Iteration counts were fixed at 80 resp. 300 to equalize the runtime for both approaches. For larger regularization parameter, the half-quadratic method outperforms the forward-backward approach as smoothness increases.

For periodic boundary conditions, Yang et al. solved (16) rapidly using FFT. In our case, Neumann boundary conditions and thus the Discrete Cosine Transform (DCT-2) [34] are appropriate. This requires 2L independent (parallelizable) individual DCTs which can be efficiently computed in O(n log n) each. By the alternating application of the above two steps, we can solve (14) for fixed β large enough for any required suboptimality bound. In practice, convergence can be sped up by starting with a small β and solving a sequence of problems for increasing β, warm-starting each with the solution for the previous problem. Given an arbitrary u0 ∈ RnL , the complete algorithm reads Algorithm 3. Inner loop, half-quadratic approach 1: while stopping criterium not satisfied do 2: compute y j+1 from (15) 3: compute uj+1 from y j+1 and (16), 4: possibly increase β 5: end while The stopping criteria can be based on the residual [15]. For our experiments, we set a fixed iteration count, as increasing β at each step turned out to lead to fastest convergence, and residua for different β are not comparable.

5

Experiments, Performance Evaluation

Inner Problem. We compared the half-quadratic approach to the conventional forward-backward method. The difficulty with the former lies in the choice of the update strategy for β. We chose a generalization of the exponential strategy outlined in [15]: Set β = βmin and update by multiplying with c := (βmax /βmin)1/K for some K until β = βmax . We made the following observations: – In order to rapidly minimize the objective function, it is best to use a continuation strategy, i.e. to increase β at each step, rather than spending time on solving (14) exactly for each β.

158

J. Lellmann et al.

– Increasing K generally improves the quality of the result. – For fixed βmax and K, there seems to be a unique optimal βmin that minimizes the final objective function value. With the continuation strategy and fixed βmax , we found the optimal βmin to usually lie in the range of 10−5 βmax to 10−3 βmax . Unfortunately, there seems to be a strong dependency on the choice of λ as well as the scale and complexity of s. We set βmin = 0.2 · 10−4 βmax , which worked well for our data. βmax was set at n/0.2 according to a suboptimality bound of ε = 0.1 (section 4). We compared the performance of the two methods in terms of the objective function value for fixed runtime of the optimized Matlab implementations (Fig. 3, Table 1). For larger τ λ,the half-quadratic method gives better results. For τ λ = 20, less than 10 iterations are required to reach the quality of 300 iterations of the forward-backward method, giving a speedup of about 4-5. However, finding the optimal parameter set is more involved than for the forward-backward method. Table 1. Run times t (in seconds), objective function values r and relative differences (rHQ − rFB )/rHQ for the experiment in Fig. 3. For larger τ λ, the half-quadratic method gives more accurate results in the same time. τλ 0.1 1 2 5 10 20 50 tHQ 1.14 1.23 1.20 1.31 0.98 0.95 1.08 tFB 1.03 1.02 1.06 1.03 1.22 1.25 1.19 rHQ 3901.9 27660.7 36778.5 40038.8 42262.8 44377.1 44752.5 rFB 3901.9 27660.4 36760.6 40104.3 42924.3 46988.6 57504.9 rel. diff. 1.17e-16 1.24e-5 4.85e-4 -1.64e-3 -0.0156 -0.0588 -0.285

Overall Problem. We evaluated the performance of our algorithm against five different methods in their publicly available implementations from the Middlebury MRF benchmark [35]: Belief Propagation (BP), Sequential Belief Propagation (BPS), Graph Cuts with alpha-expansion (GCE) and alpha-beta swap (GCS), and Sequential Tree Reweighted Belief Propagation (TRBPS). Of each of the grayscale 32 × 32 images, 20 noisy copies were generated and segmented into four gray levels with fixed intensities. In view of the last section and in order not to mix up speed with accuracy issues, we used the forward-backward approach for the inner loop. We set δinner = 1 · 10−3 , δouter = 2 · 10−2, and τ = 1. For small λ, our method shows results comparable to the other approaches with respect to the number of bad labels. We point out again that this solution to the non-binary labeling problem is achieved by solving the convex optimization problem (6) followed by local rounding as explained in section 2. In contrast to our method, the MRF benchmark algorithms optimize the anisotropic energy. To compensate, their λ was scaled by a common factor of √ ≈ 2 that was found empirically. Nevertheless, the discretization gives them a small advantage on images with axis parallel edges (experiments 1 and 2).

Convex Multi-class Image Labeling by Simplex-Constrained Total Variation

159

25

50 bp bps gce gcs trws tv

40 30

bp bps gce gcs trws tv

20 15 10

20 10 0 0

Standard deviation

Incorrect labels (mean %)

Fig. 4. Exemplary grayscale segmentation results for the benchmarked methods for four labels. Left to right: Noisy input data, final results for BP, BPS, GCE, GCS, TRWS, and the proposed method (TV). λ was manually chosen for each method. Axis-parallel edges are better recovered by the anisotropic methods, while our isotropic discretization has an advantage on diagonal edges.

5 0

0.2

0.4

0.6

λ

0.8

1

1.2

1.4

0

0.2

0.4

0.6

λ

0.8

1

1.2

1.4

Fig. 5. Error rates for the first experiment in Fig. 4. For each λ, all experiments were repeated 20 times with random noise (zero-mean Gaussian with σ = 0.45, 0.35, 0.25 resp. 0.35 for experiments 1-4 and image intensities in [0, 1]), and the percentage of incorrectly assigned labels compared to ground truth was recorded. Sequential Belief Propagation (BPS) generally performed worst, while our method (TV) was on par with the others, in particular for lower λ. The figure also reveals that belief propagation (BP) gets stuck in a good, but often inferior local optimum, and does not respond to larger values of λ, i.e. stronger regularization requested by the user.

Figure 6 demonstrates the performance of our algorithm for color segmentation. Only few outer iterations (20 in our case) are necessary for accurate optimization.

160

J. Lellmann et al.

Fig. 6. Performance of our method for four-class segmentation based on 1 color distance. Left to right: Ground truth, inspired by [29, 36]; ground truth overlaid with Gaussian noise, σ = 1; local nearest-neighbor labeling; our approach with λ = 0.7 after 20 outer iterations. The energy of the result is about 1% lower than the energy of the ground truth, suggesting that at this noise level, further improvements are limited by the model.

6

Conclusion and Future Work

In this paper, we presented a convex variational approach to solve the combinatorial multi-labeling problem for energies involving a general data term, total-variation-like regularizers, and simplex constraints. To enforce the simplex constraint, we based our approach on the globally convergent Douglas-Rachford operator splitting scheme. We evaluated two methods in order to efficiently solve the ROF-type subproblems, and showed that the half-quadratic approach allows faster convergence at the price of more involved parameter tuning. Experiments showed that the quality of the generated labelings is comparable to state of the art discrete optimization methods, and can be achieved by just solving a convex optimization problem. Due to the generality of the data term, our method allows for a wide range of features or distance measures. To fully evaluate these possibilities in connection with variations of the TV measure is a subject of our future research. Acknowledgements. Jing Yuan gratefully acknowledges support by the German National Science Foundation (DFG) under grant SCHN 457/9-1.

References 1. Boykov, Y., Veksler, O., Zabih, R.: Fast approximate energy minimization via graph cuts. PAMI 23(11), 1222–1239 (2001) 2. Boykov, Y., Kolmogorov, V.: An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision. PAMI 26(9), 1124–1137 (2004) 3. Kolmogorov, V., Zabih, R.: What energy functions can be minimized via graph cuts? PAMI 26(2), 147–159 (2004) 4. Rudin, L., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Physica D 60, 259–268 (1992) 5. Strang, G.: Maximal flow through a domain. Math. Prog. 26, 123–143 (1983)

Convex Multi-class Image Labeling by Simplex-Constrained Total Variation

161

6. Chan, T.F., Esedo¯ glu, S., Nikolova, M.: Algorithms for finding global minimizers of image segmentation and denoising models. J. Appl. Math. 66(5), 1632–1648 (2006) 7. Pock, T., Schönemann, T., Graber, G., Bischof, H., Cremers, D.: A convex formulation of continuous multi-label problems. In: ECCV, vol. 3, pp. 792–805 (2008) 8. Ishikawa, H.: Exact optimization for Markov random fields with convex priors. PAMI 25(10), 1333–1336 (2003) 9. Zach, C., Gallup, D., Frahm, J.M., Niethammer, M.: Fast global labeling for realtime stereo using multiple plane sweeps. In: VMV (2008) 10. Kleinberg, J., Tardos, E.: Approximation algorithms for classification problems with pairwise relationships: Metric labeling and MRFs. In: FOCS, pp. 14–23 (1999) 11. Ziemer, W.: Weakly Differentiable Functions. Springer, Heidelberg (1989) 12. Meyer, Y.: Oscillating Patterns in Image Processing and Nonlinear Evolution Equations. Univ. Lect. Series, vol. 22. AMS (2001) 13. Sapiro, G., Ringach, D.L.: Anisotropic diffusion of multi-valued images with applications to color filtering. Trans. Image Process. 5, 1582–1586 (1996) 14. Chan, T.F., Shen, J.: Image processing and analysis. SIAM, Philadelphia (2005) 15. Yang, J., Yin, W., Zhang, Y., Wang, Y.: A fast algorithm for edge-preserving variational multichannel image restoration. Tech. Rep. 08-09, Rice Univ. (2008) 16. Duval, V., Aujol, J.F., Vese, L.: A projected gradient algorithm for color image decomposition. CMLA Preprint (2008-21) (2008) 17. Chan, T., Esedoglu, S., Park, F., Yip, A.: Total variation image restoration: Overview and recent developments. In: The Handbook of Mathematical Models in Computer Vision. Springer, Heidelberg (2005) 18. Lellmann, J., Kappes, J., Yuan, J., Becker, F., Schnörr, C.: Convex multi-class image labeling by simplex-constrained total variation. TR, U. of Heidelberg (2008) 19. Rockafellar, R., Wets, R.J.B.: Variational Analysis, 2nd edn. Springer, Heidelberg (2004) 20. Douglas, J., Rachford, H.H.: On the numerical solution of heat conduction problems in two and three space variables. Trans. of the AMS 82(2), 421–439 (1956) 21. Lions, P.L., Mercier, B.: Splitting algorithms for the sum of two nonlinear operators. SIAM Journal on Numerical Analysis 16(6), 964–979 (1979) 22. Eckstein, J.: Splitting Methods for Monotone Operators with Application to Parallel Optimization. PhD thesis, MIT (1989) 23. Eckstein, J., Bertsekas, D.P.: On the Douglas-Rachford splitting method and the proximal point algorithm for max. mon. operators. M. Prog. 55, 293–318 (1992) 24. Michelot, C.: A finite algorithm for finding the projection of a point onto the canonical simplex of Rn . J. Optim. Theory and Appl. 50(1), 195–200 (1986) 25. Dobson, D.C., Curtis, Vogel, R.: Iterative methods for total variation denoising. J. Sci. Comput 17, 227–238 (1996) 26. Chambolle, A.: An algorithm for total variation minimization and applications. JMIV 20, 89–97 (2004) 27. Chambolle, A.: Total variation minimization and a class of binary MRF models. In: Rangarajan, A., Vemuri, B.C., Yuille, A.L. (eds.) EMMCVPR 2005. LNCS, vol. 3757, pp. 136–152. Springer, Heidelberg (2005) 28. Aujol, J.F.: Some algorithms for total variation based image restoration. CMLA Preprint (2008-05) (2008) 29. Chan, T.F., Golub, G.H., Mulet, P.: A nonlinear primal-dual method for total variation-based image restoration. J. Sci. Comput. 20, 1964–1977 (1999) 30. Combettes, P.L., Wajs, V.R.: Signal recovery by proximal forward-backward splitting. SIAM J. Multisc. Model. Sim. 4(4), 1168–1200 (2005)

162

J. Lellmann et al.

31. Bresson, X., Chan, T.: Fast minimization of the vectorial total variation norm and applications to color image processing. Tech. Rep. 07-25, UCLA (2007) 32. Geman, D., Yang, C.: Nonlinear image recovery with halfquadratic regularization. IEEE Trans. Image Proc. 4(7), 932–946 (1995) 33. Cohen, L.: Auxiliary variables and two-step iterative algorithms in computer vision problems. JMIV 6(1), 59–83 (1996) 34. Strang, G.: The discrete cosine transform. SIAM Review 41(1), 135–147 (1999) 35. Szeliski, R., Zabih, R., Scharstein, D., Veksler, O., Kolmogorov, V., Agarwala, A., Tappen, M., Rother, C.: A comparative study of energy minimization methods for Markov random fields. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3952, pp. 16–29. Springer, Heidelberg (2006) 36. Hintermüller, M., Stadler, G.: An infeasible primal-dual algorithm for total bounded variation-based inf-convolution-type image restoration. J. Sci. Comput. 28(1), 1–23 (2006)

Geodesically Linked Active Contours: Evolution Strategy Based on Minimal Paths Julien Mille and Laurent D. Cohen CEREMADE, UMR CNRS 7534, Université Paris IX-Dauphine Place du Maréchal de Lattre de Tassigny, 75016 Paris, France {mille,cohen}@ceremade.dauphine.fr

Abstract. The proposed method is related to parametric and geodesic active contours as well as minimal paths, in the context of image segmentation1 . Our geodesically linked active contour model consists in a set of vertices connected by paths of minimal cost. This makes up a closed piecewise defined curve, over which an edge or region energy functional is formulated. The greedy algorithm is used to move vertices towards a configuration minimizing the energy functional. This evolution technique ensures lower sensitivity to erroneous local minima than usual gradient descent of the energy. Our method intends to take advantage of explicit active contours, minimal paths and greedy evolution techniques.

1

Introduction

Among well known variational models for image segmentation, active contours have drawn lively interest since their introduction by Kass et al [1]. Their key principle is the research of a curve minimizing an energy functional, which mainly depends on the adequacy of the curve to the target object. Active contours are implemented either with a parametric curve - in which case they are often referred to as ’snakes’ - or in an implicit fashion based on the level set framework [2] [3]. Early active contour models are mainly parametric and boundary-based, as the data term of the energy functional is an edge indicator function integrated along the curve. The Euler-Lagrange equation, determined by calculus of variations, indicates the minimizing flow to be followed by gradient descent scheme. These models are dependent of curve parameterization and unable to adapt their topology. Moreover, gradient descent is sensitive to local minima of the energy functional. Parameterization invariance is achieved by the geodesic active contour model [4], which introduces a geometrically intrinsic functional, whereas topology adaptiveness is provided by the level set implementation. Significant attempts have been made to decrease the sensitivity to local minima, based either on the gradient descent direction or on the minimization method itself. The balloon force [5] falls into the first category, as it adds a normal-oriented inflation or retraction component, in order to increase the capture range of the snake. As regards the evolution process, several heuristics based 1

This work was partially supported by ANR grant NanoGPSCellulaire ANR-05NANO- 045-06.

X.-C. Tai et al. (Eds.): SSVM 2009, LNCS 5567, pp. 163–174, 2009. c Springer-Verlag Berlin Heidelberg 2009 

164

J. Mille and L.D. Cohen

on local searches have been proposed as alternatives to gradient descent, including dynamic programming [6] [7] and the greedy algorithm [8] [9]. The latter, which is subsequently addressed in the paper, considers the energy as a sum of curve points energies. It basically consists in iteratively moving curve points to locations minimizing their own energies, these locations belonging to a search window. On the other hand, the minimal path approach by Cohen and Kimmel [10], which seeks for a curve of minimal cost between two end-points, can be used to recover open and closed boundaries. It is closely related to the geodesic active contour with respect to the functional to be optimized, but has in addition the main benefit of finding a global minimum efficiently thanks to the Fast Marching technique [11]. In this paper, we deal with an explicit implementation of active contour, i.e. a discrete curve defined by control points, or vertices. The described method is both related to minimal paths and greedy search. Our geodesically linked active contour model is made up of a set of vertices connected by paths of minimal cost with respect to a boundary-based metric. We define search windows centered at each vertex and evolve vertices according to a greedy fashion. Making a given vertex movable and the other ones still, we consider every geodesically linked contour passing through the points in the window of the moving vertex. This last one is finally moved to the location leading to the contour of smallest energy. The motivation for this work resides in several points. Firstly, the minimal path approach alone can only find a minimizer of an edge functional, with one or several(s) fixed input end-point(s). Conversely, our model is suitable to any energy functional, which we prove by endowing it with edge-based or different regionbased energies, including the minimal variance of the Chan and Vese model [12]. We believe that describing the curve with geodesics is pertinent whatever the energy functional is. Indeed, whether the functional holds edge, region and/or even shape prior terms, the major part of the final curve will be located on more or less salient edges. In comparison to snakes driven by gradient descent, the use of search windows significantly reduces sensitivity to erroneous local minima and energy weights tuning.

2 2.1

Background Parametric and Geodesic Active Contours

The active contour model is represented as a plane curve Γ with C 2 position vector c(u) = [x(u) y(u)]. Segmentation of an object of interest is performed by finding the curve minimizing an energy functional E, which has the general form:  1 E(Γ ) = L(c, c , c )du (1) 0

where L is usually made up of internal terms regularizing the curve and external terms attaching the curve to image data. According to calculus of variations, the following variational derivative vanishes if the curve is a local minimizer of E:     δE ∂L d ∂L d2 ∂L = − + 2 (2) δΓ ∂c du ∂c du ∂c

Geodesically Linked Active Contours

165

Curve evolution is usually performed by gradient descent, taking the opposite variational derivative as a descent direction. Given an image I defined over D ∈ R2 , they use the following edge indicator g, which is a decreasing function of gradient magnitude (the image is usually convolved with the derivative of a gaussian). The original parametric snake [1] has the following energy and variational derivative:  1 2 2 Lsnake (c, c , c ) = α c  + β c  + g(c) 2 (3) δEsnake = −αc + βc + ∇g δΓ The energy functional of the snake is dependent on parameterization. This has an impact on discretization, since the energy varies in terms of sampling when the contour is implemented as a polygonal curve. The geodesic active contour (GAC) [4] solves the parameterization issue by introducing an intrinsic energy functional, weighting the edge indicator by length element c : LGAC (c, c , c ) = g(c) c  (4)

δEGAC = (∇g, n − κg) n δΓ

where n and κ are the unit inward normal vector and curvature, respectively. Hence, the flow resulting from the geometric energy also holds a regularization term. This model lends itself to level set implementation, allowing topology changes. Boundary-based models driven by gradient descent, whether parametric or geodesic, are relatively blind to neighboring structures and may get trapped in local minima induced by noise. To increase the capture range, the balloon force was introduced in [5] for parametric contours, whereas an advection term is used in [3] for level sets. Despite such techniques, gradient descent may still cause the contour to miss or pass through significant boundaries. The minimal path method addresses this issue by finding a global minimum of the energy. 2.2

Minimal Paths

The minimal path approach by Cohen and Kimmel [10] aims at finding curves of minimal lengths in a Riemannian space endowed with an heterogeneous isotropic metric. The length of path C is:  1  1 L(C) = (5) P˜ (C(s))ds = P˜ (C(u)) C  (u) du 0

0

where s is the arc length. Potential P˜ , which defines the isotropic metric, should be chosen according to the application. Curves located on image boundaries are detected by using an edge-dependent potential P˜ (x) = w + g(x), where w is a regularizing constant. Hence the cost of C may be rewritten using euclidean length:  1  1 L(C) = (w + g(C(s)))ds = wLeuclidean(C) + g(C(s))ds (6) 0

0

166

J. Mille and L.D. Cohen

With respect to the energy functional to be minimized, the minimal path approach is similar to the geodesic active contour model, as can be seen in term LGAC of eq. 4. However, the minimal path has the avantageous difference of reaching the global minimum of the energy, given two fixed end-points x0 and x1 . Starting from point x0 , the minimal action map U0 should be calculated. It corresponds to the minimal cost integrated along a path starting at x0 and ending at x:    C(0) = x0 U0 (x) = inf L(C) s.t. C(1) = x C The action map U0 is the viscosity solution of the Eikonal equation ∇ U  = P˜ with initial condition U (x0 ) = 0. This allows U0 to be computed by the Fast Marching method [11], which is similar in principle to Dijkstra’s graph search algorithm. Once the action map has been computed, the geodesic γ, i.e. the path of minimal action linking a point x1 to x0 , is found by back-propagation starting from x1 until x0 is reached: γ  = −∇U0 (γ). In its initial formulation, the minimal path method determines an open curve between two fixed end-points. It is also able to find closed contours by providing only one point on the final contour and detecting a saddle point on the minimal action map [10]. 2.3

Greedy Algorithm

Along with dynamic programming [6] [7], greedy methods deal with discrete energy functionals. The greedy algorithm for active contours, as developed in [8], seeks for a minimizer of the energy by means of a set of local optimizations. It is only applicable on explicit implementations, where the contour is represented as a polygon with n vertices {vi }1≤i≤n . The total energy is considered as a sum of vertex energies: n  E(Γ ) = Evertex (vi ) i=1

where Evertex is the discretization of the energy at a given vertex, using finite differences. Considering the snake term Lsnake of eq. 3, it comes:  1 2 2 α vi − vi−1  + β vi+1 − 2vi − vi−1  + g(vi ) (7) Evertex (vi ) = 2 Vertices are successively moved in order to minimize their own energies. At each iteration, a square window of width m is considered around the current vertex. ˜ i in the window. The energy of the latter is computed at each tested position v The vertex is then moved to the position leading to the lowest energy, which is summarized by the evolution scheme, at iteration t: (t+1)

vi

= arg min  Evertex (˜ vi ) ˜ i ∈W vi(t) v

where W(x) is the window centered at point x. The initial greedy algorithm [8] performs in O(nm2 ) operations. The window size has an obvious impact on computational cost, but also on convergence abilities. Indeed, the contour can capture

Geodesically Linked Active Contours

167

farther structures as the window is larger. The greedy algorithm is by essence a discrete optimization heuristic. The formulation of the variational derivative is not used and continuous calculus of variations is thus not necessary.

3

The Geodesically Linked Active Contour

We develop an approach taking advantage of above described methods. Our geodesically linked active contour is based simultaneously on an explicit implementation of active contours, minimal paths and the greedy algorithm. Basically, we deal with an evolving explicit closed curve, allowing initialization inside or around the target object without providing fixed points. Minimal paths coupled with a geometric energy functional allows a parameterization-free handling of the contour. The use of the greedy algorithm, as opposed to gradient descent, guarentees better robustness to local minima. 3.1

Minimal Paths to Connect Vertices

Let us consider a set of n linked vertices S = {vi }1≤i≤n . We denote as γi (u) = [xi (u) yi (u)] the geodesic path connecting vi to vi+1 :    C(0) = vi γi = arg min L(C) s.t. (8) C(1) = vi+1 C where the cost functional L is defined in eq. 5. At every step of the evolution algorithm, the set of geodesics {γi }1≤i≤n describes a closed piecewise differentiable contour Γ , which euclidean length is: Leuclidean(Γ ) =

n   i=1

0

1

γi (u) du

One may note that a concatenation of geodesics γi is not a geodesic itself, since it is forced to pass through given points. To some extent, curve Γ may be considered as a piecewise minimizer of an edge-based functional. If a uniform potential P˜ (x) = 1 was chosen, the geodesics would become straight lines of equation γi (u) = (1 − u)vi + uvi+1 , u ∈ [0, 1], in which case Γ would represent a polygon. Fig. 1 depicts geodesically linked contours with uniform potential and edge-based potential (dark smooth lines represent high image gradient areas). As described in section 2.2, path γi is determined by gradient descent of the minimal action map Ui+1 of origin vi+1 . Given start point vi+1 , the Fast Marching algorithm [11] allows to specify one or several end points (in our case vi ) so that propagation can be stopped when vi is reached. This prevents the whole image from being visited by the Fast Marching and saves computational time. In the case of edge-based segmentation, the interest of describing the evolving contour with geodesics is obvious. Indeed, in the end of deformation, the

168

J. Mille and L.D. Cohen

(a)

(b)

Fig. 1. Vertices linked by geodesics with uniform potential (a) and edge-based potential (b)

geodesics fit the actual boundaries of the sought object. On the other hand, in the case of region-based segmentation, image edges are not explicitly searched. However, we believe that linking vertices with geodesics is relevant for any usual segmentation criterion. We may assume that the final contour should be partially located on more or less salient boundaries, whatever energy functional is optimized. In subsequent sections, we formulate three energies independently implemented on the geodesically linked active contour, namely the edge, region and narrow band region energies. Before, we recall Green’s theorem, which we use to convert domain integrals into boundary integrals. For every region R and real-valued function f over R2 , we have:   f (x)dx = P dx + Qdy (9) R

∂R

where [P (x) Q(x)] is a continuously differentiable vector field such that:   1 x 1 y Q(x, y) = f (t, y)dt P (x, y) = − f (x, t)dt 2 −∞ 2 −∞

(10)

The theorem expects that ∂R should be at least piecewise smooth, it is thus applicable to the geodesically linked active contour. For instance, to express the area of region Rin enclosed by Γ , we consider eq. 9 with f (x) = 1: n  1 1 |Rin | = xi (u)yi (u) − xi (u)yi (u)du 2 i=1 0 3.2

Edge Energy

Boundary-based segmentation is performed by minimizing an edge energy. The edge indicator function g is integrated along geodesics. In order not to penalize lengthy contours, the edge energy is normalized by euclidean length: n  1  1 g(γi (u)) γi  (u) du Eedge (Γ ) = Leuclidean(Γ ) i=1 0 Note that according to eq. 6, the integral of g along γi equals Ui+1 (vi ) minus the euclidean length Leuclidean(γi ). Hence, once the action maps have been computed, the edge indicator does not need to be summed over geodesics again. With the edge energy alone, if the search space of vertex coordinates is too small, the

Geodesically Linked Active Contours

169

contour fails at capturing actual boundaries when initialized far from them. To increase the capture range, we add an area-dependent term, which minimization acts like a balloon force [5]: Eballoon (Γ ) =

|D| − |Rin | |D|

where |D| is the image area. In that case, the total energy is a weighted sum of edge and balloon energies. 3.3

Region Energy

The increasing use of region terms has proven to overcome limitations of edgebased only models, especially when dealing with data sets suffering from noise and lack of contrast between neighboring structures. Classical region-based deformable models segment images according to statistical data computed over the object of interest and the background. Image partitions should be uniform in terms of pixel intensities or higher level features like texture descriptors. We rely on the intensity variance, which is close to the two-phase Mumford-Shah segmentation model by Chan and Vese [12]. The average intensity in the inner region is expressed using Green’s theorem: μ(Rin ) =

1 |Rin |



1  |Rin | i=1 n

I(x)dx = Rin



1

0

xi P (γi ) + yi Q(γi )du

where P and Q are the summed intensities (see template formulas in eq. 10). Then, the inner intensity variance is:   1 1 2 2 σ (Rin ) = (I(x) − μ(Rin )) dx = I 2 (x)dx − μ(Rin )2 |Rin | Rin |Rin | Rin where the integral of squared intensities may also be expanded according to Green’s theorem. Corresponding quantities on the outer region may be expressed using relation    f (x)dx = Rout

3.4

D

f (x)dx −

f (x)dx Rin

Narrow Band Region Energy

The ideal case of uniform regions is rarely encountered in real applications, as the background usually contains structures of various intensities. Hence, strict homogeneity is not necessarily a desirable property. In order to account for spatially varying intensity, local statistics in region-based segmentation have emerged recently [13] [14]. The narrow band principle, which has proven its efficiency in the evolution of level sets [3], is used in our approach to formulate a local region term [15].

170

J. Mille and L.D. Cohen

Γ

Γ[−B]

Γ[B]

Bin Bout Fig. 2. Inner and outer bands for narrow band region energy

Instead of dealing with whole domains Rin and Rout , we consider an inner band Bin and an outer band Bout in the vicinity of the contour, as depicted in fig. 2. The narrow band region energy is the intensity variance over the bands: Eband (Γ ) = σ 2 (Bin ) + σ 2 (Bout ) Our narrow band region energy is based on parallel curves [16]. We define curve γ[B] i as a parallel curve of γi : γ[B] i (u) = γi (u) + Bni (u)

(11)

where B is the user-defined band thickness, constant along the curve, and ni is the inward unit normal to geodesic γi . Hereafter, we will use the index [B] to denote all quantities related to the parallel curve. Bands Bin and Bout are bounded by parallel curves of the n geodesics γi , respectively γ[B] i and γ[−B] i . We assume that geodesics are smooth enough so that their parallel curves do not self-intersect nor exhibit singularities. An important property resulting from the definition in eq. 11 is that the velocity vector of the parallel curve can be expressed as a function of the velocity vector of the initial curve, as well as its curvature and normal. Using the identity ni = − κi γi  , we have: γ[B] i  = γi  + Bni  = (1 − Bκi )γi 

(12)

By a change of variable, an integral over inner band Bin may be expressed explicitly in terms of the curve and band thickness:  n  1 B  f (x)dx = f (γi + bni ) γi  (1 − bκi ) db du (13) Bin

0

i=1

0

We use the template formula in eq. 13 to express the mean and variance of intensities in the inner band:  n  1  1 B μ(Bin ) = I(γi + bni ) γi  (1 − bκi ) db du |Bin | i=1 0 0 1  σ (Bin ) = |Bin | i=1 2

n



0

1



0

B

(I(γi + bni ) − μ(Bin ))2 γi  (1 − bκi ) db du

and similarly for the outer band, replacing b with −b.

Geodesically Linked Active Contours

3.5

171

Evolution with Greedy Algorithm

Vertices should be moved in order to minimize the selected energy. This is usually performed with gradient descent of the Euler-Lagrange equation. In our case, it is difficult to differentiate Eedge , Eregion or Eband with respect to a given vertex vi , since these energies depend on geodesics to vi (see eq. (8)). The greedy algorithm presented in section 2.3 provides us a way to evolve vertices without differentiating the energy. Motion of curve points can always be decomposed into normal and tangential components. While the geometry of the curve is modified by normal displacements, tangential motion only affects curve parameterization [4]. Since the distribution of vertices along the contour can be updated with a resampling technique, we only consider normal displacement in the greedy evolution. We define a normal-oriented window WN of length m centered at vertex vi :   m m

  WN (vi ) = vi + knvi  k ∈ − , 2 2 where nvi is the inward unit normal vector, estimated by finite difference using the second and next-to-last points of geodesics γi and γi+1 , respectively. Since steps between successive points in the window are integers, the window may be computed using a Bresenham-like algorithm.

γ˜i ˜i v

vi+1

vi

γ˜i−1

vi−1 Fig. 3. Geodesics linking neighboring vertices to points in search window

Greedy evolution is performed by moving vertex vi to the position in the window which corresponding geodesically linked contour has the smallest en˜ i belonging to the window. The associated ergy. Let us consider a test position v geodesics γ˜i−1 and γ˜i link it to the neighbors of vi , as depicted in fig. 3. The energy of the corresponding geodesically linked contour Γ˜ = {γ1 , ..., γi−2 , γ˜i−1 , γ˜i , γi+1 , ..., γn } is computed and compared to the energy of the initial contour Γ . All window points are tested in this way, so that the evolution scheme at iteration t is: (t+1)

vi

=

arg min 

˜ i ∈WN v

(t) vi



E(Γ˜ )

172

J. Mille and L.D. Cohen

where E is one of the previously described energies. If we consider set H = {1, ..., i − 2} ∪ {i + 1, ...n} holding indices of geodesics not influenced by a modification on vi , all quantities involved in the energies are written with constant ˜ i . For instance, the area of the tested inner and variable parts with respect to v region is decomposed:  1   ˜   xj (u)yj (u) − xj (u)yj (u) du Rin  = 

j∈H

+ 0

0 1

 x ˜i−1 y˜i−1

−

x ˜i−1 y˜i−1

 du + 0

1

x ˜i y˜i − x ˜i y˜i du

˜ i need to be comThis implies that the part of energies invariant with respect to v puted only once, before moving vi . Finally, once all vertices have been treated, resampling may be performed to maintain consistent distribution of vertices along the curve.

4

Experiments

We tested the geodesically linked active contour with the three different energy configurations (edge+balloon, region and narrow band region). A comparison

Fig. 4. Segmentation of left ventricle: initialization (a) and final location (b) of the geodesically linked active contour, initialization (c) and final location (d) of the parametric contour

Geodesically Linked Active Contours

173

with a parametric snake endowed with the same energies is provided. The snake was initialized as a small circle inside the area of interest, far from the target boundaries. Similarly, the initial vertices of our model are sampled on a circle. Results are shown in fig. 4. For each row, columns (a) and (b) represents the initial and final states of the geodesically linked active contour, respectively. Columns (c) and (d) represent the same states for the snake. For all experiments, the regularization weight w was set to 0.25, which achieved sufficient regularization for all tested images. The size of the window was m = 50 and the maximal inter-vertex distance for resampling was set to 20. The image in row 1, which was segmented using the edge energy, depicts the gapclosing ability of the model. The geodesically linked active contour managed to pass through false edges and reach actual boundaries. Thanks to the large search window, it turned out to be rather unsensitive to balloon strength, as values for coefficient α in the range [0.1, 4] were suitable. On the other hand, the balloon coefficient has a strong influence on the gradient descent-driven parametric snake, which yields difficult parameter tuning. Actually, it was not possible to find a correct balloon weight allowing to jump false edges while stabilizing on real ones. The image in row 2, which was segmented using the region energy, depicts a similar phenomenon. The geodesically linked contour does not get trapped in small gaps in the region, which could present an interest for segmentation of partially occluded objects. Row 3 depicts a MRI of the heart left ventricle, which was used to put the narrow band region energy into application. The band thickness B is an important parameter. Apart from its impact on the algorithmic complexity - computing intensity means and variances on the bands takes at least O(nB) operations it controls the trade-off between local and global features around the object. If B = 1, the region energy is as local as an edge term. The main image property having an effect on the minimal band thickness is the edges sharpness. Indeed, the deformable curve needs a larger band as the boundaries of the target object are fuzzy. However, B = 10 was a suitable value in our experiments. Note that we depict the state of the parametric snake before self-collision. One may note that an unconstrained region-based level set method would also properly segment images in row 2 and 3. However, this remark should be moderated by the fact our model is dedicated to applications where topology preservation is needed.

5

Conclusion and Perspectives

We proposed the geodesically linked active contour model for image segmentation. The model lies on an explicitly implemented curve moved by an evolution method based on minimal paths and a greedy algorithm. Linking curve points with geodesics solves parameterization issues and allows the contour to fit the most salient boundaries at every step of deformation. Displacing vertices according to a greedy search ensured lower sensitivity to erroneous local minima than usual gradient descent of the energy. The model was endowed with edge and

174

J. Mille and L.D. Cohen

region energies and was validated on a few datasets. Further work may focus on developing an adaptive search window for greedy evolution. Currently, the window length is constant whatever the values of energies or the previous positions of vertices. We believe the algorithm could be improved by adapting the window length with respect to these properties, in order to avoid visiting positions that would not seemingly minimize the energy.

References 1. Kass, M., Witkin, A., Terzopoulos, D.: Snakes: active contour models. International Journal of Computer Vision 1(4), 321–331 (1988) 2. Osher, S., Sethian, J.: Fronts propagation with curvature-dependent speed: algorithms based on Hamilton-Jacobi formulations. Journal of Computational Physics 79, 12–49 (1988) 3. Malladi, R., Sethian, J., Vemuri, B.: Shape modeling with front propagation: a level set approach. IEEE Transactions on Pattern Analysis and Machine Intelligence 17(2), 158–175 (1995) 4. Caselles, V., Kimmel, R., Sapiro, G.: Geodesic active contours. International Journal of Computer Vision 22(1), 61–79 (1997) 5. Cohen, L.: On active contour models and balloons. Computer Vision, Graphics, and Image Processing: Image Understanding 53(2), 211–218 (1991) 6. Amini, A., Weymouth, T., Rain, R.: Using dynamic programming for solving variational problems in vision. IEEE Transactions on Pattern Analysis and Machine Intelligence 12(9), 855–867 (1990) 7. Geiger, D., Gupta, A., Luiz, A., Vlontzos, J.: Dynamic programming for detecting, tracking, and matching deformable contours. IEEE Transactions on Pattern Analysis and Machine Intelligence 17(3), 294–302 (1995) 8. Williams, D., Shah, M.: A fast algorithm for active contours and curvature estimation. Computer Vision, Graphics, and Image Processing: Image Understanding 55(1), 14–26 (1992) 9. Sakalli, M., Lam, K.M., Yan, H.: A faster converging snake algorithm to locate object boundaries. IEEE Transactions on Image Processing 15(5), 1182–1191 (2006) 10. Cohen, L., Kimmel, R.: Global minimum for active contour models: a minimal path approach. International Journal of Computer Vision 24(1), 57–78 (1997) 11. Sethian, J.: A fast marching level set method for monotonically advancing fronts. Proceedings of the National Academy of Science 93(4), 1591–1595 (1996) 12. Chan, T., Vese, L.: Active contours without edges. IEEE Transactions on Image Processing 10(2), 266–277 (2001) 13. Piovano, J., Rousson, M., Papadopoulo, T.: Efficient segmentation of piecewise smooth images. In: Sgallari, F., Murli, A., Paragios, N. (eds.) SSVM 2007. LNCS, vol. 4485, pp. 709–720. Springer, Heidelberg (2007) 14. Lankton, S., Tannenbaum, A.: Localizing region-based active contours. IEEE Transactions on Image Processing 17(11), 2029–2039 (2008) 15. Mille, J., Boné, R., Cohen, L.: Region-based 2D deformable generalized cylinder for narrow structures segmentation. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008, Part II. LNCS, vol. 5303, pp. 392–404. Springer, Heidelberg (2008) 16. Farouki, R., Neff, C.: Analytic properties of plane offset curves. Computer Aided Geometric Design 7(1-4), 83–99 (1990)

Validation of Watershed Regions by Scale-Space Statistics Tomoya Sakai and Atsushi Imiya Institute of Media and Information Technology, Chiba University, Japan {tsakai,imiya}@faculty.chiba-u.jp

Abstract. This paper shows a potential use of scale space for statistical validation of watershed regions of a greyscale image. The watershed segmentation has difficulty in distinguishing valid watershed regions associated with real structures of the image from invalid random regions due to background noise. In this paper, a hierarchy of watershed regions is established by following merging process of the regions in a Gaussian scale space. The distribution of annihilation scales (lives) of the regional minima is investigated to statistically judge the regions as being valid or not. Recursive validation using the hierarchy prevents oversegmentation due to the randomness.

1

Introduction

The aim of this study is to develop a statistical validation scheme for segmentation of a greyscale image. If we do not have a priori knowledge on the shapes or structures of objects in the image, topographic features of the greyscale image, and the watersheds in particular, are useful for unsupervised image segmentation. A well-known phenomenon in the watershed segmentation is oversegmentation, that is, producing a large number of undesired tiny regions. Since the undesired watershed regions are mainly caused by noise in the image, it is desirable to settle the oversegmentation problem by taking account of statistical properties of the randomness. There is a body of literature dealing with the oversegmentation problem of watersheds [1,2,3,4,5,6,7,8]. In the antecedent work, most schemes for preventing the oversegmentation attempt to hierarchically merge the oversegmented regions on the basis of similarity between adjacent regions measured by the MDL [3], colour distance [8], and so on. Diffusion-based multiscale image representations are also used for merging the regions [5, 6, 8], since the scale space theory [9,10,11,12,13,14,15] mathematically underpins topological relationships among the topographic features without a priori knowledge about them. The oversegmentation can be reduced by selecting levels in the hierarchy of regions, or by setting lower bounds to the scale above and below which the watersheds are valid and invalid, respectively. In this paper, we show that the scale-space treatment of the image is also useful for the statistical analysis of the random watershed regions. The validity X.-C. Tai et al. (Eds.): SSVM 2009, LNCS 5567, pp. 175–186, 2009. c Springer-Verlag Berlin Heidelberg 2009 

176

T. Sakai and A. Imiya

of a watershed region can be quantified in terms of the statistical confidence of distinguishing it from invalid watershed regions due to randomness. We present a fully unsupervised watershed segmentation algorithm, in which the watershed regions are recursively validated according to their hierarchical relationships in the scale space.

2 2.1

Watershed Segmentation with Variable Scale Gaussian Scale Space

In the Gausian scale-space theory [9,10,11,12,14,15,16], a one-parameter family of nonnegative functions is derived from a d-dimensional greyscale image f (x), x ∈ Rd . f (x, σ) = G(x, σ) ∗ f (x) (1) Here, “∗” expresses d-dimensional convolution, and G(x, σ) is an isotropic Gaussian function with the scale σ.   |x|2 1 (2) G(x, σ) = √ d exp − 2 2σ 2π σ d We redefine the d-dimensional greyscale image and its scale-space representation in the extended real scale and space as follows. Definition 1. A d-dimensional greyscale image is defined as  a nonnegative d ¯ d with a finite net image intensity scalar function f (x), x ∈ R x∈R¯ d f (x)dx . ¯ d, R ¯ + ), is the convoDefinition 2. The scale-space image f (x, σ), (x, σ) ∈ (R lution of the greyscale image f (x) with the isotropic Gaussian kernel G(x, σ). ¯ d and R ¯ + denote the d-dimensional extended real space including a point Here, R at infinity and the extended real scale including an infinite scale, respectively. Although the domain of a greyscale image in practice is bounded within a limited area or volume, we embed such an image in the extended real scale space. The point at infinity will be theoretically used as a representative point of the background of the image in the watershed segmentation later. 2.2

Watershed Segmentation and Hierarchy of Regions

The watershed segmentation was derived from spatial partitioning on the basis of the drainage patterns of rainfall. As the topographic height map defines the boundaries of the catchment basins draining to the same lowest points, a twodimensional greyscale image defines the watershed boundary curves enclosing regions with local minima when we regard the image intensity as the topographic height. For a d-dimensional image, the entire space is partitioned by (d − 1)dimensional hypersurfaces into d-dimensional watershed regions. Each watershed region defined by a smooth function f (x) contains a unique local minimum, to

Validation of Watershed Regions by Scale-Space Statistics

177

which any point in the watershed region is connected by a gradient curve of f (x). In practice, the watershed segmentation of the gradient image |∇f (x)| is known to provide better intuitive partitions than that of the image f (x) itself [2, 5, 6, 8] because object boundaries in a scene may cause large spatial changes in the image intensity. Simple computation of the watersheds of the images results in oversegmentation caused by tiny and insignificant catchment basins. As suggested in the antecedent work [3, 5, 6, 8], hierarchical relationships among the watershed regions are of great help for merging the oversegmented regions. We employ the scale-space framework to derive the hierarchy because the scale-space axioms are acceptable in general cases where any prior information about the similarities among the unexpected watershed regions are not given. If we apply the gradient watershed segmentation to the image f (x, σ) with the variable scale σ, we can observe the evolution of the watersheds with respect to scale. The catastrophy theory applied to the gradient watershed segmentation in the Gaussian scale space [5] shows that the gradient watershed regions of f (x, σ) may be generically annihilated, merged, created and splitted with increasing scale σ. Therefore, hierarchical watershed segmentation using multiscale representation of the image [2, 6, 8] is essentially the extraction of the hierarchical relationships among the watershed regions in the scale space through the generic events. Since every watershed region is represented by its local minimum, the trajectories of the regional minima in scale space describe the relationships among the regions. For the purpose of validation of the regions, we derive the hierarchy from all the traceable regional minima from the finest scale along their trajectories in scale space. We trace the trajectories by local minimisation at every level of scale [16]. In an annihilation or merging event, two regional minima and a saddle between them are involved. We regard one of these two regional minima as a child of the resulting regional minimum after the event. We trace only one of two local minima after a creation or splitting event because we are interested in the hierarchy of the regions at the finest scale. Remark that the point at infinity is a local minimum which exists at any scale. The local minimum at infinity is the regional minimum of the image background because the rainfall in the background region is drained to this ideal point. The following algorithm RegionHierarchy traces every trajectory of the regional minimum from every pixel p ∈ P at σ = 0 until the regional minimum disappears or goes outside the image boundary toward the local minimum at infinity with increasing scale. RegionHierarchy(set of pixel centres P , image f (p ∈ P )) 1 let G be a graph with card(P ) + 1 nodes with the labels l = 0, . . . , N where l = 0 represents the point at infinity; 2 store σlt = ∞ in all nodes of G; 3 set σmax to be the size of the convex hull of P ; 4 σ := 0; 5 Q := P ; 6 while card(Q)  = 1 or σ < σmax do

178

7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

T. Sakai and A. Imiya

Q := Q; σ := σ + Δσ, where Δσ is a small value compared with the space intervals of the points Q; for each q l ∈ Q do update q l by minimising |∇f (x, σ)|2 with q l as the initial position 1 ; if q l is outside the convex hull of P then connect the two nodes of G labelled 0 and l; end if end for let L be a list of labels corresponding to the points in Q; while card(L)  = 1 do pop a label l from L; n := NearestNeighbour(L, l); if |q l − q n | < εσ, where q l , q n ∈ Q, and ε is the tolerance of minimisation then if |q l − q l | > |q n − q n |, where q l , q n ∈ Q then child := l and parent := n; else child := n and parent := l; end if connect the two nodes of G labelled parent and child; t set σchild := σ; remove q child from Q; end if end while end while return G.

The resulting graph G is a set of trees representing the hierarchy of the watershed regions of the gradient image. Any node in G represents a watershed region consisting of the pixels indicated by its subtree nodes. The annihilation or merging scale σ t is stored at the node in G corresponding to p. We utilise the bicubic spline interpolation [17] to seach for the local minimum with subpixel precision in Step 10. The function NearestNeighbour in Step 18 searches for the nearest point to pl in the set of points listed in L and returns its label. The annihilation or merging event is detected in Step 19, and one of the two regional minima with larger displacement is identified as the child in Step 20. Figure 1 shows an example of the trajectories of regional minima and the region hierarchy obtained by RegionHierarchy. Since the set of tree, G, expresses hierarchical relationships among the image pixels, any tree node with a scale σ > 0 represents a set of pixels consisting a watershed region.

1

It is trivial that the watershed regions of the gradient magnitude squared |∇f |2 are identical to those of the gradient magnitude |∇f |.

Validation of Watershed Regions by Scale-Space Statistics

179

σ

(a)

(b)

(c)

Fig. 1. Trajectories of regional minima and region hierarchy. (a) A noisy 96 × 96 image f (x) embeded in a dark background. (b) Gradient magnitude squared |∇f (x, σ = 20)|2 . The brighter the larger magnitude. (c) Trajectories of the regional minima in scale space. The thick curves (blue) are the parts of the trajectories for σ > 5. The thin straight lines (red) are the edges of G between the nodes with σ > 5.

2.3

Scale Selection Problem

We need a criterion to select the scales or the tree levels in hierarchy. One may expect that the watersheds of the image f (x, σ) at a small scale σ well approximates the boundary of true image regions. However, if noise spoils the fine structure of the image, the estimated watersheds at small scales are stochastic and experimentally less reproducible. The noise is suppressed at a large scale, but the watershed segmentation is poor in terms of detection ability and localisation: the edges of small watershed regions are smoothed out, and the boundary shapes of large regions are simplified. Since the randomness is the major cause of the oversegmentation problem in the watershed methods [1, 4, 5], the oversegmentation problem should be resolved in a statistical manner.

3 3.1

Validation of Watershed Regions Valid Watershed Regions

Generally, a greyscale image expresses spatial distribution of a measured physical quantity. The true image f true(x), which we want to measure and apply the watershed segmentation to, is inevitably spoiled by random noise through the measurement. Therefore, the actual image f (x) presents valid watersheds related to those of the true image f true (x) and invalid watersheds due to the randomness. Assertion 1. A valid watershed region of an observed gradient image |∇f (x)| is related to one of the watershed regions of the true gradient image |∇f true (x)|. Since the watershed regions are represented by the region minima, the image f (x) has the valid watershed regions of the gradient image |∇f (x)| iff the true gradient image |∇f true (x)| has corresponding local minima. Contrapositively,

180

T. Sakai and A. Imiya

iff |∇f true (x)| is a featureless image without any local minimum, then no valid watershed exists for any observation f (x), which should be considered as an image of the background only. This condition means that f true (x) = 0 everywhere ¯ d because of the Definition 1. Therefore, f (x) for f true (x) = 0, i.e., the in R noise image, produces only the invalid watershed regions. The valid watershed region must be statistically distinguishable from such invalid region. From this viewpoint, the validity of the watershed region is interpreted as the statistical confidence in rejecting the following null hypothesis. Null hypothesis H0 : The watershed region is that of the noise image. Alternative hypothesis H1 : The watershed region is not that of the noise image. The null hypothesis H0 is rejected if the regional minimum is distinguishable from that of the noise image using test statistics. 3.2

Life Distribution

An important fact is that the randomness of the image f (x, σ) is filtered out as the scale σ increases, and deterministic features of the image f (x) emerge at large scales. In other words, the deterministic features such as the valid watershed regions are established from coarse to fine. There presumably exists a critical lower bound of scale, above and below which the watersheds of f (x, σ) are valid and invalid, respectively. In order to observe how the valid regions survive until large scales against the scale-space filtering, we define the life of the watershed region. Definition 3. The life of the watershed region is defined as the annihilation scale σ t of the regional minimum. Let W be a distribution of the lives of the watershed regions of |∇f (x, σ)| for the image of random noise. If W can be parametrically modelled, a goodness-of-fit test can be performed under the null hypothesis H0 . That is, if an image f (x) is an observation of a true uniform image with noise, then the model of W fits the distribution of lives {σ t } of its watershed regions, and H0 for any watershed regions of f (x) is accepted. We investigate experimentally the life distribution W for the gradient watershed regions of a Gaussian white noise image as shown in Fig. 2(a). We averaged the frequencies of lives over one hundred noise images. We discard the lives of pixel points whose annihilations are detected in 0 < σ ≤ Δσ by RegionHierarchy because not all the pixel centres are the local minima. Figure 2(b) is the averaged histogram of life. The obtained life histogram shows an unimodal shape. This implies that there exists a scale where the merging of the regions most frequently occurs. The regional minima of the noise image are uniformly distributed random points, and the regions tend to merge with nearest regions. Therefore, we deduce that this unimodal property is associated with distribution of the nearest neighbour distances of random points. In fact, the nearest neighbour distance distribution has a unimodal shape (See appendix A). The scale of

Validation of Watershed Regions by Scale-Space Statistics

181

σ

Relative frequency (a)

(b)

Fig. 2. Noise image and the averaged life histogram for its gradient watershed regions. (a) The noise image has uncorrelated random pixel values. (b) The life histogram shows relative frequency of scale at which the regional minima of the gradient image are annihilated as Gaussian blurring of the noise image proceeds.

the mode can be used as a gauge of the density of invalid regions. The regional minima with significantly large values of life out of the unimodal distribution W can be identified to be valid, because such regional minima are distinguishable from the invalid regional minima of the noise image. 3.3

Recursive Validation

We can set a critical value of the scale to judge the watershed regions valid or invalid. Although the computation of such a critical scale requires the parametric model of the life distribution in the strict sense of statistics, the critical scale can be roughly evaluated by the peak and decaying form of the life histogram. If the image contains valid regions, the life histogram may be multimodal or may have a peak at a small scale relative to the outlying lives representative of the valid regions. According to our experimental result in Section 3.2, a regional minimum with a life which is more than six times greater than the peak can be considered to be valid with the statistical confidence level α > 99% under the assumption of uncorrelated Gaussian random pixel values of a two-dimensional image as the noise. We present an algorithm RegionDiscovery for discovery of the valid watershed regions. This algorithm recursively validates the regions in a top-down fashion using each tree T in G by RegionHierarchy. According to the hierarchy, any discovered region is split into subregions as long as they are valid. Each subregion is validated using the life histograms constructed from the lives stored in the subtrees of T corresponding to the subregion.

182

T. Sakai and A. Imiya

RegionDiscovery(tree T , set of valid regions V , significance level α) 1 let Σ be a set of life values stored in T except the root; 2 let s be the subroot node of T with the largest life value σmax ∈ Σ; 3 if IsMultimodal(Σ) or IsOutlier(σmax , Σ, α) then 4 RegionDiscovery(Subtree(T , s), V , α); 5 RegionDiscovery(T \Subtree(T , s), V , α); 6 else 7 push the region R := Pixels(T ) into V ; 8 end if. Here, the function IsMultimodal returns true if the histogram of Σ is not unimodal. IsOutlier returns true if the life σ t is greater than the critical αlevel of scale computed from the given set of lives Σ. Note that these functions discard the lives in 0 < σ ≤ Δσ. Subtree extracts the subtree with subroot node s from the tree T . Pixels returns a set of pixels whose labels are recorded at the nodes in the given tree. The following function, Watershed, executes our watershed segmentation algorithm for a given image f with a set of pixels P and a significance level α. It returns the set of valid watershed regions consisting of subsets of P . Watershed(set of pixel centres P , image f , significance level α) 1 set V := ∅; 2 G := RegionHierarchy(P, f ); 3 for each tree T in G do 4 RegionDiscovery(T , V , α); 5 end for 6 return V .

scale 30 25 20 15 10 5

(a)

(b)

(c)

Fig. 3. An example of our watershed segmentation of noisy image. (a) Original image with 20% noise. (b) Trajectories of local minima of the gradient magnitude of (a) in scale space. The trajectories reaching out of the spatial domain are subordinate to a local minimum at infinity. (c) Watershed regions of the gradient magnitude by the algorithm Watershed. The brightness indicates the order of lives.

Validation of Watershed Regions by Scale-Space Statistics

4

183

Test Example

We demonstrate our gradient watershed segmentation Watershed for a noisy greyscale image. The purpose of this section is not to test the performance of the algorithm, but to show that the statistics in scale space has potential to discover the valid watershed regions without any prior information about them. Figure 3(a) shows a 128 × 128 test image f (x) with 20% additive noise [18]. The trajectories of local minima of |f (x, σ)| traced from σ = 0 in scale space are shown in Fig. 3(b). We see a large number of local minima created by the noise at small scales. As the scale increases, the local minima are hierarchically grouped and representative local minima survive at larger scales. Figure 3(c) shows the segmentation result with a confidence level α = 99% for f (x). There are nine

σ=2

σ=6

σ = 12

Fig. 4. Watershed segmentation of Fig. 3(a) at different scales. First row: the scalespace image f (x, σ). Second row: the gradient magnitude |∇f (x, σ)|. Third row: the watersheds of |∇f (x, σ)|. Each column corresponds to the same scale indicated below.

184

T. Sakai and A. Imiya

discovered regions clearly corresponding to the major regions of the original image. The tiny faults in the regions were caused by failure in the minimisation. They were wrongly assigned to the image background, which should be fixed in the future work. For the comparision purpose, we show in Fig. 4 the simple watershed segmentation results at a few levels of scale without using the region hierarchy or statistics in scale space. We see invalid small regions at small scales while the shapes of valid regions at large scales are distorted. It is remarkable that structural and statistical analyses using scale space can reconstruct the precise edges of statistically valid watershed regions despite the significant noise.

5

Concluding Remarks

The scale-space treatment of the image clarifies not only the hierarchical relationships among the watershed regions but also their statistical properties. We can observe in the Gaussian scale space how the random features are suppressed and deterministic features emerge as the scale grows. A valid watershed region must be statistically distinguishable from unreproducible regions caused by the random features. The reproducibility is a desirable ability of image recognition techniques. On the basis of this simple requirement we described the null hypothesis H0 , which is to be rejected if the watershed region is valid. A watershed region is recognised as valid at a statistical confidence level in rejecting H0 . We presented a validation scheme for watershed segmentation using statistics in scale space. We defined the life of a watershed region, whose distribution is useful for testing H0 . We showed that the life distribution for the noise image is unimodal, and the valid regions can be identified by the regional minima with significantly large values of lives out of the unimodal distribution. The statistical properties of the life and the region hierarchy enable the recursive validation of the watershed regions. A distinctive feature of our scheme is that it does not require any definition of similarity or dissimilarity measures between watershed regions, which is used in many methods for preventing oversegmentation. Instead, we focused on the statistical differences between the valid and invalid regions in scale space. In order to take advantage of the potential of scale-space statistics, our scheme requires further investigation, especially in relation to the model of the life distribution, and improvement and acceleration of the algorithms to obtain feasible segmentation results for larger size real images.

References 1. Vincent, L., Soille, P.: Watersheds in digital spaces: An efficient algorithm based on immersion simulations. IEEE Trans. on Pattern Analysis and Machine Intelligence 13(6), 583–598 (1991)

Validation of Watershed Regions by Scale-Space Statistics

185

2. Beucher, S.: Watershed, hierarchical segmentation and waterfall algorithm. In: Proc. Math. Morphology and Its Appl. to Image Processing, pp. 69–76 (1994) 3. Maes, F., Vandermeulen, D., Suetens, P., Marchal, G.: Computer-aided interactive object delineation using an intelligent paintbrush technique. In: Ayache, N. (ed.) CVRMed 1995. LNCS, vol. 905, pp. 77–83. Springer, Heidelberg (1995) 4. Hagyard, D., Razaz, M., Atkin, P.: Analysis of watershed algorithms for greyscale images. In: Proc. of IEEE Intl. Conf. Image Procesing, vol. 3, pp. 41–44 (1996) 5. Olsen, O.F., Nielsen, M.: Generic events for the gradient squared with application to multi-scale segmentation. In: ter Haar Romeny, B.M., Florack, L.M.J., Viergever, M.A. (eds.) Scale-Space 1997. LNCS, vol. 1252, pp. 101–112. Springer, Heidelberg (1997) 6. Gauch, J.M.: Image segmentation and analysis via multiscale gradient watershed hierarchies. IEEE Trans. on Image Processing 8(1), 69–79 (1999) 7. Roerdink, J.B.T.M., Meijster, A.: The watershed transform: definitions, algorithms, and parallelization strategies. Fundamenta Informaticae 41, 187–228 (2001) 8. Vanhamel, I., Pratikakis, I., Sahli, H.: Multiscale gradient watersheds of color Images. IEEE Trans. on Image Processing 12(6), 617–626 (2003) 9. Witkin, A.P.: Scale space filtering. In: Proc. of 8th IJCAI, pp. 1019–1022 (1986) 10. Koenderink, J.J.: The structure of images. Biological Cybernetics 50, 363–370 (1984) 11. Lindeberg, T.: Scale-Space Theory in Computer Vision. Kluwer, Boston (1994) 12. Weickert, J., Ishikawa, S., Imiya, A.: Linear Scale-Space has First been Proposed in Japan. Journal of Mathematical Imaging and Vision 10, 237–252 (1999) 13. Lifshitz, L.M., Pizer, S.M.: A multiresolution hierarchical approach to image segmentation based on intensity extrema. IEEE Trans. on Pattern Analysis and Machine Intelligence 12(6), 529–540 (1990) 14. Florack, L.M.J., Kuijper, A.: The topological structure of scale-space images. Journal of Mathematical Imaging and Vision 12(1), 65–79 (2000) 15. Kuijper, A.: The deep structure of Gaussian scale-space images. PhD thesis, Utrecht University (2002) 16. Sakai, T., Imiya, A.: Gradient structure of image in scale space. Journal of Mathematical Imaging and Vision 28(3), 243–257 (2007) 17. Keys, R.: Cubic convolution interpolation for digital image processing. IEEE Trans. on Acoustics, Speech, and Signal Processing 29(6), 1153–1160 (1981) 18. SAMPL database, http://sampl.ece.ohio-state.edu/database.htm 19. Suwa, N.: Quantitative morphology: stereology for biologists. Iwanami Shoten (1977) (in Japanese)

A

Distribution of Nearest Neighbour Distances

We present a proof that the nearest neighbour distances obey the Weibull distribution if the points in Rd are uniformly distributed in a Poisson arrangement [19]. The Poisson arrangement is defined as the uniformly random distribution of points with constant density ρ such that the number of points x in a fixed volume V follows the Poission distribution. Po(x; λ) =

λx exp(−λ) x!

(3)

186

T. Sakai and A. Imiya

Here, λ = ρV is the expected number of points in the volume V . Let r be the distance from an arbitrary point. The distribution of the nearest neighbour distances, p(r), can be regarded as the probability that the nearest neighbour is found in an infinitesimal gap between r and r + δr. This is the case that no points are found within the distance r, and at least one point is found between r and r + δr. Since the volume Vd of a unit d-ball and its surface area Sd−1 has a relationship Vd d = Sd−1 , we have   p(r)δr = Po(0; ρVd rd ) 1 − Po(0; ρSd−1 rd−1 δr)   ≈ exp(−ρVd rd ) 1 − exp(ρSd−1 rd−1 δr) = exp(−ρVd rd ) · ρSd−1 rd−1 δr = exp(−ρVd rd ) · ρVd drd−1 δr √ Letting s = 1/d ρVd be the scale of the average volume of d-dimensional hypercube per point, we obtain the Weibull distribution  

d  r d−1 r d p(r; s, d) = (4) exp − s s s where s and d correspond to the so-called scale and shape parameters of the Weibull distribution, respectively. This distribution p(r; s, d) has a mode at r =  s d (d − 1)/d. For a fixed dimensionality, the mode depends only on the scale parameter s, which enables us to calculate the point density ρ from the mode.

Adaptation of Eikonal Equation over Weighted Graph Vinh-Thong Ta, Abderrahim Elmoataz, and Olivier Lézoray Université de Caen Basse-Normandie, GREYC CNRS UMR 6072, Image Team {vinhthong.ta,abderrahim.elmoataz-billah,olivier.lezoray}@unicaen.fr http://www.info.unicaen.fr/˜vta

Abstract. In this paper, an adaptation of the eikonal equation is proposed by considering the latter on weighted graphs of arbitrary structure. This novel approach is based on a family of discrete morphological local and nonlocal gradients expressed by partial difference equations (PdEs). Our formulation of the eikonal equation on weighted graphs generalizes local and nonlocal configurations in the context of image processing and extends this equation for the processing of any unorganized high dimensional discrete data that can be represented by a graph. Our approach leads to a unified formulation for image segmentation and high dimensional irregular data processing.

1

Introduction

Solutions of the nonlinear eikonal equation have found numerous applications. One can quote for instance, geometric optics, image analysis or computer vision including shape from shading [1, 2], median axis or skeleton extraction [3], topographic segmentation (watershed) [4] or geodesic distance computation on discrete and parametric surfaces [5, 6, 7, 8, 9]. The latter works consider both structured and unstructured meshes on cartesian or non-cartesian domains. The eikonal equation is a special case of the following general continuous Hamilton-Jabobi equation:  H(x, f, ∇f ) = 0 x∈Ω ⊂ IRn , (1) f (x) = φ(x) x∈Γ ⊂ Ω where φ in the boundary condition is a positive speed function defined on Ω and f (x) is the traveling time or distance from source Γ . Then, the eikonal equation can be expressed by using the following Hamiltonian: H(x, f, ∇f ) = ∇f (x) − P (x),

(2)

where P (x) is a given potential function. Solution of (1) represents the shortest distance from x to the zero distance curve given by Γ (where φ(x)=0). Solutions of (2) are usually based on a discretization of the Hamiltonian where the approximation of the derivatives is performed by the Godunov [10] or the X.-C. Tai et al. (Eds.): SSVM 2009, LNCS 5567, pp. 187–199, 2009. c Springer-Verlag Berlin Heidelberg 2009 

188

V.-T. Ta, A. Elmoataz, and O. Lézoray

Lax-Friedrich [11] schemes. Then, many numerical methods have been proposed and investigated to solve the nonlinear system described by (2). For instance, one can quote the following schemes. (i) An iterative scheme [1] relying on fixed point methods that solves a quadratic equation was proposed. (ii) The fast sweeping methods [12] that use Gauss-Seidel type of iterations to update the distance function field. The key point of fast sweeping is to update the points in a certain order. (iii) Tsitsiklis [13] was the first to develop a Dijkstra like method and proposed an optimal algorithm for solving the eikonal equation. Based on this idea, [14, 11] produced the fast marching methods. Another approach to solve (2) is to consider a time dependent version of the equation and to evolve it to the steady state. Then, (2) can be rewritten as ⎧ n ⎪ ⎨∂f (x, t)/∂t = −∇f (x) + P (x) x∈Ω ⊂ IR (3) f (x, t) = φ(x) x∈Γ ⊂ IRn . ⎪ ⎩ f (x, 0) = φ0 (x) x∈Ω This paper only considers the discrete analogue of the time dependent formulation of the eikonal equation but, in future works, the stationary case (time independent) will be also considered. Contributions. In this work, we propose an adaptation of (3) over weighted graphs of the arbitrary structure. The goal here is to provide a simple and common formulation that solves the eikonal equation for any discrete data that can be represented by a weighted graph such as images or high dimensional data defined on irregular domains. This alternative formulation for solving the eikonal equation is based on partial difference equations (PdEs) and discrete gradients over weighted graphs. Our formulation has several advantages. Any discrete domain that can be described by a graph can be considered without any spatial discretization. In the context of image processing, local and nonlocal configurations are directly enabled within a same formulation. Finally, the aim of this paper is not to solve a particular application with the eikonal equation but to show the potentialities of our proposition to address image segmentation, data clustering or distance computation. Paper Organization. The paper is organized as follows. Section 2 recalls basics, definitions and operators on weighted graphs. Section 3 introduces our formulation for solving the eikonal equation. Section 4 shows the potentialities of our proposition for the segmentation of images and unorganized data processing. Finally, last Section concludes.

2

Discrete Derivatives on Weighted Graphs

This Section recalls basics, definitions, operators and processes on weighted graphs.

Adaptation of Eikonal Equation over Weighted Graph

2.1

189

Definitions and Weighted Graphs Construction

Notations and Definitions. We consider the general situation where any discrete domain can be viewed as a weighted graph. Let G=(V, E, w) be a weighted graph composed of two finites sets: vertices V and weighted edges E⊆V ×V . An edge (u, v)∈E connects two adjacent (neighbor) vertices u and v . The neighborhood of a vertex u is noted N (u)={v∈V \{u} : (u, v)∈E}. The weight ωuv of an edge (u, v) can be defined with a function w:V ×V →IR+ such that w(u, v)=ωuv if (u, v)∈E and w(u, v)=0 otherwise. Graphs are assumed to be simple, connected and undirected implying that function w is symmetric. Let f :V →IR be a discrete real-valued function that assigns a real value f (u) to each vertex u∈V . We denote by H(V ) the Hilbert space of such functions defined on V . Weighted Graphs Construction. Any discrete domain can be represented by a weighted graph where functions of H(V ) represents the data to process. In the general case, an unorganized set of points V ⊂IRn can be seen as a function f 0 :V ⊂IRn →IRn . Then, constructing a graph from this data consists in defining the set of edges E by modeling the neighborhood. It is based on a similarity relationship between data with a pairwise distance measure μ:V ×V →IR+ . There exists several methods to transform a set of vertices V into a neighborhood (similarity) graph (see [15] for a survey on proximity and neighborhood graphs). In this paper, we focus on two particular graphs: the τ -neighborhood graphs and a modified version of k-nearest neighbors graphs. The k nearest neighbors graph, noted k-NNG is a weighted graph where each vertex u∈V is connected to its k nearest neighbors which have the smallest distance measure towards u according to function μ. Since this graph is directed, a modified version of this graph is used to make it undirected. The τ -neighborhood graph, noted Gτ is a weighted graph where the τ -neighborhood Nτ for a given vertex u∈V is defined as Nτ (u)={v∈V \{u} : μ(u, v)≤τ } with τ >0 a threshold parameter. 2D images can be viewed as functions f 0 :V ⊂ZZ 2 →IRn . In this case, the associated distance μ for construct the neighborhood graph is usually the city block or the Chebychev distances computed with the spatial coordinates of each vertex representing an image pixel. With these distances and the τ -neighborhood graphs, one recovers the two usual graphs used in image processing, the 4-adjacency grid graph (denoted G0 with the city block distance) and the 8-adjacency grid graph (denoted G1 with the Chebychev distance) with τ ≤1. Another useful graph structure in image processing is the region adjacency graph (RAG) where vertices correspond to image regions, and the set of edges is obtained by considering an adjacency distance. With the τ -neighborhood (τ =1), the RAG is the Delaunay graph of an image partition. Weights Computation. Similarities between data can be incorporated within edges’ weights according to a measure of similarity g:E→IR+ that satsfies w(u, v)=g(u, v) for (u, v)∈E. Then, the distance computation between data is performed by comparing their features that generally depend on a given initial function f 0 ∈H(V ). To this aim, each vertex u∈V is assigned with a feature vector F (f 0 , u)∈IRm . With F , the following weight functions can be considered. For

190

V.-T. Ta, A. Elmoataz, and O. Lézoray

a given edge (u, v)∈E and a distance measure ρ:V ×V →IR+ associated to F , we can have g0 (u, v) = 1 (constant weight case) , g1 (u, v) = (ρ(F (f 0 , u), F (f 0 , v)) + )−1 with >0, →0, g2 (u, v) = exp(−ρ(F (f 0 , u), F (f 0 , v))2 /σ 2 ) with σ>0, where σ controls the similarity and ρ is usually the euclidean distance. Several choices for the expression of F can be considered depending on the features to preserve. The simplest one is F (f 0 , .)=f 0 . In the context of image processing, an important feature vector F is provided by image patches, i.e., F (f 0 , u)=Fτ (f 0 , u)={f 0 (v) : v∈Nτ (u) ∪ {u}}. In the case of a grayscale image Fτ (f 0 , .) is a vector of size (2τ +1)2 corresponding to the values of f 0 in a square window of size (2τ +1)×(2τ +1) centered at vertex u (a pixel). Color images can be handled using features of dimension 3×(2τ +1)2 . Then, the resultant weight function directly incorporates local or nonlocal features [16]. This feature vector has been proposed in the context of texture synthesis [17], and further used in the context of image processing [18,19]. 2.2

Graph Based Discrete Gradients

Let G=(V, E, w) be a weighted graph. The discrete weighted gradient of a function f ∈H(V ) at a vertex u∈V is defined by 

(∇w f )(u) = (∂v f (u))(u,v)∈E

where ∂v f (u)= w(u, v)(f (v)−f (u)) corresponds to the discrete (partial) derivative of f with respect to the edge (u, v). These definitions have been used by [20] for image and mesh regularization. Based on the latter works, two discrete formulations of weighted morphological gradients on graphs have been proposed − by [21]: namely, the weighted external ∇+ w and the internal ∇w gradient operators. For u∈V + (∇+ w f )(u) = (∂v f (u))(u,v)∈E

and

− (∇− w f )(u) = (∂v f (u))(u,v)∈E ,

(4)

where the external ∂v+ f (u) and the internal ∂v− f (u) discrete partial derivatives are ∂v+ f (u) = max(0, ∂v f (u)) and ∂v− f (u) = − min(0, ∂v f (u)), with ∂v− f (u)=∂u+ f (v). When the weight is constant (w=g0 ) these definitions recover the classical directional derivative operators. The Lp -norm (with 0
1/p  w(u, v)p/2 |(f (v)−f (u))± |p and (5) (∇± w f )(u)p = v∼u

 1/2 |(f (v)−f (u))± | . (∇± w f )(u)∞ = max w(u, v) v∼u

(6)

Adaptation of Eikonal Equation over Weighted Graph

191

Notation v∼u means that vertex v is adjacent to u. ∇± w refers to both external and internal gradient (with respect to the sign) and (a)+ = max(0, a) and (a)− = min(0, a). These gradients have the following property: p − p (∇w f )(u)pp = (∇+ w f )(u)p + (∇w f )(u)p

with 0
3

Eikonal Equation on Weighted Graphs

In this Section, we present our formulation to approximate the eikonal equation (3) over weighted graphs by considering PdEs and the morphological gradients presented in the previous Section. With morphological processes described by (8), the time dependent eikonal formulation (3) can be viewed as an erosion process regarding the minus sign and a null potential function P . With the corresponding internal gradient (∇− w) involved in discrete PdEs based erosion process, (3) can be directly rewritten with weighted graphs. Given a graph G=(V, E, w) and a function f ∈H(V ), we obtain a discrete PdEs based version of the system (3) ⎧ − ⎪ ⎨∂f (u, t)/∂t = −∇w f (u)p + P (u) u∈V f (u, t)=φ(u) u∈V0 ⊂ V , ⎪ ⎩ f (u, 0)=φ0 (u) u∈V where V0 corresponds to the initial seed vertices. With f n (u) ≈ f (u, nΔt), this iterative numerical scheme is obtained for all u∈V :  n f n+1 (u)=f n (u) − Δt (∇− (9) w f )(u)p − P (u) .

192

V.-T. Ta, A. Elmoataz, and O. Lézoray

The steady state (i.e. given a fixed number n of iteration or when f n+1 −f n  < ) of this process is the solution of the eikonal equation (2). Injecting the corresponding internal gradient norm in (9), we obtain for the Lp -norm (5) and the L∞ -norm (6)

1/p  f n+1 (u)=f n (u) − Δt w(u, v)p/2 | min(0, f (v)−f (u))|p − P (u) , (10) v∼u

f

n+1

  (u)=f (u) − Δt max w(u, v)1/2 | min(0, f (v)−f (u))| − P (u) . n

v∼u

(11)

The proposed methodology leads to a simple and common formulation that constitutes an adaptative framework for the eikonal equation. Indeed, our approach only depends on the p value and the weight function w. In Sect. 4, experiments show how the framework can be adapted to address image segmentation or data clustering. Relations with other schemes. Scheme (9) has the advantage to work on any graph structures. Then, with an adapted graph topology and an appropriated weight function, the proposed formulation is linked to well-known schemes such as Osher-Sethian Hamiltonian discretization scheme or the graph based Dikjstra algorithm. Osher-Sethian scheme. Let G0 =(V, E, g0 ) be an unweighted 4-adjacency grid graph associated with an image. Then, (10) recovers the exact Osher-Sethian upwind first order Hamiltonian discretization scheme [14] when p=2 and using G0 :

1/2  f n+1 (u)=f n (u) − Δt | min(0, f (v)−f (u))|2 − P (u) . v∼u

Replacing vertices u∈V and their neighborhood by their spatial coordinates (x, y), the latter expression can be rewritten as   2 f n+1 ((x, y))=f n ((x, y))−Δt | min 0, f n ((x, y))−f n ((x−1, y))  2 +| max 0, f n ((x+1, y))−f n ((x, y))  2 +| min 0, f n ((x, y))−f n ((x, y−1))

 2 1/2 +| max 0, f n ((x, y+1))−f n ((x, y)) − P ((x, y)) , since min(0, a−b)2 = max(0, b−a)2 . This equation corresponds to the discretization scheme of the Hamilton-Jacobi equations proposed by [14]. Dikjstra scheme. Let G=(V, E, g0 ) be an unweighted graph. Then, (11) corresponds to an iterative version of the Dikjstra shortest path algorithm defined on graphs of arbitrary structure. Indeed, in the case where p=∞, Δt=1 and with G, (11) becomes, for all u∈V  f n+1 (u) = f n (u) − max | min(0, f (v)−f (u))| + P (u) = min(f n (v)) + P (u), v∼u

v∼u

Adaptation of Eikonal Equation over Weighted Graph

193

by considering the neighborhood of u as the set N (u)∪{u} and with the properties that max(0, a−b)= − min(0, b−a) and min(0, a−b)= min(a, b)−b. This equation corresponds to a shortest path algorithm for a given graph where at each step, the distance f (u) at vertex u corresponds to the minimal distance in its neighborhood.

4

Experiments

The proposed formulation of the eikonal equation and can be used to process any function defined on vertices of a graph or on any arbitrary discrete domain. This Section illustrates the potentialities of our formulation through examples of weighted distance computation, image segmentation and unorganized high dimensional data processing. Different graph structures and weight functions are also used to show the flexibility of our approach. In the sequel, all experiments are obtained with a constant potential function P =1. Clearly, a different potential function can be adapted for a particular application. The objective of the following experiments is not to solve a particular application. They only illustrate the potential and the behavior of our eikonal equation formulation. Adaptative Front Propagation and Weighted Distances. Figure 1 shows the adaptivity of our formulation in order to compute weighted distances. Indeed, this example shows results for different p values, graph topologies, weight functions and features F . The initial seed is located at the top left corner of the original grayscale image f 0 :H(V )→IR. First, second and third rows of Fig. 1 show results for p=2, 1 and ∞ respectively, where (10) and (11) are used. All the results correspond to color distance maps (red for small and blue for large distances) where iso-levels sets are superimposed in white. First and second columns of Fig. 1 show results obtained with unweighted (w=g0 ) graphs. First column uses a 4-adjacency grid graph (G0 ) and corresponds to the classical case. Second column uses a 25-adjacency grid graph (G2 ) and shows the effect of a larger neighborhood. Third and fourth columns show results obtained with weighted graphs. Third column considers graph G0 weighted by function g2 with F =f 0 . By using non constant weights, image information is automatically integrated in the distance computation that modifies the front evolution speed particularly into the textured sub-image. Fourth column shows the nonlocal case where graph G2 is constructed and weighted with function g2 associated with patches of size 11×11. In that case, repetitive information are clearly captured by the weights that stops the front propagation around the textured sub-image. Finally, segmentation of the textured sub-image can be simply obtained by thresholding the computed distances. Image Segmentation with Region Based Graphs. The goal of the following two examples is not to show a perfect segmentation but to show how we can take advantage of graph topologies in image segmentation. The basic idea is to consider that image pixels are not the only relevant components in image

194

V.-T. Ta, A. Elmoataz, and O. Lézoray

Original f 0 :H(v)→IR

G0 , w=g0

G2 , w=g0

G0 , w=g2 F =f 0

G2 , w=g2 F =F5 (f 0 , .)

p=1

p=2

p=∞

Fig. 1. Front propagation and weighted distances with different p values, graph configurations G, weights w and features F . Figures represent color distance maps with iso-level sets obtained by thresholding the distances. The seed is located at the top left corner (see text for more details).

and more abstract elements such as image regions can be used. Hence, we suggest to work directly with a reduced version of images: image partitions. Image partitions can be obtained by image pre-processing methods such as watershed. Figures 2(b) and 3(b) show such partitions computed from Figs. 2(a) and 3(a). Figures 2(c) and 3(c) are reconstructed images from partitions with the mean color value for each region. Figure 2 presents an example of image segmentation based on RAG and also shows that this graph structure can accelerate segmentation processes. This example compares segmentation obtained by a 4-adjacency grid graph G0 weighted by function g2 with pixel grayscale values (Fig. 2(d)) and segmentation result with a RAG constructed from partition (Fig. 2(b)) and weighted by function g2 with mean values (Fig. 2(e)). Color distance maps are obtained with the initial seeds (white points) in Fig. 2(a). Segmentations are performed by thresholding the obtained distances. Results show similar behaviors both on distance maps and segmentations while drastically speeding-up the segmentation process in the RAG case. Indeed, the number of vertices in the RAG represents approximatively 3% as compared to the number of vertices in the pixel based graph. The direct consequence is a decreasing of the computational complexity thanks to the reduced amount of data to consider. On a standard computer the computing time can be decreased by a 10 factor. Figure 3 shows another benefit of using a RAG structure: nonlocal (non spatially connected) object segmentation. This experiment compares segmentation results with RAG (Fig. 3(d)) and nonlocal RAG (Fig. 3(e)). Both graphs are

Adaptation of Eikonal Equation over Weighted Graph

(a)

Original and seeds

(white)

(d)

(b)

Partition (97% of

reduction)

Grid graph G0 , w=g2

(c)

195

Reconstructed im-

age

(e)

RAG, w=g2

Fig. 2. Acceleration of image segmentation process. (a) original image (150×235) with 35 250 pixels. (b) partition with 999 regions (97% of reduction in terms of image components). (c) reconstructed image with mean color value. (d) and (e): at left, distance color maps (red for small and blue for large distances) and at right, final segmentations. Images (d) are obtained with a pixel based graph computed from (a). Images (e) are obtained with a RAG constructed with (b) and (c) (see text for more details).

computed from partition 3(b) weighted by function g2 with mean color values. In nonlocal RAG case, each vertex neighborhood is extended by a 5 nearest neighborhood based on mean value feature. The obtained graph is a RAG∪5NNG graph. Figures 3(d) and 3(e) show color distance maps computed from initial seeds (white stroke) in Fig. 3(a) and final segmentations. For local case (Fig. 3(d)), object marked by seeds is well segmented with respect to close distances (red color). The other objects are far (blue color) and the final segmentation only extracts the marked one. For nonlocal case (Fig. 3(e)), the distance within the marked object is close to the initial seeds. In addition the distances to other triangles in the scene are also computed as close to seeds (red color). The consequence is that all the objects in the image are extracted by thresholding even if they are not spatially close with a minimal number of initial seeds. Unorganized High Dimensional Data Processing. The following experiments show applications of our formulation of the eikonal equation for the processing of high dimensional data in irregular domains. Figure 4 shows applications of the eikonal equation for data clustering and shortest path problems. The initial data set (Fig. 4(a)) is constituted of 133 images of head pose. Each image is of size 29×29. From this data set, two possible applications can be performed: clustering and head pose transition estimation. The goal here is not to solve machine leaning problems, but to show that these problems can be addressed by our formulation of eikonal equation. In order to process such data, a graph (|V |=133) is constructed where each vertex represents an image and is described by a feature of size 29×29 (i.e IR841 ). In the following results, initial seeds (images) are represented with white boundaries. Points that are close and

196

V.-T. Ta, A. Elmoataz, and O. Lézoray

(a)

Original and seeds

(white)

(d)

(b)

Partition (98% of re-

(c)

Reconstructed image

duction)

RAG, w=g2

(e)

RAG∪5-KNNG,w=g2

Fig. 3. Nonlocal region based image segmentation. (a) original image (256×256) with 65 536 pixels. (b) partition with 1 324 regions (98% of reduction as compared to original one). (d) and (e) at left, distance color maps (red for small and blue for large distances) and at right, final segmentations. Graphs used in (d) and (e) are computed from (b) and (c) (see text for more details).

far to seeds are respectively represented with blue and red colors in distance maps (Fig. 4(b) and 4(c)). Figures 4(b) and 4(d) show the application of the eikonal equation for data clustering. Such an application can be used for data set exploration or semisupervised learning: given an input seed (query) one wants to obtained the closest points with respect to the initial input. Figure 4(b) shows the distance map obtained from a single initial seed. Figure 4(d) shows clustering results. Initial input has a white boundary. The 10 closest images are located at the top and the 10 farthest are located at the bottom of Fig. 4(d). Figures 4(c) and 4(e) shows another example of application of the eikonal equation for data set. Given two initial images, one wants to recover a transition sequence of images that separates them. This problem can be viewed as a shortest path problem solved by the eikonal equation. Figure 4(c) shows the distance map obtained from the initial seeds. Figure 4(e) shows the obtained path from seed at top left to seed at bottom right. These experiments show satisfying results and the ability of our approach to address machine learning problems even if a simple euclidean distance is used to compare data points. Clearly, results can be improved by using well adapted distances or features estimation.

Adaptation of Eikonal Equation over Weighted Graph

(a)

Original data, 133 images of size 29×29 (|V |=133, f 0 : V →

841

IR

(b)

)

Color distance map+seed (white boundaries)

(d)

197

Local clustering with (b)

(c)

Color distance map+seeds (white boundaries)

(e)

Shortest path with (c)

Fig. 4. High dimensional data clustering and shortest path. (b) and (c) color distance maps (blue for small and red for large distance) images superimposed with white boundaries are initial seeds. (d) clustering results where at top the 10 closest and at bottom the 10 farthest with respect to the seed (white boundary). (e) shortest path from the two initial seeds (white boundary).

5

Conclusion

In this paper, a discrete version of the eikonal equation over weighted graphs of arbitrary structure is proposed. Solution of the eikonal equation based on PdEs, discrete gradients and weighted graphs is presented. The proposed formulation constitutes a simple, common and adaptative framework that recovers well-known definitions and unifies local and nonlocal configurations in the context of image processing. This framework can consider any discrete data that can be represented by weighted graphs. Through experiments, we have shown the potentiality and the flexibility of our approach to address image segmentation

198

V.-T. Ta, A. Elmoataz, and O. Lézoray

and unorganized high dimensional data processing. Finally, an ongoing work is to address the stationary (time independent) version of the eikonal equation and to solve this equation by considering fast marching like methods on arbitrary graphs within our framework.

References 1. Rouy, E., Tourin, A.: A viscosity solutions approach to shape-from-shading. SIAM J. Num. Anal. 29, 867–884 (1992) 2. Sethian, J.: A fast marching level set methods for monotonically advancing fronts. Proc. Nat. Acad. Sci. 41(2), 199–235 (1999) 3. Siddiqi, K., Bouix, S., Tannenbaum, A., Zucker, S.W.: The hamilton-jacobi skeleton. In: Proc. ICCV, pp. 828–834 (1999) 4. Maragos, P., Butt, M.: Curve evolution, differential morphology and distance transforms as applied to multiscale and eikonal problems. Fundamentae Informatica 41, 91–129 (2000) 5. Bronstein, A.M., Bronstein, M.M., Kimmel, R.: Weighted distance maps computation on parametric three-dimensional manifolds. J. Comput. Phys. 225(1), 771–784 (2007) 6. Sethian, J.A., Vladimirsky, A.: Ordered upwind methods for static hamiton-jacabi equations: Theory and algorithms. SIAM J. Num. Anal. 41(1), 325–363 (2003) 7. Abgrall, R.: Numerical discretization of the first-order hamilton-jacobi equations on triangular meshes. Comm. Pure and Applied Math. 49, 1339–1373 (1996) 8. Shu, C.-W., Zhang, Y.-T.: High order WENO schemes for hamilton-jacobi equations on triangular meshes. SIAM J. Scien. Comp. 24, 1005–1030 (2003) 9. Mémoli, F., Sapiro, G.: Fast computation of weighted distance functions and geodesics on implicit hyper-surfaces. J. Comput. Phys. 173, 730–764 (2001) 10. Leveque, R.: Finite Volume Methods for Hyperbolic Problems. Cambridge University Press, Cambridge (2002) 11. Sethian, J.A.: Level Set Methods and Fast Marching Methods. Evolving Interfaces in Computational Geometry. In: Fluid Mechanics, Computer Vision, and Materials Science, 2nd edn. Cambridge University Press, Cambridge (1999) 12. Zhao, H.K.: Fast sweeping method for eikonal equations. Math. Comp. 74, 603–627 (2005) 13. Tsitsiklis, J.: Efficient algorithms for globally optimal trajectories. IEEE Trans. Autom. Control 40(9), 1528–1538 14. Osher, S., Sethian, J.: Fronts propagating with curvature-dependent speed: Algorithms based on Hamilton-Jacobi formulations. J. Comput. Phys. 79, 12–49 (1988) 15. Jaromczyk, J., Toussaint, G.: Proc. IEEE. Relative Neighborhood Graphs and Their Relatives 80(9), 1502–1517 (1992) 16. Elmoataz, A., Lézoray, O., Bougleux, S., Ta, V.T.: Unifying local and nonlocal processing with partial difference operators on weighted graphs. In: Proc. LNLA, pp. 11–26 (2008) 17. Efros, A., Leung, T.: Texture synthesis by non-parametric sampling. In: Proc. ICCV, pp. 1033–1038 (1999) 18. Buades, A., Coll, B., Morel, J.: Nonlocal image and movie denoising. IJCV 76(2), 123–139 (2008) 19. Gilboa, G., Osher, S.: Nonlocal operators with applications to image processing. Report 07-23, UCLA (2007)

Adaptation of Eikonal Equation over Weighted Graph

199

20. Bougleux, S., Elmoataz, A., Melkemi, M.: Discrete regularization on weighted graphs for image and mesh filtering. In: Sgallari, F., Murli, A., Paragios, N. (eds.) SSVM 2007. LNCS, vol. 4485, pp. 128–139. Springer, Heidelberg (2007) 21. Ta, V.T., Elmoataz, A., Lézoray, O.: Partial difference equations over graphs: Morphological processing of arbitrary discrete data. In: Proc. ECCV, pp. 668–680 (2008) 22. Brockett, R., Maragos, P.: Evolution equations for continuous-scale morphological filtering. IEEE Trans. Signal Process. 42(12), 3377–3386 (1994)

A Variational Model for Interactive Shape Prior Segmentation and Real-Time Tracking Manuel Werlberger, Thomas Pock, Markus Unger, and Horst Bischof Institute for Computer Graphics and Vision, Graz University of Technology {werlberger,pock,unger,bischof}@icg.tugraz.at http://www.gpu4vision.org

Abstract. In this paper, we introduce a semi-automated segmentation method based on minimizing the Geodesic Active Contour energy incorporating a shape prior. We increase the robustness of the segmentation result using the additional shape information that represents the desired structure. Furthermore the user has the possibility to take corrective actions during the segmentation and adapt the shape prior position. Interaction is often desirable when processing difficult data like in medical applications. To facilitate the user interaction we add a shape deformation which allows to change the shape position manually by the user and automatically in terms of underlying image features. Using a variational formulation, the optimization can be done in a globally optimal manner for a fixed shape representation. To obtain real-time behavior, which is especially important for an interactive tool, the whole method is implemented on the GPU. Experiments are done on medical, as well as on video data and camera streams that are processed in real-time. In terms of medical data we compare our method with a segmentation done by an expert. The GPU based binaries will be available online on our homepage.

1

Introduction

Image segmentation is a very common problem in computer vision. Many segmentation methods use low-level features to obtain a division into fore- and background. Due to the need of robustness it has become a common practice to incorporate high level knowledge to gain reasonable results. The method presented in this paper enhances the robustness of segmentation by imposing shape information of the desired object which allows a precise result with difficult image data (Fig. 1). Pioneering contributions have been made by Cremers et al. [1] with their variational approach of ‘Diffusion Snakes’, the level set formulation by Leventon et al. [2, 3] as well as the region based approach by Paragios and Rousson [4, 5]. The efficient registration of the shape prior to the desired image structure is a challenging problem. Therefore we developed a semi-automated segmentation tool that allows to adjust the object position by hand and by a local optimization routine which is modelled as shape transformation in either case. For the X.-C. Tai et al. (Eds.): SSVM 2009, LNCS 5567, pp. 200–211, 2009. c Springer-Verlag Berlin Heidelberg 2009 

A Variational Model for Interactive Shape Prior Segmentation

201

Fig. 1. Segmentation of the metacarpal bone of a ring finger. The left image shows a simple intensity thresholding which clearly fails. The pure GAC segmentation in the middle image does not result into a valid segmentation either. The shape prior segmentation in the right image can also deal with the low-contrast regions and provides an accurate result.

realization of the segmentation method we use a variational formulation of the Geodesic Active Contour (GAC) energy. The minimization is done with a fast primal-dual approach that is implemented using NVIDIA graphics hardware to become real-time capable. Therefore the method can even be used for tracking objects in videos or live camera streams. Approaches based on the calculus of variation have had great success and recently it has been shown that variational methods show good parallelization capabilities that benefit from a GPU implementation [6, 7]. The main contribution of our work is the incorporation of a shape representation into a variational segmentation framework. We show that the resultant segmentation is globally optimal for a fixed shape prior and provide a GPU implementation for a fast primal-dual optimization procedure defining a definite convergence criterion. Therefore we can add the possibility to interact with the shape position and get a segmentation result in real-time. The framework permits a local optimization of the shape position to get a correct segmentation of objects with a preceding misalignment of the prior. The remainder of the paper is organized as follows: First we give an overview of related work. Section 3 discusses the method on combining a shape prior with a variational formulation of the GAC segmentation model which leads to a segmentation model utilizing a Mumford-Shah (MS) like data term as shape force. In Section 3.1 we propose a fast numerical algorithm to compute the solution of the segmentation model. In 4 we present experiments and a qualitative assessment on reference data. Finally, Section 5 gives a short conclusion.

2 2.1

Related Work Mumford-Shah Segmentation

In [8], Mumford and Shah (MS) proposed a segmentation model of the form    2 2 (u − f ) dx + α |∇u| dx + β length (Γ ) (1) min u,Γ

Ω

Ω\Γ

202

M. Werlberger et al.

where f denotes the observed image, u its piecewise smooth approximation and Γ represents the edges in u. Equation (1) is based on a piecewise smooth approximation of the intensity function and was used in the computer vision community for various tasks like denoising, inpainting, stereo matching, segmentation and many more. A special case of the MS model to segment an image into fore- and background was proposed with the so-called piecewise constant MS segmentation model (2) that was later used by Chan and Vese [9] in combination with a level-set optimization.     2 2 min Per (Σ) + λ (f − c1 ) dx + λ (f − c2 ) dx (2) Σ,c1 ,c2

Σ

Ω\Σ

In (2) f denotes the input image and c1 , c2 the mean values of the fore- and background intensities separated by the region Σ. This realization of the MS functional represents the Potts model [10] for two distinct classes. In addition Chan et al. provided a convex formulation in [11] in form of a Total Variation (TV) functional for a binary segmentation u = 1Σ :    |∇u| dx + λ us (x) dx (3) min u∈{0,1}

Ω

Ω

with

2



2

s (x) = (c1 − f (x) ) + (1 − u) (c2 − f (x) )



(4)

and with Ω |∇u| dx being the TV-norm in a distributional sense Ω |D u|. Due to the fact that we are working on image data which can be interpreted as sufficient smoothfunctions, the TV-norm is valid for any input u and we stick to the notation Ω |∇u| dx in this paper.  For this formulation the TV-norm denotes the length of the segmentation: Ω |∇u| dx = Per (Σ). Moreover Chan et al. [11] showed that a global minimizer can be found for (2) with a restricted minimization of the relaxed problem (3) so that 0 ≤ u ≤ 1. The minimization set is then given by   Σ = x ∈ Ω : u > μ , for every μ ∈ (0, 1) . (5) 2.2

Geodesic Active Contours

Based on the Snake model of Kass et al. [12], Caselles et al. [13] and Kichenassamy et al. [14, 15] proposed an energy that is invariant with respect to new parametrizations of the contour. The Geodesic Active Contour (GAC) (in 3D the model is called minimal surface) is defined as the variational problem   |C|  min g (|∇I (C(s))|) ds , (6) C

0

where |C| describes the Euclidean length of the curve C and the function g models an edge detector. The edge strength has to be restricted to an interval g ∈ (0, 1]. One common choice for computing g is κ

g(|∇I|) = e−η|∇I| ,

for some reasonable parameters κ and η.

(7)

A Variational Model for Interactive Shape Prior Segmentation

203

The general intention of the Snake model is to locate the curve at points with a high edge strength and keep a certain smoothness in the curve. The main advance of GACs is the profound mathematical framework that makes the model very versatile for different applications. The main drawbacks of the model are the non-convexity of the GAC energy and that the empty set is always a global minimizer of (6). In [16, 17, 18], several authors proposed the so-called weighted Total Variation (8) that can be used to give an alternative formulation of the GAC energy. They showed that if u = 1ΩC is a binary function with C the boundary of ΩC , the energy (8) equals the GAC energy (6).  T Vg (u) = g |∇u| dx (8) Ω

Note that the weighted TV-norm is similar to the regularization term of (3). The additional weighting function g is a pointwise constant multiplier and therefore the method of Chan et al. [11, 19] mentioned in the previous Section 2.1 is still valid. Based on these assumptions they showed that by replacing 1ΩC with u ∈ [0, 1], (8) becomes convex, allowing to compute a global minimizer. However, there remains the problem that the empty set depicts a global optimal solution. In [6, 7], Unger et al. proposed a variational formulation incorporating user constraints that avoid this drawback of the classical GAC model.

3

Shape Prior Segmentation

Our main contribution is to combine GAC segmentation and a MS-like data term to incorporate shape information. Therefore we model the GAC energy with the weighted Total Variation (8) and utilize the need of additional constraints by imposing the shape prior. Starting with a formulation of the Mumford-Shah like energy utilized by Chan and Vese in [9] we obtain a variational optimization problem like in (3). The multiplicative part in the data-term will be used as shape information s (x) to model the shape prior segmentation energy. In addition we use the weighted Total Variation (8) to model the GAC energy and add a parameter λ to balance between regularization and shape force:    min g |∇u| dx + λ s (x) u dx (9) 0≤u≤1

Ω

Ω

For a low λ the result of the GAC will be preferred, whereas for increasing λ the shape prior will be taken more into account. In Fig. 2 the effects of different parameter settings are shown. Since (9) is homogeneous of degree one, the thresholding theorem of [11] still applies in our case, allowing us to compute the global minimizer of (9) as mentioned in Section 2.1 according the MS segmentation model (3). The optimization method will be discussed in more detail in the following Section 3.1.

204

M. Werlberger et al.

Fig. 2. Evaluation of different settings of λ: The images show that with increasing λ = {0.01, 0.02, 0.1} (left to right) the segmentation is more attracted to the fixed shape. For a low λ the influence of the pure GAC energy increases and the segmentation is more attracted to significant edges.

The shape prior itself influences the segmentation by setting pixelwise foreand background constraints which are modelled in the following way: s (x) < 0 s (x) > 0

... ...

Foreground Background

(10)

Therefore one can use different types of shape representations. As a simple example we defined the shape s (x) as a binary function with s (x) = −1 within the shape region and s (x) = 1 outside similar to [20] where Cremers et al. use subspace methods to learn a representative set of shapes. Binary functions are used to encode shapes which lead to problems when interpolating between two instances. As a consequential step we use a signed-distance map with the constraints (10) which implicitly includes a distance information towards the shape boundary. That means for our algorithm that the more a pixel is within the shape boundary, the more likely this region belongs to the desired segmentation. In Fig. 5 we show the benefit on using a signed-distance map as shape representation compared to a binary one. This representation and the combination with a GAC energy allows to handle deformations with a single prior. 3.1

Solving the Shape Prior Segmentation Model

It is well known that functionals like (9) are difficult to optimize due to the L1 norm |∇u|. Chan et al. [21], Carter [22] and Chambolle [23, 24] proposed a dual formulation for optimizing the classical variational problem of Rudin, Osher and Fatemi [25] for image denoising. Such a primal-dual approach can be applied to our minimization problem (9). The main intention is to remove the singularity by introducing the dual formulation of the weighted TV-norm     g |∇u| dx = max − u div p dx , (11) Ω



 d T

||p||≤g

Ω

where p = p1 , . . . , p : Ω → Rd is the dual variable with d being the problems dimension. Combining this maximization problem with the initial minimization task (9) this leads to

A Variational Model for Interactive Shape Prior Segmentation

    min max − u div p dx + λ s (x) u dx .

0≤u≤1 ||p||≤g

Ω

205

(12)

Ω

For a fixed shape prior the outline of the primal-dual optimization algorithm is given as follows: 1. Primal update: The primal update accomplishes the segmentation update and therefore performs the optimization according to the minimization of u:    ∂  − u · div p dx + λ s (x) u dx = − div p + λs (x) (13) ∂u Ω Ω Performing a gradient descent update scheme this leads to   un+1 = Π[0,1] un − τP (− div p + λs (x) ) ,

(14)

where τP denote the steplength and the orthogonal projection Π towards the binary set [0, 1] can be done with a simple thresholding step. 2. Dual update: The maximization according to p can be stated as    ∂  p · ∇u dx + λ s (x) u dx = ∇u (15) ∂p Ω Ω with the additional constraint ||p|| ≤ g. This results into a gradient ascent method with a orthogonal reprojection to restrict the length of p to the weight g:   pn+1 = ΠB0g pn + τD ∇u (16) Here B0g denotes a d-dimensional ball centered at the origin with the radius g. The reprojection onto B0g can be formulated with   ΠB0g q =

q  ||q|| max 1, g

(17)

3. Iterate until convergence: Solving the optimization problem (12) results in a consecutive update scheme with a gradient descent (14) and a gradient ascent step (16). Such an iterative algorithm demand on a convergence criterion. Therefore we take the energy of the single steps into account: Primal energy: The primal energy can be calculated by solving (12) by maximizing the equation towards the dual variable p. Due to (11) p can be restated as ∇u g |∇u| p ∈ B0g

p=

if ∇u  =0

(18)

else

for the optimization. This results into the energy equation (19) which is the same as evaluating the energy functional (9).  g |∇u| + λ s (x) u dx. (19) EP rimal = Ω

206

M. Werlberger et al.

Fig. 3. Relation of primal to dual energy while optimizing with the proposed primaldual update scheme. The plot shows the iterations in a logarithmic scale over 100 iterations. Note that after 20 iterations the primal-dual gap is small enough to stop iterating.

Dual energy: The dual energy can be formulated by minimizing (12) towards u: 

  min u − div p + λs (x) dx , (20) 0≤u≤1

Ω

which conclude that the binary segmentation u ∈ {0, 1} is set to u = 1 if the term − div p + λs (x) < 0 and u = 0 otherwise:    min − div p + λs (x) , 0 dx (21) EDual = Ω

In [26, 27], Zhu et. al. introduce a measurement for the convergence state of primal-dual algorithms in case of the ROF model. As a criterion they use the gap between the primal and dual energy. Applied to our primal-dual optimization algorithm we get an adaption of the energies like in Fig. 3. Therefore we use a fixed stepwidth for the primal (τP ) and dual update (τD ) with the constraint that τP τD ≤ 12 . For all the results shown in this paper we used τP = τD = √12 . We also tried adaptive timesteps (τP , τD ) for the optimization steps similar to the work of Zhu and Chan in [26] but did not find a reasonable equivalent for our method. 3.2

Shape Alignment

So far, our considerations assume a spatially fixed shape prior s (x) . In order to adapt the shape prior to different locations in the image we introduce a set of transformation parameters φ = {t, R, S} with the transformation parameters t for translation, R for rotation and S for the scale. Imposing this transformation into the segmentation energy (9) leads to an additional optimization parameter:      min φ (t, R, S) ◦ s (x) u dx (22) g |∇u| dx + λ u,φ

Ω

Ω

A Variational Model for Interactive Shape Prior Segmentation

207

Fig. 4. Segmentation of a vertebra in an X-ray image of the spline. Due to very bad contrast the segmentation without prior would fail completely. The definition of the prior was prepared by us and therefore cannot be considered as reference data.

In [20], Cremers et al. show with the help of the Lipschitz continuity that with an integrated rigid body motion the energy functional remains convex and therefore can be optimized globally for fixed transformation parameters. They additionally show that for optimizing the shape position itself the complete subspace Ω where the energy (9) is defined has to be sampled on a rather fine grid with all possible shape positions for evaluating the minimization task in a global manner. This is of course not feasible for an interactive application due to the need of optimizing the problem in real-time. Therefore we added a semiautomated approach that allows the user to have influence on the shape position and in addition an automated search for the optimal transformation parameters in a local neighborhood can be done. The optimization scheme in Sect. 3.1 can be retained with an extension on optimizing the transformation φ: 1. Solve shape prior segmentation model. 2. Optimize transformation parameters φ with a fixed u: Here we use a semi-automated position optimization of the shape prior. First, the user has the possibility to do a coarse positioning of the shape and second an optimization step tries to fit the shape to desired surrounding structures. Therefore we evaluate the energy (19) for different transformations φ. The optimal position is found where (19) has its minimum. The user gets an immediate result while changing the shape position and therefore can directly interact with the segmentation algorithm. 3. Iterate until convergence. Note that the domain of φ is restricted due to performance issues. Doing a complete search over the whole parameters space of the transformation parameters, a global optimal solution of (22) can be calculated like in [20].

4

Experimental Results

To reach real-time performance we have to compute the iterative solution as fast as possible. Due to the good parallelization attributes of variational algorithms we decided to implement the method using GPGPU programming with the help of NVIDIAs CUDA. The involved enhancements offer the possibility to combine

208

M. Werlberger et al.

Fig. 5. The same dataset as in Fig. 1 was used to evaluate the shape alignment step. Therefore the shape prior is placed nearby the desired bone (left image) and the alignment step searches for the optimal position in the local neighborhood. The results (middle image) show that the segmentations are equivalent to the hand-labeled points. The right image shows the alignment step using a binary shape prior which fails for the optimization step. For the correct result in the middle image a signed distance representation of the shape term was used.

Fig. 6. The position refinement is robust against partial occlusion

user interactivity with the computational intensive variational method and get a segmentation result at interactive rates. The performance depends mainly on the size of the search region in the parameter space and on the shape prior’s size as well. As an example we achieve 80 frames per second on a NVIDIA GeForce GTX 280 for pure segmentation and 20 frames per second including the position optimization. In our Framework we have two possibilities to provide a shape prior for the segmentation. Either the user can define a shape prior directly with a segmentation of a structure using pure GAC energy by setting foreground and background seeds. This is especially useful when recurring structures have to be segmented. For more accurate results especially on difficult data we can load a predefined shape structure that is used for the segmentation method. In Fig. 1 a predefined prior is used for segmenting finger bones and compare the segmentation result with a simple intensity thresholding and a segmentation with the pure GAC-energy. The result shows an identical segmentation result for the shape prior segmentation and the reference data labeled by an expert. A more difficult example is shown in Fig. 4 which shows a segmentation of a single vertebra in an X-ray image of the spline. Due to the very bad contrast simple thresholding would obviously fail completely and also pure GAC energy would end up into setting very much seed information.

A Variational Model for Interactive Shape Prior Segmentation

209

Fig. 7. Real-time tracking of an espresso cup in a live camera stream

Fig. 8. Multiple vertebrae of the cervical spline are tracked separately through a sequence. The intention is to ascertain the movement of the vertebrae towards each other.

Fig. 9. Segmentation of bottles with a single shape prior

Examples of automated shape alignment with optimizing the transformation parameters in a local neighborhood are shown in Fig. 5–8. Fig. 5 shows again the labeled bone dataset. The overlay with reference data shows that the position optimization leads to the correct segmentation. For non-medical image data examples are presented in Fig. 6 and 7. The first one shows that the proposed method is robust against partial occlusion. Furthermore the method can be used

210

M. Werlberger et al.

for tracking a certain structure over a sequence of frames. Fig. 7 shows a sequence of an espresso cup which was directly processed from a camera image with on-thefly segmentation and position optimization. For a restricted domain of movement we gain real-time performance for tracking a sequence. In Fig. 8 we track parts of the cervical spline in a moving X-ray image series to obtain a path of movement of the vertebrae to each other during a flexion. The predefined shapes of the four vertebrae are initialized in the first frame and than automatically tracked over the complete flexion. This can be used to ascertain shapes of implants for intervertebral discs. In Fig. 9 multiple objects are segmented with the help of a single shape prior. Therefore a prior is defined in form of a bottle and placed roughly on each bottle in the image. The fine adjustments are done automatically.

5

Conclusion

In this paper, we proposed a globally optimal shape prior segmentation method with additional user interaction and automated position refinement. With this approach we can handle very different images and gain robust segmentation results. Especially the segmentation for difficult data like the low-contrast spline image benefits from the additional shape information. A great advantage of variational methods like this are the parallelization capability that especially profits by the modern graphics hardware that are able to boost the performance of such highly parallel algorithms.

References 1. Cremers, D., Tischhäuser, F., Weickert, J., Schnörr, C.: Diffusion snakes: Introducing statistical shape knowledge into the Mumford–Shah functional. International Journal of Computer Vision 50(3), 295–313 (2002) 2. Leventon, M., Faugeraus, O., Grimson, W.: Level set based segmentation with intensity and curvature priors. In: Workshop on Mathematical Methods in Biomedical Image Analysis, pp. 4–11 (2000) 3. Leventon, M., Grimson, W., Faugeras, O.: Statistical shape influence in geodesic active contours. In: Proc. IEEE Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 316–323. IEEE, Los Alamitos (2000) 4. Paragios, N., Rousson, M., Ramesh, V.: Matching distance functions: A shape-toarea variational approach for global-to-local registration. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002. LNCS, vol. 2351, pp. 775–789. Springer, Heidelberg (2002) 5. Paragios, N., Rousson, M., Ramesh, V.: Non-rigid registration using distance functions. Computer Vision and Image Understanding 89(2-3), 142–165 (2003) 6. Unger, M., Pock, T., Bischof, H.: Continuous Globally Optimal Image Segmentation with Local Constraints. In: Computer Vision Winter Workshop (2008) 7. Unger, M., Pock, T., Trobin, W., Cremers, D., Bischof, H.: TVSeg - Interactive total variation based image segmentation. In: British Machine Vision Conference (2008)

A Variational Model for Interactive Shape Prior Segmentation

211

8. Mumford, D., Shah, J.: Optimal approximations by piecewise smooth functions and variational problems. Comm. on Pure and Applied Math. XLII(5), 577–685 (1988) 9. Chan, T.F., Vese, L.A.: Active contours without edges. IEEE Trans. Image Processing 10(2), 266–277 (2001) 10. Potts, R.B.: Some generalized order-disorder transformations. Proc. Camb. Phil. Soc. 48, 106–109 (1952) 11. Chan, T.F., Esedoglu, S., Nikolova, M.: Algorithms for finding global minimizers of image segmentation and denoising models. SIAM Journal of Applied Mathematics 66(5), 1632–1648 (2006) 12. Kass, M.: Snakes: Active contour models. International Journal of Computer Vision 1(4), 321–331 (1980) 13. Caselles, V., Kimmel, R., Sapiro, G.: Geodesic active contours. International Journal of Computer Vision 22(1), 61–79 (1997) 14. Kichenassamy, S., Kumar, A., Olver, P., Tannenbaum, A., Yezzi, A.: Conformal curvature flows: From phase transitions to active vision. Archive for Rational Mechanics and Analysis, 275–301 (1996) 15. Kichenassamy, S., Kumar, A., Olver, P.J., Tannenbaum, A.R., Yezzi Jr., A.J.: Gradient flows and geometric active contour models. In: International Conference on Computer Vision, pp. 810–815 (1995) 16. Leung, S., Osher, S.: Global minimization of the active contour model with TVinpainting and two-phase denoising. In: Paragios, N., Faugeras, O., Chan, T., Schnörr, C. (eds.) VLSM 2005. LNCS, vol. 3752, pp. 149–160. Springer, Heidelberg (2005) 17. Bresson, X., Esedoglu, S., Vandergheynst, P., Thiran, J.P., Osher, S.J.: Global minimizers of the active contour/snake model. In: International Conference on Free Boundary Problems: Theory and Applications (FBP) (2005) 18. Bresson, X., Esedoglu, S., Vandergheynst, P., Thiran, J.P., Osher, S.J.: Fast global minimization of the active contour/snake model. J. of Mathematical Imaging and Vision 28(2), 151–167 (2007) 19. Chan, T.F., Esedoglu, S.: Aspects of total variation regularized L1 function approximation. SIAM Journal of Applied Mathematics 65(5), 1817–1837 (2005) 20. Cremers, D., Schmidt, F.R., Barthel, F.: Shape priors in variational image segmentation: Convexity, Lipschitz continuity and globally optimal solutions. In: Computer Vision and Pattern Recognition, pp. 1–6 (2008) 21. Chan, T.F., Golub, G.H., Mulet, P.: A nonlinear primal-dual method for total variation-based image restoration. SIAM Journal on Scientific Computing 20(6), 1964–1977 (1999) 22. Carter, J.: Dual Methods for Total Variation-based Image Restoration. PhD thesis, UCLA (2001) 23. Chambolle, A.: An algorithm for total variation minimization and applications. Journal of Mathematical Imaging and Vision 20(1-2), 89–97 (2004) 24. Chambolle, A.: Total variation minimization and a class of binary MRF models. In: Rangarajan, A., Vemuri, B.C., Yuille, A.L. (eds.) EMMCVPR 2005. LNCS, vol. 3757, pp. 136–152. Springer, Heidelberg (2005) 25. Rudin, L.I., Osher, S.J., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Physica D: Nonlinear Phenomena 60, 259–268 (1992) 26. Zhu, M., Chan, T.: An efficient primal-dual hybrid gradient algorithm for total variation image restoration. UCLA CAM Report 08-34 (2008) 27. Zhu, M., Wright, S.J., Chan, T.F.: Duality-based algorithms for total variation image restoration. UCLA CAM Report 08-33 (2008)

A Nonlinear Probabilistic Curvature Motion Filter for Positron Emission Tomography Images Musa Alrefaya, Hichem Sahli, Iris Vanhamel, and Dinh Nho Hao Vrije Universiteit Brussel, Dept. Electronics and Informatics ETRO-IRIS, Pleinlaan 2, B-1050 Brussels, Belgium {malrefay,hsahli,iuvanham}@etro.vub.ac.be http://www.etro.vub.ac.be

Abstract. Positron Emission Tomography (PET) is an important nuclear medicine imaging technique which enhances the effectiveness of diagnosing many diseases. The raw-projection data, i.e. the sinogram, from which the PET is reconstructed, contains a very high level of Poisson noise. The latter complicates the PET image’s interpretation which may lead to erroneous diagnoses. Suitable denoising techniques prior to reconstruction can significantly alleviate the problem. In this paper, we propose filtering the sinogram with a constraint curvature motion diffusion for which we compute the edge stopping function in terms of edge probability under the assumption of contamination by Poison noise. We demonstrate through simulations with images contaminated by Poisson noise that the performance of the proposed method substantially surpasses that of recently published methods, both visually and in terms of statistical measures.

1

Introduction

Positron Emission Tomography (PET) is an in vivo nuclear medicine imaging method that provides functional information of the body tissues. The PET image results from reconstructing very noisy, low resolution raw data, i.e. the sinogram, in which important features are shaped as a curved structures. Enhancing the PET image spurred a wide range of denoising models and algorithms. Some methodologies focus on enhancing the reconstructed PET image directly, where others prefer enhancing the sinogram prior to reconstruction. Existing methods may suffer drawbacks such as the careful selection of a high number of parameters, smoothing of the important features’ boundaries, or prohibitive computation. Recently, nonlinear diffusion techniques have been investigated for PET images. Many researchers did explore the application of the well-known Perona 

The authors sincerely wishes to express great thanks to Prof. M. Defrise, Division of Nuclear Medicine at AZ-VUB, for his discussions and feedback. The comparison to the TV-Nestrove scheme would not have been possible without the help of Dr. Pierre Weiss from INRIA-France, who did provide us the Matlab Code.

X.-C. Tai et al. (Eds.): SSVM 2009, LNCS 5567, pp. 212–223, 2009. c Springer-Verlag Berlin Heidelberg 2009 

A Curvature Motion Filter for PET

213

and Malik anisotropic diffusion [15] in combination with diverse diffusivity functions, on PET images [2, 4, 5, 14, 26], as well as on sinograms [6, 25]. The main drawback of this filter, with respect to the Poisson noise, which characterize such type of images, is that the diffusion produces important oscillations in the gradient, which finally leads to a poorly smoothed image [28, 29]. Moreover, the adopted diffusivity functions do not consider the special properties of the sinogram in which the preservation of the curved-shape features is paramount (see Figure1. In [28], mean curvature motion and Gaussian curvature motion of PET images have been investigated. Total variation (TV) scheme for smoothing the PET images was also discussed in [28]. Happonen et al. [29] propose filtering the sinogram in the stackgram domain where the signal along the sinusoidal trajectories of the sinogram can be filtered separately. They used and compared the Gaussian and nonlinear filters. Filtering the sinogram has the advantage that the noise distribution is known, which is not the case after reconstruction. Consequently, this work proposes filtering the sinogram by means of a curvature constrained filter in which the amount of diffusion is modulated according to a probabilistic diffusivity function that suits images contaminated with Poisson noise. In addition, a comparison of the proposed method with TV-based methodologies proposed in [1] and [12], is conducted. For this purpose, a simulated thorax PET phantom was constructed to which varying levels of Poisson noise have been added is used. For evaluating the filtering approaches, contrast noise curves (contrast versus background noise at different iteration numbers) were generated for the different filtering approaches. The reminder of the paper is organized as follows. Section 2, briefly review the notions of curvature motion, edge affected diffusion filtering, and self-snakes. The proposed filtering scheme is introduced in Sect. 3. Section 4.1 introduces the applied validation methods, the remainder of Sec.4 discusses the experimental results. Conclusions and future work are given in Sec.5.

2

Geometry Driven Scale-Space Filtering

This section reviews the formulations for mean curvature motion (MCM), Edge Affected Variable Conductance Diffusion (EA-VCD), and self-snakes. Let f be a scalar image defined on the spatial image domain Ω, then the family of diffused versions of f is given by: U (f ) : f (.) → u(., t) with u(., 0) = f (.)

(1)

where U is referred to as the scale-space filter, u is denoted the scale-space image, and the scale t ∈ R+ [23, 26]. The denoised or enhanced version of f , is a given u(., t) that is closest to the unknown noise-free version of f . 2.1

Curvature Motion

One way of introducing smoothness in the curve is to let it evolve under its Euclidean curvature k. Mean curvature motion (MCM) is considered as the

214

M. Alrefaya et al.

standard curvature evolution. MCM allows diffusion solely along the level-lines. In Gauge coordinates the corresponding PDE formulation is:   ∇u ut (., t) = uvv = k|∇u| = div |∇u| (2) |∇u| Hence diffusion solely occurs along the v-axis. 2.2

Edge Affected Variable Conductance Diffusion

Variable Conductance filtering (VCD) is based on the diffusion with a variable conduction coefficient that controls the rate of diffusion [23]. In the case of Edge Affected-VCD (EA-VCD), the conductance coefficient is inversely proportional to the edgeness. Consequently it is commonly referred to as the edge stopping function (g), in which the edgeness is typically measured by the gradient magnitude. The EA-VCD is governed by: ut = div [g(|∇u|)∇u]

(3)

The above PDE system together with the initial condition given in (1) is completed with homogenous von Neumann boundary condition on the boundary of the image domain. Note that the Perona and Malik’s antitropic diffusion [15] is an EA-VCD. 2.3

Self-Snakes

Self-snakes are a variant of the MCM in which an edge-stopping function is introduced [19]. The main goal is preventing further shrinking of the level-lines once they have reached the important image edges. For scalar images, self-snakes are governed by:   ∇u ut = |∇u|div g(|∇u|) (4) |∇u| This equation adopts the same boundary condition as (3). Furthermore, it can be decomposed in two parts [19, 23]:   ∇u ut = g|∇u|div |∇u| +(∇g).∇u (5) = gk|∇u| +(∇g).∇u The first part describes a degenerate forward diffusion along the level lines, i.e. orthogonal to the local gradient; it allows preserving the edges. Additionally, the diffusion is limited in areas with high gradient magnitude and encouraged in smooth areas. Actually the first term is the constraint curvature motion. The second term can be viewed as a shock filter since it pushes the level-lines towards valleys of high gradient, acting as Osher’s shock filter [18].

A Curvature Motion Filter for PET

3

215

The Probabilistic Curvature Motion Filter

Based on (i) the curvature motion method, and (ii) a probabilistic diffusivity function, we presented in an earlier work [16], this section introduces the proposed filtering schemes for PET sinogram, considering the following characteristics: 1. The important features in the sinogram are curved structures with high contrast values. These represent the region of interests in the reconstructed PET image, e.g. tumor. 2. The weak edges in the sinogram are the edges that contains low contrast values. In other words, edges with small |uww |. 3. The noise in the sinogram is a priori identified as a Poisson noise. The above presented schemes, namely, MCM (2), EA-VCD (3) and the SelfSnakes (5), can be derived using the following general equation: ut = g1 (|∇u|)uvv + g2 (|∇u|)uww

(6)

where the second order Gauge derivatives of the image in the (vv) and (ww) directions are given by: uvv =

uxx u2y −2ux uy uxy +uyy u2x

(u2x +u2y )

uww =

uxx u2y +2ux uy uxy +uyy u2x

(u2x +u2y )

(7)

Equation (6) comprises, a diffusion modulated by g1 along the image edges (vv) (a smoothing term), and a diffusion adjustable by g2 across the image edges (ww) (a sharpening term). Careful modeling of these terms allows efficiently denoising the PET sinograms, whilst keeping their interesting features. In the following sections, we propose the use of a probabilistic diffusivity function, and derive two diffusions schemes, for which we did apply the Gauge derivatives numerical approximation that was described in [23]. 3.1

The Probabilistic Diffusivity Function

The main idea of the Probabilistic Diffusivity Function [16] is to express the diffusivity function as the probability that the observed gradient presents no edge of interest under a suitable marginal prior distribution for the noise-free gradient histogram. The diffusivity function was defined as: gpr (x) = A(1 − P (H1 |x))

(8)

where the normalizing constant A is set to A = 1/(1 − P (H1 |0)) to ensure that gpr (0) = 1; the hypothesis H1 describes the notion whether an edge element of interest is present given the considered noise, and H0 an edge element of interest is absent. Formally, H0 : y ≤ σn , and H1 : y > σn (9)

216

M. Alrefaya et al.

with y being the ideal, noise-free, gradient magnitude, and σn the noise standard deviation in the observed gradient image. In [16] it has been demonstrated that gpr (x) = (1 + μη(0))

1 1 + μη(x)

(10)

where μ = P (H1 )/P (H0 ) is the prior odds, and η(x) = p(x|H1)/p(x|H0) is the likelihood ratio.  −1 Considering a Laplacian prior p(y) = λ2 e−λ|y| , we have μ = eλσn − 1 [16], and the parameter λ can be estimated as λ = [0.5(σ 2 − σn2 )]−1/2

(11)

with σ 2 denoting the variance of the noisy image, and σn2 , as defined above. Due to limited space, the reader is referred to [16] for the detailed expression of η(x) in (10). The proposed diffusivity function,(10), has no free parameters to optimize, and it fits well in the cluster of the reference backward-forward diffusivities. Indeed, for the considered PET sinograms, the noise standard deviation, σn ,in 11 is being estimated as σn2 = V ar(uLn ) where the image noise uLn is reconstructed from the two finest resolution levels coefficients by applying the wavelet decomposition of u, using daubechi(4) function. 3.2

Probabilistic Constraint Curvature Motion

For the probabilistic constraint curvature motion (PCCM), we start from a constraint version of mean curvature motion: the diffusion across the level lines is prohibited whilst the diffusion along the level-lines is controlled via the probabilistic diffusivity function (10): ut = gpr (|∇u|)uvv = gpr (|∇u|)k|∇u|

(12)

. Thus the function g1 in (6) is chosen to be g1 = gpr , for dealing with Poisson noise. This filter effectively smooths the image, as well as preserves edges of the important features such as lines, curve and flow-like structures. By its nature, the PCCM cannot enhance the weak edges and/or features in the sinogram. The second term in (6) allows the sharpening. Consequently, we . set g2 = g1 . In this way, weak but important edges are enhanced whilst the noise is removed efficiently. Formally, the enhanced PCCM (ePCCM) is given by:  ut = gpr (|∇u|)uvv + gpr (|∇u|)uww

3.3

(13)

Probabilistic Self Snakes (PSS)

It can be demonstrated that the diffusion of scalar images via EA-VCD can be decomposed into (5), moreover, it can be rewritten as [7, 23]: ut = g(|∇u|)uvv + [g(|∇u|) + g  (|∇u|)|∇u|] uww

(14)

A Curvature Motion Filter for PET

217

consolidating the properties of both the self-snakes and the EA-VCD into a single diffusion schema. Considering equation (6), and the proposed probabilistic . . diffusivity function, we have g1 (x) = gpr (x), and the sharpening term, g2 (x) =  gpr (x) + xgpr (x). This filter proves to be very effective and flexible for the sinogram image where the high contrast regions, which represent a tumor in the reconstructed PET, should be smoothed wisely without blurring the poor edges. Like EA-VCDT, the main advantage of this filter is that the average gray value of the image is not altered during the diffusion process which is a significant issue in the sinogram.

4 4.1

Experiments and Discussion Introduction

The goal of the conducted experiments consists of measuring the performance of the proposed filtering methods, and studying their influence on the two commonly used PET reconstruction methods. The filtered back projection (FBP) [11], and the iterative ordered subset expectation maximization (OSEM) [8], respectively. The FBP algorithm is computationally efficient while the OSEM algorithm can incorporate easily prior information on the image to improve image quality. In our experiments, the reconstruction parameters of these algorithms are set as follows: the hamming parameter in the FBP method is 0.5, while for the OSEM, we use 16 subsets and run it for 4 iterations. In the following the reconstructed PET images are denoted by UF (t) = FBP(ut ), and UO (t) = OSEM(ut ), respectively, for a given enhanced sinogam ut . A simulated thorax PET phantom, containing three regions of interest (tumors) was constructed. To which varying levels of Poisson noise have been added and used for the evaluation. 50 realizations (noisy sinograms) with added noise level of 1x106 coincident events, have been generated. Each sinogram has a mm size of 256x256 pixels and their spacing is 2x2 pixel . Figure.1(a) shows the ideal noise-free sinogram with the PET images obtain via the FBP (Fig.1(c)) and, OSEM (Fig.1(d)) reconstruction. A corresponding noise contaminated realization is shown in Fig.1(b),(e)-(f). The proposed PCCM and PSS diffusion schemes have been assessed and evaluated against resent Total Variation (TV) denoising techniques, namely, the approach of Chambolle [1], denoted here after as TV-C, and the Nesterov [12] algorithm, denoted as TV-N. 4.2

Quantitative Evaluation Measures

Two types of evaluation measures are adopted. The first set stems from measuring the quality of the filtering techniques whilst the second set originates from validating the quality of the PET reconstruction. As ground-truth information, the former uses the noise-free image, whilst the latter needs prior identification of the important areas by a medical professional.

218

M. Alrefaya et al.

(a)

(b)

(c)

(d)

(e)

(f)

Fig. 1. (a) Original simulated sinogram and its reconstructed PET image using (b) FBP (c) OSEM reconstruction. (d) An example of one realization of a noisy sinogram and (e-f) the corresponding reconstructed PET images. The tumors (ROI) are the 3 clearly visible white spots.

Denoising Quality. The idea is to verify the quality of the denoised sinogram, ut , with respect to the noise-free image I. In this work, we adopt the following measures [22]: DQ1: The Peak Signal to Noise Ratio (PSNR) is a statistical measure of error, used to determine the quality of the filtered images. It represents the ratio of a signal power to the noise power corrupting. Obviously, one sees that the higher the PSNR, the better the quality. P SN R(t) = 10log10 

Card(Ω) |I(p) − ut (p)|

(15)

p∈Ω

DQ2: The correlation (Cmρ ) between the noise-free and the filtered image. The higher this correlation the better the quality is. Cmρ (t) = ρ [I, ut ]

(16)

DQ3: The calculated variance of the noise (NV) describes the remaining noiselevel. Therefore, it should be as small as possible. NV(t) = V ar (|I − ut |)

(17)

In this work, we are interested in comparing the maximum of each measure for the different filtering approaches. The latter yields the best obtainable result per measure.

A Curvature Motion Filter for PET

219

The Contrast Recovery Curve. For evaluating the filtering on the reconstructed PET images, the filtered data, at discrete scales for the proposed PDE approaches, and regularization parameters values, for the TV approaches, were reconstructed using the FBP and the OSEM approaches. With the reconstructed data sets we determine the contrast recovery curve using a set of region of interests that were identified by a medical professional. In our case, the 3 white spots that represent tumors in Fig.1. This was accomplished by quantifying a Contrast Gain, cg, and coefficients of variations V arcg . We calculate the contrast gain cgi for each realizations i ∈ [1, N = 50] and its overall variance. Let R = {r1 , r2 , . . . , rn } be the set of identified ROIs (n = 3 in our case), and B a representative background tissue area,then:   (i)  (i) 1 1 cgi (t) = n1 U (p, t) − Card(B) U (p, t) Card(r) r∈R

V arcg (t) =

1 N

N 

p∈r

(cgi (t) −

i=1

1 N

N 

p∈B

cgj (t))

2

(18)

j=1

where p is a pixel. The V arcg versus cg plot provides a straightforward evaluation method for the contrast-noise tradeoff [3]. The best quality PET reconstruction is situated in the upper, i.e. high contrast gain, left, i.e. high stability, area of the plot. 4.3

Evaluation

A fundamental issue with scale-spaces induced by diffusion processes, as the ones proposed in this paper, is the automatic selection of the most salient scale. For our PET sinogram denoising application, we use an earlier proposed optimal scale selection approach [22], where the maximum correlation method has been adopted:     σ[no (t0 )] topt = argmax Cmp (t) = argmax σ[ut ] + σ[no (t)] (19) σ[ut0 ] withno is the so-called outlier noise, which we estimated using the proposed wavelet-based noise estimation. Note that, t0 is the zeroth scale, thus ut0 = f and n(t0 ) represents the initial amount of noise. Figures 2.(a),(d) illustrates the obtained optimal scale using the PSS approach, and and the TV-C results, respectively. The Table 1 lists the quantitative results comparing the different denoising approaches. The best performing filtering method, per measure is, displayed in bold. As it can be seen, the best performing filtering is achieved when using the PSS. Furthermore, we notice that for all the used measures, the proposed diffusion methods outperform the considered TV-based filters. Figure 3 depicts the Contrast Recovery Curves for the investigated filtering methods. Recall that, the best enhancement is obtained when the contrast gain is as high as possible whilst the variance over it remains as small as possible. Furthermore, the degree of smoothness of the curve indicates the stability and biasing level.

220

M. Alrefaya et al. Table 1. Denoising quality measures v.s. filtering approaches f PCCM ePCCM PSS TV-C TV-N P SN R(topt ) -17.510 -4.1000 -4.1000 -3.9900 -5.7200 -5.1100 NV(topt ) 7.5100 1.6000 1.6000 1.5500 1.9300 1.8000 Cmρ (topt ) 0.8500 0.9915 0.9916 0.9917 0.9876 0.9890

(a)

(b)

(c)

(d)

(e)

(f)

Fig. 2. Enhanced sinogram and reconstructed PET images. PSS-based approach first row, TV-based approach second row. 1.5

Contrast Gain

1.4

1.3

1.2

PCCM + FBP ePCCM + FBP PSS + FBP PCCM + OSEM ePCCM + OSEM PSS + OSEM TV−N + FBP TV−C + FBP TV−N + OSEM TV−C + OSEM

1.1

1

0.9

0.8 1.5

Variance x 10 2

2.5

3

−3

3.5

4

4.5

Fig. 3. The contrast-noise curve. The best performance occurs in the area where the contrast gain is high and its variance is low.

A Curvature Motion Filter for PET

221

We may observe that all investigated methods have a good performance. However, the proposed PSS yields the best performance on the given data set.

5

Conclusions

Experiments show that combining the probabilistic diffusivity function with the curvature motion diffusion produces a powerful nonlinear filtering method that is appropriate for PET sinograms. It preserves the boundaries of the curvy shape features and wisely smoothes the regions of interest as well as the other regions. Our findings show that the PCCM method smoothes the PET images and keeps the boundaries of the important features, while the weak edges in some cases are vanished. On the other hand, the ePCCM method overcome this problem and the contrast recovered better in the ROIs by the enhancing term in the filter. This filter gives a well smoothed image and preserves the edges, and gains the advantage of the curvature motion diffusion and the shock filter. The PSS approach deal better with the problem of the poor and discontinuity of edges which is common in the PET images. Using the probabilistic diffusion function has proven to be an effective and suitable tool for controlling the diffusion process in the proposed scheme. The results, as shown in the contrast-noise curves, demonstrate that this function has a great capability to detect and enhance the important features edge’s in the high noisy sinogram images. Moreover, The proposed diffusivity function has no free parameters to optimize. All parameters are image-based, and are automatically estimated and proved to give the the best results.

References 1. Chambolle, A.: An Algorithm for Total Variation Minimization and Applications. JMIV 20, 89–97 (2004) 2. Chan, T., Li, H., Lysaker, M., Tai, X.C.: Level Set Method for Positron Emission Tomography. International Journal of Biomedical Imaging 2007 (2007) 3. Comtat, C., Kinahan, P.E., Fessler, J., Beyer, T., Townsend, D.W., Defrise, M., Michel, C.: Clinically feasible reconstruction of 3D whole-body PET/CT data using blurred anatomical labels. Phys. Med. Biol. 47, 1–20 (2002) 4. Demirkaya, O.: Diffusion Filtering of Functional Images using the structural information available in Hyprid imaging modalities. In: IEEE Medical Imaging Symposium, Germany (2008) 5. Demirkaya, O.: Post-reconstruction filtering of positron emission tomography whole-body emission images and attenuation maps using nonlinear diffusion filtering. Acad. Radiol. 11, 1105–1114 (2004) 6. Demirkaya, O.: Anisotropic diffusion filtering of PET attenuation data to improve emission images. Physics in Medicine Biology 47(20), 271–278 (2002) 7. Didas, S., Weickert, J.: Combining Curvature Motion and Edge-Preserving Denoising. In: Sgallari, F., Murli, A., Paragios, N. (eds.) SSVM 2007. LNCS, vol. 4485, pp. 568–579. Springer, Heidelberg (2007)

222

M. Alrefaya et al.

8. Hudson, M., Larkin, R.: Accelerated image reconstruction using ordered subsets of projection data. IEEE Trans. Med. Imag. 13(4), 601–609 (1994) 9. Happonen, A.P., Koskinen, M.O.: Experimental Investigation of Angular Stackgram Filtering for Noise Reduction of SPECT Projection Data: Study with Linear and Nonlinear Filters. International Journal of Biomedical Imaging 2007 (2007) 10. Jonsson, E., Huang, S.C., Chan, T.: Total Variation Regularization in Positron Emission Tomography. UCLA, Tech. Rep. no. 48 (1998) 11. Kak, C.A., Slaney, M.: Principles of Computerized Tomographic Imaging. IEEE Press, Los Alamitos (1999) 12. Nesterov, Y.: Smooth minimization of non-smooth functions. Mathematic Programming, Series A 103, 127–152 (2005) 13. Osher, S., Sethian, J.A.: Fronts propagating with curvature-dependent speed: algorithms based on Hamilton Jacobi formulations. Journal of Computational Physics 79(1), 12–49 (1988) 14. Padfield, D.R., Manjeshwar, R.: Adaptive conductance filtering for spatially varying noise in PET images. Progress in biomedical optics and imaging 7(3) no. 30 (2006) 15. Perona, P., Malik, J.: Scale space and edge detection using anisotropic diffusion. IEEE Transactions on Pattern Analysis and Machine Intelligence 12, 629–639 (1990) 16. Pizurica, A., Vanhamel, I., Sahli, H., Philips, W., Katartzis, A.: A Bayesian formulation of edge-stopping functions in non-linear diffusion. IEEE Signal Processing Letters 13(8), 501–504 (2006) 17. Rudin, L.I., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Physica D 60, 259–268 (1992) 18. Rudin, L.I., Osher, S.: Feature-oriented image enhancement with shock filters. Technical Report, Department of Computer Science, California Institute of Technology (1989) 19. Sapiro, G.: Geometric partial differential equations and image analysis. University Press, Cambridge (2001) 20. Sumengen, B., Manjunath, B.S.: Edgeflow-driven Variational Image Segmentation: Theory and Performance Evaluation. Technical Report, Department of Electrical and Computer Engineering University of California, Santa Barbara (2006) 21. Turkheimer, F.E., Boussion, N., Anderson, A.N., Pavese, N., Piccini, P., Visvikis, D.: PET Image Denoising Using a Synergistic Multiresolution Analysis of Structural (MRI/CT) and Functional Datasets. The Journal of nuclear medicine 49, 657–666 (2008) 22. Vanhamel, I., Mihai, C., Sahli, H., Katartzis, A., Pratikakis, I.: Scale Selection for Compact Scale-Space Representation of Vector-Valued Images. International Journal of Computer Vision 4485 (2008) 23. Vanhamel, I.: Vector valued nonlinear diffusion and its application to image segmentation Ph.D. Thesis, Vrije Universiteit Brussel, Faculty of Engineering Sciences, Electronics and Informatics (ETRO) (2006) 24. Wang, Y., Zhou, H.: Total Variation Wavelet-Based Medical Image Denoising. International Journal of Biomedical Imaging 2006 (2006) 25. Wang, W.: Anisotropic Diffusion Filtering for Reconstruction of Poisson Noisy Sinograms. Journal of Communication and Computer 2(11), 16–23 (2005) 26. Weickert, J.: Anisotropic diffusion in image processing. ECMI Series. TeubnerVerlag, Stuttgart (1998)

A Curvature Motion Filter for PET

223

27. Weiss, P., Aubert, G., Blanc-Fraud, L.: Efficient schemes for total variation minimization under constraints in image processing. Technical Report 6260, INRIA (2007) 28. Zhu, H., Shu, H., Zhou, J., Bao, X., Luo, L.: Bayesian algorithms for PET image reconstruction with mean curvature and Gauss curvature diffusion regularizations. Computers in Biology and Medicine 37(6), 793–804 (2007) 29. Zhu, H., Shu, H., Zhou, J., Toumoulin, C., Luo, L.: Image reconstruction for positron emission tomography using fuzzy nonlinear anisotropic diffusion penalty. Med. Biol. Eng. Comput. 44(11), 983–997 (2006)

Finsler Geometry on Higher Order Tensor Fields and Applications to High Angular Resolution Diffusion Imaging Laura Astola and Luc Florack Department of mathematics and computer science, Eindhoven University of Technology, PO Box 513, NL-5600 MB Eindhoven, The Netherlands [email protected]

Abstract. We study three dimensional volumes of higher order tensors, using Finsler geometry. The application considered here is in medical image analysis, specifically High Angular Resolution Diffusion Imaging (HARDI) [1] of the brain. We want to find robust ways to reveal the architecture of the neural fibers in brain white matter. In Diffusion Tensor Imaging (DTI), the diffusion of water is modeled with a symmetric positive definite second order tensor, based on the assumption that there exists one dominant direction of fibers restricting the thermal motion of water molecules, leading naturally to a Riemannian framework. HARDI may potentially overcome the shortcomings of DTI by allowing multiple relevant directions, but invalidates the Riemannian approach. Instead Finsler geometry provides the natural geometric generalization appropriate for multi-fiber analysis. In this paper we provide the exact criterion to determine whether a field of spherical functions has a Finsler structure. We also show a fiber tracking method in Finsler setting. Our model also incorporates a scale parameter, which is beneficial in view of the noisy nature of the data. We demonstrate our methods on analytic as well as real HARDI data.

1

Introduction

High Angular Resolution Diffusion Imaging (HARDI) is a non-invasive medical imaging modality that measures the attenuation of directional MRI (Magnetic Resonance Imaging) signal due to the diffusion of water molecules. Diffusion weighted measurements are taken in several directions, typically ranging from 50 to 130 (equidistant) angular directions. It is assumed that this diffusion of water molecules reveals relevant information of the underlying tissue architecture. The so-called apparent diffusion coefficient, D(g), is computed from the StejskalTanner [2] formula S(g) = exp(−bD(g)), (1) S0 

The Netherlands Organisation for Scientific Research (NWO) is gratefully acknowledged for financial support.

X.-C. Tai et al. (Eds.): SSVM 2009, LNCS 5567, pp. 224–234, 2009. c Springer-Verlag Berlin Heidelberg 2009 

Finsler Geometry on HOT Fields

225

where S(g) is the signal associated with gradient direction g, S0 the signal obtained when no diffusion gradient is applied, and b is a parameter associated with the imaging protocol. In the Diffusion Tensor Imaging framework, (1) is interpreted as S(g) = exp(−bg T Dg) , S0

(2)

with the 3 × 3 two-tensor D describing the probability of directional diffusivity at each voxel. A natural way to do geometric analysis on the image, is to use the inverse of the diffusion tensor D as the Riemann metric tensor [3]. This approach has been exploited to some extent in the DTI literature [4], [5], [6], [7]. Since HARDI data typically contains more directional measurements than the traditional DTI, we study it as a metric space, but using a more refined model for directional information than can be accounted for by using only the local position dependent inner product i.e. Riemannian metric. Higher order tensor representations [8], [9], [10], [11] of HARDI data are well suited to differential geometric methods. We mention that Finsler geometry has already been introduced in HARDI setting. In the work of Melonakos et al. [12] the homogeneity condition is forced by normalizing the parameter-vectors, but we take a different approach, using higher order monomial tensors and an ODE-based fiber tracking method. This paper is organized as follows. In section 2, we give a very short introduction to Finsler geometry and in section 3, we show that indeed HARDI measurements can be modeled with a Finsler-structure and give the specific condition which ensures this. In section 4 we discuss how to switch back and forth between iterative polynomial tensor fitting, that allows Laplace-Beltrami smoothing, and a monomial tensor fitting convenient for constructing a Finsler-norm. In section 5 we show some results of fiber-tracking based on the local Finsler metric and demonstrate it on an analytical example as well as on a real HARDI data of a rat brain scan. In the appendix we will give the details of the construction of the strong convexity criterion.

2

Finsler Geometry

In a perfectly homogeneous and isotropic medium, geometry is Euclidean, and shortest paths are straight lines. In an inhomogeneous space, geometry is Riemannian and the shortest paths are geodesics. If a medium is not only inhomogeneous, but also anisotropic1 , i.e. has innate directional structure, the appropriate geometry is Finslerian [13] [14] and the shortest paths are correspondingly Finsler-geodesics. As a consequence the metric tensor depends on both, position and direction. This is also a natural model for high angular resolution diffusion images. 1

We will call a medium isotropic if it is endowed with a direction independent inner product, or Riemannian metric. In the literature such a medium is also often referred to as anisotropic due to the directional bias of the metric itself.

226

L. Astola and L. Florack

Definition 1. We denote the bundle of tangent spaces T(x,y) M (y = 0) as T M \ {0}. A Finsler norm is a function F : TM → [0, ∞) that satisfies each of the following criteria: 1. Differentiability: F is C ∞ on the tangent bundle T M \ {0}. 2. Homogeneity: F (x, λy) = λF (x, y). 3. Strong convexity: The Hessian matrix, with components gij (x, y) =

1 ∂ 2 F 2 (x, y) , 2 ∂y i ∂y j

(3)

is positive definite at every point (x, y) of T M \ {0}.

3

Finsler Norm on HARDI Higher Order Tensor Fields

We want to show that higher order tensors, such as those fitted to HARDI data, do define a Finsler norm, which can be used in the analysis of this data. We take as a point of departure a given orientation distribution function (ODF), which if normalized, is a probability density function on the sphere and which can be computed from the data by using one of the methods described in the literature [15], [16], [17], [18], [19]. It models the probability that a given direction corresponds to a direction of a fiber. We use the heuristics that a high probability of finding a fiber in direction y corresponds to a larger diffusivity and at the same time to a shorter travel time from the diffusing particle point of view. Just as in the Riemannian framework, we can actually take our metric tensor to be the inverse of a local  (y-dependent) two-tensor. We use the Einstein summation convention ai bi = i ai bi , and put y = (y1 , y2 , y2 ) = (sin θ cos ϕ, sin θ sin ϕ, cos θ) ,

(4)

thus y denotes a unit vector while y = ||y||y is a general vector in R3 . We denote the higher order spherical tensor (a homogeneous polynomial restricted to sphere) approximating the ODF as D. As an example, we show how a field of sixth order tensors D(x) defines a Finsler norm. This can be extended directly to all even order tensors. We put  1/6 F (x, y) = Dijklmn (x)y i y j y k y l y m y n . (5) In the following, we verify the defining criteria stated in Definition 1. 1. Differentiability: The tensor field D(x) is constructed by fitting a tensor to the set of angular samples at each voxel, using a least squares method. The data set with fixed angle is continuous in x by linear interpolation between the sample points and differentiable w.r.t. x using Gaussian derivatives. Therefore the tensor field itself is differentiable in x, and because D is always positive, differentiability of F w.r.t. x follows. The differentiability of F in y is obvious from Eq. (5).

Finsler Geometry on HOT Fields

227

Fig. 1. Left:A fourth order spherical harmonic (or tensor), representing the (not convexified) norm function and 3 ellipsoids illustrating the metric tensors corresponding to the 3 vectors with same color. Right: Similarly a sixth order spherical harmonic function with 3 metric tensors.

2. Homogeneity: Indeed for any α ∈ R+ , x ∈ M , v ∈ Tx M : 1/6  = αF (x, v) . F (x, αv) = Dijklmn (x)αv i αv j αv k αv l αv m αv n

(6)

3. Strong convexity: We now state a strong convexity criterion for a general Finsler norm in R3 , by analogy to the R2 -criterion by Bao et al [13]. We have put the derivation of the condition into the appendix, and merely state the result here. We consider the so-called indicatrix of the norm function F at any fixed x, which is the set {g | g : (θ, ϕ) → R3 , F (g) = 1}. In our case the indicatrix is the ODF, which can be easily seen from the homogeneity condition 2. in Definition 1. F (y(θ, ϕ)) =

1 =⇒ F (ODF (θ, ϕ) · y(θ, ϕ)) = 1 . ODF (θ, ϕ) 2

∂ ∂ We denote g˙ θ := ∂θ (g), g¨θ := ∂θ 2 (g) and similarly for ϕ. We define the following three matrices: ⎛ 1 2 3⎞ ⎛ 1 2 3⎞ ⎛ 1 2 3⎞ g¨ϕ g¨ϕ g¨ϕ g g g g¨θ g¨θ g¨θ m = ⎝ g˙ θ1 g˙ θ2 g˙ θ3 ⎠ , mθ = ⎝ g˙ θ1 g˙ θ2 g˙ θ3 ⎠ , mϕ = ⎝ g˙ θ1 g˙ θ2 g˙ θ3 ⎠ . (7) g˙ ϕ1 g˙ ϕ2 g˙ ϕ3 g˙ ϕ1 g˙ ϕ2 g˙ ϕ3 g˙ ϕ1 g˙ ϕ2 g˙ ϕ3

Then the strong convexity requires: (gij y˙ θi y˙ ϕj )2 det(mϕ ) det(mθ ) > 0 , and > . det(m) det(m) gij y˙ θi y˙ θj

(8)

Since we use linear interpolation between tensors, we only need to check the condition at original data-points. This condition is always met in our ODF-data, and we expect it to hold quite generally. The goal of this section was to define a Finsler-structure and in particular a Finsler metric tensor gij (x, y) corresponding to a given tensorial ODF. Indeed

228

L. Astola and L. Florack

in case the ODF is a symmetric tensor of order two, this metric tensor is equivalent to the Riemann metric tensor. Following our Finsler approach, instead of one metric tensor per voxel we obtain a bundle of metric tensors at any x. For illustration, see Fig.1.

4

Transforming a Polynomial Tensor to a Monomial Tensor

Assume we wish to apply Laplace-Beltrami smoothing to our spherical data, by which we obtain a field of spherical functions at any desired scale, and that we wish to use a tensorial representation of the data instead of spherical harmonics. As is shown in [10], this smoothing is easy to do, using iterative polynomial tensor fitting. The point here is that for Finsler analysis, we would rather work with a tensor representation of monomial form D(y) = Di1 ···in yi1 · · · yin ,

(9)

than with the equivalent polynomial expression ˜ D(y) =

n

˜ i1 ···i yi1 · · · yik , D k

(10)

k=0

but still exploit the convenient (co-domain) scale space representation of the latter: n ˜ i1 ···i yi1 · · · yik . ˜ τ) = e−τ k(k+1) D (11) D(y, k k=0

This poses no problem, since we can rather easily transform the polynomial expression to a monomial one, using the fact that our polynomials are restricted to the sphere (eq. (4)), thus we may expand a lower order tensor to a sparse higher order one and symmetrize it. We can also always transform the monomial expression to polynomial sum of irreducible monomial tensors using Clebschprojection [20].

5

Fiber Tracking in HARDI Data Using Finsler Geometry

In DTI setting the most straightforward way of tracking fibers is to follow the principal eigenvector corresponding to the largest eigenvalue of the diffusion tensor until some stopping criterion. This method cannot reveal crossings and only provides a single direction (if at all) per voxel. Instead computing the shortest paths according to the diffusion-induced Riemann metric tensor, we could expect these to be the candidates for real fibers [5]. Of course, most of the shortest paths (geodesics) are not representing actual fibers, and therefore we should extract the potential neural fibers from arbitrary geodesics based on their connectivity [6]. We show some results of solving well-connected geodesics in an analytic as well as in a real rat brain data.

Finsler Geometry on HOT Fields

5.1

229

Analytic Tensor Field

We treat an analytic norm field in R2 , but the situation can be directly extended to R3 . Let us take as a convex norm function at each spatial position  1 1 (12) F (ϕ) = (cos 4ϕ + 4) 4 = 5 cos4 ϕ + 2 cos2 ϕ sin2 ϕ + 5 sin4 ϕ 4 . This is an example of fourth order tensor on unit vectors. Such a tensor field could represent an infinitely dense field of orthogonally crossing fibers. From the fact that F has no x-dependence we conclude that the geodesic coefficients vanish and that the geodesics coincide with the Euclidean geodesics γ(t) = (t · cos ϕ, t · sin ϕ), i.e. straight lines. However the so-called connectivity of a geodesic [6], [21] is relatively large, only in cases, where the directional norm function is correspondingly small. In Finsler setting the connectivity measure m(γ) is:

 ηij γ˙ i γ˙ j dt m(γ) =  , (13) gij (γ, γ) ˙ γ˙ i γ˙ j dt where the ηij (γ) represents the covariant Euclidean metric tensor which in Cartesian coordinates reduces to the constant identity matrix, γ˙ the tangent to the curve γ and gij (γ, γ) ˙ the Finsler-metric tensor (which depends not only on the position on the curve but also on the tangent of the curve). For illustration we compute explicitly the metric tensors, using Cartesian coordinates:

1 g11 g12 gij = , (14) (5 cos ϕ4 + 2 cos ϕ2 sin ϕ2 + 5 sin ϕ4 )3/2 g21 g22 where g11 = 5(5 cos ϕ6 + 3 cos ϕ4 sin ϕ2 + 15 cos ϕ2 sin ϕ4 + sin ϕ6 ) g12 = g21 = −48 cos ϕ3 sin ϕ3 g22 = 5(cos ϕ6 + 15 cos ϕ4 sin ϕ2 + 3 cos ϕ2 sin ϕ4 + 5 sin ϕ6 ) 1 2

1 2

g˙ g ¨ 2 The strong convexity criterion gg¨˙ 1 gg˙ 2 − −g1 g˙ 2 > 0 in R [13] on the indicatrix g(ϕ), for metric (14) is satisfied for every ϕ, since

13 − 8 cos 4ϕ g¨1 g˙ 2 − g˙ 1 g¨2 = >0. 1 2 1 2 g˙ g − g g˙ (4 + cos 4ϕ)2

(15)

The connectivity measure for a (Euclidean) geodesic γ can be computed analytically:

dt , (16) m(γ) =

(4 + cos(4ϕ))1/4 dt 5π 7π which gives the maximal connectivities in directions { π4 , 3π 4 , 4 , 4 }, as expected. See Fig. 2 for an illustration. We observe that on such a norm field the Riemannian (DTI) framework would result in Euclidean geodesics and constant connectivity over all geodesics thus revealing no information at all of the angular heterogeneity.

230

L. Astola and L. Florack

20

3

2

10 1

0 3

2

1

1

2

3

1

10

2

20 20

10

0

10

20

3

Fig. 2. Left:A field of fourth order spherical harmonics as in the norm function eq. (12) representing dense crossings and some well connected geodesics, colored in red. Right: 200 equiangular metric tensors of the same norm function, and an ellipse with light blue color corresponding to the metric in direction ϕ = π4 .

Fig. 3. Left:Finsler geodesics emanating from a voxel, and the most connective ones in red. Right: Fibers through same neighborhood in the traditional DTI principal eigenvector tracking.

5.2

Real Rat Brain Data

The Subthalamic Nucleus is a small area in the brain, that is involved in physiopathology of Parkinson’s disease [22]. We computed the Finsler geodesics and their connectivities, having an initial point in several central voxels in the Subthalamic Nucleus. These voxels were located based on comparison to an atlas of rat brain [23]. We tracked Finsler geodesics using the standard equation (ODEformulation) [14](p.78) and second order Taylor approximation, with initial directions as the 49 measurement directions, stepsize 0.2 voxel size and for 10 steps. Then we selected those 30% of all geodesics that have the best connectivity. Compared to the traditional DTI-tracking, we found that one of the main

Finsler Geometry on HOT Fields

231

directions with strong connectivity typically coincide with the DTI-fibers, but we also found other potential fiber directions. For illustration see Fig. 3.

6

Conclusions and Future Work

We have seen that it is indeed possible to analyze spherical tensor fields using Finsler geometry. It gives new methods to work with the data and also has the potential to give new information on the data. Finsler geodesics and Finsler curvatures are examples of geometric measures that can be applied on HARDI fiber-analysis, and which will be a subject of extensive future work.

Acknowledgement The rat brain data acquired for a study [24], was kindly provided by Ellen Brunenberg.

References 1. Tuch, D., Reese, T., Wiegell, M., Makris, N., Belliveau, J., van Wedeen, J.: High angular resolution diffusion imaging reveals intravoxel white matter fiber heterogeneity. Magnetic Resonance in Medicine 48(6), 1358–1372 (2002) 2. Stejskal, E., Tanner, J.: Spin diffusion measurements: Spin echoes ion the presence of a time-dependent field gradient. The Journal of Chemical Physics 42(1), 288–292 (1965) 3. Cohen de Lara, M.: Geometric and symmetry properties of a nondegenerate diffusion process. The Annals of Probability 23(4), 1557–1604 (1995) 4. O’Donnell, L., Haker, S., Westin, C.F.: New approaches to estimation of white matter connectivity in diffusion tensor MRI: Elliptic PDEs and geodesics in a tensorwarped space. In: Dohi, T., Kikinis, R. (eds.) MICCAI 2002. LNCS, vol. 2488, pp. 459–466. Springer, Heidelberg (2002) 5. Lenglet, C., Deriche, R., Faugeras, O.: Inferring white matter geometry from diffusion tensor MRI: Application to connectivity mapping. In: Pajdla, T., Matas, J.G. (eds.) ECCV 2004. LNCS, vol. 3024, pp. 127–140. Springer, Heidelberg (2004) 6. Astola, L., Florack, L., ter Haar Romeny, B.: Measures for pathway analysis in brain white matter using diffusion tensor images. In: Karssemeijer, N., Lelieveldt, B. (eds.) IPMI 2007. LNCS, vol. 4584, pp. 642–649. Springer, Heidelberg (2007) 7. Astola, L., Florack, L.: Sticky vector fields and other geometric measures on diffusion tensor images. In: MMBIA 2008, IEEE Computer Society Workshop on Mathematical Methods in Biomedical Image Analysis, held in conjunction with CVPR 2008, Anchorage, Alaska, The United States. CVPR, vol. 20, pp. 1–7. Springer, Heidelberg (2008) 8. Özarslan, E., Mareci, T.: Generalized diffusion tensor imaging and analytical relationships between diffusion tensor imaging and high angular resolution diffusion imaging. Magnetic resonance in Medicine 50, 955–965 (2003) 9. Barmpoutis, A., Jian, B., Vemuri, B., Shepherd, T.: Symmetric positive 4th order tensors and their estimation from diffusion weighted MRI. In: Karssemeijer, N., Lelieveldt, B. (eds.) IPMI 2007. LNCS, vol. 4584, pp. 308–319. Springer, Heidelberg (2007)

232

L. Astola and L. Florack

10. Florack, L., Balmashnova, E.: Decomposition of high angular resolution diffusion images into a sum of self-similar polynomials on the sphere. In: Proceedings of the Eighteenth International Conference on Computer Graphics and Vision, GraphiCon 2008, Moscow, Russian Federation, June 2008, pp. 26–31 (2008) (invited paper) 11. Florack, L., Balmashnova, E.: Two canonical representations for regularized high angular resolution diffusion imaging. In: MICCAI Workshop on Computational Diffusion MRI, New York, USA, September 10, 2008, pp. 94–105 (2008) 12. Melonakos, J., Pichon, E., Angenent, S., Tannenbaum, A.: Finsler active contours. IEEE Transactions on Pattern Analysis and Machine Intelligence 30(3), 412–423 (2008) 13. Bao, D., Chern, S.S., Shen, Z.: An Introduction to Riemann-Finsler Geometry. Springer, Heidelberg (2000) 14. Shen, Z.: Lectures on Finsler Geometry. World Scientific, Singapore (2001) 15. Tuch, D.: Q-ball imaging. Magnetic Resonance in Medicine 52(4), 577–582 (2002) 16. Jansons, K., Alexander, D.: Persistent angular structure: New insights from diffusion magnetic resonance imaging data. Inverse Problems 19, 1031–1046 (2003) 17. Özarslan, E., Shepherd, T., Vemuri, B., Blackband, S., Mareci, T.: Resolution of complex tissue microarchitecture using the diffusion orientation transform. NeuroImage 31, 1086–1103 (2006) 18. Jian, B., Vemuri, B., Özarslan, E., Carney, P., Mareci, T.: A novel tensor distribution model for the diffusion-weighted MR signal. NeuroImage 37, 164–176 (2007) 19. Descoteaux, M., Angelino, E., Fitzgibbons, S., Deriche, R.: Regularized, fast and robust analytical q-ball imaging. Magnetic Resonance in Medicine 58(3), 497–510 (2006) 20. Müller, C. (ed.): Analysis of Spherical Symmetries in Euclidean Spaces. Applied Mathematical Sciences, vol. 129. Springer, New York (1998) 21. Prados, E., Soatto, S., Lenglet, C., Pons, J.P., Wotawa, N., Deriche, R., Faugeras, O.: Control Theory and Fast Marching Techniques for Brain Connectivity Mapping. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition, New York, USA, vol. 1, pp. 1076–1083. IEEE Computer Society Press, Los Alamitos (2006) 22. Hamani, C., Saint-Cyr, J., Fraser, J., Kaplitt, M., Lozano, A.: The subthalamic nucleus in the context of movement disorders. Brain, a Journal of Neurology 127, 4–20 (2004) 23. Paxinos, G., Watson, C.: The Rat Brain In Stereotaxic Coordinates. Academic Press, San Diego (1998) 24. Brunenberg, E., Prckovska, V., Platel, B., Strijkers, G., ter Haar Romeny, B.M.: Untangling a fiber bundle knot: Preliminary results on STN connectivity using DTI and HARDI on rat brains. In: Proceedings of the 17th Meeting of the International Society for Magnetic Resonance in Medicine (ISMRM), Honolulu, Hawaii (2009)

Finsler Geometry on HOT Fields

233

Appendix We seek the general condition for gij (y)v i v j > 0 ,

(17)

to be valid in R3 (= Tx M ). From the homogeneity of the norm function F , it follows that it is sufficient to have this condition on the unit level set of the norm. We consider this level surface i.e. the set of vectors y for which F (y) = 1 and a parametrization y(θ, ϕ) = (y 1 (θ, ϕ), y 2 (θ, ϕ), y 3 (θ, ϕ)). In what follows we abbreviate gij = gij (x, y). From F (y) = 1 we have gij y i y j = 1 .

(18)

Taking derivatives of both sides and using a consequence of Euler’s theorem for homogeneous functions ( [13] p.5) that says ∂gij k y =0, ∂y k

(19)

we obtain gij y˙ θi y j = 0

(20)

gij y˙ ϕi y j = 0 , implying y˙ θ ⊥g y and y˙ ϕ ⊥g y. Taking derivatives once more, we get gij y¨θi y j = −gij y˙ θi y˙ θj gij y¨ϕi y j = −gij y˙ ϕi y˙ ϕj i gij y¨θϕ yj

=

−gij y˙ θi y˙ ϕj

(21) .

We may express an arbitrary vector v as a linear combination of orthogonal basis vectors:

y˙ ϕ , y˙ θ v = αy + β y˙ θ + γ y˙ ϕ − y˙ θ . (22) y˙ θ , y˙ θ We substitute this expression for v to the left hand side of (17) and obtain:   (gij y˙ θi y˙ ϕj )2 i j 2 i j 2 i j 2 i j , (23) gij v v = α gij y y + β gij y¨θ y + γ gij y¨ϕ y − gij y˙ θi y˙ θj because the mixed terms vanish due to the orthogonality of basis vectors. On the other hand, for y’s on the indicatrix we have as a consequence of ∂F Euler’s theorem on homogeneous functions (denoting Fyi = ∂y i ): Fyi y i = F (y) = 1 .

(24)

234

L. Astola and L. Florack

Differentiating eq. (24) w.r.t. θ and ϕ, we obtain two equations: Fyi y˙ θi = 0

(25)

Fyi y˙ ϕi

(26)

=0,

for F is a homogeneous function. The matrices m, mθ , mϕ are as defined in eq. (7). Solving system of equations (24), (25) and (25) we get: Fy1 =

y˙ ϕ2 y˙ θ3 − y˙ ϕ3 y˙ θ2 y˙ ϕ3 y˙ θ1 − y˙ ϕ1 y˙ θ3 y˙ ϕ1 y˙ θ2 − y˙ ϕ2 y˙ θ1 , Fy2 = , Fy3 = . det(m) det(m) det(m)

(27)

Now using equalities gij y¨i θ y j = Fyk y¨θk ,

Fyi = gij y j , and

det(mθ ) , det(m)

−gij y¨θi y j =

gij y¨i ϕ y j = Fyk y¨ϕk ,

(28)

det(mϕ ) det(m)

(29)

−gij y¨ϕi y j =

we obtain

 2

gij v v = α − β i j

2

gij y¨θi y j

−γ

if det(mθ ) >0 det(m)

and

2

gij y¨ϕi y j

−

(gij y˙ θi y˙ ϕj )2



gij y˙ θi y˙ θj

(gij y˙ θi y˙ ϕj )2 det(mϕ ) > . det(m) gij y˙ θi y˙ θj

>0

(30)

(31)

Bregman-EM-TV Methods with Application to Optical Nanoscopy Christoph Brune, Alex Sawatzky, and Martin Burger Westfälische Wilhelms-Universität Münster, Institut für Numerische und Angewandte Mathematik, Einsteinstr. 62, D-48149 Münster, Germany {christoph.brune,alex.sawatzky,martin.burger}@wwu.de http://imaging.uni-muenster.de Abstract. Measurements in nanoscopic imaging suffer from blurring effects concerning different point spread functions (PSF). Some apparatus even have PSFs that are locally dependent on phase shifts. Additionally, raw data are affected by Poisson noise resulting from laser sampling and "photon counts" in fluorescence microscopy. In these applications standard reconstruction methods (EM, filtered backprojection) deliver unsatisfactory and noisy results. Starting from a statistical modeling in terms of a MAP likelihood estimation we combine the iterative EM algorithm with TV regularization techniques to make an efficient use of a-priori information. Typically, TV-based methods deliver reconstructed cartoon-images suffering from contrast reduction. We propose an extension to EM-TV, based on Bregman iterations and inverse scale space methods, in order to obtain improved imaging results by simultaneous contrast enhancement. We illustrate our techniques by synthetic and experimental biological data.

1

Introduction

Image reconstruction is a fundamental problem in many fields of applied sciences, e.g. nanoscopic imaging, medical imaging or astronomy. Fluorescence microscopy for example is an important imaging technique for the investigation of biological (live-) cells, up to nano-scale. In this case image reconstruction arises in form of deconvolution problems. Undesirable blurring effects can be ascribed to diffraction of light. Mathematically, image reconstruction in such applications can often be formulated as the solution of a linear inverse and ill-posed problem. The task consists of computing an estimation of an unknown object from given measurements. Typically these problems deal with Fredholm integral equations of the first kind ¯ , f¯ = Ku

(1)

¯ is a compact operator, f¯ (exact) data and u the desired image. In the where K ¯ is a convolution operator case of nanoscopic imaging K  ¯ (Ku)(x) = (k ∗ u)(x) = k(x − y)u(y)dy , Ω X.-C. Tai et al. (Eds.): SSVM 2009, LNCS 5567, pp. 235–246, 2009. c Springer-Verlag Berlin Heidelberg 2009 

236

C. Brune, A. Sawatzky, and M. Burger

where k is a convolution kernel, describing the blurring effects created by a ¯ is not suitable, nanoscopic apparatus. Determining u by direct inversion of K since (1) is ill-posed. In such cases regularization techniques are needed to produce reasonable reconstructions. A frequently used way to realize regularization techniques is the Bayesian model, whose aim is the computation of an estimate u of the unknown object by maximizing the a-posteriori probability density p(u|f ) with measurements f . The latter is given according to Bayes formula p(u|f ) ∼ p(f |u) p(u) .

(2)

This approach is called maximum a-posteriori probability (MAP) estimation. If the measurements f are given, we describe the density p(u|f ) as the a-posteriori likelihood function which depends on u only. The Bayesian approach (2) has the advantage that it allows to incorporate additional information about u via the prior probability density p(u) into the reconstruction process. The most frequently used prior densities are Gibbs functions p(u) ∼ e−α R(u) ,

(3)

where α is a positive parameter and R a convex energy. Usual models for the probability density p(f |u) in (2) are Gaussian- or Poisson-distributed raw data f , i.e. 2

p(f |u) ∼ e−Ku−f 2 /(2σ

2

)

,

p(f |u) ∼

 (Ku)fi i

fi !

i

e−(Ku)i ,

(4)

¯ In the canonical case of additive where K is a semi-discrete Operator based on K. Gaussian noise (see (4), left) the minimization of the negative log likelihood function (2) leads to classical Tikhonov regularization [1] based on minimizing a functional of the form   1 2 min Ku − f 2 + α R(u) . (5) u≥0 2 The first, so-called data-fidelity term, penalizes the deviation from equality in (1) whereas R is a regularization term as in (3). If we choose K = Id and the total variation (TV) regularization technique R(u) := |u|BV , we obtain the wellknown ROF-model [2] for image denoising. The additional positivity constraint is necessary in typical applications as the unknown represents a density image. In nanoscopic imaging measured data are stochastic and pointwise, more precisely, they are called "photon counts". This property refers to laser scanning techniques in fluorescence microscopy. Consequently, the random variables of measured data are not Gaussian- but Poisson-distributed (see (4), right), with expected value given by equation (1). Hence a MAP estimation via the negative log likelihood function (2) leads to the following variational problem [1]   min (Ku − f log Ku) dμ + α R(u) . (6) u≥0

Ω

Bregman-EM-TV Methods

237

Up to additive terms independent of u, the data-fidelity term is the so-called Kullback-Leibler functional (also known as cross entropy or I-divergence) between the two probability measures f and Ku. A particular complication of (6) compared to (5) is the strong nonlinearity in the data fidelity term and resulting issues in the computation of minimizers. In case of K = Id, i.e. in case of Poisson noise removal with total variation regularization, we refer to [3]. In the absence of regularization (α = 0) the EMalgorithm (cf. [4, 5, 6]) has become a standard scheme, which is however difficult to be generalized to regularized cases. Robust solutions of this problem for appropriate models of R is one of the novelties of this paper. The specific choice of the regularization functional R in (6) is important for how a-priori information about the expected solution is incorporated into the reconstruction process. Smooth, in particular quadratic regularizations have attracted most attention in the past, mainly due to the simplicity in analysis and computation. However, such regularization approaches always lead to blurring of the reconstructions, in particular they cannot yield reconstructions with sharp edges. Recently, singular regularization energies, in particular those of 1 or L1 -type, have attracted strong attention. In this work, we introduce an approach which uses total variation (TV) as the regularization functional. TV regularization was derived as a denoising technique in [2] and generalized to various other imaging tasks subsequently. The exact definition of TV [7], used in this paper, is  R(u) := |u|BV = sup u divg , (7) g∈C0∞ (Ω,Rd ), ||g||∞ ≤1

Ω

 which is formally (true if u is sufficiently regular) |u|BV = Ω |∇u|. The motivation for using TV is the effective suppression of noise and the realization of almost homogeneous regions with sharp edges. These features are attractive for nanoscopic imaging if the goal is to identify object shapes that are separated by sharp edges and shall be analyzed quantitatively. Unfortunately, images reconstructed by methods using TV regularization suffer from loosing contrast. In this paper, we suggest to extend EM-TV by iterative regularization to Bregman-EM-TV, attaining simultaneous contrast enhancement. More precisely, we apply total variation inverse scale space methods by employing the concept of Bregman distance regularization. The latter has been derived in [8] with a detailed analysis for Gaussian-type problems (5) and generalized to time-continuity [9] and Lp -norm data fitting terms [10]. Here, in the case of Poisson-type problems, the method consists in computing a minimizer u1 of (6) with R(u) := |u|BV first. Updates are determined successively by computing   l+1 l u = arg min (Ku − f log Ku) dμ + α ( |u|BV − p , u ) , (8) u∈BV (Ω)

Ω

l

where p is an element of the subgradient of the total variation semi norm in ul . Introducing the Bregman distance with respect to | · |BV defined via p˜ (u, u ˜) := |u|BV − |˜ u|BV − ˜ p, u − u ˜ , D|·| BV

p˜ ∈ ∂|˜ u|BV ⊆ BV ∗ (Ω) ,

(9)

238

C. Brune, A. Sawatzky, and M. Burger

where ·, · denotes the duality product, allows to characterize ul+1 in (8) as   pl l+1 l u = arg min (Ku − f log Ku) dμ + α D|·|BV (u, u ) . (10) u∈BV (Ω)

Ω

We will see that inverse scale space strategies can noticeably improve reconstructions for inverse problems with Poisson statistics like optical nanoscopy.

2

Reconstruction Methods

In literature there are two types of reconstruction methods that are used in general: analytic (direct) and algebraic (iterative) methods. A classical example for a direct method is the Fourier-based filtered backprojection (FBP). Although FBP is well understood and computationally efficient, iterative type methods obtain more and more attention in the applications mentioned above. The major reason is the high noise level (low SNR) and the type of statistics, which cannot be taken into account by direct methods. Hence, we will give a short review on the Expectation-Maximization (EM) algorithm [4, 11], which is a popular iterative algorithm to maximize the likelihood function p(u|f ) in problems with incomplete data. Then we will proceed to the presentation of the proposed EMTV and Bregman-EM-TV algorithm. 2.1

Reconstruction Method: EM Algorithm

In the absence of prior knowledge any object u has the same relevance, i.e. the Gibbs a-priori density p(u) in (3) is constant. We can then normalize p(u) such that R(u) ≡ 0. Hence (6) reduces to the constrained minimization problem  min (Ku − f log Ku) dμ . (11) u≥0

Ω

A suitable iteration scheme for computing stationary points, which also preserves positivity (assuming K preserves positivity), is the so called EM algorithm (cf. [12])   K∗ f , k = 0, 1, . . . . (12) uk+1 = uk ∗ K 1 Kuk For noise-free data f several convergence proofs of the EM algorithm to the maximum likelihood estimate, i.e. the solution of (11), can be found in literature [12,13,14,15]. Besides, it is known that the speed of convergence of iteration (12) is slow. A further property of the iteration is a lack of smoothing, whereby the so-called "checkerboard effect" arises. For noisy data f it is necessary to differentiate between discrete and continuous modeling. In the discrete case, i.e. if K is a matrix and u is a vector the existence of a minimum can be guaranteed since the smallest singular value is bounded by a positive value. Hence, the vectors are bounded during the iteration and convergence is ensured. However, if K is a general continuous operator

Bregman-EM-TV Methods

239

the convergence is not only difficult to prove, but even a divergence of the EM algorithm is possible. Again the reason is the ill-posedness of the integral equation (1), which transfers to problem (11). This aspect can be taken as a lack of additional a-priori knowledge about the unknown u resulting from R(u) = 0. The EM algorithm converges to a minimizer if it exists. Consequently, in the continuous case it is essential to ensure consistence of the given data to prevent divergence of the EM algorithm. As described in [13], the EM iterates show the following typical behavior for ill-posed problems. The (metric) distance between the iterates and the solution decreases initially before it increases as the noise is amplified during the iteration process. This issue might be regulated by using appropriate stopping rules to obtain reasonable results. In [13] it is shown that certain stopping rules indeed allow stable approximations. Ways to improve reconstruction results are TV or Bregman-TV regularization techniques that we will consider in the following section. 2.2

Reconstruction Method: EM-TV Algorithm

The EM or Richardson/Lucy algorithm is currently the standard iterative reconstruction method for deconvolution problems with Poisson noise based on the linear equation (1). However, with the assumption R(u) = 0, no a-priori knowledge about the expected solution is taken into account, i.e. different images have the same a-priori probability. Especially in case of measurements with low SNR the multiplicative fixed point iteration (12) delivers unsatisfactory and noisy results even with early termination. Therefore we propose to integrate nonlinear variational methods into the reconstruction process to make an efficient use of a-priori information and to obtain improved results. An interesting approach to improve the reconstruction is the EM-TV algorithm. In the classical EM algorithm, the negative log likelihood functional (11) is minimized. We modify the functional by adding a weighted TV term [2],   min . (13) (Ku − f log Ku) dμ + α|u|BV u∈BV (Ω) u≥0

Ω

This is exactly (6) with TV as regularization functional R. That means images with smaller total variation are preferred in the minimization (have higher prior probability). BV (Ω) is a popular function space in image processing since it can represent discontinuous functions. By minimizing TV the latter are even preferred [16, 17]. Hence, expected reconstructions are cartoon-like images. Obviously, such an approach cannot be used for studying very small structures in an object, but it is perfect for segmenting different cell structures and analyzing them quantitatively. For the solution of (13), we propose a forward-backward splitting algorithm, which can be realized by alternating classical EM steps with almost standard TV minimization steps as encountered in image denoising. The latter is solved by using duality [18] obtaining a robust and efficient algorithm. For designing the proposed alternating algorithm, we consider the first order optimality condition

240

C. Brune, A. Sawatzky, and M. Burger

of (13). Due to the total variation, this variational problem is not differentiable in the usual sense. But the latter is convex since TV is convex and since we can extend the data fidelity term to a Kullback-Leibler functional, cf. [19], without affecting the stationary points. For such problems powerful methods from convex analysis are available, e.g. a generalized derivative called the subdifferential [20], denoted by ∂. This generalized notion of gradients and the Karush-Kuhn-Tucker (KKT) conditions [20, Theorem 2.1.4] yield the existence of a Lagrange multiplier λ ≥ 0 such that   ⎧ ⎫ f ⎨ 0 ∈ K ∗1 − K ∗ + α ∂|u|BV − λ ⎬ Ku . (14) ⎩ ⎭ 0 = λu By multiplying (14) with u we can eliminate the Lagrange multiplier and derive the following semi-implicit iteration scheme   K∗ f + α ˜ uk pk+1 = 0 (15) uk+1 − uk ∗ K 1 Kuk ˜ := Kα∗ 1 . Interestingly, the second term within with pk+1 ∈ ∂|uk+1 |BV and α this iteration scheme is the EM step in (12). Consequently, method (15) solving variational problem (13), can be realized as a nested two step iteration,   ⎧ ⎫ ∗ f ⎨u 1 = u K ⎬ (EM step) k k+ 2 K ∗ 1 Kuk . (16) ⎩ ⎭ uk+1 = uk+ 12 − α ˜ uk pk+1 (TV step) Thus, we alternate an EM step with a TV correction step. The complex second half step from uk+ 12 to uk+1 can be realized by solving the following variational problem,    (u − uk+ 12 )2 1 . (17) uk+1 = arg min +α ˜ |u|BV 2 Ω uk u∈BV (Ω) Inspecting the first order optimality condition confirms the equivalence of this minimization with the TV correction step in (16). Problem (17) is just a modified version of the Rudin-Osher-Fatemi (ROF) model, with weight u1k in the fidelity term. This analogy creates the opportunity to carry over efficient numerical schemes known for the ROF-model. For the solution of (17) we use the exact definition of TV (7) with dual variable g and derive an iteration scheme for the quadratic dual problem similar to [18]. The resulting algorithm reads as follows: We initialize the dual variable g 0 with 0 (or the resulting g from the previous TV correction step) and for any n ≥ 0 we compute the update g n+1 =

α uk divg n − uk+ 12 ) g n + τ ∇(˜ 1 + τ |∇(˜ α uk divg n − uk+ 12 )|

,

0 < τ <

1 , 4α ˜ uk

with the constrained damping parameter τ to ensure stability and convergence of the algorithm. For a detailed analytical examination of EM-TV we refer to [21].

Bregman-EM-TV Methods

2.3

241

Extension to Inverse Scale Space: Bregman-EM-TV

The EM-TV algorithm (16) we presented solves the problem (13) and delivers cartoon-reconstructions with sharp edges due to TV regularization. However, the realization of TV steps via the weighted ROF-models (17) has the drawback that reconstructed images suffer from loosing contrast. Thus, we propose to extend (13) and therewith EM-TV by iterative regularization to a simultaneous contrast correction. More precisely, we perform a contrast enhancement by inverse scale space methods and by using the Bregman iteration. These techniques have been derived in [8], with a detailed analysis for Gaussian-type problems (5), and have been generalized in [9, 10]. Following these methods, an iterative refinement is realized by a sequence of modified EM-TV problems based on (13). The inverse scale space methods concerning TV, derived in [8], follow the concept of iterative regularization by the Bregman distance [22]. In case of the Poisson-model the method initially starts with a simple EM-TV algorithm, i.e. it consists in computing a minimizer u1 of (13). Then, updates are determined successively by considering variational problems with a shifted TV, namely (8), where pl is an element of the subgradient of the total variation in ul . The Bregman distance concerning TV is defined in (9). The introduction of this definition allows to characterize the sequence of modified variational problems (8) by addition of constant terms as   pl l+1 l u = arg min (Ku − f log Ku) dμ + α D|·|BV (u, u ) . (18) u∈BV (Ω)

Ω

Thus, the first iterate u1 can also be realized by the variational problem (18), if p u0 is constant and p0 := 0 ∈ ∂|u0 |BV . The Bregman distance D|·| does not BV represent a distance in the common (metric) sense, since D is not symmetric in general and the triangle inequality does not hold. Though, compared to (8), the p formulation in (18) offers the advantage that D|·| is a distance measure with BV p D|·| (u, u ˜) ≥ 0 BV

p and D|·| (u, u ˜) = 0 for u = u˜ . BV

Besides, the Bregman distance is convex in the first argument because | · |BV is convex. In general, i.e. for any convex functional J (see e.g. [10]), the Bregman distance can be interpreted as the difference between J(·) in u and the Taylor linearization of J around u ˜ if, in addition, J is continuously differentiable. Before deriving a two-step iteration corresponding to (16) we will motivate the contrast enhancement by iterative regularization in (18). The TV regularization in (13) prefers functions with only few oscillations. The iterative Bregman regularization has the advantage that, with ul as an approximation to the possible solution, additional information is available. The variational problem (18) can be interpreted as follows: search for a solution that matches the Poisson distributed data after applying K and simultaneous minimization of the residual of the Taylor approximation of | · |BV around ul . In the following we will see that this form of regularization does not change the position of gradients with respect

242

C. Brune, A. Sawatzky, and M. Burger

to the last computed EM-TV solution ul but that an increase of intensities is permitted. This leads to a noticeable contrast enhancement. For the derivation of a two-step iteration we consider the first order optimality condition of the variational problem (8) resp. (18). Due to convexity of the Bregman distance in the first argument we can determine the subdifferential of (18). Analogous to the derivation of the EM-TV iteration the subdifferential of the log likelihood functional can be expressed by the Fr´echet derivative in (14). Hence, the optimality condition is given by   f ∗ ∗ 0 ∈ K 1 − K + α ( ∂|ul+1 |BV − pl ), pl ∈ ∂|ul |BV . (19) Kul+1 For u0 constant and p0 := 0 ∈ ∂|u0 |BV this delivers a well defined update of the iterates pl ,    K∗ 1 f l+1 l p 1− ∗ ∈ ∂|ul+1 |BV , := p − α ˜ K 1 Kul+1 where α ˜ := Kα∗ 1 results from an operator normalization. Analogous to EM-TV we can apply the idea of the nested iteration (16) in every refinement step, l = 1, 2, · · · . For the solution of (18) condition (19) yields a strategy consisting of an EM-step ul+1 followed by solving the adapted weighted ROF-problem k+ 1 2

ul+1 k+1

⎫ ⎧ ⎬ ⎨ 1  (u − ul+11 )2 k+ 2 l = arg min + α ˜ ( |u| − p , u ) . BV ⎭ ul+1 u∈BV (Ω) ⎩ 2 Ω k

(20)

Following [8,9,10], we provide an opportunity to transfer the shift-term pl , u to the data-fidelity term. This approach facilitates the implementation of contrast enhancement with Bregman distance via a slightly modified EM-TV algorithm. With the scaling v l := α ˜ pl and (19) we obtain the following update formula    K∗ f l+1 l , v0 = 0 . =v − 1− ∗ (21) v K 1 Kul+1 Using this scaled update we can rewrite the second step (20) to ⎧ ⎫ ⎨ 1  (u − ul+11 )2 − 2uul+1 ⎬ vl k k+ 2 = arg min + α ˜ |u| . ul+1 BV k+1 ⎭ ul+1 u∈BV (Ω) ⎩ 2 Ω k Note that l+1 l+1 l 2 l+1 2 l 2 l+1 l+1 l l )2 − 2uul+1 (u − ul+1 k v = (u − (uk+ 1 + uk v )) + (uk ) (v ) − 2uk+ 1 uk v , k+ 1 2

2

holds, where the last two terms are independent of u. Hence (20) ⎧ l 2 ⎨ 1  (u − (ul+11 + ul+1 k v )) k+ 2 = arg min + α ˜ |u|BV ul+1 k+1 ul+1 u∈BV (Ω) ⎩ 2 Ω k

2

simplifies to ⎫ ⎬ , (22) ⎭

Bregman-EM-TV Methods

243

i.e. the second step (20) can be realized by a slight modification of the TV step introduced in (17). Obviously, the efficient numerical implementation of the weighted ROF-problem in Section 2.2 using the exact definition of TV and duality strategies can be applied in complete analogy to (22). The update variable v in (21) is an error function with reference to the optimality condition of the unregularized log-likelihood functional (11). In every refinement step of the Bregman iteration v l+1 differs from v l by the current error in the optimality condition (11). Within the TV-step (22) one observes that an iterative regularization with the Bregman distance leads to contrast enhancement. Instead of fitting to the EM solution ul+1 in the weighted norm, we use a function in the fidelity k+ 1 2

term whose intensities are increased by the error function v l . Resulting from the idea of adaptive regularization v l is weighted by ul+1 k , too. As usual for iterative methods the described reconstruction method by iterative regularization needs a stopping criterion. The latter should stop at an iteration offering a solution that approximates the true image as good as possible. This is necessary to prevent that too much noise arises by the inverse scale space strategy. In the case of Gaussian noise, the discrepancy principle is a reasonable stopping criterion, i.e.  the procedure would stop if the residual Kul − f 2 reaches the variance of the noise. In the case of Poisson noise, however, it makes sense to stop the Bregman iteration if the Kullback-Leibler distances of Kul and the given data f reach the noise level. For synthetic data the noise level is given by the KL distance of Ku∗ and f , where u∗ denotes the true, noise-free image. For experimental data it is necessary to find a suitable estimate for the noise level from counts.

3

Results

In recent years revolutionary imaging techniques have been developed in light microscopy with enormous importance for biological and material sciences or medicine. For a couple of decades the technology of light microscopy has been considered to be exhausted, as the resolution is basically limited by Abbe’s law for diffraction of light. By developing stimulated emission depletion (STED)and 4Pi-microscopy now resolutions are achieved that are way beyond these

(a)

(b)

(c)

(d)

(e)

Fig. 1. Synthetic data concerning different PSFs: (a) true image; (b) Gaussian PSF; (c) is convolved with Gaussian PSF and Poisson noise; (d) PSF appearing in 4Pi microscopy; and (e) is convolved with 4Pi PSF and Poisson noise

244

C. Brune, A. Sawatzky, and M. Burger

(a)

(b)

(c)

(f)

(e)

(d)

(g)

(h)

Fig. 2. Synthetic data: (a) raw data; (b) EM reconstruction, 20 its, KL-distance: 3.20; (c) EM-TV, α = 0.04, KL-distance: 2.43; (d) Bregman-EM-TV, α = 0.1, after 4 updates, KL-distance: 1.43; (e) true image; (f)-(h) horizontal slices EM, EM-TV and Bregman-EM-TV compared to true image slice

(a)

(b)

(c)

(d)

(e)

(f)

Fig. 3. Experimental data: (a) Protein Bruchpilot in active zones of neuromuscular synapses in larval Drosophila; (b) EM-TV; (c) Bregman-EM-TV; (d) Protein Syntaxin in cell membrane, fixed mamalian (PC12) cell; (e) EM-TV; and (f) Bregman-EM-TV

Bregman-EM-TV Methods

245

diffraction barrier [23,24]. To get an impression of nanoscopic images blurred by different convolution kernels (PSFs), we refer to Figure 1. Figure 2 illustrates our techniques at a simple synthetic object. With EM-TV (see 2(c) and 2(g)) we get rid of noise and oscillations, but we are not able to separate the objects sufficiently. Using Bregman-EM-TV a considerable improvement resulting from contrast enhancement can be achieved. This aspect is underlined by the values of the KL-distance for the different reconstructions. Figure 3, (a)-(c) demonstrate the protein Bruchpilot [25] and its EM-TV and Bregman-EM-TV reconstruction. Particularly, the latter delivers well separated object segments and a high contrast level. In Figure 3, (d)-(f) we illustrate our techniques by reconstructing Syntaxin [26], a membrane integrated protein participating in exocytosis. Here, the contrast enhancing property of Bregman-EM-TV is observable as well, compared to EM-TV. It is possible to preserve fine structures in the image.

4

Conclusions

We have derived reconstruction methods for inverse problems with Poisson noise. Particularly, we concentrated on deblurring problems in nanoscopic imaging, although the proposed methods can easily be adapted to other imaging tasks, i.e. medical imaging (PET, [27]). Motivated by a statistical modeling we developed a robust EM-TV algorithm that incorporates a-priori knowledge into the reconstruction process. By combining EM with simultaneous TV regularization we can reconstruct cartoon-images with sharp edges, that deliver a reasonable basis for quantitative investigations. To overcome the problem of contrast reduction, we extended the reconstruction to Bregman iterations and inverse scale space methods. We applied the proposed methods to optical nanoscopy and pointed out their improvements in comparison to standard reconstruction techniques. Acknowledgments. This work has been supported by the German Federal Ministry of Education and Research through the project INVERS. C.B. acknowledges further support by the Deutsche Telekom Foundation, and M.B. by the German Science Foundation DFG through the project "Regularisierung mit Singulären Energien". The authors thank Dr. Katrin Willig and Dr. Andreas Schönle (MPI Biophysical Chemistry, Göttingen) for providing experimental data and stimulating discussions.

References 1. Bertero, M., Lantéri, H., Zanni, L.: Iterative image reconstruction: a point of view. In: Mathematical Methods in Biomedical Imaging and Intensity-Modulated Radiation Therapy (IMRT). CRM series, vol. 8 (2008) 2. Rudin, L.I., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Physica D 60, 259–268 (1992) 3. Le, T., Chartrand, R., Asaki, T.J.: A variational approach to reconstructing images corrupted by Poisson noise. J. Math. Imaging Vision 27, 257–263 (2007) 4. Shepp, L.A., Vardi, Y.: Maximum likelihood reconstruction for emission tomography. IEEE Transactions on Medical Imaging 1(2), 113–122 (1982)

246

C. Brune, A. Sawatzky, and M. Burger

5. Richardson, W.H.: Bayesian-based iterative method of image restoration. J. Opt. Soc. Am. 62, 55–59 (1972) 6. Lucy, L.B.: An iterative technique for the rectification of observed distributions. The Astronomical Journal 79, 745–754 (1974) 7. Acar, R., Vogel, C.R.: Analysis of bounded variation penalty methods for ill-posed problems. Inverse Problems 10, 1217–1229 (1994) 8. Osher, S., Burger, M., Goldfarb, D., Xu, J., Yin, W.: An iterative regularization method for total variation based image restoration. Multiscale Modelling and Simulation 4, 460–489 (2005) 9. Burger, M., Gilboa, G., Osher, S., Xu, J.: Nonlinear inverse scale space methods. Commun. Math. Sci. 4(1), 179–212 (2006) 10. Burger, M., Frick, K., Osher, S., Scherzer, O.: Inverse total variation flow. SIAM Multiscale Modelling and Simulation 6(2), 366–395 (2007) 11. Dempster, A.P., Laird, N.M., Rubin, D.B.: Maximum Likelihood from Incomplete Data via the EM Algorithm. J. of the Royal Statistical Society, B 39, 1–38 (1977) 12. Natterer, F., Wübbeling, F.: Mathematical methods in image reconstruction. SIAM Monographs on Mathematical Modeling and Computation (2001) 13. Resmerita, E., et al.: The expectation-maximization algorithm for ill-posed integral equations: a convergence analysis. Inverse Problems 23, 2575–2588 (2007) 14. Vardi, Y., Shepp, L.A., Kaufman, L.: A statistical model for positron emission tomography. J. of the American Statistical Association 80(389), 8–20 (1985) 15. Iusem, A.N.: Convergence analysis for a multiplicatively relaxed EM algorithm. Mathematical Methods in the Applied Sciences 14, 573–593 (1991) 16. Evans, L.C., Gariepy, R.F.: Measure theory and fine properties of functions. Studies in Advanced Mathematics. CRC Press, Boca Raton (1992) 17. Giusti, E.: Minimal surfaces and functions of bounded variation. Birkhäuser, Basel (1984) 18. Chambolle, A.: An algorithm for total variation minimization and applications. J. of Mathematical Imaging and Vision 20, 89–97 (2004) 19. Resmerita, E., Anderssen, S.: Joint additive Kullback-Leibler residual minimization and regularization for linear inverse problems. Math. Meth. Appl. Sci. 30, 1527– 1544 (2007) 20. Hiriart-Urruty, J.B., Lemaréchal, C.: Convex Analysis and Minimization Algorithms I. Grundlehren der mathematischen Wissenschaften [Fundamental Principles of Mathematical Sciences], vol. 305. Springer, Heidelberg (1993) 21. Brune, C., Sawatzky, A., Wübbeling, F., Kösters, T., Burger, M.: EM-TV methods for inverse problems with poisson noise (in preparation) (2009) 22. Bregman, L.M.: The relaxation method for finding the common point of convex sets and its application to the solution of problems in convex programming. USSR Comp. Math. and Math. Phys. 7, 200–217 (1967) 23. Klar, T.A., et al.: Fluorescence microscopy with diffraction resolution barrier broken by stimulated emission. PNAS 97, 8206–8210 (2000) 24. Hell, S., Schönle, A.: Nanoscale resolution in far-field fluorescence microscopy. In: Hawkes, P.W., Spence, J.C.H. (eds.) Science of Microscopy. Springer, Heidelberg (2006) 25. Kittel, J., et al.: Bruchpilot promotes active zone assembly, Ca2+ channel clustering, and vesicle release. Science 312, 1051–1054 (2006) 26. Willig, K.I., Harke, B., Medda, R., Hell, S.W.: STED microscopy with continuous wave beams. Nature Meth. 4(11), 915–918 (2007) 27. Sawatzky, A., Brune, C., Wübbeling, F., Kösters, T., Schäfers, K.: Accurate EMTV algorithm in PET with low SNR. In: IEEE Nucl. Sci. Symp. (2008)

PDE-Driven Adaptive Morphology for Matrix Fields Bernhard Burgeth, Michael Breuß, Luis Pizarro, and Joachim Weickert Mathematical Image Analysis Group, Faculty of Mathematics and Computer Science, Saarland University, 66041 Saarbrücken, Germany {burgeth,breuss,pizarro,weickert}@mia.uni-saarland.de http://www.mia.uni-saarland.de

Abstract. Matrix fields are important in many applications since they are the adequate means to describe anisotropic behaviour in image processing models and physical measurements. A prominent example is diffusion tensor magnetic resonance imaging (DT-MRI) which is a medical imaging technique useful for analysing the fibre structure in the brain. Recently, morphological partial differential equations (PDEs) for dilation and erosion known for grey scale images have been extended to three dimensional fields of symmetric positive definite matrices. In this article we propose a novel method to incorporate adaptivity into the matrix-valued, PDE-driven dilation process. The approach uses a structure tensor concept for matrix data to steer anisotropic morphological evolution in a way that enhances and completes line-like structures in matrix fields. Numerical experiments performed on synthetic and realworld data confirm the gap-closing and line-completing qualities of the proposed method.

1

Introduction

Initiated in the sixties by the pioneering research of Serra and Matheron on binary morphology [23, 31], this branch of image processing has developed into a rich field of research. Numerous monographs e.g. [17, 24, 32, 33, 34] and proceedings, e.g. [16,18,22] bear witness to the variety in mathematical morphology. The building blocks of morphological operations are dilation and erosion. These are usually realised by algebraic set operations involving a probing set, a so-called structuring element, e.g. [34] for details. An alternative approach to dilation is given [1] by the nonlinear partial differential equation (PDE)  ∂t u = ∇u = |∂x u|2 + |∂y u|2 (1) with initial condition u(x, y, 0) = f (x, y). The equation mimics the dilation of a grey scale image f with respect to a ball-shaped structuring element of growing radius t. PDEs of this type using a continuous size parameter t for the structuring element give rise to continuous-scale morphology [1,2,6,29,35]. Equation (1) has been extended in two ways: X.-C. Tai et al. (Eds.): SSVM 2009, LNCS 5567, pp. 247–258, 2009. c Springer-Verlag Berlin Heidelberg 2009 

248

B. Burgeth et al.

Firstly, in [5] adaptivity has been incorporated by introducing a speed function β = β(u) into (1), ∂t u = β(u) · ∇u

(2)

Earlier attempts towards adaptivity have been made in [20, 26] where a local switch between dilation and erosion with a nonadaptive structuring element leads to a so-called morphological shock filter, and in [21] introducing morphological amoebae described in a set-theoretic framework. Secondly, in [8] scalar continuous morphology has been extended to a PDEdriven morphology of matrix-valued images, matrix fields for short. Matrix fields have received increasing attention over the recent years since they are the appropriate data type to describe anisotropy in models or measurements of physical quantities. For instance, diffusion tensor magnetic resonance imaging (DT-MRI) became a valuable tool in medicine for in vivo diagnosis. It results in three dimensional tensor fields that describe the diffusive properties of water molecules, and as such the structure of the tissue under examination. The goal of this article is to introduce adaptivity into morphology for matrix fields. As it turns out it is advantageous to start for this generalisation from a scalar adaptive formulation for d-dimensional data u in form of the PDE ∂t u = M (u) · ∇u

(3)

with ∇u as a column vector and a data dependent, symmetric, positive semidefinite d × d-matrix M = M (u)rather  than from (2). For example, for greyvalue ab images (d = 2) one has M = and (3) turns into bc  ∂t u =

(a∂x u + b∂y u)2 + (b∂x u + c∂y u)2

(4)

→ M (x, y) transforms a sphere cenAn application of the mapping (x, y)  tered around the origin into an ellipse. So, in fact, (3) describes a dilation with an ellipsoidal structuring element. The matrix M must contain directional information of the evolving u, and thus it may be derived from the so-called structure tensor. The structure tensor, going back to [14, 27, 4], is a classic tool in image processing to extract directional information from an image. It is given by      Sρ (u(x)) := Gρ ∗ ∇u(x) · (∇u(x)) = Gρ ∗ ∂xi u(x) · ∂xj u(x) i,j=1,...,d (5) Here Gρ ∗ indicates a convolution with a Gaussian of standard deviation ρ, however, more general averaging procedures can be used. For more details the reader is referred to [3] and the literature cited there. We will make use of the extended structure tensor concept for matrix fields as proposed in [10]. There it was used to steer an coherence-enhancing diffusion process for matrix fields, an anisotropic filtering process that has been proposed for scalar and colour images in [36, 37].

PDE-Driven Adaptive Morphology for Matrix Fields

249

In [38, 7, 13] Di Zenzo‘s approach [12] to construct a structure tensor for multi-channel images has been extended to matrix fields yielding a standard structure tensor (using the notation of forthcoming Section 2): Jρ (U (x)) :=  m i,j=1 Sρ (Ui,j (x)) This construction has been refined to a customisable structure tensor in [30]. The article has the following structure: We will briefly convey in Section 2 basic notions of matrix analysis needed to establish a matrix-valued PDE for an adaptively steered morphological dilation process. This includes a short account of the construction of an extended structure tensor for matrix fields. In Section 3 we introduce the steering tensor that guides the dilation process adaptively. We explain how the numerical scheme of Rouy and Tourin is generalised to the matrix valued setting in Section 4. We compare in our experiments adaptive and isotropic dilation with CED-diffusion when applied to synthetic matrix fields and real DT-MRI data sets. We report on this comparison of the results in Section 5. The remarks in Section 6 conclude this article.

2

Matrix Analysis and an Extended Structure Tensor Concept

This section contains the key definitions for the formulation of matrix-valued PDEs. For a more detailed exposition the reader is referred to [9]. A matrix field is considered as a mapping U : Ω ⊂ Rd −→ Symm (R) from a d-dimensional image domain into the set of symmetric m × m-matrices with real entries, U (x) = (Up,q (x))p,q=1,...,m . The set of positive (semi-) definite matrices, + denoted by Sym++ m (R) (resp., Symm (R)), consists of all symmetric matrices A  with v, Av := v Av > 0 (resp., ≥ 0) for v ∈ Rm \ {0} . This set is of special interest since DT-MRI produces data with this property. Note that at each point x the matrix U (x) of a field of symmetric matrices can be diagonalised yielding U (x) = V (x) D(x)V (x), where V (x) is a orthogonal matrix, while D(x) is a diagonal matrix. In the sequel we will denote m × m - diagonal matrices with entries λ1 , . . . , λm ∈ R from left to right simply by diag(λi ). The extension of a function h : R −→ R to Symm (R) is standard [19]: With a slight abuse of notation we set h(U ) := V  diag(h(λ1 ), . . . , h(λm ))V ∈ Sym+ m (R), h denoting now a function acting on matrices as well. Specifying h(s) = |s|, s ∈ R as the absolut value function leads to the absolut value |A| ∈ Sym+ m (R) of a matrix A. It is natural to define the partial derivative for matrix fields componentwise: ∂ ω U = (∂ω Up,q )p,q=1,...,m

(6)

where ω ∈ {t, x1 , . . . , xd }, that is, ∂ ω stands for a spatial or temporal derivative. Viewing a matrix as a tensor (of second order), its gradient would be a third order tensor according to the rules of differential geometry. However, we adopt a more operator-algebraic point of view by defining the generalised gradient ∇U (x) at a voxel x = (x1 , . . . , xd ) by ∇U (x) := (∂ x1 U (x), . . . , ∂ xd U (x))

(7)

250

B. Burgeth et al.

which is an element of (Symm (R))d , in close analogy tothe scalar setting where ∇u(x) ∈ Rd . For W ∈ (Symm (R))d we set |W |p := p |W1 |p + · · · + |Wd |p for 0 < p < +∞. It results in a positive semidefinite matrix from Sym+ m (R), the direct counterpart of a nonnegative real number as the length of a vector in Rd . There will be the need for a symmetric multiplication of symmetric matrices. We opt for the so-called Jordan product A •J B := 12 (AB + BA) . It produces a symmetric matrix, and it is commutative but neither associative nor distributive. Furthermore, for later use in numerical schemes we have to clarify the notion of maximum and minimum of two symmetric matrices A, B. In direct anaology with relations known to be valid for real numbers one defines [8]: max(A, B) =

1 1 (A + B + |A − B|) and min(A, B) = (A + B − |A − B|) (8) 2 2

where |F | stands for the absolut value of the matrix F . With this at our disposal we formulate the matrix-valued counterpart of (3) as ∂ t U = |M (U ) • ∇U |2

(9)

with an initial matrix field F (x) = U (x, 0). Here M (U ) denotes a symmetric md × md-block matrix with d2 blocks of size m × m that is multiplied block-wise with ∇U employing the symmetrised product "•". Note that | · |2 stands for the length of M (U ) • ∇U in the matrix valued sense. The construction of M (U ) is detailed in Section 3 and relies on the so-called full structure tensor. The full structure tensor S L for matrix fields as defined in [10] reads      S L (U ) := Gρ ∗ ∇U ·(∇U ) = Gρ ∗ ∂ xi U · ∂ xj U i,j=1,...,d (10) with Gρ ∗ indicating a convolution with a Gaussian of standard deviation ρ. S L (U (x)) is a symmetric md × md-block matrix with d2 blocks of size m × m, S L (U (x)) ∈ Symd (Symm (IR)) = Symmd (IR). Typically for the 3D medical DT-MRI data one has d = 3  and m = 3, yielding a 9 × 9-matrix S L . It can md  be diagonalised as S L (U ) = k=1 λk wk wk with real eigenvalues λk (w.l.o.g. arranged in decreasing order) and an orthonormal basis {wk }k=1,...,md of IRmd . In order to extract useful d-dimensional directional information S L (U ) ∈ Symmd (IR) is reduced to a structure tensor S(U ) ∈ Symd (IR) in a generalised projection step [10] using the block operator matrix TrA := diag(trA , . . . , trA ) containing the trace operation. We set Tr := TrIm where Im denotes the m × m unit matrix. This operator matrix acts on elements of the space (Symm (IR))d as well as on block matrices via formal block-wise matrix multiplication, ⎛ ⎞⎛ ⎞ ⎛ ⎞ trA · · · 0 M11 · · · M1d trA (M11 ) · · · trA (M1d ) . . . . . . .. .. .. ⎝ .. . . . .. ⎠ ⎝ .. . . . .. ⎠ = ⎝ ⎠, (11) . 0 · · · trA trA (Md1 ) · · · trA (Mdd ) Md1 · · · Mdd provided that the square blocks Mij have the same size as A. The projection that is conveyed by the reduction process condenses the directional information contained in S L (U ), for a more detailed reasoning we must refer the reader to [10]

PDE-Driven Adaptive Morphology for Matrix Fields

251

for the sake of brevity. The reduction operation is accompanied by an extension operation: The Im -extension is the mapping from Symd (IR) to Symmd (IR) conveyed by the Kronecker product ⊗ : ⎛ ⎞ ⎛ ⎞ ⎛ ⎞ ⎛ ⎞ v11 · · · v1d v11 · · · v1d Im · · · Im v11 Im · · · v1d Im ⎜ .. . . .. ⎟ ⎜ .. . . .. ⎟ ⎜ .. . . .. ⎟ ⎜ .. .. ⎟(12) .. −→⎝ . ⎝ . . . ⎠ . . ⎠⊗⎝ . . . ⎠:=⎝ . . . ⎠ vd1 · vdd

vd1 · · · vdd

Im · · · Im

vd1 Im · · · vdd Im

This resizing step renders a proper matrix-vector multiplication with the large generalised gradient (∇U (x)) possible. By specifying the matrix A in (11) one may invoke a priori knowledge into the direction estimation [10]. The research on these structure-tensor concepts has been initiated by [38, 7]. The approaches to matrix field regularisation suggested in [11] are based on differential geometric considerations. Comprehensive survey articles on the analysis of matrix fields using various techniques can be found in [39].

3

Steering Matrix M (U ) for Matrix Fields

With this notions we are in the position to propose the steering matrix M in the adaptive dilation process for matrix fields. We proceed in four steps: 1. The matrix field IRd  x  → U (x) provides us with a module field of generalised gradients ∇U (x) from which we construct the generalised structure tensor S L (U (x)) possibly with a certain integration scale ρ. This step corresponds exactly to the scalar case. 2. We infer d-dimensional directional information by reducing S L (U (x)) with trA by means of the block operator matrix TrA leading to a symmetric d × dmatrix S, for example S = Jρ if A = Im ,   (13) S(x) := TrA S L (U (x)) 3. The symmetric d × d-matrix S is spectrally decomposed, and the following mapping is applied:  Rd+ −→ Rd H: , (14) c (λ1 , . . . , λd )  −→ λ1 +···+λ (λd , λd−1 , . . . , Kc · λ1 ) d with constants c, K > 0. H applied to S yields the steering matrix M , M := H(S)

(15)

Observe that the ellipsoid associated with the matrix M is flipped if compared with S and, depending on the choice of K, more excentric than the one accompanying S. 4. Finally we enlarge the d × d-matrix M to a md × md-matrix M by the extension operation: ⎛ ⎞ Im · · · Im ⎜ ⎟ M = M ⊗ ⎝ ... . . . ... ⎠ (16) Im · · · Im

252

4

B. Burgeth et al.

Matrix-Valued Numerical Schemes

In the context of PDE-based mathematical morphology, first-order finite difference methods such as the Osher-Sethian scheme [25] and the Rouy-Tourin method [28] are reasonable choices for solving the scalar PDE (4). We choose the latter in our experiments. The variant we present for the sake of brevity in its two-dimensional form reads      1  x n  2 1 n+1 n x n ui,j = ui,j + τ max max −D− ui,j , 0 , max D+ ui,j , 0 hx hx   1/2   1  y n  2 1 y n + max max −D− ui,j , 0 , max D+ ui,j , 0 (17) hy hy In the latter formulation we employ the notation unij as the grey value of the image u at the pixel centred at (ihx , jhy ) ∈ R2 at the time-level nτ of the

Fig. 1. (a) Top left: 2D slice of original 3D matrix field. (b) Top right: Adaptive dilation with of the original data with K = 25, ρ = 1 after t = 0.3. (c) Bottom left: Standard PDE-based dilation mimicing a ball-shaped structuring element after t = 1. (d) Bottom right: CED-filtering with ρ = 4 after t = 10.

PDE-Driven Adaptive Morphology for Matrix Fields

253

evolution. Additionally we use standard abbreviations for forward and backward x n x n difference operators, i.e., D+ ui,j := uni+1,j −uni,j and D− ui,j := uni,j −uni−1,j . and spatial grid size hx , hy . This scheme approximates, in the pixel (ihx , jhy )     1  x n  1 x n max −D− ui,j , 0 , max D+ ui,j , 0 (18) ux ≈ max hx hx  uy ≈ max



  1  y n  1 y n max −D− ui,j , 0 , max D+ ui,j , 0 hy hy

(19)

Using this approximations, we modify the original Rouy-Tourin scheme (17) in an obvious manner to obtain a numerical scheme for the adaptive version of the PDE-based dilation (3). The extension to higher dimensions poses no problem. Since linear combinations and elementary functions such as the square, squareroot or absolute value function for matrix fields are now at our disposal it is straightforward to define one sided differences in x-direction for 2D matrix fields of m × m-matrices: x n D+ U (i, j) := U n ((i + 1)hx , jhy ) − U n (ihx , jhy ) ∈ Symm (R)

(20)

x n U (i, j) := U n (ihx , jhy ) − U n ((i − 1)hx , jhy ) ∈ Symm (R) D−

(21)

Fig. 2. (a) Left: 2D slice of 3D DT-MRI data set. (b) Right: Adaptive dilation of the original data with K = 10, ρ = 1, t = 0.5.

254

B. Burgeth et al.

In order to avoid confusion with the subscript notation for matrix components we used the notation U (i, j) to indicate the (matrix-) value of the matrix field evaluated at the voxel centred at (ihx , jhy ) ∈ R2 . The y-direction (and z-direction in 3D) is treated accordingly. The notion of supremum and infimum of two matrices – as needed in a matrix variant of Rouy-Tourin – has been provided by (8). Having these generalisations at our disposal a modified, adaptive version of the Rouy-Tourin scheme is available now in the setting of matrix fields simply by replacing grey values unij by matrices U n (i, j).

5

Experiments

The matrix data are visualised as an ellipsoid in each voxel via the level sets of quadratic form {v ∈ R2 v : v  U −2 (i, j)v = const.} associated with the matrix

Fig. 3. (a) Top left: Enlarged section of the original data of figure 2 showing the genu area. (b) Top right: Adaptive dilation of the original data with K = 10, ρ = 1, t = 0.5. (c) Bottom left: Standard PDE-based dilation mimicing a ball-shaped structuring element with t = 0.5. (d) Bottom right: CED-filtering with ρ = 1 after t = 0.5.

PDE-Driven Adaptive Morphology for Matrix Fields

255

Fig. 4. (a) Top left: Enlarged section of the original data of figure 2 showing the splenium area. (b) Top right: Adaptive dilation of the original data with K = 10, ρ = 1, t = 0.5. (c) Bottom left: Standard PDE-based dilation mimicing a ball-shaped structuring element with t = 0.5. (d) Bottom right: CED-filtering with ρ = 1 after t = 0.5.

U (i, j) ∈ Sym+ 3 (R) representing the matrix field at voxel (ihx , jhy ). By using U −2 the length of the semi-axes of the ellipsoid correspond directly with the three eigenvalues of the matrix. Changing the constant const. amounts to a mere scaling of the ellipsoids. Note that only positive definite matrices produce ellipsoids as level sets of its quadratic form. In all our experiments we compare the results of the proposed matrix-valued adaptive dilation with the isotropic dilation [8] , and with the matrix-valued coherence-enhancing diffusion from [10]. For the explicit numerical schemes we used a time step size of 0.1, grid size hx = hy = 1, and c = 0.01 · K in (14). Figure 1 shows a synthetic data set of size 32 × 32 representing an interrupted diagonal stripe built from cigar-shaped ellipsoids of equal size. All methods succeed to some degree to fill the gaps. In the case of the proposed adaptive dilation the gap is filled almost completely with tensors comparable in size with the original ones while the width of the stripe is not altered at all. However, the numerical scheme has a slight bias towards the directions of the coordinate system entailing in the appearance of mild artefacts. Standard dilation fills the gap basically as a side effect of the isotropic dilation process which leads also to a considerable widening of the ribbon-like structure. CED for matrix fields produces indeed small cigar-shaped ellipsoids at the location of the gap. But the process is considerably slower than any of the dilation processes

256

B. Burgeth et al.

and the neighbouring ellipsoids become smaller due to the property of mass conservation. Additionally an undesirable widening of the stripe is observed. We also tested the proposed method on a real DT-MRI data set of a human head consisting of a 128 × 128 × 38-field of positive definite matrices. Figure 2 shows the lateral ventricals in a 40 × 55 2D section before and after applying adaptive dilation with speed parameter K = 10, integration scale ρ = 1 and stopping time t = 0.5. For a better comparison we display two enlarged regions of interest in Figures 3 and 4, namely the genu and the splenium areas, resp.. We observe that adaptive dilation preserves the shape of the ventricles better than the isotropic dilation, while enhancing slightly the directional structure of the fibre tracts surrounding the ventricles. Due to measurement errors the fibre tracts are interrupted in the original Figures 3(a) and 4(a). These holes in the anisotropic regions (splenium) are quickly filled by the adaptive dilation while CED-filtering will take much longer to do so.

6

Conclusion

In this article we have presented a novel method for an adaptive, PDE-based dilation process in the setting of matrix fields. The evolution governed by a matrix-valued PDE is guided by a steering tensor, the construction of which relies on an extended structure tensor concept for matrix fields. A matrix-valued extension of the Rouy-Tourin-scheme that allows to include directional information is employed to solve the novel PDE. Experiments on positive semidefinite DT-MRI and synthetic data confirm that the novel adaptive dilation process displays line-enhancing and gap-closing qualities, and as such it is superior to standard isotropic dilation which extends structures in all directions. It is also a valuable alternative in terms of quality and speed to coherence-enhancing diffusion filtering for matrix fields, an anisotropic processes which aims at enhancing flow-like structures as well but may suffer from dissipative effects. Future research will concentrate on improving the numerical realisation of our adaptive dilation.

Acknowledgement The financial support of the German Academic Exchange Service (DAAD) for the third author is gratefully acknowledged.

References 1. Alvarez, L., Guichard, F., Lions, P.-L., Morel, J.-M.: Axioms and fundamental equations in image processing. Archive for Rational Mechanics and Analysis 123, 199–257 (1993) 2. Arehart, A.B., Vincent, L., Kimia, B.B.: Mathematical morphology: The Hamilton–Jacobi connection. In: Proc. Fourth International Conference on Computer Vision, Berlin, pp. 215–219. IEEE Computer Society Press, Los Alamitos (1993)

PDE-Driven Adaptive Morphology for Matrix Fields

257

3. Bigün, J.: Vision with Direction. Springer, Berlin (2006) 4. Bigün, J., Granlund, G.H., Wiklund, J.: Multidimensional orientation estimation with applications to texture analysis and optical flow. IEEE Transactions on Pattern Analysis and Machine Intelligence 13(8), 775–790 (1991) 5. Breuß, M., Burgeth, B., Weickert, J.: Anisotropic continuous-scale morphology. In: Martí, J., Benedí, J.M., Mendonça, A.M., Serrat, J. (eds.) IbPRIA 2007. LNCS, vol. 4478, pp. 515–522. Springer, Heidelberg (2007) 6. Brockett, R.W., Maragos, P.: Evolution equations for continuous-scale morphological filtering. IEEE Transactions on Signal Processing 42, 3377–3386 (1994) 7. Brox, T., Weickert, J., Burgeth, B., Mrázek, P.: Nonlinear structure tensors. Image and Vision Computing 24(1), 41–55 (2006) 8. Burgeth, B., Bruhn, A., Didas, S., Weickert, J., Welk, M.: Morphology for tensor data: Ordering versus PDE-based approach. Image and Vision Computing 25(4), 496–511 (2007) 9. Burgeth, B., Didas, S., Florack, L., Weickert, J.: A generic approach to diffusion filtering of matrix-fields. Computing 81, 179–197 (2007) 10. Burgeth, B., Didas, S., Weickert, J.: A general structure tensor concept and coherence-enhancing diffusion filtering for matrix fields. Technical Report 197, Department of Mathematics, Saarland University, Saarbrücken, Germany (July 2007); to appear in: Laidlaw, D., Weickert, J. (eds.): Visualization and Processing of Tensor Fields. Springer, Heidelberg (2009) 11. Chefd’Hotel, C., Tschumperlé, D., Deriche, R., Faugeras, O.: Constrained flows of matrix-valued functions: Application to diffusion tensor regularization. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002. LNCS, vol. 2350, pp. 251–265. Springer, Heidelberg (2002) 12. Di Zenzo, S.: A note on the gradient of a multi-image. Computer Vision, Graphics and Image Processing 33, 116–125 (1986) 13. Feddern, C., Weickert, J., Burgeth, B., Welk, M.: Curvature-driven PDE methods for matrix-valued images. International Journal of Computer Vision 69(1), 91–103 (2006) 14. Förstner, W., Gülch, E.: A fast operator for detection and precise location of distinct points, corners and centres of circular features. In: Proc. ISPRS Intercommission Conference on Fast Processing of Photogrammetric Data, Interlaken, Switzerland, June 1987, pp. 281–305 (1987) 15. Goutsias, J., Heijmans, H.J.A.M., Sivakumar, K.: Morphological operators for image sequences. Computer Vision and Image Understanding 62, 326–346 (1995) 16. Goutsias, J., Vincent, L., Bloomberg, D.S. (eds.): Mathematical Morphology and its Applications to Image and Signal Processing. Computational Imaging and Vision, vol. 18. Kluwer, Dordrecht (2000) 17. Heijmans, H.J.A.M.: Morphological Image Operators. Academic Press, Boston (1994) 18. Heijmans, H.J.A.M., Roerdink, J.B.T.M. (eds.): Mathematical Morphology and its Applications to Image and Signal Processing. Computational Imaging and Vision, vol. 12. Kluwer, Dordrecht (1998) 19. Horn, R.A., Johnson, C.R.: Matrix Analysis. Cambridge University Press, Cambridge (1990) 20. Kramer, H.P., Bruckner, J.B.: Iterations of a non-linear transformation for enhancement of digital images. Pattern Recognition 7, 53–58 (1975) 21. Lerallut, R., Decencière, E., Meyer, F.: Image filtering using morphological amoebas. Image and Vision Computing 25(4), 395–404 (2007)

258

B. Burgeth et al.

22. Louverdis, G., Vardavoulia, M.I., Andreadis, I., Tsalides, P.: A new approach to morphological color image processing. Pattern Recognition 35, 1733–1741 (2002) 23. Matheron, G.: Eléments pour une théorie des milieux poreux. Masson, Paris (1967) 24. Matheron, G.: Random Sets and Integral Geometry. Wiley, New York (1975) 25. Osher, S., Fedkiw, R.P.: Level Set Methods and Dynamic Implicit Surfaces. Applied Mathematical Sciences, vol. 153. Springer, New York (2002) 26. Osher, S., Sethian, J.A.: Fronts propagating with curvature-dependent speed: Algorithms based on Hamilton–Jacobi formulations. Journal of Computational Physics 79, 12–49 (1988) 27. Rao, A.R., Schunck, B.G.: Computing oriented texture fields. CVGIP: Graphical Models and Image Processing 53, 157–185 (1991) 28. Rouy, E., Tourin, A.: A viscosity solutions approach to shape-from-shading. SIAM Journal on Numerical Analysis 29, 867–884 (1992) 29. Sapiro, G., Kimmel, R., Shaked, D., Kimia, B.B., Bruckstein, A.M.: Implementing continuous-scale morphology via curve evolution. Pattern Recognition 26, 1363– 1372 (1993) 30. Schultz, T., Burgeth, B., Weickert, J.: Flexible segmentation and smoothing of DTMRI fields through a customizable structure tensor. In: Bebis, G., Boyle, R., Parvin, B., Koracin, D., Remagnino, P., Nefian, A., Meenakshisundaram, G., Pascucci, V., Zara, J., Molineros, J., Theisel, H., Malzbender, T. (eds.) ISVC 2006. LNCS, vol. 4291, pp. 455–464. Springer, Heidelberg (2006) 31. Serra, J.: Echantillonnage et estimation des phénomènes de transition minier. PhD thesis, University of Nancy, France (1967) 32. Serra, J.: Image Analysis and Mathematical Morphology, vol. 1. Academic Press, London (1982) 33. Serra, J.: Image Analysis and Mathematical Morphology, vol. 2. Academic Press, London (1988) 34. Soille, P.: Morphological Image Analysis, 2nd edn. Springer, Berlin (2003) 35. van den Boomgaard, R.: Mathematical Morphology: Extensions Towards Computer Vision. PhD thesis, University of Amsterdam, The Netherlands (1992) 36. Weickert, J.: Coherence-enhancing diffusion of colour images. In: Sanfeliu, A., Villanueva, J.J., Vitrià, J. (eds.) Proc. Seventh National Symposium on Pattern Recognition and Image Analysis, Barcelona, Spain, April 1997, vol. 1, pp. 239–244 (1997) 37. Weickert, J.: Coherence-enhancing diffusion filtering. International Journal of Computer Vision 31(2/3), 111–127 (1999) 38. Weickert, J., Brox, T.: Diffusion and regularization of vector- and matrix-valued images. In: Nashed, M.Z., Scherzer, O. (eds.) Inverse Problems, Image Analysis, and Medical Imaging. Contemporary Mathematics, vol. 313, pp. 251–268. AMS, Providence (2002) 39. Weickert, J., Hagen, H. (eds.): Visualization and Processing of Tensor Fields. Springer, Berlin (2006)

On Semi-implicit Splitting Schemes for the Beltrami Color Flow Lorina Dascal1 , Guy Rosman1 , Xue-Cheng Tai2,3 , and Ron Kimmel1 1

Department of Computer Science, Technion – Israel Institute of Technology, 32000, Haifa, Israel {lorina,rosman,ron}@cs.technion.ac.il 2 Division of Mathematical Sciences, SPMS, Nanyang Technological University, 50 Nanyang Avenue, 639798, Singapore [email protected] 3 Department of Mathematics, University of Bergen, Johannes Brunsgate 12, 5007, Bergen, Norway [email protected]

Abstract. The Beltrami flow is an efficient non-linear filter, that was shown to be effective for color image processing. The corresponding anisotropic diffusion operator strongly couples the spectral components. Usually, this flow is implemented by explicit schemes, that are stable only for small time steps and therefore require many iterations. In this paper we introduce a semi-implicit scheme based on the locally one-dimensional (LOD) and additive operator splitting (AOS) schemes for implementing the anisotropic Beltrami operator. The mixed spatial derivatives are treated explicitly, while the non-mixed derivatives are approximated in a semi-implicit manner. Numerical experiments demonstrate the stability of the proposed scheme. Accuracy and efficiency of the splitting schemes are tested in applications such as the scale-space analysis and denoising. In order to further accelerate the convergence of the numerical scheme, the reduced rank extrapolation (RRE) vector extrapolation technique is employed.

1

Introduction

Nonlinear diffusion filters based on partial differential equations (PDEs) have been extensively used in the last decade for different tasks in image processing. Their efficient implementation requires designing numerical schemes in which the issues of accuracy, stability, and computational cost all play important roles. The Beltrami image flow is an example of a non-linear filter, that is efficient for color image processing. It treats the image as a 2-D manifold embedded in a hybrid spatial-feature space. Minimization of the image area surface yields the Beltrami flow. The corresponding diffusion operator is anisotropic and strongly couples the spectral components. Due to its anisotropy and non-separability, so far there is no efficient implicit, nor operator-splitting-based numerical scheme for the partial differential equation that describes the Beltrami flow in color. X.-C. Tai et al. (Eds.): SSVM 2009, LNCS 5567, pp. 259–270, 2009. c Springer-Verlag Berlin Heidelberg 2009 

260

L. Dascal et al.

Usual discretizations of this filter are based on explicit schemes, that limit the time step and therefore result in a large number of iterations. In [1] an acceleration technique based on the reduced rank extrapolation (RRE) algorithm [2, 3] was proposed in order to speed-up the slow convergence of the explicit scheme. As an alternative to the explicit scheme, an approximation using the short time kernel of the Beltrami operator was suggested in [4]. Although unconditionally stable, this method is still computationally demanding, since computing the kernel involves geodesic distance computation around each pixel. The bilateral filter, which can be shown to be an Euclidean approximation of the Beltrami kernel, was studied in different contexts (see [5], [6], [7], [8], [9], [10]). Recently, a related filter, the nonlocal means filter, was proposed in [11] and shown to be useful in denoising gray-scale and color images. In this paper we propose to approximate the system of nonlinear coupled equations given by the Beltrami flow by a semi-implicit finite difference scheme based on operator splitting. Additive operator splitting (AOS) schemes were first developed for (nonlinear elliptic/parabolic) monotone equations and NavierStokes equations [12, 13]. In image processing applications, the AOS scheme was found to be an efficient way for approximating the Perona-Malik filter [14], especially if symmetry in scale-space is required. The AOS scheme is first order in time, semi-implicit, and unconditionally stable with respect to its time-step [13, 14]. In the early 1950’s (see [15]) the alternating-direction method (ADI) was introduced, and in [16] the LOD (locally one-dimensional) splitting method was proposed. The LOD scheme and other multiplicative splitting methods were employed in the context of nonlinear diffusion image filtering in [17]. We stress that the main characteristic of this class of equations, which allows splitting, is local isotropy. However, in the case of the anisotropic Beltrami operator, the main difficulty in splitting stems from the presence of the mixed derivatives. To overcome this problem, we suggest to construct the following semi-implicit scheme; the spatial mixed derivatives are discretized explicitly at the current time step nΔt, while those that do not contain mixed derivatives are approximated using an average of two levels of time steps: nΔt and (n + 1)Δt (Crank-Nicolson scheme). As our equations are nonlinear, a stability proof of the corresponding finite difference scheme is a non-trivial task. We provide numerical experiments which indicate that the LOD and the AOS splitting schemes for the nonlinear Beltrami color filter are stable for a wide range of time steps. We demonstrate the efficiency and stability of the splitting in applications such as: Beltramibased scale space and Beltrami-based denoising. In order to further expedite the LOD/AOS splitting schemes, we show how to speed-up their convergence by using the RRE (reduced rank extrapolation) technique. The RRE method was introduced by Me˘sina and Eddy [2, 3] to speed-up the convergence of general sequences of vectors without explicit knowledge of the sequence generator. This technique was applied in [1] in order to speed up the slow convergence of the standard explicit scheme for the Beltrami color flow. In this paper we show that in applications such as scale-space and denoising of color images, the semiimplicit LOD/AOS schemes can also be accelerated using the RRE technique.

On Semi-implicit Splitting Schemes for the Beltrami Color Flow

261

This paper is organized as follows: In Section 2 we briefly summarize the Beltrami framework. In Section 3 we briefly review general semi-implicit splitting operator schemes. In Section 4 we propose a semi-implicit splitting scheme for the anisotropic Beltrami operator, based on the LOD/AOS schemes. In Section 5 we demonstrate the efficiency and stability of the LOD/AOS splitting schemes for Beltrami-based scale-space and Beltrami-based denoising. Furthermore, we propose to accelerate the LOD/AOS schemes using the RRE technique. Section 6 concludes the paper.

2

The Beltrami Framework

Let us briefly review the Beltrami framework for non-linear diffusion in computer vision [18, 19, 20, 21]. We represent images as embedding maps of a Riemannian manifold in a higher dimensional space. We denote the map by U : Σ → M , where Σ is a two-dimensional surface, with (σ 1 , σ 2 ) denoting coordinates on it. M is the spatial-feature manifold, embedded in Rd+2 , where d is the number of image channels. For example, a gray-level image can be represented as a 2D surface embedded in R3 . The map U in this case is U (σ 1 , σ 2 ) = (σ 1 , σ 2 , I(σ 1 , σ 2 )), where I is the image intensity. For color images, U is given by U (σ 1 , σ 2 ) = (σ 1 , σ 2 , I 1 (σ 1 , σ 2 ), I 2 (σ 1 , σ 2 ), I 3 (σ 1 , σ 2 )), where I 1 , I 2 , I 3 are the three components of the color vector. Next, we choose a Riemannian metric on this surface, g, with elements denoted by gij . The canonical choice of coordinates in image processing is Cartesian (we denote them here by x1 and x2 ). For such a choice, which we follow in the rest of the paper, we identify σ 1 = x1 and σ 2 = x2 . In this case, σ 1 and σ 2 are the image coordinates. We denote the elements of the inverse of the metric by superscripts g ij , and the determinant by g = det(gij ). Once images are defined as embedding of Riemannian manifolds, it is natural to look for a measure on this space of embedding maps. Denote by (Σ, g) the image manifold and its metric, and by (M, h) the spacefeature manifold and its metric. Then, the functional S[U ] assigns a real number to a map U : Σ → M ,  √ S[U, gij , hab ] = ds σ g||dU ||2g,h , (1) where s is the dimension of Σ, g is the determinant of the image metric, and the range of indices is i, j = 1, 2, ... dim(Σ) and a, b = 1, 2, ... dim(M ). The integrand ||dU ||2g,h is expressed in a local coordinate system by ||dU ||2g,h = (∂xi U a )g ij (∂xj U b )hab . This functional, for dim(Σ) = 2 and hab = δab , was first proposed by Polyakov [22] in the context of high energy physics, in the theory known as string theory. The elements of the induced metric for color images with Cartesian color coordinates are   3 3 1 + β 2 a=1 (Uxa1 )2 β 2 a=1 Uxa1 Uxa2   G = (gij ) = , (2) 3 3 β 2 a=1 Uxa1 Uxa2 1 + β 2 a=1 (Uxa2 )2

262

L. Dascal et al.

where a subscript of U denotes a partial derivative and the parameter β > 0 determines the ratio between the spatial and spectral (color) distances. Using standard methods in calculus of variations, the Euler-Lagrange equations with respect to the embedding (assuming Euclidean embedding space) are 1 1 δS 0 = − √ hab b = √ div (D∇U a ), g δU g  

(3)

Δg U a

where the diffusion matrix is D =

√ −1 gG . Note that we can write 2

div(D∇U ) =

∂xq (dqr ∂xr U ).

q,r=1

The operator that acts on U is the natural generalization of the Laplacian from flat spaces to manifolds. It is called the Laplace-Beltrami operator, and denoted by Δg . The parameter β, in the elements of the metric gij , determines the nature of the flow. At the limits, where β → 0 and β → ∞, we obtain respectively a linear diffusion flow and a nonlinear flow, akin to the TV flow [23] for the case of grey-level images (see [20] for details). The Beltrami scale-space emerges as a gradient descent minimization process 1 δS Uta = − √ = Δg U a , g δU a

a = 1, 2, 3.

(4)

For Euclidean embedding, the functional in Eq. (1) reduces to  S(U ) =

√ g dx1 dx2 .

(5)

This geometric measure can be used as a regularization term for color image processing. In the variational framework, the reconstructed image is the minimizer of a cost-functional. This functional can be written in the following general form, 3

Ψ (U ) = λ ||U a − F a ||2 + S(U ), a=1

where the parameter λ controls the smoothness of the solution and F is the given image. The modified Euler-Lagrange equations as a gradient descent process are 1 δΨ 2λ Uta = − √ = − √ (U a − F a ) + Δg U a , g δU a g

a = 1, 2, 3.

(6)

On Semi-implicit Splitting Schemes for the Beltrami Color Flow

3

263

Operator Splitting Schemes

In this section we briefly review standard first order accurate splitting schemes for diffusion equations. One of the main drawbacks of the semi-implicit schemes for such equations in multiple dimensions is that the resulting inverted matrix does not have an efficient algorithm for its inversion. In order to remedy this shortcoming, splitting techniques are commonly employed in solving timedependent partial differential equations. They allow one to reduce problems in multiple spatial dimensions to a sequence of problems in one dimension, which are easier to solve. One of the simplest splitting schemes belonging to the class of multiplicative operator splitting schemes, is the locally one-dimensional (LOD) scheme [16]. The LOD scheme only needs to invert one three-diagonal matrix for each direction. It is simple to implement, is unconditionally stable and it is first order accurate. However, the system matrix is not axis symmetric, a property that may be important in some cases. If such a property is required, one could use the additive operator splitting scheme [13], which was actually invented for parallel implementation of splitting methods. Even for sequential implementations, the AOS is almost as efficient as the LOD scheme; instead of multiplying the operators, one computes them independently and then averages the sums of the inverse of the two matrices. We want to emphasize that the matrices for AOS use 2Δt instead of Δt. It is not a trivial matter to apply dimensional splitting schemes for Beltrami type of equations. Our goal is to construct a splitting scheme for the nonlinear anisotropic Beltrami operator, which would amount to inverting tridiagonal matrices, be unconditionally stable and preserve the time discretization accuracy that was obtained without applying splitting techniques.

4

The Proposed Splitting Scheme

In this section we present an operator splitting scheme for the Beltrami filter. Before splitting, we first introduce a semi-implicit approximation scheme to our equations. A semi-implicit Crank-Nicolson scheme for an equation involving mixed derivatives can rely on the following discretization of the spatial derivatives operators: mixed derivatives are computed at time step nΔt, while the non-mixed derivatives are computed as the average of the values at time steps nΔt and (n + 1)Δt. This approach for handling mixed derivatives in semiimplicit schemes for approximating linear equations has been considered in several previous works (see [24, 25, 26] for example), including the context of image processing [27], although it was not combined with the Crank-Nicolson method in the latter case. We note that in numerical experiments we have found the introduction of the Crank-Nicolson method into the splitting scheme necessary in order to maintain stability for large time steps. A simpler scheme, similar to the one used in [27], did not seem to be sufficiently stable for this PDE and the applications demonstrated in this paper. We now present the scheme we intend to use.

264

L. Dascal et al.

First, we refine our grid notations. We work on the rectangle Ω = (0, 1)×(0, 1), which we discretize by a uniform grid of m × m pixels, such that xi = iΔx, yj = jΔy, tn = nΔt, where 1 ≤ i ≤ m, 1 ≤ j ≤ m, 1 ≤ n ≤ J and JΔt = T . Let the 1 grid size be Δx = Δy = m−1 . a For each channel U , a = 1, 2, 3 of the color vector, we define the discrete approximation (U a )nij by (U a )(iΔx, jΔy, nΔt) = (U a )nij ≈ U a (iΔx, jΔy, nΔt). We impose von-Neumann boundary condition, and initially set U a to be our initial data image. 4.1

LOD/AOS Scheme for the Beltrami Scale-Space

We approximate the Beltrami filter given in Eq. (4) by the following semi implicit Crank-Nicolson scheme: 1  1 n a n+1 1 n a n (U a )n+1 − (U a )n = √ n All (U ) + All (U ) + Δt g 2 2 2

l=1

2



2

l=1



Anqr (U a )n ,

q=1 r =q

where U a is the N -dimensional vector denoting one of the components of the color vector, and Anqr is a central difference approximation of the operator ∂xq (dqr ∂xr ) at time step n. Rearranging terms, we obtain −1

2

Δt (U a )n+1 = I − √ n Anll 2 g l=1 ⎛ ⎞ 2

2



Δt Δt ⎝I + √ Anqr + √ n Anll ⎠ (U a )n , g n q=1 2 g r =q

l=1

which can also be written as ⎞ −1 ⎛

2 2

2



Δt Δt ⎝I + Δt (U a )n+1 = I − A¯nll A¯nqr + A¯nll ⎠ (U a )n , 2 2 q=1 l=1

where

r =q

l=1

1 A¯11 = √ ∂x (A∂x ), g

1 A¯22 = √ ∂y (C∂y ), g

1 A¯12 = √ ∂x (B∂y ), g

1 A¯21 = √ ∂y (B∂x ), g

and the functions A, B, C are the corresponding elements of the diffusion matrix associated with the Beltrami flow.

On Semi-implicit Splitting Schemes for the Beltrami Color Flow

265

Again, this semi-implicit scheme still has a major drawback. At each iteration one needs to solve a large linear system whose matrix of coefficients is not tridiagonal and thus costly. Instead, we employ the LOD splitting scheme  Δt ¯ −1 Δt ¯ −1  A22 A11 (U a )n+1 = I − I− 2 2 2

 

Δt ¯ Δt ¯ A11 )(I + A22 ) + Δt (I + A¯nqr (U a )n , 2 2 q=1 r =q

or the AOS scheme, that reads, (U a )n+1 =

−1  −1  1  I − ΔtA¯22 + I − ΔtA¯11 2 2

 

Δt ¯ Δt ¯ A11 )(I + A22 ) + Δt (I + A¯nqr (U a )n . 2 2 q=1 r =q

The above splitting schemes are efficient because at each time step a single tridiagonal matrix inversion is performed for each spatial dimension. The system of differential equations we deal with is nonlinear. The question of theoretical stability of the LOD/AOS based nonlinear finite difference scheme is a non-trivial challenge, with theory still lagging behind common practice. Our numerical experiments indicate that the splitting is stable for a wide variety of parameters, suitable for most applications, as will be shown in Section 5. 4.2

LOD/AOS Scheme for the Beltrami-Based Denoising

The splitting scheme in the presence of a fidelity term requires a slight modification that we detail below. In this case we solve for each channel the equation 2λ Uta = − √ (U a − F a ) + Δg U a , g

(7)

with von-Neumann boundary condition and the initial condition U a (x, 0) = F a (x). The Crank-Nicolson scheme approximating Eq. (7) is 2  Δt ¯n λ −1 (U a )n+1 = I − All + 2Δt √ n I 2 g l=1



2



Δt ¯n Δt ¯n A11 )(I + A22 ) + Δt A¯nqr (U a )n + 2 2 q=1 r =q  λ +2ΔtF a √ n . g

(I +

(8)

266

L. Dascal et al.

It is possible to use LOD/AOS approximations for the inverse of the matrix in the above scheme. √ However, we would like to treat the fidelity term in a special way. When λ/ g n is big, we find that the scheme proposed below possesses better stability properties. We now describe the details for treating the fidelity term for our CrankNicolson the nominator and the denominator by the matrix  scheme. Dividing S n = 1 + 2Δt √λgn I, and rearranging terms, we get 2  Δt n −1 ¯n −1 (S ) (U a )n+1 = I − All 2





l=1

2



Δt ¯n Δt ¯n A11 )(I + A22 ) + Δt A¯nqr (U a )n 2 2 q=1 r =q  λ +2(S n )−1 ΔtF a √ n . g

(S n )−1 (I +

Approximating the semi-implicit scheme based on the LOD-splitting, we have −1  −1  1 1 I − Δt(S n )−1 A¯n11 (U a )n+1 = I − Δt(S n )−1 A¯n22 2 2 2

 

Δt ¯n Δt ¯n A11 )(I + A22 ) + Δt (S n )−1 (I + A¯nqr (U a )n + 2 2 q=1 r =q  λ +2(S n )−1 ΔtF a √ n . g A similar splitting scheme can be developed using AOS.

5

Experimental Results

We proceed to demonstrate experimentally the stability, accuracy, and efficiency of the LOD and AOS splitting schemes for the Beltrami color flow. In Figure 1 we show the results of the Beltrami flow, implemented by employing the LOD splitting scheme for approximating Eq. (4). Next we illustrate the use of the splitting schemes in the case where the functional involves a fidelity term. A noisy image as well as the reference denoising result, based on the explicit scheme, are shown in Figure 3, next to the result of the AOS and LOD splitting schemes. Note that the visual results obtained by the two schemes are similar to the reference image. 5.1

RRE Extrapolation Technique for Acceleration of the LOD Splitting Scheme

In [28, 1] vector extrapolation was applied in order to speed up the slow convergence of the explicit schemes for the Beltrami color flow. In the experiments

On Semi-implicit Splitting Schemes for the Beltrami Color Flow

267

Fig. 1. Top row, left: The original image which contains JPEG artifacts.√Middle: Results of the LOD splitting scheme with Δt = 1, after 1 iteration, β = 103 , λ = 0. Right: Results of the LOD splitting scheme with after 2 iterations. Bottom row, left: Results of the LOD splitting scheme with after 4 iterations. Middle: a close-up of the original image. Right: a close-up of the resulting image after 4 iterations.

Fig. 2. The different image channels of an image patch taken from the images in Figure 1. Left to right: An image patch before denoising, its different color channels, the denoised image, and the denoised color channels. The color arrows indicate the direction of the gradient in the various color channels.

below we demonstrate how the RRE extrapolation technique can also be used to accelerate the convergence of implicit schemes. Figure 4 shows that the RRE method accelerates the LOD scheme. A comparison is also given to the convergence rate achieved by the method of [28,1]. Extrapolation techniques also allow us to obtain a more accurate rate, if one takes a smaller time step.

268

L. Dascal et al.

Fig. 3. Large image at the right: An image with artifacts resulting from lossy compression.. Smaller images – a close-up on a section of the image. Top row, left: The image with JPEG artifacts. Right: Beltrami-based denoising by explicit scheme, run with 4000 explicit iterations, Δt = 0.0005. Bottom row,√left: Denoising by LOD, Δt = 0.02. Right: Denoising by AOS, Δt = 0.02. λ = 1, β = 2000.

10

Residual Norm

10

10

10

10

5

Explicit Explicit+RRE LOD LOD+RRE

0

−5

−10

−15

0

10

20 30 40 CPU Time (sec)

50

Fig. 4. Graph of the residuals (LOD, explicit+RRE and LOD+RRE) versus CPU times. Parameters: Δt = 0.05 for the explicit scheme, Δt = 2.5 for LOD, λ = 0.5, β = √ 500 ≈ 22.36.

6

Conclusions

Due to its anisotropy and non-separability nature, no implicit scheme, nor operator splitting based scheme was so far introduced for the partial differential equations that describe the Beltrami color flow. In this paper we propose a

On Semi-implicit Splitting Schemes for the Beltrami Color Flow

269

semi-implicit splitting scheme based on LOD/AOS for the anisotropic Beltrami operator. The spatial mixed derivatives are discretized explicitly at time step nΔt , while the non-mixed derivatives are approximated using the average of the two time levels nΔt and (n + 1)Δt. The stability of the splitting is empirically tested in applications such as Beltrami-based scale-space and Beltrami-based denoising, which display a stable behavior. In order to further accelerate the convergence of the splitting schemes, the RRE vector extrapolation technique is employed.

Acknowledgements We thank Prof. Avram Sidi for interesting discussions. This research was supported by the United States -Israel Binational Science Foundation grant No. 2004274, by the Israeli Science Foundation grant No. 623/08, by the Ministry of Science grant No. 3-3414, and by the Elias Fund for Medical Research. XueCheng Tai is supported by the MOE (Ministry of Education) Tier II project T207N2202 and IDM project NRF2007IDMIDM002-010.

References 1. Rosman, G., Dascal, L., Kimmel, R., Sidi, A.: Efficient beltrami image filtering via vector extrapolation methods. SIAM J. Imag. Sci. (2008) (submitted) 2. Mešina, M.: Convergence acceleration for the iterative solution of the equations X = AX + f . Comp. Meth. Appl. Mech. Eng. 10, 165–173 (1977) 3. Eddy, R.: Extrapolating to the limit of a vector sequence. In: Wang, P. (ed.) Information Linkage Between Applied Mathematics and Industry, New York, pp. 387–396. Academic Press, London (1979) 4. Spira, A., Kimmel, R., Sochen, N.A.: A short-time Beltrami kernel for smoothing images and manifolds. IEEE Trans. Image Process. 16(6), 1628–1636 (2007) 5. Smith, S.M., Brady, J.: Susan - a new approach to low level image processing. Intl. J. of Comp. Vision 23, 45–78 (1997) 6. Aurich, V., Weule, J.: Non-linear gaussian filters performing edge preserving diffusion. In: Mustererkennung 1995, 17. DAGM-Symposium, London, UK, pp. 538–545. Springer, Heidelberg (1995) 7. Tomasi, C., Manduchi, R.: Bilateral filtering for gray and color images. In: Proceedings of IEEE International Conference on Computer Vision, pp. 836–846 (1998) 8. Sochen, N., Kimmel, R., Bruckstein, A.M.: Diffusions and confusions in signal and image processing. J. of Math. Imag. and Vision 14(3), 195–209 (2001) 9. Elad, M.: On the bilateral filter and ways to improve it. IEEE Trans. Image Process. 11(10), 1141–1151 (2002) 10. Barash, D.: A fundamental relationship between bilateral filtering, adaptive smoothing and the nonlinear diffusion equation. IEEE Trans. Image Process. 24(6), 844–847 (2002) 11. Buades, A., Coll, B., Morel, J.M.: A review of image denoising algorithms, with a new one. SIAM Interdisciplinary Journal 4, 490–530 (2005) 12. Lu, T., Neittaanmaki, P., Tai, X.C.: A parallel splitting up method and its application to Navier-Stokes equations. Applied Mathematics Letters 4(2), 25–29 (1991)

270

L. Dascal et al.

13. Lu, T., Neittaanmaki, P., Tai, X.C.: A parallel splitting up method for partial differential equations and its application to Navier-Stokes equations. RAIRO Mathematical Modelling and Numerical Analysis 26(6), 673–708 (1992) 14. Weickert, J., Romeny, B.M.T.H., Viergever, M.A.: Efficient and reliable schemes for nonlinear diffusion filtering. IEEE Trans. Image Process. 7(3), 398–410 (1998) 15. Peaceman, D.W., Rachford, H.H.: The numerical solution of parabolic and elliptic differential equations. Journal Soc. Ind. Appl. Math. 3, 28–41 (1955) 16. Yanenko, N.N.: The method of fractional steps. The solution of problems of mathematical physics in several variables. Springer-Verlag, New York (1971) 17. Barash, D., Schlick, T., Israeli, M., Kimmel, R.: Multiplicative operator splittings in nonlinear diffusion: from spatial splitting to multiple timesteps. J. of Math. Imag. and Vision 19(16), 33–48 (2003) 18. Kimmel, R., Malladi, R., Sochen, N.: Images as embedding maps and minimal surfaces: Movies, color, texture, and volumetric medical images. Intl. J. of Comp. Vision 39(2), 111–129 (2000) 19. Sochen, N., Kimmel, R., Maladi, R.: From high energy physics to low level vision. In: ter Haar Romeny, B.M., Florack, L.M.J., Viergever, M.A. (eds.) Scale-Space 1997. LNCS, vol. 1252, pp. 236–247. Springer, Heidelberg (1997) 20. Sochen, N., Kimmel, R., Maladi, R.: A general framework for low level vision. IEEE Trans. Image Process. 7, 310–318 (1998) 21. Yezzi, A.J.: Modified curvature motion for image smoothing and enhancement. IEEE Trans. Image Process. 7(3), 345–352 (1998) 22. Polyakov, A.M.: Quantum geometry of bosonic strings. Physics Letters 103 B, 207–210 (1981) 23. Rudin, L., Osher, S., Fatemi, E.: Non-linear total variation based noise removal algorithms. Physica D Letters 60, 259–268 (1992) 24. Yanenko, N.N.: About implicit difference methods of the calculation of the multidimensional equation of thermal conductivity. In: Proceedings of VUZ. Series of Mathematics, vol. 23(4), pp. 148–157 (1961) 25. Andreev, V.B.: Alternating direction methods for parabolic equations in two space dimensions with mixed derivatives. Zhurnal Vychislitelnoi Matematiki i Matematicheskoi Fiziki 7(2), 312–321 (1967) 26. Mckee, S., Mitchell, A.R.: Alternating direction methods for parabolic equations in three space dimensions with mixed derivatives. The Computer Journal 14(3), 25–30 (1971) 27. Weickert, J.: Coherence-enhancing diffusion filtering. Intl. J. of Comp. Vision 31(2/3), 111–127 (1999) 28. Dascal, L., Rosman, G., Kimmel, R.: Efficient Beltrami filtering of color images via vector extrapolation. In: Sgallari, F., Murli, A., Paragios, N. (eds.) SSVM 2007. LNCS, vol. 4485, pp. 92–103. Springer, Heidelberg (2007)

Multi-scale Total Variation with Automated Regularization Parameter Selection for Color Image Restoration Yiqiu Dong1 and Michael Hintermüller2 1 START-Project “Interfaces and Free Boundaries” and SFB “Mathematical Optimization and Applications in Biomedical Science”, Institute of Mathematics and Scientific Computing, University of Graz, Heinrichstrasse 36, A-8010 Graz, Austria [email protected] 2 Department of Mathematics, Humboldt-University of Berlin, Unter den Linden 6, 10099 Berlin, Germany, and START-Project “Interfaces and Free Boundaries” and SFB “Mathematical Optimization and Applications in Biomedical Science”, Institute of Mathematics and Scientific Computing, University of Graz, Heinrichstrasse 36, A-8010 Graz, Austria [email protected]

Abstract. In this paper, a multi-scale vectorial total variation model for color image restoration is introduced. The model utilizes a spatially dependent regularization parameter in order to preserve the details during noise removal. The automated adjustment strategy of the regularization parameter is based on local variance estimators combined with a confidence interval technique. Numerical results on images are presented to demonstrate the efficiency of the method.

1

Introduction

We consider the problem of recovering color images degraded by cross-channel ˆ blurring and Gaussian noise. Without loss of generality, we assume an image u is a vectorial function defined on a bounded and piecewise smooth open subset ˆ : Ω → RM , where M is the number of channels in the color Ω ∈ R2 , that is, u ˆ is given by model. The degraded form z of u ˆ + n, z = Ku where K ∈ L(L2 (Ω; RM )) is a cross-channel blurring operator, and n represents white Gaussian noise with zero mean and standard deviation σ. The problem of ˆ from z with unknown n is known to be typically ill-posed [1]. restoring u In order to preserve significant edges during restoring images, Rudin, Osher and Fatemi proposed total variation regularization [2] for gray-level images. In this approach (which we call the TV-model in what follows), the image u ˆ is recovered by solving the optimization problem   λ min |Du| + |Ku − z|2 dx, (1) u∈BV (Ω) Ω 2 Ω X.-C. Tai et al. (Eds.): SSVM 2009, LNCS 5567, pp. 271–281, 2009. c Springer-Verlag Berlin Heidelberg 2009 

272

Y. Dong and M. Hintermüller

where BV (Ω) denotes the space of functions of bounded variation and λ > 0. Because of the edge preservation ability, the TV-model is widely accepted as a reliable tool in image restoration. Over the years, various research efforts have been devoted to studying, solving and extending the TV-model; see, e.g., [3, 4, 5, 6, 7, 8, 9] as well as the monograph [1] and the many references therein. In general, images are comprised of multiple objects at different scales. This suggests that different values of λ localized at image features of different scales are desirable to obtain better restoration results. For this reason, a multi-scale total variation (MTV) model with a spatially varying choice of parameters was proposed [10]. In order to enhance image regions containing details while still sufficiently smoothing homogeneous features, a spatially dependent regularization parameter selection was proposed in [11]. In this paper, we will extend the multi-scale total variation with spatially dependent regularization parameter to restore degraded color images. The automated adjustment strategy of the regularization parameter is based on local variance estimators combined with a confidence interval technique. For speeding up the performance of the scheme we generalize the multi-scale representation according to [12, 13], and the corresponding subproblems are solved by a superlinearly convergent algorithm based on Fenchel-duality and inexact semismooth Newton techniques. The latter extends earlier work in [9]. The outline of the rest of the paper is as follows. In Section 2 we introduce the multi-scale vectorial total variation model and the primal-dual algorithm for solving the associated minimization problem. In Section 3 we extend the LVE-based parameter selection to color images. Section 4 proposes a method for color image restoration combining the multi-scale representation and spatially adaptive parameter selection. Section 5 gives numerical results to demonstrate the performance of the new method. Finally conclusions are drawn in Section 6.

2

Multi-scale Vectorial Total Variation

Based on the TV-model (1), in [14] the vectorial total variation (VTV) regularization was proposed for restoring color images:   λ min |Du| + |Ku − z|2 dx, (2) 2 Ω u∈BV (Ω;RM ) Ω where the space BV (Ω; RM  ) of vector-valued functions is the set of functions  u ∈ L1 (Ω; RM ) such that Ω |Du| < ∞, where the vectorial TV norm Ω |Du| is defined as    |Du| = sup u · divv dx : v ∈ Cc1 (Ω; RM×2 ), |v| ≤ 1 , Ω

and |v| =



Ω

M i=1 vi , vi .

The space BV (Ω; RM ) endowed with the norm  uBV (Ω;RM ) = uL1 (Ω;RM ) + |Du| Ω

is a Banach space.

Multi-scale Total Variation

273

In the VTV-model (2), the parameter λ controls the trade-off between a good fit of z and a smoothness requirement due to the vectorial total variation regularization. Since images are usually comprised of multiple objects at different scales, locally different λ is desirable. Therefore, here we consider multi-scale vectorial total variation (MVTV):   1 min |Du| + λ(x)|Ku − z|2 dx. (3) 2 Ω u∈BV (Ω;RM ) Ω Similar as in Section 2 of [11], we can obtain the same conclusion on the existence and uniqueness of the minimizer for the MVTV-model. Here, we do not repeat proof details, but rather refer to [11]. 2.1

Primal-Dual Approach to Multi-scale Vectorial Total Variation

In [9] an infeasible primal-dual algorithm of generalized Newton-type was proposed for solving (1). In the sequel we extend its key features to the case (3). Rather than operating on the MVTV-model (3) the method is based on    1 μ 2 2 min |∇u| dx + λ|Ku − z| dx + |∇u| dx, (4) 2 2 Ω u∈H01 (Ω;RM ) 2 Ω Ω ¯ for almost all x ∈ Ω and 0 < μ λ ¯ −1 . The μwhere 0 <  ≤ λ(x) ≤ λ term serves the purpose of a function space regularization for a “convenient" dualization in a Hilbert space setting. In our numerics, we typically choose μ = 0. Applying the Fenchel-Legendre calculus [15] analogously as in [9], the Fencheldual of (4) reads sup 2

M

p ∈ L (Ω; R ) |p(x)| ≤ 1 a.e. in Ω

1 1 − |||K ∗ z − divp|||2H −1 + z2L2 , 2 2

(P0 )

where |||u|||2H −1 = Hμ,K v, vH01 ,H −1 , v ∈ H −1 (Ω; RM ) with Hμ,K = (K ∗ λK − μ)−1 ,  : H01 (Ω; RM ) → H −1 (Ω; RM ), and ·, ·H01 ,H −1 denotes the duality pairing between H01 (Ω; RM ) and its dual H −1 (Ω; RM ). Moreover, L2 (Ω; RM ) = (L2 (Ω; RM ))2 . In order to avoid the non-uniqueness of the solution of (P0 ), following [9] we consider a dual regularization:  1 1 β ∗ 2 2 sup − |||K z − divp|||H −1 + zL2 − p2L2 . (P ) 2 2 2 Ω p ∈ L2 (Ω; RM ) |p(x)| ≤ 1 a.e. in Ω

where β > 0 is the regularization parameter. In order to study the effect of the βregularization of the Fenchel-dual, we apply the Fenchel-Legendre calculus once more and find that the dual of (P ) is given by    1 μ 2 2 |∇u|2 dx + λ|Ku − z| dx + Φβ (∇u)dx, (P ∗ ) min 2 Ω u∈H01 (Ω;RM ) 2 Ω Ω where for w ∈ L2 (Ω; RM ),

274

Y. Dong and M. Hintermüller

 Φβ (w)(x) =

|w(x)| − β2 , if |w(x)| ≥ β, 1 2 2β |w(x)| , if |w(x)| < β.

(5)

¯ and The first-order optimality conditions of (P ∗ ) characterize the solution u ¯ of (P ∗ ) and (P ), respectively, by p ¯ − div¯ − μ¯ u + K ∗ λK u p = K ∗ λz in H −1 (Ω; RM ), max(β, |∇¯ u|)¯ p − ∇¯ u=0

2

in L (Ω; R ). M

(6a) (6b)

Note that the system (6) is non-smooth, i.e. not necessarily Fréchet-differentiable. The discrete version of this system can be solved by a semismooth Newton method [9, 11]. The generalized Newton solver converges globally, that is regardless of the initialization, and locally at a superlinear rate [9].

3

Spatially Dependent Regularization Parameter Selection

Since the capability of multi-scale vectorial total variation is mainly limited by the selection of the parameter λ, in this section we extend the way to choose λ proposed in [11] to the MVTV-model. Suppose the variance of Gaussian noise is σ 2 , which can be estimated easily in practice. With a correct choice of λ in the TV-model (1), the restored image u can satisfy the constraint  |Ku − z|2 dx = σ 2 |Ω| (7) Ω

globally. However, the MVTV-model (3) represents a localized version of the constraint by allowing λ = λ(x). In order to enhance image details while preserving homogenous regions, the choice of λ must be based on local image features. Hence, we search for a reconstruction where the variance of the residual is closer to the noise variance in both the detail regions and the homogeneous parts. In order to achieve this goal we introduce local variance estimators (LVEs) for an automated adaptive choice of λ. 3.1

Local Variance Estimator

Consider the discrete version of the residual image rh = zh − K h uh , where uh is the restored image from the minimization problem (2) with λ > 0. If we use a relatively small parameter λ, the residual rh will include the noise as well as the details. Then, the average of the squared residual in a small window will reflect the distribution of details in the image. ω Let Ωi,j denote the set of pixel-coordinates in a ω-by-ω window centered at (i, j) (with obvious modification near the boundary), i.e., ω   ω

ω Ωi,j ≤ s, t ≤ , = (s + i, t + j) : − 2 2

Multi-scale Total Variation

275

where · means rounding to the nearest integer towards zero. Then we apply the mean filter with window size ω to the residual image rh as follows: LVEω i,j =

M 1 M ω2



rhs,t

ω k=1 (s,t)∈Ωi,j

2 k

=

M 1 M ω2



ω k=1 (s,t)∈Ωi,j

h 2 zs,t − (K h uh )s,t k .

Here LVE stands for a “Local Variance Estimator”. In general, LVEω has a large value in the detail regions, and it has a small value in the homogeneous regions. But the noise in the residual may also lead to some large LVE values in the homogeneous regions. In order to reduce the effect due to noise, we utilize the confidence interval technique well-known in statistics [16, 17] in connection with LVE. 3.2

Upper Bound for the Local Variance

In the discrete setting, all elements of n can be regarded as an array of independent normally distributed random variables with mean 0 and variance σ 2 . Then, the random variable M 1 ω Ti,j = 2 (nhs,t )2k σ ω k=1 (s,t)∈Ωi,j

ω has the χ2 -distribution with M ω 2 degrees of freedom, that is, Ti,j ∼ χ2Mω2 . Set M 1 ω Si,j := (zhs,t − (K h uh )s,t )2k . M ω2 ω k=1 (s,t)∈Ωi,j

ˆ h , then ˆ h satisfies nh = zh − K h u If uh = u ω Si,j =

M 1 M ω2



M 1 M ω2



ˆ h )s,t )2k (zhs,t − (K h u

ω k=1 (s,t)∈Ωi,j

=

(nhs,t )2k =

ω k=1 (s,t)∈Ωi,j

σ2 ω T . M ω 2 i,j

On the contrary, if the residual image zh − K h uh contains details, we expect ω Si,j =

M 1 M ω2

k=1

M 1 > M ω2



(zhs,t − (K h uh )s,t )2k

ω (s,t)∈Ωi,j



ω k=1 (s,t)∈Ωi,j

(nhs,t )2k =

σ2 ω T . M ω 2 i,j

ω > B for some pixel (i, j) Therefore, we search for a bound B such that Si,j implies that in the residual some details are left. Given m × m, the total number

276

Y. Dong and M. Hintermüller

of pixels in the color image with M channels, we propose to consider the expected σ2 ω 2 maximum of the m2 random variables Mω 2 Ts , s = 1, . . . , m , as the bound B: B ω,m :=

σ2 E( max T ω ), M ω 2 k=1,...,m2 k

(8)

where E represents the expected value of a random variable. Similar as proposed in [11], we get σ2 (Em (T ω ) + dm (T ω )), M ω2

B ω,m =

where Em (T ω ) = Td + βκm , dm (T ω ) = β π√6 , βm = m2 fm (Td ), κ = 0.577215, and m f(Td ) is the distribution of Td , which is the so-called dominant value. 3.3

Selection of the Parameter λ

Now, we use the confidence interval for S ω to reduce the effect from noise on the local variance estimators in order to distinguish the detail regions in the images correctly. Recall that LVEω denotes the mean of the squared residual in a given window. Ideally, there is only noise in the residual. Then LVEω should behave like S ω . Hence, whenever ω,m LVEω ), i,j ∈ [0, B

(9)

we assume that the window contains noise only. On the other hand, if (9) is not satisfied, we suppose that this is due to image details contained in the residual ω image in Ωi,j . This property is useful when updating the parameter λ locally. For adapting λ algorithmically we proceed as follows. Initially we assign a small positive value to λ. Then we restore the image iteratively by increasing λ according to the following rule:  ˜ k+1 λ i,j

= ζ · min

˜k λ i,j

λk+1 i,j =

 +  ω +ρ (LVEk )i,j − σ ,L ,

1 ω2



˜k+1 , λ s,t

(10a)

(10b)

ω (s,t)∈Ωi,j

where ζ ≥ 1, ρ > 0, (x)+ = max(x, 0), LVEω k is obtained from uk , L is a large ˜ k ∈ L∞ (Ω), and for each channel of the vectorial data positive value to ensure λ we use the same λk during restoration. In our numerics we choose ζ = 2 which comes from the method proposed in [12] (TNV-algorithm). Finally, we set the ˜ k ||∞ /σ in order to keep the new λ ˜ k+1 at the same scale parameter ρ = ρk = ||λ ˜ as λk .

Multi-scale Total Variation

4

277

Our Method

Recently, a multi-scale image decomposition method (TNV-algorithm) was proposed in [12], which uses the TV-model (1) to extract the details in the residual, and which varies the regularization parameter over a sequence of dyadic scales to capture different features in the image. Although this method performs better than a number of existing methods, it satisfies the constraint (7) only globally, and does not consider the local characteristic of the features in the image. Referring to this decomposition method, we intertwine its idea with the MVTV-model (3), and combine it with the spatially dependent regularization parameter selection. This results in the following algorithm: Algorithm 2

2

1: Initialize uh0 = 0 ∈ RMm , ph0 = 0 ∈ RMm ×2 , λ0 = [λ0 , · · · , λ0 ] ∈ RM with 2 λ0 ∈ Rm and k = 0. 2: If k = 0 solve the discrete version of the minimization problem    1 μ 2 0 2 ˜ 0 = arg u min |∇u|2 dx + λ |Ku − z| dx + |∇u| dx, 2 Ω u∈H01 (Ω;RM ) 2 Ω Ω else compute vkh = zh − K h uhk and solve the discrete version of the minimization problem:    1 μ 2 k 2 ˜ k = arg u min |∇u|2 dx + λ |Ku − vk | dx + |∇u| dx, 2 Ω u∈H01 (Ω;RM ) 2 Ω Ω ˜ hk . 3: Update uhk+1 = uhk + u h 4: Based on uk+1 , update

  +  ˜ k+1 = 2 · min λ ˜k + ρ λ LVEω − σ ,L , k

(λk+1 )i,j =

1 ω2



˜k+1 . λ s,t

ω (s,t)∈Ωi,j

5: Stop; or set k := k + 1 and go to step 2. A few remarks on the algorithm are in order. We initialize λ by a relatively small positive constant. In our numerical practice an 11-by-11 window turned out to yield reliable results. In Section 5, we study the influence of the window size on the restoration results. Similar to the Bregman iteration proposed in [18], we stop the iterative procedure as soon as the residual zh −