This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
= h(y) (∇V η, T(y)−∇V η, ∇A(η)) eη(Ω),T(y)−A(η(Ω)) (14) = q(y, η)∇V η, T(y) − ∇A(η) . By the chain rule applied to Ψ (q(y(x), η), p(y)), we get < Ψ (q(y, η), p(y)), V >=< q (y, η), V > ∂1 Ψ (q, p), which gives < D (Ω), V >= χ q(y, η)∂1 Ψ (q, p)∇V η, T(y) − ∇A(η)dy. We introduce C= q(y, η)∂1 Ψ (q, p) (T(y) − ∇A(η)) dy = E[∂1 Ψ (q, p) (T(Y ) − E[T(Y )])] χ
which completes the proof. A.3 Proof of Lemma1 When using the MLE, the term E[T(Y )] can be empirically estimated with T(Y ) and so derived easily with respect to the domain Ω. We propose to directly derive the expression ∇A(η) = T(Y ) which gives: κ j=1
< ηj , V >
∂2A (η) =< Ti (Y ) , V > ∂ηi ∂ηj
∀i ∈ [1, κ] ,
(15)
¨ which can be written in the compact form ∇V (T) = A(η)∇ V η. ¨ Restricting our study to the full rank exponential family, where A(η) is a symmetric positive-definite, hence invertible, matrix (Theorem 2), the domain derivative of the pa¨ −1 ∇V (T) = ∇V η where ∇V (T) is given rameters η is uniquely determined by A(η) 1 by: ∇V (T) = |Ω| ∂Ω T(y) − T(y(a)) (V · N)da(x) (taking benefit of theorem 4) and the lemma follows.
Optimization of Divergences within the Exponential Family for Image Segmentation
149
A.4 Proof of Lemma 2 Since p and q belongs to the same parametric law, they share the same value for h(y), T(y) and A(η) and then log(q) − log(p) = η − η 1 , T(y) − A(η) + A(η 1 ). The value of C is then C = s1 − s2 , with: s1 = E[(η − η 1 , T(y) − A(η) + A(η 1 ) + 1)(Ti (Y ) − E[Ti (Y )]] p s2 = E[ (Ti (Y ) − E[Ti (Y )]]Ep [(Ti (Y ) − E[Ti (Y )]] q Developing the expression of the expectation of the second term,we find s2 = Ep [(Ti (Y ) − E[Ti (Y )]] = ∇A(η 1 ) − ∇A(η). Using the linearity of the expectation and the fact that E[Tj (Y )(Ti (Y )] − E[Ti (Y )]E[Tj (Y )] designates the co¨ ij = variance matrix of the sufficient statistics T and can then be replaced by A(η) κ ¨ ¨ ¨ Cov[T(Y )]ij = A(η)ji , we find: s1 = j=1 (ηj − η1j )A(η)ij , and then C = A(η) (η − η1 ) + ∇A(η) − ∇A(η 1 ).
Convex Multi-class Image Labeling by Simplex-Constrained Total Variation Jan Lellmann, Jörg Kappes, Jing Yuan, Florian Becker, and Christoph Schnörr Image and Pattern Analysis Group (IPA) HCI, Dept. of Mathematics and Computer Science, University of Heidelberg {lellmann,kappes,yuanjing,becker,schnoerr}@math.uni-heidelberg.de
Abstract. Multi-class labeling is one of the core problems in image analysis. We show how this combinatorial problem can be approximately solved using tools from convex optimization. We suggest a novel functional based on a multidimensional total variation formulation, allowing for a broad range of data terms. Optimization is carried out in the operator splitting framework using Douglas-Rachford Splitting. In this connection, we compare two methods to solve the Rudin-Osher-Fatemi type subproblems and demonstrate the performance of our approach on single- and multichannel images.
1
Introduction
In this paper, we study the variational approach inf f (u) , f (u) = − u(x), s(x)dx + λ TV(u) , u∈C
λ>0,
(1)
Ω
for determining a labeling u : Ω → RL , that is a contextual classification of each pixel x ∈ Ω into one out of L classes, based on an arbitrary vector-valued similarity function s(x) ∈ RL as input data that has been computed from image data beforehand. The objective function (1) comprises the common form of a data term plus a regularization term. The data term is given by the L2 inner product of the assignment variables u and the similarity function s, and the regularizer is a total variation (TV) formulation for vector-valued data, TV(u) = ∇u1 2 + · · · + ∇uL 2 dx . (2) Ω
Furthermore, the constraint u ∈ C restricts the vector field u(x) at each location x ∈ Ω to lie in the standard probability simplex, that is u(x) ∈ RL + and L i=1 u(x) i = 1 for all x ∈ Ω. Our work is motivated by the following observation. Suppose that at each pixel x ∈ Ω, there is an unambiguous assignment (labeling) of the data s(x) to some class l ∈ {1, . . . , L} represented by the corresponding l-th unit vector, X.-C. Tai et al. (Eds.): SSVM 2009, LNCS 5567, pp. 150–162, 2009. c Springer-Verlag Berlin Heidelberg 2009
Convex Multi-class Image Labeling by Simplex-Constrained Total Variation
151
Fig. 1. Left: Noisy input image. Right: The labeled image based on the non-binary assignment u as global minimizer of the convex approach (1). The discrete problem is accurately solved by a continuous approach.
u(x) = el . Then, an interface with√area A between two image regions labeled with l and l , respectively, adds A 2 to the regularization term iff l = l , as all but two gradients under the square root vanish. As a result, under these √ assumptions and up to the immaterial constant 2, the TV term corresponds to the well-known Potts model that assigns constant penalties to local changes of the labeling. A significant difference between the Potts model and our approach (1), however, is that the former amounts to solve a discrete combinatorial problem, whereas the latter is a continuous convex optimization problem. Experiments show that our approach (1) approximates discrete decisions fairly well (Fig. 1 and 2) by computing a global optimum to a single convex optimization problem. By contrast, the state-of-the-art discrete approach [1] approximates the combinatorial solution by solving a non-uniquely defined sequence of binary problems via graph cuts. This fact, along with the potential of continuous convex optimization for parallel implementations and their more robust dependency on (hyper-) parameters, motivated to investigate the approach (1) as a promising model for a general “labeling submodule” within computer vision systems. To this end, – We have a closer look at the data and regularization terms (section 2). – We apply an operator splitting approach to (1) in order to decompose the computation of a globally optimal labeling into two independent computational steps: TV denoising for vector-valued data, and projection of the labeling vectors u(x) on the canonical simplex (section 3). – We evaluate two different algorithms for the TV denoising subroutine (section 4) and compare the performance of our convex method to a range of established graph cut-based approaches (section 5). Related work. In contrast to the binary case with anisotropic discretization [2], multi-class energies are generally not submodular and thus cannot be optimized globally using graph cuts [3]. Some extensions exist, which find a local minimum by solving a sequence of binary graph cuts [1]. The continuous formulation – optimization on the set of characteristic functions – is known as continuous cut [5]. Chan et al. [6] showed that this problem can be relaxed and solved on
152
J. Lellmann et al.
Fig. 2. Output of the standard TV approach [4] for scalar-valued images applied to the noisy input image depicted in Fig. 1, for different values of the regularization parameter λ. Irrespective of this value, the performance is worse than with the approach (1) (cf. Fig. 1, right), because the latter approximates the Potts model that does not depend on the size (contrast) of discontinuities. Consequently, the former approach cannot remove noise without degrading weak discontinuities, as is apparent above for the horizontal discontinuities.
a convex set, without losing global optimality. In contrast, our work is aimed at the multi-class case. In [7], a comparable approach based on [8] was presented, which relies on a natural ordering of the labels, as given in e.g. stereo reconstruction. An approach very similar to ours was recently presented in [9], where the authors use a different formulation of the total variation on vector fields, and an alternating optimization method. The (discrete) Potts model was studied in [10], where approximate solutions were computed by an LP relaxation with explicit constraints. In contrast, our approach considers the general TV term and a problem decomposition into efficiently solvable subproblems, without the need to introduce additional variables. Notation. We consider the discretized version of our approach (1). Let Ω = {1, . . . , n1 } × · · · × {1, . . . , nd } ⊆ Rd , d ∈ N, denote a regular image grid of n := |Ω| pixels. The (multidimensional) image space X := Rn×L is equipped with the Euclidean inner product ·, ·Ω over the vectorized elements. We naturally identify v = (v 1 , . . . , v L ) ∈ Rn×L with ((v 1 ) · · · (v L ) ) ∈ RnL . Superscripts v i denote a collection of vectors, while subscripts vk denote vector components. Using the notation e = (1, 1, . . . , 1) , the standard simplex on RL n×L L and its extension C on R are given by ΔL := v ∈ R v ≥ 0 , e, v = 1
and C := x∈Ω ΔL . Define δC (x) to be 0 iff x ∈ C, and +∞ otherwise. Let grad := (grad 1 , . . . , gradd ) be the d-dimensional forward difference gradient operator for Neumann boundary conditions. Accordingly, div := −grad is the backward difference divergence operator for Dirichlet boundary conditions. These operators extend to Rn×L via Grad := (IL ⊗ grad), Div := (IL ⊗ div), where IL is the L × L identity matrix. We will also need the convex sets L 12 Bλ := (p1 , . . . , pL ) ∈ Rd×L pi 22 λ , i=1
(3)
Convex Multi-class Image Labeling by Simplex-Constrained Total Variation
Dλ :=
Bλ ⊆ Rn×d×L , Eλ := {u ∈ Rn×L |u = Div p , p ∈ Dλ } .
153
(4)
x∈Ω
The discrete total variation on vector-valued data is then defined as
TV(u) := σE1 (u) = Gx u2 ,
(5)
x∈Ω
where σM (u) := supp∈M u, p is the support function from convex analysis, and Gx is an (Ld)×n matrix composed of rows of (Grad) s.t. Gx u gives the gradients of all ui in x stacked one above the other.
2
Variational Approach
Based on the introduced notation, our novel approach (1) reads inf f (u) ,
u∈C
λ TV(u) , f (u) = −u, sΩ + data term regularization term
λ>0,
(6)
As the objective function f and the constraint set C are convex, the overall problem is convex as well. We will now define and motivate each term. Data Term. The data term in (6) is fairly general. Any vector-valued similarity function s can be used, whose components s(x) i indicate the affinity of some data point at x with class i. As an example, suppose we have image features g(x), x ∈ Ω, prototypical feature vectors G = (G1 , . . . , GL ) as well as a distance measure d on the features. We might think of g as a grayscale image, of G as some prototypical gray values, and of d as a quadratic distance measure, possibly derived from a statistical noise model. The hard assignment of the pixel x ∈ Ω to a label (or class) l(x) ∈ {1, . . . , L} should then be penalized by the distance d(g(x), Gl(x) ) of the corresponding feature to the prototype of the assigned class. Denoting the negative distance by s, and summing up over the image domain, we see that
s(x), u(x) for u(x) = el(x) . d g(x), Gl(x) = − (7) x∈Ω
x∈Ω
Thus, instead of looking for l ∈ {1, . . . , L}n , we may equivalently look for u ∈ {e1 , . . . , eL }n . However, the right hand side formulation has the advantage that it extends naturally to the soft assignment u ∈ C: We may now solve the easier problem of optimizing for u on the convex set C. In our experiments, we chose d(x, y) = ||x − y||1 , as the 1 -norm is still convex but known to be more robust against noise and outliers. However, s is not restricted to representing distances. In fact, it may be arbitrarily nonlinear and nonconvex in x and g, and involve nonlocal operations on g. The complexity is completely hidden within the precomputed vector s. Regularization Term. Recall that the regularizer of (6) is defined (5) as
TV(u) = sup u, Div p = Gx u2 . (8) p∈D1
x∈Ω
154
J. Lellmann et al.
This definition for vector-valued u parallels the definition of the “isotropic” total variation measure in the scalar-valued case [11, 4, 12]. It is also known as MTV [13, 14, 15], and was recently studied in [16] in its continuous formulation. Contrary to the anisotropic discretization, where one would substitute the sum of 1-norms in (3), it is less biased towards edges parallel to the axes. See also [17] for an overview of TV-based research and applications. Optimality. After solving the relaxed problem, it remains to show that a binary solution can be recovered. For the continuous, binary case, Chan et al. [6] showed that an exact solution can be obtained by thresholding at almost any threshold. However, their results do not immediately transfer to the discrete multi-class case. In particular, the crucial “layer cake” formula holds for 1 -, but not 2 discretizations of the TV. Contrary to the binary case, it is not clear which rounding scheme to use for vector-valued u. For our experiments, we chose the final class label for each pixel x as the index l of the maximal u∗l (x) of the global optimum u∗ of (6). This defines a suboptimal discrete solution u∗t . Bounding the error f (u∗t ) − f (u∗d) with respect to the unknown discrete optimum u∗d will be subject of our future work.
3
Optimization
Two basic problems arise concerning the optimization of (6): Nondifferentiability of the objective function due to the TV term, and handling of the simplex constraint u ∈ C. We cope with the latter using the tight Douglas-Rachford splitting method as presented in the following section. We refer to [18] for the full derivations. Douglas-Rachford Splitting. Minimization of a proper, convex, lower-semicontinuous (lsc) function f : X → R can be regarded as finding a zero of its (necessarily maximal monotone [19, Chap. 12]) subgradient operator T := ∂f : X ⇒ X. In the operator splitting framework, ∂f is assumed to be decomposable into the sum of two “simple” operators, T = A + B, of which forward and backward steps can practically be computed. Here, we consider the (tight) Douglas-Rachford-Splitting iteration [20, 21], z k+1 ∈ (Jτ A (2Jτ B − I) + (I − Jτ B ))(z k ) ,
(9)
where Jτ T := (I + τ T )−1 is the resolvent of T . Under the very general constraint that A and B are maximal monotone and A + B has at least one zero, the sequence (z k ) will converge to a point z, with the additional property that x := Jτ B (z) is a zero of T ( [22, Thm. 3.15], [22, Prop. 3.20], [22, Prop. 3.19], [23]). In particular, for f = f1 +f2 , fi proper, convex, lsc with ri(dom f1 )∩ri(dom f2 ) = ∅ (ri(S) denoting the relative interior of a set S), it can be shown [19, Cor. 10.9] that ∂f = ∂f1 + ∂f2 , and the ∂fi are maximal monotone. As x ∈ Jτ ∂fi (y) ⇔ x = argmin(2τ )−1 x − y22 + fi (x), the computation of the resolvents reduces to proximal point optimization problems involving only the fi .
Convex Multi-class Image Labeling by Simplex-Constrained Total Variation
155
Application. For our specific problem, we split inf (f1 (u) + f2 (u)) , f1 (u) = −u, sΩ + λTV(u) , f2 (u) = δC (u) .
u∈C
(10)
and get the following Douglas-Rachford scheme: Algorithm 1. Outer loop (Douglas-Rachford) 1: choose some u0 and a fixed step size τ > 0 2: repeat 1 3: solve uk ← argminu { 2τ u − z k 2 − u, s + σEλ (u)} 1 k 4: solve w ← argminw { 2τ w − (2uk − z k )2 + δC (w)} 5: z k+1 ← z k + wk − uk 6: until uk − uk−1 ∞ δouter .
As f is bounded from below on the compact set C and thus attains its minimum. From the remarks in the last section, we get convergence of the scheme for the discrete case: δC (w) and σEλ are both proper, convex, lsc with dom σEλ = Rn and ri(C) = ∅. In practice, one has to deal with solutions of the subproblems with limited accuracy. While there are extensions of the convergence result that take these inexact solutions into account [22, Prop. 4.50], they require the subproblems to be solved with increasing accuracy. However, we found that the method generally converged even though these requirements were not met.
4
Inner Loop Optimization
The second subproblem (Alg. 1, step 4) is a projection on the constraint set, wk = ΠC (2uk − z k ), which requires one projection on the low-dimensional unit simplex ΔL per x ∈ Ω. These projections can be computed in a finite number of steps [24]. The first subproblem (step 3) is equivalent to 1 uk = argminu u − (z k + τ s)2 + (τ λ)T V (u), (11) 2 i.e. an extension to vector vector-valued u of the classical Rudin-Osher-Fatemi (ROF, TV-L2 ) problem with regularization parameter τ λ. Many methods have been suggested to solve the ROF problem, e.g. PDE, fixpoint, or interior point methods for primal [4, 25], dual [26, 27, 28], or mixed [29] formulations. We evaluate two approaches: First, we will formulate a particularly simple gradient projection method in the operator splitting framework, cf. [30]. This scheme was introduced in [27] and extended to the multidimensional case in [31] (see also [16]). The second approach is based on the fast half-quadratic method of Yang et al. [15]. −1 k Forward-backward approach. The optimality of step (z − k 3, τ k condition u) + s ∈ ∂σEλ (u), can be rewritten as u = τ z /τ + s − ΠEλ z /τ + s . To compute the projection ΠEλ , we use the dual representation, 1 1 2 2 ΠEλ (x) = argmin q − xΩ = Div argmin Div p − xΩ + δDλ (p) . (12) 2 p q∈Eλ 2
156
J. Lellmann et al.
Using a simple forward-backward splitting for the inner problem results in the (gradient projection) update rule pj+1 = ΠDλ p − νDiv (Div p − x) . The projection ΠDλ can be computed explicitly and is separable in x, while the inner part can be computed for all models independently. This opens up the method to parallelization. Convergence is guaranteed for ν < 2/Div Div (see e.g. [22,√Thm. 3.12]). Extending the argument in [26, Thm. 3.1], we find that div 4d. Accord1 ingly, we may set ν < 2d . In our experiments, we set ν = 0.95 2d to avoid numerical problems close to the theoretical maximum. Wrapping up, we have Algorithm 2. Inner loop, forward-backward approach 1: 2: 3: 4: 5:
k
x ← zτ + s, choose arbitrary p0 ∈ Rn×d×L repeat pj+1 = ΠDλ (pj − νDiv (Div p − x)) until pj+1 − pj ∞ δinner uk ← τ (x − Div pj+1 ).
Half-quadratic approach. While the forward-backward method is simple and easy to implement, its convergence speed is in practice not satisfactory. As an alternative, we tested an ROF specialization of the general multichannel image restoration method by Yang et al. [15]. Starting from (11), the problem is to find μ uk = argminu g(u) , g(u) := u − f 2 + T V (u) , (13) 2 where μ := τ1λ and f := z k + τ s. Using a half-quadratic approach [32, 33], Yang et al. derive the splitting/penalty formulation
μ β 2 (u, y) = argmin yx + yx − Gx u + u − f 2Ω . (14) 2 2 Ld nL yx ∈R ,x∈Ω,u∈R x∈Ω
The parameter β controls smoothing of the total variation; setting β n/(2ε) guarantees ε-suboptimality of the solution of the smoothed problem with respect to the original problem (for a derivation see [18]). Equation (14) can be solved using alternating minimization w.r.t. u and the auxiliary variables yx . The latter is highly parallelizable, as it boils down to n separate explicit operations: yxj+1 = max Gx u − β −1 , 0 (Gx u/Gx u) . (15) On the other hand, minimizing (14) for u amounts to solving μ Grad Grad + (μ/β)I(nL) uj+1 = Grad y j+1 + f, β for uj+1 , where y j+1 is a proper rearrangement of the yx .
(16)
Convex Multi-class Image Labeling by Simplex-Constrained Total Variation
157
Fig. 3. Results of the speed comparison between forward-backward (FB) and halfquadratic method (HQ) for the inner problem, applied to data from the first iteration of the outer problem (cf. Table 1). Left to right: Original input, FB with τ λ = 5, HQ with τ λ = 5, FB with τ λ = 20, HQ with τ λ = 20. Iteration counts were fixed at 80 resp. 300 to equalize the runtime for both approaches. For larger regularization parameter, the half-quadratic method outperforms the forward-backward approach as smoothness increases.
For periodic boundary conditions, Yang et al. solved (16) rapidly using FFT. In our case, Neumann boundary conditions and thus the Discrete Cosine Transform (DCT-2) [34] are appropriate. This requires 2L independent (parallelizable) individual DCTs which can be efficiently computed in O(n log n) each. By the alternating application of the above two steps, we can solve (14) for fixed β large enough for any required suboptimality bound. In practice, convergence can be sped up by starting with a small β and solving a sequence of problems for increasing β, warm-starting each with the solution for the previous problem. Given an arbitrary u0 ∈ RnL , the complete algorithm reads Algorithm 3. Inner loop, half-quadratic approach 1: while stopping criterium not satisfied do 2: compute y j+1 from (15) 3: compute uj+1 from y j+1 and (16), 4: possibly increase β 5: end while The stopping criteria can be based on the residual [15]. For our experiments, we set a fixed iteration count, as increasing β at each step turned out to lead to fastest convergence, and residua for different β are not comparable.
5
Experiments, Performance Evaluation
Inner Problem. We compared the half-quadratic approach to the conventional forward-backward method. The difficulty with the former lies in the choice of the update strategy for β. We chose a generalization of the exponential strategy outlined in [15]: Set β = βmin and update by multiplying with c := (βmax /βmin)1/K for some K until β = βmax . We made the following observations: – In order to rapidly minimize the objective function, it is best to use a continuation strategy, i.e. to increase β at each step, rather than spending time on solving (14) exactly for each β.
158
J. Lellmann et al.
– Increasing K generally improves the quality of the result. – For fixed βmax and K, there seems to be a unique optimal βmin that minimizes the final objective function value. With the continuation strategy and fixed βmax , we found the optimal βmin to usually lie in the range of 10−5 βmax to 10−3 βmax . Unfortunately, there seems to be a strong dependency on the choice of λ as well as the scale and complexity of s. We set βmin = 0.2 · 10−4 βmax , which worked well for our data. βmax was set at n/0.2 according to a suboptimality bound of ε = 0.1 (section 4). We compared the performance of the two methods in terms of the objective function value for fixed runtime of the optimized Matlab implementations (Fig. 3, Table 1). For larger τ λ,the half-quadratic method gives better results. For τ λ = 20, less than 10 iterations are required to reach the quality of 300 iterations of the forward-backward method, giving a speedup of about 4-5. However, finding the optimal parameter set is more involved than for the forward-backward method. Table 1. Run times t (in seconds), objective function values r and relative differences (rHQ − rFB )/rHQ for the experiment in Fig. 3. For larger τ λ, the half-quadratic method gives more accurate results in the same time. τλ 0.1 1 2 5 10 20 50 tHQ 1.14 1.23 1.20 1.31 0.98 0.95 1.08 tFB 1.03 1.02 1.06 1.03 1.22 1.25 1.19 rHQ 3901.9 27660.7 36778.5 40038.8 42262.8 44377.1 44752.5 rFB 3901.9 27660.4 36760.6 40104.3 42924.3 46988.6 57504.9 rel. diff. 1.17e-16 1.24e-5 4.85e-4 -1.64e-3 -0.0156 -0.0588 -0.285
Overall Problem. We evaluated the performance of our algorithm against five different methods in their publicly available implementations from the Middlebury MRF benchmark [35]: Belief Propagation (BP), Sequential Belief Propagation (BPS), Graph Cuts with alpha-expansion (GCE) and alpha-beta swap (GCS), and Sequential Tree Reweighted Belief Propagation (TRBPS). Of each of the grayscale 32 × 32 images, 20 noisy copies were generated and segmented into four gray levels with fixed intensities. In view of the last section and in order not to mix up speed with accuracy issues, we used the forward-backward approach for the inner loop. We set δinner = 1 · 10−3 , δouter = 2 · 10−2, and τ = 1. For small λ, our method shows results comparable to the other approaches with respect to the number of bad labels. We point out again that this solution to the non-binary labeling problem is achieved by solving the convex optimization problem (6) followed by local rounding as explained in section 2. In contrast to our method, the MRF benchmark algorithms optimize the anisotropic energy. To compensate, their λ was scaled by a common factor of √ ≈ 2 that was found empirically. Nevertheless, the discretization gives them a small advantage on images with axis parallel edges (experiments 1 and 2).
Convex Multi-class Image Labeling by Simplex-Constrained Total Variation
159
25
50 bp bps gce gcs trws tv
40 30
bp bps gce gcs trws tv
20 15 10
20 10 0 0
Standard deviation
Incorrect labels (mean %)
Fig. 4. Exemplary grayscale segmentation results for the benchmarked methods for four labels. Left to right: Noisy input data, final results for BP, BPS, GCE, GCS, TRWS, and the proposed method (TV). λ was manually chosen for each method. Axis-parallel edges are better recovered by the anisotropic methods, while our isotropic discretization has an advantage on diagonal edges.
5 0
0.2
0.4
0.6
λ
0.8
1
1.2
1.4
0
0.2
0.4
0.6
λ
0.8
1
1.2
1.4
Fig. 5. Error rates for the first experiment in Fig. 4. For each λ, all experiments were repeated 20 times with random noise (zero-mean Gaussian with σ = 0.45, 0.35, 0.25 resp. 0.35 for experiments 1-4 and image intensities in [0, 1]), and the percentage of incorrectly assigned labels compared to ground truth was recorded. Sequential Belief Propagation (BPS) generally performed worst, while our method (TV) was on par with the others, in particular for lower λ. The figure also reveals that belief propagation (BP) gets stuck in a good, but often inferior local optimum, and does not respond to larger values of λ, i.e. stronger regularization requested by the user.
Figure 6 demonstrates the performance of our algorithm for color segmentation. Only few outer iterations (20 in our case) are necessary for accurate optimization.
160
J. Lellmann et al.
Fig. 6. Performance of our method for four-class segmentation based on 1 color distance. Left to right: Ground truth, inspired by [29, 36]; ground truth overlaid with Gaussian noise, σ = 1; local nearest-neighbor labeling; our approach with λ = 0.7 after 20 outer iterations. The energy of the result is about 1% lower than the energy of the ground truth, suggesting that at this noise level, further improvements are limited by the model.
6
Conclusion and Future Work
In this paper, we presented a convex variational approach to solve the combinatorial multi-labeling problem for energies involving a general data term, total-variation-like regularizers, and simplex constraints. To enforce the simplex constraint, we based our approach on the globally convergent Douglas-Rachford operator splitting scheme. We evaluated two methods in order to efficiently solve the ROF-type subproblems, and showed that the half-quadratic approach allows faster convergence at the price of more involved parameter tuning. Experiments showed that the quality of the generated labelings is comparable to state of the art discrete optimization methods, and can be achieved by just solving a convex optimization problem. Due to the generality of the data term, our method allows for a wide range of features or distance measures. To fully evaluate these possibilities in connection with variations of the TV measure is a subject of our future research. Acknowledgements. Jing Yuan gratefully acknowledges support by the German National Science Foundation (DFG) under grant SCHN 457/9-1.
References 1. Boykov, Y., Veksler, O., Zabih, R.: Fast approximate energy minimization via graph cuts. PAMI 23(11), 1222–1239 (2001) 2. Boykov, Y., Kolmogorov, V.: An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision. PAMI 26(9), 1124–1137 (2004) 3. Kolmogorov, V., Zabih, R.: What energy functions can be minimized via graph cuts? PAMI 26(2), 147–159 (2004) 4. Rudin, L., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Physica D 60, 259–268 (1992) 5. Strang, G.: Maximal flow through a domain. Math. Prog. 26, 123–143 (1983)
Convex Multi-class Image Labeling by Simplex-Constrained Total Variation
161
6. Chan, T.F., Esedo¯ glu, S., Nikolova, M.: Algorithms for finding global minimizers of image segmentation and denoising models. J. Appl. Math. 66(5), 1632–1648 (2006) 7. Pock, T., Schönemann, T., Graber, G., Bischof, H., Cremers, D.: A convex formulation of continuous multi-label problems. In: ECCV, vol. 3, pp. 792–805 (2008) 8. Ishikawa, H.: Exact optimization for Markov random fields with convex priors. PAMI 25(10), 1333–1336 (2003) 9. Zach, C., Gallup, D., Frahm, J.M., Niethammer, M.: Fast global labeling for realtime stereo using multiple plane sweeps. In: VMV (2008) 10. Kleinberg, J., Tardos, E.: Approximation algorithms for classification problems with pairwise relationships: Metric labeling and MRFs. In: FOCS, pp. 14–23 (1999) 11. Ziemer, W.: Weakly Differentiable Functions. Springer, Heidelberg (1989) 12. Meyer, Y.: Oscillating Patterns in Image Processing and Nonlinear Evolution Equations. Univ. Lect. Series, vol. 22. AMS (2001) 13. Sapiro, G., Ringach, D.L.: Anisotropic diffusion of multi-valued images with applications to color filtering. Trans. Image Process. 5, 1582–1586 (1996) 14. Chan, T.F., Shen, J.: Image processing and analysis. SIAM, Philadelphia (2005) 15. Yang, J., Yin, W., Zhang, Y., Wang, Y.: A fast algorithm for edge-preserving variational multichannel image restoration. Tech. Rep. 08-09, Rice Univ. (2008) 16. Duval, V., Aujol, J.F., Vese, L.: A projected gradient algorithm for color image decomposition. CMLA Preprint (2008-21) (2008) 17. Chan, T., Esedoglu, S., Park, F., Yip, A.: Total variation image restoration: Overview and recent developments. In: The Handbook of Mathematical Models in Computer Vision. Springer, Heidelberg (2005) 18. Lellmann, J., Kappes, J., Yuan, J., Becker, F., Schnörr, C.: Convex multi-class image labeling by simplex-constrained total variation. TR, U. of Heidelberg (2008) 19. Rockafellar, R., Wets, R.J.B.: Variational Analysis, 2nd edn. Springer, Heidelberg (2004) 20. Douglas, J., Rachford, H.H.: On the numerical solution of heat conduction problems in two and three space variables. Trans. of the AMS 82(2), 421–439 (1956) 21. Lions, P.L., Mercier, B.: Splitting algorithms for the sum of two nonlinear operators. SIAM Journal on Numerical Analysis 16(6), 964–979 (1979) 22. Eckstein, J.: Splitting Methods for Monotone Operators with Application to Parallel Optimization. PhD thesis, MIT (1989) 23. Eckstein, J., Bertsekas, D.P.: On the Douglas-Rachford splitting method and the proximal point algorithm for max. mon. operators. M. Prog. 55, 293–318 (1992) 24. Michelot, C.: A finite algorithm for finding the projection of a point onto the canonical simplex of Rn . J. Optim. Theory and Appl. 50(1), 195–200 (1986) 25. Dobson, D.C., Curtis, Vogel, R.: Iterative methods for total variation denoising. J. Sci. Comput 17, 227–238 (1996) 26. Chambolle, A.: An algorithm for total variation minimization and applications. JMIV 20, 89–97 (2004) 27. Chambolle, A.: Total variation minimization and a class of binary MRF models. In: Rangarajan, A., Vemuri, B.C., Yuille, A.L. (eds.) EMMCVPR 2005. LNCS, vol. 3757, pp. 136–152. Springer, Heidelberg (2005) 28. Aujol, J.F.: Some algorithms for total variation based image restoration. CMLA Preprint (2008-05) (2008) 29. Chan, T.F., Golub, G.H., Mulet, P.: A nonlinear primal-dual method for total variation-based image restoration. J. Sci. Comput. 20, 1964–1977 (1999) 30. Combettes, P.L., Wajs, V.R.: Signal recovery by proximal forward-backward splitting. SIAM J. Multisc. Model. Sim. 4(4), 1168–1200 (2005)
162
J. Lellmann et al.
31. Bresson, X., Chan, T.: Fast minimization of the vectorial total variation norm and applications to color image processing. Tech. Rep. 07-25, UCLA (2007) 32. Geman, D., Yang, C.: Nonlinear image recovery with halfquadratic regularization. IEEE Trans. Image Proc. 4(7), 932–946 (1995) 33. Cohen, L.: Auxiliary variables and two-step iterative algorithms in computer vision problems. JMIV 6(1), 59–83 (1996) 34. Strang, G.: The discrete cosine transform. SIAM Review 41(1), 135–147 (1999) 35. Szeliski, R., Zabih, R., Scharstein, D., Veksler, O., Kolmogorov, V., Agarwala, A., Tappen, M., Rother, C.: A comparative study of energy minimization methods for Markov random fields. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3952, pp. 16–29. Springer, Heidelberg (2006) 36. Hintermüller, M., Stadler, G.: An infeasible primal-dual algorithm for total bounded variation-based inf-convolution-type image restoration. J. Sci. Comput. 28(1), 1–23 (2006)
Geodesically Linked Active Contours: Evolution Strategy Based on Minimal Paths Julien Mille and Laurent D. Cohen CEREMADE, UMR CNRS 7534, Université Paris IX-Dauphine Place du Maréchal de Lattre de Tassigny, 75016 Paris, France {mille,cohen}@ceremade.dauphine.fr
Abstract. The proposed method is related to parametric and geodesic active contours as well as minimal paths, in the context of image segmentation1 . Our geodesically linked active contour model consists in a set of vertices connected by paths of minimal cost. This makes up a closed piecewise defined curve, over which an edge or region energy functional is formulated. The greedy algorithm is used to move vertices towards a configuration minimizing the energy functional. This evolution technique ensures lower sensitivity to erroneous local minima than usual gradient descent of the energy. Our method intends to take advantage of explicit active contours, minimal paths and greedy evolution techniques.
1
Introduction
Among well known variational models for image segmentation, active contours have drawn lively interest since their introduction by Kass et al [1]. Their key principle is the research of a curve minimizing an energy functional, which mainly depends on the adequacy of the curve to the target object. Active contours are implemented either with a parametric curve - in which case they are often referred to as ’snakes’ - or in an implicit fashion based on the level set framework [2] [3]. Early active contour models are mainly parametric and boundary-based, as the data term of the energy functional is an edge indicator function integrated along the curve. The Euler-Lagrange equation, determined by calculus of variations, indicates the minimizing flow to be followed by gradient descent scheme. These models are dependent of curve parameterization and unable to adapt their topology. Moreover, gradient descent is sensitive to local minima of the energy functional. Parameterization invariance is achieved by the geodesic active contour model [4], which introduces a geometrically intrinsic functional, whereas topology adaptiveness is provided by the level set implementation. Significant attempts have been made to decrease the sensitivity to local minima, based either on the gradient descent direction or on the minimization method itself. The balloon force [5] falls into the first category, as it adds a normal-oriented inflation or retraction component, in order to increase the capture range of the snake. As regards the evolution process, several heuristics based 1
This work was partially supported by ANR grant NanoGPSCellulaire ANR-05NANO- 045-06.
X.-C. Tai et al. (Eds.): SSVM 2009, LNCS 5567, pp. 163–174, 2009. c Springer-Verlag Berlin Heidelberg 2009
164
J. Mille and L.D. Cohen
on local searches have been proposed as alternatives to gradient descent, including dynamic programming [6] [7] and the greedy algorithm [8] [9]. The latter, which is subsequently addressed in the paper, considers the energy as a sum of curve points energies. It basically consists in iteratively moving curve points to locations minimizing their own energies, these locations belonging to a search window. On the other hand, the minimal path approach by Cohen and Kimmel [10], which seeks for a curve of minimal cost between two end-points, can be used to recover open and closed boundaries. It is closely related to the geodesic active contour with respect to the functional to be optimized, but has in addition the main benefit of finding a global minimum efficiently thanks to the Fast Marching technique [11]. In this paper, we deal with an explicit implementation of active contour, i.e. a discrete curve defined by control points, or vertices. The described method is both related to minimal paths and greedy search. Our geodesically linked active contour model is made up of a set of vertices connected by paths of minimal cost with respect to a boundary-based metric. We define search windows centered at each vertex and evolve vertices according to a greedy fashion. Making a given vertex movable and the other ones still, we consider every geodesically linked contour passing through the points in the window of the moving vertex. This last one is finally moved to the location leading to the contour of smallest energy. The motivation for this work resides in several points. Firstly, the minimal path approach alone can only find a minimizer of an edge functional, with one or several(s) fixed input end-point(s). Conversely, our model is suitable to any energy functional, which we prove by endowing it with edge-based or different regionbased energies, including the minimal variance of the Chan and Vese model [12]. We believe that describing the curve with geodesics is pertinent whatever the energy functional is. Indeed, whether the functional holds edge, region and/or even shape prior terms, the major part of the final curve will be located on more or less salient edges. In comparison to snakes driven by gradient descent, the use of search windows significantly reduces sensitivity to erroneous local minima and energy weights tuning.
2 2.1
Background Parametric and Geodesic Active Contours
The active contour model is represented as a plane curve Γ with C 2 position vector c(u) = [x(u) y(u)]. Segmentation of an object of interest is performed by finding the curve minimizing an energy functional E, which has the general form: 1 E(Γ ) = L(c, c , c )du (1) 0
where L is usually made up of internal terms regularizing the curve and external terms attaching the curve to image data. According to calculus of variations, the following variational derivative vanishes if the curve is a local minimizer of E: δE ∂L d ∂L d2 ∂L = − + 2 (2) δΓ ∂c du ∂c du ∂c
Geodesically Linked Active Contours
165
Curve evolution is usually performed by gradient descent, taking the opposite variational derivative as a descent direction. Given an image I defined over D ∈ R2 , they use the following edge indicator g, which is a decreasing function of gradient magnitude (the image is usually convolved with the derivative of a gaussian). The original parametric snake [1] has the following energy and variational derivative: 1 2 2 Lsnake (c, c , c ) = α c + β c + g(c) 2 (3) δEsnake = −αc + βc + ∇g δΓ The energy functional of the snake is dependent on parameterization. This has an impact on discretization, since the energy varies in terms of sampling when the contour is implemented as a polygonal curve. The geodesic active contour (GAC) [4] solves the parameterization issue by introducing an intrinsic energy functional, weighting the edge indicator by length element c : LGAC (c, c , c ) = g(c) c (4)
δEGAC = (∇g, n − κg) n δΓ
where n and κ are the unit inward normal vector and curvature, respectively. Hence, the flow resulting from the geometric energy also holds a regularization term. This model lends itself to level set implementation, allowing topology changes. Boundary-based models driven by gradient descent, whether parametric or geodesic, are relatively blind to neighboring structures and may get trapped in local minima induced by noise. To increase the capture range, the balloon force was introduced in [5] for parametric contours, whereas an advection term is used in [3] for level sets. Despite such techniques, gradient descent may still cause the contour to miss or pass through significant boundaries. The minimal path method addresses this issue by finding a global minimum of the energy. 2.2
Minimal Paths
The minimal path approach by Cohen and Kimmel [10] aims at finding curves of minimal lengths in a Riemannian space endowed with an heterogeneous isotropic metric. The length of path C is: 1 1 L(C) = (5) P˜ (C(s))ds = P˜ (C(u)) C (u) du 0
0
where s is the arc length. Potential P˜ , which defines the isotropic metric, should be chosen according to the application. Curves located on image boundaries are detected by using an edge-dependent potential P˜ (x) = w + g(x), where w is a regularizing constant. Hence the cost of C may be rewritten using euclidean length: 1 1 L(C) = (w + g(C(s)))ds = wLeuclidean(C) + g(C(s))ds (6) 0
0
166
J. Mille and L.D. Cohen
With respect to the energy functional to be minimized, the minimal path approach is similar to the geodesic active contour model, as can be seen in term LGAC of eq. 4. However, the minimal path has the avantageous difference of reaching the global minimum of the energy, given two fixed end-points x0 and x1 . Starting from point x0 , the minimal action map U0 should be calculated. It corresponds to the minimal cost integrated along a path starting at x0 and ending at x: C(0) = x0 U0 (x) = inf L(C) s.t. C(1) = x C The action map U0 is the viscosity solution of the Eikonal equation ∇ U = P˜ with initial condition U (x0 ) = 0. This allows U0 to be computed by the Fast Marching method [11], which is similar in principle to Dijkstra’s graph search algorithm. Once the action map has been computed, the geodesic γ, i.e. the path of minimal action linking a point x1 to x0 , is found by back-propagation starting from x1 until x0 is reached: γ = −∇U0 (γ). In its initial formulation, the minimal path method determines an open curve between two fixed end-points. It is also able to find closed contours by providing only one point on the final contour and detecting a saddle point on the minimal action map [10]. 2.3
Greedy Algorithm
Along with dynamic programming [6] [7], greedy methods deal with discrete energy functionals. The greedy algorithm for active contours, as developed in [8], seeks for a minimizer of the energy by means of a set of local optimizations. It is only applicable on explicit implementations, where the contour is represented as a polygon with n vertices {vi }1≤i≤n . The total energy is considered as a sum of vertex energies: n E(Γ ) = Evertex (vi ) i=1
where Evertex is the discretization of the energy at a given vertex, using finite differences. Considering the snake term Lsnake of eq. 3, it comes: 1 2 2 α vi − vi−1 + β vi+1 − 2vi − vi−1 + g(vi ) (7) Evertex (vi ) = 2 Vertices are successively moved in order to minimize their own energies. At each iteration, a square window of width m is considered around the current vertex. ˜ i in the window. The energy of the latter is computed at each tested position v The vertex is then moved to the position leading to the lowest energy, which is summarized by the evolution scheme, at iteration t: (t+1)
vi
= arg min Evertex (˜ vi ) ˜ i ∈W vi(t) v
where W(x) is the window centered at point x. The initial greedy algorithm [8] performs in O(nm2 ) operations. The window size has an obvious impact on computational cost, but also on convergence abilities. Indeed, the contour can capture
Geodesically Linked Active Contours
167
farther structures as the window is larger. The greedy algorithm is by essence a discrete optimization heuristic. The formulation of the variational derivative is not used and continuous calculus of variations is thus not necessary.
3
The Geodesically Linked Active Contour
We develop an approach taking advantage of above described methods. Our geodesically linked active contour is based simultaneously on an explicit implementation of active contours, minimal paths and the greedy algorithm. Basically, we deal with an evolving explicit closed curve, allowing initialization inside or around the target object without providing fixed points. Minimal paths coupled with a geometric energy functional allows a parameterization-free handling of the contour. The use of the greedy algorithm, as opposed to gradient descent, guarentees better robustness to local minima. 3.1
Minimal Paths to Connect Vertices
Let us consider a set of n linked vertices S = {vi }1≤i≤n . We denote as γi (u) = [xi (u) yi (u)] the geodesic path connecting vi to vi+1 : C(0) = vi γi = arg min L(C) s.t. (8) C(1) = vi+1 C where the cost functional L is defined in eq. 5. At every step of the evolution algorithm, the set of geodesics {γi }1≤i≤n describes a closed piecewise differentiable contour Γ , which euclidean length is: Leuclidean(Γ ) =
n i=1
0
1
γi (u) du
One may note that a concatenation of geodesics γi is not a geodesic itself, since it is forced to pass through given points. To some extent, curve Γ may be considered as a piecewise minimizer of an edge-based functional. If a uniform potential P˜ (x) = 1 was chosen, the geodesics would become straight lines of equation γi (u) = (1 − u)vi + uvi+1 , u ∈ [0, 1], in which case Γ would represent a polygon. Fig. 1 depicts geodesically linked contours with uniform potential and edge-based potential (dark smooth lines represent high image gradient areas). As described in section 2.2, path γi is determined by gradient descent of the minimal action map Ui+1 of origin vi+1 . Given start point vi+1 , the Fast Marching algorithm [11] allows to specify one or several end points (in our case vi ) so that propagation can be stopped when vi is reached. This prevents the whole image from being visited by the Fast Marching and saves computational time. In the case of edge-based segmentation, the interest of describing the evolving contour with geodesics is obvious. Indeed, in the end of deformation, the
168
J. Mille and L.D. Cohen
(a)
(b)
Fig. 1. Vertices linked by geodesics with uniform potential (a) and edge-based potential (b)
geodesics fit the actual boundaries of the sought object. On the other hand, in the case of region-based segmentation, image edges are not explicitly searched. However, we believe that linking vertices with geodesics is relevant for any usual segmentation criterion. We may assume that the final contour should be partially located on more or less salient boundaries, whatever energy functional is optimized. In subsequent sections, we formulate three energies independently implemented on the geodesically linked active contour, namely the edge, region and narrow band region energies. Before, we recall Green’s theorem, which we use to convert domain integrals into boundary integrals. For every region R and real-valued function f over R2 , we have: f (x)dx = P dx + Qdy (9) R
∂R
where [P (x) Q(x)] is a continuously differentiable vector field such that: 1 x 1 y Q(x, y) = f (t, y)dt P (x, y) = − f (x, t)dt 2 −∞ 2 −∞
(10)
The theorem expects that ∂R should be at least piecewise smooth, it is thus applicable to the geodesically linked active contour. For instance, to express the area of region Rin enclosed by Γ , we consider eq. 9 with f (x) = 1: n 1 1 |Rin | = xi (u)yi (u) − xi (u)yi (u)du 2 i=1 0 3.2
Edge Energy
Boundary-based segmentation is performed by minimizing an edge energy. The edge indicator function g is integrated along geodesics. In order not to penalize lengthy contours, the edge energy is normalized by euclidean length: n 1 1 g(γi (u)) γi (u) du Eedge (Γ ) = Leuclidean(Γ ) i=1 0 Note that according to eq. 6, the integral of g along γi equals Ui+1 (vi ) minus the euclidean length Leuclidean(γi ). Hence, once the action maps have been computed, the edge indicator does not need to be summed over geodesics again. With the edge energy alone, if the search space of vertex coordinates is too small, the
Geodesically Linked Active Contours
169
contour fails at capturing actual boundaries when initialized far from them. To increase the capture range, we add an area-dependent term, which minimization acts like a balloon force [5]: Eballoon (Γ ) =
|D| − |Rin | |D|
where |D| is the image area. In that case, the total energy is a weighted sum of edge and balloon energies. 3.3
Region Energy
The increasing use of region terms has proven to overcome limitations of edgebased only models, especially when dealing with data sets suffering from noise and lack of contrast between neighboring structures. Classical region-based deformable models segment images according to statistical data computed over the object of interest and the background. Image partitions should be uniform in terms of pixel intensities or higher level features like texture descriptors. We rely on the intensity variance, which is close to the two-phase Mumford-Shah segmentation model by Chan and Vese [12]. The average intensity in the inner region is expressed using Green’s theorem: μ(Rin ) =
1 |Rin |
1 |Rin | i=1 n
I(x)dx = Rin
1
0
xi P (γi ) + yi Q(γi )du
where P and Q are the summed intensities (see template formulas in eq. 10). Then, the inner intensity variance is: 1 1 2 2 σ (Rin ) = (I(x) − μ(Rin )) dx = I 2 (x)dx − μ(Rin )2 |Rin | Rin |Rin | Rin where the integral of squared intensities may also be expanded according to Green’s theorem. Corresponding quantities on the outer region may be expressed using relation f (x)dx = Rout
3.4
D
f (x)dx −
f (x)dx Rin
Narrow Band Region Energy
The ideal case of uniform regions is rarely encountered in real applications, as the background usually contains structures of various intensities. Hence, strict homogeneity is not necessarily a desirable property. In order to account for spatially varying intensity, local statistics in region-based segmentation have emerged recently [13] [14]. The narrow band principle, which has proven its efficiency in the evolution of level sets [3], is used in our approach to formulate a local region term [15].
170
J. Mille and L.D. Cohen
Γ
Γ[−B]
Γ[B]
Bin Bout Fig. 2. Inner and outer bands for narrow band region energy
Instead of dealing with whole domains Rin and Rout , we consider an inner band Bin and an outer band Bout in the vicinity of the contour, as depicted in fig. 2. The narrow band region energy is the intensity variance over the bands: Eband (Γ ) = σ 2 (Bin ) + σ 2 (Bout ) Our narrow band region energy is based on parallel curves [16]. We define curve γ[B] i as a parallel curve of γi : γ[B] i (u) = γi (u) + Bni (u)
(11)
where B is the user-defined band thickness, constant along the curve, and ni is the inward unit normal to geodesic γi . Hereafter, we will use the index [B] to denote all quantities related to the parallel curve. Bands Bin and Bout are bounded by parallel curves of the n geodesics γi , respectively γ[B] i and γ[−B] i . We assume that geodesics are smooth enough so that their parallel curves do not self-intersect nor exhibit singularities. An important property resulting from the definition in eq. 11 is that the velocity vector of the parallel curve can be expressed as a function of the velocity vector of the initial curve, as well as its curvature and normal. Using the identity ni = − κi γi , we have: γ[B] i = γi + Bni = (1 − Bκi )γi
(12)
By a change of variable, an integral over inner band Bin may be expressed explicitly in terms of the curve and band thickness: n 1 B f (x)dx = f (γi + bni ) γi (1 − bκi ) db du (13) Bin
0
i=1
0
We use the template formula in eq. 13 to express the mean and variance of intensities in the inner band: n 1 1 B μ(Bin ) = I(γi + bni ) γi (1 − bκi ) db du |Bin | i=1 0 0 1 σ (Bin ) = |Bin | i=1 2
n
0
1
0
B
(I(γi + bni ) − μ(Bin ))2 γi (1 − bκi ) db du
and similarly for the outer band, replacing b with −b.
Geodesically Linked Active Contours
3.5
171
Evolution with Greedy Algorithm
Vertices should be moved in order to minimize the selected energy. This is usually performed with gradient descent of the Euler-Lagrange equation. In our case, it is difficult to differentiate Eedge , Eregion or Eband with respect to a given vertex vi , since these energies depend on geodesics to vi (see eq. (8)). The greedy algorithm presented in section 2.3 provides us a way to evolve vertices without differentiating the energy. Motion of curve points can always be decomposed into normal and tangential components. While the geometry of the curve is modified by normal displacements, tangential motion only affects curve parameterization [4]. Since the distribution of vertices along the contour can be updated with a resampling technique, we only consider normal displacement in the greedy evolution. We define a normal-oriented window WN of length m centered at vertex vi : m m
WN (vi ) = vi + knvi k ∈ − , 2 2 where nvi is the inward unit normal vector, estimated by finite difference using the second and next-to-last points of geodesics γi and γi+1 , respectively. Since steps between successive points in the window are integers, the window may be computed using a Bresenham-like algorithm.
γ˜i ˜i v
vi+1
vi
γ˜i−1
vi−1 Fig. 3. Geodesics linking neighboring vertices to points in search window
Greedy evolution is performed by moving vertex vi to the position in the window which corresponding geodesically linked contour has the smallest en˜ i belonging to the window. The associated ergy. Let us consider a test position v geodesics γ˜i−1 and γ˜i link it to the neighbors of vi , as depicted in fig. 3. The energy of the corresponding geodesically linked contour Γ˜ = {γ1 , ..., γi−2 , γ˜i−1 , γ˜i , γi+1 , ..., γn } is computed and compared to the energy of the initial contour Γ . All window points are tested in this way, so that the evolution scheme at iteration t is: (t+1)
vi
=
arg min
˜ i ∈WN v
(t) vi
E(Γ˜ )
172
J. Mille and L.D. Cohen
where E is one of the previously described energies. If we consider set H = {1, ..., i − 2} ∪ {i + 1, ...n} holding indices of geodesics not influenced by a modification on vi , all quantities involved in the energies are written with constant ˜ i . For instance, the area of the tested inner and variable parts with respect to v region is decomposed: 1 ˜ xj (u)yj (u) − xj (u)yj (u) du Rin =
j∈H
+ 0
0 1
x ˜i−1 y˜i−1
−
x ˜i−1 y˜i−1
du + 0
1
x ˜i y˜i − x ˜i y˜i du
˜ i need to be comThis implies that the part of energies invariant with respect to v puted only once, before moving vi . Finally, once all vertices have been treated, resampling may be performed to maintain consistent distribution of vertices along the curve.
4
Experiments
We tested the geodesically linked active contour with the three different energy configurations (edge+balloon, region and narrow band region). A comparison
Fig. 4. Segmentation of left ventricle: initialization (a) and final location (b) of the geodesically linked active contour, initialization (c) and final location (d) of the parametric contour
Geodesically Linked Active Contours
173
with a parametric snake endowed with the same energies is provided. The snake was initialized as a small circle inside the area of interest, far from the target boundaries. Similarly, the initial vertices of our model are sampled on a circle. Results are shown in fig. 4. For each row, columns (a) and (b) represents the initial and final states of the geodesically linked active contour, respectively. Columns (c) and (d) represent the same states for the snake. For all experiments, the regularization weight w was set to 0.25, which achieved sufficient regularization for all tested images. The size of the window was m = 50 and the maximal inter-vertex distance for resampling was set to 20. The image in row 1, which was segmented using the edge energy, depicts the gapclosing ability of the model. The geodesically linked active contour managed to pass through false edges and reach actual boundaries. Thanks to the large search window, it turned out to be rather unsensitive to balloon strength, as values for coefficient α in the range [0.1, 4] were suitable. On the other hand, the balloon coefficient has a strong influence on the gradient descent-driven parametric snake, which yields difficult parameter tuning. Actually, it was not possible to find a correct balloon weight allowing to jump false edges while stabilizing on real ones. The image in row 2, which was segmented using the region energy, depicts a similar phenomenon. The geodesically linked contour does not get trapped in small gaps in the region, which could present an interest for segmentation of partially occluded objects. Row 3 depicts a MRI of the heart left ventricle, which was used to put the narrow band region energy into application. The band thickness B is an important parameter. Apart from its impact on the algorithmic complexity - computing intensity means and variances on the bands takes at least O(nB) operations it controls the trade-off between local and global features around the object. If B = 1, the region energy is as local as an edge term. The main image property having an effect on the minimal band thickness is the edges sharpness. Indeed, the deformable curve needs a larger band as the boundaries of the target object are fuzzy. However, B = 10 was a suitable value in our experiments. Note that we depict the state of the parametric snake before self-collision. One may note that an unconstrained region-based level set method would also properly segment images in row 2 and 3. However, this remark should be moderated by the fact our model is dedicated to applications where topology preservation is needed.
5
Conclusion and Perspectives
We proposed the geodesically linked active contour model for image segmentation. The model lies on an explicitly implemented curve moved by an evolution method based on minimal paths and a greedy algorithm. Linking curve points with geodesics solves parameterization issues and allows the contour to fit the most salient boundaries at every step of deformation. Displacing vertices according to a greedy search ensured lower sensitivity to erroneous local minima than usual gradient descent of the energy. The model was endowed with edge and
174
J. Mille and L.D. Cohen
region energies and was validated on a few datasets. Further work may focus on developing an adaptive search window for greedy evolution. Currently, the window length is constant whatever the values of energies or the previous positions of vertices. We believe the algorithm could be improved by adapting the window length with respect to these properties, in order to avoid visiting positions that would not seemingly minimize the energy.
References 1. Kass, M., Witkin, A., Terzopoulos, D.: Snakes: active contour models. International Journal of Computer Vision 1(4), 321–331 (1988) 2. Osher, S., Sethian, J.: Fronts propagation with curvature-dependent speed: algorithms based on Hamilton-Jacobi formulations. Journal of Computational Physics 79, 12–49 (1988) 3. Malladi, R., Sethian, J., Vemuri, B.: Shape modeling with front propagation: a level set approach. IEEE Transactions on Pattern Analysis and Machine Intelligence 17(2), 158–175 (1995) 4. Caselles, V., Kimmel, R., Sapiro, G.: Geodesic active contours. International Journal of Computer Vision 22(1), 61–79 (1997) 5. Cohen, L.: On active contour models and balloons. Computer Vision, Graphics, and Image Processing: Image Understanding 53(2), 211–218 (1991) 6. Amini, A., Weymouth, T., Rain, R.: Using dynamic programming for solving variational problems in vision. IEEE Transactions on Pattern Analysis and Machine Intelligence 12(9), 855–867 (1990) 7. Geiger, D., Gupta, A., Luiz, A., Vlontzos, J.: Dynamic programming for detecting, tracking, and matching deformable contours. IEEE Transactions on Pattern Analysis and Machine Intelligence 17(3), 294–302 (1995) 8. Williams, D., Shah, M.: A fast algorithm for active contours and curvature estimation. Computer Vision, Graphics, and Image Processing: Image Understanding 55(1), 14–26 (1992) 9. Sakalli, M., Lam, K.M., Yan, H.: A faster converging snake algorithm to locate object boundaries. IEEE Transactions on Image Processing 15(5), 1182–1191 (2006) 10. Cohen, L., Kimmel, R.: Global minimum for active contour models: a minimal path approach. International Journal of Computer Vision 24(1), 57–78 (1997) 11. Sethian, J.: A fast marching level set method for monotonically advancing fronts. Proceedings of the National Academy of Science 93(4), 1591–1595 (1996) 12. Chan, T., Vese, L.: Active contours without edges. IEEE Transactions on Image Processing 10(2), 266–277 (2001) 13. Piovano, J., Rousson, M., Papadopoulo, T.: Efficient segmentation of piecewise smooth images. In: Sgallari, F., Murli, A., Paragios, N. (eds.) SSVM 2007. LNCS, vol. 4485, pp. 709–720. Springer, Heidelberg (2007) 14. Lankton, S., Tannenbaum, A.: Localizing region-based active contours. IEEE Transactions on Image Processing 17(11), 2029–2039 (2008) 15. Mille, J., Boné, R., Cohen, L.: Region-based 2D deformable generalized cylinder for narrow structures segmentation. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008, Part II. LNCS, vol. 5303, pp. 392–404. Springer, Heidelberg (2008) 16. Farouki, R., Neff, C.: Analytic properties of plane offset curves. Computer Aided Geometric Design 7(1-4), 83–99 (1990)
Validation of Watershed Regions by Scale-Space Statistics Tomoya Sakai and Atsushi Imiya Institute of Media and Information Technology, Chiba University, Japan {tsakai,imiya}@faculty.chiba-u.jp
Abstract. This paper shows a potential use of scale space for statistical validation of watershed regions of a greyscale image. The watershed segmentation has difficulty in distinguishing valid watershed regions associated with real structures of the image from invalid random regions due to background noise. In this paper, a hierarchy of watershed regions is established by following merging process of the regions in a Gaussian scale space. The distribution of annihilation scales (lives) of the regional minima is investigated to statistically judge the regions as being valid or not. Recursive validation using the hierarchy prevents oversegmentation due to the randomness.
1
Introduction
The aim of this study is to develop a statistical validation scheme for segmentation of a greyscale image. If we do not have a priori knowledge on the shapes or structures of objects in the image, topographic features of the greyscale image, and the watersheds in particular, are useful for unsupervised image segmentation. A well-known phenomenon in the watershed segmentation is oversegmentation, that is, producing a large number of undesired tiny regions. Since the undesired watershed regions are mainly caused by noise in the image, it is desirable to settle the oversegmentation problem by taking account of statistical properties of the randomness. There is a body of literature dealing with the oversegmentation problem of watersheds [1,2,3,4,5,6,7,8]. In the antecedent work, most schemes for preventing the oversegmentation attempt to hierarchically merge the oversegmented regions on the basis of similarity between adjacent regions measured by the MDL [3], colour distance [8], and so on. Diffusion-based multiscale image representations are also used for merging the regions [5, 6, 8], since the scale space theory [9,10,11,12,13,14,15] mathematically underpins topological relationships among the topographic features without a priori knowledge about them. The oversegmentation can be reduced by selecting levels in the hierarchy of regions, or by setting lower bounds to the scale above and below which the watersheds are valid and invalid, respectively. In this paper, we show that the scale-space treatment of the image is also useful for the statistical analysis of the random watershed regions. The validity X.-C. Tai et al. (Eds.): SSVM 2009, LNCS 5567, pp. 175–186, 2009. c Springer-Verlag Berlin Heidelberg 2009
176
T. Sakai and A. Imiya
of a watershed region can be quantified in terms of the statistical confidence of distinguishing it from invalid watershed regions due to randomness. We present a fully unsupervised watershed segmentation algorithm, in which the watershed regions are recursively validated according to their hierarchical relationships in the scale space.
2 2.1
Watershed Segmentation with Variable Scale Gaussian Scale Space
In the Gausian scale-space theory [9,10,11,12,14,15,16], a one-parameter family of nonnegative functions is derived from a d-dimensional greyscale image f (x), x ∈ Rd . f (x, σ) = G(x, σ) ∗ f (x) (1) Here, “∗” expresses d-dimensional convolution, and G(x, σ) is an isotropic Gaussian function with the scale σ. |x|2 1 (2) G(x, σ) = √ d exp − 2 2σ 2π σ d We redefine the d-dimensional greyscale image and its scale-space representation in the extended real scale and space as follows. Definition 1. A d-dimensional greyscale image is defined as a nonnegative d ¯ d with a finite net image intensity scalar function f (x), x ∈ R x∈R¯ d f (x)dx . ¯ d, R ¯ + ), is the convoDefinition 2. The scale-space image f (x, σ), (x, σ) ∈ (R lution of the greyscale image f (x) with the isotropic Gaussian kernel G(x, σ). ¯ d and R ¯ + denote the d-dimensional extended real space including a point Here, R at infinity and the extended real scale including an infinite scale, respectively. Although the domain of a greyscale image in practice is bounded within a limited area or volume, we embed such an image in the extended real scale space. The point at infinity will be theoretically used as a representative point of the background of the image in the watershed segmentation later. 2.2
Watershed Segmentation and Hierarchy of Regions
The watershed segmentation was derived from spatial partitioning on the basis of the drainage patterns of rainfall. As the topographic height map defines the boundaries of the catchment basins draining to the same lowest points, a twodimensional greyscale image defines the watershed boundary curves enclosing regions with local minima when we regard the image intensity as the topographic height. For a d-dimensional image, the entire space is partitioned by (d − 1)dimensional hypersurfaces into d-dimensional watershed regions. Each watershed region defined by a smooth function f (x) contains a unique local minimum, to
Validation of Watershed Regions by Scale-Space Statistics
177
which any point in the watershed region is connected by a gradient curve of f (x). In practice, the watershed segmentation of the gradient image |∇f (x)| is known to provide better intuitive partitions than that of the image f (x) itself [2, 5, 6, 8] because object boundaries in a scene may cause large spatial changes in the image intensity. Simple computation of the watersheds of the images results in oversegmentation caused by tiny and insignificant catchment basins. As suggested in the antecedent work [3, 5, 6, 8], hierarchical relationships among the watershed regions are of great help for merging the oversegmented regions. We employ the scale-space framework to derive the hierarchy because the scale-space axioms are acceptable in general cases where any prior information about the similarities among the unexpected watershed regions are not given. If we apply the gradient watershed segmentation to the image f (x, σ) with the variable scale σ, we can observe the evolution of the watersheds with respect to scale. The catastrophy theory applied to the gradient watershed segmentation in the Gaussian scale space [5] shows that the gradient watershed regions of f (x, σ) may be generically annihilated, merged, created and splitted with increasing scale σ. Therefore, hierarchical watershed segmentation using multiscale representation of the image [2, 6, 8] is essentially the extraction of the hierarchical relationships among the watershed regions in the scale space through the generic events. Since every watershed region is represented by its local minimum, the trajectories of the regional minima in scale space describe the relationships among the regions. For the purpose of validation of the regions, we derive the hierarchy from all the traceable regional minima from the finest scale along their trajectories in scale space. We trace the trajectories by local minimisation at every level of scale [16]. In an annihilation or merging event, two regional minima and a saddle between them are involved. We regard one of these two regional minima as a child of the resulting regional minimum after the event. We trace only one of two local minima after a creation or splitting event because we are interested in the hierarchy of the regions at the finest scale. Remark that the point at infinity is a local minimum which exists at any scale. The local minimum at infinity is the regional minimum of the image background because the rainfall in the background region is drained to this ideal point. The following algorithm RegionHierarchy traces every trajectory of the regional minimum from every pixel p ∈ P at σ = 0 until the regional minimum disappears or goes outside the image boundary toward the local minimum at infinity with increasing scale. RegionHierarchy(set of pixel centres P , image f (p ∈ P )) 1 let G be a graph with card(P ) + 1 nodes with the labels l = 0, . . . , N where l = 0 represents the point at infinity; 2 store σlt = ∞ in all nodes of G; 3 set σmax to be the size of the convex hull of P ; 4 σ := 0; 5 Q := P ; 6 while card(Q) = 1 or σ < σmax do
178
7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
T. Sakai and A. Imiya
Q := Q; σ := σ + Δσ, where Δσ is a small value compared with the space intervals of the points Q; for each q l ∈ Q do update q l by minimising |∇f (x, σ)|2 with q l as the initial position 1 ; if q l is outside the convex hull of P then connect the two nodes of G labelled 0 and l; end if end for let L be a list of labels corresponding to the points in Q; while card(L) = 1 do pop a label l from L; n := NearestNeighbour(L, l); if |q l − q n | < εσ, where q l , q n ∈ Q, and ε is the tolerance of minimisation then if |q l − q l | > |q n − q n |, where q l , q n ∈ Q then child := l and parent := n; else child := n and parent := l; end if connect the two nodes of G labelled parent and child; t set σchild := σ; remove q child from Q; end if end while end while return G.
The resulting graph G is a set of trees representing the hierarchy of the watershed regions of the gradient image. Any node in G represents a watershed region consisting of the pixels indicated by its subtree nodes. The annihilation or merging scale σ t is stored at the node in G corresponding to p. We utilise the bicubic spline interpolation [17] to seach for the local minimum with subpixel precision in Step 10. The function NearestNeighbour in Step 18 searches for the nearest point to pl in the set of points listed in L and returns its label. The annihilation or merging event is detected in Step 19, and one of the two regional minima with larger displacement is identified as the child in Step 20. Figure 1 shows an example of the trajectories of regional minima and the region hierarchy obtained by RegionHierarchy. Since the set of tree, G, expresses hierarchical relationships among the image pixels, any tree node with a scale σ > 0 represents a set of pixels consisting a watershed region.
1
It is trivial that the watershed regions of the gradient magnitude squared |∇f |2 are identical to those of the gradient magnitude |∇f |.
Validation of Watershed Regions by Scale-Space Statistics
179
σ
(a)
(b)
(c)
Fig. 1. Trajectories of regional minima and region hierarchy. (a) A noisy 96 × 96 image f (x) embeded in a dark background. (b) Gradient magnitude squared |∇f (x, σ = 20)|2 . The brighter the larger magnitude. (c) Trajectories of the regional minima in scale space. The thick curves (blue) are the parts of the trajectories for σ > 5. The thin straight lines (red) are the edges of G between the nodes with σ > 5.
2.3
Scale Selection Problem
We need a criterion to select the scales or the tree levels in hierarchy. One may expect that the watersheds of the image f (x, σ) at a small scale σ well approximates the boundary of true image regions. However, if noise spoils the fine structure of the image, the estimated watersheds at small scales are stochastic and experimentally less reproducible. The noise is suppressed at a large scale, but the watershed segmentation is poor in terms of detection ability and localisation: the edges of small watershed regions are smoothed out, and the boundary shapes of large regions are simplified. Since the randomness is the major cause of the oversegmentation problem in the watershed methods [1, 4, 5], the oversegmentation problem should be resolved in a statistical manner.
3 3.1
Validation of Watershed Regions Valid Watershed Regions
Generally, a greyscale image expresses spatial distribution of a measured physical quantity. The true image f true(x), which we want to measure and apply the watershed segmentation to, is inevitably spoiled by random noise through the measurement. Therefore, the actual image f (x) presents valid watersheds related to those of the true image f true (x) and invalid watersheds due to the randomness. Assertion 1. A valid watershed region of an observed gradient image |∇f (x)| is related to one of the watershed regions of the true gradient image |∇f true (x)|. Since the watershed regions are represented by the region minima, the image f (x) has the valid watershed regions of the gradient image |∇f (x)| iff the true gradient image |∇f true (x)| has corresponding local minima. Contrapositively,
180
T. Sakai and A. Imiya
iff |∇f true (x)| is a featureless image without any local minimum, then no valid watershed exists for any observation f (x), which should be considered as an image of the background only. This condition means that f true (x) = 0 everywhere ¯ d because of the Definition 1. Therefore, f (x) for f true (x) = 0, i.e., the in R noise image, produces only the invalid watershed regions. The valid watershed region must be statistically distinguishable from such invalid region. From this viewpoint, the validity of the watershed region is interpreted as the statistical confidence in rejecting the following null hypothesis. Null hypothesis H0 : The watershed region is that of the noise image. Alternative hypothesis H1 : The watershed region is not that of the noise image. The null hypothesis H0 is rejected if the regional minimum is distinguishable from that of the noise image using test statistics. 3.2
Life Distribution
An important fact is that the randomness of the image f (x, σ) is filtered out as the scale σ increases, and deterministic features of the image f (x) emerge at large scales. In other words, the deterministic features such as the valid watershed regions are established from coarse to fine. There presumably exists a critical lower bound of scale, above and below which the watersheds of f (x, σ) are valid and invalid, respectively. In order to observe how the valid regions survive until large scales against the scale-space filtering, we define the life of the watershed region. Definition 3. The life of the watershed region is defined as the annihilation scale σ t of the regional minimum. Let W be a distribution of the lives of the watershed regions of |∇f (x, σ)| for the image of random noise. If W can be parametrically modelled, a goodness-of-fit test can be performed under the null hypothesis H0 . That is, if an image f (x) is an observation of a true uniform image with noise, then the model of W fits the distribution of lives {σ t } of its watershed regions, and H0 for any watershed regions of f (x) is accepted. We investigate experimentally the life distribution W for the gradient watershed regions of a Gaussian white noise image as shown in Fig. 2(a). We averaged the frequencies of lives over one hundred noise images. We discard the lives of pixel points whose annihilations are detected in 0 < σ ≤ Δσ by RegionHierarchy because not all the pixel centres are the local minima. Figure 2(b) is the averaged histogram of life. The obtained life histogram shows an unimodal shape. This implies that there exists a scale where the merging of the regions most frequently occurs. The regional minima of the noise image are uniformly distributed random points, and the regions tend to merge with nearest regions. Therefore, we deduce that this unimodal property is associated with distribution of the nearest neighbour distances of random points. In fact, the nearest neighbour distance distribution has a unimodal shape (See appendix A). The scale of
Validation of Watershed Regions by Scale-Space Statistics
181
σ
Relative frequency (a)
(b)
Fig. 2. Noise image and the averaged life histogram for its gradient watershed regions. (a) The noise image has uncorrelated random pixel values. (b) The life histogram shows relative frequency of scale at which the regional minima of the gradient image are annihilated as Gaussian blurring of the noise image proceeds.
the mode can be used as a gauge of the density of invalid regions. The regional minima with significantly large values of life out of the unimodal distribution W can be identified to be valid, because such regional minima are distinguishable from the invalid regional minima of the noise image. 3.3
Recursive Validation
We can set a critical value of the scale to judge the watershed regions valid or invalid. Although the computation of such a critical scale requires the parametric model of the life distribution in the strict sense of statistics, the critical scale can be roughly evaluated by the peak and decaying form of the life histogram. If the image contains valid regions, the life histogram may be multimodal or may have a peak at a small scale relative to the outlying lives representative of the valid regions. According to our experimental result in Section 3.2, a regional minimum with a life which is more than six times greater than the peak can be considered to be valid with the statistical confidence level α > 99% under the assumption of uncorrelated Gaussian random pixel values of a two-dimensional image as the noise. We present an algorithm RegionDiscovery for discovery of the valid watershed regions. This algorithm recursively validates the regions in a top-down fashion using each tree T in G by RegionHierarchy. According to the hierarchy, any discovered region is split into subregions as long as they are valid. Each subregion is validated using the life histograms constructed from the lives stored in the subtrees of T corresponding to the subregion.
182
T. Sakai and A. Imiya
RegionDiscovery(tree T , set of valid regions V , significance level α) 1 let Σ be a set of life values stored in T except the root; 2 let s be the subroot node of T with the largest life value σmax ∈ Σ; 3 if IsMultimodal(Σ) or IsOutlier(σmax , Σ, α) then 4 RegionDiscovery(Subtree(T , s), V , α); 5 RegionDiscovery(T \Subtree(T , s), V , α); 6 else 7 push the region R := Pixels(T ) into V ; 8 end if. Here, the function IsMultimodal returns true if the histogram of Σ is not unimodal. IsOutlier returns true if the life σ t is greater than the critical αlevel of scale computed from the given set of lives Σ. Note that these functions discard the lives in 0 < σ ≤ Δσ. Subtree extracts the subtree with subroot node s from the tree T . Pixels returns a set of pixels whose labels are recorded at the nodes in the given tree. The following function, Watershed, executes our watershed segmentation algorithm for a given image f with a set of pixels P and a significance level α. It returns the set of valid watershed regions consisting of subsets of P . Watershed(set of pixel centres P , image f , significance level α) 1 set V := ∅; 2 G := RegionHierarchy(P, f ); 3 for each tree T in G do 4 RegionDiscovery(T , V , α); 5 end for 6 return V .
scale 30 25 20 15 10 5
(a)
(b)
(c)
Fig. 3. An example of our watershed segmentation of noisy image. (a) Original image with 20% noise. (b) Trajectories of local minima of the gradient magnitude of (a) in scale space. The trajectories reaching out of the spatial domain are subordinate to a local minimum at infinity. (c) Watershed regions of the gradient magnitude by the algorithm Watershed. The brightness indicates the order of lives.
Validation of Watershed Regions by Scale-Space Statistics
4
183
Test Example
We demonstrate our gradient watershed segmentation Watershed for a noisy greyscale image. The purpose of this section is not to test the performance of the algorithm, but to show that the statistics in scale space has potential to discover the valid watershed regions without any prior information about them. Figure 3(a) shows a 128 × 128 test image f (x) with 20% additive noise [18]. The trajectories of local minima of |f (x, σ)| traced from σ = 0 in scale space are shown in Fig. 3(b). We see a large number of local minima created by the noise at small scales. As the scale increases, the local minima are hierarchically grouped and representative local minima survive at larger scales. Figure 3(c) shows the segmentation result with a confidence level α = 99% for f (x). There are nine
σ=2
σ=6
σ = 12
Fig. 4. Watershed segmentation of Fig. 3(a) at different scales. First row: the scalespace image f (x, σ). Second row: the gradient magnitude |∇f (x, σ)|. Third row: the watersheds of |∇f (x, σ)|. Each column corresponds to the same scale indicated below.
184
T. Sakai and A. Imiya
discovered regions clearly corresponding to the major regions of the original image. The tiny faults in the regions were caused by failure in the minimisation. They were wrongly assigned to the image background, which should be fixed in the future work. For the comparision purpose, we show in Fig. 4 the simple watershed segmentation results at a few levels of scale without using the region hierarchy or statistics in scale space. We see invalid small regions at small scales while the shapes of valid regions at large scales are distorted. It is remarkable that structural and statistical analyses using scale space can reconstruct the precise edges of statistically valid watershed regions despite the significant noise.
5
Concluding Remarks
The scale-space treatment of the image clarifies not only the hierarchical relationships among the watershed regions but also their statistical properties. We can observe in the Gaussian scale space how the random features are suppressed and deterministic features emerge as the scale grows. A valid watershed region must be statistically distinguishable from unreproducible regions caused by the random features. The reproducibility is a desirable ability of image recognition techniques. On the basis of this simple requirement we described the null hypothesis H0 , which is to be rejected if the watershed region is valid. A watershed region is recognised as valid at a statistical confidence level in rejecting H0 . We presented a validation scheme for watershed segmentation using statistics in scale space. We defined the life of a watershed region, whose distribution is useful for testing H0 . We showed that the life distribution for the noise image is unimodal, and the valid regions can be identified by the regional minima with significantly large values of lives out of the unimodal distribution. The statistical properties of the life and the region hierarchy enable the recursive validation of the watershed regions. A distinctive feature of our scheme is that it does not require any definition of similarity or dissimilarity measures between watershed regions, which is used in many methods for preventing oversegmentation. Instead, we focused on the statistical differences between the valid and invalid regions in scale space. In order to take advantage of the potential of scale-space statistics, our scheme requires further investigation, especially in relation to the model of the life distribution, and improvement and acceleration of the algorithms to obtain feasible segmentation results for larger size real images.
References 1. Vincent, L., Soille, P.: Watersheds in digital spaces: An efficient algorithm based on immersion simulations. IEEE Trans. on Pattern Analysis and Machine Intelligence 13(6), 583–598 (1991)
Validation of Watershed Regions by Scale-Space Statistics
185
2. Beucher, S.: Watershed, hierarchical segmentation and waterfall algorithm. In: Proc. Math. Morphology and Its Appl. to Image Processing, pp. 69–76 (1994) 3. Maes, F., Vandermeulen, D., Suetens, P., Marchal, G.: Computer-aided interactive object delineation using an intelligent paintbrush technique. In: Ayache, N. (ed.) CVRMed 1995. LNCS, vol. 905, pp. 77–83. Springer, Heidelberg (1995) 4. Hagyard, D., Razaz, M., Atkin, P.: Analysis of watershed algorithms for greyscale images. In: Proc. of IEEE Intl. Conf. Image Procesing, vol. 3, pp. 41–44 (1996) 5. Olsen, O.F., Nielsen, M.: Generic events for the gradient squared with application to multi-scale segmentation. In: ter Haar Romeny, B.M., Florack, L.M.J., Viergever, M.A. (eds.) Scale-Space 1997. LNCS, vol. 1252, pp. 101–112. Springer, Heidelberg (1997) 6. Gauch, J.M.: Image segmentation and analysis via multiscale gradient watershed hierarchies. IEEE Trans. on Image Processing 8(1), 69–79 (1999) 7. Roerdink, J.B.T.M., Meijster, A.: The watershed transform: definitions, algorithms, and parallelization strategies. Fundamenta Informaticae 41, 187–228 (2001) 8. Vanhamel, I., Pratikakis, I., Sahli, H.: Multiscale gradient watersheds of color Images. IEEE Trans. on Image Processing 12(6), 617–626 (2003) 9. Witkin, A.P.: Scale space filtering. In: Proc. of 8th IJCAI, pp. 1019–1022 (1986) 10. Koenderink, J.J.: The structure of images. Biological Cybernetics 50, 363–370 (1984) 11. Lindeberg, T.: Scale-Space Theory in Computer Vision. Kluwer, Boston (1994) 12. Weickert, J., Ishikawa, S., Imiya, A.: Linear Scale-Space has First been Proposed in Japan. Journal of Mathematical Imaging and Vision 10, 237–252 (1999) 13. Lifshitz, L.M., Pizer, S.M.: A multiresolution hierarchical approach to image segmentation based on intensity extrema. IEEE Trans. on Pattern Analysis and Machine Intelligence 12(6), 529–540 (1990) 14. Florack, L.M.J., Kuijper, A.: The topological structure of scale-space images. Journal of Mathematical Imaging and Vision 12(1), 65–79 (2000) 15. Kuijper, A.: The deep structure of Gaussian scale-space images. PhD thesis, Utrecht University (2002) 16. Sakai, T., Imiya, A.: Gradient structure of image in scale space. Journal of Mathematical Imaging and Vision 28(3), 243–257 (2007) 17. Keys, R.: Cubic convolution interpolation for digital image processing. IEEE Trans. on Acoustics, Speech, and Signal Processing 29(6), 1153–1160 (1981) 18. SAMPL database, http://sampl.ece.ohio-state.edu/database.htm 19. Suwa, N.: Quantitative morphology: stereology for biologists. Iwanami Shoten (1977) (in Japanese)
A
Distribution of Nearest Neighbour Distances
We present a proof that the nearest neighbour distances obey the Weibull distribution if the points in Rd are uniformly distributed in a Poisson arrangement [19]. The Poisson arrangement is defined as the uniformly random distribution of points with constant density ρ such that the number of points x in a fixed volume V follows the Poission distribution. Po(x; λ) =
λx exp(−λ) x!
(3)
186
T. Sakai and A. Imiya
Here, λ = ρV is the expected number of points in the volume V . Let r be the distance from an arbitrary point. The distribution of the nearest neighbour distances, p(r), can be regarded as the probability that the nearest neighbour is found in an infinitesimal gap between r and r + δr. This is the case that no points are found within the distance r, and at least one point is found between r and r + δr. Since the volume Vd of a unit d-ball and its surface area Sd−1 has a relationship Vd d = Sd−1 , we have p(r)δr = Po(0; ρVd rd ) 1 − Po(0; ρSd−1 rd−1 δr) ≈ exp(−ρVd rd ) 1 − exp(ρSd−1 rd−1 δr) = exp(−ρVd rd ) · ρSd−1 rd−1 δr = exp(−ρVd rd ) · ρVd drd−1 δr √ Letting s = 1/d ρVd be the scale of the average volume of d-dimensional hypercube per point, we obtain the Weibull distribution
d r d−1 r d p(r; s, d) = (4) exp − s s s where s and d correspond to the so-called scale and shape parameters of the Weibull distribution, respectively. This distribution p(r; s, d) has a mode at r = s d (d − 1)/d. For a fixed dimensionality, the mode depends only on the scale parameter s, which enables us to calculate the point density ρ from the mode.
Adaptation of Eikonal Equation over Weighted Graph Vinh-Thong Ta, Abderrahim Elmoataz, and Olivier Lézoray Université de Caen Basse-Normandie, GREYC CNRS UMR 6072, Image Team {vinhthong.ta,abderrahim.elmoataz-billah,olivier.lezoray}@unicaen.fr http://www.info.unicaen.fr/˜vta
Abstract. In this paper, an adaptation of the eikonal equation is proposed by considering the latter on weighted graphs of arbitrary structure. This novel approach is based on a family of discrete morphological local and nonlocal gradients expressed by partial difference equations (PdEs). Our formulation of the eikonal equation on weighted graphs generalizes local and nonlocal configurations in the context of image processing and extends this equation for the processing of any unorganized high dimensional discrete data that can be represented by a graph. Our approach leads to a unified formulation for image segmentation and high dimensional irregular data processing.
1
Introduction
Solutions of the nonlinear eikonal equation have found numerous applications. One can quote for instance, geometric optics, image analysis or computer vision including shape from shading [1, 2], median axis or skeleton extraction [3], topographic segmentation (watershed) [4] or geodesic distance computation on discrete and parametric surfaces [5, 6, 7, 8, 9]. The latter works consider both structured and unstructured meshes on cartesian or non-cartesian domains. The eikonal equation is a special case of the following general continuous Hamilton-Jabobi equation: H(x, f, ∇f ) = 0 x∈Ω ⊂ IRn , (1) f (x) = φ(x) x∈Γ ⊂ Ω where φ in the boundary condition is a positive speed function defined on Ω and f (x) is the traveling time or distance from source Γ . Then, the eikonal equation can be expressed by using the following Hamiltonian: H(x, f, ∇f ) = ∇f (x) − P (x),
(2)
where P (x) is a given potential function. Solution of (1) represents the shortest distance from x to the zero distance curve given by Γ (where φ(x)=0). Solutions of (2) are usually based on a discretization of the Hamiltonian where the approximation of the derivatives is performed by the Godunov [10] or the X.-C. Tai et al. (Eds.): SSVM 2009, LNCS 5567, pp. 187–199, 2009. c Springer-Verlag Berlin Heidelberg 2009
188
V.-T. Ta, A. Elmoataz, and O. Lézoray
Lax-Friedrich [11] schemes. Then, many numerical methods have been proposed and investigated to solve the nonlinear system described by (2). For instance, one can quote the following schemes. (i) An iterative scheme [1] relying on fixed point methods that solves a quadratic equation was proposed. (ii) The fast sweeping methods [12] that use Gauss-Seidel type of iterations to update the distance function field. The key point of fast sweeping is to update the points in a certain order. (iii) Tsitsiklis [13] was the first to develop a Dijkstra like method and proposed an optimal algorithm for solving the eikonal equation. Based on this idea, [14, 11] produced the fast marching methods. Another approach to solve (2) is to consider a time dependent version of the equation and to evolve it to the steady state. Then, (2) can be rewritten as ⎧ n ⎪ ⎨∂f (x, t)/∂t = −∇f (x) + P (x) x∈Ω ⊂ IR (3) f (x, t) = φ(x) x∈Γ ⊂ IRn . ⎪ ⎩ f (x, 0) = φ0 (x) x∈Ω This paper only considers the discrete analogue of the time dependent formulation of the eikonal equation but, in future works, the stationary case (time independent) will be also considered. Contributions. In this work, we propose an adaptation of (3) over weighted graphs of the arbitrary structure. The goal here is to provide a simple and common formulation that solves the eikonal equation for any discrete data that can be represented by a weighted graph such as images or high dimensional data defined on irregular domains. This alternative formulation for solving the eikonal equation is based on partial difference equations (PdEs) and discrete gradients over weighted graphs. Our formulation has several advantages. Any discrete domain that can be described by a graph can be considered without any spatial discretization. In the context of image processing, local and nonlocal configurations are directly enabled within a same formulation. Finally, the aim of this paper is not to solve a particular application with the eikonal equation but to show the potentialities of our proposition to address image segmentation, data clustering or distance computation. Paper Organization. The paper is organized as follows. Section 2 recalls basics, definitions and operators on weighted graphs. Section 3 introduces our formulation for solving the eikonal equation. Section 4 shows the potentialities of our proposition for the segmentation of images and unorganized data processing. Finally, last Section concludes.
2
Discrete Derivatives on Weighted Graphs
This Section recalls basics, definitions, operators and processes on weighted graphs.
Adaptation of Eikonal Equation over Weighted Graph
2.1
189
Definitions and Weighted Graphs Construction
Notations and Definitions. We consider the general situation where any discrete domain can be viewed as a weighted graph. Let G=(V, E, w) be a weighted graph composed of two finites sets: vertices V and weighted edges E⊆V ×V . An edge (u, v)∈E connects two adjacent (neighbor) vertices u and v . The neighborhood of a vertex u is noted N (u)={v∈V \{u} : (u, v)∈E}. The weight ωuv of an edge (u, v) can be defined with a function w:V ×V →IR+ such that w(u, v)=ωuv if (u, v)∈E and w(u, v)=0 otherwise. Graphs are assumed to be simple, connected and undirected implying that function w is symmetric. Let f :V →IR be a discrete real-valued function that assigns a real value f (u) to each vertex u∈V . We denote by H(V ) the Hilbert space of such functions defined on V . Weighted Graphs Construction. Any discrete domain can be represented by a weighted graph where functions of H(V ) represents the data to process. In the general case, an unorganized set of points V ⊂IRn can be seen as a function f 0 :V ⊂IRn →IRn . Then, constructing a graph from this data consists in defining the set of edges E by modeling the neighborhood. It is based on a similarity relationship between data with a pairwise distance measure μ:V ×V →IR+ . There exists several methods to transform a set of vertices V into a neighborhood (similarity) graph (see [15] for a survey on proximity and neighborhood graphs). In this paper, we focus on two particular graphs: the τ -neighborhood graphs and a modified version of k-nearest neighbors graphs. The k nearest neighbors graph, noted k-NNG is a weighted graph where each vertex u∈V is connected to its k nearest neighbors which have the smallest distance measure towards u according to function μ. Since this graph is directed, a modified version of this graph is used to make it undirected. The τ -neighborhood graph, noted Gτ is a weighted graph where the τ -neighborhood Nτ for a given vertex u∈V is defined as Nτ (u)={v∈V \{u} : μ(u, v)≤τ } with τ >0 a threshold parameter. 2D images can be viewed as functions f 0 :V ⊂ZZ 2 →IRn . In this case, the associated distance μ for construct the neighborhood graph is usually the city block or the Chebychev distances computed with the spatial coordinates of each vertex representing an image pixel. With these distances and the τ -neighborhood graphs, one recovers the two usual graphs used in image processing, the 4-adjacency grid graph (denoted G0 with the city block distance) and the 8-adjacency grid graph (denoted G1 with the Chebychev distance) with τ ≤1. Another useful graph structure in image processing is the region adjacency graph (RAG) where vertices correspond to image regions, and the set of edges is obtained by considering an adjacency distance. With the τ -neighborhood (τ =1), the RAG is the Delaunay graph of an image partition. Weights Computation. Similarities between data can be incorporated within edges’ weights according to a measure of similarity g:E→IR+ that satsfies w(u, v)=g(u, v) for (u, v)∈E. Then, the distance computation between data is performed by comparing their features that generally depend on a given initial function f 0 ∈H(V ). To this aim, each vertex u∈V is assigned with a feature vector F (f 0 , u)∈IRm . With F , the following weight functions can be considered. For
190
V.-T. Ta, A. Elmoataz, and O. Lézoray
a given edge (u, v)∈E and a distance measure ρ:V ×V →IR+ associated to F , we can have g0 (u, v) = 1 (constant weight case) , g1 (u, v) = (ρ(F (f 0 , u), F (f 0 , v)) + )−1 with >0, →0, g2 (u, v) = exp(−ρ(F (f 0 , u), F (f 0 , v))2 /σ 2 ) with σ>0, where σ controls the similarity and ρ is usually the euclidean distance. Several choices for the expression of F can be considered depending on the features to preserve. The simplest one is F (f 0 , .)=f 0 . In the context of image processing, an important feature vector F is provided by image patches, i.e., F (f 0 , u)=Fτ (f 0 , u)={f 0 (v) : v∈Nτ (u) ∪ {u}}. In the case of a grayscale image Fτ (f 0 , .) is a vector of size (2τ +1)2 corresponding to the values of f 0 in a square window of size (2τ +1)×(2τ +1) centered at vertex u (a pixel). Color images can be handled using features of dimension 3×(2τ +1)2 . Then, the resultant weight function directly incorporates local or nonlocal features [16]. This feature vector has been proposed in the context of texture synthesis [17], and further used in the context of image processing [18,19]. 2.2
Graph Based Discrete Gradients
Let G=(V, E, w) be a weighted graph. The discrete weighted gradient of a function f ∈H(V ) at a vertex u∈V is defined by
(∇w f )(u) = (∂v f (u))(u,v)∈E
where ∂v f (u)= w(u, v)(f (v)−f (u)) corresponds to the discrete (partial) derivative of f with respect to the edge (u, v). These definitions have been used by [20] for image and mesh regularization. Based on the latter works, two discrete formulations of weighted morphological gradients on graphs have been proposed − by [21]: namely, the weighted external ∇+ w and the internal ∇w gradient operators. For u∈V + (∇+ w f )(u) = (∂v f (u))(u,v)∈E
and
− (∇− w f )(u) = (∂v f (u))(u,v)∈E ,
(4)
where the external ∂v+ f (u) and the internal ∂v− f (u) discrete partial derivatives are ∂v+ f (u) = max(0, ∂v f (u)) and ∂v− f (u) = − min(0, ∂v f (u)), with ∂v− f (u)=∂u+ f (v). When the weight is constant (w=g0 ) these definitions recover the classical directional derivative operators. The Lp -norm (with 0
1/p w(u, v)p/2 |(f (v)−f (u))± |p and (5) (∇± w f )(u)p = v∼u
1/2 |(f (v)−f (u))± | . (∇± w f )(u)∞ = max w(u, v) v∼u
(6)
Adaptation of Eikonal Equation over Weighted Graph
191
Notation v∼u means that vertex v is adjacent to u. ∇± w refers to both external and internal gradient (with respect to the sign) and (a)+ = max(0, a) and (a)− = min(0, a). These gradients have the following property: p − p (∇w f )(u)pp = (∇+ w f )(u)p + (∇w f )(u)p
with 0
3
Eikonal Equation on Weighted Graphs
In this Section, we present our formulation to approximate the eikonal equation (3) over weighted graphs by considering PdEs and the morphological gradients presented in the previous Section. With morphological processes described by (8), the time dependent eikonal formulation (3) can be viewed as an erosion process regarding the minus sign and a null potential function P . With the corresponding internal gradient (∇− w) involved in discrete PdEs based erosion process, (3) can be directly rewritten with weighted graphs. Given a graph G=(V, E, w) and a function f ∈H(V ), we obtain a discrete PdEs based version of the system (3) ⎧ − ⎪ ⎨∂f (u, t)/∂t = −∇w f (u)p + P (u) u∈V f (u, t)=φ(u) u∈V0 ⊂ V , ⎪ ⎩ f (u, 0)=φ0 (u) u∈V where V0 corresponds to the initial seed vertices. With f n (u) ≈ f (u, nΔt), this iterative numerical scheme is obtained for all u∈V : n f n+1 (u)=f n (u) − Δt (∇− (9) w f )(u)p − P (u) .
192
V.-T. Ta, A. Elmoataz, and O. Lézoray
The steady state (i.e. given a fixed number n of iteration or when f n+1 −f n < ) of this process is the solution of the eikonal equation (2). Injecting the corresponding internal gradient norm in (9), we obtain for the Lp -norm (5) and the L∞ -norm (6)
1/p f n+1 (u)=f n (u) − Δt w(u, v)p/2 | min(0, f (v)−f (u))|p − P (u) , (10) v∼u
f
n+1
(u)=f (u) − Δt max w(u, v)1/2 | min(0, f (v)−f (u))| − P (u) . n
v∼u
(11)
The proposed methodology leads to a simple and common formulation that constitutes an adaptative framework for the eikonal equation. Indeed, our approach only depends on the p value and the weight function w. In Sect. 4, experiments show how the framework can be adapted to address image segmentation or data clustering. Relations with other schemes. Scheme (9) has the advantage to work on any graph structures. Then, with an adapted graph topology and an appropriated weight function, the proposed formulation is linked to well-known schemes such as Osher-Sethian Hamiltonian discretization scheme or the graph based Dikjstra algorithm. Osher-Sethian scheme. Let G0 =(V, E, g0 ) be an unweighted 4-adjacency grid graph associated with an image. Then, (10) recovers the exact Osher-Sethian upwind first order Hamiltonian discretization scheme [14] when p=2 and using G0 :
1/2 f n+1 (u)=f n (u) − Δt | min(0, f (v)−f (u))|2 − P (u) . v∼u
Replacing vertices u∈V and their neighborhood by their spatial coordinates (x, y), the latter expression can be rewritten as 2 f n+1 ((x, y))=f n ((x, y))−Δt | min 0, f n ((x, y))−f n ((x−1, y)) 2 +| max 0, f n ((x+1, y))−f n ((x, y)) 2 +| min 0, f n ((x, y))−f n ((x, y−1))
2 1/2 +| max 0, f n ((x, y+1))−f n ((x, y)) − P ((x, y)) , since min(0, a−b)2 = max(0, b−a)2 . This equation corresponds to the discretization scheme of the Hamilton-Jacobi equations proposed by [14]. Dikjstra scheme. Let G=(V, E, g0 ) be an unweighted graph. Then, (11) corresponds to an iterative version of the Dikjstra shortest path algorithm defined on graphs of arbitrary structure. Indeed, in the case where p=∞, Δt=1 and with G, (11) becomes, for all u∈V f n+1 (u) = f n (u) − max | min(0, f (v)−f (u))| + P (u) = min(f n (v)) + P (u), v∼u
v∼u
Adaptation of Eikonal Equation over Weighted Graph
193
by considering the neighborhood of u as the set N (u)∪{u} and with the properties that max(0, a−b)= − min(0, b−a) and min(0, a−b)= min(a, b)−b. This equation corresponds to a shortest path algorithm for a given graph where at each step, the distance f (u) at vertex u corresponds to the minimal distance in its neighborhood.
4
Experiments
The proposed formulation of the eikonal equation and can be used to process any function defined on vertices of a graph or on any arbitrary discrete domain. This Section illustrates the potentialities of our formulation through examples of weighted distance computation, image segmentation and unorganized high dimensional data processing. Different graph structures and weight functions are also used to show the flexibility of our approach. In the sequel, all experiments are obtained with a constant potential function P =1. Clearly, a different potential function can be adapted for a particular application. The objective of the following experiments is not to solve a particular application. They only illustrate the potential and the behavior of our eikonal equation formulation. Adaptative Front Propagation and Weighted Distances. Figure 1 shows the adaptivity of our formulation in order to compute weighted distances. Indeed, this example shows results for different p values, graph topologies, weight functions and features F . The initial seed is located at the top left corner of the original grayscale image f 0 :H(V )→IR. First, second and third rows of Fig. 1 show results for p=2, 1 and ∞ respectively, where (10) and (11) are used. All the results correspond to color distance maps (red for small and blue for large distances) where iso-levels sets are superimposed in white. First and second columns of Fig. 1 show results obtained with unweighted (w=g0 ) graphs. First column uses a 4-adjacency grid graph (G0 ) and corresponds to the classical case. Second column uses a 25-adjacency grid graph (G2 ) and shows the effect of a larger neighborhood. Third and fourth columns show results obtained with weighted graphs. Third column considers graph G0 weighted by function g2 with F =f 0 . By using non constant weights, image information is automatically integrated in the distance computation that modifies the front evolution speed particularly into the textured sub-image. Fourth column shows the nonlocal case where graph G2 is constructed and weighted with function g2 associated with patches of size 11×11. In that case, repetitive information are clearly captured by the weights that stops the front propagation around the textured sub-image. Finally, segmentation of the textured sub-image can be simply obtained by thresholding the computed distances. Image Segmentation with Region Based Graphs. The goal of the following two examples is not to show a perfect segmentation but to show how we can take advantage of graph topologies in image segmentation. The basic idea is to consider that image pixels are not the only relevant components in image
194
V.-T. Ta, A. Elmoataz, and O. Lézoray
Original f 0 :H(v)→IR
G0 , w=g0
G2 , w=g0
G0 , w=g2 F =f 0
G2 , w=g2 F =F5 (f 0 , .)
p=1
p=2
p=∞
Fig. 1. Front propagation and weighted distances with different p values, graph configurations G, weights w and features F . Figures represent color distance maps with iso-level sets obtained by thresholding the distances. The seed is located at the top left corner (see text for more details).
and more abstract elements such as image regions can be used. Hence, we suggest to work directly with a reduced version of images: image partitions. Image partitions can be obtained by image pre-processing methods such as watershed. Figures 2(b) and 3(b) show such partitions computed from Figs. 2(a) and 3(a). Figures 2(c) and 3(c) are reconstructed images from partitions with the mean color value for each region. Figure 2 presents an example of image segmentation based on RAG and also shows that this graph structure can accelerate segmentation processes. This example compares segmentation obtained by a 4-adjacency grid graph G0 weighted by function g2 with pixel grayscale values (Fig. 2(d)) and segmentation result with a RAG constructed from partition (Fig. 2(b)) and weighted by function g2 with mean values (Fig. 2(e)). Color distance maps are obtained with the initial seeds (white points) in Fig. 2(a). Segmentations are performed by thresholding the obtained distances. Results show similar behaviors both on distance maps and segmentations while drastically speeding-up the segmentation process in the RAG case. Indeed, the number of vertices in the RAG represents approximatively 3% as compared to the number of vertices in the pixel based graph. The direct consequence is a decreasing of the computational complexity thanks to the reduced amount of data to consider. On a standard computer the computing time can be decreased by a 10 factor. Figure 3 shows another benefit of using a RAG structure: nonlocal (non spatially connected) object segmentation. This experiment compares segmentation results with RAG (Fig. 3(d)) and nonlocal RAG (Fig. 3(e)). Both graphs are
Adaptation of Eikonal Equation over Weighted Graph
(a)
Original and seeds
(white)
(d)
(b)
Partition (97% of
reduction)
Grid graph G0 , w=g2
(c)
195
Reconstructed im-
age
(e)
RAG, w=g2
Fig. 2. Acceleration of image segmentation process. (a) original image (150×235) with 35 250 pixels. (b) partition with 999 regions (97% of reduction in terms of image components). (c) reconstructed image with mean color value. (d) and (e): at left, distance color maps (red for small and blue for large distances) and at right, final segmentations. Images (d) are obtained with a pixel based graph computed from (a). Images (e) are obtained with a RAG constructed with (b) and (c) (see text for more details).
computed from partition 3(b) weighted by function g2 with mean color values. In nonlocal RAG case, each vertex neighborhood is extended by a 5 nearest neighborhood based on mean value feature. The obtained graph is a RAG∪5NNG graph. Figures 3(d) and 3(e) show color distance maps computed from initial seeds (white stroke) in Fig. 3(a) and final segmentations. For local case (Fig. 3(d)), object marked by seeds is well segmented with respect to close distances (red color). The other objects are far (blue color) and the final segmentation only extracts the marked one. For nonlocal case (Fig. 3(e)), the distance within the marked object is close to the initial seeds. In addition the distances to other triangles in the scene are also computed as close to seeds (red color). The consequence is that all the objects in the image are extracted by thresholding even if they are not spatially close with a minimal number of initial seeds. Unorganized High Dimensional Data Processing. The following experiments show applications of our formulation of the eikonal equation for the processing of high dimensional data in irregular domains. Figure 4 shows applications of the eikonal equation for data clustering and shortest path problems. The initial data set (Fig. 4(a)) is constituted of 133 images of head pose. Each image is of size 29×29. From this data set, two possible applications can be performed: clustering and head pose transition estimation. The goal here is not to solve machine leaning problems, but to show that these problems can be addressed by our formulation of eikonal equation. In order to process such data, a graph (|V |=133) is constructed where each vertex represents an image and is described by a feature of size 29×29 (i.e IR841 ). In the following results, initial seeds (images) are represented with white boundaries. Points that are close and
196
V.-T. Ta, A. Elmoataz, and O. Lézoray
(a)
Original and seeds
(white)
(d)
(b)
Partition (98% of re-
(c)
Reconstructed image
duction)
RAG, w=g2
(e)
RAG∪5-KNNG,w=g2
Fig. 3. Nonlocal region based image segmentation. (a) original image (256×256) with 65 536 pixels. (b) partition with 1 324 regions (98% of reduction as compared to original one). (d) and (e) at left, distance color maps (red for small and blue for large distances) and at right, final segmentations. Graphs used in (d) and (e) are computed from (b) and (c) (see text for more details).
far to seeds are respectively represented with blue and red colors in distance maps (Fig. 4(b) and 4(c)). Figures 4(b) and 4(d) show the application of the eikonal equation for data clustering. Such an application can be used for data set exploration or semisupervised learning: given an input seed (query) one wants to obtained the closest points with respect to the initial input. Figure 4(b) shows the distance map obtained from a single initial seed. Figure 4(d) shows clustering results. Initial input has a white boundary. The 10 closest images are located at the top and the 10 farthest are located at the bottom of Fig. 4(d). Figures 4(c) and 4(e) shows another example of application of the eikonal equation for data set. Given two initial images, one wants to recover a transition sequence of images that separates them. This problem can be viewed as a shortest path problem solved by the eikonal equation. Figure 4(c) shows the distance map obtained from the initial seeds. Figure 4(e) shows the obtained path from seed at top left to seed at bottom right. These experiments show satisfying results and the ability of our approach to address machine learning problems even if a simple euclidean distance is used to compare data points. Clearly, results can be improved by using well adapted distances or features estimation.
Adaptation of Eikonal Equation over Weighted Graph
(a)
Original data, 133 images of size 29×29 (|V |=133, f 0 : V →
841
IR
(b)
)
Color distance map+seed (white boundaries)
(d)
197
Local clustering with (b)
(c)
Color distance map+seeds (white boundaries)
(e)
Shortest path with (c)
Fig. 4. High dimensional data clustering and shortest path. (b) and (c) color distance maps (blue for small and red for large distance) images superimposed with white boundaries are initial seeds. (d) clustering results where at top the 10 closest and at bottom the 10 farthest with respect to the seed (white boundary). (e) shortest path from the two initial seeds (white boundary).
5
Conclusion
In this paper, a discrete version of the eikonal equation over weighted graphs of arbitrary structure is proposed. Solution of the eikonal equation based on PdEs, discrete gradients and weighted graphs is presented. The proposed formulation constitutes a simple, common and adaptative framework that recovers well-known definitions and unifies local and nonlocal configurations in the context of image processing. This framework can consider any discrete data that can be represented by weighted graphs. Through experiments, we have shown the potentiality and the flexibility of our approach to address image segmentation
198
V.-T. Ta, A. Elmoataz, and O. Lézoray
and unorganized high dimensional data processing. Finally, an ongoing work is to address the stationary (time independent) version of the eikonal equation and to solve this equation by considering fast marching like methods on arbitrary graphs within our framework.
References 1. Rouy, E., Tourin, A.: A viscosity solutions approach to shape-from-shading. SIAM J. Num. Anal. 29, 867–884 (1992) 2. Sethian, J.: A fast marching level set methods for monotonically advancing fronts. Proc. Nat. Acad. Sci. 41(2), 199–235 (1999) 3. Siddiqi, K., Bouix, S., Tannenbaum, A., Zucker, S.W.: The hamilton-jacobi skeleton. In: Proc. ICCV, pp. 828–834 (1999) 4. Maragos, P., Butt, M.: Curve evolution, differential morphology and distance transforms as applied to multiscale and eikonal problems. Fundamentae Informatica 41, 91–129 (2000) 5. Bronstein, A.M., Bronstein, M.M., Kimmel, R.: Weighted distance maps computation on parametric three-dimensional manifolds. J. Comput. Phys. 225(1), 771–784 (2007) 6. Sethian, J.A., Vladimirsky, A.: Ordered upwind methods for static hamiton-jacabi equations: Theory and algorithms. SIAM J. Num. Anal. 41(1), 325–363 (2003) 7. Abgrall, R.: Numerical discretization of the first-order hamilton-jacobi equations on triangular meshes. Comm. Pure and Applied Math. 49, 1339–1373 (1996) 8. Shu, C.-W., Zhang, Y.-T.: High order WENO schemes for hamilton-jacobi equations on triangular meshes. SIAM J. Scien. Comp. 24, 1005–1030 (2003) 9. Mémoli, F., Sapiro, G.: Fast computation of weighted distance functions and geodesics on implicit hyper-surfaces. J. Comput. Phys. 173, 730–764 (2001) 10. Leveque, R.: Finite Volume Methods for Hyperbolic Problems. Cambridge University Press, Cambridge (2002) 11. Sethian, J.A.: Level Set Methods and Fast Marching Methods. Evolving Interfaces in Computational Geometry. In: Fluid Mechanics, Computer Vision, and Materials Science, 2nd edn. Cambridge University Press, Cambridge (1999) 12. Zhao, H.K.: Fast sweeping method for eikonal equations. Math. Comp. 74, 603–627 (2005) 13. Tsitsiklis, J.: Efficient algorithms for globally optimal trajectories. IEEE Trans. Autom. Control 40(9), 1528–1538 14. Osher, S., Sethian, J.: Fronts propagating with curvature-dependent speed: Algorithms based on Hamilton-Jacobi formulations. J. Comput. Phys. 79, 12–49 (1988) 15. Jaromczyk, J., Toussaint, G.: Proc. IEEE. Relative Neighborhood Graphs and Their Relatives 80(9), 1502–1517 (1992) 16. Elmoataz, A., Lézoray, O., Bougleux, S., Ta, V.T.: Unifying local and nonlocal processing with partial difference operators on weighted graphs. In: Proc. LNLA, pp. 11–26 (2008) 17. Efros, A., Leung, T.: Texture synthesis by non-parametric sampling. In: Proc. ICCV, pp. 1033–1038 (1999) 18. Buades, A., Coll, B., Morel, J.: Nonlocal image and movie denoising. IJCV 76(2), 123–139 (2008) 19. Gilboa, G., Osher, S.: Nonlocal operators with applications to image processing. Report 07-23, UCLA (2007)
Adaptation of Eikonal Equation over Weighted Graph
199
20. Bougleux, S., Elmoataz, A., Melkemi, M.: Discrete regularization on weighted graphs for image and mesh filtering. In: Sgallari, F., Murli, A., Paragios, N. (eds.) SSVM 2007. LNCS, vol. 4485, pp. 128–139. Springer, Heidelberg (2007) 21. Ta, V.T., Elmoataz, A., Lézoray, O.: Partial difference equations over graphs: Morphological processing of arbitrary discrete data. In: Proc. ECCV, pp. 668–680 (2008) 22. Brockett, R., Maragos, P.: Evolution equations for continuous-scale morphological filtering. IEEE Trans. Signal Process. 42(12), 3377–3386 (1994)
A Variational Model for Interactive Shape Prior Segmentation and Real-Time Tracking Manuel Werlberger, Thomas Pock, Markus Unger, and Horst Bischof Institute for Computer Graphics and Vision, Graz University of Technology {werlberger,pock,unger,bischof}@icg.tugraz.at http://www.gpu4vision.org
Abstract. In this paper, we introduce a semi-automated segmentation method based on minimizing the Geodesic Active Contour energy incorporating a shape prior. We increase the robustness of the segmentation result using the additional shape information that represents the desired structure. Furthermore the user has the possibility to take corrective actions during the segmentation and adapt the shape prior position. Interaction is often desirable when processing difficult data like in medical applications. To facilitate the user interaction we add a shape deformation which allows to change the shape position manually by the user and automatically in terms of underlying image features. Using a variational formulation, the optimization can be done in a globally optimal manner for a fixed shape representation. To obtain real-time behavior, which is especially important for an interactive tool, the whole method is implemented on the GPU. Experiments are done on medical, as well as on video data and camera streams that are processed in real-time. In terms of medical data we compare our method with a segmentation done by an expert. The GPU based binaries will be available online on our homepage.
1
Introduction
Image segmentation is a very common problem in computer vision. Many segmentation methods use low-level features to obtain a division into fore- and background. Due to the need of robustness it has become a common practice to incorporate high level knowledge to gain reasonable results. The method presented in this paper enhances the robustness of segmentation by imposing shape information of the desired object which allows a precise result with difficult image data (Fig. 1). Pioneering contributions have been made by Cremers et al. [1] with their variational approach of ‘Diffusion Snakes’, the level set formulation by Leventon et al. [2, 3] as well as the region based approach by Paragios and Rousson [4, 5]. The efficient registration of the shape prior to the desired image structure is a challenging problem. Therefore we developed a semi-automated segmentation tool that allows to adjust the object position by hand and by a local optimization routine which is modelled as shape transformation in either case. For the X.-C. Tai et al. (Eds.): SSVM 2009, LNCS 5567, pp. 200–211, 2009. c Springer-Verlag Berlin Heidelberg 2009
A Variational Model for Interactive Shape Prior Segmentation
201
Fig. 1. Segmentation of the metacarpal bone of a ring finger. The left image shows a simple intensity thresholding which clearly fails. The pure GAC segmentation in the middle image does not result into a valid segmentation either. The shape prior segmentation in the right image can also deal with the low-contrast regions and provides an accurate result.
realization of the segmentation method we use a variational formulation of the Geodesic Active Contour (GAC) energy. The minimization is done with a fast primal-dual approach that is implemented using NVIDIA graphics hardware to become real-time capable. Therefore the method can even be used for tracking objects in videos or live camera streams. Approaches based on the calculus of variation have had great success and recently it has been shown that variational methods show good parallelization capabilities that benefit from a GPU implementation [6, 7]. The main contribution of our work is the incorporation of a shape representation into a variational segmentation framework. We show that the resultant segmentation is globally optimal for a fixed shape prior and provide a GPU implementation for a fast primal-dual optimization procedure defining a definite convergence criterion. Therefore we can add the possibility to interact with the shape position and get a segmentation result in real-time. The framework permits a local optimization of the shape position to get a correct segmentation of objects with a preceding misalignment of the prior. The remainder of the paper is organized as follows: First we give an overview of related work. Section 3 discusses the method on combining a shape prior with a variational formulation of the GAC segmentation model which leads to a segmentation model utilizing a Mumford-Shah (MS) like data term as shape force. In Section 3.1 we propose a fast numerical algorithm to compute the solution of the segmentation model. In 4 we present experiments and a qualitative assessment on reference data. Finally, Section 5 gives a short conclusion.
2 2.1
Related Work Mumford-Shah Segmentation
In [8], Mumford and Shah (MS) proposed a segmentation model of the form 2 2 (u − f ) dx + α |∇u| dx + β length (Γ ) (1) min u,Γ
Ω
Ω\Γ
202
M. Werlberger et al.
where f denotes the observed image, u its piecewise smooth approximation and Γ represents the edges in u. Equation (1) is based on a piecewise smooth approximation of the intensity function and was used in the computer vision community for various tasks like denoising, inpainting, stereo matching, segmentation and many more. A special case of the MS model to segment an image into fore- and background was proposed with the so-called piecewise constant MS segmentation model (2) that was later used by Chan and Vese [9] in combination with a level-set optimization. 2 2 min Per (Σ) + λ (f − c1 ) dx + λ (f − c2 ) dx (2) Σ,c1 ,c2
Σ
Ω\Σ
In (2) f denotes the input image and c1 , c2 the mean values of the fore- and background intensities separated by the region Σ. This realization of the MS functional represents the Potts model [10] for two distinct classes. In addition Chan et al. provided a convex formulation in [11] in form of a Total Variation (TV) functional for a binary segmentation u = 1Σ : |∇u| dx + λ us (x) dx (3) min u∈{0,1}
Ω
Ω
with
2
2
s (x) = (c1 − f (x) ) + (1 − u) (c2 − f (x) )
(4)
and with Ω |∇u| dx being the TV-norm in a distributional sense Ω |D u|. Due to the fact that we are working on image data which can be interpreted as sufficient smoothfunctions, the TV-norm is valid for any input u and we stick to the notation Ω |∇u| dx in this paper. For this formulation the TV-norm denotes the length of the segmentation: Ω |∇u| dx = Per (Σ). Moreover Chan et al. [11] showed that a global minimizer can be found for (2) with a restricted minimization of the relaxed problem (3) so that 0 ≤ u ≤ 1. The minimization set is then given by Σ = x ∈ Ω : u > μ , for every μ ∈ (0, 1) . (5) 2.2
Geodesic Active Contours
Based on the Snake model of Kass et al. [12], Caselles et al. [13] and Kichenassamy et al. [14, 15] proposed an energy that is invariant with respect to new parametrizations of the contour. The Geodesic Active Contour (GAC) (in 3D the model is called minimal surface) is defined as the variational problem |C| min g (|∇I (C(s))|) ds , (6) C
0
where |C| describes the Euclidean length of the curve C and the function g models an edge detector. The edge strength has to be restricted to an interval g ∈ (0, 1]. One common choice for computing g is κ
g(|∇I|) = e−η|∇I| ,
for some reasonable parameters κ and η.
(7)
A Variational Model for Interactive Shape Prior Segmentation
203
The general intention of the Snake model is to locate the curve at points with a high edge strength and keep a certain smoothness in the curve. The main advance of GACs is the profound mathematical framework that makes the model very versatile for different applications. The main drawbacks of the model are the non-convexity of the GAC energy and that the empty set is always a global minimizer of (6). In [16, 17, 18], several authors proposed the so-called weighted Total Variation (8) that can be used to give an alternative formulation of the GAC energy. They showed that if u = 1ΩC is a binary function with C the boundary of ΩC , the energy (8) equals the GAC energy (6). T Vg (u) = g |∇u| dx (8) Ω
Note that the weighted TV-norm is similar to the regularization term of (3). The additional weighting function g is a pointwise constant multiplier and therefore the method of Chan et al. [11, 19] mentioned in the previous Section 2.1 is still valid. Based on these assumptions they showed that by replacing 1ΩC with u ∈ [0, 1], (8) becomes convex, allowing to compute a global minimizer. However, there remains the problem that the empty set depicts a global optimal solution. In [6, 7], Unger et al. proposed a variational formulation incorporating user constraints that avoid this drawback of the classical GAC model.
3
Shape Prior Segmentation
Our main contribution is to combine GAC segmentation and a MS-like data term to incorporate shape information. Therefore we model the GAC energy with the weighted Total Variation (8) and utilize the need of additional constraints by imposing the shape prior. Starting with a formulation of the Mumford-Shah like energy utilized by Chan and Vese in [9] we obtain a variational optimization problem like in (3). The multiplicative part in the data-term will be used as shape information s (x) to model the shape prior segmentation energy. In addition we use the weighted Total Variation (8) to model the GAC energy and add a parameter λ to balance between regularization and shape force: min g |∇u| dx + λ s (x) u dx (9) 0≤u≤1
Ω
Ω
For a low λ the result of the GAC will be preferred, whereas for increasing λ the shape prior will be taken more into account. In Fig. 2 the effects of different parameter settings are shown. Since (9) is homogeneous of degree one, the thresholding theorem of [11] still applies in our case, allowing us to compute the global minimizer of (9) as mentioned in Section 2.1 according the MS segmentation model (3). The optimization method will be discussed in more detail in the following Section 3.1.
204
M. Werlberger et al.
Fig. 2. Evaluation of different settings of λ: The images show that with increasing λ = {0.01, 0.02, 0.1} (left to right) the segmentation is more attracted to the fixed shape. For a low λ the influence of the pure GAC energy increases and the segmentation is more attracted to significant edges.
The shape prior itself influences the segmentation by setting pixelwise foreand background constraints which are modelled in the following way: s (x) < 0 s (x) > 0
... ...
Foreground Background
(10)
Therefore one can use different types of shape representations. As a simple example we defined the shape s (x) as a binary function with s (x) = −1 within the shape region and s (x) = 1 outside similar to [20] where Cremers et al. use subspace methods to learn a representative set of shapes. Binary functions are used to encode shapes which lead to problems when interpolating between two instances. As a consequential step we use a signed-distance map with the constraints (10) which implicitly includes a distance information towards the shape boundary. That means for our algorithm that the more a pixel is within the shape boundary, the more likely this region belongs to the desired segmentation. In Fig. 5 we show the benefit on using a signed-distance map as shape representation compared to a binary one. This representation and the combination with a GAC energy allows to handle deformations with a single prior. 3.1
Solving the Shape Prior Segmentation Model
It is well known that functionals like (9) are difficult to optimize due to the L1 norm |∇u|. Chan et al. [21], Carter [22] and Chambolle [23, 24] proposed a dual formulation for optimizing the classical variational problem of Rudin, Osher and Fatemi [25] for image denoising. Such a primal-dual approach can be applied to our minimization problem (9). The main intention is to remove the singularity by introducing the dual formulation of the weighted TV-norm g |∇u| dx = max − u div p dx , (11) Ω
d T
||p||≤g
Ω
where p = p1 , . . . , p : Ω → Rd is the dual variable with d being the problems dimension. Combining this maximization problem with the initial minimization task (9) this leads to
A Variational Model for Interactive Shape Prior Segmentation
min max − u div p dx + λ s (x) u dx .
0≤u≤1 ||p||≤g
Ω
205
(12)
Ω
For a fixed shape prior the outline of the primal-dual optimization algorithm is given as follows: 1. Primal update: The primal update accomplishes the segmentation update and therefore performs the optimization according to the minimization of u: ∂ − u · div p dx + λ s (x) u dx = − div p + λs (x) (13) ∂u Ω Ω Performing a gradient descent update scheme this leads to un+1 = Π[0,1] un − τP (− div p + λs (x) ) ,
(14)
where τP denote the steplength and the orthogonal projection Π towards the binary set [0, 1] can be done with a simple thresholding step. 2. Dual update: The maximization according to p can be stated as ∂ p · ∇u dx + λ s (x) u dx = ∇u (15) ∂p Ω Ω with the additional constraint ||p|| ≤ g. This results into a gradient ascent method with a orthogonal reprojection to restrict the length of p to the weight g: pn+1 = ΠB0g pn + τD ∇u (16) Here B0g denotes a d-dimensional ball centered at the origin with the radius g. The reprojection onto B0g can be formulated with ΠB0g q =
q ||q|| max 1, g
(17)
3. Iterate until convergence: Solving the optimization problem (12) results in a consecutive update scheme with a gradient descent (14) and a gradient ascent step (16). Such an iterative algorithm demand on a convergence criterion. Therefore we take the energy of the single steps into account: Primal energy: The primal energy can be calculated by solving (12) by maximizing the equation towards the dual variable p. Due to (11) p can be restated as ∇u g |∇u| p ∈ B0g
p=
if ∇u =0
(18)
else
for the optimization. This results into the energy equation (19) which is the same as evaluating the energy functional (9). g |∇u| + λ s (x) u dx. (19) EP rimal = Ω
206
M. Werlberger et al.
Fig. 3. Relation of primal to dual energy while optimizing with the proposed primaldual update scheme. The plot shows the iterations in a logarithmic scale over 100 iterations. Note that after 20 iterations the primal-dual gap is small enough to stop iterating.
Dual energy: The dual energy can be formulated by minimizing (12) towards u:
min u − div p + λs (x) dx , (20) 0≤u≤1
Ω
which conclude that the binary segmentation u ∈ {0, 1} is set to u = 1 if the term − div p + λs (x) < 0 and u = 0 otherwise: min − div p + λs (x) , 0 dx (21) EDual = Ω
In [26, 27], Zhu et. al. introduce a measurement for the convergence state of primal-dual algorithms in case of the ROF model. As a criterion they use the gap between the primal and dual energy. Applied to our primal-dual optimization algorithm we get an adaption of the energies like in Fig. 3. Therefore we use a fixed stepwidth for the primal (τP ) and dual update (τD ) with the constraint that τP τD ≤ 12 . For all the results shown in this paper we used τP = τD = √12 . We also tried adaptive timesteps (τP , τD ) for the optimization steps similar to the work of Zhu and Chan in [26] but did not find a reasonable equivalent for our method. 3.2
Shape Alignment
So far, our considerations assume a spatially fixed shape prior s (x) . In order to adapt the shape prior to different locations in the image we introduce a set of transformation parameters φ = {t, R, S} with the transformation parameters t for translation, R for rotation and S for the scale. Imposing this transformation into the segmentation energy (9) leads to an additional optimization parameter: min φ (t, R, S) ◦ s (x) u dx (22) g |∇u| dx + λ u,φ
Ω
Ω
A Variational Model for Interactive Shape Prior Segmentation
207
Fig. 4. Segmentation of a vertebra in an X-ray image of the spline. Due to very bad contrast the segmentation without prior would fail completely. The definition of the prior was prepared by us and therefore cannot be considered as reference data.
In [20], Cremers et al. show with the help of the Lipschitz continuity that with an integrated rigid body motion the energy functional remains convex and therefore can be optimized globally for fixed transformation parameters. They additionally show that for optimizing the shape position itself the complete subspace Ω where the energy (9) is defined has to be sampled on a rather fine grid with all possible shape positions for evaluating the minimization task in a global manner. This is of course not feasible for an interactive application due to the need of optimizing the problem in real-time. Therefore we added a semiautomated approach that allows the user to have influence on the shape position and in addition an automated search for the optimal transformation parameters in a local neighborhood can be done. The optimization scheme in Sect. 3.1 can be retained with an extension on optimizing the transformation φ: 1. Solve shape prior segmentation model. 2. Optimize transformation parameters φ with a fixed u: Here we use a semi-automated position optimization of the shape prior. First, the user has the possibility to do a coarse positioning of the shape and second an optimization step tries to fit the shape to desired surrounding structures. Therefore we evaluate the energy (19) for different transformations φ. The optimal position is found where (19) has its minimum. The user gets an immediate result while changing the shape position and therefore can directly interact with the segmentation algorithm. 3. Iterate until convergence. Note that the domain of φ is restricted due to performance issues. Doing a complete search over the whole parameters space of the transformation parameters, a global optimal solution of (22) can be calculated like in [20].
4
Experimental Results
To reach real-time performance we have to compute the iterative solution as fast as possible. Due to the good parallelization attributes of variational algorithms we decided to implement the method using GPGPU programming with the help of NVIDIAs CUDA. The involved enhancements offer the possibility to combine
208
M. Werlberger et al.
Fig. 5. The same dataset as in Fig. 1 was used to evaluate the shape alignment step. Therefore the shape prior is placed nearby the desired bone (left image) and the alignment step searches for the optimal position in the local neighborhood. The results (middle image) show that the segmentations are equivalent to the hand-labeled points. The right image shows the alignment step using a binary shape prior which fails for the optimization step. For the correct result in the middle image a signed distance representation of the shape term was used.
Fig. 6. The position refinement is robust against partial occlusion
user interactivity with the computational intensive variational method and get a segmentation result at interactive rates. The performance depends mainly on the size of the search region in the parameter space and on the shape prior’s size as well. As an example we achieve 80 frames per second on a NVIDIA GeForce GTX 280 for pure segmentation and 20 frames per second including the position optimization. In our Framework we have two possibilities to provide a shape prior for the segmentation. Either the user can define a shape prior directly with a segmentation of a structure using pure GAC energy by setting foreground and background seeds. This is especially useful when recurring structures have to be segmented. For more accurate results especially on difficult data we can load a predefined shape structure that is used for the segmentation method. In Fig. 1 a predefined prior is used for segmenting finger bones and compare the segmentation result with a simple intensity thresholding and a segmentation with the pure GAC-energy. The result shows an identical segmentation result for the shape prior segmentation and the reference data labeled by an expert. A more difficult example is shown in Fig. 4 which shows a segmentation of a single vertebra in an X-ray image of the spline. Due to the very bad contrast simple thresholding would obviously fail completely and also pure GAC energy would end up into setting very much seed information.
A Variational Model for Interactive Shape Prior Segmentation
209
Fig. 7. Real-time tracking of an espresso cup in a live camera stream
Fig. 8. Multiple vertebrae of the cervical spline are tracked separately through a sequence. The intention is to ascertain the movement of the vertebrae towards each other.
Fig. 9. Segmentation of bottles with a single shape prior
Examples of automated shape alignment with optimizing the transformation parameters in a local neighborhood are shown in Fig. 5–8. Fig. 5 shows again the labeled bone dataset. The overlay with reference data shows that the position optimization leads to the correct segmentation. For non-medical image data examples are presented in Fig. 6 and 7. The first one shows that the proposed method is robust against partial occlusion. Furthermore the method can be used
210
M. Werlberger et al.
for tracking a certain structure over a sequence of frames. Fig. 7 shows a sequence of an espresso cup which was directly processed from a camera image with on-thefly segmentation and position optimization. For a restricted domain of movement we gain real-time performance for tracking a sequence. In Fig. 8 we track parts of the cervical spline in a moving X-ray image series to obtain a path of movement of the vertebrae to each other during a flexion. The predefined shapes of the four vertebrae are initialized in the first frame and than automatically tracked over the complete flexion. This can be used to ascertain shapes of implants for intervertebral discs. In Fig. 9 multiple objects are segmented with the help of a single shape prior. Therefore a prior is defined in form of a bottle and placed roughly on each bottle in the image. The fine adjustments are done automatically.
5
Conclusion
In this paper, we proposed a globally optimal shape prior segmentation method with additional user interaction and automated position refinement. With this approach we can handle very different images and gain robust segmentation results. Especially the segmentation for difficult data like the low-contrast spline image benefits from the additional shape information. A great advantage of variational methods like this are the parallelization capability that especially profits by the modern graphics hardware that are able to boost the performance of such highly parallel algorithms.
References 1. Cremers, D., Tischhäuser, F., Weickert, J., Schnörr, C.: Diffusion snakes: Introducing statistical shape knowledge into the Mumford–Shah functional. International Journal of Computer Vision 50(3), 295–313 (2002) 2. Leventon, M., Faugeraus, O., Grimson, W.: Level set based segmentation with intensity and curvature priors. In: Workshop on Mathematical Methods in Biomedical Image Analysis, pp. 4–11 (2000) 3. Leventon, M., Grimson, W., Faugeras, O.: Statistical shape influence in geodesic active contours. In: Proc. IEEE Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 316–323. IEEE, Los Alamitos (2000) 4. Paragios, N., Rousson, M., Ramesh, V.: Matching distance functions: A shape-toarea variational approach for global-to-local registration. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002. LNCS, vol. 2351, pp. 775–789. Springer, Heidelberg (2002) 5. Paragios, N., Rousson, M., Ramesh, V.: Non-rigid registration using distance functions. Computer Vision and Image Understanding 89(2-3), 142–165 (2003) 6. Unger, M., Pock, T., Bischof, H.: Continuous Globally Optimal Image Segmentation with Local Constraints. In: Computer Vision Winter Workshop (2008) 7. Unger, M., Pock, T., Trobin, W., Cremers, D., Bischof, H.: TVSeg - Interactive total variation based image segmentation. In: British Machine Vision Conference (2008)
A Variational Model for Interactive Shape Prior Segmentation
211
8. Mumford, D., Shah, J.: Optimal approximations by piecewise smooth functions and variational problems. Comm. on Pure and Applied Math. XLII(5), 577–685 (1988) 9. Chan, T.F., Vese, L.A.: Active contours without edges. IEEE Trans. Image Processing 10(2), 266–277 (2001) 10. Potts, R.B.: Some generalized order-disorder transformations. Proc. Camb. Phil. Soc. 48, 106–109 (1952) 11. Chan, T.F., Esedoglu, S., Nikolova, M.: Algorithms for finding global minimizers of image segmentation and denoising models. SIAM Journal of Applied Mathematics 66(5), 1632–1648 (2006) 12. Kass, M.: Snakes: Active contour models. International Journal of Computer Vision 1(4), 321–331 (1980) 13. Caselles, V., Kimmel, R., Sapiro, G.: Geodesic active contours. International Journal of Computer Vision 22(1), 61–79 (1997) 14. Kichenassamy, S., Kumar, A., Olver, P., Tannenbaum, A., Yezzi, A.: Conformal curvature flows: From phase transitions to active vision. Archive for Rational Mechanics and Analysis, 275–301 (1996) 15. Kichenassamy, S., Kumar, A., Olver, P.J., Tannenbaum, A.R., Yezzi Jr., A.J.: Gradient flows and geometric active contour models. In: International Conference on Computer Vision, pp. 810–815 (1995) 16. Leung, S., Osher, S.: Global minimization of the active contour model with TVinpainting and two-phase denoising. In: Paragios, N., Faugeras, O., Chan, T., Schnörr, C. (eds.) VLSM 2005. LNCS, vol. 3752, pp. 149–160. Springer, Heidelberg (2005) 17. Bresson, X., Esedoglu, S., Vandergheynst, P., Thiran, J.P., Osher, S.J.: Global minimizers of the active contour/snake model. In: International Conference on Free Boundary Problems: Theory and Applications (FBP) (2005) 18. Bresson, X., Esedoglu, S., Vandergheynst, P., Thiran, J.P., Osher, S.J.: Fast global minimization of the active contour/snake model. J. of Mathematical Imaging and Vision 28(2), 151–167 (2007) 19. Chan, T.F., Esedoglu, S.: Aspects of total variation regularized L1 function approximation. SIAM Journal of Applied Mathematics 65(5), 1817–1837 (2005) 20. Cremers, D., Schmidt, F.R., Barthel, F.: Shape priors in variational image segmentation: Convexity, Lipschitz continuity and globally optimal solutions. In: Computer Vision and Pattern Recognition, pp. 1–6 (2008) 21. Chan, T.F., Golub, G.H., Mulet, P.: A nonlinear primal-dual method for total variation-based image restoration. SIAM Journal on Scientific Computing 20(6), 1964–1977 (1999) 22. Carter, J.: Dual Methods for Total Variation-based Image Restoration. PhD thesis, UCLA (2001) 23. Chambolle, A.: An algorithm for total variation minimization and applications. Journal of Mathematical Imaging and Vision 20(1-2), 89–97 (2004) 24. Chambolle, A.: Total variation minimization and a class of binary MRF models. In: Rangarajan, A., Vemuri, B.C., Yuille, A.L. (eds.) EMMCVPR 2005. LNCS, vol. 3757, pp. 136–152. Springer, Heidelberg (2005) 25. Rudin, L.I., Osher, S.J., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Physica D: Nonlinear Phenomena 60, 259–268 (1992) 26. Zhu, M., Chan, T.: An efficient primal-dual hybrid gradient algorithm for total variation image restoration. UCLA CAM Report 08-34 (2008) 27. Zhu, M., Wright, S.J., Chan, T.F.: Duality-based algorithms for total variation image restoration. UCLA CAM Report 08-33 (2008)
A Nonlinear Probabilistic Curvature Motion Filter for Positron Emission Tomography Images Musa Alrefaya, Hichem Sahli, Iris Vanhamel, and Dinh Nho Hao Vrije Universiteit Brussel, Dept. Electronics and Informatics ETRO-IRIS, Pleinlaan 2, B-1050 Brussels, Belgium {malrefay,hsahli,iuvanham}@etro.vub.ac.be http://www.etro.vub.ac.be
Abstract. Positron Emission Tomography (PET) is an important nuclear medicine imaging technique which enhances the effectiveness of diagnosing many diseases. The raw-projection data, i.e. the sinogram, from which the PET is reconstructed, contains a very high level of Poisson noise. The latter complicates the PET image’s interpretation which may lead to erroneous diagnoses. Suitable denoising techniques prior to reconstruction can significantly alleviate the problem. In this paper, we propose filtering the sinogram with a constraint curvature motion diffusion for which we compute the edge stopping function in terms of edge probability under the assumption of contamination by Poison noise. We demonstrate through simulations with images contaminated by Poisson noise that the performance of the proposed method substantially surpasses that of recently published methods, both visually and in terms of statistical measures.
1
Introduction
Positron Emission Tomography (PET) is an in vivo nuclear medicine imaging method that provides functional information of the body tissues. The PET image results from reconstructing very noisy, low resolution raw data, i.e. the sinogram, in which important features are shaped as a curved structures. Enhancing the PET image spurred a wide range of denoising models and algorithms. Some methodologies focus on enhancing the reconstructed PET image directly, where others prefer enhancing the sinogram prior to reconstruction. Existing methods may suffer drawbacks such as the careful selection of a high number of parameters, smoothing of the important features’ boundaries, or prohibitive computation. Recently, nonlinear diffusion techniques have been investigated for PET images. Many researchers did explore the application of the well-known Perona
The authors sincerely wishes to express great thanks to Prof. M. Defrise, Division of Nuclear Medicine at AZ-VUB, for his discussions and feedback. The comparison to the TV-Nestrove scheme would not have been possible without the help of Dr. Pierre Weiss from INRIA-France, who did provide us the Matlab Code.
X.-C. Tai et al. (Eds.): SSVM 2009, LNCS 5567, pp. 212–223, 2009. c Springer-Verlag Berlin Heidelberg 2009
A Curvature Motion Filter for PET
213
and Malik anisotropic diffusion [15] in combination with diverse diffusivity functions, on PET images [2, 4, 5, 14, 26], as well as on sinograms [6, 25]. The main drawback of this filter, with respect to the Poisson noise, which characterize such type of images, is that the diffusion produces important oscillations in the gradient, which finally leads to a poorly smoothed image [28, 29]. Moreover, the adopted diffusivity functions do not consider the special properties of the sinogram in which the preservation of the curved-shape features is paramount (see Figure1. In [28], mean curvature motion and Gaussian curvature motion of PET images have been investigated. Total variation (TV) scheme for smoothing the PET images was also discussed in [28]. Happonen et al. [29] propose filtering the sinogram in the stackgram domain where the signal along the sinusoidal trajectories of the sinogram can be filtered separately. They used and compared the Gaussian and nonlinear filters. Filtering the sinogram has the advantage that the noise distribution is known, which is not the case after reconstruction. Consequently, this work proposes filtering the sinogram by means of a curvature constrained filter in which the amount of diffusion is modulated according to a probabilistic diffusivity function that suits images contaminated with Poisson noise. In addition, a comparison of the proposed method with TV-based methodologies proposed in [1] and [12], is conducted. For this purpose, a simulated thorax PET phantom was constructed to which varying levels of Poisson noise have been added is used. For evaluating the filtering approaches, contrast noise curves (contrast versus background noise at different iteration numbers) were generated for the different filtering approaches. The reminder of the paper is organized as follows. Section 2, briefly review the notions of curvature motion, edge affected diffusion filtering, and self-snakes. The proposed filtering scheme is introduced in Sect. 3. Section 4.1 introduces the applied validation methods, the remainder of Sec.4 discusses the experimental results. Conclusions and future work are given in Sec.5.
2
Geometry Driven Scale-Space Filtering
This section reviews the formulations for mean curvature motion (MCM), Edge Affected Variable Conductance Diffusion (EA-VCD), and self-snakes. Let f be a scalar image defined on the spatial image domain Ω, then the family of diffused versions of f is given by: U (f ) : f (.) → u(., t) with u(., 0) = f (.)
(1)
where U is referred to as the scale-space filter, u is denoted the scale-space image, and the scale t ∈ R+ [23, 26]. The denoised or enhanced version of f , is a given u(., t) that is closest to the unknown noise-free version of f . 2.1
Curvature Motion
One way of introducing smoothness in the curve is to let it evolve under its Euclidean curvature k. Mean curvature motion (MCM) is considered as the
214
M. Alrefaya et al.
standard curvature evolution. MCM allows diffusion solely along the level-lines. In Gauge coordinates the corresponding PDE formulation is: ∇u ut (., t) = uvv = k|∇u| = div |∇u| (2) |∇u| Hence diffusion solely occurs along the v-axis. 2.2
Edge Affected Variable Conductance Diffusion
Variable Conductance filtering (VCD) is based on the diffusion with a variable conduction coefficient that controls the rate of diffusion [23]. In the case of Edge Affected-VCD (EA-VCD), the conductance coefficient is inversely proportional to the edgeness. Consequently it is commonly referred to as the edge stopping function (g), in which the edgeness is typically measured by the gradient magnitude. The EA-VCD is governed by: ut = div [g(|∇u|)∇u]
(3)
The above PDE system together with the initial condition given in (1) is completed with homogenous von Neumann boundary condition on the boundary of the image domain. Note that the Perona and Malik’s antitropic diffusion [15] is an EA-VCD. 2.3
Self-Snakes
Self-snakes are a variant of the MCM in which an edge-stopping function is introduced [19]. The main goal is preventing further shrinking of the level-lines once they have reached the important image edges. For scalar images, self-snakes are governed by: ∇u ut = |∇u|div g(|∇u|) (4) |∇u| This equation adopts the same boundary condition as (3). Furthermore, it can be decomposed in two parts [19, 23]: ∇u ut = g|∇u|div |∇u| +(∇g).∇u (5) = gk|∇u| +(∇g).∇u The first part describes a degenerate forward diffusion along the level lines, i.e. orthogonal to the local gradient; it allows preserving the edges. Additionally, the diffusion is limited in areas with high gradient magnitude and encouraged in smooth areas. Actually the first term is the constraint curvature motion. The second term can be viewed as a shock filter since it pushes the level-lines towards valleys of high gradient, acting as Osher’s shock filter [18].
A Curvature Motion Filter for PET
3
215
The Probabilistic Curvature Motion Filter
Based on (i) the curvature motion method, and (ii) a probabilistic diffusivity function, we presented in an earlier work [16], this section introduces the proposed filtering schemes for PET sinogram, considering the following characteristics: 1. The important features in the sinogram are curved structures with high contrast values. These represent the region of interests in the reconstructed PET image, e.g. tumor. 2. The weak edges in the sinogram are the edges that contains low contrast values. In other words, edges with small |uww |. 3. The noise in the sinogram is a priori identified as a Poisson noise. The above presented schemes, namely, MCM (2), EA-VCD (3) and the SelfSnakes (5), can be derived using the following general equation: ut = g1 (|∇u|)uvv + g2 (|∇u|)uww
(6)
where the second order Gauge derivatives of the image in the (vv) and (ww) directions are given by: uvv =
uxx u2y −2ux uy uxy +uyy u2x
(u2x +u2y )
uww =
uxx u2y +2ux uy uxy +uyy u2x
(u2x +u2y )
(7)
Equation (6) comprises, a diffusion modulated by g1 along the image edges (vv) (a smoothing term), and a diffusion adjustable by g2 across the image edges (ww) (a sharpening term). Careful modeling of these terms allows efficiently denoising the PET sinograms, whilst keeping their interesting features. In the following sections, we propose the use of a probabilistic diffusivity function, and derive two diffusions schemes, for which we did apply the Gauge derivatives numerical approximation that was described in [23]. 3.1
The Probabilistic Diffusivity Function
The main idea of the Probabilistic Diffusivity Function [16] is to express the diffusivity function as the probability that the observed gradient presents no edge of interest under a suitable marginal prior distribution for the noise-free gradient histogram. The diffusivity function was defined as: gpr (x) = A(1 − P (H1 |x))
(8)
where the normalizing constant A is set to A = 1/(1 − P (H1 |0)) to ensure that gpr (0) = 1; the hypothesis H1 describes the notion whether an edge element of interest is present given the considered noise, and H0 an edge element of interest is absent. Formally, H0 : y ≤ σn , and H1 : y > σn (9)
216
M. Alrefaya et al.
with y being the ideal, noise-free, gradient magnitude, and σn the noise standard deviation in the observed gradient image. In [16] it has been demonstrated that gpr (x) = (1 + μη(0))
1 1 + μη(x)
(10)
where μ = P (H1 )/P (H0 ) is the prior odds, and η(x) = p(x|H1)/p(x|H0) is the likelihood ratio. −1 Considering a Laplacian prior p(y) = λ2 e−λ|y| , we have μ = eλσn − 1 [16], and the parameter λ can be estimated as λ = [0.5(σ 2 − σn2 )]−1/2
(11)
with σ 2 denoting the variance of the noisy image, and σn2 , as defined above. Due to limited space, the reader is referred to [16] for the detailed expression of η(x) in (10). The proposed diffusivity function,(10), has no free parameters to optimize, and it fits well in the cluster of the reference backward-forward diffusivities. Indeed, for the considered PET sinograms, the noise standard deviation, σn ,in 11 is being estimated as σn2 = V ar(uLn ) where the image noise uLn is reconstructed from the two finest resolution levels coefficients by applying the wavelet decomposition of u, using daubechi(4) function. 3.2
Probabilistic Constraint Curvature Motion
For the probabilistic constraint curvature motion (PCCM), we start from a constraint version of mean curvature motion: the diffusion across the level lines is prohibited whilst the diffusion along the level-lines is controlled via the probabilistic diffusivity function (10): ut = gpr (|∇u|)uvv = gpr (|∇u|)k|∇u|
(12)
. Thus the function g1 in (6) is chosen to be g1 = gpr , for dealing with Poisson noise. This filter effectively smooths the image, as well as preserves edges of the important features such as lines, curve and flow-like structures. By its nature, the PCCM cannot enhance the weak edges and/or features in the sinogram. The second term in (6) allows the sharpening. Consequently, we . set g2 = g1 . In this way, weak but important edges are enhanced whilst the noise is removed efficiently. Formally, the enhanced PCCM (ePCCM) is given by: ut = gpr (|∇u|)uvv + gpr (|∇u|)uww
3.3
(13)
Probabilistic Self Snakes (PSS)
It can be demonstrated that the diffusion of scalar images via EA-VCD can be decomposed into (5), moreover, it can be rewritten as [7, 23]: ut = g(|∇u|)uvv + [g(|∇u|) + g (|∇u|)|∇u|] uww
(14)
A Curvature Motion Filter for PET
217
consolidating the properties of both the self-snakes and the EA-VCD into a single diffusion schema. Considering equation (6), and the proposed probabilistic . . diffusivity function, we have g1 (x) = gpr (x), and the sharpening term, g2 (x) = gpr (x) + xgpr (x). This filter proves to be very effective and flexible for the sinogram image where the high contrast regions, which represent a tumor in the reconstructed PET, should be smoothed wisely without blurring the poor edges. Like EA-VCDT, the main advantage of this filter is that the average gray value of the image is not altered during the diffusion process which is a significant issue in the sinogram.
4 4.1
Experiments and Discussion Introduction
The goal of the conducted experiments consists of measuring the performance of the proposed filtering methods, and studying their influence on the two commonly used PET reconstruction methods. The filtered back projection (FBP) [11], and the iterative ordered subset expectation maximization (OSEM) [8], respectively. The FBP algorithm is computationally efficient while the OSEM algorithm can incorporate easily prior information on the image to improve image quality. In our experiments, the reconstruction parameters of these algorithms are set as follows: the hamming parameter in the FBP method is 0.5, while for the OSEM, we use 16 subsets and run it for 4 iterations. In the following the reconstructed PET images are denoted by UF (t) = FBP(ut ), and UO (t) = OSEM(ut ), respectively, for a given enhanced sinogam ut . A simulated thorax PET phantom, containing three regions of interest (tumors) was constructed. To which varying levels of Poisson noise have been added and used for the evaluation. 50 realizations (noisy sinograms) with added noise level of 1x106 coincident events, have been generated. Each sinogram has a mm size of 256x256 pixels and their spacing is 2x2 pixel . Figure.1(a) shows the ideal noise-free sinogram with the PET images obtain via the FBP (Fig.1(c)) and, OSEM (Fig.1(d)) reconstruction. A corresponding noise contaminated realization is shown in Fig.1(b),(e)-(f). The proposed PCCM and PSS diffusion schemes have been assessed and evaluated against resent Total Variation (TV) denoising techniques, namely, the approach of Chambolle [1], denoted here after as TV-C, and the Nesterov [12] algorithm, denoted as TV-N. 4.2
Quantitative Evaluation Measures
Two types of evaluation measures are adopted. The first set stems from measuring the quality of the filtering techniques whilst the second set originates from validating the quality of the PET reconstruction. As ground-truth information, the former uses the noise-free image, whilst the latter needs prior identification of the important areas by a medical professional.
218
M. Alrefaya et al.
(a)
(b)
(c)
(d)
(e)
(f)
Fig. 1. (a) Original simulated sinogram and its reconstructed PET image using (b) FBP (c) OSEM reconstruction. (d) An example of one realization of a noisy sinogram and (e-f) the corresponding reconstructed PET images. The tumors (ROI) are the 3 clearly visible white spots.
Denoising Quality. The idea is to verify the quality of the denoised sinogram, ut , with respect to the noise-free image I. In this work, we adopt the following measures [22]: DQ1: The Peak Signal to Noise Ratio (PSNR) is a statistical measure of error, used to determine the quality of the filtered images. It represents the ratio of a signal power to the noise power corrupting. Obviously, one sees that the higher the PSNR, the better the quality. P SN R(t) = 10log10
Card(Ω) |I(p) − ut (p)|
(15)
p∈Ω
DQ2: The correlation (Cmρ ) between the noise-free and the filtered image. The higher this correlation the better the quality is. Cmρ (t) = ρ [I, ut ]
(16)
DQ3: The calculated variance of the noise (NV) describes the remaining noiselevel. Therefore, it should be as small as possible. NV(t) = V ar (|I − ut |)
(17)
In this work, we are interested in comparing the maximum of each measure for the different filtering approaches. The latter yields the best obtainable result per measure.
A Curvature Motion Filter for PET
219
The Contrast Recovery Curve. For evaluating the filtering on the reconstructed PET images, the filtered data, at discrete scales for the proposed PDE approaches, and regularization parameters values, for the TV approaches, were reconstructed using the FBP and the OSEM approaches. With the reconstructed data sets we determine the contrast recovery curve using a set of region of interests that were identified by a medical professional. In our case, the 3 white spots that represent tumors in Fig.1. This was accomplished by quantifying a Contrast Gain, cg, and coefficients of variations V arcg . We calculate the contrast gain cgi for each realizations i ∈ [1, N = 50] and its overall variance. Let R = {r1 , r2 , . . . , rn } be the set of identified ROIs (n = 3 in our case), and B a representative background tissue area,then: (i) (i) 1 1 cgi (t) = n1 U (p, t) − Card(B) U (p, t) Card(r) r∈R
V arcg (t) =
1 N
N
p∈r
(cgi (t) −
i=1
1 N
N
p∈B
cgj (t))
2
(18)
j=1
where p is a pixel. The V arcg versus cg plot provides a straightforward evaluation method for the contrast-noise tradeoff [3]. The best quality PET reconstruction is situated in the upper, i.e. high contrast gain, left, i.e. high stability, area of the plot. 4.3
Evaluation
A fundamental issue with scale-spaces induced by diffusion processes, as the ones proposed in this paper, is the automatic selection of the most salient scale. For our PET sinogram denoising application, we use an earlier proposed optimal scale selection approach [22], where the maximum correlation method has been adopted: σ[no (t0 )] topt = argmax Cmp (t) = argmax σ[ut ] + σ[no (t)] (19) σ[ut0 ] withno is the so-called outlier noise, which we estimated using the proposed wavelet-based noise estimation. Note that, t0 is the zeroth scale, thus ut0 = f and n(t0 ) represents the initial amount of noise. Figures 2.(a),(d) illustrates the obtained optimal scale using the PSS approach, and and the TV-C results, respectively. The Table 1 lists the quantitative results comparing the different denoising approaches. The best performing filtering method, per measure is, displayed in bold. As it can be seen, the best performing filtering is achieved when using the PSS. Furthermore, we notice that for all the used measures, the proposed diffusion methods outperform the considered TV-based filters. Figure 3 depicts the Contrast Recovery Curves for the investigated filtering methods. Recall that, the best enhancement is obtained when the contrast gain is as high as possible whilst the variance over it remains as small as possible. Furthermore, the degree of smoothness of the curve indicates the stability and biasing level.
220
M. Alrefaya et al. Table 1. Denoising quality measures v.s. filtering approaches f PCCM ePCCM PSS TV-C TV-N P SN R(topt ) -17.510 -4.1000 -4.1000 -3.9900 -5.7200 -5.1100 NV(topt ) 7.5100 1.6000 1.6000 1.5500 1.9300 1.8000 Cmρ (topt ) 0.8500 0.9915 0.9916 0.9917 0.9876 0.9890
(a)
(b)
(c)
(d)
(e)
(f)
Fig. 2. Enhanced sinogram and reconstructed PET images. PSS-based approach first row, TV-based approach second row. 1.5
Contrast Gain
1.4
1.3
1.2
PCCM + FBP ePCCM + FBP PSS + FBP PCCM + OSEM ePCCM + OSEM PSS + OSEM TV−N + FBP TV−C + FBP TV−N + OSEM TV−C + OSEM
1.1
1
0.9
0.8 1.5
Variance x 10 2
2.5
3
−3
3.5
4
4.5
Fig. 3. The contrast-noise curve. The best performance occurs in the area where the contrast gain is high and its variance is low.
A Curvature Motion Filter for PET
221
We may observe that all investigated methods have a good performance. However, the proposed PSS yields the best performance on the given data set.
5
Conclusions
Experiments show that combining the probabilistic diffusivity function with the curvature motion diffusion produces a powerful nonlinear filtering method that is appropriate for PET sinograms. It preserves the boundaries of the curvy shape features and wisely smoothes the regions of interest as well as the other regions. Our findings show that the PCCM method smoothes the PET images and keeps the boundaries of the important features, while the weak edges in some cases are vanished. On the other hand, the ePCCM method overcome this problem and the contrast recovered better in the ROIs by the enhancing term in the filter. This filter gives a well smoothed image and preserves the edges, and gains the advantage of the curvature motion diffusion and the shock filter. The PSS approach deal better with the problem of the poor and discontinuity of edges which is common in the PET images. Using the probabilistic diffusion function has proven to be an effective and suitable tool for controlling the diffusion process in the proposed scheme. The results, as shown in the contrast-noise curves, demonstrate that this function has a great capability to detect and enhance the important features edge’s in the high noisy sinogram images. Moreover, The proposed diffusivity function has no free parameters to optimize. All parameters are image-based, and are automatically estimated and proved to give the the best results.
References 1. Chambolle, A.: An Algorithm for Total Variation Minimization and Applications. JMIV 20, 89–97 (2004) 2. Chan, T., Li, H., Lysaker, M., Tai, X.C.: Level Set Method for Positron Emission Tomography. International Journal of Biomedical Imaging 2007 (2007) 3. Comtat, C., Kinahan, P.E., Fessler, J., Beyer, T., Townsend, D.W., Defrise, M., Michel, C.: Clinically feasible reconstruction of 3D whole-body PET/CT data using blurred anatomical labels. Phys. Med. Biol. 47, 1–20 (2002) 4. Demirkaya, O.: Diffusion Filtering of Functional Images using the structural information available in Hyprid imaging modalities. In: IEEE Medical Imaging Symposium, Germany (2008) 5. Demirkaya, O.: Post-reconstruction filtering of positron emission tomography whole-body emission images and attenuation maps using nonlinear diffusion filtering. Acad. Radiol. 11, 1105–1114 (2004) 6. Demirkaya, O.: Anisotropic diffusion filtering of PET attenuation data to improve emission images. Physics in Medicine Biology 47(20), 271–278 (2002) 7. Didas, S., Weickert, J.: Combining Curvature Motion and Edge-Preserving Denoising. In: Sgallari, F., Murli, A., Paragios, N. (eds.) SSVM 2007. LNCS, vol. 4485, pp. 568–579. Springer, Heidelberg (2007)
222
M. Alrefaya et al.
8. Hudson, M., Larkin, R.: Accelerated image reconstruction using ordered subsets of projection data. IEEE Trans. Med. Imag. 13(4), 601–609 (1994) 9. Happonen, A.P., Koskinen, M.O.: Experimental Investigation of Angular Stackgram Filtering for Noise Reduction of SPECT Projection Data: Study with Linear and Nonlinear Filters. International Journal of Biomedical Imaging 2007 (2007) 10. Jonsson, E., Huang, S.C., Chan, T.: Total Variation Regularization in Positron Emission Tomography. UCLA, Tech. Rep. no. 48 (1998) 11. Kak, C.A., Slaney, M.: Principles of Computerized Tomographic Imaging. IEEE Press, Los Alamitos (1999) 12. Nesterov, Y.: Smooth minimization of non-smooth functions. Mathematic Programming, Series A 103, 127–152 (2005) 13. Osher, S., Sethian, J.A.: Fronts propagating with curvature-dependent speed: algorithms based on Hamilton Jacobi formulations. Journal of Computational Physics 79(1), 12–49 (1988) 14. Padfield, D.R., Manjeshwar, R.: Adaptive conductance filtering for spatially varying noise in PET images. Progress in biomedical optics and imaging 7(3) no. 30 (2006) 15. Perona, P., Malik, J.: Scale space and edge detection using anisotropic diffusion. IEEE Transactions on Pattern Analysis and Machine Intelligence 12, 629–639 (1990) 16. Pizurica, A., Vanhamel, I., Sahli, H., Philips, W., Katartzis, A.: A Bayesian formulation of edge-stopping functions in non-linear diffusion. IEEE Signal Processing Letters 13(8), 501–504 (2006) 17. Rudin, L.I., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Physica D 60, 259–268 (1992) 18. Rudin, L.I., Osher, S.: Feature-oriented image enhancement with shock filters. Technical Report, Department of Computer Science, California Institute of Technology (1989) 19. Sapiro, G.: Geometric partial differential equations and image analysis. University Press, Cambridge (2001) 20. Sumengen, B., Manjunath, B.S.: Edgeflow-driven Variational Image Segmentation: Theory and Performance Evaluation. Technical Report, Department of Electrical and Computer Engineering University of California, Santa Barbara (2006) 21. Turkheimer, F.E., Boussion, N., Anderson, A.N., Pavese, N., Piccini, P., Visvikis, D.: PET Image Denoising Using a Synergistic Multiresolution Analysis of Structural (MRI/CT) and Functional Datasets. The Journal of nuclear medicine 49, 657–666 (2008) 22. Vanhamel, I., Mihai, C., Sahli, H., Katartzis, A., Pratikakis, I.: Scale Selection for Compact Scale-Space Representation of Vector-Valued Images. International Journal of Computer Vision 4485 (2008) 23. Vanhamel, I.: Vector valued nonlinear diffusion and its application to image segmentation Ph.D. Thesis, Vrije Universiteit Brussel, Faculty of Engineering Sciences, Electronics and Informatics (ETRO) (2006) 24. Wang, Y., Zhou, H.: Total Variation Wavelet-Based Medical Image Denoising. International Journal of Biomedical Imaging 2006 (2006) 25. Wang, W.: Anisotropic Diffusion Filtering for Reconstruction of Poisson Noisy Sinograms. Journal of Communication and Computer 2(11), 16–23 (2005) 26. Weickert, J.: Anisotropic diffusion in image processing. ECMI Series. TeubnerVerlag, Stuttgart (1998)
A Curvature Motion Filter for PET
223
27. Weiss, P., Aubert, G., Blanc-Fraud, L.: Efficient schemes for total variation minimization under constraints in image processing. Technical Report 6260, INRIA (2007) 28. Zhu, H., Shu, H., Zhou, J., Bao, X., Luo, L.: Bayesian algorithms for PET image reconstruction with mean curvature and Gauss curvature diffusion regularizations. Computers in Biology and Medicine 37(6), 793–804 (2007) 29. Zhu, H., Shu, H., Zhou, J., Toumoulin, C., Luo, L.: Image reconstruction for positron emission tomography using fuzzy nonlinear anisotropic diffusion penalty. Med. Biol. Eng. Comput. 44(11), 983–997 (2006)
Finsler Geometry on Higher Order Tensor Fields and Applications to High Angular Resolution Diffusion Imaging Laura Astola and Luc Florack Department of mathematics and computer science, Eindhoven University of Technology, PO Box 513, NL-5600 MB Eindhoven, The Netherlands [email protected]
Abstract. We study three dimensional volumes of higher order tensors, using Finsler geometry. The application considered here is in medical image analysis, specifically High Angular Resolution Diffusion Imaging (HARDI) [1] of the brain. We want to find robust ways to reveal the architecture of the neural fibers in brain white matter. In Diffusion Tensor Imaging (DTI), the diffusion of water is modeled with a symmetric positive definite second order tensor, based on the assumption that there exists one dominant direction of fibers restricting the thermal motion of water molecules, leading naturally to a Riemannian framework. HARDI may potentially overcome the shortcomings of DTI by allowing multiple relevant directions, but invalidates the Riemannian approach. Instead Finsler geometry provides the natural geometric generalization appropriate for multi-fiber analysis. In this paper we provide the exact criterion to determine whether a field of spherical functions has a Finsler structure. We also show a fiber tracking method in Finsler setting. Our model also incorporates a scale parameter, which is beneficial in view of the noisy nature of the data. We demonstrate our methods on analytic as well as real HARDI data.
1
Introduction
High Angular Resolution Diffusion Imaging (HARDI) is a non-invasive medical imaging modality that measures the attenuation of directional MRI (Magnetic Resonance Imaging) signal due to the diffusion of water molecules. Diffusion weighted measurements are taken in several directions, typically ranging from 50 to 130 (equidistant) angular directions. It is assumed that this diffusion of water molecules reveals relevant information of the underlying tissue architecture. The so-called apparent diffusion coefficient, D(g), is computed from the StejskalTanner [2] formula S(g) = exp(−bD(g)), (1) S0
The Netherlands Organisation for Scientific Research (NWO) is gratefully acknowledged for financial support.
X.-C. Tai et al. (Eds.): SSVM 2009, LNCS 5567, pp. 224–234, 2009. c Springer-Verlag Berlin Heidelberg 2009
Finsler Geometry on HOT Fields
225
where S(g) is the signal associated with gradient direction g, S0 the signal obtained when no diffusion gradient is applied, and b is a parameter associated with the imaging protocol. In the Diffusion Tensor Imaging framework, (1) is interpreted as S(g) = exp(−bg T Dg) , S0
(2)
with the 3 × 3 two-tensor D describing the probability of directional diffusivity at each voxel. A natural way to do geometric analysis on the image, is to use the inverse of the diffusion tensor D as the Riemann metric tensor [3]. This approach has been exploited to some extent in the DTI literature [4], [5], [6], [7]. Since HARDI data typically contains more directional measurements than the traditional DTI, we study it as a metric space, but using a more refined model for directional information than can be accounted for by using only the local position dependent inner product i.e. Riemannian metric. Higher order tensor representations [8], [9], [10], [11] of HARDI data are well suited to differential geometric methods. We mention that Finsler geometry has already been introduced in HARDI setting. In the work of Melonakos et al. [12] the homogeneity condition is forced by normalizing the parameter-vectors, but we take a different approach, using higher order monomial tensors and an ODE-based fiber tracking method. This paper is organized as follows. In section 2, we give a very short introduction to Finsler geometry and in section 3, we show that indeed HARDI measurements can be modeled with a Finsler-structure and give the specific condition which ensures this. In section 4 we discuss how to switch back and forth between iterative polynomial tensor fitting, that allows Laplace-Beltrami smoothing, and a monomial tensor fitting convenient for constructing a Finsler-norm. In section 5 we show some results of fiber-tracking based on the local Finsler metric and demonstrate it on an analytical example as well as on a real HARDI data of a rat brain scan. In the appendix we will give the details of the construction of the strong convexity criterion.
2
Finsler Geometry
In a perfectly homogeneous and isotropic medium, geometry is Euclidean, and shortest paths are straight lines. In an inhomogeneous space, geometry is Riemannian and the shortest paths are geodesics. If a medium is not only inhomogeneous, but also anisotropic1 , i.e. has innate directional structure, the appropriate geometry is Finslerian [13] [14] and the shortest paths are correspondingly Finsler-geodesics. As a consequence the metric tensor depends on both, position and direction. This is also a natural model for high angular resolution diffusion images. 1
We will call a medium isotropic if it is endowed with a direction independent inner product, or Riemannian metric. In the literature such a medium is also often referred to as anisotropic due to the directional bias of the metric itself.
226
L. Astola and L. Florack
Definition 1. We denote the bundle of tangent spaces T(x,y) M (y = 0) as T M \ {0}. A Finsler norm is a function F : TM → [0, ∞) that satisfies each of the following criteria: 1. Differentiability: F is C ∞ on the tangent bundle T M \ {0}. 2. Homogeneity: F (x, λy) = λF (x, y). 3. Strong convexity: The Hessian matrix, with components gij (x, y) =
1 ∂ 2 F 2 (x, y) , 2 ∂y i ∂y j
(3)
is positive definite at every point (x, y) of T M \ {0}.
3
Finsler Norm on HARDI Higher Order Tensor Fields
We want to show that higher order tensors, such as those fitted to HARDI data, do define a Finsler norm, which can be used in the analysis of this data. We take as a point of departure a given orientation distribution function (ODF), which if normalized, is a probability density function on the sphere and which can be computed from the data by using one of the methods described in the literature [15], [16], [17], [18], [19]. It models the probability that a given direction corresponds to a direction of a fiber. We use the heuristics that a high probability of finding a fiber in direction y corresponds to a larger diffusivity and at the same time to a shorter travel time from the diffusing particle point of view. Just as in the Riemannian framework, we can actually take our metric tensor to be the inverse of a local (y-dependent) two-tensor. We use the Einstein summation convention ai bi = i ai bi , and put y = (y1 , y2 , y2 ) = (sin θ cos ϕ, sin θ sin ϕ, cos θ) ,
(4)
thus y denotes a unit vector while y = ||y||y is a general vector in R3 . We denote the higher order spherical tensor (a homogeneous polynomial restricted to sphere) approximating the ODF as D. As an example, we show how a field of sixth order tensors D(x) defines a Finsler norm. This can be extended directly to all even order tensors. We put 1/6 F (x, y) = Dijklmn (x)y i y j y k y l y m y n . (5) In the following, we verify the defining criteria stated in Definition 1. 1. Differentiability: The tensor field D(x) is constructed by fitting a tensor to the set of angular samples at each voxel, using a least squares method. The data set with fixed angle is continuous in x by linear interpolation between the sample points and differentiable w.r.t. x using Gaussian derivatives. Therefore the tensor field itself is differentiable in x, and because D is always positive, differentiability of F w.r.t. x follows. The differentiability of F in y is obvious from Eq. (5).
Finsler Geometry on HOT Fields
227
Fig. 1. Left:A fourth order spherical harmonic (or tensor), representing the (not convexified) norm function and 3 ellipsoids illustrating the metric tensors corresponding to the 3 vectors with same color. Right: Similarly a sixth order spherical harmonic function with 3 metric tensors.
2. Homogeneity: Indeed for any α ∈ R+ , x ∈ M , v ∈ Tx M : 1/6 = αF (x, v) . F (x, αv) = Dijklmn (x)αv i αv j αv k αv l αv m αv n
(6)
3. Strong convexity: We now state a strong convexity criterion for a general Finsler norm in R3 , by analogy to the R2 -criterion by Bao et al [13]. We have put the derivation of the condition into the appendix, and merely state the result here. We consider the so-called indicatrix of the norm function F at any fixed x, which is the set {g | g : (θ, ϕ) → R3 , F (g) = 1}. In our case the indicatrix is the ODF, which can be easily seen from the homogeneity condition 2. in Definition 1. F (y(θ, ϕ)) =
1 =⇒ F (ODF (θ, ϕ) · y(θ, ϕ)) = 1 . ODF (θ, ϕ) 2
∂ ∂ We denote g˙ θ := ∂θ (g), g¨θ := ∂θ 2 (g) and similarly for ϕ. We define the following three matrices: ⎛ 1 2 3⎞ ⎛ 1 2 3⎞ ⎛ 1 2 3⎞ g¨ϕ g¨ϕ g¨ϕ g g g g¨θ g¨θ g¨θ m = ⎝ g˙ θ1 g˙ θ2 g˙ θ3 ⎠ , mθ = ⎝ g˙ θ1 g˙ θ2 g˙ θ3 ⎠ , mϕ = ⎝ g˙ θ1 g˙ θ2 g˙ θ3 ⎠ . (7) g˙ ϕ1 g˙ ϕ2 g˙ ϕ3 g˙ ϕ1 g˙ ϕ2 g˙ ϕ3 g˙ ϕ1 g˙ ϕ2 g˙ ϕ3
Then the strong convexity requires: (gij y˙ θi y˙ ϕj )2 det(mϕ ) det(mθ ) > 0 , and > . det(m) det(m) gij y˙ θi y˙ θj
(8)
Since we use linear interpolation between tensors, we only need to check the condition at original data-points. This condition is always met in our ODF-data, and we expect it to hold quite generally. The goal of this section was to define a Finsler-structure and in particular a Finsler metric tensor gij (x, y) corresponding to a given tensorial ODF. Indeed
228
L. Astola and L. Florack
in case the ODF is a symmetric tensor of order two, this metric tensor is equivalent to the Riemann metric tensor. Following our Finsler approach, instead of one metric tensor per voxel we obtain a bundle of metric tensors at any x. For illustration, see Fig.1.
4
Transforming a Polynomial Tensor to a Monomial Tensor
Assume we wish to apply Laplace-Beltrami smoothing to our spherical data, by which we obtain a field of spherical functions at any desired scale, and that we wish to use a tensorial representation of the data instead of spherical harmonics. As is shown in [10], this smoothing is easy to do, using iterative polynomial tensor fitting. The point here is that for Finsler analysis, we would rather work with a tensor representation of monomial form D(y) = Di1 ···in yi1 · · · yin ,
(9)
than with the equivalent polynomial expression ˜ D(y) =
n
˜ i1 ···i yi1 · · · yik , D k
(10)
k=0
but still exploit the convenient (co-domain) scale space representation of the latter: n ˜ i1 ···i yi1 · · · yik . ˜ τ) = e−τ k(k+1) D (11) D(y, k k=0
This poses no problem, since we can rather easily transform the polynomial expression to a monomial one, using the fact that our polynomials are restricted to the sphere (eq. (4)), thus we may expand a lower order tensor to a sparse higher order one and symmetrize it. We can also always transform the monomial expression to polynomial sum of irreducible monomial tensors using Clebschprojection [20].
5
Fiber Tracking in HARDI Data Using Finsler Geometry
In DTI setting the most straightforward way of tracking fibers is to follow the principal eigenvector corresponding to the largest eigenvalue of the diffusion tensor until some stopping criterion. This method cannot reveal crossings and only provides a single direction (if at all) per voxel. Instead computing the shortest paths according to the diffusion-induced Riemann metric tensor, we could expect these to be the candidates for real fibers [5]. Of course, most of the shortest paths (geodesics) are not representing actual fibers, and therefore we should extract the potential neural fibers from arbitrary geodesics based on their connectivity [6]. We show some results of solving well-connected geodesics in an analytic as well as in a real rat brain data.
Finsler Geometry on HOT Fields
5.1
229
Analytic Tensor Field
We treat an analytic norm field in R2 , but the situation can be directly extended to R3 . Let us take as a convex norm function at each spatial position 1 1 (12) F (ϕ) = (cos 4ϕ + 4) 4 = 5 cos4 ϕ + 2 cos2 ϕ sin2 ϕ + 5 sin4 ϕ 4 . This is an example of fourth order tensor on unit vectors. Such a tensor field could represent an infinitely dense field of orthogonally crossing fibers. From the fact that F has no x-dependence we conclude that the geodesic coefficients vanish and that the geodesics coincide with the Euclidean geodesics γ(t) = (t · cos ϕ, t · sin ϕ), i.e. straight lines. However the so-called connectivity of a geodesic [6], [21] is relatively large, only in cases, where the directional norm function is correspondingly small. In Finsler setting the connectivity measure m(γ) is:
ηij γ˙ i γ˙ j dt m(γ) = , (13) gij (γ, γ) ˙ γ˙ i γ˙ j dt where the ηij (γ) represents the covariant Euclidean metric tensor which in Cartesian coordinates reduces to the constant identity matrix, γ˙ the tangent to the curve γ and gij (γ, γ) ˙ the Finsler-metric tensor (which depends not only on the position on the curve but also on the tangent of the curve). For illustration we compute explicitly the metric tensors, using Cartesian coordinates:
1 g11 g12 gij = , (14) (5 cos ϕ4 + 2 cos ϕ2 sin ϕ2 + 5 sin ϕ4 )3/2 g21 g22 where g11 = 5(5 cos ϕ6 + 3 cos ϕ4 sin ϕ2 + 15 cos ϕ2 sin ϕ4 + sin ϕ6 ) g12 = g21 = −48 cos ϕ3 sin ϕ3 g22 = 5(cos ϕ6 + 15 cos ϕ4 sin ϕ2 + 3 cos ϕ2 sin ϕ4 + 5 sin ϕ6 ) 1 2
1 2
g˙ g ¨ 2 The strong convexity criterion gg¨˙ 1 gg˙ 2 − −g1 g˙ 2 > 0 in R [13] on the indicatrix g(ϕ), for metric (14) is satisfied for every ϕ, since
13 − 8 cos 4ϕ g¨1 g˙ 2 − g˙ 1 g¨2 = >0. 1 2 1 2 g˙ g − g g˙ (4 + cos 4ϕ)2
(15)
The connectivity measure for a (Euclidean) geodesic γ can be computed analytically:
dt , (16) m(γ) =
(4 + cos(4ϕ))1/4 dt 5π 7π which gives the maximal connectivities in directions { π4 , 3π 4 , 4 , 4 }, as expected. See Fig. 2 for an illustration. We observe that on such a norm field the Riemannian (DTI) framework would result in Euclidean geodesics and constant connectivity over all geodesics thus revealing no information at all of the angular heterogeneity.
230
L. Astola and L. Florack
20
3
2
10 1
0 3
2
1
1
2
3
1
10
2
20 20
10
0
10
20
3
Fig. 2. Left:A field of fourth order spherical harmonics as in the norm function eq. (12) representing dense crossings and some well connected geodesics, colored in red. Right: 200 equiangular metric tensors of the same norm function, and an ellipse with light blue color corresponding to the metric in direction ϕ = π4 .
Fig. 3. Left:Finsler geodesics emanating from a voxel, and the most connective ones in red. Right: Fibers through same neighborhood in the traditional DTI principal eigenvector tracking.
5.2
Real Rat Brain Data
The Subthalamic Nucleus is a small area in the brain, that is involved in physiopathology of Parkinson’s disease [22]. We computed the Finsler geodesics and their connectivities, having an initial point in several central voxels in the Subthalamic Nucleus. These voxels were located based on comparison to an atlas of rat brain [23]. We tracked Finsler geodesics using the standard equation (ODEformulation) [14](p.78) and second order Taylor approximation, with initial directions as the 49 measurement directions, stepsize 0.2 voxel size and for 10 steps. Then we selected those 30% of all geodesics that have the best connectivity. Compared to the traditional DTI-tracking, we found that one of the main
Finsler Geometry on HOT Fields
231
directions with strong connectivity typically coincide with the DTI-fibers, but we also found other potential fiber directions. For illustration see Fig. 3.
6
Conclusions and Future Work
We have seen that it is indeed possible to analyze spherical tensor fields using Finsler geometry. It gives new methods to work with the data and also has the potential to give new information on the data. Finsler geodesics and Finsler curvatures are examples of geometric measures that can be applied on HARDI fiber-analysis, and which will be a subject of extensive future work.
Acknowledgement The rat brain data acquired for a study [24], was kindly provided by Ellen Brunenberg.
References 1. Tuch, D., Reese, T., Wiegell, M., Makris, N., Belliveau, J., van Wedeen, J.: High angular resolution diffusion imaging reveals intravoxel white matter fiber heterogeneity. Magnetic Resonance in Medicine 48(6), 1358–1372 (2002) 2. Stejskal, E., Tanner, J.: Spin diffusion measurements: Spin echoes ion the presence of a time-dependent field gradient. The Journal of Chemical Physics 42(1), 288–292 (1965) 3. Cohen de Lara, M.: Geometric and symmetry properties of a nondegenerate diffusion process. The Annals of Probability 23(4), 1557–1604 (1995) 4. O’Donnell, L., Haker, S., Westin, C.F.: New approaches to estimation of white matter connectivity in diffusion tensor MRI: Elliptic PDEs and geodesics in a tensorwarped space. In: Dohi, T., Kikinis, R. (eds.) MICCAI 2002. LNCS, vol. 2488, pp. 459–466. Springer, Heidelberg (2002) 5. Lenglet, C., Deriche, R., Faugeras, O.: Inferring white matter geometry from diffusion tensor MRI: Application to connectivity mapping. In: Pajdla, T., Matas, J.G. (eds.) ECCV 2004. LNCS, vol. 3024, pp. 127–140. Springer, Heidelberg (2004) 6. Astola, L., Florack, L., ter Haar Romeny, B.: Measures for pathway analysis in brain white matter using diffusion tensor images. In: Karssemeijer, N., Lelieveldt, B. (eds.) IPMI 2007. LNCS, vol. 4584, pp. 642–649. Springer, Heidelberg (2007) 7. Astola, L., Florack, L.: Sticky vector fields and other geometric measures on diffusion tensor images. In: MMBIA 2008, IEEE Computer Society Workshop on Mathematical Methods in Biomedical Image Analysis, held in conjunction with CVPR 2008, Anchorage, Alaska, The United States. CVPR, vol. 20, pp. 1–7. Springer, Heidelberg (2008) 8. Özarslan, E., Mareci, T.: Generalized diffusion tensor imaging and analytical relationships between diffusion tensor imaging and high angular resolution diffusion imaging. Magnetic resonance in Medicine 50, 955–965 (2003) 9. Barmpoutis, A., Jian, B., Vemuri, B., Shepherd, T.: Symmetric positive 4th order tensors and their estimation from diffusion weighted MRI. In: Karssemeijer, N., Lelieveldt, B. (eds.) IPMI 2007. LNCS, vol. 4584, pp. 308–319. Springer, Heidelberg (2007)
232
L. Astola and L. Florack
10. Florack, L., Balmashnova, E.: Decomposition of high angular resolution diffusion images into a sum of self-similar polynomials on the sphere. In: Proceedings of the Eighteenth International Conference on Computer Graphics and Vision, GraphiCon 2008, Moscow, Russian Federation, June 2008, pp. 26–31 (2008) (invited paper) 11. Florack, L., Balmashnova, E.: Two canonical representations for regularized high angular resolution diffusion imaging. In: MICCAI Workshop on Computational Diffusion MRI, New York, USA, September 10, 2008, pp. 94–105 (2008) 12. Melonakos, J., Pichon, E., Angenent, S., Tannenbaum, A.: Finsler active contours. IEEE Transactions on Pattern Analysis and Machine Intelligence 30(3), 412–423 (2008) 13. Bao, D., Chern, S.S., Shen, Z.: An Introduction to Riemann-Finsler Geometry. Springer, Heidelberg (2000) 14. Shen, Z.: Lectures on Finsler Geometry. World Scientific, Singapore (2001) 15. Tuch, D.: Q-ball imaging. Magnetic Resonance in Medicine 52(4), 577–582 (2002) 16. Jansons, K., Alexander, D.: Persistent angular structure: New insights from diffusion magnetic resonance imaging data. Inverse Problems 19, 1031–1046 (2003) 17. Özarslan, E., Shepherd, T., Vemuri, B., Blackband, S., Mareci, T.: Resolution of complex tissue microarchitecture using the diffusion orientation transform. NeuroImage 31, 1086–1103 (2006) 18. Jian, B., Vemuri, B., Özarslan, E., Carney, P., Mareci, T.: A novel tensor distribution model for the diffusion-weighted MR signal. NeuroImage 37, 164–176 (2007) 19. Descoteaux, M., Angelino, E., Fitzgibbons, S., Deriche, R.: Regularized, fast and robust analytical q-ball imaging. Magnetic Resonance in Medicine 58(3), 497–510 (2006) 20. Müller, C. (ed.): Analysis of Spherical Symmetries in Euclidean Spaces. Applied Mathematical Sciences, vol. 129. Springer, New York (1998) 21. Prados, E., Soatto, S., Lenglet, C., Pons, J.P., Wotawa, N., Deriche, R., Faugeras, O.: Control Theory and Fast Marching Techniques for Brain Connectivity Mapping. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition, New York, USA, vol. 1, pp. 1076–1083. IEEE Computer Society Press, Los Alamitos (2006) 22. Hamani, C., Saint-Cyr, J., Fraser, J., Kaplitt, M., Lozano, A.: The subthalamic nucleus in the context of movement disorders. Brain, a Journal of Neurology 127, 4–20 (2004) 23. Paxinos, G., Watson, C.: The Rat Brain In Stereotaxic Coordinates. Academic Press, San Diego (1998) 24. Brunenberg, E., Prckovska, V., Platel, B., Strijkers, G., ter Haar Romeny, B.M.: Untangling a fiber bundle knot: Preliminary results on STN connectivity using DTI and HARDI on rat brains. In: Proceedings of the 17th Meeting of the International Society for Magnetic Resonance in Medicine (ISMRM), Honolulu, Hawaii (2009)
Finsler Geometry on HOT Fields
233
Appendix We seek the general condition for gij (y)v i v j > 0 ,
(17)
to be valid in R3 (= Tx M ). From the homogeneity of the norm function F , it follows that it is sufficient to have this condition on the unit level set of the norm. We consider this level surface i.e. the set of vectors y for which F (y) = 1 and a parametrization y(θ, ϕ) = (y 1 (θ, ϕ), y 2 (θ, ϕ), y 3 (θ, ϕ)). In what follows we abbreviate gij = gij (x, y). From F (y) = 1 we have gij y i y j = 1 .
(18)
Taking derivatives of both sides and using a consequence of Euler’s theorem for homogeneous functions ( [13] p.5) that says ∂gij k y =0, ∂y k
(19)
we obtain gij y˙ θi y j = 0
(20)
gij y˙ ϕi y j = 0 , implying y˙ θ ⊥g y and y˙ ϕ ⊥g y. Taking derivatives once more, we get gij y¨θi y j = −gij y˙ θi y˙ θj gij y¨ϕi y j = −gij y˙ ϕi y˙ ϕj i gij y¨θϕ yj
=
−gij y˙ θi y˙ ϕj
(21) .
We may express an arbitrary vector v as a linear combination of orthogonal basis vectors:
y˙ ϕ , y˙ θ v = αy + β y˙ θ + γ y˙ ϕ − y˙ θ . (22) y˙ θ , y˙ θ We substitute this expression for v to the left hand side of (17) and obtain: (gij y˙ θi y˙ ϕj )2 i j 2 i j 2 i j 2 i j , (23) gij v v = α gij y y + β gij y¨θ y + γ gij y¨ϕ y − gij y˙ θi y˙ θj because the mixed terms vanish due to the orthogonality of basis vectors. On the other hand, for y’s on the indicatrix we have as a consequence of ∂F Euler’s theorem on homogeneous functions (denoting Fyi = ∂y i ): Fyi y i = F (y) = 1 .
(24)
234
L. Astola and L. Florack
Differentiating eq. (24) w.r.t. θ and ϕ, we obtain two equations: Fyi y˙ θi = 0
(25)
Fyi y˙ ϕi
(26)
=0,
for F is a homogeneous function. The matrices m, mθ , mϕ are as defined in eq. (7). Solving system of equations (24), (25) and (25) we get: Fy1 =
y˙ ϕ2 y˙ θ3 − y˙ ϕ3 y˙ θ2 y˙ ϕ3 y˙ θ1 − y˙ ϕ1 y˙ θ3 y˙ ϕ1 y˙ θ2 − y˙ ϕ2 y˙ θ1 , Fy2 = , Fy3 = . det(m) det(m) det(m)
(27)
Now using equalities gij y¨i θ y j = Fyk y¨θk ,
Fyi = gij y j , and
det(mθ ) , det(m)
−gij y¨θi y j =
gij y¨i ϕ y j = Fyk y¨ϕk ,
(28)
det(mϕ ) det(m)
(29)
−gij y¨ϕi y j =
we obtain
2
gij v v = α − β i j
2
gij y¨θi y j
−γ
if det(mθ ) >0 det(m)
and
2
gij y¨ϕi y j
−
(gij y˙ θi y˙ ϕj )2
gij y˙ θi y˙ θj
(gij y˙ θi y˙ ϕj )2 det(mϕ ) > . det(m) gij y˙ θi y˙ θj
>0
(30)
(31)
Bregman-EM-TV Methods with Application to Optical Nanoscopy Christoph Brune, Alex Sawatzky, and Martin Burger Westfälische Wilhelms-Universität Münster, Institut für Numerische und Angewandte Mathematik, Einsteinstr. 62, D-48149 Münster, Germany {christoph.brune,alex.sawatzky,martin.burger}@wwu.de http://imaging.uni-muenster.de Abstract. Measurements in nanoscopic imaging suffer from blurring effects concerning different point spread functions (PSF). Some apparatus even have PSFs that are locally dependent on phase shifts. Additionally, raw data are affected by Poisson noise resulting from laser sampling and "photon counts" in fluorescence microscopy. In these applications standard reconstruction methods (EM, filtered backprojection) deliver unsatisfactory and noisy results. Starting from a statistical modeling in terms of a MAP likelihood estimation we combine the iterative EM algorithm with TV regularization techniques to make an efficient use of a-priori information. Typically, TV-based methods deliver reconstructed cartoon-images suffering from contrast reduction. We propose an extension to EM-TV, based on Bregman iterations and inverse scale space methods, in order to obtain improved imaging results by simultaneous contrast enhancement. We illustrate our techniques by synthetic and experimental biological data.
1
Introduction
Image reconstruction is a fundamental problem in many fields of applied sciences, e.g. nanoscopic imaging, medical imaging or astronomy. Fluorescence microscopy for example is an important imaging technique for the investigation of biological (live-) cells, up to nano-scale. In this case image reconstruction arises in form of deconvolution problems. Undesirable blurring effects can be ascribed to diffraction of light. Mathematically, image reconstruction in such applications can often be formulated as the solution of a linear inverse and ill-posed problem. The task consists of computing an estimation of an unknown object from given measurements. Typically these problems deal with Fredholm integral equations of the first kind ¯ , f¯ = Ku
(1)
¯ is a compact operator, f¯ (exact) data and u the desired image. In the where K ¯ is a convolution operator case of nanoscopic imaging K ¯ (Ku)(x) = (k ∗ u)(x) = k(x − y)u(y)dy , Ω X.-C. Tai et al. (Eds.): SSVM 2009, LNCS 5567, pp. 235–246, 2009. c Springer-Verlag Berlin Heidelberg 2009
236
C. Brune, A. Sawatzky, and M. Burger
where k is a convolution kernel, describing the blurring effects created by a ¯ is not suitable, nanoscopic apparatus. Determining u by direct inversion of K since (1) is ill-posed. In such cases regularization techniques are needed to produce reasonable reconstructions. A frequently used way to realize regularization techniques is the Bayesian model, whose aim is the computation of an estimate u of the unknown object by maximizing the a-posteriori probability density p(u|f ) with measurements f . The latter is given according to Bayes formula p(u|f ) ∼ p(f |u) p(u) .
(2)
This approach is called maximum a-posteriori probability (MAP) estimation. If the measurements f are given, we describe the density p(u|f ) as the a-posteriori likelihood function which depends on u only. The Bayesian approach (2) has the advantage that it allows to incorporate additional information about u via the prior probability density p(u) into the reconstruction process. The most frequently used prior densities are Gibbs functions p(u) ∼ e−α R(u) ,
(3)
where α is a positive parameter and R a convex energy. Usual models for the probability density p(f |u) in (2) are Gaussian- or Poisson-distributed raw data f , i.e. 2
p(f |u) ∼ e−Ku−f 2 /(2σ
2
)
,
p(f |u) ∼
(Ku)fi i
fi !
i
e−(Ku)i ,
(4)
¯ In the canonical case of additive where K is a semi-discrete Operator based on K. Gaussian noise (see (4), left) the minimization of the negative log likelihood function (2) leads to classical Tikhonov regularization [1] based on minimizing a functional of the form 1 2 min Ku − f 2 + α R(u) . (5) u≥0 2 The first, so-called data-fidelity term, penalizes the deviation from equality in (1) whereas R is a regularization term as in (3). If we choose K = Id and the total variation (TV) regularization technique R(u) := |u|BV , we obtain the wellknown ROF-model [2] for image denoising. The additional positivity constraint is necessary in typical applications as the unknown represents a density image. In nanoscopic imaging measured data are stochastic and pointwise, more precisely, they are called "photon counts". This property refers to laser scanning techniques in fluorescence microscopy. Consequently, the random variables of measured data are not Gaussian- but Poisson-distributed (see (4), right), with expected value given by equation (1). Hence a MAP estimation via the negative log likelihood function (2) leads to the following variational problem [1] min (Ku − f log Ku) dμ + α R(u) . (6) u≥0
Ω
Bregman-EM-TV Methods
237
Up to additive terms independent of u, the data-fidelity term is the so-called Kullback-Leibler functional (also known as cross entropy or I-divergence) between the two probability measures f and Ku. A particular complication of (6) compared to (5) is the strong nonlinearity in the data fidelity term and resulting issues in the computation of minimizers. In case of K = Id, i.e. in case of Poisson noise removal with total variation regularization, we refer to [3]. In the absence of regularization (α = 0) the EMalgorithm (cf. [4, 5, 6]) has become a standard scheme, which is however difficult to be generalized to regularized cases. Robust solutions of this problem for appropriate models of R is one of the novelties of this paper. The specific choice of the regularization functional R in (6) is important for how a-priori information about the expected solution is incorporated into the reconstruction process. Smooth, in particular quadratic regularizations have attracted most attention in the past, mainly due to the simplicity in analysis and computation. However, such regularization approaches always lead to blurring of the reconstructions, in particular they cannot yield reconstructions with sharp edges. Recently, singular regularization energies, in particular those of 1 or L1 -type, have attracted strong attention. In this work, we introduce an approach which uses total variation (TV) as the regularization functional. TV regularization was derived as a denoising technique in [2] and generalized to various other imaging tasks subsequently. The exact definition of TV [7], used in this paper, is R(u) := |u|BV = sup u divg , (7) g∈C0∞ (Ω,Rd ), ||g||∞ ≤1
Ω
which is formally (true if u is sufficiently regular) |u|BV = Ω |∇u|. The motivation for using TV is the effective suppression of noise and the realization of almost homogeneous regions with sharp edges. These features are attractive for nanoscopic imaging if the goal is to identify object shapes that are separated by sharp edges and shall be analyzed quantitatively. Unfortunately, images reconstructed by methods using TV regularization suffer from loosing contrast. In this paper, we suggest to extend EM-TV by iterative regularization to Bregman-EM-TV, attaining simultaneous contrast enhancement. More precisely, we apply total variation inverse scale space methods by employing the concept of Bregman distance regularization. The latter has been derived in [8] with a detailed analysis for Gaussian-type problems (5) and generalized to time-continuity [9] and Lp -norm data fitting terms [10]. Here, in the case of Poisson-type problems, the method consists in computing a minimizer u1 of (6) with R(u) := |u|BV first. Updates are determined successively by computing l+1 l u = arg min (Ku − f log Ku) dμ + α ( |u|BV − p , u ) , (8) u∈BV (Ω)
Ω
l
where p is an element of the subgradient of the total variation semi norm in ul . Introducing the Bregman distance with respect to | · |BV defined via p˜ (u, u ˜) := |u|BV − |˜ u|BV − ˜ p, u − u ˜ , D|·| BV
p˜ ∈ ∂|˜ u|BV ⊆ BV ∗ (Ω) ,
(9)
238
C. Brune, A. Sawatzky, and M. Burger
where ·, · denotes the duality product, allows to characterize ul+1 in (8) as pl l+1 l u = arg min (Ku − f log Ku) dμ + α D|·|BV (u, u ) . (10) u∈BV (Ω)
Ω
We will see that inverse scale space strategies can noticeably improve reconstructions for inverse problems with Poisson statistics like optical nanoscopy.
2
Reconstruction Methods
In literature there are two types of reconstruction methods that are used in general: analytic (direct) and algebraic (iterative) methods. A classical example for a direct method is the Fourier-based filtered backprojection (FBP). Although FBP is well understood and computationally efficient, iterative type methods obtain more and more attention in the applications mentioned above. The major reason is the high noise level (low SNR) and the type of statistics, which cannot be taken into account by direct methods. Hence, we will give a short review on the Expectation-Maximization (EM) algorithm [4, 11], which is a popular iterative algorithm to maximize the likelihood function p(u|f ) in problems with incomplete data. Then we will proceed to the presentation of the proposed EMTV and Bregman-EM-TV algorithm. 2.1
Reconstruction Method: EM Algorithm
In the absence of prior knowledge any object u has the same relevance, i.e. the Gibbs a-priori density p(u) in (3) is constant. We can then normalize p(u) such that R(u) ≡ 0. Hence (6) reduces to the constrained minimization problem min (Ku − f log Ku) dμ . (11) u≥0
Ω
A suitable iteration scheme for computing stationary points, which also preserves positivity (assuming K preserves positivity), is the so called EM algorithm (cf. [12]) K∗ f , k = 0, 1, . . . . (12) uk+1 = uk ∗ K 1 Kuk For noise-free data f several convergence proofs of the EM algorithm to the maximum likelihood estimate, i.e. the solution of (11), can be found in literature [12,13,14,15]. Besides, it is known that the speed of convergence of iteration (12) is slow. A further property of the iteration is a lack of smoothing, whereby the so-called "checkerboard effect" arises. For noisy data f it is necessary to differentiate between discrete and continuous modeling. In the discrete case, i.e. if K is a matrix and u is a vector the existence of a minimum can be guaranteed since the smallest singular value is bounded by a positive value. Hence, the vectors are bounded during the iteration and convergence is ensured. However, if K is a general continuous operator
Bregman-EM-TV Methods
239
the convergence is not only difficult to prove, but even a divergence of the EM algorithm is possible. Again the reason is the ill-posedness of the integral equation (1), which transfers to problem (11). This aspect can be taken as a lack of additional a-priori knowledge about the unknown u resulting from R(u) = 0. The EM algorithm converges to a minimizer if it exists. Consequently, in the continuous case it is essential to ensure consistence of the given data to prevent divergence of the EM algorithm. As described in [13], the EM iterates show the following typical behavior for ill-posed problems. The (metric) distance between the iterates and the solution decreases initially before it increases as the noise is amplified during the iteration process. This issue might be regulated by using appropriate stopping rules to obtain reasonable results. In [13] it is shown that certain stopping rules indeed allow stable approximations. Ways to improve reconstruction results are TV or Bregman-TV regularization techniques that we will consider in the following section. 2.2
Reconstruction Method: EM-TV Algorithm
The EM or Richardson/Lucy algorithm is currently the standard iterative reconstruction method for deconvolution problems with Poisson noise based on the linear equation (1). However, with the assumption R(u) = 0, no a-priori knowledge about the expected solution is taken into account, i.e. different images have the same a-priori probability. Especially in case of measurements with low SNR the multiplicative fixed point iteration (12) delivers unsatisfactory and noisy results even with early termination. Therefore we propose to integrate nonlinear variational methods into the reconstruction process to make an efficient use of a-priori information and to obtain improved results. An interesting approach to improve the reconstruction is the EM-TV algorithm. In the classical EM algorithm, the negative log likelihood functional (11) is minimized. We modify the functional by adding a weighted TV term [2], min . (13) (Ku − f log Ku) dμ + α|u|BV u∈BV (Ω) u≥0
Ω
This is exactly (6) with TV as regularization functional R. That means images with smaller total variation are preferred in the minimization (have higher prior probability). BV (Ω) is a popular function space in image processing since it can represent discontinuous functions. By minimizing TV the latter are even preferred [16, 17]. Hence, expected reconstructions are cartoon-like images. Obviously, such an approach cannot be used for studying very small structures in an object, but it is perfect for segmenting different cell structures and analyzing them quantitatively. For the solution of (13), we propose a forward-backward splitting algorithm, which can be realized by alternating classical EM steps with almost standard TV minimization steps as encountered in image denoising. The latter is solved by using duality [18] obtaining a robust and efficient algorithm. For designing the proposed alternating algorithm, we consider the first order optimality condition
240
C. Brune, A. Sawatzky, and M. Burger
of (13). Due to the total variation, this variational problem is not differentiable in the usual sense. But the latter is convex since TV is convex and since we can extend the data fidelity term to a Kullback-Leibler functional, cf. [19], without affecting the stationary points. For such problems powerful methods from convex analysis are available, e.g. a generalized derivative called the subdifferential [20], denoted by ∂. This generalized notion of gradients and the Karush-Kuhn-Tucker (KKT) conditions [20, Theorem 2.1.4] yield the existence of a Lagrange multiplier λ ≥ 0 such that ⎧ ⎫ f ⎨ 0 ∈ K ∗1 − K ∗ + α ∂|u|BV − λ ⎬ Ku . (14) ⎩ ⎭ 0 = λu By multiplying (14) with u we can eliminate the Lagrange multiplier and derive the following semi-implicit iteration scheme K∗ f + α ˜ uk pk+1 = 0 (15) uk+1 − uk ∗ K 1 Kuk ˜ := Kα∗ 1 . Interestingly, the second term within with pk+1 ∈ ∂|uk+1 |BV and α this iteration scheme is the EM step in (12). Consequently, method (15) solving variational problem (13), can be realized as a nested two step iteration, ⎧ ⎫ ∗ f ⎨u 1 = u K ⎬ (EM step) k k+ 2 K ∗ 1 Kuk . (16) ⎩ ⎭ uk+1 = uk+ 12 − α ˜ uk pk+1 (TV step) Thus, we alternate an EM step with a TV correction step. The complex second half step from uk+ 12 to uk+1 can be realized by solving the following variational problem, (u − uk+ 12 )2 1 . (17) uk+1 = arg min +α ˜ |u|BV 2 Ω uk u∈BV (Ω) Inspecting the first order optimality condition confirms the equivalence of this minimization with the TV correction step in (16). Problem (17) is just a modified version of the Rudin-Osher-Fatemi (ROF) model, with weight u1k in the fidelity term. This analogy creates the opportunity to carry over efficient numerical schemes known for the ROF-model. For the solution of (17) we use the exact definition of TV (7) with dual variable g and derive an iteration scheme for the quadratic dual problem similar to [18]. The resulting algorithm reads as follows: We initialize the dual variable g 0 with 0 (or the resulting g from the previous TV correction step) and for any n ≥ 0 we compute the update g n+1 =
α uk divg n − uk+ 12 ) g n + τ ∇(˜ 1 + τ |∇(˜ α uk divg n − uk+ 12 )|
,
0 < τ <
1 , 4α ˜ uk
with the constrained damping parameter τ to ensure stability and convergence of the algorithm. For a detailed analytical examination of EM-TV we refer to [21].
Bregman-EM-TV Methods
2.3
241
Extension to Inverse Scale Space: Bregman-EM-TV
The EM-TV algorithm (16) we presented solves the problem (13) and delivers cartoon-reconstructions with sharp edges due to TV regularization. However, the realization of TV steps via the weighted ROF-models (17) has the drawback that reconstructed images suffer from loosing contrast. Thus, we propose to extend (13) and therewith EM-TV by iterative regularization to a simultaneous contrast correction. More precisely, we perform a contrast enhancement by inverse scale space methods and by using the Bregman iteration. These techniques have been derived in [8], with a detailed analysis for Gaussian-type problems (5), and have been generalized in [9, 10]. Following these methods, an iterative refinement is realized by a sequence of modified EM-TV problems based on (13). The inverse scale space methods concerning TV, derived in [8], follow the concept of iterative regularization by the Bregman distance [22]. In case of the Poisson-model the method initially starts with a simple EM-TV algorithm, i.e. it consists in computing a minimizer u1 of (13). Then, updates are determined successively by considering variational problems with a shifted TV, namely (8), where pl is an element of the subgradient of the total variation in ul . The Bregman distance concerning TV is defined in (9). The introduction of this definition allows to characterize the sequence of modified variational problems (8) by addition of constant terms as pl l+1 l u = arg min (Ku − f log Ku) dμ + α D|·|BV (u, u ) . (18) u∈BV (Ω)
Ω
Thus, the first iterate u1 can also be realized by the variational problem (18), if p u0 is constant and p0 := 0 ∈ ∂|u0 |BV . The Bregman distance D|·| does not BV represent a distance in the common (metric) sense, since D is not symmetric in general and the triangle inequality does not hold. Though, compared to (8), the p formulation in (18) offers the advantage that D|·| is a distance measure with BV p D|·| (u, u ˜) ≥ 0 BV
p and D|·| (u, u ˜) = 0 for u = u˜ . BV
Besides, the Bregman distance is convex in the first argument because | · |BV is convex. In general, i.e. for any convex functional J (see e.g. [10]), the Bregman distance can be interpreted as the difference between J(·) in u and the Taylor linearization of J around u ˜ if, in addition, J is continuously differentiable. Before deriving a two-step iteration corresponding to (16) we will motivate the contrast enhancement by iterative regularization in (18). The TV regularization in (13) prefers functions with only few oscillations. The iterative Bregman regularization has the advantage that, with ul as an approximation to the possible solution, additional information is available. The variational problem (18) can be interpreted as follows: search for a solution that matches the Poisson distributed data after applying K and simultaneous minimization of the residual of the Taylor approximation of | · |BV around ul . In the following we will see that this form of regularization does not change the position of gradients with respect
242
C. Brune, A. Sawatzky, and M. Burger
to the last computed EM-TV solution ul but that an increase of intensities is permitted. This leads to a noticeable contrast enhancement. For the derivation of a two-step iteration we consider the first order optimality condition of the variational problem (8) resp. (18). Due to convexity of the Bregman distance in the first argument we can determine the subdifferential of (18). Analogous to the derivation of the EM-TV iteration the subdifferential of the log likelihood functional can be expressed by the Fr´echet derivative in (14). Hence, the optimality condition is given by f ∗ ∗ 0 ∈ K 1 − K + α ( ∂|ul+1 |BV − pl ), pl ∈ ∂|ul |BV . (19) Kul+1 For u0 constant and p0 := 0 ∈ ∂|u0 |BV this delivers a well defined update of the iterates pl , K∗ 1 f l+1 l p 1− ∗ ∈ ∂|ul+1 |BV , := p − α ˜ K 1 Kul+1 where α ˜ := Kα∗ 1 results from an operator normalization. Analogous to EM-TV we can apply the idea of the nested iteration (16) in every refinement step, l = 1, 2, · · · . For the solution of (18) condition (19) yields a strategy consisting of an EM-step ul+1 followed by solving the adapted weighted ROF-problem k+ 1 2
ul+1 k+1
⎫ ⎧ ⎬ ⎨ 1 (u − ul+11 )2 k+ 2 l = arg min + α ˜ ( |u| − p , u ) . BV ⎭ ul+1 u∈BV (Ω) ⎩ 2 Ω k
(20)
Following [8,9,10], we provide an opportunity to transfer the shift-term pl , u to the data-fidelity term. This approach facilitates the implementation of contrast enhancement with Bregman distance via a slightly modified EM-TV algorithm. With the scaling v l := α ˜ pl and (19) we obtain the following update formula K∗ f l+1 l , v0 = 0 . =v − 1− ∗ (21) v K 1 Kul+1 Using this scaled update we can rewrite the second step (20) to ⎧ ⎫ ⎨ 1 (u − ul+11 )2 − 2uul+1 ⎬ vl k k+ 2 = arg min + α ˜ |u| . ul+1 BV k+1 ⎭ ul+1 u∈BV (Ω) ⎩ 2 Ω k Note that l+1 l+1 l 2 l+1 2 l 2 l+1 l+1 l l )2 − 2uul+1 (u − ul+1 k v = (u − (uk+ 1 + uk v )) + (uk ) (v ) − 2uk+ 1 uk v , k+ 1 2
2
holds, where the last two terms are independent of u. Hence (20) ⎧ l 2 ⎨ 1 (u − (ul+11 + ul+1 k v )) k+ 2 = arg min + α ˜ |u|BV ul+1 k+1 ul+1 u∈BV (Ω) ⎩ 2 Ω k
2
simplifies to ⎫ ⎬ , (22) ⎭
Bregman-EM-TV Methods
243
i.e. the second step (20) can be realized by a slight modification of the TV step introduced in (17). Obviously, the efficient numerical implementation of the weighted ROF-problem in Section 2.2 using the exact definition of TV and duality strategies can be applied in complete analogy to (22). The update variable v in (21) is an error function with reference to the optimality condition of the unregularized log-likelihood functional (11). In every refinement step of the Bregman iteration v l+1 differs from v l by the current error in the optimality condition (11). Within the TV-step (22) one observes that an iterative regularization with the Bregman distance leads to contrast enhancement. Instead of fitting to the EM solution ul+1 in the weighted norm, we use a function in the fidelity k+ 1 2
term whose intensities are increased by the error function v l . Resulting from the idea of adaptive regularization v l is weighted by ul+1 k , too. As usual for iterative methods the described reconstruction method by iterative regularization needs a stopping criterion. The latter should stop at an iteration offering a solution that approximates the true image as good as possible. This is necessary to prevent that too much noise arises by the inverse scale space strategy. In the case of Gaussian noise, the discrepancy principle is a reasonable stopping criterion, i.e. the procedure would stop if the residual Kul − f 2 reaches the variance of the noise. In the case of Poisson noise, however, it makes sense to stop the Bregman iteration if the Kullback-Leibler distances of Kul and the given data f reach the noise level. For synthetic data the noise level is given by the KL distance of Ku∗ and f , where u∗ denotes the true, noise-free image. For experimental data it is necessary to find a suitable estimate for the noise level from counts.
3
Results
In recent years revolutionary imaging techniques have been developed in light microscopy with enormous importance for biological and material sciences or medicine. For a couple of decades the technology of light microscopy has been considered to be exhausted, as the resolution is basically limited by Abbe’s law for diffraction of light. By developing stimulated emission depletion (STED)and 4Pi-microscopy now resolutions are achieved that are way beyond these
(a)
(b)
(c)
(d)
(e)
Fig. 1. Synthetic data concerning different PSFs: (a) true image; (b) Gaussian PSF; (c) is convolved with Gaussian PSF and Poisson noise; (d) PSF appearing in 4Pi microscopy; and (e) is convolved with 4Pi PSF and Poisson noise
244
C. Brune, A. Sawatzky, and M. Burger
(a)
(b)
(c)
(f)
(e)
(d)
(g)
(h)
Fig. 2. Synthetic data: (a) raw data; (b) EM reconstruction, 20 its, KL-distance: 3.20; (c) EM-TV, α = 0.04, KL-distance: 2.43; (d) Bregman-EM-TV, α = 0.1, after 4 updates, KL-distance: 1.43; (e) true image; (f)-(h) horizontal slices EM, EM-TV and Bregman-EM-TV compared to true image slice
(a)
(b)
(c)
(d)
(e)
(f)
Fig. 3. Experimental data: (a) Protein Bruchpilot in active zones of neuromuscular synapses in larval Drosophila; (b) EM-TV; (c) Bregman-EM-TV; (d) Protein Syntaxin in cell membrane, fixed mamalian (PC12) cell; (e) EM-TV; and (f) Bregman-EM-TV
Bregman-EM-TV Methods
245
diffraction barrier [23,24]. To get an impression of nanoscopic images blurred by different convolution kernels (PSFs), we refer to Figure 1. Figure 2 illustrates our techniques at a simple synthetic object. With EM-TV (see 2(c) and 2(g)) we get rid of noise and oscillations, but we are not able to separate the objects sufficiently. Using Bregman-EM-TV a considerable improvement resulting from contrast enhancement can be achieved. This aspect is underlined by the values of the KL-distance for the different reconstructions. Figure 3, (a)-(c) demonstrate the protein Bruchpilot [25] and its EM-TV and Bregman-EM-TV reconstruction. Particularly, the latter delivers well separated object segments and a high contrast level. In Figure 3, (d)-(f) we illustrate our techniques by reconstructing Syntaxin [26], a membrane integrated protein participating in exocytosis. Here, the contrast enhancing property of Bregman-EM-TV is observable as well, compared to EM-TV. It is possible to preserve fine structures in the image.
4
Conclusions
We have derived reconstruction methods for inverse problems with Poisson noise. Particularly, we concentrated on deblurring problems in nanoscopic imaging, although the proposed methods can easily be adapted to other imaging tasks, i.e. medical imaging (PET, [27]). Motivated by a statistical modeling we developed a robust EM-TV algorithm that incorporates a-priori knowledge into the reconstruction process. By combining EM with simultaneous TV regularization we can reconstruct cartoon-images with sharp edges, that deliver a reasonable basis for quantitative investigations. To overcome the problem of contrast reduction, we extended the reconstruction to Bregman iterations and inverse scale space methods. We applied the proposed methods to optical nanoscopy and pointed out their improvements in comparison to standard reconstruction techniques. Acknowledgments. This work has been supported by the German Federal Ministry of Education and Research through the project INVERS. C.B. acknowledges further support by the Deutsche Telekom Foundation, and M.B. by the German Science Foundation DFG through the project "Regularisierung mit Singulären Energien". The authors thank Dr. Katrin Willig and Dr. Andreas Schönle (MPI Biophysical Chemistry, Göttingen) for providing experimental data and stimulating discussions.
References 1. Bertero, M., Lantéri, H., Zanni, L.: Iterative image reconstruction: a point of view. In: Mathematical Methods in Biomedical Imaging and Intensity-Modulated Radiation Therapy (IMRT). CRM series, vol. 8 (2008) 2. Rudin, L.I., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Physica D 60, 259–268 (1992) 3. Le, T., Chartrand, R., Asaki, T.J.: A variational approach to reconstructing images corrupted by Poisson noise. J. Math. Imaging Vision 27, 257–263 (2007) 4. Shepp, L.A., Vardi, Y.: Maximum likelihood reconstruction for emission tomography. IEEE Transactions on Medical Imaging 1(2), 113–122 (1982)
246
C. Brune, A. Sawatzky, and M. Burger
5. Richardson, W.H.: Bayesian-based iterative method of image restoration. J. Opt. Soc. Am. 62, 55–59 (1972) 6. Lucy, L.B.: An iterative technique for the rectification of observed distributions. The Astronomical Journal 79, 745–754 (1974) 7. Acar, R., Vogel, C.R.: Analysis of bounded variation penalty methods for ill-posed problems. Inverse Problems 10, 1217–1229 (1994) 8. Osher, S., Burger, M., Goldfarb, D., Xu, J., Yin, W.: An iterative regularization method for total variation based image restoration. Multiscale Modelling and Simulation 4, 460–489 (2005) 9. Burger, M., Gilboa, G., Osher, S., Xu, J.: Nonlinear inverse scale space methods. Commun. Math. Sci. 4(1), 179–212 (2006) 10. Burger, M., Frick, K., Osher, S., Scherzer, O.: Inverse total variation flow. SIAM Multiscale Modelling and Simulation 6(2), 366–395 (2007) 11. Dempster, A.P., Laird, N.M., Rubin, D.B.: Maximum Likelihood from Incomplete Data via the EM Algorithm. J. of the Royal Statistical Society, B 39, 1–38 (1977) 12. Natterer, F., Wübbeling, F.: Mathematical methods in image reconstruction. SIAM Monographs on Mathematical Modeling and Computation (2001) 13. Resmerita, E., et al.: The expectation-maximization algorithm for ill-posed integral equations: a convergence analysis. Inverse Problems 23, 2575–2588 (2007) 14. Vardi, Y., Shepp, L.A., Kaufman, L.: A statistical model for positron emission tomography. J. of the American Statistical Association 80(389), 8–20 (1985) 15. Iusem, A.N.: Convergence analysis for a multiplicatively relaxed EM algorithm. Mathematical Methods in the Applied Sciences 14, 573–593 (1991) 16. Evans, L.C., Gariepy, R.F.: Measure theory and fine properties of functions. Studies in Advanced Mathematics. CRC Press, Boca Raton (1992) 17. Giusti, E.: Minimal surfaces and functions of bounded variation. Birkhäuser, Basel (1984) 18. Chambolle, A.: An algorithm for total variation minimization and applications. J. of Mathematical Imaging and Vision 20, 89–97 (2004) 19. Resmerita, E., Anderssen, S.: Joint additive Kullback-Leibler residual minimization and regularization for linear inverse problems. Math. Meth. Appl. Sci. 30, 1527– 1544 (2007) 20. Hiriart-Urruty, J.B., Lemaréchal, C.: Convex Analysis and Minimization Algorithms I. Grundlehren der mathematischen Wissenschaften [Fundamental Principles of Mathematical Sciences], vol. 305. Springer, Heidelberg (1993) 21. Brune, C., Sawatzky, A., Wübbeling, F., Kösters, T., Burger, M.: EM-TV methods for inverse problems with poisson noise (in preparation) (2009) 22. Bregman, L.M.: The relaxation method for finding the common point of convex sets and its application to the solution of problems in convex programming. USSR Comp. Math. and Math. Phys. 7, 200–217 (1967) 23. Klar, T.A., et al.: Fluorescence microscopy with diffraction resolution barrier broken by stimulated emission. PNAS 97, 8206–8210 (2000) 24. Hell, S., Schönle, A.: Nanoscale resolution in far-field fluorescence microscopy. In: Hawkes, P.W., Spence, J.C.H. (eds.) Science of Microscopy. Springer, Heidelberg (2006) 25. Kittel, J., et al.: Bruchpilot promotes active zone assembly, Ca2+ channel clustering, and vesicle release. Science 312, 1051–1054 (2006) 26. Willig, K.I., Harke, B., Medda, R., Hell, S.W.: STED microscopy with continuous wave beams. Nature Meth. 4(11), 915–918 (2007) 27. Sawatzky, A., Brune, C., Wübbeling, F., Kösters, T., Schäfers, K.: Accurate EMTV algorithm in PET with low SNR. In: IEEE Nucl. Sci. Symp. (2008)
PDE-Driven Adaptive Morphology for Matrix Fields Bernhard Burgeth, Michael Breuß, Luis Pizarro, and Joachim Weickert Mathematical Image Analysis Group, Faculty of Mathematics and Computer Science, Saarland University, 66041 Saarbrücken, Germany {burgeth,breuss,pizarro,weickert}@mia.uni-saarland.de http://www.mia.uni-saarland.de
Abstract. Matrix fields are important in many applications since they are the adequate means to describe anisotropic behaviour in image processing models and physical measurements. A prominent example is diffusion tensor magnetic resonance imaging (DT-MRI) which is a medical imaging technique useful for analysing the fibre structure in the brain. Recently, morphological partial differential equations (PDEs) for dilation and erosion known for grey scale images have been extended to three dimensional fields of symmetric positive definite matrices. In this article we propose a novel method to incorporate adaptivity into the matrix-valued, PDE-driven dilation process. The approach uses a structure tensor concept for matrix data to steer anisotropic morphological evolution in a way that enhances and completes line-like structures in matrix fields. Numerical experiments performed on synthetic and realworld data confirm the gap-closing and line-completing qualities of the proposed method.
1
Introduction
Initiated in the sixties by the pioneering research of Serra and Matheron on binary morphology [23, 31], this branch of image processing has developed into a rich field of research. Numerous monographs e.g. [17, 24, 32, 33, 34] and proceedings, e.g. [16,18,22] bear witness to the variety in mathematical morphology. The building blocks of morphological operations are dilation and erosion. These are usually realised by algebraic set operations involving a probing set, a so-called structuring element, e.g. [34] for details. An alternative approach to dilation is given [1] by the nonlinear partial differential equation (PDE) ∂t u = ∇u = |∂x u|2 + |∂y u|2 (1) with initial condition u(x, y, 0) = f (x, y). The equation mimics the dilation of a grey scale image f with respect to a ball-shaped structuring element of growing radius t. PDEs of this type using a continuous size parameter t for the structuring element give rise to continuous-scale morphology [1,2,6,29,35]. Equation (1) has been extended in two ways: X.-C. Tai et al. (Eds.): SSVM 2009, LNCS 5567, pp. 247–258, 2009. c Springer-Verlag Berlin Heidelberg 2009
248
B. Burgeth et al.
Firstly, in [5] adaptivity has been incorporated by introducing a speed function β = β(u) into (1), ∂t u = β(u) · ∇u
(2)
Earlier attempts towards adaptivity have been made in [20, 26] where a local switch between dilation and erosion with a nonadaptive structuring element leads to a so-called morphological shock filter, and in [21] introducing morphological amoebae described in a set-theoretic framework. Secondly, in [8] scalar continuous morphology has been extended to a PDEdriven morphology of matrix-valued images, matrix fields for short. Matrix fields have received increasing attention over the recent years since they are the appropriate data type to describe anisotropy in models or measurements of physical quantities. For instance, diffusion tensor magnetic resonance imaging (DT-MRI) became a valuable tool in medicine for in vivo diagnosis. It results in three dimensional tensor fields that describe the diffusive properties of water molecules, and as such the structure of the tissue under examination. The goal of this article is to introduce adaptivity into morphology for matrix fields. As it turns out it is advantageous to start for this generalisation from a scalar adaptive formulation for d-dimensional data u in form of the PDE ∂t u = M (u) · ∇u
(3)
with ∇u as a column vector and a data dependent, symmetric, positive semidefinite d × d-matrix M = M (u)rather than from (2). For example, for greyvalue ab images (d = 2) one has M = and (3) turns into bc ∂t u =
(a∂x u + b∂y u)2 + (b∂x u + c∂y u)2
(4)
→ M (x, y) transforms a sphere cenAn application of the mapping (x, y) tered around the origin into an ellipse. So, in fact, (3) describes a dilation with an ellipsoidal structuring element. The matrix M must contain directional information of the evolving u, and thus it may be derived from the so-called structure tensor. The structure tensor, going back to [14, 27, 4], is a classic tool in image processing to extract directional information from an image. It is given by Sρ (u(x)) := Gρ ∗ ∇u(x) · (∇u(x)) = Gρ ∗ ∂xi u(x) · ∂xj u(x) i,j=1,...,d (5) Here Gρ ∗ indicates a convolution with a Gaussian of standard deviation ρ, however, more general averaging procedures can be used. For more details the reader is referred to [3] and the literature cited there. We will make use of the extended structure tensor concept for matrix fields as proposed in [10]. There it was used to steer an coherence-enhancing diffusion process for matrix fields, an anisotropic filtering process that has been proposed for scalar and colour images in [36, 37].
PDE-Driven Adaptive Morphology for Matrix Fields
249
In [38, 7, 13] Di Zenzo‘s approach [12] to construct a structure tensor for multi-channel images has been extended to matrix fields yielding a standard structure tensor (using the notation of forthcoming Section 2): Jρ (U (x)) := m i,j=1 Sρ (Ui,j (x)) This construction has been refined to a customisable structure tensor in [30]. The article has the following structure: We will briefly convey in Section 2 basic notions of matrix analysis needed to establish a matrix-valued PDE for an adaptively steered morphological dilation process. This includes a short account of the construction of an extended structure tensor for matrix fields. In Section 3 we introduce the steering tensor that guides the dilation process adaptively. We explain how the numerical scheme of Rouy and Tourin is generalised to the matrix valued setting in Section 4. We compare in our experiments adaptive and isotropic dilation with CED-diffusion when applied to synthetic matrix fields and real DT-MRI data sets. We report on this comparison of the results in Section 5. The remarks in Section 6 conclude this article.
2
Matrix Analysis and an Extended Structure Tensor Concept
This section contains the key definitions for the formulation of matrix-valued PDEs. For a more detailed exposition the reader is referred to [9]. A matrix field is considered as a mapping U : Ω ⊂ Rd −→ Symm (R) from a d-dimensional image domain into the set of symmetric m × m-matrices with real entries, U (x) = (Up,q (x))p,q=1,...,m . The set of positive (semi-) definite matrices, + denoted by Sym++ m (R) (resp., Symm (R)), consists of all symmetric matrices A with v, Av := v Av > 0 (resp., ≥ 0) for v ∈ Rm \ {0} . This set is of special interest since DT-MRI produces data with this property. Note that at each point x the matrix U (x) of a field of symmetric matrices can be diagonalised yielding U (x) = V (x) D(x)V (x), where V (x) is a orthogonal matrix, while D(x) is a diagonal matrix. In the sequel we will denote m × m - diagonal matrices with entries λ1 , . . . , λm ∈ R from left to right simply by diag(λi ). The extension of a function h : R −→ R to Symm (R) is standard [19]: With a slight abuse of notation we set h(U ) := V diag(h(λ1 ), . . . , h(λm ))V ∈ Sym+ m (R), h denoting now a function acting on matrices as well. Specifying h(s) = |s|, s ∈ R as the absolut value function leads to the absolut value |A| ∈ Sym+ m (R) of a matrix A. It is natural to define the partial derivative for matrix fields componentwise: ∂ ω U = (∂ω Up,q )p,q=1,...,m
(6)
where ω ∈ {t, x1 , . . . , xd }, that is, ∂ ω stands for a spatial or temporal derivative. Viewing a matrix as a tensor (of second order), its gradient would be a third order tensor according to the rules of differential geometry. However, we adopt a more operator-algebraic point of view by defining the generalised gradient ∇U (x) at a voxel x = (x1 , . . . , xd ) by ∇U (x) := (∂ x1 U (x), . . . , ∂ xd U (x))
(7)
250
B. Burgeth et al.
which is an element of (Symm (R))d , in close analogy tothe scalar setting where ∇u(x) ∈ Rd . For W ∈ (Symm (R))d we set |W |p := p |W1 |p + · · · + |Wd |p for 0 < p < +∞. It results in a positive semidefinite matrix from Sym+ m (R), the direct counterpart of a nonnegative real number as the length of a vector in Rd . There will be the need for a symmetric multiplication of symmetric matrices. We opt for the so-called Jordan product A •J B := 12 (AB + BA) . It produces a symmetric matrix, and it is commutative but neither associative nor distributive. Furthermore, for later use in numerical schemes we have to clarify the notion of maximum and minimum of two symmetric matrices A, B. In direct anaology with relations known to be valid for real numbers one defines [8]: max(A, B) =
1 1 (A + B + |A − B|) and min(A, B) = (A + B − |A − B|) (8) 2 2
where |F | stands for the absolut value of the matrix F . With this at our disposal we formulate the matrix-valued counterpart of (3) as ∂ t U = |M (U ) • ∇U |2
(9)
with an initial matrix field F (x) = U (x, 0). Here M (U ) denotes a symmetric md × md-block matrix with d2 blocks of size m × m that is multiplied block-wise with ∇U employing the symmetrised product "•". Note that | · |2 stands for the length of M (U ) • ∇U in the matrix valued sense. The construction of M (U ) is detailed in Section 3 and relies on the so-called full structure tensor. The full structure tensor S L for matrix fields as defined in [10] reads S L (U ) := Gρ ∗ ∇U ·(∇U ) = Gρ ∗ ∂ xi U · ∂ xj U i,j=1,...,d (10) with Gρ ∗ indicating a convolution with a Gaussian of standard deviation ρ. S L (U (x)) is a symmetric md × md-block matrix with d2 blocks of size m × m, S L (U (x)) ∈ Symd (Symm (IR)) = Symmd (IR). Typically for the 3D medical DT-MRI data one has d = 3 and m = 3, yielding a 9 × 9-matrix S L . It can md be diagonalised as S L (U ) = k=1 λk wk wk with real eigenvalues λk (w.l.o.g. arranged in decreasing order) and an orthonormal basis {wk }k=1,...,md of IRmd . In order to extract useful d-dimensional directional information S L (U ) ∈ Symmd (IR) is reduced to a structure tensor S(U ) ∈ Symd (IR) in a generalised projection step [10] using the block operator matrix TrA := diag(trA , . . . , trA ) containing the trace operation. We set Tr := TrIm where Im denotes the m × m unit matrix. This operator matrix acts on elements of the space (Symm (IR))d as well as on block matrices via formal block-wise matrix multiplication, ⎛ ⎞⎛ ⎞ ⎛ ⎞ trA · · · 0 M11 · · · M1d trA (M11 ) · · · trA (M1d ) . . . . . . .. .. .. ⎝ .. . . . .. ⎠ ⎝ .. . . . .. ⎠ = ⎝ ⎠, (11) . 0 · · · trA trA (Md1 ) · · · trA (Mdd ) Md1 · · · Mdd provided that the square blocks Mij have the same size as A. The projection that is conveyed by the reduction process condenses the directional information contained in S L (U ), for a more detailed reasoning we must refer the reader to [10]
PDE-Driven Adaptive Morphology for Matrix Fields
251
for the sake of brevity. The reduction operation is accompanied by an extension operation: The Im -extension is the mapping from Symd (IR) to Symmd (IR) conveyed by the Kronecker product ⊗ : ⎛ ⎞ ⎛ ⎞ ⎛ ⎞ ⎛ ⎞ v11 · · · v1d v11 · · · v1d Im · · · Im v11 Im · · · v1d Im ⎜ .. . . .. ⎟ ⎜ .. . . .. ⎟ ⎜ .. . . .. ⎟ ⎜ .. .. ⎟(12) .. −→⎝ . ⎝ . . . ⎠ . . ⎠⊗⎝ . . . ⎠:=⎝ . . . ⎠ vd1 · vdd
vd1 · · · vdd
Im · · · Im
vd1 Im · · · vdd Im
This resizing step renders a proper matrix-vector multiplication with the large generalised gradient (∇U (x)) possible. By specifying the matrix A in (11) one may invoke a priori knowledge into the direction estimation [10]. The research on these structure-tensor concepts has been initiated by [38, 7]. The approaches to matrix field regularisation suggested in [11] are based on differential geometric considerations. Comprehensive survey articles on the analysis of matrix fields using various techniques can be found in [39].
3
Steering Matrix M (U ) for Matrix Fields
With this notions we are in the position to propose the steering matrix M in the adaptive dilation process for matrix fields. We proceed in four steps: 1. The matrix field IRd x → U (x) provides us with a module field of generalised gradients ∇U (x) from which we construct the generalised structure tensor S L (U (x)) possibly with a certain integration scale ρ. This step corresponds exactly to the scalar case. 2. We infer d-dimensional directional information by reducing S L (U (x)) with trA by means of the block operator matrix TrA leading to a symmetric d × dmatrix S, for example S = Jρ if A = Im , (13) S(x) := TrA S L (U (x)) 3. The symmetric d × d-matrix S is spectrally decomposed, and the following mapping is applied: Rd+ −→ Rd H: , (14) c (λ1 , . . . , λd ) −→ λ1 +···+λ (λd , λd−1 , . . . , Kc · λ1 ) d with constants c, K > 0. H applied to S yields the steering matrix M , M := H(S)
(15)
Observe that the ellipsoid associated with the matrix M is flipped if compared with S and, depending on the choice of K, more excentric than the one accompanying S. 4. Finally we enlarge the d × d-matrix M to a md × md-matrix M by the extension operation: ⎛ ⎞ Im · · · Im ⎜ ⎟ M = M ⊗ ⎝ ... . . . ... ⎠ (16) Im · · · Im
252
4
B. Burgeth et al.
Matrix-Valued Numerical Schemes
In the context of PDE-based mathematical morphology, first-order finite difference methods such as the Osher-Sethian scheme [25] and the Rouy-Tourin method [28] are reasonable choices for solving the scalar PDE (4). We choose the latter in our experiments. The variant we present for the sake of brevity in its two-dimensional form reads 1 x n 2 1 n+1 n x n ui,j = ui,j + τ max max −D− ui,j , 0 , max D+ ui,j , 0 hx hx 1/2 1 y n 2 1 y n + max max −D− ui,j , 0 , max D+ ui,j , 0 (17) hy hy In the latter formulation we employ the notation unij as the grey value of the image u at the pixel centred at (ihx , jhy ) ∈ R2 at the time-level nτ of the
Fig. 1. (a) Top left: 2D slice of original 3D matrix field. (b) Top right: Adaptive dilation with of the original data with K = 25, ρ = 1 after t = 0.3. (c) Bottom left: Standard PDE-based dilation mimicing a ball-shaped structuring element after t = 1. (d) Bottom right: CED-filtering with ρ = 4 after t = 10.
PDE-Driven Adaptive Morphology for Matrix Fields
253
evolution. Additionally we use standard abbreviations for forward and backward x n x n difference operators, i.e., D+ ui,j := uni+1,j −uni,j and D− ui,j := uni,j −uni−1,j . and spatial grid size hx , hy . This scheme approximates, in the pixel (ihx , jhy ) 1 x n 1 x n max −D− ui,j , 0 , max D+ ui,j , 0 (18) ux ≈ max hx hx uy ≈ max
1 y n 1 y n max −D− ui,j , 0 , max D+ ui,j , 0 hy hy
(19)
Using this approximations, we modify the original Rouy-Tourin scheme (17) in an obvious manner to obtain a numerical scheme for the adaptive version of the PDE-based dilation (3). The extension to higher dimensions poses no problem. Since linear combinations and elementary functions such as the square, squareroot or absolute value function for matrix fields are now at our disposal it is straightforward to define one sided differences in x-direction for 2D matrix fields of m × m-matrices: x n D+ U (i, j) := U n ((i + 1)hx , jhy ) − U n (ihx , jhy ) ∈ Symm (R)
(20)
x n U (i, j) := U n (ihx , jhy ) − U n ((i − 1)hx , jhy ) ∈ Symm (R) D−
(21)
Fig. 2. (a) Left: 2D slice of 3D DT-MRI data set. (b) Right: Adaptive dilation of the original data with K = 10, ρ = 1, t = 0.5.
254
B. Burgeth et al.
In order to avoid confusion with the subscript notation for matrix components we used the notation U (i, j) to indicate the (matrix-) value of the matrix field evaluated at the voxel centred at (ihx , jhy ) ∈ R2 . The y-direction (and z-direction in 3D) is treated accordingly. The notion of supremum and infimum of two matrices – as needed in a matrix variant of Rouy-Tourin – has been provided by (8). Having these generalisations at our disposal a modified, adaptive version of the Rouy-Tourin scheme is available now in the setting of matrix fields simply by replacing grey values unij by matrices U n (i, j).
5
Experiments
The matrix data are visualised as an ellipsoid in each voxel via the level sets of quadratic form {v ∈ R2 v : v U −2 (i, j)v = const.} associated with the matrix
Fig. 3. (a) Top left: Enlarged section of the original data of figure 2 showing the genu area. (b) Top right: Adaptive dilation of the original data with K = 10, ρ = 1, t = 0.5. (c) Bottom left: Standard PDE-based dilation mimicing a ball-shaped structuring element with t = 0.5. (d) Bottom right: CED-filtering with ρ = 1 after t = 0.5.
PDE-Driven Adaptive Morphology for Matrix Fields
255
Fig. 4. (a) Top left: Enlarged section of the original data of figure 2 showing the splenium area. (b) Top right: Adaptive dilation of the original data with K = 10, ρ = 1, t = 0.5. (c) Bottom left: Standard PDE-based dilation mimicing a ball-shaped structuring element with t = 0.5. (d) Bottom right: CED-filtering with ρ = 1 after t = 0.5.
U (i, j) ∈ Sym+ 3 (R) representing the matrix field at voxel (ihx , jhy ). By using U −2 the length of the semi-axes of the ellipsoid correspond directly with the three eigenvalues of the matrix. Changing the constant const. amounts to a mere scaling of the ellipsoids. Note that only positive definite matrices produce ellipsoids as level sets of its quadratic form. In all our experiments we compare the results of the proposed matrix-valued adaptive dilation with the isotropic dilation [8] , and with the matrix-valued coherence-enhancing diffusion from [10]. For the explicit numerical schemes we used a time step size of 0.1, grid size hx = hy = 1, and c = 0.01 · K in (14). Figure 1 shows a synthetic data set of size 32 × 32 representing an interrupted diagonal stripe built from cigar-shaped ellipsoids of equal size. All methods succeed to some degree to fill the gaps. In the case of the proposed adaptive dilation the gap is filled almost completely with tensors comparable in size with the original ones while the width of the stripe is not altered at all. However, the numerical scheme has a slight bias towards the directions of the coordinate system entailing in the appearance of mild artefacts. Standard dilation fills the gap basically as a side effect of the isotropic dilation process which leads also to a considerable widening of the ribbon-like structure. CED for matrix fields produces indeed small cigar-shaped ellipsoids at the location of the gap. But the process is considerably slower than any of the dilation processes
256
B. Burgeth et al.
and the neighbouring ellipsoids become smaller due to the property of mass conservation. Additionally an undesirable widening of the stripe is observed. We also tested the proposed method on a real DT-MRI data set of a human head consisting of a 128 × 128 × 38-field of positive definite matrices. Figure 2 shows the lateral ventricals in a 40 × 55 2D section before and after applying adaptive dilation with speed parameter K = 10, integration scale ρ = 1 and stopping time t = 0.5. For a better comparison we display two enlarged regions of interest in Figures 3 and 4, namely the genu and the splenium areas, resp.. We observe that adaptive dilation preserves the shape of the ventricles better than the isotropic dilation, while enhancing slightly the directional structure of the fibre tracts surrounding the ventricles. Due to measurement errors the fibre tracts are interrupted in the original Figures 3(a) and 4(a). These holes in the anisotropic regions (splenium) are quickly filled by the adaptive dilation while CED-filtering will take much longer to do so.
6
Conclusion
In this article we have presented a novel method for an adaptive, PDE-based dilation process in the setting of matrix fields. The evolution governed by a matrix-valued PDE is guided by a steering tensor, the construction of which relies on an extended structure tensor concept for matrix fields. A matrix-valued extension of the Rouy-Tourin-scheme that allows to include directional information is employed to solve the novel PDE. Experiments on positive semidefinite DT-MRI and synthetic data confirm that the novel adaptive dilation process displays line-enhancing and gap-closing qualities, and as such it is superior to standard isotropic dilation which extends structures in all directions. It is also a valuable alternative in terms of quality and speed to coherence-enhancing diffusion filtering for matrix fields, an anisotropic processes which aims at enhancing flow-like structures as well but may suffer from dissipative effects. Future research will concentrate on improving the numerical realisation of our adaptive dilation.
Acknowledgement The financial support of the German Academic Exchange Service (DAAD) for the third author is gratefully acknowledged.
References 1. Alvarez, L., Guichard, F., Lions, P.-L., Morel, J.-M.: Axioms and fundamental equations in image processing. Archive for Rational Mechanics and Analysis 123, 199–257 (1993) 2. Arehart, A.B., Vincent, L., Kimia, B.B.: Mathematical morphology: The Hamilton–Jacobi connection. In: Proc. Fourth International Conference on Computer Vision, Berlin, pp. 215–219. IEEE Computer Society Press, Los Alamitos (1993)
PDE-Driven Adaptive Morphology for Matrix Fields
257
3. Bigün, J.: Vision with Direction. Springer, Berlin (2006) 4. Bigün, J., Granlund, G.H., Wiklund, J.: Multidimensional orientation estimation with applications to texture analysis and optical flow. IEEE Transactions on Pattern Analysis and Machine Intelligence 13(8), 775–790 (1991) 5. Breuß, M., Burgeth, B., Weickert, J.: Anisotropic continuous-scale morphology. In: Martí, J., Benedí, J.M., Mendonça, A.M., Serrat, J. (eds.) IbPRIA 2007. LNCS, vol. 4478, pp. 515–522. Springer, Heidelberg (2007) 6. Brockett, R.W., Maragos, P.: Evolution equations for continuous-scale morphological filtering. IEEE Transactions on Signal Processing 42, 3377–3386 (1994) 7. Brox, T., Weickert, J., Burgeth, B., Mrázek, P.: Nonlinear structure tensors. Image and Vision Computing 24(1), 41–55 (2006) 8. Burgeth, B., Bruhn, A., Didas, S., Weickert, J., Welk, M.: Morphology for tensor data: Ordering versus PDE-based approach. Image and Vision Computing 25(4), 496–511 (2007) 9. Burgeth, B., Didas, S., Florack, L., Weickert, J.: A generic approach to diffusion filtering of matrix-fields. Computing 81, 179–197 (2007) 10. Burgeth, B., Didas, S., Weickert, J.: A general structure tensor concept and coherence-enhancing diffusion filtering for matrix fields. Technical Report 197, Department of Mathematics, Saarland University, Saarbrücken, Germany (July 2007); to appear in: Laidlaw, D., Weickert, J. (eds.): Visualization and Processing of Tensor Fields. Springer, Heidelberg (2009) 11. Chefd’Hotel, C., Tschumperlé, D., Deriche, R., Faugeras, O.: Constrained flows of matrix-valued functions: Application to diffusion tensor regularization. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002. LNCS, vol. 2350, pp. 251–265. Springer, Heidelberg (2002) 12. Di Zenzo, S.: A note on the gradient of a multi-image. Computer Vision, Graphics and Image Processing 33, 116–125 (1986) 13. Feddern, C., Weickert, J., Burgeth, B., Welk, M.: Curvature-driven PDE methods for matrix-valued images. International Journal of Computer Vision 69(1), 91–103 (2006) 14. Förstner, W., Gülch, E.: A fast operator for detection and precise location of distinct points, corners and centres of circular features. In: Proc. ISPRS Intercommission Conference on Fast Processing of Photogrammetric Data, Interlaken, Switzerland, June 1987, pp. 281–305 (1987) 15. Goutsias, J., Heijmans, H.J.A.M., Sivakumar, K.: Morphological operators for image sequences. Computer Vision and Image Understanding 62, 326–346 (1995) 16. Goutsias, J., Vincent, L., Bloomberg, D.S. (eds.): Mathematical Morphology and its Applications to Image and Signal Processing. Computational Imaging and Vision, vol. 18. Kluwer, Dordrecht (2000) 17. Heijmans, H.J.A.M.: Morphological Image Operators. Academic Press, Boston (1994) 18. Heijmans, H.J.A.M., Roerdink, J.B.T.M. (eds.): Mathematical Morphology and its Applications to Image and Signal Processing. Computational Imaging and Vision, vol. 12. Kluwer, Dordrecht (1998) 19. Horn, R.A., Johnson, C.R.: Matrix Analysis. Cambridge University Press, Cambridge (1990) 20. Kramer, H.P., Bruckner, J.B.: Iterations of a non-linear transformation for enhancement of digital images. Pattern Recognition 7, 53–58 (1975) 21. Lerallut, R., Decencière, E., Meyer, F.: Image filtering using morphological amoebas. Image and Vision Computing 25(4), 395–404 (2007)
258
B. Burgeth et al.
22. Louverdis, G., Vardavoulia, M.I., Andreadis, I., Tsalides, P.: A new approach to morphological color image processing. Pattern Recognition 35, 1733–1741 (2002) 23. Matheron, G.: Eléments pour une théorie des milieux poreux. Masson, Paris (1967) 24. Matheron, G.: Random Sets and Integral Geometry. Wiley, New York (1975) 25. Osher, S., Fedkiw, R.P.: Level Set Methods and Dynamic Implicit Surfaces. Applied Mathematical Sciences, vol. 153. Springer, New York (2002) 26. Osher, S., Sethian, J.A.: Fronts propagating with curvature-dependent speed: Algorithms based on Hamilton–Jacobi formulations. Journal of Computational Physics 79, 12–49 (1988) 27. Rao, A.R., Schunck, B.G.: Computing oriented texture fields. CVGIP: Graphical Models and Image Processing 53, 157–185 (1991) 28. Rouy, E., Tourin, A.: A viscosity solutions approach to shape-from-shading. SIAM Journal on Numerical Analysis 29, 867–884 (1992) 29. Sapiro, G., Kimmel, R., Shaked, D., Kimia, B.B., Bruckstein, A.M.: Implementing continuous-scale morphology via curve evolution. Pattern Recognition 26, 1363– 1372 (1993) 30. Schultz, T., Burgeth, B., Weickert, J.: Flexible segmentation and smoothing of DTMRI fields through a customizable structure tensor. In: Bebis, G., Boyle, R., Parvin, B., Koracin, D., Remagnino, P., Nefian, A., Meenakshisundaram, G., Pascucci, V., Zara, J., Molineros, J., Theisel, H., Malzbender, T. (eds.) ISVC 2006. LNCS, vol. 4291, pp. 455–464. Springer, Heidelberg (2006) 31. Serra, J.: Echantillonnage et estimation des phénomènes de transition minier. PhD thesis, University of Nancy, France (1967) 32. Serra, J.: Image Analysis and Mathematical Morphology, vol. 1. Academic Press, London (1982) 33. Serra, J.: Image Analysis and Mathematical Morphology, vol. 2. Academic Press, London (1988) 34. Soille, P.: Morphological Image Analysis, 2nd edn. Springer, Berlin (2003) 35. van den Boomgaard, R.: Mathematical Morphology: Extensions Towards Computer Vision. PhD thesis, University of Amsterdam, The Netherlands (1992) 36. Weickert, J.: Coherence-enhancing diffusion of colour images. In: Sanfeliu, A., Villanueva, J.J., Vitrià, J. (eds.) Proc. Seventh National Symposium on Pattern Recognition and Image Analysis, Barcelona, Spain, April 1997, vol. 1, pp. 239–244 (1997) 37. Weickert, J.: Coherence-enhancing diffusion filtering. International Journal of Computer Vision 31(2/3), 111–127 (1999) 38. Weickert, J., Brox, T.: Diffusion and regularization of vector- and matrix-valued images. In: Nashed, M.Z., Scherzer, O. (eds.) Inverse Problems, Image Analysis, and Medical Imaging. Contemporary Mathematics, vol. 313, pp. 251–268. AMS, Providence (2002) 39. Weickert, J., Hagen, H. (eds.): Visualization and Processing of Tensor Fields. Springer, Berlin (2006)
On Semi-implicit Splitting Schemes for the Beltrami Color Flow Lorina Dascal1 , Guy Rosman1 , Xue-Cheng Tai2,3 , and Ron Kimmel1 1
Department of Computer Science, Technion – Israel Institute of Technology, 32000, Haifa, Israel {lorina,rosman,ron}@cs.technion.ac.il 2 Division of Mathematical Sciences, SPMS, Nanyang Technological University, 50 Nanyang Avenue, 639798, Singapore [email protected] 3 Department of Mathematics, University of Bergen, Johannes Brunsgate 12, 5007, Bergen, Norway [email protected]
Abstract. The Beltrami flow is an efficient non-linear filter, that was shown to be effective for color image processing. The corresponding anisotropic diffusion operator strongly couples the spectral components. Usually, this flow is implemented by explicit schemes, that are stable only for small time steps and therefore require many iterations. In this paper we introduce a semi-implicit scheme based on the locally one-dimensional (LOD) and additive operator splitting (AOS) schemes for implementing the anisotropic Beltrami operator. The mixed spatial derivatives are treated explicitly, while the non-mixed derivatives are approximated in a semi-implicit manner. Numerical experiments demonstrate the stability of the proposed scheme. Accuracy and efficiency of the splitting schemes are tested in applications such as the scale-space analysis and denoising. In order to further accelerate the convergence of the numerical scheme, the reduced rank extrapolation (RRE) vector extrapolation technique is employed.
1
Introduction
Nonlinear diffusion filters based on partial differential equations (PDEs) have been extensively used in the last decade for different tasks in image processing. Their efficient implementation requires designing numerical schemes in which the issues of accuracy, stability, and computational cost all play important roles. The Beltrami image flow is an example of a non-linear filter, that is efficient for color image processing. It treats the image as a 2-D manifold embedded in a hybrid spatial-feature space. Minimization of the image area surface yields the Beltrami flow. The corresponding diffusion operator is anisotropic and strongly couples the spectral components. Due to its anisotropy and non-separability, so far there is no efficient implicit, nor operator-splitting-based numerical scheme for the partial differential equation that describes the Beltrami flow in color. X.-C. Tai et al. (Eds.): SSVM 2009, LNCS 5567, pp. 259–270, 2009. c Springer-Verlag Berlin Heidelberg 2009
260
L. Dascal et al.
Usual discretizations of this filter are based on explicit schemes, that limit the time step and therefore result in a large number of iterations. In [1] an acceleration technique based on the reduced rank extrapolation (RRE) algorithm [2, 3] was proposed in order to speed-up the slow convergence of the explicit scheme. As an alternative to the explicit scheme, an approximation using the short time kernel of the Beltrami operator was suggested in [4]. Although unconditionally stable, this method is still computationally demanding, since computing the kernel involves geodesic distance computation around each pixel. The bilateral filter, which can be shown to be an Euclidean approximation of the Beltrami kernel, was studied in different contexts (see [5], [6], [7], [8], [9], [10]). Recently, a related filter, the nonlocal means filter, was proposed in [11] and shown to be useful in denoising gray-scale and color images. In this paper we propose to approximate the system of nonlinear coupled equations given by the Beltrami flow by a semi-implicit finite difference scheme based on operator splitting. Additive operator splitting (AOS) schemes were first developed for (nonlinear elliptic/parabolic) monotone equations and NavierStokes equations [12, 13]. In image processing applications, the AOS scheme was found to be an efficient way for approximating the Perona-Malik filter [14], especially if symmetry in scale-space is required. The AOS scheme is first order in time, semi-implicit, and unconditionally stable with respect to its time-step [13, 14]. In the early 1950’s (see [15]) the alternating-direction method (ADI) was introduced, and in [16] the LOD (locally one-dimensional) splitting method was proposed. The LOD scheme and other multiplicative splitting methods were employed in the context of nonlinear diffusion image filtering in [17]. We stress that the main characteristic of this class of equations, which allows splitting, is local isotropy. However, in the case of the anisotropic Beltrami operator, the main difficulty in splitting stems from the presence of the mixed derivatives. To overcome this problem, we suggest to construct the following semi-implicit scheme; the spatial mixed derivatives are discretized explicitly at the current time step nΔt, while those that do not contain mixed derivatives are approximated using an average of two levels of time steps: nΔt and (n + 1)Δt (Crank-Nicolson scheme). As our equations are nonlinear, a stability proof of the corresponding finite difference scheme is a non-trivial task. We provide numerical experiments which indicate that the LOD and the AOS splitting schemes for the nonlinear Beltrami color filter are stable for a wide range of time steps. We demonstrate the efficiency and stability of the splitting in applications such as: Beltramibased scale space and Beltrami-based denoising. In order to further expedite the LOD/AOS splitting schemes, we show how to speed-up their convergence by using the RRE (reduced rank extrapolation) technique. The RRE method was introduced by Me˘sina and Eddy [2, 3] to speed-up the convergence of general sequences of vectors without explicit knowledge of the sequence generator. This technique was applied in [1] in order to speed up the slow convergence of the standard explicit scheme for the Beltrami color flow. In this paper we show that in applications such as scale-space and denoising of color images, the semiimplicit LOD/AOS schemes can also be accelerated using the RRE technique.
On Semi-implicit Splitting Schemes for the Beltrami Color Flow
261
This paper is organized as follows: In Section 2 we briefly summarize the Beltrami framework. In Section 3 we briefly review general semi-implicit splitting operator schemes. In Section 4 we propose a semi-implicit splitting scheme for the anisotropic Beltrami operator, based on the LOD/AOS schemes. In Section 5 we demonstrate the efficiency and stability of the LOD/AOS splitting schemes for Beltrami-based scale-space and Beltrami-based denoising. Furthermore, we propose to accelerate the LOD/AOS schemes using the RRE technique. Section 6 concludes the paper.
2
The Beltrami Framework
Let us briefly review the Beltrami framework for non-linear diffusion in computer vision [18, 19, 20, 21]. We represent images as embedding maps of a Riemannian manifold in a higher dimensional space. We denote the map by U : Σ → M , where Σ is a two-dimensional surface, with (σ 1 , σ 2 ) denoting coordinates on it. M is the spatial-feature manifold, embedded in Rd+2 , where d is the number of image channels. For example, a gray-level image can be represented as a 2D surface embedded in R3 . The map U in this case is U (σ 1 , σ 2 ) = (σ 1 , σ 2 , I(σ 1 , σ 2 )), where I is the image intensity. For color images, U is given by U (σ 1 , σ 2 ) = (σ 1 , σ 2 , I 1 (σ 1 , σ 2 ), I 2 (σ 1 , σ 2 ), I 3 (σ 1 , σ 2 )), where I 1 , I 2 , I 3 are the three components of the color vector. Next, we choose a Riemannian metric on this surface, g, with elements denoted by gij . The canonical choice of coordinates in image processing is Cartesian (we denote them here by x1 and x2 ). For such a choice, which we follow in the rest of the paper, we identify σ 1 = x1 and σ 2 = x2 . In this case, σ 1 and σ 2 are the image coordinates. We denote the elements of the inverse of the metric by superscripts g ij , and the determinant by g = det(gij ). Once images are defined as embedding of Riemannian manifolds, it is natural to look for a measure on this space of embedding maps. Denote by (Σ, g) the image manifold and its metric, and by (M, h) the spacefeature manifold and its metric. Then, the functional S[U ] assigns a real number to a map U : Σ → M , √ S[U, gij , hab ] = ds σ g||dU ||2g,h , (1) where s is the dimension of Σ, g is the determinant of the image metric, and the range of indices is i, j = 1, 2, ... dim(Σ) and a, b = 1, 2, ... dim(M ). The integrand ||dU ||2g,h is expressed in a local coordinate system by ||dU ||2g,h = (∂xi U a )g ij (∂xj U b )hab . This functional, for dim(Σ) = 2 and hab = δab , was first proposed by Polyakov [22] in the context of high energy physics, in the theory known as string theory. The elements of the induced metric for color images with Cartesian color coordinates are 3 3 1 + β 2 a=1 (Uxa1 )2 β 2 a=1 Uxa1 Uxa2 G = (gij ) = , (2) 3 3 β 2 a=1 Uxa1 Uxa2 1 + β 2 a=1 (Uxa2 )2
262
L. Dascal et al.
where a subscript of U denotes a partial derivative and the parameter β > 0 determines the ratio between the spatial and spectral (color) distances. Using standard methods in calculus of variations, the Euler-Lagrange equations with respect to the embedding (assuming Euclidean embedding space) are 1 1 δS 0 = − √ hab b = √ div (D∇U a ), g δU g
(3)
Δg U a
where the diffusion matrix is D =
√ −1 gG . Note that we can write 2
div(D∇U ) =
∂xq (dqr ∂xr U ).
q,r=1
The operator that acts on U is the natural generalization of the Laplacian from flat spaces to manifolds. It is called the Laplace-Beltrami operator, and denoted by Δg . The parameter β, in the elements of the metric gij , determines the nature of the flow. At the limits, where β → 0 and β → ∞, we obtain respectively a linear diffusion flow and a nonlinear flow, akin to the TV flow [23] for the case of grey-level images (see [20] for details). The Beltrami scale-space emerges as a gradient descent minimization process 1 δS Uta = − √ = Δg U a , g δU a
a = 1, 2, 3.
(4)
For Euclidean embedding, the functional in Eq. (1) reduces to S(U ) =
√ g dx1 dx2 .
(5)
This geometric measure can be used as a regularization term for color image processing. In the variational framework, the reconstructed image is the minimizer of a cost-functional. This functional can be written in the following general form, 3
Ψ (U ) = λ ||U a − F a ||2 + S(U ), a=1
where the parameter λ controls the smoothness of the solution and F is the given image. The modified Euler-Lagrange equations as a gradient descent process are 1 δΨ 2λ Uta = − √ = − √ (U a − F a ) + Δg U a , g δU a g
a = 1, 2, 3.
(6)
On Semi-implicit Splitting Schemes for the Beltrami Color Flow
3
263
Operator Splitting Schemes
In this section we briefly review standard first order accurate splitting schemes for diffusion equations. One of the main drawbacks of the semi-implicit schemes for such equations in multiple dimensions is that the resulting inverted matrix does not have an efficient algorithm for its inversion. In order to remedy this shortcoming, splitting techniques are commonly employed in solving timedependent partial differential equations. They allow one to reduce problems in multiple spatial dimensions to a sequence of problems in one dimension, which are easier to solve. One of the simplest splitting schemes belonging to the class of multiplicative operator splitting schemes, is the locally one-dimensional (LOD) scheme [16]. The LOD scheme only needs to invert one three-diagonal matrix for each direction. It is simple to implement, is unconditionally stable and it is first order accurate. However, the system matrix is not axis symmetric, a property that may be important in some cases. If such a property is required, one could use the additive operator splitting scheme [13], which was actually invented for parallel implementation of splitting methods. Even for sequential implementations, the AOS is almost as efficient as the LOD scheme; instead of multiplying the operators, one computes them independently and then averages the sums of the inverse of the two matrices. We want to emphasize that the matrices for AOS use 2Δt instead of Δt. It is not a trivial matter to apply dimensional splitting schemes for Beltrami type of equations. Our goal is to construct a splitting scheme for the nonlinear anisotropic Beltrami operator, which would amount to inverting tridiagonal matrices, be unconditionally stable and preserve the time discretization accuracy that was obtained without applying splitting techniques.
4
The Proposed Splitting Scheme
In this section we present an operator splitting scheme for the Beltrami filter. Before splitting, we first introduce a semi-implicit approximation scheme to our equations. A semi-implicit Crank-Nicolson scheme for an equation involving mixed derivatives can rely on the following discretization of the spatial derivatives operators: mixed derivatives are computed at time step nΔt, while the non-mixed derivatives are computed as the average of the values at time steps nΔt and (n + 1)Δt. This approach for handling mixed derivatives in semiimplicit schemes for approximating linear equations has been considered in several previous works (see [24, 25, 26] for example), including the context of image processing [27], although it was not combined with the Crank-Nicolson method in the latter case. We note that in numerical experiments we have found the introduction of the Crank-Nicolson method into the splitting scheme necessary in order to maintain stability for large time steps. A simpler scheme, similar to the one used in [27], did not seem to be sufficiently stable for this PDE and the applications demonstrated in this paper. We now present the scheme we intend to use.
264
L. Dascal et al.
First, we refine our grid notations. We work on the rectangle Ω = (0, 1)×(0, 1), which we discretize by a uniform grid of m × m pixels, such that xi = iΔx, yj = jΔy, tn = nΔt, where 1 ≤ i ≤ m, 1 ≤ j ≤ m, 1 ≤ n ≤ J and JΔt = T . Let the 1 grid size be Δx = Δy = m−1 . a For each channel U , a = 1, 2, 3 of the color vector, we define the discrete approximation (U a )nij by (U a )(iΔx, jΔy, nΔt) = (U a )nij ≈ U a (iΔx, jΔy, nΔt). We impose von-Neumann boundary condition, and initially set U a to be our initial data image. 4.1
LOD/AOS Scheme for the Beltrami Scale-Space
We approximate the Beltrami filter given in Eq. (4) by the following semi implicit Crank-Nicolson scheme: 1 1 n a n+1 1 n a n (U a )n+1 − (U a )n = √ n All (U ) + All (U ) + Δt g 2 2 2
l=1
2
2
l=1
Anqr (U a )n ,
q=1 r =q
where U a is the N -dimensional vector denoting one of the components of the color vector, and Anqr is a central difference approximation of the operator ∂xq (dqr ∂xr ) at time step n. Rearranging terms, we obtain −1
2
Δt (U a )n+1 = I − √ n Anll 2 g l=1 ⎛ ⎞ 2
2
Δt Δt ⎝I + √ Anqr + √ n Anll ⎠ (U a )n , g n q=1 2 g r =q
l=1
which can also be written as ⎞ −1 ⎛
2 2
2
Δt Δt ⎝I + Δt (U a )n+1 = I − A¯nll A¯nqr + A¯nll ⎠ (U a )n , 2 2 q=1 l=1
where
r =q
l=1
1 A¯11 = √ ∂x (A∂x ), g
1 A¯22 = √ ∂y (C∂y ), g
1 A¯12 = √ ∂x (B∂y ), g
1 A¯21 = √ ∂y (B∂x ), g
and the functions A, B, C are the corresponding elements of the diffusion matrix associated with the Beltrami flow.
On Semi-implicit Splitting Schemes for the Beltrami Color Flow
265
Again, this semi-implicit scheme still has a major drawback. At each iteration one needs to solve a large linear system whose matrix of coefficients is not tridiagonal and thus costly. Instead, we employ the LOD splitting scheme Δt ¯ −1 Δt ¯ −1 A22 A11 (U a )n+1 = I − I− 2 2 2
Δt ¯ Δt ¯ A11 )(I + A22 ) + Δt (I + A¯nqr (U a )n , 2 2 q=1 r =q
or the AOS scheme, that reads, (U a )n+1 =
−1 −1 1 I − ΔtA¯22 + I − ΔtA¯11 2 2
Δt ¯ Δt ¯ A11 )(I + A22 ) + Δt (I + A¯nqr (U a )n . 2 2 q=1 r =q
The above splitting schemes are efficient because at each time step a single tridiagonal matrix inversion is performed for each spatial dimension. The system of differential equations we deal with is nonlinear. The question of theoretical stability of the LOD/AOS based nonlinear finite difference scheme is a non-trivial challenge, with theory still lagging behind common practice. Our numerical experiments indicate that the splitting is stable for a wide variety of parameters, suitable for most applications, as will be shown in Section 5. 4.2
LOD/AOS Scheme for the Beltrami-Based Denoising
The splitting scheme in the presence of a fidelity term requires a slight modification that we detail below. In this case we solve for each channel the equation 2λ Uta = − √ (U a − F a ) + Δg U a , g
(7)
with von-Neumann boundary condition and the initial condition U a (x, 0) = F a (x). The Crank-Nicolson scheme approximating Eq. (7) is 2 Δt ¯n λ −1 (U a )n+1 = I − All + 2Δt √ n I 2 g l=1
2
Δt ¯n Δt ¯n A11 )(I + A22 ) + Δt A¯nqr (U a )n + 2 2 q=1 r =q λ +2ΔtF a √ n . g
(I +
(8)
266
L. Dascal et al.
It is possible to use LOD/AOS approximations for the inverse of the matrix in the above scheme. √ However, we would like to treat the fidelity term in a special way. When λ/ g n is big, we find that the scheme proposed below possesses better stability properties. We now describe the details for treating the fidelity term for our CrankNicolson the nominator and the denominator by the matrix scheme. Dividing S n = 1 + 2Δt √λgn I, and rearranging terms, we get 2 Δt n −1 ¯n −1 (S ) (U a )n+1 = I − All 2
l=1
2
Δt ¯n Δt ¯n A11 )(I + A22 ) + Δt A¯nqr (U a )n 2 2 q=1 r =q λ +2(S n )−1 ΔtF a √ n . g
(S n )−1 (I +
Approximating the semi-implicit scheme based on the LOD-splitting, we have −1 −1 1 1 I − Δt(S n )−1 A¯n11 (U a )n+1 = I − Δt(S n )−1 A¯n22 2 2 2
Δt ¯n Δt ¯n A11 )(I + A22 ) + Δt (S n )−1 (I + A¯nqr (U a )n + 2 2 q=1 r =q λ +2(S n )−1 ΔtF a √ n . g A similar splitting scheme can be developed using AOS.
5
Experimental Results
We proceed to demonstrate experimentally the stability, accuracy, and efficiency of the LOD and AOS splitting schemes for the Beltrami color flow. In Figure 1 we show the results of the Beltrami flow, implemented by employing the LOD splitting scheme for approximating Eq. (4). Next we illustrate the use of the splitting schemes in the case where the functional involves a fidelity term. A noisy image as well as the reference denoising result, based on the explicit scheme, are shown in Figure 3, next to the result of the AOS and LOD splitting schemes. Note that the visual results obtained by the two schemes are similar to the reference image. 5.1
RRE Extrapolation Technique for Acceleration of the LOD Splitting Scheme
In [28, 1] vector extrapolation was applied in order to speed up the slow convergence of the explicit schemes for the Beltrami color flow. In the experiments
On Semi-implicit Splitting Schemes for the Beltrami Color Flow
267
Fig. 1. Top row, left: The original image which contains JPEG artifacts.√Middle: Results of the LOD splitting scheme with Δt = 1, after 1 iteration, β = 103 , λ = 0. Right: Results of the LOD splitting scheme with after 2 iterations. Bottom row, left: Results of the LOD splitting scheme with after 4 iterations. Middle: a close-up of the original image. Right: a close-up of the resulting image after 4 iterations.
Fig. 2. The different image channels of an image patch taken from the images in Figure 1. Left to right: An image patch before denoising, its different color channels, the denoised image, and the denoised color channels. The color arrows indicate the direction of the gradient in the various color channels.
below we demonstrate how the RRE extrapolation technique can also be used to accelerate the convergence of implicit schemes. Figure 4 shows that the RRE method accelerates the LOD scheme. A comparison is also given to the convergence rate achieved by the method of [28,1]. Extrapolation techniques also allow us to obtain a more accurate rate, if one takes a smaller time step.
268
L. Dascal et al.
Fig. 3. Large image at the right: An image with artifacts resulting from lossy compression.. Smaller images – a close-up on a section of the image. Top row, left: The image with JPEG artifacts. Right: Beltrami-based denoising by explicit scheme, run with 4000 explicit iterations, Δt = 0.0005. Bottom row,√left: Denoising by LOD, Δt = 0.02. Right: Denoising by AOS, Δt = 0.02. λ = 1, β = 2000.
10
Residual Norm
10
10
10
10
5
Explicit Explicit+RRE LOD LOD+RRE
0
−5
−10
−15
0
10
20 30 40 CPU Time (sec)
50
Fig. 4. Graph of the residuals (LOD, explicit+RRE and LOD+RRE) versus CPU times. Parameters: Δt = 0.05 for the explicit scheme, Δt = 2.5 for LOD, λ = 0.5, β = √ 500 ≈ 22.36.
6
Conclusions
Due to its anisotropy and non-separability nature, no implicit scheme, nor operator splitting based scheme was so far introduced for the partial differential equations that describe the Beltrami color flow. In this paper we propose a
On Semi-implicit Splitting Schemes for the Beltrami Color Flow
269
semi-implicit splitting scheme based on LOD/AOS for the anisotropic Beltrami operator. The spatial mixed derivatives are discretized explicitly at time step nΔt , while the non-mixed derivatives are approximated using the average of the two time levels nΔt and (n + 1)Δt. The stability of the splitting is empirically tested in applications such as Beltrami-based scale-space and Beltrami-based denoising, which display a stable behavior. In order to further accelerate the convergence of the splitting schemes, the RRE vector extrapolation technique is employed.
Acknowledgements We thank Prof. Avram Sidi for interesting discussions. This research was supported by the United States -Israel Binational Science Foundation grant No. 2004274, by the Israeli Science Foundation grant No. 623/08, by the Ministry of Science grant No. 3-3414, and by the Elias Fund for Medical Research. XueCheng Tai is supported by the MOE (Ministry of Education) Tier II project T207N2202 and IDM project NRF2007IDMIDM002-010.
References 1. Rosman, G., Dascal, L., Kimmel, R., Sidi, A.: Efficient beltrami image filtering via vector extrapolation methods. SIAM J. Imag. Sci. (2008) (submitted) 2. Mešina, M.: Convergence acceleration for the iterative solution of the equations X = AX + f . Comp. Meth. Appl. Mech. Eng. 10, 165–173 (1977) 3. Eddy, R.: Extrapolating to the limit of a vector sequence. In: Wang, P. (ed.) Information Linkage Between Applied Mathematics and Industry, New York, pp. 387–396. Academic Press, London (1979) 4. Spira, A., Kimmel, R., Sochen, N.A.: A short-time Beltrami kernel for smoothing images and manifolds. IEEE Trans. Image Process. 16(6), 1628–1636 (2007) 5. Smith, S.M., Brady, J.: Susan - a new approach to low level image processing. Intl. J. of Comp. Vision 23, 45–78 (1997) 6. Aurich, V., Weule, J.: Non-linear gaussian filters performing edge preserving diffusion. In: Mustererkennung 1995, 17. DAGM-Symposium, London, UK, pp. 538–545. Springer, Heidelberg (1995) 7. Tomasi, C., Manduchi, R.: Bilateral filtering for gray and color images. In: Proceedings of IEEE International Conference on Computer Vision, pp. 836–846 (1998) 8. Sochen, N., Kimmel, R., Bruckstein, A.M.: Diffusions and confusions in signal and image processing. J. of Math. Imag. and Vision 14(3), 195–209 (2001) 9. Elad, M.: On the bilateral filter and ways to improve it. IEEE Trans. Image Process. 11(10), 1141–1151 (2002) 10. Barash, D.: A fundamental relationship between bilateral filtering, adaptive smoothing and the nonlinear diffusion equation. IEEE Trans. Image Process. 24(6), 844–847 (2002) 11. Buades, A., Coll, B., Morel, J.M.: A review of image denoising algorithms, with a new one. SIAM Interdisciplinary Journal 4, 490–530 (2005) 12. Lu, T., Neittaanmaki, P., Tai, X.C.: A parallel splitting up method and its application to Navier-Stokes equations. Applied Mathematics Letters 4(2), 25–29 (1991)
270
L. Dascal et al.
13. Lu, T., Neittaanmaki, P., Tai, X.C.: A parallel splitting up method for partial differential equations and its application to Navier-Stokes equations. RAIRO Mathematical Modelling and Numerical Analysis 26(6), 673–708 (1992) 14. Weickert, J., Romeny, B.M.T.H., Viergever, M.A.: Efficient and reliable schemes for nonlinear diffusion filtering. IEEE Trans. Image Process. 7(3), 398–410 (1998) 15. Peaceman, D.W., Rachford, H.H.: The numerical solution of parabolic and elliptic differential equations. Journal Soc. Ind. Appl. Math. 3, 28–41 (1955) 16. Yanenko, N.N.: The method of fractional steps. The solution of problems of mathematical physics in several variables. Springer-Verlag, New York (1971) 17. Barash, D., Schlick, T., Israeli, M., Kimmel, R.: Multiplicative operator splittings in nonlinear diffusion: from spatial splitting to multiple timesteps. J. of Math. Imag. and Vision 19(16), 33–48 (2003) 18. Kimmel, R., Malladi, R., Sochen, N.: Images as embedding maps and minimal surfaces: Movies, color, texture, and volumetric medical images. Intl. J. of Comp. Vision 39(2), 111–129 (2000) 19. Sochen, N., Kimmel, R., Maladi, R.: From high energy physics to low level vision. In: ter Haar Romeny, B.M., Florack, L.M.J., Viergever, M.A. (eds.) Scale-Space 1997. LNCS, vol. 1252, pp. 236–247. Springer, Heidelberg (1997) 20. Sochen, N., Kimmel, R., Maladi, R.: A general framework for low level vision. IEEE Trans. Image Process. 7, 310–318 (1998) 21. Yezzi, A.J.: Modified curvature motion for image smoothing and enhancement. IEEE Trans. Image Process. 7(3), 345–352 (1998) 22. Polyakov, A.M.: Quantum geometry of bosonic strings. Physics Letters 103 B, 207–210 (1981) 23. Rudin, L., Osher, S., Fatemi, E.: Non-linear total variation based noise removal algorithms. Physica D Letters 60, 259–268 (1992) 24. Yanenko, N.N.: About implicit difference methods of the calculation of the multidimensional equation of thermal conductivity. In: Proceedings of VUZ. Series of Mathematics, vol. 23(4), pp. 148–157 (1961) 25. Andreev, V.B.: Alternating direction methods for parabolic equations in two space dimensions with mixed derivatives. Zhurnal Vychislitelnoi Matematiki i Matematicheskoi Fiziki 7(2), 312–321 (1967) 26. Mckee, S., Mitchell, A.R.: Alternating direction methods for parabolic equations in three space dimensions with mixed derivatives. The Computer Journal 14(3), 25–30 (1971) 27. Weickert, J.: Coherence-enhancing diffusion filtering. Intl. J. of Comp. Vision 31(2/3), 111–127 (1999) 28. Dascal, L., Rosman, G., Kimmel, R.: Efficient Beltrami filtering of color images via vector extrapolation. In: Sgallari, F., Murli, A., Paragios, N. (eds.) SSVM 2007. LNCS, vol. 4485, pp. 92–103. Springer, Heidelberg (2007)
Multi-scale Total Variation with Automated Regularization Parameter Selection for Color Image Restoration Yiqiu Dong1 and Michael Hintermüller2 1 START-Project “Interfaces and Free Boundaries” and SFB “Mathematical Optimization and Applications in Biomedical Science”, Institute of Mathematics and Scientific Computing, University of Graz, Heinrichstrasse 36, A-8010 Graz, Austria [email protected] 2 Department of Mathematics, Humboldt-University of Berlin, Unter den Linden 6, 10099 Berlin, Germany, and START-Project “Interfaces and Free Boundaries” and SFB “Mathematical Optimization and Applications in Biomedical Science”, Institute of Mathematics and Scientific Computing, University of Graz, Heinrichstrasse 36, A-8010 Graz, Austria [email protected]
Abstract. In this paper, a multi-scale vectorial total variation model for color image restoration is introduced. The model utilizes a spatially dependent regularization parameter in order to preserve the details during noise removal. The automated adjustment strategy of the regularization parameter is based on local variance estimators combined with a confidence interval technique. Numerical results on images are presented to demonstrate the efficiency of the method.
1
Introduction
We consider the problem of recovering color images degraded by cross-channel ˆ blurring and Gaussian noise. Without loss of generality, we assume an image u is a vectorial function defined on a bounded and piecewise smooth open subset ˆ : Ω → RM , where M is the number of channels in the color Ω ∈ R2 , that is, u ˆ is given by model. The degraded form z of u ˆ + n, z = Ku where K ∈ L(L2 (Ω; RM )) is a cross-channel blurring operator, and n represents white Gaussian noise with zero mean and standard deviation σ. The problem of ˆ from z with unknown n is known to be typically ill-posed [1]. restoring u In order to preserve significant edges during restoring images, Rudin, Osher and Fatemi proposed total variation regularization [2] for gray-level images. In this approach (which we call the TV-model in what follows), the image u ˆ is recovered by solving the optimization problem λ min |Du| + |Ku − z|2 dx, (1) u∈BV (Ω) Ω 2 Ω X.-C. Tai et al. (Eds.): SSVM 2009, LNCS 5567, pp. 271–281, 2009. c Springer-Verlag Berlin Heidelberg 2009
272
Y. Dong and M. Hintermüller
where BV (Ω) denotes the space of functions of bounded variation and λ > 0. Because of the edge preservation ability, the TV-model is widely accepted as a reliable tool in image restoration. Over the years, various research efforts have been devoted to studying, solving and extending the TV-model; see, e.g., [3, 4, 5, 6, 7, 8, 9] as well as the monograph [1] and the many references therein. In general, images are comprised of multiple objects at different scales. This suggests that different values of λ localized at image features of different scales are desirable to obtain better restoration results. For this reason, a multi-scale total variation (MTV) model with a spatially varying choice of parameters was proposed [10]. In order to enhance image regions containing details while still sufficiently smoothing homogeneous features, a spatially dependent regularization parameter selection was proposed in [11]. In this paper, we will extend the multi-scale total variation with spatially dependent regularization parameter to restore degraded color images. The automated adjustment strategy of the regularization parameter is based on local variance estimators combined with a confidence interval technique. For speeding up the performance of the scheme we generalize the multi-scale representation according to [12, 13], and the corresponding subproblems are solved by a superlinearly convergent algorithm based on Fenchel-duality and inexact semismooth Newton techniques. The latter extends earlier work in [9]. The outline of the rest of the paper is as follows. In Section 2 we introduce the multi-scale vectorial total variation model and the primal-dual algorithm for solving the associated minimization problem. In Section 3 we extend the LVE-based parameter selection to color images. Section 4 proposes a method for color image restoration combining the multi-scale representation and spatially adaptive parameter selection. Section 5 gives numerical results to demonstrate the performance of the new method. Finally conclusions are drawn in Section 6.
2
Multi-scale Vectorial Total Variation
Based on the TV-model (1), in [14] the vectorial total variation (VTV) regularization was proposed for restoring color images: λ min |Du| + |Ku − z|2 dx, (2) 2 Ω u∈BV (Ω;RM ) Ω where the space BV (Ω; RM ) of vector-valued functions is the set of functions u ∈ L1 (Ω; RM ) such that Ω |Du| < ∞, where the vectorial TV norm Ω |Du| is defined as |Du| = sup u · divv dx : v ∈ Cc1 (Ω; RM×2 ), |v| ≤ 1 , Ω
and |v| =
Ω
M i=1 vi , vi .
The space BV (Ω; RM ) endowed with the norm uBV (Ω;RM ) = uL1 (Ω;RM ) + |Du| Ω
is a Banach space.
Multi-scale Total Variation
273
In the VTV-model (2), the parameter λ controls the trade-off between a good fit of z and a smoothness requirement due to the vectorial total variation regularization. Since images are usually comprised of multiple objects at different scales, locally different λ is desirable. Therefore, here we consider multi-scale vectorial total variation (MVTV): 1 min |Du| + λ(x)|Ku − z|2 dx. (3) 2 Ω u∈BV (Ω;RM ) Ω Similar as in Section 2 of [11], we can obtain the same conclusion on the existence and uniqueness of the minimizer for the MVTV-model. Here, we do not repeat proof details, but rather refer to [11]. 2.1
Primal-Dual Approach to Multi-scale Vectorial Total Variation
In [9] an infeasible primal-dual algorithm of generalized Newton-type was proposed for solving (1). In the sequel we extend its key features to the case (3). Rather than operating on the MVTV-model (3) the method is based on 1 μ 2 2 min |∇u| dx + λ|Ku − z| dx + |∇u| dx, (4) 2 2 Ω u∈H01 (Ω;RM ) 2 Ω Ω ¯ for almost all x ∈ Ω and 0 < μ λ ¯ −1 . The μwhere 0 < ≤ λ(x) ≤ λ term serves the purpose of a function space regularization for a “convenient" dualization in a Hilbert space setting. In our numerics, we typically choose μ = 0. Applying the Fenchel-Legendre calculus [15] analogously as in [9], the Fencheldual of (4) reads sup 2
M
p ∈ L (Ω; R ) |p(x)| ≤ 1 a.e. in Ω
1 1 − |||K ∗ z − divp|||2H −1 + z2L2 , 2 2
(P0 )
where |||u|||2H −1 = Hμ,K v, vH01 ,H −1 , v ∈ H −1 (Ω; RM ) with Hμ,K = (K ∗ λK − μ)−1 , : H01 (Ω; RM ) → H −1 (Ω; RM ), and ·, ·H01 ,H −1 denotes the duality pairing between H01 (Ω; RM ) and its dual H −1 (Ω; RM ). Moreover, L2 (Ω; RM ) = (L2 (Ω; RM ))2 . In order to avoid the non-uniqueness of the solution of (P0 ), following [9] we consider a dual regularization: 1 1 β ∗ 2 2 sup − |||K z − divp|||H −1 + zL2 − p2L2 . (P ) 2 2 2 Ω p ∈ L2 (Ω; RM ) |p(x)| ≤ 1 a.e. in Ω
where β > 0 is the regularization parameter. In order to study the effect of the βregularization of the Fenchel-dual, we apply the Fenchel-Legendre calculus once more and find that the dual of (P ) is given by 1 μ 2 2 |∇u|2 dx + λ|Ku − z| dx + Φβ (∇u)dx, (P ∗ ) min 2 Ω u∈H01 (Ω;RM ) 2 Ω Ω where for w ∈ L2 (Ω; RM ),
274
Y. Dong and M. Hintermüller
Φβ (w)(x) =
|w(x)| − β2 , if |w(x)| ≥ β, 1 2 2β |w(x)| , if |w(x)| < β.
(5)
¯ and The first-order optimality conditions of (P ∗ ) characterize the solution u ¯ of (P ∗ ) and (P ), respectively, by p ¯ − div¯ − μ¯ u + K ∗ λK u p = K ∗ λz in H −1 (Ω; RM ), max(β, |∇¯ u|)¯ p − ∇¯ u=0
2
in L (Ω; R ). M
(6a) (6b)
Note that the system (6) is non-smooth, i.e. not necessarily Fréchet-differentiable. The discrete version of this system can be solved by a semismooth Newton method [9, 11]. The generalized Newton solver converges globally, that is regardless of the initialization, and locally at a superlinear rate [9].
3
Spatially Dependent Regularization Parameter Selection
Since the capability of multi-scale vectorial total variation is mainly limited by the selection of the parameter λ, in this section we extend the way to choose λ proposed in [11] to the MVTV-model. Suppose the variance of Gaussian noise is σ 2 , which can be estimated easily in practice. With a correct choice of λ in the TV-model (1), the restored image u can satisfy the constraint |Ku − z|2 dx = σ 2 |Ω| (7) Ω
globally. However, the MVTV-model (3) represents a localized version of the constraint by allowing λ = λ(x). In order to enhance image details while preserving homogenous regions, the choice of λ must be based on local image features. Hence, we search for a reconstruction where the variance of the residual is closer to the noise variance in both the detail regions and the homogeneous parts. In order to achieve this goal we introduce local variance estimators (LVEs) for an automated adaptive choice of λ. 3.1
Local Variance Estimator
Consider the discrete version of the residual image rh = zh − K h uh , where uh is the restored image from the minimization problem (2) with λ > 0. If we use a relatively small parameter λ, the residual rh will include the noise as well as the details. Then, the average of the squared residual in a small window will reflect the distribution of details in the image. ω Let Ωi,j denote the set of pixel-coordinates in a ω-by-ω window centered at (i, j) (with obvious modification near the boundary), i.e., ω ω
ω Ωi,j ≤ s, t ≤ , = (s + i, t + j) : − 2 2
Multi-scale Total Variation
275
where · means rounding to the nearest integer towards zero. Then we apply the mean filter with window size ω to the residual image rh as follows: LVEω i,j =
M 1 M ω2
rhs,t
ω k=1 (s,t)∈Ωi,j
2 k
=
M 1 M ω2
ω k=1 (s,t)∈Ωi,j
h 2 zs,t − (K h uh )s,t k .
Here LVE stands for a “Local Variance Estimator”. In general, LVEω has a large value in the detail regions, and it has a small value in the homogeneous regions. But the noise in the residual may also lead to some large LVE values in the homogeneous regions. In order to reduce the effect due to noise, we utilize the confidence interval technique well-known in statistics [16, 17] in connection with LVE. 3.2
Upper Bound for the Local Variance
In the discrete setting, all elements of n can be regarded as an array of independent normally distributed random variables with mean 0 and variance σ 2 . Then, the random variable M 1 ω Ti,j = 2 (nhs,t )2k σ ω k=1 (s,t)∈Ωi,j
ω has the χ2 -distribution with M ω 2 degrees of freedom, that is, Ti,j ∼ χ2Mω2 . Set M 1 ω Si,j := (zhs,t − (K h uh )s,t )2k . M ω2 ω k=1 (s,t)∈Ωi,j
ˆ h , then ˆ h satisfies nh = zh − K h u If uh = u ω Si,j =
M 1 M ω2
M 1 M ω2
ˆ h )s,t )2k (zhs,t − (K h u
ω k=1 (s,t)∈Ωi,j
=
(nhs,t )2k =
ω k=1 (s,t)∈Ωi,j
σ2 ω T . M ω 2 i,j
On the contrary, if the residual image zh − K h uh contains details, we expect ω Si,j =
M 1 M ω2
k=1
M 1 > M ω2
(zhs,t − (K h uh )s,t )2k
ω (s,t)∈Ωi,j
ω k=1 (s,t)∈Ωi,j
(nhs,t )2k =
σ2 ω T . M ω 2 i,j
ω > B for some pixel (i, j) Therefore, we search for a bound B such that Si,j implies that in the residual some details are left. Given m × m, the total number
276
Y. Dong and M. Hintermüller
of pixels in the color image with M channels, we propose to consider the expected σ2 ω 2 maximum of the m2 random variables Mω 2 Ts , s = 1, . . . , m , as the bound B: B ω,m :=
σ2 E( max T ω ), M ω 2 k=1,...,m2 k
(8)
where E represents the expected value of a random variable. Similar as proposed in [11], we get σ2 (Em (T ω ) + dm (T ω )), M ω2
B ω,m =
where Em (T ω ) = Td + βκm , dm (T ω ) = β π√6 , βm = m2 fm (Td ), κ = 0.577215, and m f(Td ) is the distribution of Td , which is the so-called dominant value. 3.3
Selection of the Parameter λ
Now, we use the confidence interval for S ω to reduce the effect from noise on the local variance estimators in order to distinguish the detail regions in the images correctly. Recall that LVEω denotes the mean of the squared residual in a given window. Ideally, there is only noise in the residual. Then LVEω should behave like S ω . Hence, whenever ω,m LVEω ), i,j ∈ [0, B
(9)
we assume that the window contains noise only. On the other hand, if (9) is not satisfied, we suppose that this is due to image details contained in the residual ω image in Ωi,j . This property is useful when updating the parameter λ locally. For adapting λ algorithmically we proceed as follows. Initially we assign a small positive value to λ. Then we restore the image iteratively by increasing λ according to the following rule: ˜ k+1 λ i,j
= ζ · min
˜k λ i,j
λk+1 i,j =
+ ω +ρ (LVEk )i,j − σ ,L ,
1 ω2
˜k+1 , λ s,t
(10a)
(10b)
ω (s,t)∈Ωi,j
where ζ ≥ 1, ρ > 0, (x)+ = max(x, 0), LVEω k is obtained from uk , L is a large ˜ k ∈ L∞ (Ω), and for each channel of the vectorial data positive value to ensure λ we use the same λk during restoration. In our numerics we choose ζ = 2 which comes from the method proposed in [12] (TNV-algorithm). Finally, we set the ˜ k ||∞ /σ in order to keep the new λ ˜ k+1 at the same scale parameter ρ = ρk = ||λ ˜ as λk .
Multi-scale Total Variation
4
277
Our Method
Recently, a multi-scale image decomposition method (TNV-algorithm) was proposed in [12], which uses the TV-model (1) to extract the details in the residual, and which varies the regularization parameter over a sequence of dyadic scales to capture different features in the image. Although this method performs better than a number of existing methods, it satisfies the constraint (7) only globally, and does not consider the local characteristic of the features in the image. Referring to this decomposition method, we intertwine its idea with the MVTV-model (3), and combine it with the spatially dependent regularization parameter selection. This results in the following algorithm: Algorithm 2
2
1: Initialize uh0 = 0 ∈ RMm , ph0 = 0 ∈ RMm ×2 , λ0 = [λ0 , · · · , λ0 ] ∈ RM with 2 λ0 ∈ Rm and k = 0. 2: If k = 0 solve the discrete version of the minimization problem 1 μ 2 0 2 ˜ 0 = arg u min |∇u|2 dx + λ |Ku − z| dx + |∇u| dx, 2 Ω u∈H01 (Ω;RM ) 2 Ω Ω else compute vkh = zh − K h uhk and solve the discrete version of the minimization problem: 1 μ 2 k 2 ˜ k = arg u min |∇u|2 dx + λ |Ku − vk | dx + |∇u| dx, 2 Ω u∈H01 (Ω;RM ) 2 Ω Ω ˜ hk . 3: Update uhk+1 = uhk + u h 4: Based on uk+1 , update
+ ˜ k+1 = 2 · min λ ˜k + ρ λ LVEω − σ ,L , k
(λk+1 )i,j =
1 ω2
˜k+1 . λ s,t
ω (s,t)∈Ωi,j
5: Stop; or set k := k + 1 and go to step 2. A few remarks on the algorithm are in order. We initialize λ by a relatively small positive constant. In our numerical practice an 11-by-11 window turned out to yield reliable results. In Section 5, we study the influence of the window size on the restoration results. Similar to the Bregman iteration proposed in [18], we stop the iterative procedure as soon as the residual zh −K h uhk 2 drops below ξσ, where ξ > 1 relates to the image size. For m → ∞ we have ξ → 1.
5
Numerical Results
In this section we provide numerical results to study the behavior of our method with respect to its image restoration capabilities. We use two RGB color images
278
Y. Dong and M. Hintermüller
(a)
(b)
Fig. 1. Original images: (a) “Barbara”, (b) “Lena”
(a)
(b)
(c)
Fig. 2. Results of denoising image “Barbara” (the 1st row) and “Lena” (the 2nd row): (a) Noisy images, (b) Restored images (k = 3), (c) Final values of λ
(i.e., M = 3), “Barbara” (576-by-720) and “Lena” (512-by-512), as shown in Figure 1. Furthermore, from the experiments conducted on a broad variety of images we found that our method is robust with respect to the initial choice of λ. Thus, in all experiments listed here we use the same initial choice λ = 2.5. 5.1
Color Image Denoising
Here, we concentrate on image denoising, i.e., K h is the identity matrix. The degraded images containing Gaussian white noise with the noise level σ = 0.1. For a study of our method in the case of texture-like structures we zoom the
Multi-scale Total Variation
279
Fig. 3. Restored images by our method with different ω: (a) ω = 5, (b) ω = 11, (c) ω = 17
(a)
(b)
(c)
Fig. 4. Results of restoring blurred noisy image “Barbara” (the 1st row) and “Lena” (the 2nd row): (a) Blurred noisy images, (b) Restored images (k = 5), (c) Final values of λ
two images in Figure 1 into certain regions. In all of our experiments the image intensity range is scaled to [0, 1]. The results are shown in Figure 2 together with the number of iterations k. We can see that our method suppresses the noise successfully while preserving the details. In addition, we also show the final values of λ obtained by our choice rule. We find that in detail regions λ is large in order to preserve the details, and it is small in the homogeneous regions to remove noise.
280
Y. Dong and M. Hintermüller
In order to test our method for different values of the window size ω, Figure 3 shows the restored images with ω = 5, 11, 17. Except for some slight effects, we observe a remarkable stability with respect to ω. 5.2
Color Image Deblurring and Denoising
In this section, we illustrate the restoration ability of our method for noisy blurred images. The blurring operator K is a cross-channel blurring operator with the kernel: ⎡ ⎤ ⎡ ⎤ Krr Krg Krb 0.8 · (M, 7, 135) 0.1 · (G, 9, 7) 0.1 · (A, 7) ⎣ Kgr Kgg Kgb ⎦ = ⎣ 0.1 · (A, 9) 0.8 · (M, 7, 90) 0.1 · (G, 5, 1) ⎦ , Kbr Kbg Kbb 0.1 · (G, 7, 5) 0.1 · (M, 7, 45) 0.8 · (A, 11) where (A, r) denotes the average blur with window size r, (G, r, σ) denotes the Gaussian blur with window size r and standard deviation σ, (M, l, θ) denotes the motion blur with length l and angle θ, and (r, g, b) are the three channels in the RGB color model. Further we have Gaussian white noise with σ = 0.02. Figure 4 depicts a part of the noisy blurred “Barbara” and “Lena” images with the restored results and final values of λ. We find that our method still can preserve most of the details; see, e.g., the features on the scarf. Furthermore, for noisy blurred images our method is still able to distinguish most of the detail regions properly.
6
Conclusion
A multi-scale vectorial total variation model with spatially adapted regularization parameter λ for color image restoration is proposed in this paper. The local variance estimator LVE of the residual image is extended to the multi-channel case, and turns out to be an accurate instrument for updating λ within an iterative procedure. Assuming that the noise variance σ 2 is known, the present algorithm is completely automatized, i.e., there is no necessity of tuning parameters. The numerical results show that the new method can restore the degraded images efficiently while preserving most details.
References 1. Vogel, C.: Computational Methods for Inverse Problems. Frontiers Appl. Math., vol. 23. SIAM, Philadelphia (2002) 2. Rudin, L., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Physica D 60, 259–268 (1992) 3. Dobson, D., Vogel, C.: Convergence of an iterative method for total variation denoising. SIAM J. Numer. Anal. 34, 1779–1791 (1997) 4. Chambolle, A., Lions, P.L.: Image recovery via total variation minimization and related problems. Numerische Mathematik 76, 167–188 (1997)
Multi-scale Total Variation
281
5. Chang, Q., Chern, I.L.: Acceleration methods for total variation-based image denoising. SIAM J. Applied Mathematics 25, 982–994 (2003) 6. Strong, D., Chan, T.: Edge-preserving and scale-dependent properties of total variation regularization. Inverse Problems 19, 165–187 (2003) 7. Chambolle, A.: An algorithm for total variation minimization and application. Journal of Mathematical Imaging and Vision 20, 89–97 (2004) 8. Hintermüller, M., Kunisch, K.: Total bounded variation regularization as bilaterally constrained optimization problem. SIAM J. Appl. Math. 64, 1311–1333 (2004) 9. Hintermüller, M., Stadler, G.: An infeasible primal-dual algorithm for total bounded variation-based inf-convolution-type image restoration. SIAM Journal on Scientific Computing 28(1), 1–23 (2006) 10. Almansa, A., Ballester, C., Caselles, V., Haro, G.: A TV based restoration model with local constraints. J. Sci. Comput. 34(3), 209–236 (2008) 11. Dong, Y., Hintermüller, M., Rincon-Camacho, M.: Automated parameter selection in a multi-scale total variation model. IFB-Report No. 22, Institute of Mathematics and Scientific Computing, University of Graz (November 2008) 12. Tadmor, E., Nezzar, S., Vese, L.: A multiscale image representation using hierarchical (BV, L2 ) decompositions. Multiscale Model. Simul. 2, 554–579 (2004) 13. Tadmor, E., Nezzar, S., Vese, L.: Multiscale hierarchical decomposition of images with applications to deblurring, denoising and segmentation. Comm. Math. Sci. 6, 1–26 (2008) 14. Bresson, X., Chan, T.: Fast dual minimization of the vectorial total variation norm and applications to color image processing. Inverse Problems and Imaging 2(4), 455–484 (2008) 15. Ekeland, I., Témam, R.: Convex Analysis and Variational Problems. Classics Appl. Math., vol. 28. SIAM, Philadelphia (1999) 16. Papoulis, A.: Probability, Random Variables, Stochastic Processes. McGraw Hill, New York (1991) 17. Mood, A.: Introduction to the Theory of Statistics. McGraw-Hill, New York (1974) 18. Osher, S., Burger, M., Goldfarb, D., Xu, J., Yin, W.: An iterative regularization method for total variation-based image restoration. SIAM Multiscale Model. and Simu. 4, 460–489 (2005)
Multiplicative Noise Cleaning via a Variational Method Involving Curvelet Coefficients Sylvain Durand1 , Jalal Fadili2 , and Mila Nikolova3 1
2
3
M.A.P. 5 - CNRS, University Paris Descartes, France [email protected] http://www.math-info.univ-paris5.fr/∼sdurand/ GREYC CNRS-ENSICAEN-Université de Caen, France [email protected] http://www.greyc.ensicaen.fr/∼jfadili/ CMLA - CNRS, ENS Cachan, PRES UniverSud, France [email protected] http://www.cmla.ens-cachan.fr/∼nikolova/
Abstract. Classical ways to denoise images contaminated with multiplicative noise (e.g. speckle noise) are filtering, statistical (Bayesian) methods, variational methods and methods that convert the multiplicative noise into additive noise (using a logarithmic function) in order to apply a shrinkage estimation for the log-image data and transform back the result using an exponential function. We propose a new method that involves several stages: we apply a reasonable under-optimal hard-thresholding on the curvelet transform of the log-image; the latter is restored using a specialized hybrid variational method combining an 1 data-fitting to the thresholded coefficients and a Total Variation regularization (TV) in the image domain; the restored image is an exponential of the obtained minimizer, weighted so that the mean of the original image is preserved. The minimization stage is realized using a properly adapted fast Douglas-Rachford splitting. The existence of a minimizer of our specialized criterion and the convergence of the minimization scheme are proved. The obtained numerical results outperform the main alternative methods.
1
Introduction
In many active imaging systems (e.g. synthetic aperture radar, laser or ultrasound imaging), the data for the unknown image S0 : Ω → R+ , Ω ⊂ R2 , are severely corrupted with multiplicative noise. Then several independent measurements for the same image are needed: Sk = S0 ηk + nk ,
∀k ∈ {1, · · · , K},
(1)
where ηk : Ω → R+ , and nk represent the multiplicative and a typically zeromean additive noise, ∀k. Commonly (see e.g. [27]) ηk is modeled as a onesided exponential probability density function (pdf) (cf. Fig. 1(a)): pdf(ηk ) = X.-C. Tai et al. (Eds.): SSVM 2009, LNCS 5567, pp. 282–294, 2009. c Springer-Verlag Berlin Heidelberg 2009
Multiplicative Noise Cleaning
283
μ e−μηk 1lR+ (ηk ) for μ = 1. In practice, one takes an average of all measurements, 1 K see e.g. Fig. 2(b). Since K k=1 nk ≈ 0, the data read (cf. e.g. [27, 1, 30]): S=
K K 1 1 Sk = S 0 ηk = S0 η . K K k=1
(2)
k=1
Usually all ηk are independent. Denoting by Γ the usual Gamma-function, the mean of the noise η in (2) has a Gamma distribution (cf. Fig. 1(b)): η=
K 1 ηk : K
pdf(η) =
k=1
K K η K−1 exp (−Kη) . Γ (K)
(3)
Various adaptive filters have been proposed, see e.g. [31,17]: they work well when the noise is moderate or weak, i.e. for K large. Bayesian, variational or diffusionbased methods have been proposed as well; see e.g. [28, 24, 18, 2]. Numerous methods convert the multiplicative noise into additive noise by v = log S = log S0 + log η = u0 + n,
(4)
see e.g. [16, 30, 1, 23]. Then the pdf of n reads (cf. Fig. 1(c)): −1 exp − K(n − en ) . n = log η : pdf(n) = K K Γ (K)
(5)
One can prove that E [n] = ψ0 (K) − log K and Var [n] = ψ1 (K), where ψk (z) = d k+1 log Γ (z) is the polygamma function. A common strategy is to decompose dz the log-data v into a multiscale frame for L2 (R2 ) (an over-complete basis), say W ≡ {wi : i ∈ I} where I is a set of indexes: y = W v = W u0 + W n.
(6)
By the Central Limit Theorem, the noise W n in y is nearly Gaussian — cf. Fig. 1(d). Then coefficients y are denoised using shrinkage estimators T : R → R: yT [i] = T (W v)[i] , ∀i ∈ I. (7) Shrinkage functions designed for multiplicative noise were proposed e.g. in [30,1]. ≡ {w Let W i : i ∈ I} be a left inverse of W . Then a denoised log-image vT reads T ((W v)[i]) w i = T (y[i]) w i . (8) vT = i∈I
i∈I
Then the sought-after image is of the form ST = exp vT . 1
1
0
10
(a) ηk
0
1
1
(b) η =
K 2
1 K
k=1
1
−2
0
1
ηk (c) n = log η
Fig. 1. Noise distributions
−1
0
(d) W n
1
284
S. Durand, J. Fadili, and M. Nikolova
Our approach. We apply (4) and consider a tight-frame transform of the logdata. The restored log-image (section 2) minimizes a criterion composed of an 1 -fitting to the (suboptimally) hard-thresholded frame coefficients and a Total Variation (TV) regularization in the image domain. The minimization (section 3) uses a specialized Douglas-Rachford splitting. The full algorithm, involving a bias correction, is given in section 4. Experiments are presented in section 5. Some notations. (.T ) means transposed, (.∗ ) means convex conjugate and (. ) means adjoint.
2
Restoration of the Log-Image
Here we consider how to restore a good log-image given data v : Ωmega → R obtained using (4). We focus on methods which, for a given preprocessed data set, lead to convex optimization problems. We comment only variational methods and shrinkage estimators since they underly our specialized hybrid objective function. 2.1
Drawbacks of Shrinkage Restoration and Variational Methods
Shrinkage restoration. The main problems with these methods, sketched in (7)(8), is that shrinking large coefficients entails an erosion of the spiky features, while shrinking small coefficients yields Gibbs-like oscillations in the vicinity of edges and a loss of details in the textured area. On the other hand, if shrinkage is insufficient, some coefficients bearing mainly noise can remain almost unchanged—we call such coefficients outliers—and (8) shows that they yield artifacts with the shape of the functions w i , see Fig. 2. Even though various improvements were brought, these artifacts remain visible—see the results on Fig. 3(d) and Fig. 4(c) in Section 5 using the very recent Stein-block thresholding [8].
(a) Noisy, K = 10
(b) T = 2 Var [n] (c) T = 4 Var [n] (d) T = 6 Var [n]
Fig. 2. (a) Noisy Lena obtained according to (1)-(2) for K = 10. (b)-(d) Restorations exp vTH where data v are denoised by hard-thresholding of its curvelet coefficients, see (12)-(13), for different choices of T .
Multiplicative Noise Cleaning
285
(b) Noisy: μ = 1, K = 10 (c)ˆ u by (11) and Sˆ by (34) see (2)-(3) psnr=26.2 db, mae=8.5
(a) Original (256 × 256)
(d) Stein-block [8] (e) AA algorithm [2] (f) Our method psnr=25.5 db, mae=9.4 psnr=25.4 db, mae=9.4 psnr=26.05 db, mae=8.8 Fig. 3. Restoration of (b) using modern methods. Note that (c) is a slightly improved version of [26] and that the restoration in (d) is done in the curvelet domain.
(a) Original
(b) Noisy
(c) Stein-block th.
(d) Our method
Fig. 4. (a) Shepp-Logan phantom (256 × 256). (b) Noisy, K = 10. (c) Denoised with Stein-block thresholding in the curvelet domain [8] PSNR=24.73dB, MAE=4. (d) Denoised with our algorithm PSNR=31.25dB, MAE=1.87.
Variational methods. In these methods, the restored function minimizes a criterion Fv of the form Fv (u) = ρ Ω
ψ u(t), v(t) dt +
ϕ(|∇u(t)|) dt, Ω
(9)
286
S. Durand, J. Fadili, and M. Nikolova
where ψ : R+→ R+ measures closeness to data and ϕ(|∇u(·)|) introduces priors 2 via a trade-off parameter ρ > 0. A classical choice is ψ = u(·)−v(·) . It is usually required that the potential function ϕ : R+ → R+ promotes images involving edges. Analysing the minimizers of Fv as solutions of PDE’s on Ω, Rudin, Osher and Fatemi [25] exhibited that ϕ(|∇u(t)|) = |∇u(t)|, leads to such images, where def for any z(t) = (z1 (t), z2 (t)) ∈ R2 , t ∈ Ω, one sets |z(t)| = z1 (t)2 + z2 (t)2 . The resulting regularization term is known as Total Variation (TV) and will be denoted by · TV . However, whatever smooth data-fitting is chosen, this regularization yields images containing numerous constant regions (called staircasing effect), hence textures and fine details are removed, see [22]. The method in [2] is of this kind and operates in the image domain; the fitting term is derived ˆ defined by from (3) and the denoised image S, Sˆ = arg min FS for FS (Σ) = ρ(K) log Σ(t) + S(t)/Σ(t) dt + Σ TV , (10) Σ
exhibits constant regions (see section 5). In [26], the regularization Σ TV is changed into log Σ TV so as to reformulate the model as a convex problem, and not to over smooth the image parts with higher gray values. To recover the denoised image, we applied Sˆ ∝ exp(ˆ u) for u ˆ = arg min where Fv (u) = ρ u − v 2 + u TV . u
(11)
Following [25], various edge-preserving convex functions ϕ have been proposed; see [3] for a recent overview. Even though ϕ (0) = 0 alleviates stair-casing, a systematic drawback of the resulting restored images is that the amplitude of edges is underestimated; thus neat edges or spiky areas are subjected to erosion. 2.2
Hybrid Methods
Hybrid methods, see e.g. [9, 19, 5, 14], combine the information contained in the large coefficients y[i] obtained according to (6) with priors directly on the image u. They amount to define the restored function u ˆ by minimize Φ(u) subject to u ˆ ∈ {u : |(W (u − v)) [i]| ≤ μi , ∀i ∈ I} . Using an edge-preserving regularization, such as Φ = TV is a pertinent choice. The selection of parameters {μi }i∈J is more tricky. This choice must take into account the magnitude of the relevant data coefficient y[i]. However, choosing μi based solely on y[i], as done in these papers, is too rigid since there are either correct data coefficients that incur smoothing (μi > 0), or noisy coefficients that are left unchanged (μi = 0). A good compromise that we adopt is to determine (μi )i∈I based both on the data and on the prior term. 2.3
A Specialized Hybrid Criterion
Given the log-data v obtained by (4), we apply a frame transform as in (6) to get y = W v = W u0 + W n. The noise contained in the i-th datum reads n, wi .
Multiplicative Noise Cleaning
287
The low frequency approximation coefficients carry important information on the image. Therefore, a good choice is to keep them intact at this stage. Let I∗ ⊂ I denote the subset of all such elements of the frame. Then we apply a hard-thresholding operator TH [12] to all coefficients I \ I∗ :
0 if |t| ≤ T, def yTH [i] = TH y[i] , ∀i ∈ I \ I∗ , where TH (t) = (12) t otherwise, where T is an underoptimal threshold in order to preserve the information relevant to edges and to some fine details in textured areas, contained in the small coefficients. Let us consider vTH = W v[i] w i , where I1 = {i ∈ I : |y[i]| > T } ∪ I∗ . (13) i∈I1
The image vTH contains a lot of artifacts with the shape of the w i for those y[i] that are noisy but above the threshold T , as well as information on the fine details in the original log-image u0 . In all cases, whatever the choice of T , an image of the form vTH is unsatisfactory—see Fig. 2. The denoised coefficients, denoted by x ˆ, are obtained based on the under-thresholded data yTH . We focus on hybrid methods of the form: x ˆ = arg minx F (x) for x), where Ψ is a data-fitting term in the frame domain and F (x) = Ψ (x, yTH ) + Φ(W Φ is an edge-preserving regularization term in the log-image domain. Let us denote I0 = I \ I1 = {i ∈ I \ I∗ : |y[i]| ≤ T }.
(14)
Coefficients y[i] for i ∈ I0 can be of the two types. 1. Coefficients y[i] bearing mainly noise—then the best choice is x ˆ[i] = 0; 2. Coefficients y[i] relevant to edges and other details in u0 . Since y[i] is difficult to distinguish from the noise, the relevant x ˆ[i] should be restored using the edge-preserving prior Φ. Note that a careful restoration must find a nonzero x ˆ[i] in order to avoid Gibbs-like oscillations in u ˆ. Coefficients y[i] for i ∈ I1 are of the following two types. 1. Large coefficients which carry the main features of the sought-after function. They verify y[i] ≈ wi , u0 and can be kept intact. 2. Coefficients highly contaminated by noise, i.e. |y[i]| | wi , u0 |. We call them outliers because if we had x ˆ[i] = y[i], then u ˆ would contain an artifact with the shape of w i since by (13) we get vTH = ˆ[j]w j + y[i]w i . Instead, x ˆ[i] must be restored according to the j\i x prior Φ. This analysis clearly defines the goals that the minimizer x ˆ of F is expected to achieve: restored coefficients x ˆ[i] have to fit yTH [i] exactly if they are coherent with the prior Φ, otherwise they have to be restored according to Φ. Since [21] it is known that such requirements can be satisfied by criteria F where Ψ is non-smooth at the origin (e.g. 1 ), see also [13]. For these reasons, we focus on F (x) = Ψ (x) + Φ(x),
(15)
288
S. Durand, J. Fadili, and M. Nikolova
where, for Λ = diag(λi )i∈I , λi |(x − y)[i]| + λi |x[i]| = Λ(x − yTH ) 1 , Ψ (x) = i∈I1 ∪I∗
Φ(x) =
Ω
(16)
i∈I0
x| ds = W x . |∇W TV
(17)
In the pre-processing step (12) we do not recommend the use of a shrinkage function other than TH since it will alter all the data coefficients without restoring them faithfully. Via TH , we base our restoration on data yTH where all nonthresholded coefficients keep the original information on the sought-after image. The theorem stated next addresses the existence and the uniqueness of a minimizer for F . Given y, let Gy be the (convex) set of all minimizers of F : def Gy = x ˆ ∈ 2 (I) : F (ˆ x) = min F (x) . (18) 2 x∈ (I)
2
Theorem 1. [13] For y ∈ (I) and T > 0 given, consider F as defined in (15), where Ω ∈ R2 is open, bounded and its boundary ∂Ω is Lipschitz. Suppose is the pseudo-inverse that {wi }i∈I is a frame of L2 (Ω) and the operator W of W . Assume also that λmin = min λi > 0. Then Gy is nonempty, and for all i∈I
x x x ˆ1 , x ˆ2 ∈ Gy , ∇W ˆ1 ∝ ∇W ˆ2 , a.e. on Ω. x x In words, Sˆ1 = W ˆ1 and Sˆ2 = W ˆ2 have the same level lines, i.e. they differ by a local change of contrast; the latter is usually invisible to the naked eye. The choice of λi is investigated in [13]. Following this analysis, we use only two values for λi , depending only on the set I the index i belongs to. We focus on curvelets transforms of the log-data because (a) such a transform captures efficiently the main features of the data and (b) it is a tight-frame which is helpful for the subsequent numerical stage.
3
Minimization for the Log-Image
Let Γ0 (H) denote the class of proper lower-semicontinuous convex functions on a Hilbert space H. Now we focus on the minimization problem find xˆ such that F (ˆ x) =
min F for F = Ψ + Φ, x
(19)
where Ψ and Φ are defined in (16)-(17). Clearly, Ψ, Φ ∈ Γ0 (2 (I)), hence F ∈ Γ0 (2 (I)). The set Gy in (18) is non-empty by Theorem 1 and can be rewritten as Gy = {x ∈ 2 (I) x ∈ (∂F )−1 (0)}, where ∂F stands for subdifferential. Minimizing F amounts to finding a solution to the fixed point equation x = (Id + γ∂F )−1 (x) ,
(20)
where (Id + γ∂F )−1 is the resolvent operator associated to ∂F , γ > 0 is the proximal stepsize and Id is the identity map on 2 (I). Since (Id + γ(∂Ψ + ∂Φ))−1 cannot be calculated in closed-form, we focus on splitting methods that use separately the resolvent operators (Id + γ∂Ψ )−1 and (Id + γ∂Φ))−1 .
Multiplicative Noise Cleaning
3.1
289
Specialized Douglas-Rachford (D-R) Splitting Algorithm
The D-R family is the most general class of monotone operator splitting methods. Given a sequence μt ∈ (0, 2), D-R methods can be expressed via the recursion μt μt Id+ (2(Id+γ∂Ψ )−1− Id) ◦ (2(Id+γ∂Φ)−1− Id) x(t) . (21) x(t+1)= 1− 2 2 Since problem (19) has solutions, we have the following convergence result: Theorem 2. Let γ > 0 and μt ∈ (0, 2) be such that t∈N μt (2 − μt ) = +∞. Take x(0) ∈ 2 (I) and consider the sequence of iterates defined by (21). Then, (x(t) )t∈N converges weakly to some point x ˆ ∈ 2 (I) and (Id+γ∂Φ)−1 (ˆ x) ∈ Gy . The statement follows from [10, Corollary 5.2]. The sequence μt = 1, ∀t ∈ N fits. 3.2
Proximal Calculus
Proximity operators, invented in [20], generalize convex projection. Definition 1 (Moreau [20]). Let ϕ ∈ Γ0 (H). Then ∀x ∈ H the function z → 2 ϕ(z)+ x−z /2, for z ∈ H, achieves its infimum at a unique point denoted by proxϕ x. The relevant operator proxϕ : H → H is the proximity operator of ϕ. By the minimality condition for proxϕ , it is easy to see that ∀x, p ∈ H we have p = proxϕ x ⇐⇒ x − p ∈ ∂ϕ(p) ⇐⇒ (Id + ∂ϕ)−1 = proxϕ . By introducing def
the reflection operator rproxϕ = 2proxϕ − Id, the D-R iteration (21) reads μt μt Id + rproxγΨ ◦ rproxγΦ x(t) . (22) x(t+1) = 1 − 2 2 Proximity operator of Ψ
Lemma 1. Let x ∈ 2 (I). Then proxγΨ (x) = yTH [i]+TS γλi (x[i]−yTH [i]) , i∈I where TS γλi (z[i]) = max 0, z[i] − γλi sign(z[i]) . The proof is quite standard and can be found in our Report [15]. Note that rproxγΨ (x) = 2 yTH [i] + TS γλi (x[i] − yTH [i]) −x . (23) i∈I
(x). Computing proxγΦ Proximity operator of Φ. Clearly, Φ(x) = · TV ◦ W for an arbitrary W may be intractable. We assume that : 2 (I) → L2 (Ω) is surjective; (w1) W W = Id and W = c−1 W for 0 < c < ∞; note that W W = c Id; (w2) W (w3) W is bounded.
Let X = L2 (Ω) × L2 (Ω), ·, · X be the inner product in X and
·
p , p ∈ γ
[1, ∞] the Lp -norm on X . DefineB ∞ (X ) as the γ-radius closed L∞ -ballin X ,
γ def B ∞ = z ∈ X :
z
∞ ≤ γ = z = (z1 , z2 ) ∈ X : |z(t)| ≤ γ, ∀t ∈ Ω , and γ
PB γ (X ) : X → B ∞ (X ) the associated projector. ∞
290
S. Durand, J. Fadili, and M. Nikolova γ
Lemma 2. Let x ∈ 2 (I) and B ∞ (X ) is as defined above. Then: (x) ; proxγΦ (x) = Id − W ◦ Id − proxc−1 γ·TV ◦ W
(24)
(25) proxc−1 γ·TV (u) = u − PC (u) ,
γ/c where C = div(z) ∈ L2 (Ω) z ∈ Cc∞ (Ω × Ω), z ∈ B ∞ (X ) . (26) 2 Sketch of the proof. By (w1), range(W ) = L (Ω). Using that domain( · TV ) = = {0}. Statement (i) follows from L2 (Ω), we find cone dom · TV − range W
applying [11, Proposition 11] whose requirements are satisfied. If ϕ ∈ Γ0 (L2 (Ω)) and ϕ∗ is its convex conjugate, the Moreau decomposition [20, Proposition 4.a] asserts proxϕ + proxϕ∗ = Id .
(27)
Since the conjugate function of a norm is the indicator function of the ball of ∗ ∈ C. where C is given its dual norm, c−1 γ · TV (z) = 0 if z ∈ C, +∞ if z ∗ = PC . Identifying c−1 γ . TV with in (26). Using Definition 1, prox −1 c γ.TV ∗ ϕ and c−1 γ . TV with ϕ∗ , equation (27) leads to (ii)1 . From (24)-(25) we easily find that (x) . rproxγΦ (x) = Id − 2W ◦ PC ◦ W
(28)
Calculation of the projection PC in (25) on a discrete grid. In this case, W is an M×N tight frame with M= #I N = #Ω and assumption (w2) reads W = Id and W = c−1 W T , c ∈ (0, ∞) hence W T W = c Id). The discrete W counterpart of X is X = 2 (Ω) × 2 (Ω). We denote the discrete gradient by ¨ (cf. [6] or [29]) and the discrete divergence Div : X → 2 (Ω) is defined as ∇ ¨ . Moreover, C in (26) admits a simpler expression: Div = −∇
γ/c (29) C = Div(z) ∈ 2 (Ω) z ∈ B ∞ (X ) , γ/c
where B ∞ (X ) is defined using the new discrete notations. The projection PC in (25) does not admit an explicit form so we provide an iterative scheme for its calculation in the next lemma. Lemma 3. We adapt all assumptions of Lemma 2 to the new discrete setting, as explained above. Consider the forward-backward iteration ¨ Div(z (t) ) − cu/γ z (t+1) = PB 1 (X ) z (t) + βt ∇ (30) ∞
(31) 0 < inf βt ≤ sup βt < 1/4 t t z[i, j] if |z[i, j]| ≤ 1; PB 1 (X ) (z)[i, j] = (32) ∞ z[i, j]/|z[i, j]| otherwise . for
where ∀(i, j) ∈ Ω , 1
Note that our argument (27) to compute proxc−1 γ·TV (u) is not used in [6], which instead uses conjugates and bi-conjugates of the objective function.
Multiplicative Noise Cleaning
291
Then 1
(i) (z (t) )t∈N converges to a point zˆ ∈ B ∞ (X ); −1 (t) (ii) c γDiv(z ) converges to c−1 γDiv(ˆ z ) = (Id − proxc−1 γ·TV )(u). t∈N
The proof of this lemma can be found in our Report [15]. The iteration proposed in (30) to compute the proximity operator of the TV-norm is different from the projection algorithm of [6]. A similar iteration was proposed in [7] and in some other articles. The proof we gave is however simpler as it uses known properties of proximity operators. Note that computing prox·TV amounts to solving a discrete ROF-denoising. Our iteration to solve this problem is one possibility among others, see e.g. a recent report [4]. A crucial property of the D-R scheme (22) is its robustness to numerical errors that may occur when computing the proximity operators proxΨ and proxΦ , see [10]. More precisely, let at ∈ 2 (I) be an error term that models the inexact computation of proxγΦ in (24), as the latter is obtained through (30). If the sequence of error terms (at )t∈N and stepsizes (μt )t∈N in Theorem 2 obey t∈N μt at < +∞, then the D-R algorithm (22) converges [10, Corollary 6.2]. In our experiments, using 200 inner iterations in (30) is sufficient to satisfy this requirement. 3.3
Bias Correction to Recover the Sought-After Image
x Recall from (4) that u0 = log S0 and set u ˆ=W ˆ(NDR ) as the estimator of u0 , where NDR is the number of D-R iterations in (22). Unfortunately, the estimator u ˆ is prone to bias, i.e. E [ˆ u] = u0 − buˆ . A problem that classically arises in statistical estimation is how to correct such a bias. More importantly is how this bias affects the estimate after applying the inverse transformation, here the ˆ exponential. Our goal is then to ensure that for the estimate S of the image, we ˆ ˆ have E S = S0 . Expanding S in the neighborhood of E [ˆ u], we have u])(1+Var [ˆ u] /2+R2 ) = S0 exp (−buˆ )(1+Var [ˆ u] /2+R2 ) , (33) E euˆ = exp (E [ˆ where R2 is expectation of the Lagrange remainder in the Taylor series. One can observe that the posterior distribution of u ˆ is nearly symmetric, hence R2 ≈ 0. Then buˆ ≈ log(1v +Var [ˆ u] /2) ensures unbiasedness. Consequently, finite sample (nearly) unbiased estimates of u0 and S0 are respectively u ˆ + log(1 + Var [ˆ u] /2), and exp (ˆ u) (1 + Var [ˆ u] /2). Var [ˆ u] can be reasonably estimated by ψ1 (K), the variance of the noise n in (4) being given in (1). Thus, given the restored logimage u ˆ, our denoised image read: Sˆ = exp (ˆ u) (1 + ψ1 (K)/2) .
4
(34)
Full Algorithm to Suppress Multiplicative Noise
Piecing together Lemmas 1 and 2, and Theorem 2, we write down the full multiplicative noise removal algorithm:
292
S. Durand, J. Fadili, and M. Nikolova
Task: Denoise an image S corrupted with multiplicative noise according to (2). Parameters: The observed noisy image S, number of iterations NDR (DouglasRachford outer iterations) and NFB (Forward-Backward inner iterations), stepsizes μt ∈ (0, 2), 0 < βt < 1/4 and γ > 0, tight-frame transform W and initial threshold T (e.g. T = 2 ψ1 (K)), regularization parameters λ0,1 associated to the sets I0,1 . Specific operators: (a) TS γλi (z) = max 0, z[i] − γλi sign(z[i]) , ∀z ∈ R#I . i∈I
z[i, j] if |z[i, j]| ≤ 1 (b) ∀(i, j) ∈ Ω, PB 1 (X ) (z)[i, j] = ∞ z[i, j]/|z[i, j]| else. ¨ and Div—the discrete versions of the continuous operators ∇ and div. (c) ∇ (d) ψ1 (·) defined according to (1) (built-in Matlab function). Initialization: Compute v = log S and transform coefficients y = W v. Hardthreshold y at T to get yTH . Choose x(0) . Main iteration: For t = 1 to NDR , x(t) . (1) Inverse curvelet transform of x(t) according to u(t) = W (0) (2) Initialize z ; For s= 0 to NFB− 1 ¨ Div(z (s) ) − c u(t) . z (s) + βt ∇ z (s+1) = P 1 B ∞ (X )
γ
(3) Set z (t) = z (NFB) and compute w(t) = c−1 γ Div(z (t) ). (4) Forward curvelet transform: α(t) = W w(t) . (5) Compute r(t) = rproxγΦ (x(t) ) = x(t) − 2α(t) . (6) Find q (t)= rproxγΨ ◦ rproxγΦ x(t)= 2 yTH [i]+TS γλi r(t) [i]−yTH [i] −r(t) . (t) (7) Update x(t+1) : x(t+1) = (1 − μt /2) x(t) + (μt /2)q . x(NDR ) (1 + ψ1 (K)/2). Output: Denoised image Sˆ = exp W
5
i∈I
Experiments
In all experiments, our algorithm was run using second-generation curvelet tight frame along with the following set of parameters: ∀t, μt ≡ 1, βt = 0.24, γ = 10 and NDR = 50. The initial threshold T was set to 2 ψ1 (K). For comparison purposes, some very recent multiplicative noise removal algorithms from the literature are considered: the AA algorithm [2] minimizing the criterion in (10), and the Stein-block denoising method [8] in the curvelet domain, applied on the log transformed image. The latter is a sophisticated shrinkage-based denoiser that thresholds the coefficients by blocks rather than individually, and has been shown to be nearly minimax over a large class of images in presence of various additive bounded noises. We also tried the L2-TV method where the restored log-image u ˆ minimizes (11) and the denoised image Sˆ involves the bias correction (34). Thanks to the bias correction, it can be seen as an improved version of the first method proposed in the recent Report [26, § 4.1]. For fair comparison, the hyperparameters for all competitors were tweaked to reach their best level of performance on each noisy realization.
Multiplicative Noise Cleaning
293
The denoising algorithms were tested on two images: Lena and Boat, all of size 256×256 and gray-scale in the range [1, 256]. For each image, a noisy observation is generated by multiplying the original image by a realization of noise according to (2)-(3) for K = 10. The running time of our denoising method is 1 minute 3 seconds for 50 iterations on Intel 2.5 GHz Core Duo. The denoising performance of any algorithm is measured in terms of peak signal-to-noise ratio (PSNR) and mean absolute-deviation MAE, namely √ PSNR = 20 log10 N S0 ∞ / Sˆ − S0 dB and MAE = Sˆ − S0 /N . 2
1
The results are depicted in Figs. 3 and 4. Note that the AA algorithm tends to over-regularize the solution. Our denoiser clearly outperforms its competitors.
References 1. Achim, A., Tsakalides, P., Bezerianos, A.: Sar image denoising via bayesian wavelet shrinkage based on heavy-tailed modeling. IEEE Trans. Geosci. Remote Sens. 41(8), 1773–1784 (2003) 2. Aubert, G., Aujol, J.-F.: A variational approach to remove multiplicative noise. J. on Applied Mathematics 68(4), 925–946 (2008) 3. Aubert, G., Kornprobst, P.: Mathematical problems in image processing, 2nd edn. Springer, Berlin (2006) 4. Aujol, J.-F.: Some algorithms for total variation based image restoration. Report CLMA 2008-05 (2008) 5. Candès, E.J., Guo, F.: New multiscale transforms, minimum total variation synthesis. Applications to edge-preserving image reconstruction. Signal Processing 82 (2002) 6. Chambolle, A.: An algorithm for total variation minimization and application. J. of Mathematical Imaging and Vision 20(1) (2004) 7. Chambolle, A.: Total variation minimization and a class of binary MRF models. In: Rangarajan, A., Vemuri, B.C., Yuille, A.L. (eds.) EMMCVPR 2005. LNCS, vol. 3757, pp. 136–152. Springer, Heidelberg (2005) 8. Chesneau, C., Fadili, J., Starck, J.-L.: Stein block thresholding for image denoising. Technical report (2008) 9. Coifman, R.R., Sowa, A.: Combining the calculus of variations and wavelets for image enhancement. Applied and Computational Harmonic Analysis 9 (2000) 10. Combettes, P.L.: Solving monotone inclusions via compositions of nonexpansive averaged operators. Optimization 53(5) (2004) 11. Combettes, P.L., Pesquet, J.-C.: A Douglas-Rachford splittting approach to nonsmooth convex variational signal recovery. IEEE J. of Selected Topics in Signal Processing 1(4), 564–574 (2007) 12. Donoho, D.L., Johnstone, I.M.: Ideal spatial adaptation by wavelet shrinkage. Biometrika 81(3), 425–455 (1994) 13. Durand, S., Nikolova, M.: Denoising of frame coefficients using l1 data-fidelity term and edge-preserving regularization. SIAM J. on Multiscale Modeling and Simulation 6(2), 547–576 (2007) 14. Durand, S., Froment, J.: Reconstruction of wavelet coefficients using total variation minimization. SIAM J. on Scientific Computing 24(5), 1754–1767 (2003)
294
S. Durand, J. Fadili, and M. Nikolova
15. Durand, S., Fadili, J., Nikolova, M.: Multiplicative noise removal using L1 fidelity on frame coefficients. Report CMLA n.2008-40 (2008) 16. Fukuda, S., Hirosawa, H.: Suppression of speckle in synthetic aperture radar images using wavelet. Int. J. Remote Sens. 19(3), 507–519 (1998) 17. Krissian, K., Westin, C.-F., Kikinis, R., Vosburgh, K.G.: Oriented speckle reducing anisotropic diffusion. IEEE Trans. on Image Processing 16(5), 1412–1424 (2007) 18. Ma, J., Plonka, G.: Combined Curvelet Shrinkage and Nonlinear Anisotropic Diffusion. IEEE Trans. on Image Processing 16(9), 2198–2206 (2007) 19. Malgouyres, F.: Mathematical analysis of a model which combines total variation and wavelet for image restoration. J. of information processes 2(1), 1–10 (2002) 20. Moreau, J.-J.: Fonctions convexes duales et points proximaux dans un espace hilbertien. CRAS Sér. A Math 21. Nikolova, M.: Minimizers of cost-functions involving nonsmooth data-fidelity terms. Application to the processing of outliers. SIAM J. on Numerical Analysis 40(3), 965–994 (2002) 22. Nikolova, M.: Weakly constrained minimization. Application to the estimation of images and signals involving constant regions. J. of Mathematical Imaging and Vision 21(2), 155–175 (2004) 23. Pizurica, A., Wink, A.M., Vansteenkiste, E., Philips, W., Roerdink, J.B.T.M.: A review of wavelet denoising in mri and ultrasound brain imaging. Current Medical Imaging Reviews 2(2), 247–260 (2006) 24. Rudin, L., Lions, P.-L., Osher, S.: Multiplicative denoising and deblurring: Theory and algorithms. In: Osher, S., Paragios, N. (eds.), pp. 103–119. Springer, Heidelberg (2003) 25. Rudin, L., Osher, S., Fatemi, C.: Nonlinear total variation based noise removal algorithm. Physica 60D, 259–268 (1992) 26. Shi, J., Osher, S.: A nonlinear inverse scale space method for a convex mutiplicative noise model. In: UCLA 2007 (2007) 27. Ulaby, F., Dobson, M.C.: Handbook of Radar Scattering Statistics for Terrain. Artech House, Norwood (1989) 28. Walessa, M., Datcu, M.: Model-based despeckling and information extraction from sar images. IEEE Trans. Geosci. Remote Sens. 38(9), 2258–2269 (2000) 29. Welk, M., Steidl, G., Weickert, J.: Locally analytic schemes: A link between diffusion filtering and wavelets shrinkage. Applied and Computational Harmonic Analysis 24, 195–224 (2008) 30. Xie, H., Pierce, L.E., Ulaby, F.T.: SAR speckle reduction using wavelet denoising and markov random field modeling. IEEE Trans. Geosci. Remote Sensing 40(10), 2196–2212 (2002) 31. Yu, Y., Acton, S.T.: Speckle reducing anisotropic diffusion. IEEE Trans. on Image Processing 11(11), 1260–1270 (2002)
Projected Gradient Based Color Image Decomposition Vincent Duval, Jean-François Aujol, and Luminita Vese 1
Institut TELECOM, TELECOM ParisTech, CNRS UMR 5141 [email protected] 2 CMLA, ENS Cachan, CNRS, UniverSud [email protected] 3 UCLA, Mathematics Department [email protected]
Abstract. This work deals with color image processing, with a focus on color image decomposition. The problem of image decomposition consists in splitting an original image f into two components u and v = f − u. u contains the geometric information of the original image, while v is made of the oscillating patterns of f , such as textures. We propose a numerical scheme based on a projected gradient algorithm to compute the solution of various decomposition models for color images or vectorvalued images. A direct convergence proof of the scheme is provided, and some analysis on color texture modeling is given. Keywords: Color image decomposition, projected gradient algorithm, color texture modeling.
1
Introduction
Total variation regularization was introduced almost 20 years ago for image restoration in the seminal work by Rudin et al [1]. It has now grown as a popular and widely used tool in image processing (see [2, 3] and references therein for instance). If we denote by f the original image, the problem we are interested in consists in minimizing energies of the type: |Du| + μf − ukT . (1) Here |Du| is the total variation of u; we simply have |Du| = |∇u| dx in the case when u is regular. .T stands for a norm which favors the noise and/or the textures of the original image f (in the sense that it is small for such features) and k is a positive exponent. The most basic choice for .T is the L2 norm, and k = 2. However, inspired from the book by Y. Meyer [4], and also motivated by work of Mumford-Gidas [5], other spaces have been considered for modeling natural images and oscillating patterns such as textures or noise. [4] was the inspiration source of many works, e.g. to name a few [6, 7, 8, 9, 10, 11, 12, 13, 14]. Image decomposition consists in splitting an original image f into two components, u X.-C. Tai et al. (Eds.): SSVM 2009, LNCS 5567, pp. 295–306, 2009. c Springer-Verlag Berlin Heidelberg 2009
296
V. Duval, J.-F. Aujol, and L. Vese
and v = f − u. u contains the geometrical component of the original image (it can be seen as a sketch of the original image), while v is made of the oscillatory component (when the original image f is noise free, v is the texture component). In this work, we are concerned with color image processing. While some authors deal with color images using a Riemannian framework, like G. Sapiro and D. L. Ringach [15] or N. Sochen et al [16], others combine a functional analysis viewpoint with the Chromaticity-Brightness representation [17]. The model we use is more basic: it is the same as the one used in [18] (and related with [19]). Its advantage is to have a rich functional analysis interpretation. Note that in [20], the authors also propose a cartoon + texture color decomposition and denoising model inspired from Y. Meyer [4], using the vectorial versions of total variation and approximations of the space G(Ω) for textures (to be defined later); unlike the work presented here, they use Euler-Lagrange equations and a gradient descent scheme for the minimization. Here, we give some insight into the definition of a texture space for color images. In [21], a TV-Hilbert model was proposed for image restoration and/or decomposition: |Du| + μf − u2H (2) where .H stands for the norm of some Hilbert space H. This is a particular case of problem (1). Thanks to the Hilbert structure of H, different methods can be used to minimize (2), such as a projection algorithm [21]. We extend (2) to the case of color images. From a numerical point of view, (1) is not straightforward to minimize. Depending on the choice of .T , the minimization of (1) can be quite challenging. Even in the simplestcase when .T is the L2 norm and k = 2, handling the total variation term |Du| needs to be done with care. The most classical approach consists in writing the associated Euler-Lagrange equation to problem (1). In [1], a fixed step gradient descent scheme is used to compute the solution. This method has on the one hand the advantage of being very easy to implement, and on the other hand the disadvantage of being quite slow. To improve the convergence speed, quasi-Newton methods have been proposed [22]. Duality based schemes have also drawn a lot of attention to solve (1): first by Chan, Golub and Mulet in [23], later by A. Chambolle in [24] with a projection algorithm. This projection algorithm has recently been extended to the case of color images in [18]. It has been shown that graph cuts based algorithms could also be used [25,26]. Let us notice that it is shown in [27,28] that Nesterov’s schemes provide fast algorithms for minimizing (1). Another variant of Chambolle projection algorithm [24] is to use a projected gradient algorithm [25, 28, 29]. Here we have decided to use this approach which has both advantages of being easy to implement and of being quite efficient. The plan of the paper is the following. In Sect. 2, we define and provide some analysis about the spaces we consider in the paper. In Sect. 3, we extend the TVHilbert model originally introduced in [21] to the case of color images. In Sect. 4, we present a projected gradient algorithm to compute a minimizer of problem (2). This projected gradient algorithm has first been proposed by A. Chambolle
Projected Gradient Based Color Image Decomposition
297
in [25] for total variation regularization. A proof of convergence was given in [28] relying on optimization results by Bermudez and Moreno [30]. We derive here a simple and direct proof of convergence. In Sect. 5, we apply this scheme to solve various classical denoising and decomposition problems. We illustrate our approach with many numerical examples.
2
Definitions and Properties of the Considered Color Spaces
In this section, we introduce some notations, and we provide some analysis of the functional analysis spaces we consider to model color textures. 2.1
Introduction
Let Ω be a Lipschitz convex bounded open set in R2 . We model color images as RM -valued functions defined on Ω. The inner product in L2 (Ω, RM ) is denoted M as u, vL2 (Ω,RM ) = Ω i=1 ui vi . For a vector ξ ∈ RM , we define the norms: M M |ξ|1 = |ξi |, |ξ|2 = ξi2 , |ξ|∞ = max |ξi | . i=1
i=1
i=1...M
We say that a function f ∈ L1 (Ω, RM ) has bounded variation if the following quantity is finite: |f |T V = supξ∈B f , div ξL2 (Ω,RM ) , with B = {ξ ∈ Cc1 (Ω, R2×M )/∀x ∈ Ω, |ξ(x)|2 ≤ 1} .
(3)
This quantity is called the total variation. For more information on its properties, we refer the reader to [3]. The set of functions with bounded variation is a vector space classically denoted by BV (Ω, RM ). For f smooth enough, the total M 2 variation of f is |f |T V = Ω i=1 |∇fi | dx. Other choices of sets B are possible (see [18] for a discussion), which are mathematically equivalent and define the same BV space. But in practice, in image processing, it is crucial to have a coupling between the channels as in (3) in order to avoid visual artifacts. 2.2
The Color G(Ω) Space
The G(R2 ) space was introduced by Y. Meyer in [4] to model textures in grayscale images. For the generalization to color images, we will adopt the framework of [8]; the color space G(Ω) is also used in [20], as a generalization of [6] to color image decomposition and color image denoising. Definition 1. The space G(Ω) is defined by: G(Ω) = {v ∈ L2 (Ω, RM )/ ∃ξ ∈ L∞ (Ω, (R2 )M ), ∀i = 1, . . . , M, vi = div ξi and ξi · N = 0 on ∂Ω}
298
V. Duval, J.-F. Aujol, and L. Vese
(where ξi · N refers to the normal trace of ξi over ∂Ω). One can endow it with the norm: vG = inf{ξ∞ , ∀i = 1, . . . , M, vi = div ξi , ξi · N = 0 on ∂Ω} M 2 with ξ∞ = ess sup i=1 |ξi | . The following result was proved in [9] for grayscale images: it characterizes G(Ω). Working component by component, it is straightforward to extend it to color images (see [31]). Proposition 1 G(Ω) =
v ∈ L2 (Ω, RM )/ v=0 . Ω
Remark 1. The topology induced by the G-norm on G(Ω) is coarser than the one induced by the L2 norm. Let us consider, for m ∈ N∗ , the sequence ∀k = (k) 1 . . . M, fm (x, y) = cos mx + cos my defined on (−π, π)M . The vector field 1 1 ξ (k) = ( m sin(mx), m sin(my)) satisfies the boundary condition, and its diver√
2M and limm→+∞ f m G = gence is equal to f m . As a consequence f m G ≤ m 0. Yet, it is easy to see that f m 2L2 (Ω,RM ) = 4M π 2 . The sequence f m converges to 0 for the topology induced by the G-norm, but not for the one induced by the L2 norm.
More generally, oscillating patterns with zero mean have a small G norm (see [4] for more details).
3 3.1
Color TV-Hilbert Model: Presentation and Mathematical Analysis Presentation
The TV-Hilbert framework was introduced for grayscale images by J.-F. Aujol and G. Gilboa in [21] as a way to approximate the BV-G model. They prove that one can extend Chambolle’s algorithm to this model. In this section we show that this is still true for color images. We are interested in solving the following problem: 1 inf |u|T V + f − u2H (4) u 2λ where H is the space of zero-mean functions of L2 (Ω, RM ), regarded as a Hilbert space endowed with the following norm : v2H = v, KvL2 (Ω,RM ) . Here we assume that K : H → L2 (Ω, RM ) is a symmetric positive definite, bounded linear operator (for the topology induced by the L2 (Ω, RM ) norm on H) and K −1 is bounded on Im(K).
Projected Gradient Based Color Image Decomposition
299
Example 1 (The Rudin-Osher-Fatemi model). It was proposed in [1] for grayscale images, then extended to color images using different methods (e.g. [15] or [19]). In [18], the authors use another kind of color total variation, which is the one we use in this paper. The idea is to minimize the functional: |u|T V +
1 f − u2L2 (Ω,RM ) . 2λ
(5)
Without loss of generality, we can assume that f has zero mean. Then this model becomes a particular case of the TV-Hilbert model with K = Id. Example 2 (The OSV model). In [7], S. Osher, A. Solé and L. Vese propose to model textures by the H −1 space. In order to generalize this model, we introduce the following functional : 1 inf |u|T V + |∇Δ−1 (f − u)|2 (6) u 2λ Ω ⎛ −1 ⎞ ⎛ ⎞ Δ v1 ∇ρ1 M ⎜ ⎟ ⎜ . ⎟ .. 2 where Δ−1 v = ⎝ ⎠, ∇ρ = ⎝ .. ⎠, |∇ρ|2 = j=1 |∇ρj | and .
Δ−1 vM ∇ρM M −1 2 −1 i |∇Δ (f −u)| = Ω i=1 |∇Δ (f −ui )|2 = f −u, −Δ−1 (f −u)L2 (Ω,RM ) . Ω For K = −Δ−1 , the Osher-Solé-Vese problem is a particular case of the TV-Hilbert framework. We also refer to L. Lieu, L. Vese [14] for more general (BV, H −s ) models, as particular cases of the TV-Hilbert formulation.
3.2
Mathematical Study
For f ∈ L2 (Ω, RM ), the existence and uniqueness of the minimizer u of (4) can be proved using standard methods (see [3]). Now, let us introduce the notation v = f − u, when u is a minimizer of (4). Following Y. Meyer’s steps, one can extend the result proposed in [4] for grayscale images (see [31] for a detailed proof): Theorem 1 (Characterization of minimizers) Let f ∈ L2 (Ω, RM ). (i) If KfG ≤ λ then the solution of the TV-Hilbert problem is given by (u, v) = (0, f ). (ii) If Kf G > λ then the solution (u, v) is characterized by: KvG = λ and u, KvL2 (Ω,RM ) = λ|u|T V . For λ > 0, the set Gλ = {v ∈ L2 (Ω, RM ), vG ≤ λ} is a closed convex set, as well as K −1 Gλ . The orthogonal projection of this set is well-defined and we can notice that Theorem 1 reformulates:
H v = PK −1 G (f ) λ . u= f −v
300
V. Duval, J.-F. Aujol, and L. Vese
That is, v is the orthogonal projection of f on the set K −1 Gλ . Therefore, the problem is equivalent to its dual formulation, with v = λK −1 div p : inf λK −1 div p − f 2H .
|p|≤1
4
(7)
Projected Gradient Algorithm
We present here a projection algorithm for solving this dual formulation, inspired from [24, 18], and we provide a complete proof of convergence of this scheme. 4.1
Discrete Setting
From now on, we will work in the discrete case, using the following convention. A grayscale image is a matrix of size N × N . We write X = RN ×N the space of grayscale images. Their gradients belong to the space Y = X × X. The L2 inner product is u, vX = 1≤i,j≤N ui,j vi,j . For the gradient and divergence operators on grayscale images, we use the same discretizations as in [24]. A color image is an element of X M and its gradient belongs to Y M . The gradient and the divergence are defined component by component, so that the color divergence is still the opposite of the adjoint of the color gradient. Notice that in this framework, we have ∇2 = div2 = 8 (see [24]). 4.2
Projected Gradient
It was recently noticed ([25], [28]), that problem (7) for grayscale images could be solved using a projected gradient descent. This is the algorithm we decided to extend to the case of color images. Let B = {v ∈ Y M , ∀ 1 ≤ i, j ≤ N, |vi,j |2 ≤ 1} be the discrete version of our set of test-functions. Theorthogonal projection on x1 x2 B is easily computed: PB (x) = max{1,|x| . The projected gradient , 2 } max{1,|x|2 } descent scheme is defined by : pm+1 = PB pm + τ ∇(K −1 div pm − f /λ (8) which amounts to: pm+1 = i,j
pm + τ ∇(K −1 div pm − fλ )i,j . i,j −1 div pm − f ) | max 1, |pm i,j 2 i,j + τ ∇(K λ
(9)
Since the functional is not elliptic, the standard proof of convergence of the projected gradient algorithm (see [32] for instance) needs to be adapted to this particular case. Proposition 2. If 0 < τ < 4 K1−1 , then algorithm (9) converges. More precisely, there exists p ˜ ∈ B such that : lim (K −1 div pm ) = K −1 div p ˜
m→∞
˜ − f 2H = inf p∈B λK −1 div p − f 2H . and λK −1 div p
Projected Gradient Based Color Image Decomposition
301
Proof. We only give here a sketch of the proof. Let us first notice that p is a minimizer iff p ∈ B and ∀q ∈ B, ∀τ > 0, q − p, p − (p + τ ∇(K−1 div p − f /λ))L2 ≥ 0. Or equivalently: p = PB p + τ (∇(K −1 div p − f /λ) , where PB is the orthogonal projection on B with respect to the L2 inner product. Let p be such a minimizer. • Now let us consider a sequence defined by (8), and write A = −∇K −1 div . We have : pk+1 − p2 ≤ (I − τ A)(p − pk )2 since PB is 1-Lipschitz [32]. Provided I − τ A ≤ 1, we can deduce : pk+1 − p ≤ pk − p
(10)
and the sequence (pk − p) is convergent. • A is a symmetric positive semi-definite operator. By writing E = ker A and ⊥
F = ImA, we have Y M = E ⊕ F , and we can decompose any q ∈ Y M as the sum of two orthogonal components q E ∈ E and q F ∈ F . Notice that by injectivity of K −1 , E is actually equal to the kernel of the divergence operator. Let μ1 = 0 < μ2 ≤ . . . ≤ μa be the ordered eigenvalues of A. I − τ A = max(|1 − τ μ1 |, |1 − τ μa |) = 1 for 0 ≤ τ ≤
2 . μa
We can restrict I − τ A to F and then define : g(τ ) = (I − τ A)|F < 1 for 0 < τ < μ2a • Now we assume that 0 < τ < μ2a . Therefore, inequality (10) is true and the sequence (pk ) is bounded, and so is the sequence (K −1 div pk ). We are going to prove that the sequence (K −1 div pk ) has a unique cluster point. Let (K −1 div pϕ(k) ) be a convergent subsequence. By extraction, one ˜ its limit. Passcan assume that pϕ(k) is convergent too, and denote by p ϕ(k)+1 ing to the limit in (8), the sequence (p ) is convergent towards p ˆ = PB p ˜ + τ ∇(K −1 div p ˜ − f /λ) . Using (10), we also notice that ˜ p − p = ˆ p − p. As a consequence: ˜ p − p2 = PB p ˜ − f /λ) − PB p + τ ∇(K −1 div p − f /λ) 2 ˜ + τ ∇(K −1 div p ≤ (I − τ A)(˜ p − p)2 = (˜ p − p)E 2 + g(τ )2 (˜ p − p)F 2 < ˜ p − p2 if (˜ p − p)F =0 .
Of course, this last inequality cannot hold, which means that (˜ p −p)F = 0. Hence (˜ p − p) ∈ E = ker A and K −1 div p ˜ = K −1 div p: the sequence (K −1 div pk ) is convergent. • Since div 2 = ∇2 = 8 (see [24]), we conclude by noticing that μa ≤ 8K −1 .
Since we are only interested in v = λK −1 div p, Proposition (2) justifies the validity of algorithm (8). We can actually prove that the sequence (pm ) defined by (8) converges (see [31] Corollary 4.1).
302
5
V. Duval, J.-F. Aujol, and L. Vese
Applications to Color Image Denoising and Decomposition
In this last section, we apply the projected gradient algorithm to solve various color image problems. 5.1
TV-Hilbert Model
The Color ROF Model. As an application of (9), we use the following scheme for the ROF model (5): pm+1 = i,j
pm + τ ∇(div pm − fλ )i,j . i,j f m max 1, |pm i,j + τ ∇(div p − λ )i,j |2
(11)
The Color OSV Model: As for the OSV model (6), we use: = pm+1 i,j 5.2
pm − τ ∇(Δdiv pm + fλ )i,j . i,j m + f) | max 1, |pm − τ ∇(Δdiv p i,j 2 i,j λ
(12)
The Color A2BC Algorithm
Following Y. Meyer [4], one can use the G(Ω) space to model textures, and try to solve the problem: inf u (|u|BV + αf − uG ). In [8], the authors approximate this problem by minimizing the following functional: 1 f − u − v2L2 (Ω) + χGμ (v n ) 2λ
0 if v ∈ Gμ with χGμ (v) = . +∞ otherwise
Fμ,λ (u, v) = |u|BV +
(13)
Following [8,17,18], it is straightforward to extend the A2BC algorithm using the projection algorithm. We start by initializing with u0 = v 0 = 0, and then compute iteratively until convergence1: v n+1 = PGμ (f − un ) 5.3
and
un+1 = f − v n+1 − PGλ (f − v n+1 ) .
The Color TV-L1 Model
The TV-L1 model is very popular for grayscale images. It benefits from having both good theoretical properties (it is a morphological filter) and fast algorithms (see [26]). In order to extend it to color images, we consider the problem: M 2 inf u |u|T V + λf − u1 with the notation u1 = Ω l=1 |ul | . As for the A2BC algorithm, we are led to consider the approximation, for α > 0: inf |u|BV +
u,v 1
1 f − u − v22 + λv1 . 2α
The proof of convergence of this algorithm is the same as the one in [8].
Projected Gradient Based Color Image Decomposition
303
Fig. 1. From left to right: original and noisy images (WG, PSNR = 57.3 dB), denoised with color ROF (λ = 25, PSNR= 74.2 dB) and with color OSV (λ = 25, PSNR= 74.1 dB)
Fig. 2. Cartoon-texture decomposition using color A2BC algorithm (upper row) and color TVL1 (lower row). On top, the original image.
In order to generalize the TV-L1 algorithm proposed by Aujol et al ( [33]), we aim at solving the alternate minimization problem: 1 1 f − u − v22 f − u − v22 + λv1 . inf |u|BV + and inf u v 2α 2α
304
V. Duval, J.-F. Aujol, and L. Vese
Fig. 3. From left to right: original and noisy images (using salt and pepper noise, PSNR= 34.6 dB), denoised with color TVL1 (PSNR= 67.5 dB) and noise part
The first problem is a Rudin-Osher-Fatemi problem. Scheme (9) with K = Id is well adapted for solving it. The second one can be solved by a "vectorial soft thresholding": Proposition 3. The solution of the second problem is given by: v(x) = V Tαλ (f (x) − u(x)) =
f (x) − u(x) max (|f (x) − u(x)|2 − αλ, 0) a.e. |f (x) − u(x)|2
The proof of this last result is given in [31]. Therefore, we propose to generalize the TV-L1 algorithm by initializing with u0 = v 0 = 0, then computing iteratively until convergence (the proof of convergence is the same as the one in [33]): v n+1 = V Tαλ (f − un ) 5.4
and
un+1 = f − v n+1 − PGα (f − v n+1 ).
Numerical Experiments
Figure 1 displays denoising results using ROF (5) and OSV (6) models. The images look very similar but since the OSV model penalizes much more the highest frequencies than the ROF model [33], the denoised image still shows the lowest frequencies of the noise. The convergence speed in the ROF model is roughly the same as with the Bresson-Chan algorithm (see [18], [31]). Figure 2 displays a cartoon-texture decomposition experiment using different kinds of texture. The algorithms used were A2BC and TVL1. Both results look good. On Figure 3, a denoising experiment was performed using salt-and-pepper noise. The denoised picture looks quite good and surprisingly better than the original image! This is because the picture we used had some compression artifacts that the algorithm removed.
Acknowledgements This work has been supported by the French "Agence Nationale de la Recherche" (ANR), under grant FREEDOM (ANR07-JCJC-0048-01), "Films, REstauration Et DOnnées Manquantes", and by the National Science Foundation under Grants DMS-0312222 and DMS-0714945. Part of this work was done while the first author was visiting the Department of Mathematics, UCLA.
Projected Gradient Based Color Image Decomposition
305
References 1. Rudin, L., Osher, S., Fatemi, E.: Non linear total variation based noise removal algorithms. Physica D 60, 259–268 (1992) 2. Chan, T., Shen, J.: Image processing and analysis - Variational, PDE, wavelet, and stochastic methods. SIAM Publisher, Philadelphia (2005) 3. Aubert, G., Kornprobst, P.: Mathematical Problems in Image Processing: Partial Differential Equations and the Calculus of Variations. Applied Mathematical Sciences, vol. 147. Springer, Heidelberg (2001) 4. Meyer, Y.: Oscillating patterns in image processing and nonlinear evolution equations. In: The fifteenth Dean Jacqueline B. Lewis memorial lectures. University Lecture Series, vol. 22. American Mathematical Society, Providence, RI (2001) 5. Mumford, D., Gidas, B.: Stochastic models for generic images. Quarterly of Applied Mathematics LIV(1) (2001) 6. Vese, L., Osher, S.J.: Modeling textures with total variation minimization and oscillating patterns in image processing. Journal of Scientific Computing 19(1-3), 553–572 (2003) 7. Osher, S., Solé, A., Vese, L.: Image decomposition and restoration using total variation minimization and the H −1 norm. SIAM Journal on Multiscale Modeling and Simulation 1(3), 349–370 (2003) 8. Aujol, J.F., Aubert, G., Blanc-Féraud, L., Chambolle, A.: Image decomposition into a bounded variation component and an oscillating component. Journal of Mathematical Imaging and Vision 22(1), 71–88 (2005) 9. Aubert, G., Aujol, J.: Modeling very oscillating signals. Application to image processing. Applied Mathematics and Optimization 51(2), 163–182 (2005) 10. Aujol, J.F., Chambolle, A.: Dual norms and image decomposition models. International Journal on Computer Vision 63(1), 85–104 (2005) 11. Yin, W., Goldfarb, D., Osher, S.: A comparison of three total variation based texture extraction models. Journal of Visual Communication and Image Representation 18(3), 240–252 (2007) 12. Garnett, J., Jones, P., Le, T., Vese, L.: Modeling oscillatory components with the homogeneous spaces BM O−α and W −α,p . Pure and Applied Mathematics Quarterly (to appear) 13. Le, T., Vese, L.: Image decomposition using total variation and div (BMO). Multiscale Modeling and Simulation, SIAM Interdisciplinary Journal 4(2), 390–423 (2005) 14. Lieu, L., Vese, L.: Image restoration and decomposition via bounded total variation and negative hilbert-sobolev spaces. Applied Mathematics & Optimization 58, 167– 193 (2008) 15. Sapiro, G., Ringach, D.L.: Anisotropic diffusion of multivalued images with applications to color filtering. IEEE Transactions on Image Processing 5(11), 1582–1586 (1996) 16. Sochen, N., Kimmel, R., Malladi, R.: A general framework for low level vision. IEEE Transactions on Image Processing 7(3), 310–318 (1998) 17. Aujol, J.F., Kang, S.H.: Color image decomposition and restoration. Journal of Visual Communication and Image Representation 17(4), 916–928 (2006) 18. Bresson, X., Chan, T.: Fast minimization of the vectorial total variation norm and applications to color image processing. Inverse Problems and Imaging (IPI) (accepted) (2007)
306
V. Duval, J.-F. Aujol, and L. Vese
19. Blomgren, P., Chan, T.: Color TV: Total variation methods for restoration of vector valued images. IEEE Transactions on Image Processing 7(3), 304–309 (1998) 20. Vese, L., Osher, S.: Color texture modeling and color image decomposition in a variational-PDE approach. In: Proceedings of the Eighth International Symposium on Symbolic and Numeric Algorithms for Scientific Computing (SYNASC 2006), pp. 103–110. IEEE, Los Alamitos (2006) 21. Aujol, J., Gilboa, G.: Constrained and SNR-based solutions for TV-Hilbert space image denoising. Journal of Mathematical Imaging and Vision 26(1-2), 217–237 (2006) 22. Vogel, C.: Computational Methods for Inverse Problems. Frontiers in Applied Mathematics, vol. 23. SIAM, Philadelphia (2002) 23. Chan, T., Golub, G., Mulet, P.: A nonlinear primal-dual method for total variationbased image restoration. SIAM Journal on Scientific Computing 20(6), 1964–1977 (1999) 24. Chambolle, A.: An algorithm for total variation minimization and its applications. JMIV 20, 89–97 (2004) 25. Chambolle, A.: Total variation minimization and a class of binary MRF models. In: Rangarajan, A., Vemuri, B.C., Yuille, A.L. (eds.) EMMCVPR 2005. LNCS, vol. 3757, pp. 136–152. Springer, Heidelberg (2005) 26. Darbon, J., Sigelle, M.: Image restoration with discrete constrained total variation part I: Fast and exact optimization. Journal of Mathematical Imaging and Vision 26(3), 277–291 (2006) 27. Weiss, P., Aubert, G., Blanc-Féraud, L.: Efficient schemes for total variation minimization under constraints in image processing. SIAM Journal on Scientific Computing (to appear) (2007) 28. Aujol, J.: Some algorithms for total variation based image restoration. CMLA Preprint 2008-05 (2008), http://hal.archives-ouvertes.fr/hal-00260494/en/ 29. Zhu, M., Wright, S., Chan, T.: Duality-based algorithms for total variation image restoration, UCLA CAM Report 08-33 (May 2008) 30. Bermudez, A., Moreno, C.: Duality methods for solving variational inequalities. Comp. and Maths. with Appls. 7(1), 43–58 (1981) 31. Duval, V., Aujol, J.F., Vese, L.: A projected gradient algorithm for color image decomposition. Technical report, UCLA, CAM Report 08-40 (2008) 32. Ciarlet, P.G.: Introduction á l’Analyse Numérique Matricielle et á l’Optimisation. Dunod (1998) 33. Aujol, J., Gilboa, G., Chan, T., Osher, S.: Structure-texture image decomposition modeling, algorithms, and parameter selection. International Journal of Computer Vision 67(1), 111–136 (2006)
A Dual Formulation of the TV-Stokes Algorithm for Image Denoising Christoffer A. Elo1 , Alexander Malyshev1 , and Talal Rahman2 1
2
Department of Mathematics, University of Bergen Johannes Bruns gate 12, 5007 Bergen, Norway [email protected], [email protected] Bergen University College, Faculty of Engineering, Nygårdsgaten 112, 5020 Bergen [email protected]
Abstract. We propose a fast algorithm for image denoising, which is based on a dual formulation of a recent denoising model involving the total variation minimization of the tangential vector field under the incompressibility condition stating that the tangential vector field should be divergence free. The model turns noisy images into smooth and visually pleasant ones and preserves the edges quite well. While the original TV-Stokes algorithm, based on the primal formulation, is extremely slow, our new dual algorithm drastically improves the computational speed and possesses the same quality of denoising. Numerical experiments are provided to demonstrate practical efficiency of our algorithm.
1
Introduction
We suppose that the observed image d0 (x, y), (x, y) ∈ Ω ⊂ R2 , is an original image d(x, y) perturbed by an additive noise η, d0 = d + η.
(1)
The problem of recovering the image d from the noisy image d0 is an inverse problem that is often solved by variational methods using the total variation (TV) minimization. The corresponding Euler equation, which is a set of nonlinear partial differential equations, is typically solved by applying a gradient-descent method to a finite difference approximation of these equations. A classical total variation denoising model is the primal formulation due to Rudin, Osher and Fatemi [1] (the ROF model): λ d − d0 2L2 . (2) 2 The parameter λ > 0 can be chosen, e.g., to approximately fulfill the condition d − d0 L2 ≤ σ, where σ is an estimate of ηL2 . The Euler equation −div (∇d/|∇d|) + λ(d − d0 ) = 0 is usually replaced by a regularized one, ∇d + λ(d − d0 ) = 0, (3) −div |∇d|β min ∇dL1 + d
X.-C. Tai et al. (Eds.): SSVM 2009, LNCS 5567, pp. 307–318, 2009. c Springer-Verlag Berlin Heidelberg 2009
308
C.A. Elo, A. Malyshev, and T. Rahman
where |∇d|β =
|∇d|2 + β 2 is a necessary regularization, since images contain
flat areas where |∇d| = d2x + d2y ≈ 0. When solving (3) numerically, an explicit time marching scheme with an artificial time variable, t, is typically used. However, such an algorithm is rather slow due to severe restrictions requiring small time steps for the convergence. It is well known that the ROF model suffers from the so called staircase effect, which is a disadvantage when denoising images with affine regions. To overcome this defect, we motivate for a two-step approach, where the fourth-order model, studied in [2, 3, 4], is decoupled into two second-order problems. Such methods are known to overcome the staircase effect, but tend to have computational difficulties due to very large conditioning. The authors of [5, 6] used the same two-step approach as in [7], but adopting ideas from [8, 9] they proposed to preserve the divergence-free condition on the tangential vector field. Recall that the tangential vector field τ is orthogonal to the normal (gradient) vector field n of the image d: n = ∇d = (dx , dy ),
τ = ∇⊥ d = (−dy , dx )T .
(4)
Hence div τ = 0. The first step of the TV-Stokes algorithm smoothes the tangential vector field τ0 = ∇⊥ d0 for a given noisy image d0 and then solve the minimization problem 1 min ∇τ L1 + τ − τ0 2L2 τ 2δ
subject to div τ = 0,
(5)
where δ > 0 is some carefully chosen parameter. Once a smoothed tangential vector field τ is obtained, the second step reconstructs the image d by fitting it to the normal vector field by solving the minimization problem min ∇dL1 d
n − ∇d, |n| L2
subject to d − d0 L2 = σ,
(6)
where σ is an estimate of ηL2 . In [5] the minimization problems (5) and (6) are numerically solved by means of a time marching explicit scheme, while existence and uniqueness are proven for the Modified TV-Stokes in [6]. The TV-Stokes approach resulted in an algorithm which does not suffer from the staircase effect, preserves the edges, and the denoised images look visually pleasant. However, the TV-Stokes algorithm from [5] is extremely slow convergent and therefore practically unusable as demonstrated in the last section of the present paper. We adopt the TV-Stokes denoising model but reduce the above presented primal formulation to the so called dual formulation, which is then numerically solved by a variant of fast Chambolle’s iteration [10]. The reduction exploits the orthogonal projector ΠK onto the subspace K = {τ : div τ = 0} for elimination of the divergence-free constraint.
A Dual Formulation of the TV-Stokes Algorithm for Image Denoising
2
309
The TV-Stokes Denoising Algorithm in Dual Formulation
To overcome difficulties with non-differentiability in the primal formulation, Carter [11], Chambolle [10] and Chan, Golub and Mulet [12] have proposed dual formulations of the ROF model, where a dual variable p = (p1 (x, y), p2 (x, y)) is used to express the total variation: ∇dL1 = max {(d, divp)L2 : |pj (x, y)| ≤ 1 ∀(x, y) ∈ Ω, j = 1, 2} . p
(7)
For instance, a variant of dual formulation from [10] consists in minimization of the distance divp − λd0 L2 . In [10] Chambolle also proposed a fast iteration for solving this minimization problem that produces a denoised image after a few steps only. Below we show how to reduce the TV-Stokes model to a dual formulation. 2.1
Step 1
To derive a dual formulation of the first step we take advantage of the following analog of (7) for the total variation of the tangential vector field τ = (τ1 , τ2 )T : ∇τ L1 = max {(τ, divp)L2 : |pi (x, y)| ≤ 1 ∀(x, y) ∈ Ω, i = 1, 2} , p
(8)
where the dual variable p is a pair of two rows, p1 = (p11 , p12 ) and p2 = (p21 , p22 ). The divergence is defined as follows: divp = (divp1 , divp2 )T , where divpi =
∂pi2 ∂pi1 + , i = 1, 2. ∂x ∂y
(9)
This definition is similar to the vectorial dual norm from [13] for vectorial images, e.g. color images. Plugging (8) into (5) yields 1 min max (τ, divp)L2 + (τ − τo , τ − τo )L2 . (10) div τ =0 |pi |≤1 2δ Results from convex analysis, see for instance Theorem 9.3-1 in [14], allow us to exchange the order of max and min in (10) and obtain an equivalent optimization problem 1 max min (τ, divp)L2 + (τ − τo , τ − τo )L2 . (11) 2δ |pi |≤1 div τ =0 Now comes a trick. Let us introduce the orthogonal projection ΠK onto the constrained subspace K = {τ : div τ = 0}. Note that τ0 ∈ K. By means of the pseudoinverse Δ+ we may write that τ1 τ1 τ + = − ∇Δ div 1 . (12) ΠK τ2 τ2 τ2
310
C.A. Elo, A. Malyshev, and T. Rahman
The constraint div τ = 0 means that ΠK τ = τ , and the latter implies the equalities (τ, divp) = (ΠK τ, divp) = (τ, ΠK divp). Hence (11) is equivalent to 1 (13) max min (τ, ΠK divp)L2 + (τ − τo , τ − τo )L2 . |pi |≤1 div τ =0 2δ Solution to the minimization problem (without constraint div τ = 0!) 1 min (τ, ΠK divp)L2 + (τ − τo , τ − τo )L2 τ 2δ is τ = τ0 − δΠK divp
(14)
and satisfies the constraint div τ = 0. Owing to (14) we have the equality 1 1 (τ − τo , τ − τo ) = 2δ [(τ0 , τ0 ) − (δΠK divp − τ0 , δΠK divp − τ0 )] , (τ, ΠK divp) + 2δ which together with (13) gives our dual formulation:
min ΠK divp − δ −1 τ0 L2 : |pi | ≤ 1, i = 1, 2 . (15) p
Numerical solution of (15) is computed by Chambolle’s iteration from [10]:
pn + Δt ∇ ΠK divpn − δ −1 τ0 0 n+1 p = 0, p . (16) = 1 + Δt |∇ (ΠK divpn − δ −1 τ0 )| The iteration converges rapidly when Δt ≤ 14 . The smoothed tangential field after n iterations is given by τn = τ0 − δΠK divpn . 2.2
Step 2
The image d is reconstructed at the second step by fitting it to the normal vector field built from the tangential vector field computed at step 1, (n1 , n2 ) = (τ2 , −τ1 ). Again we introduce a dual variable r = (r1 (x, y), r2 (x, y)) and use the formula ∇dL1 = max|r|≤1 (∇d, −r)L2 . Then the minimization problem (6) is equivalent to the problem n 1 d − d0 2L2 , d, div r + (17) min max + d |r|≤1 |n| 2μ L2 where μ > 0 is a Lagrangian multiplier. After interchanging min and max in (17) we find conditions for attaining the minimum: n d = d0 − μ div r + . (18) |n| By analogy with (15) we can derive the dual formulation for step 2: d0 n − : |r| ≤ 1 . min div r + r |n| μ L2
(19)
A Dual Formulation of the TV-Stokes Algorithm for Image Denoising
Chambolle’s iteration for (19) is as follows: n − μ−1 d0 rn + Δt ∇ div rn + |n| . rn+1 = n − μ−1 d0 1 + Δt ∇ div rn + |n| 2.3
311
(20)
The Discrete Algorithm
The staggered grid is used for discretization as in [5]. For convenience we introduce the differentiation matrices ⎞ ⎛ ⎞ ⎛ 1 −1 1 ⎟ ⎜ −1 1 ⎟ ⎟ −1 1 1⎜ 1⎜ ⎟ ⎟ ⎜ ⎜ T .. .. (21) , −B = ⎜ B= ⎜ ⎟, ⎟ . . . . .. .. ⎠ ⎟ h⎝ h⎜ ⎝ −1 1 ⎠ −1 1 −1 where B is the forward difference operator and −B T is the backward difference operator. The discrete gradient operator applied to a matrix d is then defined as ∇h d = dBxT , By d , (22) where Bx (By ) stands for differentiation in the x (resp. y) direction. The discrete divergence operator is given by divh (p1 , p2 ) = −p1 Bx − ByT p2 .
(23)
The discrete analog of the projection operator ΠK has the form h ΠK = I − ∇h (Δh )+ divh ,
(24)
where the gradient and divergence are applied in a slightly different manner: T τ1 dBx h T h div = −τ1 Bx − By τ2 , ∇ d= . (25) τ2 By d To complete the definition (24) we need a description of the pseudoinverse operator (Δh )+ for the discrete Laplacian Δh d = −dBxT Bx − ByT By d.
(26)
Let us introduce the orthogonal N × N matrix of the Discrete Cosine Transform, C, which is defined by dst(eye(N)) in MATLAB. The symmetric matrix defined in MATLAB by dst(eye(N-1)), satof the Discrete Sine Transform, S, T isfies the equation S S = (N/2) ∗ I, where I is theidentity matrix. We prefer N/2 of order N − 1. The to use the orthogonal symmetric matrix S = −S/ singular value decomposition of B has the form B = S[0, Σ]C,
Σ = diag(σ1 , . . . , σN −1 ),
where the diagonal matrix Σ has the diagonal entries
(27)
312
C.A. Elo, A. Malyshev, and T. Rahman
σk =
πk 2 sin , h 2N
k = 1, 2, . . . , N − 1.
(28)
By the aid of (27) equation (26) can be rewritten as f = Δh d = −dC T
0
Σx2
C − CT
0
Σy2
Cd.
Denoting f = Cf C T and d = CdC T we arrive at the equation 0 0 − d. f = −d Σy2 Σx2
(29)
(30)
Suppose that the matrices f and This equation is easily solved with respect to d. d have the entries fij and dij for i, j = 0, 1, . . . . Note that in our case f00 = 0. Then the solution d = G(f) is as follows: d00 = 0, 2 , di,0 = −fi,0 /σi,y
i = 1, 2, . . . ,
2 , d0,j = −f0,j /σj,x
j = 1, 2, . . . ,
(31)
2 2 + σj,x ), i, j = 1, 2, . . . . dij = −fij /(σi,y
Thus the pseudoinverse operator (Δh )+ can be efficiently computed with the help of the Discrete Cosine Transform: (Δh )+ f = C T G(Cf C T )C,
(32)
where the function G is defined in (31). In conclusion we recall that multiplication of an N × N matrix by C or C T = C −1 is typically implemented by the aid of the fast Fourier transform and requires only O(N 2 log2 N ) arithmetical operations. All other computations have the cost O(N 2 ).
(a) Lena, 200 × 200
(b) Cameraman, 256 × 256 Fig. 1. Original images
(c) Barbara, 512 × 512
A Dual Formulation of the TV-Stokes Algorithm for Image Denoising
313
Algorithm. Dual TV-Stokes Given d0 , k, δ and μ ; Step one; Let p0 = 0 and q 0 = 0 ; Calculate τ 0 = (v 0 , u0 ) : v 0 = −Bd and u0 = dB T ; Initialize counter: n = 0 ; while not converged do Calculate projections: h (πp , πq ) = ΠK (divh pn , divh q n )
pn + k ∇h πp − δ −1 v0 . 1 + k |(∇h (πp − δ −1 v0 ))| q n + k ∇h πq − δ −1 u0 . q n+1 = 1 + k |(∇h (πq − δ −1 u0 ))| Update counter: n = n + 1 ; end Calculate τ : pn+1 =
h τ = τ0 − ΠK (δdivh pn+1 , δdivh q n+1 )
(33)
(34)
(35)
(36)
Step two; Let r 0 = 0 and calculate the normal field: n = (n1 , n2 ), n1 = u(v 2 + u2 )−1/2 and n2 = −v(v 2 + u2 )−1/2 ; Initialize counter: n = 0 ; while not converged do Calculate projections: r
n+1
r n + k ∇h divh (r n + n) − μ−1 v0 . = 1 + k ∇h divh (rn + n) − μ−1 v0
(37)
Update counter: n = n + 1 ; end Recover image d: d = d0 − μdivh r n+1
(38)
Algorithm 1. Dual TV-Stokes algorithm for image denoising 2.4
Numerical Experiments
In what follows we present several examples to show how the TV-Stokes method works for different images. All the images we have tested are normalized into gray-scale values, ranging from 0 (black) to 1 (white). In the experiments we start with a clean image, shown in figure 1, and then add random noise with zero mean. This is done by the imnoise MATLAB command, where the variance
314
C.A. Elo, A. Malyshev, and T. Rahman 5
130
4.4
x 10
4.2
120
4
110 3.8
100 3.6
90 3.4
80
3.2
3
70
2.8
60
2.6
50
0
10
20
30
40
50
60
70
80
90
0
1
2
3
4
5
6
(a) Dual TV-Stokes algorithm 1
7
8 4
100
x 10
(b) TV-Stokes [5]
Fig. 2. Energy vs. iterations plot for the first step
parameter is set to 0.001 for the Barbara image and 0.005 for the Lena image. The Cameraman image is taken directly from the paper [5], so we compare the results with the same noisy image as input. In [5] this model is further compared to the two-step method LOT and famous ROF model. The signal-to-noise ratio is measured in decibels before denoising: " ! (d − d)2 dx Ω SN R = 20 log10 ! , (39) (η − η)2 dx Ω ! ! 1 1 where d = |Ω| and η = |Ω| Ω d dx, Ω η dx The numerical procedures used in [5] were based on explicit finite difference schemes. This process is very slow, as the constraint converges slowly. However, in the proposed dual method the constraint is satisfied on each step by the orthogonal projection. The energy and number of iterations required for convergence in step one are shown in figure 2. The figure clearly illustrates that the dual TV-Stokes algorithm requires less iterations before the energy is stable than the primal TV-Stokes algorithm. Although the iterations in the dual TV-Stokes algorithm require more computational effort in each iteration, it is much faster than using sparse linear solvers. Inverting the Laplacian for the orthogonal projection in each iteration is a bottleneck for very large images. In all these examples the projection was applied by the aid of the Fast Fourier Transform, which needs O(n2 log(n)) operations in each iteration. For very large images, one should consider using a multigrid solver method for applying the projection. This will reduce the operations cost to O(N 2 ). All methods were coded in MATLAB, and in table 1 the CPU time is given in seconds for each test image. The figure shows the dual TV-Stokes algorithm vs. the primal TV-Stokes algorithm from [5]. We measure the L2 -norm of the energy in (15) and (19) for stopping criteria, and stop the iteration when the difference of the energy is below 10−3 . For the TV-Stokes algorithm we used the same stopping criteria as in [5], where the tolerance of the L2 -norm of the
A Dual Formulation of the TV-Stokes Algorithm for Image Denoising
315
Table 1. Runtimes of the dual TV-Stokes algorithm compared to the TV-Stokes algorithm [5]. The test system is a 2 Opteron 270 dualcore 64-bit processor and 8GB RAM. Both steps in the dual TV-Stokes algorithm are computed with 150 iterations, while the first step in the primal TV-Stokes algorithm is calculated with 75000 iterations and the second step with 25000 iterations. Algorithm Dual TV-Stokes algorithm Image First step Second step Lena 9.8 1.12 Cameraman 17.4 2.2 Barbara 128.2 20.7
TV-Stokes algorithm, [5] First step Second step 9083.2 1992.5 11189.0 2259.4 80602.5 14926.3
constraint is equal to 5 × 10−3 and the difference in the energy tolerance is equal to 10−3 . The time steps were set to 10−3 and 5 × 10−3 respecitvely for the first and second step of the TV-Stokes algorithm. Our first test is the well known Lena image, which we will recover from highly added noise. We have cropped the image to show the face, which consists of smooth areas and edges that are important to preserve. The denoised image in Figure 3, shows that the dual TV-Stokes method has recovered the smooth areas without inducing any staircase-effect. The smoothing parameter δ is equal to 0.0835 and μ is equal to 0.17. Since this is a highly noisy image, the ROF model fails to give a visually pleasant image, because the smooth surfaces are piecewise continuous. The TV-Stokes algorithm however, has nearly the same quality as the dual TV-Stokes algorithm. For the TV-Stokes algorithm, δ was equal to 0.045. The next test is the Cameraman image, which consists of a smooth skyline and some low-intensity buildings in the background. The buildings are difficult to recover, as they get smeared out by the denoising. The results are shown in figure 4 with δ equal to 0.055 and μ equal to 0.08. The TV-Stokes result is taken from [5] where the SNR are the same as the one we report, 20 log10 (8.21) ≈ 18.28. Figure 4.d shows the TV-Stokes reconstruction for the same noisy image, where the delta parameter is equal to 0.06. The last example is the Barbara image, which is quite detailed, with high and low intensity textures. The high intensity textures and the smooth areas are preserved quite well, but the low intensity textures disappear in the same way as for the Cameraman. This image is 512 × 512 in size, which makes the algorithm slower, because of the rather large number of matrix operations per iteration. However, reaching a result for the optimal parameters is still obtainable, since the method has a denoised image after a few steps. Thus, one can run the method multiple times to find the optimal parameters. For this image we used δ equal to 0.05 and μ equal to 0.15. We do not report on an optimal result for this particular case of the TV-Stokes algorithm, due to page limitation and the amount of running time. Clearly, using the dual formulation is more effective than solving the model with the explicit gradient descent method. The CPU time is found for only one runtime, since computing an average of many runtimes is very time consuming
316
C.A. Elo, A. Malyshev, and T. Rahman
(a) Noisy image, SN R ≈ 14.0
(b) Denoised using the dual TVStokes algorithm
(c) Contour plot, dual TV-Stokes (d) Difference image, dual TVimage Stokes
(e) Denoised using ROF [1]
(f) Difference image, ROF
(g) Denoised using the TV-Stokes (h) Difference image, TV-Stokes algorithm [5] Fig. 3. Lena image (200 × 200), denoised using the dual TV-Stokes, TV-Stokes and the ROF algorithm
A Dual Formulation of the TV-Stokes Algorithm for Image Denoising
317
(a) Noisy image, SN R ≈ 18.28 (b) Denoised using the dual TVStokes algorithm
(c) Difference image, dual TV- (d) Denoised using Stokes Stokes algorithm [5]
the
TV-
Fig. 4. Cameraman (256 × 256), denoised using the dual and the primal formulation of the TV-Stokes algorithm
(a) Noisy image, SN R ≈ 20.0
(b) Denoised image
Fig. 5. Barbara (512 × 512), denoised using the dual formulation of the TV-Stokes algorithm
for the TV-Stokes method. Although, the time shown are for one runtime, they clearly give the indication that our method is much faster and stable. The comparison with the primal method also shows that the proposed dual method has the same denoising quality.
318
C.A. Elo, A. Malyshev, and T. Rahman
References 1. Rudin, L.I., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Phys. D 60(1-4), 259–268 (1992) 2. Chan, T., Marquina, A., Mulet, P.: High-order total variation-based image restoration. SIAM J. Sci. Comput. 22(2), 503–516 (2000) 3. Chambolle, A., Lions, P.L.: Image recovery via total variation minimization and related problems. Numer. Math. 76, 167–188 (1997) 4. Lysaker, O., Lundervold, A., Tai, X.C.: Noise removal using fourth-order partial differential equation with applications to medical magnetic resonance images in space and time. IEEE Trans. Imag. Proc. 12, 1579–1590 (2003) 5. Rahman, T., Tai, X.C., Osher, S.: A tv-stokes denoising algorithm. In: Sgallari, F., Murli, A., Paragios, N. (eds.) SSVM 2007. LNCS, vol. 4485, pp. 473–483. Springer, Heidelberg (2007) 6. Litvinov, W., Rahman, T., Tai, X.C.: A modified tv-stokes model for image processing (submitted) (2008) 7. Lysaker, O.M., Osher, S., Tai, X.C.: Noise removal using smoothed normals and surface fitting. IEEE Transaction on Image Processing 13(10), 1345–1357 (2004) 8. Bertalmio, M., Bertozzi, A., Sapiro, G.: Navier-stokes, fluid dynamics, and image and video inpainting. In: Proc. IEEE Computer Vision and Pattern Recognition (CVPR) (2001) 9. Tai, X., Osher, S., Holm, R.: Image inpainting using tv-stokes equation. Image Processing based on partial differential equations (2006) 10. Chambolle, A.: An algorithm for total variation minimization and applications. J. Math. Imaging Vis. 20(1-2), 89–97 (2004) 11. Carter, J.: Dual methods for total variation-based image restoration. PhD thesis, UCLA (2001) 12. Chan, T.F., Golub, G.H., Mulet, P.: A nonlinear primal-dual method for total variation-based image restoration. SIAM J. Sci. Comput. 20(6), 1964–1977 (1999) 13. Bresson, X., Cham, T.F.: Fast minimization of the vectorial total variation norm and applications to color image processing. CAM Report 07-25 (2007) 14. Ciarlet, P.G., Jean-Marie, T., Bernadette, M.: Introduction to numerical linear algebra and optimisation. Cambridge University Press, Cambridge (1989)
Anisotropic Regularization for Inverse Problems with Application to the Wiener Filter with Gaussian and Impulse Noise Micha Feigin and Nir Sochen School of Mathematics, Tel Aviv University [email protected], [email protected]
Abstract. Most inverse problems require a regularization term on the data. The classic approach for the variational formulation is to use the L2 norm on the data gradient as a penalty term. This however acts as a low pass filter and thus is not good at preserving edges in the reconstructed data. In this paper we propose a novel approach whereby an anisotropic regularization is used to preserve object edges. This is achieved by calculating the data gradient over a Riemannian manifold instead of the standard Euclidean space using the Laplace-Beltrami approach. We also employ a modified fidelity term to handle impulse noise. This approach is applicable to both scalar and vector valued images. The result is demonstrate via the Wiener filter with several approaches for minimizing the functional including a novel GSVD based spectral approach applicable to functionals containing gradient based features.
1
Introduction
Handling degraded images, both due to blur and noise, is a practical reality in any imaging field. The common image degradation model is I = I0 ∗ h + n
(1)
where I, the observed image, is the result of a convolving the input image (or ideal image) I0 with some blurring kernel h. The result is then summed with additive noise n. This is a common model for any system that contains a lens and sensor. Both the blur and noise are a combination of several processes. Some typical causes for image blue are out of focus images, motion blur due to an unstable camera and/or object and a low pass filter resulting from the finite aperture and anti aliasing filter on the sensor. Noise can result from the sensor and amplifier due to low light, heat, dead pixels and background radiation or from memory and communication corruption. Each of these processes has it’s own typical blur kernel and noise distribution statistics [1, 2]. A direct naive approach to handle the blur can be given using a spectral (Fourier) approach manipulation of the degradation model equation. To see the ˆ +n difficulty though, look at the Fourier transform of this equation Iˆ = Iˆ0 · h ˆ X.-C. Tai et al. (Eds.): SSVM 2009, LNCS 5567, pp. 319–330, 2009. c Springer-Verlag Berlin Heidelberg 2009
320
M. Feigin and N. Sochen
(where the hat notation denotes the Fourier transform). This transforms the convolution into a multiplication which allows for an easy rearrangement of the ˆ Any L2 kernel h will the equation. Extracting Iˆ0 gives us Iˆ0 = Iˆ − n ˆ /h. decay to zero at infinity. This results with a divide by zero issue at least for high frequencies. Add to that the issue that the SNR usually drops at these frequencies, which makes this procedure very sensitive to noise. One solution is this case is the Wiener filter [3], which can be derived from the standard variational formulation for ill posed inverse problems by adding prior knowledge (or assumptions) via an additional penalty term to the reconstruction. That is to minimize an energy functional of the form S (I0 ) = I0 ∗ h − I + μ Φ (I0 ) . fidelity term penalty
(2)
Here Φ is some function of the parameter I0 that imposes the assumptions on the model. A common constraint term is Φ (I0 ) = ∇I0 which penalizes high frequencies as these are often the source of instability. The side effect of this constraint is that while high frequency noise is reduced in the reconstruction, edge detail is lost as well as is demonstrated in Fig. 1.
(a) Original Image (b) Degraded Input
(c) μ = 5 · 10−4
(d) μ = 5 · 10−5
Fig. 1. Edge preservation vs. Noise suppression with the Wiener filter. The input image 1(a) is degraded using Gaussian white noise 1(b). The results show the difference between preferring noise suppression 1(c) to edge preservation 1(d).
This functional is often minimized under the L2 norm which is appropriate for Gaussian noise. This is mainly due to the fact that the resulting Euler Lagrange equations are linear and are thus (relatively) easy to solve. That is, the classic Wiener filter functional based on the L2 norm 2 2 (3) S (I0 ) = I0 ∗ h − IL2 + ∇I0 L2 = |I0 ∗ g − I|2 + |∇I0 |2 dA . results with the following Euler Lagrange equations (see [4] for the derivation of the Euler Lagrange formulation of the convolution) −h (−¯ x) ∗ (h (¯ x) ∗ I0 − I) − μΔI0 = 0 .
(4)
Anisotropic Regularization for Inverse Problems
321
Here x ¯ is the coordinate vector x ¯ = (x, y) for the two dimensional case. This can be solved as before by applying the Fourier transform, which results with ˆ (−ω) · h ˆ (¯ h ω ) · Iˆ0 − Iˆ + μ |¯ ω |2 Iˆ0 = 0 (5) where ω ¯ = (ωx , ωy ) is the frequency vector for the resulting frequencies along the x and y axes respectively. Now, assuming that the convolution kernel is real we ˆ (−ω) = h ˆ ∗ (ω) (where h∗ is the conjugate of h) to rewrite can use the identity h the equation as ˆ h∗ (ω) Iˆ0 = Iˆ . (6) 2 ˆ 2 h (ω) + μ |ω| Despite being easy to solve, there are two main issues with the L2 norm approach, both for the constraint and the fidelity term. The first issue is that it fails to preserve object boundaries (Fig. 1). The main reason is the penalty term that penalizes high frequencies. As the fidelity term is also L2 it does little to alleviate this problem. The second issue is that the fidelity term is designed to handle Gaussian noise and behaves poorly in the presence of impulse noise One solution to both these issues is to use the L1 or total variation (TV) norm [5,6,7]. When used for the fidelity term it improves behavior with impulse noise. For the constraint it improves edge preservation. For the functional S (I0 ) = I0 ∗ h − IT V + μ ∇I0 T V = |I0 ∗ h − I| + μ |∇I2 | dA (7) the resulting Euler-Lagrange equations are −h (−¯ x) ∗
h (¯ x) ∗ I0 − I − μdiv |h (¯ x) ∗ I0 − I|
∇I |∇I|
= 0.
(8)
Unfortunately though the solution of which is unstable. One approach to improve on this is to use an augmented TV norm [5] 2 2 (I0 ∗ h − I) + η + μ |∇I0 | + ηdA (9) S (I0 ) = with 0 < η 1. The resulting modified Euler-Lagrange equation are ⎛ ⎞ ∇I h (¯ x) ∗ I0 − I ⎠ = 0. − μdiv ⎝ −h (−¯ x) ∗ 2 2 (h (¯ x) ∗ I0 ) + η |∇I| + η
(10)
This greatly improves the response of the fidelity term to impulsive noise, but not so much for the edge preservation of the constraint. It also doesn’t account explicitly for the edges in the image. Other approaches include using Mumford-Shah like techniques of edge detection into the functional [4], weighing the Laplacian based on edge detection [8], Perona-Malik like regularizers [9], maximal likelihood estimators [10], certainty maps [11] and channel pairing on color images [12].
322
M. Feigin and N. Sochen
We propose two novelties in this paper. The first is to combine the augmented L1 norm on the fidelity term for handling impulse noise with anisotropic regularization based on the Laplace Beltrami operator for edge preservation. This is achieved by keeping the L2 norm of the gradient, however this is calculated over a Riemannian manifold instead of the standard Euclidean space using a Laplace-Beltrami approach [13]. When combined with the augmented TV norm (9), this approach also produces exceptional results for impulsive noise (Sec. (4)) The second is the use of the GSVD (generalized singular value decomposition) for the minimization of functionals that employ a gradient based penalty term. It’s direct contribution is the ability easily minimize non-local operators and functionals defined on non square domains where the Fourier transform is inapplicable. For isotropic operators it can be very efficient as the decomposition needs to be calculated once only off line. One interesting point to both these approaches is the relation to other frameworks. In particular it enables to better understand the relation to sparse representation and K-SVD [14]. It is important to note that both these ideas are easily applicable to general ill posed inverse problems over general feature spaces, and specifically for this case, also for color images [15] and textures [16]. The rest of this paper is organized as follows: Sec. 2 discusses the anisotropic approach. Sec. 3 discusses several approaches to minimizing the functional, including a novel approach using the GSVD. Sec. 4 shows some results of the method.
2
Anisotropic Regularization for the Wiener Filter
The problem with edge preservation lies with the gradient based penalty term. In the Euler-Lagrange equations it manifests as a Laplacian that acts as a low pass filter. In order to correctly formulate the anisotropic penalty term, we start with the Euler Lagrange equation for the Wiener filter −h (−x) ∗ (h (x) ∗ I0 − I) − μΔI0 = 0
(11)
and replace the Laplacian with an anisotropic operator, namely the LaplaceBeltrami operator [13] resulting with −h (−x) ∗ (h (x) ∗ I0 − I) − μΔg I0 = 0 .
(12)
The Laplace Beltrami operator is defined as √ 1 Δg I = √ div gG−1 ∇I g where for the gray-scale case
1 + Ix2 Ix Iy , G= Ix Iy 1 + Iy2
g = det (G) .
(13)
(14)
What this does is apply the Laplacian diffusion operator, but instead of applying it under the standard Euclidean norm, it is applied over the image manifold [13].
Anisotropic Regularization for Inverse Problems
323
This means that we are looking at the image as a two dimensional manifold in three dimensional space for gray scale images and in 5 dimensional space for color images. When applying the diffusion operator, distance between pixels is measured over this manifold so the distance takes into account not only spatial offset but also intensity offset. The result is that pixels on different side of an edge are farther apart than pixels on the same homogeneous region and the edges act as insulators so that image data doesn’t flow across edges. This can be extended to color images by applying the diffusion √ on a per- channel basis, that is for each channel I i the process is Δg I i= √1g div gG−1 ∇I i with 2 i i 1 + i Ixi i IxIy . (15) G= i i 2 1 + i Iyi i Ix Iy The metric itself takes into account all the channels coupling them in the final process to remove misalignment of the edges across the different channels. Note that the image channels can be color channels such as RGB, CMY or more general features such as textures [16]). When extending the functional to handle impulse noise using the augmented L1 fidelity term, the Euler-Lagrange equations become instead −h (−¯ x) ∗
3
h (¯ x) ∗ I0 − I 2
(h (¯ x) ∗ I0 ) + η
− μΔg I = 0 .
(16)
Finding the Minimizer
There are several approaches to minimizing the resulting functional. We already have the Euler-Lagrange equations, i.e Eq. (12) and 16. Using the direct Fourier space approach, even for the L2 fidelity term, is not applicable here since the Fourier transform doesn’t diagonalize the LaplaceBeltrami operator. A different relatively simple direct approach approach is to use the gradient descent equations h (¯ x) ∗ I0 − I ∂ I0 = h (−¯ + μΔg I x) ∗ ∂t (h (¯ x) ∗ I0 )2 + η
(17)
For the L2 fidelity term there are two other spectral approaches that can be applied here, and eigen transform and the GSVD. The advantage of these among other things is that they provide a direct solution and thus prove the existence of the minimizer, same as for the standard Wiener filter. Proving the existence of a minimizer for the proposed Tikhonov functional is much more difficult and beyond the scope of this paper, but can be done using similar lines to those taken in [5]. 3.1
The Laplace-Beltrami Eigen-Space
We can use the same approach implemented in [17] to diagonalize the LaplaceBeltrami operator. The problem is that the Eigenvectors of the Laplace-Beltrami
324
M. Feigin and N. Sochen
operator don’t convert the convolution into a multiplication, so we need to combine this approach with the Fourier transform. We start with the Euler-Lagrange equations for the anisotropic Wiener filter, Eq. (12). If we linearize the Laplace Beltrami operator by fixing the metric, it becomes a self adjoint negative (semi) definite operator and thus it’s eigenspace is a bases to the function space under the L2 norm. Insert into this equation the eigen decomposition of the image using this eigen space I0 = c0i φi , I = ci φi (18) i
i
This produces h (−x) ∗
h (x) ∗
i
c0i φi −
+μ
ci φi
i
λi c0i φi = 0
(19)
i
which after rearrangement gives c0i h (−x) ∗ h (x) ∗ φi + μλi c0i φi = ci h (−x) ∗ φi .
(20)
i
Now, to handle the convolution, apply the Fourier transform ˆ∗ · h ˆ · c0 φˆi + μλi c0 φˆi = − −h ci ˆh∗ φˆi i i i
(21)
i
which can be rewritten as
c0i
i
ˆ 2 ˆ ∗ φˆi . ci h h − μλi φˆi =
(22)
i
˜ Here I˜ = (ci ) and This is a linear set of equations of the form AI˜0 = B I. 0 ˜ I0 = ci are the coefficient vectors in the Laplace-Beltrami eigen-space. This is a system of equations needs to be solved for I˜0 . Using these coefficients the ideal image I0 can be reconstructed. For a full solution this needs to be combined with fixed point iterations updating the metric, although it is stable with respect to the flow so in effect this is rarely need. There are two things to note here. First, the coefficients of I decay rather quickly so we can truncate I˜ and thus not calculate the right hand side of B. The same assumption can be made for I˜0 and thus for A. 3.2
Using the GSVD
Consider an energy functional with two linear operators La and Lb using the L2 norm 2 2 S (f ) = |La f | + μ |Lb f | dA . (23)
Anisotropic Regularization for Inverse Problems
325
Assuming that these operators can be discretized as matrices A and B respectively this can written as equations with v a vector representation of the function f S (v) = Av2 + μ Bv2 . (24) The two matrices A and B have a joint diagonalization based on the general singular value decomposition (GSVD) of the form [18] A = U Σ1 X T ,
B = V Σ2 X T
(25)
with U and V unitary matrices and Σ1 and Σ2 positive diagonal (not necessarily square). U and V must have the same number of columns but not necessarily the same number of rows (this last property we will need later on). Thus Eq. (24) can be rewritten as 2 2 S (v) = U Σ1 X T v L2 + μ V Σ2 X T v L2 . (26) Now, we can substitute v˜ = X T v to construct a functional in v˜. Also note that the L2 norm is invariant to unitary transformations, thus this functional is equivalent to 2 2 S (˜ v ) = Σ1 v˜L2 + μ Σ2 v˜L2 . (27) This new functional can be minimized according to v˜ resulting with Σ1T Σ1 v˜ + μΣ2T Σ2 v˜ = 0
(28)
We would like to do something similar with the Wiener-Filter formulation. The problem is that the gradient operator can not be discretized as a matrix operator since it takes a function and returns a vector. Luckily, what we need is an operator operating on I such that the norm would be equal to that of the gradient. For the L2 case this can be achieved as follows 2 2 S (I0 ) = |h ∗ I0 − I| + μ |∇I0 | dA 2 Dx 2 2 (29) ⇒ HI0 − I + Dy I0 = HI0 − IL2 + DI0 L2 L2 where H is the convolution matrix (which is block cyclic but not cyclic in the x 2D case) and D = D is the matrix resulting from stacking the matrix for the Dy derivative in the x direction and the one for the derivative in the y direction. For the L2 case we get 2 Dx 2 2 2 (30) Dy I0 = Dx I0 L2 + Dy I0 L2 = ∇I0 L2 . L2 Now we can use the fact that the GSVD can be applied to matrices with a different number of rows to diagonalize this equation H = U Σ1 X T ,
D = V Σ2 X T
(31)
326
M. Feigin and N. Sochen
Using this we can do the same procedure as before 2 2 HI0 − I2L2 + DI0 2L2 ⇒ U Σ1 X T I0 − I L + V Σ2 X T I0 L 2
2
(32)
and again based on U and V being unitary and substituting I˜0 = X T I0 and I˜ = U −1 I = U T I results with 2 2 (33) S I˜0 = Σ1 I˜0 − I˜ + Σ2 I˜0 L2
L2
this can be minimized according to I˜0 to produce Σ1T Σ1 I˜0 − I˜ + Σ2T Σ2 I˜0 = 0
(34)
or after rearrangement and back-substitution −1 T T Σ1 U I0 . I = X −T Σ1T Σ1 + μΣ2T Σ1
(35)
Note that Σ1T Σ1 + μΣ2T Σ1 is a diagonal matrix and thus easy to invert (in fact for μ = 1 it is the identity matrix). To apply the same idea to the anisotropic case, we need to formulate the prior to the Laplace-Beltrami operator as a gradient over a manifold instead. The operator is the minimizer of the following symmetric positive definite √ √ −1 2 ∇I T G−1 ∇I gdm σ = Dg ∇I dm σ, gG = Dg2 (36) and the discrete formulation for the anisotropic derivative matrix Dg (which replaces D in Eq. 29) can be found via an eigen decomposition of the matrix √ −1 gG √ ⎛ 2 √ 2 2 2 ⎞ Ix + 1+Ix +Iy Iy Ix Iy (1− 1+Ix2 +Iy2 )
√ √ D + D x y 4 4 2 2 2 2 2 2 2 2 Dx 1+Ix +Iy ⎜ (I +Iy )√ 1+Ix +Iy ⎟ (Ix +I√ y) = ⎝ I xI 1− Dg = A (37) ⎠ 1+Ix2 +Iy2 ) Iy2 + 1+Ix2 +Iy2 Ix2 x y( Dy √ √ D + D (Ix2 +Iy2 ) 4 1+Ix2 +Iy2 x (Ix2 +Iy2 ) 4 1+Ix2 +Iy2 y One advantage of this approach is that it is applicable to non-local operators and to non square domains where the Fourier transform as applied to the original Wiener filter fails. For the isotropic case it needs to be calculated once off line as the transform is constant and thus can be very efficient for reoccurring problems (or by splitting the problem into constant sized patches as described in [17]).
4
Numerical Results
Comparing the reconstruction quality based on standard measurements alone such as SNR and PSNR doesn’t do justice to the method. This is due to the fact that these values are not good assessors for edge reconstruction being L2 based measures. Despite this and for a lack of a better objective comparison method,
Anisotropic Regularization for Inverse Problems
327
we do see an improvement in the reconstruction based on these measurements. It is important to also note the subjective difference when looking at the images themselves. The biggest difference is seen near pronounced edges and textures which are much better preserved than with the standard wiener filter. This method also removes ringing (Gibbs effect) seen around strong edges and color skews in color images. The results are cropped and zoomed to better accent the difference due to the limit of the medium.
(a) Input
(b) Degraded
(c) Standard W.F.
(d) Anisotropic W.F.
Fig. 2. Reconstruction of a gray-scale image (2(a)) degraded using a Gaussian kernel and Gaussian noise (2(b)) with standard deviation of 10%. The image is reconstructed using the standard (2(c)) and anisotropic Wiener filter (2(d)).
The first example (Fig. 2) shows the results for a gray scale image degraded by a Gaussian kernel and Gaussian noise with standard deviation of 10% (with a resulting SNR of 16.34db). The reconstruction for both the standard Wiener filter (2(c)) and the anisotropic version (2(d)) is done based on the L2 fidelity term. The SNR of the reconstructed images are 20.72db and 21.08db respectively. The anisotropic reconstruction displays less noise, especially visible in homogeneous areas such as the white background and skin. The edges in the isotropic version on the other hand display both blur (such as the back, hands and hair) and ringing around pronounced edges not appearing in the anisotropic version. This is most pronounced around the dominant edges of the back and the hair. Figure 3 shows the results of applying the Wiener filter to an image with impulse noise (11% density, with 8.47db SNR). The first two examples (3(b), 3(e)) display the result of applying the standard and anisotropic Wiener filters respectively, both using the L2 fidelity term. Despite improving SNR values (15.9db and 16.48db) the results are still rather miserable, although the anisotropic version still displays more pronounced edges (teeth, wall) as well as less noise. On the other hand, looking at the versions employing the augmented L1 fidelity term (3(c) and 3(f)), on first look one can mistake them for the input image. Despite this the anisotropic version still displays much sharper results up close, as well as improved SNR (22.48db compared to 22.98db). The following examples for color images show the extendability of the method to vector valued images.
328
M. Feigin and N. Sochen
(a) Input Image
(b) Std. W.F. L2 fidelity
(c) Std. W.F., L1 fidelity
(d) Degraded Image
(e) AI W.F., L2 fidelity
(f) AI W.F., L1 fidelity
Fig. 3. Restoration of a gray scale image corrupted by impulse noise of density 0.11. Figures 3(b) and 3(e) show the reconstruction using regular and anisotropic Wiener filter with L2 fidelity. Figures 3(c) and 3(f) show the reconstruction using the L1 fidelity term.
Figure 4 shows the results for a color image degraded by a Gaussian kernel and Gaussian noise with a standard deviation of 10% (SNR of 16.7db). As can be seen, the anisotropic reconstruction produces sharper edges without the color shifts and ringing which is visible around sharp edges. Additionally, there is less overall noise and color shifts due to the smoothing of the noise. SNR for the isotropic case is 21.04db compared to 21.6db for the anisotropic variation. Fig. 5 shows the results of applying both the regular and anisotropic Wiener filter, both based on the L1 fidelity term, to a color image degraded by a Gaussian kernel and impulse noise with 11% density (SNR of 11db). The anisotropic variation shows sharper edges, better color restoration and less color skews around edge boundaries. This, like the previous results, is most pronounced around bright edges such as the teeth, eyes and wall. The SNR of the reconstruction is 20db and 23.1db for the isotropic and anisotropic varieties respectively.
Anisotropic Regularization for Inverse Problems
(a) Degraded image
(b) Standard W.F.
329
(c) Anisotropic W.F.
Fig. 4. Color image degraded by a gaussian kernel and uncorrelated Gaussian noise (4(a)) with standard deviation of 10%. Figures 4(b) and 4(c) show the results for the standard and the anisotropic reconstruction.
(a) Degraded image
(b) Std. W.F. L1 fidelity
(c) AI W.F. L1 fidelity
Fig. 5. Color image degraded by a gaussian kernel and uncorrelated impulse noise (5(a)) with density 0.11. Figures 5(b) and 5(c) show the results for the standard and anisotropic restoration based on the L1 fidelity term.
5
Conclusion
In this work we presented an anisotropic regularization term for inverse problems that allows to better preserve object edges while at the same time improving noise suppression. Combined with an augmented L1 fidelity term it provides remarkable results for images corrupted by impulse noise.
References 1. Goodman, J.: Introduction to Fourier Optics. McGraw-Hill Book Company, New York (1996) 2. Jähne, B.: Digital Image Processing, 5th edn. Springer, Heidelberg (2002)
330
M. Feigin and N. Sochen
3. Gonzalez, R.C., Woods, R.E.: Digital image processing, 2nd edn. Prentice-Hall, Englewood Cliffs (2002) 4. Bar, L., Sochen, N., Kiryati, N.: Semi-blind image restoration via mumford-shah regularization. IEEE Trans. on Image Processing 15(2), 483–493 (2005) 5. Bar, L., Kiryati, N., Sochen, N.: Image deblurring in the presence of impulsive noise. Int. J. Comput. Vision 70(3), 279–298 (2006) 6. Blomgren, P., Chan, T.F.: Color tv: Total variation methods for restoration of vector-valued images. IEEE Trans. Image Processing 7, 304–309 (1998) 7. Chan, T.F., Vese, L.A.: Image segmentation using level sets and the piecewiseconstant mumford-shah model. Technical Report 00-14, UCLA CAM (2000) 8. Charbonnier, P., Blanc-féraud, L., Aubert, G., Barlaud, M.: Deterministic edgepreserving regularization in computed imaging. IEEE Trans. Image Processing 6, 298–311 (1997) 9. Welk, M., Theis, D., Weickert, J.: Variational deblurring of images with uncertain and spatially variant blurs. In: Kropatsch, W.G., Sablatnig, R., Hanbury, A. (eds.) DAGM 2005. LNCS, vol. 3663, pp. 485–492. Springer, Heidelberg (2005) 10. Jalobeanu, A., Blanc-Feraud, L., Zerubia, J.: An adaptive gaussian model for satellite image deblurring. IEEE Transactions on Image Processing (4), 613–621 (2004) 11. Krajsek, K., Mester, R.: The edge preserving wiener filter for scalar and tensor valued images. In: DAGM-Symposium, pp. 91–100 (2006) 12. Kaftory, R., Sochen, N., Zeevi, Y.Y.: Variational blind deconvolution of multichannel images. Int. J. Imaging Science and Technology 15(1), 56–63 (2005) 13. Sochen, N., Kimmel, R., Malladi, R.: A general framework for low level vision. IEEE Trans. Image Processing, Special Issue on Geometry Driven Diffusion 7, 310–318 (1998) 14. Aharon, M., Elad, M., Bruckstein, A.: The K-SVD: An algorithm for designing of overcomplete dictionaries for sparse representation. IEEE Trans. On Signal Processing 54(11), 4311–4322 (2006) 15. Kimmel, R., Malladi, R., Sochen, N.: Images as embedded maps and minimal surfaces: Movies, color, texture, and volumetric medical images. International Journal of Computer Vision 39, 111–129 (2000) 16. Sagiv, C., Sochen, N., Zeevi, Y.: Gabor features diffusion via the minimal weighted area method. In: EMMCVPR (September 2001) 17. Feigin, M., Sochen, N., Vemuri, B.C.: Efficient anisotropic α-kernels decompositions and flows. In: POCV (2008) 18. Golub, G.H., Loan, C.F.V.: Matrix computations, 3rd edn. Johns Hopkins University Press, Baltimore (1996)
Locally Adaptive Total Variation Regularization Markus Grasmair Department of Mathematics, University of Innsbruck, Technikerstr. 21a, A-6020 Innsbruck, Austria [email protected] http://infmath.uibk.ac.at
Abstract. We introduce a locally adaptive parameter selection method for total variation regularization applied to image denoising. The algorithm iteratively updates the regularization parameter depending on the local smoothness of the outcome of the previous smoothing step. In addition, we propose an anisotropic total variation regularization step for edge enhancement. Test examples demonstrate the capability of our method to deal with varying, unknown noise levels.
1
Introduction
Because of its ability to generate images with piecewise smooth structures that are well separated by pronounced edges, total variation regularization is one of the most widely used techniques for image denoising and related tasks. Since the first proposal by Rudin, Osher, and Fatemi [14] of using the total variation for denoising purposes, that is, the L1 -norm of the gradient, this method has been applied to a wide range of applications in imaging and inverse problems. We refer to [1, 2, 3, 5, 12, 13, 15] to name but a few contributions to this field. Given a noisy function f ∈ L2 (Ω) on some open and bounded domain Ω ⊂ IRn , n ∈ IN, the goal of denoising is to find a new function u close to f that retains the important features of f while noise, consisting of fast oscillations, is removed. Noting that edges belong to the most prominent features in images, this task can be achieved by minimizing the total variation functional 2 1 T (u; α) := u(x) − f (x) dx + α|Du|(Ω) (1) 2 Ω with respect to u ∈ BV(Ω). The regularization parameter α > 0 in (1) controls the amount of smoothing that is desired: the larger α, the more the regularized function uα tends to consist of well separated homogeneous regions. Conversely, a small parameter α implies a function lying close to the input data, but also possibly exhibiting a significant number of oscillations. The relation between α and uα , however, exists only on a qualitative level. There is no simple connection between the value of α and the smoothness of uα , or even between α and the difference f − uα , which is simply the part of the data classified as noise by the functional T . The necessity of taking into X.-C. Tai et al. (Eds.): SSVM 2009, LNCS 5567, pp. 331–342, 2009. c Springer-Verlag Berlin Heidelberg 2009
332
M. Grasmair
account both the data and the expected noise level is a well established fact in the theory of inverse problems (see for instance [8]). Because for many applications of mathematical imaging, in particular tasks that are to be completely automated, a precise knowledge of the noise is not available, this leads to the conclusion that, in these cases, a-priori parameter choices are not feasible. Instead, one should adapt α until both uα and the perceived noise f − uα are satisfactory. Though better than a fixed a-priori choice, also adaptation of the regularization parameter need not be sufficient for good results. It may happen that the noise on the image f is not identically distributed but varies locally. In this case, it is difficult to find a compromise between oversmoothing in noise-free regions caused by too large a parameter choice, and a still noisy output resulting from a small parameter. Similar effects can be observed, if the structure of the noise-free data itself changes over the image. Then, the regularization parameter should be larger for homogeneous parts of the image than for parts with small details. The problem of finding a parameter that is suited for the whole image can be circumvented by passing from a global parameter α > 0 to a parameter function α : Ω → IR>0 . Then, the regularization functional reads as 1 T (u; α) = 2
Ω
2 u(x) − f (x) dx +
α(x) d|Du|(x) .
(2)
Ω
This functional is well-defined, if α is continuous, and, using direct methods, can readily be shown to attain a minimizer, if α is bounded away from zero. Total variation regularization with non-constant regularization parameter has already been studied in several other articles [6, 9, 10, 11, 16, 17]. In [16, 17], the choice of α is based on the scale of the features one wants to recover. In [10], at first the uniform problem is solved with an automatically identified optimal regularization parameter α. The result of the first denosing attempt is used for extracting the edges in the image, at which subsequently the regularization parameter is locally increased. Then the minimization problem is solved a second time with the localized parameter α(x). The approach in [11] uses statistical properties of the residual in order to decide whether the local regularization parameter is suited. The criterion employed there is based on the local variance of the residual: If it is close to the noise level, one can expect that mostly noise has been filtered. It it is higher, then the residual probably contains texture and therefore the regularization parameter has to be decreased. The estimates in [11] are closely related to the inequalities in [10], though the approaches by which they are reached differ considerably. Note moreover that the same idea has already been employed in [6] for one-dimensional total variation regularization. In this paper, we propose to target some a-priori specified smoothness of the output uα , which is measured in terms of the oscillations of the direction ∇uα /|∇uα | of the gradient of the image. This direction can be determined by passing to a dual formulation, as it essentially equals the rescaled dual variable. This idea of parameter adaptation based on the properties of the dual function is taken from [6].
Locally Adaptive Total Variation Regularization
333
The main concept of this paper of using a dual variable to provide a guess on the smoothness of the regularized image is introduced in Section 2. For further improving this smoothed image by enhancing the edges, we propose to subsequently apply anisotropic total variation regularization with an anisotropy that is estimated from the same dual variable that has determined the isotropic regularization parameter (see Section 3). A complete description of the algorithm can be found in Section 4. Finally, we apply this method in Section 5 to two test examples that show its suitability for adaptive noise removal.
2
Parameter Adaptation via Dual Variables
Consider the dual formulation of T (·; α), which consists in solving the constrained minimization problem 2 J (V ) := div V (x) + f (x) dx → min , Ω
|V (x)| ≤ α(x) almost everywhere on Ω , V (x) · ν(x) = 0 almost everywhere on ∂Ω ,
(3)
over the space of vector valued essentially bounded functions L∞ (Ω; IRn ). In (3), ν denotes the outward normal to the domain Ω, and the equation V · ν = 0 is understood in a distributional sense. Also, the divergence of an essentially bounded function is defined distributionally. To be precise, the functions V and div V satisfy the equation ∇φ(x) · V (x) dx = − φ(x) div V (x) dx Ω
Ω
for every φ ∈ C 1 (IR ). Minimization of Tα is equivalent to solving the dual problem (3) in the sense that a function Vα ∈ L∞ (Ω; IRn ) solves (3), if and only if uα := f + div Vα minimizes Tα . We refer to [4], which treats the dual formulation of total variation regularization, and to [7] for a detailed introduction to infinite dimensional convex analysis. We now examine the dual variable V more closely. Formally, the optimality condition for a minimizer uα of the functional T reads as ∇uα (x) for almost every x ∈ Ω . uα (x) − f (x) div α(x) |∇uα (x)| n
Since uα − f = div Vα , one sees that the dual minimizer Vα introduced above in fact coincides with the direction of the gradient of uα , multiplied by α(x). In particular, for almost every x ∈ Ω, we either have that |Vα (x)| = α(x) or the gradient of uα at x is zero, that is, uα is approximately constant near x. Even more, the local behaviour of Vα is strongly related to a certain kind of regularity of the regularized function uα : Large variations of Vα /α on the unit
334
M. Grasmair
sphere imply equally large variations of the direction of the gradient of uα . In other words, variations of Vα /α imply small oscillations of uα . The method we propose in the following takes advantage of these properties of Vα and uα and exploits their relation. Let r > 0 be some fixed parameter. We define the r-local mean of a vector valued, essentially bounded function W ∈ L∞ (Ω; IRn ) at x ∈ Ω by 1 Mr (x; W )(x) := − W (y) dy := n W (y) dy . L Br (x) ∩ Ω Br (x)∩Ω Br (x)∩Ω Here, Ln denotes the n-dimensional Lebesgue measure. In addition, we define the r-local variation of W by Σr (x; W )(x) := W (x) − Mr (x; W ) . (4) The definition of Σr directly implies that Σr (x; W ) ≤ 2 ess sup |W (y)| : y ∈ Br (x) ∩ Ω for almost every x ∈ Ω. Applying the above inequality to the scaled solution Wα (x) := Vα (x)/α(x) of (3), one immediately sees that 0 ≤ Σr (x; Wα ) ≤ 2 max |Vα (y)|/α(y) : y ∈ Br (x) ∩ Ω ≤ 2 . Moreover, the actual size of the value Σr (x; Wα ) provides an indication of the oscillation of the function uα near x: If Σr (x; Wα ) is close to zero, then the gradient ∇uα points in roughly the same direction on the whole set Br (x). Conversely, a value above one implies that the orientation of ∇uα (x) vastly differs from the majority of directions present in Br (x). See Figure 1 for an example of a smoothed image with corresponding local variation of the dual variable Vα . In this manner, the function Σr (x; Wα ) can serve as a local criterion for the smoothness of the regularized function uα . If the finally desired smoothness is not yet reached, that is, if Σr (x; Wα ) is too large, it is necessary to increase the local regularization parameter α(x). Conversely, if the function uα appears too smooth, that is, Σr (x; Wα ) is close to zero, then α(x) is decreased and a new tentative solution uα is computed. This process of computing Σr (x; Wα ) and updating α is repeated until the update of uα becomes small enough. In order to reach a uniform smoothness of the regularized image uα over its whole domain, we propose to prescribe some target smoothness 0 < θ < 1. Then one can compute a suitable update α ˜ of α setting s α(x) ˜ = α(x) θ + Σr (x; Wα )/2 (5) for some parameter s > 0 determining the size of the update. Iteration of this update will lead to a uniform smoothness Σr (x; Wα ) ≈ 2(1 − θ). The choice
Locally Adaptive Total Variation Regularization
335
Fig. 1. Smoothed image (left) and corresponding function Σr (right). Bright pixel values indicate a higher value of Σr .
of the target smoothness should reflect the properties of the image one wants to recover: A large parameter (θ ≥ 0.7) means that only the structures about the size of r are of interest. Small values (θ ≈ 0.55) put more emphasis on the structures of size smaller than r (see also Figure 4). In order to avoid too rapid changes of the parameter α(x), it is necessary smooth the update α ˜ computed by means of (5). Also from a theoretical point of view, this smoothing procedure is required for obtaining a continuous regularization function α. We propose to simply replace the update α ˜ (x) by its local mean value Mr (x; α). ˜ In this way, the average smoothness in the balls Br (x) will be almost independent of x.
3
Edge Enhancement by Anisotropy
Having determined the size of the local regularization parameter α(x) by means of the scaled dual variable Wα , it is in addition possible to use the distribution of the values of Wα on the unit sphere for sharpening edges and, in particular, thin ridges, which usually tend to get oversmoothed. To that end, instead of applying isotropic regularization, we introduce an anisotropy the direction of which is determined by the local covariance of Wα . For R > 0 we define the IRn×n -valued function CovR (x; W ), the covariance of W on BR (x) ∩ Ω, by defining its (i, j)-th component as (i) (i,j) (i) (j) W (y) − MR (x; W ) W (j) (y) − MR (x; W ) dy . CovR (x; W ) := − BR (x)∩Ω
(6) Again using the property that Wα is proportional to ∇uα , one sees that the principal component of CovR (x; Wα ) indicates, up to sign, the prevailing direction of ∇uα near x. This dominant direction can be pronounced further by replacing the isotropic bound |Vα (x)| ≤ α(x) in (3) by an anisotropic one defined by CovR (x; Wα ). This is achieved by minimizing J (V ) respecting the constraints V · ν = 0 on ∂Ω and
336
M. Grasmair
c(x) V (x)t CovR (x; Wα )V (x) ≤ 1
on Ω .
(7)
Here, the scalar valued function c : Ω → IR>0 has to be chosen in such a way that a similar amount of smoothing is reached as for isotropic regularization with parameter α(x). For determining a suitable size for c, note that the amount of smoothing induced by the bound (7) can be estimated by the determinant of the matrix c(x) CovR (x; Wα ), which, for consistency with the constraint |V (x)| ≤ α(x), should equal α(x)−2n . Thus one obtains for the function c the value −1/n c(x) = α(x)−2 det CovR (x; Wα ) . We therefore propose an edge enhancement via solving the minimization problem 2 J (V ) = div V (x) + f (x) dx → min , Ω
V (x)t A(x)V (x) ≤ 1 almost everywhere on Ω ,
(8)
V (x) · ν(x) = 0 almost everywhere on ∂Ω . Here
−1/n CovR (x; Wα ) , A(x) = α(x)−2 det CovR (x; Wα )
and Wα = Vα /α, where Vα is the solution of (3). Denoting the solution of (8) by VA and defining uA := f + div VA , we obtain an enhanced version of the isotropic total variation minimizer uα .
4
Summary of the Algorithm
We now summarize the method developed in the previous sections for adaptive denoising of a noisy image f ∈ L2 (Ω). Algorithm 1. Set k = 1, choose some initial regularization function α1 : Ω → IR>0 , a smoothness parameter 0 < θ < 1, some r > 0, R > 0, s > 0, and ε > 0. 1. Compute
Vk := arg min J (V ) : |V (x)| ≤ αk (x) on Ω, V · ν = 0 on ∂Ω . 2. Define Wk := Vk /αk and compute Σr (x; Wk ) (see (4)). 3. If Vk − Vk−1 < ε go to 5. 4. Compute s α ˆ k+1 (x) := αk (x) θ + Σr (x; Wα )/2 and ˆ k+1 ) , αk+1 (x) := Mr (x; α increase k by one, and go to 1.
Locally Adaptive Total Variation Regularization
337
5. Compute CovR (x; Wk ) (see (6)) and −1/n CovR (x; Wk ) . A(x) := α(x)−2 det CovR (x; Wk ) 6. Compute
VA := arg min J (V ) : V (x)t A(x)V (x) ≤ 1 on Ω, V · ν = 0 on ∂Ω .
Define the regularized function uA := f + div VA . In steps 1–4, only the regularization function α is determined. For this, it is not necessary to compute the minimizers of J precisely. Instead, a reasonable approximation of a minimizer is sufficient to provide a good update of α, at least during the first iterations. In particular if an iterative method is used for the minimization of J , the computation time can be improved by stopping the iteration well before convergence is reached. In the numerical examples below, the functions Vk and VA were computed by alternating between gradient of J and descent steps for the minimization t projections of V on the sets V : |V (x)| ≤ α (x) and V : V (x) A(x)V (x) ≤ k 1 , respectively. The function Vk−1 was used as initial guess for the computation of Vk .
5
Examples
The algorithm presented in Section 4 is tested by means of two images. The first, synthetic image shows a collection of ellipses and rectangles of different size and intensity (see Figure 2, upper left). These clean data were distorted by normally distributed random noise. In order to illustrate the capability of the algorithm for dealing with varying noise level, the standard deviation of the random noise was chosen to increase towards the right bottom of the image from about 10% of the maximal intensities to 150% (see Figure 2, lower left). The original image only consisting of simple geometric forms without any texture, it should be perfectly suited for total variation regularization. The changing noise level within the distorted data, however, makes a uniform parameter choice almost impossible: If the regularization parameter is chosen too small, then the noise on the right hand side of the data is barely removed. In particular, the right hand edges of the lower ellipses can hardly be recovered. On the other hand, a too large regularization parameter leads to the disappearance of the small circle at the left hand side of the image (see Figure 2, middle column). Only a very small range of parameters removes the noise reasonably well while still preserving the small scale structure—and even then the contrast deteriorates. Figure 2, upper right, shows the smoothed image obtained with Algorithm 1. Since the original image is very smooth, the smoothness parameter was chosen rather large as θ = 0.85. The variance Σr was evaluated on balls with a radius of 3 pixels, the complete image measuring 256 × 256 pixels. The lower right image in Figure 2 shows the distribution of the finally chosen regularization function α. As expected, it increases to the right bottom, where more noise is present. Over the whole image, the maxima and minima of α differ by a factor of 12.
338
M. Grasmair
Fig. 2. Left column: original and noisy image; the noise level increases to the right bottom of the image. Middle column: denoising without parameter adaptation; either small details are lost or the smoothing effects are partially insufficient. Right column, upper row: denoised image for smoothness parameter θ = 0.85. Right column, lower row: logarithm of the finally chosen regularization function α; the minima and maxima of α differ by a factor of 12.
One can see in the resulting image that the noise is efficiently removed. Also, the shape of the two lower ellipses is reconstructed in a reasonable way, considering that rather more noise than signal is present in these regions. Moreover, the small circle on the left is clearly visible, though some contrast was lost. As a second test example, we consider the photographer image. In a first experiment we add different levels of random noise (see Figure 3). The outcome of the adaptive Algorithm 1 (right column) is compared with the solution of standard total variation regularization with constant parameter choice independent of the noise level (middle column). The smoothness parameter for the adaptive algorithm was chosen as θ = 0.60; the regularization parameter for the standard algorithm was selected in such a way that the results for moderate noise level (third row) are comparable. The results show that, as expected, a constant regularization parameter only yields good results for a very specific noise level. For stronger noise, almost no smoothing is obtained, whereas the image is oversmoothed in case it is already quite clean. In contrast, the adaptive algorithm yields comparable results for different noise levels, and is also able to treat noise-free images (first row). In order to illustrate the effect of the smoothness parameter, we apply Algorithm 1 to the noise-free photographer image and vary θ (see Figure 4). For a value of θ = 0.55 mainly the grass and details of the camera are smoothed. As
Locally Adaptive Total Variation Regularization
339
Fig. 3. Left column: image with Gaussian noise; the noise level increases with each row (σ = 0, 30, 50, 100). Middle column: total variation denoised image with constant parameter choice. The regularization parameter is kept the same for all images. Right column: denoised images with adaptive parameter choice for a smoothness parameter θ = 0.60.
θ increases, more and more details are lost until only the large scale structures in the image remain. Thus, the smoothness parameter works in some sense like the regularization parameter of standard total variation regularization.
340
M. Grasmair
Fig. 4. Influence of the smoothness parameter θ. First row: original image and smoothed images with θ = 0.55 and θ = 0.60. Second row: smoothed images with θ = 0.65, θ = 0.70, and θ = 0.80. Table 1. Comparison between standard TV regularization, the method proposed in [11], and our method for different smoothness parameters. The table provides signal to noise ratios for the photographer image with various levels of Gaussian noise added (σ = 20, 30, 40, 50). original uniform TV adaptive ( [11]) θ = 0.55 θ = 0.60 θ = 0.65 9.47 6.31 3.86 1.93
14.63 12.13 10.05 8.38
15.63 13.35 11.59 10.11
15.30 13.28 11.47 10.00
14.73 13.46 12.45 11.54
13.71 12.84 12.18 11.51
There is, however, a notable difference. In the standard method, the time when structures in the image disappear depends on their scale, which is basically the ratio between contrast, that is, the difference of the intensities of the structure and the background, and the perimeter of the structure. As opposed to this, the model presented here puts much less emphasis on the contrast. Low contrast but distinct parts of the image tend to disappear much later than with uniform regularization. Compare for instance the rightmost building in the images of Figure 4 with the outcome of the standard method (Figure 3, first row, middle image). Finally, Table 1 compares the performance of our algorithm with uniform total variation regularization and the adaptive method from [11]. The regularization parameter for the comparison was chosen in such a way that the norm of the
Locally Adaptive Total Variation Regularization
341
residual equals the norm of the noise. At small noise levels, the texture enhancing method [11] and even uniform regularization perform better. On the other hand, our algorithm provides good results if much noise is present. Note moreover that the here proposed method does not require a guess on the noise level, whereas the other methods do.
6
Conclusion
We have introduced an algorithm for the local adaptation of the regularization parameter in total variation regularization applied to the task of image denoising. The main idea of the method is to base the parameter choice on the smoothness of the output image, which is measured in terms of the variation of the direction of its gradient. This variation can be obtained when employing a dual method for the actual minimization of the total variation regularization functional. Starting from an initial guess of the regularization function, the proposed algorithm consecutively computes the corresponding minimizer of the total variation functional and updates the regularization function depending on the smoothness of the update. The iteration stops when the update is sufficiently small. As a post-processing step, we propose to apply an anisotropic regularization method intended to sharpen edges. Again, the regularization is determined by the dual variable. This anisotropic regularization step reduces the contrast loss due to isotropic smoothing and, in particular, is suited for the enhancement of ridges. The examples presented in Section 5 indicate the suitability of the proposed method for denoising images with unknown, varying noise levels. In particular, they show its ability to provide an estimate for the amount of smoothing required to obtain a certain smoothness of the output.
Acknowledgement This work has been supported by the Austrian Science Fund (FWF) within the national research network Industrial Geometry, project 9203-N12.
References 1. Acar, R., Vogel, C.R.: Analysis of bounded variation penalty methods for ill-posed problems. Inverse Probl. 10(6), 1217–1229 (1994) 2. Aubert, G., Kornprobst, P.: Mathematical problems in image processing. In: Partial differential equations and the calculus of variations, With a foreword by Olivier Faugeras, 2nd edn. Applied Mathematical Sciences, vol. 147. Springer, New York (2006) 3. Burger, M., Osher, S.: Convergence rates of convex variational regularization. Inverse Probl. 20(5), 1411–1421 (2004) 4. Chambolle, A.: An algorithm for total variation minimization and applications. J. Math. Imaging Vision 20(1–2), 89–97 (2004)
342
M. Grasmair
5. Chambolle, A., Lions, P.-L.: Image recovery via total variation minimization and related problems. Numer. Math. 76(2), 167–188 (1997) 6. Davies, P.L., Kovac, A.: Local extremes, runs, strings and multiresolution. Ann. Statist. 29(1), 1–65 (2001) 7. Ekeland, I., Temam, R.: Convex Analysis and Variational Problems. NorthHolland, Amsterdam (1976) 8. Engl, H.W., Hanke, M., Neubauer, A.: Regularization of inverse problems. Mathematics and its Applications, vol. 375. Kluwer Academic Publishers Group, Dordrecht (1996) 9. Frigaard, I.A., Ngwa, G., Scherzer, O.: On effective stopping time selection for visco-plastic nonlinear BV diffusion filters used in image denoising. SIAM J. Appl. Math. 63(6), 1911–1934 (electronic) (2003) 10. Frigaard, I.A., Scherzer, O.: Herschel–Bulkley diffusion filtering: non-Newtonian fluid mechanics in image processing. Z. Angew. Math. Mech. 86(6), 474–494 (2006) 11. Gilboa, G., Sochen, N., Zeevi, Y.Y.: Variational denoising of partly-textured images by spatially varying constraints. IEEE Trans. Image Process. 15(8), 2281–2289 (2006) 12. Ito, K., Kunisch, K.: Augmented Lagrangian methods for nonsmooth, convex optimization in Hilbert spaces. Nonlinear Anal. 41A, 591–616 (2000) 13. Nashed, M.Z., Scherzer, O.: Least squares and bounded variation regularization with nondifferentiable functional. Numer. Funct. Anal. Optim. 19(7-8), 873–901 (1998) 14. Rudin, L.I., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Phys. D 60(1–4), 259–268 (1992) 15. Scherzer, O., Grasmair, M., Grossauer, H., Haltmeier, M., Lenzen, F.: Variational Methods in Imaging. Applied Mathematical Sciences, vol. 167. Springer, New York (2008) 16. Strong, D.M.: Adaptive Total Variation Minimizing Image Restoration. CAM Report 97-38, University of California, Los Angeles (1997) 17. Strong, D.M., Aujol, J.-F., Chan, T.F.: Scale recognition, regularization parameter selection, and Meyer’s G norm in total variation regularization. Multiscale Model. Simul. 5(1), 273–303 (electronic) (2006)
Basic Image Features (BIFs) Arising from Approximate Symmetry Type Lewis D. Griffin1, Martin Lillholm1, Mike Crosier1, and Justus van Sande2 2
1 Computer Science, University College London, London WC1E 6BT, UK Biomedical Engineering, Eindhoven University of Technology, The Netherlands [email protected]
Abstract. We consider detection of local image symmetry using linear filters. We prove a simple criterion for determining if a filter is sensitive to a group of symmetries. We show that derivative-of-Gaussian (DtG) filters are excellent at detecting local image symmetry. Building on this, we propose a very simple algorithm that, based on the responses of a bank of six DtG filters, classifies each location of an image into one of seven Basic Image Features (BIFs). This effectively and efficiently realizes Marr’s proposal for an image primal sketch. We summarize results on the use of BIFs for texture classification, object category detection, and pixel classification. Keywords: Gaussian Derivatives, Hermite Transform, Group Theory.
1 Introduction Previous schemes for detection of image symmetry are fairly complex [1-6]; requiring, for example, comparison of the outputs of filters at multiple positions. Herein we show that symmetries may be detected by single linear filters. Building on this we present a simple algorithm that computes a Marr-type primal sketch [7] by categorizing local image structure according to its approximate symmetry. The paper is organized as follows. In section 2 we present results on image symmetries. In 3 we show how to test whether a linear filter is sensitive to a symmetry. In 4 we review image measurement with derivative-of-Gaussian (DtG) filters. In 5 we consider the symmetry-sensitivity of these DtG filters. In 6 we show how this sensitivity gives rise to a system of Basic Image Features (BIFs). In 7 we summarize results on using BIFs for texture categorization, object category detection and pixel classification. In 8 we conclude. Sections 2-5 are a distillation of work published, in press and under review in fuller form elsewhere [8-14]; 6 is new; parts of 7 have been presented or are under review in fuller form elsewhere [9, 11].
2 Image Symmetries Symmetry of a structure (X) is always relative to some class of admissible transformations. A structure is said to have a symmetry when a non-trivial group of X.-C. Tai et al. (Eds.): SSVM 2009, LNCS 5567, pp. 343–355, 2009. © Springer-Verlag Berlin Heidelberg 2009
344
L.D. Griffin et al.
admissible transformations, known as the automorphism group, each leave it indistinguishable from the original. This is denoted by Aut [ X ] := {t | t D X = X } .
Considering images, an obvious class of transformations are the spatial isometries; and the possible symmetries, relative to this class, have long been catalogued [15-17]. A broader class of transformations, where each spatial isometry is combined with a permutation of a finite set of image ‘colour’ values, has also been considered. These allow the symmetries of, for example, Escher’s ‘Reptiles’ to be expressed [18]. The gamut of possible ‘colour symmetries’ has been fully determined [19, 20]. We propose that the class of ‘image isometries’, defined as a spatial isometry combined with an intensity isometry, is appropriate for images. We write an image isometry as g = ( i, s ) , where i : \ → \ is an intensity isometry, and s : \ 2 → \ 2 is a spatial isometry. Such an image isometry is applied to an image I : \ 2 → \
(
)
according to g D I = i D I ( s D _ ) = i I ( s ( _ ) ) . Choosing a class of transformations is tantamount to choosing a geometry [21], and the geometry that corresponds to the class of image isometries has previously been considered for images [22] and much earlier, abstractly, as one of a larger class of possible geometries [23]. We have employed a method for determining the possible automorphism groups of images, relative to the class of image isometries. The method relies on two results. First, that the projection of a group of image isometries onto their spatial or intensity components in both cases makes a group. Second, that (except for a special case) the intensity projection group must be isomorphic to a factor group of the spatial projection group [8]. Using the method, we have determined the possible automorphism groups of 2-D images, except for cases that contain discrete periodic translations. A summary of these possible symmetries, together with our notational system is shown in fig. 1. The symmetries include: familiar ones, such as reflectional (J2,1), reflect-and-negate (J6,1 ), and Yin-Yang type (J7,2); simple but often ignored ones, such as variation in one direction only (J3); simple but novel, such as continuous translate-and-increment in one direction, plus a line of reflection parallel to that direction (J11); and some wholly novel, such as continuous translate-and-increment in one direction, plus a continuous line of centres of Yin-Yang type symmetry (J12).
3 Sensitivity of Linear Filters to Symmetries Detection of a symmetry seems to require multiple measurements, but this is incorrect. Consider a +1/-1 filter, such as used in finite-difference schemes. When positioned so that it straddles a putative line of reflection, a necessary criterion for the symmetry is that the filter gives a 0 response. We generalizes this: a filter F is sensitive to a symmetry K if it gives the same response to all images that have the symmetry (i.e. ∃ f ∈ \ Aut [ I ] ⊇ K ⇒ F I = f ). This definition is impractical because it requires assessment across all images. However, we have found a necessary and sufficient test that requires only a single integral to be computed. We present this below in Theorem 1, after introducing some notation.
J6,3, J6,5, …
…
J6,2
J6,1
J1,4, J1,6, …
…
… J2,3, J2,5, …
…
J1,2
J2,4, J2,6, …
…
…
…
J0
J2,1
J7,4, J7,6, …
J8,4, J8,6, …
J2,2
J3
J4
J8,2
J7,2
J9
J11
Jslope
J10
Fig. 1. The group/subgroup lattice of the possible image symmetries, excluding those with discrete periodic translation
J1,3, J1,5, …
…
J6,4, J6,6, …
J5
Jconst
J12
Basic Image Features (BIFs) Arising from Approximate Symmetry Type 345
346
L.D. Griffin et al.
We use an inner product notation ( F I :=
G G ∫ F( x) I(x) )
to denote the
G x∈\ 2
measurement of an image I : \ 2 → \ by a filter F : \ 2 → \ ; and we define an K operator F ( ) := ∑ i s D F which, roughly speaking, ‘smears’ a filter by a group. ( i , s )∈K
Theorem: Symmetry-Sensitivity Test for Filters
F is sensitive to K if and only if F F ( K ) = 0 Proof A formal proof will be published elsewhere [14]. Intuitively the truth of the theorem can be understood as follows. The signal that a filter ‘sees’ best is a copy of itself. Of all the symmetric signals, a symmetrised version of the filter should be the most easily seen. If the filter cannot see a symmetrised version of itself, then it is insensitive to the symmetry.
4 Gaussian Derivative Filters Gaussian Derivative (DtG) filters are defined in 1-D by
Gσ ( x ) := ( 2πσ
)
1 2 −2
e
−
x2 2σ 2
, Gσ(
n)
( x ) :=
n
dn ⎛ −1 ⎞ ⎛ x ⎞ Gσ ( x ) = ⎜ ⎟ Hn ⎜ ⎟ Gσ ( x ) dx n 2 σ ⎝ ⎠ ⎝σ 2 ⎠
where Hn is the nth Hermite polynomial; and in 2-D by Gσ(
m, n )
( x, y ) := Gσ( m)( x ) Gσ( n)( y ) .
They are used as a general-purpose method to probe an image location (which for simplicity we assume is at the origin 0 ) by computation of inner products jmn = Gσ( m, n ) I . Typically, one measures with a family of DtG filters up to some order e.g. the 2nd
{
}
order family Gσ( m, n ) | 0 ≤ m + n ≤ 2 . Scale-normalized filter responses c pq := σ p + q j pq make later equations simpler. The suitability of DtGs as the front-end of an uncommitted computational vision system arises from the symmetries that individual filters and families possess [24]. First amongst these is a scale symmetry, which manifests as a change of size, but not of shape, when a DtG is rescaled by blurring with a Gaussian kernel. Second is that the linear span of a family of DtGs is rotationally symmetric. The responses of a bank of DtG filters entangle intrinsic and extrinsic aspects of image structure. For example, an in-plane rotation of the image about the measurement point causes the DtG responses to change. A representation that
Basic Image Features (BIFs) Arising from Approximate Symmetry Type
347
disentangles these aspects for measurement up to 2nd order has been developed [13]. The representation works by factoring out of the 6-D 2nd order DtG response space the changes due to the group of image isometries that fix the measurement point and do not invert the intensity axis which we denote D∞ ( 0 ) × A+ (1) . 12.1
=
-2.3
=
-0.1
2nd order DtG family
image patch
0.8
4.1
-3.7
point in 6-D jet space
6-D jet space D∞ ( 0 ) × A+ (1) - The group of centred rotations and reflections, and positive affine intensity re-scalings
= The 2nd order local-image-structure orbifold
Fig. 2. The top part illustrates schematically the probing of an image patch by a bank of DtG filters resulting in a point in jet space; the bottom, the factoring of the jet space by a group of transformations resulting in the local-image-structure orbifold
The result is an orbifold – a type of manifold with boundaries, creases and corners allowed – consisting of a 3-D and a 0-D component (figure 2). The intrinsic aspect of a 6-tuple of filter responses corresponds to a particular location in the orbifold, and is invariant to rotating the image about the measurement point, reflecting it in a line through the measurement point, or affinely scaling the intensity. When the responses of the 1st and 2nd order DtG filters are all zero, the intrinsic aspect is the 0-D part of the orbifold; all other responses map to the 3-D component. A coordinate system ( l , b, a ) ∈ ⎡⎣ − π2 , π2 ⎤⎦ × ⎡⎣ 0, π2 ⎤⎦ × ⎡⎣0, π2 ⎤⎦ for the 3-D component is given by [13]:
(
)
2 ⎛ ⎞ 2 2 2 l = arctan ⎜ 4 c10 + c01 + ( c20 − c02 ) + 4 c11 , c20 + c02 ⎟ ⎝ ⎠
⎛ 2 2 b = arctan ⎜ 2 c10 + c01 , ⎝
a=
1 2
((
( c20 − c02 )2 + 4 c112 ⎞⎟ ⎠
((
)
)
2 2 arctan c01 − c10 ( c02 − c20 ) + 4c10 c01 c11 , 2 c012 − c102 c11 + c10 c01 ( c20 − c02 )
))
The orbifold has been equipped with a metric, induced by one on the filter response space, which expressed as a line element in the lba-system is
(
−1
)
ds 2 = dl 2 + cos 2 l db 2 + da 2 2 ( 5 − 3cos2b ) sin 2 2b . The orbifold is intrinsically curved, but it can be embedded into Euclidean 3-space with only mild distortion.
348
L.D. Griffin et al.
5 Symmetry-Sensitivity of DtG Filters Using the elements of sections 2-4, we can determine which DtG filters are sensitive to which symmetries. We consider not just canonical filter forms (e.g. an x-derivative) but any linear combination of filters in the 2nd order filter family. This allows us to determine the symmetry-sensitivity of the entire filter family, independent of the particular basis filters used. For example, while the x-derivative filter is sensitive to a reflectional symmetry with a vertical mirror line through the measurement point, the x- and y-derivatives together are sensitive to any reflectional symmetry in a line through the measurement point. J 6,c {3, J 7,c {6, c J 8,4
J1,c{3,
}
J1,c{4, J
c 2,3 +
c J1,2
J
}
J
c 5
J 6,c {6, } J 7,c {12, } J 8,c {6, } J const
} }
c J 6,2 c J 7,4
a 4
J11a
c J 2,2
a J 2,1
a J 2,2 + a+ J 8,2 +
J3
J10
J11g J12g
c J8,2 J 9a
c J 7,2
c J 8,4
J12a J slope
J 4g J 9g
a J 6,1
a J 6,2 + a− J 8,2 +
J 5a
SS is the exterior only
J0
g g J1,2 + J 2,1+ g g J 6,1 + J 7,2+ g J 8,2 +
SS is the entire volume
Fig. 3. The sensitivity-submanifolds (SS) of different symmetry types are shown in red. The different possible SS are arranged in a lattice induced by inclusion relations. The symmetry type labels correspond to those used in figure 1. Superscripts indicate the spatial relationship between the symmetry and the origin: a c indicates origin-centred rotation; an a+ that the origin is contained in a line of reflection, but is not a centre of rotation; similarly for a- and anti-reflections; a g indicates general position, neither centred nor aligned. All symmetries labelled in a box have the indicated SS; those on the left are minimal.
The filter family sensitivities can be projected into the orbifold to determine where the intrinsic component of the jet responses must lie whenever the image has any of a class of symmetries equivalent by conjugation with an element of D∞ ( 0 ) × A+ (1) . We
Basic Image Features (BIFs) Arising from Approximate Symmetry Type
349
call the restricted set of possible responses the sensitivity-submanifold (SS). For example the SS is the orbifold exterior ( a = 0 ∧ a = π 2 ) for reflectional symmetry in a mirror through the measurement point. The results are summarized in fig 3.
6 Symmetry-Based Basic Image Features (BIFs) We have used the symmetry sensitivities of the DtG filters as a starting point in defining a set of Basic Image Features (BIFs) that realize Marr’s idea of a primal sketch of image structure, in a computationally simple scheme. We do not claim that the scheme is derived as rigorously as the results on symmetry sensitivity. Our scheme works by considering the orbifold projection of jets, and classifying them according to the SS that they are closest to i.e. we define a Voronoi cell partitioning of the orbifold with the SS as cell centres. We find that this works best when only seven 0-D SS (the first and second rows of figure 3) are used, though we cannot justify this beyond that it produces nice results. The resulting orbifold partitioning is shown in the top-left of figure 4.
Fig. 4. Top left: the partitioning of the orbifold into BIF categories. Bottom left: BIFs calculated across a range of scales for a simple image of a figure ‘8’; in each cube scale increases right-to-left. Lower cubes sectioned for visualisation. Right: an example complex greyscale image, with BIFs calculated at one particular scale.
The orbifold distance to the six of these SS that lie in the 3-D component of the c orbifold are simple to compute; for example, the distance to the J 7,2 SS is tan −1
(
1 c2 2 20
2 2 + c11 + 12 c02
) (c
2 10
)
2 + c01 . To find which distance is shortest it is
350
L.D. Griffin et al.
computationally equivalent but simpler to find which of six quantities is maximum. The distance to the seventh SS, which corresponds to the origin of jet space where all the 1st and 2nd derivative filters have zero response, is not well-defined. We incorporate it into our scheme by using a multiple of the 0th order jet response. The full resulting scheme for computing BIFs is as follows. i) compute scale-normalized DtG filter responses as described in section 7. ii) compute λ :=
1 2
( c20 + c02 )
and γ :=
1 4
( c20 − c02 )2 + c112
iii)classify according to which is the largest of
{
M = ε . c00 ,
2 2 c10 + c01 , λ , − λ , (γ + λ )
2 , (γ − λ )
}
2, γ .
In our scheme the only free parameters, that have to be tuned to the application are the filter scale σ and ε which controls the amount of image classified as flat; a setting of ε = 0.05 is an effective default. For display purposes we find the following colour scheme effective: if ε .c00 is the largest of M then colour the pixel pink; if 2 2 c10 + c01 is largest colour it grey; then black, white, blue, yellow and green.
7 Example Applications Using BIFs We summarize results on using BIFs for texture, object and pixel classification. 7.1 Texture Classification
Textures are often classified based on a representation of them by a histogram over a texton vocabulary [25-29]. Textons are categorical patch classifications [25, 30]. To define the texton vocabulary, a space of patch descriptions is typically Voronoi partitioned into on-the-order-of 1000 texton categories, usually around centres found by k-means clustering of the responses from many images. Textures are then classified by nearest-neighbour matching of histograms. We have investigated the classification performance of an approach in which images are labelled using spatial complexes of BIFs instead of Voronoi cells in a local description vector space. Our approach is (i) simpler because we have eliminated the clustering step needed to produce a dictionary of features, and (ii) faster because we assign image patches to histogram bins without having to use a high-dimensional nearest-neighbour computation. We call the spatial complexes of BIFs that we use analogously to textons, Basic Image Patterns (BIPs). The type of BIP that we have found effective for texture description is a scale-template of the BIFs at the same location but at four, octave-separated scales. Unlike spatial-template BIPs, these scale-templates retain the rotation invariance of BIFs, which has been shown [30] to be advantageous in texture classification tasks. For textures, we do not use the pink/flat BIF category, so four scales produces a 64=1296 bin histogram representation, which seems to capture the right trade-off between specificity and generality (see figure 5).
Basic Image Features (BIFs) Arising from Approximate Symmetry Type
351
Fig. 5. Left: An image from the CUReT 'polyester' texture class. Centre: BIFs computed at four octave-separated scales, stacked to form an array of 'column-BIPs'. Right: Occurrence histogram of column-BIPs from every position in the image".
Our method has been tested on the CUReT texture dataset [31]. As reported in [9], the simple column-BIP representation and nearest-neighbour matching using the Bhattacharyya distance correctly classifies 98.2±0.1% of the remaining 49 images per class, which is at least as good as other methods using nearest-neighbour classifiers. Extending this method by using a multi-scale histogram comparison [9] results in an improvement to 98.6±0.1% on CUReT, which is comparable to methods [27] using SVMs for classification; and produces what are, to the best of our knowledge, the best reported results [9] on the more challenging UIUCTex [32] and KTH-TIPS [27] datasets, which include variations in scale. 7.2 Object Categorization
Texton approaches have also been shown to be useful for object categorization [28, 33]. Similar to texture, the ‘standard’ approach is to partition a patch descriptor space, such as that used by SIFT [34, 35], into on-the-order-of 1000 categories (visual words) and then to describe each image to be analyzed by what visual words it contains, and to use machine learning techniques to determine a classifier that can predict the category of object based on such descriptions. We have conducted preliminary experiments to assess whether visual words built from BIFs could be used rather than SIFT-space categories. As with texture this would be simpler and faster. For our initial experiments, we have labelled pixels according to their BIF type and, inspired by SIFT, with an orientation, quantized at the π 4 level. The orientation depends on the BIF type: grey BIFs have one of eight possible orientations based on 1st order structure; yellow, green and blue BIFs have one of four possible orientations based on 2nd order structure; black, white and pink BIFs are unoriented. Thus we have twenty-three possible orientation-augmented BIF (oBIF) labels. oBIFs are a natural 2nd order generalization of the gradient orientation alphabet typically used in SIFT [34, 35]. See figure 6 for an example image and calculated oBIFs.
352
L.D. Griffin et al.
Fig. 6. The top row (left) shows an image from the PASCAL challenge, labeled with direction-augmented BIFs at right. On the bottom are shown the 4ä4 template BIPs whose occurrence in an image most informatively signals the presence of a car.
We have tested three different types of visual word, which when built from BIFs or oBIFs we call Basic Image Patterns (BIPs); two based on geometrical partitioning of patch space and one based on more standard data-driven quantization. Each BIP system has been used with simple un-optimised of-the-shelf classifiers and applied to the 20-class PASCAL VOC 2008 [33] object recognition challenge dataset. Our score in figure 7 is based on a late fusion of the three schemes and is mid-field: above other first-time entrants and below well-optimised veteran entries. Using the PASCAL VOC 2008 [33] dataset, examples of 4x4 template BIPs whose presence in images is approximately independent, and which are maximally informative for the ‘car’ category are shown in figure 6. SurreyUv A_SRKDA UvA_TreeSFS LEAR_shotgun Uv A_ FullSFS Uv A_ Soft5ColorSift LEAR_ flat XRCE TKK_ ALL_ SFBS TKK_MAXVAL BerlinFIRSTNikon UCL ECPLIAMA CASIA_ LinSVM INRIASaclay_ CMA CASIA_ NonLinSVM INRIASaclay_ MEVO FIRST_ SCST FIRST_ SC1C CASIA_ NeuralNet 0
10
20
30
40
50
Fig. 7. Results for the PASCAL VOC 2008 challenge. Each bar in the chart is a challenge entry - our result is highlighted.
Basic Image Features (BIFs) Arising from Approximate Symmetry Type
353
7.3 Pixel Classification
Many image problems involve inferring one of a small class of labels for each pixel of an image. For complex images with unpredictable global structure, most approaches balance the likelihood of the labels, given the local image structure, and the likelihood of the local arrangement of inferred labels. In both cases the likelihoods are computed on the basis of statistics learnt from groundtruth-labelled training data. We have experimented with the use of BIFs in the computation of label likelihoods given the image i.e. ignoring the likelihood of arrangements of inferred labels. For our experiments we have used 2-D Electron Microscopy images of neuronal grey matter tissue, stained to enhance neuronal membranes. We trained on four images with hand-drawn groundtruth data, indicating the position of membranes, and evaluated on a further four images. We use a k-Nearest Neighbour (k-NN) approach to classification. NN classification starts by compiling a list of descriptors of all the patches in the training data, together with the groundtruth label of the pixel in the patch centre. The classifier is used by extracting a patch around each pixel to be classified, forming a description of it, comparing the description to each the compiled descriptions, finding the k which are most similar, and assigning the pixel being analyzed with the label associated with the majority of the k. We evaluated a baseline solution based on pixel values. The distance between two patch descriptions is simply the Euclidean distance between their blurred pixel values, minimized over allowing one patch to be rotated or reflected into eight configurations. We jointly optimize blur, patch size and k. The best settings that we find are: no blur, 7ä7 patches, and k=14. At these settings membrane-labelled pixels overlap (intersection divided by union) with the groundtruth by 48%. Our solution uses a patch of BIF labels as a patch descriptor. The distance between two descriptors is simply the number of pixels where the label does not agree. As in the baseline, we minimize the distance over one of the patches being rotated or reflected. We jointly optimize the scale ( σ ) at which the BIFs are computed, the parameter ε which controls the amount of the flat BIF class, patch size and k. The best settings that we find are σ = 1.2 , ε = 0.15 , 9ä9 patches, and k=10. At these settings we achieve an overlap of 55%. See figure 8. image
groundtruth
greylevel-based classification
BIF-based classification
BIFs
Fig. 8. Typical results of our pixel-classification system
So, using BIF- rather than greylevel-description raises the score from 48% to 55%. Computation is also faster because the kNN lookup dominates the cost of computing patch descriptions, and with BIFs the distances that need to be computed are of a Hamming rather than Euclidean type.
354
L.D. Griffin et al.
8 Conclusions We have derived a scheme for classifying image structure into one of seven BIF types based on the outputs of a bank of six DtG filters. Applied to an entire image, the output realizes Marr’s notion of an image primal sketch. Presented results show that BIF description is simple, fast and effective for texture, object and pixel classification. The BIF system was derived by considering the sensitivity of DtG filters to image symmetry. Although the final algorithm is pleasingly simple, there are some weak points in the derivation of the BIFs from symmetry sensitivities. Specifically, why are only the 0-D SS considered, how exactly does orbifold distance correspond to degree of failure of symmetry, why should least-approximate local symmetry be an effective feature label? We hope that the foundation of symmetry-sensitivity of DtGs can eventually answer all of these questions in a scheme where arbitrary choice has been eliminated. Such a scheme will be extendable to higher-order filter families (where appeal to visual evidence and past practice are less effective), for which a richer alphabet of feature labels is to be expected. We predict that such a richer alphabet will give more effective solutions in the application areas that we have reviewed.
References 1. Liu, Y.X., Collins, R.T., Tsin, Y.H.: A computational model for periodic pattern perception based on frieze and wallpaper groups. IEEE Transactions on Pattern Analysis and Machine Intelligence 26(3), 354–371 (2004) 2. Scognamillo, R., et al.: A feature-based model of symmetry detection. Proceedings of the Royal Society of London Series B-Biological Sciences 270(1525), 1727–1733 (2003) 3. Mellor, M., Brady, M.: A new technique for local symmetry estimation. In: Kimmel, R., Sochen, N.A., Weickert, J. (eds.) Scale-Space 2005. LNCS, vol. 3459, pp. 38–49. Springer, Heidelberg (2005) 4. Bonneh, Y., Reisfeld, D., Yeshurun, Y.: Quantification of local symmetry - application to texture-discrimination. Spatial Vision 8(4), 515–530 (1994) 5. Mancini, S., Sally, S.L., Gurnsey, R.: Detection of symmetry and anti-symmetry. Vision Research 45(16), 2145–2160 (2005) 6. Baylis, G.C., Driver, J.: Perception of symmetry and repetition within and across visual shapes: Part-descriptions and object-based attention. Visual Cognition 8(2), 163–196 (2001) 7. Marr, D.: Vision. W H Freeman & co., New York (1982) 8. Griffin, L.D.: Symmetries of 1-D Images. Journal of Mathematical Imaging and Vision 31(2-3), 157–164 (2008) 9. Crosier, M., Griffin, L.D.: Texture classification with a dictionary of basic image features. In: CVPR 2008. IEEE, Los Alamitos (2008) 10. Lillholm, M., Griffin, L.D.: Statistics and category systems for the shape index descriptor of local image. Image and Vision Computing (in press) (2008) 11. Lillholm, M., Griffin, L.D.: Novel image feature alphabets for object recognition. In: ICPR 2008 (2008) 12. Griffin, L.D.: Symmetries of 2D images: cases without periodic translation. Journal of Mathematical Imaging and Vision (in press)
Basic Image Features (BIFs) Arising from Approximate Symmetry Type
355
13. Griffin, L.D.: The 2nd order local-image-structure solid. IEEE Transactions on Pattern Analysis and Machine Intelligence 29(8), 1355–1366 (2007) 14. Griffin, L.D., Lillholm, M.: Symmetry-sensitivity of derivative of gaussian filters. IEEE Transactions on Pattern Analysis and Machine Intelligence (in press) 15. Bieberbach, L.: Über die bewegungsgruppen der euklidischen raume I. Mathematische Annalen 70, 297 (1911) 16. Conway, J.H., et al.: On three-dimensional space groups. Contributions to Algebra and Geometry 42(2), 475–507 (2001) 17. Grünbaum, B., Shephard, G.C.: Tilings and Patterns. WH Freeman & co., New York (1987) 18. Schattschneider, D.: MC Escher. Visions of Symmetry. Plenum Press (1990) 19. Holser, W.T.: Classification of symmetry groups. Acta Crystallographica 14, 1236–1242 (1961) 20. Loeb, A.A.: Color and Symmetry. Robert E. Krieger (1978) 21. Klein, F.: A comparative review of recent researches in geometry (trans. by MW Haskell). Bulletin of the New York Mathematical Society 2, 215–249 (1892) 22. Koenderink, J.J., van Doorn, A.J.: Image processing done right. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002. LNCS, vol. 2350, pp. 158–172. Springer, Heidelberg (2002) 23. Cayley, A.: Sixth memoir upon the quantics. Philosophical Transactions of the Royal Society 149, 61–70 (1859) 24. Koenderink, J.J., van Doorn, A.J.: Generic Neighborhood Operators. IEEE Transactions on Pattern Analysis and Machine Intelligence 14(6), 597–605 (1992) 25. Varma, M., Zisserman, A.: Texture classification: are filter banks necessary? In: CVPR 2003. IEEE, Los Alamitos (2003) 26. Varma, M., Zisserman, A.: A statistical approach to texture classification from single images. International Journal of Computer Vision 62(1), 61–81 (2005) 27. Hayman, E., et al.: On the signifigance of real-world conditions for material classification. In: Pajdla, T., Matas, J(G.) (eds.) ECCV 2004. LNCS, vol. 3024, pp. 253–266. Springer, Heidelberg (2004) 28. Zhang, J., et al.: Local features and kernels for classification of texture and object categories: a comprehensive study. In: CVPR 2006 (2006) 29. Perronnin, F., et al.: Adapted vocabularies for generic visual categorization. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3954, pp. 464–475. Springer, Heidelberg (2006) 30. Varma, M., Zisserman, A.: Unifying Statistical Texture Classification Frameworks. Image and Vision Computing (in press) (2005) 31. Cula, O.G., Dana, K.J.: Compact representation of bidirectional texture functions. In: CVPR 2001. IEEE, Los Alamitos (2001) 32. Lazebnik, S.C., Schmid, C., Ponce, J.: A spare texture representation using local affine regions. IEEE Transactions on Pattern Analysis and Machine Intelligence 27(8), 1265– 1278 (2005) 33. Csurka, G., et al.: Visual categorization with a bag of keypoints. In: ECCV 2004, pp. 1–22 (2004) 34. Lowe, D.G.: Towards a computational model for object recognition in IT cortex. In: Biologically Motivated Computer Vision, Proceeding, pp. 20–31 (2000) 35. Lowe, D.G.: Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision 60(2), 91–110 (2004)
An Anisotropic Fourth-Order Partial Differential Equation for Noise Removal Mohammad Reza Hajiaboli Department of Electrical and Computer Engineering Concordia University, Montreal, Canada [email protected]
Abstract. Fourth-order nonlinear diffusion filters are isotropic filters in which the strength of diffusion at regions with strong image features such as regions with an edge or texture is reduced leading to their preservation. However, the optimal choice of parameter in the numerical solver of these filters for having a minimal distortion of the image features results in a very slow convergence rate and formation of speckle noise on the denoised image especially when the noise level is moderately high. In this paper, a new fourth-order nonlinear diffusion filter is introduced, which have an anisotropic behavior on the image features. In the proposed filter, it is shown that a suitable design of a set of diffusivity functions to unevenly control the diffusion on the directions of level set and gradient leads to a fast convergent filter with a good edge preservation capability. The comparison of the results obtained by the proposed filter with that of the classical and recently developed techniques shows that the proposed method produces a noticeable improvement in the quality of denoised images evaluated subjectively and quantitatively as well as a substantial increment of the convergence rate comparing to the classical filter.
1
Introduction
Nonlinear diffusion denoising filters are known for their good edge preservation capabilities. In these techniques, the denoised image is a solution of a partial differential equation (PDE). The first kind of these denoising methods is introduced by Perona and Malik [1] in 1990 based on solving a nonlinear second-order PDE (i.e. the so-called Perona-Malik equation). Since then, there has been a great deal of research in this filed which led to introduction of variety of nonlinear diffusion denoising techniques (see [2], [3] as a few examples). In spite of the good edge preservation obtained by these techniques, these methods tend to produce blocky effects in the images [4]. In fact, the solution of Perona-Malik equation is a piecewise constant solution, therefore these filters create blocky effects on the smooth regions of the image. A spatially regularized version of the nonlinear diffusion filter has been introduced by Catte et al. [2] to reduce the formation of the these artifacts on the denoised image. You and Kaveh [4] proposed a more effective solution to this problem by using a fourthorder PDE for noise removal, where a planar approximation of the noisy image X.-C. Tai et al. (Eds.): SSVM 2009, LNCS 5567, pp. 356–367, 2009. c Springer-Verlag Berlin Heidelberg 2009
An Anisotropic Fourth-Order Diffusion Filter
357
is supported in the solution of the PDE resulting in a significant improvement of the ramp edge preservation and a dramatic reduction of blocky effects. Based on this idea a variety of the fourth-order PDE based denoising techniques have been developed such as the filters given in [5], [6], and [7]. However, the fourthorder diffusion filters damp high spatial frequency components (i.e. noise and step edges) much faster than the second-order ones [5]. This feature can result in distorting of the step edges during the evolutionary process of the image denoising especially when smoothing strength of the filter for the detected edges is not effectively reduced by a diffusivity function. Setting a small threshold value in the diffusivity function substantially reduces the diffusivity on the edges with the expense of a very slow convergence rate, as reported in [4] and [5]. All of previously mentioned techniques belong to a class of diffusion-based denoising filters known as isotropic nonlinear diffusion denoising methods. It means that total amount of the diffusion controlled by the diffusivity function is applied on the different regions of the image regardless of the direction of the image features. To improve edge preservation of these filters, the other class of diffusion-based denoising techniques have been emerged in which the diffusion is adapted to the direction of the local image features [8], [9] and [10]. It means that the filter minimizes the diffusion strength on the direction perpendicular to the direction of local features and maximizes it in the direction of the local features. However, these techniques have been developed in the context of the second-order diffusion filters. In this paper, an anisotropic fourth-order diffusion filter is proposed in which the diffusion strength is adjusted respecting the direction of the local features. Two different diffusivity functions are designed to extremely minimize the diffusion perpendicular to the feature orientation, while allowing the diffusion parallel to the edge orientation and on the smooth regions to proceed with normal strength. The comparison of the results obtained by the proposed filter with that of the classical and newly developed ones reveals a noticeable improvement in the quality of the denoised images evaluated subjectively and quantitatively as well as a substantial increment of the convergence rate comparing to the classical filter.
2 2.1
A Brief Review From Second to Fourth-Order Filters
The nonlinear diffusion filters are evolutionary processes. The fundamental PDE of the nonlinear diffusion filter introduced by Perona and Malik [1] is given by ∂u/∂t = div. (c (∇u) ∇u) ,
(1)
where u is the image intensity function, c(.) is a diffusivity function by which the diffusion coefficient is calculated and t is the evolution time. Symbols of div. and . are used for mathematical notation of Euclidean norm and divergence respectively. The diffusivity function is a positive and none increasing function of ∇u. One of these diffusivity functions defined by Perona and Malik is given by
358
M.R. Hajiaboli
c (∇u) = k 2 / k 2 + ∇u2 ,
(2)
where k is the so-called contrast parameter. You and his colleagues [11], carried out a detailed analysis to show that the solution of (1) is equal to the minimization of an energy functional. If the diffusivity function of (2) is used then the energy functional is R (u) = Ω
k2 2 ln k + ∇u2 dxdy , 2
(3)
where Ω is the region of support of u. R (u) is minimized when ∇u2 is minimum, which leads to a piecewise constant approximation of u. Therefore, formation of staircase artifacts on the ramp edges is unavoidable. In order to resolve this problem, You and Kaveh [4] introduced a fourth-order PDE-based denoising method in which the denoised image is obtained by minimization of the potential function given by E (u) = f |∇2 u| dxdy , (4) Ω
2
where f (s) = sc (s)and |∇ u| is the absolute value of Laplacian of u. Therefore, for the same diffusivity function in (2), E(u) is in form of E (u) = Ω
k2 2 ln k + |∇2 u| dxdy , 2
(5)
meaning that E(u) is minimized when |∇2 u| is minimum. Therefore, the ramp region of u (i.e. the regions where |∇2 u| = 0) are fit in the solution of the associated fourth-order PDE. The solution of the Minimization problem of (4) after using Euler equation followed by gradient descent procedure is given by ∂u/∂t = −∇2 c |∇2 u| ∇2 u , (6) By the forward Euler approximation of the ∂u/∂t , the numerical solver of (6) is given by un+1 = un − dt × ∇2 c |∇2 un | ∇2 un , u0 = u0
and
n = 0, 1, · · · , N ,
(7)
where n is the number of iterations, dt is the time step-size and u0 is a noisy image. This process is an iterative process. In order to protect the edges from over-smoothing, the process needs to be ceased at a certain number of iterations denoted by N. Besides these nonlinear diffusion filters, another class of techniques known as regularization techniques based on solving the nonlinear PDE has been widely used for image restoration. The classical paper of Rudin, Osher and Fatemi [12] is introduced one of the first kind of these filters in which PDE to be solved is
An Anisotropic Fourth-Order Diffusion Filter
359
of the second order. Therefore, the same problem of formation of staircases on the ramp regions of the image motivates the researchers to introduce the new regularization techniques by solving the higher order PDE such as [13], [14]. However, the focus of this paper is on the diffusion based techniques as they have been reviewed earlier. 2.2
Edge Preservation and Convergence Rate
Apart from a significant advancement in reduction of the blocky effects on the denoised image using (6), the optimal parameter setting for numerical solver in (7) leads to very slow convergence rate in its numerical solver especially when the level of contaminating noise is moderately high. A recently developed technique known as a fourth-order hybrid model [6] uses a relaxed median filter [15] to improve the quality of the denoised image when the observed image is heavily contaminated by noise. The numerical model of this filter is given by un+1 = RMαω un − dt × ∇2 c |∇2 un | ∇2 un , (8) where RM denotes the relaxed median filter with a lower bound of α and upper bound of ω. This filtering process needs a lower number of iterations to give a denoised image. However, the denoised image is highly affected by using the relaxed median filter and the main advantage of using fourth-order diffusion filters (i.e. the ramp edge preservation) is hindered as it is shown later. Moreover, the computational burden per iteration is dramatically higher than that of the You and kaveh. Another recently introduced technique [7] demonstrates a significant improvement in the convergence rate along with a good ramp edge preservation. In this technique, the diffusivity function of the You and Kaveh filter, c |∇2 u| , is replaced by c (∇u) and the PDE of the filter is given by (9) ∂u/∂t = −∇2 c (∇u) ∇2 u , Although the energy functional of (9) does not have a closed form, it can be seen that the filter can still support the planar approximation of the image. The ramp edge preservation of this fourth-order diffusion filter comes from the fact that 2 2 ∂u/∂t 2 → 0 when ∇ u → 0. However, as|∇ u| ≥ ∇u the diffusivity function of c |∇ u| gives the smaller diffusion coefficient for the step edges compared to c (∇u) . Therefore, in spite of the good convergence rate obtained by (9), the step edges are still facing the higher amount of the distortion comparing to that of the classical methods. 2.3
Anisotropic Diffusion Filters
The so-called anisotropic diffusion filters refer to the schemes in which the diffusion rate is specifically controlled based on the direction of the local features such as the ones introduced in [8], [9] and [10]. The coherence-enhancing diffusion filter [9] is one this kind in which the scalar diffusion coefficient in (1) is
360
M.R. Hajiaboli
replaced by a tensor diffusion coefficient to reduce the diffusivity of the filter in perpendicular to the orientation of the local features, while let the diffusion with high strength is performed at the direction of the level set. Another anisotropic filter introduced by Carmona and Zhong [10] uses the scalar diffusivity functions to perform anisotropic diffusion. The PDE of this filter is given by ∂u/∂t = c1 (c2 uηη + c3 uξξ ) ,
(10)
where c1 ,c2 and c3 are different diffusivity functions and uηη and uξξ are the second-order directional derivative. Let η denote the perpendicular direction to the orientation of the feature or the so-called gradient direction and ξ denote the direction of the contour or level set. All of these techniques belong to a class of filters known as the secondorder diffusion filters. Some techniques such as [16] for surface smoothing by anisotropic diffusion filtering of the normals to the surface or its other variant for image denoising [17] can be considered as fourth-order anisotropic filters, however these filters are two phase filters meaning that at the first phase, an anisotropic filter applies on the normal map of the surface or image and at the second phase, a surface is fitted to the processed normals. In Section 3, a new setting of the fourth-order anisotropic diffusion filter is proposed, which is a single phase filter and can be seen as a generalization of the classical fourth-order nonlinear diffusion filter of You and Kaveh.
3 3.1
The Proposed Model Diffusion Equation
The previously mentioned fourth-order diffusion filters are isotropic in which the extent of the diffusion is controlled by the diffusivity function regardless of the orientation of the edges. The only anisotropic behavior of those filters is limited to the anisotropic response of the discrete Laplacian operator. Most of the discrete Laplacian operators exhibit an anisotropic response to the edge with respect to x and y (i.e. the Cartesian coordination) [18]. However, in order to give an anisotropic realization of the fourth-order diffusion filter, one should consider the second-order directional derivative of the image. Two normalized and orthogonal vectors of η and ξ pointing at the direction of the gradient and level set respectively are given by [ux uy ] [−uy ux ] η= and ξ = . u2x + u2y u2x + u2y
(11)
Based on the definition in (11) , one can derive the second order derivative of the image in the direction of the gradient and level set as uηη = and
uxx u2x + 2ux uy uxy + uyy u2y u2x + u2y
(12)
An Anisotropic Fourth-Order Diffusion Filter
uξξ =
uxxu2y − 2ux uy uxy + uyy u2x . u2x + u2y
361
(13)
However, it can be simply shown that the summation of these second directional derivatives is equal to the Laplacian of the image, ∇2 u = uxx + uyy = uξξ + uηη .
(14)
Therefore, the proposed fourth-order diffusion equation, which is of a generalization of (6) can be written as ∂u/∂t = −∇2 (c1 (c2 uηη + c3 uξξ )) .
(15)
In the proposed model, c1 , c2 and c3 are the diffusivity functions, where c1 controls total amount of diffusion and c2 and c3 control the uneven diffusion in the direction of η and ξ . Apparently, choosing c2 = c3 and c1 ∗ c2 = c will lead to the nonlinear diffusion filter of (6) or (9) depending on the definition of c. In the next section, the criteria of a suitable choice for these diffusivity functions are discussed. 3.2
Diffusivity Functions
Different diffusivity functions in context of nonlinear diffusion denoising have been introduced and depending on the choice of the diffusivity function, the behavior of the filter can be varied. The most commonly used diffusivity function in fourth-order diffusion filters is the one in (2) as c (s), where s is the modulus of the derivative of the image (s = |∇2 u| in (6) or s = ∇u in (9). This diffusivity function regardless of the choice of s is a function bounded in (0,1]. However, a low computational cost and suitable choice of these diffusivity functions in our proposed model is given by c1 (s) = c2 (s) = c (∇u) and c3 = 1 .
(16)
Similar to (9), s in the function c1 is the modulus of the gradient of u which leads to a fast convergence rate and c2 = c1 is an optimal choice in terms of overall computational cost of the filter. Therefore, the proposed model in (15) can be rewritten in the form of ∂u/∂t = −∇2 c (∇u)2 uηη + c (∇u) uξξ . (17) Since the function c is bounded in (0, 1], the overall diffusivity in η direction is smaller than the one in ξ direction. Before presenting comparative results in the next section, the performance of the filter is compared to the second order filter of Perona Malik in Fig.1, which can show the ability of the proposed filter to preserve the ramp edges. In fact, the proposed filter supports the planar approximation of the image similar to (6) and (9), since for planar regions, uηη → 0 and uξξ → 0 which lead to ∂u/∂t → 0.
362
M.R. Hajiaboli
(a)
(b)
(c)
Fig. 1. Comparing the results obtained by a second-order filter and the proposed filter, (a) noisy image, (b) denoised image by the Perona and Malik filter, (c) denoised image by the proposed filter
3.3
Inverse Diffusion
The classical fourth-order filter of You and Kaveh [4] in (6) is a well-posed process because its potential function, (5), is a positive potential function with a global minimum. On the other hand, deriving the potential function of the proposed filter, (17), is not as simple as (6). However, in order to demonstrate that the uneven weighed summation of uηη and uξξ may lead to the inverse diffusion, it is sufficient to show that at least for a sub-region of u 2 sing c (∇u) uηη + c (∇u) uξξ = sign ∇2 u . (18) In this case, the dynamic flow of (17) performs an inverse diffusion, which results in the edge enhancement. The maximum of the uneven weight between coefficients of uηη and uξξ happens, when c (∇u) = 1/2. In this case, the linear version of the (17) can be written in the form of u uξξ ηη ∂u/∂t = −∇2 + 2 4u uξξ uξξ ηη 2 = −∇ + + 4 4 24 u u ∇ ξξ + . (19) = −∇2 4 4 Knowing (6) has a positive potential function, if sign ∇2 u/4 + uξξ /4 = 2 that sign ∇ u , it results in a positive potential function for filter (19). It means that |∇2 u| > |uξξ | should be valid throughout the whole image, which does not hold true. An example shown in Fig.2 can demonstrate the fact that the linear diffusion equation of (19) performs an inverse diffusion on the edges. The signal shown in Fig.2-(b) is the extracted intensity profile of the standard test image of "disk" in Fig.2-(a) at the middle row. The signal in Fig.2-(c) is the same intensity
An Anisotropic Fourth-Order Diffusion Filter
363
Fig. 2. Inverse diffusion as a result of the uneven diffusion in the directions of η and ξ, (a) is the original image of "disk", (b),(c) and (d) are the intensity of the original, diffused image by (19) and diffused image by the proposed filter (17) at the middle row
profile of the image after being filtered by (19). The inverse diffusion in this case leads to the edge enhancement. However, if the filter is run on the nonlinear fashion as it is proposed in (17), the image shown in Fig.2-(d) shows that process of uplifting of the edges is dramatically reduced. In the other word, in the general application of the image denoising, the process of the inverse diffusion in the proposed filter does not lead to instability of the filter and formation of ringing artifacts around the edges.
4
Comparative Results
In this section, we are presenting the comparative results of the proposed method with the other fourth-order nonlinear diffusion filters. The results of the following filters are going to be compared: 1. The Proposed filter with the PDE of (17) with k=7 and dt=0.031 (i.e. the time-step size that provides a data independent stability in the numerical solver [7]). 2. The filter of (7) introduced by You and Kaveh [4] with the suggested parameters setting of dt=0.25 and k=1. 3. The filter of (8) introduced in [12] with the suggested parameters setting of dt=0.1, k=3, α = 3 and ω = 5. 4. The filter in (9) introduced in [7] that is a self-governing filter. In this filter, the diffusivity function of Pernoa and Malik, c(s) has been used with s = ∇u, the contrast parameter of k is estimated by histogram-based mechanism used in [1] and dt=0.031. Three test images of "Pepper", "Cameraman" and "House" have been corrupted by white additive Gaussian noise with standard deviation of 15. In Table 1, an objective comparison between the performances of these filters in terms of signal-to-noise ratio (SNR) of the denoised image and their computational complexity are presented.
364
M.R. Hajiaboli Table 1. Quantitative comparison of the results
Noisy Image SNR(dB) Pepper 10.98 Cameraman 12.38 House 9.68
Method Proposed (9) (7) (8) Proposed (9) (7) (8) Proposed (9) (7) (8)
SNR(dB) 17.84 17.32 15.83 15.21 17.08 16.83 16.59 13.59 17.44 17.08 15.80 15.39
Denoised Image Num. of Iter. CPU/Iter. Convergence(s) 80 0.038 3.04 14 0.080 1.12 3133 0.031 97.12 2 0.155 0.31 35 0.038 1.33 6 0.082 0.492 3015 0.031 93.46 1 0.160 0.16 89 0.038 3.382 36 0.081 2.916 3907 0.031 121.12 2 0.160 0.32
18 17 16
SNR(dB)
15 14 13 12 filter (7) filter (9) proposed filter (17) filter(8)
11 10 9 0 10
1
10
2
10 Number of Iteration −1
3
10
4
10
Fig. 3. Comparing the convergence rate of the filters for denoising of test image "House"
The results exhibit that the proposed method constantly produces the denoised image with higher SNR. It is important to note that the results are obtained at the optimal number of iterations in which the maximum SNR in evolutionary process of the filters are achieved. If the iterative filtering process is continued after the optimal number of iterations, the SNR of the denoised image is reduced due to over-smoothness of edges. The other important feature in the proposed method is its fast convergence rate. As it is shown in Fig.3, for the test image of "House", the convergence rate in the proposed method is much higher than the filter of You and Kaveh. The computational burden of the filters is measured as CPU time of each iteration provided that they are filtering the same image on the same computer. Thus, the total convergence time for filtering process is a multiplication of CPU/iteration by number of iterations. The relaxed median regularized filter converges faster
An Anisotropic Fourth-Order Diffusion Filter
(a)
(d)
(b)
(c)
(e)
(f)
365
Fig. 4. Comparing the perceptual quality of the results. The pair of images labeled (a) to (f) are as the following: (a) noiseless image, (b) noisy image, (c) denoised image using (7), (d) denoised image using (8), (e) denoised image using (9), (f) is denoised image using proposed filter (17).
366
M.R. Hajiaboli
than the proposed method, however the maximum SNR is significantly lower than that of other methods, and the decay rate of SNR due to over-smoothness of the edges is also very fast. The computational cost of the proposed filter compared to the one in (9) is slightly higher, however the higher SNR obtained by the proposed filter justifies this amount of the higher computational burden. In Fig.4, the perceptual quality of the denoised image by the proposed method is compared with that of the other methods. In the first row, the whole image and in the second row, a magnified portion of the image are shown. Each pair of the images is labeled from (a)-(f). The first two images (a) and (b) are the noiseless and the noisy images. In Fig.4-(c), the denoised image by You and Kaveh filter is shown in which formation of some speckle noise on the denoised image is visible. This drawback is known and addressed in [4] and it is as a result of choosing small value for k in diffusivity function, however this setting of k is necessary to protect the edges from over-smoothing. In Fig.4-(d), the denoised image by the relaxed median regularized filter using (8) is shown. This denoised image is blurred and some staircase artifacts on smooth regions of the image are formed. The next image, shown in Fig.4-(e) is the result of the filter in (9) in which the extent of denoising and edge preservation is noticeably better than that of the filters of (7) and (8). However, comparing this result with the one obtained by the proposed filter in Fig. 4-(f) reveals that the extent of edge preservation in the proposed filter is noticeably higher.
5
Conclusion
An anisotropic fourth-order PDE for noise removal has been proposed. A brief theoretical review of the second and fourth-order diffusion denoising filters has been presented with highlighting the fact that previously developed fourth-order filters are isotropic filters in which the extent of the edge preservation is controlled by reduction of the diffusivity of the filters near the edge regardless of its orientation. A major challenge in these filters is that the optimal choice of the model parameters for good edge preservation leads to a dramatically slow convergence rate. However, in the proposed filter, the diffusion strength has been adjusted with respect to the direction of the local features. Two different diffusivity functions have been designed to extremely minimize the diffusion in perpendicular to the feature orientation (i.e. gradient direction), while let the diffusion on the direction parallel to the orientation of the edge (i.e. direction of the level set) proceed with normal speed. Therefore, the proposed filter leads to a faster reduction of the uncorrelated noise and overall faster convergence rate with a good edge preservation due to reduction of the diffusivity of the filter in the gradient direction. The comparison of the results obtained by the proposed filter with that of the classical and newly developed ones has shown that the proposed method produces a noticeable improvement in the quality of the denoised images evaluated subjectively and quantitatively as well as a substantial increment of the convergence rate compared to the classical filter.
An Anisotropic Fourth-Order Diffusion Filter
367
References 1. Perona, P., Malik, J.: Scale-space and edge detection using anisotropic diffusion. IEEE Trans. on Pattern Analysis and Machine Intelligence 12(7), 629–639 (1990) 2. Catte, F., et al.: Image selective smoothing and edge detection by nonlinear diffusion. SIAM J. Numer. Anal. 29(1), 182–193 (1992) 3. Black, M.J., et al.: Robust anisotropic diffusion. IEEE Transactions on Image Processing 7(3), 421–432 (1998) 4. You, Y.L., Kaveh, M.: Fourth-order partial differential equations for noise removal. IEEE Transactions on Image Processing 9(10), 1723–1730 (2000) 5. Lysaker, M., Lundervold, A., Tai, X.-C.: Noise removal using fourth-order partial differential equation with applications to medical magnetic resonance images in space and time. IEEE Tran. on Image Processing 12(12), 1579–1590 (2003) 6. Rajan, J., Kannan, K., Kaimal, M.R.: An Improved hybrid model for molecular image denoising. Journal of Mathematical Imaging and Vision 31, 73–79 (2008) 7. Hajiaboli, M.R.: A self-governing hybrid model for noise removal. In: Wada, T., Huang, F., Lin, S. (eds.) PSIVT 2009. LNCS, vol. 5414, pp. 295–305. Springer, Heidelberg (2009) 8. Weickert, J.: Anisotropic Diffusion in Image Processing. B. G. Teubner (1998) 9. Weickert, J.: Coherence-enhancing diffusion filtering. International Journal of Computer Vision 31(2-3), 111–127 (1998) 10. Carmona, R.A., Zhong, S.: Adaptive smoothing respecting feature directions. IEEE Transactions on Image Processing 7(3), 353–358 (1998) 11. You, Y.-L., et al.: Behavioral analysis of anisotropic diffusion in image processing. IEEE Trans. Image Processing 5, 1539–1553 (1996) 12. Rudin, L., Osher, S., Fatemi, E.: Nonlinear Total Variation based noise removal algorithms. Physica D 60, 259–268 (1992) 13. Chan, T., Marquina, A., Mulet, R.: High Order Total Variation-based Image Restoration. SIAM J. on Scientific Computing 22(2), 503–516 (2000) 14. Fang, L., et al.: Image restoration combining a total variational filter and a fourthorder filter. Journal of Visual Communication and Image Representation 18(4), 322–330 (2007) 15. Hamza, A.B., et al.: Removing noise and preserving details with relaxed median filters. Journal of Mathematical Imaging and Vision 11(2), 161–177 (1999) 16. Tasdizen, T., et al.: Geometric surface smoothing via anisotropic diffusion of normals. IEEE visualization 1(1), 125–132 (2002) 17. Lysaker, M., Osher, S., Tai, X.-C.: Noise removal using smoothed normals and surface fitting. IEEE Transactions on Image Processing 13(10), 1345–1357 (2004) 18. Kamgar-Parsi, B., Rosenfeld, A.: Optimally isotropic Laplacian operator. IEEE Transactions on Image Processing 8(10), 1467–1472 (1999)
Enhancement of Blurred and Noisy Images Based on an Original Variant of the Total Variation Khalid Jalalzai and Antonin Chambolle Centre de Mathématiques Appliquées (CMAP), École Polytechnique, 91128 Palaiseau Cedex, France [email protected], [email protected]
Abstract. In this paper, we introduce a new variant of the total variation (T V ). Its purpose is to simplify T V -based restoration when the image is degraded by some kernel which is easily computed in the Fourier domain (blur, Radon transform...). We actually replace the T V term by a mere L1 norm of some field, for which the optimization is much easier. This approach permits us to use a recent and fast algorithm to enhance, in particular, blurred and noisy images. We also compare our approach with standard total variation based denoising and show that it avoids the famous staircasing effect.
1
Introduction
In 1992, Rudin, Osher and Fatemi (ROF) introduced the total variation in their founding article [13] as a regularizing criterion for inverse problems in imaging. This has been fruitful in image restoration since it can regularize images without smoothing the edges. A possible approach to tackle the minimization of ROF’s problem consists in the generic forward-backward splitting method studied for instance by Combettes and Wajs in [3]. This consists in minimizing (ϕ + ψ) where ϕ and ψ are both convex functions with certain regularity properties. Usually in signal restoration, given a signal u, ϕ(u) is the so-called data fidelity term and is equal to 12 Au − g2 where g is a noisy signal which also underwent a linear perturbation A. The second term ψ usually reflects a priori knowledge about the noise for instance. In case ψ = T V as in ROF’s problem, the main drawback is that it is usually difficult to compute (even with a small error) the minimizer, namely the proximal operator proxT V (see Moreau [9] or again Combettes and Wajs [3] for more about this). Therefore, what we propose in this article is a variant of ROF’s problem where ψ is simply the L1 norm of some field p. This new term preserves the nice properties of the original total variation. Its relevance is due to the fact that its proximal operator is easy to compute and leaves the way open to compressed sensing-type algorithms (see Nesterov [12] or Beck and Teboulle [2] for instance). X.-C. Tai et al. (Eds.): SSVM 2009, LNCS 5567, pp. 368–376, 2009. c Springer-Verlag Berlin Heidelberg 2009
Enhancement of Blurred and Noisy Images
369
However our idea is different from the "Augmented lagrangian" (see Tai and Wu [15]) or "Split Bregman" (see Goldstein and Osher [6]) methods where the field p must satisfy at convergence (sometimes approximately) the constraint p = ∇u, while in our approach p might be quite far from being a gradient.
2
Few Notations
From now on, an image u will be represented by an n×n matrix with real entries i.e. an element of X = IRn×n . To simplify matters in the sequel, especially when we shall consider the discrete Fourier transform of u, we assume that the image u is also periodic and defined for all k ∈ ZZ by ui+kn,j+kn = ui,j with i, j ∈ {1, ..., n}. To define the total variation of the image u, we first have to introduce a discretized version of the gradient. For u ∈ X, it is the vector ∇u of Y = X × X given by ui+1,j − ui,j (∇u)i,j = , ui,j+1 − ui,j for i, j = 1, ..., n. Finally, the most simple approximation of the total variation of u ∈ X is defined by T V (u) = |(∇u)i,j | i,j
where | · | is simply the Euclidian norm of IR2 . Let us also introduce two important operators: the divergence div p of an element p ∈ Y and the laplacian Δv of an image v. By analogy with the continuous setting, we want them to satisfy div p, uX = −p, ∇uY and Δv = div ∇v,
(1)
for all u ∈ X.
3
The TV-Based Classical Approach
Given a noisy image g which has also been exposed to a linear perturbation A, the Rudin, Osher and Fatemi method suggests to minimize the following functional 1 2 F (u) = Au − g + λT V (u) (2) 2 to restore the image g. The positive parameter λ controls the regularization level. Actually the T V term is not differentiable and in practice it is often replaced by another approximation of the total variation: ε2 + |(∇u)i,j |2 T Vε (u) = i,j
370
K. Jalalzai and A. Chambolle
where ε is a positive real number. Therefore, we are led to find the unique uε which minimizes 1 2 Fε (u) = Au − g + λT Vε (u). 2 We are actually facing a smooth convex optimization problem which can be solved easily with the gradient method. To do so, it is enough to consider a sequence (un ) of images and a small enough gradient step h > 0 that satisfy un+1 = un − h AT (Aun − g) + λ∇T Vε (un ) ⎛ ⎞ (∇un )i,j ⎠, ∇T Vε (un ) i,j = − div ⎝ ε2 + |(∇un )i,j |2
with
for any i, j = 1, ..., n. It remains to choose u0 : it would be wiser to take it as close as possible to the minimizer, consequently u0 = g seems to be a good choice. Unfortunatelly, the simple scheme which is suggested above is fairly slow since it converges as O n1 which means that there exists a positive real C such that Fε (un ) − Fε (uε ) ≤
C . n
A proof of this classical result can be found in [10], [11] or even [2]. Actually, in [11], Nesterov proposes a variant of the gradient algorithm with convergence rate O( n12 ) which solves the problem. It is as follows:
L 2 vn = argmin Fε (un ) + v − un , ∇Fε (un )X + v − un , v ∈ X , 2 n 1 2 [Fε (uk ) + w − uk , ∇Fε (uk )X ] + w − u0 , w ∈ X , wn = argmin 2 k=0
un+1
2 k+1 wn + vn , = k+3 k+3
where L is the Lipschitz constant of ∇Fε . This algorithm combines efficiently classical gradient method (for the calculation of vn ) and conjugate gradient method (calculation of wn ). We refer to Nesterov [10] for further explanations on these two techniques. See also Beck-Teboulle [2] for a recent, simpler variant.
4
A Variant of TV
Let u ∈ X be an image. The main idea is to replace the T V term in (2) by J(u) = min p1 p∈Y Πp=∇u
Enhancement of Blurred and Noisy Images
371
2 2 where on the one hand, p1 = i,j (p1i,j ) + (p2i,j ) when p = (p1 , p2 ) ∈ X ×X and on the other hand, Π is the projection on the gradients defined by Πp = ∇¯ v, where v¯ realizes the minimum min ∇v − p.
v∈X
(3)
Here · is the Euclidian norm of Y . Remark by the way that we have J(u) ≤ T V (u) ≤ T Vε (u). for any u ∈ X. This is a straightforward consequence of the definition. In the sequel, we shall detail some other interesting properties of this functional which makes us believe that it behaves the same way as T V . Let us get back to work: the solution of (3) is characterized by the EulerLagrange equation ∇∗ (∇u − p) = 0 or, using the notation introduced in (1), Δu = div p, (we recall that our operators ∇, div and Δ are here discrete operators with periodic boundary conditions). Therefore, J(u) =
min
p∈Y div p=Δu
p1 .
Hence, the Rudin, Osher and Fatemi’s problem expressed in terms of this new functional consists in minimizing G(p) =
1 2 Au − g + λp1 2
over (p, u) which satisfy the constraint Δu = div p. Lately, minimization of such functionals has attracted much attention in data compression in particular and was the subject of many papers. Among those, two recent articles by Nesterov [12] and by Beck and Teboulle [2] focus on the minimization of objective functions which can be decomposed as a sum ϕ+ψ where ϕ is a continuously differentiable convex function whose gradient is Lipschitz continuous and ψ is a continuous convex function which is possibly nonsmooth but is simple in the sense that its proximal operator is easy to compute (see Combettes and Wajs [3] for the definition). These characteristics suit perfectly the two terms composing G and we henceforth denote ϕ(p) =
1 2 AΔ−1 div p − g and ψ(p) = p1 . 2
372
K. Jalalzai and A. Chambolle
In their article, Beck and Teboulle describe the following scheme to construct a minimizing sequence (pn ) for G: q1 = p0 ∈ Y, t1 = 1, L 2 pn = argmin ϕ(qn ) + p − qn , ∇ϕ(qn )Y + p − qn + p1 , p ∈ Y , 2 2 1 + 1 + 4tn , tn+1 = 2 tn − 1 qn+1 = pn + (pn − pn−1 ), tn+1 −1
is the Lipschitz constant of ∇ϕ. where L = 12 (1 − cos( 2π n )) Remark that in this algorithm, each iteration needs only one computation of the gradient if things are done properly. As for Nesterov’s algorithm, which converges as O( n12 ) as does Beck and Teboulle’s one, and which is again a clever combination of gradient method and conjugate gradient, it demands two calculations of the gradient which slows down notably each iteration.
5
The Continuous Setting
Let us mention in this section some properties of the functional J in the continuous setting. We refer to Jalalzai [7] for proofs and further results. First of all, let us fix some notations especially for this section. Henceforth, Ω will designate an open set of IRn with a smooth enough boundary and to simplify matters we first place ourselves in the context of functions u whose distributional derivatives are integrable functions that we denote Du, i.e. u lies in Sobolev space H 1 (Ω). The functional we previously introduced is a discretization of n 2 J(u) = inf |φ|, φ ∈ L (Ω) and Πφ = Du . Ω
where Π is the orthogonal projection on gradients as in section 4. Formally n speaking, given a function φ ∈ L2 (Ω) we set Πφ = D¯ v where v¯ minimizes min Dv − φL2 (Ω)n .
v∈H 1 (Ω)
n
It is actually easy to see that there exists a unique ψ ∈ L2 (Ω) such that we have the so-called Helmholtz decomposition ¯ φ = Πφ + ψ where ∇v · ψ = 0 given any v ∈ C 1 (Ω). Ω
(we refer to Dautray-Lions [4] or Temam [14] for more about this topic). If we put things together, we proved that n 2 1 ¯ |Du + ψ|, ψ ∈ L (Ω) and ∇v · ψ = 0 ∀v ∈ C (Ω) . J(u) = inf Ω
Ω
Enhancement of Blurred and Noisy Images
373
Nonetheless, this new formulation of J stays meaningful even when u is simply a function of bounded variation in Ω (denoted u ∈ BV (Ω)) which means that its n distributional derivative Du is this time in Mb (Ω) , the space of IRn -valued finite Radon measures on Ω. Henceforth, we also let ψ range in the space Mb (Ω)n . We refer to Ambrosio, Fusco and Pallara [1] or even Giusti [5] for properties of bounded variation functions and for other measure theory considerations. All this motivated a new definition of J when u ∈ BV (Ω), namely: n 1 ¯ J(u) = inf |Du + ψ|, ψ ∈ Mb (Ω) and ∇v dψ = 0 ∀v ∈ C (Ω) . Ω
Ω
Note by the way that J(u) is obviously well-defined for any u ∈ BV (Ω) since |Du|. (4) J(u) ≤ Ω
Thanks to a classical convex duality argument, it is possible to show that under some additional assumptions on Ω, we have ∇w · Du, w ∈ C 1 (Ω) and ∇w∞ ≤ 1 . J(u) = sup Ω
Using this dual formulation one can prove the following result: Theorem 1. Let Ω be an open set in IRn and u = χE be the characteristic of a finite-perimeter set E ⊂ Ω, or even let u ∈ BV (Ω) with a derivative Du concentrated on the jump set. Then J(u) = |Du|. Ω
The proof is mostly based on the fact that rectifiable sets admit approximate tangent hyperplanes. Remark that when Du has a diffuse part, inequality (4) may be strict. The latter theorem legitimates the use of functional J in the image processing context since it shows that J behaves the same way as T V .
6
Preliminary Numerical Simulations
In this last section, we compare the two different approaches based on the functionals T V and J. For this purpose, we use the two algorithms we presented above. In our implementation, Beck-Teboulle’s algorithm does 2 times less iterations since it needs to compute four Fourier transforms per iteration whereas Nesterov’s algorithm needs only two. We think that one can do much better especially in the case we consider J since it makes extensive use of Fourier transform methods and therefore is easily parallelizable. Moreover, the functional J seems to avoid the famous staircasing effect (see Louchet and Moisan’s article [8]) produced by the T V minimization. Indeed, the latter yields images with peculiar local configurations. J does not. All along these tests, the regularization parameter λ is maintained equal to 1. For all the simulations, we used a personal computer with a 2 Ghz Core2 Duo processor and we let the two Matlab programs run for exactly 20 seconds.
374
6.1
K. Jalalzai and A. Chambolle
First Example
We look at the 256 × 256 Lenna image. This photo went through a Gaussian blur of standard deviation σblur = 1.5 followed by an additive zero-mean Gaussian noise with standard deviation σnoise = 4. The original image is represented in Fig. 1. We then implemented and runned Beck-Teboulle and Nesterov’s algorithms to restore the image. The results of these two experiments are shown in Fig. 3 and 4.
Fig. 1. Original Lenna photo
Fig. 3. J-processed iterations
6.2
Lenna,
Fig. 2. σblur = 1.5, σnoise = 4
600
Fig. 4. T V -processed iterations
Lenna,
1000
Second Example
The second example aims to compare the deblurring for a text scan. The first figure is the original image. In Fig. 2 the 256 × 256 poem image underwent the same disruption process with parameters σblur = 1 and σnoise = 4.
Enhancement of Blurred and Noisy Images
Fig. 5. The first verse of a famous Verlaine’s poem
Fig. 7. J-processed iterations
poem,
600
375
Fig. 6. σblur = 1, σnoise = 4
Fig. 8. T V -processed iterations
poem,
1200
References 1. Ambrosio, L., Fusco, N., Pallara, D.: Functions of Bounded Variation and Free Discontinuity Problems. Oxford University Press, Oxford (2000) 2. Beck, A., Teboulle, M.: A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM Journal on Imaging Sciences (accepted) (2008) 3. Combettes, P.L., Wajs, V.R.: Signal recovery by proximal forward-backward splitting. SIAM Journal on Multiscale Modeling and Simulation 4(4), 1168–1200 (2005) 4. Dautray, R., Lions, J.-L.: Mathematical Analysis VI and Numerical Methods for Science and Technology. Evolution Problems II, vol. 6. Springer, Heidelberg (1993) 5. Giusti, E.: Minimal Surfaces and Functions of Bounded Variation. Birkhäuser, Basel (1984) 6. Goldstein, T., Osher, S.: The Split Bregman Method for L1 Regularized Problems. UCLA CAAM Report 08-29 (2008)
376
K. Jalalzai and A. Chambolle
7. Jalalzai, K.: Étude des propriétés d’une variante de la variation totale. Master thesis (2008) 8. Louchet, C., Moisan, L.: Total variation denoising using posterior expectation (2008), http://hal.archives-ouvertes.fr 9. Moreau, J.-J.: Fonctions convexes duales et points proximaux dans un espace hilbertien. C. R. Acad. Sci. Paris Sér. A Math. 255, 2897–2899 (1962) 10. Nesterov, Y.: Introductory lectures on convex optimization. Kluwer Academic Publishers, Dordrecht (2004) 11. Nesterov, Y.: Smooth minimization of non-smooth functions. Mathematical Programming (A), pp. 127–152 (2005) 12. Nesterov, Y.: Gradient methods for minimizing composite objective function. CORE Report (2007) 13. Rudin, L.I., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Physica D 60, 259–268 (1992) 14. Temam, R.: Navier-Stokes Equations Theory and Numerical Analysis. AMS Bookstore (2001) 15. Tai, X.-C., Wu, C.: Augmented Lagrangian Method, Dual Methods and Split Bregman Iteration for ROF Model. UCLA CAAM Report 09-05 (2009)
Coarse-to-Fine Image Reconstruction Based on Weighted Differential Features and Background Gauge Fields Bart Janssen, Remco Duits, and Luc Florack Eindhoven University of Technology, Dept. of Biomedical Engineering & Dept. of Mathematics and Computer Science {B.J.Janssen,R.Duits,L.M.J.Florack}@tue.nl
Abstract. We propose an iterative approximate reconstruction method where we minimize the difference between reconstructions from subsets of multi scale measurements. To this end we interpret images not as scalar-valued functions but as sections through a fibered space. Information from previous reconstructions, which can be obtained at a coarser scale than the current one, is propagated by means of covariant derivatives on a vector bundle. The gauge field that is used to define the covariant derivatives is defined by the previously reconstructed image. An advantage of using covariant derivatives in the variational formulation of the reconstruction method is that with the number of iterations the accuracy of the approximation increases. The presented reconstruction method allows for a reconstruction at a resolution of choice, which can also be used to speed up the approximation at a finer level. An application of our method to reconstruction from a sparse set of differential features of a scale space representation of an image allows for a weighting of the features based on the sensitivity of those features to noise. To demonstrate the method we apply it to the reconstruction from singular points of a scale space representation of an image.
1
Introduction
Reconstruction from signal samples is a long standing problem in signal and image analysis [20]. We present a method for the approximation of a signal or image from its generalized samples, i.e. the samples are given on a non-equidistant grid and were obtained by means of spatially varying filters. Variational reconstruction of non-equidistant image samples has recently become of interest to the image compression community [9] where significant gains in reconstruction quality have been obtained by introducing anisotropic non-linear regularization strategies. In the scale space community a general interest in reconstruction from generalized samples has been there for quite some time [19, 18, 14, 12, 13]. We propose a method that produces an image that approximately satisfies all features. Features that are more robust to perturbations of the source image are given a higher weight, which steers the reconstruction method such that those X.-C. Tai et al. (Eds.): SSVM 2009, LNCS 5567, pp. 377–388, 2009. c Springer-Verlag Berlin Heidelberg 2009
378
B. Janssen, R. Duits, and L. Florack
features are better approximated than those that are more sensitive to noise. This leads to a more robust method compared to interpolating methods. A gauge field is introduced by means of covariant derivatives on a vector bundle. This way a model of the to be reconstructed image can be incorporated in the energy functional which is minimized to find a suitable reconstruction. Using this gauge field we can construct a coarse-to-fine image reconstruction method. A coarse-to-fine approach naturally leads to a more efficient algorithm in terms of memory consumption and computational efficiency.
2
Image Reconstruction
In the reconstruction problem we aim for a reconstruction from a set of linear functionals on an image. These functionals represent measurements on the image and are henceforth called features. More rigorously: a feature di ∈ R of an image f ∈ L2 (R2 ) measured with a filter ψi ∈ L2 (R2 ) is given by di = (ψi , f )L2 , i = 1 . . . P in which (·, ·)L2 denotes the L2 -inner product. In general the set of features do not describe the input image f unambiguously (they do not constitute a frame), and there is need for a model to which the reconstruction should adhere. When such a model can be described by a (semi-)norm the reconstruction can be obtained directly by means of an orthogonal projection onto the features [14]. Nielsen and Lillholm [19, 18] proposed to find a reconstruction from its features using a nonlinear regularization term (model). Their so called observationconstrained evolution ensures that the features are interpolated. When measurements are contaminated by noise approximation is often favored over interpolation. In the following we will not discuss the interpolation but approximation of a set of P features {di }P i=1 that were obtained by means of the filters {ψi }i=1 .
3
Approximation
Instead of searching for a signal that interpolates the given features one can try to find a signal that approximates the features. In the case of noisy measurements the latter approach is often preferred. We now aim for the function g ∈ H1 (R2 ) that minimizes P λ 2 E(g) = ((g, ψi )L2 − di ) + ||∇g||2 dV , (1) 2 2 R i=1 regularization term data term where λ ∈ R+ a parameter that controlls the quality of the approximation. As λ tends to 0 the approximation will approach the interpolation of the features. The minimizer of this linear functional can be found by finding the unique g that solves the following Euler equation: P ψi ((ψi , g)L2 − di ) − λΔg = 0 . (2) i=1
Coarse-to-Fine Image Reconstruction
379
The parameter λ takes into account each feature with the same weight. This is not desirable when the features are not normalized and even after normalization one can improve on the selection of the weights. We allow for these improvements by introducing P extra parameters (which we will call feature weights), αi ∈ R+ , i = 1 . . . P , that will be set to a fixed value based on the properties of the features. In case of reconstruction from differential features of a scale space representation of an image, which is the main motivation for our method, we can select the newly introduced parameters based on the noise propagation in the scale space representation of an image. The global parameter λ can be absorbed by these parameters but will be maintained in our formulation for the sake of clarity. For fixed αi we now search for the g that satisfies arg min E(g) = arg min g∈L2 (R2 )
P
g∈L2 (R2 ) i=1
2
αi ((g, ψi )L2 − di ) +
λ 2
R2
||∇g||2 dV .
(3)
In the next section we will discuss how the feature weights can be selected. 3.1
Noise Propagation
In order to be able to select sensible values for the αi parameters that appear in eq. (3), we need to make some assumptions on the noise and the set of filters {ψi }P i=1 that are used to extract the measurements. With regard to the noise we assume additive zero-mean white Gaussian noise which has a correlation distance of τ pixels. In recent work about stability of toppoints [2] (which are singular points of a Gaussian scale space representation of an image) this was found to be a sensible assumption. In our application we will reconstruct from differential structure taken from the Gaussian scale space representation of the input image f , therefore we assume that the set of filters {ψi }P i=1 consists of Gaussian kernels or derivatives thereof. The idea is now to construct the weights αi according to the sensitivity of their associated differential features to noise. In order to estimate the sensitivity of a feature di of the image f that is contaminated by additive noise we can adopt work on noise propagation in scale space by Blom [4]. He proposes to 2 compute at a certain scale t > 0 the momenta Mm = Nmx ,my , Nnx ,ny x ,my ,nx ,ny of derivatives of orders mx , my , nx , and ny of the fiducial noise function N . He assumes only the covariance matrix N 2 of the noise to be given. In case the correlation distance τ is much smaller than the scale t, 2 Mm N 2 x ,my ,nx ,ny
τ −1 12 (mx +my +nx +ny ) 2t
4t
Qmx +nx Qmy +ny ,
(4)
with Qn = (n + 1)!! for n even and Qn = 0 otherwise. Features that are sensitive to perturbations on the source image f should influence the final result less than features that are relatively insensitive to these perturbations. Therefore we i compute αi from eq. (4) such that αi ∝ Mn−2 i ,ni ,ni ,ni at scale t . The parameters x
y
x
y
nix , niy , and ti are the derivative order in the x direction, the derivative order
380
B. Janssen, R. Duits, and L. Florack
in the y direction and the scale of the ith filter ψi . Here we stress that these estimations are based on the assumption that P the filters are partial derivatives of a Gaussian. We furthermore ensure that i=1 αi = 1, which essentially makes αi independent of the value of N 2 and τ . 3.2
Discretization
We can try to solve an approximation to g by discretizing eq. (2) (augmented with the feature weights) or discretize the energy functional in eq. (3), and thereafter finding a discrete minimizer of the discretized energy. These two approaches can be equivalent for a slick choice of so called test functions that are involved in the former method. We will proceed by elaborating on directly discretizing the energy. To solve g from eq. (3) we will approximate g by a β-spline of order n:
(eiω/2 − e−iω/2 )n+1 β n (x) = F −1 ω → (x) , (5) (iω)n+1 where F −1 denotes inverse Fourier transformation. Equality (5) is equivalent to the (n + 1)-fold convolution of the β-spline of order 0 ⎧ ⎨ 1 − 12 < x < 12 0 β (x) = 21 |x| = 12 ; (6) ⎩ 0 otherwise a rectangle. Further details concerning β-splines can be found in eg. [22]. It was shown in the context of optic flow [17] and registration [21] that such an approach has computational advantages over a finite difference approach. Arigovindan et al. [1] showed good results in his application of this approach to (a multigrid scheme for) image and vector field interpolation. Moreover it allows for a coarse-to-fine implementation in a elegant way because of the 2-scale relation
x x n+1 n −n n β = 2 − k . (7) β k 2j 2j−1 k∈Z
The n-th order β-spline approximation of g in two spatial dimensions at resolution a > 0, is given by g˜a (x, y) =
M−1 −1 N l=0 k=0
ck,l β n (
x y − k)β n ( − l) , a a
(8)
with ck,l , x, y ∈ R, β n (·) the central β-spline of order n ∈ N, resolution parameter a, and N, M ∈ N correspond to the width and height of the image in pixels. Notice this is a representation of the image in the continuous domain and that g˜a ∈ C n (R2 ), i.e. n-times continuously differentiable.
Coarse-to-Fine Image Reconstruction
381
The regularization term in eq. (3), R2 ||∇g||2 dxdy, can be approximated with the help of eq. (8) by ∞ ∞ ∞ 1 M−1 −1 N ∂β n xi 2 −k ||∇ga (x, y)|| dxdy = ck,l cm,n a −∞ −∞ −∞ ∂xi i=0 l,n=0 k,m=0 x ∞
x ∂β n xi 1−i 1−i − m dxi − l βn − n dx1−i , (9) βn ∂xi a a a −∞ where (x1 , x2 ) correspond to (x, y) in eq. (8). When we consider the integrals in the previous equations we notice that it can be expressed by a convolution: ∞ ∞ ∂β n x ∂β n ∂β n x ∂β n −k − m dx = −a (u) ((m − k) − u) du . a ∂x a ∂u −∞ ∂x −∞ ∂u (10) This is easily verified by substitution of integration variable (u = xa − k) and noting that β n (x) = β n (−x) for all x ∈ R. We furthermore note that a derivative of a central β-spline of degree n is again a linear combination of β-splines at the expense of lowering its degree to (n − 1) ∂ n β (x) = β n−1 (x + 1/2) − β n−1 (x − 1/2) . ∂x As a result we can write eq. (9) in matrix-vector notation as ∞ ∞ ||∇ga (x, y)||2 dxdy = cT Rc , −∞
(M−1)(N −1)
with c = {ci }i=0
(11)
(12)
−∞
and
N −1 M−1 ∂β 2n (m − k) R = aβ 2n (n − l) n,l=0 ⊗ −a + ∂x m,k=0 M−1 N −1 ∂β 2n (n − l) −a ⊗ aβ 2n (m − k) m,k=0 . ∂y n,l=0
(13)
We will express the inner product in the data term in equation (3) in terms of β-splines as well. This leads to an expression similar to eq. (10), (ga , ψi )L2 (R2 ) = (−1)ni +mi
N −1,M−1
ck,l (β n ∗ ψi )(k − xi , l − yi ) ,
(14)
k,l=0
where (xi , yi ) and (ni , mi ) are the location and differential order of the ith filter ψi respectively. In contrast to the discretization of the regularization we will not derive a closed form expression for this convolution but we will approximate the β-spline in eq. (14) by a Gaussian. Where we use the observation in [23] that 6x2 6 n ∼ β (x) = e− (n+1) . (15) π(n + 1)
382
B. Janssen, R. Duits, and L. Florack
The data term can be expressed in matrix-vector notation by Edata (c) = ||Sc − d||2 ,
(16)
(N −1)(M−1),P
and d = {di }P where S = {(β n ∗ ψi )(k − xi , l − yi )}k,l=0,i=1 i=1 . Now we can write the minimizer of equation (3) in matrix-vector notation as T S S − λR c = S T d . (17) This linear system of equations can be solved using a conjugate gradient (CG) method [3]. In case the matrix S is sparse it is beneficial to apply a multigrid method [5]. Mainly due to the non-sparseness of S, the conjugate gradient method is preferred. Notice that, in this specific case, R can be expressed as a convolution. For large images it is infeasible to explicitly compute S T S, therefore we compute the matrix vector product ˆ c = S T Sc that appears in a conjugate ˜ = Sc and thereafter evaluating ˆc = S T c ˜. gradient iteration by first evaluating c
4
Adaptation to a Gauge Field
In the previous sections we used a very simple model as a regularization term. For several applications it would be beneficial if we were able to introduce a more sophisticated model of the image we want to reconstruct. Feature based image editing [16] and optic flow estimation [13,8] are applications that potentially have great benefit of such a refinement. Recently an image in-painting method was introduced that achieves a model refinement by means of covariant derivatives on a vector bundle that are guided by a user selectable gauge field [10]. We will adapt a similar approach. The basic idea is to replace the gradient that appears in the regularization term of eq. (3) by a covariant derivative DAh that is biased by a gauge field h ∈ H2 (R2 ). To this covariant derivative the gauge field h should be “invisible”, i.e DAh h = 0. If we were able to put h to be the original image f the approximation would exactly produce f again. To this end we interpret f not as a scalar function but as a section through a fibered space E = R2 × R+ . Heuristically this means that we rescale intensity by a spatially varying factor, the unit section σ. Thus we consider f σ instead of f to model intensity values in the image (the latter is a special case in which σ(x) = (x, 1) ∀x ∈ R2 ). This implies that when we consider derivatives, we need to account for the spatial variability of σ. In the next subsection we will introduce to this end a connection on a vector bundle. There, we will also make the heuristic description of our approach presented here a bit more rigorous. For the reader who is not familiar with the concept of vector bundles it could be useful to take notice of Fig. 1 before reading the next subsection, since it aids in developing the right geometrical interpretation of the presented material. 4.1
Connections on Vector Bundles
Consider a vector bundle (E, π, M ), with total space E = R2 × R+ , base space M = R2 , and projection π : E → M . π projects a point in E (a point in M augmented with an intensity L ∈ R+ ) to M in the following manner
Coarse-to-Fine Image Reconstruction
383
(18)
π(x, y, L) = (x, y) .
L amounts to a certain physical quantity such as luminous intensity, which is expressed in candela (cd). Next we define a section s : M → E such that π◦s = idM , where idM denotes the identity map on M . We define the association of a section σf with unique image f ∈ L2 (R2 ) as f ↔ σf ⇔ ∀(x,y)∈R2 σf (x, y) = (x, y, f (x, y)) .
(19)
The multiplication of such a section σf by an image g is given by (20)
gσf = σf g .
Let σ ˜ denote the unit section σ ˜ (x, y) = (x, y, L0 ), with L0 a fixed luminous intensity unit (eg. 1cd). We want to define a connection D over the space of sections Γ (E) on E. Let L (Γ (T M ), Γ (E)) denote the space of linear operators that map a section of a tangent bundle on M to a section of a vector bundle. Here we stress that a section of a tangent bundle, V ∈ Γ (T M ), is just a vector field on M . A map D : Γ (E) → L (Γ (T M ), Γ (E))
(21)
is a connection on a vector bundle iff it possesses the following properties, cf. [15], pp.106. In the following we will use standard notation DV σ = (Dσ) (V ). 1. D is tensorial in V : DV +W σ = DV σ + DW σ for V, W ∈ Γ (T M ), σ ∈ Γ (E) Df V σ = f DV σ for f ∈ C ∞ (M, R), V ∈ Γ (T M ) .
(22) (23)
2. D is R-linear in σ: DV (σ + τ ) = DV σ + DV τ for V ∈ Γ (T M ), σ, τ ∈ Γ (E)
(24)
and it satisfies the Leibniz product rule: DV (f σ) = V (f )σ + f DV σ for f ∈ C ∞ (M, R) .
(25)
Suppose we have a section D on a vector bundle. Then it must satisfy the four properties (eqs. (22) to (25)) mentioned above. Therefore we must have the following identity σ+ Dσ(X)(c(t)) = D(z σ ˜ )(X)(c(t)) = X|c(t) (z)˜
2
z(c(t))c˙i (t)D∂xi σ ˜
(26)
i=1
2 for all sections σ = z σ ˜ , and vector fields X = i=1 c˙i ∂xi . Here c : (0, 1) → M ˙ i = 1, 2, with c(t) ˙ = ddt c(t), and is a smooth curve on M , c˙i (t) = dxi , c(t) z ∈ C ∞ (M, R) an arbitrary image. By {dxi }2i=1 = {dx, dy} we denote the dual frame in the cotangent bundle T ∗ M .
384
B. Janssen, R. Duits, and L. Florack
For each i = 1, 2