Nonlinear Smoothing and Multiresolution Analysis

ISNM International Series of Numerical Mathematics Volume 150 Managing Editors: K.-H. Hoffmann, Bonn D. Mittelmann, Tem...

Author: Carl Rohwer

51 downloads 812 Views 962KB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!

Report copyright / DMCA form

DOWNLOAD PDF

ISNM International Series of Numerical Mathematics Volume 150 Managing Editors: K.-H. Hoffmann, Bonn D. Mittelmann, Tempe Associate Editors: R. E. Bank, La Jolla H. Kawarada, Chiba R. J. LeVeque, Seattle C. Verdi, Milano Honorary Editor: J. Todd, Pasadena

Nonlinear Smoothing and Multiresolution Analysis

Carl Rohwer

Birkhäuser Verlag Basel . Boston . Berlin

Author: Carl Rohwer Department of Mathematics University of Stellenbosch 7602 Stellenbosch South Africa

2000 Mathematics Subject Classification: Primary 00A69, 41A46, 42C40; Secondary 06, 47, 62, 65, 94

A CIP catalogue record for this book is available from the Library of Congress, Washington D.C., USA Bibliographic information published by Die Deutsche Bibliothek Die Deutsche Bibliothek lists this publication in the Deutsche Nationalbibliografie; detailed bibliographic data is available in the Internet at http://dnb.ddb.de.

ISBN 3-7643-7229-X Birkhäuser Verlag, Basel – Boston – Berlin This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, broadcasting, reproduction on microfilms or in other ways, and storage in data banks. For any kind of use whatsoever, permission from the copyright owner must be obtained. © 2005 Birkhäuser Verlag, P.O. Box 133, CH-4010 Basel, Switzerland Part of Springer Science+Business Media Printed on acid-free paper produced of chlorine-free pulp. TCF ∞ Printed in Germany ISBN-10: 3-7643-7229-X ISBN-13: 978-3-7643-7229-3 987654321

www.birkhauser.ch

Contents Foreword . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

vii

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

ix

1. Operators on Sequences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1

2. Basic Rank Selectors, Pulses and Impulses . . . . . . . . . . . . . . . . . . . . . . . . . .

9

3. LU LU -Smoothers, Signals and Ambiguity . . . . . . . . . . . . . . . . . . . . . . . . . .

21

4. LU LU -Intervals and Similar Smoothers . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

31

5. Smoothing and Approximation with Signals . . . . . . . . . . . . . . . . . . . . . . . . .

43

6. Variation Reduction and Shape Preservation . . . . . . . . . . . . . . . . . . . . . . . .

51

7. Multiresolution Analysis of Sequences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

71

8. The Discrete Pulse Transform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

91

9. Fair Comparison with Linear Smoothers . . . . . . . . . . . . . . . . . . . . . . . . . . . .

109

10. Interpretation and Future . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

127

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

131

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

135

Foreword This monograph is intended as a simple introduction to the so-called LU LU -theory and the practical use of LU LU -smoothers leading up to a full Multiresolution Analysis of any ﬁnite sequence. The attempt has been to present the subject in a way that is retrospectively ordered to some extent, but preserves some of the twisted paths that intuition initially suggested. It is based on the few publications in the subject, some already submitted and lectures given at conferences and on invitation at public and private institutions. The ideas originated more than ﬁfteen years ago when the author was involved in practical problems at the Institute for Maritime Technology in Simonstown and advanced while the author was at the BMI in Stellenbosch. The author hopes that publication of this book will stimulate the interest of other researches and takes responsibility for any remaining typographical or technical errors. A growing love for this particular subject was one of the prime reasons for a return to academic life, and a perceived opportunity to consolidate various perspectives and results. It is however only the encouragement, guidance and help of several respected people that have resulted in the eﬀort presented here. There is a deep gratitude for; Siegfried G¨ oldner for teaching Mathematics, Gunther Meinardus for patience, good advice and courage, Johan de Villiers for researching with me in Spline Function, Vilmos Totic for pointing us towards Wavelets, Charles Chui for teaching these and Doron Lubinsky for trust, encouragement and walking many an academic mile. I also thank many colleagues and friends who have contributed by discussion or collaboration to the subject: Willie Conradie, Johannes Cronje, Tertius de Wet, Hans Eggers, Sven Ehrich, Frank Hampel, Trevor Hastie, Rolf Jeltsch, Henri Johnson, Gerhard Joubert, Odei Kao, Sergue Lapine, Dirk Laurie, Eberhard Malkowsky, Colin Mallows, Dana Murray, Lothar Reichel, Birgit Rohwer, Lourens Toerien, Johan van der Merwe, Marcel Wild, and the students that have attended my courses on the subject, especially J.P. Harper for his help with the sketches. Finally, most of all, I thank Jan Malan for encouragement and support during the birth of the ideas, Lauretta Adams for her courage, patience and excellent word processing, Marguerite and our children for sharing me with LU LU , and my parents to whom I dedicate this eﬀort.

Introduction The order that I create, my father said, is the order of life. To create this order I shape a face permitting adoration. Order is a sign of life and not it’s cause. Introducing life creates order, and introducing order co-opts death. Order for the sake of order is a parody of life. Antoine de Saint-Exup´ery. ∗

Unlikely events do occur, but science, measurement and information are primarily concerned with tendencies, which we refer to in this book as “trends”. Perhaps the most basic trend is monotonicity, as exhibited in the arrow of time. Tides and waves exhibit a form of local trend, if the minor local trends and large global trends are overlooked (smoothed out). Anomalies, characterized by an apparent lack of trend do occur, and are often inherently brief in space and time. When there is a lack of obvious trend, there is still possibly a statistical trend in the shape of a distribution. Numbers and mathematics have evolved from measurement and economical information storage and transfer. If mathematics is considered the language of science, and the functions of a language are examined, it is reasonable to deduce an order in the functions (uses) as for example in the ﬁrst four functions of a language as given by Popper (18). Corresponding to each function, an associated value comparison is apparent. Function

Value

1.

Expressive

Revealing/not revealing

2.

Signalling

Eﬃciency/ineﬃciency

3.

Descriptive

Truth/falsity

4.

Argumentative

Validity/invalidity

It is natural to consider higher functions (uses), like the poetic function, since even a strict formal language like mathematics can convey wisdom by using metaphors and analogies for modelling events, and leads to the power of interpolation and extrapolation as evident in the impressive accomplishments of modern science. Science develops from simple models of relatively constant things, including recurring ∗ Imprompto

translated by the author from a German translation.

x

Introduction

events of constant period or frequency, and builds up to more complicated trends. Increasing speed, accuracy and data transfer rates have often led to an overabundance of data from which a desired “signal” or trend is to be extracted in a stable and computationally economical way. An early example is the eﬀort produced by Gauss to determine an elliptic orbit (“signal”) from a sequence of measurements corrupted by undetermined and partly undeterminable perturbations produced by ray bending, instrument resolution, computational approximation, truncation and human error. A recent example is the transmission of information from an instrument on a distant planet. The relatively high data rate possible is an advantage but needs to be handled economically for good extraction of “signal”, given perturbations essentially of similar type as those handled by Gauss. Two distinct changes are visible; The amount of work involved to extract the wanted signal and the type of noise regularly introduced by digital processing. Celestial orbits and the determination of their parameters, particularly with the limits of technology of the times, have been replaced by more complicated models for which parameters have to be estimated. Speciﬁcally, the advent of digital measuring, calculation and transmission of an ever increasing amount of data has led to problems that are not only diﬀerent in detail but in philosophical basis. Whereas extrapolation requires the use of well motivated non-local models, interpolation can use simpler local models for predicting values between measurements under reasonably mild assumptions on the behavior of a function. Spline functions have been a natural development linking global continuity requirements with local support in a computationally convenient way, permitting a well-known idea, the Raleigh–Ritz–Galerkin approximation for solutions of diﬀerential equations, to expand into a powerful tool called the Finite Element Method. Using computationally convenient interpolants by functions with local support, and comparing with the best global approximant with the Lebesgue Inequality, justiﬁes the procedure in that it shows the error to be no greater than a small factor of the best possible in a given norm. Thus local procedures have been made respectable, and the art of numerical analysis has been saved from the suspicion of being a collection of clever ad-hoc techniques to calculate parameters in a complex model for the purpose of explaining or predicting events. What spline theory has done to link global and local analysis, wavelet theory has done to periodic events; localizing some frequency based phenomena coherently (8). Intelligent ad-hoc local procedures for the removal of “impulsive” noise from data have received considerable attention in the last two decades. Wisdom and experience has resulted in the widespread use of the (running-) median smoother and compositions of these. A lack of clear and consistent behavior has been widely lamented, but experience indicates that these are eﬀective procedures. Examples of problems from practice suﬃce to indicate the extent of the problem. Suppose a sequence of numbers is transmitted with a reasonable trend in that all are measurements of light intensity at a pixel in a camera. If any one of the pixels now and then registers a “glint”, or if one of the numbers now and then has a spurious inﬂuence in a signiﬁcant bit, like a sign bit or exponent bit

Introduction

xi

of the number, there is the annoying presence of “impulsive” noise. This noise is characterized by briefness and unreasonable size relative to neighboring data. Even though this type of noise was also present in previous times, human intervention and wisdom easily identiﬁed such an “error” and interpolated from neighboring data to leave a relatively undamaged signal. For high speed data processing the problem is to balance complexity with speed and automation. Two diﬀerent concrete examples will suﬃce to illustrate the problem of impulsive noise in measurement. In the ﬁrst a high sample rate representation of a missile trajectory, as measured by radar, is to be analyzed for various purposes. Various sections are seriously contaminated by large impulses. An elaborate analysis can reconstruct the general trajectory shape, and even features like control oscillations and other possible vibrations can be extracted by elaborate techniques. If this is to be done in real time, it is a diﬀerent problem altogether.

Figure 1. Trajectory data corrupted by impulsive noise.

The second example is data from a speedometer of a small fast boat on a rough sea. The downward impulses, clearly due to the boat leaping out of the water, are signiﬁcantly larger than the upward impulses. This bias is dependent on sea state. How is the bias to be removed to give a good speed estimate over a large range of sea states? Statistics has provided an impressive intellectual framework for containing the uncertainty introduced by unexplained or random perturbation (“noise”) as exhibited by deviation from the expected “signals” A philosophically consistent way of treating impulsive noise, in the presence of the usual random noise, is to consider the distribution of the random noise as “heavy tailed”. This leads to Robust Statistics, as developed by Huber, Hampel and others. When economy of eﬀort or speed is required a diﬀerent approach was advocated by Tukey and others. The idea is to view a sequence x of data as the sum of signals, well-behaved noise from a reasonable distribution and added impulsive noise (arbitrarily large amplitude impulses occurring occasionally). A simple selector, like the running median, is used to remove impulses of prescribed width. This is followed by linear smoothing and/or linear signal extraction. The underlying idea is that a linear process “smears” the energy of impulses over neighboring sequence elements, and makes it more diﬃcult to extract afterwards. It is therefore advan-

xii

Introduction

40

35

30

25 0

2

4

6

8

10

12

14

16

18

Figure 2. Data from a boat speedometer with impulsive noise.

tageous to remove and replace it, by comparison to immediate neighbors as early as possible, to lessen the damage by the subsequent linear smoothing processes. This is easy to understand and implement and relatively fast and economical, and leads to the subsequent established theory for signal extraction. These simple nonlinear smoothers (selectors and combinations of them) have over the last two decades received considerable attention, particularly in the engineering literature. Comparisons and analysis have been based mainly on the transfer of distributions if the noise is identically independently distributed random noise. A characterization of “roots”, or sequences that remain unchanged by smoothing, did slowly emerge in terms of “edges”, “constant regions” and “oscillations”, but the “enigmatic behavior” and lack of theory has been almost universally lamented (31). The design and choice of such smoothers has been based mainly on wisdom and experience. Tukey and others popularized smoothers like the three point median (M1 ), the three point median applied twice and followed by the ﬁve point running median (M2 M12 ), the repeated three point median (R = M1∞ ) and the recursive version (M1∗ ) of the three point median. Other rank order selectors are often mentioned, particularly the use of min/max operators in image processing, but the emphasis was on running medians and compositions (“concatenations”) of them. The idea of the median smoother links naturally with that of linear smoothers (ﬁlters) in the following way. Instead of projecting a given sequence x onto a chosen subspace (“signals”), a process that may be non-local, it is practical to consider sections of the sequence separately, by passing a “window” over it. Considering an element xi of x and the neighbors in the window {xi−n , . . . , xi , . . . , xi+n }, they are used in a projection onto a subspace (constant sequences, linear sequences or sampled higher order polynomials or trigonometric functions). The norm usually used

Introduction

xiii

is the p = 2 norm, leading to a linear mapping, for instance onto the best constant approximation to {xi−n , . . . , xi , . . . , xi+n }, from which a smoothed value, (the average of the elements in the window), is substituted for. This is repeated for each i. Another motivation for this particular norm is usually loosely given as the “Central Limit Theorem”. Many decades ago Whitaker and Robinson, in their book “Calculus of Observations”, already pointed out the often inappropriate addiction to, or misinterpretation of, the central limit theorem by quoting a witty remark of Poincar´e; “Everybody believes in the exponential law of errors; the experimenters, because they think it can be proved by mathematicians; and the mathematicians, because they believe it has been established by observation.” The Gaussian (normal-) distribution for “noise” on measurement can be motivated by the observation that if a large amount n of numbers, with typical roundoﬀ error from a uniform distribution, are added or subtracted, the result is a spline distribution of order n results. For n = 6 this already is almost indistinguishable on a typical graph from a Gaussian. When other operations like division are however also employed, they may result in vastly diﬀerent error distributions. This problem has been ampliﬁed by the vast increase in the number of calculations performed in a modern system. Diﬀerent methods of adjusting have to be explored. The convenience of a linear projection associated with the least squares norm is however a good motivation, leading to a simple computational and conceptual procedure. The value xi in a sequence is replaced by the weighted average of it and its neighbors. (Or the output sequence is the discrete convolute with a ﬁxed local sequence.) Although the linear projection in the window is idempotent, the global smoothing of the sequence, considered as a mapping between input sequence and output sequence, is not. In the presence of impulsive noise it is natural to replace the least squares projection with an approximation in a more robust norm, the 1 -norm. This leads to the replacing of xi by median {xi−n , . . . , xi+n }, and thus the (“running-”) median smoother Mn . Since a median is a good estimator of the average, the median and running average smoothers are similar in eﬀect when the noise is “reasonable”, but when the noise is from a “heavy-tailed” distribution the median becomes superior. It is natural to compare a median and similar smoothers by considering its “linear part” (14), providing a method of comparison. A general perception did persist that other methods of comparison and analysis are required to augment this comparison, but the concept of eigenanalysis of nonlinear operators, which is natural for linear operators, reeks of folly. There may however be considerable merit in it. Even a clear deﬁnition of signal, in the context of median smoothing, has generally been avoided and only emerged slowly. Once a clear characterization of “signals”, or the signiﬁcant “roots” of median smoothers, emerged, it became natural to consider the problem of “best approximation” onto signals (4). Although this leads to a computationally unappetizing process, there is merit in the insight obtained and a surprising twist in interpretation.

xiv

Introduction

A consistent framework for analysis and comparison of smoothers has been lacking. For linear ﬁlters the vectorspace framework has been useful, leading naturally to eigenanalysis and the associated analysis in the “frequency domain”. A natural alternative framework seems to be the semi-group generated by the unsymmetric minimum and maximum selectors, and the natural order on this semi-group. Not only does this lead to computationally convenient smoothers, almost naturally vectorisable, but it leads to a coherent conceptual framework for analysis and comparison of other rank order smoothers and their composition. The so-called LU LU -structure has emerged as worth aiming for, also in dimensions higher than 1. The LU LU -operators are actually particular cases of Morphological Filters, as developed by Serra (29) and others. This was discovered by the author in the mid-1990s, and independently by Maragos and Schafer (15), who informed me shortly afterwards. There is however a strong belief that the speciﬁc case has particular value. It is an interesting quirk of logic that that slight relaxation of one of the smoother axioms of Mallows leads to the possibility of adding additional useful and natural axioms, that can be satisﬁed by practical and consistent smoothers. The development of the so called LU LU -theory in the following chapters is based on initial practical work on real problems, for which the author had not been educationally prepared, and subsequent occasional publications on the subject (19)–(25). It introduces the ideas and methods of proof developed previously, but deviates where hindsight suggests a development that is simpler, clearer and more suggestive in indicating search directions in the many facets not yet investigated. It attempts to introduce scientists and mathematicians with a general background into a complementary method of analysis of measurements. As Collatz (10) pointed out long ago, the concepts of order, inequalities and lattices are perhaps the more fundamental. The ﬁnal result presented is a novel Multiresolution Analysis (MRA), complementary to that of the discrete wavelet (MRA), which decomposes a sequence into subsequences of linearly decreasing resolution. It does so in a very consistent fashion and explains the accepted good performance of Median Transforms and provides many pleasant and unexpected surprises. To prove a strong consistency, very important attributes of an operator, like Full Trend Preservation, Neighbor Trend Preservation and Co-idempotence are introduced, and demonstrated to be useful and natural. A similarly important measure of smoothness, Local Monotonicity is introduced, motivated and used extensively, and shown to link with the excepted natural measure of smoothness; Total Variation. Beauty, simplicity and elegance of poetic metaphors have always been useful when exploring in science and engineering, as in chess. The author is convinced, from his own experience, that the LU LU -theory is the natural self-contained route into the wonderful ﬁeld of Mathematical Morphology for all scientists and engineers educated in the usual type of undergraduate curriculum. The hope is that it becomes a well used and useful scenic highway.

1. Operators on Sequences Tu mir keine Wunder zulieb Gib deinen Gesetzen zurecht die von Geschlecht zu Geschlecht sichtbarer sind. Rainer Maria Rilke

For the study of sequences it is convenient to use a framework that yields creative insight. The vectorspace framework is convenient and it utilizes natural geometric insight. Let X be the set of bi-inﬁnite sequences x = xi of real numbers. As usual, the deﬁnition of addition and scalar multiplication is given by: Deﬁnition.

x ⊕ y = xi + yi , for x = xi and y = yi in X. α y = αxi , for y ∈ X and α a real number.

With these deﬁnitions X becomes a vectorspace, and the simpler notation x + y and αx can be used when no misunderstanding should arise. Depending on need and utility, diﬀerent metrics, norms and inner products can be chosen to make X a metric-, normed- and inner product space respectively. The following norms usually suﬃce, particularly in the cases of p = 1, 2 and ∞. ||x||p =

Deﬁnition.

∞

p1 p

|xi |

, for p = 1, 2, . . . ,

−∞

and ||x||∞ = sup{|xi |}, for p = ∞. i

Deﬁnition. (x, y) =

∞ −∞

12 x2i

is the usual inner product.

Remarks (a) Although sequences may generally in practice be ﬁnite, zeros can be added to make them bi-inﬁnite and bounded in any of the usual norms. (b) The topologies induced by the above norms usually suﬃce. (c) Unless otherwise stated we shall assume that all sequences are in 1 , or ||x||1 < ∞.

2

1. Operators on Sequences

Collatz considered orders to be even more basic and useful in numerical mathematics, and this turns out to be justiﬁed in the theory of nonlinear smoothers. As is usual an order relation is deﬁned using the usual deﬁnition of a relation as an ordered pair of elements from a set. Deﬁnition. Let A be a set and R a relation in A. R is a partial order (partial order relation) if the following are satisﬁed. (a) R is reﬂexive: (a, a) ∈ R, ∀a ∈ A. (b) R is anti-symmetric: (a, b) ∈ R and (b, a) ∈ R implies a = b. (c) R is transitive: (a, b) ∈ R and (b, c) ∈ R imply (a, c) ∈ R. The usual notation a ≤ b for (a, b) ∈ R and a < b if (a, b) ∈ R and a = b can be introduced. The word “partial” is introduced to suggest that not all pairs in A need be in R. If this is so, it becomes a total order, like the usual order on the real numbers, which is chosen as the point of departure here. Deﬁnition. For x, y ∈ X, x ≤ y if and only if xi ≤ yi ∀i. It is standard theory that the above deﬁnition results in a partial order on X, but not in a total order. This order on X induces a natural partial (not total) order on the set of operators on X (mappings from X to X) in the following way. Deﬁnition. If A and B are operators on X, then A ≤ B if and only if Ax ≤ Bx, ∀x ∈ X. It is useful to introduce brieﬂy the usual notation and operators on X. Deﬁnition. Let F (X) be the set of operators on X. Then: (a) (A + B)x = Ax + Bx, ∀x ∈ X. (b) Ix = x, ∀x ∈ X. (c) Ox = 0, ∀x ∈ X and 0 the null sequence. (d) (αA)x = α(Ax), ∀x ∈ X. (e) (AB)x = A(Bx), ∀x ∈ X. (f) (Ex)i = xi+1 , for each i and x ∈ X. (g) N x = −x, ∀x ∈ X. With the above deﬁnition it is standard theory that the addition deﬁnes a commutative group of operators, and with the scalar multiplication also a vectorspace for the linear ones. The deﬁnition of multiplication (composition) results in a semigroup with identity I, which is non-commutative. Combining all the deﬁnitions results in a near-ring of operators. Important to note is that the operators are generally right-distributive, since, by deﬁnition of addition, (A + B)C = AC + BC for all A, B, C ∈ F (X). Unless the operators are all linear, the operators are not generally left-distributive. Although composition is not commutative, non-negative integer powers of a single operator can be shown to commute. Thus, the following can be introduced in a prevalent way. Deﬁnition. A0 = I and An+1 = AAn , n = 0, 1, 2, . . ..

1. Operators on Sequences

3

Theorem 1.1. Am An = An Am for n, m non-negative integers. Proof. The proof is standard, utilizing induction.

Deﬁnition. An operator S is syntone if x ≥ y ⇒ Sx ≥ Sy. Theorem 1.2. If A and B are syntone operators, then AB is also syntone. Proof. Let x ≥ y. Then Bx ≥ By and ABx = A(Bx) ≥ A(By) ≥ ABy.

Theorem 1.3. Let A, B and D be operators on X and A syntone. Then B ≤ D ⇒ AB ≤ AD. Proof. ABx = A(Bx) ≤ A(Dx) = ADx, for each x ∈ X.

As point of departure the heuristic motivation for smoothing, namely separating some “signal” from a sequence where the signal is “hidden” (or contaminated) by “noise”, should be called to mind. Some simple criteria for design and comparison are appropriate. Eﬀectiveness Consistency Stability Eﬃciency

For each x, P x should be a signal and (I − P )x noise. Signals should be preserved and noise mapped onto 0. Small input perturbations should not distort output excessively. The computations should be economical.

Some of these have been formalized implicitly by sets of axioms, such as the axioms of Mallows. Serra suggests idempotence and syntoneness as axioms (29) for “morphological ﬁlters”. For the purpose at hand, the axioms of Mallows can be taken as the point of departure, weakened slightly to yield the principal smoother axioms. For an elite type, idempotence and co-idempotence can be added to yield the axioms of separators. For consistency it is reasonable and necessary to demand that the concepts of signal and noise should be translation independent on both axes. This leads to the demands that P (x + c) = P (x) + c and P (E j x) = E j (P x)), which essentially yield the ﬁrst two axioms of Mallows. It is also reasonable to demand scale independence. This requires that P (αx) = αP (x), for α ≥ 0. (Mallows omits the restriction to α ≥ 0, which seems too lenient). If 0 is a signal it is also noise, since (I − P )0 = 0. This yields the following axioms. Smoother Axioms. An operator P on X is a smoother if: 1. P E = EP . 2. P (x + c) = P (x) + c, for each x, c ∈ X such that c is a constant sequence. 3. P (αx) = αP (x), for each x ∈ X and scalar α ≥ 0. Separator Axioms. A smoother P is a separator if it also satisﬁes the additional axioms: 4. P 2 = P 5. (I − P )2 = I − P

(Idempotence). (Co-idempotence).

4

1. Operators on Sequences

The separator axioms can be heuristically argued by a simple analogy to be reasonable demands. Suppose P is a separator that separates milk x into two components; P x, the curd, and (I − P )x, the whey. If P is the only instrument at our disposal, what simple tests could be performed to test the performance of P and what procedure could be followed to improve the performance? A simple test would be to take the curd, and pass it through the separator again. Alternatively, or additionally, we could pass the whey through the separator again. This is schematically illustrated in the following diagram.

x

P

Px

(I − P )x

P

P 2x

P

P (I − P )x (I − P )P x

(I − P )2 x

Figure 1.1. Diagram of a two-stage separator cascade.

Interpretations of what happens include the following, where the curd is arbitrarily chosen to be the signal and the whey the noise. (This arbitrary choice is signiﬁcant.) 1. If P 2 x = P x, for each sample x, then the separator can be considered signalconsistent. 2. If (I − P )2 x = (I − P )x, for each sample x, the separator can be called co-idempotent and considered noise-consistent. 3. A perfect consistent separator is both idempotent and co-idempotent. 4. If a separator P is not idempotent it can be considered a deﬁcient noiseextractor. A better separation could then be achieved by passing the signal through P again. Then P 2 x can be considered the signal and (I − P )P x + (I − P )x the noise.

1. Operators on Sequences

5

5. If a separator P is not co-idempotent, then it can be considered a deﬁcient signal-extractor. A better separation could be achieved by passing (I − P )x through P again. Then (I −P )2 x can be considered the noise and P x+P (I − P )x the signal. 6. If P is neither idempotent nor co-idempotent, then a better separation can be achieved by passing P x through P again and also passing (I −P )x through P again, and considering P 2 x + P (I − P )x as the signal yield and (I − P )P x + (I − P )2 x as the noise yield. The above interpretations are self-evident but often overlooked. One of the reasons for this is that for projections the separation is consistent, since then the separator is idempotent and co-idempotent. Even for nonlinear operators like best approximations from subspaces, the operator is idempotent and co-idempotent, as 0 is in each subspace. The following is a useful observation. Lemma. P is co-idempotent if and only if P (I − P ) = 0, where 0 is the operator mapping all sequences onto the zero sequence. Proof. I − P = (I − P )2 = (I − P )(I − P ) = I − P − P (I − P ). This is true if and only if P (I − P ) = 0.

The lemma conﬁrms directly that for linear operators, idempotence and coidempotence coincide. This may be the reason why co-idempotence has been generally overlooked as a useful concept. The following simple examples, communicated privately by van der Walt and le Riche respectively, demonstrate that not all nonlinear idempotent operators are co-idempotent and that some are. It therefore seems to make sense to distinguish between them. Example 1. Let P x = |x| = {|xi |}. Clearly P 2 = P , since P 2 x = P |x| = {||xi ||} = {|xi |} = P x. (I − P )x = {xi − |xi |} and (I − P )2 x = 2(I − P )x. Therefore P is not co-idempotent. Example 2. Let P x = {yi } such that yi =

xi 0

if xi ≥ 0 . if x ≤ 0

Then P 2 x = P x and (I − P )x = {ti } such that xi if xi < 0 . ti = 0 if xi ≥ 0 But then P (I − P )x = 0 so that (I − P )2 = I − P and P is idempotent. P is therefore a separator. (It is also clear that there are only two eigenvalues; 1 and 0.) Important classes of basic smoothers are order selectors (rank order selectors), order based selectors and selectors. An n-window is successively centered at

6

1. Operators on Sequences

each index i to yield the support {i − n, i − n + 1, . . . , i, . . . , i + n} so that the set of values Wi = {xi−n , . . . , xi , . . . , xi+n } is selected from the sequence x. Deﬁnition. An order selector (rank order selector) S maps the sequence x onto Sx such that (Sx)i = xj ∈ Wi , and such that xj is a given rank in the order on Wi . A rank based selector uses rank and other information to select an element from Wi and a selector need not use the rank or order in Wi . To prove that order selectors are smoothers is simple, but additionally they are syntone. Theorem 1.4. Rank Order selectors are syntone. Proof. Let S be an order selector. Then (Sx)i is a given rank, say the kth largest element of {xi−n , . . . , xi , . . . , xi+n }. If y ≥ x then the kth largest element of {yi−n , . . . , yi , . . . , yi+n } is not smaller than (Sx)i . Thus (Sx)i ≤ (Sy)i . But this is true for each index i, and therefore Sx ≤ Sy, by the deﬁnition of order on sequences. All selectors, and also nonrecursive digital (linear-) ﬁlters, are local operators in the sense that the value of the image of x at the index i is independent of the values of xj outside the n-window centered at i. Features of a sequence far away do not inﬂuence the image locally. Theorem 1.5: The set of smoothers form a semigroup with composition and the usual identity. Proof. I is a smoother, as it satisﬁes all the axioms. Let A and B be smoothers. Then clearly (AB)(0) = A(B(0)) = A(0) = 0, by axiom 1. Also AB(x + c) = A(B(x) + c) = A(B(x)) + c, by axiom 2. If E is the shift operator then (AB)E = A(EB) = (AE)B = (EA)B = E(AB) by Axiom 3. It is clear that the composition of two idempotent operators is again idempotent if they commute. However, compositions of idempotent operators can be idempotent without commuting. (The operator LnU n, as deﬁned later, is idempotent, but LnU n = U nLn.) Theorem 1.6. Let A, B be smoothers and α + β = 1. Then αA + βB is a smoother. Proof. (αA + βB)(x + c) = αA(x + c) + βB(x + c) = α(A(x) + c) + β(B(x) + c) = αA(x) + βB(x) + c = (αA + βB)(x) + c, (αA + βB)E = αAE + βBE = αEA + βEB = E(αA) + E(βB) = E(αA + βB), (αA + βB)(γx) = αA(γx) + βB(γx) = αγA(x) + βγB(x), if γ ≥ 0 = γ(αA + βB)(x).

1. Operators on Sequences

7

Deﬁnition. An operator A is linear if A(αx+βy) = αA(x)+βA(y) for all x, y ∈ X, α ∈ R. The linear operators form a ring of operators since they are right-distributive, and are a very important class of operators. Linear (digital-) ﬁlters are a special class of smoothers that have extensive applications in many ﬁelds. There is a welldeveloped theory for analysis, design and comparison (11). The theory is based on eigenvalue analysis and statistical analysis, and is associated with approximation, and closely linked to Fourier Analysis, Wavelet Analysis etc. Digital ﬁlters are operators that are linear combinations of the form F xi = j αj E j xi−j . In this sense the shift operators are a basis for the vectorspace of linear operators on sequences. For nonlinear operators an alternative is to consider compositions of “basic” smoothers in a semi-group structure.

2. Basic Rank Selectors, Pulses and Impulses Zwei mal zwei gleich vier ist Wahrheit. Schade, daß sie leicht und leer ist. Denn ich wollte lieber Klarheit. ¨ Uber das, was voll und schwer ist. Wilhelm Busch

A general perception has been prevalent that medians are the “basic” smoothers required to generate other speciﬁcally designed composite smoothers. The smoothers Mn , Mnk and R = lim Mnk have been popularized by Tukey. k→∞

Mallows and others have extensively investigated them for statistical properties, and by analogy with digital ﬁlters, studied the “linear” component of such smoothers. Other popular smoothers derived from the “basic” smoothers Mn are the smoothers like M2 , M12 , 2Mn − Mn2 and the recursive versions of Mn . Winsorisers are based on the “basic” rank selectors that use quartiles of the elements in a window. Extensive experience in linear smoothing and statistics may have obscured the possibility of obtaining insight by breaking up the conceptual “atoms” into “positive” and “negative” constituents, that are more basic selectors. Although they do not make up the “atoms” exactly, they suggest a possibility, using order and syntoneness in the semi-group structure, to compare and analyze many smoothers, including the popular rank order smoothers and composition. They also suggest search directions for the construction of smoothers that have many optimal, or near optimal properties. Considering the medians as “basic” smoother may have had the result of an almost universal exceptance that analysis of nonlinear smoothers is diﬃcult, if not impossible. Considering the extreme selectors, the minimum (“erosions”) and the maximum (“dilations”), as “basic” constituents of composite smoothers for the removal of impulsive noise may well be the better alternative. The use of compositions (“openings”) and (“closings”) of these “basic” selectors has been well known in image processing (29), but a comprehensive theory for understanding the “enigmatic” behavior of nonlinear smoothers comes from their compositions. They can be shown to allow a slightly stricter set of axioms than had been accepted, in the form of those of Mallows. The following development is not the historical path of discovery, but provides some additional insight, and suggests eﬃcient recursive evaluation.

10

2. Basic Rank Selectors, Pulses and Impulses

Deﬁnition. Let the operators and that map the sequence x onto y be deﬁned by : y= x = yi = max{xi , xi+1 }, : y= x = yi = min{xi−1 , xi }. Theorem 2.1.

, and I are syntone.

Proof. Let x > y. Then, for each index i it follows that, ( x)i = max{xi , xi+1 } ≥ max{yi , yi+1 } = ( y)i .

is syntone, and the proof for I is trivial. Although classiﬁed as smoother, since it is a selector, or ( ) does not comply with our intuition as far as smoothers are concerned. This is easily demonstrated on a constant sequence witha single superimposed impulse, as is given by the sequence x = xi = δij when x = ( x)i = δi,j + δi,j+1 . The pulse has been spread and needs “deconvolution” with so that x = x. This seems a futile exercise, but a sequence with a downward pulse, like −x will have it removed by , and the operator cannot recreate it, so that (−x) = (−x). To remove both upward and downward isolated single impulses we need the compositions and , which are not the same, as the above example demonstrates. The following result is somewhat more subtle than the obvious inequality ≤ I ≤ . Theorem 2.2. ≤I ≤ . Proof. ( x)i = max{ xi , xi+1 } Similarly it follows that

= max{min{xi , xi−1 }, min{xi , xi+1 }} ≤ max{xi , xi } = (Ix)i .

The rest follows similarly. Theorem 2.3. For each non-negative integer n, n+1 n+1

Proof.

n+1 n+1

≤

n n

and

n n = ( )

≤

n

I

n

n+1 n+1

≥

n n

.

(associativity). (syntoneness of

n

).

A similar proof follows for the rest. Theorem 2.4. (The ﬁrst swallowing theorem). For each n ≤ m it follows that n n m m

=

m m

and

n n m m

=

m m

.

2. Basic Rank Selectors, Pulses and Impulses

Figure 2.1. Graphs of a random sequence x,

11

x,

x,

2 2

x and

2 2

x.

n n n m−n m n m−n m m m ( ) ≤ I ≤ .

Proof. Furthermore

m m n n

≥I

m m ,

so that the ﬁrst part of the theorem is proved. The rest follows similarly.

When n = m this can also be called the ﬁrst idempotence theorem. Theorem 2.5. For each non-negative integer n, n n n n n n n n n n n n ≤ ≤ n n n n

Proof.

n n

= ≤ ≤ =

≤

n n n n n n

n n n n n n n n n n n n n n 0 0 n n n n n n

.

A similar argument proves the rest.

≤

n n

and

.

(Theorem 2.4)

(Theorem 2.2, 2.3)

(Theorem 2.3)

12

2. Basic Rank Selectors, Pulses and Impulses

Figure 2.2. Graphs of x,

x,

x,

x and

x.

Numerical experiments indicate that x≤ x, and although this is true, it cannot be proved from the semi-group structure and the order alone. However the following three theorems follow algebraically. Theorem 2.6. (The second idempotence theorem). For each non-negative integer n;

n n n n 2 n n n n =

Proof.

n n n n

=

n n n n n n

n n n n n n n n

≥

≥ =

and

n n 2 n n n n n n . =

n n n n n n

=

=

(Theorem 2.4)

(Theorem 2.2, 2.3)

n n n n n n n n n n

I

n n n n

n n n n

.

(Theorem 2.4)

The rest follows similarly. Theorem 2.7.

n n n n n n

≤

n n n n n n

.

2. Basic Rank Selectors, Pulses and Impulses n n n n n n

Proof.

≤I ≤

13

n n n n

≤

n n n n

n n n n n n

I

.

Theorem 2.8. (The third idempotence theorem). For each non-negative integer n;

n n n n n n 2

n n n n n n 2

Proof.

=

=

n n n n n n

n n n n n n

and

.

n n n n n n n n n n n n

=

=

=

n n n n n n n n n n

(Theorem 2.4)

n n n n n n n n n n n n n n n n

.

(Theorem 2.6)

The rest follows similarly. With the suggestive notation Ln =

n n

,

Un =

n n ,

it follows that; 1. Ln+1 ≤ Ln ≤ L0 = I = U0 ≤ Un ≤ Un+1 .

(Theorem 2.2, 2.3)

2. (Ln Lm = Lm and Un Um = Um ) for m ≥ n.

(Theorem 2.4)

3. Ln ≤ Ln Un Ln ≤ Ln Un and Un Ln ≤ Un Ln Un ≤ Un .

(Theorem 2.5)

4. (Ln Un )2 = Ln Un and (Un Ln )2 = Un Ln

(Theorem 2.6)

5. Ln Un Ln ≤ Un Ln Un .

(Theorem 2.7)

6. (Ln Un Ln )2 = Ln Un Ln and (Un Ln Un )2 = Un Ln Un .

14

2. Basic Rank Selectors, Pulses and Impulses

The following theorem is very useful. Theorem 2.9. For each integer n ≥ 0 the following are equivalent. (a) LnU n ≥ U nLn. (b) LnU nLn = U nLn. (c) U nLnU n = LnU n. Proof. Let (a) be true. Then Ln(U nLn) ≤ IU nLn = U nLn, since Ln ≤ I. But (LnU n)Ln ≥ (U nLn)Ln = U n(LnLn) = U nLn, since LnU n ≥ U nLn and Ln2 = Ln. Therefore (a) implies (b). Suppose that LnU nLn = U nLn. Then (U nLn)U n = (LnU nLn)U n = (LnU n)(LnU n), = LnU n, by the idempotence of LnU n. Therefore (b) ⇒ (c). Suppose U nLnU n = LnU n. Then LnU n = (U nLn)U n ≥ U nLnI, since U nLn is syntone. Therefore (c) ⇒ (a). This completes the proof of the theorem.

A useful insight obtained is that, for each positive integer n, the operators L = Ln and U = U n form a semi-group that is completely ordered if the inequality LU ≥ U L can be proved. In such a case the ordered semi-group shall be called an LU LU -structure, and its complete multiplication table is; L

U

UL

LU

L

L

LU

UL

LU

U

UL

U

UL

LU

UL

UL

LU

UL

LU

LU

UL

LU

UL

LU

The order is then completely determined by:

L ≤ U L ≤ LU ≤ U.

The fourth swallowing theorem LU L = U L and U LU = LU will be proved via the signiﬁcant inequality LnU n ≥ M n ≥ U nLn, where M n is the well-known median smoother. It should be clear that in the deﬁnition of and there is an alternative choice for the shift of index in the maximum and minimum. However, when the combinations Ln = n n and U n = n n are considered, it is useful to note that they are unsymmetric only in that Ln(−x) = −U nx. This can be formalised by

2. Basic Rank Selectors, Pulses and Impulses

15

Theorem 2.10. LnN = N U n. Proof.

n

xi = max{xi , xi+1 , . . . , xx+n } = − min{−xi , −xi+1 , . . . , −xi+n }

n

and

xi = min{xi−n , . . . , xi } = − max{−xi−n , . . . , −xi }.

But

LnN x = Ln(−x) =

n n n n − x = (−x)

=

−

n n

xi

i

=N

n n

i

x,

for each x. Hence LnN = N U n.

Deﬁnition. An operator B is the dual of A if AN = N B. It is clear that Ln and U n are duals of each other and that all compositions of operators of these two types are again dual to the operator formed by interchanging Ln and U n, for each n. Any inequality comparing two such compositions changes the inequality around when such duals are interchanged, for example if LU ≤ U , then N L = U N ≥ LU N = LN L = N U L, so that L ≤ U L. It is therefore expedient to speak of dual theorems, and the duality can be used to prove one from the other by simple algebra. Remarks 1) Wenote that and are not duals of each other but that E −1 N = N or EN = N , by the deﬁnition and the fact that E commutes with , and N . 2) Any operator that commutes with N can be considered a dual of itself, and any pair of operators A and B such that AN = BN are duals of each other. Therefore N, I, O, E are all self-dual operators. At this stage the concept of “signal” and “noise”, which have been undeﬁned, can slowly be developed, more or less on historical lines by considering blockpulses. Deﬁnition.

A (primitive) n-blockpulse is a blockpulse of length n. It is called upward if it is non-negative and downward if it is non-positive.

Deﬁnition.

A (primitive) n-pulse is a sequence x such that xj = 0 for i ∈ [j + 1, j + n] for some integer j.

Theorem 2.11: Let x be an upward n-blockpulse and z a downward n-pulse. For each integer k ≥ 0 the following are true:

16

2. Basic Rank Selectors, Pulses and Impulses

Figure 2.3. An upward blockpulse, a downward blockpulse and a pulse in between.

(a)

k

(b)

k

(c)

k

(d)

k

x is an upward blockpulse of length max{0, n − k}. x is an upward blockpulse of length n + k. z is a downward blockpulse of length n + k. z is a downward blockpulse of length max{0, n − k}.

0 0 = I so that x is an upward blockpulse of length n. Proof. (a) Induction on k. Assume (a) holds for k < n. Then ∃j such that

k

= 0 for

x

i ∈ {j + 1, j + 2, . . . , (j + max{0, n − k})}.

i

But then

k k+1 k x = x = min x i

i

k , ( x)i

i−1

for all i ∈ {j + 2, j + 3, . . . , (j + max{0, n − k})}. But this is a blockpulse of width max{0, n − (k + 1)}. If k = n − 1 then k+1 x = 0. The proof for the rest of the theorem follows similarly, from the deﬁnition of and , by induction. Clearly upward blockpulses are widened by and downnarrowed by and ward blockpulses are widened by and narrowed by . Once either has been narrowed to a width of 0, the result is the zero sequence (an elementary blockpulse!).

2. Basic Rank Selectors, Pulses and Impulses

17

Theorem 2.12. Let x be an upward blockpulse of length n. Then: m m 0, for m ≥ n x= x, for 0 ≤ m < n and

m m

x = x, for each m ≥ 0.

Proof. Let x be an upward n-blockpulse of amplitude mα. x is a blockpulse of length If m ≥ n, then by Theorem 5.1(a) it follows that max{0, n − m} = 0. m xis ablockpulse of length n − m, by Theorem 5.1(a). If m < n, then m m ( x) is a blockpulse of length n − m + m = n. By Theorem 5.1(b) Clearly the amplitude of a blockpulse remains at thesame value α through m m mapping by and . By comparing indexes where x and x are zero they are shown to beidentical. Similarly, m x is a blockpulse of length n + m and m m x is a blockpulse of length n and amplitude α. Comparing indexes where they are zero it is shown m m that x = x. Corollary. The Dual theorem of 5.2 holds, since if x is anupward n-blockpulse, m m m m m m N = N , since and N x is a downward n-blockpulse and m m are dual operators. Clearly, the last theorem shows that n-blockpulses are annihilated (mapped Lm upward onto 0) by provided they are respectively if m ≥ n and Um

downward

identically preserved if m < n. If blockpulses of length k ≤ n are considered as noise, then Lm and U m are smoothers for the upward and downward blockpulses respectively, and LmU m would remove both such upward and downward blockpulses, since U m removes the downward blockpulses and preserves the upward pulses, which are then removed by Lm. Similarly U mLm removes both types. When the blockpulses are wide enough they are perfectly preserved by both composite smoothers U mLm and LmU m. These operators clearly meet the primary requirements of smoothers, and are syntone as well. Syntone operators are order preserving. This useful property will be used to prove that they remove a wider class of noise impulses, at least when these are added to a constant sequence. To show this the following deﬁnition is required. Deﬁnition. An Impulse (n-impulse)is a sequence x such that xi = 0 for i ∈ {j + 1, . . . , j + n}, for some integer j. This deﬁnition is the same as for an n-pulse, but signiﬁes that it is chosen as impulsive noise. Theorem 2.13. Let x be an n-impulse. Then LmU mx = U mLmx = 0 for m ≥ n.

18

2. Basic Rank Selectors, Pulses and Impulses

Proof. There exist n-blockpulses u and −u such that −u ≤ x ≤ u with ||u|| < 0. U mLm and LmU m are syntone, so that U mLm(−u) ≤ U mLm(x) ≤ LmU m(x) ≤ LmU m(u). But LmU m(u) = 0 and −LmU m(u) = U mLm(−u) = 0, so that U mLm(x) = LmU m(x) = 0.

Corollary: An n-pulse is removed from any constant sequence by LmU m and U mLm, if m ≥ n. The operators U and L as deﬁned here meet all the requirements of Morphological Filters (29), and in the terminology used there; U is “extensive”, as I ≤ U , and L is “anti-extensive” as I ≥ L. Furthermore U is a “closing” as it is “increasing” (syntone), extensive and idempotent and L is an “opening” as it is increasing, anti-extensive and idempotent. There is an alternative to using compositions of U and L to remove upward and downward impulses from constant sequences. The diﬀerence between the pair LU and U L may turn out to be illuminating, but if an unbiased smoother is desired the median is available, since it is an unbiased estimator of the average of the same elements. The following heuristic argument seems sound; since U removes downward pulses and L upward pulses the sum should be twice the signal plus both impulses. Subtracting the signal plus both impulses, namely the sequence x, should leave virtually only the signal. Deﬁnition. Qn = U n + Ln − I. Theorem 2.14. Let x be an n-blockpulse. Then Qnx = 0. Proof. If x ≥ 0 then U nx = x and Lnx = 0 by the above theorem. Thus Qx = U x + Lx − x = x + 0 − x = 0. If x ≤ 0 then Lnx = x and U nx = 0, and again Qx = 0.

The operator Q is selfdual since QN = U N + LN − N = I − U − L = N Q. Thus it is unbiased, like the median smoother. Theorem 2.15. Qn is a smoother. Proof. From Theorem 1.6 it follows that 2U n − I and 2Ln − I are smoothers. By the same theorem then, their average is also a smoother 1 1 2 (2Ln − I) + 2 (2U n − I) = U n + Ln − I. Remarks. (a) Qn is not idempotent, since if: x = (−1)i , U nx = 1 and Lnx = −1. Therefore Qn x = −x, and similarly Q2n x = x.

2. Basic Rank Selectors, Pulses and Impulses

19

(b) Qn is not necessarily syntone, and not necessarily a selector. However, Q1 is a syntone selector. It is easy to show that Q1 = M1 but this identity only holds in n = 1. Clearly Qn seems as useful and at least as enigmatic in behavior as the median Mn . At this point some problems also become clearer, since if an n-impulse is added to a general monotone sequence, its removal is not certain. In fact, even its identiﬁcation is problematic. Consider the case of an impulse xi = δij added to a monotone sequence sj = αi, for some scalar α. If α > 1 then si + xi is monotone increasing, and so is si − xi . If α is not known, how is the presence of impulsive noise to be identiﬁed at the index j? Even more basic. If a primitive 1-impulse nj is removed by L1 U1 and U1 L1 , and another nk also, what about the sum of two such “noise”-impulses? If j − k > n then they are both removed from constant sequences by L1 U1 and U1 L1 , but if j − k = ±1 or j − k = ±2 they are not. In both the above examples an ambiguity arises. How is this to be interpreted, particularly with respect to the relatively simple task of extending the deﬁnition of signal and noise to larger classes of sequences? Is there any unambiguous deﬁnition of noise or signal? If so what are they, and is the characterization illuminating? Furthermore; how much has been achieved in clarifying the enigmatic, erratic behavior of the popular median smoothers? It is clear that the median smoothers, deﬁned by (Mn x)i = median{xi−m , . . . , xi , . . . , xi+m }, remove n-blockpulses from constant sequences as well as Lm Um and Um Lm , since the majority of the set {xi−m , . . . , xi+m } are equal to the constant value of the constant sequence. What has been gained? How are smoothers like Lm Um , Um Lm and Mm to be compared? And what about the many others that seem competitive, are popular and treated as “basic”? Is there a sensible, useful deﬁnition of signal and noise which separates X into two sets, sharing only the zero sequence? At this stage it seems reasonable that if pulses of width (n + 1) are to be considered signals, then at least, impulses of lesser width are noise. The concept will be generalized after this has been done to the concept of signal.

3. LU LU -Smoothers, Signals and Ambiguity Wollest mit Freuden Und wollest mit Leiden Mich nicht u ¨bersch¨ utten Doch in der Mitten Liegt holdes Bescheiden. Eduard M¨ orike

It is clear how the operators LnU n and U nLn treat pulses of suﬃcient briefness when they are added to constant sequences or to sequences that are constant for a suﬃciently large neighborhood of the pulse. Since the operators are non-linear they clearly could be treated diﬀerently when added to another sequence, since they can only be detected as pulses with respect to their neighboring values. One way is to characterize the sequences that pass unaltered through LnU n and U nLn and consider these signals, and then to consider the residue (I − LnU n)x, and (I − U nLn)x as the noise that is superimposed on the signal. This turns out to be more illuminating than the alternative. This idea was used with the median smoother M n, and resulted in a less than simple characterization in terms of “edges”, “constant regions” etc. In the case of the LU LU -operators, this turns out to be simpler. Theorem 3.1. Let s < q < t and |t − s| < n + 2. If (Lnx)s < (Lnx)q , if

(U nx)s > (U nx)q ,

then (Lnx)t ≥ (Lnx)q ,

and

then (U nx)t ≤ (U nx)q .

Proof. Suppose (Lx)s < (Lx)q and (Lx)t < (Lx)q , with L denoting Ln. Since (Lx)s < (Lx)q , (Lx)q > min{xj−n , . . . , xj }, for each j ∈ [s, s + n]. Since (Lx)t < (Lx)q , (Lx)q > min{xj−n , . . . , xi } for each j ∈ [t, t + n]. Since |s − t| < n + 2, min{xj−n , . . . , xj }, < (Lx)q for j ∈ [s, t + n], and (Lx)q is one of these minima, which is a contradiction. Therefore (Lx)t ≮ (Lx)q . The rest of the theorem follows, since U n is the dual of Ln. The above theorem demonstrates that the image y = Lnx of a sequence x cannot have local outliers upwards. Similarly y = U nx so that it cannot have downward outliers. This can be formally proved in the following theorem. Deﬁnition. A sequence x is n-monotone if for each j, {xj , xj+1 , . . . , xj+n+1 } is monotone.

22

3. LU LU -Smoothers, Signals and Ambiguity

Figure 3.1. The output L1 x of a random sequence, showing no points larger than both neighbors.

Theorem 3.2. For each n, Lnx = U nx if and only if x is n-monotone Proof. Let Lx = Lnx = U nx = U x and X = {xi−n , xi−n+1 , . . . , xi , xi+1 }. Since Lx ≤ x ≤ U x, ∀x ∈ X, it follows that x = Lx = U x. Let xq be the ﬁrst element in X that diﬀers from xi−n . Assume that xq > xi−n . Then (Lx)q > (Lx)i−n and, by the previous theorem, it follows that (Lx)q+1 ≥ (Lx)q and therefore xq+1 ≥ xq . By induction it follows that xj+1 ≥ xj , for each j ∈ {i − n, . . . , i} showing that the n + 2 successive numbers are monotone increasing. If xq < xi−n , a similar argument with U proves that X is monotone decreasing, and thus that {xi−n , . . . , xi+1 } is monotone. Since this is true for each index i, the sequence x is n-monotone. Conversely, consider the set A = {xi−n , . . . , xi+n }, with x being n-monotone. If the set is monotone increasing, min{xj , . . . , xj+n } = xj , for each ∈ [i − n, i − 1]. But then (Lx)i = max{xi−n , xi−n+1 , . . . , xi } = xi , and a similar argument shows that if A is increasing then again (Lx)i = xi . Suppose therefore that A is not monotone. Then there is a j and two subsets {xj−1 , . . . , xj+n } and {xj , . . . , xj+n+1 } of A, such that one is monotone increasing and the other monotone decreasing. The intersection {xj , . . . , xj+n } must then be constant and contain xi . Therefore (Lx)i , which is the maximum of the minima of such subsets, one of which is min{xj , . . . , xj+n } = xi , is not smaller than xi . Since Lx ≤ x for any sequence, this means that (Lx)i = xi . A similar argument, or duality, shows that (U x)i = xi , and since the whole argument is valid for each index i, it follows that Lx = x = U x. What is more important for the characterization of the output of the smoothers LU and U L are the following theorems, which characterize the range of both the smoothers in terms of an important concept. Theorem 3.3. For each n, LnU nx = x = U nLnx if and only if x is n-monotone. Proof. Let x be n-monotone. Then, by the previous theorem, U x = Lx = x. But then U (Lx) = U x = Lx = L(U x) = x. Conversely, let U Lx = LU x = x. Then Lx = L(LU x) = LL(U x) = LU x = x and U x = U (U Lx) = U U (Lx) = U Lx = x. By the previous theorem, this implies that x is n-monotone.

3. LU LU -Smoothers, Signals and Ambiguity

23

Corollary. U nLnx = x if and only if x is n-monotone, and U nLnx = x if and only if x = LnU nx. Proof. U nx = U n(U nLnx) = U nLnx = x and similarly Lnx = Ln(U nLnx) = U nLnx = x, by the following theorem’s corollary. It should be remarked that U nLnx = LnU nx does not imply that x is nmonotone. This contrasts with the case when U nx = Lnx, where the fact that Ln ≤ I ≤ U n, proves that U nx = Lnx = x. But generally U nLnx = LnU nx does not imply that x = LnU nx = U nLnx, since generally it is not true that x lies between U nLnx and LnU nx.

Figure 3.2. U Lx and LU x for a random sequence x demonstrating 1-monotone output.

Theorem 3.4. For each integer n ≥ 0, U nLn ≤ M n ≤ LnU n. Proof. Assume (M nx)i > (LnU nx)i for some sequence x and index i. From the deﬁnition of Ln it follows that there are two indexes s and t such that (U nx)s , (U nx)t < (M nx)i , with i ∈ [s, t] and [s − t] ≤ n. But then, from the deﬁnition of U n, there exist two indexes j ∈ [s, s + n] and q ∈ [t, t + n] such that max{xj−n , . . . , xj }, max{xq−n , . . . , xq } < (M nx)i . Consider the union {xj−n , . . . , xq } = {xj−n , . . . , xj } ∪ {xq−n , . . . , xq }. It contains at least n + 1 elements from the set {xi−n , . . . , xi+n }, which are all smaller than the median, which in turn cannot be larger than more than n of the elements. This contradiction proves the one inequality. A similar proof yields the other, or alternatively, an argument using the duals of M n and LnU n concludes the proof of the theorem. Corollary. U nLnU n = LnU n and LnU nLn = U nLn, by Theorem 2.9. Theorem 3.5. For each sequence x ∈ X, U nLnx and LnU nx are n-monotone.

3. LU LU -Smoothers, Signals and Ambiguity

24

Proof. By the above corollary it follows that U n(U nLnx) = (U nU n)(Lnx) = U n(Lnx) = U nLnx and Ln(U nLnx) = (LnU n)(Lnx) = (LnU nLn)x = U nLnx. But then, by Theorem 3.2. U nLnx is n-monotone. A similar proof follows for LnU nx.

It is illuminating to take stock of what has been achieved. The operators LnU n and U nLn share the range Mn , given by the following deﬁnition. Deﬁnition. Mn is the set of all sequences in x that are n-monotone. The sequences in Mn can therefore be considered the signals, as interpreted by both operators, and LnU n and U nLn can be considered to be extensions of the identity operator on the set Mn . The previous two theorems also imply that the sequences are all roots of the operators Mn , Mn2 and the (nonexistent) operator R = M ∞ . Thus all the popular smoothers used to remove the n-impulses share these common “roots”, as they were also often called. Some ﬁnite powers of Mn do have roots that are not in Mn , but these can be considered an insigniﬁcant minority. These roots, not in Mn , can be called spurious roots. An example is the sequence x = (−1)i which is a root of M2 , since clearly M2 x = x. (This sequence is precisely a sampling of the Nyquist-frequency , which should hardly be considered as signal.) It is important to note that the median operator M n is not equal to either U nLn or LnU n. Example. The sequence x = (−1)i yields M n(x) = (−1)i+n , whereas U nLn(x) = Ln(x) = −1 and LnU n(x) = U n(x) = 1. The operators are however equal on the very large set Mn , and on a larger set , namely the set of all roots of the operators U n or Ln. This is demonstrated by the following theorem. Theorem 3.6. For x = U n(x) or x = Ln(x), M n(x) = LnU n(x) = U nLn(x). Proof. LU x = LU (U x) ≥ M (U x) ≥ U L(U x) = (U LU )x = LU x.

Corollary. M nU n = LnU n and M nLn = U nLn. It is important to understand that the operators M nU n and M nLn are not the same as U nM n and LnM n respectively. The latter two are not idempotent operators whereas the above corollary shows that M nU n and M nLn are idempotent. Theorem 3.7. For each n ≥ 0 and j ≥ 0, using the notation M = Mn , LnM j ≤ LnM j+1

and U nM j ≥ U nM j+1 .

3. LU LU -Smoothers, Signals and Ambiguity

Proof.

LnM n ≤ U nLnM n,

25

since I ≤ U n,

= LnU nLnM n, since U nLn = LnU nLn, = LnM nLnM n, since U nLn = M nLn, ≤ LnM nIM n. j

Therefore LnM = (LnM n)M j−1 ≤ LnM 2 nM j−1 ≤ LnM j+1 . A similar argument (or an argument with duals) proves the rest of the theorem. It is easy to construct particular sequences to show that the inequalities in the theorem cannot be replaced by equalities. Example. The sequence x = xi = δi,j−1 − δi,j + δi,j+1 containing an alternating triplet of pulses. Since m1 x = mi = δi,j and M12 x = 0, it is clear that U1 M1 x = M1 x has a unit pulse, whereas U1 M12 x = 0. This example also demonstrates that U1 M1 is not idempotent, since applying M1 to U1 M1 yields the zero sequence and therefore also U1 M1 (U1 M1 x) = 0 = U1 M1 x.

Figure 3.3. The operator U1 M1 applied to the sequence of three alternating inputs.

The operators U M j and LM j do however yield a root quickly (after two applications), as the following theorem demonstrates. Theorem 3.8. For each n ≥ 0, (LnM n)3 = (LnM n)2 and (U nM n)3 = (U nM n)2 . Proof.

(U nM n)(U nM n) = U n(M nU n)M n = U n(LnU n)M n, since M nU n = LnU n, = LnU nM n,

since U nLnU n = LnU n.

Using this it follows that (U nM n)(U nM nU nM n) = (U nM n)(LnU nM n). Since U n(M nLn)U nM n = U n(U nLn)U nM n = LnU nM n, it follows that (U nM n)2 = (U nM n)3 = LnU nM n. This proves one half of the theorem, and a similar argument completes the proof of the theorem.

3. LU LU -Smoothers, Signals and Ambiguity

26

Investigating the proof of the above theorem makes it clear that the operators (U nM n)2 and (LnM n)2 map any sequence onto an n-monotone sequence. The same is true for the operators (U nM nj )2 and LnM nj )2 as the following theorem demonstrates. Theorem 3.9. For each n ≥ 0 and j ≥ 0, (U nM nj )2 = (U nM nj )3 Proof.

and (LnM nj )2 = (LnM nj )3 .

(U nM nj )2 = U n(M nj U n)M nj = U nM nj−1 (M nU n)M nj = U nM nj−1 (LnU n)M nj .

A standard induction argument proves the crucial equality (U nM nj )2 = U nLnU nM nj = LnU nM nj . For each x ∈ X, LnU n(M nj x) = LnU n(y) is n-monotone, by Theorem 2.5. Since M n preserves such sequences, U nM nj preserves LnU ny as well, proving one half of the theorem. The other half is proved similarly, or using an argument with duals. Theorem 3.10. Let m = max{n, k}. Then U mLm ≤ U nLnU kLk ≤ LnU nLkU k ≤ LmU m. Proof. The operators are all syntone and LU ≥ U L so that LnU n(LkU k) ≥ LnU nU kLk ≥ U nLnU kLk. Suppose that m = n. Then LnU nLkU k ≥ LnU nLnU k, because Lk ≥ Ln, ≥ LnU nLn, because U k ≥ I, = U nLn,

by Theorem 2.9.

Similarly LnU nLkU k ≤ LnU nIU k, since Lk ≤ I, ≤ LnU nU n, since U k ≤ U n, = LnU n,

since U n is idempotent.

A similar argument when m = k completes the proof of the theorem.

The above theorem reﬁnes the order relation on the set of selectors considerably, and yields a proof that several other classes of selectors and compositions map into sets of locally monotone sequences, and do so by mapping a sequence x into a sequence between U mLmx and LmU mx. Examples are given by the following corollary.

3. LU LU -Smoothers, Signals and Ambiguity

27

Corollary. If m = max{n, k} then U mLm ≤ Mnj Mki ≤ LmU m for all i, j > 0, U nLn ≤

Mni

≤ LnU n and U kLk ≤

Mki

since

≤ LkU k.

Apart from the popular smoothers M n∞ and M n∗ , there are now a whole class of smoothers composed of various LnU n and M n that map consistently onto the class Mn of n-monotone sequences. They are thus all eﬀective, and since they are all idempotent, the criteria that can now be used to select from these are eﬃciency and stability. Consistency does however have other criteria, apart from idempotence and preservation of signals. One of these is co-idempotence. There is however a philosophical problem to address. Do we really need to have an n-monotone sequence as output? This question is closely linked to the question of ambiguity suggested by the fact that Ln and U n do not commute and therefore generally yield diﬀerent n-monotone sequences. The same holds for the alternative smoothers mapping onto Mn . They all preserve each other’s output, thus they agree with the respective concept of what signals are, but often yield diﬀerent signals. We consider the smoothers addressed by Theorems 2.14 and 2.15. Theorem 3.11. For each N ≥ 0 the operator Qn has the following properties. (a) Ln ≤ Qn ≤ U n. (b) QnLn = U nLn and QnU n = LnU n. (c) (LnQn)2 = (LnQn)3 and (U nQn)2 = (U nQn)3 . Proof. (a) (U n + Ln − I)x ≤ U nx for each x, since Lnx − x ≤ 0. Similarly Qn ≥ Ln. (b) (U n + Ln − I)Ln = U nLn + LnLn − Ln = U nLn, since Ln is idempotent. Similarly QnU n = LnU n. (c) LnQn)2 = Ln(QnLn)Qn = LnU nLnQn = U nLnQn, (LnQn)3 = LnQn(U nLnQn) = Ln(QnU n)LnQn = Ln(LnU n)LnQn = LnU nLnQn = U nLnQn. The above properties have a remarkable similarity with those of the median smoother Mn . This is extremely odd, since Qn is generally not a selector. A selector S is an operator such that (Sx)i = xj , for some j, but the sequence z = zi = 2δi,j−2 −δi,j−1 + 12 δi,j −δi,j+1 +2δi,j+2 is such that (U3 z)j = 2, (L3 z)j = −1 and Q3 z = 1 − 12 = zk , ∀k. Letting x = z + 12 δi,j shows that x > z, but (Q3 x)j < (Q3 z)j , and therefore that Q3 is not syntone. The following theorem demonstrates, that there is also considerable similarity between the average of LnU n and U nLn, Mn and Qn . Theorem 3.12. The operator Gn = 12 [LnU n + U nLn] has the following properties: (a) Ln ≤ Gn ≤ U n. (b) GnLn = U nLn and GnU n = LnU n. (c) (LnGn)2 = (LnGn)3 and (U nGn)2 = (U nGn)3 .

3. LU LU -Smoothers, Signals and Ambiguity

28

Proof. (a) Ln ≤ U nLn ≤ Gn ≤ LnU n ≤ U n. (b) GnLn = 12 [U nLn + LnU n]Ln = 12 [U n(LnLn) + Ln(U nLn)] = U nLn] = U nLn.

1 2 [U nLn

+

A similar argument proves that GnLn = LnU n. Finally, (c) (LnGn)2 = Ln(GnLn)Gn = Ln(U nLn)Gn = U nLnGn, (LnGn)3 = LnGn(U nLnGn) = Ln(GnU n)LnGn = Ln(LnU n)LnGn = U nLnGn. Corollary. (LnGn)2 maps X onto Mn , since, for each x ∈ X, the sequences U nLnGn(x) = U nLn(Gn(x)) ∈ Mn . The same is true for (Un Gn)2 . Having a selection of smoothers available that map onto Mn , is more than could be hoped for. It is thus possible to investigate and compare them as to their merit in secondary considerations. Eﬀectiveness and some consistency is guaranteed, and the important criteria that remain are stability and eﬃciency. Stability seems to be a strong priority and can be addressed by demonstrating one unwanted instability in the popular smoothers mapping M ∞ and M ∗ onto Mn . Both have a particularly annoying instability due to the fact that they are not suﬃciently local. For each n, both Mn∞ and Mn∗ can undergo radical changes in sequences Mn∞ x and Mn∗ x at a particular index i, by having the sequence x changed at another index j, arbitrarily far away. This is illustrated with two examples diﬀering at only one value in the following ﬁgures.

Figure 3.4. An illustration of the arbitrarily distant error inﬂuence with M ∞ .

In the next chapter it will be seen that such an occurrence is rare, and not as serious as it may seem. As far as values of the smoothed sequence are concerned, it will be seen later that all the operators will be Lipschitz-continuous so that perturbations in x will not have disproportionate perturbations in the image under smoothing.

3. LU LU -Smoothers, Signals and Ambiguity

29

Figure 3.5. An illustration of the arbitrarily distant error inﬂuence with M ∗ .

Of academic interest is that repeated application of Gn to a sequence x normally quickly produces a unique unbiased n-monotone estimate of the signal in x. The convergence in practice is much faster than that of repeated application by M n. The guaranteed rate of convergence is given by the following theorem. Theorem 3.13. ||LnU nGnx − U nLnGnx||∞ ≤ 12 ||LnU nx − U nLnx||∞ . Proof. Let w = LU x − U Lx, c = ||w||∞ and z = 12 (LU x + U Lx) = U Lx + 1 2 (LU x − U Lx). LU z = LU (U Lx + 12 w) ≤ LU (U Lx + 12 c), since c ≥ w and LU is syntone. U Lz = U L(U Lx + 12 w) ≥ U L(U Lx), since w ≥ 0 and U L is syntone. Therefore LU z − U Lz ≤ LU (U Lx) + 12 c − U L(U Lx), and since LU U L = U L is idempotent it follows that 0 ≤ LU z − U Lz ≤ 12 c. But then ||LU z − U Lz||∞ ≤ 12 ||w||∞ , proving the theorem. Remark. The constant

1 2

is the best possible as the following example shows.

Example. Let e = (−a)i . U ei = min{max{(−a)i−1 , (−a)i }, max{(−a)i , (−a)i+1 }} for i even ai = . i−1 a for i odd −ai−1 for i even Lei = −ai for i odd. Both are monotone sequences so that U Lei = Lei and LU ei = U ei . 0 ≤ LU ei − U Lei = ai + ai−1 = ai (1 + a1 ).

3. LU LU -Smoothers, Signals and Ambiguity

30

(1 − a1 ) , then 2 1 1 − a1 LU zi − U Lzi = LU ei − U Lei . 2 1 + a1

If zi = 12 (LU ei + U Lei ) = (−1)i ai

1 a−1 ||LU z − U Lz||p ≤ ||LU e − U Le||p , ∀p ≥ 1. 2 a+1 1 a−1 For large a, is arbitrarily close to 12 . 2 a+1 For the same example (M ne)i = (−a)i−1 = − a1 (−a)i . From this it is easy to show that 1 ||M k nx − M k−1 nx||, ||M k+1 nx − M k nx||p = |a|

Thus

demonstrating arbitrary slow convergence if |a| is near 1. Computationally this is inconvenient, compared to Gn . It will become apparent that the eﬀectiveness, measured in terms of producing a signal, as deﬁned, is not necessarily to be interpreted too strictly. Consistency can similarly be relaxed slightly, so as not to strictly require idempotence of the smoother. At this stage there are suﬃciently many idempotent smoothers available, and also suﬃciently many that preserve Mn . The other half of a strict measure of consistency is the co-idempotence of the smoother. The LU LU -operators will be shown to be co-idempotent and thus strictly consistent, but again this will be shown to permit relaxation. The important measure of eﬃciency remains a consideration, but other, arguably relatively minor, considerations will still be addressed after the further consideration of consistency, using the concept of the LU LU -interval. A friend, Johannes Cronje, pointed out that in Quantum Mechanics the Heisenberg Uncertainty Principle comes into play precisely when the commutators of observables are nonzero. With this image in mind several problems in nonlinear smoothing, and later in the Multiresolution Analysis, can be seen in a clearer perspective. The Heisenberg Uncertainty Principle is central in the mapping between a function and its Fourier Transform, and a function and its Wavelet Transforms. A diﬀerent, but similar, uncertainty may be fundamental between a sequence and its Pulse Transforms, as deﬁned later. This idea turns out to be a guide towards useful and beautiful results. A further obvious idea is to use secondary attributes to select amongst the “equivalent” smoothers inside the LU LU -interval. It is in the selection of these secondary attributes that more insight is required. Some surprisingly useful ones may present themselves.

4. LU LU -Intervals, Noise and Co-idempotence Vom Ungl¨ uck erst Zieh ab die Schuld Was u ¨brig ist, trag in Geduld. Theodor Storm

Choosing the n-monotone sequences as signals is not only convenient, being precisely the range of the LU LU -operators LnU n and U nLn, but also appropriate. The n-monotone sequences are precisely those that have no upward pulse of width not more than n, if this is measured relative to the neighboring values. What is a philosophical puzzle, is why there should be such natural idempotent mappings onto Mn , that came in unequal pairs. Noting that the inequality LnU n ≥ U nLn is strict, and should be written as LnU n > U nLn, it is clear that there generally is a pair of equally valid n-monotone “approximations” of a sequence x ∈ X. This ambiguity turns out to be natural and signiﬁcant rather than an oddity of logic, when interpreted heuristically. There is a natural ambiguity inherent in the deﬁnition of impulsive noise, which can be interpreted as follows. Consider the sequences y = xi = δi,j + δi,j−1 + δi,j+1 and z = 0 in M1 . How is the sequence x = xi = δi,j−1 + δi,j+1 to be interpreted. Is it y with a single downward impulse at j or z with a pair of almost adjacent impulses upward? Both are equally valid simple interpretations. This can be considered a fundamental ambiguity. Experimentally, the ambiguity is very illuminating. If a smooth function is sampled, reasonably distributed (near Gaussian) noise added, and impulses, to yield a sequence x, then a norm of the diﬀerence between LnU nx and U nLnx seems to yield a good indication of the “amount” of well distributed noise. When there is no reasonable ambiguity, such as when there is an isolated n-blockpulse on a constant sequence c, the interval [U nLnx, LnU nx] is precisely the interval [c, c]. It is to be noted that, for n > 1, reﬁned ambiguity interpretations could be argued for generally narrower “intervals of ambiguity”, as for example U2 L2 U1 L1 x and L2 U2 L1 U1 x which, by Theorem 3.10, lie in the interval between U2 L2 x and L2 U2 x. It is convenient to consider the diﬀerence as a robust indicator of the uncertainty in interpretation. The maximum ambiguity is attained when a periodic section

4. LU LU -Intervals, Noise and Co-idempotence

32

Sampled Input

Smoother Ln Un

Interpretation Mn

Un Ln Signal

Noise

Figure 4.1. The fundamental ambiguity in the concept of impulsive noise.

is sampled at precisely the Nyquist-frequency. As an example it is illustrative to sample a function of the type f (t) = sin(αt + β), which has a linearly increasing frequency α. For low frequencies the diﬀerence between LU x and U Lx is very small, where x = xi = f (hi ). As i increases, so does the frequency. Progressively the interval between LU x and U Lx grows until it is an envelope at the Nyquist frequency, after which we have the familiar aliasing eﬀect. This is well known to signal analysts, anybody familiar with digital ﬁlters and with the Shannon– Whitney sampling theorem. A constant sequence has an impulse as a Fourier Transform and conversely, an impulse has a constant Fourier Transform. These are extremes of the general uncertainty in the essential supports of a function and its Fourier Transform. If a function is narrow in time support, it is wide in frequency content. This can be more precisely formulated, and is known as Heisenberg’s uncertainty principle (8), and is central in the analysis of Windowed Transforms, and in the wide range of Wavelet Transforms receiving considerable attention currently. Aliasing is not restricted to trigonometric functions either, but to any sampling for linear approximation by functions from a subspace of functions, such as polynomials of degree m. Thus even when, as often in the case of linear signal processing, high frequencies are considered noise, and low frequencies signal, the LU LU -operators indicate the danger of aliasing by the fundamental separation at multiples of the Nyquist frequency. There clearly is signiﬁcant room for further interpretation, but to obtain more insight it is necessary to formalize the ideas further. Deﬁnition. The closed interval [U nLnx, LnU nx] is called the n-signal interval of x ∈ X and denoted by [x, x]n , or [x, x] when n is implied.

4. LU LU -Intervals, Noise and Co-idempotence

33

n=1

n=2

Figure 4.2. A sampling x of a sinusoidal with increasing frequency and the LU LU -images. The median interpretation is contained in between.

Remarks. (a) [x, x]0 can be identiﬁed with x itself. (b) [x, x]n is identiﬁed with the sequence x if x ∈ Mn . The deﬁnition of the LU LU -interval , or n-signal interval (n-interval), can be expanded to map sequence intervals [x, y], with y ≤ x, onto [U nLnx, LnU ny]. Also it is useful to deﬁne operator intervals like [U L, LU ]. Deﬁnition. Let x ≤ y be sequences from X. Then the n-LU LU interval of [x, y] is [U nLn(x), LnU n(y)]. Deﬁnition. The interval [A, B] can be deﬁned to be the set of all operators C such that A ≤ C ≤ B. Deﬁnition. x, xn = [x, x]n ∩ Mn is the set of all n-monotone sequences in [x, x]n . All the above concepts permit a consistent framework for the comparison of diﬀerent smoothers, amongst them the popular “repeated till convergence” operator Rn = Mn∞ . (In practice the sequences that are to be smoothed are ﬁnite and can be appended with zeroes to be bi-inﬁnite, yet permit the existence of Rn .) Another popular smoother, the recursive median Mn∗ , is also comparable. To make these concepts precise, the following development is expedient. Since the ultimate objective is not to produce an n-monotone sequence as such, but to obtain a sequence that is suﬃciently free of impulsive noise of a prescribed type, in the precise sense of being within the primary “interval of ambiguity” as to interpretation of impulsive noise, it is not important whether Rn or Mn∗ produce n-monotone sequences or not. There are other good smoothers as well that do not. It would therefore be suﬃcient if an operator maps a general sequence x into the interval [x, x]n . This idea prompts the following.

4. LU LU -Intervals, Noise and Co-idempotence

34

Deﬁnition. A sequence interval [a, b] is included in [c, d] if c ≤ a ≤ b ≤ d and is denoted by [a, b] ⊂ [c, d]. Theorem 4.1. Interval inclusion is an order relation. Proof. (a) [a, b] ⊂ [a, b]. (b) [a, b] ⊂ [c, d] and [c, d] ⊂ [a, b] ⇒ [a, b] = [c, d]. (c) [a, b] ⊂ [c, d] and [c, d] ⊂ [e, f ] ⇒ [a, b] ⊂ [e, f ].

Deﬁnition. Two sequences x and y are n-LU LU equivalent if [x, x]n = [y, y]n . Theorem 4.2. LU LU -equivalence is an equivalence relation. Proof. The relation is easily proven to be reﬂexive, symmetric and transitive.

The LU LU -equivalence clearly partitions X into diﬀerent “ancestor” classes; all those sequences mapping onto the same LU LU -interval. Noise added to x ∈ X can be considered removable if it keeps the sequence in the same ancestor class. This is a less restrictive concept of removable noise than that used by Callager and Wise, who consider removable noise as that which changes a sequence into one that has the same “root”. Yet even this concept is too restrictive, since it would be acceptable if the addition of noise only keeps the new LU LU -interval included in the old one. In a similar spirit an operator A could be called n-LU LU -similar (n-similar) if it merely is such that A ⊂ [U nLn, LnU n], and a sequence z called n-LU LU similar to x if z ∈ [x, x]. (These relations are not equivalence relations.) Several important smoothers are n-LU LU -similar. Included are the operators Rn = Mnk ,

Mn∗ ,

1 [U nLn + LnU n], 2

and so is

Qn

if n = 1 or 2.

The operator M2 M12 , popular with Tukey, is speciﬁcally 2-LU LU -similar. Theorem 4.3. Let An be an n-LU LU -similar operator. Then An annihilates nimpulses and preserves n-monotone sequences. Proof. Let x be an n-impulse. Then 0 = U nLn(x) ≤ A(x) ≤ LnLn(x) = 0. If x is n-monotone, then similarly x = U nLn(x) ≤ A(x) ≤ LnU n(x) = x. To prove the similarity for some of the important smoothers, the following theorems are required. Theorem 4.4. For each integer k ≥ 1, Mnk and Gkn are n-LU LU -similar. Proof. The smoothers U n, Ln and M n are all syntone, so that all compositions are syntone. Noting that U nLn ≤ M n ≤ LnU n, it follows that (U nLn)2 ≤ U nLnM n ≤ M nM n and (LnU n)2 ≥ LnU nM n ≥ M nM n. Since LnU n and U nLn are idempotent, U nLn ≤ M n2 ≤ LnU n. The proof is completed by induction. The proof for Gkn is similar.

4. LU LU -Intervals, Noise and Co-idempotence

35

∞ Figure 4.3. The smoother Rn mapping x into the LU LU -interval

Corollary. If Rn∞ x exists it is also in [U nLnx, LnU nx]. To prove that the recursive medians are LU LU -similar requires the following theorem and its required deﬁnitions. (The result is remarkable, to anyone familiar with recursive digital ﬁlters.) To show this it is required to deﬁne the recursive variants, for which the starting procedure is important. It is therefore assumed that the sequences used in practice are ﬁnite, so that somewhere in the past (for suﬃciently small index) the sequences have zero elements only. for j ≥ i, xj , Deﬁnition. U ∗ n(x)i = U n(y)i , where yj = ∗ U n(x) , for j < i. j for j ≥ i, xj , ∗ L n(x)i = Ln(y)i , where yj = L∗ n(x)j , for j < i. for j ≥ i, xj , M ∗ n(x)i = M n(y)i , where yj = M ∗ n(x)j , for j < i. Theorem 4.5. U ∗ n = U n and L∗ n = Ln. Proof. There is an integer s such that U ∗ n(x)i = U n(x)i = xi = 0 for all i ≤ s. Suppose that U ∗ n(x)i = U n(x)i for all i ≤ k. U ∗ n(x)k+1 = U n(y)k+1 , where for j ≥ k + 1, xj , yi = U ∗ n(x)j , for j < k + 1. It is clear that y ≥ x yields U ∗ n(x)k+1 ≡ U n(y)k+1 ≥ U n(x)k+1 . But also y ≤ U nx yields U ∗ n(x)k+1 = U n(y)k+1 ≤ U n(U nx)k+1 = U n(x)k+1 . The proof is completed by the standard induction argument. Lemma. Ln ≤ M ∗ n ≤ U n. Proof. There is an index s such that Ln(x)i ≤ M ∗ n(x)i ≤ U n(x)i , for all i ≤ s. Suppose that M ∗ n(x)k ≤ U n(x)k . Then j ≥ i, xj , M ∗ n(x)k+1 = M n(y)k+1 , with yj = M ∗ n(x)j , j < i.

4. LU LU -Intervals, Noise and Co-idempotence

36

But, by the assumption, y ≤ U n(x), so that M ∗ n(x)k+1 ≤ M n(U nx)k+1 ≤ U n(U nx)k+1 = U n(x)k+1 . A standard induction argument proves that M ∗ n ≤ U n, and a similar argument to the above proves the rest of the lemma. Theorem 4.6. M ∗ n ⊂ [U nLn, LnU n]. Proof. There is an s such that 0 = LnU n(x)i = M ∗ n(x)i = U nLn(x)i , for all i ≤ s. Suppose that M ∗ n(x)k ≤ LnU n(x)k . for j ≥ i, xj , M ∗ n(x)k+1 = M n(y)k+1 , with yi = M ∗ n(x)j , for j < i. M n(y)k+1 ≤ LnU n(y)k+1 ≤ LnU n(U nx)k+1 , since U xn ≥ y. This proves that M ∗ n(x)k+1 ≤ LnU n(x)k+1 , and a standard induction argument proves that M ∗ n(x) ≤ LnU n(x). A similar argument proves that M ∗ n(x) ≥ U nLn(x); since the two inequalities are true for all sequences concerned, the theorem is proved. The fact that the recursive median M ∗ n is in the interval [U nLn, LnU n] is astonishing, though not much more so than the innocent looking fact that M k n is also. This becomes clear when considering the supports of M k n and M ∗ n. Note that the support of LnU n and U nLn is an index interval of [i − 2n, i + 2n], and that of M n is [i − n, i + n]. M 2 n has a support interval [i − 2n, i + 2n] and this increases with each additional power. Similarly, the support of the recursive median smoother is arbitrarily large, as was demonstrated by a previous argument for the instability of the recursive median. How can a local smoother bound a nonlocal one, or even just one with an arbitrarily larger support? It takes some careful thought to digest this seemingly paradoxical fact, which could not be possible in linear (digital) ﬁlters. Two further examples of useful smoothers that are n-LU LU similar are the following. Deﬁnition. Let Bn be the operator ⎧ ⎨ xi , U nLnxi , Bn(x)i = ⎩ LnU nxi ,

deﬁned by if xi ∈ [U nLn(x)i , LnU n(x)i ], if xi < U nLn(x)i , if xi > LnU n(x)i .

Theorem 4.7. Bn ∈ [U nLn, LnU n]. Proof. For each x ∈ X, U nLnxi ≤ Bn(x)i ≤ LnU n(x)i for each index i. This means that U nLn(x) ≤ Bn(x) ≤ LnU n(x), proving the theorem. An alternative to Bn is often useful, which is also n-LU LU -similar, but in the real example below is somewhat better than Bn.

4. LU LU -Intervals, Noise and Co-idempotence

37

Figure 4.4. Removal of impulsive noise by the operator An : 6500 of 430 000 points from the trajectory of Figure 1 in the introduction. (n is large).

Deﬁnition. An =

xi , if xi ∈ [U nLnxi , LnU nxi ], 1 [U Lnx + LnU nx ], otherwise. i i 2

For the sake of consistency it was argued that an operator should ideally be idempotent and co-idempotent. The demand of the idempotence has been shown to be needlessly strict, since it could be argued that it is suﬃcient to be within an interval of ambiguity provided by consistent operators, in this case the appropriate LU LU -operators. A similar relaxation could be suﬃcient for co-idempotence, but this needs to be investigated. The smoothers U n and Ln can be shown to be co-idempotent by the following theorems. Theorem 4.8. Ln (I − Ln ) ≥ 0 and Un (I − Un ) ≤ 0. Proof. I − Ln ≥ 0 ≥ I − Un since Ln ≤ I ≤ Un . Since Ln and Un are syntone it follows that Ln (I − Ln ) ≥ Ln 0 = 0 and 0 = Un 0 ≥ Un (I − Un ). The above theorem seems to be as far as one can get, using only the properties listed above, to proving the co-idempotence of Ln and Un . To complete the proof of the co-idempotence seems to require an operator speciﬁc proof of the following theorem. Theorem 4.9. Ln (I − Ln ) ≤ 0 and Un (I − Un ) ≥ 0. Proof. Suppose Ln (I − Ln )xi > 0. Using the notation L = Ln and uj = xj − Lxj , this means that max{min{ui−n , . . . , ui }, . . . , min{ui , . . . , ui+n }} > 0. Thus min{uj . . . , uj+n } > 0 for at least one j ∈ [i−n, i], or; xj −Lxj > 0, xj+1 −Lxj+1 > 0, . . . , xj+n − Lxj+n > 0 which means that xk > Lxk for all k ∈ [j, j + n]. But Lxk = max{min{xk−n , . . . , xk }, . . . , min{xk , . . . , xk+n }} ≥ min{xj , . . . , xj+n }.

4. LU LU -Intervals, Noise and Co-idempotence

38

This means there is a value xs in {xj , . . . , xj+n } which is not larger than any of the values Lxk , and thus also Lxs . This is a contradiction. A similar proof yields Un (I − Un ) ≥ 0. The following partial result is a simple consequence of the above results. Theorem 4.10. Un Ln (I − Un Ln ) ≤ 0 and Ln Un (I − Ln Un ) ≥ 0. Proof. Un Ln (I − Ln ) = 0, since Ln (I − Ln ) = 0. Since I − Un Ln ≤ I − Ln it follows, because Un Ln is syntone, that Un Ln (I − Un Ln ) ≤ Un Ln (I − Ln ) = 0. A similar proof holds for the other part of the theorem. Theorem 4.11. Un Ln (I − Un Ln ) = 0 and Ln Un (I − Ln Un ) = 0. Proof. With U = Un and L = Ln , assume that LU (I − LU )xi > 0 for some x and i. With the notation wj = U (I − LU )xj , this means that max{min{wi−n , . . . , wi }, . . . , min{wi , . . . , wi+n }} > 0. Assume that min{wk−n , . . . , wk } > 0, with k ∈ [i, i + n]. This implies that U (I − LU )x > 0 for each ∈ [k − n, k]. With the notation vj = (I − LU )xj , this means that min{max{v−n , . . . , v }, . . . , max{v , . . . , v+n }} > 0. This means that xj > LU xj for one value j in each of the sets {k − 2n, . . . , k − n}, . . . , {k, . . . , k + n}. Let ε = xm = min{xj ; xj > LU xj , k − 2n ≤ j ≤ k + n}. For each i ∈ [k − 2n, k + n] it follows that U xi

= ≥

min{max{xi−n , . . . , xi }, . . . , max{xj , . . . , xi+n }} min{ε, ε, . . . , ε} = ε.

But then LU xi = max{min{U xi−n , . . . , U xi }, . . . , min{U xi , . . . , U xi+n }} ≥ ε, and since this is also true for i = m it follows that LU xm ≥ ε > LU xm . This contradiction, together with the above theorem, yields LU (I − LU ) = 0. A similar proof holds for U L(I − U L) = 0. Corollary. (a) Ln (I − Un Ln ) ≤ 0

and

Un (I − Ln Un ) ≥ 0

since

Ln (I − Un Ln ) ≤ Un Ln (I − Un Ln ) = 0 and Un (I − Ln Un ) ≥ Ln Un (I − Ln Un ) = 0. (b) Ln Un (Un Ln − I) = 0

and Um Ln (Ln Un − I) = 0

since

0 = Ln Un (I − Ln Un )(−x) = Ln Un (I(−x) − Ln Un (−x) = Ln Un (−Ix − Ln (−Ln x)) = Ln Un (−Ix + Un Ln x). A similar proof holds for Un Ln (Ln Un − I) = 0.

4. LU LU -Intervals, Noise and Co-idempotence

39

All of the above inequalities and equalities can be veriﬁed by testing on randomly generated sequences. The really worthwhile results are that Ln Un (I − Ln Un ) = 0 and Un Ln (I − Un Ln ) = 0, since this implies Ln Un and Un Ln are co-idempotent. Apart from the conﬁrmation of consistency argued previously, one of the consequences is that Ln Un and Un Ln have only eigenvalues 0 and 1, since Ln Un (λx) = λLn Un (x) and Un Ln (λx) = λLn Un (x) for λ > 0. It is worthwhile proving this with the following general result. Recalling that an operator P is scale independent if P (λx) = λP x for λ ≥ 0 (third smoother axiom), we get; Theorem 4.12. If P is a scale independent separator, then it has only the eigenvalues 1 and 0. Proof. Let P e = λe. (a) If λ ≥ 0, then P 2 e = P (λe) = λP e = λ2 e = P e = λe. But (λ2 − λ)e = 0 implies λ2 − λ = 0 or λ = 1, 0. (b) If λ < 0, then (I −P )e = e−λe = (1−λ)e. Since P (I −P )e = 0 = P (1−λ)e = (1 − λ)P e = (1 − λ)λe, it follows that λ(1 − λ) = 0, which is impossible for λ < 0. Corollary. The range of a scale independent separator P is the set of eigensequences w.r.t. 1, unless the range contains only the zero sequence. Similarly, the range of I − P is the set of eigensequences of 0. Since each sequence x can be decomposed into x = P x + (I − P )x it is clear that, unless P = I or P = 0, x can be decomposed into P x, the eigensequence w.r.t. the eigenvalue 1 (“signal”) and (I − P )x, the eigensequence w.r.t. the eigenvalue 0 (“noise”). It is important to note that, if x = y + z, with y in the range of P , and therefore a “signal”, it does not follow that y = P z and z = (I − P )x. This can be seen from the examples Ln Un and Un Ln , which are not identical, though their ranges are precisely the set of n-monotone sequences, and are therefore identical. What is signiﬁcant is that; Range(I − Ln Un ) = Range(I − Un Ln ). Even when x is such that Ln Un x = Un Ln x, the decomposition of x into an nmonotone sequence and an eigensequence w.r.t. the eigenvalue 0 is not unique in general, as is demonstrated by the examples x = (i < 0), w = (i < 1) and v = (i < 2), which are all monotone and w − x and v − w are such that L1 U1 (w − x) = U1 L1 (w − x) = L1 U1 (v − w) = U1 L1 (v − w) = 0. By choosing counterexamples the operators M k n, Qn and Gn can be shown to be not co-idempotent. The other operators M ∗ n, M ∞ n, U nLnQn, LnU nQ etc have not been investigated. There does not seem to be a need for this, in light of the fact that there is an interval of uncertainty associated with the noise, as determined by LnU n and U nLn. They conﬁrm each other’s interpretation of signal but not noise. Heuristically, it seems suﬃcient to have the residual of an extracted signal within this interval of ambiguity, as interpreted by the consistent pair I − LnU n

4. LU LU -Intervals, Noise and Co-idempotence

40

and I − U nLn. It is clear that I − LnU n ≤ I − U nLn, so that it is reasonable to consider the following deﬁnition. Deﬁnition. The n-noise interval of a sequence x is [(I − LnU n)x, (I − U nLn)x]. Theorem 4.13. Let An be an operator such that An ∈ [U nLn, LnU n]. Then I − An ∈ [(I − LnU n)x, (I − U nLn)x]. Proof. If U nLnx ≤ Anx ≤ LnU nx, then −U nLnx ≥ −Anx ≥ −LnU nx and (I − U nLn)x ≥ (I − An)x ≥ (I − LnU n)x. Since this is true for each x ∈ X, the theorem is proved. Since I − An − (I − An)2 = An(I − An), the ambiguity as to the amount of signal still left in the residual (I − An)x can be considered to be in [U nLn(I − LnU n)x, LnU n(I − U nLn)], which is the union of two LU LU -intervals. Since it has been proved that U2 L2 ≤ U2 L2 U1 L1 ≤ L2 U2 L1 U1 ≤ L2 U2 , it follows that [U2 L2 U1 L1 , L2 U2 L1 U2 ] ⊂ [U2 L2 , L2 U2 ], so that a narrower interval of ambiguity exists inside the 2-interval. Similarly it may be of use to deﬁne operators Cn and Fn recursively as follows: Deﬁnition. C1 = L1 U1 and Cn+1 = Ln+1 Un+1 Cn , F1 = U1 L1 and Fn+1 = Un+1 Ln+1 Fn . Theorem 4.14. For each n; U nLn ≤ F n ≤ Cn ≤ LnU n. Proof. The theorem is true for n = 1. Assume it is true for n = k, or Uk Lk ≤ Fk ≤ Ck ≤ Lk Uk . But Uk+1 Lk+1 ≤ Uk+1 Lk+1 Uk Lk , ≤ Uk+1 Lk+1 Fk , ≤ Uk+1 Lk+1 Ck

by Theorem 3.10, by assumption,

≤ Lk+1 Uk+1 Ck ≤ Lk+1 Uk+1 Lk Uk ,

by assumption,

≤ Lk+1 Uk+1 ,

by Theorem 3.10.

The standard induction argument completes the proof.

Looking at an example will demonstrate the idea that the operators Cn and Fn may be more realistic smoothers than Ln and Un for the removal of pulses of length not exceeding n. Let x = xi = δi,j + δi,j+5 . Clearly x is a zero sequence with two single impulses at j and at j + 5. U6 L6 and L6 U6 give outputs that are both 0 everywhere, U5 L5 and L5 U5 give diﬀerent outputs. U5 L5 x = 0, but L5 U5 is a 5-blockpulse stretching from j to j + 5. The interpretation can be made that, if the noise consists of randomly located impulses,

4. LU LU -Intervals, Noise and Co-idempotence

41

L5 U5 x implies that the signal is a pulse of width 5 with 4 negative blockpulses at j + 1, j + 2, j + 3 and j + 4. This is a less probable occurrence than that assumed by U5 L5 ; that the signal was zero with two single pulses located coincidentally at j and j +5. Clearly the expected distribution of impulses will determine the design of an appropriate smoother from smoothers that are compositions of Uj Lj and Lj Uj with j ≤ n rather than U nLn and LnU n, to yield more likely interpretations if pulses of briefness not exceeding n are considered to have to be removed. A choice that yields a small interval of ambiguity (interpretation) seems [Fn , Cn ], although there may be subtle arguments to include other, previously discussed smoothers in the LU LU -interval. A further reason for choosing the smoothers Fn and Cn will appear later, in connection with the problem of decomposing a given sequence into pulses of diﬀerent resolution levels – Multiresolution Analysis (MRA) with (block-) pulses. Interestingly, the apparent increase in computational eﬀort to compute Fn and Cn as opposed to simpler smoothers like U nLn and LnU n turns out to be misleading.

5. Smoothing and Approximation with Signals Ich lebe mein Leben in wachsenden Ringen die sich u ¨ber die Dinge ziehn. Ich werde den letzten vielleicht nicht vollbringen aber versuchen will ich ihn. Rainer Maria Rilke

The set Mn of n-monotone sequences is clearly of crucial importance in the subject of nonlinear smoothing. Many smoothers map onto or into Mn . Is there a possibility of selecting any that are optimal in some sense? Considering the operator B that maps x onto a “nearest” element in Mn , in the sense of the metric chosen, the questions that come to mind are: For which metric does B exist, and if it maps onto an interval of sequences, can a suitable choice be made to deﬁne an unambiguous n-monotone image? How does the LU LU -interval compare with the best? The questions come from experience in Approximation Theory. Given a function f , we seek a best approximation Bf in a subspace S of approximating functions. For some norms the function Bf , though uniquely deﬁned, is computationally complex. The idea then is to use a simpler operator P that is computationally eﬃcient, like an interpolant, and use the Lebesgue inequality to show that this sub-optimal approximation P f is not “far” from the best approximation Bf . Thus P f suﬃces for practical purposes, or can possibly be used as a good start for an iterative procedure converging to Bf . Restrepo and Bovic did a logical thing by considering the problem of best approximation in all the usual p-norms and seminorms as well (0 < p < 1). This idea had been relatively unexplored, but seems sound. The p-norms are mappings from X × X to R are positive deﬁnite and symmetric and thus form a semi-metric. They are positive homogenous (dp (γx, γy) = |γ|dp (x, y)) and translation invariant. For p ≥ 1 a metric is obtained, but for 0 < p < 1 the triangle inequality does not hold. They make a case for the semi-metric, but it is suﬃcient here to consider only the cases p ≥ 1. They prove the existence of and suggest computational procedures for obtaining such “best” locally monotone sequences. The computational complexity is however uninviting and non-local. As in the case of function approximation, it is thus natural to seek simple, possibly local, suboptimal methods that yield sequences in Mn that are near the

44

5. Smoothing and Approximation with Signals

optimal p-projection image, or best approximation. In linear approximation, the Lebesgue inequality provides such a comparison method, provided the operator norm is small. But in the problem at hand the operators that are candidates cannot be linear for previously argued reasons. It is however worthwhile to pursue the idea for possible generalization. It may even be speculated that the best approximation to a sequence x lies in the LU LU -interval! This turns out to be not true in general, but still possibly true for a large class of sequences. When this is not so, this also may actually reﬂect more on the best approximation than on the LU LU -interval! The Lebesgue inequality requires that the operator is linear, idempotent and of bounded norm. Is the idea of the Lebesgue inequality transferable to the problem at hand; where the approximating set Mn is not a subspace and the operators not linear, though often idempotent? It is useful to review the Lebesgue inequality and a proof for the purpose of possible analogy. Theorem. Let S be a subspace of a normed space X and P a linear mapping into S that is the identity on S and has bounded induced norm µ = ||P ||. Then, for each x ∈ X and s ∈ S. ||P x − x|| ≤ (||P || + 1)||x − s||. Proof.

||P x − x|| ≤ ||P x − s|| + ||s − x|| ≤ ||P x − P s|| + ||s − x||

(Triangle inequality) (P is identity on S)

≤ ||P (x − s)|| + ||s − x|| (P is linear) ≤ ||P || ||x − s|| + ||s − x|| (Induced norm) ≤ (1 + µ)||s − x||.

Replacing the subspace S by Mn , a subset in X, it is clear that the triangle inequality is still valid, and the linearity demand on P can be circumvented by directly ﬁnding a Lipschitz-constant µ for P . Providing that P maps X onto Mn and has a Lipschitz-constant µ, the comparison between P x and any s in Mn is still valid, and thus P x can be comparable with the “best” sequence in Mn . Considering the set of Lipschitz constants for an operator on X and choosing its minimum yields a minimum Lipschitz constant, which is a mapping µ onto R, the real numbers, deﬁned by; Deﬁnition. (minlip constant) µ(P ) = sup

||P x − P y|| , x, y ∈ X, x = y . ||x − y||

The minlip mapping µ has the properties; i) Clearly µ(C) = 0, where C is a constant mapping C(x) = c. Conversely; let µ(P ) = 0. Then, ||P x − P y|| = 0, ∀x, y ∈ X. Thus, P x = P y = c ∈ X.

5. Smoothing and Approximation with Signals

ii)

||αP x − αP y|| , x, y ∈ X, x = y µ(αP ) = sup ||x − y|| ||P x − P y|| = sup |α| ||x − y|| ||P x − P y|| = |α| sup = |α|µ(P ). ||x − y||

45

iii)

||(P + A)x − (P + A)y|| ||x − y|| ||P x − P y|| ||Ax − Ay|| + ≤ sup . ||x − y|| ||x − y||

µ(P + A) = sup

But then µ(P + A) ≤ sup{µ(P ) + µ(A)}, since µ(P ) and µ(A) are upper bounds and µ(P + A) ≤ µ(P ) + µ(A). The two idempotent smoothers LU, U L are candidates for comparison with the best approximation in Mn . The following lemmas are useful. Lemma: For each operator P , µ(N P ) = µ(P N ) = µ(N ), where N is the negation operator. ||P N x − P N y|| , x, y ∈ X, x = y Proof. µ(P N ) = sup ||x − y|| ||P (−x) − P (−y)|| = sup , −x, −y ∈ X, −x = −y = µ(P ), || − x − (−y)|| ||N P x − N P y|| µ(N P ) = sup , x, y ∈ X, x = y ||x − y|| = µ(P ), since || − P x + P y|| = ||P x − P y||.

Lemma. For any two operators AB and B on X, µ(AB) ≤ µ(A)µ(B). ||ABx − ABy|| , x, y ∈ X, x = y ||x − y|| µ(A)µ(B)||x − y|| ≤ sup , x, y ∈ X, x = y = µ(A)µ(B), ||x − y||

Proof. µ(AB) ≤ sup

since ||ABx − ABy|| ≤ µ(A)||Bx − By|| ≤ µ(A)µ(B), as µ(A) and µ(B) are Lipschitz constants. n n 1 Theorem 5.1. µp ( ) = µp ( ) = (n + 1) p , for p ≥ 1. n n Proof. | xi − yi | = | max{xi , xi+1 , . . . , xi+n } − max{yi , yi+1 , . . . , yi+n }|. (Without loss of generality we may assume all the elements concerned to be posi-

46

5. Smoothing and Approximation with Signals

tive.) Assuming |

n n

xi − xi −

n n

yi > 0 it follows that for some k ∈ {i, i + 1, . . . , i + n}

yi |P = (xk − max{yi , yi+1 , . . . , yi+n })p ≤ |xk − yk |p , ≤

i+n

|xj − yj |p .

j=i

But then ||

n

x−

n

y||pp ≤

∞

⎛ ⎝

i=−∞

≤ (n + 1)

i+n

⎞ |xj − yj |p ⎠

j=i ∞

|xk − yk |p ≤ (n + 1)||x − y||pp .

k=−∞

n n 1 x− y||p ≤ (n + 1) P . Thus || To prove that this inequality is sharp, it is suﬃcient to consider the sequences x = xi = δij and y = 0. Then ||x − y||p = ||x||p = 1,

and ||

n

x−

n

y||p = ||

n

1

x||p = (n + 1) p .

n n The other half follows, since =N N, by a previous lemma. n n n n 2 Corollary. µp ( ) = µp ( ) ≤ (n + 1) p , by the lemma on compositions. The above upper bound is not sharp, as becomes clear from the following theorem. Theorem 5.2. For each x, y ∈ X and p ≥ 1, 1

||Lnx − Lny||p ≤ (2n + 1) P ||x − y||p . Proof. Consider |Lnxi − Lnyi |. Without loss of generality it can be assumed that Lnxi ≥ Lnyi , and since Ln is local, all the xj and yj concerned can be assumed positive, since adding an arbitrary constant sequence does not change the inequality. Since Lnxi = min{xk−n , . . . , xk }, for some k ∈ [i, i + n], and Lnyi ≥ min{yk−n , . . . , yk } as well, (it is the maximum of such sets), Lnxi − Lnyi ≤ min{xk−n , . . . , xk } − min{yk−n , . . . , yk } ≤ min{xk−n , . . . , xk } − y , for some ∈ [k − n, k]. But then Lnxi − Lnyi ≤ x − y , so that |Lnxi − Lnyi |p ≤ |x − y |p , for some ∈ [i − n, i + n].

5. Smoothing and Approximation with Signals

Since |x − y |p ≤

i+n

47

|x − y |p

=i−n

it follows that ||Lnx −

Lny||pp

≤

∞ i=−∞

i+n

p

|x − y |

∞

≤ (2n + 1)

|xi − yi |p

i=−∞

=i=n

≤ (2n + 1)||x − y||pp .

Taking the pth root on each side proves the theorem. 1 p

Theorem 5.3. For each p ≥ 1, µp (Ln) = µp (Un ) = (2n + 1) . 1

Proof. By the previous theorem it follows that µp (Ln) ≤ (2n + 1) p . To prove that this is an equality as well, it is suﬃcient to consider

j−1 j+n j+n δi + δj δ0j . and y = yi = x = xi = =j−n

=j+1

=j−n

But then ||x − y||p = 1 and Lny = y, where as Lnx = 0 since min{xi , xi+1 , . . . , xi+n } = 0, ∀i. Therefore 1

1

||Lny − Lnx||p = ||Lny||p = (2n + 1) p ,

and µp (Ln) ≥ (2n + 1) p .

By a previous lemma the dual yields the same minlip constant, proving the theorem. 2

Corollary. µp (LnU n) = µp (U nLn) ≤ (2n + 1) p . In all the above cases p = ∞ can be taken as the limiting case when p tends to ∞. It is easily conﬁrmed that the following theorem holds. Theorem 5.4. Let S be a syntone selector. Then µ∞ (S) = 1. Proof. It is clear that −||x − y||∞ ≤ xi − yi ≤ ||x − y||∞ . Let c = ||x − y||∞ . Then, since S is syntone, yi − c ≤ xi ≤ yi + c implies that S(yi − c) ≤ Sxi ≤ S(yi + c). Since a selector is a smoother, it follows that Syi − c ≤ Sxi ≤ Syi + c

and

− c ≤ |Sxi − Syi | ≤ c.

This means that ||Sx − S||∞ ≤ ||x − y||∞ and that µ∞ (S) ≤ 1. A simple example with a constant sequence suﬃces to show that µ∞ (S) = 1.

48

5. Smoothing and Approximation with Signals

The last theorem suggests that possibly the previous upper bound for µp (LnU n) is pessimistic, at least for the case where p is large. This can be pursued further, but since the main purpose of a sharp bound for µp is the comparison with the “best” or other operators, and investigating the proof of the Lebesgue-type inequality, it is clear that the following restriction to the more general inequality will suﬃce. Theorem 5.5. For x ∈ X and y ∈ Mn , 1

||LnU nx − LnU ny||p ≤ (2n + 1) p ||x − y||p and 1

||U nLnx − U nLny||p ≤ (2n + 1) p ||x − y||p . Proof. Consider LnU nx and y ∈ Mn . Assume both LnU nx and U nLny = y are positive. (This can be done without loss of generality since an arbitrary constant sequence can be added to x and y.) Suppose LU xi ≥ yi . Then |LU xi − yi | ≤ |U xi − yi |, since U xi ≥ LU xi ≥ yi . If LU xi < yi , then |LU xi − yi | ≤ |Lxi − yi |, since Lxi ≤ LU xi < yi . But |U xi − yi |, |Lxi − yi | ≤

max

j∈[i−n,i+n]

|xj − yj |,

and therefore |LU xi − yi |p ≤ ||LU x − y||pp ≤

max

j∈[i−n,i+n] ∞ i=−∞

⎛ ⎝

|xj − yj |p ≤

i+n

|xj − yj |p ,

j=i−n

⎞

i+n

|xj − yj |p ⎠ = (2n + 1)

j=i−n

= (2n + 1||x −

∞

|xj − yj |p

j=−∞

y||pp . 1

This implies that ||LU x − y||p ≤ (2n + 1) p ||x − y||p .

This inequality is sharp, as the example used previously, with the sequences

j−1 j+n j+n x = xi = δ i + δ i and y = yi = δij , =j−n

=j+1

=j−n

demonstrates that 1

||Lny − Lnx||p = ||U nLny − 0||p ≥ (2n + 1) p

but ||x − y||p = 1.

At this stage, for the operators U nLn and LnU n, a Lebesgue type Comparison Theorem can be formulated for the set Mn of n-monotone sequences.

5. Smoothing and Approximation with Signals

49

Theorem 5.6. Let P be either U nLn or LnU n. Then, for each s ∈ Mn , ||P x−x||p ≤ 1 (1 + (2n + 1) p )||s − x||p . Proof.

||P x − x||p ≤ ||P x − s||p + ||s − x||p , ∀s ∈ Mn 1

≤ (2n + 1) p ||s − x||p + ||s − x||p 1

≤ (1 + (2n + 1) p )||s − x||p .

If there is a “best” approximation to x in Mn then clearly this inequality reassures that LnU n and U nLn yield approximations that are “less than best” by 1 the factor 1 + (2n + 1) p . Whether this factor is optimal is not established yet. The above theorem can be generalized easily to facilitate the comparison of more general smoothers with the best approximation in Mn . This can be done with the following theorem. Theorem 5.7. Let x ∈ X and y ∈ Mn . If P ∈ [Ln , Un ], then ||P x − y||p ≤ 1 (2n + 1) p ||x − y||p . Proof. Consider the ith elements of x and y. (Assume both are positive. If not, a constant can be added temporarily for the neighborhood of i under consideration.) If yi > Pn xi , then |yi − P xi | ≤ |yi − Ln xi |. If yi ≤ P xi , then |P xi − yi | ≤ |Un xi − yi |. Since Ln y = Un y = y it follows that |P xi − yi |p ≤ max{|Un xi − Un yi |p , |Ln xi − Ln yi |p }. By the argument used in the proof of Theorem 5.2, it follows that p

|P xi − yi | ≤

i+n

|x − y |p ,

1

or ||P x − y||pp ≤ (2n + 1) p ||x − y||pp .

=i−n

This proves the theorem.

There are several popular operators that meet the requirements of the above theorem, such as medians and their powers. For example a simple popular smoother like M2 M12 can be compared with the best approximation in M2 by noting that L2 U2 L1 U1 is in the interval [L2 , U2 ], since L2 U2 L1 U1 ≤ L2 U2 U1 ≤ L2 U2 U2 = L2 U2 ≤ U2 etc. Others are the operators 12 (U L + LU ), U + L − I, etc. It is almost natural, given the results of previous comparisons, to guess that the “best approximations” are in the corresponding LU LU -interval. This turns out to be not true in general, as the following example will demonstrate. Example. Let x = xi = δi,j + δi,1+j be a sequence, that clearly has two consecutive impulses superimposed on the zero sequence. With L2 U2 x = U2 L2 x = 0, it is clear that 0 is not a best approximation in any norm with p < ∞ since y = yi = δi,j−1 + δi,j + δi,j+1 is better.

50

5. Smoothing and Approximation with Signals

It is worth noting that a best approximation from M2 does not even lie between L2 x and U2 x, and does not remove the basic impulse for which U2 L2 , L2 U2 , M2 etc. were designed! (The possible exception is the p = ∞ norm, where a best approximation may be in the LU LU -interval.) The above example illustrates this since y is a best approximation as ||x − y||p = 1 cannot be improved. This casts doubt on the basic idea of approximating with Mn as a general method of impulsive noise removal, and highlights the remarkable ability of the LU LU -smoothers and similar local smoothers. The indications are however that for n = 1 the “best approximation” from M1 may well be in the LU LU -interval. This needs further investigation, but does not seem very important. An alternative, secondary approximation problem presents itself naturally. Given a choice of a smoother S amongst the smoothers in a class C that map into the range of S, and being aware of the fact that a smoother can at best destroy information selectively, the question of comparison with alternatives comes to mind. Is there an associated smoother that can somehow reduce the damage done by S? If so, is there a smoother SL such that the residual (SL S − I)x is minimal, for each sequence x, in some relevant norm? Similarly, is there a smoother SR such that (SSR − I)x is minimal? A simple example is suﬃcient to show that SR does not exist in respect of, for example, the 1-norm. With respect to the Total Variation, which is a norm in 1 , there always is a left quasi-inverse. Indications are also that a left quasi-inverse SL exists, with respect to the 1-norm. Thus (SL S − I)x1 ≤ (AS − I)x1 , for each sequence x ∈ 1 and each composition A of the basic operators ∨ and ∧. In particular each idempotent composition, (which includes all LU LU -operators) has the identity operator as quasi-inverse. These operators are thus best approximations to the identity operator, with respect to left-composition. Articles investigating these issues, and interesting incidental results may appear soon.

6. Variation Reduction and Shape Preservation Ich Forscher – ich? Oh spart dies Wort! – Ich bin nur schwer – so manche Pfund! Ich falle, falle immerfort Und endlich auf den Grund Nietzsche

There are a variety of concepts of smoothness in functions; Continuity, Diﬀerentiability, Integrability, Measurability, to name but a few. None of these generalize suitably to sequences. There are other concepts that are linked to these (27), two of which are relevant here; Bounded Variation and (Piece-wise-) Monotonicity. Two examples that are well known can be taken to illustrate. 1) If f is an increasing real-valued function on [a, b], then f is diﬀerentiable b f (x) ≤ f (b) − f (a). almost everywhere and a

2) A function f is of bounded variation on [a, b] if and only if f is the diﬀerence of two monotone real-valued functions. Clearly piecewise monotonicity is suﬃcient in both. The digital world has impressed us with the reality that, in practice, measurements are sequences. So are transmitted signals, yet the concepts of smoothness often used are based on concepts of smoothness of an “underlying” function or “true function”. For the Fast Fourier Transform (FFT) and Fast Wavelet Transform the underlying assumption of a band-limited function is usual. There is something unsatisfactory in basing the concept of smoothing used in the subject of Nonlinear Smoothing on properties of an underlying function which is not at hand. Similarly concepts like “trend” and “impulse” are often also based by implication on linear functions and delta functions. Most people using median smoothers feel that there are some very useful properties that are “almost always” there, the behavior is “enigmatic” and unpredictable. Yet they generally behave very well. If measures and concepts of smoothness that are more directly attributable to a given sequence are to be used, it seems natural to turn to variation and monotonicity. When these concepts are chosen appropriately they turn out to be illuminating and reveal a very strong underlying mathematical framework for the analysis of signals, even with a wide variety of contamination.

52

6. Variation Reduction and Shape Preservation

Both concepts can be used globally and locally. For practical purposes of measurements sequences are always ﬁnite, but there is no problem generalizing to X = p and a natural choice is p = 1. We have already become used to local monotonicity as a local concept of smoothness (there are no pulses of lesser briefness), and a connection to variation can be envisaged as follows. Deﬁnition. T x = T (x) = ||x||1 , where is the (forward-) diﬀerence operator. T x is called the (total-) variation of x. It is clear that if x ∈ 1 , then x ∈ 1 , since N

N

|xi+1 − xi | ≤

i=−N

(|xi | + |xi+1 |) ≤ 2||x||1 .

i=−N

Theorem 6.1: (i) T (Ex) = T (x), where E is the shift operator. (ii) T (x + y) ≤ T (x) + T (y) (subadditivity). (iii) T (αx) = |α|T (x) and T (x) = 0 only if x = 0. Proof. (i)

N i=−N

|Exi+1 − Exi | =

N i=−N

|xi+2 − xi+1 | =

N +1

|xi+1 − xi |.

i=−N +1

Taking limits proves that T (Ex) = T (x). (ii) |(x + y)i+1 − (x + y)i | = |xi+1 − xi + yi+1 − yi | ≤ |xi+1 − xi | + |yi+1 − yi |. (iii) |(αx)i+1 − (αx)i | ≤ |α| |xi+1 − xi |. Summing and taking limits proves T (x+y) ≤ T (x)+T (y) and T (αx) = αT (x). Since a sequence that is n-monotone is (n − 1)-monotone, M0 ⊃ M1 ⊃ M2 ⊃ · · · ⊃ Mn ⊃ · · · is a nested sequence of subsets of 1 whose intersection contains only the zero sequence. Intuitively it is clear that the progressive mapping of a sequence x ∈ 1 into these subsets should reduce the variation, ultimately arbitrarily close to 0. There must be a connection between local monotonicity and total variation. Since it is possible to decompose the operators Un Ln , Ln Un , Ln and Un into oftwo basic local maximum and minimum operators, compositions by Un = ( n n ) and Ln = n n , the following approach is logical. For convenience U1 can be denoted by U and L1 by L provisionally. Lemma. If j is a point where U xj = xj , then U xj − xj = min{xj−1 , xj+1 } − xj > 0 and U xj+1 = xj+1 . Proof. Let U xj = xj . Then U xj > xj , and by deﬁnition (U x)j = min{max{xj−1 , xj }, max{xj , xj+1 }} = min{xj−1 , xj+1 },

6. Variation Reduction and Shape Preservation

53

since both maxima must be larger than xj . Noting that (U x)j+1 = min{max{xj , xj+1 }, max{xj+1 , xj+2 }} = min{xj+1 , max{xj+1 , xj+2 }} ≤ xj+1 ,

it follows that U xj+1 = xj+1 .

It is clear that U x cannot diﬀer from x at consecutive points and that they diﬀer at precisely a point where the value is smaller than both neighbors. Theorem 6.2. T (x) = T ( x) + 2||U x − x||1 for all x ∈ 1 . Proof. Consider the sequence t = {tj } of integers where U xj > xj . Then T (x) = tj+1 −2 ∞ τj where τj = |xi+1 − xi |, simply partitions the sum. j=−∞

i=tj −1

Consider a speciﬁc j, and letting tj = k and tj+1 = m, let n be the ﬁrst index after k where xn > xn+1 . Since xk+1 > xk and xm < xm−1 , n must exist and be between k and m, and xk , xk+1 , . . . , xn is non-decreasing and xn , xn+1 , . . . , xm non-increasing, and n−1

µj =

|

xi+1 −

xi | +

|

xi+1 −

xi |

i=n

i=k−1

=|

m−2

xk −

xk−1 | +

n−2

|xi+2 − xi+1 | +

xn −

|xi+1 − xi |,

i=n

i=k

since

m−2

xn−1 = 0.

Except for three terms this yields Tj and therefore; τj = µj − |xk+1 − xk−1 | + |xk − xk−1 | + |xk+1 − xk | = µj − |xk+1 − xk−1 | + xk−1 − xk + xk+1 − xk = µj − max{xk+1 , xk−1 } + min{xk+1 , xk−1 } + xk−1 + xk+1 − 2xk = µj + 2(min{xk+1 , xk−1 } − xk ) = µj + 2(U xk − xk )

(From the previous lemma)

Therefore ∞ ∞ ∞ T ( x) = µj = τj − 2 |U xtj − xtj | j=−∞

= T (x) − 2

j=−∞ ∞ i=−∞

j=−∞

|U xi − xi | = T (x) − 2||U x − x||1 .

54

6. Variation Reduction and Shape Preservation

It is clear that the operator is variation decreasing in a precise sense. A similar property must exist for , and can be demonstrated by the following corollary to Theorem 6.2. Corollary. T (x) = T ( x) + 2||Lx − x||1 . Proof. T ( x) = T (E x) = T ( (Ex)) = T (− (Ey)), and with y = −x, T ( x) = T ( y) = T (y) − 2||U y − y||1 = T (−Ex) − 2||U (−Ex) + Ex||1 = T (Ex) − 2|| − L(Ex) + Ex||1 = T (x) − 2||E(x − Lx)||1 = T (x) − 2||x − Lx||1 , which proves the corollary.

Corollary. Any operator O that is a composition of and is variation diminishing (variation non-increasing). Proof. For and the statement follows from Theorem 6.2 and its corollary, since T ( x) and T ( x) exceed T (x) by amounts 2||Lx − x||1 and 2||U x − x||1 respectively. The rest of the proof follows by induction. The following theorem shows that does not reduce variation on any output of . Theorem 6.3. T ( x) = T ( x) and T ( x) = T ( x) for each x ∈ 1 . Proof. Suppose that U ( x)i > xi for some index i. Then ( x)i−1 > ( x)i < ( x)i+1 . If ( x)i is smaller than min{xi−2 , xi−1 } and min{xi , xi+1 } it follows that it is smaller than xi−1 and x . This is a contradiction since ( x)i is the i smaller of xi−1 and xi . Thus U ( x) = x. But then T ( x) = T ( x) + 2||U ( x) − x||1 = T( x). A similar proof can be given for the other part of the theorem, or the usual duality argument can be used; If y = −x, then T ( x) = T (− y) = T ( y) = T ( y) = T (− x) = T ( x). Corollary. T ( U x) = T (U x) and T ( Lx) = T (Lx). Example 1. Let

xi =

1 0

for i = −1, 1 . elsewhere

6. Variation Reduction and Shape Preservation

Then

U xi =

1 0

for i = −1, 0, 1 elsewhere

and

55

x=

1 for i = −2, −1, 0, 1 . 0 elsewhere

Elementary computation yields T (x) = 4,

T (U x) = 2,

Example 2. Let

T ( x) = 2 and ||U x − x||1 = 1.

(−1)i , i2

i = 0 and x0 = 1. ∞ ∞ ∞ 1 1 1 1 T (x) = 2 2 + + + 2 = 4 + 2 2 2 2 i (i + 1) i (i + 1)2 i=1 i=1 i=1 π2 2 π2 +2 − 1 = π2 + 2 =2+2 6 6 3 xi =

and

T (U x) = T ( x) = 2,

since x is monotone increasing up to −1 and monotone decreasing from 0. Elementary calculation gives π2 . ||U x − x|| = 3 It should be clear, since U = U1 = and L = L1 = , that the following theorem should hold, because of the fact that points where U1 xi = xi are separated by at least one index. Theorem 6.4. T (x) = T (U x) + T (U x − x) and T (x) = T (Lx) + T (Lx − x). Proof. Let ti be the sequence of points such that U xtj > xtj . By the ﬁrst lemma such a point xtj has neighbors x in xr such that U x − x = 0 and U xr − xr = 0. Therefore, by Theorem 6.1, T (U x − x) = =

∞ j=−∞ ∞

|U xtj − xtj − (U x − x )| + |U xtj − xtj − (U xr − xr )| 2|U xtj − xtj | = 2||U x − x||1 ,

j=−∞

T (x) = T ( x) + 2||U x − x||1 = T (U x) + 2||U x − x||1 , = T (U x) + T (U x − x).

by Theorem 6.3,

The other inequality follows similarly, or by a duality argument, as before.

56

6. Variation Reduction and Shape Preservation

The idempotent operators U and L do not have a common range, but the range of LU and U L is common and is M1 . The precise variation reduction relations for the composite operators are surprisingly simple to prove. This yields a quantitative measure for the contribution to the total variation of x by single downward pulses and single upward pulses, as interpreted by LU and U L respectively. Theorem 6.5. T (x) = T (LU x)+T (LU x−x) and T (x) = T (U Lx)+T (U Lx−x). Proof. By Theorem 6.1, T (x) ≤ T (LU x) + T (LU x − x), by the triangle inequality. By Theorem 6.4, T (x) = T (U x) + T (U x − x) = T (LU x) + T (LU x − U x) + T (U x − x) ≥ T (LU x) + T (LU x − x),

by the subadditivity.

This proves the ﬁrst half of the theorem and the second is proved similarly, or by the usual duality argument. Corollary.

T (LU x − x) = T (LU x − U x) + T (U x − x) = 2||LU x − U x||1 + 2||U x − x||1 , T (U Lx − x) = T (U Lx − Lx) + T (Lx − x) = 2||U Lx − Lx||1 + 2||Lx − x||1 .

For comparison it can be shown that for the median operator M1 the decrease in variation can be arbitrarily slow. Example 3. Let

x=

(−1)i 0

for i ∈ [−n, n], . elsewhere

Then n(n + 1) +2 T (x) = 4 2

and M1 x =

−(−1)i 0

for i ∈ [−n + 1, n − 1], . elsewhere

Therefore T (M1 x) = 2(n − 1)n + 2 and the relative decrease in variation is 4n n 2 < < . 2n(n + 1) + 2 n(n + 1) n (For the same example) T (U1 x) = T (L1 U1 x) = 2 and T (L1 x) = T (U1 L1 x) = 2. The above example again illustrates the general “enigmatic” behavior of the median smoothers and their compositions, despite the fact that they are so closely related to the LU LU operators, and the preference given to their iterates, in spite of the computational eﬀort and generally marginal changes produced by repetition.

6. Variation Reduction and Shape Preservation

57

At this stage a generalization of the previous theorem to general Un Ln , Ln Un and Un Ln is possible in several ways, but the proofs become diﬃcult, if pursued along similar lines. There is a surprising shortcut, yielding in the process some new results which are very surprising. As it turns out there are stronger trend preserving properties , and variation preserving properties, of the LU LU operators. The median smoothers Mn are trend preserving, in the sense that if xi−n , . . . , xi , . . . , xi+n is monotone, then (Mn x)i = xi . The operators Ln Un and Un Ln share this property. Another, stronger, result is the global sequence preservation Ln Un x = Un Ln x = x if x is n-monotone. The operator Mn shares this property. This is however a strong requirement, which can be weakened if only local order preservation is required. There is a diﬀerent, more local, and strong trend preservation, which the medians do not share. Deﬁnition. An operator A is called neighbor trend preserving (ntp) if, for each sequence x ∈ X; xi+1 ≤ xi ⇒ Axi+1 ≤ Axi , and xi+1 ≥ xi ⇒ Axi+1 ≥ Axi for each index i. Theorem 6.6. The operators and , as well as all compositions of them, are neighbor trend preserving. Proof. Suppose that xi+1 ≥ xi . Then ( x)i = max{xi+1 , xi } = xi+1 , and

( x)i+1 = max{xi+2 , xi+1 } ≥ xi+1 = ( x)i .

But then and

( x)i+1 = min{ xi−1 , xi } = xi xi = ( x)i+1 . ( x)i = min{ xi , xi−1 } ≤

A similar argument holds for the case when xi+1 ≤ xi , proving that is trend preserving. Similarly, or for some variation, a typical duality argument can be used to prove the other half. Considering the sequence x and its negative −x;

Since

But

xi+1 xi ⇔ (−x)i+1 (−x)i . is trend preserving, (−x))i . ( (−x))i+1 (

(−x) = − x and ( (−x) = − (x) so that x)i or ( x)i+1 ( x)i . (− x)i+1 (−

58

6. Variation Reduction and Shape Preservation

The above theoremisstrong but not very general, since the only compositions = and L = are U1 L1 , and L1 U1 . (The operators of the operators U 1 1 and themselves are not ntp, as a simple example can prove: x0 = 1, x1 = −1, x2 = 2, but ( x)0 = 1, ( x)1 = 2). The important general LU LU -operators Ln and Un are however ntp, as the following theorem proves. Theorem 6.7. Un and Ln are neighbor trend preserving. Proof. Suppose xi+1 ≥ xi and Un xi+1 < Un xi . For notational convenience i can be assumed to be 0. Un x0 = min{max{x−n , . . . , x0 }, . . . , max{x0 , . . . , xn }} ≤ min{max{x−n+1 , . . . , x0 , x1 }, . . . , max{x0 , . . . , xn }}, Un x1 = min{max{x−n+1 , . . . , x0 , x1 }, . . . , max{x1 , . . . , xn+1 }}. If the latter minimum is smaller than Un x0 it means that it must be because max{x1 , . . . , xn+1 } is smaller than all the others involved. In particular max{x1 , . . . , xn+1 } < max{x0 , . . . , xn }. This can only be true if x0 > x1 , . . . , xn+1 , and in particular x0 > x1 is a contradiction. Thus Un x1 ≥ Un x0 . A similar argument shows that xi ≥ xi+1 implies Un xi ≥ Un xi+1 . As in the previous proof a simple duality argument proves the other half of the theorem. What is surprising is the weak demand for (Un x)i (Un x)i+1 to hold. This demand is independent of n! It only requires xi xi+1 . One of the consequences is a simple proof of a surprising general theorem, which is simple to envisage and prove, but which would not have been easy to see as signiﬁcant, due to the strong assumptions required for it to hold. Since these strong demands are demonstrated to hold for all the operators Ln and Un , as well as all compositions of them, the signiﬁcance of the theorem becomes clear. Deﬁnition. An operator S is diﬀerence reducing if |Sxi+1 − Sxi | ≤ |xi+1 − xi | for each sequence x ∈ X and each index i. Deﬁnition. An operator S is fully trend preserving if it is ntp and diﬀerence reducing. Theorem 6.8. Let P be a fully trend preserving operator. Then T (x) = T (P x) + T (x − P x) for each sequence x ∈ p. Proof. |xi+1 − xi | = |P xi+1 − P xi + xi+1 − P xi+1 − (xi − P xi )|. If xi+1 ≥ xi , then P xi+1 − P xi ≥ 0 and xi+1 − P xi+1 − (xi − P xi ) = (xi+1 − xi ) − (P xi+1 − P xi ) ≥ 0. Thus |xi+1 − xi | = |P xi+1 − P xi | + |xi+1 − P xi+1 − (xi − P xi )|.

6. Variation Reduction and Shape Preservation

59

If xi ≤ xi+1 , a similar argument yields the same. Summing both sides gives N i=−N

|xi+1 − xi | =

N

|P xi+1 − P xi | +

i=−N

N

|(I − P )xi+1 − (I − P )xi |.

i=−N

Taking limits yields the required result.

Theorem 6.9. Un and Ln are diﬀerence reducing. Proof. Consider a sequence x and an index i. Without loss of generality, we may assume that xi+1 ≥ xi and therefore U xi+1 ≥ U xi . Since equality would yield a trivial result, we assume that U xi+1 > U xi . If U xi+1 = xi+1 , then 0 ≤ U xi+1 −U xi ≤ xi+1 −U xi ≤ xi+1 −xi , since U xj ≥ xj , for each index j. Assume therefore that Un xi+1 = xi+1 , so that Un xi+1 > xi+1 . Un xi+1 = min{max{xi+1−n , . . . , xi+1 }, . . . , max{xi+1 , . . . , xi+1+n , }} > xi+1 , and each of max{xi+1−n , . . . , xi+1 }, . . . , max{xi+1 , . . . , xi+1+n } > U xi+1 . Investigating each of the subsets involved in Un xi = min{max{xi−n , . . . , xi }, . . . , max{xi , . . . , xi+n }}, we see that each must have an element not smaller than Un xi+1 , and thus Un xi ≥ Un xi+1 . Together with the inequality Un xi+1 ≥ U xi above, it follows that Un xi+1 = Un xi . Thus |Un xi+1 − Un xi | ≤ |xi+1 − xi |, for all the cases, and all indexes i. A similar proof for L = Ln completes the proof. (or the usual duality argument can derive the inequality for L from that of U , since U (−x) = −L(x)). Corollary. (a) T (x) = T (Un x) + T (x − Un x). (b) T (x) = T (Ln x) + T (x − Ln x). (c) T (x) = T (Ln Un x) + T (x − Ln Un x). (d) T (x) = T (Un Ln x) + T (x − Un Ln x). Proof. Each of the operators satisﬁes the demands of Theorem 6.8.

It is instructive to illustrate with some simple examples, some direct application of the above results. If a signal is expected in a measurement contaminated by impulsive noise and identically independently distributed noise from a “good” (perhaps near Gaussian-) distribution, then there may be an a priori estimate of the total variation of the signal expected. If s is the sequence of the pure signal, r the well-behaved noise and e the impulsive noise, then the sequence x = s + r + e and generally T (x) ≤ T (s) + T (r) + T (e). Assuming that the impulsive sequence has brief large impulses, (its contribution may be large) and that T (r) T (x) (the signal-to-noise ratio is good) we can approximate T (x) T (s) + T (e) + T (r). Experimentation with the type of random noise, or theoretical estimation, can determine how large T (r) is approximately. Smoothing successively by L1 U1 , L2 U2 , . . . , Ln Un , yields a monotone reduction of T (Li Ui ). As

60

6. Variation Reduction and Shape Preservation

soon as large reductions in variation stop, it indicates that the impulsive noise has been adequately removed, and provided T (Ln Un ) is near the expected variation of T (s), not too much damage has been incurred. At each level the variation that is removed is calculable, and therefore indicates the contribution by noise at that resolution level. The alternative decomposition with Un Ln will yield an interval of ambiguity. Example 4. The LU LU -decompositions of a sequence of identically, independently generated random noise from a cubic B-spline distribution, show the fast reduction to an expected 0 variation in the smoothed components. (This could be theoretically derived.)

T(x) = 12.497

+ T(L1U1x) = 5.309

T((I–L U )x) = 7.188 1 1 T((I–U1L1)x) = 7.085

T(U1L1x) = 5.413

+ T((I–C2)C1x) = 2.587

T(C2x) = 2.722 T(F2x) = 2.533

T((I–F2)F1x) = 2.88

+ T(C3x) = 1.436 T(F x) = 0.799 3

T((I–C3)C2x) = 1.286 T((I–F3)F2x) = 1.734

Figure 6.1. The LU LU -decompositions of random noise r and the variations involved.

Example 5. A signal composed of three randomly placed, but isolated, pulses of amplitude 1 and width 1, 2 and 3, with 15 of r added as noise, undergoes a LU LU decomposition. The result demonstrates the removal of each resolution feature of the signal at the appropriate level, and at the exact location. The total variation of the signal is 6, so that the total variation will be roughly that, plus the variation of the noise r. Comparing the contribution expected from r in Example 4, the reduction in variation is predictable. Example 6. A slowly oscillating sinusoid is considered as the signal and the signal of Example 5 is added as noise. Decomposition yields a fair approximation to the signal, despite the impulses of considerable amplitude. (Multiplying the impulses by 10 or 1010 would not signiﬁcantly change the result, except for diﬀerent total variations peeled oﬀ at each layer.

6. Variation Reduction and Shape Preservation

61

T(x) = 10.638

+ T((I–L U )x) = 5.535 1 1 T((I–U1L1)x) = 5.292

T(L U x) = 5.103 1 1 T(U L x) = 5.346 1 1

+ T((I–C )C x) = 2.177 2 1

T(C2x) = 2.926 T(F x) = 3.104

T((I–F )F x) = 2.242 2 1

2

+ T(C3x) = 0.994 T(F x) = 0.854 3

T((I–C3)C2x) = 1.932 T((I–F )F x) = 2.251 3

2

Figure 6.2. The LU LU -decompositions of 3 pulses with random noise r.

The above simple examples demonstrate why the LU LU -decompositions have been so practical in applications of measurements contaminated by bad impulsive noise, and well-distributed noise. Noting that the diﬀerences between the decompositions of Ln Un and Un Ln were very small, it is clear that decomposition with median operators (a highly regarded procedure, in two dimensions – considered much better than wavelet-decomposition in many applications (1)) would yield similar results, but certainly not better. The computational and conceptual simplicity of the LU LU -smoothers yields a signiﬁcant advantage. When the two LU LU -decompositions diﬀer signiﬁcantly at any level, it implies that there is a signiﬁcant amount of oscillation at the Nyquist-frequency of that level. The median would behave unpredictably there (Example 3). Fully trend preserving operators have further remarkable properties with surprising strong results. The following theorems are basic and easy, if somewhat surprising in the last case and the corollary. Theorem 6.10. Let A and B be ftp on sequences in p. Then: (a) AB and BA are ftp. (b) αA + (1 − α)B is ftp for all α ∈ [0, 1]. (c) I − A is ftp. Proof. (a) Suppose xi+1 ≥ xi . Then Axi+1 ≥ Axi , because A is ftp and B(Ax)i+1 ≥ B(Ax)i because B is ftp. Therefore (BA)xi+1 ≥ (BA)xi . A similar argument demonstrates that if xi+1 ≤ xi , then (BA)xi+1 ≤ (BA)xi . Also |Axi+1 − Axi | ≤ |xi+1 − xi | and |B(Ax)i+1 − B(Ax)i | ≤ |Axi+1 − Axi | ≤ |xi+1 − xi |. This proves that BA is ftp, and a similar argument demonstrates that AB is ftp.

62

6. Variation Reduction and Shape Preservation

T(x) = 12.971

+ T((I–L U )x) = 5.57 1 1 T((I–U1L1)x) = 5.528

T(L U x) = 7.401 1 1 T(U1L1x) = 7.443

+ T((I–C )C x) = 2.692 2 1 T((I–F )F x) = 2.519

T(C2x) = 4.709 T(F x) = 4.924 2

2

1

+ T(C x) = 3.148 3 T(F x) = 3.215 3

T((I–C )C x) = 1.562 3 2 T((I–F3)F2x) = 1.709

Figure 6.3. The LU LU -decomposition of a sinusoid, impulsive noise and r.

(b) |(αA + (1 − α)B)xi+1 − (αA + (1 − α)B)xi | ≤ α|Axi+1 − Axi | + (1 − α)|Bxi+1 − Bxi | ≤ α|xi+1 − xi | + (1 − α)|xi+1 − xi | ≤ |xi+1 − xi |. Clearly as A and B are ftp, (αA + (1 − α)B)xi+1 ≥ (αA + (1 − x)B)xi if xi+1 ≥ 0 etc. (c) Suppose xi+1 ≥ xi . Then (I − A)xi+1 ≥ (I − A)xi iﬀ xi+1 − xi ≥ Axi+1 − Axi ≥ 0. Also |(I − A)xi+1 − (I − A)xi | = |Axi − Axi+1 | ≤ |xi+1 − xi |. A similar argument completes the proof if xi+1 ≤ xi . Corollary. A is ftp iﬀ I − A is ftp. What is important to note is that Ln and Un are ftp but M1 is not, as the example x = xi = (−1)i demonstrates, when xi ≥ xi+1 implies M1 xi < M1 xi+1 and also |M1 xi − M1 xi+1 | = 2|xi − xi+1 |. Although the sequence is not in p, the example is relevant as a ﬁnite part can be appended with zeroes to bring it into p. In the more general theory of Morphological Filters (29) there are several related ﬁlters that are useful, for practical and theoretical purposes. It may be useful to consider some of them here in the context of LU LU -operators. Serra calls syntone operators increasing and deﬁnes a ﬁlter (Morphological-) as a syntone and idempotent operator. He calls F extensive if I ≤ F and anti-extensive if G ≤ I. An extensive ﬁlter is called a closing and an anti-extensive ﬁlter an opening . An underﬁlter F has F 2 ≤ F and an overﬁlter F 2 ≥ F . A v-underﬁlter F has F (F ∨ I) = F and a ∧-ﬁlter has F (F ∧ I) = F , and strong ﬁlter both. The (Morphological-) center of two ﬁlters G and F with F ≥ G is deﬁned as B = (F ∧ I) ∨ G = (F ∨ G) ∧ (I ∨ G) = F ∧ (I ∨ G). (In these deﬁnitions (A ∨ B)xi = (Axi ) ∨ (Bxi ) = max{Axi , Bxi } and (A ∧ B)xi = (Axi ) ∧ (Bxi ) = min{Axi , Bxi }.)

6. Variation Reduction and Shape Preservation

63

In particular Ln and Un are strong ﬁlters and their morphological center is I, and Ln Un and Un Ln are strong ﬁlters and their center is idempotent. It is also co-idempotent, provided (Ln Un )∨I and (Un Ln ∧I) are co-idempotent or provided (Un Ln ) ∨ I and (Ln Un ) ∧ I) are co-idempotent. See Wild (52) for a proof that the center is co-idempotent. The rest follows already from Serra. Theorem 6.11. If A and B are ftp then A ∧ B and A ∨ B are ftp and the morphological center, if it exists, is ftp. Proof. (a) Let W = A ∨ B and xi+1 ≥ xi . Then (A ∨ B)xi+1 ≥ Axi+1 , Bxi+1 . Since A and B are ftp, (A ∨ B)xi+1 ≥ Axi , Bxi and therefore (A ∨ B)xi+1 ≥ (Axi ) ∨ (Bxi ) = (A ∨ B)xi . But then also 0 ≤ (A ∨ B)xi+1 − (A ∨ B)xi = Axi+1 − (A ∨ B)xi , if Axi+1 ≥ Bxi+1 = Bxi+1 − (A ∨ B)xi , if Axi+1 ≤ Bxi+1 and |(A∨B)|xi+1 −(A∨B)xi | ≤ Axi+1 −Axi ≤ xi+1 −xi or |(A∨B)xi+1 −(A∨B)xi | ≤ Bxi+1 − Bxi ≤ xi+1 − xi . A similar argument holds if xi+1 ≤ xi . (b) Let W = A ∧ B and xi+1 ≥ xi . Then (A ∧ B)xi ≤ Axi , Bxi . But A and B are ftp, so that Axi ≤ Axi+1 and Bxi ≤ Bxi+1 . Therefore (A ∧ B)xi ≤ Axi+1 ∧ Bxi+1 = (A ∧ B)xi+1 and also 0 ≤ (A ∧ B)xi+1 − (A ∧ B)xi = (A ∧ B)xi+1 − Axi , if Axi ≤ Bxi . = (A ∧ B)xi+1 − Bxi , if Bxi ≤ Axi Therefore |(A ∧ B)xi+1 − (A ∧ B)xi | ≤ Axi+1 Axi , if Axi ≤ Bxi ≤ Bxi+1 − Bxi , if Bxi ≤ Axi . Both quantities on the right are not larger than xi+1 − xi . (A similar argument again holds if xi+1 ≤ xi .) (c) The morphological center, if it exists, is of the type A∨(B∧I) or A∧(B∨I), both of which are compositions of ftp operators. Theorem 6.12. Let x ∈ Mn and A ntp. Then Ax ∈ Mn . Proof. Consider x ∈ Mn . Then {xi , xi+1 , . . . , xi+n , xi+n+1 } are monotone for each i. Thus xi ≤ xi+1 ≤ · · · ≤ xi+n+1 or xi ≥ xi+1 ≥ · · · ≥ xi+n+1 . In both cases the output of A inherits the inequalities, so that {Axi , Axi+1 , . . . , Axi+n+1 } is monotone, for each i. This means that Ax ∈ Mn . Corollary. If A is ntp, then, for all j ≤ n, since Lj and Uj preserve n-monotone sequences, Lj ALn Un = Uj ALn Un = ALn Un , Lj AUn Ln = Uj Un Ln = AUn Ln , Mj AUn Ln = Mj ALn Un = ALn Un . The corollary is very useful for simplifying compositions of LU LU -operators and other ftp, operators, and can be partially generalized to equalities like

64

6. Variation Reduction and Shape Preservation

Ln ALn = ALn etc, using the concepts of upward n-monotone and downward n-monotone (24). We consider the deﬁnition and basic result although the important corollary is easily proved directly. Deﬁnition. For k ≥ 1 call a segment (xi , xi+1 , . . . , xi+k , xi+k+1 ) of a series x ∈ X a k-upwards arc (resp. k-downwards arc) if xi > xi+1 = · · · = xi+k < xi+k+1 (resp. xi < xi+1 = · · · = xi+k > xi+k+1 . Let n ≥ 0. We say that x ∈ X is upwards n-monotone (downwards n-monotone) if k ≥ n + 1 for each k-upwards (k)-downwards) arc contained in x. Theorem 6.13. Let A be fully trend preserving (ftp). Then A preserves upwards n-monotone and downwards n-monotone sequences. Proof. The invariant sequences of Un (respectively Lm ) are exactly the upwards n-monotone (respectively downwards m-monotone) sequences in p . Let x be upwards n-monotone, and Axi > Axi+1 = · · · = Axi+k < Ai+k+1 be an upward arc of A, and k < n + 1. Then xi > xi+1 and xi+k < xi+k+1 (otherwise contradictions arise). Suppose xj , with i + 1 < j < i + k + 1 is such that it diﬀers from a neighbor. Then an arc with smaller support results. Repeating the argument yields an arc with xj > xi+1 = · · · = xj+ < xj++1 and < k < m + 1. This contradicts the fact that x is upward n-monotone. A similar argument for downward n-monotone sequences proves the theorem. Corollary. If A is ftp, then for all j ≤ n, Lj ALn = ALn and Uj AUn = AUn . This corollary is the important result here, as in the analysis of smoothers, the following composition of the basic LU LU smoothers Ln and Un with diﬀerent n, is often useful. Deﬁnition. Cn = Ln Un Cn−1 and Fn = Un Ln Fn−1 , with C0 = F0 = I. For simpliﬁcation it is convenient to note that if n = max{k, m}, then Uk Um = Un and Lk Lm = Ln . A similar result holds for Cn and Fn . Theorem 6.14. Cm Ck = Cn and Fm Fk = Fn , where n = max{k, m}. Proof. Suppose m ≤ k. The output of Ck is k-monotone and Cm preserves this. Thus Cm cannot change the output and Cm Ck = Ck . Suppose m = k +1. Cm Ck = Ck+1 Ck = Lk+1 Uk+1 Ck Ck = Lk+1 Uk+1 Ck , by the ﬁrst part of the proof. But this is Ck+1 , by deﬁnition. By induction it follows that the theorem is also true for any m > n. Clearly a similar proof holds for the other part of the theorem. The fact that the output of Cn is n-monotone implies also that the Fn Cn = Cn and Cn Fn = Fn , since all the operators involved are syntone. It follows by induction that Un Ln ≤ Fn ≤ Cn ≤ Ln Un , since Cn = Ln Un Cn−1 ≤ Ln Un Ln−1 Un−1 ≤ Ln Un Un−1 = Ln Un , by the swallowing theorem Un Uj = Un for j ≤ n.

6. Variation Reduction and Shape Preservation

65

In the following graph a section of a random sequence x together with images of ftp operators L1 U1 , I − L1 U1 , U1 L1 , I − U1 L1 , C2 , (I − C2 )C1 , F2 , (I − F2 )F1 , C3 , (I − C3 )C2 , F3 , (I − F3 )F2 are plotted on the same graph, demonstrating the trend preservation. Lines are drawn to connect consecutive values of the sequences for visibility and circles identify x, which has the largest variation.

Figure 6.4. The output of x for several ftp operators to illustrate local trend preservation.

Theorem 6.15. Let x ∈ Mn−1 and A ftp. Then Un (I − AUn )x = Un x − Un AUn x and Ln (I − ALn )x = Ln x − Ln ALn x. Proof. Consider a sequence x ∈ Mn−1 and index i. Assume ﬁrst that U xi = Un xi = xi . Since U xi = min{max{xi−n , . . . , xi }, . . . , max{xi , xi+1 , . . . , xi+n }}, there is an index j ∈ [i − n, i] such that max{xj , . . . , xj+n } = xi . This means that xj ≤ xj+1 ≤ · · · ≤ xi−1 ≤ xi ≥ xi+1 ≥ · · · ≥ xj+n , since x ∈ Mn−1 means that the consecutive values are monotone. The operators U, AU and I − AU are ftp, so that the corresponding values of the outputs inherit this order. Therefore U (I − AU )xi = (I − AU )xi = xi − AU xi = U xi − U AU xi . Assume therefore that U xi = xi , and therefore U xi > xi . Since x ∈ Mn−1 each of the sets {xk , xk+1 , . . . , xk+n } involved is monotone, and each has a maximum larger than xi , the sequence must have a subset such that xj > xj+1 = xj+2 = · · · = xi = xi+1 = · · · = xj+n < xj+n+1 . Because the relevant operators are ftp, the outputs of the operators U, AU and I − AU must inherit the order structure, except for the strict inequalities weakening to inequalities. Noting that U xi = min{xj , xj+n+1 } it is clear that similarly

66

6. Variation Reduction and Shape Preservation

U AU xi = min{AU xj , AU xj+n+1 } and U (I − AU )xi = min{(I − AU )xj , (I − AU )xj+n+1 }. Furthermore, since max{xj , . . . , xj+n } = xj and max{xj+1 , . . . , xj+1+n } = xj+n+1 , it follows that U xj = xj , U xj+n+1 = xj+n+1 , and similarly U AU xj+n+1 = AU xj+n+1 , U AU xj = AU xj , U (I − AU )xj = (I − AU )xj , U (I − AU )xj+n+1 = (I − AU )xj+n+1 . Therefore U (I − AU )xj = min{(I − AU )xj , (I − AU )xj+n+1 } = min{xj − AU xj , xj+n+1 − AU xj+n+1 } = min{U xj − U AU xj , U xj+n+1 − U AU xj+n+1 }. One of the values xj , xj+n+1 , say xj for convenience, is equal to U xi = U xi−1 = · · · = U xj . AU x inherits this sequence of equalities and so does U AU xj . Therefore U (I − AU )xi = min{U xi − U AU xi , U xj+n+1 − U AU xj+n+1 } = U xi − U AU xi . The last step follows from the fact that I − U A is ftp, so that U xj+n+1 − U AU xj+n+1 = U xi + (U xj+n+1 − U xi − U AU xj+n+1 + U AU xi ) − U AU xi ≥ U xi − U AU xi . This is because the term in brackets is positive, since U xi = U xj+n and U AU xi = U AU xj+n and U xj+n+1 − U xj+1 ≥ U A(U x)j+n+1 − U A(U x)j+n , as U A is ftp. Thus the theorem is proved for the ﬁrst statement. The other case follows from a similar argument with Ln instead of Un . Theorem 6.16. If A is ftp and x ∈ Mn−1 , Ln Un (I − ALn Un )x = Ln Un x − Ln Un ALn Un x

and

Un Ln (I − AUn Ln )x = Un Ln x − Un Ln AUn Ln x. Proof. It follows from the previous theorem that Ln Un (I − ALn Un )x = Ln (Un x − Un ALn Un x) = Ln (I − Un ALn )Un x = (Ln − Ln Un ALn )Un x = Ln Un x − Ln Un ALn Un x. Theorem 6.17.

Cj (I − Cn ) = Cj − Cn , for j ≤ n and Fj (I − Fn ) = Fj − Fn , for j ≤ n.

Proof. Let j = 1. Then C1 = L1 U1 and Cn = Cn L1 U1 so that, C1 (I − Cn ) = L1 (U1 (I − Cn L1 U1 )) = L1 (U1 − U1 Cn L1 U1 ), = L1 (I − U1 Cn L1 )U1

by Theorem 1, as Cn L1 is ftp

= (L1 − L1 U1 Cn L1 )U1 , by Theorem 1, as U1 Cn is ftp = L1 U1 − Cn = C1 − Cn .

6. Variation Reduction and Shape Preservation

67

The theorem is therefore true for j = 1. The rest of the proof follows by induction. Assume it is true for j = m < n. Then Cm+1 (I − Cn ) = Lm+1 Um+1 Cm (I − Cn ) = Lm+1 Un+1 (Cm − Cn ) = Lm+1 Um+1 (Cm − Cn Cm ) = Lm+1 Um+1 (I − Cn )Cm = (Lm+1 Um+1 − Ln+1 Um+1 Cn )Cm = Lm+1 Um+1 Cm − Lm+1 Um+1 + Cn = Cm+1 − Cn , if m + 1 ≤ n. A similar proof holds for the second part with L and U interchanged.

Corollary. Cn and Fn are co-idempotent. The co-idempotence of more general compositions of diﬀerent Li Ui operators may also hold, but this is not the concern here. Example 1: L3 U3 L1 U1 (I − L3 U3 L1 U1 ) = L3 U3 (I − L1 U1 L3 U3 )L1 U1 . Whether this simpliﬁes to L3 U3 (I − L3 U3 )L1 U1 = 0L1 U1 = 0, is as yet unclear. Theorem 6.18. The operators Cn , Cm , I − Cn and I − Cm all commute for all positive integers n and m. Proof. (i) Cn Cm = Cmax{n,m} , by Theorem 4. (ii) Cn (I − Cm ) = Cn − Cm , for all n ≤ m = Cn − Cm Cn = (I − Cm )Cn , Cn (I − Cm ) = Cn Cm (I − Cm ), for all n ≥ m = Cn O, by the co-idempotence of Cm . Also (I − Cm )Cn = Cn − Cm Cn = Cn − Cn = 0. (iii) (I − Cn )(I − Cm ) = I − Cm − Cn (I − Cm ) = I − Cm − (I − Cm )Cn = I − Cm − Cn + Cmax{n,m} = I − Cmin{n,m} . Corollary. The operators Fn and I − Fn all commute. (The proof can be done as before, or using the duality Cn (−x) = −Fn (x).) An operator A that is idempotent and co-idempotent is a separator in that it consistently separates (decomposes) a sequence x into Ax and (I − A)x. This is as near to a projection as we can hope to get a nonlinear operator. Since even the composition of two idempotent smoothers does not have to be idempotent, it is of crucial importance that smoothers that are selected to smooth sequentially are as consistent as possible. Since Un and Ln are the primary smoothers under consideration here, the idempotence and co-idempotence of the basic compositions like Ln Un , Un Ln , Cn and Fn go a long way towards consistent behavior. For further consistency the following results are also important. Theorem 6.19. Let x ∈ Mn−1 , and A, B ntp. For all α, β ≥ 0:

68

6. Variation Reduction and Shape Preservation

(a) Un (αA + βBUn )x ≥ αUn Ax + βBUn x and Ln (αA + βBLn )x ≤ αLn Ax + βBLn x. (b) If A also commutes with Un on such a sequence, then Un (αA + βBUn )x = αUn Ax + βBUn x. (c) If A also commutes with Ln on such a sequence, then Ln (αA + βBLn )x = αLn Ax + βBLn x. Proof. (a) With the notation U = Un , let x ∈ Mn−1 and i an index. Assume ﬁrst that U xi = xi . Then there is an index j ∈ [i − n, i] such that max{xj , . . . , xj+n } ≤ xi . But the sets {xj , . . . , xi } and {xi , xj+n } are monotone, thus xj ≤ xj+1 ≤ · · · ≤ xi ≥ · ≥ xj+n . A, BU, U A, U BU, αA + βB and U (αA + βBU ) all transfer these inequalities, to their output, since they are ntp. Therefore U Axi = Axi , U BU xi = BU xi

and

U (αA + βBU )xi = αAxi + βBU xi = αU Axi + βU BU xi . Assume therefore that Un xi = xi (and therefore Un xi > xi ). As previously argued there is a j ∈ [i − n, i] such that xj−1 > xj = xj+1 = · · · = xi = · · · = xj+n−1 < xj+n , and

U xj−1 ≥ U xi = min{xj−1 , xj+n } = U xj−1 = · · · = U xi = · · · = U xj+n−1 ≤ U xj+n .

All the operators in question are ntp so that again the inequalities are inherited by the outputs and U Axi = min{Axj−1 , Axj+n }, U BU xi = min{BU xj−1 , BU xj+n } = BU xi , since B is ntp. U (αA + βBU )xi = min{(αA + βBU )xj−1 , (αA + βBU )xj+n } = min{αAxj−1 + βBU xj−1 , αAxj+n + βBU xj+n }. Now αU Axi + βBU xi ≤ αAxj−1 + βBU xj−1 and αU Axi + βBU xi ≤ αAxj+n + βBU xj+n . The inequality therefore also holds for the minimum of the two, proving that αU Axi + βBU xi ≤ U (αA + βB)xi . A similar argument proves the other part of (a). (b) Noting that U Axi = min{Axj−1 , Axj+n }, assume that Axj−1 ≤ Axj+n so that U Axi = Axj−1 . (A similar argument holds if the other is smaller.) Then U xj−1 = U xj = · · · = U xi = · · · = U xj+n−1 ≤ U xj+n Since A and B are ntp the

6. Variation Reduction and Shape Preservation

69

equalities are inherited by AU x and BU x so that AU xi = AU xj−1 ≤ AU xj+n and BU xi = BU xj−1 ≤ BU xj+n , and so that U (αA + βBU )xi ≤ αAxj−1 + βBU xj−1 ≤ αU Axj−1 + βBU xi . If now U Axj−1 = AU xj−1 then since AU xj−1 = AU xi , we get U (αA + βBU )xi ≤ αAU xi + βBU xi = αU Axi + βBU xi . This, together with the inequality of part (a) of the proof yields the result U (αB + βBU )xi = αU Axi + βBU xi , which proves part (b) of the theorem. (c) A similar proof to the above, or using a duality argument, yields the required equality. Theorem 6.20. For x ∈ Mn−1 and α, β ≥ 0: Un (α(I − Ln Un ) + βLn Un )x = αUn (I − Ln Un )x + βLn Un x and Ln (α(I − Un Ln ) + βUn Ln )x = αLn (I − Un Ln )x + βUn Ln x, Ln Un (α(I − Ln Un ) + βLn Un )x = βLn Un x and Un Ln (α(I − Un Ln ) + βUn Ln )x = βUn Ln x. Proof. Ln and I −Ln Un are ftp and thus ntp. By Theorem 6.15 I −Ln Un commutes with Un , and by Theorem 6.19(b) the ﬁrst equality holds. A similar proof holds for the second equality. For the third equality we apply the ﬁrst equality. Then Ln Un (α(I − Ln Un ) + βLn (Un )x = Ln [αUn (I − Ln Un )x + βLn Un x] = Ln [α(Un x − Un Ln Un x) + βLn Un x], by Theorem 6.15. = Ln [α(I − Ln ) + βLn ]Un x,

since Un Ln Un = Ln Un .

Since Ln commutes with I − Ln , we have by Theorem 6.19 that the last expression becomes [αLn (I − Ln ) + βLn ]Un x = βLn Un x, since Ln is co-idempotent. The last equality is proved similarly, or by duality.

The theorems above that have resulted from the simple concept of ntp turn out to be extremely strong. Recalling the problem of selecting good smoothers amongst the n-LULU similar ones (those that lie inside the LULU interval [Un Ln , Ln Un ]), the natural candidates now seem to identify themselves, if our understanding of the behavior is to be optimal. Ln Un and Un Ln themselves had all the nice properties, but in practice it becomes clear that, for large n, their bias is excessive. (They are too far from the center of the LU LU -interval). Furthermore it is often natural to smooth successively, in order to decide how much smoothing is required. Philosophically, we know that smoothing destroys information, unless

70

6. Variation Reduction and Shape Preservation

that which is removed is stored separately. A smoother is designed to selectively remove unwanted information to highlight wanted information. This is why it is considered an art. If some form of automated selection of degree of smoothing of data is required, it seems natural to smooth recursively, until some selected criterion is met. This leads naturally to the choice of Fn or Cn , for some n, (to be selected as the computation progresses). We shall ﬁnd that computationally this does not become excessive. Such a recursive removal of successive levels of “noise” in a systematic way immediately brings to mind a similar procedure; that of Multiresolution Analysis in Wavelet Theory. This idea is worth investigating in the context of LU LU -theory.

7. Multiresolution Analysis of Sequences “Treu der Natur!” – wie f¨ angt er’s an; Wann w¨ are je Natur im Bilde abgetan Unendlich ist das kleinste St¨ uck der Welt! – Er malt zuletzt davon, was ihm gef¨ allt Und was gef¨ allt ihm? was er malen kann. Nietzsche

The Fast Fourier Transform (FFT) has often been called the most important mathematical tool in modern technology. In a similar way the Fast Wavelet Transform (FWT) may have an impact, for instance, in image processing and transmission. A diﬀerent idea has been pursued, and is considered to be even better in the “Multiresolution Analysis” of digital images. In the book, Image Processing and Data Analysis, Stark Murtagh and Bijaoui follow a discussion of the Wavelet Transform with a section on Multiresolution Analysis (MRA) based on the Median Transform. Roughly speaking, the averaging ﬁlter that maps a function onto a function in a space of lower dimension in the Wavelet Transform is replaced by a median smoother. As usual this can heuristically be motivated by yielding a more robust estimator of an average and therefore outliers do less damage on the “smoother component” in the mapping. Iterative and non-iterative algorithms for Median Transforms are presented. The claim is made that this MRA is well suited when image reconstruction is done from a subset of the (additive-) decomposition for purposes of restoration, compression and partial reconstruction. The reconstructed image is often found to have fewer artifacts than in the case of wavelet decomposition. These artifacts are often in the form of the speciﬁc wavelets chosen. An example is the negative ring surrounding bright point sources. Shapes are found to be closer to those of the input image. The claim is that this is due to the nonlinearity of the median ﬁlter (strictly speaking the nonlinearity permits this better preservation). Computational requirements are listed as high, although there is a saving in that these transforms can be performed in integer values only and decimation can yield a considerable economization. Other morphological tools, speciﬁcally N erosions followed by N dilations are mentioned, with the observation that results were found to be better with the median. Morphological transforms are presented, with mention of the good estimate of the image background that is obtained, especially for images with small structures, as in astronomy.

72

7. Multiresolution Analysis of Sequences

In one-dimensional signal analysis the origin of many artifacts (irritating signiﬁcant distortions) in signals partially reconstructed from wavelet decompositions, can often be understood in the context of approximation theory. The linear projections used in wavelet decompositions may perform relatively well in audio signals. The explanation may ultimately be physiological, but for the purpose at hand, it is suﬃcient to observe that an audio signal is often well approximated locally by trigonometric polynomials, so that the FFT can be truncated for compression. Such local approximation does some damage (Gibbs’ phenomenon), but “softer” windows can be employed to lessen this. Depending upon the support of a speciﬁc wavelet, and the norm of an associated mapping onto a subset of lesser dimension, the Gibbs’ phenomenon may be acceptable for the partial reconstruction. Visual signals or images have a diﬀerent problem, also in one dimension. Sharp edges and constant regions play a signiﬁcant part in the acceptability or recognisability of an image. Such data, approximated by smooth functions, or sequences of sampled smooth functions, bring Gibbs into play, with possible “overshoot” and “undershoot” near edges. An edge can be interpreted as having an impulse in the derivative, or an impulse in the sequence of diﬀerences of samples of such an image. Impulses are not handled well by linear ﬁlters, or smooth functions in approximation theory. Convoluting a wavelet with an impulse yields precisely the wavelet concerned so that impulsive noise on a sequence can be expected to yield spurious features, often close to the shape of the wavelets, or sums of these, at all resolution levels. In the extreme case of the non-local Fourier Transform, the impulse results in a constant function in the frequency domain. This contamination is diﬃcult to remove afterwards. A linear ﬁlter preserves the “energy” in an impulse. Even in the case of the simplest Haar-wavelet decomposition, where the projection onto the lower frequency sequence has minimal norm, an impulse is merely spread onto all levels of the wavelet decomposition. With the LU LU -theory an alternative arises. There are operators with pulses as eigensequences, and where signals are naturally composed of pulses of various widths, a corresponding pulse-decomposition/trend-decomposition can be envisaged. In modern control theory, digital transmission theory, chromatography, image processing, to name but a few examples, it is often natural to assume a composition of pulses inherent in a measurement. In earthquakes there can be a natural assumption of pulses of vibrations, which can be handled by wavelets, pulse decomposition or a combination. In such cases the pulse can also be considered to be in the frequency domain. Due to the inherent ambiguity in the concept of pulses, the decomposition has to be nonlinear, and the natural framework of analysis and computation may by the LU LU -theory. The nonlinearity is not to be seen merely as a generalization of the linear case, but a fundamentally diﬀerent perspective, with a very strong underlying logical structure. This fundamentally speciﬁc use of the term “nonlinear” is often insuﬃciently appreciated by precisely those that are very familiar with linear signal processing.

7. Multiresolution Analysis of Sequences

73

The idea of Multiresolution Analysis with pulses, or Trend Analysis is best developed by considering the simplest Spline Wavelet Analysis, or Haar-decomposition, as a comparison. For a background any textbook on wavelet theory can be consulted, for instance (8). For the purpose of introducing and motivating the subsequent results we exploit the simple one-to-one correspondence of a sequence as a zero-degree B-spline function sampled at the centers of intervals. Local monotonicity as a concept of smoothness then automatically comes into play in both the wavelet decomposition and the LU LU -decompositions. Given a function f , a subset of a function space is chosen so that it is spanned by translations of a so-called scaling function ϕ, which is itself a linear combination ∞ αi φ(2t − i). Generalizations are possible but not required of the type φ(t) = i=−∞

here, in fact it will be suﬃcient for the purpose at hand to consider only the simple Haar-wavelet decomposition. The “scaling function” φ in this case is simply the characteristic function of the interval [0, 1), thus a B spline of order 1 (degree 0). It is clear that φ(t) is a linear combination of φ(2t) and φ(2t − 1), and that in this case it is simply the sum. Thus φ(t) = φ(2t) + φ(2t − 1), and by induction 2k φ(2k t − i). φ(t) = i=0

Consider the space, which is the span of these functions, and consider a sequence x = {xi : i = 0, . . . , N − 1} which is a sampling of a function of f at i+δ the values ti = k , where 0 ≤ δ < 1. Clearly, in this case, the sequence can be 2 identiﬁed with the sequence of coeﬃcients w.r.t. the basis {φi ; φi (k) = φ(2k t − i)}. Letting Bj be the space such that Bj = span {φi ; φi (t) = φ(2j t − i)}, a best least squares estimate P x from Bk−1 to a function x =

i=0

from Bk is easily obtained as N

P x(t) =

N −1

2 1

j=0

2

(α2j + α2j+1 )φ(2k−1 t − j)

and N

(x − P x)(t) =

2 1

j=0

2

(α2j+1 − α2j )ψ(2k−1 t − j),

where ψ(t) = ϕ(2t − 1) − ϕ(2t), is the Haar-wavelet.

αi φ(2k t − i)

74

7. Multiresolution Analysis of Sequences

The basis is orthonormal w.r.t. the inner product (x, y) =

∞

−∞

x(t)y(t)dt,

and the set of wavelets ψi span the orthogonal complement of Bk−1 , if ψi (t) = ψ(2k−1 t−i). For later comparison it is suﬃcient to note that there is a preservation of “energy” in the sequences in that 2 2 ∞ ∞ ∞ α2j + α2j+1 α2j+1 − α2j 2 2 ||α||2 = αi = 2 + 2 . 2 2 i=−∞ i=−∞ i=−∞ In Figure 7.1 the “scaling sequence” and “wavelet sequence” are depicted, and in Figure 7.2 the ﬁrst stage of a decomposition.

Figure 7.1. The Haar scaling functions and its wavelet of minimum support, identiﬁed with sequences.

As an example we can consider a digitized proﬁle of a fort and its ﬁrst decomposition. Thus the sequence x is decomposed into a “smoother” sequence P x, which is pairwise constant, and a “rougher” sequence x − P x which has pairwise elements equal in absolute value, but diﬀering in signs. Signiﬁcant, for later argument, is that P x has every three consecutive elements monotone. Since the wavelet ψ has a deﬁnable frequency, it is natural to view the decomposition as a “smoother” spline P x and a wavelet component, which is a sequence with an associated frequency locally. Repeating such a decomposition, the original sequence x can eventually be decomposed into a constant sequence and several “layers” of wavelets at frequencies that are successively an octave lower than the previous. Schematically the wavelet decomposition can be viewed in the following diagram. From the theory of wavelets the projections Pi have the following properties: (i) Pi f ⊥ {f, P1 f, . . . , Pi−1 f }, (ii) Pi is idempotent, co-idempotent, linear and eigenvalues are only 0 and 1.

7. Multiresolution Analysis of Sequences

75

Figure 7.2. A sequence x decomposed into P x and (I − P )x.

Reconstruction can be achieved by adding the diﬀerent “layers”, and partial reconstruction by a subset of the layers, or a set of subsets of each layer. For data compression, for instance, the coeﬃcients at each wavelet level can be “quantized” and only the nonzero quantized values stored or transmitted. Thus it is reasonable to speak of a “local frequency content” by considering the size of coeﬃcients of wavelets with support at a chosen location. What seems clear is that a signal with sections of almost constant value will have small frequency content at all frequencies there. Only in regions where the sequence varies signiﬁcantly will there be a need for high frequency information being transmitted. For purposes of automatic analysis, signiﬁcant changes in the sequences are identiﬁed with “wavelet activity” nearby. The eﬀect seems local due to the small support of the scaling function ϕ and the wavelet ψ. In a sense therefore it is reasonable to say that global shape is determined by the low frequency content and higher resolution features reside in the wavelet coeﬃcients. Hence the name Multiresolution Analysis (MRA). But should this name not be reserved for a stricter interpretation? A linear transform can be characterized by its response to an “impulse”. Letting di be the sequence {δij : j = 0, . . . , N − 1}, where δij is the Kronecker-delta it is easy to see that, in spite of its minimal support, the wavelet decomposition will have exponential decaying amplitudes in exponentially growing support intervals. A partial reconstruction will therefore inevitably have deviation from the original sequence in an arbitrarily large region. This is the essential problem of the response of a linear mapping to “impulsive noise”, if impulsive noise is precisely deﬁned as an arbitrary multiple of a Kronecker-delta sequence. The “energy” in such an impulse is “spread” or “smeared”, and, depending on the amplitude of the impulse, can completely swamp the essential signal in an arbitrarily large region.

76

7. Multiresolution Analysis of Sequences

f W1 =(I−P1 )

P1

W1 f

P1 f W2 =(I−P2 )

P2

W2 f

P2 P1 f W3 =(I−P3 )

P3

W3 f

P3 P2 P1 f W4 =(I−P4 )

W4 f Figure 7.3. A schematic decomposition of a function f into wavelet sums.

If impulses like these can be expected in a measuring (or transmitting-) device, there will have to be some precaution taken, preferably before any linear transformation is performed. This led to the widespread use of pre-smoothing with running medians. The problem is essentially similar in all wavelet decompositions, and a progressively more local feature has progressively wider frequency content in general; the time-frequency window has an area exceeding a ﬁxed positive quantity. Moreover, the damage done is not restricted to impulses, but to impulses in diﬀerences of the sequences too. Thus a sampling of a simple step function will generally have signiﬁcant distortions spreading into all frequency levels. This behavior is also phase dependent. It is illuminating to consider the above observations in the simple case of a sequence of samplings of a quadratic polynomial, with a simple unit step function (Heaviside function), uniformly distributed random noise and two isolated impulses (Kronecker-delta sequences) added. Choosing a Haar wavelet decomposition, the two successive least squares approximations from subspaces of a half

7. Multiresolution Analysis of Sequences

77

and a quarter of the original dimension are depicted in Figure 7.4. In Figure 7.5 the “wavelet”-components of the ﬁrst two frequency levels illustrate the geometric decay of the amplitude of features introduced by the impulses and the step discontinuity. In both cases two choices of spline knots are compared to show the phase dependence.

Figure 7.4. The original sequence and the sequences representing the partially reconstructed original sequence minus the highest frequency level and minus the two highest frequency levels.

In the above decompositions, it is clear that the synthetic wavelet activity due to the edge and the impulses is ampliﬁed by the factor α if the original impulses and jump discontinuity are, resulting in arbitrary synthetic features arbitrarily far in the decomposition. In image processes, edges are signiﬁcant for picture quality. Linear mappings generally do not preserve monotonicity in a sequence as Median smoothers do, and do so in a precise local sense, as do all rank order selectors. Can this be exploited? Clearly experience suggests that Multiresolution Analysis with medians works well in image processing (1). Are there computationally eﬃcient alternatives, and can an underlying theory provide reassurance of predictable, comparable performance? And if a “feature” or a deviation from a “smooth” surrounding trend is at a precisely deﬁned “resolution”, in that it is suﬃciently local, can it be separated without excessive contamination? The concept of local monotonicity is a more compact characterization of the so-called “roots” of median smoothers. The behavior of Mn itself is enigmatic, and this can be associated with the existence of sections of “spurious” roots in a sequence. The “spurious” roots have recently been shown to be precisely the periodic ones (33) that are not in p . In the case of M2 it is essentially only one sequence, namely xi = (−1)i , and its multiples.

78

7. Multiresolution Analysis of Sequences

Figure 7.5. The sequences representing the highest and second highest frequency levels in the Haar-decomposition.

If x ∈ X then x is 0-monotone and therefore M0 = X. Furthermore M0 ⊃ M1 ⊃ · · · ⊃ Mn ⊃ · · · form a sequence of nested subsets. (A Haar-decomposition projects into M1 with the ﬁrst decomposition onto M3 with the second and so forth, and this is dependent on the phase, or therefore on the choice of the nodes of the splines involved.) If an alternative decomposition is to be constructed it can be considered prudent to aim for an elementary separator P in such a way that the criteria for a separator listed in Chapter 1 are not compromised too much. Eﬀectiveness: The output P must be a sequence without higher resolution detail. Eﬃciency:

The computations must be economical in terms of basic digital operations like logical comparisons, additions, multiplications, divisions etc. Consistency: Mapping the output again should preserve it, or conﬁrm it as good. Stability: Input perturbations should not result in excessive output perturbations. Considering the simple Haar-wavelet, the projection operator P , is eﬀective since it is a projection onto a spline-subspace of half the original dimension of the (order 1 -) spline space at the sampling resolution. It is achieved eﬃciently by a simple averaging ﬁlter on the sequence. Since a projection operator is idempotent it preserves its own output. (It is noteworthy that this necessitates a ﬁlter that is not translation invariant (and therefore phase dependent as it depends on the nodes of the spline subspace of lower dimension onto which it is mapped.) Thus it does not meet the requirements of one of the axioms of a smoother, as introduced by Mallows. Stability has been argued to be suspect in the case of features that are “brief” impulses. Excessive output perturbations can result.

7. Multiresolution Analysis of Sequences

79

Choosing as the ﬁrst separator in an alternative Multiresolution Analysis the operator P1 = L1 U1 , it is clear that the output is 1-monotone. (U1 L1 would be another choice leading to a similar scheme.) A general sequence x ∈ X is then eﬀectively mapped onto a 1-monotone sequence P1 x in M1 . This operator is eﬃcient, as will be shown later. Since L1 U1 is idempotent it is consistent in preserving its output, but, since the operator is nonlinear, consistency demands somewhat more. Since P1 is not a projection, the component (I − P2 )x that is removed must also be consistently removed, in that (I − P1 )(I − P1 )x = (I − P1 )x. This means that I − P1 must also be idempotent, and this “co-idempotence” of Pn = Ln Un is equivalent to having (I − Pn )x being a null-sequence of the operator Pn , for each x. The separation can thus be considered to be consistent. Stability is good since the operator has a Lipschitz constant, so that small amplitude perturbations cannot be ampliﬁed. Furthermore a single (large) impulse has an inﬂuence restricted to amplitudes of neighbors, and the inﬂuence is local. Under the heading of eﬀectiveness, a further consideration arises when P is not a projection. A projection onto a subspace S is automatically a good approximation in the appropriate norm, and since the Lebesgue inequality is applicable, also if the norm of P is ﬁnite, in any other norm. If Pn is merely a separator, it is important to consider whether the image Pn x is a good approximation from Mn to the sequence x. This was shown to be so in the cases of Ln Un and Un Ln . Thus the separator L1 U1 eﬀectively separates a sequence x into a good approximation L1 U1 x in M1 and a (high resolution) sequence (I − L1 U1 )x, which is a null-sequence of L1 U1 , and thus can be considered to consist of suﬃciently local impulses, suﬃciently separated not to yield a lower resolution nonzero output when mapped by L1 U1 . It can be considered as a sum of “noiselets”. It must be stressed that the sum of such null-sequences is not automatically a null-sequence again! Furthermore, the operator Un Ln , although considering Ln Un x as a “signal”, since it is in Mn , does not necessarily consider (conﬁrm) (I − Ln Un )x as being noise. This is because Un Ln (I − Ln Un ) is not the zero operator, although Ln Un (I − Ln Un ) is. Ln Un and Un Ln have a common range but I − Ln Un and I − Un Ln do not. These observations are associated with a fundamental “uncertainty principle” with respect to the concept of impulse, and will result in a similar uncertainty in the concept of resolution, if made strict. The two decompositions, with LU and U L can be eﬀectively done in parallel, but separately. In the following example it is instructive to view the two large impulses, as well as the “jump-discontinuity” as being suﬃciently large, so that the “smoother” parts L1 U1 x and U1 L1 x will not be aﬀected at all if the impulses are multiplied by an arbitrarily larger number. All the extra amplitude will be restricted to the noise-components x − L1 U1 x and x − U1 L1 x. No change will result in lower layers of decompositions. Only when the amplitude decreases to the level of the local variation of the uncontaminated signal will there be a minor change in L1 U1 x and U1 L1 x, minor meaning of the order of the diﬀerence between the two. Since the Haar-decomposition is based on linear operators the multiplication of the impulses by an arbitrary α will result in a proportional distortion by the same factor

80

7. Multiresolution Analysis of Sequences

in the wavelet component resulting in proportional distortion with exponentially increasing width. With suﬃciently large amplitude impulses this can swamp all signiﬁcant features of the original sequence. In Figure 7.6 a sequence like that in Figure 7.4 is decomposed once by each of L1 U1 and U1 L1 to illustrate the claims made. This can conveniently be called a “trend decomposition”. The Median transform would be between those of Ln U1 and U1 L1 and very similar. x L1U1x U1L1x

(I–L1U1)x (I–U1L1)x

Figure 7.6. Illustration of a “trend decomposition” using the operators L1 U1 and U1 L1 .

It is worthwhile comparing a Haar decomposition of the same sequence x. x Px 1

(I–P1x)

Figure 7.7. Haar (wavelet-) decomposition of the same sequence as above.

In the Haar-decomposition the large scale level change smears this signiﬁcant feature to exponentially growing large sections of the successively smoother decompositions. The synthetic wavelet activity at the jump will have a correspondingly large amplitude in all subsequent wavelet layers. The well-known idea of thresholding the wavelet coeﬃcient sequences to handle impulsive noise is clearly

7. Multiresolution Analysis of Sequences

81

limited in its eﬀectiveness, quite apart from the diﬃculty of choosing an appropriate threshold. The observations above are demonstrated with optimal clarity in the simple example of a constant signal and a single impulse of width 1. The ﬁrst two Haar-decompositions in Figure 7.8 clearly demonstrate the exponentially growing width in both components of the decompositions, and because the operators involved are linear this behavior is scale independent. Omitting one or more levels of wavelet components will result in a large distortion in the reconstructed (-smoothed signal). The LU LU -decomposition of the same signal and impulse has the full energy of the impulse in its ﬁrst noise-component and omitting of this (highest resolution) component and any other will leave a perfectly smoothed original constant signal. No sketch is required to see this.

Figure 7.8. The impulse response of a Haar-decomposition.

Clearly, since the Haar decomposition is linear, this behavior will result in a similar distortion when added to any signal. Since the LU LU -decompositions are not linear, the superposition of this impulse on a given signal x will not necessarily result in an undistorted removal of the impulse from x. But all the distortions will not exceed the magnitude of the local variation of the signal at the position of the impulse. Strictly speaking L1 U1 x and U1 L1 x cannot be distorted by more than the factor max{|xi+1 − xi |, |xi − xi−1 |} at i, if the arbitrarily large impulse is added at i, since both have either the value xi−1 or xi+1 . No distortion larger than this can result in any lower level of decomposition, since the induced 1-norms of all the operators Ln Un and Un Ln are 1. A further distortion occurs at the two neighboring points i − 1 and i + 1, but this distortion cannot exceed |xi−2 − xi−1 | and |xi+2 − xi+1 | respectively. Clearly the contamination can spread, but this cannot exceed the maximum amplitude of the signal in the corresponding region. Furthermore, it does not necessarily spread as far as it can theoretically, depending upon the given underlying signal. A precise analysis is not intended here, and may become exceedingly diﬃcult. Experience suggests very limited growth in the contamination support.

82

7. Multiresolution Analysis of Sequences

A practical typical comparison that is illustrative of the speciﬁc advantage of the LU LU -decomposition argued above is the following example, where a broad nimpulse is decomposed by LU LU -decomposition and Haar decomposition. Except for the, progressively more unlikely event, as n increases, of the n-pulse starting and ending at a node of the lower dimensional spline subspaces, this is typical.

Figure 7.9. One stage of decomposition by L1 U1 −, U1 L1 − and Haar-decomposition.

Having suggested and heuristically argued many “advantages” the LU LU decompositions have, it would be appropriate to derive some theorems that can support some of these known, observed or believed properties of the LU LU decomposition. In linear orthogonal decompositions the squares of 2-norm are preserved. A pure n-pulse has the same “energy” as n 1-pulses of the same amplitude, but, depending upon how these 1-pulses are distributed, can have a total variation between 2α and 2nα. The total variation is therefore a measure of the “resolution” of features. Pursuing this idea leads to the following substantial results. The operators involved were originally intended for nonlinear smoothing, or pre-smoothing for the removal of impulsive noise. On attempting to clarify and quantify some observed experimental observations and case studies, it is natural to choose total variation of a sequence as a measure of smoothness. A wavelet decomposition of a sequence x ∈ p with order 1 or 2 spline wavelets will eventually lead to a constant sequence, which has to be the null sequence. Thus all the “energy” has been peeled oﬀ into the (wavelet-) frequency layers, since the squares of the 2-norms of P x and x − P x add up to the square of the 2-norm of x, so that ||x − P1 x||22 + ||P1 x − P2 P1 x||22 + · · · = ||x||22 . The behavior of the total variation of x and all the components of the decomposition at the intermediate stages is of interest, if smoothing is desired for the purpose of exposing signiﬁcant features in the sequence x without too much dam-

7. Multiresolution Analysis of Sequences

83

age. In a wavelet-decomposition, as in a Fourier decomposition, it is natural to stop the decomposition process when the frequency layers do not contain signiﬁcant energy any more. Similarly it would be convenient if the reduction in variation has some natural measure giving precise indication of what fraction of the total variation has been removed. We assume that the features to be exposed by smoothing are given by a sequence e to which higher resolution noise, given by a sequence r, has been added to produce a signiﬁcant increase in variation. Since T (x) = T (e + r) ≤ T (e) + T (r) in general we are therefore assuming that T (x) ≈ T (e)+T (r). We should like to stop smoothing when the total variation goes signiﬁcantly below that of T (e), which is not known but perhaps estimated. If the noise r is of much higher “resolution” than the desired features of the sequence, the successive peeling oﬀ of resolution layers could be expected to decrease the variation steadily until most of the noisy features are removed. Reaching the expected resolution level of e should result in a further strong reduction, perhaps signaling that the unknown resolution level of e has been reached, and further “smoothing” would partially erase these. This Heuristic motivation could be experimentally supported, but clearly the underlying assumption of a “proportional” allocation of the total variation to each resolution layer, is crucial in the appropriateness. The LU LU -decompositions have such a remarkable property. It can be developed in the following way. Lemma. Let x be (n−1)-monotone. If j is a point where Un xj = xj and xj−1 = xj , then Un xj = min{xj−1 , xj+n } and Un xj−1 = xj−1 . Proof. If Un xj diﬀers from xj it must be larger, since Un x ≥ x. Un xj = min{max{xj−n , . . . , xj }, . . . , max{xj , . . . , xj+n }} > xj , implies that each of the maxima is larger that xj , so that there are at least two values x ∈ {xj−n , . . . , xj } and xr ∈ {xj , . . . , xj+n }, such that x , xr > xj . But x is (n − 1)-monotone, so that each set of n successive elements of x are monotone. Then xj−n ≥ · · · ≥ x ≥ · · · xj−1 ≥ xj ≤ xj+1 ≤ · · · ≤ xr ≤ · · · ≤ xj+n . Noting xj−1 = xj , from the assumption of the lemma, it must follow that xj−1 > xj . From this it follows that there must be a constant section of equal values, since xj−1 > xj ≥ · · · ≥ xj+n−1 and xj ≤ · · · ≤ xj+n−1 ≤ xj+n , imply that xj = xj+1 = · · · = xj+n−1 . At least one of the points xj+1 , . . . , xj+n must be strictly larger than xj , so that this value must be xj+n . Therefore also Un xj+n = xj+n = xr and r = j + n. Thus Un xj = min{max{xj−1 , . . . , xj+n−1 }, xj+n } = min{xj−1 , xj+n }. Further more, clearly Un xj−1 = xj−1 , since max{xj−1 , xj , . . . , xj−1 } = xj−1 . Although a more general theorem was previously derived, it is instructive to prove the following theorem again, so that the features of the sequences that are separated become clearer in the case of successive separation into resolution levels.

84

7. Multiresolution Analysis of Sequences

Theorem 7.1. For x ∈ Mn−1 , n ≥ 1, n n T (x) = T ( x) + T (Un x − x) = T ( x) + T (Ln x − x).

Proof. Consider the sequence t = {tj } of integers where Un xtj > xtj and xtj +1 = xtj . (Recall the previous lemma, and note that {xtj −n , . . . , xtj } are (n − 1)monotone, xtj −1 > xtj .) The sequence t cannot contain two consecutive integers, so that tj+1 −n−1 ∞ τj , with τj = |xi+1 − xi |. T (x) = j=−∞

i=tj −n

Consider a speciﬁc j, with k = tj and m = tj+1 , and for notational convenience, let cj+1 = w be the ﬁrst index after k such that xw > xw+1 . Clearly w exists in [k + 1, m − 1]. Since x is (n − 1)-monotone, and therefore xk−1 > xk ≥ xk+1 ≥ · · · ≥ xk+n−1

and xk ≤ xk+1 ≤ · · · ≤ xk+n ,

it follows that xk = xk+1 = · · · = xk+n−1 . A similar argument yields xw−n = · · · = xw−1 = xw . By the previous lemma U xk = min{xk−1 , xk+n } implies that xk−1 , xk+n and w must be larger than xk . Since tj+1 is the ﬁrst integer after tj where the sequence changes from a monotone decreasing to monotone increasing set of values, {xk+n , . . . , xm−n−1 } has the following structure; {xk−n , . . . , xk−1 } is a monotone decreasing section, xk−1 > xk = xk+1 = · · · = xk+n−1 , {xk , . . . , xw } is monotone increasing and xw > xw+1 ≥ · · · ≥ xm . Let: µj =

m−n−1

|

n

xi+1 −

n

xi |

i=k−n

=

k−2 i=k−n

|

n

xi+1 −

n

xi | +

k−1

|

n

xi+1 −

n

i=k−1

xi | +

m−n−1

|

n

xi+1 −

n

xi |.

i=k

The ﬁrst of these three sums is equal to k−2

|xi+1 − xi |,

i=k−n

since the set {xk−n , . . . , xk+n−1 } is monotone decreasing. n n xk − xk−1 | = |xk+n − xk−1 |, and the last sum has The next sum is | two cases:

7. Multiresolution Analysis of Sequences

85

(i) Suppose w > m − n − 1. Then m−n−1

|

n

n

xi+1 −

xi | = |xk+n+1 − xk+n | + · · · |xw−n + xw−n−1 |,

i=k

since m − n − 1 > w − n − 1 if w < m and xw−n = xw−n+1 = · · · = xw−1 = xw . This is the variation on a monotone increasing section and yields |xw − xk+n |. Thus k−2 µj = |xi+1 − xi | + |xk+n − xk−1 | + xw − xk+n i=k−n k−1

=

|xi+1 − xi | − |xk − xk−1 | + |xk+n − xk−1 |.

i=k−n

But xw − xk+n = xw − xk+n−1 + xk+n−1 − xk =

m−n−1

|xi+1 − xi | + xk+n−1 − xk

i=k

= τj + (xk − xk−1 ) + |xk+n − xk−1 | + xk − xk+n . Therefore µj = τj + 2xk − 2 min{xk+n , xk−1 } = τj − 2(Un xk − xk ), from the previous lemma = τj −

m−n−1 m−n−1 2 |Un xi − xi | = τj + |Un xi+1 − xi+1 − Un xi + xi |. m i=k−n

i=k−n

The last equality comes from the fact that U xi − xi = U xk − xk for i = k, . . . , k + n−1 and zero elsewhere in the interval [k−n, . . . , m−n−1], and then the variation of this block pulse is simply twice the height. (ii) Suppose m − n − 1 ≥ w. Then m−n−1

n

i=k m−n−1

n

| |

xi+1 − xi+1 −

n

n

xi | = xw − xk+n + xw − xm−n

or

xi | = xw − xk + xw − xm−n + xk − xk+n .

i=k

This again yields µj = τj −

m−n−1 2 |Un xi − xi | = τj − 2(Un xk − xk ). n i=k−n

86

7. Multiresolution Analysis of Sequences

In both cases therefore n ∞ ∞ ∞ T ( x) = µj = τj − j=−∞

j=−∞

j=−∞

2 n

tj+1 −n−1

|Un xi − xi |

i=tj −n

= T (x) − T (Un x − x) n or T (x) = T ( x) + T (x − Un x). n x) + By a similar argument, or by the usual duality argument, T (x) = T ( T (x−Ln x). This proves the theorem. As was proved previously, a corollary follows easily: Corollary: For x ∈ Mn−1 , n ≥ 1: T (x) = T (Un x) + T (x − Un x) and T (x) = T (Ln x) + T (x − Ln x), T (x) = T (Ln Un x) + T (x − Ln Un x) and T (x) = T (Un Ln x) + T (x − Ln Un x). Theorem 7.2. Then ||Un x − Ln x||1 = ||Un x − x||1 + ||x − Ln x||1 for n ≥ 0. Proof. Since Un xi ≥ xi ≥ Ln xi it follows easily that ∞

|Un xi − Ln xi | =

i=−∞

=

=

∞

Un xi − Ln xi

i=−∞ ∞

(Un xi − xi + xi − Ln xi )

i=−∞ ∞ i=−∞

Un xi − xi +

∞

xi − Ln xi

i=−∞

= ||Un x − x||1 + ||x − Ln x||1 .

In the above comparisons the complications at endpoints of a ﬁnite sequence are avoided for the sake of simplicity. These problems can usually be treated satisfactorily in a variety of diﬀerent ways by letting both beginning and end of a ﬁnite sequence move to zero in a variety of diﬀerent reasonable ways. Furthermore, there is an arguably unfair advantage that is easily overlooked if only the ﬁrst decompositions are considered. If the sequences are ﬁnite, then a Haar-decomposition projects the sequence x onto a nested set of subspaces, each being of half the dimension of the previous until a constant is reached. The LU LU -operators map, in a way that is projection-

7. Multiresolution Analysis of Sequences

87

like, (roughly speaking, as near to a projection as can be expected with a nonlinear operator) onto nested subsets M1 ⊃ M2 ⊃ M3 ⊃ · · · etc. Since these are not subspaces, there is no question of dimension, but to be fair it should be realized that there is not a decomposition into octaves, but rather a linear decrease in “ﬂexibility” (for lack of a better word). A fairer comparison would be if the decomposition were from M0 to M1 to M3 to M5 etc., since, if the range of the projections Pj of the Haar-decomposition is Rj we have that Rj ⊂ M2j −1 . The proof of this is obvious if we observe that a sequence in Rj is a sampled B-spline of order 1, which is (2j − 1)-monotone, since it is a sampling of a piecewise constant function. The sequence is therefore made up of successive sections of 2j equal values, and clearly every 2j + 1 successive values are monotone. At this stage there has been no attempt to suggest comparison in a signiﬁcant advantage of the wavelet-decomposition, namely the economizing in representation and computation by the fact that the smoother component and the wavelet component can be economically stored by the respective spline and wavelet basis. This yields an eﬀective representation in no more numbers than the original number of elements of the sequence. These coeﬃcients can furthermore be quantized without major distortion in the reconstructed sequence. In the LU LU -decomposition there are no bases that are generally useful for economization. There are several possibilities for savings by coding the noise components, which can be progressively more sparse. Similarly the smoother part could have progressively larger constant sections permitting some economizing. This whole issue is not addressed here, and it is generally complicated, as in the case of two-dimensional image processing, with both wavelet decompositions and Median decompositions. The primary comparisons to be made here are more fundamental in nature. The LU LU -decompositions are an alternative to the prevalent median transform of the same type. These seem generally to be good in the two-dimensional case of image processing (3). A disadvantage listed is the computational complexity. A more serious disadvantage seems to be the lack of theory. Wavelet theory has a comprehensive and beautiful theory for analysis. For linear operators, Fourier Transforms and their inverses provide a framework for analysis. For a large class of operators, Mathematical Morphology provides a framework of analysis, and is well developed, especially in the two-dimensional case. Starck, Murtagh and Bijaoni state that median transforms are considered better than Morphological Filters. This may be when considering only the simpler compositions like Un and Ln , in the one-dimensional case, when Ln ≤ Mn ≤ Un is true, but the interval [Ln , Un ] is too large and Ln and Un are only comparable to Mn in approximating properties when the signiﬁcant features in the noise are one-sided. The inequality Un Ln ≤ Mn ≤ Ln Un is much sharper and the operators Un Ln and Ln Un are good approximations to the Median. This is visible in the foregoing examples where the ﬁrst decompositions with Ln Un and Un Ln are compared. Since this diﬀerence can be argued to be within the fundamental uncertainty interval of the concept “impulse” and an associated concept of “resolution”, they must both be expected to be equivalent to the Median for the purpose at hand.

88

7. Multiresolution Analysis of Sequences

The advantage argued here is in the application to analysis of measurements. Signiﬁcant features of measurements that are fundamental are “edges”, “local trend”, and “pulses”. The words “edges”, “trend”, “impulse”, “pulse” and “resolution” are all widely used in science and technology and are very often not clearly deﬁned in the context. For a multiresolution decomposition to be called such, it may be prudent to become somewhat more precise. In chemistry, for example, a spectroscopy apparatus can be said to have a resolution suﬃcient to distinguish between spectral lines of two chemical compounds. These may be of Gaussian-shape and the instrument must be able to separate two deﬁnite maxima, even in the presence of some reasonable expected noise. If the instrument has a higher resolution it can separate a spectrum of each of these compounds into several Gaussian pulses from the presence of a few ionisation species in the compounds. When the instrument is even better it could even break up these pulses into pulses of diﬀerent isotopes in the species. It is instructive to consider again an ambiguity intrinsic in the concept of a pulse with a simple example. In Figure 7.10, a pair of level 1 pulses is seen to be ambiguously interpretable as a pulse at level three with two level 1 pulses superimposed, or as two level 1 impulses. Clearly alternate interpretations are also possible, but the ambiguity is at least apparent from the diﬀerent sequences x − L1 U1 x and x − U1 L1 x.

Figure 7.10. The ﬁrst decompositions of a two pulse signal by L1 U1 and U1 L1 .

After the next two decompositions no pulse of lower resolution will be present. The only activity is in the ﬁrst three layers indicating the fundamental resolution interval [1, 3]. The Haar decomposition of the same pulse in Figure 7.11 smears activity into all levels, making a stricter use of the term “resolution” diﬃcult. It is not the purpose here to expand on all the various advantages over wavelet transforms, but to indicate the type of use that has been, and can be made in the detection of detail at various resolution levels. The method of analysis is

7. Multiresolution Analysis of Sequences

89

Figure 7.11

relatively unknown, and based on the strong mathematical structure on the basic operators. Having deﬁned a pure n-impulse pn it may be useful to deﬁne a pulse p of resolution between m and n if pm ≤ p ≤ pn . If superimposed noise is comparatively small, it is clear that, by the syntoneness of the operators involved in the LU LU decomposition, the pulse will appear in the mth level and disappear in the nth level. Thus it may be a more precise way of deﬁning what is meant by a pulse at a resolution level (or interval). If two such pulses are separated suﬃciently they can be resolved. When they are superimposed, or close enough, the fundamental uncertainty at that resolution level can be recognized by the diﬀerence between the decomposition with Ln Un and Un Ln . At each level n the operators Ln , Un , Un Ln and Ln Un are all idempotent and co-idempotent and all compositions are one of these four. This near-ring of operators has a complete order given by Ln ≤ Un Ln ≤ Ln Un ≤ Un . In the decomposition procedure several layers of such operators are used, and since they are all syntone operators, relative orders are inherited and remain useful for analysis. The well-established Median operators, popularly used for smoothing out impulsive noise and accepted to be very good for preservation of edges and other signiﬁcant features, are contained between such LU LU -outputs. When decompositions with Ln Un and Un Ln are compared, the diﬀerence indicates an “amount of ambiguity, which turns out to be very useful. Using the “commutators” like Ln Un − Un Ln resulted in the design of recording instruments for times of arrival of shock pulses for location purposes. In the ﬁrst application attempted, accuracy of such time of arrival estimates have exceeded the best previous designs of an international company by a large margin. Figure 7.12 below shows the histogram of arrival errors of the LU LU -based trigger compared to the triggers then in use. The advantage seems due to the strong edge preservation (shape preservation) in the presence of large quantities of noise.

90

7. Multiresolution Analysis of Sequences LULU triggers

50

0

50 100 150 Distance from P

200

STA triggers

50

0

50 100 150 Distance from P

200

Figure 7.12. Comparison of time-of-arrival errors between LU LU -estimates and the current estimates based on ratios of linear smoothers, in histogram form.

At this stage the merit of an alternative Multiresolution Analysis is suﬃciently apparent, but the fundamental problem that looms over the whole idea is the nonlinearity of the decomposition operators (separators). If we have a Wavelet (or Fourier-) decomposition we have a simple change of basis and any linear combination of the basis vectors are guaranteed to be decomposed into precisely that set of sequences again. Suppose however that a sequence x ∈ M0 has been decomposed into a sequence of resolution sequences r(n) ; n = 1, 2, . . . by a Multiresolution Decomposition with the operators Ln Un , for n = 1, 2, . . .. Several questions arise immediately: 1. Will a basic modiﬁcation of any resolution level, such as multiplication by a non-negative factor, have a decomposition that will have simply that resolution level similarly modiﬁed, but otherwise identical? 2. What is to be expected when several high, or low or arbitrary, resolution sequences undergo basic modiﬁcation and are added together and decomposed? Such questions need to be resolved before any degree of conﬁdence can be attached to the value of the whole decomposition procedure.

8. The Discrete Pulse Transform Bleib nicht auf ebnem Feld! Steig nicht zu hoch hinaus! Am sch¨ onsten sieht die Welt von halber H¨ ohe aus. Nietzsche

To address the question of consistency of the DPT, it is convenient to introduce a notation and compare with the Direct Fourier Transform (DFT or FFT). As in the case of the Fourier Transform, we often seek a “band pass ﬁlter”. The information required needs to have the high resolution detail and the low resolution drift removed. What is the distortion that results? In the DFT the removal of high frequencies results in smearing of edges (detail), and if the trend (drift) is not removed there will be unwanted distortion at the edges. Consistency of the DPT seems to be unobtainable, since we have compositions of essentially nonlinear operators at work. The remarkable shape preservation properties of the basic operators do however keep suﬃcient hope alive to try the seemingly impossible. Conceptually a sequence x = xi ; i = 0, 1, 2, . . . , N = 2m , with xN = x0 N is decomposed into N2 + 1 sequences s0) , s(1) , . . . , s( 2 ) , where s(0) is a constant sequence and s(n) is a periodic sequence with frequency n times the fundamental frequency. As is well known, the mappings Dn that map x onto s(n) are projections onto a one-dimensional space when n = 0 and a two-dimensional space otherwise. Each component sequence can thus be associated with two numbers, either coeﬃcients with respect to a chosen basis or an amplitude and phase pair. The notation D(x) = DF T (x) = [D0 (x), D1 (x), . . . , DM (x)], with M = 2m−1 , is convenient, with s(i) = Di (x). Since the process is essentially a basis transformation it is obvious that the DFT is component consistent in the following sense; If z =

M

αi Di (x),

then

Dn (z) = αn Dn (x),

for each n = 0, 1, . . . , M.

i=0

Furthermore, the orthonormality yields the “Energy” preservation law; ||x||22 =

M i=0

||Di (x)||22 ,

92

8. The Discrete Pulse Transform

so that it is possible to allocate a percentage of the energy in x to each frequency level. The proposed DPT (Discrete Pulse Transform), can then conveniently be viewed as the mapping of a sequence x = xi ; i = 0, 1, . . . , N = 2m , with xN = x0 onto a vector of sequences r(n) = Dn (x), at diﬀerent “resolution levels”, such that D(x) = DP T (x) = [D1 (x), D2 (x), . . . , DN (x), D0 (x)],

with

D0 (x) = CN x.

There are two equivalent natural primary choices for such decomposition procedures, based on the smoothers Ln Un , or Un Ln . Considering the ﬁrst choice, the decomposition proceeds recursively. The sequence x is separated by the separator L1 U1 into a “smoother” sequence L1 U1 x and the highest resolution sequence (I − L1 U1 )x = D1 (x) = r(1) . The smoother part L1 U1 x is then separated by L2 U2 to yield the second “resolution component” r(2) = D2 (x) = (I − L2 U2 )L1 U1 x and the smoother part L2 U2 L1 U1 x. This is continued until after the N th separation only a constant sequence D0 (x) remains. If the restriction xN = x0 on x is omitted, the “lowest resolution” component D0 (x) will be monotone, but not necessarily constant. We shall however demand xN = x0 , as this is also done with the DFT where “trend” is ﬁrst removed. Clearly the properties of the DPT will result from the properties of the smoothers Ln Un . The DPT is not a basis transformation like the DFT, it is not even linear in any of its separators. It is therefore not even clear that, if we take an individual resolution component r(n) of x, the decomposition of the sequence r(n) will yield r(n) at level n, or that the other resolution levels would have zero sequences. More generally it can be stated (any example is a proof) that the decomposition of a sum is not the sum of the decompositions nor is DP T (αx) = αDP T (x), for all α. Furthermore the LU decomposition does not even conﬁrm that of U L, in the sense that if the nth resolution component r(n) of x, as decomposed by the operators LU , is decomposed by the U L decomposition, then in general r(n) will not be the nth resolution component of x. The consistency of each DPT will therefore require careful analysis. As it turns out a remarkable consistency exists, which needs to be proved carefully. The consistency of the individual separators Ln Un (or Un Ln ) is the key to initial consistency analysis. It is worth reviewing the argument here. If an operator S is to separate a sequence into a “signal” Sx and (additive-) “noise” (I − S)x then this should ideally be consistent. A projection is an example of a consistent “separator” in that having obtained Sx and (I − S)x from a given sequence, separation of Sx would yield only S(Sx) = Sx and (I − S)Sx = 0, which signiﬁes no noise. Similarly if (I − S)x is separated (I − S)((I − S)x) = (I − S)x, as a projection is linear and S(I −S)x = 0, which signiﬁes pure “noise” and no “signal”. If an operator is not linear, then idempotence results in consistent signal extraction as then S(Sx) = Sx, but S(I−S)x is not necessarily the zero sequence. For consistent separation both idempotence and co-idempotence have to be required.

8. The Discrete Pulse Transform

93

Such operators are separators, and were shown to have only two eigenvalues 1 and 0, corresponding to the “signals” and “noise” as eigensequences respectively and Mn is exactly the set of eigensequences with respect to 1 of Ln Un and Un Ln , but they do not form subspaces, since the sum of two signals need not be a signal. (As was shown, the operators I − Ln Un and I − Un Ln do not share their eigensequences w.r.t. the eigenvalue 0. Ln Un and Un Ln do not always agree on what is “noise”, although they agree on what is “signal”, since Ln Un (Un Ln ) = Un Ln and Un Ln (Ln Un ) = Ln Un .). If the DPT with Ln Un or the DPT with Un Ln are considered they can be viewed as separating a given sequence x into N + 1 component sequences at diﬀerent resolution levels. Primary consistency demands that if any resolution level is taken through the separation process, it should yield zero sequences everywhere else and the full sequence at the appropriate resolution level. The following theorem guarantees just that. Theorem 8.1. Let x be a given sequence in M0 and DP T (x) = [D1 x, D2 x, . . . , DN x, D0 x]. Then Dj (Di x) = δij Di x, where δij is the Kronecker delta. Proof. Consider the DPT with Ln Un as separators. Firstly D1 x = (I − L1 U1 )x and (I − L1 U1 )D1 x = (I − L1 U1 )((I − L1 U1 )x) = (I − L1 U1 )2 x = (I − L1 U1 )x, by the co-idempotency of L1 U1 . Thus D1 (D1 x) = Dn (x), and since L1 U1 (I − L1 U1 ) = 0 the zero sequence is passed to the next separators and yields the zero sequence at each resolution level. For n > 1 the sequence Dn x = (I − Ln Un )Cn−1 x, which is in Mn−1 , passes unchanged through the smoothers Lj Uj and clearly yields zero sequences r(j) = 0 for j ≤ n − 1. At the separator Ln Un the output 0 is passed to the next separator yielding subsequent zeroes, since (I − Ln Un )Dn x = (I − Ln Un )(I − Ln Un )Cn−1 x = (I − Ln Un )Cn−1 x = Dn x. Thus Dn x appears in full at resolution level n, and Dj (Dn x) = δjn Dn x. A similar proof holds for the DPT with Un Ln .

Thus individual resolution components of a sequence are consistently decomposed and, as a next stage, it is reasonable to investigate the consistency as a “low-pass” smoother. A sequence z is formed by omitting the ﬁrst (highest-) resolution components of a sequence x. The following theorem demonstrates such consistent decomposition of z. Theorem 8.2. Let x ∈ in M0 that is decomposed by the DPT of Theorem 8.1, and n 0 for m < n, (i) . z =x− r , with r(i) = Di x. Then Dm (z) = (m) for m ≥ n r i=1

94

8. The Discrete Pulse Transform

Proof. By induction on n, z =x−

n

r(i) = Cn x.

i=1

Since Cn x ∈ Mn the smoothers Lm Um leave Cn x unchanged while m < n. Thus all the high resolution components of z are zero until smoother Ln+1 Un+1 is reached, which separates z = Cn x into Cn+1 x and the resolution component Dn+1 z = (I − Ln+1 Un+1 )Cn x = Dn+1 x. The rest of the decomposition clearly follows as for x, yielding the same values for the low resolution components. A similar proof holds for the DPT with Un Ln .

x

F x 1 F1z

F2x

(1) (2) z=r +r

(I–F )x = r(1) 1 (I–F1)z = r(1)

(I–F2)F1x = r(2)

F2z

(I–F2)F1z = r(2)

F x 3

(I–F3)F2x = r(3)

F3z

(I–F3)F2z = 0

Figure 8.1. Theorem 8.2 veriﬁed by a random sequence.

A further level of consistency would be to omit the low resolution components , as would be the case in a “high-pass” smoother. Here however the result requires some additional theory. Suppose, for instance that x has only two nonzero resolution levels r(1) = D1 x and r(2) = D2 x. By the previous theorems, omitting any of the resolution levels to form a sequence z, the decomposition would be consistent with that of x. But if there are three nonzero resolution levels, there are two partial sums that are problematic.

8. The Discrete Pulse Transform

95

Assume z = r(1) + r(2) = x − r(3) . D1 z = (I − L1 U1 )(I − L1 U1 )x + (I − L2 U2 )(C1 x), and since neither L1 U1 nor I − L1 U are linear, it is not clear that D1 z = D1 x = r(1) . The result seems to hinge on the fact that L1 U1 acts linearly on the particular sum r(1) + r(2) = x − C2 x = (I − C2 )x. Omitting the resolution component r(2) yields similar problems in the proof of the consistent decomposition of the partial sum. Since experimentation suggests consistency of this type, there must exist some theory not yet explored. The consistency seems too good to be true. We start with the “band pass” case. Theorem 8.3. Let z = r(m) +r(m+1) +· · ·+r(n) . Then z is decomposed consistently. Proof. For j < m, and noting that Cm−1 (x) is m − 1 monotone and z = (I − Cn )Cm−1 (x), Cj z = Cj (I − Cn )Cm−1 (x) = (Cj − Cn )Cm−1 (x),

by Theorem 6.18.

= (I − Cn )Cj Cm−1 (x) = (I − Cn )Cm−1 (x) = z. The sequence z is therefore passed unaltered through the ﬁrst m − 1 separators and means that Dj (z) = 0, for j < m. For m ≤ j ≤ n; Cj (z) = (I − Cn )Cj Cm−1 x = (I − Cn )Cj (x) and Dj (z) = (Cj−1 − Cj )z = (I − Cn )Cj−1 x − (I − Cn )Cj x = Cj−1 x − Cj x − Cn Cj−1 x + Cn Cj x = Cn−j x − Cj x = Dj (x). Finally Cn (z) = Cn (I − Cn )Cm−1 x = 0, by the co-idempotence of Cn , or Cn (I − Cn ) = 0. Thus the components of z come out fully consistently. Clearly the DPT is consistent as a “high pass” smoother. (The case where m = 1) and as a “band pass” smoother. (General m < n). A further result, demonstrating further linear behavior under speciﬁc conditions is required to prove additional consistency. This result comes from Theorem 6.19 by the following remarkably general theorem. Theorem 8.4. Let x ∈ M0 and DP T (x) = [D1 x, D2 x, . . . , DN x, D0 x], with D0 x = n αi Di x is decomposed consistently, for n ≤ N . CN x. If αi ≥ 0, then z = i=1

Proof. Let z =

n

αi Di x. Noting that Di = (1 − Ci )Ci−1 are all ftp we see that

i=1 n i=m

αi Di x =

n

αi (I − Ci )Ci−1

x = Am Cm−1 x ∈ Mm−1 ,

i=m

since Ci−1 = Ci−1 Cm for i ≤ m, where n αi (I − Ci )Ci−1 Am = i=m

is ntp, since it is a convex combination of ftp operators.

96

8. The Discrete Pulse Transform

(2) (3) z=r +r

x

(I–F )x = r(1) 1

F x 1 F1z

(I–F1)z = 0

(2) (I–F2)F1x = r

F2x F2z

(I–F2)F1z = r(2)

F x 3

(I–F3)F2x = r(3)

F3z

(I–F3)F2z = r(3)

Figure 8.2. Theorem 8.3 veriﬁed by a random sequence.

Assume that ⎛ Cj−1 z = ⎝

n

⎞ αi Di ⎠ Cj−1 x

for j < n.

i=j

(Clearly it is true for j = 1). ⎛ Cj z = Lj Uj (αj Dj + ⎝

n

⎞ αi Di ⎠ Cj−1 x

i=j+1

= Lj (αj Uj Dj + Aj+1 Cj )Cj−1 x,

by Theorem 6.19

= Lj (αj Uj (I − Lj Uj ) + Aj+1 Lj Uj )Cj−1 x = Lj (αj (I − Lj )Uj + Aj+1 Lj Uj )Cj−1 x, = Lj (αj (I − Lj ) + Aj+1 Lj )Uj Cj−1 x = (αi Lj (I − Lj ) + Aj+1 Lj )Uj Cj−1 x = 0 + Aj+1 Cj x.

by Theorem 6.15

8. The Discrete Pulse Transform

97

(1) (3) z=r +r

x

(I F )x = r(1) 1

F x 1

(I F 1)z = r(1)

F1z

(I F 2)F1x = r(2)

F x 2

(I F 2)F1z = 0

F2z

(I F )F x = r(3) 3 2

F x 3 F3z

(I F 3)F2z = r(3)

Figure 8.3. Omitting more general selected resolution levels.

Thus ⎛ Dj = (Cj−1 − Cj )z = αj Dj x

and Cj z = ⎝

n

⎞ αi Di ⎠ Cj x.

i=j+1

A standard induction argument completes the proof of the theorem. Corollary. Any sequence

N i=0

αi Di x, with αi ≥ 0 is decomposed consistently.

The basic alternative DPT using the operators Un Ln has a similar consistency and this can be proved in the same way as above. What is important to note is that they generally give diﬀerent resolution components. When the sequence is strongly correlated, the two decompositions yield essentially similar results, but the addition of additive random noise yields some separation, proportional to the amplitude of this noise. What is important is that the two DPT processes yield ﬁrst resolution levels that include between them the resolution levels that would appear out of an equivalent Median Decomposition, since the median operators involved are in the LU LU -interval. However at later resolution levels this need

98

8. The Discrete Pulse Transform

not be, and some order holds only on average, since the sequence that is passed on from the mth separator to the m + 1th separator is between those of the other two. The theorems above therefore not only justify the corresponding DPT procedures, but also, partly, the related Median decomposition, which has until now been heuristic in the motivation. It has, in fact almost none of the strong consistency involved in the LU LU -cases, but in practice is “almost everywhere” consistent. Concensus is that it ”works well”. Clearly it can be expected to work well if both the LU LU -decompositions work well, and give similar decompositions. In the case of the two LU LU -decompositions there is a further consistency, that is relatively easy to prove, and is easily motivated. In the case of a discrete Fourier Transform each “resolution level” (frequency level) is a two-dimensional space of sequences. + Since the mapping from a sequence x onto α− n sin(ni) + αn cos(ni) is simply a transformation to a diﬀerent basis of which two vectors are sin(ni) and cos(ni), it is clear that the sequence of arbitrary linear combinations of the sin and cos components is decomposed consistently, so that each component comes out with its corresponding amplitude. The pulse decomposition decomposes a sequence x onto diﬀerent resolution levels. In each of these, Dn (x) consists of positive and negative pulses, namely Dn− (x) = (I − Un )Cn−1 (x), which is negative where it is not zero, and Dn+ (x) = (I − Ln )Un Cn−1 (x), which is positive when it is not zero. This is because I − Un ≤ 0 and (I − Ln ) ≥ 0. Clearly (I−Un )Cn−1 (x)+(I−Ln )Un Cn−1 (x) = (I−Ln Un )Cn−1 (x) = Dn (x), and the crucial interest is whether a further consistency is present, in that, for all + α− j , αj ≥ 0, n − + + z= α− i Di (x) + αi Di (x) i=1

− + + + is decomposed consistently to yield Dn− (z) = α− n Dn (x) and Dn (z) = αn Dn (x). This should hold for each n > 0. The next theorem proves this, and has further useful consequences, but needs the following lemma. Deﬁning, as usual, Bxi if Bxi > 0 Bxi if Bxi < 0 (B+ x)i = and (B− x)i = 0 otherwise, 0 otherwise.

Lemma.

((I − Ln Un )Cn−1 )− = (I − Un )Cn−1 = Dn− , ((I − Ln Un )Cn−1 )+ = (I − Ln )Un Cn−1 = Dn+ .

Proof. (I −Un )Cn−1 ≤ 0, (the zero operator) since I −Un ≤ 0 and (I −Ln )Un Cn−1 ≥ 0, since I − Ln ≥ 0. What is needed to complete the proof is to show that the set of indexes i, where Dn− xi < 0 and the set of indexes where Dn+ xi > 0 are disjoint.

8. The Discrete Pulse Transform

99

Let x be a sequence. Then Cn−1 x = z is in Mn−1 . Assume that Dn− zi < 0, and Dn− zi−1 = zi−1 . Then (Un z)i = min{max{zi−n , . . . , zi }, . . . , max{zi , . . . , zi+n }} > zi . To the left of zi and to the right of zi there are z , zr > zi with |r − | < n + 1. Since z ∈ Mn−1 we have a constant section zi = zi+1 = · · · = zi+n−1 , with Un zi = min{zi−1 , zi+n } > zi . This is because z > zi are both in {zi−n , . . . , zi } which is therefore monotone decreasing as z ∈ Mn−1 . Similarly {zi , . . . , zr , . . . , zi+n } is monotone increasing, and the intersection must be constant. But if j ∈ [i, i + n], then (Ln Un z)j = max{min{Un zi−n , . . . , Un zi }, . . . , min{Un zi , . . . , Un zi+n }} has at least one of the minima equal to Un zi = Un zj , therefore Ln Un zj = Un z, and Dn+ zj = 0. Thus Dn+ z is zero where Dn− z is negative. Clearly therefore Dn+ z can only be positive where Dn− z is zero. The other part of the theorem is proved similarly, or by duality. Since (I − Un )Cn−1 and (I − Ln )Un Cn−1 are ftp, the following theorem is easy to prove in analogy with the previous one. Theorem 8.5. Let x ∈ M0 and DP T (x) = [D1 x, D2 x, . . . , DN x, D0 x], with D0 x = CN (x). N + + − (α− z= i Dn (x) + αi Dn (x)) i=1

is decomposed consistently. − Proof. The proof is done as in Theorem 7, with αi Di x replaced by α− i Di x + + + αi Di x. − + + Cj z = Lj Uj (α− j Dj + αj Dj +

N i=j+1

− + + (α− i Di + αi Di ))Cj−1 (x).

Applying Uj ﬁrst we get ⎛

⎛

− + ⎝α+ Uj z = Uj ⎝α− j Dj + j Dj +

=

Uj (α− j (I

⎞⎞

N i=j+1

+

− + ⎠⎠ Cj−1 (x) α− i Di + αi Di

− Uj ) + (Aj+1 Lj )Uj )Cj−1 (x),

with Aj+1 ftp. This implies by Theorem 4, (since Cj−1 (x) ∈ Mj−1 ), that Uj z = (α− j Uj (I − Uj ) + Aj+1 Lj Uj )Cj−1 (x) = Aj+1 Lj Uj Cj−1 (x), since Uj (I − Uj ) = 0.

100

8. The Discrete Pulse Transform

Applying Lj to Uj z yields in a similar fashion that Cj z =

n i=j+1

− + + (α− i Di + αi Di )Cj−1 (x).

The resolution components that are removed at resolution layer j are therefore − − − + + α− j Dj Cj−1 (x) = αj Dj (x) and αj Dj (x) and the induction argument used in Theorem 6 completes the proof. In a wavelet decomposition, which is just a basis transformation, it is clear that if each individual wavelet in a decomposition of a sequence x is multiplied by a constant and the sum is decomposed, then it decomposes consistently in the sense that the exact multiple of each wavelet appears in the resolution level it came from. Since the pulse decomposition with LU LU -operators is nonlinear there is no basis transformation, yet the remarkable result of the previous theorem provokes an almost unthinkable idea: That if each individual pulse in a resolution level is multiplied by its own non-negative constant, the sum could still be decomposed consistently! If this idea is correct the implication for image processing would be very important. For example, if a square white slab were barely visible against a background that is nearly as white, a pulse decomposition on the rows (or columns) of a matrix of luminosity values obtained from a photo can locate the slab by its expected ratio of width to height. Highlighting this particular pulse would make the slab clearly visible, without distorting any of the surroundings. Quite considerable testing has conﬁrmed the idea, but an elegant or illuminating proof seems elusive. Proofs in the case of only a few resolution levels exist, but do not provide suﬃcient insight to complete even with laborious calculations. Promising ideas are being pursued, which have the merit of classifying many diﬀerent useful ideas that are not addressed here. We therefore state the idea as a conjecture: The Highlighting Conjecture. Let z be a sequence formed by adding all the resolution levels of x, after multiplying each particular pulse at a resolution level by a chosen non-negative number. Then z decomposes consistently. In the following example a random sequence x is generated and (I − U1 )x and (I − L1 )U1 x, in Figure 8.4 below, have each of their pulses ampliﬁed by a positive random number. All the resolution levels are then added together again and the sum is decomposed. Comparing the ﬁrst resolution levels demonstrates the identical modiﬁed sequences. The last few theorems that have led to the establishment of a guaranteed type of consistency in the Discrete Pulse Transform (DPT) are perhaps stronger than the results up to here indicate. Noting that many related operators that are used in various decompositions, for the purpose of fast economical coding and transmission of data, often have suﬃciently many of the properties like idempotence, co-idempotence, syntoneness, or commuting with the LULU-operators, to permit proof of further levels of consistency of the DPT. Some typical examples

8. The Discrete Pulse Transform

101

Figure 8.4. The sequences (I − U1 )x and (I − L1 )U1 x and their modiﬁed version.

of such operators can be given with some of the relevant properties. Firstly, the following lemma is clear and trivial to prove. Lemma. If an operator P on M0 has a support of 1, (that is (P x)i = f (xi ), where f (x) is a non-negative, non-decreasing, real-valued function, then P is syntone; and neighbor trend preserving (ntp). The following operators are examples that might occur in practice, and can be demonstrated to have simple properties. Deﬁnition. The Rounding Operator R. For x ∈ M0 and index i; (Rx)i ≡ h sgn(xi )int

|xi | 1 + h 2

.

Properties. (a) R is idempotent, syntone, ntp and R(−x) = −R(x). (b) R commutes with ∨ and ∧: R∨ = ∨R and R∧ = ∧R. (c) R commutes with all compositions of Ln and Un . Proof. (a)

R(Rx)i = h sgn(Rxi )int

|Rxi | 1 + h 2

|hsgn(xi )int( |xhi | + 12 )| 1 + = h sgn(xi )int h 2 1 |xi | 1 + + = Rxi . = h sgn(xi )int int h 2 2

102

8. The Discrete Pulse Transform

R is syntone by the previous lemma and

| − xi | 1 + h 2 |xi | 1 + = −h sgn(xi )int = −(Rx)i . h 2

(R(−x))i = h sgn(−xi )int

(b) Consider (R∨)xi = h sgn(∨xi )int

| ∨ xi | 1 + h 2

.

If xi+1 ≥ xi ≥ 0 (similarly if xi ≥ xi+1 ≥ 0) then; (R∨)xi = h sgn(xi+1 )int

|xi+1 | 1 + h 2

= max{Rxi+1 , Rxi } = (∨R)xi . If xi+1 ≥ 0 > xi (similarly if xi ≥ 0 > xi+1 ) then; |xi+1 | 1 (R∨)xi = h sgn(xi+1 )int + h 2 |xi+1 | 1 |xi | 1 + + = max h sgn(xi+1 )int , hsgn(xi )int h 2 h 2 = (∨R)xi . If 0 ≥ xi+1 > xi (similarly if 0 ≥ xi > xi+1 ) then;

| ∨ xi | 1 |xi+1 | 1 + + (R∨)xi = h sgn(∨xi )int = −hint h 2 h 2 |xi+1 | 1 |xi | 1 = max −h int + + , −hint h 2 h 2 = (∨R)xi , since |xi | > |xi+1 |. In all cases therefore (R∨)xi = (∨R)xi . (c) Clearly R∨n = ∨n R by induction, and similarly R∧n = ∧n R. Similarly RLn = R(∨n ∧n ) = (T ∨n )∧n = (∨n T )∧n = ∨n (R∧n ) ∨n (∧n R) = (∨n ∧n )R = Ln R. For all compositions of Ln and Un a similar argument proves commutativity with R.

8. The Discrete Pulse Transform

103

Note: R is “nearly” co-idempotent. If we were to replace the “upward rounding” by “downward rounding” to the nearest multiple of h and noting that then −h h 2 < xi − Rxi < 2 , it is clear that |xi − Rxi | 1 1 |xi − Rxi | < so that int + = 0. h 2 h 2 Hence R(I − R) = 0. (This can be achieved by subtracting the smallest machine number eps from the argument of the integer function, or a similar idea.) A similar operator is a truncating operator of the type |xi | (T x)i = h sgn(xi )int , h where we expect similar properties, with the addition of co-idempotence. Deﬁnition. A Thresholding Operator T . ∀x ∈ M0 , xi if |xi | ≤ t, (Tt x)i = (T x)i ≡ . sgn(xi )t if |xi | ≥ t Properties. (a) T is Idempotent and T (−x) = −T x. (b) T is fully trend preserving (ftp). (c) T commutes with ∨ and ∧: T ∨ = ∨T and T ∧ = ∧T . (d) T commutes with Ln , Un , Ln Un , Un Ln , Cn and Fn . Proof. (a) Trivial, or by the previous lemma. (xi ≥ xi+1 is handled similarly). Then (T x)i+1 ≥ (T x)i (b) Let xi+1 ≥ xi . trivially, so that T is ntp. Consider (I − T )xi = xi − (T x)i and (I − T )xi+1 = xi+1 − (T x)i+1 . If xi , xi+1 ∈ [−t, t], then both terms are zero. If xi ∈ [−t, t] but xi+1 > t, the ﬁrst term is zero and the second is positive. If xi+1 ∈ [−t, t] but xi < −t, then the ﬁrst term is negative and the second zero. If xi , xi+1 ∈ [−t, t], then xi+1 > t > −t > xi . Then the ﬁrst is negative and the second positive. In each case therefore (I − T )xi+1 ≥ (I − T )xi , and I − T is ntp. (c) Consider (∨T )xi = max{T xi , T xi+1 }. If xi+1 , xi ∈ [−t, t], then (∨T )xi = ∨xi = max{xi , xi+1 } = (T ∨)xi . If xi+1 ∈ [−t, t] and (a) xi > t, then (∨T )xi = max{t, xi+1 } = t = (T ∨)xi , (b) xi < −t, then (∨T )xi = max{xi+1 , −t} = xi+1 = (T ∨)xi . (Similarly if xi ∈ [−t, t] and xi+1 ∈ [−t, t].) If xi+1 , xi ∈ [−t, t] and (a) xi+1 , xi > t; then (∨T )xi = max{t, t} = (T ∨)xi , (b) xi+1 , x + i < −t, then (∨T )xi = max{−t, t} = (T ∨)xi , (c) xi+1 > t > −t > xi , then (∨T )xi = max{−t, t} = t = (T ∨)xi , (d) xi > t > −t > xi+1 , then (∨T )xi = max{t, −t} = t = (T ∨)xi . Therefore T commutes with ∨, and a similar argument proves that T ∧ = ∧T .

104

8. The Discrete Pulse Transform

(d) Clearly T ∨n = ∨n T by induction and similarly T ∧n = ∧n T . Similarly T Ln = T (∨n ∧n ) = (T ∨n )∧n = (∨n T )∧n = ∨n (T ∧n ) = ∨n (∧n T ) = (∨n ∧n )T = Ln T. For all other compositions of Ln and Un a similar argument proves commutativity. Note: T is not co-idempotent, as a simple example of a constant sequence xi = c = 3t shows. T (I − T )x = T (c − T c) = T (3t − t) = x = 0. A similar operator is another thresholding operator; (⊥ x)i =

xi 0

if |xi | > t, if |xi | < t.

Similar properties to the above case can be expected, as well as co-idempotence. With the properties of the above operators it seems that the eﬀect on the DDT can be investigated with conﬁdence, when this is required. Clearly if the conjecture following Theorem 8 is true we expect consistent decomposition after modiﬁcation of the resolution levels by the above operators. From the deﬁnition of LU LU -operators it appears to have a considerable computational complexity, but for the purpose of successive decompositions by Ln Un (or Un Ln ) there are considerable computational simpliﬁcations. The example with Ln Un can be chosen to illustrate. Given a sequence x = {xi : i = 1, 2, . . . , N } in M0 the ﬁrst decomposition is with U1 , followed by L1 to yield a sequence L1 U1 x in M1 and a residual R1 x = x − L1 U1 x. At each stage, the sequence that is decomposed by Ln Un is in Mn−1 , which permits a simpliﬁed calculation. Consider (Un x)i = min{max{xi−n , . . . , xi }, . . . , max{xi , . . . , xi+n }. Near endpoints, where left (or right-) neighbors are not deﬁned, it is simple to omit them in the maximum (or minimum) calculations. This is clearly equivalent to appending suﬃcient values equal to x1 to the left (and equal to xN at the right). Given that n ≥ 1 and x is (n − 1)-monotone, the ﬁrst (and the last) values of Un x can be copied from x. This is because the following arguments hold. Since {xk , . . . , xk+n+1 } is monotone; (i) Assume x1 ≤ x2 ≤ · · · ≤ xn+1 , then for j = 1, . . . , n, Un xj = min{ max{x1 , . . . , xj }, max{x2 , . . . , xj , xj+1 }, . . . . . . , max{xj , . . . , xj } = xj , since xj = max{x1 , . . . , xj } and xj is not larger than all the others since they are upper bounds.

8. The Discrete Pulse Transform

105

(ii) Assume x1 ≥ x2 ≥ · · · ≥ xn+1 . Noting that xn ≥ xn+1 implies that xn ≥ xn+1 ≥ · · · ≥ xn+n , it is clear that max{xj , xj+1 , . . . , xj+n } = xj , so that Un xj = min{max{1, . . . , xj }, . . . , max{xj , xj+1 , . . . , xj+n } = xj , since the last maximum is xj and the previous cannot be less. Thus Un xj = xj for j ∈ {1, . . . , n}. A similar argument also holds for j ∈ {N − n + 1, . . . , N } so that 2n values of Un xj are already calculated. The following sequences are calculated successively. ei = max{xi , xi+1 , . . . , xi+n } = max{xi , xi+n }, ti = min{ei−n , . . . , ei } = min{ei−n , ei },

i = 2 to N − n − 1, for i = n + 1 to N − n − 1.

This requires a total of 2N − 3n − 4 comparisons and yields Un x. This is followed by a similar process for Ln Un x. For a full decomposition up to and including level n therefore, a total of n(4N − 3n − 7) =

N

(4N − 6i − 4)

i=1

comparisons are needed. To compute the residual x − Ln Un x at each level, only values diﬀerent from 0 need a subtraction. (The minority of points, deﬁnitely less N , but generally much less.) There is no point in letting n be more than n+1 than N − 1, since then x − Ln Un x = 0 and Ln Un x is a constant. Generally therefore the total number of comparisons is less than N 2 − 11N + 8. There are indications that a further saving can be achieved, as the above estimates are not sharp. Certainly it is clear that a great amount of parallelization is possible, depending upon the equipment available. Some simple examples demonstrate very economic coding if some structures are present. Such economizing is likely to occur often when quantizing (and/or thresholding) is introduced for the amplitude, as is the case with measurements in ﬁxed bit format It should be noted that in such a case none but a ﬁnite number of possible numbers available are suﬃcient for all decompositions, since the only operations performed are selections and diﬀerences. This is an important economization, compared to other values that are introduced by wavelet decompositions, where averages and diﬀerences are required. There are a variety of alternative computational procedures that can be researched for further economization, depending on the number of resolution levels required. This vast ﬁeld of algorithmic analysis is still open.

106

8. The Discrete Pulse Transform

Example 1. Let N = 2k and x = {xi = |i − k|}. At level 1. At level 3.

(x − L1 U1 x)i = δik (The Kronecker delta) (x − L3 U3 )L2 U2 L1 U1 xi = δik−1

At level 5. .. .

= δkk−2 .. .

At level 2m + 1.

= δik−m up to 2N − 1.

Thus there are precisely N pulses. Each is of width 2m−1 and constant amplitude, and coding requires N starting indexes and N amplitudes. Example 2. Let N = 2k and x = {xi = (−1)i }. Then, at level 1, there are pulses at each even integer, except 1, and at each odd integer, after which there is only one pulse of width N − 1 and a remaining value of −1 (or a pulse of width N and amplitude −1). Thus a total of k indexes and k + 2 amplitudes. Example 3. When random noise from a cubic-B-spline distribution is added to three uniform level pulses of duration 1, 2 and 3, the visually signiﬁcant counts are 23, 6 and 5 out of a sequence length of 50. Quantizing and thresholding appropriately, and reconstituting from only three signiﬁcant pulses would yield an image that is substantially the signiﬁcant features from three indexes and amplitudes. The thresholds can be chosen a priori to estimates corresponding to amplitudes and distribution of the random noise. Impulsive noise of a prescribed width can be removed by omitting higher levels altogether. One important aspect of the decomposition of an arbitrary sequence, the question of the total number of pulses in the resolution levels, can be answered by the following simple argument: Taking the case of a ﬁnite sequence of N points and appending zeroes (or assuming periodicity with xN +i − xi ) on either side, it becomes clear that a pulse appears in a resolution level when one of two neighboring values of the sequence smoothed by some Ln Un (or Un Ln ) is replaced by the other. A pulse therefore has the amplitude of such a diﬀerence. The total variation contains not more than N + 1 such diﬀerences and each pulse removes at least one such term (Theorem 7.1). The last pulse to be removed takes with it two such terms. Therefore no more than N pulses appear in total and the original sequence can be fully reconstructed from these. This is veriﬁed by the chosen examples above, and the value of the constructive proof of Theorem 7.1 in substantiating this argument is apparent. Apart from the overheads of keeping track of the index at which a particular pulse is located, the coding economy of storing/transmitting the resolution levels is no more than that of the original sequence. Since a sequence is generally contaminated by some random noise from a distribution, we can expect from experience that this appears mainly in the higher resolution levels, where it could be removed with minimal change to the essential features remaining.

8. The Discrete Pulse Transform

107

How the identically, independently distributed (i.i.d.-) random noise will be distributed by the PPT seems the key question here. The operators are not linear and the addition of such noise to a signal will not yield a decomposition that is predictable if the decompositions of signal and noises are understood separately. Nevertheless, practical experience suggests that the behavior can be expected to be fairly well predicted by understanding the case of pure noise. The issue is being researched and results are promising. At this stage it is however instructive to introduce the basic initial results and ideas in the context of some fair comparisons of the LU LU -operators with linear ﬁlters of comparable type.

9. Fair Comparison with Linear Smoothers Seit ich des Suchens M¨ ude ward Erlernte ich das Finden. Seit mir ein Wind hielt Widerpart Segl’ ich mit allen Winden. Nietzsche

A theory of linear (digital-) ﬁlters is well known and established. A good introduction is provided by Hamming (11). The underlying ideas can however be brieﬂy, and casually, introduced, for the purpose of comparison with other smoothers, speciﬁcally the LU LU -operators and popular selectors and compositions. As in the case of smoothers like the (running-) median M n, a window Wn with width 2n + 1 considers the subset {xi−n , . . . , xi , . . . , xi+n } of a sequence x, for successive i. This window can be considered to “move over” the sequence x, or the sequence can be considered to “pass through” the window. The elements in the window are used to compute an output value (Fn x)i =

i+n

αj xj =

j=i−n

n

αk xi−k ,

k=−n

with αj constants. The simplest way of motivating this type of smoother is to consider it as a local least squares projection of the values xi−n , . . . , xi+n onto a best constant value, and using this value to replace xi . This corresponds to the case of the median smoother which does the same, but projects in the 1 -norm, so as to be more “robust”. This least squares local projection yields the value (Fn x)i =

n k=−n

1 xi+k . 2n + 1

Clearly the operator Fn maps sequences in X onto sequences in X and meets all the requirements of a smoother. The important requirements are easy to verify; Fn is axis independent and scale independent. If also preserves sequences that are of the form x = xi = αi + β, where α and β are constants.

110

9. Fair Comparison with Linear Smoothers

A similar argument, using local projections to replace xi by the value at i of the least squares quadratic polynomial to the values {xi−n , . . . , xi+n }, yields, for example with n = 2 the output, yi = (F2 x)i =

1 [−3xi−2 + 12xi−1 + 17xi + 12xi+1 − 3xi+2 ]. 35

It becomes clear that, in each case, there is a weighted average of the values in the window that replaces xi with the smoothed value yi . For the second example it is easy to verify that it preserves sequences that are sampled quadratic polynomials, which results in a better “shape preservation” of a sequence. The penalty is that the weighted average is not a convex combina3 3 12 17 12 add up to 1 tion anymore, since, although the coeﬃcients − , , , , − 13 35 35 35 35 (preserving constant sequences), the ﬁrst and last are negative. Smoothers of the above type are easily seen to be linear and the set of all such smoothers form a vector subspace of the set of smoothers from X to X. Such a smoother can be considered to yield an output that is, for each x ∈ X, a convolute of x with a speciﬁc constant sequence α. Deﬁnition. A sequence y is the (discrete) convolute of sequences x and α iﬀ. yi =

∞

xj αi−j .

j=−∞

Smoothers (ﬁlters) of the following types appear in literature under the names “FIR ﬁlters” (“ﬁnite response ﬁlters”), “transversal ﬁlters”, “tapped delay line ﬁlters”, “moving Average ﬁlters” or just “non-recursive (digital-) ﬁlters”. They shall be called non-recursive ﬁlters here, or just ﬁlters, where there is no misunderstanding likely. It is often useful to use values that are already smoothed in the window, so that the output is of the form yi =

∞ i=−∞

αj xi−j +

∞

βj yi−j ,

with α, β constant sequences.

j=−∞

This is an implicit deﬁnition of y, so that evaluation is awkward, and generally the version that is recursively calculable is used, where βj is chosen 0 for j non-positive. Such smoothers are called “inﬁnite impulse response ﬁlters”, “ladder ﬁlters”, “lattice ﬁlters”, “autoregressive moving average ﬁlters”, “ARIMA-ﬁlters”. Here they shall be called recursive ﬁlters. Well-known versions are: a) yi = yi−1 + 12 (xi + xi−1 ). (The Trapezium Rule for integration.) b) yi = αxi +(1−α)yi−1 , α ∈ (0, 1). (The exponential smoothing prediction.) c) yi = α(xi − xi−1 ) + (1 − α)yi−1 , α ∈ (0, 1). (The “trend indicator”.)

9. Fair Comparison with Linear Smoothers

111

Noting that sequences formed by sampling of polynomials are not generally absolutely summable, the above ideas of local projection could be generalized to rather yield local projections (or global projections onto sampled spline functions). This idea is inherent in the currently very active research ﬁeld of wavelet analysis, speciﬁcally spline wavelet analysis, where a sequence is mapped onto a spline function. The spline function undergoes a wavelet decomposition, and smoothing simply is the removal of “high-frequency” wavelets. The idea for semi-orthogonal spline wavelets is an illuminating version of general spline smoothing as done by Schoenberg (28). Wavelet smoothers are not translation invariant and therefore, as previously argued, have additional complications to attend to. For the purpose of comparison the usual digital ﬁlters will however provisionally suﬃce. They are traditionally popular because of the electronic analogy in RC-networks, and the smoothing (ﬁltering) has therefore developed in conjunction with electronic transmission of signals where sinusoidal electric vibrations are used as carriers with other (locally-) periodic information superimposed. The sequences x = cos(vi) and y = sin(vi), for various frequencies v, are thus naturally prevalent. Since these are all eigensequences of linear mappings from X to X, the natural ideas used for analysis are Fourier Analysis and projections onto subspaces of sequences such as x and y. The idea of a frequency is central, and will be measured in cycles per second (Herz) or in radians per second (angular frequency). Sequences of constant frequency are periodic, and the fundamental period is the shortest interval within which the sequence repeats itself. Deﬁnition. Let TN be the subspace of (periodic) functions of the form f (x) = α0 +

N

αi cos(ix) + βi sin(ix)

i=1

and FN be the subspace of (periodic) sequences that are samplings of these at a given sampling rate. With the above deﬁnition of the signals (sequences in FN ) it is soon clear that the requirement that a smoother preserves sequences in FN is diﬃcult to meet as there is a fundamental ambiguity related to the concept of “aliasing”. Aliasing is observed in typical “Western” ﬁlms, where the spokes of a coach which are observed to undergo a peculiar behavior when the coach accelerating. Stroboscopic light ﬂashes of a sinusoidal movement produces similar eﬀects. Aliasing is simply due to the fact that frequencies higher than sampling frequency (“Nyquist-frequency”) are undistinguishable from a lower frequency. Consider the sequence x(i) = cos(2π(n + α)i + β), where n is an integer, α a positive fraction and β a ﬁxed (phase-) angle. xi = x(i) = cos(2π(n + α)i + β) = cos(2παi + β).

112

9. Fair Comparison with Linear Smoothers

It is clear that every cosine sequence is thus indistinguishable from a particular cosine frequency with angular frequency between 0 and 12 . High frequencies are thus aliased under a low frequency, and only sequences with frequencies between 0 and 12 can be preserved by a ﬁlter using the sampled values x = xi . When a general sequence that is periodic in a section is ﬁltered, the result has a fundamental ambiguity as to the original frequency sampled. This is an analogy of the fundamental ambiguity for impulsive noise, which yields the LU LU -interval. Noting that a ﬁlter yields a unique sequence in some subspace FN , it is clear that it does not warn us of the ambiguity present in the smoothed sequence. Even when a slowly increasing frequency is sampled, the approach of the ﬁrst multiple of the Nyquist frequency is diﬃcult to detect, whereas the separation of the two smoothed samplings LU x and U Lx explicitly warns of ambiguity. It is also noteworthy to remark that aliasing is not restricted to trigonometric polynomials but to all polynomials, and other classes of functions as well. This problem of aliasing should be kept in mind during all the subsequent analysis of ﬁlters and smoothers in general, and viewed as a fundamental ambiguity as to the signal content in a sequence. For linear ﬁlters, as with linear operators in general, the general idea for analysis is eigenanalysis. Consider a (linear-) ﬁlter Ln mapping x ∈ X onto y ∈ X such that yi = (Lnx)i =

n

δj xi−j .

j=−n

Let

xi = sin(αi + θ). yi = (Lx)i =

n

Then elementary trigonometric identities yield δj sin(α(i + j) + θ) =

j=−n

with

of

n

δj (sin(αi + θ + αj )

j=−n

= A sin(αi + θ) + B cos(αi + θ) = A2 + B 2 sin(αi + θ + β), n n B A= δj cos αj, B = δj sin αj en β = arctan( ). A j=−n j=−n

√ The sequence thus undergoes a phase change of β and an amplitude change A2 + B 2 . When a ﬁlter is symmetric (δj = δ−j ) it is clear that B=

n

(δj − δ−j ) sin αj = 0,

j=−n

so that the phase change is zero, and (Lx)i = A sin(αi + θ). Thus the sequence xi is an eigensequence with eigenvalue A.

9. Fair Comparison with Linear Smoothers n

When

113

δj = 1, which is required for the preservation of constant se-

j=−n

quences by smoothers, n

A = δ0 +

(δj + δ−j ) cos αj = δ0 +

j=1 n

=

(δj + δ−j )(1 − 2 sin2 (

j=1

δj −

j=−n

n

αj )) 2

n

α (δj + δ−j )2 sin2 ( j). 2 j=1

The ﬁrst term is 1 and the second is 0 at α = 0, and grows as α grows δj ≥ 0. Thus the low frequencies are almost unchanged but higher frequencies are decreased in amplitude by the ﬁlter. For the speciﬁc case of the ﬁlter with coeﬃcients [ 15 , 15 , 15 , 15 , 15 ], 2 1 sin 52 α 1 1 cos αj = [1 + 2 cos α + 2 cos 2α] = A= . 5 j=−2 5 5 sin α2 Similarly, for a moving average ﬁlter with δj = that

1 for j ∈ [−n, n] it follows 2n + 1

n

A(α) =

2 cos(αj) sin(nα + α2 ) 1 + = , 2n + 1 j=1 2n + 1 (2n + 1) sin α2

since n

n

2 cos αj sin

j=1

α α α α α = [sin(αj + ) − sin(αj − )] = sin(nα + ) − sin . 2 2 2 2 2 j=1

Since the ﬁlter is linear, any periodic function will pass through the ﬁlter as if its constituent frequencies pass through separately and recombine, and since xi =

n

αk sin(wk i) + βk cos(wk i) =

k=1

n α2k + βk2 sin(wk i + θk ), k=1

it follows that yi = (Lx)i = =

n k=1

n k=1

A(wk ) α2k + p2k sin(wk i + θk )

a(wk )[αk sin(wk i) + βk cos(wk i)].

114

9. Fair Comparison with Linear Smoothers

If cos x + i sin x = eix

cos x − i sin x = e−ix , the trigonometric iden-

and

tities sin(x + y) = sin x cos y + cos x sin y are replaced by

and

cos(x + y) = cos x cos y − sin x sin y

eix eiy = ei(x+y) .

It becomes apparent that the complex power sequences are eigensequences of the linear translation operator (T u)j = uj+1 ,

since eiw(t+h) = eiwh eiwt = λ(w)eiwt .

This idea also is applicable in the following cases. m i) The non-recursive ﬁlters of the form yn = cj xn−j yield, with substituj=−m

tion of xj = eiwj , the output m

yn = eiwn

cj e−iwj = λ(w)eiwn ,

j=−m

with λ(w) =

m

cj e−iwj .

j=−m

ii) The diﬀerence operator f (t) = f (t + h) − f (t), since eiwt = eiw(t+h) − eiwt = eiwt (eiwh − 1). iii) The summation operator f (t), since n k=1

eiw(t+kh) = eiwt

n k=1

iwnh e −1 eiwhk = eiwt eiwt = λ(w)eiwt . eiwh − 1

The transfer function of a recursive linear ﬁlter of the form A(α) = δ0 +

n

(δj + δ−j ) cos(αj)

j=1

can be considered a cosine transform of the sequence δj . The output yi =

n j=−n

δj xi−j =

∞

δj xi−j

−∞

is considered as a convolute of xi and δj , with δj = 0 for |j| > n.

9. Fair Comparison with Linear Smoothers

115

For a smoother it is required that ∞

δj = 1.

j=−∞

The transfer function can now be considered as the Fourier transform δ of the sequence δ. The Fourier transform of the sequence x multiplied by δ is the transform of the convolute of the sequence x with δ, and yields the required smoothed x by inverting. Filtering is thus simply multiplication in the “frequency domain”. This is in analogy to the case of functions, as is used in the development of B-splines (8) and in the development of Wavelet Theory. For the design of smoothers, this permits the idea of designing ﬁlters as follows. Since A(α) = δ0 + 2δ1 cos α + 2δ2 cos 2α + · · · + 2δn cos nα for a symmetric ﬁlter, the ideal transfer can be approximated with a cosine polynomial. 1 for |α| < β, Example. Let T (α) = This is an ideal ﬁlter as it is a projection. 0 otherwise. A Fourier expansion gives. T (x) = γ0 +

∞

γi cos i(x),

i=0

with γi =

1 π

π

0

T (t) cos(it)dt =

sin iβ , πi

and γ0 =

β . π

γi , but the disadvantage 2 is an inﬁnite window. The coeﬃcients also decay with tempo 0( 1i ), because of the discontinuity of T at ±β. Other approximations, say with Lanczos factors, would still have slow convergence, since the fundamental Heisenberg inequality (8) assures a wide window if the transform is to be narrow. Recursive ﬁlters could improve things somewhat, but not suﬃciently, since the support of a ﬁlter needs to be small if good shape preservation of signals is desired. It turns out that there is an alternative set of eigensequences that are common to linear smoothers and some selectors and compositions of these, and thus permits comparison. The sequences that are formed by sampling exponentials present themselves naturally. Considering the sequences α = xi = αi , it is clear that, when α > 0, they are monotone. By the previous theory LU a = LU a = a, so that a is an eigensequence of the LU LU -operators, for each n, and also of the medians and similar smoothers inside the LU LU -interval. A linear ﬁlter F , with coeﬃcient sequence δ, will have an output given by The ﬁlter coeﬃcients are now δ0 = γ0 and δi =

(F a)i =

n j=−n

δj ai+j =

n j=−n

δj ai+j = αi

n j=−n

δj αj .

116

9. Fair Comparison with Linear Smoothers

This means that a is an eigensequence of F with the eigenvalue λα =

n

δj αj ,

with λα ≥ 1.

j=−n

This eigenvalue can be 1 with at most 2n values of α, unless α = 1. Thus at best only constant sequences can be preserved in general. Areas in which the signal undergoes fast changes (exponential sections) will therefore undergo distortion. In image processing this becomes clear by the typical “edge-smearing” of linear smoothers. In auditory signals this may not be so noticeable, since the ear detects in the “frequency domain” rather than in the “time-domain”, and amplitude is perhaps not picked up with the same resolution as frequency. Since edges can be arbitrarily steep, linear smoothers can yield an arbitrarily large distortion. For negative values of α, a is still an eigensequence and the eigenvalue is now smaller. For negative α, the medians still have the sequence as an eigensequence, but with a peculiar behavior of the eigenvalue. median[αi−1 , αi , αi+1 } = αi α, if |α| < 1. Thus when α = −1, the eigenvalue −1 results, as was previously established. This is because a is a sequence that is periodic at precisely the Nyquist frequency. In this respect the median applied twice is better, at least not permitting a negative eigenvalue. Linear combinations like 12 [M1 + M12 ] and 14 [M1 + 2M12 + M13 ] exhibit eigenvalues closer to zero, thus conﬁrming practical experience. It is easy to show that a is not an eigensequence of the LU LU -operators when α < 0. They yield outputs that provide an “envelope of uncertainty” as (L1 U1 a)i = max{αi , min{αi+1 , αi−1 }} and (U1 L1 a)i = min{αi , max{αi+1 , αi−1 }}, in a growing or diminishing oscillation in a sequence, thus warning of ambiguity. This corresponds to previous argument and experience. The operator G = 12 [LU + U L] does however have a as eigensequence for α < 0. 1 1+α (Ga)i = (αi + αi+1 ) = αi . 2 2 Therefore λa = Similarly

1+α is the eigenvalue corresponding to a. 2 n 1+α n i (G a)i = α . 2

Thus repeating G results in an output converging to the zero line for −1 ≤ α < 0, and near α = −1 this convergence is rapid.

9. Fair Comparison with Linear Smoothers

117

When comparing smoothers that are index-symmetric the sequences a = αi and ( α1 )i are clearly handled identically and thus this eigenanalysis can be restricted to |α| ≤ 1. The comparison with this eigenanalysis does not reﬂect favorably on the linear smoothers. The linear smoothers demonstrate eigenvalues that approximate 1 and 0 only at α = ±1, whereas medians and some combinations have eigenvalue 1 for all α > 0 and approximating 0 better for α < 0. Having proved that LU and U L have only two eigenvalues 1 and 0, an ideal situation, it is clear that their resistance to identifying an oscillation as noise can be considered a quality of merit rather than a defect. On the ﬁrst measure of comparison therefore, namely eigenanalysis on a common set of eigenvalues, the popular nonlinear smoothers indicate a marked superiority over linear ﬁlters. Lastly, the following argument yields reassurance of no ampliﬁcation of either “signal” or noise for a large class of smoothers. Clearly |Sx|∞ ≤ |x|∞ for a selector, so that |Sx|∞ = |λx|∞ = |λ| |x|∞ ≤ |x|∞ . Also if S and Su are selectors and S ≤ A ≤ Su , then S x ≤ Ax ≤ Su x for each x ∈ X. Since −|x|∞ ≤ (S x)i ≤ (Ax)i ≤ (Su x)i ≤ ||x||∞ , for each sequence x ∈ X and each index i, it follows that ||Ax||∞ ≤ ||x||∞ . Another method of comparison, on a level playing ﬁeld again, is the behavior of random noise. This has received attention in the case of median smoothers (2), but can be considered again, with the inclusion of comparison with LU LU smoothers. We consider ﬁrstly a sequence x = xi of identically independent random numbers from a probability density (pdf) fx (t) with t fx (u)du. Fx (t) = −a

We consider an operator Rk that selects from the windowed subset Wi = {xi−n , . . . , xi+n } the value ranked kth. Let (Rk x)i be this value and the operator be deﬁned accordingly. The following theorem is easy to prove and well known, but included for comparison. Theorem 9.1. The probability that the kth rank is larger than z is given by 2n+1 2n + 1 Fy (z) = P (yi ≤ z) = Fxj (z)(1 − Fx (z))2n+1−j . j j=k

Proof. The probability is given by the sum from k to 2n + 1 of the probabilities that j of the elements of Wi are not larger than z, which is 2n + 1 Fxj (z)(1 − Fx (z))2n+1−j , j

118

9. Fair Comparison with Linear Smoothers

since the elements of Wi can be arranged in 2n+1 ways such that j elements j are not larger than z (probability Fxj (z)) and 2n + 1 − j elements are not smaller (probability (1 − Fx (z))2n+1−j . Theorem 9.2. The probability density function of Rk xi is given by 2n fy (z) = (2n + 1) [Fxk−1 (z)(1 − Fx (z))2n+1−k ]fx (z). k−1 2n+1 2n + 1 Proof. Fy (z) = Fxj (z)(1 − Fx (z))2n+1−j , j j=k

2n+1 2n + 1 d Fy (z) = fx (z) fy (z) = jFxj−1 (z)(1 − Fx (z))2n−(j−1) j dz j=k 2n+1 2n + 1 − Fxj (z)(1 − Fx (z))2n−j (2n − (j − 1)) j j=k

2n+1 2n + 1 = fx (z) Tk + jFxj−1 (z)(1 − Fx (z))2n−(j−1) j j=k+1

2n + 1 i−1 2n−(i−1) (2n + 1 − (i − 1))Fx (z)(1 − Fx (z)) − i−1 i=k+1 2n + 1 kFxk−1 (z)(1 − Fx (z))2n−(k−1) . = fx (z)Tk , where Tk = k 2n+1

Examples. 1. The maximum operator R2n+1 . If k = 2n + 1, then Fy (z) = Fx2n+1 (z) and fy (z) = (2n + 1)Fx2n (z)fx (z). 2. The minimum operator R1 . 2n+1 2n + 1 Fy (z) = Fxj (z)(1 − Fx (z))2n+1−j . j j=1 But, from the binomial theorem, the sum from 0 to 2n + 1 equals (Fx (z) + 1 − Fx (z))2n+1 = 1. Thus Fy (z) = 1 − (1 − Fx (z))2n+1

of 1 − Fy (z) = (1 − Fx (z))2n+1 .

3. With n = 1 the cumulative distribution of the output of M1 is given by Fy (z) =

3

(3j )Fxj (z)(1 − Fx (z))3−j = 3Fx2 (z)(1 − Fx (z)) + Fx3 (z)

j=2

= 3Fx2 (z) − 2Fx3 (z) = Fx2 (z)(3 − 2Fx (z))

9. Fair Comparison with Linear Smoothers

119

and fy (z) = [6Fx (z) − 6Fx2 (z)]fx (z) = 6fx (z)Fx (z)(1 − Fx (z)). (a) With fx (z) = h1 (|z| < h2 ), in [− h2 , h2 ]: Fy (z) =

2 h 6 h h (x + )2 (h − x), fy (z) = 3 (x + )( − xt) h3 2 h 2 2

(b) With fx (z) = Fx (z) =

the uniform distribution with σz2 =

4 h h2 ( 2

− |z|)(|z| <

h ) 2

σx2 =

and

h2 24 ,

h2 12 ,

and σy2 =

gives h2 . 20

we get

2 h 2 h h h h (x+ )2 −(z ∈ [− , 0])+(1− 2 (x− )2 )(z ∈ [0, ])+1·(z > ), h2 2 2 h 2 2 2

and Fy (z) =

4 h4 (z

+ h2 )4 (3 −

(1 −

2 h2 (x

4 h2 (z

+ h2 )2 ),

− h2 )2 )2 (1 +

4 h2 (x

(z ∈ [− h2 , 0])

− h2 )2 ),

(z ∈ [0, h2 ]) (z > h2 ).

1 Therefore, also fy (z) =

3(z + h2 )3 (1 −

2 h2 (z

−3(z − h2 )3 (1 −

+ h2 )2 )( h2 )4 ,

2 h2 (z

− h2 )2 )( h2 )4

(z ∈ [− h2 , 0]) (z ∈ [0, h2 ])

23h2 . 20.78 Comparing the transfer of distributions above with that of some simple ﬁlters can be done as follows. Let F be the 2n + 1 average ﬁlter and x a random sequence from a distribution fx (z), then the variable s = nx comes from the distribution with σ 2 y =

fs (z) = nfx (nz) =

1 1 z fx ( ) , with h = . h h n

If φk = fx is a B-spline of order k, then the output of the average of two variables is ∞ ∞ 1 1 x t−x t c(t) = 2 )dx = 2 φk ( )φk ( φk (w)φk ( − w)h dw, h −∞ h h h −∞ h so that c(t) = h1 φ2k ( ht ), from the basic convolution properties of B-splines. Similarly the sum of m variables from fs (z) has the distribution fy (z) =

1 z φnk ( ) = nφnk (nz). h h

120

9. Fair Comparison with Linear Smoothers

Comparing with the ﬁrst example on M1 we let (F x)i = 13 [xi−1 + xi + xi+1 ]. If (z) φ1 = (|z| < 12 ), the output yi = 13 [xi−1 + xi + xi+1 ] is a sequence of numbers from fy (z) = 3φ3 (3z), where φ3 is the B-spline of order 3 given by; φ3 (x)

=

1 3 2 2 (x + 2 ) 1 3 2 2 (x − 2 ) 3 2 4 −x

, x ∈ [− 32 , − 21 ] , x ∈ [ 12 , 32 ] , x ∈ [− 12 , 12 ]

0

otherwise.

From formulas for the moments of the splines, µ2 (3) = σy2 =

∞

−∞

z 2 3φ3 (3z)dz =

1 9

∞

−∞

3 12

x2 φ3 (x)dx =

and

1 1 µ2 (3) = . 9 36

Thus the variance is reduced by a factor of 3, demonstrating the similarity in 1 reduction of variance to that of M1 which yields σy2 = 20 . This is relatively easily 1 for the smoother Mn , demonstrating the usual (slow) generalized to 2(4n + 6) 1 order of approximation of the standard deviation 0 √ to zero. n When attempting to compare with the transfer of distributions of the LU LU operators we run into some computational complications, which can be witnessed in the simplest cases. Let x = xi be an i.i.d. with density f . Let u = U xi = U1 xi = min{mj = max{xi−n+j , . . . , xi+j }, j ∈ {0, 1, . . . , n}}. Then p(u ≤ z) = 1 − p(u > z) = 1 − p(mj > z, ∀j ). But p(mj > z, ∀j ) = p(xi > z) + ε. ε = p(xi ≤ z).p(x > z, for some i − n ≤ < i)p(xk > z, i < k ≤ i + n). Therefore p(mj > 2∀j ) = 1 − p(xi ≤ z) + p(xi ≤ z)(1 − p(xj ≤ z, ∀j
9. Fair Comparison with Linear Smoothers

121

Noting that fL (z) = fU (−z) and integrating we get z z FL (z) = fL (w)dw = fU (−w)dw − 12

=−

− 12

−2 1 2

fU (t)dt = −

−2

− 12

fU (t)dt +

1 2

− 12

fU (t)dt

= 1 − FU (−z). Thus the distributions of the output of L1 can be simply obtained from those of U1 . Example. Let f be the centered uniform distribution, then the output of U1 has the distribution given by fu (z) = (z + 12 )(2.5 − 3z)(x < 12 ). This is clearly biased in the positive direction with average value given by ∞ ∞ 1 1 and variance . tf (t)dt = t2 f (t)dt = 12 15 −∞ −∞ The bias of Un increases with n, with the average value given by which tends to

1 2

as n grows. The variance tends to 14 .

n2 2(n + 1)(n + 2)

Noting that the output of Ln Un cannot be obtained from the composition of the individual mappings, since the output of Un is increasingly correlated for increasing n, we have to go through probabilistic arguments that become increasingly tedious. Even for n = 1 it requires some work to yield, FLU (z) = F (z)2 (1 + (1 − F (z))(1 + F 2 (z)) and fLU (z) = f (z)F (z)(1 − F (z))(4 + F (z) + 5F (z)2 ). These distributions still suggest an expected bias, which with the uniform f , can 1 39 1 1 1 be calculated to yield an average of 20 and a variance of σ 2 = 400 − 21 ∈ [ 21 , 20 ]. This compares well with the variance of the median output. Intuition and simulation suggest that this equivalence will deteriorate as n grows. Seeking less bias, we often smooth with the smoothers Cn and Fn instead, which yield even more diﬃcult calculations to yield a closed form expression for the distribution of the output. Though bounds may be obtainable, we might prefer simulation. Since real noise generally has to be experimentally characterized there seems to be no real motivation to explore much further. Initial simulations suggest that the variance reduction of the corresponding median compositions approximate those of Cn and Fn quite well, with a small bias remaining in the expected value. Further, and more systematic, evaluation of the transfer of distribution by LU LU -operators has been done by Wild and others, but calculation becomes tedious. Perhaps the more important statistical estimates for random error are those with the probability of pulses appearing in the resolution levels of a DPT. We consider again the response of the DPT to noise in the sequence e = ei , where

122

9. Fair Comparison with Linear Smoothers

ei is an i.i.d. random number from a probability density function f that is some B-spline (or in the limit, Gaussian). An interesting question is what number of pulses can be expected in a particular resolution level, the expected amplitude and the variance. Taking the ﬁrst stage of a DPT with Un Ln we consider only downward pulses in the ﬁrst (highest-) resolution level. The pulses appear when L1 xi > xi , that is when max{xi−1 , xi+1 } < xi . z p(max{xi−1 , xi+1 } < z)k = F (z)2 , where F (z) = f (t)dt. −a

The density function of max{xi−1 , xi+1 } is the derivative of F (z)2 , or 2F (z)f (z), and the distribution of xi − max{xi−1 , xi+1 } is given by the convolute of f and 2F f . Example. In the case where f is uniform and centered, F (z) = (z − 12 )(|z| ≤ 12 ) and the convolute yields a distribution function given by 1 − z 2 when z > 0. A downward pulse appears only when this distribution is positive. The average pulse therefore has an amplitude of 1 t(1 − t2 )dt 1 0 = . 1 6 (1 − t2 )dt 0

The total variation of the ﬁrst part of the resolution layer, or T (x − L1 x), has an expected value of 2 N3 · 16 , since the probability of such a pulse appearing is exactly 1 3 = p(xi > max{xi−1 , xi+2 }). The negative pulses in the ﬁrst resolution level can be handled similarly, but now it is clear that there is a lesser expectation, because of the following argument. Let zj = (L1 x)j . Then xi < U1 L1 xi = min{max{zi−1 zi }, max{zi , zi+1 } ⇔ zi−1 , zi+1 > xi ⇔ min{xi−2 , xi−1 }, min{xi+1 , xi+2 } > xi ⇔ xi−2 , xi−1 , xi+1 , xi+2 > xi . The probability of this being true is precisely 15 , irrespective of the distribution f . Thus ∞ 1 F (z)4 f (z)dz = . P (U1 L1 xi > xi ) = 5 −∞ The total probability of a pulse appearing at a given point in the ﬁrst reso8 lution level is therefore 13 + 15 = 15 . Computing the average amplitude will result 1 in a value less than 6 . Variance can also be computed. The total variation is thus also predictable. Simulations easily conﬁrm the above calculations. Clearly general expected values of amplitudes, variance, total variation and even probabilities of pulses appearing at a given resolution level are increasingly

9. Fair Comparison with Linear Smoothers

123

complicated to calculate. For practical problems the noise can however often be estimated by constructing histograms from simulation. Thus criteria for thresholding the ﬁrst few levels to remove such noise with a reasonable probability can be fairly well selected and employed. What is clear is that for such a sequence e more than half the pulses in the DPT appear in the ﬁrst resolution level! (In a diadic wavelet decomposition this is almost half.) We can therefore expect a rapid lessening of the number of pulses and their amplitude in the higher resolution levels. The following graphs depict the expected values obtained by simulation of the fraction of the total variation remaining in the case of the uniform distribution. Various distributions yield similar results. The modelling of noise in the case of selectors has some advantages and some disadvantages over the case of linear smoothers. Some key advantages seem to be the simple formula for the probability of a pulse appearing, the variation preservation and the associated full trend preservation. These may lead to some insight, simpliﬁcation and estimation/bounding possibilities. This needs further research. For some background of noise modelling in image processing, Chapter 2 of the book of Starck et al. (3) can be consulted. What is clear however is that 1

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0

0

2

4

6

8

10 n

12

14

16

18

20

Figure 9.1. Graph of the fraction of the total variation left after peeling of the nth resolution level in the case of a uniform distribution.

124

9. Fair Comparison with Linear Smoothers

even with linear Multiresolution Analysis, the approximation ultimately stored, or transmitted have quantizing and thresholding attached. Thus they also become nonlinear approximations of the original data. Comparing linear and nonlinear smoothing in practice has another diﬃculty, since they are generally complementary procedures, and each has its merits. A simple example is the case of a contaminated sampling of a simple periodic function, where a Multiresolution Analysis with pulses reveals isolated large impulses as well as Gaussian noise. Using this diagnosis a simple low order LU LU -based smoother (or median-) removes the large impulses and standard linear approximation yields a good reconstruction. An example is given in the ﬁgure below.

Figure 9.2. A simple reconstruction of a cosine function sampled with noise.

Going back to the ﬁrst examples on page xi, we see that after analysis a morphological center of a LU LU -operator with suitably large support leaves the trajectory with minimal damage available for further analysis. Linear smoothing attempts were not worth displaying. The second example on page xii yields a fair comparison with small support linear ﬁlters and equivalent LU LU -smoothers. The advantage was signiﬁcant, and examples with Ln Un with n = 1, 2 and 3 can be shown with those of equivalent averages. The comparison here was entirely fair, although the author was on the boat when the measurements were taken and knew the intended speed. The boat was maintained at a constant speed of 35 knots (measured by other means) for a while, followed by a (linear) increase. This is clearly reﬂected in both the graphs, if interpreted by the intelligent human observer, whose descerning eye rejects the impulses still present. For a discussion of the essentially nonlinear nature of visual interpre-

9. Fair Comparison with Linear Smoothers

125

Figure 9.3. Decontamination of data with (real) impulsive noise. 40

35

30 35

30

25

Figure 9.4. Comparison of nonlinear- and linear smoothers on speedometer data.

tation of the human brain, further reading is worthwhile. An example is the article of S. Mallet (12) in the further background reference. It is also worth considering how many simple optical illusions are constructable (18), whereas the ear seems less deceivable in such a simple fashion, how easily parallel lines are confused in the presence of another (monotone) line, and how exhausting and confusing (to say the least) a swaying horizon is at sea. Does the latter associate with the reference of the base as the outcome of successive LU LU -smoothing, as in the ﬁnal example? As a ﬁnal example of the properties of the pulse transform we consider again the LU LU -decomposition of a sampled fort of Figure 7.2 distorted by noise (impulsive and Gaussian). It has the merit of demonstrating the economical coding of the essential recognizable shape (of C5 x) in four integers and four indexes, with code pulses appearing at some lower resolution levels, and the superior shape preservation compared to wavelet decompositions.

126

9. Fair Comparison with Linear Smoothers

15

15

10

10

5

5

0

0

–5

300

400

500

–5 castle with noise

15 10

10 5

0

0

–10

–5 15

C x 1

15

10

10

5

5

0

0

–5 15

–5 C4x

15

10

10

5

5

0

0

–5 15

D4x

–5 C x 32

15

10

10

5

5

0

0

–5

–5

15

D1x

C148x

D148x

15

10

10

5

5

0

0

–5

D32x

–5 C200x

D200x

Figure 9.5. Graphs of impulsive noise and the sequences Ci and Di for i = 1, 4, 32, 148, 200.

10. Interpretation and Future Leg ich mich aus, so leg’ ich mich hinein Ich kann nicht selbst mein Interprete sein. Doch wer nur steigt auf seiner eignen Bahn Tragt auch mein Bild zu hellem Licht hinan. Nietzsche

Hearing and Seeing are two physiologically diﬀerent means of receiving information. Hearing and the Frequency Domain belong together, but in Vision the natural framework may be Mathematical Morphology. Analyzing sequences for information with linear operators can be naturally associated with the Frequency Domain, since the eigensequences of such operators are periodic. Analyzing them for features like pulses and trends is diﬀerent. Wavelet Theory has held promises but the constraints have become apparent. Mathematical Morphology, as a framework for analysis and interpretation, has been developed over several years, but has only recently been linked with other attempts at establishing mathematical structure. LU LU -theory developed from the culture of intuitive association with the linear framework (of Electronic Engineers and Statisticians), but is a special part of Mathematical Morphology. Its perspectives yield a rich harvest of strong and intuitively simple results which the biased author risks calling beautiful. This eﬀort is intended to seduce mathematicians, scientists and engineers to explore this beautiful structure for interpretation and usefulness. It is deliberately written with passion rather than complete detail, an impressionistic painting rather than a photograph. The ideas have however grown out of practical problems tackled over more than two decades in several diﬀerent environments. The simplicity, power and regularity of the practical successes has fed the passion that drove the analysis, for which the author never seemed or felt adequately prepared. In attempting to interpret what LU LU -theory has yielded, the following can be claimed. 1. A framework for analysis and comparison of several useful nonlinear smoothers has been established that contributes to the selection, design and understanding of smoothers for a speciﬁc purpose.

128

10. Interpretation and Future

2. Conceptual links have been established with the existing (linear) framework for analysis, comparison and design of smoothers, as well as Mathematical Morphology. 3. A concept of “Smoothness” and “Trend” (Local Monotonicity) has been established and linked to an established natural measure of smoothness (Total Variation). 4. A Concept of “Pulse” has been established that is complementary to that of “Trend”. 5. Fundamental operators (Cn and Fn ) have been deﬁned and investigated for establishing a class of (nonlinear-) Multiresolution Analysis that decompose a sequence for such “Trend” and “Pulse” content. 6. The attributes of Neighbor Trend Preservation (ntp) and Full Trend Preservation (ftp) and that of Co-idempotence have been deﬁned and demonstrated to be as useful and important as the well-known concepts of Idempotence and Syntoneness. 7. Heuristically derived procedures (Median Transforms) in Image Processing, which have been known to work well (but lacking theory), have had their success supported by theory and the perceived superiority over Morphological Filters has been shown to be due to lack of fair selection. 8. Heuristic truths have emerged that have often resulted in good results in particular applications in Science and Industry. Involvement with several partners in Laser Spectroscopy, Positioning Problems, Earthquake Research, Image Processing, Financial Analysis, as well as Research in Mathematics and Statistics, is ongoing and growing beyond expectation. 9. Relevance (value) of some mathematical structures has been exposed. Mathematical research into these should yield many simpler arguments and clearer insights. 10. A Discrete Pulse Transform (DPT) has been established in analogy to a Discrete Fourier Transform (DFT). The primary use may be in image processing. A remarkable (and unexpected) consistent behavior has been proved and indications for further consistency presented. Unlike the DFT the resolution levels are not subspaces and there is no hope, nor justiﬁcation, for expecting consistent decomposition of arbitrary linear combinations of sequences from such resolution levels. If however a speciﬁc sequence is decomposed, then those speciﬁc resolution levels can be used to choose an arbitrary linear combination with non-negative coeﬃcients, and have guaranteed consistent decomposition. Noting that there is no negative luminosity in image processing, this seems suﬃcient for the purpose at hand, and as much as can be expected. Elementary modiﬁcations of such a decomposition can be stored and transmitted, and the reconstructed image will be predictably decomposed by the DPT, including the strong consistency formulated as “The Highlighting Conjecture”.

10. Interpretation and Future

129

11. Since the manuscript has been delivered to the publisher, some additional research has yielded results on some diverse lines. The interested reader can consult the list for further background reading for some published results. An article, written long ago with Malkowsky, introduces an analogy in Real Analysis. The article on quasi-inverses links up with Chapter 5, and shows that the LU LU -operators and compositions are optimal in damage minimization amongst general min-max operators. Included are some basic results on further links in LU LU -theory, like a simple proof that the median operator M1 is Variation Reducing, but arbitrarily slowly. Further advances in this direction have been submitted. Two further articles on applications to Financial Data, and on the transfer of distributions have appeared. Two further articles are being prepared on diﬀerent aspects of the consistency of the DPT. Prospects are good that one will contain a proof of the Highlight Conjecture. Future. The author has exposed his ideas in the hope that contact, debate and criticism can lead to more understanding of his favorite research area. It is hoped that more competent researchers can solidify and extend the ideas that have been behind this eﬀort. If only a few of the avenues of research that have been opened yield some result, the author would be grateful.

References [1] E. Ataman, V.K. Aarte and K.M. Wong, A Fast Method for Real-Time Median Filtering, IEEE Trans. Acoust. Speech Signal Process. ASSP-28, No. 4 (1980). [2] E. Ataman, V.K. Aarte and K.M. Wong, Some Statistical Properties of Median Filters, IEEE Trans. Acoust. Speech Signal Process. ASSP-29, No. 5 (1981). [3] A. Bijaoui, F. Murtagh, J.L. Starck, Image Processing and Data Analysis – The Multiscale Approach. Cambridge University Press, 1998. [4] A.C. Bovic, A. Restrepo, Locally Monotonic Regression, IEEE Transactions on Signal Processing. Sept. 1993, Vol. 41, No. 9. [5] A.R. Butz, A Class of Rank Order Smoothers, IEEE Trans. Acoust. Speech Signal Process. ASSP-34, No. 1 (1986). [6] A.R. Butz, Some Properties of a Class of Rank Order Smoothers, IEEE Trans. Acoust. Speech Signal Process. ASSP-34, No. 3 (1986). [7] N.C. Callagher and G.L. Wise, A theoretical analysis of the properties of median ﬁlters, IEEE Trans. Acoust. Speech Signal Process. ASSP-29, No. 6 (1981). [8] C.K. Chui, Wavelets: A Mathematical Tool for Signal Analysis. SIAM, Philadelphia, 1997. [9] E. Cloete, B. Opperman, An Asynchronous parallel computing algorithm for a nonlinear ﬁlter: Proc. 18th SANUM Conference, Durban, 1992. [10] L. Collatz, Monotonicity in Numerical Mathematics, Proc. 1st SANUM Conference, Durban 1973. [11] R.W. Hamming, Digital Filters Prentice-Hall, Englewood Cliﬀs, N.Y., 1977. [12] R.L. Hamming, The Frequency Approach to Numerical Mathematics in Studies in Numerical Analysis, Editor: B.K.P. SCAIFE, Academic Press. [13] C. Lanczos, Applied Analysis, Prentice Hall, N.J., 1956. [14] C.L. Mallows, Some Theory of Nonlinear Smoothers, Ann. Statist. 8, No. 4 (1980), 695–715. [15] P. Maragos, R.W. Schafer, Morphological Filters - Part II: Their relations to Median, Order-Statistic, and Stack-Filters, IEEE Transactions on Acoustics, Speech and Signal Processing, Vol. ASSP-35 (1987), 1170–1184. [16] A.E. Marquardt, L.M. Toerien and F. Terblanche, Applying nonlinear, Smoothers to Remove Impulsive Noise from Experimentally sampled data, R and D. Journal, SA I Mech E. Vol. 7, No. 1 (1991). [17] T.A. Nodes and N.C. Callagher, Median Filters: Some Modiﬁcations and Their Properties, IEEE Trans. Acoust. Speech Signal Process. ASSP-35, No. 5 (1982).

132

References

[18] K.R. Popper and J.C. Eccles, The Self and its Brain, Springer International, 1981. [19] C.H. Rohwer, Idempotent One-Sided Approximation of Median Smoothers, Journal of Approximation Theory. Vol. 58, No. 2 (1989), 151–163. [20] C.H. Rohwer, L.M. Toerien, Locally Monotone Robust Approximation of Sequences, Journal of Computational and Applied Mathematics 36 (1991), 399–408. [21] C.H. Rohwer, Projections and Separators, Quaestiones Mathematicae 22 (1999), 219–230. [22] C.H. Rohwer, Fast Approximation with Locally Monotone Sequences. Proceedings 4th FAAT Conference, Maratea. Supplemento ai rendiconti del Circolo matimatico di Palermo, Series II numero 68 – anno 2002. [23] C.H. Rohwer, Variation Reduction and LULU-smoothing, Quaestiones Mathematicae 25 (2002), 163–176. [24] C.H. Rohwer, M. Wild, Natural Alternative for One Dimensional Median Filtering. Quaestiones Mathematicae. 25 (2002), 135–162. [25] C.H. Rohwer, Multiresolution Analysis with Pulses, Advanced Problems in Constructive Approximation (Editors: M.D. Buhmann, D.A. Mache), International Series of Numerical Mathematics Vol 142, 165–186, Birkh¨ auser Verlag Basel, 2002. [26] C.H. Rohwer, Fully Trend Preserving Operators. Quaestiones Methematicae 27 (2004), 217–229. [27] H.L. Royden, Real Analysis, Macmillan, N.J., 1969. [28] I.J. Schoenberg, On Spline Functions, in Inequalities. (Editor: O. Shisha) Academic Press New York, 1967. [29] J. Serra, Image Analysis and Mathematical Morphology, Acad. Press London 1982. [30] J.W. Tukey, Exploratory Data Analysis, Addison-Wesley, Reading, MA, 1977. [31] P.F. Velleman, Robust Nonlinear Data Smoothers: Deﬁnitions and Recommendations, Proc. Natl. Acad. Sci. USA, 74, No. 2 (1977), 434–436. [32] M. Wild, On the Idempotency and Co-idempotency of the Morphological Centre, International Journal of Pattern Recognition and Artiﬁcial Intelligence, Vol. 15, No. 7 (2001), 119–1128. [33] Zhou Xing-Wei, Yang De-Yun, Ding Run-Tao, Inﬁnite Length Roots of Median Filters, Science in China, Ser. A, Vol. 35 (1992), 1496–1508.

Some further background [1] G.R. Arce and N.C. Gallagher, Jr., State description for the root signal set of median ﬁlters, IEEE Trans. Acoust. Speech Signal Process. ASSP-30 (1982), 894– 902. [2] G.R. Arce and M.P. McLoughlin, Theoretical analysis of the max/median ﬁlter, IEEE trans. Acoust. Speech Signal Process. ASSP-35 (1987), 60–69. [3] N. Balakrishman and C.R. Rao (eds), Handbook of Statistics, Elsevier Science B.V. 1998. [4] C.K. Chui, An introduction to Wavelets, Academic Press, Onlander. 1992.

References

133

[5] R. Coifman, F. Geschund and Y. Meyer, Applied and computational analysis 10, 27–44, (2001). [6] W.J. Conradie, T. de Wet, M.D. Jankowitz, An Overview of LU LU Smoothers with Application to Financial Data. S.E.E. (To appear). [7] W.J. Conradie, T. de Wet, M.D. Jankowitz, Exact and Asymptotic Distribution of LU LU Smoothers. Journal of Computational and Applied Mathematics. (To appear). [8] J.P. Fitch, Software and VLSI algorithms for generalized ranked order ﬁltering, IEEE Trans. Circuits and Systems. CAS-34 (1987), 553–559. [9] K.J. Hahn and K.M. Wong, Median ﬁltering of spectra, Proc. IEEE International Electrical and Electronic Conference, Toronto, Sept. 1983, pp. 352–355. [10] O. Kao, Modiﬁcation of the LU LU operators for preservation of critical image details, The 2001 International Conference on Imaging Science, Systems and Technology, Las Vegas, Nevada, USA. [11] E. Malkowsky, C.H. Rohwer, The LU LU -semigroup for Envelopes of Functions, Quaestiones Mathematicae 26 (2004). [12] S. Mallet, Applied mathematics meets signal processing, DOC. MATH. Extra volume ICM. I. 319–338 (1998). [13] M.P. McLoughlin and G.R. Arce, Deterministic properties of the recursive separable median ﬁlter, IEEE Trans. Acoust. Speech Signal Process. ASSP-35, (1987), 98–106. [14] P.M. Narendra, A separable median ﬁlter for image noise smoothing, IEEE Trans. Pattern Anal. Machine Intelligence. PAMI-3 (1981), 20–29. [15] T.A. Nodes and N.C. Gallagher, Jr., Two-dimensional root structures and convergence properties of the separable median ﬁlter, IEEE Trans Acoust. Speech Signal Process. ASSP-31 (1983), 1350–1365. [16] L.R. Rabiner, M.R. Sambur and C.E. Schmidt, Applications of a nonlinear smoothing algorithm to speech processing, IEEE Trans, Acoust. Speech Signal Process. ASSP-23, No. 6 (1975). [17] C.R. Rao, Has statistics a future? If so, in what form? Submitted World Scientiﬁc, 2002. [18] C.H. Rohwer, Quais-inverses and Approximation with min-max Operators, Proceedings 5th FAAT Conference, Maratea. Supplemento ai rendiconti del Circolo matematico di Palermo. (To appear). [19] U.E. Ruttimann and R.L. Webber, Fast computing median ﬁlters on general purpose processing systems, Optical Engineering, 25, No. 9 (1986). [20] A.F. Siegel, Robust regression using repeated medians, Biometrika, 69, 1 (1982), 224–242. [21] R.N. Strickland and M.R. Gerber, Estimation of ship proﬁles from a time sequence of forward-looking infrared images, Optical Engineering. 25, No. 8 (1986), 995–1000. [22] M. Wild, Idempotent and co-idempotent stack ﬁlters and min-max operators, Theoretical Computer Science 299 (2003), 603–631.

Index aliasing, 111 ambiguity, 31, 41, 72 analysis eigen∼, 117 Fourier ∼, 7 framework for ∼, 127 multiresolution ∼, 70, 71, 75 spline wavelet ∼, 73 trend ∼, 73 wavelet ∼, 7 annihilate, 17, 34 anti-extensive ﬁlter, 62 approximation theory, 43 B-splines, 119 band pass case, 95 best approximation, 44 bi-inﬁnite sequences of real numbers, 1 blockpulse n-∼, 15, 18 downward ∼, 16 upward ∼, 16 bounded variation, 51 class Mn of n-monotone sequences, 27 closing, 62 co-idempotence, 3, 128 co-idempotent, 37, 67 separator, 4 concept of idempotence, 128 of inequalities, xiv of lattices, xiv of order, xiv of pulse, 128 of smoothness, 128 of syntoness, 128 of trend, 128 consistency, 3, 28, 78

diﬀerence reducing, 58 digital ﬁlter, 7, 9, 109 discrete Fourier transform, 128 pulse transform, 128 downward blockpulse, 16 downwards n-monotone, 64 dual, 15 dual theorem, 15 eﬀectiveness, 3, 28, 78 eﬃciency, 3, 28, 78 eigenanalysis, 117 eigensequence, 116, 127 eigenvalue, 5, 39 extensive ﬁlter, 62 fast Fourier transform, 51, 71 wavelet transform, 51, 71 ﬁlter anti-extensive ∼, 62 digital ∼, 7, 9, 109 extensive ∼, 62 linear digital ∼, 7 morphological ∼, 3, 62 over∼, 62 recursive ∼, 115 strong ∼, 62 under∼, 62 Fourier analysis, 7 framework for analysis, 127 frequency domain, xiv Nyquist-∼, 24, 32, 61, 111 fully trend preserving, 58, 61, 128 Gibbs’ phenomenon, 72 Haar decomposition, 73, 78 wavelet, 73

136 heavy-tailed distribution, xiii Heisenberg inequality, 115 high-pass smoother, 94 highlighting, 100 highlighting conjecture, 100, 128 idempotence, 3 concept of ∼, 128 idempotent separator, 4 identically independent random numbers, 117 identically, independently distributed, 107 image processing, 71, 128 impulse, 10, 17 n-∼, 17 response, 81 impulsive noise, x inequalities, xiv interval LU LU -∼, 30, 33, 40, 44 n-noise ∼, 40 n-signal ∼, 32 Kronecker delta, 106 1 -norm, xiii lattices, xiv Lebesgue inequality, 43, 44 Lebesgue-type inequality, 48 linear digital ﬁlter, 7 Lipschitz-constant, 44 local operator, 6 low-pass smoother, 93 LU LU -interval, 30, 33, 40, 44 LU LU -smoothing, 125 mathematical morphology, 127 measure of smoothness, xiv, 128 median, 9, 23 recursive ∼, 35 smoother, 18, 19 transform, 71 minlip constant, 44 monotone, 21 n-∼, 21–23 piece-wise ∼, 51

Index morphological center, 62 ﬁlter, 3, 62 transform, 71 morphology mathematical ∼, 127 multiresolution analysis, 70, 71, 75 n-blockpulse, 15, 18, 19 n-impulse, 17 n-LU LU -similar, 34 n-monotone, 21–23 class Mn of ∼ sequences, 27 downwards ∼, 64 upwards ∼, 64 n-noise interval, 40 n-pulse, 15 n-signal interval, 32 n-similar, 34 neighbor trend preservation, 128 neighbor trend preserving, 57 noise, 3 impulsive ∼, x n-∼ interval, 40 random ∼, 117 nonlinear smoother, 9 nonlinear smoothing, 43 norms, 1 Nyquist-frequency, 24, 32, 61, 111 opening, 62 operator local ∼, 6 rounding ∼, 101 self-dual ∼, 15 thresholding ∼, 103 truncating ∼, 103 order, xiv partial ∼, 2 outlier, 21 overﬁlter, 62 parallelization, 105 partial order, 2 piece-wise monotone, 51 preservation neighbor trend ∼, 128

Index properties trend preserving ∼, 57 variation preserving ∼, 57 pulse n-∼, 15 concept of ∼, 128 transform, 125 pulses and trends, 127 random noise, 117 rank based selector, 6 rank order selector, 6 recursive ﬁlter, 115 recursive median, 35 residual, 105 resolution component, 93–95, 97 level, 97, 128 sequence, 90 robust statistics, xi root, 24, 25, 77 rounding operator, 101 scale independent separator, 39 selector, 6 rank based ∼, 6 rank order ∼, 6 self-dual operator, 15 semigroup, 2 separator, 3, 67 co-idempotent ∼, 4 idempotent ∼, 4 scale independent ∼, 39 signal, 3 smoother, 3, 6, 9 high-pass ∼, 94 low-pass ∼, 93 median ∼, 18, 19 nonlinear ∼, 9 smoothing nonlinear ∼, 43 smoothness concept of ∼, 128 measure of ∼, 128 spline wavelet analysis, 73 stability, 3, 28, 78 strong ﬁlter, 62 swallowing theorem, 10

137 syntone, 3, 6 syntoness concept of ∼, 128 theorem dual ∼, 15 swallowing ∼, 10 thresholding operator, 103 total variation, 52 transfer of distributions, 119, 121 transform discrete Fourier ∼, 128 discrete pulse ∼, 128 fast Fourier ∼, 51, 71 fast wavelet ∼, 51, 71 median ∼, 71 morphological ∼, 71 pulse ∼, 125 translation independent, 3 trend analysis, 73 concept of ∼, 128 trend preserving fully ∼, 58, 61, 128 neighbor ∼, 57 properties, 57 truncating operator, 103 underﬁlter, 62 upward blockpulse, 16 upwards n-monotone, 64 variation bounded ∼, 51 decreasing, 54 diminishing, 54 non-increasing, 54 preserving properties, 57 total ∼, 52 vectorspace, 1 wavelet analysis, 7 spline ∼, 73

Nonlinear Smoothing and Multiresolution Analysis

Read more

Nonlinear Smoothing and Multiresolution Analysis (International Series of Numerical Mathematics, 150)

Read more

Nonlinear Smoothing and Multiresolution Analysis (International Series of Numerical Mathematics, 150)

Read more

Nonlinear Analysis and Optimization

Read more

Nonlinear Analysis and Optimization

Read more

Nonlinear analysis

Read more

Nonlinear Analysis

Read more

Kernel smoothing

Read more

Nonlinear analysis and differential equations

Read more

convex analysis and nonlinear optimization

Read more

Surface Fitting and Multiresolution Methods

Read more

Smoothing Splines: Methods and Applications

Read more

Nonparametric Regression and Spline Smoothing

Read more

Elements of nonlinear analysis

Read more

Nonlinear Functional Analysis

Read more

Introduction to Nonlinear Analysis

Read more

Methods of nonlinear analysis,

Read more

Geometric nonlinear functional analysis

Read more

Applied nonlinear analysis

Read more

Nonlinear functional analysis

Read more

Methods of nonlinear analysis,

Read more

Nonlinear systems analysis

Read more

Nonlinear systems analysis

Read more

Nonlinear Functional Analysis

Read more

Nonlinear Analysis of Structures

Read more

Nonlinear Time Series Analysis

Read more

Nonlinear Functional Analysis

Read more

Methods in nonlinear analysis

Read more

Nonlinear Functional Analysis

Read more

Nonlinear Functional Analysis

Read more

Recommend Documents

Nonlinear Smoothing and Multiresolution Analysis

Nonlinear Smoothing and Multiresolution Analysis (International Series of Numerical Mathematics, 150)

ISNM International Series of Numerical Mathematics Volume 150 Managing Editors: K.-H. Hoffmann, Bonn D. Mittelmann, Tem...

Nonlinear Smoothing and Multiresolution Analysis (International Series of Numerical Mathematics, 150)

ISNM International Series of Numerical Mathematics Volume 150 Managing Editors: K.-H. Hoffmann, Bonn D. Mittelmann, Tem...

Nonlinear Analysis and Optimization

Nonlinear Analysis and Optimization

Nonlinear analysis

Series in Mathematical Analysis and Applications Edited by Ravi P. Agarwal and Donal O’Regan VOLUME 9 NONLINEAR ANALYS...

Nonlinear Analysis

Series in Mathematical Analysis and Applications Edited by Ravi P. Agarwal and Donal O’Regan VOLUME 9 NONLINEAR ANALYS...

Kernel smoothing

Nonlinear analysis and differential equations

convex analysis and nonlinear optimization

!#" $ $&%(' ,/0* )* $&+ ,%-*.$ *12$3*,...