Advances in Electronics and Electron Physics, Volume 88

ADVANCES IN ELECTRONICS AND ELECTRON PHYSICS VOLUME 88 EDITOR-IN-CHIEF PETER W . HAWKES Centre National de la Reche...

Author: Peter W. Hawkes

22 downloads 1453 Views 16MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!

Report copyright / DMCA form

DOWNLOAD PDF

ADVANCES IN ELECTRONICS AND ELECTRON PHYSICS

VOLUME 88

EDITOR-IN-CHIEF

PETER W . HAWKES Centre National de la Recherche Scientifique Toulouse, France

ASSOCIATE EDITOR

BENJAMIN KAZAN Xerox Corporation Palo AIfo Research Center Palo Alto, California

Advances in

Electronics and Electron Physics EDITEDBY PETER W. HAWKES CEMES/Laboratoire d’Optique Electronique du Centre National de la Recherche Scientifique Toulouse, France

VOLUME 88

ACADEMIC PRESS Harcourt Brace and Company Boston San Diego New York London Sydney Tokyo Toronto

This book is printed on acid-free paper.

@

@ 1994 BY ACADEMIC PRESS, INC. ALLRIGHTS RESERVED. NO PART OP THIS PUBLICATION MAY BE REPRODUCED COPYRIGHT

OR

TRANSMITTED IN ANY FORM OR BY ANY MEANS, ELECTRONIC OR MECHANICAL, INCLUDING PHOTOCOPY, RECORDING, OR ANY INFORMATION STORAGE AND RETRIEVAL SYSTEM, WITHOUT PERMISSION IN WRITING FROM THE PUBLISHER.

ACADEMIC PRESS, INC. 525

B

Street, Suite 1900, San Diego, CA 92101-4495

United Kingdom Edition published by ACADEMIC PRESS LIMITED 24-28 Oval Road, London NWI 7DX

LIBRARYOF CONGRESS CATALOG CARDNUMBER: 49-7504 ISSN 0065-2539 ISBN 0-12-014730-0 PRINTED IN THE UNITED STATES OF AMERICA 94

95

96

97

BC

9

8

7

6

5

4

3

2

I

CONTENTS CONTRIFIUTORS . . PREFACE . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

vii ix

Integer Sinusoidal Transforms WAI KUEN CHAM I. I1. I11. IV . V.

I. I1. I11. IV .

v.

VI . VII . VIII . IX . X.

Introduction . . . . . . . . . . . . . . . . . . . . . Sinusoidal Transforms . . . . . . . . . . . . . . . . Dyadic Symmetry and Walsh Transforms . . . . . . . . Integer Sinusoidal Transforms . . . . . . . . . . . . . Integer Cosine Transforms . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . .

1 10 15 24 41 59

Data Structures for Image Processing in C M . R . DOBIEAND P . H . LEWIS Introduction . . . . . . . . . . . . . . . . . . . . . Image Representations . . . . . . . . . . . . . . . . Previous Work . . . . . . . . . . . . . . . . . . . . Standards for Image Processing . . . . . . . . . . . . . Data Structure Design in C . . . . . . . . . . . . . . . Function Interface Design in C . . . . . . . . . . . . . Error Handling . . . . . . . . . . . . . . . . . . . . A Small Example . . . . . . . . . . . . . . . . . . . Implementation . . . . . . . . . . . . . . . . . . . Conclusions . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . .

63 65 67 72 73 74 78 78 80

Electron Crystallography of Organic Molecules DOUQLAS L . DORSET I . Introduction . . . . . . . . . . . . . . . . . . . . . I1. Historical Background . . . . . . . . . . . . . . . 111. Methodology . . . . . . . . . . . . . . . . . . . IV . Perturbations to Diffraction Intensities . . . . . . . . V . Applications . . . . . . . . . . . . . . . . . . . . . VI. Conclusions . . . . . . . . . . . . . . . . . . . . . Acknowledgments . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . V

. . . .

108

109

111 114 117 146 157 184 185 185

vi

CONTENTS

I. I1. I11. IV . V. VI . VII . VIII .

I. I1. I11.

. VI .

IV

v.

VII . VIII . IX

.

I. I1.

.

111

IV . V. VI .

Fractal Signal Analysis Using Mathematical Morphology PETROS MARAWS Introduction . . . . . . . . . . . . . . . . . . . . . Morphological Signal Transformations . . . . . . . . . Fractal Dimensions . . . . . . . . . . . . . . . . . . Fractal Signals . . . . . . . . . . . . . . . . . . . . Measuring the Fractal Dimension of 1D Signals . . . . . . Measuring the Fractal Dimension of 2D Signals . . . . . . Modeling Fractal Images Using Iterated Function Systems Conclusions . . . . . . . . . . . . . . . . . . . . . Acknowledgments . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . .

199 201 203 213 218 230 237 242 243 243

Fuzzy Set Theoretic Tools for Image Analysis SANKARK . PAL Introduction . . . . . . . . . . . . . . . . . . . . . Uncertainties in a Recognition System and Relevance of Fuzzy Set Theory . . . . . . . . . . . . . . . . . . . Image Ambiguity and Uncertainty Measures . . . . . . . Flexibility in Membership Functions . . . . . . . . . . Some Examples of Fuzzy Image-Processing Operations . . . Feature/Knowledge Acquisition. Matching. and Recognition Fusion of Fuzzy Sets and Neural Networks: Neuro-Fuzzy Approach . . . . . . . . . . . . . . . . . . . . . . Use of Genetic Algorithms . . . . . . . . . . . . . . . Discussion . . . . . . . . . . . . . . . . . . . . . . Acknowledgments . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . .

278 288 290 291 292

The Differentiating Filter Approach to Edge Detection MARIAPETROU Introduction . . . . . . . . . . . . . . . . . . . . Putting Things in Perspective . . . . . . . . . . . . . Theory . . . . . . . . . . . . . . . . . . . . . . . Theory Extensions . . . . . . . . . . . . . . . . . Postprocessing . . . . . . . . . . . . . . . . . . . . Conclusions . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . .

297 307 309 324 333 339 343

INDEX.

. . . . . . . . . . . . . . . . . . . . . . . . . .

. .

247 249 251 260 264 272

347

CONTRIBUTORS Numbers in parentheses indicate the pages on which the authors’ contributions begin.

WAI KUEN CHAM( l ) , Department of Electronic Engineering, The Chinese University of Hong Kong, Shatin, N.T., Hong Kong M. R. DOBIE(63), Department of Electronics and Computer Science, University of Southampton SO9 5NH, England DOUGLASL. DORSET(11 I), Electron Diffraction Department, Medical Foundation of Buffalo, Inc., 73 High Street, Buffalo, New York 14203-1196 P. H. LEWIS (63), Department of Electronics and Computer Science, University of Southampton SO9 5NH, England PETROS MARAGOS (199), School of Electrical Engineering, Georgia Institute of Technology, Atlanta, Georgia 30332-0250 SANKARK. PAL(247), Machine Intelligence Unit, Indian Statistical Unit, Calcutta 700035, India MARIAPETROU (297), Department of Electronic and Electrical Engineering, University of Surrey, Guildford GU2 5XH, England

vii

This Page Intentionally Left Blank

PREFACE

Most of the chapters in this latest volume of Advances in Electronics a17d Electron Physics are concerned in one way or another with imaging. The first two chapters deal with mathematical and computational aspects of image handling. In the first, W. K . Cham presents the theory of the various integer sinusoidal transforms, with image coding and, hence, compression as the principal application. In the second, M. R. Dobie and P. H . Lewis systematically examine data structures for image processing in the language C. Their discussion includes standards and error handling as well as numerous examples of real image-processing algorithms in C. The third chapter, by D. L. Dorset, forms a very complete account of the crystallographic study of organic molecules, using the electron microscope. The author has been one of the principal contributors to this important field, and his mastery of the subject has enabled him to present a vast amount of material in an easily readable text. This chapter should be of interest to the crystallographic community as well as electron microscopists studying specimens of this type. The next chapter, by P. Maragos. brings together two subjects that are attracting a great deal of interest, namely, the role of fractals and mathematical morphology. Professor Maragos is already well known for his extremely original work in another branch of mathematical morphology and I am delighted to publish this extended account of his very recent work in these A ~ I Y I I I L Y ~AS .welcome feature of this chapter is the introductory material, which should enable those who are not yet experts in the field of fractals to appreciate the very new material presented here. Several other contributions on related topics are planned for forthcoming volumes. Another growing area of study is concerned with the roles that fuzzy sets have to play in practical questions. For some years, these sets were regarded by many with some suspicion and remained the distractions of a limited community. They have now become respectable, however, and their place in image processing is being recognized. The fifth chapter, by S. K. Pal, explains why they are useful in both the traditional territory of image processing and in the newer world of neural networks. The volume ends with a chapter by M. Petrou on a problem in image processing that has been with us since image processing began but which has still not been completely solved, namely, edge detection. The everyday difficulties are discussed carefully and the various traps and pitfalls that make this basic task so hazardous are made very vivid. Methods of avoiding them IX

X

PREFACE

and of detecting edges with an acceptable degree of reliability are then set out clearly. I am convinced that many users of these techniques will be grateful for this account of the subject. As usual, I wish to assure all of the contributors of my gratitude for agreeing to write for these Advances, and I conclude with a list of forthcoming articles in this series. Peter W . Huwkes FORTHCOMING ARTICLES Electron holography Nanofabrication Use of the hypermatrix Image processing with signal-dependent noise The Wigner distribution Parallel detection Hexagon-based image processing Microscopic imaging with mass-selected secondary ions Nanoemission Magnetic reconnection Sampling theory ODE methods Interference effects in mesoscopic structures The artificial visual system concept Projection methods for image processing Minimax algebra and its applications Corrected lenses for charged particles The development of electron microscopy in Italy The study of dynamic phenomena in solids using field emission Gabor filters and texture analysis Miniaturization in electron optics Amorphous semiconductors Stack filtering Median filters Bayesian image analysis RF tubes in space

G . Ade H. Ahmed D. Antzoulatos H. H. Arsenault M. J. Bastiaans P. E. Batson S. B. M. Bell M. T. Bernius Vu Thien Binh A. Bratenahl and P. J. Baum J. L. Brown J. C. Butcher M. Cahay J. M. Coggins P. L. Combettes R. A. CuninghameGreen R. L. Dalglish G. Donelli M. Drechsler J. M. H. Du Buf A. Feinerman W. Fuhs M. Gabbouj N. C. Gallagher and E. Coyle S. and D. Geman A. S. Gilmour

PREFACE

Relativistic microwave electronics Theory of morphological operators The quantum flux parametron The de Broglie-Bohm theory Contrast transfer and crystal images Mathematical morphology Electrostatic energy analysers

Fuzzy relations and their applications Applications of speech recognition technology Spin-polarized SEM Sideband imaging High-definition television Regularization SEM image processing Electronic tools in parapsychology Image formation in STEM Phase-space treatment of photon beams New developments in electron diffraction theory Z-contrast in materials science Electron scattering and nuclear structure Multislice theory of electron lenses The wave-particle dualism Electrostatic lenses Scientific work of Reinhold Rudenberg Electron holography X-ray microscopy Accelerator mass spectroscopy Applications of mathematical morphology Set-theoretic methods in image processing Texture analysis Focus-deflection systems and their applications Information energy New developments in ferroelectrics Orientation analysis The suprenum project Knowledge-based vision

xi V. L. Granatstein H. J. A. M. Heijmans W. Hioe and M. Hosoya P. Holland K. Ishizuka R. Jones S. P. Karetskaya, L. G. Glikman, L. G. Beizina, and Y. V. Goloskokov E. E. Kerre H. R. Kirby K. Koike W. Krakow M. Kunt A. Lannes N. C. MacDonald R. L. Morris C. Mory and C. Colliex G. Nemes L. M. Peng S. J. Pennycook G. A. Peterson G. Pozzi H. Rauch F. H. Read and 1. W. Drummond H. G. Rudenberg D. Saldin G. Schmahl J. P. F. Sellschop J. Serra M. 1. Sezan H. C. Shen T. Soma I. J. Taneja J. Toulouse K. Tovey 0. Trottenberg J. K. Tsotsos

xii

PREFACE

Electron gun optics Very high resolution electron microscopy Spin-polarized SEM Morphology on graphs Cathode-ray tube projection TV systems

Diode-controlled liquid-crystal display panels Image enhancement Signal description The Aharonov-Casher effect

Y. Uchikawa D. Van Dyck T. R. van Zandt and R. Browning L. Vincent L. Vriens, T. G. Spanjer, and R. Raue Z. Yaniv P. Zamperoni A. Zayezdny and I. Druckmann A. Zeilinger, E. Rasel, and H. Weinfurter

ADVANCES IN ELECTRONICS AND ELECTRON PHYSICS, VOL. 88

Integer Sinusoidal Transforms WAI KUEN CHAM Department of Electronic Engineering, The Chinese University of Hong Kong. Shatin, Hong Kong

.

I. Introduction . . . . . . . . . . . . . . . . . . . . , . A. Transform Coding of Image Data . . . . . . . . . . . . B. Orthogonal Transforms for Image Coding . . . . . . . . . . . . 11. Sinusoidal Transforms . . . . . . . . . . . . . . . , . 111. Dyadic Symmetry and Walsh Transforms . . . . * * . . . . . . A. Background . . . . . . . . . * . . . . . . . B. Dyadic Symmetry and Walsh Transforms . . . . . . . . . . C. Dyadic Decompositions . . . . . . . . . , . . . . . . . IV. Integer Sinusoidal Transforms . . . . . . . . . . . . . . . A. Definition . . , , . . . . , , . . . . . . . . . B. Generation Method . . . . . . . . . . . . . . . C. Examples of Order-8 Integer Sinusoidal Tran,s f or rns . . . . , . V. Integer Cosine Transforms . . . . . . . . . . . . . . . . . A. Derivation . . . . . . . . . . . . . . . . . . . B. Performance of lCTs . . . . . . . . . . . . . . C. Implementation of ICT(10, 9, 6, 2, 3, I ) . . . . * . . . . , . D. Fixed-Point Error Performance . . . . . . . . . . . . . . . E. Fast Computation Algorithm . . . . . . . . . . . . . . . References . . . . . . . . . . . . . , . . . . .

. . .

.

. . .

.

. .

.

.

. . . . . . . .

. .

.

. .

. .

.

.

1

1 5 10

I5 I5

. .

11 23 24 24 21 31 41

.

41 44

49 52 51 59

I . INTRODUCTION A. Transform Coding of Image Data

Sinusoidal transforms in simplicity are transforms whose kernel elements are generated using sinusoidal functions. Basis vectors of some sinusoidal transforms are eigenvectors of covariance matrices of certain image models. Hence, these transforms have excellent compression ability for image data, and, in fact, image coding is a major application of sinusoidal transforms. Kernel elements of sinusoidal transforms are functions of sinusoidal functions and so generally are real numbers that are difficult or expensive to implement. The ideal case is to have integer versions of these sinusoidal transforms, which on one hand have the excellent performance of the sinusoidal transforms and on the other hand can be implemented easily and economically. This chapter explains how integer sinusoidal transforms are 1

Copyright 0 1994 by Academic Press. Inc. All rights of reproduction in any form reserved. ISBN 0-12-014730-0

2

WAI KUEN CHAM TABLE 1 STANDARDS AND DRAFT FOR IMAGE CODING Scheme

JPEG H.261 MPEG

Status

Application

Standard Standard Draft

Coding of still images Video conferencing Video coding

generated and analyzes the integer cosine transform, which is probably the most important integer sinusoidal transform, in more detail. Spurred by both market needs and technology development, a number of digital image coding standards are being developed. Table 1 lists some of the schemes that have become standards or in drafting stage. Analysts also predict that the high-definition television (HDTV) market could be worth US$500 billion over the next two decades. The world’s first HDTV system, MUSE, was pioneered by NHK of Japan; however, this system was not accepted by the U.S. F.C.C., because MUSE is not compatible with the existing TV system. Several schemes were proposed to satisfy the contradicting requirements of compatibility and quality. Table 2 shows some of the major proposals. All the image coding standards and HDTV proposals listed in Tables 1 and 2 relate to one class of coding algorithm called transform coding and use the discrete cosine transform (DCT) for the transformation process. Figure 1 shows the block diagram of a one-dimensional transform coding system. The original image to be encoded is divided into subpictures that are represented by n-vectors. Each of these vectors, say X,is transformed into a vector C of weakly correlated coefficients (c(i ) )using a transform [TI,i.e.,

c = [TIX.

(1)

The coefficients are then quantized to form a vector C,,which is then coded for transmission. At the receiver, the received bit stream is decoded into C, , and the inverse transform of [TI is applied to convert C,into X,, which is the quantized version of X. TABLE 2 MAJOR HDTV PROPOSALS HDTV systems

Proposed by

DigiCipher SC-HDTV Advanced Digital HDTV Progressive HDTV

General Instrument Zenith and AT&T Philips and others American Television Alliance

INTEGER SINUSOIDAL TRANSFORMS

3

C

X

Transform [TI

Quantizers

Bit Aseigner

bit stream to channel

bit stream

Decoder

from channel

FIGURE1. Block diagram of a transform coding system.

Compression of image data is achieved by the transformation of signal vector X in spatial domain into transform coefficient vector C , in which maximum information is packed into a minimum number of coefficients. Quantization is performed in transform domain, so that quantization error can be minimized by allocating more bits to coefficients having larger variances and fewer bits to coefficients having smaller variances. The following summarizes the basic theory of transform coding. 1. Orthogonal Tranforms

The transform [TI is orthogonal, i.e., [TI-' = [TI',

(2)

so that the average energy of the quantization error in transform domain is equal to the average energy of the quantization error in spatial domain, i.e., E[lC -

cqIZ1 = E[IX - X,l'l,

(3)

where E [ . ] stands for "the expected value of." In an efficient transform coding system, an adaptive scheme is used to minimize the quantization error in the transform domain. The use of an orthogonal transform ensures that the minimization of quantization error in transform domain leads to the minimization of quantization error in spatial domain, which implies a close resemblance of the decoded image and the original. The derivation of Eq. (3) from Eq. (2) is straightforward and is given as follows: E [ ( C- C9l2I= E [ ( C - CJ(C - CJl = E[([TIX - [TIX,)'([TlX - [TIXJl = E[(X -

X,)'[Tl'[TI(X - X9>l

= E [ ( X - X,>'(X - X,)] = E [ ( X- X912]

(Q.E.D.)

4

WAI

KUEN CHAM

Let T(i) be the ith basis vector of [TI.Equation (2) is equivalent to Eqs. (4) and (9,which means that basis vectors of [TI are orthogonal and of unity magnitude respectively. T(i)'-T ( j ) = 0,

IT(i)l = 1,

i~

j i , j, = 0, 1, ..., n - 1

i = 0,1,

(4)

...,n - 1.

(5)

If Eq. (5) is relaxed to

k any real number and i = 0, 1 , ., ., n

I T(i)l = k,

-

1

(6)

then it can be easily proved that Eq. (3) becomes

E"C - Cq12] = k2E[IX - Xq12].

(7)

Eq. (7) is sufficient to ensure that the minimization of quantization error in transform domain leads to the minimization of quantization error in spatial domain so [TI that satisfies Eqs. (4) and (6) is also used in a transform coding system.

2. The Optimal Bit Allocation The optimal bit allocation that results the minimum distortion can be derived from the rate distortion theory (Berger, 1971). The optimal number of bits for quantization of c(i) is

where 1. e2 is a parameter that reflects the performance of a practical quantizer,

whose value depends on the probability density function of the signal to be quantized and also the type of the quantizer (e.g., uniform or nonuniform); 2. Np is the average number of bits allocated to a pixel; and 3. oc(i)2is the variance of the ith coefficient c(i)in vector C . 3 . Quantization Error

If the optimal bit allocation is used, then the quantization error is

Detailed derivation of Eqs. (8) and (9) and a listing of values of e2 can be found in a paper by Wang and Goldberg (1988).

5

INTEGER SINUSOIDAL TRANSFORMS

4. The Optimal Tramform

Equation (9) implies that the optimal transform that results in minimum quantization error is the one that minimizes the geometric mean uc(i)2. The Karhunen-Lohe transform (KLT), whose basis vectors are composed of eigenvectors of the covariance matrix of c(i),can completely decorrelate the transform coefficients c(i) and minimize the geometric mean ac(i)2. Therefore, the KLT can minimize the quantization error, and so it is the optimal transform.

n n

5 . Separable Two-Dimensional Transforms

In practice, a separable two-dimensional orthogonal transform instead of a one-dimensional orthogonal transform is used. Consider a n x n matrix of pixels [XI = [X,(O),X"(1),... ,Xu@ - 111 where vector X , ( i ) represents the ith column of [ X I . A separable twodimensional transform can be performed on [XI in two steps: 1. [XI is first transformed into [D] by a premultiplication of [TI

[Dl = [ T l [ X l . This implies that the ith column of [ D ] is the one-dimensional transformation of X J i ) , i.e., D,(i) = [ T ] X , ( i ) .

2 . [ D ] is then transformed into [ C ] by a postmultiplication of [TI

This is to convert every row vector of [D] into a row vector of [ C ] . Therefore, (10) [CI = t TI [XI [TI' and [XI = [Tl'tCI[Tl = c(WW)TW'

cc i i

where c ( i , j ) is the (i,j)th element of [ C ] . B. Orthogonal Transforms for Image Coding

The first transform to be used in a transform coding system is discrete Fourier transform (DFT) (Andrews and Pratt, 1968), whose fast computational algorithm is called fast Fourier transform (FFT).DFT is one of four forms of Fourier analysis.

6

WAI KUEN CHAM TABLE 3 THEFOURFORMS OP FOURIER ANALYSIS Characteristics ________~

____~

Transform

Spatial domain

Frequency domain

Fourier Transform Fourier series Z-Transform DFT

Aperiodic and continuous Periodic and continuous Aperiodic and discrete Periodic and discrete

Aperiodic and continuous Aperiodic and discrete Periodic and continuous Periodic and discrete

Fourier analysis has four forms, depending on whether the spatial domain signal is aperiodic or periodic, continuous or discrete, as shown in Table 3. Generally, if a signal in one domain (either spatial or frequency) is periodic, then the corresponding signal in the other domain is discrete. Also, if a signal in one domain is aperiodic, then the corresponding signal in the other domain is continuous. The DFT is one whose spatial domain as well as frequency domain signals are periodic and discrete. Consider a continuous and periodic signal x(r) of period R . It is sampled to form a vector X, x = (x(O),X(l), .* *, x(n - l))', where x(i) is the data sampled at iR, and nR, = R. The DFT or the frequency-domain signal of vector X is

c = [FIX, where matrix [F] is the DFT. Because the signal is periodic, a linear shift of the signal is equivalent to a cyclic shift. Therefore, delay of X by m samples where m < n can be obtained easily as X , ( - m ) = (x(m),x(m

+ I), ,..,x(n - l),

x(O),

I , .

,x(m

-

1))'.

In this chapter, unless specified otherwise, transforms are in matrix form, and signals are discrete and in vector form like the DFT. However, it should be noted that the signal is not necessary periodic, as in the case of DFT. Hence, a linear shift may not be equivalent to a cyclic shift. However, many sinusoidal transforms can be realized using the DFT by repeating and reshuffling the signal. The use of DFT has assumed that such repeated and reshuffled signal is periodic. Attention on the DFT for transform coding quickly switched to the Walsh transform, whose computation requires only additions and subtractions (Pratt, Kane, and Andrews, 1969). Section I11 will give a more detailed description of the transform. In 1971, investigation began into

INTEGER SINUSOIDAL TRANSFORMS

7

the application of the Karlunun-Loeve Transform (KLT). In image coding, image data is often represented by a first-order Markov process x(i) = px(i - 1)

+ s(i),

i = 2, 3,

..., n ,

where p = E[x(i)x(i - 111, and s(i) is a zero-mean random variable. Eigenvectors of the covariance matrix of the Markov process form the KLT, which has been proved to be the optimal transform in the sense that it results in the minimum quantization error. The covariance matrix model whose (i, j ) t h element is ~ x( i, J ) = p l ~ - J l . i , j = o , 1 , ...,n - 1 ,

(1 1)

is a widely accepted model for image data because it is simple, and experiments confirmed its effectiveness for representation of image data. Ray and Driver (1970) found that the eigenvalues, ek , and the mth component of the kth eigenvectors, arn(k),of this covariance matrix were

m , k = 0 , 1 , ..., n - 1 ,

(13)

where (Q,) are the positive roots of the equation tan(nM) = -

(1 - pz)sin cos M - 2p + p2 cos n ’

n even.

Although eigenvalues and eigenvectors of covariance matrix [R,] given by Eq. (1 1) were found analytically, the KLT has no known fast computation algorithm. Therefore, they are seldomly used in practical systems. At about the same time, Enomoto and Shibata designed a new 8 x 8 transform to match typical image vectors (1971). Pratt, Welch, and Chen generalized this transform (1972), which was then known as slant transform, and later applied it to image coding with a fast computation algorithm (1974) resulting in a lower mean square error (MSE) for moderate block sizes in comparison to other unitary transforms. Many other transforms, such as discrete linear basis (Haralick and Shanmugam, 1974), slant Haar transform (Fino and Algazi, 1974), singular value decomposition (Andrews and Patterson, 1976), and modified slant transform and modified slant Haar transform (Kekre and Solanki, 1977) have also been proposed for image coding. However, the discovery of the discrete cosine transform (DCT) in 1974 (Ahmed, Natarajan, and Rao, 1974) and its fast computational

8

WAI KUEN CHAM

algorithm (Chen, Smith, and Fralick, 1977) has generated much interest. The (i,j)th component of an order-n DCT kernel is T ( i , j ) = (2/n)"*u(i) cos(i(j + i ) n / n ) ,

i,j = 0,1, ..., n - 1, (14)

where u(i) =

1/dl

=1

for i = o or n otherwise.

Comparisons between the DCT and other suboptimal transforms using the Markov process have shown that the DCT results in the least MSE (Chen and Smith, 1977), and in fact the DCT is asymptotically close to the KLT given by Eq. (13) (Shanmugam, 1975; Yemini and Pearl, 1979; Clarke, 1981) when the adjacent element correlation coefficient, p, approaches unity. Flickner and Ahmed (1982) showed that the DCT is also asymptotically close to the KLT for n approaching infinite. The high-energy packing ability of the DCT is not only useful for image coding but also for analysis, filtering, decimation, and interpolation of speech, electrocardiograms and electroencephalograms, and other signals, as well as for the processing of images for pattern recognition and image enhancement (Rao and Yip, 1990). All these processes are performed in the transform domain in much the same way as they would be in the frequency domain in Fourier analysis. For pattern recognition or waveform analysis, one can search for recognizable configurations of transform coefficients. To filter and enhance an image, each coefficient is multiplied by an appropriate function (and, possibly, added to some linear combination of other coefficients) before inverse transformation back to the signal domain. While the DCT is asymptotically close to the optimal transform, the search of sinusoidal transforms of even higher energy packing ability is continual. The following lists some of the findings. 1. Sine Transforms

Jain (1976) showed that by decomposing the Markov process into two mutually uncorrelated processes, namely the boundary response and the residual process, one can find the KLT for the residual process. The KLT is a discrete sine transform (DST), which can be computed using fast computational algorithms and whose transform kernel is i , j = 1,..., n.

(1 5 )

Farrelle and Jain (1986) proposed a coding technique, called recursive block coding, in which image data are divided into blocks with one pixel overlapping another. The overlapping pixels are transmitted and used for

9

INTEGER SINUSOIDAL TRANSFORMS

prediction of those in between. The difference between the original data and the predicted ones thus forms the residual process, which is then coded using the DST. Meiri and Yudilevich (1981) also proposed a similar technique using another transform called the pinned sine transform.

2. Symmetry Cosine Transform Kitajima (1980) proposed a transform whose kernel is @ ( i , j )= ( 2 / t ~ ) ” ~ u ( i ) ucos(ijn/n), (j)

i , j = 0, 1,

..., n ,

(16)

where

u(i) = l / f i = I

for i = o or n, otherwise.

The transform kernel is a symmetric matrix and is called the symmetry cosine transform (SCT). The SCT is asymptotically equivalent to the KLT for n approaching infinite and requires fewer multiplications than the DCT. Its ability to compress a Markov process signal vector with the first and last elements weighted by 1/fiis higher than the ability of the DCT to compress a Markov process signal vector. The SCT is also called the version I DCT (Wang, 1984). 3 . Weighted Cosine Transforms

Wang (1986), by putting a little phase change to the even basis vectors of the DCT, developed the phase shift cosine transform (PSCT), which has better performance but requires more computation than the DCT. Lo and Cham (1990), by introducing a weighting into the odd basis vectors of the PSCT, developed the weighted cosine transform (WCT), which has improved performance but requires more computation in comparison with the PSCT. Both PSCT and WCT can be computed using fast computational algorithms. Basis vectors of sinusoidal transforms are eigenvectors or asymptotically equivalent to the eigenvectors of certain covariance matrices. Therefore, their performance is optimal or near optimal. On the other hand, many nonsinusoidal transforms, such as the Walsh transform or slant transform, are integer transforms whose implementation is simpler but whose performance is inferior in comparison with the sinusoidal transforms. The idea of integer sinusoidal transforms is to combine the advantages of both. In Section 11, we shall first describe two unified treatments of sinusoidal transforms proposed by Jain (1976) and Wang (1984). The two treatments will provide us with a deeper understanding of sinusoidal transforms.

10

WAI KUEN CHAM

In Section 111, dyadic symmetry and its relation to the Walsh transform is given. The results described in these two sections are used in Section IV to generate integer sinusoidal transforms. As the DCT plays a very important role in image coding, we shall give a detailed analysis of the integer cosine transform in Section V.

11, SINUSOIDAL TRANSFORMS

The term sinusoidal transform was used by Jain (1976, 1989) to suggest that the KLT, DFT, DCT, DST, and pinned sine transform can all be regarded as members of a sinusoidal family by means of a parametric matrix [J(k, k2 k3)I where 9

3

[J(ki > k2 k3)I 9

=

I

1 - kla -a

0

-a

1

-a

0

-a

1

0

.

I .

........ ........

1

k3a 0

-a

I

*

(17)

A transform [a] = [ao, 0 1 ..., , is a member of the family if am’sare eigenvectors of [J(k,,k2, k3)]. Table 4 gives some members of the sinusoidal family generated by [J(kl, k2,O)]. Each member of the family is a KLT of a unique, first-order nonstationary (in general) Markov process x(i) = r(i)x(i - 1) + s(i) i = 2, 3 , ...,n ,

where [r(i)]and { s ( i ) )are deterministic and random white-noise sequences respectively. The variances of the initial state x(1) and random variable s(2) are

P2

E[x(1)21 = 1 E [ s ( ~ ) ~=] where

-

r(2) + k, a

klcr

(18)

11

INTEGER SINUSOIDAL TRANSFORMS

SOME MEMBERS OF

THE

TABLE 4 SINUSOIDAL FAMILY GENERATED BY ./(k,, k, , 0)

P I

k,

k,

P

P

KLT

1

1

Even cosine-1 or the DCT

m, k

Eigenvectors Qm(k),

.

a,,, sin(Q,,k

E

[ O , n - I]

+ 4,) 1/h,

,

m

=

~

m#O

1

0

-1

0

Even sine-1 or the DST

-1

Even sine-2

or pinned sine transform

sin (m + I)(k + I)n

nsin

n + l

(2k + I) ( m + l)n , 2n

I/&,

-I

1

Even sine-3

1

0

Odd cosine-]

ficos

0

1

Odd sine-1

2 (2m + l)(k + 1)n -sin 2n + 1 4 2 n + 1)

0

-

-1

0

1

Odd sine-2

Odd sine-3

2

2

W G q sin 2

2(k

+ 1)(2m + 2(2n

m # n - 1 m = n -

I

l)n

+ 1)

+ I)(k + I)n 2n + 1 (2k + I)(m + 1)n 2n + 1 2(m

and p is the adjacent element correlation coefficient. The covariance matrix of ( x ( i ) l is (20) [R,] = E [ x * x'] = p 2 [ J ( k l kz, , k3)I-l. For example, the KLT of the first-order stationary Markov process of the covariance matrix given by Eq. (1 1) can be constructed from [J(p,p, O ) ] . The even sine-1 transform (i.e., the DST) is constructed from (J(0,0, O ) ] . The even sine-2 transform (i.e., the pinned sine transform) and even cosine-1 transform (i.e., the DCT) are constructed from [J(-1, - 1, O)] and [J(1, 1, O)] respectively.

12

WAI KUEN CHAM

The J matrix given by Eq. ( 1 7) can be used for performance evaluation of the sinusoidal transforms. Two sinusoidal transforms are compared by computing their J matrix distances with the KLT

II [J(kl

k2 Y k3)I - [J(P, P , 0)l (I2.

(21) For example, Jain found that the DCT does not always perform better than the DST for the Markov process having covariance matrix given by Eq. (1 1). For p < 0.5, the J matrix distance of the DST is smaller than that of the DCT. The J matrix can also be used for finding a sinusoidal transform approximation to the KLT of a random process of covariance matrix [R,]. If [R,] commutes with J matrix, i.e., 4 k l k2 k3) = 1

9

9

[R,I[Jl = [JI[R,I, (22) then [ J ] and [R,] will have the same set of eigenvectors. Thus, the best sinusoidal transform may be chosen as the one whose J matrix minimizes the J matrix distance given by Eq. (21). Another unified approach was proposed by Wang (1984), who suggested that there are four versions of cosine transforms and four versions of sine

i , j = O , l , ..., n

(23)

i , j = O , l , ..., n - 1 (24) i , j = O , 1 , ..., n - 1 (25) i , j = O , 1 ,..., n - 1 (26)

i , j = 1 , 2 ,..., n - 1

(27)

i , j = 1 , 2 ,..., n

(28)

i , j = l , 2 ,..., n

(29)

i , j = O , l , ..., n - 1 (30)

13

INTEGER SINUSOIDAL TRANSFORMS

where

u(i) = 1/v9 =1

for i = o or n otherwise.

[C"(n)] is the DCT given by Eq. (14) and [C'(n)] is the SCT given by Eq. (16). All these transforms are orthogonal. Version I and IV transforms are symmetric, and so their inverses equal themselves. Transpositions of version I1 matrices are version 111; therefore,

[ ~ " (+ n I)]-' = [C"'(n + I)] [~"(+ n I)]-' = [S"'(n + I)].

(31) (32)

Generally, all versions of the cosine or sine transforms can be expressed in terms of [C"] of lower orders. 1 . Version I:

(34)

2. Version 11:

n>3

(35)

n>3

(36)

3. Version 111:

n>3

(37)

n>3

(38)

1

14

WAI KUEN CHAM

[

[AIV(n)] = 1 -U t n ) -rC+n,]

\rz I(+n)

I&)

-

I =

(43)

-

1 0 0 0

0 . . . . . . . 1 0 . . . . . .

0 0 0 0

0 0 0 0

. . . 1

0 1 0 0

. . . . . . . . . 0 . 0 1 0 . . . 0

f o r j odd

(44)

f o r j even

(45)

0 . . . 0 1 0 . 0 0 . . 0 1 0 . . 0 1 0 . . . 0 0 . 0 0 . . . . o o . 1 0 1 0 . . 0 0 . 0 0

.

.

.

.

0

0

1

0

. . . . . . . . . 0 . 0 1 0 . . . 0

~ 0 . . . 0 1 0 . 0 1 -1 1 *

.

-1

1.

INTEGER SINUSOIDAL TRANSFORMS

15

Wang (1984) showed that [CIv(n)] can be expressed into a product of 2 log, n + 1 sparse matrices. This expression leads to a fast computation algorithm of [CIv(n)] and so to fast computational algorithms for all versions of cosine and sine transforms. 111. DYADICSYMMETRY AND WALSHTRANSFORMS

A . Background Not all sinusoidal transforms have integer versions. Some do but their integer transform kernels are composed of integers of large magnitudes. Such integer transforms may not be very useful because their implementation is not simple. Fortunately, some sinusoidal transforms have integer versions composed of integers of small magnitude. For example, the DCT and the pinned sine transform have integer versions composing of integers having magnitude equal to five or less. The implementation of these transforms is very simple. Generally, those transforms that have several dyadic symmetries within their basis vectors tend to have integer versions composed of small integers. Furthermore, these dyadic symmetries also allow fast computation algorithms to be derived for the integer transforms. In this section, the definition of dyadic symmetry, its properties, and its relationship to the Walsh transform will be given. We shall prove that every basis vector of the Walsh transform has all dyadic symmetries. More detailed results on the dyadic symmetry and Walsh transforms can be found in (Cham and Clarke, 1987). Some of the theorems derived in this section will be used in Section IV to derive the integer sinusoidal transforms. The Walsh transform, which is probably the most popular integer transform, has close relationship to some integer transforms. For example, the HCT and LCT were derived by modifications of the Walsh transform (Cham and Clarke, 1986). Jones, Hein, and Knaver (1978); Srinivasan and Rao (1983); and Kwak, Srinivasan, and Rao (1983) also derived an integer version of the DCT, called the C-matrix transform, via the Walsh transform. Based on the work by Haar (1910), Walsh derived a complete orthonormal set of continuous rectangular functions in the interval [0, 11, now known as Walsh functions (1923). Harmuth (1977), using the Walsh functions as an example, generalized the concept of frequency (for sinusoidal functions only), to “sequency” for any type of function. Pratt, Kane, and Andrews then used the Walsh transform, derived from the Walsh functions, in place of the FFT for image coding (1969). As for the Walsh functions, the Walsh matrix contains only the values - 1 and + l . Therefore, conversion of a signal vector into the Walsh

16

WAI KUEN CHAM

transform domain involves only simple additions and subtractions. Moreover, there exists a fast Walsh transform algorithm similar to that of the FFT, and therefore the computational requirement of the Walsh transform is much less than that of the FFT. In the early 1970s, the simplicity and ease of implementation of the Walsh transform resulted in a wide range of applications (Proc. Symp. Applications of Walsh Functions, 1970, 1971, 1972, 1973), including analysis, filtering, and data compression of speech, electrocardiograms and electroencephalograms, and other signals, as well as for the multiplexing of communication channels and the processing of images for pattern recognition, data compression, and image enhancement. The effectiveness of most of these applications, however, especially filtering and data compression, depends on one single important property. In the case of the Walsh transform, it is the ability to pack the signal energy into a few transform coefficients. For the Walsh function, which is continuous, it is the ability to represent a signal waveform accurately using as few terms as possible. Unfortunately, the Walsh functions and Walsh transform are inferior to Fourier series representation and DFT in that respect (Blachman, 1974). It is found that, to represent a smooth signal, far more terms are required in the Walsh series representation. Even for discontinuous signals, the Walsh series may also need a lot more terms. On the other hand, many other sinusoidal transforms have been found to have higher energy packing ability than the Walsh transform. Thus, the interest in applications of the Walsh functions and Walsh transform was diminishing. During the development, different researchers adopted different nomenclatures and so created a lot of confusion. Ahmed, Schreiber, and Lopresti (1973) thus proposed a set of terms and related definitions for sequency, Walsh functions, and transform. They pointed out that the set of Walsh functions and its discrete conterpart the Walsh transform are classified into three groups according to their ordering. 1. Sequency-Ordered Walsh Functions and Transforms

These functions are also known as sequency-ordered Walsh functions (Yuen, 1972), Walsh-ordered Walsh functions (Harmuth, 1968) or simply Walsh functions. Their discrete counterpart, the sequency-ordered Walsh transform, is called the Walsh transform. 2. Dyadic-Ordered Walsh Functions and Transform These functions are also known as Paley-ordered Walsh functions, and their discrete counterpart is called the Paley-ordered Walsh transform.

INTEGER SINUSOlDAL TRANSFORMS

17

3. Natural-Ordered Walsh Functions and Transform The functions are also known as Hadamard-ordered Walsh functions or the binary Fourier representation (BIFORE) (Ahmed, Rao, and Abdussattar, 1971). Their discrete counterpart, the natural-ordered Walsh transform, is sometimes called the BIFORE Walsh transform. On the other hand, the Hadamard matrix is defined here as a square matrix of only plus and minus one, whose rows and columns are orthogonal to one another and can be of any size. Hadamard functions will refer to their counterpart in the continuous case, which are also called Walsh-like functions (Larsen and Madych, 1976). Therefore, under this nomenclature, Walsh functions and transforms are of order 2", m being an integer, and are particular cases of Hadamard functions and transforms. In the next section, we shall introduce the concept of dyadic symmetry and independent and dependent dyadic symmetry and their application on the Walsh matrix. Some of these concepts are used in the generation of integer sinusoidal transforms in Section IV and a fast computation algorithm in Section V.

B. Dyadic Symmetry and Walsh Transforms

In comparison with the DFT whose power spectrum is linear shift invariant, the power spectrum of the Walsh transform is dyadic shift invariant (Robinson, 1972). In fact, generation methods, fast computation algorithms, and other properties of the Walsh transform relate closely to dyadic symmetry. Let F be a number field. In this chapter, unless stated otherwise, all vectors in F are column vectors. Let M be 2", where m is an integer. Definition I . A vector in F of M elements [a(O),a ( l ) ,a(2), . . ., a(M - 111 is said to have even or odd Sth dyadic symmetry if and only if j = O , l , ..., M - 1 a n d S = 1 , 2 ,..., M - 1

a(j)=a(S)-a(j@S),

where @) = 1, =

-1

for S even, and for S odd.

@ is "exclusive-OR" operation.

Definition 2. Two M-vectors, U and V , in F are said to have a common dyadic symmetry S , where S E [l, M - 11 if U has even or odd Sth dyadic symmetry and V has even or odd Sth dyadic symmetry.

18

WAI KUEN CHAM

Definition 3. Two M-vectors U and V in F are said t o have the same dyadic symmetry S if U and V have a common dyadic symmetry S and the types (even or odd) of the dyadic symmetry of the two vectors are equal. Theorem 4. Two M-vectors U and V in Fare orthogonal if U and V have a common dyadic symmetry and the types of the dyadic symmetry are different.

froof, Let S be the common dyadic symmetry and U be the vector having even S dyadic symmetry. Elements of U can be grouped into M / 2 pairs, (u(a,), u(bl)), (u(a2), u(bd),. . ., (u(a~/2),u ( b ~ 2 )with ) u(ai) = u(b;),

(46)

where a; and bi are integers within [0, M - I] and i is an integer within the range [ l , $t4]. As vector V has odd S dyadic symmetry, so elements of V can also be grouped into M / 2 pairs, (v(a,), ~ ( b , ) ) ,(u(a2), v(b2)), . .., ( ~ ( a ~ v/ (~b)~,/ d with ) v(ai) = -v(bi). (47) The dot product of U and V is

u'.v =

M- 1

c u ( k ) .v(k) c u(ai) - v(ai) +

k=O

=

M/2 - 1

M/2- 1

i=O

j = O

C

u(bj)* v(bj)

M/2 - 1

=

c

('(ak)

*

d a k ) + u(bk) * v(bk)

(48)

k=l

By substitution of Eq. (46) and Eq. (47) into Eq. (48), the dot product of U and V is U' V = 0 (Q.E.D.)

-

Theorem 5. If a M-vector in F has dyadic symmetries S , , S 2 , ...,S,, then this vector also has dyadic symmetry Sk =

with type given by

S, @

s 2

@

' * *

@ S,

(49)

19

INTEGER SlNUSOIDAL TRANSFORMS

Proof. Let vector A be [a(O),a ( l ) ,a(2), ..., a(M - l)] having dyadic symmetry S1, S2,..., S,. As given by the definition of dyadic symmetry, we have

0S , ) d j ) = a ( S 2 ) 4 j0 S2) 4j)=4 S M j

j = 0 , 1,

a ( j ) = a(S,)a(j 0 Sr),

..., M

- 1

and i

=

1,2, ..., r.

Combine the r.h.s. of the first two equations together, we have d j

0 S , ) = a ( S , ) 4 S 2 ) a ( j0 S2).

Since both j and j 0 S , are dummy variables within [0, m - 11, j can be replaced by j 0 S1, resulting in

4-i 0 S , 0 S , ) = 4 S , ) 4 S 2 ) a ( j0 Sl 0 S2) or 4 j ) = a(S,)4S2)a(0 j S,

0

Continuing the same procedure for S3,S,, , . ., and S , , we have dj)=4 S M j

0 &I

where 4 S k )

= cr(S,)cr(S2).* .

4%)

and Sk

= S,

0 S2 0

0S,.

(Q.E.D.)

Corollary 6. The maximum possible number of dyadic symmetries in a 2"-vector in F is 2" - 1 . Example 7. If an 8-vector has even first and odd second dyadic symmetry, then by Theorem 5 this vector also has odd third dyadic symmetry. Example 8. Consider two vectors X and Y . Vector X has dyadic symmetries 1 , 2, and 4, while Y has dyadic symmetries 1, 2, and 3 . Theorem 5 implies that vector X also has dyadic symmetries 3, 5 , 6, and 7 because 3=1@2,

5=1@4, 6 = 2 @ 4,

and

7=1@2@4.

20

WAI KUEN CHAM

On the other hand, no more new dyadic symmetry can be deduced from dyadic symmetries 1, 2, and 3 on vector Y . This is because dyadic symmetries 1 , 2, and 4 are linearly independent while dyadic symmetries I , 2, and 3 are linearly dependent. The concept of (linearly) independence and dependence can be easily comprehended if we consider dyadic symmetries as vectors in a binary field FB that has 0 and 1 as its elements, “logicalAND” as the multiplication operation and “exclusive-OR” as the addition operation. The multiplication and addition operations will be represented respectively as * and 0. Consider a dyadic symmetry S whose binary representation is s(l),s(2), ...,s(m), where s(1) is the most significant bit and s(m) is the least significant bit. We shall use S to represent the vector [s(l),s(2), . ..,s(m)]in the binary field FB. Unless specified otherwise, all vectors in FB will be row vectors.

Definition 9. Dyadic symmetries S1, S 2 , ...,S,,,are said to be dependent if there exist m elements k,, k 2 ,...,k, = 0 or 1, not all zero, such that k,.S,+k,.S,+...+k;S,=O

.

Otherwise, the m symmetries are said to be linearly independent.

Theorem 10. Consider a vector in F that has 2” - 1 dyadic symmetries. The types of these dyadic symmetries can be derived from the types of any m independent dyadic symmetries. Proof. Let S1, S 2 , . .., S, be the m m-vectors representing the m independent dyadic symmetries. Since m independent vectors can span an m-dimensional vector space, any one of the 2” - 1 dyadic symmetries, represented by a m-vector S in FB, can be expressed as S = k1 * S 1 + k2 * S 2 +

.--+ k, *S,,

k l , k 2 ,..., k, = 0 or 1.

Suppose k a l ,k,, , ...,,.k are those scaling constants not equal to 0. Equation (50) implies that

49 = &,,)4S,,)

* * ’

4Sap).

(51)

The type of S thus can be determined by the sign of a(S). (Q.E.D.) Let the binary Walsh matrix [ B ] be the binary representation of a Walsh matrix [ W ] ,so its ( i , j)th element is b(i,j) = 0 = 1

if w(i,j ) = 1 if w ( i , j ) = -1.

Note that indices i and j , like the dyadic symmetry S, are vectors in F B . However, an index may assume 0, while S may not.

INTEGER SINUSOIDAL TRANSFORMS

21

Definition 21. A 2" x 2" sequency-ordered binary Walsh matrix [B,] is a matrix whose (i,j)th element is b z ( i , j ) = j[z]-'i' = [ j ~ ) , j ( 2 )..., , j(m)l[zI-'[i(l), i(2), ...,WII',

i , j = O , l , ..., 2"- 1 ,

(53)

where the m x m matrix [Z] is called the sequency-ordered dyadic symmetry matrix and is equal to

(54) 1

1

...

1

1

1

For example, when m = 3, we have

[ N ]=

1 0 0 1

... 0 0 0 ... 0 0 0

. ... . . . 0 0 ... 0 1 0 0 0 ... 0 0 1 *

(56)

22

WAI KUEN CHAM

Definition 23. A 2" x 2" dyadic-ordered binary Walsh matrix [BD]is a matrix whose (i,j)th element is

bD(i,j) = j[DI-'i'

...

.

= [ j ~ ) , j ( 2 ) , , ~ w I P I - ' [ ~i(2), u ) ,.., i(m)I',

i , j = 0, 1 , ..., 2"

- 1,

(57)

where the m x m matrix [D] is called the dyadic-ordered dyadic symmetry matrix and is equal to 0 0 ... 0 0 1

[D

::: ...0 0: 00 1 .

[Dl = 0 01 0 1

0 0

...

(58)

0 0

Obviously, [D]-' = [D] and

b D ( i , j )= i(l)j(m)0 i(2)j(m - 1 )

0

0 i(m)j(l).

Theorem 14. AN basis vectors of a Walsh matrix have all 2" - 1 dyadic symmetries.

Proof. This theorem is equivalent to the statement that in any ith basis vector and for any dyadic symmetry S, the product of the j t h element and the ( j 0 S)th element is the same for all j ; i.e.,

b ( i , j ) @ b(i,j

0 S ) = d(i, S ) .

By Eq. (52), we have

b(i,j) = j[S]-'i'

and

b(i,j 0 S ) = j

0 S[S]-'i'.

Therefore

b ( i , j ) 0 b(i,j

0 S ) = S[S]-'i' = d(i, S )

is the same for all j . (Q.E.D.)

Theorem 15. Conversion between sequency, dyadic, and natural ordering can be achieved using the follo wing equations, where iz , iD , and iNare the corresponding row indices: iz = [ z ] [ D ] - ' ~=, [ z ] [ N ] - ' ~ , ,

(59)

iD = [ D ] [ z ] - ' ~=, [ D ] [ N ] - ' ~ ~ ,

(60)

iN = [ N ] [ z ] - ' ~ = , [N][D]-'~,.

(61)

and

INTEGER SINUSOIDAL TRANSFORMS

23

Proof. Since bZ (iZ j ) =

bD

(iD j ) = bN(iN j ) 3

by means of Eqs. (53), ( 5 3 , and (57), we have

j[Zl-'iz

= j [ D ] - ' i= ~ j[N]-'iN,

j = 1,2,..., m.

Therefore, we have Eqs. (59)to (61). (Q.E.D.)

C. Dyadic Decompositions

Theorem 16. If all basis vectors of a M x M matrix [TI have a common dyadic symmetry S with half of them having even S dyadic symmetry and the other half having odd S dyadic symmetry, then the transformation of n-vector X into C by [T I, i.e.,

c = [T IX , can be composed into two )M x ) M transformations, and Such decomposition is called the Sth dyadic decomposition, and [T,] and [T,]are called the even and odd transforms of the decomposition respectively (Cham and Clarke, 1987). The elements in C, together with the i M elements in C, form vector C. Elements of X, are obtained by summing two elements in X,i.e., x ( i ) + x(i 0 S ) , and elements of X, are obtained by subtracting two elements in X , i.e., x ( i ) - x(i 0 S ) . Instead of proving this theorem, we shall give an example to show how this theorem may be used to derive a fast computation algorithm. Consider an order-8 sequency-ordered Walsh transform that converts vector X into vector C, 1.e.. - -40) 1 1 1 1 1 1 1 1 x(0) dl) 1 1 1 1 - 1 - I -1 -1 x(1) c(2) 1 1 - 1 -1 -1 -1 1 1 x(2) 1 1 1 1 1 1 - 1 -1 43) x(3) * (64) 44) 1 - 1 -1 1 1 - 1 -1 1 x(4) c(5) 1 - 1 -1 1-1 1 1-1 x(5) c(6) 1-1 1 - 1 -1 1-1 1 ~(6) 47)-1-1 1-1 1-1 1 - 1 - - x(7)

-

24

WAI KUEN CHAM

All eight basis vectors of the Walsh transform kernel have the seventh dyadic symmetry with basis vectors W(O), W(2), W(4), and W(6) having even and W(1), W(3), W(5), and W(7) having odd seventh dyadic symmetry. Therefore, we may decompose the order-8 Walsh transform into two order-4 Walsh transforms, i.e., 1 1 - 1 -1 1 - 1 -1 1 1 -1 1 - 1l 1

1 1 - 1 -1 1 - 1 -1 1 - 1I - 1 1 - 1l 1 By recursively applying the dyadic decomposition on the Walsh transform kernels, we may generate a fast computation algorithm for the Walsh transformation. The Walsh transform has all possible dyadic symmetries and so can be decomposed in many different ways to generate all kinds of fast computation algorithms. Figure 2 shows the signal flow diagram of a fast computation algorithm obtained by applying the seventh, third, and first dyadic decompositions on the order-8, -4, and -2 Walsh transform kernels respectively. IV. INTEGERSINUSOIDAL TRANSFORMS A . Definition The term integer sinusoidal transform was first used by Cham and Yip (1991) to represent those integer transforms obtained from the family of sinusoidal transforms proposed by Jain (1 979). Apparently, an integer sinusoidal transform [TI, as indicated by its name, needs to satisfy the following two conditions: 1. It must have similar characteristics to the corresponding sinusoidal transform [@I; and, 2. its kernel elements [ t ( i , j ) lmust be integers. However, for mathematical tractability, we shall require integer sinusoidal transforms [TI to be orthogonal, which implies (T(i)l equals 1 as given by Eq. (5). Obviously, condition (2) and Eq. ( 5 ) contradict each other.

INTEGER SINUSOIDAL TRANSFORMS

x x x

X FIGURE2. A fast computational algorithm generated using the seventh, third, and first dyadic decompositions for the order-8 Walsh transform.

Therefore, we give up condition (2) and have the following definition for an integer transform:

Definition 27. An order-n transform [TI whose (i,j)th kernel element is t ( i , j ) is said to be an integer transform if there exists a real number k(i) such that T(i),the ith row of [ T I , is

..., r(i, n - 1)) = k(i)(e(i,O), e(i, l), ... , e(i, n - l)), (67) where i = 0, 1, . . ., n - 1 and e(i, 0), e(i, I ) , . . . and e(i, n - 1) are integers. (t(i,0 ) , r(i, l),

26

WAI KUEN CHAM

Equation (67) implies that scaling factors [ k ( i ) ]of an integer transform are always greater than 0 and less than one. It also implies that in an integer transform the ratio of any two kernel components in a row is a rational number, i.e., t(i, r) t(i, s)

-

e(i, r) e(i,s) '

e(i, r) and e(i, s) are integers.

The primary objective of deriving an integer transform from a real transform is to simplify the implementation. Can an integer transform as given by Definition 17 achieve this objective? Let's consider the four possible ways to realize a transform: 1. implemented by a dedicated hardware (hardwired approach) via direct matrix multiplication, 2. implemented by a dedicated hardware (hardwired approach) via a fast computation algorithm, 3. implemented by a microprocessor-based system (firmware approach) via a fast computation algorithm, or 4. implemented by a microprocessor-based system (firmware approach) via direct matrix multiplication.

Method 4 is seldom used in practice because this method requires more memory and computation time then does method 3. While direct matrix multiplication is not an effective way to implement a sinusoidal transform in a firmware approach, it is often used in a hardwired approach because of design simplicity and its highly regular structure. If method 1 is used for realization, obviously the use of an integer transform that does not require real number multiplications implies a simpler implementation. If method 2 or 3 is used, then an integer transform needs to have a fast computation algorithm similar or comparable to those of the real transform to maintain its superiority in implementation. For the order-8 and order-16 ICTs, fast computation algorithms have been found (Cham, 1989; Cham and Chan, 1991). They are derived from the orthogonal conditions and the dyadic symmetry existing inside the DCT. We anticipate that fast algorithms for other integer sinusoidal transforms could be derived in a similar way. An order-n integer transform as defined by Definition 17 requires n real multiplication operations for scaling factors [ k ( i ) J .Fortunately, in some applications such as transform coding, the n real multiplication operations can be eliminated and do not cause extra computation. Consider an integer transform [TI. Let [TI = [K"I (69)

27

INTEGER SlNUSOlDAL TRANSFORMS

where [ K ] is a diagonal matrix whose (i, i)th element is the scaling factor k(i) and matrix [El has integer e(i,j) as its (i,j)th element. Therefore, transform [TI can be implemented by the multiplication of the integer matrix [El followed by the multiplication of the real diagonal matrix [ K ] , which requires only n multiplication operations. As [TI is orthogonal, we have [TI-' = [TI' = [E]"K]'.

(70)

Therefore, the inverse of transform [TI can be implemented by a real diagonal matrix multiplication followed by an integer matrix multiplication. In image transform coding, the postmultiplication of [ K ] required in the encoder can be absorbed into the quantization process of transform coefficients, while the premultiplication of [K]'required in the decoder can be absorbed into the decoding process; hence, the real number multiplication can be completely eliminated. We shall give a more detailed description in part C of Section V, using the order-8 ICT as an example. Based on Definition 17, we have the following definition for integer sinusoidal transforms.

Definition 28. An order-n integer transform [TI is said to be an integer sinusoidal transform or an integer version of a sinusoidal transform [@I if [TI satisfies the following four conditions: 1. 2. 3. 4.

[TI is orthogonal. [TI is an integer transform as given by Definition 17. If l@(i,j)l L I@(i, k)l, then It(i,j)( 2 It(i, k) for i, j , k Sign (t(i, j ) ) = sign(@(i,j)) for i, j E [0, n - 1 1 .

E

[0,n - 1 1 .

In transform coding, condition 1 is vital, because it ensures that the quantization errors in the transform domain and the signal domain have the same energy level as given by Eq. (3). In fact, orthogonality is also required in applications such as analysis and filtering. Condition 2 is to enable simple implementation. Conditions 3 and 4 ensure that the new integer transform [TI retains similar structure and so similar performance as the corresponding sinusoidal transform [@I. B. Generation Method

We shall first use the order-8 DST as an example to show how an integer transform can be derived. Let the transform kernel of the order-8 DST be

28

WAI KUEN CHAM

where @(i)s are basis vectors of the transform kernel. The (i,j)th component of the order-8 DST is a(i,j)= g)”’sin(

(i + l ) ( j + 1)n , 9

)

i, j

E [0,7].

(72)

Let components of the transform kernel with the same magnitude be represented by a single variable. For example, kernel components cP(0, 0), W, 71, W, 31, W , 41, W 3 , 11, @(3,6), @(4, I), @(4,6), W6,3), @(6,4),
k(0) (a k(1) (b k(2) (c k(3) (d [TI = k(4) (d k(5) (c k(6) (b k(7) (a

b d

c

d

c

u -a

d

0 -c -c -b b -C b b -c 0 c -c -d c -a -a -b c -d d c

u -a

-C

-C

b -d

0

c

c

c -a -C - a 0 c c -d -C b

Note that as required by condition 4 of Definition 18, variables a, 6, c, and d are positive. Strictly speaking, variables may assume zero value by the definition of positive. However, our search restricts the variables by Eq. (74), which will make the integer transform resemble the sinusoidal transform more. a, b, c, and d > 0. (74) Condition 1 of Definition (18) requires [TI of Eq. (73) to be orthogonal. Table 5 lists the conditions under which any two basis vectors T(i)and T ( j ) are orthogonal. For example, vectors T(0)and T(1) are always orthogonal by Theorem 4 because they both have the seventh dyadic symmetry with one even and the other odd. Vectors T(0) and T(2) are orthogonal when the equation c * ( a+ b - d ) = 0 (75)

29

INTEGER SINUSOIDAL TRANSFORMS

la

2b

1

3

1

3

1

1

'3

1

2

1

3

T(0) T(1)

1

2

1

2

1

T(2)

1

2

1

3

773)

1

3

1

T(4)

1

2

T(5)

1

T(6)

1 is orthogonal because of the seventh dyadic symmetry. 2 is orthogonal is c . (a + 6 - d ) = 0. ' 3 is orthogonal if u * (d - b) - c2 + 6 . d = 0. a

is satisfied. Similarly, vectors T(0) and T(4) are orthogonal when the equation a . ( d - b)-C2+ bd=O

(76)

is satisfied. From Table 5 , we find that Eqs. (75) and (76) are sufficient conditions for the newly formed transform [TI being orthogonal. With inequality (74), Eqs. (75)and (76)can be simplified and become Eqs. (77) and (78). d=a+b (77)

c

= d(a2

+ b2 + a

*

6)

(78)

Condition 2 of Definition 18 requires variables a, b, c, and d to be integers. Condition 3 requires d 1 c 2 b 2 a.

(79)

Integer solutions that satisfy Eqs. (77), (78),and inequality (79)are used to generate the integer DST. The solutions of c and d can be easily found by computer search, using a and b as independent variables. All integer solutions with d I128 are tabulated in Table 6. Note that we may get an additional set of integer DST by representing @(2,2), @(2, 5 ) , @(5,2), and O(5,5) by another variable, say e, whose value is not equal to zero. In this case, Eq. (75)becomes a + b - d + e = 0. The method of finding integer transforms from a sinusoidal transform [O] has been shown using the DST as an example. The procedure of finding

30

WAI KUEN CHAM

TABLE 6 INTEGER SOLUTIONS FOR INTEGER DST a

3 I

5 11 I 13 16 9 32 17 40 11 19 55 40 24

b

5 8 16 24 33 35 39 56 45 63 51 85 80 51

I1 95

C

I 13 19 31 31 43 49 61 61 13 19 91 91 91 103 I09

d 8 15 21 35 40 48 55 65 11

80 91 86 99 112 111 119

an integer sinusoidal transform as given by Definition 18 is summarized as follows: 1. Write down the transform kernel [@I. 2. Represent the kernel components that have the same magnitude by a variable that will assume only positive values. Minus signs should be placed appropriately so as to satisfy condition 4 of Definition 18. 3. Express the kernel of the integer transform [TI in terms of the variables. Each basis vector T(i)should be scaled by a factor k(i) so that T(i) have unity magnitude (e.g., Eq. (73)). 4. Derive the inequality for the variables so as to satisfy condition 3 (e.g., inequality (79)). 5 . Derive the conditions under which any two basis vectors are orthogonal so as to satisfy condition 1 (e.g., Table 5, Eqs. (77) and (78)). 6. Find integer solutions that satisfy the inequality and orthogonal conditions obtained from procedures 4 and 5 respectively (e.g., Table 6). 7. Find scaling factors ( k ( i ) ]so that IT(i)l = 1.

The technique of finding integer sinusoidal transforms from a sinusoidal transform has been described. Cham and Yip (1991) applied the technique on the sinusoidal transform family proposed by Jain (1976) for order 8. In Section C, we shall list the integer sinusoidal transforms thus found.

31

INTEGER SINUSOIDAL TRANSFORMS

C. Examples of Order-8 Integer Sinusoidal Transforms

1. Integer Even Cosine-I Transform or Integer Cosine Transform (ICT) Jain (1976) called the DCT given by Eq. (14) the even cosine-1 transform, which is also called the version I1 cosine transform by Wang (1984). For simplicity, we shall call the integer even cosine-1 transform as integer cosine transform (ICT). As the DCT plays an important role in image coding, so Section V will look into various aspects of this integer transform in details. ( g g g (b, b, b2

(a,

a, -a,

(b, -b3 -b, (g - g -g (b, -b, b3 (a, - a , a, (b3 -b, 6,

[TI =

g g g g g ) b3 -b3 -b, - b , -b,) - a , -ao - a , a, a,) -b2 6 2 bo b3 -b,) g g -g -g g ) 6 , - b , -b3 b, -b2) -a, -a, a, -ao a,) -b, b, - b , b2 -b3)

(79)

Inequalities governing magnitude restrictions: b, 1 b, 2 b2

L

b3

(80)

a, 1 a ,

(81)

From Table 7 , one equation is found to restrict four variables, bo,bl , b ~ , and 6 , . There are no restrictions on a, and a, except inequality ( 8 1 ) . Many

CONDITIONS UNDER

1"

TABLE 7 WHICHANYTwo BASISVECTORSOF AN ICT ARE ORTHOGONAL

2b

1

3c

1

2

1

1

4d

1

1

5

1

2

4 1

5

1

1

5e

1

I

2

4 1 4

1

1

1 is orthogonal because of the seventh dyadic symmetry. 2 is orthogonal because of the third dyadic symmetry. 3 is orthogonal because of the first dyadic symmetry. d 4 is orthogonal if b, . ( b , - b,) - b, . ( b , + b2) = 0. 5 is orthogonal because T,' 7;. = 0. a

32

WAI KUEN CHAM TABLE 8 INTEGER SOLUTIONS FOR ICTs 60

b,

b2

63

5 6 7 8

3 6 4 5 5 8 6

2 3 3 3

1 2 1 2

9 9

10 10 11 11 12 12 13 13 14 14

9

6 1 1 10 7 8 4 8

1

4 4 4 6 5 4 5

3 2 2 1

3

2 3 1 3 2

6

6 5 3 6

2

integer solutions of 6, , b l , b 2 ,and 6 , satisfying the equation and inequality are found. Table 8 lists some of the solutions. We shall denote an ICT by the values of the six variables. For example, ICT(lO,9,6,2,3, 1) refers to the ICT with bo = 10, bl = 9, b2 = 6, b, = 2, a, = 3, and a, = 1 . 2 . Even Cosine-2 Transform

(a b c d (b e h - f (c h - d - b (d - f - b h [TI = (e - c -g a 4 5 ) ( f -a e g k(6) ( g - d a -c - k ( 7 ) (h -g f -e

k(0) k(1) k(2) k(3) k(4)

e -c

-g

f

g

h)

-a -d

-g)

e

a

f)

a g - c -e) -h -b f d) -b d h -c) f h -e b) d -c b -a)

Magnitude restrictions: aibzc?d>e>f>g?h

33

I N T E G E R SINUSOIDAL TRANSFORMS

TABLE 9 CONDITIONS UNDERWHICH ANY Two BASISVECTORS OF EVENCOSINE-2 TRANSFORM ARE ORTHOGONAL

la

2b 3'

3 2 2

3

1

1

3 4 I 2

1 4'

2 4 3

AN

4 2 1 3 3 2 1

1

2 3

INTEGER

T(0) T(1) T(2) T(3) T(4) T(5) T(6)

From Table 9, three equations are found to restrict eight variables. Integer solutions are presented in Table 10.

INTEGER

SOLUTIONS FOR

TABLE 10 INTEGER EVENCOSINE-2 TRANSFORMS

a

b

C

d

e

f

g

h

42 51 62 63

38 48 61 52 76 80 19 16 81 80 86 88 93 84 84 107 85

31 42 49 48 64 74 68 74 14 10 66 16 73 61 76

32 39 41 43 61 67 62

22 21 37 23

19 24 31 26 38 40 31 38 35 40 58 42 51 44 50 61 38

10 I5 21 15 25 23 31 20 33 25 2 34 26 26 10 29 32

4 1 5 1 7 13 6 8

81

81 82 84 84 87 92 93 94 91 104 108 111

81

80

64 12 65

62 66 70 65 68 76 71

41

53 50 44 42 43 54 58 58 37 35 10 25

18

7 1

1 6 1

4 1 1

34

WAI KUEN CHAM

3. Even Sine-2 Transform

[TI =

b

c

d

k(1) (e k(2) (b

f d

f

e - e -f -f - e )

k(3) (g k(4) (c

g -g

a -C

a -d

k(5) (f - e - e k(6) ( d - c

-g

b

d

b

k(0) (a

c

g

g -g

b -d

-g

g

b) -g)

a

e

f -f

c)

e -f)

b -c

b -a -a

- 4 7 ) (g - g

d

u

-C

a)

g -g

d)

g -g)

Magnitude restrictions:

d i c z b r a

fie

(86)

From Table 11, one equation is found to restrict variables a, b, c, and d, but there are no restrictions on variables e, f,g. Solutions can be easily found by solving the equation. Results of the first fifteen sets of integer solutions of a, b, c, and d are tabulated in Table 12.

CONDITIONS UNDER WHICH

TABLE 11 ANYT W O BASIS VECTORS

OF AN INTEGER

EVENSINE-2 TRANSFORM ARE ORTHOGONAL

T(5)

T(6)

2

1

I

4d

3 1

3 1

3

T(1)

T(2)

T(3)

T(4)

la

2b

1 3c 1

1

1 1

2 I 2 1

T(7) 1 4

1

s 1 3 1

1 is orthogonal because of the seventh dyadic symmetry. b 2 is orthogonal if ab + bd + ac - cd = 0 . '3 is orthogonal because of the third dyadic symmetry. 4 is orthogonal because 7'/ . = 0. 5 is orthogonal because of the first dyadic symmetry. a

35

INTEGER SINUSOIDAL TRANSFORMS TABLE 12 INTEGER SOLUTIONS FORINTEGER EVEN SINE-2 TRANSFORMS 4

b

c

d

1 2 1 2

2 3 3 3 4 2 6 5 4 3

3 6 4 5 5 3

5

1

2 2 I 3 3 2 4 3 1 3

6 1

8 9

10 10 11 11 12 12 12 12 13 13

9

6

1 5 7 7 10 7 8

5 5 6

6 5

4 . Even Sine-3 Transform

[TI =

k(0) (a

b

c

d

e

k(1) ( b

e

h

f

c -a

(c

h

k(2) k(3) k(4) k(5)

f

g

-d

h) -g)

a

f) (d f - b - h - a g c -e) (e c - g - a h -b -a d) (f - a - e g - b - d h -c)

k(6) ( g - d - k ( 7 ) (h -g

d - b -g

a

c

f -e

-f

e

h -e

d -c

(87)

b)

b -a)

Magnitude restrictions:

Three equations are found to restrict eight variables. A computer search confirms that an integer solution does not exist with h E [I, 1281 (see Table 13).

36

WAI KUEN CHAM

CONDITIONS UNDER WHICH

TABLE 13 ANY T W O BASIS VECTORS OF AN INTEGER

EVENSINE-3 TRANSFORM ARE ORTHOGONAL

2b 3c

la

3

3 2 2

1 3

1 1

4

1 2

4d

2

4

4

2

3 1

1 3 3

2 3

2 1

1 is orthogonal b 2 is orthogonal c 3 is orthogonal * 4 is orthogonal a

+ +

if ab be + ch + df + ec - fa - gd - hg = 0. if ac bh + cd - db - eg -fe + ga hf = 0. if ad + bf- cb - dh - ea +fg + gc - he = 0. because Ti' * = 0.

+

5 . Odd Cosine-l Transform

[TI

=

k(0) (a

b

c

f

g

h)

k(1) ( b

e

h -g - d - a

-c

-f)

k(2) (c

h -e -a

-f

k(3) (d - g

-a -h

c

e -f

-b)

k(4) (e - d

-f

g -b -h

a)

d

c

e

e -b

4 5 ) (f - a

g

k(6) ( g - c

b -f

k(7) (h -f

d -b

-h

g

h

b

d -c)

d -a

a -c

d)

e)

e -g)

Magnitude restriction: a r b r c r d r e rf r g r h

(90)

From Table 14, three equations are found to restrict eight variables. Integer solutions are presented in Table 15.

37

INTEGER SINUSOIDAL TRANSFORMS TABLE 14 CONDITIONS UNDERWHICH ANYTwo BASISVECTORS OF AN INTEGER ODD ARE ORTHOGONAL COSINE-] TRANSFORM

1"

2b 4d

2 3' 3

3 1 2 I

1

3 3 4 2

4 2

3 2

T(0)

1

1

1 3 2

2 4 1 3

T(2) T(3) T(4)

T(I)

T(5) T(6)

TABLE 15 INTEGER SOLUTIONS FOR INTEGER ODD COSINE1 TRANSFORMS U

b

29 37 58 58 66 74 78 78 87 100 105 110 111 112 113 113 116 116 116 118 119 122

23 30 46 52 48 60 59 77 69 92 89 93 90 111 83 107 92 93 104 113 96 114

C

d

e

f

g

18 28 36 48 46 56 53 75 54 86 88 77 84 90 17 89 72 75 96 99 90 108

17 23 34 43 41 46

15 19 30 30 30 38 36 52 45 68 59 60

13 16 26 23 29 32 33 40 39 57 52

57

48 60

11 13 22 22 18 26 24 28 33 32 31 40 39 48 30 42 44 37 44 42 41 41

50

66 51

77 78 75 69 84 73 85 68 69

86 94 84 96

80 52 79 60 68 60 82 51

81

50

51

63 52 63 46 65 42 66

h 1

1 2 2 6 2 6 14

3 24 17 10

3 16 14 23 4 18 4 26 2 25

38

WAI KUEN CHAM

6. Odd Sine-1 Transform

[TI =

f

k(0) (a

b

c

d

e

k(1) (c

f

h

e

b -a -d

k(2) (e

g

b -c -h - d

4 3 ) (g

c -d

44)

(h - a - g

k(5) ( f - e - a

-f b

a

h

g

-g)

a

k(6) (d - h

e -a -c

k(7) ( b - d

f -h

d)

h -c)

g -f

g -e

f )

b -e)

f -c -e

g -d -b

d)

b)

c -a)

Magnitude restriction: h r g r f r e r d r c r b r a

(92)

As shown in Table 16, three equations have been found to restrict eight variables. A computer search confirms that an integer solution with h E [ l , 1281 does not exist. However, integer solutions may exist for larger values of h. TABLE 16 CONDITIONS UNDERWHICH ANYTWO BASISVECTORS OF AN INTEGER ARE ORTHOGONAL SINE-1TRANSFORM

la

2b

2

3

1

4

4d

3c

1 2 1

3 3

2 1

3

4 2

1

3 2

ODD

3 2 1 2

4 1

3 " 1 is orthogonal if a . (c -f) + b . ( e + f ) - d . (g - e) - h . (g - c) = 0. b 2 is orthogonal if a . ( e + g) + be ( c + g ) - d . (c+f) + h * (f- e) = 0. ' 3 is orthogonal if a (h - b) - c * (f+g) + d * ( b + h) + e * (f+g) = 0. d 4is always orthogonal because T,' * = 0.

INTEGER SINUSOIDAL TRANSFORMS

39

7. Odd Sine-2 Transform

[TI =

k(0) (a

b

c

d

4 1 ) (b

d

f

h -g

k(2) (c

f - h -e

-b

4 3 ) (d

h -e -a

c

c

k(5) (f - e

a

g -d

k(6) (g - c

d

k(7) (h - a

g -b

h)

g

-e - c a

-a)

d

g)

g -f - b )

h -d

k(4) (e - g - b

-f

f

e

a

h -c)

b

h-b

a

(93)

f )

f -c

e)

e -d)

Magnitude restriction: d r e r c i f r b r g r a r h

(94)

As shown in Table 17, three equations are found to restrict eight variables. A computer search confirms that an integer solution with d E [ l , 1281 does not exist. However, integer solutions may exist for larger values of d .

CONDITIONS

T(1) 1"

TABLE 17 UNDERWHICHANYT W O BAsS VECTORSOF S I N E - 2 TRANSFORM ARE ORTHOGONAL

AN INTEGER

T(2)

T(3)

T(4)

T(5)

T(6)

T(7)

2b

3c

4*

1

4 2 3 2

2 2 1 4

4 4 1 2 1 3

1 3 2

4

1

1

4 4 2

" 1 isorthogonalifab+bd+cf+dh-eg-fe-gc-ha= 0. b 2 is orthogonal if ac+ bf - ch - de - eb + f a + g d + hg = 0. ' 3 is always orthogonal because T' . 7; = 0. d 4 is orthogonal is ae - bg - cb + dc + eh -fd + ga + hf = 0.

ODD

40

WAI KUEN CHAM

8. Odd Sine-3 Transform

[TI =

f

k(0) (a

b

c

d

e

k(1) ( h

f

d

b -a

k(2) ( b

e

h - g -d -a

k(3) ( g

c -b

-f

h

k(4) (c

h -e

a

f

g

- c -e c

d -a

(95) -g

-b

k(5) (f - a - g

e -b -h

d

k(6) (d - g

a

h -c

k(7) (e - d

f -c

e

-f

g -b

h

Magnitude restriction:

Table 18 shows that magnitudes of these eight variables are restricted by three equations. Integer solutions with e E [ I , 1281 are presented in Table 19. TABLE 18 CONDITIONS UNDERWHICHANYTwo BASISVECTORSOF AN INTEGERODD SINE-3 TRANSFORM ARE ORTHOGONAL

la

2b

'3

4d

1

4

4 2 3 2

2 2 1 4

4 4

1

1

1

2 3

1

T(0)

3 2

T(1)

1 4

4 2

T(2)

T(3) T(4) T(5) T(6)

41

INTEGER SINUSOIDAL TRANSFORMS TABLE 19 INTEGER SOLUTIONS FOR a

1 1

2 2 6 2 6 14 3 6 16 24 14 17 1

10 3

INTEGER ODD

SINE-3 TRANSFORMS

b

C

d

e

f

g

h

13 16 26 23 29 32 33 40 39 37 49 57

17 23 34 43 41 46 50 66 51 59 63 77 61 78 54 75 69

23 30 46 52 48 60 59 77 69 70 77 92 77 89 74 93 90

29 37 58 58 66 74 78 78 87 90 94 100 I04 105 I10

18 28 36 48 46 56 53 75 54 60 70 86 62 88 58 77 84

15 19 30 30 30 38 36 52 45 42 54 68 52 59 47 60 57

11 13 22 22 18 26 24 28 33 30 28 32 30 31 36 40 39

51

52 46 50 48

110 111

V. INTEGER COSINETRANSFORMS A. Derivation

As described in Section I, the DCT has become an industrial standard in image coding. However, the DCT kernel components are real numbers, hence its implementation is more complicated than that of integer transforms. One early attempt to solve this problem is by Jones, Hein, and Knaver (1978). They found that the order-8 DCT can be approximated using the orthogonal C-matrix transform [ T C M r ] with small performance degradation. The C-matrix transform is computed via the bit-reversed sequency-ordered Walsh transform [W,]and the C-matrix [ TCM] as follows: LTCMTI

=

[TCMI

12

5

[Wl,

(97)

-

13 where

-5

12 12 0 4 3 4 0 12 - 3 -4 3 12 4 - 3 -4 0 12

.

42

WAI KUEN CHAM

As [TCM]is a sparse block diagonal matrix containing only integers 13, 12, 5 , 4, 3, - 3 , -4, and - 5 , and [ W,] contains only +1 and -1, the C-matrix transform can be implemented using simple integer arithmetic. The work was then extended by Srinivasan and Rao (1983) to order 16 and by Kwak, Srinivasan, and Rao (1983) to order 32. On the other hand, Cham (1989) derived a number of order-8 ICTs using the technique described in Section IV. Table 8 lists some of the solutions. The ICTs, in comparison with the order-8 CMT, have three advantages: 1. While there is only one CMT, there are many ICTs, which have different complexity and performance. This provides an engineer with the freedom to trade off performance for simple implementation. 2. The ICT has a fast computation algorithm similar to that of the DCT while the CMT does not. 3. Some ICTs whose structure is simpler than the CMT, for example ICT(lO,9,6,2,3, l), have performance better than the CMT and in fact very close to the DCT. Cham and Chan (1991) and Koh and Huang (1991) found a number of order-16 ICTs. We shall explain the derivation using Eq. (39, which expresses the order-n DCT (i.e., the version I1 DCT) in terms of the order-in version IV DCT and the DCT.

For example

ien n = 8, Eq. (35) implies that the order-8 DCT

[C1*(8)]=

can be expressed in terms of

INTEGER SINUSOIDAL TRANSFORMS

From Eqs. (99), (loo), and (101), we can see that 1. [C"(4)] is expanded with the even seventh dyadic symmetry and put in the even rows of [C"(8)]; and 2. [CIV(4)]is expanded with the odd seventh dyadic symmetry and put in the odd rows of [C"(8)]. The above two observations and Theorem 4 imply that any even basis vector of [C1'(8)] is always orthogonal to any of its odd basis vector (which is also reflected in condition 1 in Table 7). Therefore, [C"(8)] obtained from [C"(4)] and [CIV(4)],using Eq. ( 3 9 , are always orthogonal. In the same way, an orthogonal order-8 ICT, [TICT(8)],can be obtained from [&T(4)] and [TIV(4)], which are the order-4 ICT and integer version IV DCT, respectively.

Similarly, the order-16 ICT is

As we already have order-8 ICTs (Table 8), so an order-16 ICT can be obtained if we have an order-8 integer version IV DCT. The order-8 version IV DCT can be expressed as

[C"(8)] =

44

WAI KUEN CHAM

TABLE 20 THE SOLUTIONS THAT SATISFY EQS. (105), (106), AND (107) AND INEQUALITY (108) FOR c, < 135 CO

CI

c2

134 128 121 121 120 120 117 108 94 87 81 81 62 42

119 124 119 111 114 108 106 107 93 80 80 76 61 38

118 119 107 105 103 104 90 81 73 70 74 64 49 37

c3

c4

c5

c6

'1

98 100 97 89 94 85 82 76 70

70 88 79 69 68 69 59 70 58 43 53 41 37 22

69 67 68 63 57 52 50 61 51 40 40 38 31 19

11 22 19 15 34 32 42 29 26 25 23 25 21 10

10 12 15 9 14 2 1 1 6 7 13 7 5 4

65

67 61 47 32

where kd3(i) is the scaling factor such that the ith basis vector is of unity magnitude. An integer version IV DCT generated from [CrV(8)] needs to satisfy all four conditions of Definition 18. To satisfy condition 1, which requires that basis vectors are orthogonal to each other, Eqs. (105) to (107) need to be satisfied.

+ cIc4 + czc7 = c3c5 + cZc4 + cOc5 + c3c6 + cgc7 cOc2 + C1c.I + c4c5 + COCg + c5c7 = c2c3 + C l c 3 + c4cg cOc3 + cOc4 + c3c7 + c5c6 = C1c.j + C l c z + c2c6 + c4c7 COC,

(105)

(106) (107)

Condition 2 requires that co, . .., cl0and cll are integers. Conditions 3 and 4 require that CO 2 c1 2 c2 2 c3 2

c4 2 c5 2

c6

2

c7

> 0.

(108)

The solutions that satisfy Eqs. (105), (106), (107), and inequality (108) for co less than 135 are not many and are listed in Table 20. B. Performance of ICTs

In this section, we wish to answer two questions: 1. Among the many ICTs, which ones have better performance? 2. How well do these better ICTs perform in comparison with the DCT?

INTEGER SINUSOIDAL TRANSFORMS

45

To answer the first question, we shall use a criterion called transform efficiency. There are other criteria such as basis restrictions MSE (Jain, 1981), maximum reducible bit (Wang, 1986), and residue correlation (Hamidi and Peral, 1976). As results suggested by these criteria are similar, we shall not go through the same comparison procedure using different criteria. To answer the second question, we shall use basis restriction MSE based on a Markov model and real images. In transform coding of images, transforms are used to convert highly correlated signals into coefficients of low correlation. Such decorrelation ability may be measured by transform efficiency E d , which is defined on the first-order Markov process having covariance matrix [R,], given by Eq. (11). A larger Ed implies a higher decorrelation ability. The optimal KLT, which converts signals into completely uncorrelated coefficients, has transform efficiency equal to 100% for all adjacent element correlation p. Let the n-dimensional vector X be a sample from the one-dimensional, zero-mean, iunit-variance first-order Markov process. The transform efficiency of a transform [TI, which converts vector X into Y , i.e.,

Y

=

[TIX,

is defined as

where

[R,]

=

E [ Y *Y'] = E [ [ T ] X X ' [ T ] '=] [ T ] [ R , ] [ T ] ' .

(111)

Cham (1989) found the set of (a,, a1, b, , 6 , , b, ,b,) that gives the order-8 ICT the highest transform efficiency for 6, less than or equal to 255, and (a,, a,) equal to (1, 0), (4, l), (3, l), (2, l ) , and (1, 1) by means of exhaustive search. Table 21 lists the five order-8 ICTs that have the highest transform efficiencies for p equal to 0.9 and 6, less than or equal to 255. It can be seen that the transform efficiencies of these five ICTs are very close and higher than the order-8 DCT. The search also found that (00,

all =

( 3 9 1 )

usually gives a higher transform efficiency for the same (b,, bl , b, , b J . The implementation complexity of an ICT depends on the magnitude of the variable b,. Therefore, a search was also carried out to find the ICTs that have the highest transform efficiency with bo limited to different values. As given by Table 22, ICT(10,9,6,2,3, l), which requires only four bits for representation of its kernel components, has transform efficiency very close to the best ICT(230,201, 134,46, 3, l), which requires eight bits.

46

WAI KUEN CHAM TABLE 21 THETWELVE ORDER-8 ICTS THATHAVETHE HIGHEST TRANSFORM EFFICIENCIES FOR p = 0.9 AND b, 5 255 Transform efficiency 90.2 90.2 90.2 90.2 90.2 89.8 86.8 85.8

77.1

Transform ICT(230,201, 134,46,3, 1) ICT(175, 153, 102,35,3,1) ICT(I20, 105,70,24,3, 1) ICT(I85, 162, 108,37,3, 1) ICT(250.219. 146,50,3, 1) DCT CMT Slant transform Walsh transform

Figure 3 plots transform efficiencies of the DCT, the Walsh transform, the CMT, ICT(230, 201, 134, 46, 3, l), ICT(55, 48, 32, 1 1 , 3, l), and ICT(10, 9, 6, 2, 3, 1) against adjacent element correlation p. It shows that transform efficiencies of the DCT and the ICTs are very close to each other and are always better than those of the CMT and the Walsh transform for adjacent element coefficient between 0.1 and 0.9. The effectiveness of a transform in image coding and other applications, such as filtering and analysis, depends on the transform’s capability to pack TABLE 22 OF THE DCT, CMT, SLANT TRANSFORM, WALSH TRANSFORM EFFICIENCY AND THE SEVEN ICTs THATHAVETHE HIGHEST TRANSFORM TRANSFORM, FOR b, I 255, 127, 63, 31, 15, 7, 3 AND p = 0.9 EFFICIENCY Magnitude restriction

bo 5 255 b O s 127 b, 5 63 b, 5 31 b O s 15 b, 5 7 b, 5 3

Transform efficiency

Transform DCT CMT Slant transform Walsh transform ICT(230 201 134 46 3 ICT(I20 105 70 24 3 ICT( 55 48 32 11 3 ICT( 10 9 6 2 3 ICT( 10 9 6 2 3 ICT( 6 6 3 2 3 ICT( 3 2 1 1 3

89.8 86.8 85.8

I) 1)

1) 1)

2) 1) 1)

77.1 90.2 90.2 90.2 90.2 90.2 83.2 80.0

47

INTEGER SINUSOIDAL TRANSFORMS 96 I

I

70

-

68

-

66

1

I

02

03

0 ICT(10.9.6.2)

+

I

I

I

0 1

I

3 1 0 5 06 a d j o c e i r e l e m e n t correlation

ICT(55.48.32.1 1 )

0 ICT(230.201.134.46)

I

0C DCT

0 8

x

00 Wokh

'J CMT

FIGURE3. Transform efficiency for different values of adjacent element correlation. +, ICT(55,48,32, 11); 0 , ICT(230,201, 134,46); A, DCT; X , Walsh;

0, ICTIO, 9 , 6 , 2 ) ; V, CMT.

energy into a few transform coefficients. Such capability can be measured directly by means of basis restriction MSE (Jain, 1981). Consider a twodimensional zero-mean nonseparable isotropic Markov process with covariance function Rx(i,j ; p , q) = E [ x ( i , j )* x ( p , q)] = pv'(i-p)2+G-q)2,

(1 12)

where p is the adjacent element correlation in vertical and horizontal directions. Let the n x n matrix [XI be a sample of the Markov process. Suppose [XI is transformed into [ C ] by transform [TI using Eq. (lo), i.e., [ C ] = [TI ' [XI ' [TI'.

Let the (i,j)th elements of [XI and [ C ] be x(i, j ) and c ( i , j ) respectively. The covariance function of [ C ] is

Rc(u,U ; r, S ) = E [ c ( ~U), *

C(T, s)]

1 1 1 1 R , ( i , j ; p , q ) . T(u,i )

=

i *

j

p

q

T(U,j)* T(r,p)' T(S,4).

48

WAI KUEN CHAM

Hence, we can obtain the variance of c(u, u) from R x ( i , j ; p q) , because

a,(u,u)2 = R,(u, u; u, u). Let Q be the set containing N1 index pairs (u, u) corresponding to the largest N 1 a&, v)*. The basis restriction MSE is defined as

e(M) = 1 -

[c c

oc(u,u)2

u,ven

I.c c "

Oc(u, u)z

].

(113)

Table 23 shows comparisons of basis restrictions MSE of the KLT, the DCT, the Walsh transform, the CMT, ICT(230,201,134,46,3, 1) and ICT(lO,9,6,2, 3, 1) for p equal to 0.95. The results show that basis restriction MSEs of the two ICTs, KLT, and the DCT are very close and always smaller than those of the CMT and the Walsh transform, The basis restriction MSE of the KLT is slightly smaller than that of the DCT, which in turn is slightly smaller than those of the two ICTs. An order-16 ICT is given by Eq. (103) as

Hence, an integer version IV DCT given by a solution for ( c i ) needs to combine with an order-8 ICT to form an order-16 ICT. We found that the transform efficiency of an order-16 ICT does not vary significantly with the order-8 ICT if the order-8 ICT has a high transform efficiency. As TABLE 23 THE BASISRESTRICTION MSE OF THE KLT, THE DCT, THE WALSHTRANSFORM, THE CMT, ICT(lO,9,6,2, 3, I), AND ICT(230,201, 134.46, 3, 1) FOR ADJACENT ELEMENT CORRELATION EQUAL TO 0.95

Number of coefficients retained

KLT

DCT

ICT(230,201, 134.46.3,l)

ICT(lO,9, 6,2,3,1)

CMT

2 6

0.1372 0.0567

10

0.0406

14 18 22 26 30 34

0.0320 0.0263 0.0221 0.0189 0.0160 0.0136

0.1381 0.0572 0.0409 0.0322 0.0264 0.0222 0.0189 0.0160 0.0136

0.1381 0.0573 0.0410 0.0323 0.0266 0.0223 0.0190 0.0162 0.0137

0.13821 0.0573 0.0410 0.0323 0.0266 0.0224 0.0190 0.0162 0.0137

0.1387 0.0587 0.0431 0.0348 0.0287 0.0238 0.0198 0.0165 0.0140

Walsh transform 0.1468 0.0785 0.0541

0.0441 0.0361 0.0300 0.025 1 0.0205 0.0170

49

INTEGER SINUSOIDAL TRANSFORMS TABLE 24 ORDER-16 ICTS THATHAVETHE HIGHEST TRANSFORM EFFICIENCY FOR c, 5 63 AND c, 5 127 ~

~~

~~

Transforms

Efficiency

DCT ICT(8I 80 74 67 53 40 23 13) c,, 5 63 ICT(42 38 37 32 22 19 10 4)

88.5 87.5 86.2

c,, 5 127

ICT(10,9,6,2,3,1) has both simple structure and high transform efficiency, it is used as the GCT(8)in Eq. (103) for generation of an order-16 ICT that will be denoted as ICT(c,, c, ,c,, c3,c, ,c5, c,, c,). The order-16 ICTs that have the highest transform efficiency for p = 0.9 and c,, less than 63 and 127 are given in Table 24.

C. Implementation of ICT(10,9, 6,2,3, 1) A major application of ICT is transform coding of image data. In this section, we shall answer two questions: 1. How do we implement [ K ] without real-number multiplication

operations in a transform coding system? 2. What are the optimal values of the scaling factors in [K]?

As a transform order equal to 8 seems to be a good choice in consideration of implementation complexity and capability to adapt to local statistics, nearly all recent transform coding standards and proposals use transform order equal to 8. Therefore, we shall only consider order-8 ICT in this section. An order-8 ICT can be expressed as the product of [K] and [El as given by Eq. (69) as follows:

IKI

=

I

..

50

WAI KUEN CHAM

g

g

bo

b,

g

g

g

g

“1

g

b3 -b3 -bz - b , -bo a, - a , -ao -ao - a , a, a,

a.

bz

[K] is a diagonal matrix containing real scaling factors [kb3(i)) and [El is a matrix containing only integers. Consider a transform coding system that uses ICT for transformation. From Eqs. (69) and (70), we have and where C, is the quantized version of C. Let F and G, be vectors that relate to C and C, as follows: c = [KIF, (1 18) G, = [KIC,;

(1 19)

and letf(i) and g,(i) be their ith elements respectively. Equations (1 16) and (118) imply that F = [EIX. ( 120) Equations (1 17) and (119) imply that X, = [EI‘G,.

(121)

Figure 4 shows the block diagram of a transform coding system using ICT. Each transform coefficient c ( i ) is quantized by a different quantizer. Let the quantizer designed for coefficient c ( i ) be an m-level quantizer X

---- -F

+

L

X

4

C

[Kl

[El A

-

-

cq

Q u a n t i z e r e H B i t Aseigner

-

Gq

4 ‘ [Elt

t+

C

b i t etream

to channel

b i t stream

q

m’HIK1 Decoder

+

- 1

from Channel

FIGURE4. Block diagram of a transform coding system using ICT.

51

INTEGER SINUSOIDAL TRANSFORMS

F

X

n~

t-f

U

X

¶

1-

%

Quantizere scaled by [K1

[El I

Bit Aesigner 1

-

G

9 u

4 ' [Elt

Decoder scaled by [KI

I

bit stream to channel

bit stream

%

q

+

i-,

from

channel

I

FIGURE5. Block diagram of a transform coding system using ICT.

characterized by the transfer function W d i , j ) , l , ( i , j N , j = 1,

..., m )

(122)

where ld (i,j ) is a decision level and l,(i, j ) is the corresponding quantization level. The bit assigner is to represent the quantized coefficient c,(i) by a bit pattern that will be transmitted across the channel. The bit pattern for each quantization level may be determined using techniques such as Huffman coding. At the receiver, the bit pattern will be decoded into the corresponding quantization level fq(i,j ) by the decoder. The two multiplication operations of ( K ]as shown in Fig. 4 can be incorporated into the quantizer at the transmitter and the decoder at receiver as shown in Fig. 5 . The multiplication of [K]to convert F into C by Eq. (1 18) can be incorporated into the quantization process if the transfer function of the quantizer is changed from (122) to

[(-

, j = 1,

..., m

1.

At the receiver, the multiplication of [ K ]to convert C, into G, by Eq. (1 19) can be incorporated into the decoder by simply changing the decoder output lq(i,j) to kb3(i) l q ( i , j ) for i = 0, 1, ...,7 a n d j = 1,2, ..., m. Another point that needs to be considered in the implementation of an order-8 ICT is the determination of values for g, ao, and a , , which in turn determines the values of [ k b 3 ( i ) )We . need to find a set of (kb3(i)) that results in the minimum rounding error in ( f ( i ) ]and ( g ( i ) ] .Suppose that the image vector X has element x(i) varying from - 128 to 127, requiring eight bits for representation. The maximum magnitude of coefficient f ( i ) can be determined from Eq. (1 15) and is given in Table 25. In the case of ICT(10,9,6,2,3, 1) with g equal to one, we have

-

256(b,

+ b, + bt + b,) = 6912; 512(a, + al) = 2048; 1024g = 1024.

(124) (125) ( 126)

52

WAI KUEN CHAM

TABLE 25

THE MAXIMUM MAGNITUDE OF COEFFICIENTSf ( i ) FOR SIGNAL ELEMENT x(i) IN THE RANGEf - 128,127) Maximum Magnitude

f(i)

1024g 256(bo + bl + b, 512(00 + U l ) 256(b, + b , + b, 1024g 256(bo + b, + b2 512(U0 + U l ) 256(bo + b1 + b2

+ 6,) + b,) + b,)

+ b,)

Let the weighting of the least significant bit (LSB) for the representation of coefficients { f ( i ) ]be LF. If each coefficient f ( i ) is represented by 14 bits (including the sign bit), then LF is zero, and each f(i) can be represented exactly. If f ( i ) is represented by fewer than 14 bits, then a rounding error will result. As the maximum magnitude of f(0)and f(4) is 1024, so their rounding errors can be minimized by assigning a larger value to g under the constraint 1024g s 8192. Therefore, g should be equal to eight. Similarly, the maximum magnitude of f(2) and f(6) is 2048; rounding errors can be minimized by assigning larger values to a, and a, under the constraints 512(Uo + a l ) I8192

and

ao/al = 3.

Therefore, (ao,a,) should be equal to (12,4). However, as the implementation of (9,3) is simpler because bl also equals 9, so we suggest the use of (9,3) instead of (12,4). The scaling factors are therefore equal to kbO(0) = kb3(4) = 1/m; kb3(1) = kb3(3) = kb3(5) = kb3(7) = 11-

(127) 1.07628kb,(O);

kb3(2) = kb3(6) = 1 / & 6 = 1.19257kb3(0).

(128) (129)

D. Fixed-Point Error Performance The DCT kernel components are irrational numbers, and so are the transform coefficients. These components and coefficients can be represented accurately in floating point form, whose processing, however, requires expensive floating point processor. Hence, in practice, components and

53

INTEGER SINUSOIDAL TRANSFORMS

X

r'

'cr

1-4 -I

I

[TIt

[TI

- u FIGURE6. Errors in C, and X,,are due to finite bit representation of C and X .

coefficients are usually represented in binary form, thus rounding or truncation errors will be introduced into coefficients and pixels. In Fig. 6, the error in C, is due to finite bit representation of C, and the error in X,, is due to the error propagated from C , . This section will estimate the mean square value of the rounding errors in X,, for the one-dimensional order-8 DCT and ICT (Wu and Cham, 1990). We shall assume that the DCT kernel components and the ICT scaling factors are perfectly represented, and so no rounding or truncation error is due to the transform kernel. Let [C"(8)] and [T,,,(8)] be such DCT and ICT kernels respectively, and let X be the image vector with element x ( i ) varying from - 128 to 127, thus requiring eight bits (including a sign bit) for representation. Therefore, the most significant bit (MSB) weight of x ( i ) is 6 and L X , which is the LSB, weighs 0. 1 . Rounding Errors in a DCT System

The one-dimensional DCT coefficient vector C is C = [C"(8)]X.

(130)

As a finite number of bits is assigned to represent c(i), which is real, a rounding error e,,(i) will result. Let the rounded C be C,, and its ith component is

c,(i)

=

c(i) + e,,(i).

(131)

Since c(i) is usually much larger than ecr(i),e,,(i) can be assumed to be uniformly distributed within [-2Lc-', 2Lc-'1, where LC is the LSB weighting of c(i). The MSE due to rounding of c ( i ) is therefore

Note that the maximum magnitude of the order-8 DCT coefficients is 363 when i equals zero. The MSB weight of the DCT coefficient is thus eight, and the bit length of (c(i)l is L,dcr =

8 - Lc

+ 1.

(133)

54

WAI KUEN CHAM

As shown in Fig. 6, C, is then inversely transformed to the reconstructed image vector X,, = [C"(8)]'Cr. (134) Suppose xcr(i),the ith element of X,, is represented with the LSB weight equal to zero, thus no new rounding error is introduced. The error residing within Xcr(i)is (135) excr(i)= xCAO - ~ ( 0 , which is solely due to the propagation of rounding error ecr(i).By the central limit theorem, we may assume that the distribution of error excr(i)is Gaussian. Also, we assume that (excr(i)] are the same for i = 0,2, ..., 7 and has the same variance u:~,.Hence, we shall denote excr(i)simply as ex,, and

As LX is equal to zero, errors of magnitude less than 0.5 may be neglected. The MSE is therefore equal to

As [C"(8)] is orthogonal, by Eq. (3), U& is equal to E:dct, whose value is given by Eq. (132). Table 26 shows the values of E,,, as a function of the bit length for Ic(i)l. The relation between E:cr and E:dct is also given.

2. Rounding Errors in an ICT(I0, 9, 6, 2, 9, 3) System In this section, we shall estimate the rounding error of ICT(lO,9,6,2,9, 3) with g equal to 8 and (uo,a l ) equal to (9,3) as described in Section C . We shall estimate the MSE between X and Xfr in a system as shown in Fig. 7. E,c,

AS A

TABLE 26 FUNCTION OF E:d,, , L c , AND BIT LENGTH FOR ( c ( i ) (

Bit length for Ic(i)l

LC

5

4

6 I 8

3 2 1 0

9 10 11

-1 -2

0.997338E,2,,, 0.979582E,2,,, 0.861385E:dcf 0.391625E,2d,, 0.007383E:dc, 0

21.3 5.3 1.3 0.28 0.033 0.0001 0

55

INTEGER SINUSOIDAL TRANSFORMS

X

Fr n 4[El [Kl

‘r

1i-

1 - 1 [Kl u U

u

xf f

Gr

n

[El

I----

U

FIGURE7. Errors in F, and Xu,are due to finite bit representation of F and X.

The maximum magnitudes of f ( i ) and g ( i ) are 8192 and 16, respectively, when i = 0. Therefore, the MSB weighting off(i) and g ( i ) are 12 and 3. Let the LSB weighting off(i) and g ( i ) be LF and LG respectively. The bit length for If(i)l and Ig(i)l are therefore

+ 1, 3 - LG + 1 .

Lfi,, = 12 - LF

(137)

Lnict= (138) Obviously, f ( i ) , which is an integer, will have no rounding error if LF is equal to zero or Lfic,is equal to 13. In practice, we would like a shorter Lfict, which means that LF > 0. In this case, a uniformly distributed error efr(i)is introduced:

=o

for LF

> 0,

for LF

I0.

With no quantization error introduced into C, as shown in Fig. 7, Eq. (1 19) becomes G = [KIC. Equations (118) and (141) imply that c(i) = kb3(i)f(i)

i = 0, 1,

...,7,

g ( i ) = kb3(i12f(i)

i

...,7.

and =

0 , 1,

The values of (kb3(i)2)can be derived from Eqs. (127) to (129). kb3(0)’ = kb3(4)’ = 1/512 kb3(1)’ = kb3(3)’

=

kb3(5)’

=

2-’

= kb3(7)’

= 1 / 4 2 = 1.158 x

kb3(2)2 = /~b3(6)~ = 11360 = 1.422 x 2-’

56

WAI KUEN CHAM

What is the minimum bit length for g ( i ) so that no new rounding error will be introduced? As given by Eqs. (144) to (146), all scaling factors (kb3(i))are equal to or larger than T9,so all (f,(i))can be represented exactly by (g,(i)] if LG ILF - 9. (147) Equation (147) together with Eqs. (137) and (138) imply that no new rounding error will be introduced to g ( i ) if g ( i ) is represented using a bit length not shorter than that of f ( i ) . Therefore, from Eq. (143), the MSE in g ( i ) is Bdr(i)2= kb3(i)4Ej. (148) G, is then inversely transformed by [El' to the reconstructed image vector xfr

( 149)

= [EI'Gr

whose ith element, xf,(i), is represented with LX equal to zero. The error residing within xf,(i) is eqr(i) = xfr(i) - ~ ( 0 , (150) which is solely due to rounding errors in F,. Truncation error e& is of Gaussian distribution by the central limit theorem, and its mean square value can be estimated using Eq. (136). In the case of the DCT, the mean square value of the rounding error in c ( i ) is Ecdcr,which is the same for all i. In the case of ICT, the mean square value of the error in c(i), which is solely due to the rounding error in f ( i ) , varies with i and will be called Ecict(i).As given by Eq. (142), it is equal to

Ecict(i)2= kb3(i)2Ei, i = 0, 1 , The Ecict(i)2averaged over i is therefore 1

Ecic, = -

8 i

..., 7.

(151)

Ecicf(i)2= 1.1845 x ~ ~ E f 2 , .

We shall compare E&, under the condition that ICT transform coefficients f ( i ) and DCT transform coefficients c(i) are represented using the same bit length. From Eqs. (133) and (137), we have LF = LC

+ 4.

(153) From Eqs. (140), (152), and (153), E&', the mean square value of the rounding error in ICT transform coefficient c(i), is

E&,

= 1.1845 X

X

1 12

-22Lc+8= 0.5923 X E:dcr,

(154)

which is about 60% of the Ecdcr. Therefore, we may conclude that ICT(10,9,6,2,9,3)has better fixed-point error performance than the DCT.

INTEGER SINUSOIDAL TRANSFORMS

57

E. Fast Computation Algorithm

The order-8 ICT can be computed using a fast computational algorithm similar to the one developed by Chen, Smith, and Fralick (1977) for the DCT. Consider the transformation

F = [EIX. (155) As given by Eq. (99), even and odd basis vectors of the order-8 ICT have respectively even and odd seventh dyadic symmetry. By Theorem 16, Eq. (155) can be decomposed into two order-4 transformations as follows:

I;:[

f(1)

f(7)

[i: 1:; -:: -::][$; I:;] b*

=

b, - b2

b2

x(0) - x(7)

bl -bo

(157)

~(3) ~(4)

The even and odd basis vectors of the transform kernel in Eq. (156) have even and odd third dyadic symmetry and so can be decomposed again. The even and odd transforms of the third dyadic symmetry decomposition are resDectivelv

On the other hand, basis vectors of the kernel in Eq. (157) do not have dyadic symmetry. To further decompose the transform, we need to use the orthogonal condition given in Table 7, i.e., *

( b , + b2) = 0.

(1 58)

58

x

WAI KUEN CHAM

x

f (0)/g

f ( 4 1 /g

-a0

f(6)

x X X

b3

FIGURE8. A fast computational algorithm for the order-8 ICT.

Equations (158)’ (160), and (161) imply that

Figure 8 shows the signal flow diagram of the fast computational algorithm given by Eqs. (156), (157), and (164) to (167).

INTEGER SINUSOIDAL TRANSFORMS

59

REFERENCES Ahmed, N., Rao, K. R., and Abdussattar, A. L. (1971). “BIFORE or Hadarmard Transform,” IEEE Trans. Audio Electroacoust. AU-19, 225-234. Ahmed, N., Schreiber, H. H., and Lopresti, P. V . (1973). “On Notation and Definition of Terms Related to a Class of Complete Orthogonal Functions,” IEEE Trans. EMC-15, 75-80. Ahmed, N., Natarajan, T., and Rao, K. R. (1974). “Discrete Cosine Transform,” IEEE Trans. Computers 90-93. Andrews, H. C., and Patterson, C. L. (1976). ‘‘Singular Value Decomposition lmage Coding,” IEEE Trans. Comm. 425-532. Andrews, H. C., and Pratt, W. K. (1968). “Fourier Transform Coding Images,” Proceeding of Hawaii International Conference on System Sciences 677-679. Berger, T. (1971). Rate Distortion Theory: A Mathematical Basis for Data Compression, Prentice-Hall, Englewood Cliffs, New Jersey. Blachman, N. M. (1974). “Sinusoids versus Walsh Functions,” Proc. IEEE 62(3), 346-354. Cham, W. K., and Clarke, R. J . (1986). “Application of the Principle of Dyadic Symmetry to the Generation of the Orthogonal Transforms,” IEE Proceedings, Part F 133(3), 264-270. Cham, W. K . , and Clarke, R. J . (1987). “Dyadic Symmetry and Walsh Matrices,” IEE Proceedings, Part F 134(2), 141-145. Cham, W. K. (1989). “Development of Integer Cosine Transforms by the Principle of Dyadic Symmetry,” IEE Proceedings, Part I 136, 276-282. Cham, W. K., and Chan, Y. T. (1991). “An Order-16 Integer Cosine Transform,” IEEE Trans, on Signal Processing 39(5), 1205-1208. Cham, W. K . , and Yip, P. C. (1991). “Integer Sinusoidal Transforms for lmage Processing,” International Journal of Electronics 70(6), 1015-1030. Chen, W., and Smith, C. H. (1977). “Adaptive Coding of Monochrome and Color Images,” IEEE Trans. on Comm. COM-25(11), 1285-1292. Chen, W., Smith, C. H., and Fralick, S. C. (1977). “A Fast Computational Algorithm for the DCT,” IEEE Trans. on Comm COM-25(9), 1004-1009. Clarke, R. J . (1981). “Relation Between the Karhunen-Loeve and Cosine Transforms,” IEE Proc. 128(6), 359-360. Enomoto, H.. and Shibata, K. (1971). “Orthogonal Transform Coding System for Television Signals,” IEEE Trans. Electromagn. Comp. EMC-13, 11-17, Farrelle, P., and Jain, A. K. (1986). “Recursive Block Coding-A New Approach to Transform Coding,” IEEE Trans. on Comm. COM-34(2), 161-179. Fino, B. J., and Algazi, V. R. (1974). “Slant Haar Transform,” Proc. IEEE 62, 653-654. Flickner, M. D., and Ahmed, N. (1982). “Some Considerations of the Discrete Cosine Transform,” proceedings of 16th Asilomar Conf. on Circuits, Systems and Computers, Pacific Grove, CA, 295-299. Hamidi, M., and Peral, J. (1976). “Comparison of the Cosine and Fourier Transforms of Markov-l Signals,” IEEE Trans. ASSP-24, 428-429. Haralick, R . M., and Shanmugam, K. (1974). “Comparative Study of a Discrete Linear Basis for Image Data Compression,” IEEE Trans. on Systems, Man and Cybernetics SMC-4(1), 16-27. Harmuth, H. F. (1968). “A Generalized Concept of Frequency and Some Applications,” IEEE Trans. on Information Theory IT-14(3), 375-382. Harmuth, H. F. (1977). “Sequency Theory Foundations and Applications,” Advances in Electronics and Electron Physics. New York: Academic Press. Haarr (1910). Mathematische Annalen 69, 331-371.

60

WAI KUEN CHAM

Jain. A. K. (1976). “A Fast KL Transform for a Class of Random Processes,” IEEE Trans. on Computers 24, 1023-1029. Jain, A. K. (1979). “A Sinusoidal Family of Unitary Transforms,” IEEE Trans. on Pattern Analysis and Machine Intelligence PAMI-1(4), 356-365. Jain, A. K . (1981). “Advances in Mathematical Models for Image Processing,” Proc. IEEE 69(5), 502-528. Jain, A. K. (1989). Fundamental of Digital Image Processing. Prentice-Hall, Englewood Cliffs, New Jersey. Jones, H. W., Hein, D. N., and Knauer, S. C. (1978). “The Karhunen-Loeve Discrete Cosine and Related Transforms Obtained via the Hadamard Transform,” Proc. of the International Telemetering Conf., Los Angeles 14, 87-98. Kekre, H. B., and Solanki, J. K. (1977). “Modified Slant and Modified Slant Haar Transforms for Image Data Compression,” Compt. and Electr. Eng 4(3), 199-206. Kitajima, H. (1980). “A Symmetry Cosine Transform,” IEEE Trans. on Computers C-29(4), 1980. Koh. S. N., and Huang. S. J. (1991). “Development of Order-16 Integer Transform.” Signal Processing 24(3), 283-289. Kwak, H. S., Srinivasan, R.,and Rao, K. R. (1983). “C-Matrix Transform,” IEEE Trans. ASSP31(5), 1304- 1307. Larsen, R. D., and Madych, W. R. (1976). “Walsh-like Expansions and Hadamard Matrices, Orthogonal System Generation,’’ IEEE Trans. ASSP-24(1), 71-75. Lo, K. T., and Cham, W. K. (1990). “Image Coding Using Weighted Cosine Transform,” Proceeding of the TENCOW90 on Computer and Communication System, Sept. 1990, 464-468. Meiri. A. Z., and Yudilevich, E. (1981), “A Pinned Sine Transform Image Coder,” IEEE Trans. on Comm. COM-29(12), 1728-1735. Pratt, W. K., Kane, J. and Andrews, H. C. (1969). “Hadamard Transform Image Coding,” PrOC. IEEE 57(1), 58-68. Pratt, W. K., Welch, L. R., and Chen, W. (1972). “Slant Transform for Image Coding,” Proc. Sym. Application of Walsh Functions. Pratt, W. K., Chen, W.H., and Welch, L. R. (1974). “Slant Transform Image Coding,” IEEE Trans. Comm. COM-22(8), 1075-1093. Proc. Symposium Applications of Walsh Functions, Washington, D.C. (1970). Proc. Symposium Applications of Walsh Functions, Washington, D.C. (1971). Proc. Symposium Applications of Walsh Functions, Washington, D.C. (1972). Proc. Symposium Applications of Walsh Functions, Washington, D.C. (1973). Rao, K. R., and Yip, P. (1990). Discrete Cosine Transform. Boston: Academic Press. Ray, W. D., and Driver, R. M. (1970). “Further Decomposition of the Karhunen-Loeve Series Representation of a Stationary Random Process,” IEEE Trans. In$ Theory lT-16, 663-668. Robinson, 0.S. (1972). “Logical Convolution and Discrete Walsh and Fourier Power Spectra,” IEEE Trans. Audio Electroacoust. 271-280. Shanmugam, K. S. (1975). “Comments on ‘Discrete Cosine Transform,”’ IEEE Trans. on Computers C-24(7), 759. Srinivasan, R., and Rao, K. R. (1983). “An Approximation to the Discrete Cosine Transform for N = 16”” Signal Processing V, pp. 81-85. Walsh, J. L. (1923). “A Closed Set of Orthogonal Functions,” American . I of .Mathematics 45, 5-24. Wang, L., and Goldberg, M. (1988). “Progressive Image Transmission by Transform Coefficient Residual Error Quantization,” IEEE Trans. on Comm. 36(1), 75-87.

INTEGER SINUSOIDAL TRANSFORMS

61

Wang, Z. (1984). “Fast Algorithms for the Discrete W Transform and for the Discrete Fourier Transform,” ASSP-32(4), 803-816. Wang, Z. (1986). “The Phase Shift Cosine Transform,” ACTA Electronic Sinica 14(6), 11-19. Wu, F. S., and Cham, W. K. (1990). “A Comparison of Error Behaviour in the Implernentation of the DCT and the ICT,” Proceeding of IEEE TENCON on Computer and Communication Systems, Sept. 1990, 450-453. Yemini, Y., and Pearl, J. (1979). “Asymptotic Properties of Discrete Unitary Transforms,” IEEE Trans. PAMI-1(4), 366-371. Yuen, C. K. (1972). “Remarks on the Ordering of Walsh Functions,” IEEE Trans. Computer C-21, 1452.

This Page Intentionally Left Blank

ADVANCES IN ELECFRONICS AND ELECTRON PHYSICS. VOL. 88

Data Structures for Image Processing in C M. R. DOBIE and P. H. LEWIS Department of Electronics and Computer Science, University of Southampton, England

. . . . . , . . . . . . . . 111. Previous Work , . . . . . . . A. Designing for Speed . , . . . . . . . B. Object-Oriented Systems , . . . . . ~. C. Designing for Flexibility , . . . . . . . D. Designing for Portability , . . . . . . . IV. Standards for Image Processing . . . . . . . V. Data Structure Design in C . . . . . . . VI. Function Interface Design in C . . . . . . . VII. Error Handling . , , . . . . . . . . . VIII. A Small Example . . . . . . . . . . . . IX. Implementation . . . . . . . . . . . . A. Implementing the Data Structures . . . , . B. Image-Related Data Structures . . . . . . C. Miscellaneous Data Structures . . . . . . I. Introduction

.

. . . . . . . . . .

11. Image Representations

,

,

I

D. Implementing Image-Processing Routines . . . . E. Efficiency Considerations F. Portability Considerations . . . . X. Conclusions . . . . . . . . . . References . . . . . . . . . . . .

.

. .

.

. . .

. . . . . ,

. .

. . . . . . . . . .

. .

. . . . .

. . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . .

. . . . . .

. . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . , . , . . . . . .

.

,

. . . . . . . . . . . . . . . , . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . , . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ,

63 65

67 61 68 69

I1 12

13 14 78 78 80 81

82 88 91 104 10s 108

109

I . INTRODUCTION Image processing is one of the fastest-growing application areas of computing. Its popularity has been enhanced by the relatively recent introduction of low-cost image capture systems and the availability of lowcost computers with sufficient memory and processing power to manipulate good quality images at acceptable speeds. The techniques of image processing have gained in importance for a wide range of disciplines, including, for example, the analysis of medical images, the manipulation of satellite and other remotely sensed images, the processing of document images in office systems and desktop publishing systems, and the interpretation of images and image sequences in industrial vision systems for inspection and robot control. The importance of the subject is reflected in the inclusion of image processing units within many degree programmes, often including coverage of the related areas of image 63

Copyright 0 1994 by Academic Press. Inc. All rights of reproduction in any form reserved. ISBN 0-12-0147304

64

M. R. DOBIE and P. H.LEWIS

understanding and computer vision. There are now many excellent text books on concepts, techniques and applications (Gonzalez and Wintz, 1987; Rosenfeld and Kak, 1982; Low, 1991; Jaehne, 1991; Horn, 1986; Banks, 1990; Schalkoff, 1989). Not many years ago, the only way to undertake image processing was by the use of dedicated image-processing hardware, which was not only expensive but also required knowledge of low-level, hardware-specific programming languages. The arrival of workstations and personal computers and the moves towards standard software environments have resulted in a demand for hardware facilities for image capture, display, storage, and manipulation which can be used with a range of standard platforms and utilising popular high-level languages such as Fortran and C. The dramatic improvements in display quality and processing power available with today’s personal computers mean that for many basic image-processing applications the only specialist facilities needed are those required for initial image capture. Specialised image-processing hardware and application-specific integrated circuits (ASICS) are still popular, particularly for computationally intensive applications or for handling very large images. Chips are available for many image-manipulation tasks including Fourier transform operations, convolution, edge detection, and even such specialist algorithms as the Hough transform. Commercial software for image processing is available in a variety of forms. At the lowest level, libraries of routines for utilising particular hardware are usually available with the hardware itself. These libraries typically provide a minimal, low-level interface to a traditional programming language like C or a device driver interface to an operating system. More comprehensive libraries of basic image handling and analysis functions on the host computer are also available both from the imageprocessing hardware manufacturers and from independent software vendors. Image-processing functionality is usually present in scientific visualisation systems such as Sunvision, AVS, and Explorer. It is very difficult for such systems to fulfill the needs of many users. There is such a wide variety of image-processing methods and specialised applications that it may be impossible to find an existing package to meet a particular set of requirements. For example, the flexibility required by one user may make a system too large or too slow for other users. The situation is made worse by the large number of external image representations, which may introduce another problem of converting image data for use in different systems. Most systems do not allow users to add their own routines, and so it is quite common for users to implement their own image-processing algorithms from scratch, especially if they are exploring new methods or applications.

DATA STRUCTURES FOR IMAGE PROCESSING IN C

65

Several internal representations for image data are described in Section 11, and some approaches to the design and implementation of image-processing systems are described in Section 111. The remaining sections describe some specific problems with implementing image-processing routines in the C language, and some solutions are illustrated with code. 11. IMAGEREPRESENTATIONS The most common representation for digital images is a square or rectangular grid of points. Each point is called a pixel (for picture element) and has some attributes associated with it. In a binary image each pixel is either on or off (which may be displayed as white and black). An intensity (often called grey-level or monochrome) image has pixel values which represent the brightness of each point. These are often displayed as a number of shades of grey from black to white. A color image usually has three values for each pixel, which represent the colour and brightness of the point in some colour space. Many systems are in use, including RGB, HSV, HVC, and YIQ (Foley et al., 1990). In addition there are other types of image with different needs. In an edge map image each pixel has an edge magnitude and orientation value. A LANDSAT satellite image has seven values for each pixel, representing responses in different parts of the spectrum. All these types of information can be represented as a two dimensional (2D) array of points. Some image structures require several dimensions. For example, some applications work with three-dimensional (3D) spatial data, so a three-dimensional array is required. It may be necessary to include information about time (in a sequence of images, perhaps) so this adds another dimension to the representation. A more general solution is to allow n dimensional arrays to represent images. Such structures require large amounts of storage and processing time, and there has been much research in finding ways to reduce the requirements without compromising flexibility, A popular image-representation scheme, particularly appropriate for images which are square and have 2" pixels along each side, is the quad-tree representation. A root node for the image-tree data structure is established, and the image is examined against some homogeneity criterion. If it is found to be homogeneous, its homogeneous properties are recorded in the single root node, and processing stops. If it is not homogeneous, the image is recursively divided into quadrants. Each is examined separately for homogeneity. If a quadrant is homogeneous a tree leaf is added to the quadtree recording the quadrant properties. If a quadrant is not homogeneous,

66

M. R. DOBIE and P. H. LEWIS

it is again subdivided into quadrants, and processing proceeds in this way until all quadrants are homogeneous or individual pixels have been reached. Many variations on the quad-tree have been published with specific properties for image processing, and these have been well described in Samet (1984). The quad-tree is an example of a hierarchical or pyramid data structure for image processing (Tanimoto and Klinger, 1980), and selecting a level in the quad-tree above the lowest level is equivalent to using an image with a reduced spatial resolution, rising one level being equivalent to dividing the resolution by two. Many variations and applications of pyramid structures may be found (Wang and Goldberg, 1989; Yau and Srihari, 1983). Hartman and Tanimoto (1984) describe a pyramidal data structure where each level of the pyramid is hexagonal (instead of square) and the pixels are triangular (instead of square). This uses less storage than a square pyramid for images of similar spatial resolution. The idea of the quad-tree may be extended to represent 3D images using an oct-tree. Three-dimensional images are increasingly important for computer vision as techniques for depth and structure estimation are developed. The oct-tree may be used whether the full 3D shapes of objects are to be represented or just 3D representations of the visible surfaces in a scene. The 3D image data should be a cube of values. If all the values satisfy some criterion of homogeneity, only the root node is recorded with the appropriate value. If it is not homogeneous, it is recursively decomposed into eight subcubes and each one tested for homogeneity and either entered as a leaf node in the oct-tree or subdivided again (Jackins and Tanimoto, 1980). Well-established techniques are available for creating and manipulating the resulting tree structures in C (Wyk, 1988) and other languages (Page and Wilson, 1983; Gonnet, 1984). As for the quad-tree, modifications and extensions to the oct-tree to provide particular properties have been published. A variation described in Gargantani (1982) is more space-efficient than the basic oct-tree, and an alternative to the oct-tree for 3D image representation presented in Iyengar and Gadagkar (1988) is a 3D extension to the 2D TID (Translation Invariant Data structure) presented in Scott and Iyengar (1986). It overcomes the inherent lack of translation invariance in the oct-tree representation. Iyengar also presents a novel data structure specifically for 3D object boundary representation in vision tasks using surface curvature maps (Wang and Iyengar, 1992). In addition there is a whole range of symbolic image representations that are useful for specific tasks. These include lists, sets, and networks of nodes that represent regions, features, and objects in a scene. These structures are used in computer vision and image understanding to study the relationships between parts of images.

DATA STRUCTURES FOR IMAGE PROCESSING IN C

67

Although these representations are useful they are somewhat specialised. In this chapter we will concentrate on the traditional representation of an image as a grid of pixels. 111. PREVIOUS WORK

There are many aspects to designing and implementing an image-processing system, and the problems involved can be approached in a variety of ways. This section describes some of the work that has appeared in the literature. The approach that is taken often depends on the priorities of the particular application for which a system is designed. The following list shows several features of a system that can be traded off against each other.

Speed: Some systems are required to operate in real time or interactively. This may require the use of specialised hardware which reduces the portability of the system. Flexibility: A flexible system is capable of processing many types of image data and supports a wide variety of operations yielding different types of results. In practice there is no upper limit to the flexibility of the system, as there will always be some tasks that it cannot perform. Size: A system may be constrained by the environment in which it is to be used. There is usually a compromise between size and flexibility. Ease of development: This feature encourages experimentation and the development of new algorithms in the system. If a system is both interactive and flexible, then these features aid development. Code reuse: A system may be required to make use of large bodies of existing software or allow code to be used from a variety of sources. Portability: It may be desirable for a system t o be easily used on a wide range of hardware and software configurations. This increases the number of users who can make use of the system. A . Designing for Speed

Many applications of image processing (and subsequent image understanding) require that images are processed in real time (or close to real time). Examples are autonomous vehicle guidance and target tracking. If a system is operated interactively by a user, then a faster system is often easier to use. Traditional approaches to speeding up image processing systems involve adding hardware. This can be done either by adding more processors and executing algorithms in parallel or by adding specialised hardware for

68

M. R. DOBIE and P. H.LEWIS

performing specific tasks quickly. There is a large body of research that deals with the design of parallel machines. This includes the hardware architecture and methods for programming algorithms (or adapting existing algorithms) to take advantage of parallelism. The DIPOD system described in Sleigh and Bailey (1987) uses a mixture of these methods by linking specialist processors on a small network. The network nodes have individual tasks such as program execution, program development, framestore, and image Y O . Each program execution node is a specially designed processor, and programs are written in a language that takes advantage of the processor features. This system achieves close to real-time performance with reasonably sized images, and one of the benefits of this is that the immediate response to changes in an algorithm allows the user to gain a better understanding of how various parameters affect a given method. A problem with this approach is that the system is tied closely to the hardware, and as the hardware is superseded the system becomes obsolete; it is often difficult to reuse parts of the system. With today’s rapid pace of hardware development, it is becoming more possible to use widely available, general-purpose computers for image processing. The cost of developing specialised hardware for an increasing number of algorithms starts to outweigh the speed advantages that can be gained. Hardware solutions are best for specific implementations where speed is crucial.

B. Object-Oriented Systems If a system is to be used for experimenting and developing new algorithms, then flexibility is a desirable feature. A researcher may use a large number of tools with different types of image data and generate different types of results. The emphasis in such a system is the ease with which existing methods can be adapted and new methods added. An object-oriented approach to system design can often provide such flexibility. In an object-oriented system, images and other data structures in the system are represented by objects. Each object has a set of methods, which define operations that can be applied to the object. For example, an object representing an image may have a method to calculate the mean intensity of the image. The actual code to perform this calculation may be different for different types of image object. For example, different calculations would be required for grey-level and colour images. An object also has internal data. For example, an image object may keep information about the size of the image. Some of this information will be useful to the user, and some will not. The information that is useful is

DATA STRUCTURES FOR IMAGE PROCESSING IN C

69

made available using access functions. These are simply methods of the object that return the value of specific information held within the object. Another feature of object oriented systems is inheritance. This allows new objects to be created as specialised versions of existing objects. For example, a satellite image object may be a specialised version of an image object. The satellite image object shares the methods of the image object and replaces those that are inappropriate with its own specialised versions. This allows a degree of code reuse, as common methods that are applicable to many image types are only implemented once. Finally, many object oriented systems exhibit polymorphism. This property is the ability of methods in the system to operate on several different object types using the same code. For example, if a method is written that displays two images side-by-side, the method would work for any combination of image types. In a nonpolymorphic system different versions (though very similar) of the method would have to be written for displaying each combination of image types.

C. Designing for Flexibility Carlsen and Haaks (1992) describe the design of a very comprehensive object-oriented system. The system is very wide ranging, including objects for storing and acquiring images, processing images data visualisation, and several types of user interfaces. The system is desi5;lled so that changes to one part of the system do not affect other parts. This allows the system to be developed and updated by several people at once. Although this system is designed to perform image-to-image processing, it does support the creation and manipulation of symbolic data structures (such as sets and lists) that are used for higher-level image understanding tasks. These structures can then be used by other operations. The image objects in this system are very flexible. Images can have pixels of arbitrary type and can hold symbolic or graphical representations. There are objects that can hold a set of images and other associated data and objects that can hold several related images, such as an image sequence or a stereo pair. Algorithms that operate on images are stored in separate operator objects. These are capable of processing many different image types. Some operations are decomposed into several objects. For example, neighbourhood convolution operations are performed with one object traversing the image and another object applying the operation to each neighbourhood in turn. This system allows a high degree of code reuse, since code can be shared between many operations. In addition, an operator can call existing code written in C, Fortran, or Pascal.

70

M.R. DOBIE and P. H. LEWIS

This system also implements a set of objects that provide a user interface. The user interface is independent of the image-processing objects.The system is implemented in a mixture of Objective C, Prolog, and Common Lisp. Paulus (1992) describes another image-analysis system that was designed using a strictly object-oriented approach. He describes a set of recursive structures that can be used to represent regions in an image with attributes and relationships between them. The system he describes is implemented in C+ + and has over 70 classes representing different image-analysis objects and geometric image features. Some implementation and performance details are given, and the need for a machine independent external representation for image structures is emphasised. Flexible systems such as this are very good for image-processing applications development, but they do have disadvantages. They are often very large, and sometimes they can be slow. To take full advantage of objectoriented design, they are usually implemented in object-oriented languages such as C++ and Smalltalk. Although these languages are becoming more available, they are still not as common as traditional languages like C and Fortran. As a result, systems implemented in object-oriented languages are less portable. An alternative approach is to build an object-oriented system using a traditional language. Piper and Rutovitz (1985) discuss data structures for image processing. The main conclusions are that a systematic and organised object-oriented approach to data structures allows a very flexible system to be built. Although an object-oriented approach is taken, the actual implementation of the system is in the C language. Pointers are used to refer to objects, and it is possible to create polymorphic functions. The system is also capable of processing arbitrarily shaped images. Some example structures are shown which demonstrate how processing functions can be implemented. The paper does not explore the problems of handling multiple data types in C. Piper and Rutovitz describe how their routines, together with a suitable intermediate file format, have been compiled into filter programs for use in the UNIX shell environment to provide an interactive set of imageprocessing tools. Flexibility is gained here by combining image-processing functions with the already flexible command structure of most UNIX shells. An alternative approach to using an existing programming language is to develop a new one and implement an image-processing system with the new language. This may be necessary if specially developed hardware is used and there are no compilers or interpreters available for existing languages, as with the DIPOD system described in Section I1I.A. Another reason for adopting this approach is that a new language can provide elegant expressions and structures for manipulating image data types. Lawton and McConnell (1988) describe an image-understanding

DATA STRUCTURES FOR IMAGE PROCESSING IN C

71

environment and illustrate the implementation of a convolution algorithm in three different languages within the system. Hamey et al. (1989) describe a specialist language called Apply that allows the programmer easily to implement an algorithm that can exploit the processing power of a parallel machine.

D. Designing for Portability In many environments it is desirable for image-processing software to work with several different hardware and software configurations. Unfortunately, many image-processing packages only operate on a single type (or family) of computers. As hardware develops and users upgrade, they are faced with the task of replacing their software so that they can continue work with their new system. This may have disadvantages such as extra cost, retraining, and possibly the conversion of existing data. This problem can be reduced by using image-processing software that is portable. This allows an image-processing package (and any applications that use it) to work on several different types of machines. One way t o design a portable image-processing system is to implement the system in a portable programming language and refrain from using machine-specific features. It is also important to use a portable external representation for the images that the system uses. This approach was taken by the authors (Dobie and Lewis, 1991) to implement a modest imageprocessing system. The portable programming language chosen is the C language. C compiliers are available on many different machines, from home computers to workstations and mainframes. There is an international standard for the C language, so it is possible to rely on a minimum set of features being available. In addition, the language provides a rich enough set of features to allow the low-level parts of a system to be implemented efficiently as well as the creation and manipulation of higher-level structures. There is also an international standard (called POSIX) that defines ways for applications to interact with an operating system (among other things). There are standard ways for manipulating files and processes, getting system information, and controlling a terminal. By developing to these standards it is possible to implement an image-processing system that is independent of some features of particular computer systems. This approach allows systems to be created that are easily portable from one machine to another. A disadvantage is that the number of features that the system can use is quite small. For example, although there are standard ways to manipulate files and memory objects, there are currently no standard ways to display images on a screen or to save images in files.

72

M. R. DOBIE and P. H.LEWIS

IV. STANDARDS FOR IMAGE PROCESSING

The next logical step is to develop a more specialised standard that caters specifically for the needs of image-processing systems. This process is currently under way, and an international standard is being drafted. Once it is finished, applications can be written to use the features that the standard defines, and these features will be available on all conforming implementations of the standard. The requirements for the standard are described by Blum et al. (1991) and by Clark (1992). The standard is composed of three parts. The Common Imaging Architecture will define the data types that are used to represent images and associated data. An Application Programming Interface will define the routines that an application can use to manipulate image data, and an Image Interchange Format will define how images are to be represented for storage and communication. Several applications for image processing were considered, and from these a large set of requirements was derived for the data representation and manipulation capabilities of the standard. The intention is to provide this functionality in a well-defined and portable manner and allow the standard to be extended as required by future applications. Images can be represented using a wide variety of elementary types such as bits, characters, integers, and real numbers, all of varying sizes. These can be combined into compound types such as arrays, sets, lists, and records. The pixel of an image can be an arbitrary combination of these. Images themselves can also be represented as compound types. For example, an image sequence might be represented as a list of images. There is also additional information that is used to describe image data. This may define the image geometry or the colour and spectral information that the image represents. Arbitrary data can be associated with an image to allow for application-specific needs. In addition to image data, many applications need to store related information such as histograms, region information, text, and sound. Where possible, the standard will define these in terms of existing standards for representation. The Image Interchange Format is designed to allow the storage and transmission of image data for any application. This includes all the image representations that are possible under the Common Imaging Architecture. It will include a complete description of image geometry and colour specification to allow accurate image reproduction. Compression of image data is defined in terms of existing compression standards. The standard is still under development and currently approaching the first draft stage. A binding for the C language is being developed, and one for the FORTRAN language is planned. Images can have up to five

DATA STRUCTURES FOR IMAGE PROCESSING IN C

73

dimensions, allowing three spatial coordinates, time, and spectral bands. A set of over 200 processing functions is planned, slthough different levels of conformance to the standard will allow smaller subsets to be used. V. DATASTRUCTURE DESIGN IN C

Some approaches to designing image-processing software have been discussed in the preceding sections. Many of the ideas described are object-oriented in nature (even if they are not implemented using an objectoriented language). In the following sections we will discuss the specific problems of implementing an image-processing system in the C language. Examples from the authors’ experience will be used to illustrate some approaches to the problems. A basic object-oriented approach can be adopted. The main aim is to take advantage of the good features of an object-oriented design while implementing the whole system in standard C so we retain portability. A secondary goal is that the system should be fast. The system is implemented as a collection of C types and functions. The functions are stored in a C library. As with Piper and Rutovitz (1985), it was decided that an object is best represented by a C structure. Each structure has an associated C type, which represents the type of the object. A pointer to the structure is used to refer to the object. Each object has a set of functions associated with it, which correspond to the methods of the object. These are used to create, manipulate, and destroy the object. The main advantage of this technique is that it is conceptually simple. The user (of the library) need only declare one variable to refer to an object which is passed to all the manipulation and processing routines. All the internal details of an object are hidden from the user (in the structure) and maintained by the methods (the library routines). Any details which the user can use should be made available using an accessfunction, which the user calls to obtain the values of specific fields within the object. A small execution time advantage is also gained, since a single pointer is all that is passed between routines. This is fast and requires little stack space when compared to passing a large set of associated parameters to routines. It also removes one source of programming errors, since the internal integrity of the objects is maintained by the library and cannot (if the routines are well designed) be disrupted by the user, Another advantage of using single pointers to objects is that one routine can process different types of objects that are referenced via the same type of pointer. For example, the same edge detection routine is capable of

74

M.R. DOBIE and P. H.LEWIS

detecting edges in monochrome and colour images of any size and shape without a change to the function call. All the information which a routine might need about the object (like its type, size, and shape) is available from the object via the pointer. In an object-oriented language this behaviour is called polymorphism. In the C language it has to be implemented explicitly using a type field in each object and functions that respond to different types of objects appropriately. A brief discussion of the object-oriented approach is given in Section 1II.B. This idea is applied inside the objects as well. Many objects contain pointers which can point to a number of other different objects depending on the type and representation of the object itself. This approach cuts down the memory requirements of the system as a whole, since there are never any redundant fields in the objects to allow for more complex structures, which may not be present. The approach leads to elegant interfaces to the routines, with all the detailed information about the object contained within the object itself, rather than being passed in a large parameter list (which can be confusing to the user and inefficient). The simplified interfaces allow a newcomer to learn the library routines quickly and use them effectively with a reduced chance of making programming errors. There is a disadvantage with adopting this polymorphic approach in C. When implementing a routine to manipulate objects, the routine must check the type of each object and execute a relevant piece of code for that type of object or signal an error condition if an object has been supplied for which the routine is inappropriate. The flexibility and ease of use from the caller’s point of view place a burden on the routine implementor, since he or she must cope with all the possible types of objects that may be passed to the routine and decide how each should be processed.

VI. FUNCTION INTERFACE DESIGN IN c The previous section describes how the data objects are represented as C structures and how they can be referred to by a pointer. This simplifies the input parameters to the library routines. It is not necessary to pass a list of associated parameters for each object because a single pointer is all that is required. This section discusses several approaches for passing data to the library routines and returning results from them. The data required by library functions are passed as function parameters. These usually include a mixture of pointers to objects and extra parameters which are required by the function. For example, an image-thresholding

DATA STRUCTURES FOR IMAGE PROCESSING IN C

75

function might have two parameters; a pointer to an object representing an image and the value of the threshold. Several schemes for returning output parameters from library functions have been considered. In particular, it is consistent that functions return pointers to objects for their output, since pointers to objects are used for their input parameters. There are several types of functions with different needs for their interfaces. Some functions create new objects (for example, threshold image A and create a new result image B). Other functions retrieve information about existing objects (for example, calculate the area of image A), and there are functions that don’t return anything (for example, destroy image A). Some functions need the ability to communciate error conditions, and others do not. If a function returns an object, there are several ways it can be done. 0

0

0

The function can be given a pointer to an empty object which it can fill in. The return value of the function can be used for returning an error status flag, indicating whether the function was successful or not. The function can create a new object, fill it in, and return a pointer to it as its return value. Error conditions need to be returned by another method. The function can be given a pointer to an object and actually alter the object to produce the result, effectively destroying the object it was originally given.

The third technique can be eliminated straight away. Although some image processing operations can be performed in situ, many give a result of a completely different type to the inputs. This method would involve an object changing its type halfway through a program (which would be confusing to the user) and would require explicit copying of the input objects if they were to be preserved. These disadvantages outweigh the slight overhead of creating new objects when they may actually be unnecessary. The other two methods are more promising. Taking the thresholding example from above and assuming a structure type called IMAGE we compare, in Fig. 1, the C code that would be required to call a library function. The first technique needs one more line of code than the second, but it handles the error code neatly. Its main disadvantage is that the programmer needs to know the relationship between the input and output images of the threshold function (for example, are they the same size or type?) in order to create the correct empty IMAGE object to pass t o the function. Using the second technique, this relationship is encapsulated within the threshold function itself, so the function creates the correct output image

M. R. DOBIE and P. H. LEWIS

76

The first technique results in this code:

...

image-B = create-empty-image( ; error = threshold(image-A, image-B, if (error == . . . )

...

- ..

)

;

1

The second technique results in this code: (

IMAGE *image-A, int error ;

*image-B

;

image-B = ihreshold (image-A, & e r r o r , if (error == . . . ) ...

. ..

)

;

1 FIGURE 1 . Comparing code for returning results.

object, depending on the input image, and the programmer doesn’t have to worry about it. The second technique achieves a higher level of abstraction, which makes it easier to use and reduces the chance of programming errors. This is the method adopted for this library. The method for handling errror conditions is discussed in more detail in Section VII. If a function returns several results (for example, most edge detectors return an edge magnitude image and an edge orientation image), these can all be returned as one IMAGE object. The type of the object will reflect the fact that it contains both the edge-magnitude and edge-orientation information. This is appropriate for logically related results that can be combined into an object type. Independent multiple results can be handled in the traditional C fashion by passing several parameters by reference, which can {

IMAGE COORD DWORD

*image ; width, heighc ; area ;

get-image-dimensions

(image, &width, &height, &area) ;

1 FIGURE 2. Returning several results at once.

DATA STRUCTURES FOR IMAGE PROCESSING IN C

77

{

display-image ( threshold( acquire-image

...

...

)

(

. ..

)

,

1 1

;

1 FIGURE3. Cascaded function calls.

be altered by the function. An example is shown in Fig. 2, where a function that returns several statistics about an image is called. The function sets the values of the variables width, height, and area. Both methods are used in this library. Another point to note is that with the second technique it is possible to cascade function calls. This is illustrated in Fig. 3. The pointer returned by acquire-image points to the input image for threshold, and the thresholded image is the parameter for display-image. The notational convenience of this code increases the usability of the functions. One consequence is that each function should fail gracefully if its input parameters are NULL to prevent a program crashing within a library routine. One disadvantage with this code in C is that it will create objects in memory which are not subsequently destroyed, and therefore the available pool of free memory will eventually be used up, A garbage collection system (found in other languages such as Lisp and Smalltalk) or automatic destruction of objects as they go out of scope (like that in the C + + language) would solve this problem. To solve the problem in standard C, the intermediate pointers have to be saved in variables and the intermediate objects destroyed explicitly. A simple technique is shown in Fig. 4. (

IMAGE

*trnpll *trnp2 ;

display-image ( tmp2 = threshold( trnpl = acquire-irnage(

...

... )

),

;

destroy-image ( t r n p i ) ; destroy-image (trnp2 1 ; 1

FIC~URE 4. Cascaded function calls in C.

...

78

M. R. DOBIE and P. H. LEWIS

VII. ERRORHANDLING There are several ways in which the library functions could notify their caller of an error condition. The neatest way is for the function to return an error code, but this has been ruled out by the function interface design, which returns pointers to new objects as the function-return value. An alterantive is for the caller to pass in the address of a variable, which the function can set to the error code, as illustrated in Fig. 1. This is not very readable and unnecessarily complicates the arguments to the function. A compromise method similar to that used by the standard C run-time library (Kernighan and Ritchie, 1978, 1988) was chosen. A global variable (called image-error-code) is provided, which is set to an error code if an error occurs. An error condition is signalled by a function returning an exceptional (and otherwise invalid) value. Functions which return pointers can return NULL if there is an error. Functions which return information about an object can return an invalid value to signal an error. Functions which would normally have no return value can return a boolean result, with FALSE indicating an error. Error messages can be optionally displayed for information or debugging purposes, as well as the program taking appropriate action after examining the image-error-code variable. This is similar to the technique proposed for the IPAL library being developed by the Numerical Algorithms Group (Carter et al., 1989). Note that functions which take pointers to objects as arguments do not attempt to check whether the pointer points to a valid object, since there is no reliable way of doing this in C . It is up to the caller to ensure the integrity of pointers passed into the library functions. Other object-oriented languages, such as Smalltalk, do not suffer from this problem because they do not allow variables to undergo arbitrary type conversions. A function will check to make sure that its input pointers are not NULL. If they are (as in a cascaded function call where an inner call has failed) the function being called must signal an error too. It should be impossible for a program to crash within a library function. A function will also check whether the type of object that it is passed is appropriate for that function’s operation. VIII. A SMALL EXAMPLE Figure 5 shows a small example program demonstrating how the library functions are called in practice. This program acquires a monochrome image from a frame grabber and thresholds the centre portion of it, displaying the results on the frame store. The result is shown in Fig. 6.

DATA STRUCTURES FOR IMAGE PROCESSING IN C

# i nc 1ude

'' image 1 i b .h

79

I'

main ( 1 (

* small-image *smal.l-area ;

IMAGE IM-WINDOW

*big-image,

;

/ * acquire a full screen image from t h e frame grabber * /

/ * d e f i n e a window for the centre portion * / small-ared

- create ._ window(128,128,384,384,NORMAL,NULL) ;

/ * copy t h e area from rhe frame store and threshold it * / small-image

=

i m ~ t h r e s h o ~ d ( f g ~ c o p y _ m o n o O , PLANE - MONO, 128) :

/ * clear t h e €rame s t o r e * /

all -b o a r d s ( s c l e a r ( 0 ) ) ;

/ * display the resu1t.s * / fg-display (big-image) ; fg-display (small-image)

;

1

FIGURE5 .

A thresholding example.

The program declares some variables. There are two IMAGE structures and one IM-WINDOW structure. The IMAGE structures hold an image, and the IM-WINDOW structure defines the shape and size of an area. These structures are discussed in more detail in Section 1X.B. The first function call acquires a full-screen (512 x 512 pixels in this case) monochrome image from the frame grabber. It creates a new IMAGE structure to hold the image and returns a pointer to it, which is saved in big-image. Next an IM-WINDOW structure is created which represents the centre portion of the image. The coordinates give a square area, and the NORMAL parameter indicates a rectangular window (rather than an arbitrarily shaped area).

80

M. R . DOBIE and P. H. LEWIS

FIGURE6. The output of the thresholding example program.

Next, a cascaded function called fg-copy-mono copies the area from the frame store, and this image is used as the argument to the thresholding function. The area is thresholded at a level of 128 using the monochrome plane of the image (specified by PLANE-MONO), and the resulting binary image is saved in small-image. After clearing the frame store the original image is displayed with the thresholded area overlaid on top of it. Figure 6 shows this applied to a picture of two bears. IX. IMPLEMENTATION The library has been initially implemented in the C language running under a UNIX operating system. In addition to the reasons given in Section III.D, this combination allows easy manipulation of large data objects and is ideal for experimenting with the design ideas described in this chapter. The library has also been successfully ported to Silicon Graphics workstations, PC systems running UNIX, and PC systems running Microsoft windows. The library can be considered in two parts. There are routines which create and manipulate the data structures themselves and routines which operate on the image data. These are discussed in Sections 1X.A and 1X.D.

DATA STRUCTURES FOR IMAGE PROCESSING IN C

81

A . Implementing the Data Structures The routines to manipulate the data structures have been implemented in an object-oriented manner. Each data structure and its associated functions correspond to an object and its methods in an object-oriented system (see Section 1II.B for a description of object-oriented approaches). Each object is encapsulated in one source file and one header file. The header file provides the calling interface to the data structure. It declares all the functions that are used to manipulate the structure and any types that may be required as parameters to those functions. In the header file the structure itself is declared as a void type. This prevents any code that uses the structure from gaining access to the structures’ internal fields. The only way that other code can use the structure is by calling the functions that are defined in the header file. This data hiding approach helps to guarantee the integrity of the data in the structure. The source file defines the details of the data structure itself and the functions that operate on the structure, These include routines to create, copy, and destroy an object. There are often file I/O routines too. Many structures have accessfunctions to retrieve and set some of their fields, and routines to perform simple manipulations on the object they represent. Most of the data structures are made up of several memory objects linked by pointers. A routine to create a structure needs to create the required blocks of memory and link them together in the proper way. The same is true for many other operations that may be applied to a structure (for example, copying and writing to a file). There is a set of common low-level memory-management routines which deal with the memory objects that make up a data structure. These routines are used by the structure manipulation routines. The memory-management routines are invoked using macros with the type of the object as a parameter. This increases readability of the code because the programmer doesn’t need to cast the pointers returned or calculate the size of the object; the macro puts this in automatically. The macro is defined in Fig. 7, where create-structf is a function which calls the system to allocate some memory and checks to see if enough was available. Figure 8 shows how the macro might be invoked by a programmer implementing a function to create a NEW-OBJECT object.

i t d e f i n e create-struct (type)\

( t y p e *)create-structf(sizeof(type))

FIGURE7. The structure creation macro.

82

M.R. DOBIE and P. H. LEWIS

NEW-OBJECT new

=

*new

;

create-struct(NEW-OBJECT)

; /*

v e r y readable * /

1

FIGURE8.

Using the structure creation macro.

A similar abstraction is available for copying blocks of memory and for reading and writing them to files. Destroying memory blocks is all done by the same function which calls the system to free the memory, independently of the type of structure.

B. Image-Related Data Structures The top-level data structures available in this library are IMAGE, SEQUENCE, and TRACE. Each of these has several components and associated routines to manipulate them. They are described in detail in the following sections, along with any substructures that are used. There are also several miscellaneous structures that are independent, but not used directly for image processing. These are described in Section 1X.C. 1 . The IMAGE Structure

+

The IMAGE structure represents a whole image. It is used as the input and output of most of the routines in the library. The component fields of an IMAGE structure are shown in this diagram.

IMAGE

image type data storage type number of frames image size WINDOW pointer to the image data

The image type defines the type of image that this structure represents. The data storage type defines the type of the raw image data (for example, bytes, integers or floating point). A number of separate images (frames) can be stored in a single IMAGE structure. This is indicated by the number of frames being greater than one. The image size WINDOW structure defines the size, shape, and position of the image. There is a pointer to the image data itself. The amount of data and its format depend on the type of image, the size of the image, the number of frames, and the storage type of the data.

DATA STRUCTURES FOR IMAGE PROCESSING IN C

83

Several types of images have already been incorporated into the library. These represent monochrome images, colour images (using red, green, blue (RGB), and hue, saturation, brightness (HSB) colour models), edge maps, binary thresholds, and Hough transform accumulator images. Depending on the type, an image may have several planes in each of its frames. A colour HSB image has three planes, one each for hue, saturation, and brightness. An edge map has two planes, one for edge magnitude and one for edge orientation. A threshold image is simply a binary image of 0s and 1s which is generated by some functions to indicate which areas of an image satisfy some criterion. Such an image can be used by other functions to apply an operation to specific areas of an image. A Hough transform accumulator image holds a Hough accumulator array and is generated by Hough transform based matching functions. This structure is only suitable for storing two-dimensional image data. If higher dimensions are required (for example, to store a Hough transform accumulator with three or four parameters), then it is possible to use an array of IMAGE objects. Ideally a structure capable of representing any number of dimensions would be provided, but even the draft international standard (described in Section IV) draws the limit at five dimensions. Given that an image has several planes, there are two methods for storing the image data. The plane components for each pixel can be stored together, so that they are all availabe in one location. This is an interleaved representation. An alternative method is to keep all the data for each plane together. In this planar representation the components of each pixel are spread throughout the structure. In this library we chose the planar representation because there are many image-processing structures (such as edge maps and satellite images) where it is useful to examine planes of an image separately. A set of routines has been implemented to manipulate IMAGE structures. There are the basic methods for creating an empty IMAGE, copying an existing IMAGE, and destroying an IMAGE structure. There are also functions for reading and writing IMAGE structures to disk files. Images are written to disk in a Pbmplus format. This is a common interchange format on many machines and is capable of storing binary, monochrome, and RGB colour images. There are many existing utilities to convert Pbmplus formats to other common image-file formats, so we achieve portability without having to implement numerous file input and output routines. In addition there are many access functions which allow the retrieving and setting of some of the IMAGE structure’s fields, including the image size WINDOW, image type, storage type, and a number of frames. There are functions to compute other attributes of the IMAGE such as width, height, and area. To access the image data there are functions to provide a pointer to any part of the image data, specified by x and y coordinates.

84

M. R. DOBIE and P. H. LEWIS

There are several routines to manipulate the image data. These allow individual frames (or a range of frames) to be extracted from an IMAGE structure. The frames of two IMAGE structures can be combined to give a new structure containing all the frames from both images. There is a routine to average frames together to give a composite image from many frames. An image can be separated into its individual planes and an arbitrary area (specified by a WINDOW structure) can be extracted from an image. Where an image has several frames and several planes within each frame, they are generally all treated the same way, and the resulting image has the same number of frames and planes too. For example, extracting an area from a multiframe image gives a new image with the same number of frames and planes per frame as the original, but each contains only the specified area of the image. Other operations change the number of planes or frames in the resulting image. For example, extracting one plane from a multiframe image results in a new multiframe image where each frame contains just the extracted plane.

2 . The WINDOW Structure A

WINDOW structure

defines the size and shape of an area. It is used in an structure to define the shape of the image. A WINDOW can also be used independently to refer to part of an image, as in the example program shown in Fig. 5 . A WINDOW has the following structure:

IMAGE

window type bounding coordinates extra data

t

WINDOW

The window type defines the representation used for the area. A normal a rectangular area specified by its top left and bottom right coordinates. This simple representation is sufficient for most requirements. The window type may indicate that a window represents an arbitrarily shaped area. The bounding coordinates specify the smallest upright rectangle enclosing the represented area. For a normal WINDOW, these coordinates are the sole representation of the area. The example program in Fig. 5 creates a WINDOW that represents a square area in the centre of a 512 x 512 pixel screen. If a WINDOW represents an arbitrarily shaped area, an additional pointer t o a BOUNDARY structure is present. This is the extra data field in the diagram above. For a normal WINDOW this field is NULL. In the future there may be more WINDOW types, in which case this field can point to whatever representation is used. WINDOW is just

DATA STRUCTURES FOR IMAGE PROCESSING IN C

85

The usual routines to create, copy and destroy a WINDOW structure have been implemented. There are access functions to retrieve the bounding coordinates and type of a WINDOW, as well as the pointer to extra window data. WINDOW structures can be written to and read from disk files. There is a function to calculate the area of a WINDOW. This function is called by the routine to calculate the area of an image, since the shape of an image is represented by a WINDOW structure. There are several routines to manipulate the shape of a WINDOW. A WINDOW can be translated, or scaled in the x and y directions by different factors, either about its centre or about an arbitrary point. There are also routines to draw a representation of a WINDOW on a frame store, although this is hardware specific. 3 . The BOUNDARY Structure BOUNDARY structures are used to define an arbitrary shape. They may be attached to a WINDOW structure to describe an arbitrarily shaped area of an image. A BOUNDARY structure has two main components:

boundary list BOUNDARY{

vertex list

The boundary list contains elements which represent the set of scan line segments that make up the shape being represented by the BOUNDARY structure. Each element uses a left and right x coordinate and a y coordinate to describe a segment of a scan line. By only considering points in image data that correspond to points in the scan line segments, a routine can process an arbitrarily shaped region in an image. The scan line segments are stored in left-to-right, top-to-bottom order to ease processing. Section 1X.E compares the efficiency of different approaches to storing and traversing image data in this way. The vertex list is a list of points that form the boundary of the shape when connected in order by straight lines. This alternative representation of the shape is useful for drawing an outline of the shape. It is typically much more compact than the boundary list. The two lists are created and maintained independently, as conversion between them can be very complex. There are routines to create, copy, and destroy BOUNDARY structures, and read and write them to disk files. Elements can be appended to either list and retrieved in order from the head of the list. There is a routine to calculate the area enclosed by a BOUNDARY and routines to translate and draw a representation of the shape.

86

M. R. DOBIE and P. H. LEWIS

4. The SEQUENCE Structure

The SEQUENCE is another top-level structure. It is an abstract representation for a set of images. It does not contain the images themselves, but it does contain information about where they are stored along with other information about the sequence. A set of images could all be stored in an IMAGE structure with several frames. The advantage of using a SEQUENCE structure is that it provides a layer of abstraction above the image data. This allows alternative representations for the frames of the sequence while preserving a common interface through the sequence structure and its routines. Indeed, one of the representations that a SEQUENCE supports is simply a set of images stored in the frames of a single IMAGE structure. A SEQUENCE has the following components:

+-

SEQUENCE

sequence type frame number data sequence source

The sequence type defines how the frames of the sequence are represented. A sequence can be stored in a single IMAGE structure on disk, which is useful

for small, computer-generated sequences. For longer colour sequences this approach uses too much storage. A sequence can alternatively be digitised, when required, from a laserdisc. This requires a laserdisc player and a frame grabber and is a very convenient way of handling large volumes of image data. The frame number data indicates how long the sequence is and keeps track of which frames are accessed. The sequence source points to a filename and an IMAGE object or a laserdisc identifier and a VIDEODISC object, depending on the sequence type. There are routines to create both types of SEQUENCE and to destroy a SEQUENCE, as well as reading and writing to disk files. Before a SEQUENCE is used it must be initialised. This will read the frames into memory (for an IMAGE structure sequence) or open a connection to the laserdisc player (for a laserdisc sequence). There is a routine to deinitialise a SEQUENCE too. To access the frames of the SEQUENCE there are functions that return an individual frame in an IMAGE structure, either as a colour or monochrome image. The functions will return the first, last, next, and previous frame, allowing sequential access to the sequence. There are routines to get the current frame number and sequence length and to determine whether the end or start of the sequence has been reached.

DATA STRUCTURES FOR IMAGE PROCESSING IN C

87

There are functions to convert a SEQUENCE to a single IMAGE structure (colour or monochrome) and to display the frames of a SEQUENCE one after the other on a frame store, although this depends on the presence of a frame store.

5 . The VIDEODISC Structure A VIDEODISC structure is used to refer to one side of a laserdisc in a laserdisc player. The player is under computer control, and the VIDEODISC structure allows individual frames of the disc to be accessed. There are only two fields in a VIDEODISC object: last frame VIDEoD1sC< handle The lust frame i s the frame number of the last frame on one side of the disc and is used to identify the disc. The handle is a connection to the laserdisc player and is used to control it. There are functions to create and destroy a VIDEODISC object, which cause connections to the laserdisc player to be opened and closed. The player can be given commands to go to a particular frame, go to the last frame on the disc, or go forward and backwards by a frame. 6 . The TRACE Structure

The TRACE is the third top-level structure in the library. A trace is a highlevel abstraction that can be used when tracking an object through an image sequence. It represents an object (being tracked), a sequence (in which the object moves), and a set of data about the position of the object as it moves through the sequence. As the object is tracked, the locations found by different tracking algorithms can be compared against the trace data to assess their performance. This is an example of a specialised structure designed to meet the needs of a specific project. However, it serves to demonstrate the way the data structures can be easily extended to meet particular requirements. A TRACE has several components:

+

TRACE

trace object trace sequence trace length trace data arrays

The truce object field is a pointer to an IMAGE structure which represents the object being tracked. The truce sequence is a pointer to a SEQUENCE

88

M.R. DOBIE and P. H.LEWIS

structure which refers to the sequence of images containing the object. The trace length field is the number of frames in the sequence and also the length of the trace data arrays. The trace data arrays contain information about the object’s location, orientation, and size in each frame of the trace. Data need not be present for every frame of the trace sequence. There are routines to create and destroy a TRACE and to save and restore to disk files. A TRACE must be initialised and deinitialised before and after use, just like a SEQUENCE object. There are access functions to retrieve the trace object and the trace sequence fields. Trace data can be set or retrieved for a specific frame or for the current frame of the trace sequence. C. Miscellaneous Data Structures

There are a few data structures that are provided but are not used for processing image data directly. They have proved very useful in implementing specialist image analysis software. 1 . The K-STATE Structure The K-STATE structure provides a method for using Kalman filters. Kalman filters are used increasingly in computer vision and allow discrete systems to be modelled and predictions of future behaviour to be made from the model. An example application is the modelling of object motion through image sequences. Details of Kalman filters may be found in Haykin (1986), which may help in understanding the routines that are implemented for manipulating K-STATE structures. A K-STATE structure has three fields: K-STATE

-€

values first derivative second derivative

The K-STATE structure represents the state of a system at a given time. The value of the system is modelled by the Kalman filter, which can provide estimates of the value in the future and estimates of the first and second derivative of the value. For example, if the value corresponds to an x coordinate, then the K-STATE describes the x position and the velocity and acceleration in the x-direction. There are functions to create, copy, and destroy K-STATE structures, as well as access functions to retrieve the three fields. Two routines are used to implement a Kalman filter. One routine initialises the Kalman filter given standard deviations of the noise components of the system model and the

DATA STRUCTURES FOR IMAGE PROCESSING IN C

89

measurement model. This creates and returns the Kalman gain vector in a K-STATE structure, which is used to make estimates of future states. The second routine calculates an estimate of the next state of the system, given the estimate of the current state and a current measurement taken from the system. The Kalman gain vector must also be passed as a parameter. The estimate of the next state of the system is returned in a new K-STATE structure.

2. The POLYLINE Structure A POLYLINE is a representation of a set of ordered points which may (but not necessarily) form a line. The structure is intended to be fairly general purpose so it can be used wherever a list of points needs to be stored. This is useful when an image has some vector data associated with it (for example, the boundary of a region or the course of a river) or just a set of points (for example, the locations of a particular type of image feature). There are no specific fields of a POLYLINE that are accessible to the user. Although the structure represents a list, the implementation may be more sophisticated than a simple linked list to allow for fast random access and searching within the list:

PoLYmE-list

of points

There are routines to create, copy, and destroy a POLYLINE, together with routines t o append a point to the list and to insert a point before an existing point in the list. There are functions to retrieve the first, last, and next point in the list and also to search the list for a specific point. There is a routine to translate the points in a POLYLINE by given offsets in the x and y directions. 3 . The DEVICE Structure

The DEVICE structure is an attempt to provide a system independent method for displaying an image on the computer screen. There is currently no standard way that a program can use to interact with a graphical user interface (GUI), although this problem is being looked at as part of the POSIX family of standards (described in Section 1II.D). Section 1X.F describes some of the implementation techniques used to achieve this portability in more detail. Since the internal details of a DEVICE will be different from system to system, there are no fields that the user of the structure can access. DEVICE-system

dependent image representation

90

M. R. DOBIE and P.H. LEWIS

A routine is provided to create a DEVICE from an IMAGE structure. This is the only method for creating a DEVICE structure. There is a routine to destroy a DEVICE,too. Once the structure has been created, it can be displayed on the screen by calling one of several display functions. These will display the IMAGE in its normal coordinates or at a specific position. There is also the facility to display only part of an IMAGE at a given scale factor. The DEVICE structure was originally designed to work with GUIs. The DEVICE display routines are the low-level functions that are called by a more sophisticated image-manipulation program. They simply display the image on the screen. Currently DEVICE structures have been implemented for the Microsoft windows interface on PC compatible machines, and for the X windowing system on UNIX machines. 4. Error Handling

Several routines have been provided for controlling error handling in the library. An error is signalled by a library routine returning an exceptional value (as discussed in Section VII). When an error occurs the global variable image-error-code is set to a value that reflects the cause of the error. In addition there is the concept of error status, which is either visible or invisible. When the error status is visible, error messages are displayed for the user to see when an error occurs. If the error status is invisible errors occur silently, but a program can still react to an error by examining the value of image-error-code. There is one routine to signal an error. It is given an error message and an error code to identify the cause of the error. There are two routines to retrieve and set the image error status.

5 . Scalar Types There are some scalar types that the library defines for different purposes. These are listed here with a description of their values and intended use.

IM-TYPE: This is the type of an IMAGE structure and can take the values IM-MONO, IM-RGB, IM-EDGEMAP, IM-THRESHOLD, IM-HOUGH-ACCUMULATOR, and IM-HSB. BYTE: This is an 8-bit storage type used for pixel data. WORD: This is a 16-bit storage type used for pixel data. DWORD: This is a 32-bit storage type used for pixel data. FLOAT: This is a floating-point storage type used for pixel data. DOUBLE: This is a double-precision floating-point storage type used for pixel data.

DATA STRUCTURES FOR IMAGE PROCESSING IN C

91

IM-STORAGE: This refers to a storage type for an IMAGE structure and can take the values BYTES, WORDS, DWORDS, FLOATS, or DOUBLES, reflecting which of the above types has been used to store the image data. COORD: This is used to refer to the dimensions of an image and positions within it. BOOLEAN: This takes values TRUE and FALSE and is used for boolean conditions such as the success or failure of a function. IM-PLANE: This refers to a plane in an IMAGE structure and can take values which represent the order in which the plane data appears in an IMAGE structure. The possible values are PLANE-MONO for monochrome images; PLANE-RED, PLANE-GREEN, and PLANE-BLUE for RGB colour images; PLANE-HUE, PLANE-SATURATION, and PLANE-BRIGHTNESS for HSB colour images; and PLANEMAGNITUDE and PLANE-DIRECTION for edge images. The special value PLANE-ALL is used to refer to all the planes in an image. DISC-ID: This type is used to refer to a side of a laserdisc. VIDEO-FRAME: This is used to identify a particular frame on a side of a laserdisc. D. Implemen ting Image-Processing Routines

The actual image-processing routines (as opposed to the data-structure manipulation routines) are in a separate module for each routine. These routines are quite complicated as they have to deal with all the possible types of data structure which may be passed to them. Although this makes things easier for the library user, it complicates matters for the function implementor. One point to note is that all the manipulation of the image data is performed using pointers and pointer arithmetic. The alternative is to use C arrays, but this would limit the size of images that the library could handle. Also, by casting pointer variables appropriately, the polymorphism described in Section V can be easily implemented. This approach also has some efficiency benefits which are described in Section 1X.E. There are two main ways in which IMAGE structures differ. The first is the image type. Some functions (for example, displaying and copying areas) are applicable to all types of images, whereas other functions only make sense for some types. Each function must examine the types of the images it is given and decide how to process them, even if it just means signalling an error condition when an inappropriate image type is passed in.

92

M. R. DOBIE and P. H. LEWIS

The other difference between IMAGE structures is the storage type. This affects how large the structure is and how the function has to access the image data. It also defines the range of values that each point in the IMAGE can take, so each function that assumes a particular range of pixel values has to check that images of a suitable storage type are used. The way a routine is implemented also depends on the routine itself and the way it treats the frames, planes, and pixels in the IMAGE structure. Some routines only process a single plane of an image, but this is usually applied to every frame. Other routines process all the planes in the same way. Some routines only process part of the image data, and others process all pixels in the image. There are several aspects of implementation that are shared by most processing routines. A function implementing an image-processing operation typically has the structure shown in Fig. 9. The final switch statement in Fig. 9 deals with the storage type of the image data. This is necessary because the C language has no facility for generic types. In an ideal language, the code for the image processing operation would be written independently of the data type of the image data. The actual data type of the image data would be specified when the code is invoked. This is possible using generic types in Ada or templates in C++. The solution adopted here is to use a switch statement and a macro. The macro is expanded for each of the possible types of image data, and the switch statement decides at run time which expansion to execute. Figure 10 shows how the macro might be written. As this is a C macro definition, every line should end with a trailing backslash (\). These have been omitted to aid readability in this and all the following macro definitions. The first parameter of the macro is the data type of the image data (referred to as TYPE within the macro definition). The macro declares and initialises some pointer variables, which can be used to manipulate the image data (of type TYPE). There are also some variables which are useful for traversing the image data. The code that processes the image data can all be written in terms of TYPE. It has to be written only once, which reduces the chance of programmer errors. A disadvantage is that the code is expanded several times when the program is compiled, so the library will contain a version for every possible storage type for the image data. The body of the processing macro will be different for different types of operation. In the following sections several types of operations are described and illustrated with code from the library. The approach that is used depends on the level of flexibility that the implementor wishes to provide.

93

DATA STRUCTURES FOR IMAGE PROCESSING IN C

IMAGE * p r o c e s s ( image,

...

)

IMAGE *image : (

IMAGE

*new-image

;

/ * Check f o r NULL IMAGE pointers and other valid parameters * / / * Check i t input imagefs) are sensible f o r t h i s function * / /*

-

Create Lhe imdge L O hold the result u b i i d l l y check t o r b d m e size, storage type and number of frames. L h e image type o f t e n depends on the operation,

*/

/ * Process the image . i c c o r d n q to s t o r d g e type ' /

swjr c h (srorage cypc: ( CdSO

8Y'I'ES:

proct.ss--i.ype (DY'I't:, image,new-image,

break

,

.,

1;

;

case WOHIIS: process-t.ype(WORD, image,new-image, break ;

...

);

/ * , . . one zdse tor all storage types * / 1 reLiirri (new-irnegt:) ;

t

FIGURE9. Implementing the top level of a processing routine.

Udetirit! p r o c e s s type('I'Yl'~,source,dest,

...

)

1

/ * Image data poinlers * i TYPE *s = get-image-daLa (source) ;

TYPE

*d

=

WORI) I M- I' I, AN ti:

n

= qet-number-of-images(source)

DWORD

COOHD

get-image-data (dest)

; ;

get-planes-per-frame (source) : d r e a = get-image-area (source) ; width = gat-image-width(source) ;

ppf

=

/ * Process Ltie image data..

.

*/

i

FIGURE10. Implementing the bottom level of a processing routine.

94

M. R. DOBIE and P. H . LEWIS

1. Traversing The Image Data

In an IMAGE structure, all the image data is stored in a single block of memory. The frames of an IMAGE are stored in sequence. Within each frame there may be several planes. Each plane is the size of the image, and the pixels are stored in left-to-right, top-to-bottom scanline order. Thus if a pointer starts at the beginning of the image data, it represents the top left corner of the first plane of the first frame of the IMAGE. As the pointer is incremented, it moves along the top row of the image, then onto the start of the second row, and so on until the end of the plane, where it starts at the top left corner of the second plane (if there is one). Each group of planes is a frame, and there may be several frames in the structure. The macro in Fig. 10 declares and initialises the variables width, area,ppf, and n for traversing the image data. In the C language, pointer arithmetic is defined so that adding one to a pointer variable causes it to point to the next item in memory, where the type of item is the same as the type to which the pointer variable points (Kernighan and Ritchie, 1988, page 205). In the macro, this type is passed as the first parameter and is the type of a pixel in the image data. Table 1 shows the pointer expressions which can be used to traverse the image data given a pointer p to some pixel in the image. These expressions are used by the processing routines described in the following sections. 2 . Processing Every Pixel in the Same Way

Figure 11 shows a macro that calculates the absolute difference between two images. It is the same as Fig. 10, except that there are two source I M A ~ Epointers. After initialising pointers to the image data of source and TABLE I EXPRESSIONS USEDTO TRAVERSE IMAGE DATA Expression

Refers to

P

A pixel in the image

P + l P-1

One pixel to the right One pixel to the left One pixel below One pixel above Corresponding pixel in the next plane Corresponding pixel in the previous plane Corresponding pixel in the next frame Corresponding pixel in the previous frame Total number of pixels in the image

p

+ width

p - width p area

+ +

p - area p ppf * area p - ppf *area n * ppf * area

DATA STRUCTURES FOR IMAGE PROCESSING IN C

95

Udef ine di fference-type ('I'YPE, sourcel, source2,dest)

t / * image data pointers * / TYPE * s 1 = get-image-data (sourcel) ; wPE * s 2 = get-image-data(source2) ; TYPE *d = get-image-data (dest) ; WORD n = get-number-of-images(source1) ; IM-PLAN E ppf = get-planes-per-frame (sourcel) ; DWORD area = get-image-area(source1) ; wpb: *d-limit ; / * Process the image data

...

*/

d-limit = d i (n*p*area) ; while (d
-

*s2+t))) ;

FIGURE 11. Processing every pixel in the same way.

destination images the number of pixels to be processed is calculated. This is the product of the number of frames in the IMAGE structure, the number of planes in each frame, and the area (in square pixels) of each plane (as shown in Table 1). The product is added to the destination pointer to give the address just beyond the last pixel in the destination image. The processing loop iterates over all the pixels in the destination image calculating the absolute difference between the corresponding pixels in the two source images and incrementing all three image-data pointers. There are several points to note about this implementation:

0

The images have already been checked to make sure they are the same size and type and that the destination image of the correct size has been created. This happens in the top-level function (Fig. 9). The iteration could be controlled using a source pointer instead of a destination pointer. A pointer is used since this saves having a loop counter variable and incrementing it. The macro uses a system abs( ) function that uses a long as its argument and result. This makes it unsuitable for floating-point data, so this macro would not be used in the floating-point case of the top-level switch statement. The result of the abs( ) function is cast to the correct type for the destination image.

96

M. R. DOBIE and P. H. LEWIS

3 . Processing One Plane in Each Frame

Some image processing operations are only applicable to a single plane of an image. This example is a function that applies a single threshold to a given plane of an IMAGE structure and generates a binary threshold image which indicates which pixels are above or equal to the threshold and which pixels are below the threshold. The macro that performs the processing is shown in Fig. 12. The extra parameters for this macro are the plane to be thresholded and the value of the threshold. The plane is of type IM-PLANE (see Section IX.C.S), and the value represents the position of the plane within each frame. The macro increments the source image pointer, s, to point to the start of the plane to process. The outer loop iterates over the frames of the image. The inner loop is similar to the differencing example (Section IX.D.2) but only processes one plane of data, incrementing source and destination pointers as it goes. At the end of the inner loop, s points to the start of the following plane, so it is moved forward ppf - 1 planes to point to the start of the required plane in the next frame. The inner loop performs the thresholding using a conditional operator to set the corresponding pixel in the destination image to THRESH-ON or THRESH-OFF. The destination image always has a storage type of BYTE for this function. #define threshold-type(TYPE,source,dest,plane,value) (

/ * image data pointers * / TYPE *s = get_image-data(source) ; TYPE 'd = get-image-data(dest) ;

WORD I M-PLANE DWORD

n

= get-nurnber-o€-images(source) ppf = getglanesger-frarne(source) area = get_image_area(source) ; *s-limit ;

TYPE WORD

; ;

i ;

/ * Process the image data. (plane'area) €or (i=O ; i
s +=

..

*/

;

itt)

I s-limit

for

(;

=

s t

area

scs-Limit

;

; stt,dtt)

* d = (BYTE)( ( ( * s ) >= v a l u e ) ? THRESH-ON

:

THRESH-OFF) s += (ppf-l)*area ;

1

FIOURE12. Processing one plane of an image.

;

97

DATA STRUCTURES FOR lMAGE PROCESSING IN C

4. Processing Several Planes Together There are some image processing operations that work with several planes at once. These are usually only applicable to a particular type of image. This example is a function that converts a red, green, and blue colour image to the hue, saturation, and brightness representation. The function only works with red, green, and blue colour images. The macro which performs the processing is shown in Fig. 13. The macro is just given the source and destination images, which are of fixed types. The outer loop iterates over the frames of the IMAGE structure as in the previous example. The inner loop iterates over the first plane of each frame, but in fact it processes three planes at once. It calls a function, pix-rgb-to-hsb( ), which converts the red, green, and blue pixel values to hue, saturation, and brightness values and places the values at three addresses (given as the last three parameters of the function call). The pixel values are given by referring to the red, green, and blue planes using the values defined for the IM-PLANE type (described in Section IX.C.5). The single source image pointer, s, is combined with the #define rgb-to-hsb-type(TYPE,source,dest) {

image data pointers * / *s = g e t ~ i m a g e ~ d a t a ( s o u r c e 1; *d = get-image-data (destl ; TYPE WORD 11 = get_number_of_images(sourcel IM-PLANE ppf = qet_planes-per_frame(source) DWORD a r e a = get-image-area(source) ; TYPE *d-limit ; WORD i ; /*

TYPE

/ * Process the image data : i
for t i - 0

d-limit

for I

(;

...

;

;

*/

; i i i ]

-

dtarea ;

d
; s i t , d ~ i )

pix_rgb_to_hsb('(s+PLANE-RED*area), *(s+PLRNE-GREEN*area), "(s+PLANE-BLUE*area), dtPLANE_WUE*area, dtPLANE-SATURATION'area, d tPLANE-BRIGHTNESS*area) ; 1 s += Z'area d t = 2'area

; ;

I

FIGURE13. Processing all planes of an image.

98

M. R. DOBIE and P. H. LEWIS

area of a plane to refer to the corresponding pixels in the other planes. The addresses for the hue, saturation, and brightness values are calculated in a similar way relative to the destination image pointer. At the end of the inner loop the source and destination pointers are pointing to the second plane of the frame. Since the whole frame has already been processed, they are incremented to skip over the remaining two planes in the frame to point to the first plane of the next frame. The only point to note about this implementation is that it is limited to operate on the type of data that the pix-rgb-to-hsb( ) function accepts as its parameters. This problem could be alleviated by casting the three input parameters and using temporary variables for the output parameters. The function would have to be rewritten to process values of a type that could be converted to each of the possible storage types.

5 . Performing a Convolution Operation Many image-processing operations can be expressed as a convolution of an image with a small mask (Gonzalez and Wintz, 1987). This example shows how a general convolution routine is implemented in the library. The routine applies a convolution mask to all the planes in an image. The result is a new image with the same number of planes. The intention is to create a routine that can be used for arbitrary convolution operations, The mask should be of an arbitrary size, and the coefficients should be capable of representing arbitrary values. One mask might be applied to several images, so we have separated the specification of the mask from the convolution operation itself. A suitably general representation of the convolution mask is a small image with pixels of type DOUBLE. A library function is provided that creates such an image given the size of the mask and the coefficients. The function uses the ability of C to allow functions with variable numbers of parameters to create a concise function interface. The convolution function itself is now simplified because the mask can be passed as a single parameter. Another point for consideration is the result of the convolution operation. Because the mask has real coefficients, the result of the convolution will be real too. In many cases the result will need to be adjusted so that it becomes a new image. The approach we have taken here is to allow the user to specify a factor and an offset. The convolution result at each pixel is divided by the factor, added to the offset, and converted into the same type as the source image. The processing macro is shown in Fig. 14. This complex macro uses many of the techniques discussed in the previous sections. The macro is called once for each plane in the image. The pointers, s and d , point to the

DATA STRUCTURES FOR IMAGE PROCESSING IN C

99

appropriate planes in the source and destination images. m points to the image representing the convolution mask. We also pass the dimensions of the image and mask and the factor and offset used to transform the result. A problem with image convolution in general is how to interpret the result at the edges of the image, where there is not enough image data to convolve with the mask. One standard solution (adopted here) is to omit #define convolve-plane (TYPE,s,d,m,iw, ih,mw,mh,factor,offset) (

COORD

TYPE COOHD COORD

j ; 'row-lim, 'skip-lim row-skip ; col-skip ;

;

/ ' calculate rows and cols to skip * / row-skip = (mh/2l ; col-skip = (mw/2) ; /*

first mh/Z rows set to offset skip-lim = dtrow-skip*iw ; for ( ; dcskip-lim ; d+t) *d = (TYPE)offset ; s t = row skip*iw ;

*/

/ * process image rows * / for (j=row_skip ; j
/ * process the row a pixel at a time * / f o r ( ; d
last mw/2 columns set to offset * / skip-llm = d+col-skip ; f o r (; dcskip-lim ; dttl *d = (TYPEloffset ; s i - -col-skip ; /*

\

/ * I ~ S mh/2 L rows seL to offset * / skip-lim dirow-skip'iw ; for ( ; d
I

FIGURE14. Performing a convolution operation (outer level).

100

M. R . DOBIE and P. H. LEWIS

the edges from the processing and only calculate a result where there is enough image data to perform the convolution. The first two statements calculate the number of rows and columns t o skip when processing the image data. The two for loops at the start and end of the macro set the first and last rows of the destination plane to the value of offset, as well as skipping these rows in the source plane. The inner loop (indexed b y j ) processes the rows of the image. The appropriate number of columns are set to the value of offset, and in the middle of the loop the pixels are processed using the macro convolve-pixel-type. The macro is passed the TYPE of the pixels, pointers to the current source and destination pixels and information about the mask and image dimensions. The pixel processing macro is shown in Fig. 15. The pointer coeff-ptr is used to access the mask coefficients. This is simply using every pixel in the mask image in turn. The pointer pix-ptr is used to traverse the source image pixels under each mask coefficient. It is initialised to start at the top left of the mask. rskip and cskip are used to move half the mask width to the left and half the mask height up in the image using expressions from Table 1. #define c o n v o l v e _ p i x e l _ t y p e l T Y P E , s , d , m , r s k i p , c s k i p , i w , i h , m w , m h , f a c L o ~ , ~ f f s e L )

i COOKD DOUBLE DOUBLE

TYPE

xoff, y o f f ; result ; *coeff-ptr i *pix_ptr ;

/ * start a t t o p left o f mask * / coeff-ptr = m i pix-ptr = s - (rskip*iw) cskip result = 0.0 ;

-

i

/ * Apply the mask * / f o r lyoff=O

i

yoff<mh ; yofftt)

I tor(xoff=O

;

xoffmw

i

xofftt)

(

DOUBLE

coeff, pixel ;

c o e f f = 'coeff-ptrtt ; pixel = ( D O U B L E ) I*pix-ptrt+) result = result t c o e f f pixel

;

I

I * move pix-ptr under next mask row * / pix-ptr += (iw-mwl ; )

/ * transform t h e result a n d store * I *d = (TYPE)((resiilt/factor) t offset)

;

1

FIGURE15. Performing a convolution operation (inner level).

DATA STRUCTURES FOR IMAGE PROCESSING I N C

101

Two nested for loops traverse the mask coefficients. The inner calculation simply multiplies a source image pixel with a mask coefficient and accumulates the result in the variable result. After processing each row of the mask, pix-ptr is adjusted to point to the correct pixel on the next row in the image. Finally the result is divided by the scale factor and added to the offset. This is converted to the type of the image pixels and stored in the destination image. This implementation is very general and can be used for any operation that can be expressed as a single convolution. This includes averaging, blurring, sharpening, and very simple edge-detection operations. 6. Performing a Neighbourhood Operation Although the convolution routine described in the previous section is very useful, many image-processing methods are expressed as the result of applying several convolution masks to an image and combining the results in some way. If one were to use the previous routine to d o this, all the intermediate result images would have to be saved and a new routine to combine them would have to be implemented. This approach can have a large storage overhead, so it is usually preferable to process the image once, performing several convolutions during a single pass and combining the results in one routine. This example shows how a Sobel edge detector is implemented in the library. The detector applies two convolution masks to an image to calculate horizontal and vertical edge strengths. These are used to calculate the maximum edge strength and edge orientation. The routine is intended to operate on monochrome images but will process the first plane of any image type. It processes the image data and produces an edgemap image which contains two planes per frame, one with edge magnitude information and one with edge orientation information. The processing macro is shown in Fig. 16. This macro is essentially the same as the one used in the previous example (Section IX.D.5) Except that it can make some assumptions about the convolution mask size. The outer loop (indexed by i ) iterates over the frames in the IMAGE structure. Each iteration round this loop processes one plane of source image and generates two planes in the destination image. The last statement in the loop increments the source pointer to skip over any remaining planes in the source frame, since only the first plane in each frame is processed. The last but one statement increments the destination pointer to skip over the extra frame of orientation information that was generated, in a similar way to the example in Section IX.D.3.

102

M. R. DOBIE and P. H. LEWIS

#define sobel-type(TYPE,source,dest) (

TYPE TYPE WORD

*d n

= get_image_data(source) ; = get-image-data (dest) ; = get-number-of-images(source)

;

IM-PLANE

ppf

=

get-planes-per-frame(source)

;

COORD COORD DWORD WORD

width = get-image-widthbource) height = get-image-height(source) area = get-image-area(source)

*S

; ;

;

1 ;

/ * Process the image data

for (i=O

;

icn

;

...

*/

it+)

(

COORD

TYPE

j ; *d-limit ;

/ * first row set to zero * / d-limit = dtwidth ; for ( ; d
;

f o r ( j = 1 ; jchelght-1 ; j++) (

d-limit

=

dtwidth-1

;

/ * first column set to zero * / 'd = *[dtarea) = (TYPEjO ; st+, dtt f o r (;

;

dcd_limit ; stt,dtt) sobel-pixel-type(TYPE,s,d,area,width);

/ * last column set to zero * / *d = *(dtarea) = (TYPE10 ; stt, dtt

;

/ * last row set to zero * / d-limit = dtwidth ; f o r ( ; d
d

t=

s +=

area ; Ippf-l)*area

;

;

) )

FIGURE16. Performing a neighbourhood operation (outer level).

The edges of the image are dealt with in the same manner as the previous example. The Sobel edge detector uses convolution masks of size 3 x 3 pixels, so a border one pixel wide must be ignored all around the image. The two for loops at the start and end of the main loop set the first and last row of the destination planes to zero, as well as skipping these rows in the source plane.

103

DATA STRUCTURES FOR IMAGE PROCESSING IN C

The inner loop (indexed b y j ) processes the remaining rows. The first and last columns are set to zero, and in the middle of the loop the rest of the pixels are processed using a macro to process each pixel. The macro is passed the TYPE of the pixels, pointers to the current source, and destination pixels and the area and width of the images. The pixel-processing macro is shown in Fig. 17. First, the pointers above and below are set to point to the pixels one row above and one row below the current source pixel (using two expressions from Table 1). Next, the horizontal and vertical edge gradients are calculated by applying the Sobel masks in the macro sobel-pixel-type. All the pixel values are cast to type double straight away to avoid overflow problems and to preserve accuracy. These gradients are used to calculate the maximum edge strength and the orientation. The strength is in the same units as the image data and the orientation is transformed to a number between zero and 255. These values are cast to the storage type TYPE before being stored in the appropriate planes of the destination IMAGE structure. # d e f i n e sobel-pixel-type (TYPE,s , d, area, w i d t h )

double

horiz,vert ; *above, * b e l o w ;

TYPE

/ * get horiz and vert edge strengths * / above = s-width ; below = s t w i d t h ; vert

=

horiL

=

(double)* ( a b o v e t l )-(double) * (above-1) t (double)* ( b e l o w t 1 ) - ( d o u b l e )* (below-1) t ( ( ( d o u b l e )* ( s t l ) -(double) * (s-1))* 2 ) ; ( d o u b l e )* ( a b o v e - l ) - ( d o u b l e )* ( b e l o w - 1 )t ( d o u b l e ) * ( a b o v e t l ) - ( d o u b l e )* ( b e l o w t l )t ( ( ( d o u b l e )* a b o v e - ( d o u b l e )* b e l o w )* 2 ) ;

/ * c a l c u l a t e strengtlh and direction * / id = ( T Y P E ) h y p o t( h o r i z / 4 , v e r t / 4 ) ; i f ( ( h o r i z ! = 0 . 0 ) I I (vert ! = 0 . 0 ) ) * ( d t a r e a ) = (TYPE) ((atan2 ( v e r t ,horiz)+M-PI) 127.0 / M-PI) ; else * ( d t a r e a ) :0 ;

FIGURE17. Performing a neighbourhood operation (inner level).

*

104

M. R. DOBIE and P. H.LEWIS

This implementation will work for any storage type since all the intermediate calculations are performed as doubles, so accuracy is preserved throughout. Performing intermediate calculations using the type double would also solve the type problem with the function in the colour conversion example (Section IX.D.4), but at the expense of speed.

E. Efficiency Considerations One of the aims during the design and implementation of this library has been efficiency. There are trade-offs to be made between types of efficiency. In the design of the function interfaces pointers have been used to pass whole image structures to functions. This gives efficiency of expression as well as a speed advantage when compared to functions with many parameters, The main area where efficiency gains can be made is in the implementation of the processing routines themselves. Here a decision was made to achieve speed efficiency over space efficiency. It is well known that sequential access to memory using pointer manipulation is faster than similar access using an array (Kernighan and Ritchie, 1988, page 97). Given the static nature of arrays in C, the decision was made to use pointers and dynamic memory allocation throughout the library. This allows the processing of images of arbitrary size without the overheads of wasted memory and without a maximum size constraint imposed by the library. 1. Pixel Manipulation

In the lowest level macro of a function implementation (like those described in Sections IX.D.2 to IX.D.6) all the pixel manipulation is performed with pointers. Typically there is at least one pointer to source image(s) and a pointer to a destination image. For a rectangular image the method used to traverse the image data depends on the operation being applied. For a simple pixel operation (like the image differencing described in Section IX.D.2) one loop is sufficient, incrementing the pointers for each pixel processed. For an operation using x and y coordinates, two nested loops are used with the pointers being incremented in the inner loop. For a neighbourhood operation, surrounding pixel data is accessed by adding and subtracting offsets from the pointers, as shown in Section IX.D.6 and Table 1. The expressions in Table 1 are easily extended for larger neighbourhoods. For an arbitrarily shaped image, there is a BOUNDARY structure associated with the image’s image size WINDOW, which is described in Section 1X.B. 1. The actual data stored in the IMAGE structure corresponds to the rectangular

DATA STRUCTURES FOR IMAGE PROCESSING IN C

105

bounding area which encloses the shape of the image. There is some space redundancy here, as more data is stored than strictly required, but this is offset by the fact that the simple pointer manipulations just described can be still be used for traversing arbitrarily shaped images, leading to simpler, faster, and more robust code. The only difference need be in the loops which control the traversal of the data. These loops consist of an outer loop (to follow the linked list in the BOUNDARY structure) and an inner loop to move along each horizontal line segment. However, if it is not vital that only the area of the image is processed, then the whole rectangular bounding area can be processed as though it were a rectangular image. For example, to display an arbitrarily shaped image, it is vital that only pixels within the image are displayed. To threshold the image it doesn’t matter if all the pixels in the bounding rectangle are processed, as long as all the image pixels are thresholded. Clearly, there are more trade-offs here. Treating the image as rectangular allows the same (simpler) code to be used for all cases. The disadvantage is that more data is processed than is strictly required. The cost of processing the extra pixels needs to be weighed against the extra complexity of the traversal loops. This depends on the cost of processing an individual pixel and the particular shape of the image. Note that for the rectangular case, no special action has to be taken to account for edge effects in neighbourhood operations, whereas the traversal code for a shape would have to be altered to allow a border around the arbitrary area. So far, arbitrarily shaped images have been treated as rectangular except where the shape of the image is an integral part of the operation. These cases include image display, template matching, and generalised Hough transform matching.

F. Portability Considerations During the implementation of the library an effort was made to keep the C code portable. It is very difficult to write code that is portable across many platforms, but there are several easy techniques which can make the porting process much simpler. The image-processing library was originally developed under Sun UNIX using a compiler that supports the original Kernighan and Ritchie variant of C (described in Kernighan and Ritchie, 1978). It has since been ported to several other UNIX workstations and PC systems running UNIX and Microsoft windows. There are several different types of problems that are encountered when porting software from one platform to another. These are concerned with the way that facilities are provided on different systems. The aim is to

106

M. R. DOBIE and P. H. LEWIS

keep the amount of system-dependent code in the library as small as possible. The following list describes the areas where problems occur.

Hardware: There are obvious differences in hardware between systems, but the programmer is often protected from these by the operating system. Access to some hardware (for example, a file system) is defined by the standard C libraries. Problems occur when the methods for using other common hardware (for example, a serial port) differ from operating system to operating system. There are also problems using more specialised hardware (for example, a frame capture board). Systems: This category refers to facilities that are not provided by the operating system or the standard C libraries but are common on many platforms. An example is a graphical user interface (GUI), where a common operation is to display an image in a window on the screen. The implementations for this can vary considerably from system to system. Standard library: Although the standard C libraries are defined, the implementations vary on different systems. The degree of variation ranges from slightly different functionality to some functions being omitted altogether. Language: There are two main variations of the C language. The original language (described in Kernighan and Ritchie, 1978) is known as Kernighan and Ritchie (or K&R) C, named after its developers. The second variety (described in Kernighan and Ritchie, 1988) is an attempt by the American National Standards Institute to standardise the C language. This variation (known as ANSI C) can differ considerably from K&R C. K&R C is a subset of ANSI C, so any program written in the earlier language should be accepted by an ANSI system. Compilers: There are some aspects of the C language that are undefined and left to be decided by the compiler. This makes it easier to implement a conforming compiler on a wider variety of systems but allows more pitfalls for the inexperienced programmer to fall into. For example, the actual size of the basic C language types is largely implementation defined, Some compilers offer additional features to the language, and it may be necessary to use these features to enable the library to be compiled. For example, under Microsoft windows the library needs to take advantage of the “huge” memory model provided by the compiler. The library source code must accommodate all these differences, yet still compile on other platforms without the need for substantial modifications. The techniques that have been used to achieve this are described in the following sections.

DATA STRUCTURES FOR IMAGE PROCESSING IN C

107

1. Graphical User Interface Portability

Many systems today support graphical user interfaces (GUIs), but the programming interface to use them differs considerably. The DEVICE structure in the image library (described in Section IX.C.3) is an attempt to provide a system-independent method for displaying an image using a GUI. The code for implementing the DEVICE functions will be completely different on different systems and will not even compile on other systems. For this reason, the code for different systems is kept in separate source modules and the library makefile only builds the module required for the current system. Consider the function that displays a DEVICE in a window. There are two versions of the function (one for X windows and one for Microsoft windows). Each has two parameters. The first is a pointer to a DEVICE structure, and the second is a system-dependent identifier in which the window can display the image. Because of this system-dependent parameter, the two versions are named differently (the library has xwin-display( ) and mswin-display( )). The library source code that calls these functions should only refer to the function by one name, so that it is independent of which system is being used. This can be achieved by using a macro to map one generic name, win-display( ) to either xwin-display( ) or mswin-display( ). This is done with conditional compilation in a header file that defines the macros.

2 . Standard Library Portability Many aspects of the C standard libraries have been defined by ANSI, but there are still areas where they differ between systems. One example is the naming of header files. This is simply accommodated by using conditional compilation to include the relevant header file. Some standard libraries omit functions. In this case extra code has to be written to provide the missing functionality, but this code is only required for compilation on one system. Conditional compilation is again used to decide which piece of code to compile. As an extreme case, it is not possible to refer to the standard input, standard output, or standard error output when using Microsoft windows. A consequence of this is that the library’s error handling code has to be rewritten to display error messages in a window rather than on the error output. Another major problem is encountered with the standard library memory management functions. The smaller size of the basic C types in Microsoft C means that the standard memory functions cannot manipulate memory blocks larger than 65536 bytes, but there is a set of Microsoft windows functions that can do this.

108

M. R. DOBIE and P. H. LEWIS

To solve this problem the library uses its own set of memory management functions. On most systems these functions are just mapped on to the standard library memory management functions. On a Microsoft windows system they are mapped to a specially written set of functions. This is achieved in the same way as the GUI functions described in Section 1X.F. 1.

3. Type Portability The basic types of the C language are largely implementation-defined. For this reason, the library cannot assume that a given basic type can take a particular range of values. This problem is overcome by the library defining its own set of basic types. These are the scalar types described in Section IX.C.5. Each scalar type is defined in terms of C basic types, but the definition may be different from system to system. For example, on a SUN system the type DWORD (which represents 32 bits) is defined as the C type unsigned int, but on a Microsoft system the type DWORD has to be defined as unsigned long, since the type int is smaller on the Microsoft system. There is a separate set of definitions for the library storage types for each system to which the library has been ported. At compilation time the correct set of definitions is selected by setting a symbol in the makefile, which controls how the library is built. The library code uses these types instead of the C language types.

X. CONCLUSIONS The future for imaging processing is particularly bright. All forecasts suggest that the field will go on expanding for several years to come. There will be a continuing demand for new software to satisfy the many changing and developing application areas, and an essential requirement will be for well-engineered, portable code meeting the emerging national and international standards. In this chapter we have attempted to summarise many of the issues surrounding the development of image processing software in C, concentrating in particular on the design of sound structures for data and code. We have described how an object-oriented approach can be taken to provide routines that are easy to use, flexible, and robust. We have illustrated our discussion with examples from our own libraries and further information and copies of the library may be obtained from the authors.

DATA STRUCTURES FOR IMAGE PROCESSING IN C

109

REFERENCES Banks, S. (1990). Signal Processing, Image Processing and Pattern Recognition. Prentice Hall, Englewood Cliffs, New Jersey. Bluni, C., Hofmann, G., and Kroemker, D. (1991). “Requirements for the First International Imaging Standard,” IEEE Computer Graphics and Applications, 61-70. Carlsen, I. C., and Haaks, D. (1992). “Concept and Implementation of an Object-Oriented Framework for Image Processing,” Philips Journal of Research 46(6), 3 1 1-340. Carter, M. K., Crennell, K. M., Golton, E., Maybury, R., Bartlett, A, Hammarling, S., and Oldfield, R. (1989). “The Design and Implementation of a Portable Image Processing Algorithms Library in FORTRAN and C,” Proceedings of the 3rd IEE International Conference on Image Processing and Its Applications, 5 16-520. Clark, A. F. (1992). “The International Image Processing Standard-The Story So Far,” Proceedings of the 4th IEE International Conference on Image Processing and Its Applications, 575-578. Dobie, M. R., and Lewis, P. H. (1991). “Data Structures for Image Processing in C,” Partern Recognition Letlers 12, 457-466. Foley, J. D., van Dam, A., Feiner, S. K., and Hughes, J. F. (1990). Computer Graphics. Principles and Practice. Addison-Wesley, Reading, Massachusetts. Gargantani, I. (1982). “Linear Oct-trees for Fast Processing of Three Dimensional Objects,” Computer Graphics and Image Processing 20(4), 365-374. Gonnet, G. H. (1984). Handbook of Algorithms and Data Structures. Addison-Wesley, Reading, Massachusetts. Gonzalez, R. C., and Wintz, P. (1987). Digital Image Processing. Addison-Wesley, Reading, Massachusetts. Hamey, L. G. C., Webb, J. A., and Wu I-Chen (1989). “An Architecture Independent Programming Language for Low Level Vision,” Computer Vision, Graphics and Image Processing 48, 246-264. Hartman, N., and Tanimoto, S. L. (1984). “A Hexagonal Pyramid Data Structure for Image Processing,” IEEE Transactions on Systems, Man and Cybernetics 14(2). 247-256. Haykin, S. (1986). Adaptive Filter Theory. Prentice Hall, Englewood Cliffs, New Jersey. Horn, B. K. P. (1986). Robot Vision. The MIT Press, Cambridge, Massachusetts. Iyengar. S., and Gadagkar, H. (1988). “Translation Invariant Data-structure for 3D Binary Images,” Pattern Recognition Letters 7, 313-318. Jackins, C. L., and Tanimoto, S. L. (1980). “Oct-trees and Their Use in Representing ThreeDimensional Objects,” Computer Graphics and Image Processing 14(3), 249-270. Jaehne, 9. (1991). Digital Image Processing. Springer Verlag, Berlin. Kernighan, 9 . W., and Ritchie, D. M. (1978). The C Programming Language. Prentice Hall, Englewood Cliffs, New Jersey. Kernighan, 9. W., and Ritchie, D. M. (1988). TheC Programming Language, 2nd ed. Prentice Hall, Englewood Cliffs, New Jersey. Lawton, D. T., and McConnell, C. (1988). “Image Understanding Environments,” Proceedings of the IEEE 76(8), 1036-1050. Low, A. (1991). Introduclory Computer Vision and Image Processing. McGraw-Hill, New York. Page, E. S., and Wilson, L. B. (1983). Informarion Representation and Manipulation Using Pascal. Cambridge University Press, Cambridge, England. Paulus, D. (1992). “Object Oriented Image Segmentation,” Proceedings of the 4th IEE International Conference on Image Processing and Its Applications, pp. 482-485.

110

M. R. DOBIE and P. H. LEWIS

Piper, J., and Rutovitz, D. (1985). “Data Structures for Image Processing in a C language and UNlX Environment.” Pattern Recognition Letters 3, 119-129. Rosenfeld, A., and Kak, A. C. (1982). Digital Picture Processing, Vols. 1 and 2. Academic Press, Boston. Samet, H. (1984). “The Quadtree and Related Hierarchical Data Structures.” Computer Survey 16(2), 187-260. Schalkoff, R. J. (1989). Digital Image Processing and Computer Vision. John Wiley and Sons, New York. Scott, D. S., and Iyengar, S . S . (1986). “TID-A Translation Invariant Data Structure for Storing Images,” Communications of the ACM 29(5). 418-429. Sleigh, A., C., and Bailey, P. K. (1987). “DIPOD: An Image Understanding Development and Implementation System,” Pattern Recognition Letters 6(2), 101-106. Tanimoto, S., and Klinger, A. (1980). Structured Computer Vision: Machine Perception Through Hierarchical Computation Structures. Academic Press, New York. Wang, L., and Goldberg, M. (1989). “Reduced-difference Pyramid: A Data Structure for Progressive Image Transmission,” Optimal Engineering 28(7), 708-7 16. Wang, W., and Iyengar, S . S. (1992). “Efficient Data Structures for Model Based 3D Object Recognition and Localization from Range Images,” IEEE Transactionson Pattern Analysis and Machine Intelligence 14(10), 1035-1045. Wyk, C. J. Van (1988). Data Structures and C Programs. Addison-Wesley, Reading, Massachusetts. Yau, M.-M.,and Srihari, S. N. (1983). “Hierarchical data Structure for Multidimensional Digital Images,” Communications of the ACM 26(7), 504-515.

ADVANCES IN ELECTRONICS AND ELECTRON PHYSICS. VOL. 88

Electron Crystallography of Organic Molecules DOUGLAS L. DORSET Electron Diffraction Department, Medical Foundation of Buffalo, Inc., Byffalo, New York Nuscht macht dem Forscher so gnatzig wie Zeijen die er selbst zerdappert hat! Siegfried Lenz

. . . . . . . . . . . . . .

I. Introduction. . . . , . . . . . 11. Historical Background . . . . 111. Methodology . . . . . . A. Crystal Growth . . . . . . . B. Electron Diffraction , . . . . . . . . C. Electron Microscopy . . D. Determination of Crystallographic Phases E. Structure Refinement . . . . IV. Perturbations to Diffraction Intensities . . . . . . A. Dynamical Scattering . B. Secondary Scattering . . . . . C. Diffraction Incoherence . . . . . . D. Radiation Damage . . . . . V. Applications , . . , . , . . . . A. Single-Crystal Structures . . . . . 9. Imperfect Crystals , . . . . VI. Conclusions . . , . , . . , . . . . . . . Acknowledgments . . . , . . References . . , . , .

.

.

. . . .

. .

. . .

.

. . . . . . . . . . . . , . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . , . .

. . . , . . . . . . . . . . . . . . . . . . . , . . . . . . . . . , . . , . . . . . . . . . . . , . . . . . . . , . , . . . . . . . . . . . . . . . . . . . . . . , . , . . . . . . . . . . . . . , . , .

,

.

,

.

. . . . . . . . . . . .

, . . . . . . . , . . . . . . . . . . . , . . . . . . . . . . . . . . . . . . . . . . . . . . . , . , . . . . . . . . .

111

I14 117 117

122 126 130 143 146 146 151 152 156 157 157 171 184 185 185

I. INTRODUCTION Electron crystallography is a term that has been used to describe a quantitative crystal structure determination based on electron diffraction intensity data and/or high resolution electron microscope images. In the first sense, it is very similar to x-ray crystallography, requiring the same sort of procedures for finding crystallographic phases to derive a potential map with details of the molecular architecture. In the latter sense, it is an optical technique, wherein the details of the molecular structure are visualized, one hopes, in a phase-contrast image, perhaps after some sort of averaging technique is used to emphasize the repeating motif in the crystal projection. Since any converging lens (Fig. 1) has principal planes where, respectively, the diffraction pattern or image of the object are to be found, the two 111

Copyright 0 1994 by Academic Press, Inc. All rights of reproduction in any form reserved. ISBN 0-12-01473CM

112

DOUGLAS L. DORSET

I

Back focal Dlanc2

Imagu

FIGURE 1 . Planes of a converging lens, e.g., in an electron microscope. At the back focal plane, the diffraction pattern of the thin crystal object is produced before its image. The specimen is copper perchlorophthalocyanine. (Electron micrograph taken by J. R. Fryer at Medical Foundation of Buffalo, Inc.)

techniques are, of course, intimately connected (via the Fourier transform operator)-and the combination of information from both planes provides a potential advantage not realizable by any other crystallographic approach. Although the interpretation of x-ray diffraction data from organic crystals has been a mainstay of modern structure research and will continue to hold its dominant position, there are some good reasons why electron crystallography, at times, would be a procedure to be preferred over x-ray methods. The major reason that has attracted users in many applications of materials science and biology is the scattering cross-section of matter for electrons compared to x-rays or neutrons. As pointed out by Vainshtein (1964a), for example, in terms of intensity, matter scatters electrons lo6 more efficiently than x-rays. Thus, if a sample is difficult to prepare in anything other than a thin microcrystalline plate, then electron diffraction

ELECTRON CRYSTALLOGRAPHY OF ORGANIC MOLECULES

113

(or microscopy) may provide the only simple way to visualize the undistorted reciprocal (or space) lattice from a single crystal. X-ray experiments on an aggregate of the same sample, on the other hand, would produce powder or fiber patterns where the reciprocal lattice would be randomly oriented over a certain angular range (Klug and Alexander, 1974; Alexander, 1985) and, if different sets of crystal planes have nearly the same reciprocal spacing, any diffraction amplitude may include numerous overlapping contributors, The second possible advantage of electron diffraction over x-ray techniques is that the scattering factor ratios for certain pairs of atoms have different values depending on the radiation used (Doyle and Turner, 1968; Vainshtein et al., 1992). For example, there was once considerable interest in exploiting electron diffraction techniques to locate hydrogen atoms accurately in organic crystal structures (Vainshtein, 1964b). Hence, electron diffraction analysis enjoys a long history as a technique used in organic structural research, at least in its qualitative application for determining space-group or plane-group symmetry as well as unit cell constants. This is particularly true in polymer science, as seen in compendia published by Geil (1973) and Wunderlich (1973). Even in fiber or powder x-ray analysis of some materials, electron diffraction patterns are often employed to help the indexing of the diffraction maxima so that a quantitative analysis can then be carried out based on the nearly kinematical intensities from the disordered bulk specimen (Atkins, 1989). Although quantitative structure analyses based on electron diffraction patterns were also begun at an early date (Rigamonti, 1936), it would be incorrect to state that this approach enjoys the same favor in the spectrum of crystallographic methods as does x-ray crystal structure analysis. The same estimate can be made about the interpretation of high-resolution electron micrographs of organic lattices, despite the possible utility of this methodology for producing useful results at suitable resolution to portray molecular packing geometry. It will be a primary objective of this review, therefore, to show that, while electron crystallographic techniques certainly are subject to their own unique constraints for collection of scattering data that can be interpreted directly in an ab inito structure determination (Cowley, 198l), the overwhelmingly negative view of these techniques by much of the crystallographic community often is, in fact, totally unwarranted and is based largely on a number of unfortunate historical misunderstandings. It will be demonstrated, moreover, how experimental diffraction intensity data or images can be used in structure analyses of a wide variety of organic and biological materials to yield results that can be compared favorably with x-ray crystal structures, but, on the other hand, do not require the existence of an x-ray structure for such analyses to be carried out.

114

DOUGLAS L. DORSE’I

11. HISTORICAL BACKGROUND In the earliest studies, the major application of electron diffraction experiments to organic molecules was to study boundary lubricant films on various metal surfaces. Unlike the transmission experiment that is nowadays most often employed for quantitative structure investigations, most of the early work employed the Bragg surface diffraction geometry (see James, 1950), which is nowadays termed RHEED (reflection highenergy electron diffraction). The glancing angle method, which probes the outermost surface of the crystal packing, was used most often to determine how the alkyl chains of various polymethylene chain compounds (mostly heavy metal soaps) were inclined to the surface normal at various temperatures. The history of this application has been reviewed recently (Dorset, 1990a), and it is also seen that the methodology has enjoyed a renaissance after many years of inactivity, given the interest in using Langmuir-Blodgett films to construct various devices (e.g., Peterson and Russell, 1984; Fryer et al., 1985;Garoff et al., 1986; Jones et al., 1986; Heard et al., 1988). In the early work, it is significant to mention that very careful diffraction experiments could be used to depict the polymethylene chain lattice, even in a quasi-single-crystal array (Thiessen and Schoon, 1937),and that accurate values of chain orientation could be obtained from several polymorphs of a single compound that match single-crystal x-ray structures well. Methods were also found to determine average chain tilt for more disorderd polycrystalline arrays on a support surface (Karle and Brockway, 1947). The initial transmission electron diffraction experiments on organic specimens also examined the structure of polymethylene chain compounds. After the first powder diagrams of these materials (Lebedeff, 1931), singlecrystal patterns were soon obtained from paraffins, fatty acids, and wax esters (Thomson and Murison, 1933;Natta et al., 1935), and data from the former compounds were employed for the first quantitative structure analysis (Rigamonti, 1936). In this analysis, values were sought for the “setting angle” of a paraffin chain to one of the orthorhombic unit cell axes, yielding a value that is only slightly different from modern measurements. The C-H bond distance obtained from this determination was somewhat too large. Nevertheless, these results, based on visual estimate of diffracted intensities and scattering factors calculated from the Mott formula, were quite good. Five years later, the second determination of an organic structure, i.e., l-methyl-7-isopropylphenanthrene,was reported from Moscow (Karpov, 1941) and served as the basis of a large-scale effort in “electronography” in the Soviet Union, which has been recently reviewed (Vainshtein el al., 1992). For this determination, 35-40 kV electrons were

ELECTRON CRYSTALLOGRAPHY OF ORGANIC MOLECULES

115

used in a diffraction camera with a specimen stage allowing tilt orientation of the samples. Both single-crystal and texture-diffraction data were used in the analysis, and, after model building, there was a generally good agreement with the visually estimated intensity values. A decade later, a more accurate determination of the paraffin crystal structure was published by a Soviet group (Vainshtein and Pinsker, 1950), and, in this study, the potential use of the technique for accurately determining hydrogen atom positions was mentioned. It was assumed, moreover, that the positions of the carbon backbone could be taken from x-ray determinations so that the refinement process would locate the lighter proton positions. This methodology was used to determine a large number of other organic and light atom structures, summarized in Table 1, where, again, the heavy atom positions were taken from contemporary x-ray structures and the hydrogens found by difference Fourier syntheses. (Of course, this required that these syntheses would have sufficient accuracy [Vainshtein, 1956al.) At the same time, more accurate electron atomic scattering factor tables were determined (Ibers and Vainshtein, 1959; Vainshtein and Ibers, 1958) and Lorentz corrections established for a number of crystalline or polycrystalline textures (Vainshtein, 1956b,c; 1957). In all of this work, it was assumed that any deviations from kinematical scattering could be treated in much the same way as primary TABLE 1 QUANTITATIVE ELECTRON DIFFRACTION ANALYSES OF LIGHTATOMSTRUCTURES Material

Type of data

R-value

Reference Cowley (1953) Lobachev (1954) Vainshtein (1955) Vainshtein el o/. (1958) Lobachev and Vainshtein (1961) Dvoryankin and Vainshtein ( 1960) Dvoryankin and Vainshtein (1962) Udalova and Pinsker (1964) Kuwabara (1959)

Boric acid Urotropine Diketopiperazine n-paraffins Urea

Spot (hM)) Powder Texture (hkl) Spot + texture spot (/lo/)

0.20 Not given 0.20 0.12 0.23

Thiourea (25°C)

Texture (hkl)

0.20

Thiourea (-140°C)

Texture ( h k l )

0.18

Ammonium sulfate Ammonium chloride

spot (hkO + Okl) Powder (hkl)

0.36 0.10 0.08

(25°C) (- 40°C)

Copper DL alaninate Mixed copper (11) complexes: DL alanine and DL serine

Spot +texture (hkl) spot ( h k f )

0.19 Not given

Vainshtein e t a / . (1971) D’yakon er a/. (1977)

116

DOUGLAS L. DORSET

extinction corrections are carried out in x-ray crystallography (Vainshtein and Lobachev, 1956)-i.e., the Blackman (1939) two-beam model was assumed to be valid, even for an imagined mosaic single crystal, where each reflection was proposed to originate individually from appropriately oriented crystallite blocks. It is correct to state that these results were not often well received by the crystallographic community. For example, because of significant deviations from kinematical scattering observed for heavier atom structures, which could not be predicted by the Blackman theory, theoreticians began to formulate a more accurate model for dynamical electron scattering, which required that the coupling of all simultaneously excited electron beams be considered (Cowley and Moodie, 1957), rather than assigning individual intensity deviations to separate, localized two-beam interactions with the incident beam. It soon became clear that such a model was much more successful for explaining the details of this dynamical scattering, and, because such a model is also structure-dependent (i.e., requiring foreknowledge of a crystal structure before it is solved), it was imagined that ab initio determinations may not be possible with electron diffraction intensities (Heidenreich, 1964). The pessimistic viewpoint persists that a quasi-kinematical interpretation of diffraction intensities “is generally of little quantitative use” (Humphreys and Bithell, 1992). X-ray crystallographers, furthermore, were dissatisfied by the somewhat high crystallographic R-factors reported in these early determinations (see Table 1). Worse than that, the fact that structures determined by another technique were used as a starting model for the electronographic analyses meant that most of the crystallographic phase information was provided a priori. Since it is known that Fourier maps calculated with scrambled or unitary structure factor amplitudes still resemble the actual map if the crystallographic phases are correct (Ramachandran and Srinivasan, 1976), it was argued that these early studies never established the ab intio determinations were indeed possible. (This is somewhat of a misinterpretation, since, e.g., for layer silicate structures, model building, and interpretation of Patterson functions in another contemporary Soviet effort [Zvyagin, 1957; Zvyagin and Mischenko, 1961, 1963; Zvyagin, 1967; see also Vainshtein et al., 19921 played a significant role in solving these crystal structures with electron diffraction intensities.) The possible use of electron micrographs for depiction of organic crystal structures was met with similar skepticism. Although considerable progress was made in the high-resolution electron microscopy of inorganic specimens, it was also recognized that dynamical distortion of phase values might often lead to a lattice image that could not be interpreted directly in terms of an atom packing motif (Spence, 1980; Cowley, 1981). Even the

ELECTRON CRYSTALLOGRAPHY OF ORGANIC MOLECULES

117

uniqueness of lattice simulations, where dynamical scattering and the objective lens transfer function have been included, has been questioned (Humphreys and Bithell, 1992). These elastic scattering limitations, however, were not the ones imagined to be most serious detriment to the direct visualization of organic structures. Rather, the inelastic interactions of the electron beam with the sample, leading to its damage, was thought to be the major limiting factor, and this was a subject that captivated the (pessimistic) attention of many workers for a long time, even leading to workshop meetings on the subject (Zeitler, 1982). It was recognized that the radiation sensitivity of organics could be differentiated, with n-delocalized molecules being much more stable in the electron beam than were aliphatics (Reimer, 1975), thus explaining how early images of the platinum phthalocyanine crystal could be recorded at 12 A resolution (Menter, 1956). From the Rose equation, it was imagined that the case of the aliphatic molecules was nearly hopeless (Thomas and Ast, 1974; Glaeser, 1975), with best detectable resolutions in direct images predicted from 30 to 100 A. Biologists dealing with macromolecules, accordingly, encased their preparations in some sort of “negative stain” comprised of a heavy metal salt, and were content with the 20 A resolution of the remaining aggregate seen in their micrographs (Vainshtein, 1978) after “normal” radiation doses were used that were equivalent to the detonation of a 10-megaton nuclear device 30 meters from the sample (Grubb, 1974)! (Nevertheless, the coupling of crystallographic principles with image analysis on such preparations resulted in procedures that serve as a framework for modern electron crystallography [DeRosier and Klug, 19681.) Thus, unless Moire techniques were employed, it was imagined that no useful direct micrographs could be obtained from aliphatic materials at molecular resolution because of their extreme sensitivity to the electron beam. Gradually, this viewpoint was changed after low-dose imaging techniques were developed to make the most efficient use of the electron beam. Image analysis (e.g., Fourier filtration) was then used on the statistically noisy electron micrographs to produce an average representation of the unit cell (Amos et al., 1982). Ill. METHODOLOGY A . Crystal Growth

1. Solution Crystallization Solution crystallization has served as a basis of numerous electron diffraction and electron microscopic experiments and is actually based on rather simple procedures (Dorset, 1985a). For small molecules, it is generally

118

DOUGLAS L. DORSET

sufficient to prepare a dilute solution of the compound in a suitable solvent and then to dry a drop of this solution on a carbon-film-covered electron microscope grid. These are examined in the electron microscope at low beam currents and at a suitably low magnification (e.g., 1 0 , 0 0 0 ~ )to estimate the thickness (by transmission of the beam). One can, of course, shadow these specimens at a known oblique angle with C-Pt to obtain a more accurate value of the average thickness (Hall, 1966). If the crystals are found to be too thick, then the stock solution is merely diluted somewhat until favorable results are obtained. Other equivalent methods of preparation include growth from the melt, sublimation in vacuo from a heated boat containing the sample (Uyeda et af., 1972), and sonication of a larger crystal in a nonwetting solvent into smaller fragments (Vainshtein et al., 1992). Aside from the sublimation technique, solution growth is the easiest way to control specimen thickness, although samples crystallized from the melt can be thinned by rubbing if they are deposited onto some hard surface such as a glass slide, a metal block, or a salt-crystal face (Menter and Tabor, 1951). Such a worked specimen can then be coated with a carbon film and then floated onto a water surface to be picked up with bare electron microscope grids. For linear polymer samples, solution crystallization has to be somewhat better controlled to achieve useful specimens. The technique of “selfseeding” of an infinite polymer requires a solvent to be used that permits only limited solvation at room temperature (Blundell et a f . , 1966). A suspension of the polymer is made in this solvent at a low concentration, and this is raised in temperature until total solution is achieved. The solution is then cooled to produce a cloudy suspension that consists mainly of dendritic crystals. The temperature is again raised until the suspension just clears. After adjusting to a slightly lower constant temperature (but still well above ambience) single crystals are nucleated by the remaining seeds, which remain in suspension. After the growth is achieved, the suspension can be cooled again to ambience, and the preparation for electron crystallography is made by placing a drop of the suspension on the carbon film of the grid surface, allowing the crystals to settle onto the support film before the drop of solvent is blotted away. 2 . L angmuir-Blodget t Films Specimens formed on a Langmuir trough are similar to those preparations that might be grown by evaporation of a dilute solution, because the molecular axes are more or less perpendicular to the plane of the resulting organic layer (Jensen, 1970). However, single monolayers are possible with this technique when the surfactant molecules are aggregated on the water

ELECTRON CRYSTALLOGRAPHY OF ORGANIC MOLECULES

119

surface by compression with a floating barrier to a desired surface pressure (Losche et al., 1984). Carbon-covered electron microscope grids can be stored in the bulk water face below this monolayer film, and, as they are drawn through the organic film, a single monolayer can be deposited for examination in the electron microscope. Whether or not a polar or nonpolar surface of the organic layer adheres to the carbon film depends on how it was prepared before the experiment. For example, hydrophilic surfaces can be made by glow discharge of a gas around the carbon film before it is used (Hui, 1989). Either of these preparations can then be dried before they are examined in the microscope, and such layers serve as the basis for many of the LangmuirBlodgett (LB) films examined in the electron microscope, as shown in many early studies (e.g., Germer and Storks, 1938; Havinga and DeWael, 1937). Alternatively, the solvation of organic specimens can be preserved if the sample is quickly frozen by plunging into a suitable cryogen (such as liquid ethane), requiring the use of a cryo-transfer specimen holder and a lowtemperature anticontamination device in the electron microscope (Chanzy et al., 1977). Specimens have also been prepared as free-standing bilayers spanning small apertures on, e.g., 1000-mesh electron microscope grids. This requires transfer at 100% relative humidity to a special differentially pumped environmental chamber (Hui et al., 1976), as described by Hui (1989). 3. Epitaxial Orientation It is often useful to have a crystal projection of the specimen that is nearly orthogonal to the one achieved by the techniques discussed above. Most commonly, this is achieved by some epitaxial orientation technique on a suitable substrate with a surface lattice spacing that nearly matches some desired set of densely packed planes in the crystal. Historically, the most frequently used substrates for epitaxial nucleation have been inorganic salt faces (Uyeda et al., 1972; Ueda and Ashida, 1980). For this application, the substrate crystal is freshly cleaved and then outgassed by heating in vacm desorb adherent polar substances. After cooling this substrate to some lower temperature, a thin film of the organic substance is deposited by heating a perforated metal boat containing it. The organic material will sublime so that the vapor will recrystallize onto the salt face. The choice of substrate temperature is important for determining whether or not an epitaxial orientation is achieved (Fryer, 1979a). After the organic layer is formed, a thin carbon film can then be deposited on top of it from a second source. Air is admitted into the vacuum chamber, and the salt plate containing the organic film is then removed. The organic film is then floated away on a clean water surface to be picked up with bare

120

DOUGLAS L. DORSET

electron microscope grids, and the preparation is allowed to dry in air before it is examined in the electron microscope. Other procedures for epitaxial orientation on a salt crystal require that it should be cleaved after immersion into a solution of the organic material in a suitable solvent, and crystallization is achieved by adjusting the temperature of the solution (Wellinghoff et al., 1974). As shown by the extensive work of Wittmann and Lotz (1990, 1991), organic substrates are often more convenient for epitaxial orientation of such samples for several reasons. First, organic substrates generally nucleate with a nonpolar surface. Hence, if necessary, they can be stored in air without being contaminated by adsorbed gases or water. Second, the organic layers crystallize in a lower symmetry than the inorganic compounds. Hence the epitaxial nucleation is anisotropic, allowing crystalline films to be formed with the same orientation extending over great distances (e.g., millimeters). Third, they can also be easily removed after the overgrowth is achievedin some cases by floating on a water surface, if some organic salt is used, for example, or by sublimation in vacuo. Preparations can be made by evaporation from a source (e.g., experiments using potassium hydrogen phthalate as a substrate) or by cooling a co-melt. The latter cooling of a co-melt can be done in the bulk phase as shown by Fryer (1981) or in a thin film. In the latter technique (Dorset, 1989a), a thin layer of the substance to be studied is deposited onto a glass or cleaved mica sheet by evaporation of a dilute solution in some solvent. A thicker layer of the nucleating material is laid down on top of this thin film (e.g., by simply adding crystals), and a second glass slide or mica sheet is placed on top of the physical mixture to form a sandwich. The sandwich is then moved on a temperature gradient to melt the ingredients and then translated to a cooler end to cause recrystallization. After mechanically opening the sandwich, a carbon film can be deposited onto the surface and the assembly removed by floating onto a water surface. Alternatively, the carbon-covered grids could have been added to the sandwich before it was closed with the cover slide, and the nucleating material removed by sublimation in a vacuum after the sandwich is reopened. Before the assembly is reopened after initial crystallization, it is also possible to anneal the samples at some temperature on a controlled heating stage (this is advantageous for the growth of polymer crystals). The whole process has been described in terms of an eutectic phase diagram (Hodge et al., 1982; Dorset et al., 1989b; Dorset, 1990b).

4. Reconstituted Layers Considerable effort has been made to apply electron crystallography to the study of two-dimensional protein crystals, most notably those integral

ELECTRON CRYSTALLOGRAPHY OF ORGANIC MOLECULES

121

proteins found in membrane bilayers. In some instances, two-dimensionally crystalline motifs already exist on a membrane surface, as found in a famous example, the purple membrane of Halobacterium halobium (Henderson and Unwin, 1975). Much less ordered preparations have been found in, e.g., the VDAC porin array in the outer membrane of mitochondria (Mannella et al., 1983) and the crystalline order of these can be improved by light digestion with a phospholipase (Mannella, 1984). In many cases, however, the constituent proteins are not packed as a twodimensional lattice in the natural membrane and must be coaxed to form such lattices after the protein is purified and reconstituted in a suitable phospholipid bilayer. The example of the Omp F porin from the outer membrane of E. coli is selected as a typical study. As shown by Rosenbusch (1990), a suitable detergent was needed to separate the membrane protein from its natural cell membrane before it could be purified by column chromatography. The type of detergent must be suited to the physical chemical requirements of the protein itself. The porins require nonionic amphiphiles such as the alky glucosides whereas other proteins may tolerate something like sodium dodecyl sulfate (Rosenbusch, 1974). After the protein is purified, its buffer solution in the presence of excess detergent is placed in a dialysis bag in the presence of phospholipid vesicles (e.g., DMPC), and the detergent is slowly dialyzed away to force the phospholipid to replace it. Under favorable conditions, collapsed bilayer vesicles of the lipid containing crystalline arrays of the membrane protein are formed (Dorset et al., 1983b), with the resultant porin lattice array depending on the type of lipid used and its concentration (Engel et af., 1992). If lattice imperfections are found to be due to an excess of lipid, the excess can again be removed by mild phospholipase digestion, as mentioned above. Isolation and crystallization of the points has been described in detail by Garavito and Rosenbusch (1 986). Suspensions of the protein crystals in vesicles can then be deposited onto a hydrophilic carbon-film-covered electron microscope grid and then be stabilized by a number of techniques. Initial studies employed negative stains such as uranyl acetate or potassium phosphotungstate (see Unwin and Henderson, 1975). This is added as an additional drop of solution before the drop of suspension is blotted with a filter paper to dry the preparation. Alternatively, for higher-resolution studies, various sugars, such as glucose (Unwin and Henderson, 1975), could be used to replace the hydrogenbonded water (e.g., with bacteriorhodopsin), and, after drying, such preparations were examined by low-beam dose techniques in the electron microscope. Studies have also been made on thin three-dimensional macromolecular crystals (Chiu and Jeng, 1982; Fujiyoshi et al., 1984) as well as quasicrystalline motifs in tubular arrangements (Toyoshima

122

DOUGLAS L. DORSET

and Unwin, 1988). For the three-dimensional crystals, problems can be experienced in data collection because one no longer samples a continuous Fourier transform along the direction normal to the discrete reciprocal net (Amos et al., 1982).

B. Electron Diffraction 1. Illumination of the Sample and Recording Diffraction Patterns Much has been said about the radiation damage of organic crystals by the electron beam to the point where it is a common misconception that such materials cannot be studied at all without some gross change occurring during the experiment. For electron diffraction experiments especially, beam illumination conditions can be controlled adequately so that, compounded with a suitable recording medium, diffraction patterns can be obtained without any noticeable change to the specimen. Even if some damage does occur, sequential diffraction patterns will allow one to extrapolate to zero exposure conditions (Perez and Chanzy, 1989), a procedure similar to the use of reflection standards in x-ray data collection. Working with these low beam currents requires that the microscopist be well adapted to the darkness of the room. (It often helps to dim or extinguish all those superfluous instrument panel lights that seem to delight manufacturers. Now that computer interfaces are often placed on modern electron microscopes to automate various alignment procedures, the fact that one has to stare into a CRT screen at about the same wavelength as the microscope screen phosphor emission certainly does not help this situation! A similar complaint had been stated earlier by Grubb [1974].) One should also be familiar with the location of all necessary controls in the dark. The optimal procedure for illumination of organic crystals are basically those established by Glaser and Thomas (1969). That is, the double condenser lenses are both nearly maximally excited (also producing nearly parallel illumination) and a small aperture (e.g., 20 pm diameter) is inserted into the second condenser lens to easily produce current densities on the level of 10-’A/cm2 or less. If necessary, it is also possible to make an adjustment of the Wehnelt cylinder by retracting the filament to further limit the illumination. After these adjustments, it should be possible to observe bend contours on paraffin monolayer crystals at 10,000 x magnification without them moving for several minutes. Although it has been pointed out that all photographic films exposed to electron beams have similar quantum efficiencies per grain (Farnell and Flint, 1969), there are still practical matters to consider when one selects a film for recording electron diffraction patterns. One consideration, of

ELECTRON CRYSTALLOGRAPHY OF ORGANIC MOLECULES

123

course, is how linear the film response is and over how many intervals of optical density this linearity holds (Vainshtein, 1964a). Secondly, one has practical spatial considerations for useful size of diffraction spots and their separation when one wants to scan them, with a densitometer. In our experience, a large grain, direct exposure x-ray film such as Kodak DEF-5 is quite satisfactory for most quantitative electron diffraction experiments. Another consideration for radiation-sensitive specimens is to minimize the dose to the sample during “dead times” in the instrument operation, particularly when new film cassettes are being translated into position, e.g., after recording a selected area diffraction pattern, if one wishes to photograph a bright-field image of the crystal producing the diffraction pattern. Although minimal dose kits exist for automatically deflecting the beam (Fujiyoshi et al., 1980), it is often just as convenient to misalign the dark field controls of the condenser lens deflector coils so that the beam can be easily removed from the specimen area during these intermediate periods. 2. Diffraction Geometries Any converging lens produces a diffraction pattern (which, of course, can be continuous for a nonrepeating object (see Taylor and Lipson, 1964) at its back focal plane. Alternatively, one can have a parallel source illuminating the object, and, without a lens being required, its Fraunhofer diffraction can be observed on a screen at some suitable distance from the object. These rather basic principles can be used to describe the alternative ways of setting up a diffraction geometry in an electron microscope (Cowley, 1992). In the former case, where the diffraction pattern is observed at the objective lens back focal plane, one can use variable focus electromagnetic lenses located below the objective lens to magnify the pattern according to the formula L = f , n , M i , where L is the effective camera length, f , is the focal length of the objective lens to the object, and II,denotes a product of successive lens magnifications M , (Ferrier, 1969). The typical diffraction mode used in electron microscopes is the selected area geometry, where an aperture is inserted into the image plane of the objective lens to restrict the cystalline area contributing to the electron diffraction pattern (Reimer, 1984). Typical camera lengths may vary from 0.20 to 3.6m. If a larger camera length is needed, high-dispersion diffraction conditions can be set up in the electron microscope. Here the actual objective lens is not excited, and the first intermediate is used as an effective objective to give a very large value for f , in the above expression. Although selected area limitation might also be possible, the illuminated area is limited with the condenser lenses and apertures (Unwin and Henderson, 1975). Especially for selected area diffraction where the value off, is small enough so that undulations of the

124

DOUGLAS L. DORSET

specimen support would cause a significant variation of L, it is important t o have an internal diffraction standard (e.g., the powder diffraction pattern from an evaporated gold film) to calibrate the measured lattice spacings for any specimen. Most published transmission electron diffraction patterns have been obtained from specimens oriented perpendicular to the beam direction. Nevetheless, goniometer stages exist for electron microscopes that permit collection of three dimensional data. Originally tilt cartridges were designed for top-entry specimen stages to provide the best antivibrational stability (for observation of images) but these were generally noneucentric, meaning that the specimen would translate out of the selected area as it was being tilted. For beam-sensitive objects, this would require additional adjustment time for recentering the crystal, resulting in needless radiation exposure. Nearly eucentric stages have been provided for most modern instruments with a side-entry geometry, where the specimen is at the end of a long rod and the tilt is made around the rod axis. It is possible to obtain side-entry holders which additionally permit in-plane rotation, either 360" or f 180", thus enabling any accessible diffraction axis to be oriented parallel to the tilt axis (Turner et a/., 1991). Since electrons travel a helical path down the electron microscope column, it is necessary on many instruments to establish where the stage tilt axis is on the diffraction pattern for any camera length. This can be conveniently calibrated with a substance that gives a recognizable diffraction pattern as the specimen is tilted, say, 30" (e.g., the fatty acids) just when the inclination axis of the crystalline standard coincides with the goniometer tilt axis (Dorset, 1967a). Goniometry also permits carrying out dynamic experiments such as heating and cooling of the specimen. Liquid nitrogen-refrigerated cryoholders generally have heating elements near the specimen tips so that phase transition behavior of organic crystals can be followed from, e.g., -170" to +15O"C. Rapidly quenched aqueous samples can also be examined in a vitreous ice state (Taylor and Glaeser, 1974) and they can be oriented with a double tilt cryo-stage (which, although not as convenient as the tilt, rotation stage, can locate the same regions of reciprocal space). Goniometer stages have also been designed for differentially pumped environmental chambers so that it is then possible to look at fully hydrated specimens at ambient and elevated temperatures (Turner et a/., 1991). The alternative diffraction geometry without lenses (or using all lenses as a condenser system to obtain a parallel electron source) was actually the one used in earliest experiments with electron diffraction cameras (e.g., see Pinsker [1953] for a description of the apparatus). Many electron microscopes at one time allowed a sample holder for high-resolution diffraction to be inserted below all of the magnetic lenses, the term "high-resolution"

ELECTRON CRYSTALLOGRAPHY OF ORGANIC MOLECULES

125

referring to the fixed camera length between the specimen position and the fluorescent screen. With such holders it was possible to carry out transmission diffraction as well as grazing incidence reflection diffraction (i.e., RHEED), and, if the separation of diffraction spots or bands was not sufficient, a lower voltage (longer wavelength A) could be used (i.e., Bragg’s law: nA = 2dsin 0). As mentioned above, the reflection geometry was historically the most important for early electron diffraction experiments on long-chain lubricants (see also Germer and Storks, 1938, as well as Havinga and DeWael, 1937). These materials were rubbed onto a metal surface to form a monolayer (Menter and Tabor, 1951), the surface of which was oriented nearly parallel to the incident beam. The holder permits fine adjustments of the angle of incidence as well as translation into and along the beam and rotation perpendicular to the specimen surface. These arrangements do not exhaust all the possible diffraction geometries possible in an electron microscope. Others, such as convergent beam diffraction or microbeam diffraction, are most suitable for inorganic specimens which are faily resistant to beam damage (e.g., see the recent review by Cowley, 1992). 3 . Extraction of Intensity Data

If electron diffraction patterns are recorded on film, then intensity data can be measured from these as would be done in x-ray crystallography. Earliest experiments estimated film intensities by eye, using a calibrated exposure series as a comparative guide (e.g., Karpov, 1941), exactly as in the early days of x-ray crystallography. The relation between intensity and spot diameter was also utilized (Cowley et al., 1951; Zvyagin, 1993). More quantitative measurements are made with a microdensitometer , following the guidelines for accurate extraction of intensity discussed by Wooster (1964). If the diffraction patterns are scanned with a narrow slit on a flatbed microdensitometer, the most convenient measure of integrated intensity is a triangle approximation of the peak. Other laboratories have used other densitometer systems and have written computer software for automatic subtraction of background intensity (Chanzy et af., 1987). Alternatively, one can build in two perpendicular sawtooth deflector systems into the electron microscope (below the other lenses) to produce an integrating electron diffraction camera (Cowley, 1953). Here the spots are spread into squares so that one need only measure peak height. The disadvantage of this camera is that low intensities tend to be lost in the background fog of the film (Stout and Jensen, 1968). Charge coupled devices (CCD) and vidicon tubes have also been used as detectors in electron microscopes for recording electron diffraction patterns.

126

DOUGLAS L. DORSET

If the stored density array from a particular pattern, captured with a framegrabber, is integrated, there is a very satisfactory agreement to densitometer results (J. R. Fryer and D. L. Dorset, unpublished results). CCD cameras can also be used as high-resolution densitometers for measurement of diffraction intensities from films illuminated on a light box, as long as the linear response of the camera is used. Facility for such measurement is included into the CRISP package described by S . Hovmoller (1992). Whatever detector system is used for recording diffraction patterns, it is mandatory that a multiple exposure series be used to avoid saturation of the detector response, as discussed by Vainshtein (1964a). An appraisal of the accuracy in measuring intensities by these various means is given by Vainshtein et al. (1992). A thorough theoretical treatment was published for extraction of structure factor values from measured intensity data (Vainshtein, 1965b, c; 1957), but, in this early work, very little actual observation of crystal morphology was made because diffraction experiments were originally carried out in a diffraction camera and not an electron microscope. Since much early work was also done with oriented crystal textures illuminated with a rather large source diameter (mm instead of pm), the phenomenological Lorentz corrections seem to be correct for this case. However, the physical model for single crystals was not entirely correct (Dorset, 1976b), since a mosaic structure was assumed. It also does not apply to polymer lamellar crystals (Thomas et al., 1974; Dorset, 1979a). In agreement with Cowley’s (1981) statement that most thin single crystals do not appear to have a large enough defect content to justify the mosaic model, we have found that the best assumption to make is that the crystal is an elastically deformed foil illuminated by a source with high spatial coherence. As shown experimentally (Dorset, 1976b), a phenomenological Lorentz correction is not required for the resulting intensity data unless the crystals are additionally deformed by paracrystalline (curvilinear) disorder (Dorset et al., 1987) so that the densiometer trace does not measure all of the scattered intensity. In this case, corrections similar to the ones proposed by Vainshtein (1956b) for mosaics, i.e., IFhkrI2= klhkr(obs)d&, seem to be appropriate, as is also found in x-ray diffraction of, e.g., phospholipid multilayers (Franks, 1976).

C. Electron Microscopy 1. Low-Magnification Diffraction Contrast Images

When considering the use of electron microscope images to aid the interpretation of electron diffraction patterns, low magnification (e.g., 1 0 , 0 0 0 ~ )

ELECTRON CRYSTALLOGRAPHY OF ORGANIC MOLECULES

127

diffraction contrast (bright field, dark field) is often invaluable and yet places no serious demands on the operator when one is photographing selected area diffraction patterns, for example. It is assumed that the microscope is adequately aligned and the objective lens corrected for astigmatism before these experiments. At such a low magnification, it is even possible to focus the image by using an image wobbler function on the electron microscope, and, if the specimen support film is adequately flat, this focus need only be checked at a beginning of an experiment on one grid. After alignment, most modern electron microscopes are configured so that it is possible to view quickly either the back focal or image planes of the objective merely by selecting a function on the control panel. After setting up the beam illumination level for selected area diffraction and then finding the diffraction pattern by translation of the specimen stage, an aperture is placed into the back focal plane of the objective to isolate either the central beam or one of the diffracted spots, forming a bright- or dark-field image after removal of the diffraction aperture and selection of a so-called magnification function. With familiarity of the instrument, the diffraction and diffraction-contrast images can be viewed sequentially with little trouble so that a series of diffraction patterns alternating with these images can be photographed with the same illumination conditions and film. Using a timer function for exposure, appropriate exposure conditions can be adjusted as required. The utility of such diffraction-contrast images cannot be underestimated. Bend contours, for example, allow one to estimate the elastic bend distortion of a thin crystal (Cowley, 1967; Dorset, 1978a). Sometimes directional bands correspond to other features of the specimen such as the collapse of a three-dimensional habit found for some polymer and higher n-paraffin lamellae (Revol and Manley, 1986; Dorset, 1986a). Also, twin behavior can be characterized by identifying, in dark-field micrographs, the crystalline regions that correspond to a portion of a combined diffraction pattern (Pradere et al., 1988a). Moire magnification from mutilamellar crystals also enables one to detect defects such as edge dislocations or even to observe the crystalline lattice itself in an indirect way (Uyeda, 1974; Holland, 1964; Holland and Lindenmeyer, 1965; Holland et al., 1963). 2. High-Resolution, Lo w-Dose Electron Microscopy The use of the electron microscope to observe the packing of individual organic molecules in crystal lattices originated with the pioneering work of N. Uyeda and his colleagues (1976) followed shortly by the work of J. R. Fryer (see his review, published in 1989). Obviously one is faced with the problem of radiation damage because, unlike the case of diffraction,

128

DOUGLAS L. DORSET

where high and low resolution information about the average structure is concentrated in individual diffraction spots (hence allowing the rapid recording of a pattern), one wishes to map out the same average detail, point by point on an image. These constraints imposed by beam damage require that a noisy image be photographed. The average detail in a unit cell then can be discerned after later processing of an image in which no details of the structure may be visible to the naked eye (although this limit is true only for certain aliphatic samples). Although the procedure for recording high-resolution electron micrographs at low beam doses has been automated (Fujiyoshi et al., 1980), it is instructive to discuss how it can be carried out manually (see also Fryer, 1979a). First, the electron microscope is aligned well and corrected for astigmatism at a very high magnification (e.g., 100,000~)although the lowdose microscopy itself will utilize much lower magnifications (e.g., 20,000 to 60,000~).After adjusting the beam flux to low-dose levels, a grid with the thin organic crystals is placed into the instrument, and this is scanned in the selected area electron-diffraction mode until a pattern with suitable resolution is found. The image mode of the microscope is selected at the desired magnification, and the beam or specimen is then translated to a nearby area to permit focussing. (When the beam is deflected, the focal can often be carried out at a higher dose rate using the grains of the carbon support, and a fine adjustment can also be made for astigmatism; when the sample is merely translated to another area, it is easier to focus on a particle to find a minimum contrast without any adjustments made for astigmatism.) After finding the specimen again (automatic with the beam deflector but with the diffraction pattern if by translation), a focus series of (two of three) images is recorded sequentially at known intervals of defocus. For lower magnification, it is still possible to use the x-ray film employed for electron diffraction, but a finer grain emulsion will prove to be useful for higher magnifications. For macromolecular samples it is not possible to combine the high dispersion diffraction mode with high-resolution imaging. Thus, if the crystalline areas are too small to be examined conveniently by selected area diffraction, a sample can first be scanned in the diffraction mode to evaluate the quality of the specimens on the grid for low-dose imaging, and then a sequence of images is taken after the microscope is realigned. It is important to stress, in any case, that the procedure is somewhat blind, requiring the use of many films that will be evaluated later. The procedure attempts to make nearly every electron count, i.e., by giving the samples the least possible dose before its image is recorded. When carrying out these procedures at high magnifications in particular, it is also advantageous to have an image intensifier to facilitate the focus and astigmatism adjustments. It is also possible to take an image at higher doses

ELECTRON CRYSTALLOGRAPHY OF ORGANIC MOLECULES

129

after this low-dose series to obtain an image of the carbon film to permit evaluation of the astigmatism correction. After the images are photographed, the negatives are evaluated by placing them on a laser-illuminated optical bench (see Misell, 1978). From the optimal transforms of the average lattice on the negative, it is then possible to see which images have the best diffraction symmetry and resolution, i.e., which are the most promising candidates for image processing. These are then scanned with a densitometer to produce a digital pixel array of image density. Image processing to produce an average representation of the unit cell can be carried out in either of two ways available on any of several computer program packages (e.g., IMAGIC [Van Heel and Keegstra, 19811) used for such analysis. The most usual Fourierfiltration computes the Fourier transform of the digitalized image, and a mask is constructed to band-pass only the Bragg maxima in the two-dimensional diffraction pattern (e.g., see Misell, 1978). A reverse Fourier transform of these Bragg peaks will then result in an averaged image free of noncrystallographic details due to damaged areas or other defects. If the lattice is deformed by curvilinear disorder, limiting the maximum resolution of the computed image transform, it is also possible to start with the average motif found by Fourier filtration and scan this across the image to look for lattice regions that exactly match its orientation. (Slight rotations [Henderson et al., 1985) can also be performed on the test motif and adjustments made to wellmatched lattice regions with this orientational disorder.) When lattice regions are aligned to one another by this cross-correlation technique, the ensuing correlation average (Frank, 1980; Saxton, 1980), i.e., the arithmetic superposition of all unit cells, can, in principle, result in an average image with resolution higher than found originally on the optical bench. Interpretation of these images also implies that the phase-contrast transfer function of the objective lens (Erickson, 1973; Reimer, 1984), i.e., due to spherical aberration, xl(s) = n Af Alsl’

+ 0.5nCsA31s14

and chromatic aberration x 2 ( s ) = 0.5n2A2(~lZD2

are known for the micrograph being analyzed. In fact, this is the point of taking a high-dose image of the specimen area after the crystal image is photographed at low dose. However, as shown by Uyeda et al. (1972), because of elastic curvature of real organic crystals, there can be local changes of Af that corresponds to contrast reversals on the micrograph. Since C,and D are constants for a given electron microscope, Li (1991)

I30

DOUGLAS L. DORSET

has reviewed several methods by which individual micrographs can be deconvoluted to remove the phase transfer function if the electron diffraction intensities are also recorded. These include: use of Wilson statistics (Liu et al., 1990), prediction of amplitudes with the Sayre equation (Han, Fan, and Li, 1986), use of the maximum entropy criterion to find the best map (Hu and Li, 1991), and the match of theoretical and observed transfer functions (Tang and Li, 1988). One of the simplest ways, which was also proposed earlier by Unwin and Henderson (1975), is to compare the image transform wave amplitudes t o the electron diffraction structure factor magnitudes to find zeros of the transfer function. When these corrections are made, image phase values are obtained that can be converted to crystallographic phases after an appropriate translational shift is made, i.e., 6(r + ro) (see Gaskill, 1976) to a permissible unit cell origin such that g(r) = p(r)*W + ro) = p(r

+ ro)

will transform to G(s) = F(s) exp(2nis ro) = IF(s)l exp[i(9, + 277s * ro)J where 9, are crystallographic phases and 2ns ro is the phase shift due t o origin translation. 6

D. Determination of Crystallographic Phases

In our discussion of image analysis above, it was shown that the Fourier transform of an electron micrograph could be used to determine crystallographic phases after a correction was made for the phase contrast transfer function of the objective lens. The image resolution of low-dose electron micrographs from organic crystals is often 3 to 4 A (Fryer, 1989); that of two-dimensional protein crystals is often 4 to 15 A and, more rarely, 3 A (Glaeser, 1985). Therefore, the crystallographic phases obtained from such images can be used to assist the determination of a higher-resolution structure, if indeed higher-resolution electron diffraction data exist. This is one important interface between microscopy and traditional diffraction analyses that can be exploited in electron crystallography, i.e., the utilization of phase-contrast images is a powerful advantage of this technique not available to any other branch of crystallography. However, despite many attempts to achieve true atomic resolution in the electron microscope, including the stunning success made in the imaging of copper perchlorophthalocyanine, where all heavy-atom positions (but not those of the organic residue) are resolved (Uyeda et al., 1978-1979), it is clear that most atomic resolution structure analyses will employ more traditional techniques for phase determination from diffraction intensities, familiar to other crystallographers, and phases from images will play an important

ELECTRON CRYSTALLOGRAPHY OF ORGANIC MOLECULES

131

adjunct role to provide additional lower-resolution information. Some of the phasing techniques used to solve structures from electron diffraction intensity data are reviewed below. 1 . Trial-and-Error Techniques

One of the earliest techniques used to solve crystal structures (Karpov, 1941) was to construct a conformational model of the molecule or asymmetric unit to find a suitable location and orientation that would fit the observed intensity data. This is an expression of the phase problem in crystallography. If we have a group of atomsj in a crystal at positions r,, and we know their scattering factors f(s), then it is quite easy to calculate the structure factors a priori by a Fourier transform (here approximated by a summation):

F(s) = Cjfi(s)exp(2nis * rj) However, if we can only measure IF@)/,it is not a trivial matter to carry out the reverse transform. We have seen above how translation of mass in real space results in a phase term in reciprocal space; hence knowledge of atomic positions easily transforms to crystallographic phases, but, conversely, one needs phases a priori to find mass centers. It is evidence that, for any unit cell, there are an infinite number of possible packing arrays for a given molecule that will fill this cell. (However, not all of them are energetically reasonable [Kitaigorodsky, 1973]!) Therefore, it may not be obvious how our ability t o calculate phased structure factors from many trial structures may be useful for finding the actual crystallographic structure, unless we happen to arrive accidently at a model packing that gives an acceptably low value for the crystallographic residual, defined R

=

c I IFoI

-

klFcI

I/c IFoI

where IF,[ are the observed structure factor magnitudes and IF,[ are the magnitudes calculated from the trial model. On the other hand, if there are symmetry elements shared by both the molecule and unit cell, it may be easy to restrict the possible choices to a suitably small number. A word of caution must be added about the use of the crystallographic residual or R-factor defined above. As pointed out by Hamilton (1964), this is a statistically meaningful figure of merit only (!) as long as the number n of observable data greatly exceeds the number p of parameters being refined in a structure analysis. Given these parameters, one can define a quantity

where F is a distribution function and a is the confidence level in claiming the particular structural model to be correct. If there are two possible

132

DOUGLAS L. DORSET

solutions with residuals R , and R,, where R , is the smaller quantity, then the condition R , > I?,% must be satisfied before the second model can be rejected. This becomes more problematic as p approaches n in magnitude. Although this caveat will be important for rather small electron diffraction data sets, it can be shown that the R-factor is also often abused as a figure of merit in x-ray applications. The area where well-constrained trial and error techniques have been most useful has been in the solution of polymer crystal structures from electron diffraction data (Brisse, 1989; Perez and Chanzy, 1989). Given a polymer chain, it may be possible to grow suitably large single crystals of monomer and/or oligomer units so that their x-ray crystal structures can be determined (Remillard and Brisse, 1982a,b). These structures will inform the crystallographer what regions of a chain sequence can be kept conformationally rigid. With a chain fiber repeat distance as a constraint (obtained from a fiber x-ray diffraction pattern), it is then possible to concatenate these rigid subunits via so-called linkage bonds, around which conformational twists are allowed to occur. If one records a zonal electron diffraction pattern in a projection down the chain axes (i.e., from an untilted solution-crystallized polymer lamella), it will be possible to test various conformational models to minimize the R-factor to the electron diffraction IF,I values, as defined above, simultaneously with an internal energy E calculated with a nonbonded atom-atom potential function of the form, (e.g., see Pertsin and Kitaigorodsky, 1987)

where A and B are constants for respective repulsive and attractive contributions, m < n, and a is interatomic distance. This has led to a large number of ab initio structure determinations for linear polymers, as reviewed in Table 2. This technique, by the way, is a direct adaptation of methodology used in fiber x-ray analyses (Tadokoro, 1979; Atkins, 1989). 2. Patterson Function The autocorrelation function of the unit cell potential, p(r), defined 0 p(r), where 0 is the correlation operation, is Fourier transformed to F(s) F(s)* = Z(s), the * denoting a complex conjugate (Henry and Lonsdale, 1952; Cowley, 1981). This autocorrelation operation, which is the direct Fourier transform of the diffraction intensities (assuming all phases to be equal to zero) is more commonly known as a Patterson function. This can be stated P(uvw) = v-' Zhk, cos 2n(hu + kv + Iw)

p(r)

c

ELECTRON CRYSTALLOGRAPHY OF ORGANIC MOLECULES

133

TABLE 2 OF LINEAR POLYMERS ELECTRON DIFFRACTION STRUCTURE ANALYSIS Polymer

Data

R -v a I u e

References

Polyethylene

Spot (hkl)

0.19

Poly(tetrafluoroethy1ene) oligomer Poly(ethy1ene oxide) oligomer segment Poly(ethy1ene sulfide)

spot (hM))

0.11

Hu and Dorset (1989) Dorset (1991b) Dorset (1977a)

spot (hM))

0.24

Dorset (1983a)

spot (hkO)

0.19

Spot+fiber ( h k / )

0.19

Moss and Dorset (1983a) Boudeulle (1975)

Powder (hkl) Texture (hM))

0.13 0.09

Poly(diacety1ene) Poly(1, I 1-dodecadiyne)

spot ( O M ) spot (h01)

0.12 0.13

a-poly(3,3bis-chloromethyloxacyclobutane) Poly(t-caprolactone)

spot (hM))

0.25

spot ( h k l )

0.21

Poly(trimethy1ene terephthalate) Poly(hexamethy1ene terephthalate) Poly( 1,4-trans-cyclohexanediyl dimethylene succinate) y-poly(pivalo1actone) Poly(butene-1). I11 Poly(y-methyl-L- glutamate) 0-form

spot (hM))

0.34

Spot (hM))

0.17

spot (hkl)

0.22

Brisse et a/. (1984b) Dorset (1991a)

spot ( h k l ) spot (hkl)

0.14 0.26

Meille er a/. (1989) Dorset et a/. (1993)

Fiber (hkl)

Not given

Poly(su1fur nitride) Poly(acety1ene) CIS

trans

8-form

Texture (h01)

0.38

Cellulose

Texture (hkl)

0.25

Cellulose triacetate Nigeran (anhydrous) Dextran low temperature high temperature Mannan I Chitosan V, amylose

spot (hkO) spot (hM))

0.26 0.25

(hM) (hkO) (hkl) (hkl) (hkO)

0.29 0.19 0.24 < 0.20 0.24

spot spot spot spot spot

Chien et a/ . (1982) Shimamura et a/. (1981) Day and Lando (1930) Thakur and Lando (1983) Claffey et a/.(1 974) Hu and Dorset (1990) Dorset (1991~) Poulin-Dandurand el a/. (1979) Brisse et a/. (1984a)

Tatarinova and Vainshtein (1962) Vainshtein and Tatarinova (1967) Claffey and Blackwell (1976) Roche er at. (1978) Perez el a/. (1979) Guizard el a/. (1985) Guizard et a/.(1984) Chanzy et a/. (1987) Mazeau et a/. (1992) Brisson el a/.(1991)

134

DOUGLAS L. DORSET

For an atomic resolution structure, this is equivalent to all interatomic vectors being translated to a common origin (Vainshtein, 1964a). Given the type of unit cell symmetry in the crystal structure being investigated, the vector distributions can be concentrated in certain Harker lines or sections (Stout and Jensen, 1968). Historically, this was the most important way of determining x-ray crystal structures before the discovery of direct-phasing methods. If the molecule contains a heavy atom or includes a salient motif that makes a strong contribution to the total scattering from the unit cell, its position will often be easily located in the Patterson map. If the found atomic position(s) is (are) then used to calculate a trial phase set, additional atoms can often be located in ensuing electron density maps until the total crystal structure is resolved. Application of Patterson techniques to electron crystallography is of less obvious utility than it is in x-ray crystallography. This is because the relative range of electron scattering factor magnitudes is much narrower than for the x-ray form factors (Doyle and Turner, 1968). It is generally accepted that the visibility of the heavy atom in the structure C flea& where the scattering factors are evaluated at sin 811 = 0, should be near 1.O (Stout and Jensen, 1968). In the determination of the structure of copper DL-a-alaninate (D’yakon et al., 1977), the visibility of the Cu atom is 0.47. The prognosis for the use of heavy atom derivatives in protein electron crystallography is even more pessimistic (Glaeser, 1985). Nevertheless, the Patterson function has been used to solve some small molecule structures from electron diffraction intensities listed in Table 1, as well as some of the polymer structures in Table 2. 3 . Direct Methods

a. Evaluation of Phase Invariant Sums. Although the interpretation of electron microscope images is certainly a direct method for determination of crystallographic phases, the terms as it is used in x-ray crystallography is based on the probabilistic estimate of linear combinations of phases (Hauptman and Karle, 1953; Hauptman, 1972) W = +(h,) +

+(h2)

+ +(h3) +

such that the Miller index sum Zihi

Zihikili

=

O,O, 0

This linear combination of phases is known as a “structure invariant” since its value is true no matter where the unit cell origin is chosen. If there are three phases in this invariant it is known as a triple. If hl # h2 # h , , it is known as a Z2-triple, and if h, = h2 = - h 3 / 2 , one defines a &-triple.

ELECTRON CRYSTALLOGRAPHY OF ORGANIC MOLECULES

135

Four phases in the invariant defines a quartet (Green and Hauptman, 1976) and so on. For any data set with Miller index values hi, therefore, it is possible to generate arbitrarily a number of structure invariants, which really are a list of simultaneous equations where crystallographic phases are the unknowns. If it is also possible to evaluate the probability of w being correctly predicted, then these simultaneous equations can be ranked from most to least probable. This is possible, in principle. For &-triples, one calculates = ( I E b l 2 - 1)1E2t11/m and, for &-triples, -42 = (2/JN)IE(h,)E(h,)E(h,)I and, finally, for quartets, B = (2/N)IE(hl)E(h2)E(h3)E(h,)I(IE(+h,h2)12

+ IQh2 +

b3)12

+ IE(h, + h,)12

-

2)

Thus, it is seen that, in all cases, the probability of a structure invariant sum being correctly predicted is directly related to the magnitudes of normalized structure factor magnitudes defined: lEbI2

= (Fb12/&Cif,”

These are normalized so that ( [ E l 2 )= 1.000. Obviously, this is not strictly correct for limited data sets, but, for most purposes, the approximation seems to be sufficiently accurate to permit ab initio structure analyses to be carried out. The electron scattering factors are corrected for thermal motion after a Wilson (1942) plot of the experimental data is made to estimate an overall value for Biso. Thus, f ’ = f exp(- Bi,,(sin2 8/A2)]. If we were able to obtain a small set of starting phase values, then it might be possible to solve these ranked simultaneous equations to determine new phase values $,, , enough in fact to allow an interpretable potential map to be calculated from p(r) = V - ’

C IF,^ exp(i4,)exp(-2rri(hx + ky + 1.z)

which is a partial reverse Fourier transform based on a limited phase set. Initial phase values for evaluation of these simultaneous equations can be obtained by defining the origin of the unit cell. If we represent the projection of a unit cell in Fig. 2, we can see that there are, for this example, four equivalent points that can be used as legitimate origins of a crystal structure corresponding to translations (0, 0), (1/2,0), (0, 1/2), and (112, 1/2), where translations are fractions of a and b, which are the lengths of the unit cell projection. This particular example has centrosymmetry, but similar

136

DOUGLAS L. DORSET

0

0 -Y

0 I

I 0

i0

X

FIGURE2. Plane group pgg with symmetry operations shown. The origin can be chosen at any equivalent two-fold rotation site, 1 through 4.

equivalent origins are also found for noncentrosymmetric cells. Because of centrosymmetry, it can be shown that

F(h, k) = C i f i exp(2ni(hx

+ ky)) = 2Cifi cos 2n(hx + ky)

Similarly if we consider the consequence of moving the origin from (0, 0) in Fig. 2 to (1/2,0), it can be shown that

F(h, k)' = 2&fi cos 2n(hx

+

Thus an origin shift of a / 2 will change the phase of reflections with odd h index. Equivalent changes for the other origin translations are given in Table 3 for reflections where the indices h, k individually are either g = even or u = odd. It is possible, therefore, for any primitive space group, to arbitrarily set the values of three phases (PLkl such that the index parity groups are not repeated or that none of the indices or their sums result in a ggg combination (which is an invariant). In this way, the origin position is defined. One chooses, therefore, a number of reflections that appear most frequently in the most probable phase invariant sums. If there are not enough phases that result from this solution of phase sums, it is also possible to define other frequently interacting phases as n algebraic values, (Pnkr = a, b, c, . .., etc., with no restriction on the indices imposed in this

ELECTRON CRYSTALLOGRAPHY OF ORGANIC MOLECULES TABLE 3 EFFECT OF ORIGIN SHIFTS IN PLANE GROUPpgg ON SIGNOF STRUCTURE FACTOR (see Figure 2)

137

THE

Miller index parity Origin

Shift

1 2

0 a/2 b/2 ( a + b)/2

3 4

gg

,JP

ug

uu

+ +

+ +

+

+

t

-

+

-

-

+ -

-

+

case. For centrosymmetric structures, then, each algebraic unknown can be permuted a = 0, n ; b = 0, n, ..., etc., requiring the generation of 2" potential maps to search for the correct structure (e.g., with chemical knowledge of molecular geometry). The solution of the simultaneous equations in phase also depends on phase relationships dictated by the space group (Henry and Lonsdale, 1952). The plane group pgg is shown in Fig. 2. If the major symmetry elements are considered in the structure factor expression above, i.e., the center of symmetry that relates fractional positions (x,y) to (-x, - y ) and the glide element that transforms ( x , y ) to (1/2 + x, 1/2 - y ) , then the expression reduces to: F(h, k) = 4X,if, cos(2nhx) cos(2nky),

for ( h + k) even,

F(h, k) = -4Cjf, sin(2nhx) sin(2nky),

for (h + k) odd

and Thus when the summed indices are even, F(h, k) = F ( - h , -k) = F ( - h , k), but if the sum is odd, F(h, k) = F( - h , -k) = -F( - h , k). Similar relationships exist for each plane and space group and must be kept in mind as the phase invariants are evaluated. The first actual application of direct methods to experimental electron diffraction data evaluated new phases from &triples and quartets generated for hM) data from an n-paraffin and the acyl chain packing of a phospholipid (Dorset and Hauptman, 1976). The direct analyses yielded structures in good agreement with those estimated by evaluation of Patterson maps. Since then the technique of hand-phasing has been applied to numerous other structures, including some of the early data sets from organics published in the early Soviet literature. An example of a zonal phase determination for the polymer chitosan data provided by Dr. H. Chanzy is outlined in Table 4. Since two solutions are possible with the unknown a = kn/2, the correct solution must be found from tests of the

138

DOUGLAS L. DORSET

TABLE 4 DIRECTPHASEDETERMINATION FOR CHITOSAN [C,H I I04Nl, a = 8.07, b = 8.44, c = 10.54 A , space group P2,2,2,

hkO

IEI

020 110 120 130 140 150 210 220 230 240 250

2.17 1.89 2.08 1.60 1.24 1.20 0.29

a - n/2 R/2 n/2

0.45

n n

(P

a a R/2 R

0.43 0.62 0.55

Origin definition: Q l l 0 = n/2, let (

a + n/2 a + n/2

(PIZO = P

=~a ~

~

E2triple invariants: A = 1.69

(PO20

+

n/2 A = 1.98

(PO20

(PI20

+

(PI10

a - n/2 A = 1.26

A = 0.56

(PO20

+

a

n/2

-

(PI10

(PI20

A = 0.35

(PI20

+ a

(P-(150)

(P-(240)

a +

(PI30

+

(P-(250)

+

(P-CZ30)

a

n/2 A = 0.51

(P130

+ (P-Cl30) n/2

+ (Pl30 +

n/2 A = 0.55

+ (0-(140)

-a

+

(PllO

n/2

n/2

(PI20

+ (PI,-l,O

n/2

R/2

+

(P-(210)

Quartet invariants: B = 0.77 (Po20 + (P-(llo) + (P-130) + 9 2 2 0 a - n / 2 -n/2 -a B = 0.63 (PO20 + (P-(llO) + (P-(l40) + (PZ30 a - n / 2 -n/2 -a

(P220

=

(PZ30 =

=

'

Correct solution established from X i p: on potential maps, a = 1712.

ELECTRON CRYSTALLOGRAPHY OF ORGANIC MOLECULES

139

potential map (see below). A review of electron diffraction data analyzed by this method is given in Table 5 , and further details of some of these analyses will be discussed below.

6. The Tangent Formula. It is clear that the valuation of individual phase invariants singly is best suited to the solution of centrosymmetric structures. A noncentrosymmetric polymer structure (poly(&-caprolactone)) TABLE 5 ELECTRON DIFFRACTION DATAPHASED BY DIRECT METHODS (number of phaseslnumber of data) Evaluation of triples and quartets a. Two-dimensional n-paraffin (24142) HSl subcell of cephalin (29142) Urea (13160) Boric acid (281133) Copper perchlorophthalocynine (271198) Poly(ethy1ene sulfide) (17/22) y-Poly(pivalo1actone) (22/45) Poly(3,3-bis-chloromethyloxacyclobutane) (19/29) Chitosan (1 1/22) Mannan I(24/34) Cellulose triacetate (14/22) Anhydrous nigeran (21/43) Celadonite (19/41 + 14/41) Phlogopite/biotite (29/59 + 17/40) Muscovite (23/5 1 + 18/77)

Dorset and Hauptman (1976) Dorset and Hauptman (1976) Dorset (1991a) Dorset ( I 992a) Dorset el al. (1991a) Dorset (1992b) Dorset (1 992b) Dorset (1992b) Table 4 Dorset (1992b) Dorset (1992b) Dorset (1992b) Dorset (1992~) Dorset (1992~) Dorset (1992~)

b. Three-dimensional Diketopiperazine (133/289) Room temperature thiourea (821187) Poly( 1,4-t-cyclohexanediyl dimethylene succinate) (5 I /87) Polyethylene (40/50) Poly(&-caprolactone)(30/47)

Dorset Dorset Dorset Dorset Dorset

(1991d) (1991a) (1991a) (1991b) (1991~)

c. With image data n-hexatriacontane (26/52) n-tritriacontane (17/40) 1 : 1 paraffin solid solution (12120) DHPE (13/16)

Dorset Dorset Dorset Dorset

and Zemlin (1990) and Zhang (1991) (1990~) et al. (1990a)

Tangent formula Thiourea (room temperature) (134/187) Thiourea (low temperature) (183/240) Diketopiperazine (230/287) Copper perchorophthalocyanine (various results)

Dorset (1992d) Dorset (1992d) Unpublished data Unpublished data and Fan ef a / . (1985, 1991)

140

DOUGLAS L. DORSET

was also solved by this procedure, but the dominant sublattice contained in that structure predisposed that many of the noncentric phases would lie near centrosymmetric values, For noncentrosymmetric structures where this condition does not apply, it is often profitable to use a phasing technique that considers all contributors t o an individual phase, , i.e.,

+,,

4b

($k

+ 6h-k)k

Such an average can be obtained from the tangent formula (Karle and Hauptman, 1956). If we define: A =

Ck(r)

B=

x k ( r ) W h I E k l IEh-kI

IEh-kl

sin(&

+h-kh

and cos(6k + # h - k )

then the tangent formula is defined as tan +h = A / B . The reliability of the phase estimate depends on a phase variance, which can be evaluated. For this determination, X2-triples are generated to give the possible vectoral relationships indicated above. One then defines the origin as outlined above and perhaps also accepts a few phase estimates from &-triples, which seem to be correct. For a noncentrosymmetric space group it is also possible to define the phase of another type of reflection &k, with index hkl forming an invariant with one or more of the orgin-defining reflections. This phase must have the property, e.g., in the commonly encountered space group P21212,,that, if the origin-defining phases add to a sum that is independent of structure inversion, this additional phase can cause the invariant sum to add to k n / 2 depending on the handedness of the structure. This is known as enantiomorph selection (see Ladd and Palmer, 1980). In electron crystallography, it is also permissible to add phases found from image analysis to this basis set (Fan et al., 1985, 1991). After the basis set is defined, a number of other phases that interact strongly in the C2-triples are given algebraic signs and are permuted to produce a number of possible solutions. Unknown centrosymmetric phases are permuted therefore through 0, R , whereas the noncentrosymmetric phases are typically stepped through n = 4 quadrants by defining (45" + n 90") as the permissible values. The correct solution is then located by some figure of merit such as (DeTitta et al., 1975):

where only those quartets for which B is negative are used in the evaluation to seek the most negative value of this figure. Alternative figures such as RKarle(Karle and Karle, 1966) do not appear to be particularly useful for electron diffraction applications.

ELECTRON CRYSTALLOGRAPHY OF ORGANIC MOLECULES

141

The tangent formula exists in various versions, often depending on what weights w h are used. In our work, we have used a version known as QTAN (Langs and DeTitta, 1975). With this version, we have been able to determine the structure of the noncentrosymmetric ferroelectric phase of thiourea, independently arriving at the phase solution depicted by Dvoryankin and Vainshtein (1962) at the lowest value of NQEST. It is not only restricted to noncentric structures, of course, and it has also been used to solve the centrosymmetric room temperature thiourea structure as well as the structure of diketopiperazine, but also more recently, the noncentrosymmetric structure of poly(butene-1) (Dorset et al., 1993). Fan and his coworkers (1985, 1991) have used two versions of the tangent formula (MULTAN [Germain et al., 19711 and RANTAN [Yao, 19811, the latter generating random starting basis sets) for phase extension from the 2 A resolution electron micrographs of Uyeda et al. (1978-1979) to the resolution of the electron diffraction pattern.

c. Density Modification. Another method of phase determination, which is best suited to refining or extending a partial phase set, is the Hoppe-Gassmann density modification procedure (Hoppe and Gassmann, 1968; Gassmann and Zechmeister, 1972; Gassmann, 1976). The procedure is very straightforward but is also very computer-intensive. Starting with a small set of Fb, one calculates an initial potential map p(r) by its Fourier transform. This map is then modified by a procedure that restricts peak size to a maximum value and removes all negative density regions in the map. The modified map p(r)t is then Fourier-transformed to produce a new set of phased structure factors F h t . New phase values are accepted via a 2 p , where p is some threshreciprocal space filter, accepting (Ecalcl/~Eobs( old. The new phase set is then used to calculate another potential map, which is again modified and transformed to a new set of phased structure factors, which are again filtered, and so on, until the solution is found by convergence (acceptable lowering of the crystallographic R-factor). Ishizuka et al. (1982) assessed the applicability of the density modification procedure to electron crystallography using simulated image and diffraction data from copper perchlorophthalocyanine. The method successfully found phase values corresponding to reciprocal space regions where the phase contrast transfer function had very low values. It was also found to be useful for phase extension (resolution enhancement), e.g., from 2.01 A to 1.21 A. d. Convolution Techniques. One of the first relationships ever derived for direct phase determination is the Sayre (1952) equation,

142

DOUGLAS L. DORSET

which is a simple convolution of phased structure factors, multiplied by a function of the atomic scattering factors. For structures with nonoverlapping atoms, consisting of one atomic species, it is an exact expression. Although the convolution term resembles part of the tangent formula, it should be emphasized that there is no statistical averaging implied. Since, relative to x-ray scattering factors, electron scattering factors span a narrower range of magnitudes at sin O/L = 0 (Doyle and Turner, 1968), it might be argued that the Sayre equation would be particularly useful in electron crystallography, despite the fact that it is not often utilized in x-ray determinations. We have shown that its utility is limited somewhat by data resolution-e.g., it is sufficient for analysis of a paraffin structure at atomic resolution (Dorset, 1992e) but not a phospholipid bilayer at e.g., 3 data resolution (Dorset, 1991e). Liu et al. (1988) were able to extend phases for simulated data from copper phthalocyanine with the Sayre equation, starting with the 2 A image resolution seen in the best micrograph and extending to the 1 A limit of the electron diffraction pattern. Fan (1992) (see also Mo et a)., 1992) has also used it to solve some superlattice mineral structures from electron diffraction data.

a

e. Other Real-Space Functions, When projected resolution in a potential map is poor, eithei due to limited data resolution or atomic overlap, some independent evaluation of the map density distribution can be used to determine which structure from a multisolution technique is most probable. One possible procedure is to use the maximum entropy criterion (Li, 1991): S = -Zipi In Pi where Pi = pi/Cipj, and pi is a map pixel density. In combination with direct phasing methods, this has been used very effectively to solve the crystal structure of perchlorocoronene in projection with electron micrographs and electron diffraction intensities (Dong et al., 1992). Certain other figures of merit, such as minimum value of ( A P ) ~where , A p is a difference density found when the Foooterm is set to 0.0, have been stated to be related to the maximum entropy criterion (Luzzati et al., 1988) and have been found to have some success in identifying lamellar profiles of lipid bilayers, even with electron diffraction data (Dorset, 1991e, f). Other criteria such as the smoothness of the density profile, defined by (ap/dr), have also been found to be useful (Dorset, 1991e). There seem to be other instances where maximum entropy does not seem to be applicable. For example, in the direct phasing of zonal electron diffraction data from a number of linear polymers (Table 5 ) , it was found (Dorset, 1992b) that an algebraic unknown was needed sometimes to link phase terms for enough reflections to calculate a meaningful potential map

ELECTRON CRYSTALLOGRAPHY OF ORGANIC MOLECULES

143

(as indicated in the phase determination outlined in Table 4). If atoms are not well-resolved in the projection down the chain axis and an independent determination by conformational refinement is not available, how does one determine which structure is correct? This was indeed the situation when H . Chanzy sent a data set from the polysaccharide chitosan to this laboratory, and the choice between two possible maps could be made only after the trial-and-error determination with a conformational model was completed in his laboratory. Calculation of the entropy term S above was very unsatisfactory, because the values for the two phase choices were not very different. On the other hand, a test for map “peakiness,” i.e., Zip:, suggested by Stanley (1986), worked very well (e.g., when n = 5 ) for this structure and also in tests with two other polymers (poly[3,3-bis-chloromethyloxacyclobutane] and nigeran). Again, the value of S calculated from maps generated for the other two structures was also insensitive to the phase choice. E. Structure Refinement 1 . Least-Squares Refinement

In x-ray crystallography, after initial atom positions are found on an electron density map, the structure is refined against the observed structure factor magnitudes with a least-squares technique, where parameters such as scale factors, atomic shifts, and thermal parameters can be continuously varied to minimize the crystallographic R-factor (Stout and Jensen, 1968). It is assumed that the measured data are highly accurate and that there may be as many as 10 observable data for each refinable heavy atom parameter. It is obvious that electron diffraction data often are not so plentiful as this, even in the best of circumstances. Hence, principally for polymer structures, the linked atom least-squares technique of Arnott and Wonacott (1966) used in fiber x-ray diffraction has been adapted for electron crystallographic studies to give the best fit to the measured data, which, understandably, may not be as accurate as the theoretical kinematical intensities. This procedure has been often demonstrated to give reasonable results and requires that bond distances and angles remain fixed during the fit to the observed data while conformational angles are refined. There is much less experience with the unconstrained least-squares refinement of structures in electron crystallography. In this laboratory, we have recently tested such a refinement with data sets from polyethylene (one unique heavy atom for 50 observed data) and diketopiperazine (four unique heavy atoms for 289 observed data). Using full matrix least squares, only the atomic positions were allowed to change via a dampening factor to retard the magnitude of the shifts and thermal parameters were then refined

144

DOUGLAS L. DORSET

in a separate cycle. Polyethylene required the presence of the hydrogen positions in the refinement but their movement was not stable; it was necessary that their positions be readjusted to theoretical locations on the carbon atom after several cycles. Temperature factors for carbon and hydrogens were respectively B = 5.2 and 7.2 A2,with bond distance and angles in good agreement with x-ray crystal structures of polymethylene compounds. The final R-value is 0.19. Rigid body refinement was also carried out for this chain molecule in an attempt to find the best value for a setting angle (Dorset and Moss, 1983). For diketopiperazine, the positional refinement resulted in bond distances and angles closer to the x-ray crystal structure than previously found in the original potential maps. Temperature factors could not be refined due to the perturbation of intensity data by dynamical scattering. The final R value was 0.25. On the other hand, attempts to refine the structure of poly(butene-1) form I11 by least squares, with eight atoms in the asymmetric unit against 122 measured intensity data, was unsuccessful. We had earlier demonstrated that such a refinement would be possible with even fewer (73) ideal kinematical data, but, obviously, inaccuracies in intensity measurement did not permit the refinement when even more observed intensities were used. 2. Fourier Refinement

It may be possible that, for the initial phasing model, not all of the atoms of the structure have been included, or, alternatively, there are some errors in the placement of atomic positions. One method of structure refinement that has proved to be effective in electron crystallography (and, in fact, was begun in the earlier work in Moscow [Vainshtein, 1955; Dvoryankin and Vainshtein, 1960, 1962, Vainshtein et al., 19921) is to use F, Fourier syntheses with the initial estimate of crystallographic phases to seek the location of other or better atomic positions in the ensuing potential maps. Interpretation of these maps often implies a judgment is made based on the chemical knowledge of the crystallographer-i.e., the expected composition and bonding of a particular molecule. The process is iterative. After one finds new or better atomic positions, structure factors are calculated from these coordinates to derive new crystallographic phases 4, and these are then used to calculate new potential maps: p(r) = V-'ZhZk&IFol exp(i&)exp[-2ni(hx

+ ky + k>l

If no new details appear, it is sometimes advantageous to calculate a difference Fourier map of the type: Ap(r)

=

V-lZhCkZI(IFol- IF,))exp(i4,) exp[-2ni(hx

+ ky + lz)]

ELECTRON CRYSTALLOGRAPHY OF ORGANIC MOLECULES

145

to seek other possible peaks. This synthesis can be useful for finding improved atomic positions when the initial ones are somewhat inaccurate. It is also helpful at times to weight the IFoI synthesis by the computed structure factors for refining the atomic positions, hence:

The efficacy of Fourier refinement techniques in electron crystallography can be illustrated with the structure determination of the copper perhalophthalocyanines. For the perchloro-derivative, direct phasing enabled all of the heavy atoms (C1, Cu) positions to be visualized in the initial potential map (with positions identical to the ones found in the 2 A resolution images of Uyeda et al. (1978-1979), but none of the lighter ( C , N) atomic positions could be found (Dorset et al., 1991a). Successive use of the equations above permitted location of all remaining atoms. The analysis of the perbromo-analog was even more interesting (Dorset et al., 1992a). It cannot be said that direct phasing was especially successful; nevertheless, the first map revealed positions for the Cu and two possible Br sites. Successive iterations located the remaining Br positions and all lighter atoms except two carbon positions (which, however, could be added from known benzene ring geometry). The surprising thing about this analysis is that, in retrospect, the 1200 kV data were seen to be perturbed significantly by dynamical scattering. The final R-value to kinematical structure factor magnitudes was 0.41 and this could be lowered to 0.26 only after a multislice calculation of n-beam dynamical scattering was carried out with the trial model. During the refinements process, it is necessary to evaluate the molecular geometry to make sure that the bond distances and angles remain near chemically reasonable values. Bond lengths I are calculated for any general unit cell by the formula (Stout and Jensen, 1968): l 2 = (x2 - x1)2a2+ ( y 2 - yl)2b2+ (z2 - z1)2c2 + 2(x, - x,)(y, - y,)ab cos y

+ 2 0 2 - y1)(z2 - 2l)bCCOS a + 2(22 - ZJX,

- x,)cacosp

Here x,, y,, z,, where n = 1,2, refer to fractional coordinates of two atoms forming a bond and, as usual, a, b, c, a, p, y are the unit cell parameters. For any atoms BAC forming two consecutive bonds A - B, and A - C , the angle 0 between them can be calculated by: COSO= [(AB)’ + (AC)2 - ( B C ) 2 ] / [ 2 ( A B ) ( A C ) ] where A B , AC, BC are interatomic distances.

146

DOUGLAS L. DORSET

3. Density Flattening The atomicity constraint is a powerful boundary condition for structure determination and refinement, because it allows one to confine a structure search to discrete loci of real space and permits use of a well-known atomic scattering model for testing a trial geometry against an observed set of diffracted intensities. If intensity data are not observed to a resolution where single atoms are resolved, it may be difficult to carry out a structure refinement. Certain structures, such as globular proteins, contain extensive solvent regions, and, as described by Wang (1985), a flattening of these regions during a refinement can be an invaluable tool for improvement of crystallographic phases, and, hence, the appearance of the final map. In our study of phospholipid structures (Dorset, 1991e,f), we found that similar constraints could be placed on hydrocarbon regions, as suggested in earlier x-ray determinations (Worthington et al., 1973; Nagle and Wiener, 1989), and, despite the rather low data resolution (e.g., 3 A), a reasonable density profile could be obtained. This was of particular use when direct-phasing methods did not find values for all electron diffraction maxima, resulting in a potential profile where the headgroup region was reasonably depicted but the hydrocarbon density was obviously distorted by ripples. IV. PERTURBATIONS TO DIFFRACTION INTENSITIES As indicated in the historical introduction above, there are a number of possible perturbations to electron diffraction intensity data which, in the most extreme case, can frustrate any attempts to carry out an ab initio structure analysis with experimental data. Although organic structures are made of light atoms, favoring a near single-scattering approximation in many instances, it is important to recognize what major factors can cause the observed diffraction data to deviate from the direct Fourier transform of the crystal unit cell. This knowledge will define conditions for datacollection that will permit direct interpretation of the measured intensities for an ab initio structure determination. A. Dynamical Scattering

Because of the strong elastic interaction of electrons with matter, it has been known for many years that deviations from single scattering are often observed in experimental electron diffraction data (Vainshtein and Lobachev, 1956). We have mentioned above how, in the earliest work, a correction for this dynamical scattering was attempted based on the

ELECTRON CRYSTALLOGRAPHY OF ORGANIC MOLECULES

147

Blackman (1939) two-beam theory, with the justification that a mosaic crystal model would cause individual diffracted beams to originate independently from appropriately oriented blocks (Vainshtein, 1964a). However, it soon became apparent that this theoretical construct was overly simplistic. The mosaic model for thin microcrystals, on one hand, is not justified, since there are too few observable defects in such samples to make such a mosaic (Cowley, 1981). One is therefore faced with the fact that a highly coherent electron beam is illuminating a nearly perfect (if somewhat elastically deformed) thin crystal to excite simultaneously many diffracted beams, due to the rather low wavelength used in the experiment. Multiple-beam dynamical scattering was treated theoretically by several diffraction physicists (Cowley and Moodie, 1957; Fujiwara, 1959; Howie and Whelan, 1961; Sturkey, 1962) leading to scattering models shown later to be equivalent to one another (Goodman and Moodie, 1974). One of the most convenient ways to carry out a calculation of n-beam dynamical scattering, particularly if crystal thickness is a continuous variable to be monitored, is the slice method of Cowley and Moodie (1957). Given the potential distribution of a crystal p(r), one can write the transmission function of a very thin slice n with thickness Az as the phase grating approximation 4n = ~XP( iadr)1 where a is an interaction term dependent on the electron wavelength. Its Fourier transform will result in a quantity Qn(s). Progress of the electron beam through many slices will involve a process of successive convolutions, so that the exit wave after n slices will have the Fourier transform

The reciprocal space multiplication with the propagation function transform, P(s) = exp(2nic[(s)), accounts for the Fresnel diffraction of the wave on transmission through distance Az. It includes the curvature of the Ewald sphere through the excitation error [(s). An alternative model is to expand the Fourier transform of the phase grating expression above to give (Cowley and Moodie, 1959) ~ ~ ~ "= (6(s) s ) + ia'tF(s) - [(a't)2/2!]~(s)*~(s)

- i[(a't)3/3!]~(s)*~(s)*~(s) + where the modified interaction constant 0' also includes a conversion of structure factors to voltages and t = n Az. Here, no compensation is made for Ewald sphere curvature and, at a suitable high voltage limit, Yn(s) = Fdy"(s)is found to be a very good approximation. Actually, for

148

DOUGLAS L. DORSET

conditions used to obtain selected area electron diffraction patterns from organic crystals-i.e., a rather large field illumination of an elastically deformed crystalline foil-neither model is a totally faithfully representation of n-beam scattering from the sample (see Cowley, 1992). Elastic deformation is contrary to the flat crystal model used in the more rigorous multislice calculation, while a flat Ewald sphere surface cannot be a good model at lower voltages. The most rigorous calculation would include an average of multislice computations over all possible crystal orientations dictated by the bend deformation (Turner and Cowley, 1969). Multislice calculations for a continuously undulating crystal had been carried out for two representative organic structures by Moss and Dorset (1983b). Are n-beam dynamical interactions, in fact, observed in electron diffraction intensities from thin organic crystals? Considerable time was spent investigating this possibility in our early work, and the answer was overwhelmingly affirmative, even for the case of monolamellar paraffin crystals. For example, even though graphical correlations for two-beam dynamical scattering give an overall improved agreement between calculated and observed structure factors (Li, 1963), only an n-beam calculation identifies which reflections are significantly changed by this scattering (Dorset, 1976b). If two-beam interactions were important, then there should be a limiting accelerating voltage where these interactions should vanish (Honjo and Kitamura, 1957). High-voltage electron diffraction experiments on monolamellar paraffin crystals to 1000 kV, in fact, demonstrate that the measurable n-beam scattering persits (Dorset, 1967~).Using a homologous series of chain lengths to accurately control monolamellar thickness in the same crystal structure, one can also demonstrate a close match of thicknessdependent dynamical scattering intensity with predicted values (Dorset, 1980). Finally, in experiments with wax ester crystals where one reciprocal lattice row was used as a rotation axis, it was shown that the relative intensities of these continuously excited reflections change with tilt (Dorset, 1980). This observation is only explained by n-beam dynamical theory (Cowley and Kuwabara, 1962) because the relative intensities on this reciprocal lattice row depend, in part, on the number and intensities of all other reflections excited in any crystal projection. That dynamical scattering effects can be measured even in the thinnest crystals of carbon-containing molecules does not mean that intensities close enough to the kinematical condition cannot be measured, i.e., we are not quite driven to the pessimism evidenced when the theory is applied to materials composed of heavy atoms (Humphreys and Bithell, 1992)! If this were true, this whole discussion of electron crystallography would be rather pointless. The effect of dynamical scattering on the success of crystal structure determination was evaluated by calculating n-beam diffraction

ELECTRON CRYSTALLOGRAPHY OF ORGANIC MOLECULES

149

data from crystals of increasing thickness irradiated by electrons with several wavelengths (using the multislice method) and then using the dynamical structure factors for a phase determination with the tangent formula (Dorset et al., 1979). I t was shown that expected failures could be observed if the accelerating voltages were too low or the crystals were too thick, but, more importantly, the conditions for obtaining directly analyzable data were well within experimental limits. Later theoretical predictions of the most favorable accelerating voltage for electron diffraction experiments indicated that an optimal voltage exists around 300-400 kV, since the minimization of the above interaction constant (i with increasing voltage is countered by the intersection of more high-angle reflections at higher voltage as the Ewald sphere radius l / A increases (Jap and Glaeser, 1980). More recent experimental tests with copper perchlorophthalocyanine actually show that conditions continue to improve as the accelerating voltage is increased (Tivol el al., 1993). The reason for this discrepancy with the earlier simulation is that the theoretical model failed to consider the real limitations of high-angle scattering resolution by thermal motion; hence, the earlier theoretical model was valid only at absolute zero. The most surprising result from the direct-phasing experiments discussed above is that direct phasing is in fact more robust than one would expect it to be. Even though n-beam dynamical scattering is observed to change measured values of lFobsI and thus l&bgl, it is apparent that phase determination can be somewhat insensitive to the strict hierarchy of experimental IEobs(,even though gross changes, which cause weak values to become strong and vice versa, cannot be tolerated. Since the normalized structure factors are used to calculate the relative probabilities that the phase invariant sums are correct, and then are used to rank them, this success of ab initio direct phase determinations-especially with the earlier Soviet data sets obtained at rather low voltage-is rather startling. The reasons for this success now are beginning to emerge. Suppose in the expanded phase grating expression for dynamical structure factor above that the interaction constant is large enough that the F(s)*F(s) term and the F(s)*F(s)*F(s) term, respectively, dominate the real and imaginary parts of the series. If it is also true that F(s)*F(s)*F(s) = kF(s)*F(s)also holds, as it seems to for some light atom structures (Dorset, 1992e), then it is possible that the first significant manifestation of n-beam dynamical scattering will be approximated by the single convolution of phased structure factors. As shown above, this convolution is an important part of the Sayre (1952) equation, which states that this operation will not change the crystallographic phase information contained in the observed structure factors even though their magnitudes will be changed (since the Sayre compensation for scattering factors is not contained in the phase-grating expression). The veracity of

150

DOUGLAS L. DORSET

this statement was tested with the structure of poly(ethy1ene sulfide) where it was shown that not only simulated dynamical data could be phased successfully but also intensities formed by Z = (F*F)’ (D. L. Dorset and M. P. McCourt, unpublished data). Although this somewhat overly simplistic approximation only holds for the first actual appearance of dynamical scattering in the intensity values for lighter atom structures (see Dorset, 1992e), it illustrates, nevertheless, the connection between the phasegrating series and the Sayre equation, observed originally by Moodie (1969). In other words, dynamical scattering does not cause a random scrambling of diffrated intensity and, for reasons that are not too surprising, this data perturbation can be tolerated somewhat in direct-phase determinations. The major effect of dynamical scattering on structure determination, if direct phasing is possible, is its effect on the calculated potential maps. Evaluation of experimental data from thiourea (Dorset, 1991a) showed that the sulfur atom position was correctly defined, but the positions of the lighter nitrogen and carbon atoms could not be determined accurately. Such a result is anticipated from the effect of dynamical scattering on the Patterson function (Cowley and Moodie, 1959). In the analysis of the copper perhalophthalocyanines, molecular geometry could be monitored regularly by calculation of bond distances and angles corresponding to peak positions on the map, but the resulting R-factor for the geometrically most reasonable structure was not necessarily the absolute minimum obtainable by unconstrained refinement (Dorset et al., 1992a). This “true” minimum actually corresponded to a distorted molecular geometry, reflecting the influence of altered IFobs I values. Dynamical scattering effects are actually expressed in electron microscope images well before the scattered diffraction intensities are significantly changed-this is apparent to anyone who has ever carried out a multislice scattering calculation (e.g., see Barry, 1992). Uyeda et al. (1978-1979), for example, suspected distortion of the copper perchlorophthalocyanine images by dynamical effects even when they used very thin crystals and 500 kV electrons. Theoretical simulations for such molecules have explained experimental results (Ischizuka and Uyeda, 1977; O’Keefe et al., 1983). As we predicted earlier, it can be advantageous to analyze the diffraction data directly, since this will produce kinematical phases (Dorset, 1978b). Of course, if the electron accelerating voltage is too low or the crystal thickness too large, dynamical intensity data can be recorded for which no direct analysis can be carried out. This is the situation most often encountered in applications to inorganic structures but, also in the example of copper perchlorophthelocyanine, we have found that data obtained at 1200 kV can be used to find a structure solution via direct methods, whereas patterns obtained at 400 kV (at the same 100 A crystal thickness) cannot

ELECTRON CRYSTALLOGRAPHY OF ORGANIC MOLECULES

151

(M. P . McCourt and D. L. Dorset, unpublished data). In such cases, one needs to know the crystal structure before it is solved to find a model suitable for multislice calculations. Another problem of this kind was encountered in the analysis of copper phthalocyanine (Clark et at., 1979). A method has been suggected by Vainshtein et at. (1992) for correcting intensities most affected by dynamical scattering. The medium and weak intensities at medium and large values of sin i3/A are used for the structure analysis and then the strongest intensities (mostly at low angle) are subjected to a Blackman correction. However, given the specific structure dependence of n-beam dynamical interactions, it cannot be imagined that this approach would be totally satisfactory for all applications. A convolutional correction is suggested for other cases (Vainshtein el at., 1992).

B. Secondary Scattering As described originally by Cowley, Rees, and Spink (1951), growth of somewhat thick layered crystals can result in a perturbation to diffracted intensities caused by strongly diffracted beams from upper layers acting as primary beams for lower layers. The perturbation, known as secondary scattering, is an incoherent multiple scattering phenomenon, represented by

J, = I ,

+ mI:I, +

so that the measured Jh deviates from the intensity I , one wishes to detect by the series of weighted convolutional terms in intensity. Not only does this alter the intensity values themselves, it can also cause a spurious increase of data resolution and cause the appearance of space group forbidden reflections. (Note, however, that dynamical scattering causes the disappearance of such systematic absences only in certain cases [Gjmnes and Moodie, 19651. This is the basis of convergent beam diffraction analysis of space group symmetry [see Eades, 19921 which, unfortunately, cannot be used for most organic materials because of the large beam current densities required.) We have already shown how this perturbation can lead to erroneous results in a structure analysis of an n-paraffin by effectively causing the R-factor minimum to be much shallower than it would be if only single scattering data were detected (Dorset and Moss, 1983). Not only is this phenomenon observed in diffraction from solution-crystallized samples, but also from epitaxially oriented crystals (Hu et al., 1989). Only in favorable cases, moreover, is the single weighted convolution above sufficient for correction of the observed data. In many respects, the existence of secondary scattering is a more compelling argument for limiting crystal

152

DOUGLAS L. DORSET

thickness than is n-beam dynamical scattering. If one cannot correctly identify space-group forbidden reflections, a structure determination will be carried out in the wrong space group and lead to incorrect results. An example of this was seen in the evaluation of paraffin chain setting angle from the optical transform of a mask based on an electron diffraction pattern in which systematic absences were faithfully reproduced (Kawaguchi et al., 1979). When secondary scattering is suspected, it is always advisable to record many diffraction patterns from a given crystal orientation to test for the consistency of the intensity distribution. In some cases, this is the only good way to determine if this perturbation plays a significant role in the observed diffraction. C. Diffraction Incoherence

It is very difficult to produce a strictly flat support film for thin crystals. Hence, any surface undulations will be expressed by elastic bend deformations of the thin organic crystals examined in electron diffraction experiments (see Dorset, 1978a, 1980). It is also true that, for selected area diffraction experiments, the incident electron beam has only a small angular divergence, meaning that it is a more highly spatially coherent source (e.g., extending over pm) than are laboratory x-ray sources used for difffraction experiments. If the projected unit cell length is very large, then, as shown by Cowley (1961), the diffracted electron beams no longer represent the total unit cell contents, i.e., there is a coherence limit for diffraction along the beam direction, so that the projected intensity data may no longer correspond to the central section theorem of crystallography (Crowther el al., 1970) but rather an average of subprojections. For example, in layer silicates, the hkO diffraction pattern from a monoclinic unit cell did not have the expected intensity distribution but rather that of the hexagonally packed sublayers (Cowley and Goswami, 1961). This phenomenon can be modeled by a perturbed Patterson function, I(S) =

xi ~

( sexp(27cir; ) s) exp(- 7c2c2s2z:)

That is to say, if the autocorrelation function of a crystal structure results in vector positions wi(r)that Fourier transform to K$(s), then the first part of the expression represents the transform of the undistorted Patterson function to the diffracted intensities. Modification of this transform by the latter Gaussian function, however, will change the intensities of vector magnitudes, depending on the amount of bend c (in radians), the position of the reflection in reciprocal space s = IsI, and the projected length of the Patterson vector along the beam direction, zi.

ELECTRON CRYSTALLOGRAPHY OF ORGANIC MOLECULES

153

Despite the fact that this original work is largely overlooked by most electron diffractionists, it has far-reaching significance for anyone who wishes to carry out a quantitative atomic resolution structure analysis with observed intensity data. For example, in our early work on solutioncrystallized n-paraffins, it was soon observed that the intensity data did not correspond to the bilayer packing of the unit cell, but rather, most often, to the single-layer packing (Dorset, 1977b, 1980), despite the fact that multilamellar crystals were often used for data collection. This single-layer structure can also be seen in maps included in the early Soviet study (Vainshtein et al., 1958). More confusing was the observation that sometimes the lower-angle reflections had the correct relative values, while the higher angle data corresponded to the single-layer scattering (hence a sin O/A dependence) (Dorset, 1977b). Diffraction data from solution-grown phospholipid crystals, or those from any other alkane chain derivative, similarly appeared to originate from just the polymethylene lattice when the electron beam was projected down the chain axis, as if there were no other parts to the structure (Dorset, 1983b). Application of Cowley’s (1961) straightforward kinematical Patterson model above allowed this phenomenon to be explained quite well. Bend distortion causes long Patterson vectors to become smeared out (minimized in weight) at the expense of shorter vector contributions to the intensity (Dorset, 1978a, 1980). The kinematical argument given above was later justified with a more quantitative n-beam dynamical scattering calculation for a periodically continued bend-deformed structure (i.e., a wave-form crystal) to produce identical results (Moss and Dorset, 1983b). Since small amounts of elastic bend deformation are unavoidable for thin crystal films, it was obvious that this intensity perturbation could lead to cases where a total structure determination would not be possible if the projected unit cell dimension could not be reduced. Unfortunately, organic crystals formed from solution often pack with the longest molecular axes more or less perpendicular to the best-developed crystal face (Jensen, 1970), and if there is no substructure, such as the methylene subcell of an alkane derivative, perhaps nothing at all could be said about the molecular packing (Moss and Dorset, 1982a). It is fortunate that many linear polymers crystallize from solution with a small enough fiber repeat along the chain that the intensity data from them are directly analyzable. (However, nothing can be said about the lamellar fold regions which may be spaced 100 A. Thus, these features will not contribute to the overall scattering in a meaningful way.) Nevetheless, an a posteriori correction for this diffraction incoherence may be useful for structure refinement (Moss and Dorset, 1982b). For another type of linear molecule, because details of the total crystal structure were not directly

154

DOUGLAS L. DORSET

observable with solution crystallized samples, attempts were made to determine lipid structures by analogy, comparing patterns from the natural glycero-compounds to those from conformationally locked analogs, but this was only partially successful (Dorset and Hancock, 1977; Dorset et al., 1978a,b; Dorset et al., 1983a). The danger of accepting such electron diffraction intensity data at face value (Le., as representing the total unit cell) is illustrated by an recently attempted analysis of a phospholipid structure (Lando and Sudiwala, 1990). The molecule, containing acetylenic fatty acids, crystallized in tubules and the electron diffraction pattern resembled those obtained from the 0, orthorhombic perpendicular subcell of an n-paraffin, even though the indices were given as hk,7h. From the central section theorem (Crowther et al., 1970), one can just as well define these indices as hkO since nothing is known about the third dimension. Using the given atom stoichiometry to calculate one can solve the structure by direct methods. The only density apparent in the resultant potential map is that due to the methylene chain atoms with no details of the molecular headgroup visible. Using atomic coordinates for the chains to calculate structure factors, the comparison to the observed data is R = 0.21. Although R = 0.16 was claimed in the original analysis, there were at least eight dihedral angles refined against 19 observed data. Thus, one cannot distinguish between the two models, but the simplest is to be favored here, since any model where R 5 0.28 is equally valid. What actually made the complete study of such linear chain molecules possible was the availability of the epitaxial growth techniques discussed above. Since the molecules are oblong, if one can project a much shorter axis in the beam direction, it turns out that the intensity data can be used with confidence for ab initio determinations of the total unit cell contents. (In the projection down the long molecular axis, however, only the methylene-chain-packing contribution is seen, and this can be used separately to determine this part of the layer structure as is shown by the review in Table 6 for a variety of materials based on paraffin-chain moieties.) Later, in our reanalysis of texture diffraction data sets published in the Soviet literature, it became apparent that the high-tilt angles used for such experiments achieved essentially the same thing as the epitaxial growth by projecting the electron beam down a suitably small axial direction. For example, selected area diffraction work on untilted diketopiperazine microcrystals (D. L. Dorset, unpublished data) readily demonstrates that the intensity data do not correspond to the total unit cell contents, because of the large bend deformation of the crystalline plates, whereas texture diffraction data from highly tilted samples can be used for structure determination.

ELECTRON CRYSTALLOGRAPHY OF ORGANIC MOLECULES

155

TABLE 6 QUANTITATIVE DETERMINATION OF METHYLENE SUBCELLS FROM ELECTRON DIFFRACTION INTENSITYDATA(R-factor)

Hexagonal subcell Rac glycerol 1,2-dipalmitate (0.13) 1,2-dimyristoyl-sn-glyerophosphocholine (0.03) I-palmitoyl-sn-glycerophosphocholine (0.01) N-nervonyl-4-sphingosyl-I-phosphocholine (0.01) 1,2-dihexadecyl-sn-glycerophosphoethanolamine (0.06) 1,2-dipalmitoyl-sn-glycerophosphocholinehydrated bilayer (0.03) Octadecanol (not given) Glycerol trilaurate (not given)

Frede and Precht (1974) Buchheim (1970)

Rotationally disordered orthorhombic subcell Trihexadecyl glycerol (0.13)

Dorset and Pangborn (1982)

Orthorhombic perpendicular subcell n-octadecane (0.22) n-triacontane (0.28) n-hexatriacontane (0.09) Polyethylene (0.23) n-octadecanol (0.26) Cetyl palmitate, 2 forms (0.14, 0.12) Stearic acid (E-form) (0.20) Behenic acid (B-form) (0.20) 1,IO-dibromodecane (0.19) 1-bromooctadecane (0.20) Ethylene di( 1I-bromoundecanoate) (0.17)

Dorset Dorset Dorset Dorset Dorset

(1974) (1975a) (1975a) (1975a) et a/. ( I 976) Hui ef a/.(1974)

1-iodooctadecane (0.24) 1,2-dihexadecylglycerol (0.20) n-hentriacontane (not given) Tricosanol (not given) Glycerol trilaurate (not given) n-dotriacontane (not given)

Vainshtein et a/. (1958) Vainshtein ef a/. (1958) Dorset (1976b) Dorset and Moss (1983) Dorset (1979b) Dorset (1976d) Dorset (1983b) Dorset (1983b) Dorset (1983b) Dorset (1983b) Dorset (1983b) Dorset and Ghiradella (1983) Dorset (1983b) Dorset and Pangborn (1982) Rigamonti (1936) Li ( I 963) Buchheim (1970) Cowley el a/.(1951)

Triclinic parallel subcell Glycerol tripalmitate (0.10) Glycerol triheptadecanoate (0.14) Glycerol 1,3-dipalmitate (0.13) 1,3-dihexadecylglycerol (0.13) Glycerol trilaurate (not given)

Dorset (1988~) Dorset (1983~) Dorset and Pangborn (1979) Dorset and Pangborn (1982) Buchheim (1970)

Hybrid orthorhombic subcell 1,2-dipalmitoyl-rac-glycerophosphoethanolamine (0.24)

Dorset (1967e)

Orthorhombic, similar to monoclinic parallel n-docecyl octa(ethy1eneoxide) (0.19)

Dorset (1983a)

Monoclinic parallel subcell Cyclooctatetracontane (0.24)

Dorset and Hsu (1989)

15-oxotetratriacontyl-13-oxodotriaconanoate (0.19)

156

DOUGLAS L. DORSET

It is also clear that it may not be possible to obtain analyzable highresolution diffraction data from other types of specimens for which all unit cell dimensions are large. If, for example, one seeks atomic resolution in the structure analysis of proteins by phase extension with electron diffraction data, there may be a sin &A limit to the intensities that represent the total unit cell contents, even for crystals that are only slightly bent (e.g., 2" over the lateral illumination diameter). This has been verified by simulations with Cowley's distorted Patterson function, using the rubredoxin structure as a representative protein (Dorset, 1986b). No known protein structure has been determined with electron diffraction data at a resolution where this perturbation can be evaluated experimentally.

D. Radiation Damage There is much that has been written about the effect of inelastic collisions of electrons with organic crystals, often to the point of becoming obsessive. Unfortunately, many of the early pessimistic predictions of attainable image resolutions based on the Rose equation did more to retard the progress of electron crystallography than advance it, since convincing arguments were made that certain experiments were not even worth trying. For example, predictions that the best image resolution available from an aliphatic specimen should be in the 30-100 A range was quickly shown to be grossly in error once researchers had the courage just to try the experiment. For example, John Fryer (1981) was able to obtain 16 A resolution images from epitaxially crystallized n-paraffins under the same conditions he used to obtain images from other organics. This was later found to correspond to the most radiation-sensitive row of reflections from this particular structure (Dorset et al., 1984a)! Later the 2.5 resolution images from solution-crystallized samples by Zemlin et al. (1985) appeared to be a result possible only after averaging and use of very low specimen temperatures, but the nearly simultaneous work of Revol and Manley (1986) on polyethylene demonstrated that, with care, similar resolution images can be obtained at room temperature and can even be observed directly on the image plates. It is currently believed that specimen movement (Dorset and Zemlin, 1985, 1987) caused by radiation damage may be what actually degrades the image resolution (Henderson and Glaeser, 1985). For this reason, spot illumination techniques have been adapted t o the imaging of aliphatics (Downing and Glaeser, 1986), leading to a 2.1 A resolution image of a paraffin (Brink and Chiu, 1991). Thus, as shown by the above examples, the real challenge was to find methods for doing the experiment rather than condemning it as being impossible from the very start!

A

ELECTRON CRYSTALLOGRAPHY OF ORGANIC MOLECULES

157

Although it certainly is useful to be aware of the comparable radiation sensitivities of organic materials, the most important questions to be asked seem to be: What resolution is required to observe an interesting structural detail, and what effect will radiation damage have on realizing this goal? Even though damage to the specimen can be demonstrated at the doses used to obtain the high-resolution image of, say, an n-paraffin, in terms of the electron diffraction pattern, this change to the specimen cannot be discerned in the image due to the transfer function constraints. Thus, as shown in Table 7, there is a growing catalog of organic specimens from which average detail has been reported in electron microscope images at molecular resolution, and, if these images can be used to visualize noncrystallographic details such as localized defects, as in the case of linear polymers (Isoda et al., 1983) and aromatics (Fryer, 1980; Zhang et a/., 1989), then they are very valuable adjuncts to a structure analysis as are average images from which one can extract additional crystallographic phase information. However, these observations of localized detail place the most demands on the specimen in terms of electron dose. Although illumination levels can be adjusted to minimize the effects of such damage to electron diffraction intensity data collection, as mentioned above, at least one laboratory routinely uses decay series for individual reflections to extrapolate to a zero-dose set of intensities (Perez and Chanzy, 1989), a procedure that may lead to better data sets.

V . APPLICATIONS A . Single-Crystal Structures

Application of the principles discussed above to actual ab initio determinations can now be discussed in terms of representative examples, to illustrate what can be expected from the derived structures after refinement is completed. In many cases, it will be shown that the results are equivalent to those obtained by x-ray crystallography (although they may not be quite so accurate). 1. Small Molecules As mentioned above, texture electron diffraction data from small organics published in early Soviet papers were used to test the efficacy of direct methods for solving such crystal structures without any preconceived knowledge of the molecular geometry. One example is diketopiperazine (Vainshtein, 1955; Dorset, 1991d). The space group is P 2 , / a with unit cell

158

DOUGLAS L. DORSET TABLE 7 OF ORGANIC CRYSTALS LATTICE IMAGES Compound

Resolution

Reference

Paraffin Paraffin Polyethylene Poly(tetrafluoroethy1ene) Poly(4-methyl-1-pentane) Poly( 8-hydroxybutyrate) a-chitin 8-chitin Cellulose I11 Poly(p-xylylene)

A A A A A A 5.1 A 4.8 A 4.3 A

Isotactic polystyrene Poly(tetramethy1-p-silphenylene sioxane) Poly(p-phenylene sulfide) Poly(sulfur nitride) p-hexaphenyl Poly( 1,4-phenylene) n-hexatriacontane (epitaxial) Copper perchlorophthalocyanine Copper perbromophthalocyanine Anthanthrene Quaterrylene Zinc phthalocyanine Copper phthalocyanine Phthalocyanine Silver 7,7,8,8-tetracyano-quindodimethane Perchlorocoronene Lanthaninde phthalocyanines Octacyano metal phthalocyanine; metal complexes

5.5 A 3.2 8, 4.3 A 2.9 A 3.2 A 4.5 A 5.0 A 2.0 A 3.5 A 4.8 A 3.5 A 2.8 8, ca. 3 A ca. 3 A 2.4 A 3.2 A ca. 3 A 4.4 A

Zemlin et a/. (1985) Brink and Chiu (1991) Revol and Manley (1986) Chanzy el a/. (1986) Pradere et a/. (1988a,b) Revol et a/. (1989) Revol (1989) Revol and Chanzy (1986) Sugiyama et a/. (1987) Tsuji et a/. (1982) Zhang and Thomas (1991) Tsuji et a/. (1984) Tsuji el a/. (1989) Uemura e t a / . (1988) Kawaguchi et a/. (1984) Kawaguchi e t a / . (1986) Kawaguchi et a/. (1986) Dorset and Zemlin (1990) Uyeda et a/. (1978-1979) Fryer and Holland (1983) Fryer (1978) Smith and Fryer (1981) Kobayashi et a/. (1981) Murata et a/. (1976) Fryer (1979b) Uyeda et a/. (1980) Dong el a/. (1992) Zhang et a/. (1989) Ashida (1991)

2.5 2.1 3.7 4.9 4.3 3.5

2.8 8,

constants: a = 5.20, b = 11.45, c = 3.97 A, B = 81.9'. A Wilson plot made from 317 intensity values listed in Vainshtein's (1955) paper (289 of whkh are observed) indicated that the overall isotropic temperature factor was B = 0.0.kj2, so no adjustment was made to the electron scattering factors for calculations of I E h l . (It was also assumed that the earlier-used Lorentz correction was appropriate, as was the Blackman correction for dynamical scattering to some low-angle reflections.) Average values of I, moreover, correspond to the values expected for a centrosymmetric distribution (Karle et al., 1965), despite the limited size of the data set. Phase values were obtained after origin definition by evaluation of &-triples and quartets; one phase from a &-triple was also accepted. This resulted in a set of 133 phased reflections, which led to a potential map

ELECTRON CRYSTALLOGRAPHY OF ORGANIC MOLECULES

159

with the details of the structure immediately apparent (Fig. 3a, b) to the experimentalist. The R-factor so far for observed data is 0.32. Leastsquares refinement was next carried out, fixing the thermal parameters and allowing only the scale factor and atomic shifts to be varied. A dampening factor of 0.1 was necessary for the atomic shifts so that they would not move too far for each cycle of refinement. This resulted in a final R = 0.25 for all nonhydrogen atoms. Final atomic coordinates are used to calculate bond distances and angles that agree favorably with the earlier x-ray crystal structure determination (Fig. 3e). It was also possible to solve this crystal structure directly using the tangent formula in its QTAN form. After origin definition, acceptance of one phase from a &-triple, and permutation of four other phase values, the correct solution was found (239 phases) within the four most negative NQEST values, with the structure at the most negative value being a translated molecular fragment (Fig. 3c, d). The failure of this figure of merit to identify the correct structure at its absolute minimum is due to some perturbation of intensity data by dynamical scattering, indicated by the physically unreasonable B = 0.0 A*.After ascertaining that a kinematical data set from this molecule could be correctly identified by NQEST, it was clear that the dynamical scattering, which, to a first approximation, produces a convolutional smearing of intensity, causes the lowest values of to be incorrectly estimated. Although we have shown that a single convolutional smearing of high I&, I values is not necessarily detrimental to direct phasing by hand evaluation of triple invariants, the accuracy of lower estimates is crucial for the use of NQEST, since it relies on the small cross-terms in the expression for B to identify negative quartets. Hence, a more robust figure of merit must be found for the tangent formula for its application to electron diffraction intensity data. Similar applications of hand-phasing with triples and quartets were made successfully for the centrosymmetric urea, boric acid, and thiourea structures, the former two involving zonal data sets and the latter another three-dimensional set. QTAN was also used successfully to solve this room temperature, paraelectric thiourea structure (Dorset, 1992d). The space group is Pnma with cell constants: a = 7.66, b = 8.54, c = 5.52 A; and there were 187 independent 3D data. After origin definition, acceptance of seven phases from &-triples, and permutation of five unknown phase values, 134 phase values were found to yield the correct solution at the lowest value of NQEST. (Only 82 phases were found by hand evaluation of individual triples and quartets.) Because of dynamical scattering, the structure could not be refined and, hence, the placement of carbon and nitrogen atoms is somewhat in error, although the sulfur atom position agrees with the x-ray analysis.

160

DOUGLAS L. DORSET

0 FIGURE 3. Potential maps for diketopiperazineafter direct phase determinationwith QTAN. Correct solution: (a) [OOI]projection; @) [I 101projection. False solution: (c) [OOI] projection; (d) [lo01 projection. (e) Final bond distances and angles after least-squares refinement.

ELECTRON CRYSTALLOGRAPHY OF ORGANIC MOLECULES

161

QTAN was also used to solve the crystal structure of the low-temperaure, ferroelectric phase of thiourea (Dorset, 1992d). The space group is Pmc2, with cell constants: a = 8.52, b = 5.49, c = 7.54 A . Starting with origin definition and definition of five unknown phases to be permuted, 183 phases were found for 240 unique 3D data. At the lowest NQEST value, the structure corresponds to the earlier x-ray determination, although no information from this earlier work was used for this analysis. After adjustment for a different origin along the 2,-axis, the rms phase difference between the two structures is 33". 2. Larger Molecules Copper perchlorophthalocyanine was used for many years as a standard in high-resolution electron microscope imaging experiments to probe what resolution limits could be realized from organic crystals. Microcrystals tilted 26.5" around the unit cell b-axis were used to collect a hkO intensity data set of 198 reflections with 1200 k V electrons. The unit cell constants are a = 19.62, b = 26.08, c = 3.76 A , B = 116.5", space group C2/c. Because the projection has a plane group symmetry cmm (h + k = even for all zonal reflections), only one reflection can be used for origin definition. Giving algebraic values for three other reflections, one must calculate eight maps from the 27 phase terms. In one of these, it is possible to find positions for the Cu and C1 atoms and then complete the structure determination by Fourier refinement to give a final R = 0.32 for all data (Dorset et al., 1991a) and this corresponds to geometrically reasonable bond distance and angles. Because evaluation of individual triples and quartets is not a task that easily leads to an unequivocal result for this structure, the appropriateness of automated methods such as the tangent formula was also evaluated. Fan and coworkers (1985, 1991) have already shown that a phase extension via the tangent formula is possible if the phases from 2 A resolution images are available. In our attempts to use the tangent formula, we have found that a phase extension is also possible if the 27 phases determined above are used as the basis set for QTAN, a set that is actually smaller than the one used by Fan et a/. Data from a 2.4 A resolution image, supplemented by the origin definition and one &-phase, will also find the correct solution, but this is not located at the lowest NQEST value. Smaller initial phase sets will not lead to a structure solution, in agreement with earlier work. Related to the perchloro-structure, we have also carried out an analysis of the perbromo-analog (Dorset et al., 1992a). Direct phasing is not particularly successful, since half of the predicted phases are incorrect. Nevertheless, as mentioned above, trial positions can be found for Cu and two Br atoms in one map, from which the structure can be completed by Fourier refinement.

162

DOUGLAS L. DORSET

3. Lamellar Structures As is discussed above, the limitations to analysis of intensity data from alkane chain derivatives due to elastic bend distortion result in electron diffraction patterns that dissect out the polymethylene chain contribution to the total scattering when the electron beam is oriented along the chain axis. Nevertheless (see Table 6), this can be used to advantage to determine quantitatively polymethylene subcell packing (Abrahamsson et al., 1978) and chain orientation for many structures. The technique has also been quite useful for the qualitative analysis of natural products, for example, the waxes isolated from insect sources (Hurst, 1948, 1950, 1952; Dorset, 1975b). Dorset and Ghiradella (1983) demonstrated how the growth of tubular insect wax fibers could be reconciled with the physical rules for chain aggregation in flat lamellea (Boistelle and Aquilano, 1978). Dairy triglycerides were also well characterized in electron diffraction experiments (Buchheim, 1970; Precht, 1979). After the effects of solvation were recognized (e.g., see Carlemalm and Wieslander, 1975), even the physical state of lips in hydrated cell membranes could be studied (Hui and Parsons, 1974). Finally, the progress of a solid-state acyl shift could be followed in microcrystals of a 1,Zdiglyceride (Dorset and Pangborn, 1979; Dorset, 1987e) by electron diffraction. Here, it was shown that the a-phase packing in the hexagonal methylene subcell was not responsible for this internal reaction. Rather the &-structure is the crystal phase that transforms irreversibly to the 1,3-diglyceride. With the x-ray structures of both starting and final products known (Hybl and Dorset, 1971; Dorset and Pangborn, 19881, it was possible to propose a geometrically reasonable reaction mechanism after the electron diffraction studies (Dorset, 1987e). It is apparent that the reaction must proceed through an intermolecular mechanism rather than the intramolecular scheme often proposed (Fischer, 1920). In general, crystal structures with a strongly scattering subcell structure have always presented a challenge to direct phase determination (Fan, 1992). This is because many of the large [Eh[values correspond to this sublattice, whereas more interesting features from the other part of the structure are often not represented in highly probable triple and quartet invariants. Lamellar structures such as the n-paraffins are rather elementary examples of this class. For example, electron diffraction patterns from an epitaxially oriented paraffin, n-hexatriacontane (Dorset and Zemlin, 1990) contain strong reflections due to the polyethylene part of the molecule. The space group is noncentrosymmetric, Pca2,, with cell constants: a = 7.42, b = 4.96, c = 95.14 A . Of the 52 measured Okl and hhf intensities, 21 are 001 reflections. High-resolution (5 A) electron microscope images have been used to determine phases for five of the low-angle 001 lamellar data,

ELECTRON CRYSTALLOGRAPHY OF ORGANIC MOLECULES

163

which otherwise are not accessed by high-probability triple and quartet invariant sums. If advantage is taken of special &-triple relationships calculated for such lamellar data alone (see below), this subset can be extended to 13 phases. Combining these values with the values obtained from E,-triples and quartets (mostly involving subcell reflections), a total of 26 phases are evaluated to give a clear representation of the paraffin molecules in the unit cell. It is clear that this analysis is expedited by combining two sources of crystallographic phase information. Similar analyses were carried out for the odd-chain paraffin, n-tritriacontane, space group A2,arn (Dorset and Zhang, 1991), and the pseudo-even-chain structure of the approximately 1 : 1 solid solution of n-C3&6/n-C36H,4. In the latter, which is the first quantitative determination of a n-paraffin solid solution (Dorset, 1990c), it was possible to refine on the occupancies of the chain-end atoms, yielding a solution matching theoretical expectations. The analysis of the pure paraffins actually confirmed earlier determinations based on models (Moss et al., 1984; Dorset, 1986c) but the determination of the solid solution structure actually corrected an earlier model (Dorset, 1985b). Compared to earlier x-ray determinations, the structure of n-C,,H,, is essentially the same orthorhombic packing described by Teare (1959). However, the odd chain paraffin packing is a high-energy form described qualitatively by Piesczek et al. (1974) and not the lower-energy form given by Nyburg and Potworowski (1973); this appears to be the first quantitative determination of this crystal form. By our measurement (Hu et al., 1989), the unit cell is orthorhombic and not the monoclinic Aa cell given by Piesczek et al. (1973). It was also possible to study another class of compounds related to the n-alkanes. The perfluoroalkanes are found not to be very soluble in most organic solvents and hence are difficult to crystallize for electron diffraction studies, although one quantitative study of the hexagonal layer packing has been made for n-C&4 (Dorset, 1977a). Crystals of n-C,,F,, were epitaxially grown from the vapor phase on freshly cleaved KC1 crystals, and electron diffraction intensity data were used to solve the structure of the room-temperature form (Zhang and Dorset, 1990a). Lattice images of these lamellar crystals also resemble those from the n-paraffins. As mentioned above, epitaxial crystallization has also been used to orient phospholipid molecules, but the paracrystalline lamellae are curvilinearly distorted (Fryer and Dorset, 1987) so that only the lamellar reflections are observed. Nevertheless, use of such data for structure analysis has realized a goal proposed originally by Parsons (1967), i.e., t o determine such structures from electron diffraction intensities. The procedure is equivalent to the analysis of x-ray diffraction data from multilamellar arrays. Following earlier experience with such data (Khare and Worthington, 1978), we

164

DOUGLAS L. DORSET

initially approached the problem by using one-dimensional Patterson functions to determine the intergroup spacings in the polar headgroups (Dorset, 1987a). Since the x-ray crystal structures of several phospholipids are known (Elder et al., 1977; Pearson and Pascher, 1979; Harlos et al., 1984; Pascher and Sundell, 1986; Pascher etal., 1987), it was then possible to propose a molecular model that could be translated in the projected unit cell (space group P i ) to find a minimization of the crystallographic R-value. This procedure mimicked an earlier analysis of an x-ray data set from a phosphatidylethanolamine homolog (Hitchcock et al., 1975). Several phospholipids (Dorset et al., 1987; Dorset, 1987b; Dorset, 1988a, b; Dorset and Zhang, 1990) and a 1,2-diglyceride (Dorset and Pangborn, 1988) were analyzed in this way, in some cases postulating alternative headgroup conformations for ether-linked lipids to the ones found for ester-linked compounds in single-crystal x-ray structures. However, it was not certain that the structure solutions corresponding to the lowest R-value were truly unique. If two or more minima are found with nearly the same depths, how can one distinguish among them, given the poor statistical significance of this figure of merit (Hamilton, 1964) for limited data sets? It was possible to obtain high-resolution (6 images (Dorset et a / . , 1990a) from these lamellae, e.g., from 1,2-dihexadecyl-sn-glycerophosphoethanolamine, on a cryo-electron microscope, and use their Fourier transform to phase the electron-diffraction data. Combining these images with the evaluation of C1- and &triples, it was possible to phase 13 of 16 reflections to determine the cross-sectional potential distribution of the membrane bilayer. (Later, with a proposed approximate correction for dynamical scattering, which allows a more accurate hierachy of IE,,I values to be listed, 15 of these reflections were phased.) Even without electron micrographs, it was also possible to just use the triple invariants to phase the most intense lamellar electron diffraction reflections from other phospholipids (Dorset, 1990~).From these partial determinations, the complete phase set can be found with refinements that impose the condition of density flatness to the hydrocarbon chain area. The correctness of the model can be evaluated with a test for density smoothness (Dorset, 1991e,f).

A)

4. Linear Polymers

As mentioned above, linear polymer structures are often determined from zonal electron-diffraction intensity data after a conformational model of the chain is constructed and a solution is sought by simultaneous minimization of the crystallographic R-factor and an atom-atom nonbonded potential energy. Would a direct phase determination be possible with such 2D data to obtain an independent measure of the phase values

ELECTRON CRYSTALLOGRAPHY OF ORGANIC MOLECULES

165

b

FIGURE4. Direct structure analysis of chitosan based on two-dimensional electron diffraction data (see Table 4). (a) a = n/2; (b) a = - n/2.

given from the conformational search? For the two-dimensional data set from polymers in Table 5 , direct methods proved to be of particular use for six of seven structures analyzed. The phase determination of poly(pivalo1actone), on the other hand, contained too many errors to produce a map where the atoms could be identified at correct sites. The result of the structure determination for chitosan is illustrated in Fig, 4. Although there are many atoms that overlap in the projection down the fiber axis, the density outline is clearly defined. Since an algebraic phase ambiguity is needed to link enough reflections, requiring the calculation of two maps, some independent means was therefore needed to identify the correct structure. Tests of peakiness via the integral value of map density raised to a power (Stanley, 1986), i.e., Z i p ' , is found to suit this purpose very well, as seen for chitosan and two other examples. What can be said about the determination of crystal structures from three-dimensional data taken from goniometer tilts of solution-grown polymer crystals? This too is possible. For example we have analyzed (Dorset, 1991a) the structure of poly( 1,4-trans-cyclohexanediyldimethylene succinate) from previously published data (Brisse et al., 1984b), finding phase values for 5 1 of 87 reflections in this centrosymmetric structure. Analysis of a noncentrosymmetric polysaccharide structure, mannan I, was not so successful, despite the fact that the evaluation of individual triples and quarters find 41 phases from 58 reflections. (The tangent formula is not of much use here since only 21 hkO reflections are assigned phase values, but no hkf reflections are phased.) From the ensuing potential maps, it is

166

DOUGLAS L. DORSET

impossible to locate atomic positions unambiguously. After analyzing a model data set at the same resolution, it was found that this problem is due to series termination error. In other words, not enough of reciprocal space is sampled during the data collection to permit calculation of an interpretable potential map, although the fit of a model structure to the density outline may be useful. The poor sampling of data is also partially due to the missing cone of information imposed by the goniometer tilt limits. Hence data along the chain axis cannot be collected from solutiongrown samples, even if this information is among the most important part of the reciprocal lattice for many linear molecules. One way of solving this sampling problem is to use data from crystals representing two orthogonal orientations of the crystal structure, i.e., combine the intensity data from solution- and epitaxially crystallized samples. The first determination of this kind was carried out for polyethyene (Dorset, 1991b), where benzoic acid was used as the epitaxial substrate and the crystallinity of the preparation was improved by annealing the sample in presence of the substrate. After densitometry of diffraction films from the two molecular orientations, 50 unique data were used in the structure analysis (Hu and Dorset, 1989). If we define b to be the fiber axis, the space group is Pnma with unit cell constants: a = 7.48, b = 2.55, c = 4.97 A. Evaluation of triple and quartet invariants finds phase values for 40 reflections, and details of the structure are immediately apparent in the ensuing potential maps (Fig. 5a). It is possible to carry out a least-squares refinement as long as the hydrogen atom positions are included and the final R = 0.19 corresponding to a chain setting angle of 47". (Note, however, that an accurate value for this angle could not be found by rotational refinement with a rigid chain model [Dorset and Moss, 19831, because of the statistical imprecision of the R-factor.) Similar data sets were obtained from similarly crystallized samples of poly (8-caprolactone), which yield 47 unique electron diffraction intensities (Hu and Dorset, 1990). The noncentrosymmetric space group is P2,2,2, with unit-cell constants: a = 7.48, b = 4.98, c = 17.26 A. Largely because of a strong pseudocell component (compare unit cell constants to those of polyethylene), it was possible to find phase values for 30 reflections after origin definition, by evaluation of three- and four-phase invariants (Dorset, 1991~).Reflections related by three-dimensional &-triples require the use of one algebraic unknown. Nevertheless, a clear representation of the chain is found in the (100) potential map (Fig. 5b), permitting construction of the three-dimensional molecular model after calculating slices along c in the three-dimensional map, corresponding to atomic centers along the chain. The resultant structure agrees with an earlier fiber x-ray analysis requiring a conformational twist to the chain (Chatani et al., 1970) and disagrees

ELECTRON CRYSTALLOGRAPHY OF ORGANIC MOLECULES

(a>

167

(b)

FIGURE5. Potential maps after direct phasing of three-dimensional electron diffraction data from linear polymers. (a) Polyethylene, projection onto chain axes along [llO]. (b) Poly(&-caprolactone),projection onto chain axes along [ 1001.

with another (Bittiger et al., 1970), which states that the chains are perfectly flat, like polyethylene. Crystal structure analyses (Dorset and Hybl, 1972) and energy calculations (Liau and Boyd, 1990) of long chain esters favor the conformationally twisted model. An attempt to phase the electron diffraction data with QTAN yielded 26 phase values with three errors. The potential map could be used to locate the zigzag chain atoms accurately but the carbonyl oxygen was not found. Attempts to refine the initial structure by unconstrained least squares were not successful, no doubt due t o the small number of measured data for each refinable parameter. It is clear that one of the major advantages of electron diffraction techniques over the commonly used fiber x-ray analyses is that one can study metastable polymeric crystalline forms that are intolerant of mechanical stresses. Such a case is the form 111 structure of poly(butene-1). Solutionand epitaxially grown crystals formed under quiescent conditions were used to obtain three-dimensional electron-diffraction data. The space group is P212,2, with cell constants: a = 12.38, b = 8.88, c = 7.56 A. With the tangent formula, it was possible to obtain 106 phase values for 122 3D intensities (Dorset et a/., 1993). The noncrystallographic 4,-axis was immediately visualized in the potential maps, as were the individual positions of the eight unique atoms in the unit cell. After Fourier refinement

168

DOUGLAS L. DORSET

to R = 0.26 for all data, a structure very similar to one found in an earlier analysis of powder x-ray was obtained. However, this is the first single crystal determination of this polymorph and, with 122 well-resolved intensities, the structure is better defined than with the 21 powder data, some maxima of which had as many as 15 contributors (Cojazzi et al., 1976).

5 . Proteins After the groundbreaking study of glucose-embedded bacteriorhodopsin by Unwin and Henderson (1979, the study of thin protein crystals, principally two-dimensional crystals of integral membrane proteins, by electron crystallography has been one of the most widespread applications of the technique. After many years of work, most recently with a cryo-electron microscope, the bacteriorhodopsin structure has been resolved to 3.5 8, resolution (Henderson et al., 1985), compared to the 7 A seen in the original study. The crystallographic phases to be combined with electron diffraction intensity data have been obtained solely from high-resolution electron microscope images, and at present, work has begun on fitting the polypeptide backbone to the potential map, a structure that is largely comprised of a-helices. Theoretical justification for the quasikinematical approach to the problem (weak phase object approximation) has been given by Ho et al. (1979). It is beyond the scope of this review to give a complete overview of the electron crystallographic work on proteins, which has been recently surveyed by Glaeser (1985). For purposes of illustration, our own experience with another class of integral membrane protein, the bacterial porins, can be discussed in this section. The Omp F porin from the outer membrane of E. coli was originally isolated and purified by J. P. Rosenbusch (1974), who found that the molecule preferred to aggregate as trimers and that it was denatured by ionic detergents, such as sodium dodecyl sulfate. Spectroscopic studies also indicated that the molecules contained very little a-helix and, in contrast to the bacteriorhodopsin, appeared to be mostly composed of p-sheet. Conductance measurements demonstrated that the protein was weakly cation selective and that it formed a diffusion barrier to polar molecules with about a 10 8, diameter (Schindler and Rosenbusch, 1978, 1981). Using the nonionic detergents to solubilize this protein, it was possible to crystallize the trimers in several polymorphic forms and to obtain highresolution x-ray diffraction data (Garavito et al., 1983). Electron microscopic studies began initially with the work of Steven et al. (1977) on outer membrane fragments, visualizing what appeared to be the mouths of three pore openings per trimer in the images of the

ELECTRON CRYSTALLOGRAPHY OF ORGANIC MOLECULES

169

negatively stained preparations. (The limitations to interpretation of such images imposed by dynamical scattering were also assessed [Dorset, 19841.) Three polymorphs of two-dimensional crystals of preparations reconstituted in phospholipid showed similar details at 20 A resolution (Dorset et a/., 1983b), and after three-dimensional data were obtained, the continuous channel could be traced through the bilayer (Engel et al., 1985). However, from these negatively stained preparations, nothing could be said about the constriction, which served as a diffusion barrier to polar molecules. Meanwhile, similar studies were carried out on the genetically related Omp C (Chang et a/., 1985) and Pho E (Jap, 1988) porins from the same outer membrane, to produce images very similar to those from the Omp F trimer. Similar features were found for a porin from another Gram-negative bacterium (Kessel et al., 1988). Higher resolution studies of the Omp F porin were first hindered by paracrystalline lattice distortions limiting diffraction resolution from sugarembedded samples to about 10 A. Following a technique developed by Mannella (1984) for the VDAC porin from the outer membrane of mitochondria, a mild phospholipase digestion was carried out to remove extraneous phospholipid, and these preparations were found to be quite suitable for ensuing high-resolution studies, since the glucose-embedded crystals diffracted to 3.2 A. Extensive high-resolution microscopic studies were carried out on the same cryo-electron microscope used to study the bacteriorhodopsin, and, eventually, after averaging many images by Fourier filtration and correlation techniques (Sass et a/., 1988), a consistent 2.5 A resolution two-dimensional image of the protein was obtained (Fig. 6), revealing features of the 8-sheet structure, A similar high-resolution study was underway on the Pho E porin by B. K. Jap and his coworkers (1990), who produced a two-dimensional image with similar features and then extended the study to three dimensions (Jap et al., 1991). Thus the pore could be traced through the protein, but, because of the mising cone of data due to the limited goniometer tilt, all the details of the molecular geometry could not be discerned, It was clear from these three-dimensional studies, however, that the protein consists mostly of a 8-barrel tilted about 35" to the membrane normal, in agreement with independent x-ray (Kleffel et a/., 1985) and spectroscopic (Nabedryk et al., 1988) measurements. More recently, the full x-ray crystal structures have been carried out (Pauptit et ad., 1991; Cowan et a/., 1992) to, e.g., 2.4A resolution for Omp F and Pho E porins. It is significant to mention here that, despite the apparent success in high-resolution work on the bacteriorhodopsin structure determination, the study of the porins is actually the first benchmark experiment where x-ray crystallographic results can be used to evaluate the information content of high-resolution electron crystallographic analyses.

170

DOUGLAS L. DORSET

FIGURE6. High-resolution image of glucose-embedded Omp F porin after averaging. The white areas correspond to the 8-barrel structure of the protein (analysis by A. Massalski).

The agreement is very good. Details of the P-barrel structure are confirmed by x-ray crystallography as well as the other details of the projected protein structure, including a segment of protein appearing at the center of the pore seen in averaged electron micrographs. As shown by the x-ray analysis, this feature turns out to be the polypeptide loop responsible for the construction of the transmembrane channel. The actual shortcoming of the electron crystallographic study appears to be due to the tilt limitations of the goniometer stage. For example, an apparent indentation at the center of the trimer was proposed (Jap, 1989) to be the binding site for the lipopolysaccharide responsible for the porin function-a feature that could not be confirmed in the x-ray structure determination. Finally, although averaging phase-contrast electron microscope images is now demonstrated to be a valid means for directly deriving crystallographic phases for such macromolecular structures, it is also a very tedious

ELECTRON CRYSTALLOGRAPHY OF ORGANIC MOLECULES

171

procedure, especially for high-resolution work. Recently, it has been shown that an alternative procedure based on maximum entropy techniques was employed, starting, e.g., with a 6 resolution phase set obtained from Fourier transforms of images, to extend readily to the resolution of the electron-diffracted pattern to reproduce density features found after more tedious high-resolution image analyses (Gilmore, Shankland, and Fryer, 1992). Such a combination of microscopic technique with direct phasing of electron diffraction data, therefore, may remove much of the drudgery involved in such studies. Given new prospects for phase retrieval, it may be possible to determine structures of water-containing proteins in their natural state. Although frozen-hydrated specimens have often been examined (e.g., studies begun by Taylor and Glaeser 1974), the possibility of diffraction experiments on such solvated structures at room temperature (Matricardi et al., 1972; Dorset and Parsons, 1975; Tivol et al., 1982) has never been fully exploited. Since environmental chambers exist for high-voltage electron microscopes allowing goniometry (Turner et al., 1991), useful data may be available for structure determination.

a

B. Imperfect Crystals 1. Manifestations of Disorder in Real or Reciprocal Space

A major goal of high-resolution work in the electron microscope has been the desire to visualize noncrystallographic details of crystalline latticesi.e., to examine directly the structure of defects or lattice inclusions and how they are accommodated by the average crystal structure. Structural changes accompanying phase transitions could also be included into this effort. As defined by Spence (1980), for example, this is what distinguishes electron crystallography, which depicts the average unit cell contents, from the direct visualization of structure images, which implies examination of localized detail. For organic materials, search for such localized structure, of course, places the most constraints on the sample, in that the radiation dose required for such visualization might be too prohibitively large to make it possible, since more electrons are needed per unit area than for the experiments where only an average unit cell representation is required. Although the results from aromatic crystals are favorable, it may be that only an indirect inference of localized deviations can be made by examination of changes in the electron diffraction pattern. Even so, in comparison to x-ray studies, one has a unique opportunity to observe single patterns from very small crystallites so that the type of disorder is very clearly depicted.

172

DOUGLAS L. DORSET

n

n N=4

1 1.0

FIGURE7. Fourier transform pairs useful for the analysis of diffraction patterns.

Analysis of electron diffraction patterns for evidence of disorder requires the knowledge of a limited number of Fourier transform pairs, as depicted in Fig. 7, so that parts of a linear signal (i.e., the diffraction pattern) can be analyzed separately and the results recombined to suggest a plausible model. This is strictly comparable to the electrical engineering approach to signal analysis, where such transform pairs are used to analyze a complicated waveform from a “black box” to decide what circuitry could be responsible for it (analyzed, e.g., via its power spectrum-see Mason and

ELECTRON CRYSTALLOGRAPHY OF ORGANIC MOLECULES

173

Zimmerman, 1960). It is unfortunate that these intuitive concepts, with rare but significant exceptions (Lipson and Cochran, 1966), seem to be alien to many crystallographers. The most important Fourier transform pairs (represented here by the symbol u) required to disassemble and analyze the components of a diffraction pattern are as follows: First, one must relate the operations, convolution, represented by *, and correlation, represented by 0, to multiplication, represented by . Thus, if we have two functions in image space, f(r) and g(r), with respective Fourier transform in reciprocal space, F(s) and G(s),f(r)*g(r) = f(r) 0g( -r), then:

-

f(r)*g(r) u F(s) * G(s)

-

f(r) g(r) u Rs)*G(s)

0g(r) u F*(s) - G(s) f ( r ) - g(r) * F*(s) 0 G(s)

fir)

Now, iff(r) = rect(Na), i.e., a rectangle function with length Na (see Fig. 7), its Fourier transform, the so-called shape transform, is simply F(s) = sin nNau/nNau, with an intersection of the 0 line at a-I. Hence, if this is imagined to define the envelope of a lattice, the transform of a restricted lattice must have continuous streaks according to its shape transform. The other important transform pair starts withf(r) = A exp(-cZx2), a Gaussian function. Its Fourier transform is G(s) = (An”2/c)exp(- rrz/cz),which is another Gaussian. Suppose some sort of static disorder or thermal motion results in a Gaussian distribution of mass centers in a crystal. As one can appreciate from inspection of Fig. 7, the diffraction pattern will be restricted in resolution, because the Gaussian function, which is the Fourier transform, is narrower in its distribution. To complete the analysis, therefore, the first Gaussian function is convoluted with the space lattice to affect all mass centers, whereas the latter is multiplied by the reciprocal lattice to lead to a restricted diffraction resolution. Detailed discussions of these principles are found in books by Cowley (1981), Gaskill ( 1 9 7 0 Champeney (1973), and Lipson and Lipson (1969). Use of an optical bench to produce Fourier transforms of transparent masks is described by Taylor and Lipson (1964) and Harburn et a/. (1975). If the discrete maxima of an electron diffraction pattern are affected by a disorder or lattice irregularity, as indicated, then the apparent loss of intensity, e.g., from the Gaussian restriction of resolution, must be made up in some way, since, as stated by Guinier (1963), the amount of scattered radiation from a given mass must be constant. This will often appear as a continuous scattering signal, which also can be analyzed in terms of an

174

DOUGLAS L. DORSET

average structure. For example, the diffuse signal from a thermally agitated structure can be written (Amoros and Amoros, 1968):

lTDs = (FmoJz{l - e x p ( - ~ ~ ~ ( s ( ~ ) where B is the isotropic temperature factor introduced above. In this expression, it is assumed that the molecular motions are uncorrelated. Certain types of substituted disorder, on the other hand, where the molecular or group scattering factor fA is replaced with another fB, will result in diffuse scattered intensity given by the Laue formula (Guinier, 1963):

I(s) = CA cs(fA- fS)’(1

-k

2

cos 2ns ’ x,)

This allows for lattice correlations a, along x, . A more complete description of these analyses is given by Welberry and his coworkers (Welberry and Jones, 1980; Epstein et al., 1982; Welberry, 1985; Khanna and Welberry, 1987). It is important to notice here that the continuous signals from these disordered lattices are often related to the molecular or unit-cell transforms and have been used to aid crystal structure analysis by locating features of the molecular transform that would be lost in the coherent Patterson function (Hoppe, 1956, 1957; Hoppe and Baumgartner, 1957; Hoppe et al., 1957; Hoppe and Rauch, 1960). 2. Observations of Crystal Disorder Visualization of lattice disorder in organic crystals places different demands on the experimentalist, depending on what level of magnification is required to observe the phenomenon. For example, the screw dislocation instrumental for growth of paraffin crystals was first observed by Dawson (1952), who used metal-shadowed specimens. Such easily found morphological details are often used by polymer physicists, as illustrated in a recent atlas (Woodward, 1989). However, the change of local-packing due to the collapse of a three-dimensional crystal habit as it is flattened on an electron microscope grid surface, corresponding to sectorized bands in bright-field electron micrographs, requires a high-resolution experiment at low-electron beam doses (Dorset et al., 1990b). Observation of twin domains in diffraction contrast micrographs requires only very low magnifications but a detailed study of the packing at these boundaries again places more demands on the microscopist, although they may be satisfied (Kawaguchi et al., 1984). Evidence of edge dislocations was first observed indirectly by Moire magnification of bilayer crystals (Holland and Lindenmeyer, 1965), but the direct visualization at molecular resolution has only been recently reported for an aromatic polymer structure (Isoda et al., 1983) as well as lamellar structures (Fryer and Dorset, 1987; Voigt-Martin et al., 1990).

ELECTRON CRYSTALLOGRAPHY OF ORGANIC MOLECULES

175

Another claim of a high-resolution observation for an aliphatic material (Zemlin et al., 1985) must be tempered by the possibility of false detail in phase-contrast images caused by phase noise in Fourier-filtered images (Pradere et al., 1988). In fact, sympathy must be expressed for the questioned uniqueness of lattice simulations (Humphreys and Bithell, 1992) since even spurious detail (e.g., a putative edge dislocation) can be justified with an appropriate model (Dorset and Zemlin, 1987). Finally, non-Gaussian positional disorder, i.e., a form of paracrystallinity, has also been directly observed in lattice images of lamellar layer structures (Fryer and Dorset, 1987; Voigt-Martin et al., 1990). Although it is always the goal of the electron microscopist to obtain directly interpretable structure images of real organic crystals, it is often advisable to utilize information in electron diffraction patterns, at least to guide the search for suitable areas of the specimen. Small regions of twinned crystals with the same structural polarity, for example, can be expressed by streaks where the length of the sinnxlnx broadening is, as shown above, directly related to the limited real space dimension. Other types of streaking, which are more complicated to analyze, can result from, e.g., random loss of solvent layers (Cowley, 1976; Dorset, 1985a). Analysis of continuous diffuse diffraction signals is equally important. For example, the diffuse signal observed in the electron diffraction patterns from methylene subcell structures (Dorset, 1977c, 1978c) correspond well to the expectations of a thermally disordered crystal structure, although intermolecular correlations were required to explain this signal. In fact the first observation of such diffuse scattering was made with electrondiffraction patterns from anthracene microcrystals (Charlesby et al., 1939). Other types of diffuse scattering, on the other hand, cannot be attributed to thermal motion, as will be discussed below. Also, it should be pointed out that non-Gaussian disorder is also possible in the form of paracrystallinity (Hosemann and Bagchi, 1962) as applied, for example, to polymer problems (Granier et al., 1986). The foregoing only intends to give an overview of what can be observed. Its application to real structural problems is best illustrated by two general examples, i.e., the study of thermally induced disorder accompanying phase transitions and the assembly of molecules in binary solids. 3. Disorder Due to Phase Transitions Since the early use of electron diffraction to study heating stages have been included in the diffraction dynamic studies of phase transitions, both in reflection transmission (Charlesby, 1945) geometries. The only

organic molecules, cameras to permit (Tanaka, 1938) and restriction to these

176

DOUGLAS L. DORSET

thermotropic measurements is that the sample might be heated to a temperature where it will sublime under the high-vacuum environment. In many cases, however, the results are similar to those obtained in bulk samples. In our work, we have been primarily interested in the structural changes that occur in heated linear molecules. The easiest experiments to carry out are with solution-crystallized samples, e.g., to study the orthorhombic to rotator phase transition of n-paraffins (Dorset et al., 1984b). It is possible to make a plot of the unit cell axial ratio measured from the electron diffraction pattern at various temperatures to show that, at the rotator phase, there is an abrupt change to a = f i b . For the even-chain paraffins, one observes this transition up to ~Z-C,~H,, (Asbach and Kilian, 1970), and beyond this chain length the axial ratio only reaches a value around 1.55 (Dorset, 1991g; Dorset et al., 1992b). Intermediate stages have been observed when, for example, n-C3,&4 is compared to its perdeuterated analog. Although the protonated alkane is transformed to the hexagonal layer packing, the axial ratio of n-C,,D7, only reaches the value 1.68 (Dorset, 1991g), consistent with the fact that the premelt transition is observed by DSC only when samples are cooled from the melt. Although the longer n-paraffins do not undergo a premelt transition, there is another expression of increasing volume at these higher temperatures, because, near the melting point, longitudinal transitions take place, eventually leading to an oblique chain packing (Fischer, 1971). Local cooperative chain motions for heated crystals of all n-paraffins have been observed in electron micrographs as corrugations with roof structures (Keller, 1961; Fischer, 1971; Piesczek et al., 1974; Takamizawa et al., 1982). More information is obtained when epitaxially oriented samples are examined. For example, electron diffraction patterns of unheated n-C3&4 can be shown to correspond to well-ordered single crystallite areas, since the resolution of the lamellar (001) reflections is very high. As these crystals are heated toward the premelt transition, the resolution of this row of reflection dramatically decreases (Fischer, 1971; Dorset et al., 1984b). If one imagines a lamellar interface that accrues conformational disorder (as shown by vibrational spectroscopy (Maroncelli et al., 1985a; Kim et al., 1989a; Basson and Reynhardt, 1991; Jarrett et al., 1992), then an average model of this interface would contain a Gaussian distribution of atom occupancies. Since it is seen in Fig. 7 that the Fourier transform of a Gaussian function is another Gaussian with an inverse relationship of the original half-width, this disorder will limit the resolution of reflections that directly express the perfection of the lamellar repeat. The other intense reflections, which correspond to the polymethylene repeat, will not be so greatly changed. Even longer paraffins, such as n-C,,H,,, , which undergo no such premelt transitions, will exhibit similar changes in their electron diffraction patterns

ELECTRON CRYSTALLOGRAPHY OF ORGANIC MOLECULES

177

as they are heated toward the melting point (Dorset et ul., 1992b). Such information had also been obtained in earlier low-angle x-ray measurements (Fisher, 1971; Craievich et al., 1984a), but the electron diffraction study is the first visualization of these phenomena in terms of a single crystal. Also we see that a Gaussian disorder model is easier to analyze than the polynomial expressions given before (Craievich et al., 1984a). Using the lamellar repeat distance to index the intense 011 reflections (which are related to the carbon number CnH2n+Z of the average chain in the lamella, i.e., I = n, n + 2), it is possible to show that a reversible longitudinal chain translation occurs just below the melting point to produce a quasi-C,,H,, structure (Dorset et al., 1992b). This slightly thickened lamella reversibly shrinks back to the original structure when the sample is recooled. Continuous diffuse scattering can also provide information about the mechanism of such phase transitions. For example, the weak continuous signal observed in the projection down the chain axes (see above) was accounted for with a thermal model. This can be verified experimentally by cooling solution-crystallized paraffins to, e.g., 6 K to demonstrate that the diffuse scattering vanishes (Dorset et ul., 1991b). However, another very intense diffuse component in electron diffraction patterns from epitaxially crystallized n-paraffins or polyethylene does not disappear at low temperatures (e.g., 4 K). Although a model based on thermal vibrations predicts the location of the diffuse bands, a reverse ordering of the observed intensity is given by these calculations (Dorset el al., 1985). The only sensible model that correctly accounts for the distribution of diffuse intensity is one including slight longitudinal slip disorders of the chains in the crystal, obviously a residual disorder of a high-temperature crystallization (Dorset et al., 1991b). It is also possible to extend such studies to other types of molecules, including the cholesteryl esters. Using samples epitaxially oriented on benzoic acid, it was possible to heat them into the smectic state and even obtain diffraction patterns at a transition region where both the crystalline and mesostate coexisted (Dorset, 1985~).These experiments were useful for later work on binary solids to help interpret the phase diagram constructed from DSC scans (Dorset, 1987c), as will be discussed below. With the aid of a differentially pumped environmental stage, it has also been possible to follow gel state to smectic transitions in suspended, hydrated phospholipid bilayers (Hui and Strozewski, 1979). As will be discussed below in greater detail, such measurements again are of great assistance for interpretation of binary phase diagrams. Using a liquid-nitrogen-cooled specimen stage, it is also possible to detect subambient phase transitions, For example, DSC measurements of n-C,,F,, indicate that a subambient crystal-crystal transition occurs near - 50°C.

178

DOUGLAS L. DORSET

.

FIGURE8. Subambient phase transition of n-C,,F,, (a) DSC trace showing subambient endotherm. (b) Room-temperature hhl electron diffraction pattern (from epitaxially oriented sample). (c) Low-temperature hhl pattern.

Electron diffraction patterns from samples oriented on KC1 demonstrate that this endotherm corresponds to a marked increase in crystalline order (Fig. 8). Similar results have also been obtained recently from oriented films of poly(tetrafluoroethy1ene). Finally, it has also been possible to study a slow crystal-crystal phase transition of a membrane protein in the electron microscope (Dorset et al., 1989a). When the Omp F porin from the outer membrane of E. coli is reconstituted in a small amount of phospholipid, a trigonal layer form is seen with a lattice constant, a = 79 A. During our initial characterization of negatively stained electron micrographs by Fourier filtration, a computed power spectrum of the crystalline lattice was observed to contain directional continuous scattering (Fig. 9a) see later also in electron diffraction patterns (Fig. 9b). After satisfying to ourselves that this diffuse signal was not due to a Gaussian distribution of trimer centers around a unit cell origin (quasithermal disorder), we decided to monitor the sample periodically to see what else would occur. With time, FFT patterns computed from the electron microscope images were next observed to contain sharp streaks (Fig. 9c). These resolved into spots of an apparent superlattice, but sometimes the superlattice rows were confined to only two-fold or single orientation, thus breaking the hexagonal symmetry (Fig. 9d, e, f). When there was a single orientation, the pattern was recognized to be identical

ELECTRON CRYSTALLOGRAPHY OF ORGANIC MOLECULES

I79

FIGURE9. Diffraction evidence for a new orthorhombic phase growth in two-dimensional hexagonal crystals of Omp F porin. (a) FFT of electron microscope image showing initial diffuse signal. (b) Electron diffraction patterns with identical information; sequence of later events revealed by FFT of electron micrographs. (c) Sharp streaks eventually turn into individual reflections if a (d) three-fold, (e) two-fold, or (f) single orientation of the orthorhombic phase.

180

DOUGLAS L. DORSET

to that of a rectangular layer polymorph seen only rarely in original experiments, with cell constants: a = 79, b = 137 A , crystallizing in plane group pmg. A model hypothesizing a slow phase transition readily explained this sequence, which took a year at 4°C to go to completion. As shown by the structure analysis, the first event is a random rotation of a few isolated porin trimer sites by 60", and this will account for the observed diffuse signal. As small patches of the orthorhombic structure begin to grow, streaks will appear corresponding to the shape transform of a lattice with limited size. When these islands grow large enough so that the shape transform broadening of reflections is not significant, the streaks will be resolved into spots. Finally, a polar rectangular lattice grows in a lattice with three-fold symmetry, and there are three possible nucleations directions, accounting for the three possible diffraction patterns, depending on how many microareas with different orientations were imagined. For areas with just two orientations, it is possible to form an average of one area, rotate it, and then search the map for the other orthorhombic patch using the cross-correlation operation. To our knowledge, this is the first direct evidence for rotational diffusion of membrane proteins observed in the electron microscope. Earlier attempts to describe the effective potential function of integral membrane proteins relied on radial distribution analysis of particles in freeze-fracture electron micrographs (Abney et al., 1989; Braun et al., 1987). The porin trimer appears to behave as if it were a small van der Waals particle.

4. Crystallization of Multicomponent Organic Solids Recently, electron diffraction has proven itself to be a powerful technique for elucidating localized structural details about multicomponent organic solids, leading to some revisions of concepts based originally on measurements of the bulk state (Dorset, 1989a, 1990d). When the solids are comprised of similar materials, rules for the stabilization of solid solutions, originally formulated by Kitaigorodskii (1961), can be evaluated with single-crystal data. When they are composed of dissimilar materials, one can also look for specific interactions at intercrystalline boundaries in the eutectic solid. Phase diagrams were constructed for polymethylene compounds, such as the paraffins (Dorset et al., 1989b) or phospholipids (Dorset, 1990b), with aromatic materials such as naphthalene, anthracene, acridine, and benzoic acid, based on DSC measurements on solids formed at different molar concentrations. After removal of the diluent material and examination of the electron diffraction patterns from the remaining polymethylene compound, it was possible to evaluate what sort of interactions occur between

ELECTRON CRYSTALLOGRAPHY OF ORGANIC MOLECULES

181

the dissimilar ingredients in the solid at either side of the eutectic point. In general, when the solid is rich in diluent so that it is the first to crystallize from the melt, the ensuing growth of polymethylene component is epitaxially nucleated by specific lattice interactions between the components; this is the basis for the preparative procedure mentioned above. When the polymethylene component is the major ingredient, it crystallizes first from the co-melt with chain axes normal to the crystal face and with diluent filling in between the remaining gaps, perhaps with a specific interaction of the diluent (001) face with favorable match to the sublattice planes of the chain packing. Witmann and his coworkers have made the first extensive electron diffraction study of similar polymer-diluent interactions (Wittmann and Manley, 1977, 1978; Wittmann and Lotz, 1981a,b; Wittmann et al., 1983; Hodge et al., 1982) as they worked out the procedures for epitaxial crystallization. In the liquid crystalline state, Hui and his coworkers have been able to examine the phase behavior of cholesteroVphospholipid combinations in suspended hydrated bilayers, showing how domains of these ingredients grow below a certain temperature (Hui and Parsons, 1975; Stewart et al., 1979; Hui, 1981). When the two ingrediants are the same kind of molecule, the experiments are even more interesting. For example, in the n-paraffins, it is possible to follow the sequence of events from stable solid solutions to fully fractionated eutectics (Dorset, 1990e). All the binary solids of n-paraffins can be epitiaxially oriented on benzoic acid to permit their study by electron diffraction in a projection onto the chain axes. (Obviously, the projection down the chain axes, provided in solution-crystallized samples, is less useful, since the major expression of volume differences is manifested by change in lamellar spacing.) From electron diffraction measurements, the following information could be obtained about the structure of solid solutions: 0

0

0

Volume rules could be established for stabilization of an n-paraffin (and also fluorocarbon) solid solutions (Dorset, 1990f, g) and were shown to be in agreement with previous estimations (Matheson and Smith, 1985). A quantitative single-crystal structure of a solid solution was determined for the first time (Dorset, 1990~);the earlier single crystal x-ray analysis (Luth et al., 1974) was only qualitative. In this study it was shown that the chain-end carbons had a fractional occupancy in accord with the volume differences and the opportunity for longitudinal translation of the shorter component. (This was also apparent from low-angle x-ray analyses (Asbach et al., 1979; Craievich et al., 1984b).) Examination of localized crystal structures for adjacent microcrystalline areas (made possible with the indexing rules given above for

182

0

DOUGLAS L. DORSET

patterns from epitaxial paraffin crystals) reveals that there are many variations of unit cell symmetry and average lamellar spacing for any nominal bulk concentration, each closely mimicking a specific pure paraffin chain packing (Dorset, 1987d). This was not anticipated by Kitaigorodskii (or us) after his (or our) early analysis of diffraction data from bulk samples (Kitaigorodskii et al., 1958; Dorset, 1985d). Deviation from strict Vegard’s law behavior observed earlier was misinterpreted (Mnyukh, 1960) to mean that a longer component dominates the crystal packing in regions where it is a major component. The actual situation is more complicated than this, involving a comixture of the several possible crystalline forms found for a given concentration (Dorset, 1987d, 1990f). The structural rules also apply to polydisperse solid solutions. For example, electron diffraction measurements have been made (Dorset, 1987d) on a commercial paraffin wax, i.e., Gulfwax, which has a Gaussian distribution of component concentrations, as well as an artificial equimolar paraffin combination of all even chain lengths from n-C,,H5, to ~ Z - C ~ ~ H , , .

It should be pointed out, however, that the distribution of conformational defects in such solid solutions cannot be determined from diffraction data (Bragg or diffuse) and, here, spectroscopic measurements are of optimal use (Snyder eta/., 1982; Maroncelli eta)., 1985b, Kim et al., 1989b; Basson and Reynhardt, 1992). When the volume difference between the two components is increased, there is a narrow range where metastable solid solutions are initially formed from the melt. These can persist over several hours, days, weeks, months, the time frame depending on the two components of the solid. The first observation of this metastable solid was made by low-angle x-ray diffraction (Mazee, 1958). Electron diffraction patterns of these freshly grown solids, correspondingly, resemble those from stable solid solutions. Upon standing, a fractionation process occurs (Dorset, 1986d, 1990f), corresponding to the appearance of a binodal phase boundary in the phase diagram. The mechanism of this local fractionation is still being worked out (aided in part by neutron scattering [White et al., 19901 and vibrational spectroscopic measurements [Snyder et al., 1992; Dorset et al., 1992~1). However, in electron diffraction patterns, the lamellar row of reflections of the initial solid solution transforms into a superlattice-like repeat (Dorset, 1986d), the spacings of which are dependent on the molar concentrations of the ingredients. A structure that accounts for this lamellar repeat involves a random sequence of pure chain packings. Spectroscopic (as well as neutron scattering) measurements are made to evaluate how wide this pure domain

ELECTRON CRYSTALLOGRAPHY OF ORGANIC MOLECULES

183

is for any lamellar surface (Snyder el al., 1992). Low-dose electron microscope lattice images also can be used to depict the growth of this separate phase from the solid solution (Dorset, 1986c, 1990f). With further volume difference, eutectics of solid solutions are grown (Dorset, 1990f). Electron diffraction patterns from the epitaxially oriented solid consist of a superlattice (Dorset, 1986d), similar to that formed from the metastable solid solution, coexisting with a separated phase of a nearly pure longer or shorter component (depending on the melt concentration). Direct low-dose lattice images reveal that the phase boundary between the two domains is a methyl plane interface-i.e., an exact crystallographic match of two lamellar surfaces (Zhang and Dorset, 1989a). Totally fractionated eutectics have a very similar structure. These are produced when the chain length difference of the two components is too large (Dorset, 1990f,g) or when the two components are incompatible, e.g., when an n-paraffin is cocrystallized with a wax ester of nearly the same chain length (Dorset, 1989b). In the latter case, there is also an exact one-dimensional epitaxial match of lamellar interfaces between the two domains. Another variation can be imposed when samples are epitaxially crystallized from the vapor phase (Zhang and Dorset, 1989a). For n-C,,H,,,/ n-C60H122binaries, nematically disordered structures are initially formed, and the stable lamellar structure is seen only after annealing (Zhang and Dorset, 1990b). Similar results have been observed spectroscopically for shorter chain alkanes (Hagemann et a / . , 1987). The experiments reviewed above demonstrated that the sequence of solids with increasing molecular volume differences must be treated as a sequence of crystal structures. The concept of mechanical mixtures of components in eutectics (Hsu and Johnson, 1973), for example, has no meaning. We have also investigated the influence of perdeuteration of one component on the behavior of binary solids with electron diffraction patterns. For the same chain length, the phase diagrams are nearly ideal in behavior, also in the orthorhombic to hexagonal transition (Dorset, 1991g). When the chain lengths differ, electron diffraction measurements confirm the presence of apparently anomalous behavior when n-CnH2n+2/n-CnD2n+2 compared to n-CnD2n+2/n-CnH2n+2, especially in the range where the metastable solid solutions are formed. When the shorter component is perdeuterated, there is a tendency for the phase separation to occur much more quickly. This has been explained in terms of the volume difference between deuterium and hydrogen which adds to a significant difference for such molecules (Snyder et al., 1992; D. L. Dorset, unpublished data). Similar studies have been carried out on binary solids composed of cholesteryl esters (Dorset, 1988c, 1990g,h), based on the electron diffraction measurements of samples epitaxially oriented on benzoic acid (Dorset,

184

DOUGLAS L. DORSET

1987~).In this study, chain length series in two different layer packings, namely, monolayer I and the bilayer forms (Craven, 1986), were investigated. Within the same layer packing, the volume rules for cosolubilization of two components are similar to those found for the n-paraffins, even for crystal structures where no sequestered methylene subcell packing exists. However, no matter how close the formal volume differences, if two components each favor a separate crystal structure, they are shown to fractionate. The most trivial example is for esters (e.g., laurate, tridecanoate) which are polymorphic (i.e., no volume difference), for which electron diffraction patterns can be found from either form at random across a grid surface. Electron diffraction studies of heated binary samples have also been important for the interpretation of phase diagrams. Plots of lamellar spacing versus concentration can also detect structural details in solid solutions similar to the ones found for the alkanes, i.e., the dominance of certain domains by a given average crystal structure. The detection of continuous electron diffraction spacings for the solid solutions gave us the courage to attempt the growth of larger crystals for collection of single-crystal x-ray data, leading to the first x-ray structures of such linear molecule solid solutions, i.e., for cholesteryl undecanoate/cholesteryl laurate (Dorset and Pangborn, 1992), where there is a statistical occupancy of the two chains in the average structure, and for cholesteryl caprate/ cholesteryl laurate, where a microfractionation occurs somewhat similar to the metastable alkane solid solutions (McCourt et al., 1993). It is also possible to epitaxially orient binary solids of phospholipids (Dorset and Massalski, 1987). Parameters such as chain linkages, relative chain lengths and headgroup types have been considered in one study that included electron diffraction and low-dose, high-resolution electron microscopy. In this work on anhydrous paracrystals, small volume differences were again found to be tolerated as solid solutions, similar to other diffraction results from the lyotropic liquid crystalline forms (Knoll et al., 1981). It is also been possible to investigate the cosolubilization of phospholipids in hydrated phospholipid bilayers using electron diffraction and diffraction contrast microscopy (the latter to characterize domain formation when phase separation occurs) (Stewart et al., 1979). VI, CONCLUSIONS

When conditions dictate that microcrystals are the only chance for obtaining single-crystal preparations, or when one wishes preferentially to probe structures in microdomains, electron diffraction and low-dose microscopy are found to be powerful quantitative tools for crystallographic structure

ELECTRON CRYSTALLOGRAPHY OF ORGANIC MOLECULES

185

determination. Although the several possible perturbations to the measured electron scattering are recognized to be present and even important, it has been shown that the importance of their influence for affecting ab initio analyses is often overrated to the point that the actual experiment is never attempted. Realistic knowledge of their influence on measured intensities allows one to adjust experimental conditions appropriately so that data adequately near the single scattering approximation from the entire unit cell can be collected. From these data an ab initio crystal structure analysis can be carried out without referring to any other structure previously determined by any other means. From many comparisons to independent x-ray structure analyses, moreover, it is seen now that the results obtained by electron crystallography are equivalent to those from any other crystallographic technique. Thus, although the technique of electron crystallography will never replace x-ray crystallography, there are applications where its use is actually the most appropriate one. ACKNOWLEDGMENTS Work carried out in this laboratory has been supported by grants from various US and foreign governmental, as well as private, agencies: National Institute of General Medical Sciences (GM-21047, GM-46733), the National Science Foundation (PCM78-16041, CHE79-16916, CHE/DMR81-16318, INT82-13903, INT84-01669, DMR86- 10783, CHE91- 13899), the Swiss Nationalfonds (to Prof. J. P . Rosenbusch), the Roche Research Foundation for Scientific Exchange and Biomedical Collaboration with Switzerland, the Cummings Foundation, The Manufacturers and Traders Trust Company, the Baird Foundation, and the Helen Woodward Rivas Memorial Fund. I am grateful for fruitful collaborations with many colleagues, with especial thanks to Herb Hauptman, John Fryer, Bob Snyder, Jean-Claude Wittmann, Bernard Lotz, Fritz Zemlin, Elmar Zeitler, and Jurg Rosenbusch. I am grateful to Dr. H. Chanzy for sending a data set from an unknown polymer structure. Harvey Fishman is the electrical engineer who unwittingly taught me crystallography, for which I am grateful. Also many thanks are due to coworkers who have been in this lab, including Barb Moss, Walt Pangborn, Andrew Massalski, Dale Hu, Weiping Zhang, Sophie Kopp, and Mary McCourt. REFERENCES Abney, J. R . , Scalettar, B. A., and Owicki, J . C. (1989). Biophys. J. 55, 817-833. Abrahamsson, S., Dahlen, B., Lofgren. H . , and Pascher, I . (1978). Progr. Chem. Fats. Other Lipids 16, 125-143.

186

DOUGLAS L. DORSET

Alexander, L. E. (1985). X-ray Diffraction Methods in Polymer Science. Krieger, Malabar, Florida. Amoros, J . L., and Amoros, M. (1968). Molecular Crystals: Their Transforms and Diffuse Scattering. Wiley, New York. Amos, L. A., Henderson, R., and Unwin, P. N. T. (1982). Progr. Biophys. Molec. Biol. 39, 183-231,

Arnott, S., and Wonacott, A. J . (1966). Polymer 7, 157-166. Asbach, G. I., Geiger, K., and Wilke, W. (1979). Colloid Polym. Sci. 257, 1049-1059. Asbach, G. I., and Kilian, H. G. (1970). Ber. Bunsenges. Phy. Chem. 74, 814-823. Ashida, M. (1991). In Electron Crystallography of Organic Molecules (J. R. Fryer and D. L. Dorset, eds.), pp. 227-240. Kluwer, Dordrecht. Atkins, E. D. T. (1989). In Comprehensive Polymer Science, Vol. 1, Polymer Characterization (C. Booth and C. Price, eds.), pp. 613-650. Pergamon, Oxford. Barry, J. (1992). In ElectronDiffraction Techniques(J.M. Cowley,, ed.), Vol. 1, pp. 170-211. Oxford University Press, Oxford. Basson, I., and Reynhardt, E. C. (1991). J. Chem. Phys. 95, 1215-1222. Basson, I. and Reynhardt, E. C. (1992). J. Chem. Phys. 97, 1287-1295. Bittiger, H.,Marchessault, R. H., and Niegisch, W. D. (1970). Acta Cryst. B26, 1923-1927. Blackman, M. (1939). Proc. Roy. SOC. (London) 173A,68-82. Blundell, D. J., Keller, A., and Kovacs, A. J . (1966). J. Polym. Sci.-Polym. Lett. 4,481-486. Boistelle, R., and Aquilano, D. (1978), Acta Cryst. A34, 406-413. Boudeulle, M. (1975). Cryst. Struct. Commun. 4, 9-12. Braun, J., Abney, J. R., and Owicki, J. C. (1987). Biophys. J. 52, 427-439. Brink, J . , and Chiu, W. (1991). J. Microsc. (Oxford) 161,279-295. Brisse, F. (1989). J. Electr. Microsc. Techn. 11, 272-279. Brisse, F., Palmer, A., Moss, B., Dorset, D., Roughhead, W. A., and Miller, D. P. (1984a). Eur. Polym. J. 20, 791-797. Brisse, F., Remillard, B., and Chanzy, H. (1984b). Macromolecules 17, 1980-1987. Brisson, J., Chanzy, H., and Winter, W. T. (1991). Int. J. Biol. Macromol. 13, 31-39. Buchheim, W. (1970), Kiel. Milchwirtsch. Forschungsber. 22, 3-61. Carlemalm, E., and Wieslander, A. (1975). Nature 254, 537-538. Champeney, D. C. (1973). Fourier Transforms and Their Physical Applications. Academic Press, London. Chang, C. F., Mizushima, S . , and Glaeser, R. M. (1985). Biophys. J. 47, 629-639. Chanzy, H.,Folda, T., Smith, P., Gardner, K. H., and Revol, J . F. (1986). J. Muter. Sci. Lett. 5, 1045-1047.

Chanzy, H., Guizard, C., and Vuong, R. (1977). J. Microsc. (Oxford) 111, 143-150. Chanzy, H., Perez, S . , Miller, D. P., Paradossi, G., and Winter, W. T. (1987). Macromolecules 20, 2407-241 3. Charlesby, A. (1945). Proc. Phys. SOC. (London) 51, 510-518. Charlesby, A., Finch, G. I., and Wilman, H. (1939). Proc. Phys. SOC. (London) 51, 479-528.

Chatani, Y., Okita, Y., Tadokoro, H., and Yamashita, Y. (1970). Polym. J. 1, 555-562. Chien, J . C. W., Karasz, F. E., and Shimamura, K. (1982). Macromolecules 15, 1012-1017. Chiu, W.,and Jeng, T. W. (1982). Ultramicroscopy 10, 63-70. Claffey, W., and Blackwell, J . (1976). Biopolymers 15, 1903-1915. Claffey, W., Gardner, K.,Blackwell, J., Lando, J., and Geil, P. H. (19674). Phil. Mag. 30, 1223-1232.

Clark, W. R. K., Chapman, J. N., and Ferrier, R. P. (1979). Nature 277, 368-370. Cojazzi, G., Malta, V., Celotti, G., and Zannetti, R. (1976). Makromol. Chem. 177,915-926.

ELECTRON CRYSTALLOGRAPHY OFORGANICMOLECULES

187

Cowan, S. W., Schirmer, T., Rummel, G . , Steiert, M., Ghosh, R., Pauptit, R. A,, Jansonius, J. N., and Rosenbusch, J . P. (1992). Nature 358, 727-733. Cowley, J. M. (1953). Acta Cryst. 6, 522-529. Cowley, J . M. (1961). Actu Cryst. 14, 920-927. Cowley, J . M. (1967). Progr. Mater. sci. 13, 269-321. Cowley, J . M. (1976). Acra Cryst. A32, 88-91. Cowley, J . M. (1981). Dgjruction Physics (2nd rev. ed.). North-Holland, Amsterdam. Cowley, J. M. (1992). In Electron Diffraction Techniques (J. M. Cowley, ed.), Vol. 1, pp. 1-74. Oxford University Press, Oxford. Cowley, J . M., and Goswami, A. (1961). Acra Cryst. 14, 1071-1079. Cowley, J . M., and Kuwabara, S. (1962). Acta Cryst. 15, 260-269. Cowley, J . M., and Moodie, A. F. (1957). Acta Cryst. 10, 609-619. Cowley, J . M., and Moodie, A. F. (1959). Acta Cryst. 12, 360-367. Cowley, J. M., Rees, A. L. G., and Spink, J . A. (1951). Proc. Phys. SOC.(London) A64, 609-61 9. Craievich, A. F., Denicolo, I., and Doucet, J . (1984a). Phys. Rev. B30, 4782-4787. Craievich, A . F., Doucet, J., and Denicolo, I. (1984b). J. Phys. (Paris) 45, 1473-1477. Craven, B. M. (1986). In Handbook of Lipid Research. Vol. 4, The Physical Chemistry of Lipids (D. M. Small, ed.), pp. 149-182. Plenum, New York. Crowther, R. A,, DeRosier, D. J., and Klug, A. (1970). Proc. Roy. SOC.(London) A317, 319-340. Dawson, I. M. (1952). Proc. Roy. SOC.(London) A214, 72-79. Day, D., and Lando, J. B. (1980). Macromolecules 13, 1483-1487. DeRosier, D. J., and Klug, A. (1968). Nature 217, 130-134. DeTitta, G. T., Edmonds, J. W., Langs, D. A., and Hauptman, H . (1975). Acta Cryst. A31, 472-479. Dong, W., Baird, T., Fryer, J . R., Gilmore, C . J., MacNicol, D. D . , Bricogne, G., Smith, D. J . , O’Keefe, M. A., and Hovmoller, S. (1992). Nature 355, 605-609. Dorset, D. L. (1974). Chem. Phys. Lipids 13, 133-140. Dorset, D. L. (1975a). Biochim. Biophys. Acta 380, 257-263. Dorset, D. L. (1975b). Chem. Phys. Lipids 14, 291-296. Dorset, D. L. (1976a). J. Appl. Cryst. 9, 142-144. Dorset, D. L. (1976b). Acta Cryst. A32, 207-215. Dorset, D. L. (1976~).J. Appl. Phys. 47, 780-782. Dorset, D. L. (1976d). Bioorg. Khim. 2, 781-788. Dorset, D. L. (1976e). Biochim. Biophys. Acta 424, 396-403. Dorset, D. L. (1977a). Chem. Phys. Lipids 20, 13-19. Dorset, D. L. (1977b). Z. Narurjorsch. 32e. 1167-1172. Dorset, D. L. (1977~).Z. Naturjorsch. 32a, 1161-1165. Dorset, D. L. (1978a). Z. Nuturforsch. 33a, 964-982. Dorset, D. L. (1978b). In Direct Merhods in CrystaNography (H. Hauptman, ed.), pp. 87-92. Edwards, Buffalo. Dorset, D. L. (1978~).Z. Naturforsch. 33a, 1090-1092. Dorset, D. L. (1979a). J. Polym. Sci. Polym. Phys. Ed. 17, 1797-1808. Dorset, D. L. (1979b). Chem. Phys. Lipids 23, 337-347. Dorset, D. L. (1980). Acra Cryst. A36, 592-600. Dorset, D. L. (1983a). J . Colloid fnterfuc. Sci. 96, 172-181. Dorset, D. L. (1983b). LIItrumicroscopy 12, 19-28. Dorset, D. L. (1983~).Z. Nururforsch. 38c, 511-514. Dorset, D. L. (1984). Ultramicroscopy 13, 305-310.

188

DOUGLAS L. DORSET

Dorset, D. L. (1985a). J. Electr. Misrosc. Techn. 2, 90-128. Dorset, D. L. (1985b). Macromolecules 18,2158-2163. Dorset, D. L. (1985~).J. LipidRes. 26, 1142-1150. Dorset, D. L. (1985d). Macromolecules 18,2158-2163. Dorset, D. L. (1986a). J. Macromol. Sci.-Phys. B25, 1-20. Dorset, D. L. (1986b). Ultramicroscopy 19, 31 1-316. Dorset, D. L. (1986~).J. Polym. Sci.-Polym. Phys. Ed. 24,79-87. Dorset, D. L. (1986d). Macromolecules 19,2965-2973. Dorset, D. L. (1987a). J. Electr. Microsc. Techn. 7, 35-46. Dorset, D. L. (1987b). Biochim. Biophys. Acta 989, 121-128. Dorset, D. L. (1987~).J. Lipid Res. 28, 993-1005. Dorset, D. L. (1987d). Macromolecules 20, 2782-2788. Dorset, D. L. (1987e). Chem. Phys. Lipids 43, 179-191. Dorset, D. L. (1988a). Biochim. Biophys. Acta 938,279-292. Dorset, D. L. (1988b). Z . Naturforsch. 43c,319-327. Dorset, D. L. (1988~).Biochim. Biophys. Acta 963, 88-97. Dorset, D. L. (1989a). J. Electr. Microsc. Techn. 11, 298-309. Dorset, D. L. (1989b). J. Polym. Sci. B. Polym. 27, 1161-1171. Dorset, D. L. (1990a). E.M.S.A. Bulletin 20(1), 54-63. Dorset, D. L. (1990b). J. Phys. Chem. 94,6854-6858. Dorset, D. L. (1990~).Biophys. J. 58, 1077-1087. Dorset, D. L. (1990d). Chemtracts: Macromol. Chem. 1, 311-319. Dorset, D. L. (199Oe). Macromolecules 23, 623-633. Dorset, D. L. (1990f). Macromolecules 23, 894-901. Dorset, D. L. (1990g). Biochim. Biophys. Acta 1046, 195-201. Dorset, D. L. (1990h). Biochim. Biophys. Acta 1046, 57-63. Dorset, D. L. (1991a). Ultramicroscopy 38,23-40. Dorset, D. L. (1991b). Macromolecules 24, 1175-1178. Dorset, D. L. (1991~).Proc. Natl. Acad. Sci. USA 88, 5499-5502. Dorset, D. L. (1991d). Acta Cryst. A47, 510-515. Dorset, D. L. (1991e). Biophys. J. 60, 1356-1365. Dorset, D. L. (19910. Biophys. J. 60, 1366-1373. Dorset, D. L. (1991g). Macromolecules 24,6521-6526. Dorset, D. L. (1992a). Acta Cryst. A48, 568-574. Dorset, D. L. (1992b). Macromolecules 25, 4425-4430. Dorset, D. L. (1992~).Ultramicroscopy 45, 5-14. Dorset, D. L. (1992d). Ultramicroscopy 45, 357-364. Dorset, D. L. (1992e). Ultramicroscopy 41, 349-357. Dorset, D. L.,Alamo, R. G., and Mandelkern, L. (1992b). Macromolecules 25, 6284-6288. Dorset, D. L., Beckmann, E., and Zemlin, F. (1990a). Proc. Natl. Acad. Sci. USA 87, 7570-7573. Dorset, D., L.,Engel, A,, Haner, M., Massalski, A., and Rosenbusch, J. P. (1983b). J. Mol. Biol. 165,701-710. Dorset, D. L., and Ghiradella, H. (1983). Biochim. Biophys. Acta 760, 136-142. Dorset, D. L., and Hancock, A. J. (1977). Z . Naturforsch. 32c, 572-580. Dorset, D. L., Hanlon, J., and Karet, G. (1989b). Macromolecules 22, 2169-2176. Dorset, D. L., Hanlon, J., McConnell, C. H., Fryer, J. R., Lotz, B., Wittmann, J. C., Beckmann, E., and Zemlin, F. (1990b). Proc. Natl. Acad. Sci. USA 87, 1696-1700. Dorset, D. L., and Hauptman, H. A. (1976). Ultramicroscopy 1, 195-201. Dorset, D. L.,Holland, F. M., and Fryer, J. R. (1984a). Ultramicroscopy 13,311-324.

ELECTRON CRYSTALLOGRAPHY OF ORGANIC MOLECULES

189

Dorset, D. L.. and Hsu, S . L. (1989). Polymer 30, 1595-1602. Dorset, D. L., Hu, H., and Jager, J . (1991b). Actu Cryst. A47, 545-549. Dorset, D. L., Hui, S. W., and Strozewski, C. M. (1976). J. Suprumol. Strucr. 5 , 1-14. Dorset, D. L., and Hybl, A. (1972). Science 176, 806-808. Dorset, D. L., Jap, B. K., Ho,M. S., and Glaeser, R. M. (1979). Acfu Cryst. A35, 1001-1009. Dorset, D. L., and Massalski, A. K. (1987). Biochim. Biophys. Actu 903, 319-332. Dorset, D. L., Massalski, A. K., and Fryer, J. R. (1987). Z. Nuturforsch. 42a. 381-391. Dorset, D. L., Massalski, A. K., and Rosenbusch, J. P. (1989a). Proc. Natl. Acud. Sci. USA 86, 6143-6147. Dorset, D. L., McCourt, M. P., Kopp, S . , Wittmann, J . C., and Lotz, B. (1933). Actu Cryst., in press. Dorset, D. L., and Moss, B. (1983). Polymer 24, 291-294. Dorset, D. L., Moss, B., Wittmann. J. C., and Lotz, B. (1984b). Proc. Natl. Acud. Sci. USA 81, 1913-1917. Dorset, D. L., Moss, B., and Zemlin, F. (1985). J. Mucromol. Sci.-Phys. B24, 87-97. Dorset, D. L., and Pangborn, W. A. (1979). Chem. Phys. Lipids 25, 179-189. Dorset, D. L., and Pangborn, W. A. (1982). Chem. Phys. Lipids30, 1-15. Dorset, D. L., and Pangborn, W. A. (1988). Chem. Phys. Lipids 48, 19-28. Dorset, D. L., and Pangborn, W. A. (1992). Proc. Nail. Acud. Sci. USA 89, 1822-1826. Dorset, D. L., Pangborn, W.A.. and Hancock, A. J. (1933a). J. Biochem. Biophys. Meth. 8 , 29-40. Dorset, D. L., Pangborn, W. A., Hancock, A. J., and Lee, I. S. (1978a). Z. Nuturforsch. 33e, 39-49. Dorset, D. L., Pangborn, W. A., Hancock, A. J., Van Soest, T. C., and Greenwald, S. M. (197813). 2. Nuturforsch. 3312, 50-55. Dorset, D. L., and Parsons, D. F. (1975). Actu Cryst. A31, 210-215. Dorset, D. L., Snyder, R. G., Strauss, H. L., Goh, M. C., Conti, G., and Srivatsavoy, V. J. P. (1992~).Amer. Chem. SOC.Div. Polym. Chem. Inc., Polymer Preprinfs 33(1). 339. Dorset, D.L., Tivol, W. F., and Turner, J . N. (1991a). Ultramicroscopy 38, 41-45. Dorset, D. L., Tivol, W. F., and Turner, J. N. (1992a). Actu Crysf. A48, 562-568. Dorset, D. L., and Zemlin, F. (1985). Ultramicroscopy 17, 229-236. Dorset, D. L., and Zemlin, F. (1987). Ultramicroscopy 21, 263-270. Dorset, D. L.,and Zemlin, F. (1990). Ultrumicroscopy 33, 227-236. Dorset, D. L., and Zhang, W. P. (1990). Biochem. Biophys. Actu 1028.299-303. Dorset, D. L., and Zhang, W. P. (1991). J. Electr. Microsc. Techn. 18, 142-147. Downing, K. H., and Glaeser, R. M. (1986). Ultramicroscopy 20, 269-278. Doyle, P. A., and Turner, P. S. (1968). Actu Cryst. A24, 390-397. Dvoryankin, V. F., and Vainshtein, B. K. (1960). SOV.Phys. Crystullogr. 5 , 564-574. Dvoryankin, V. F., and Vainshtein. B. K. (1962). SOV. Phys. Crysfullogr. 6 , 765-772. D’yakon, I. A.. Kairyak, L. N., Ablov, A. V., and Chapurina, L. F. (1977). Dokl. Akad. Nauk SSSR 236, 103-105. Eades, J. A. (1992). In Electron Diffraction Techniques (J. M. Cowley, ed.), Vol. 1, pp. 313-359. Oxford University Press, Oxford. Elder, M., Hitchcock, P., Mason, R., and Shipley, G. G. (1977). Proc. Roy. SOC.(London) B354, 157-170. Engel, A., Hoenger, A., and Zulauf, M. (1992). Trans. Amer. Crysf Assoc., in press. Engel, A., Massalski, A., Schindler, H., Dorset, D. L., and Rosenbusch, J. P. (1985). Nuture 317, 643-645. Epstein, J., Welberry, T. R., and Jones, R. D. G. (1982). Actu Cryst. A38, 611-618. Erickson, H. P. (1963). Adv. Optical Electron Microsc. 5 , 163-199.

190

DOUGLAS L. DORSET

Fan, H. F. (1992). In Direct Methods of Solving Crystal Structures (H. Schenk, ed.), pp. 265-272. Plenum, New York. Fan, H. F., Xiang, S. B., Li, F. H., Pan, Q., Uyeda, N., and Fujiyoshi, Y. (1991). Ultramicroscopy 36, 361-365. Fan, H. F., Zhong, 2. Y., Zheng, C. D., and Li, F. H. (1985). Acta Cryst. A41, 163-165. Farnell, G. C., and Flint, R. B. (1969). J . Microsc. (Oxford) 89, 37-41. Ferrier, R. P. (1969). Adv. Optic. Electron Microsc. 3, 155-216. Fischer, E. (1920). Chem. Ber. 53, 1621-1633. Fischer, E. W. (1971). Pure Appl. Chem. 26, 385-421. Frank, J. (1980). In Computer Processing of Electron Microscope Images (P.W . Hawkes, ed.), pp. 187-222. Springer-Verlag, Berlin. Franks, N. P. (1976). J. Mol. Biol. 100, 345-348. Frede, E., and Precht, D. (1974). Kiel. Milchwirtsch. Forschungsber. 26, 325-332. Fryer, J. R. (1978). Acta Cryst. A34, 603-607. Fryer, J. R. (1978a). The Chemical Applications of Transmission Electron Microscopy. Academic Press, London. Fryer, J. R. (1979b). Acta Cryst. A35, 327-332. Fryer, J. R. (1980), J. Microsc. (Oxford) 120, 1-14. Fryer, J . R. (1981). Inst. Phys. Conf. Ser., NO. 61, Chapter 1, pp. 19-22. Fryer, J. R. (1989). J. Electr. Microsc. Techn. 11, 310-325. Fryer, J . R., and Dorset, D. L. (1987). J. Microsc. (Oxford) 145,61-68. Fryer, J . R., Hann, R. A., and Eyres, B. L. (1985). Nature 313, 382-384. Fryer, J. R., and Holland, F. (1983). Ultramicroscopy 11, 67-70. Fujiwara, K. (1959). J. Phys. SOC.Japan 14, 1513-1524. Fujiyoshi, Y., Kobayashi, T., Ishizuka, K., Uyeda, N., Ishida, Y., and Harada, Y. (1980). Ultramicroscopy 5, 459-468. Fujiyoshi, Y., Uyeda, N., Morikawa, K . , and Yamagishi, H. (1984). J. Mol. Eiol. 172, 347-354. Garavito, R. M.,Jenkins, J., Jansonius, J. N., Karlson, R., and Rosenbusch, J. P. (1983). J. Mol. B i d . 164, 313-327. Garavito, R. M., and Rosenbusch, J. P. (1986). Meth. Enzymology 125,309-328. Garoff, S., Deckmann, H. W., Dunsmuir, J . H., Alvarez, M. S., and Bloch, J . M. (1986). J. Phys. (Paris) 47, 701-709. Gaskill, J. D. (1978). Linear Systems, Fourier Transforms, and Optics. Wiley, New York. Gassmann, J. (1976). In Crystallographic Computing Techniques (F. R. Ahmed, ed.), pp. 144-154. Munksgaard, Copenhagen. Gassmann, J., and Zechmeister, K. (1972). Acta Cryst. A28, 270-280. Geil, P. H. (1973). Polymer Single Crystals. Krieger, Huntington, New York. Germain, G., Main, P., and Woolfson, M. M. (1971). Acta Cryst. A27, 368-376. Germer, L. H., and Storks, K. H. (1938). J. Chem. Phys. 6, 280-293. Gilmore, C. J., Shankland, K., and Fryer, J. R. (1992). Trans. Amer. Cryst. Assoc., in press. Gjmnes, J., and Moodie, A. F. (1965). Acta Cryst. 19, 65-67. Glaeser, R. M. (1975). In Physical Aspects of Electron Microscopy and Microbeam Analysis (B. M. Siege1 and D. R. Beaman, eds.), pp. 205-229. Wiley, New York. Glaeser, R. M. (1985). Ann. Rev. Phys. Chem. 36,243-275. Glaeser, R. M., and Thomas, G. (1969). Eiophys. J. 9, 1073-1099. Goodman, P., and Moodie, A. F. (1974). Acta Cryst. A30, 280-290. Granier, T.,Thomas, E. L., Gagnon, D. R., Karasz, F. E., and Lenz, R. W. (1986). J. Polym. Sci. B. Polym. Phys. 24,2793-2804.

ELECTRON CRYSTALLOGRAPHY OF ORGANIC MOLECULES

191

Green, E. A., and Hauptman, H. (1976). Acta Cryst. A32, 43-49. Grubb, D. T. (1974). J. Mater. Sci. 9, 1715-1736. Guinier, A. (1963). X-ray DSffraction in Crystals, Imperfect Crystals, and Amorphous Bodies. Freeman, San Francisco. Guizard, C., Chanzy, H., and Sarko, A. (1984). Macromolecules 17, 100-107. Guizard, C., Chanzy, H., and Sarko, A. (1985). J. Mol. Eiol. 183, 397-408. Hagemann, H., Strauss, H. L . , and Snyder, R. A. (1987). Macromolecules 20, 2810-2819. Hall, C. E. (1966). Introduction lo Electron Microscopy, pp. 300-301. McGraw-Hill, New York. Hamilton, W. C. (1964). Statistics in Physical Science. Estimation, Hypothesis Testing and Least Squares, pp. 157-162. Ronald, New York. Han, F. S., Fan, H . F., and Li, F. H. (1986). Acta Cryst. A42, 353-356. Harburn, G., Taylor, C. A., and Welberry, T. A. (1975). Atlasof Optical Transforms, Cornell University, Ithaca, New York. Harlos, K., Eibl, H., Pascher, I., and Sundell, S. (1984). Chem. Phys. Lipids 34, 116-126. Hauptman, H. A. (1972). Crystal Structure Determination. The Role of the Cosine Seminvariants. Plenum, New York. Hauptman, H., and Karle, J. (1953). Solution of the Phase Problem. I. The Centrosymmetric Crystal, American Crystallographic Association Monograph No. 3. Havinga, E., and DeWael, J . (1937). Chem. Weekblad 34, 694-701. Heard, D., Roberts, G. G . , Holcrift, B., and Goringe, M. J. (1988). Thin Solid Films 160, 39 1 -499. Heidenreich, R. D. (1964). Fundamentals of Transmission Electron Microscopy, pp. 199-203. Interscience, New York. Henderson, R., Baldwin, J. M., Downing, K . H., Lepault, J., and Zemlin, F. (1985). Ultramicroscopy 19, 147- 178. Henderson, R., and Glaeser, R. M. (1985). Ultramicroscopy 16, 139-150. Henderson, R., and Unwin, P. N. T. (1975). Nature 257, 28-32. Henry, N. F. M., and Lonsdale, K. (eds.) (1952). International Tables f o r X-ray Crystallography, Vol. I , Symmetry Groups. Kynoch, Birmingham, England. Hitchcock, P. B., Mason, R., and Shipley, , G. G. (1975). J. Mol. Eiol. 94, 297-299. Ho. M. H., Jap, B. K., and Glaeser, R. M. (1979). Proc. 37th Ann EMSA Meeting (G. W. Bailey, ed.), pp. 118-119, Claitor’s Publ., Baton Rouge, Louisiana. Hodge, A. M., Kiss, G . , Lotz, B., and Wittrnann, J. C. (1982). Polymer 23, 985-990. Holland, V. F. (1964). J. Appl. Phys. 35, 3235-3241. Holland, V. F., and Lindenmeyer, P. H. (1965). J. Appl. Phys. 36, 3049-3056. Holland, V. F., Lindenmeyer, P. H., Trivedi, R . , and Amelinckx, S. (1965). Phys. Stat. Sol. 10, 543-569. Honjo, G., and Kitarnura, N. (1957). Acfa Cryst. 10, 533-534. Hoppe, W. (1956). 2. Krist. 107, 46-432. Hoppe, W. (1957). 2. Krist. 108, 335-340. Hoppe, W., and Baumgartner, F. (1957). 2. Krist. 108, 328-334. Hoppe, W., and Gassmann, J . (1968). Acta Cryst. B24, 97-107. Hoppe, W., Lenne, H. U., and Moranti, G. (1957). 2. Krist. 108, 321-327. Hoppe, W., and Rauch, R. (1960). 2. Krist. 115, 141-155. Hosernann, R., and Bagchi, S. N. (1962). Direct Analysis of Diffraction by Matter. North-Holland, Amsterdam. Hovmoller, S. (1992). Ultramicroscopy 41, 121-135. Howie, A., and Whelan, M. J. (1961). Proc. Roy. Soc. (London) A263, 217-237. Hsu, E. C. H., and Johnson, J. F. (1973). Mol. Cryst. Liq. Cryst. 27, 95-104.

192

DOUGLAS L. DORSET

Hu, H., and Dorset, D. L. (1989). Acta Cryst. B45, 283-290. Hu, H., and Dorset, D. L. (1990). Macromolecules 23, 4604-4607. Hu, H., Dorset, D. L., and Moss, B. (1989). Ultramicroscopy 27, 161-170. Hu,J. J., and Li, F. H. (1991). Ultramicroscopy 35, 339-350. Hui, S. W. (1981). Biophys. J. 34, 383-395. Hui, S. W. (1989). J. Electr. Microsc. Techn. 11, 286-297. Hui, S. W., Hausner, G. G., and Parsons, D. F. (1976). J. Phys. E. Scientific Instr. 9,69-72. Hui, S. W., and Parsons, D. F. (1974). Science 184,77-78. Hui, S. W., and Parsons, D. F. (1975). Science 190, 383-384. Hui, S. W., Parsons, D. F., and Cowden, M. (1974). Proc. Natl. Acad. Sci. USA 71, 5068-5072. Hui, S. W., and Strozewski, C. M. (1979). Biochim. Biophys. Acta 555, 417-425. Humphreys, C. J., and Bithell, E. G. (1992). In Electron Diffraction Techniques (J. M. Cowley, ed.), Vol. 1, pp. 75-151. Oxford University Press, Oxford. Hurst, H. (1948). Discuss. Faraday SOC. 3, 193-210. Hurst, H. (1950). J. Exp. Biol. 27,238-252. Hurst, H. (1952). J. Exp. Biol. 29, 30-53. Hybl, A., and Dorset, D. (1971). Acta Cryst. 827, 977-986. Ibers, J . A., and Vainshtein, B. K. (1959). Sov. Phys. Crystallogr. 4,601-605. Ishizuka, K. and Uyeda, N. (1977). Acta Cryst. A33, 740-749. Ishizuka, K., Miyazaki, M., and Uyeda, N. (1982). Acta Cryst. A38, 408-413. Isoda, S., Tsuji, M., Chara, M., Kawaguchi, A., and Katayama, K. (1983). Makromol. Chem., Rapid Commun. 4, 141-144. James, R. W. (1950). The OpticalPrinciples of Diffraction of X-Rays. Bell and Sons, London. Jap, B. K. (1988). J. Mol. Biol. 199,229-231. Jap, B. K. (1989). J. Mol. Biol. 205,407-419. Jap, B. K., Downing, K. H., and Walian, P. 619907. J. Struct. Biol. 103, 57-63. Jap, B. K., and Glaeser, R . M. (1980). Acta Cryst. A36, 57-67. Jap, B. K., Walian, P. J., and Gehring, K. (1991). Nature 350, 167-170. Jarrett, W. L., Mathias, L. J., Alamo, R. G., Mandelkern, L., and Dorset, D. L. (1992). Macromolecules 25,3468-3472. Jensen, L. H. (1970). J. Polym. Sci. C29, 47-63. Jones, C. A , , Russell, G. J., Petty, M. C., and Roberts, G. G. (1986). Phil. Mag. B54, L89-L94. Karle, I. L., Dragonette, K. S., and Brenner, S. A. (1965). Acta Cryst. 19, 713-716. Karle, J., and Brockway, L. 0. (1947). J. Chem. Phys. 19, 213-225. Karle, J., and Hauptman, H. (1956). Acta Cryst. 9,635-651. Karle, J., and Karle, I. L. (1966). Acra Cryst. 21, 849-859. Karpov, V. L. (1941). Zh. Fiz. Khim. 15,577-591. Kawaguchi, A.,Isoda, S., Petermann, J., and Katayama, K. (1984). Colloid Polym. Sci. 262, 429-434. Kawaguchi, A., Ohara, M., and Kobayashi, K. (1979). J. Macromol. Sci.-Phys. B16, 193-212. Kawaguchi, A., Tsuji, M., Moriguchi, S., Uemura, A., Isoda, S., Ohara, M., Petermann, J . , and Katayama, K. (1986). Bull. Inst. Chem. Res. Kyoto Univ . 64, 54-65. Keller, A. (1961). Phil. Mag. 6, 329-343. Kessel, M., Brennan, M. J., Trus, B. L., Bisher, M. E., and Steven, A. C. (1988).J. Mol. Biol. 203, 275-278. Khanna, R., and Welberry, T. R. (1987). Acta Cryst. A43, 718-727.

ELECTRON CRYSTALLOGRAPHY OF ORGANIC MOLECULES

193

Khare, R. S . , and Worthington, C. R . (1971). Biochim. Biophys. Acta 514,239-254. Kim, Y., Strauss, H. L., and Snyder, R. G. (1989a). J. Phys. Chem. 93, 7520-7526. Kim, Y., Strauss, H. L., and Snyder, R. G. (1989b). J. Phys. Chm. 93, 485-490. Kitaigorodskii, A. I . (1961). Organic Chemical Crystallography. Consultants Bureau, New York. Kitaigorodskii, A. I., Mnyukh, Y. V., and Nechitailo, N. A. (1958). Sov. Phys. Crystallogr. 3, 303-307. Kitaigorodsky, A. I. (1973). Molecular Crystals and Molecules, Academic Press, New York. Kleffel, B., Garavito, R. M., Baumeister, W., and Rosenbusch, J. P. (1985). EMBO J. 4, 1589- 1592. Klug, H.P., and Alexander, L. E. (1974). X-ray Diffraction Procedures for Polycrystalline and Amorphous Materials, 2nd ed. Wiley, New York. Knoll, W., Ibel, K., and Sackmann, E. (1981). Biochemistry 20, 6379-6383. Kobayashi, T., Fujiyoshi, Y., Iwatsu, F., and Uyeda, N. (1981). Acta Cryst. A37, 692-697. Kuwabara, S. (1959). J. Phys. SOC. Japan 14, 1205-1216. Ladd, M. F. C., and Palmer, R. A. (1980). In Theory and Practice of Direct Methods in Crystallography (M. F. C. Ladd and R. A. Palmer, eds.), pp. 93-150. Plenum, New York. Lando, J. B., and Sudiwala, R. V. (1990). Chem. Muter. 2, 594-599. Langs, D. A., and DeTitta., G. T. (1975). Acta Cryst. A31, S16. Lebedeff, A. (1931). Nature 128,491. Li, F. H. (1963). Acta Phys. Sinica 19, 735-740. Li, F. H. (1991). In Electron Crystallography of Organic Molecules (J. R. Fryer and D. L. Dorset, eds.), pp. 153-167. Kluwer, Dordrecht. Liau, W. B., and Boyd, R. H. (1990). Macromolecules 23, 1531-1539. Lipson, H., and Cochran, W. (1966). The Determination of Crystal Structures. Cornell University Press, Ithaca, New York. Lipson, S. G., and Lipson, H. (1969). Optical Physics. Cambridge University Press, Cambridge, England. Liu, Y. W., Fan, H. F., and Zheng, C. D. (1988). Acta Cryst. A U , 61-63. Liu, Y. W., Xiang, S . B., Fan, H. F., Tang, D., Li, F. H., Pan, Q . , Uyeda, N., and Fujiyoshi, Y. (1990). Acta Cryst. AM,459-463. Lobachev, A. N. (1954). Trud. Inst. Krist. Akad. Nauk SSSR 10,71-75. Lobachev, A. N., and Vainshtein, B. K. (1961). Sov. Phys. Crystallogr. 6 , 313-317. Losche, M., Rabe, J . , Fischer, A,, Rucha, U., Knoll, W., and Mohwald, H. (1984). Thin Solid Films 117,269-280. Liith, H . , Nyburg, S. C., Robinson, P. M., and Scott, H. G. (1974). Mol. Cryst. Liq. Cryst. 27, 337-357. Luzzati, V., Mariani, P., and Delacroix, H. (1988). Makromol. Chem. Macromol. Symp. 15, 1-17. McCourt, M. P., Strong, P., Pangborn, W. A., and Dorset, D. L. (1993). J. Lipid R e x , submitted. Mannella, C. A. (1984). Science 224, 165-166. Mannella, C. A., Colombini, M., and Frank, J. (1983). Proc. Natl. Acad. Sci. USA 80, 2243-2247. Martoncelli, M., Strauss, H. L., and Snyder, R. G. (1985a). J. Chem. Phys. 82, 2811-2824. Marconcelli, M., Strauss, H. L., and Snyder, R. G. (1985b). J. Phys. Chem. 89, 5260-5267. Mason, S. J . , and Zimmermann, H. J. (1960). Electric Circuits, Signals, and Systems. Wiley, New York. Mathesen, R. R., Jr., and Smith, P . (1985). Polymer 26, 288-292. Matricardi, V. M., Moretz, R. C., and Parson, D. F. (1972). Science 177,268-270.

194

DOUGLAS L. DORSET

Mazeau, K., Chanzy, H., and Winter, W. T., (1992). Amer. Chem. Soc. Div. Polym. Chem., Inc., Polymer Preprints 33(1), 244. Mazee, W. M. (1958). Amer. Chem. Soc., Div. Petro. Chem. Preprints 3(4), 35-47. Meille, S. V., Briickner, S., and Lando, J. B . (1989). Polymer 30, 786-792. Menter, J. W. (1956). Proc. Roy. SOC. (London) A236, 119-135. Menter, J. W., and Tabor, D. (1951). Proc. Roy. SOC. (London) A204, 514-524. Misell, D. L. (1978). Image Analysis, Enhancement, and Interpretation. North-Holland, Amsterdam. Mnyukh, Y. V. (1960). Zh. Strukt. Khim. 1, 370-388. Mo, Y. D., Cheng, T. Z., Fan, H. F., Li, J. Q.,Sha, B. D., Zheng, C. D., Li, F. H., and Zhao, 2. X. (1992). Supercond. Sci. Technol. 5, 69-72. Moodie, A. F. (1969). International Conference on Electron Diffraction and Crystal Defects, Melbourne, Australia, Paper ID-I . Moss, B., and Dorset, D. L. (1982a). Acta Cryst. A38, 207-211. Moss, B., and Dorset, D. L. (1982b). J. Polym. Sci.-Polym. Phys. Ed. 20, 1789-1804. Moss, B., and Dorset, D. L. (1983a). J. Macromol. Sci.-Phys. B22, 69-77. Moss, B., and Dorset, D. L. (1983b). Acta Cryst. A39, 609-615. Moss, B., Dorset, D. L., Wittmann, J. C., and Lotz, B. (1984). J. Polym. Sci.-Polym. Phys. Ed. 22, 1919-1929. Murata, Y., Fryer, J. R., and Baird, T. (1976). J. Microsc. (Oxford) 108,261-275. Nabedryk, E.,Garaviuto, R. M., and Breton, J. (1988). Biophys. J. 53, 671-676. Nagle, J. F., and Wiener, M. D. (1989). Biophys. J. 55, 309-325. Natta, E., Baccaredda, M., and Rigamonti, R. (1935). Gazz. Chim. Ital. 65, 182-198. Nyburg, S. C., and Potworowski, J. A. (1973). Acta Cryst. B29, 347-352. O’Keefe, M. A., Fryer, J. R., and Smith, D. J. (1983). Acta Cryst. A39, 838-847. Parsons, D. F. (1967). Canad. Cander Con$ 7, 193-246. Pascher, I., and Sundell, S. (1986). Biochim. Biophys. Acta 855,68-78. Pascher, I., Sundell, S., Harlos, K., and Eibl, H. (1987). Biochim. Biophys. Acta 896, 77-88. Pauptit, R. A., Schirmer, T., Jansonius, J . N., Rosenbusch, J. P., Parker, M. W., Tucker, A. D., Tsernoglou, D., Weiss, M. S., and Schulz, G. E. (1991). J. Struct. Biol. 107, 136-145. Pearson, R. H., and Pascher, I. (1979). Nature 281,499-501. Perez, S.,and Chanzy, H. (1989). J. Electr. Microsc. Techn. 11, 280-295. Perez, S.,Roux, M., Revol, J. F., andMarchessault, R. H. (1979). J. Mol. Biol. 129,113-133. Pertsin, A. J., and Kitaigorodsky, A. I. (1987). The Atom-Atom Potential Method. SpringerVerlag, Berlin. Peterson, I. R.,and Russell, G. J. (1984). Phil. Mag. 49, 463-473. Piesczek, W., Strobl, G. R., and Malzahn, K. (1974). Acta Cryst. B30, 1278-1288. Pinsker, Z.G. (1953). Electron Diffraction. Butterworth, London. Poulin-Dandurand, S . , Perez, S., Revol, J. F., and Brisse, F. (1979). Polymer 20, 419-426. Pradere, P., Revol, J. F., and Manley, R. St. J. (1988a). Macromolecules 21,2747-2751. Pradere, P., Revol, J. F., Nguyen, L., and Manley, R. St. J. (1988b). Ultramicroscopy 25, 69-80. Precht, D. (1979). Fett. Seif. Anstrichm. 81,227-233. Ramachandran, G. N., and Srinvasan, R. (1970). Fourier Methods in Crystallography, pp. 62-67. Wiley Interscience, New York. Reimer, L. (1975). In Physical Aspects of Electron Microscopy and Microbeam Analysis (B. M. Siege1 and D. R. Beaman, eds.), pp. 231-245. Wiley, New York. Reimer, L. (1984). Transmission Electron Microscopy. Springer-Verlag, Berlin. Remillard, B., and Brisse, F. (1982a). Acta Cryst. B38, 1220-1224. Remillard, B., and Brisse, F. (1982b). Polymer 23, 1960-1964.

ELECTRON CRYSTALLOGRAPHY OF ORGANlC MOLECULES

195

Revol, J. F. (1989). Int. J. Biol. Macromol. 11, 233-235. Revol, J. F., and Chanzy, H. (1986). Biopolymers 25, 1599-1601. Revol, J. F., and Manley, R. St. J. (1986). J. Muter. Sci. Lett. 5, 249-251. Revol, J. F., Chanzy, H. D., Deslandes, Y., and Marchessault, R. H. (1989). Polymer 30, 1973- 1 976. Rigamonti, R. (1936). Gazz. Chim. Ital66, 174-182. Roche, E., Chanzy, H., Boudeulle, M., Marchessault, R. H., and Sandararajan, P. (1978). Macromolecules 11, 86-94. Rosenbusch, J . P. (1974). J. Biol. Chem. 249, 8019-8229. Rosenbusch, J . P., (1990). J. Slruct. Biol. 104, 134-138. Sass, H. J . , Biildt, G., Beckermann, E., Zemlin, F., van Heel, M., Zeitler, E., Rosenbusch, J. P., Dorset, D. L., and Massalski, A. (1989). J. Mol. Biol. 209, 171-175. Saxton, W. 0. (1980). In Electron Microscopy at Molecular Dimensions (W. Baumeister and W. Vogell, eds.), pp. 245-255. Springer-Verlag, Berlin. Sayre, D. (1952). Acta Crysf. 5, 60-65. Schindler, H . , and Rosenbusch, J. P. (1978). Proc. Natl. Acad. Sci. USA 75, 3751-3755. Schindler, H., and Rosenbusch, J . P. (1981). Proc. Natl. Acad. Sci.. USA 78, 2302-2306. Shimamura, K., Karasz, F. E., Hirsch, J . A., and Chien, J. C. W. (1981). Makromol. Sci.Rapid Commun. 2, 473-480. Smith, D. J., and Fryer, J. R. (1981). Nature 291, 481-482. Snyder, R. G., Strauss, H. L., and Elliger, C. A. (1982). J. Phys. Chem. 86. 5145-5150. Snyder, R. G . , Goh, M. C., Srivatsavoy, V. J. P., Strauss, H. L., and Dorset, D. L. (1992). J. Phys. Chem. 96, 10008-10019. Spence, J. C. H. (1980). Experimental High-Resolution Electron Microscopy. Clarendon, Oxford. Stanley, E. (1986). Acra Cryst. A42, 297-299. Steven, A. C., ten Heggler, B., Miiller, R., Kistler, J., and Rosenbusch, J. P. (1977). J. Cell Bid. 72, 292-301. Stewart, T. P., Hui, S. W., Portis, A. R., Jr., and Papahadjopoulos, D. (1979). Biochim. Biophys. Acta 556, 1-6. Stout, G. H., and Jensen, L. H. (1%8). X-ray Structure Determination: A Practical Guide. Macmillan, New York. Sturkey, L. (1962). Proc. Phys. SOC.(London) 80, 321-354. Sugiyama, J., Harada, H., and Saiki, H . (1987). Int. J . Biol. Macromol. 9 , 122-130. Tadokoro, H. (1979). Structure of Crystalline Polymers. Wiley Interscience, New York. Takamizawa, K., Ogawa, Y., and Oyama, T. (1982). Polym. J. 14, 441-456. Tanaka, K. (1938). Mem. Kyoto Univ. A21, 85-88. Tang, D., and ti, F. H. (1988). Ultramicroscopy 25, 61-68. Tatarinova, L. I., and Vainshtein, B. K. (1962). Visokomelek. Soed. 4, 261-269. Taylor, C. A., and Lipson, H. (1964). Optimal Transforms. Cornell University Press, Ithaca, New York. Taylor, K. A., and Glaeser, R. M. (1974). Science 186, 1036-1037. Teare, P. W. (1959). Acta Crysf. 12, 294-300. Thakur, M., and Lando, J. B. (1983). Macromolecules 16, 143-146. Thiessen, P. A., and Schoon, T. (1937). Z. Phys. Chem. 836, 216-231. Thomas, E. L., and Ast, D. G. (1974). Polymer 15, 37-41. Thomas, E. L., Sass, S. L., and Kramer, E. J. (1974). J . Polym. Sci. Polym. Phys. Ed. 12, 1015-1022. Thomson, G. P., and Murison, C. A. (1933). Nafure 131, 237. Tivol, W . F., Chang, B. W. B., and Parsons, D. F. (1982). Ultramicroscopy 9, 117-130.

196

DOUGLAS L. DORSET

Tivol, W. F., Dorset, D. L., McCourt, M. P., and Turner, J . N. (1933). Microsc. SOC. Amer. Bull. 23(1), 91-98. Toyoshima, C., and Unwin, P. N. T. (1988). Ultramicroscopy 25, 279-292. Tsuji, M., Isoda, S., Ohara, M., Kawaguchi, A., and Katayama, K. (1982). Polymer 23, 1568-1574. Tsuji, M., Ohara, M., Isoda, S., Kawaguchi, A,, and Katayama, K. (1989). Phil. Mag. B54, 393-403. Tsuji, M., Roy, S. K., and Manley, R. St. J. (1984). Polymer 25, 1573-1576. Turner, J. N., Barnard, D. P., and McCauley, P. (1991). In Electron Crystallography of Organic Molecules (J. R. Fryer and D. L. Dorset, eds.), pp. 55-62. Kluwer, Dordrecht. Turner, P. S., and Cowley, J. M. (1969). Acta Cryst. A25, 475-481. Udalova, V. V., and Pinsker, 2. G. (1964). Sov. Phys. Crystallogr. 8, 433-440. Ueda, Y., and Ashida, M. (1980). J. Electron Microsc. 29, 38-44. Uemura, A., Tsuji, M., Kawaguchi, A., and Katayama, K. (1988). J. Mater. Sci. 23, 1506- 1509. Unwin, P. N. T., and Henderson, R. (1975). J. Mol. Biol. 94, 425-440. Uyeda, N., Kobayashi, T., Ishizuka, K., and Fujiyoshi. Y. (1978-1979). Chem. Srripta 14, 47-61. Uyeda, N., Kobayashi, T., Ishizuka, K., and Fujiyoshi, Y. (1980). Nature 285, 95-97. Uyeda, N.,Kobayashi, T., Suito, E., Harada, Y.,and Watanabe, M. (1972). J . Appl. Phys. 43, 5181-5189. Uyeda, R. (1974). J. Appl. Cryst. 7, 1-18. Vainshtein, B. K. (1955). Zh. Fiz. Khim. 29, 327-344. Vainshtein, B. K. (1956a). Sov. Phys. Crystallogr. I , 394-398. Vainshtein, B. K. (1956b). Sov. Phys. Crystallogr. 1, 15-21. Vainshtein, B. K . (1956~).Sov. Phys. Crystallogr. 1, 117-122. Vainshtein, B. K. (1957). Sov. Phys. Crystallogr. 2, 334-345. Vainshtein, B. K. (1964a). Structure Analysis by Electron Dtffraction. Pergamon, Oxford. Vainshtein, B. K. (1964b). In Advances in Structure Research by Dlffraction Methods (R. Brill, ed.), Vol. 1 , pp. 24-54. Wiley Interscience, New York. Vainshtein, B. K. (1978). Adv. Optical Electron Microsc. 7, 281-377. Vainshtein, B. K., D’yakon, I. A. and Ablov, A. V. (1971). Sov. Phys. Doklady 15, 645-647. Vainshtein, B. K., and Ibers, J. A. (1958). Sov. Phys. Cystallogr. 3, 417-420. Vainshtein, B. K., and Lobachev, A. N. (1956). Sov. Phys. Crystallogr. 1, 370-371. Vainshtein, B. K., Lobachev, A. N., and Stasova, M. M. (1958). Sov. Phys. Crystallogr. 3, 452-459. Vainshtein, B. K., and Pinsker, 2. G. (1950). Dokl. Akad. Nauk SSSR 72, 53-56. Vainshtein, B. K., and Tatarinova, L. I. (1967). Sov. Phys. Crystallogr. 11, 494-498. Vainshtein, B. K., Zvyagin, B. B., and Avilov, A. S. (1992). In Electron Diffraction Techniques (J. M. Cowley, ed.). Vol. 1 , pp. 216-312. Oxford University Press, Oxford. Van Heel, M., and Keegstra, W. (1981). Ultramicroscopy 7, 113-130. Voigt-Martin, I . G., Krug, H., and Van Dyck, D. (1990). J. Phys. (Paris) 51, 2347-2371. Wang, B. C. (1985). Adv. Enzymology 115, 90-113. Welberry, T. R. (1985). Rept. Progr. Phys. 48, 1543-1595. Welberry, T. R., and Jones, R. D. G. (1980). J. Appl. Cryst. 13, 244-251. Wellinghoff, S.,Rybnikar, F., and Baer, E. (1974). J. Macromol. Sci.-Phys. B10, 1-39. White, J. W., Dorset, D. L., Epperson, J. E., and Snyder, R. G. (1990). Chem. Phys. Lett. 166, 560-564. Wilson, A. J. C. (1942). Nature 150, 151-152.

ELECTRON CRYSTALLOGRAPHY O F ORGANIC MOLECULES

197

Wittmann, J. C., Hodge, A. M., and Lotz, B. (1983). J. Polym. Sci.-Polym. Phys. Ed. 21, 2495-2509. Wittmann, J. C., and Lotz, B. (1981a). J. Polym. Sci.-Polym. Phys. Ed. 19, 1837-1851. Wittmann, J. C., and Lotz, B. (1981b). J. Polym. Sci.-Polym. Phys. Ed. 19, 1853-1864. Wittmann. J. C., and Lotz, B. (1989). Polymer 30, 27-34. Wittmann, J. C., and Lotz, B. (1990). Progr. Polym. Sci. 15, 909-948. Wittmann, J. C., and Lotz, B. (1991). In Electric Crystallography of Organic Molecules ( J . R. Fryer and D. L. Dorset, eds.), pp. 241-254. Kluwer, Dordrecht. Wittmann, J. C., and Manley, R. St. J. (1977). J. Polym. Sci-Polym. Phys. Ed. 15, 1089- 1100. Wittman,, J . C., and Manley, R. St. J. (1978). J. Polym. Sci.-Polym. Phys. Ed. 16, 1891-1895. Woodward, A. E. (1989). Atlas of Polymer Morphology. Hanser, Munich. Wooster, W. A. (1964). Acru Cryst. 17. 878-882. Worthington, C. R., King, G. I., and Mclntosh, T. J. (1973). Biophys. J. 13, 480-494. Wunderlich, B. (1973). Macromolecular Physics, vol. 1. Crystal Structure, Morphology, Defects. Academic Press, New York. Yao, J. X. (1981). Acta Cryst. A37, 642-644. Zeitler, E. (ed.) (1982). Cryomicroscopy and Rodiution Dumage. North-Holland. Amsterdam. Zemlin, F., Reuber, E.,Beckemann, E., Zeitler, E., and Dorset, D. L. (1985). Science 229, 46 1-462. Zhang, W. P., and Dorset, D. L. (1989a). J . Polym. Sci. Polym. Phys. Ed. 27, 1433-1447. Zhang, W. P., and Dorset, D. L. (1989b). Proc. 47th. Ann. Meeting Electr. Microsc. Soc. Americu (G. W. Bailey, ed.), pp. 702-703. San Francisco Press, San Francisco. Zhang, W. P., and Dorset, D. L. (1990a). Macromolecules 23, 4322-4326. Zhang, W. P., and Dorset, D. L. (1990b). J. Polym. Sci. B. Polym. Phys. 28, 1223-1232. Zhang, W. P., Kuo, K. H., Dorset, D. L., Hou, Y. F., and Ni, J. 2. (1989). J . Electr. Microsc. Techn. 11. 326-332. Zhang, W. P., and Thomas, E. I.. (1991). Polym. Commun. 32, 482-485. Zvyagin, B. B. (1957). Sov. Phys. Crystallogr. 2, 388-394. Zvyagin, B. B. (1967). Electron Diffraction Analysis of Clay Mineral Structures, pp. 247-280. Plenum, New York. Zvyagin, B. B. (1993). Microsc. Soc. America Bull., in press. Zvyagin, B. B., and Mischenko, K. S. (1961). Sov. Phys. Crystallogr. 5, 575-579. Zvyagin, B. B., and Mischenko, K. S. (1963). Sov. Phys. Crystallogr. 7, 502-505.

This Page Intentionally Left Blank

.

ADVANCES IN ELECTRONICS AND ELECTRON PHYSICS VOL . 88

Fractal Signal Analysis Using Mathematical Morphology PETROS MARAGOS School of Electrical Engineering. Georgia Institute of Technology. Atlanta. Georgia

I . Introduction . . . . . . . . . . . . . . . . . I1. Morphological Signal Transformations . . . . . . . . A . Set Operations . . . . . . . . . . . . . . . B. Function Operations . . . . . . . . . . . . . 111. Fractal Dimensions . . . . . . . . . . . . . . . A . Hausdorff Dimension . . . . . . . . . . . . . B . Similarity Dimension . . . . . . . . . . . . . C . Minkowski-Bouligand Dimension . . . . . . . . . D Box-Counting Dimension . . . . . . . . . . . . E . Entropy Dimension . . . . . . . . . . . . . . F . Relations among Dimensions . . . . . . . . . . IV . Fractal Signals . . . . . . . . . . . . . . . . A . Weierstrass Function . . . . . . . . . . . . . B. Fractal Interpolation Functions . . . . . . . . . . C . Fractional Brownian Motion . . . . . . . . . . . V . Measuring the Fractal Dimension of ID SIGNALS. . . . . A . 2D Covers via 2D Set Operations . . . . . . . . . B . 2D Covers via ID Function Operations . . . . . . . C . Algorithm for Discrete-Time Signals . . . . . . . . D . Application to Speech Signals . . . . . . . . . . VI . Measuring the Fractal Dimension of 2D SIGNALS. . . . . A . 3D Covers via 3D Set Operations . . . . . . . . . B . 3D Covers via 2D Function Operations . . . . . . . C . Discrete Algorithm . . . . . . . . . . . . . . VII . Modeling Fractal Images Using Iterated Function Systems . . A . Modeling Fractals with Collages . . . . . . . . . . B. Finding the Collage Parameters via Morphological Skeletons VIII . Conclusions . . . . . . . . . . . . . . . . . Acknowledgments . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . .

.

. . . . . . . . . .

. . . . .

. . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . . . . . . . . . . . . . . . . .

199 201 202 202 203 204 206 206 211 212 212 213 213 214 216 218 218 218 222 226 230 230 231 235 231 231 239 242 243 243

I . INTRODUCTION

Natural scenes contain many classes of objects that have a high degree of geometrical complexity . Examples include clouds. mountains. trees. and coastlines . In addition. many nonlinear dynamical systems give rise to limit sets whose images exhibit a high degree of geometrical complexity . Mandelbrot (1982) has demonstrated in his pioneering work that a large 199

.

Copyright 0 1994 by Academic Press Inc . All rights of reproduction in any form reserved. ISBN 0-12-014730-0

200

PETROS MARAGOS

class of mathematical sets, calledfractals, can model well many such image classes. Fractal images have become aesthetically attractive through synthesis via computer graphics (Mandelbrot, 1982; Voss, 1988; Barnsley, 1988). Although the fractal images are the most popularized class of fractals, there are also numerous natural processes described by time-series measurements (e.g., l/f noises, econometric and demographic data, pitch variations in music signals) that are fractals (Mandelbrot, 1982; Voss, 1988). The one-dimensional (abbreviated as 1 D)’ signals representing these measurements are fractals in the sense that their graph is a fractal set. In addition, the geometrical complexity of fractal surfaces of physical objects is often inherited in the 2D image intensity signals emanating from such objects (Pentland, 1984). Thus, analyzing and modeling fractal signals is of great interest both from a scientific and an engineering viewpoint. Perhaps the most important characteristic of fractals is that they have similar structure at multiple scales. Thus, in this chapter we address two problems related to this multiscale structure of fractal signals. The first is an analysis problem and deals with the estimation of the fractal dimension. This is an important parameter measuring the degree of fragmentation of fractal signals and is useful for their description and classification. Intuitively, it measures the degree of their fragmentation or irregularity over multiple scales. It makes meaningful the measurement of metric aspects such as the length of fractal curves and the area of surfaces. The second problem deals with modeling fractal images by collages, i.e., nonlinear combinations of down-scaled, rotated, and shifted versions of the original image. The unifying theme in the approaches presented herein to both problems is the extensive use of morphological filters for their efficiency as well as their ability to rigously extract size information from a signal at multiple scales. These morphological filters are based on elementary operators of morphological signal analysis (Serra, 1982; Maragos and Schafer, 1987, 1990). This chapter begins by providing in Section I1 the definitions of some basic morphological transformations for sets and signals, i .e., the erosion, dilation, and opening operations, which are required for the analysis in this chapter. This is followed by a brief survey of the theory of fractal dimensions. There is a proliferation of fractal dimensions, all of which are more or less capable of measuring the degree of fragmentation of a signal’s graph. In Section I11 we review their definitions and interrelationships. Emphasis is given on the Minkowski-Bouligand dimension, whose analysis is done using morphological operations. There are numerous classes of

’

In this chapter the notation nD will mean “n-dimensional,” where n = 1 , 2 , 3 , ... . An nD signal will imply a function with n independent variables, whereas an nD set will mean a set of points in the Euclidean space R”.

FRACTAL SIGNAL ANALYSIS USING MATHEMATICAL MORPHOLOGY

201

fractal signals. In Section IV we review three classes of parametric fractal signals and related algorithms for their synthesis. The performance of the presented morphological method for measuring fractal dimension is tested by applying it to the above synthetic fractal signals. Section V focuses on the covering methods, a class of general and efficient approaches to compute the fractal dimension of arbitrary fractal signals. It essentially reviews the work of Maragos and Sun (1991) where a general framework was presented, based on multiscale morphological erosions and dilations with varying structuring elements, that provides the theoretical support for and underlies many of the digital implementations of covering methods, e.g., in Dubuc et al. (1989) and the 1D analogs of the methods in Peleg et al. (1984), Stein (1987), and Peli et al. (1989). We shall refer to these unified algorithms as the morphological covering method. This approach originally attempts to cover the graph of a 1D signal with 2D sets at multiple scales. Thus, for an N-sample N-level ID digital signal, the set-cover methods require a O(N2)computational complexity at each scale. However, covering the signal’s graph with properly chosen 1D functions via morphological filtering yields identical results and involves 1D processing of the signal. Hence, the morphological filtering approach reduces the original set-cover complexity from quadratic to linear, since for an N-sample 1D signal the function-cover method has complexity O ( N )at each scale. A morphological covering algorithm for estimating the fractal dimension of discrete-time signals is also presented and applied to three classes of fractal signals. The morphological covering method applies to arbitrary signals. In Section V.D we briefly describe (from Maragos , 1991) its application to measuring the short-time fractal dimension of speech signals. Section VI extends the morphological covering approach to finding the fractal dimension of 2D signals and provides a related discrete algorithm. Section VII deals with modeling fractal binary images using collages. The theory of collages is first reviewed from Barnsley (1988) and then an approach is presented from Libeskind-Hadas and Maragos (1987) to finding a good collage based on morphological skeletonization. Finally, Section VIII concludes with some suggestions for future work. 11. MORPHOLOGICAL SIGNAL TRANSFORMATIONS

In this section we review the definitions of the elementary morphological transformations for sets and signals. More details, the properties, and many applications of these operators can be found in Serra (1982), Sternberg (1986), Maragos and Schafer (1987, 1990), Haralick et al. (1987), Heijmans and Ronse (1988), and Serra and Vincent (1992).

202

PETROS MARAGOS

A . Set Operations Consider sets X , B in the Euclidean space Rd or the discrete space Z d , d = 1,2, 3, ..., where R is the set of reals and Z is the set of all integers. Let

x +2

4 (x

* 2 : x E XI

(1)

denote the translate of X by the vector + z , and let B 4 { - b : b E B ) be the reflection of B . The fundamental morphological operators for sets are the dilation 0 and erosion 0 of X by B , which are defined as follows:

x g B 4 U X + b = ( z : @ + Z ) n x # 01

(2)

bsB

X

0B 4

n X - b = { z :B +

Z,

E

X)

(3)

bsB

In applications where X is an input set to some system, the second set B is usually compact and has a simple shape and small size; B is then called a structuring element. Thus, the output of the dilation operator is the set of translation points such that the translate of the reflection of B has a nonempty intersection with the input set. Similarly, the output of the erosion operator is the set of translation points such that the translated structuring element is contained in the input set. Another fundamental operator is the opening 0 of X by B :

X O B P ( X eB ) 0 B Note that X

0B

E

(4)

X for all X and B , because X O B =

u

B+z

(5)

b+zsX

To visualize the geometrical behavior of these operators, it is helpful to consider a 2D set X representing a binary image and the structuring element B being a disk centered at the origin. Then the erosion shrinks the set X, whereas dilation expands X . The opening suppresses the sharp capes and cuts the narrow isthmuses of X , inside which B cannot fit. Thus the opening by a disk performs a nonlinear smoothing of the image contour. Clearly, if we vary the structuring element B , then its shape and size will determine the nature and the degree of shrinking, expansion, or smoothing during the above morphological operations. B. Function Operations Consider signals f and g whose domain is the set E, equal either to the Euclidean space R d , d = 1,2, 3, ..., or the discrete space Z d , and whose

FRACTAL SIGNAL ANALYSIS USING MATHEMATICAL MORPHOLOGY

range is a subset of R U ( - 0 0 , of its domain:

The support off is the following subset

00).

Spt(f) g

203

(X E

E:f(x) > -00)

(6)

The dilation 0 and 0 of the signal f by the (structuring) signal g are defined as the signal operations

( S O g)W

SUP I f ( Y ) + g(x - Y ) I y

E

y

E

( f 0g)(x)

(7)

t;+l

inf l f ( Y ) -

-

x)l

(8)

G+X

where G

=

Spt(g)

(9)

The structuring function g usually has a compact support G and simple shape. A special, but quite useful in applications, case results when g is a flat function, i.e., assumes only two values on E. Specficially if

then the general dilation and erosion o f f by g reduce to the following moving local maxima and minima:

111. FRACTAL DIMENSIONS

In this section we review several fractal dimensions,2 which are more or less capable of quantifying the degree of fragmentation of curves and surfaces. More general and detailed discussions on these topics can be found in the books by Mandelbrot (1982), Barnsley (1988), and Falconer (1990). Unless otherwise stated, we shall assume in this section that F is a nonempty compact subset of the Euclidean space Rd, d = 1 , 2 , 3 , . ..

* All the fractal dimensions discussed in this chapter are related only to the geometry of a set and its metric aspects. For fractals that are sets of attracting points of chaotic dynamical systems, Farmer et al. (1983) discuss other types of dimensions that depend on the probability mass of parts of the set; such dimensions are not discussed in this chapter.

204

PETROS MARAGOS

A . Hausdorff Dimension

Let E 2 0 be the scale parameter. An &-cover of F is any countable set collection X ( E )= [ X i: i = 1,2, ...) such that F C UiXi and 0 < diam(Xi) 5 E for all i , where diam(Xi) is the largest distance between any two points of Xi.The 8-dimensional Hausdorff measure of F is defined as

where cg = y(8)/2* is a normalizing constant and

. ~ that, if d = 1,2,3, . ..,then y(d).cd where r() is the gamma f ~ n c t i o nNote is the volume of the d-dimensional ball of radius E . There is a critical real number DH 1 0 such that

This critical DH is the Hausdorff dimension of F and is equal to

DH(F) = inf (6 : X,(F) = 0 )

(16)

This dimension was introduced by Hausdorff (1 918) and further analyzed by Besicovitch (1934) and Besicovitch and Ursell(l937). Mandelbrot (1982) defines formally the fractal dimension of F as equal to DH. Further, he calls a set fractal if DH strictly exceeds its topological dimension DT. Hence set F is fractal

e,

Hausdorff dim D,(F) > topological dim D,(F)

The topological dimension is always an integer, and for a continuous curve represented by a function, DT is the number of independent variables of this function. Whenever the set F is implied, we will drop it as argument of the various dimensions. General categories of fractal sets in R3 are:

DT = 0 < DH

I1

* F = fractal dust

DT = 1 < DH

I2

* F = fractal curve

DT = 2 < DH

I3

F = fractal surface

The gamma function is defined as T(p) = j;xP-'exp(-x)dx, 0 < p < 00. Note that I'(1/2) = fi, and T(a + n) = (a t n - I)(a + n - 2) ... &(a) for n = 1,2, 3, ... and O C U l l .

FRACTAL SIGNAL ANALYSIS USING MATHEMATICAL MORPHOLOGY

205

Example 1. Cantor set: Define a set sequence IC,,]~=,through the following recursion: C, = [0, 11 = ( r E R:O

Ir I1)

(17)

c, = [O, f l u rS, 11 c,. =. [O, .$1 u I t , 41 u r8, +I u 16,11 .. ..

(18) (19)

..

C,, = (fC,,-J U [ ( f C n - J+

*I,

n = 1 , 2 , 3 , ...

(20)

where, given an arbitrary set X E Rd,d = 1,2, 3, .. . , the set

rx

( r x : xE XI

(21)

is its scaling (i.e., positive homothetic) by the real number r > 0. Thus, each member of the sequence C,, is equal to the union of two scalings of by 1/3, one of which is also translated by the vector 213. The sequence (C,,)is a monotonically decreasing sequence of closed sets whose limit

is the Cantor set. At each n, C , consists of 2" intervals of length En

= (f)"

(23)

The Hausdorff measure can be found as

X, = lim c*H(E,, 6) n-m

where

In general, the tightest covers X(E,) will be when, for each i , diam(Xi) = (1/3)"' for some integer ni 2 n. If 2 > 3', then the tightest cover occurs if ni = n for all i, because using diam(Xi) = (1/3)"' with n' > n for some i yields

Therefore, H(E,,, 6)

and hence

=

(2/3*)"

206

PETROS MARAGOS

Further, we have

Since if 6 > log(2)/log(3) if 6 c log(2)/log(3)

0, 00,

the Hausdorff dimension of the Cantor set is

B. Similarity Dimension If F can be decomposed into the union of n disjoint of just-touching copies of itself that are (possibly translated, rotated, and) scaled by ratios T i , i = 1, ..., n, then the similarity dimension (Mandelbrot, 1982) is the solution D , of the equation n

C+=1 i= 1

If all ratios ri are equal to r = r i , then

In several cases we have D , = DH (Hutchinson, 1981; Mandelbrot, 1982).

Example 2. Consider the Cantor set C defined as the limit of the set sequence C, in (20). Since each Cn is the union of N = 2 copies of C n - l scaled by r = 1/3, the limit C will be the union of two copies of itself scaled by 1/3. Hence

C. Minkowski-Bouligand Dimension

1. Sets in R 3 This dimension is based conceptually on an idea by Minkowski (1901, 1903) of finding the area of irregular surfaces or length of irregular curves F in R 3 . Specifically, dilate F with spheres of radius E by forming the union of

207

FRACTAL SIGNAL ANALYSIS USING MATHEMATICAL MORPHOLOGY

these spheres centered at all points of F and thus create the set

F,

=

u ( ( ~+b

Z) E

R 3 : llbll

II],

E 2

0

(28)

ZEF

where, for d = 1 , 2 , 3 , ..., 11 * I( is the Euclidean norm

F, is called a Minkowski cover.4 Then find the volume vol(FE) of the dilated set at all scales E , and set the volume, area, and length of the original set F as equal to5 vol(F) = lim vol(FE) E+O

area(F) = Elim vol(F,)/2& -O

(30)

len(F) = lim vol(F,)/m2 E-.O

For d = 1 , 2 , 3 , . .., it follows from (14) that the volume of a d-dimensional ball of radius E is y(d)cd = ( ( ~ bE)R d :llbll I1)

(31)

Now the Minkowski &content of F is defined as &content of F f lim &-.O

Example 3.

v o w y(3 - 6)&3-*

Square: If S is the square

s=(I

J

then VOl(s,) = 212&+ 2R1&2+ 4?TE3/3 and 6 &content

2

1

length =

m

area = I 2

3 volume

=

0

Thus, in general, for any set F there is a critical number DM such that &content of F =

00,

0,

i f 6 < D, if 6 > D M

(33)

Bouligand (1928) and Mandelbrot (1982) attribute this cover construction also to Cantor. where morphological dilations are used in stereology.

’ Serra (1982) also has a related discussion

208

PETROS MARAGOS

Bouligand (1928) extended these ideas to cases where DM is not only integer but also fractional. Hence, the Minkowski-Bouligand dimension is defined as

=

3 - A[vol(F,)]

(35 )

where we define

A(f) A sup

I

p: lim - = x-0

xp

as the infinitesimal order of a functionf(x), around x Lemma 4.

= 0.

The infinitesimal order of a function f(x) can be obtained by

Proof. If we denote

then

A Note that p

If(x)I

E

=

sup(p:p E PI

P if and only if for all E > 0 there exists a 6 > 0 such that

s ~ 1 x and 1 ~ hence

log(If(x)O

1x1 < 8

Plog(Ix0 + log(&),

This implies

Thus in the limit 1x1 + 0 we obtain

Since also p E P implies that p - E E P for all implies that

which completes the proof of (37). (Q.E.D.)

E

> 0, the above analysis

FRACTAL SIGNAL ANALYSIS USING MATHEMATICAL MORPHOLOGY

209

From the above lemma it follows that

which implies that log[vol(F&)/E3] log(l/&)

DM(F)= lim e-0

It is also possible to replace the limit i.e.,

-

&(F) = lim n

m

E

-+

(39)

0 with the limit of a sequence;

log [VOl(F&")/E~] log(1/en)

where [E,I;=, is a decreasing sequence of scales such that E, = prn for all n, for some 0 < r < 1 and p > 0. The intuitive meaning of the dimension D = DM is that vol(F,) = C I E X , area(Fe) = c , E ~ - ~ , as E len(F,) = c ~ E ' - ~ ,

+

0

(41)

where cl, c2, c3are proportionality constants. Thus if F is a curve in R3and

D > 1, then its length is infinite. 2. Sets in R2 To find the area and length of a compact set F E R2 we can create a 2D Minkowski cover

F,

=

u [(Eb + z)

E

RZ:llbll

I1)

Z€F

(42)

by dilating F with disks of radius E , find the area of the dilated set at all scales E , and set the area and length of the original set F as equal to area(F)

=

lim area(F,) E-0

len(F)

=

lim area(Fe)/2e E-0

Then the Minkowski-Bouligand dimension of F is equal to

(43)

210

PETROS MARAGOS

Exumple 5. Linear Segment: Consider the 1D set

s=

1

Then the Minkowski cover area is area(S,) = 21.5

+ ?I&'

Hence

&(S)

=2

- A[area(S,)]

=

2

-1=1

3. Sets in R

To find the length of a compact set F G R we can create a 1D Minkowski cover F, = U ( ( ~+b Z) E R :-1 Ib 5 11 (46) teF

by dilating F with intervals as equal to

[ - E , E],

and set the length of the original set F

lenQ = lim len(F,)

(47)

E-0

Then the Minkowski-Bouligand dimension of F is equal to

&(F) 4 1 - A[len(F,)] = lim e-0

log[len(F,)/e] lOg(l/E)

(49)

Example 6. Consider the Cantor set C, which is the limit of the set sequence [C,] defined in (20). Since C,,, E C, for all n and C = n, C , ,

Dilating each C, with an interval [ - E , , dilated sets (ck)&n

=

('n-I)E,

E,],

where 2.5, = (1/3)", creates the

-

Hence the ID Minkowski cover of C at scale E, has length

which implies that

(51)

FRACTAL SIGNAL ANALYSIS USING MATHEMATICAL MORPHOLOGY

21 1

D. Box-Counting Dimension For compact planar set F 5 R2 let us partition the plane with a grid of square boxes of side E and count the number N ( E )of boxes that intersect that set F. Then, if we replace the Minkowski cover area in (45) with the box cover area (54) Aboxc(F, E ) E2N(&)

'

we obtain the box dimension (Bouligand, 1928)

Lemma 7 . For any compact set F C R2, the Minkowski cover area and the box cover area have the same infinitesimal order.

Proof. Every disk of radius 2&in the Minkowski cover of F will contain as subset the grid box that contains the coresponding disk center. Hence Abo,dF, d 5 a r e a ( F d

(57)

Also, Bouligand (1928) has shown that area(F,) 2 r2 area(FE) for 0 Ir I1, which implies that

area(F,) 2 $ area(F2,)

(58)

In addition, every disk of radius E in the Minkowski cover of F is a subset of the union of the box that contains the disk center and its eight neighbors; hence (59) area(FE) I9Aboxc(F,E ) The three above inequalities imply that area(F,) 9

IAbo,.c(F, E ) I4

area(F,)

Taking logarithms on all sides of this inequality, dividing by log(&),and taking the limit as E -, 0 yields

which implies that A[Aboxc(F,

=

This completes the proof of the lemma.

A[area(Fe)l (Q.E.D.)

(62)

212

PETROS MARAGOS

As a direct corollary of the above lemma, we see that DB = D , for all planar sets F. The definitions and results in this section can also be extended to compact sets F c Rd of any dimensionality d = 1,2,3, ... . For d = 1 the boxes will become intervals of length e , whereas for d = 3, the boxes will become cubes of side e. Thus, in general,

DB = D ,

(63)

E. Entropy Dimension The entropy dimension (Kolmogorov and Tihomirov, 1959) of a compact set F C Rd is defined as

where Nrnin(e)is the smallest number of d-dimensional balls with radii E required to cover F. (It is also called the "capacity" dimension in Farmer et al. (1983).) In Barnsley (1988) and in Falconer (1990) it is shown that

DE = DB (65) Example 8. Consider the Cantor set C , which is the limit of the set 1 defined in (20). Each set C,, consists of 2" intervals of length sequence (C,, 2&, = (1/3)". For each scale E , , the smallest cover of C will be the set C , , which consists of Nrnin(e)= 2" intervals of length 2.5,. Hence

F. Relations among Dimensions For each compact subset of Rd, the dimensions discussed in the previous sections satisfy the general relationships 0 5 D , I DH

5

D , = DB = DE 5 d

DH 5 Ds

(67)

In general, DH # DM (Mandelbrot, 1985; McMullen, 1984; Falconer, 1990). However, in this chapter we focus on the Minkowski-Bouligand dimension D,, which we shall henceforth call fractal dimension D , because; (1) it is clearly related to DH, and hence able to quantify the fractal aspects of a signal, (2) it concides (in the continuous case) with DH in many cases of practical interest; (3) it is much easier to compute than DH; (4) it is more robust to compute than DB for discrete-variable signals.

FRACTAL SIGNAL ANALYSIS USING MATHEMATICAL MORPHOLOGY

21 3

Although DB = D M in the continuous case, they correspond to two different algorithms (with different performances) in the discrete case. In general, D M can be more robustly estimated than D,, which suffers from uncertainties due to the grid translation or its spacing. This is further explained in Section V.C. In conclusion, a practical algorithm to estimate the dimension D M is from the slope of the following approximately linear relation in log(l/&); i.e., for sets F c R3: l o g [ y ] = DM * log(:)

+ constant,

as E

+ constant,

as

+

0

(68)

0

(69)

and for planar sets F E R 2 : log[

7 1

= D,

log(:)

E

3

IV. FRACTAL SIGNALS A d-dimensional signal represented by a functionf: Rd -, R is called fractal if its graph Gr(f ) { ( x ,y ) E Rd x R :y = f ( x ) ) (70) is a fractal set in Rd+'. Further, i f f is continuous, then its graph is a continuous curve with topological dimension equal to d . Hence

f is continuous * d IDH[Gr(f ) ] IDM[[Gr(f

+

(71) In this section we briefly describe three classes of parametric fractal test signals. These are the deterministic Weierstrass functions (WCFs) (Hardy, 1916; Mandelbrot, 1982; Berry and Lewis, 1980), the deterministic fractal interpolation functions (FIFs) (Barnsley, 1986: Barnsley, 1988; Hutchinson, 1981), and the random functions of fractional Brownian motion (FBM) (Mandelbrot and van Ness, 1968; Mandelbrot, 1982). These factals have been used in a variety of applications. For example, there are many natural phenomena that can be modeled using such parametric fractals (Mandelbrot, 1982). In addition, the FBM and FIFs have proven to be valuable in computer synthesis of images of natural scenes (Voss, 1988; Barnsley, 1988). A . Weierstrass Function The Weierstrass cosine function (WCF) is defined as

c y-kHcos(2nykt) m

W,(t)

=

k=O

)] Id

1

214

PETROS MARAGOS

with real positive parameters H and y which, for convergence of the above infinite series, should be in the ranges O
y > l

If y is integer, then the WCF is periodic with period one. This function is continuous, but nowhere differentiable because its derivative is given by the infinite series

which generally diverges since /?> 1. The fractal dimension of the WCF is D=2-H. In our computer experiments, we synthesized discrete-time signals from WCFs by sampling t E [0, 1 ) at N + 1 equidistant points, using a fixed y = 5 , and truncating the infinite series so that the summation is done only for 0 Ik Ik,,,, where k,,, was determined by requiring 2nyk I so that the cosine’s argument does not exceed the computer’s doubleprecision. Figure l a shows three sampled WCFs whose fragmentation increases with their dimension D . B. Fractal Interpolation Functions

The basic ideas in the theory of fractal interpolation functions were developed by Hutchinson (1981) and Barnsley (1986). Given is a set of data points ((xk,y k ) E R2; k = 0 , 1, 2 , , , ,,K > 11 on the plane, where xk-1 < xk for all k. In the complete metric space Q of all continuous functions q : [xo,xK] -,R such that q(x,) = yo and q(xK) = y K define the function mapping Y by

where k = 1 , 2 , ..., K , the V , E (-I, 1 ) are free parameters, and the 4K parameters ak , bk ,ck ,dk are uniquely determined by a&

+ bk = Xk-1,

+ bk = x&

~ & X K

(75)

v&yof CkXO + dk = Yk-1, (76) v./kYK + CkXK + dk = yk Under the action of Y the graph of the input function q is mapped to the graph of the output Y(q)via affine mappings ( ~ , yy) (ax

+ b, Vu + cx + d ) ,

which include contractions and shifts of the domain and range of q. Y is a contraction mapping in Q and has a unique fixed point that is a

FRACTAL SIGNAL ANALYSIS USING MATHEMATICAL MORPHOLOGY 215

D=1.8

0

la,

2w

300

400

0

100

200

300

4w

500

0

100

200

3M

4w

500

500

, -

FIGURE1. (a) Signals from sampling WCFs over [0, I] with y = 5 and various D. (b) Signals from sampling FIFs that interpolate the sequence 0, 1, 4, 2, 5, 3 with various D. (c) FBM signals obtained via a 512-point inverse FFT on random spectra with average All three signals in each class have N = 5 0 0 and are scaled to have the magnitude a lwlD-2.5. same amplitude range. (From Maragos and Sun, 1991); 01993 IEEE.)

216

PETROS MARAGOS

continuous function F : [xo,X K ] + R that interpolates the given data; i.e., F(xk) = yk for k = 0, 1 , ...,K. F is called a fractal interpolation function (FIF)6 because quite often the fractal dimension D of its graph exceeds 1 . Specifically (Barnsley, 1988; Hardin and Massopust, 1986), if IVkl > 1 and (xk,yk) are not all collinear, then D is the unique real solution of

Et=

K

C JVkJaF-'= 1 k=l

(77)

Otherwise, D = 1 . Thus by choosing the vertical scaling ratios Vk's we can synthesize a fractal interpolation function of any desired fractal dimension. F can be synthesized by iterating Y on any initial function q in Q; i.e., F = lim,,,Y'""(q) where Yo"(q)= YIY'@"'l)(q)l. Given a finite-length discrete-time signal f , [ k ] , k = 0 , 1 , ..., K, an algorithm was described in Maragos (1991) to fractally interpolate f , by an integer factor M by sampling a FIF whose fractal dimension can be controlled via a single parameter. Specifically, we start from the K + 1 data pairs (xk = kM,yk = f , [ k ] ) with xK = MK = N , set ak = 1/K, bk = and select a constant Vk = V E (-1, l), where J V J= K D - 2 , 1
Based on F D we can up-sample f , to a 1 :M interpolated signal FD(n), n = 0 , 1, . . .,N . The larger ( V (is, the larger D,and the more fragmented F D . Figure l b shows examples of FIFs that interpolate a fixed data sequence of K + 1 = 6 points by a factor M = 100 using positive ratios V = 5O-'. C. Fractional Brownian Motion

The fractional Brownian motion (FBM) (Mandelbrot and van Ness, 1968) B H ( t )with parameter 0 < H < 1 is a time-varying random function with stationary, Gaussian-distributed, and statistically self-affine increments; In Barnsley (1988) and in Maze1 and Hayes (1991) more general FIFs are also discussed using hidden variables.

FRACTAL SIGNAL ANALYSIS USING MATHEMATICAL MORPHOLOGY

21 7

the latter means that [BH(t+ T ) - B H ( t ) ]is statistically indistinguishable from r- H[ BH(+ f r T ) - B H ( t ) Jfor any T and any r > 0. The variance of FBM obeys the power law Var[B,(t

+ T)- ~

~ ( = t ) ]

(80)

where VH is a constant depending on H. The fractal dimension of B H ( t )is D = 2 - H . Its power spectrum' is

sH(o)OC

1

~

1 0y+'

(81)

Hence, an efficient algorithm (Voss, 1988) to synthesize an FBM is to create a random sampled spectrum whose average magnitude is ~ / ) u I ~ + ~ . and ~ its random phase is uniformly distributed over [0,2n]. In our experiments we synthesized and then transformed this spectrum via an inverse FFT to obtain an FBM sequence from which we retained the first N + 1 samples. Figure lc shows synthesized FBM sequences of varying D. The larger D (the smaller H ) , the more fragmented these fractal signals look. In addition to the FFT method, there are several other methods to synthesize FBM signals (Mandelbrot and Wallis, 1969; Voss, 1988). One rigorous approach discussed by Lundahl et al. (1986) involves Cholesky decomposition of the correlation matrix of discrete fractional Gaussian noise (i.e., sequence of increments of FBM) and synthesizing the FBM as a running average of the fractional noise. This approach, however, is computationally more complex than the FFT approach. Some special methods to measure D for FBM signals include: 1 . Fitting a straight line to the data (log S H ( o ) ,l o g ( o ( )and measuring the slope yields D . This is perhaps the most popular method because of the simplicity of computing spectra using FFT. The power spectrum estimation part of this approach has been improved in various ways which include using Gabor filters (for 2D FBM) by Super and Bovik (1991) and wavelet decomposition of l/lols processes in noise (Wornell and Oppenheim, 1990). 2. The statistical self-affinity of FBM yields a power scaling law for many of its moments; linear regression on these data can measure D (Pentland, 1984). 3. Maximum likelihood methods for estimating the H of discrete fractional Gaussian noise have been developed by Lundahl et al. (1986) and by Tewfik and Deriche (1991).

' Strictly speaking, the power spectrum of the nonstationary FBM is not well-defined. However, for w # 0, we can approximately interpret S,(w) as proportional to the average power of B H ( t )within a narrow frequency band around w (Mandelbrot, 1982).

218

PETROS MARAGOS

4. Mallat (1989) showed that the ratio of the energies of the detail signals at any two consecutive scales in a dyadic wavelet decomposition of an FBM signal is equal to 22N;this can be used to estimate H . V. MEASURING THE FRACTAL DIMENSION OF 1D SIGNALS A . 2 0 Covers via 2 0 Set Operations

In this section we focus on a generalized version of the Minkowski cover method. Specifically, given a nonempty compact set B E R2,consider the positive homothetics EB = ( ~ bb :E B ) at all scales E L 0, and define the generalized cover C,(E)of a planar set F E R2as its morphological dilation by the structuring element EB: CB(E)= F @ EB.

(82)

Henceforth we call CB a morphological cover. The Minkowski cover corresponds to using a disk for B. Bouligand’s work implies that the Minkowski-Bouligand dimension of a compact planar set F can also be obtained by replacing the disks in the Minkowski cover with arbitrarily shaped planar compact sets B that have a nonzero area, contain the origin, and possess a nonzero minimum (6,) and maximum (A,) distance from the origin to their boundary. Specifically,

(21

area(F,)

($y

(83)

Hence

1[area(C,

(E))] =

A [area(F,)]

(84) Thus if we replace the Minkowski cover area in (45) with the area of the generalized cover C , , the fractal dimension will remain the same. B. 2 0 Covers via ID Function Operations

In this section we deal only with continuous-time finite-length signals f ( t ) , 0 s t IT , in which case the curve F of the discussion in Section 111 becomes the graph Gr(f) = ( ( t , f ( t ) )0: 5 t 5 TI (85) off. If (x, y) are the Cartesian coordinates of the plane R2,the time t-axis will henceforth coincide with the x-axis, whereas the signal amplitude f ( t ) assumes values on the y-axis.

FRACTAL SIGNAL ANALYSIS USING MATHEMATICAL MORPHOLOGY

2 19

The discussion in Section V.A implies that the fractal dimension of Gr(f) can be found by using general morphological covers

of the signal’s graph with compact planar sets B E R2: The digital implementations of such morphological coverings by disk-like or other (e.g., horizontal line segment) structuring elements were done in Tricot et a/. (1988) and in Dubuc et a/. (1989) by viewing Gr(f) as a binary image signal and dilating this binary image. However, this 2D processing of a 1D signal, on the one hand is unnecessary and on the other hand increases the requirements in storage space and the time complexity for implementing the covering method. Thus, for purposes of computational efficiency, it is desirable to obtain the area of C, by using ID operations on f,i.e., dilations and erosions o f f by a function g with a compact support C. Specifically, for a properly chosen g , we could obtain the cover area by integrating the difference signal f @ g - f 0 g . However, since f is defined only* over [0, TI and the morphological cover C , involves points t from outside this interval, we modify the cover and the signal operations f @ g , f @ g to handle the boundaries o f f properly. Thus, we replace the covers C,(E) with their restriction on the vertical strip [0, T ] x ( - 0 0 , a),i.e., with the truncated morphological cover

c ~ * (PE[Gr(f) )

o E B In ( [ o , TI x (-m,=))

(87)

We also modify the definitions of 1D dilations and erosions, so that they do not require any values o f f outside [0, T I . Thus, we define the supportlimited dilation and erosion off by g with respect to a support set S s R :

(f0

SUP

sg)(t) XE

(C;+r)ns

If(x) + g ( t - x ) ) ,

tES

(88)

In what follows we shall find a proper g such that the integral of the difference signal between the support-limited dilation and the erosion off by g is equal to the area of the set cover C;(E)at all scales E , if B satisfies certain constraints. The main theoretical result requires a series of individual steps explained next.

* Assuming thatf(t) is defined over all t E R by settingf(t) f 0 g computed as

= -m for t

t+

[0, TI, the erosion

in (8) gives a signal that is not the (desirable) lower envelope of the

morphological cover Gr(f) @ B.

220

PETROS MARAGOS

First, for each scale E 2 0, the upper and lower envelope of the morphological set cover are defined respectively as the signals

'

u&(x) sup(Y : (xi y ) E CE (&) 1

(90)

L,(x)

(91)

inf(y: (x, y ) E CE(&))

Further, let us define a function (92)

g(x) 4 SUPIY : ( x ,Y ) E BI and its &-scaledversion by

g,(x) 4 sup[y: ( x ,y ) E E B J , Then we have the following result.

E 2

0

(93)

Lemma 9. (Maragos and Sun, 1991). Let f:S .+ R be a continuous function, where S = [0, TI. Let B C R2be a compact set that is symmetric with respect to both the x- and the y-axis of the plane. Then

Further, if we define the function-cover area

I(f O s g e ) - (f Osg&)I(X)dX

Ag(&)

(95)

SO*

we have the following.

Theorem 10. (Maragos and Sun, 1991). Let f:S R be a continuous function, where S = [0, T I . Let B E R2 be a compact set that is also single-connected (i.e., connected with no holes) and symmetric with respect to both the x- and the y-axis of the plane. Then +

area[C&)] =

loT

[UJx) - L,(x)]dx

and thus the set-cover and function-cover areas are identical: area[C,*(~)]= A&)

(97)

Thus, instead of creating the cover of a 1D signal by dilating its graph in the plane by a 2D set B (which means 2D processing), the original signal can be filtered with an erosion and a dilation by a 1D function g . For example, if B is a unit-radius disk or rhombus, then

B = ( ( x , Y ) : x+~y 2 I 1 ) * g(x) = dC?, B = ( ( x , Y )1:x1

+ ( y (s

1)

g(X) = 1 - 1x1,

1x1

1

(98)

1x1 5 1

(99)

I

FRACTAL SIGNAL ANALYSIS USING MATHEMATICAL MORPHOLOGY

22 1

In a related work, Tricot et al. (1988) and Dubuc et a/. (1989) showed that we can find D, by using set covers CB where B is the unit horizontal segment [- 1 , 11 x (0) and that

W

=

loT[

SUP

IYI 5

I

l f ( x + Y ) ) - inf ( f ( x+ Y ) ) dx,

6

IY I 5

&

( 100)

which is called the variation off, is equal to area[C;(&)] if B = [- 1 , 1) x lo). Their result becomes a special case of Theorem 10. Specifically the assumptions of Theorem 10 allow for B to be equal to horizontal segments [ - w , w] x lo), in which case g(t) = 0 for t E [- w , w ] and g(t) = -a for It( > w. Thus the horizontal structuring element case corresponds to selecting a flat function g equal to zero on its support. The following theorem shows that we can find the fractal dimension D, of the signal’s graph by using covers with functions g . Theorem 11. Let the function f and set B satisfy all the assumptions of Theorem 10, and also assume that B # ((0,O)). Then the MinkowskiBouligand dimension of the graph o f f is equal to DM[[Gr( f)]= 2 - A(A,) = lim c-0

b[.qW21 log(l/&)

Proof. Both in the case where B has nonzero area and possesses a nonzero minimum distance from the origin to its boundary (Bouligand, 1928), and in the case where B is the horizontal segment [-1, 11 x 10) (Dubuc et al., 1989), D, remains unchanged if we replace the area of the Minkowski cover by disks in (45) with the area of covers CBby the above generalized compact sets B. Then the area of CB(&) is equal to the sum of a r e a [ C i ( ~ ) plus ] some residual term that is due to dilations of the graph’s boundary points. The infinitesimal order of this residual term is 2, because it scales proportionally to c2. Hence, since A[area(C,)] = 2 - DM 5 1 , we can ignore the above residual term and use as cover area in (45) the area of the truncated cover. Then, Theorem 10 completes the proof, since it allows to replace the area of covers by sets with the area of covers by functions. (Q.E.D.) In practice, assuming that A,(&)= (constant) * c X for yields that A&) log 7= D M * log

E

very close to 0

as&-0

This leads to the following practical algorithm to compute Dw in the discrete case.

222

PETROS MARAGOS

C . Algorithm for Discrete-Time Signals To estimate the fractal dimension of a discrete-time finite-length signalf [n], n = 0, 1 , . , .,N,we must adapt our discussion in Section V.B. First, covers at discrete scales E = I , 2,3, ...,emax are used. The set B, E R2 used for covers is also restricted to be convex, because then EB, = BFe for E = 0, 1,2, ... where B@' 4i B @ B ... @ B (E times). Hence, for B, convex and integer E, g, is equal to the &-folddilation of g with itself, denoted as g@e

g g

0g

... 0 g

(E

times)

(103)

Then it can be shown that

f o s P = ((fO s g ) O s g . . . ) o s g f osg@& = ((fOsg) O s g . . . ) o s g E

(104) ( 105)

times

All the above ideas lead to the following algorithm for digitally implementing the morphological covering method using function covers (Maragos and Sun, 1991).

Step 1. Select a unit-radius discrete set B C Zz that is a convex symmetric subset of the 3 x 3 square set of pixels with integer coordinates (n, m),which correspond to points (n, mv) in RZwhere u > 0 is the vertical grid spacing. There are only three choices for such a By and the corresponding g [ n ] is a three-sample function: '1. If B is the 3 x 3-pixel square, the corresponding g is shaped like a

rectangle: (106) grL-11 = gr[Ol = gr[ll = h > 0 2. If B is the five-pixel rhombus, then g is shaped like a triangle: ( 107) gti-11 = g,[ll = 0, g,[Ol = h > 0 3. If B is the three-pixel horizontal segment, then the corresponding g can be viewed as resulting either from g, or from gr by setting h = 0. In this case g is a flat function equal to zero on its support.

Step 2. Perform recursively the support-limited dilations and erosions of f by g@' at scales E = 1,2, ..., E,,,. That is, set G = [ - 1 , 0, l ) , S = (0, 1, ..., N), and use (88) and (104), which yield

Likewise for the erosions f Osg@'. The dashed lines in Fig. 2 show these multiscale erosions/dilations by the three different functions g .

FRACTAL SIGNAL ANALYSIS USING MATHEMATICAL MORPHOLOGY

223

Step 3. Compute the cover areas N

A,[&] =

C

((fOsgBE) - ( f ~ g@&))[nl, s

E

= 1, ***,Emax5

N

y

(109)

n=O

Step 4. Fit a straight line using least squares to the graph of log(A,[&]/(~)~) versus log(l/&),for E = 1 2, . . .,E,,, . The slope of this line gives us an approximate estimate of the fractal dimension off, as implied by (102). Although the shape of the structuring function g is not very crucial, its height h , however, plays an important role. Although h does not affect the morphological covering method in the continuous case, in the discrete case large h will sample the plot of (102) very coarsely and produce poor results. Thus small h are preferred for finer multiscale covering area distributions. However, the smaller h is, the more computations are needed to span a given signal’s range. In addition, as noted by Mandelbrot (1985), the covering method with 2D discrete disks (as well as the box-counting dimension) greatly depends on the relationship between the grid spacing v and the dynamic range off. Henceforth, we assume that u is approximately equal to the signal’s dynamic range divided by the number of its samples. This is a good practical rule, because it attempts to consider the quantization grid in the domain and range of the function as square as possible. Further, whenever h > 0, we select h = u. Therefore, assuming that for an N-sample signal, its range has been divided into N amplitude levels, the above algorithm that uses function-cover areas A, has a linear complexity ~(NE,,,) with respect to the signal’s length, whereas using set-cover areas with 2D sets yields quadratic complexity O(NZ~,,); further, both approaches give the same dimension, as Theorem 10 implies. Among previous approaches, the 1D version of the work in Peleg et af. (1984), Stein (1987), and Peli et al. (1989) corresponds to the morphological covering method using g, with h = 1 . The “horizontal structuring element method” in Tricot et af. (1988) and in Dubuc et af. (1989) corresponds to using h = 0. The fractal dimension of the graph off resulting from the morphological covering method using function-covers (in both the continuous and discrete case) has the following attractive properties. (See Maragos and Sun, 1991, for proofs.) I f f is shifted with respect to its argument and/or amplitude, then its fractal dimension remains unchanged; i.e.,

f’(4= f(x - X O ) + b * D,w[Wf)I

= D~[Gr(f‘)l

(1 10)

for arbitrary b, x,. Further, if h = 0, then the fractal dimension estimated via erosions/dilations by a flat g also remains invariant with respect to any

224

PETROS MARAGOS

'T

..........

-* 7 0 100 1

200

300

m

500

100

200

300

4M)

I 500

I00

2W

300

400

500

*T

-4

4 0

'T

I

I

4

0

SAMPLE

(c)

FIGURE2. An FBM signal (solid line) with D = 1.5, N = 500, and its erosionddilations (dashed lines) by gee at scales E = 20,40. (a) Rectangular g = g, with h = 0.01. (b) Triangular g = g, with h = 0.01. (c) Rectangular g with h = 0. (From Maragos and Sun, 1991; 01993 IEEE.)

FRACTAL SIGNAL ANALYSIS USING MATHEMATICAL MORPHOLOGY

225

affine scaling of the amplitude off: i.e.,

f ’ ( x ) = uf(x) + b

and

h

=

0

D M [ G r ( f ) ]= D,[Gr(f’)]

(111)

for arbitrary b and u # 0. The property (1 11) was also noted in Dubuc et al. (1989). The morphological covering and the box counting method give identical fractal dimension for continuous-time signals f. However, in the discrete case they have different performances, and it is because of properties (110) and (1 11) that the morphological covering is more robust than the box counting method. The latter is affected by arbitrary shifts of the argument of f, by adding constant offsets to f, and (more seriously) by scaling its amplitude range, because all these affect the number of grid boxes intersected by the graph off. However, the morphological covering method using covers with 1D functions g become completely independent from affine scalings of the signal’s range if we choose h = 0. In addition, since for the case h = 0, the erosions/dilations by g can be performed faster, we henceforth set h = 0 in all our computer experiments with the morphological covering method. Table 1 shows the estimated dimension D* and the percent estimation error 100 * ID - D*(/D using a two-pass9morphologicalcovering method on signals with N + 1 = 501 samples synthesized from sampling WCFs and FIFs of various D.The WCFs were defined for t E [0, 11 with y = 5. The FIFs interpolated the six-point data sequence 0, 1 , 4 , 2 , 5 , 3 using positive scaling ratios V = 5D-2. These experimental results and many others reported in Maragos and Sun (1991) indicate that, for these two classes of deterministic fractal signals, the morphological covering method performs very well for various combinations of dimensions D E [ 1.2, 1.81 and signal lengths N E [loo, 20001 since the average percent error for estimating D was 2 to 3% for both WCFs and FIFs. The maximum scale E,,,

and in general the scale interval [ l , E , ~ , ] over which we attempt required for a good estimation of D may exhibit considerable variations and depends on the dimension D , on the signal’s length N , and on the specific class of fractal signals. Maragos and Sun (1991) used the following heuristic rule for determining E,,,: to fit a line to the log-log plot of (102) is an important parameter. The E,,,

E,,~ =

MaxScale(D, N ) = min

Thus, t o apply the morphological covering method to a signal, a two-pass procedure consists of first applying the covering method with a small scale interval emox = 10, t o obtain some estimate D , of the fractal dimension. Then the covering method is reapplied t o the same signal by using E,,, = MaxScale(D, , N ) t o obtain a second estimate, which is considered as the final estimate D* of D.

226

PETROS MARAGOS TABLE 1 MORPHOLOOICAL COVERING METHODON WCFs, FIFs, AND FBM Signal ~

True D ~~

WCF WCF WCF FIF FIF FIF FBM FBM FBM

Estimated

Error

~~

1.4 1 .s 1.6 1.4 1 I .6 1.4 1.5 1.6

.s

1.424 1.515 1.606 I .384 1A78 1.576 1.393 1.474 1.553

1.71%

I .03% 0.39% 1 12% 1.45% 1.53% 0.5% 1.7% 2.9% I

Table 1 also shows the results from applying the (two-pass) morphological covering method on FBM signals. For each true D, it reports the sample mean D* of the estimates and the percent mean estimation error 100 * ( D - D*(/D by averaging results over 100 random FBM realizations. All FBM signals had N + 1 = 512 samples and were synthesized using a 512-point FFT. Maragos and Sun (1991) compared the performance of the morphological covering method with that of the power spectrum method to estimate the fractal dimension of FBM signals in a noise-free case as well as in the presence of additive white Gaussian noise. Their experiments, over 7 x 5 combinations (D,N ) of dimensions D E [1.2, 1.81 and signal lengths N + 1 E (2’, 2*, 2’, 21°, 211) with 100 random FBM realizations each, indicate that in the absence of noise both methods yield a similar average error of about 3 to 4%, whereas in the presence of noise the morphological covering yields much smaller error than the power spectrum method. Concluding, we emphasize that, since all three classes of fractal signals are sampled versions of nonbandlimited fractal functions, some degree of fragmentation is irreversibly lost during sampling. Hence, since the true D refers to the continuous-time signal, the discrete morphological covering algorithm (as well as any other discrete algorithm) can offer only an approximation of D. In addition, the specific approach used to synthesize the discrete fractal signals (e.g., the FFT for FBM) affects the relationship between the degree of their fragmentation and the true D, and hence it may also affect the performance of the D estimation algorithms. D. Application to Speech Signals The nonlinear dynamics of air flow during speech production may often result in some small or large degree of turbulence. In this section we quantify the geometry of speech turbulence, as reflected in the fragmentation

FRACTAL SIGNAL ANALYSIS USING MATHEMATICAL MORPHOLOGY

227

of time signal, by using the short-time fractal dimension of speech signals. Some possible applications are also outlined for speech segmentation and sound classification. During speech production a vortex is a flow region of similar (or constant) vorticity vector. Vortices in the speech air flow have been experimentally found above the glottis by Thomas (1986) and theoretically predicted in Teager and Teager (1989) and McGowan (1989) using simple geometries. There are several mechanisms for the creation of vortices: (1) velocity gradients in boundary layers; (2) separation of flow, which can easily happen at cavity inlets due to adverse pressure gradients; (3) curved geometry of tract boundaries, where due to the dominant inertia forces the flow follows the curvature and develops rotational components. After a vortex has been created, it can propagate downstream (Tritton, 1988) through vortex twisting and stretching as well as through diffusion of vorticity. The Reynolds number Re = pUL/,u characterizes the type of flow, where U is a velocity scale; L is a typical length scale, e.g., the tract diameter; p is the air density; and ,u is the air viscosity. As Re increases (e.g., in fricative sounds or during loud speech), all these phenomena may lead to instabilities and eventually result in turbulent flow, which is a “state of continuous instability” (Tritton, 1988) characterized by broad-spectrum rapidly varying (in space and time) velocity and vorticity. Modern theories that attempt to explain turbulence predict the existence of eddies (vortices with a characteristic size A) at multiple scales. According to the energy cascade theory, energy produced by eddies with large size is transferred hierarchically to the small-size eddies that dissipate it due to viscosity. A related result is the famous Kolmogorov law,

E(k, r) 0: r2’3k-5’3 (k in a finite range)

(113)

where k = 2n/A is the wavenumber, r is the energy dissipation rate, and

E(k,r) is the velocity wavenumber spectrum, i.e., Fourier transform of spatial correlations. In some cases this multiscale structure of turbulence can be quantified by fractals. Mandelbrot (1982) and others have conjectured that several geometrical aspects of turbulence (e.g., shapes of turbulent spots, boundaries of some vortex types found in turbulent flows, shape of particle paths) are fractal. In addition, processes similar to the ones that in high-Re speech flows cause vortices to twist, stretch, and fold (due to the bounded tract geometry) have also been found in low-order nonlinear dynamical systems to give rise to fractal attractors. All the above theoretical considerations and experimental evidence motivated our use of fractals as a mathematical and computational vehicle to analyze and synthesize various degrees of turbulence in speech signals. The main quantitative idea that we focus on is the fractal dimension of

228

PETROS MARAGOS

speech signals, because it can quantify their graph's fragmentation. Since the relationship between turbulence and its fractal geometry or the fractal dimension of the resulting signals is currently very little understood, herein we conceptually equate the amount of turbulence in a speech sound with its fractal dimension. Although this may be a somewhat simplistic analogy, we have found the short-time fractal dimension of speech to be a feature useful for speech sound classification and segmentation. To measure it, we use the morphological covering algorithm described in Section V .I11 with a flat function g, i.e., with height h = 0. The speech signals used in our computer experiments were sampled at 30 kHz. Hence the smallest ( E = 1) time scale at which their fractal dimension D was computed was 1/15 msec. The dimension D was computed over moving speech segments of 30 msec ( N = 900 samples) as a short-time feature. Figure 3 shows the waveform of a word and its short-time fractal dimension as function of time. While D behaves similarly with the average zero-crossings rate, it has several advantages: For example, it can distinguish between a vowel and a voiced fricative, whereas the zero-crossings can fail because the rapid fluctuations of the voiced fricative may not appear as zero-mean oscillations, which would increase the zero-crossing rate, but ZERO-CROSSINGS MEAN SQUARED AMPLITUDE

..

:'

).$

% h

I

".I

.+

I

r

20

I SPEECH SIGNAL

1

(I

2

4

6

.8

I

TIME (in SEC)

FIGURE3. Speech waveform of the word /sieving/ sampled at 30 kHz and its short-time fractal dimension, average zero-crossings rate, and mean squared amplitude estimated over a moving 10 msec window, computed every 2 msec.

FRACTAL SIGNAL ANALYSIS USING MATHEMATICAL MORPHOLOGY

229

as a graph fragmentation that increases D. We have also observed cases where D could detect voiced stops but the zero-crossings could not. Thus, the short-time fractal dimension is a promising feature that can be used for segmentation of speech waveforms. However, as Fig. 3 shows, the silence portions of the signal (due to their noise-like geometrical structure) incur a high fractal dimension similar to that of the unvoiced fricatives. Therefore, for applying it to speech segmentation, the fractal dimension should be supplemented by some additional features that can distinguish between speech and silence. Several experiments reported in Maragos (1991) lead to the following conclusions: 1. Unvoiced fricatives ( I F / , / O / , /W), affricates, stops (during their turbulent phase), and some voiced fricatives like / Z / have a high fractal dimension E [1.6, 1.91, consistent with the turbulence phenomena present during their production. 2. Vowels (at time scales < 0.1 msec) have a small fractal dimension E [ l , 1.31. This is consistent with the absence or small degree of turbulence (e.g., for loud or breathy speech) during their production. 3. Some voiced fricatives like / V / and / T H / , if they don’t contain a fully developed turbulence state, at scales < 0.1 msec have a medium fractal dimension D E [1.3, 1.61. Otherwise, their dimension is high ( > 1.6), although often somewhat lower than that of their unvoiced counterparts. Thus, for normal conversational speech, we have found that its short-time (e.g., over 10-30 msec frames) fractal dimension D (evaluated at a time scale < 0.1 msec) can roughly distinguish these three broad classes of speech sounds by quantifying the amount of their waveform’s fragmentation. However, for loud speech (where the air velocity and Re increase, and hence the onset of turbulence is easier) or for breathy voice (especially for female speakers) the dimension of several speech sounds, e.g., vowels may significantly increase. In general, the D estimates may be affected by several factors including (a) the time scale, (b) the speaking state, and (c) the specific discrete algorithm for estimating D. Therefore, we often don’t assign any particular importance to the absolute D estimates but only to their average ranges and relative differences. Related to the Kolmogorov 5/3-law (113) is the fact that the variance of velocity differences between two points at distance AX varies a (AX)2’3. These distributions have identical form to the case of fractional Brownian motions whose variances scale with time differences AT as ( A T ) 2 H , 0 < H c 1 , the frequency spectra vary a 1/(uIZH+’, and time signals are fractal with dimension D = 2 - H. Thus, putting H = 1/3 leads to D = 5/3

-

230

PETROS MARAGOS

for speech turbulence. Of course, Kolmogorov’s law refers to wavenumber (not frequency) spectra, and we dealt with pressure (not velocity) signals from the speech flow. Thus we should be cautious on how we interpret this result for speech. However, it is interesting to note that in our experiments with fricative sounds we often observed D (for time scales c 0.1 msec) in the range [1.65, 1.71. Pickover and Khorasani (1986) reported a global dimension D = 1.66 for speech signals but no mention of the 5/3 law was made, their D estimation algorithm was different, and the time scales were much longer, i.e., 10 msec to 2 sec; thus in their work the time scales were above the phoneme level, whereas our work is clearly below the phoneme time scale. VI

. MEASURING THE FRACTAL DIMENSION OF 2D SIGNALS A. 3 0 Covers via 30 Set Operations

Bouligand (1928) showed that the dimension DMof compact sets F E R3 can also be obtained by replacing the spheres in the Minkowski cover with more generally shaped sets. Specifically, let B be a compact subset of R’ with Cartesian coordinates (x, y , 2). Replacing the spheres of radius E with the &-scaled version of B, i.e., the positive homothetic EB = ( ~ bb :E B ) leads to the 3D morphological cover CB(E)

=F@

EB

(1 14)

The Minkowski cover F, is a special case where B is a unit-radius sphere. If B has a nonzero volume and its interior contains the origin, let us define the (nonzero) minimum and maximum distance from the origin to the boundary of B by BE and A s , respectively. The Bouligand showed that

Hence, the infinitesimal orders of vol[CB(&)] and vol(F,) are the same. Therefore, the fractal dimension of a set F can also be obtained from general morphological covers:

For the case of a continuous nonconstant function f ( x , y ) Dubuc et al. (1988) showed that B does not have to have nonzero volume, but it can also be a square parallel to the x , y plane; they called this special case the “horizontal structuring element met hod. ’’ Bouligand’s result (116) also applies to the special case where the set F becomes equal to the graph of some real function f ( x , y ) . In this case,

FRACTAL SIGNAL ANALYSIS USING MATHEMATICAL MORPHOLOGY

23 1

however, the digital implementation would require covering the image surface by 3D sets, which can be done by viewing Gr(f) as a set in the 3D discrete space and dilating this set of voxels. However, this 3D processing of a 2D signal on the one hand is unnecessary and on the other hand increases the requirements in storage space and the time complexity for implementing the covering method. Thus, for purposes of computational efficiency, it is desirable to obtain the volume of C,(E)by using 2D operations on f ( x , y ) , i.e., dilations @ and erosions 0 off by a structuring function g(x, y). This is explained in the next section.

B. 3D Covers via 2 0 Function Operations Let f ( x , y ) be a continuous real-valued function defined on the rectangular support S = [ ( x , y ) E R2 :0 Ix IX , 0 Iy 5 Y ) (117) and assuming its values on the z-axis. Dilating its graph Gr(f) by EB yields the cover

The goal here is to obtain the volume of this cover not by performing the above set dilation, but by first computing the cover’s upper and lower envelopes via morphologically dilating and eroding f by a function g related t o B and then obtaining the original cover volume by integrating the difference signal between these envelopes over S . Of course, certain restrictions have to be set on B . Specifically, let the cover’s upper and lower envelope be defined respectively as the 2D signals uc(x, y ) = SUPb : ( x , y , z ) E C B ( & ) f

L,(x,Y)

=

i n f k : ( x , Y ,z ) E CB(EN

(1 19) ( 120)

Since f ( x , y ) is defined only over S , and computing vol[CB(&)]involves points from outside this interval, we modify the signal operations f 0 g, f 0g so that they do not require any values off outside S . Thus, we define the support-limited dilation and erosion off by g with respect to a support S E R2:

(f0

s S)(Xl Y ) =

(f0

s g)(x, Y ) =

SUP

W P , 4) + g(x - P , Y - 4%

( x , v) E S (121)

inf

( f ( P ,4) - g ( P - x, 4 - u)l,

(XlY) E

@ , aE IC;+(X,.Y)I~S @.Y) E ic+(x,y)ins

S (122)

232

PETROS MARAGOS

Further, if we define the function g by g(x, Y ) = SUP(Z: (x, y , z) E BI

(123)

and its &-scaledversion by g,(x, Y ) =

SUP^: ( x , y , z) E EBI

(1 24)

then the following is true.

Lemma 12. If B c R3is compact and symmetric with respect to the x, y , z-axes, then for each E 2 0 ,

Proof. Let G = ( ( x ,U) : ( x , Y , Z ) E BI = S P ~ W

(126)

Since B is symmetric with respect to the x , y-axes, g,(x, y ) = g,( -x, - y ) and G = 6. Since B is symmetric with respect to the z-axis, g,(x,y) 5. 0 for all ( x , ~ in ) its domain EG.If

I(a, b) = [ c :(a, b, c) E E B ) for any (a, b) E EG, then note that sup@: c E Z(a, b)) = g,(a, b) inf(c: c E I(a, b)) = -g,(a, b)

To prove (94) we have U , ( X , Y ) = sup(z:x = P

+ a , y = q + b , z = f ( p , q )+ c, ( p , d E S , (a, 6, c) E &BI

= sup[f(x - a, y - b) = SUP(f(P, 4)+ g,(x =

(f0

+ c :(a, b) E EG n [S + (x, y ) ] , c E I(u, b))

- P , Y - 4):(x, Y ) E s n [&G+ ( P , 4)Il

s g,)(x, Y )

Likewise, L , ( x , y ) = inf[z: x = P

+ a,y

=q

+ 6 , z = f ( p , q) + c, (P,q) E S, (a, b, c) E EBI

= inf [ f ( x - a, y - b) = inflf(p, =

(f0

+ c :(a, b) E EG f l [S + (x,y ) ] ,c E I(a, b))

4) - g,(x - P,Y - 4): ( x , Y ) E S n t&G+ (P,d11

s g,)(X, Y ) (Q-E.D.1

FRACTAL SIGNAL ANALYSIS USING MATHEMATICAL MORPHOLOGY

233

By using the support-limited dilations and erosions we cannot account for the volume of the part of the original cover CB(c)outside the support off but only for the volume of the truncated morphological cover C ~ ( E= )[Gr(f)

0 E B ]n [ S X (-00,

m)]

( 127)

In what follows we shall show that the volume

{Jyrrasg& -fOsg,I(x?Y)dxdY

W)

0

(128)

resulting from integrating the difference signal between the support-limited dilation and the erosion o f f by g is equal to the volume of the truncated cover at all scales, if B satisfies certain constraints.

Lemma 13. If B E R3 is compact, symmetric with respect to the x , y , z-axes, and single-connected, then f o r each E 2 0 , VOl[CB*(&)]= =

Proof. Since g,(O, 0)

2

:‘1 loY

[U&(X, Y ) - L&(X’u)l dx dY

Vg(d

( 1 29) (130)

0, it can be easily shown that

U&Y) 2 f ( X , Y )

2

(x,H E S

L,(X,Y),

Define the set Q ( E ) = I(x, Y , z ) : (x,

v) E S , L,(x, Y ) 5 z 5

U&, Y)J

We shall prove that Q(E) = C ~ ( E )First, . let ( x , y , z ) E C&). Then, ( x , y ) E S and (x,y, z) E Gr(f) @ E B . Hence, x = p + a, y = q + b, and z = f ( x , y ) + c for some ( p , q) E S and (a, b , c) E EB. But then, from the definition of U e , it follows that z IU,(x, y ) ; likewise, z 2 L J x , y ) . Therefore, (x,y , z) E Q(E), and thus C&) E Q(E). Now let ( x , y , z ) E Q(E). Define the set

K = EBfl [(S + ( ~ , y )x )( - 0 0 , +w)] =

( ( a ,6, c) : (a, 6) E EG n (3 + (x,y ) ) ,c

E

Z(a, 6 ) )

Then, K is a connected set. Define the function

$(a, b, c) = f ( x - a, y

-

b) + c

on K. The function $ is continuous and has a connected domain K . The value z lies between the maximum U,(x,y ) = sup(+(a, b, c) : (a, b, c) E K J and the minimum LJx, y ) = inf ($(a,6, c) : (a, 6, c) E K ) value of 9 on K . Hence, from Bolzano’s intermediate value theorem (Bartle, 1976, p. 153),

234

PETROS MARAGOS

there is a point ( a f ,b’, c’) in K at which 4 takes the value z . By setting p = x - a ’ , q = y - b ‘ , and f ( p , q ) = z - c ’ w e have(p,q,f(p,q))EGr(f) and (a’,b’, c‘) E EB. Hence (x, y , z ) E C$(E) and thus Q(E) E C;(E). Therefore, we proved that Q(E) = C$(E).This set equality proves (129). The result (130) follows from (129) and Lemma 12. Thus the proof is complete. (Q.E. D. ) Thus, instead of creating the cover of a 2D signal by dilating its graph by a 3D set B (which means 3D processing), the original signal can be filtered with an erosion and a dilation by a 2D function g. As an example, if B = ((x,y,z ) : x 2+ y 2 + z2 I1 ) is the unit-radius sphere, then

g ( x , y ) = 41 - x 2 - y 2 , Theorem 14. Let f: S

+

x 2 + y2

I1.

R be a continuous function, where

S = [O,X] x [0, Y ] E R2.

Let B E R3 be a compact set that is also single-connected, symmetric with respect to thex, y , z-axes, and assume B # [ ( O , O , 0)).Then the MinkowskiBouligand dimension of the graph off is equal to DM[Gr(f)] = 3 - A(K) = lim e+o

WV,( E V E 31 log(l/&)

Proof. Both in the case where (a) B has nonzero volume and possesses a nonzero minimum distance from the origin to its boundary (Bouligand, 1928), and in the case where (b) B is the horizontal unit square (Dubuc et al., 1988), DM remains unchanged if we replace the volume of the Minkowski cover by spheres in (39) with the volume of covers CB by the above generalized compact sets B. Now, if aGr(f ) is the boundary of Gr(f), then the volume of CB(&) is equal to vol[CB(&)]= voI[C$(e)]

+ vol[aGr(f) 0 E B ]

The infinitesimal order of the volume of the dilated graph boundary is two, because it scales proportionally to e2. For example, in case (a) let iSB and AB be the minimum and maximum distance from the origin to the boundary of B . Then 7t1(6,E)2

2

s vol[dGr(f) 0 E B ]i

ZI(A,

E)’

2

where I is the (assumed finite) length of the boundary of Gr(f). Hence, since n[vol(CB)] = 3 - D,,, I1, we can ignore the term vol[dGr(f ) 0 E B ] and use as cover volume in (1 16) the volume of the truncated cover. Then, Lemma 13 completes the proof, since it allows to replace the volume of covers by sets with the volume of covers by functions. (Q.E.D.)

FRACTAL SIGNAL ANALYSIS USING MATHEMATICAL MORPHOLOGY

235

C. Discrete Algorithm

In practice we deal with 2D functions that are both quantized and spatially sampled, e.g., digital images. Thus, the theory in section V1.B must be adapted as follows. Assume that we have a 2D discrete-space finite-support signal f [n,m ] , n = 0, 1, . . . ,N , m = 0 , 1, .. .,M . We shall use covers at discrete scales E = 1 , 2 , 3 , ...,t m oThe x . 3D set B, C R3 used for covers, in addition to the restrictions of Theorem 14, is also restricted to be convex so that its corresponding function g at integer scales E is given by the &-fold dilation g@' = g @ g . . . @ g. The 3D space is then assumed to be sampled by the cubic grid (n, m, k ) of integer coordinates corresponding to the real coordinates (n,m, ku) where u > 0 is the grid spacing. We assume that u is approximately equal to the dynamic range o f f divided by the average number of samples in one dimension. Finally, the discrete set B E Z3 corresponding to B, is assumed to have a unit-radius, because larger radii would create coarser volume distributions. Hence, B must be a convex, symmetric subset of the 3 x 3 x 3 set of voxels around the origin. This yields only six choices for B : 1. B is the 27-voxel cube with horizontal cross-section the 3 x 3-pixel square G, E Z2, and the corresponding function g has square support and cubic shape:

2. B is the 11-voxel octahedron with horizontal cross-section the square G , , and the corresponding function g has a square support and pyramid shape: [n,ml E G,\I(O, 0 )) 0, [n,ml = [O, 01 (133) [n,ml 4 Gs 3. B is the 15-voxel rhomboid with horizontal cross-section the five-pixel rhombus G, C Z2, and the corresponding function g has a rhombus support and cubic shape:

4. B is the seven-pixel rhombo-octahedron with horizontal cross-section the rhombus G , , and the corresponding function g has rhombus support and pyramid shape: 0,

[n,ml E Gr\I(O, 011 [n,ml = [0,01 [n,ml @ Gr

(135)

236

PETROS MARACOS

5 . B is the nine-voxel square G, x (0). The corresponding function g is flat and can be obtained from the functions g , or g , by setting their heights h = 0. 6. B is the five-voxel rhombus G, x lo). The corresponding function g is flat and can be obtained from the functions g , or ,g by setting their heights h = 0. The morphological covering algorithm for 2D signals consists of the following steps:

Step 1. Select a 3D set structuring element B from the above six choices, and let g be its corresponding function. Step 2. Perform recursively the support-limited dilations and erosions o f f by gBeat scales E = 1,2, ., E ~ That ~ is,~ set. G equal to G,or G,, S = [O, 1, .,.,N j x (0,1, ..., M ) , and use (121) and (104). If G = G,, this yields for E = 1

..

f

OSg [ n , rn]

=

max

max (f[n

-1sisl - 1 s j 5 1

+ i, m + j ] + g [ i , j ] )

Then, for any G,

Likewise for the erosions f OSg B E .

Step 3. Compute the volumes

n=O m=O

Step 4. Fit a straight line using least-squares to the plot of (log ~

[ E ] / E log ~ ,

ih).

The slope of this line gives an estimate of the fractal dimension of the graph off. Among previous approaches, the work in Peleg et al. (1984); Stein (1987); and Peli et a/. (1989) corresponds to using gsp or g , with h = 1 . The variation method in Dubuc et al. (1988) corresponds to using a horizontal square B, i.e., a flat function g,, with h = 0. Assuming that M = N and v = (max,,,[ f [n,rnll - min,,,If [n,rnll)/N, the computational complexity of using covers with 3D sets is ~ ( N ' E , ~ ~ ) , whereas using covers with 2D functions yields a complexity 0 ( N 2 ~ , , , ) . (In both cases, if h > 0, it is assumed that h = u.)

FRACTAL SIGNAL ANALYSIS USING MATHEMATICAL MORPHOLOGY

237

VII. MODELING FRACTAL IMAGESUSINGITERATED FUNCTION SYSTEMS There is a rich class of nonlinear dynamical systems that consists of combinations of contraction maps on the Euclidean space and converge to attractor sets that are fractals. These fractal attractors include many of the well-known mathematical fractal sets and can model well images of natural scenes. These systems are known as iterated function systems and their theory was developed mainly by Hutchinson (1981) and Barnsley (1988) and his coworkers. Currently, there are many computer algorithms to generate fractals. Examples include the FFT-based synthesis of images modeled as 2D fractional Brownian motion (Voss, 1988) and the synthesis via iterated function systems (Barnsley and Demko, 1985; Diaconis and Shahshahani, 1986). However, the inverse problem, i.e., given a fractal image find a signal model and an algorithm to generate it, is much more important and very difficult. An approach that is promising for solving this inverse problem is modeling fractal images with collages; the basic theory is summarized in Section V1I.A. Then an algorithm is described in Section VI1.B to find the collage model parameters via morphological skeletonization. A. Modeling Fractals with Collages

The key idea in the collage modeling (Barnsley et al., 1986) of a fractal set F is that if we can closely cover it with a collage of m small patches that are reduced distorted copies of F, then we can approximately reconstruct F (within a controllable error) as the attractor of a set of m contraction maps (each map is responsible for one patch). To simplify the analysis let us assume that we deal with compact planar sets F G R2.Let w i : R2 R2 be contraction maps; i.e., -+

IIwi(X)

- wi(y)II

5 siI(X

-

vII,

VX,Y E

R2

(136)

where 0 5 si c 1 are constant contractivity factors. Let X be the collection of all nonempty compact subsets of R2 and define the collage map W :X X by -+

ni

m

W(X)

u w;(X) u[ w ; ( x ) : x E X ) , =

i= 1

XE

x

(137)

i= 1

Then Hutchinson (1981) showed that the map W is a contraction map on 3C with respect to the Hausdorff metric h, defined by

h(X, Y ) f inf(r 2 0 : X E Y @ rB, Y E X @ r B ] ,

X,Y E X (138)

238

PETROS MARAGOS

where B is the unit-radius disk. Namely where the contractivity factor s is equal to s = max ( s i ] lsism

Thus, the contraction mapping theorem implies that, if we iterate the map W starting from any initial set X E X, a unique fixed point Q. = lim W""(X)=

W(a)

n+m

will be reached. The limit set a, called the attractor, is independent of the initial set X and is often a fractal set. The foIlowing theorem goes one step further and states that if we can approximate well (with respect to the Hausdorff metric) an original set F with the collage W(F) of an iterated function system { w i : i = 1, ...,m), then the attractor of this system will also approximate well the original set F. Theorem 15. (Barnsley et al., 1986). Given a set F E X,

h ( E W F ) )c &

if (142)

then, for any X E X,

(

)

&

h F,lim Won(X) < n--

l-s

(143)

Thus, if we can find maps w ithat have small contractivities (i-e., s 4, 1) and make a good collage (i,e., with small distance E ) , then by iterating on an arbitrary compact set X the collage map W we can synthesize in the limit an attractor set that approximates well the original set F. In practical applications, analytically simple choices for the maps wi are the affine maps

Each wi, operating on all points ( x , y ) of F, gives a version of F that is rotated by an angle el, shrunk by a scale factor ri, and translated by the vector (tXi,tYi). The collage theorem and a related synthesis algorithm have been very successful for fractal image modeling and coding (Barnsley, 1988). These ideas work very well for images that have considerable degree of selfsimilarity. The difficulty, however, lies in finding appropriate maps wi , which (by variation of their scaling, rotation, and translation parameters)

FRACTAL SIGNAL ANALYSIS USING MATHEMATICAL MORPHOLOGY 239

can collage F well. The majority of earlier solutions required either considerable human intervention or exhaustive searching of all parameters in a discretized space. An approximate solution to this problem has been provided for binary images by Libeskind-Hadas and Maragos (1987) who used the morphological skeleton transform to efficiently extract the parameters of these affine maps, as explained in the next section. The collage models have also been extended to gray-level images by modeling image functions as measures and using the Hutchinson metric to quantify the goodness of the collage approximation (Barnsley, 1988). This measure-theoretic framework, however, is difficult to apply to images with discrete-domain. Recent improvements of the gray-level collage models for images with discrete-domain include the works of Jacquin (1992) and Lundheim (1992). Lundheim has also developed a least-squares approach to find optimal collage parameters, which is efficient and mathematically tractable. B. Finding the Collage Parameters via Morphological Skeletons

First we summarize the morphological skeleton transform for binary images, and then we outline its usage for finding the collage parameters. Since the medial axis transform (also known as symmetric axis or skeleton transform) was first introduced by Blum (1967), it has been studied extensively for shape representation and description, which are important issues in computer vision. Among the many approaches (Rosenfeld and Kak, 1982) to obtain the medial axis transform, it can also be obtained via erosions and openings (Mott-Smith, 1970; Lantuejoul, 1980; Serra, 1982; Maragos and Schafer, 1986). Let F E Z2 represent a finite discrete binary image, and let B G Z2be a binary structuring element containing the origin. The nth skeleton component of F with respect to B is the set S, = (FO nB)\[(F@ nB) 0 B ] ,

n

=

0, 1, ...,N

(145)

where N = max[n: F 0 nB # (211 and \ denotes set difference. The S,,are disjoint subsets of F, whose union is the morphological skeleton of F. (If B is a disk, then the morphological skeleton becomes identical with the medial axis.) We define the morphological skeleton transform of F to be the finite sequence (So,S , , . , S N ) . From this sequence we can reconstruct F exactly or partially; i.e., FOkB=

u

S,@nB,

O s k s N

(146)

ksnsN

Thus, if k = 0 (i.e., if we use all the skeleton subsets), F 0 kB = F and we have exact reconstruction. If 1 s k 5 N, we obtain a partial reconstruction,

240

PETROS MARAGOS

n=O

n= 1

n=2

n=3

FIGURE4. Morphological skeletonization of a binary image F (top left image) with respect to a 21-pixel octagon structuring element E . (a) Erosions F 0 nE,n = 0, 1 , 2 , 3 ; (b) openings of erosions (F 0nE) 0E . (c) Skeleton subsets S,, (d) Dilated skeleton subsets S, 0 nE. (e) Partial unions of skeleton subsets UN+kZn S, . (f) Partial unions of dilated skeleton subsets U N a k a n S k 0 kB. (From Maragos and Schafer, 1986; 0 1986 IEEE.)

.

i.e., the opening (smoothed version) of F by kR. The larger the size index k, the larger the degree of smoothing. Figure 4 shows a detailed description of the skeletal decomposition and reconstruction of an image. Thus, we can view the S,,as shape components. That is, skeleton components of small size indices n are associated with the lack of smoothness of the boundary of F, whereas skeleton components of large indices n are related to the bulky interior parts of F that are shaped similarly to nR. Libeskind-Hadas and Maragos (1987) used the information in the morphological skeleton transform in the following way to obtain the collage model parameters. First note (referring to the notation of Section VI1.A) that the collage theorem does not change if the collage map W is modified to contain a fixed condensation set C: m

W(F) =

c iu WJF) =

( 147)

1

The set C is set equal to the dilation of the skeleton subset SN corresponding to the largest scale index. This will model the bulky parts of the interior of an image F. (The origin of the plane is set equal to the mass centroid of SN .) Then, every major skeleton branch is associated with a map w i n

FRACTAL SIGNAL ANALYSIS USING MATHEMATICAL MORPHOLOGY

241

The translation vector ( t x i ,tYi)is taken as the vector of pixel coordinates of the skeleton branch point b. (The selection of the major skeleton branch points, which also determines the number of affine maps, is the only part of the algorithm done by visual inspection.) The rotation angle Bi is found as the angle that the skeleton branch forms with the horizontal. (Estimates of the rotation angle can also be obtained from fitting a line via least-squares to several known points on the specific branch.) Finally, the scaling factor is set equal to r = n / N , where n is the index of the skeleton subset containing b. This algorithm can model images F that exhibit some degree of self-similarity; i.e., when local details of F closely resemble F as a whole. Figure 5 shows an example of the application of morphological skeletonization to find the parameters of a collage model for the fractal

FIGURE5 . (a) Original binary image F (fractal Koch island). (b) Recursive process to construct the boundary of F. (c) Morphological skeleton of F (using a discrete disk for B ) . (d) Three of the six affine transformations of F. (From Libeskind-Hadas and Maragos, 1987; 0 1987 SPIE.)

242

PETROS MARAGOS

image of a Koch island. (The boundary of this 2D fractal set is generated through the recursive process of Fig. 5b and has similarity dimension log(4)/log(3).) Note that, due to the rotational symmetry of the Koch island with respect to its center, the rotation angles can also be set equal to zero in this example. Since the Koch island can be perfectly modeled as a collage of six affine maps (scaled by r = 1/3) and a large disk in the middle as condensation set, the attractor synthesized from the corresponding iterated function system is identical to the original image. VIII. CONCLUSIONS

In this chapter two important aspects of fractal signals have been analyzed using concepts and operations from morphological signal processing: the measurement of the fractal dimension of 1D and 2D signals and the modeling of binary images as attractors of iterated systems of affine maps. The major emphasis of the discussion was on the fractal dimension measurement. In this area a theoretical approach was presented for measuring the fractal dimension of arbitrary continuous-domain signals by using morphological erosion and dilation function operations to create covers around a signal’s graph at multiple scales. A related algorithm was also described for discrete-domain signals. This morphological covering approach unifies and extends the theoretical aspects and digital implementations of several other covering methods. Many empirical experiments on synthetic fractal signals indicate that the performance of this method is good since it yields average estimation errors in the order of 0 to 4%. It also has a low computational complexity, which is linear with respect to both the signal’s size of support and the maximum scale. It can be implemented very efficiently by using morphological filtering and can yield results that are invariant with respect to shifting the signal’s domain and affine scaling of its dynamic range. The latter advantage makes the morphological covering method more robust than the box-counting method in the digital case. An interesting area of future research could be the investigation of the performance of this method in the presence of noise. Modeling binary images with large degree of self-similarity as the attractors of iterated systems of affine maps is very promising for applications. However, efficient methods must be developed to find the parameters of these affine maps. A preliminary approach toward this goal was described based on the morphological skeleton transform. This approach is promising but it needs further work in automating the part for finding good branch points to place the collage patches; using connected skeletons may help finding such branch points. In addition, for the collage of an image F,

FRACTAL SIGNAL ANALYSIS USING MATHEMATICAL MORPHOLOGY 243

improved rotation angles and scaling factors for each affine map wi can be found by searching in a relatively small discretized space around the initial estimates found via morphological skeletonization and minimizing the area difference between F and (F 0 kB) U w i ( F ) ,where k is a smoothing scale at which the fractal details do not exist. An extension of these idea to graylevel images using gray-level skeletonization could also be interesting. Overall, the main characteristic of the morphological signal operators that enables them to be efficient in measuring the fractal dimension or finding the collage model parameters is their ability to extract information about the geometrical structure of signals at multiple scales. ACKNOWLEDGMENTS This chapter was written while the author was a visting professor at the National Technical University of Athens, Greece. The research work reported herein was supported by the U.S. National Science Foundation’s Presidential Young Investigator Award under the NSF Grant MIPS-8658150 with matching funds from Xerox, and in part by the National Science Foundation Grant MIP-91-20624.

REFERENCES Barnsley, M. F. (1986). “Fractal Interpolation,” Constr. Approx. 2, 303-329. Barnsley, M. F. (1988). Fractals Everywhere. Academic Press, Boston. Barnsley. M. F., and Demko, S. (1985). “Iterated Function Systems and the Global Construction of Fractals,” Proc. Royal SOC.London A-399,243-275. Barnsley, M. F., Ervin, V., Hardin, D., and Lancaster, J . (1986). “Solution of an Inverse Problem for Fractals and Other Sets,” Proc. National Acad. Sci. 83, 1975-1977. Bartle, R. G. (1976). The Elements of Real Analysis. Wiley, New York. Berry, M. V., and Lewis, Z. V. (1980). “On the Weierstrasse-Mandelbrot Fractal Function,” Proc. R. Soc. Lond. A 370, 459-484. Besicovitch, A. S. (1934). “On the Sum of Digits of Real Numbers Represented in the Dyadic System. (On Sets of Fractional Dimension II).” Math. Annalen 110, 321-329; “Sets of Fractional Dimension (IV): On Rational Approximation to Real Numbers,” J. London MaIh. SOC.9 , 126-131. Besicovitch, A. S., and Ursell, H. D. (1937). “Sets of Fractional Dimension (V): On Dimensional Numbers of Some Continuous Curves,” J. London Math. Soc. 12, 18-25. Blum, H. (1967). “A Transformation for Extracting New Descriptions of Shape.” In Models for the Perception of Speech and Visual Forms (W. Wathen-Dunn, ed.), MIT Press, Cambridge, Massachusetts. Bouligand, G. (1928). “Ensembles impropres et nombre dimensionnel,” Bull. Sci. Math. 11-52, 320-344, 361-376; Bull. Sci. Math. 11-53,185-192, 1929. Diaconis, P. M., and Shahshahani, M. (1986). “Products of Random Matrices and Computer Image Generation,” Contemporary Mathematics 50, 173-182.

244

PETROS MARAGOS

Dubuc, B., Zucker, S. W., Tricot, C., Quiniou, J. F., and Wehbi, D. (1988). “Evaluating the Fractal Dimension of Surfaces,” Tech. Report TR-CIM-87-19, Computer Vision & Robotics Lab. McGill University, Montreal, Canada, July. Dubuc, B., Quiniou, J. F., Roques-Carmes, C., Tricot, C., and Zucker, S. W. (1989). “Evaluating the Fractal Dimension of Profiles,” Phys. Rev. A 39, 1500-1512. Falconer, K , (1 990). Fractal Geometry: Mathematical Foundations and Applications. John Wiley & Sons, New York. Farmer, J. D., Ott, E., and Yorke, J . A. (1983). “The Dimension of Chaotic Attractors,” Physica 7D, 153-180. Hadwiger, H. (1957). Vorlesungen iiber Inhalt, Oberfache, und Isoperimetrie. Springer Verlag, Berlin. Haralick, R. M., Sternberg, S. R., and Zhuang, X. (1987). “Image Analysis Using Mathematical Morphology,” IEEE Trans. Pattern Anal. Mach. Intell. PAMI-9, 523-550. Hardin, D. P., and Massopust, P. R. (1986). “The Capacity for a Class of Fractals Functions,” Commun. Math. Phys. 105, 455-460. Hardy, G. H. (1916). “Weierstrass’s Non-Differentiable Function,” Trans. Amer. Math. SOC. 17, 322-323. Hausdorff, F. (1918). “Dimension and Ausseres Mass,” Math. Annalen 79, 157-179. Heijmans, H. J . A. M., and Ronse, C. (1990). “The Algebraic Basis of Mathematical Morphology. Part I: Dilations and Erosions,” Comput. Vision, Graphics, Image Process. 50, 245-295. Hutchinson, J. (1981). “Fractals and Self-Similarity,” Indiana Univ. Math. J. 30, 713-747. Jacquin, A. (1992). “Image Coding Based on a Fractal Theory of Iterated Contractive Image Transformations,” IEEE Trans. Image Processing 1 , 18-30. Kolmogorov, A. N., and Tihomirov, V. M. (1961). “Epsilon-Entropy and Epsilon-Capacity of Sets in Functional Spaces,” Uspekhi Matematicheskikh Nauk (N.S.) 14, 3-86, 1959. Translated in Trans. Amer. Math. SOC. (Series 2), 17, 277-364. Libeskind-Hadas, R., and Maragos, P. (1987). “Application of Iterated Function Systems and Skeletonization to Synthesis of Fractal Images.” In Visual Communications and Image Processing I1 (T. R. Hsing, ed.), Proc. SPIE 845, 276-284. Lantuejoul, C. (1980). “Skeletonization in Quantitative Metallography.” In Issues of Digital Image Processing (R. M. Haralick and J. C. Simon, eds.). Groningen, Sijthoff and Noordhoff, The Netherlands. Lundahl, T., Ohley, W. J., Kay, S. M., and Siffert, R. (1986). “Fractional Brownian Motion: A Maximum Likelihood Estimator and Its Application to Image Texture,” IEEE Trans. Med. h a g . MI-5, 152-160. Lundheim, L. (1992). “Fractal Signal Modeling for Source Coding,” Ph.D. Thesis, Norwegian Inst. Technology, Trondheim, Norway. Mallat, S. G. (1989). “A Theory for Multiresolution Signal Decomposition: The Wavelet Representation,” IEEE Trans. Pattern Analysis Machine Intelligence PAMI-11, 674-693. Mandelbrot, B. B. (1982). The Fractal Geometry of Nature. W. H. Freeman, New York. Mandelbrot, B. B. (1985). “Self-Affine Fractals and Fractal Dimension,” Phys. Scripta 32, 257-260. Mandelbrot, B. B., and van Ness, J. (1968). “Fractional Brownian Motion, Fractional Noise and Applications,” SIAM Review 10(4), 422-437. Mandelbrot, B. B., and Wallis, J. R. (1969). “Computer Experiments with Fractional Brownian Motion. Parts 1-3,” Water Resources Research 5, 228-267. Maragos, P. (1991). “Fractal Aspects of Speech Signals: Dimension and Interpolation.” In Proc. IEEE Int’l Conf. Acoust., Speech, and Signal Processing, Toronto, May.

FRACTAL SIGNAL ANALYSIS USING MATHEMATICAL MORPHOLOGY 245 Maragos, P., and Schafer, R. W. (1986). “Morphological Skeleton Representation and Coding of Binary Images,” IEEE Trans. Acoust., Speech, Signal Process ASSP-34, 1228-1 244.

Maragos, P., and Schafer, R. W. (1987). “Morphological Filters-Part I: Their Set-Theoretic Analysis and Relations to Linear Shift-Invariant Filters,” IEEE Trans. Acoust. Speech, Signal Processing ASSP-35, 1153-1 169. Maragos, P., and Schafer, R. W . (1990). “Morphological Systems for Multidimensional Signal Processing,” Proc. IEEE 78, 690-710. Maragos, P., and Sun, F.-K. (1991). “Measuring the Fractal Dimension of Signals: Morphological Covers and Iterative Optimization,” Technical Report 91-14, Harvard Robotics Lab., Harvard University. Also in IEEE Trans. Signal Processing, Jan. 1993. Mazel, D. S., and Hayes, M. H., I I I (1991). “Hidden-Variable Fractal Interpolation of Discrete Sequences.” In Proc. IEEE Int’l Conf, Acoust., Speech, and Signal Processing, Toronto, May 1991. McGowan, R. S. (1989). “An Aeroacoustics Approach to Phonation,” J. Acoust. SOC.A m . 83(2), 696-704.

McMullen, C . (1984). “The Hausdorff Dimension of General Sierpinski Carpets,” Nagoya Math. J . 96, 1-9. Minkowski, H. (1901). “Uber die Begriffe Lange, Oberflache und Volumen,” Jahresber. Deutch. Mathematikerverein 9, 115-121. Minkowski, H. (1903). “Volumen und Oberflache,” Math. Annalen 57, 447-495. Mott-Smith, J. C. (1970). “Medical Axis Transformations.” In Picture Processing and Psychopictorics (B. S . Lipkin and A. Rosenfeld, eds.),. Academic Press, New York. Peleg, S., Naor, J., Hartley, R., and Avnir, D. (1984). “Multiple Resolution Texture Analysis and Classification,” IEEE Trans. Pattern, Anal, Mach. Intell. PAMI-6, 518-523. Peli, T., Tom, V., and Lee, B. (1989). “Multi-Scale Fractal and Correlation Signatures for Image Screening and Natural Clutter Suppression.” In Proc. SPIE, Vol. 1199: Visual Communications and Image Processing IV, pp. 402-415. Pentland, A. P. (1984). “Fractal-Based Description of Natural Scenes,” IEEE Trans. Pattern Anal. Mach. Intell. PAMI-6, 661-614. Pickover, C., and Khorasani, A. (1986). “Fractal Characterization of Speech Waveform Graphs,” Comp. & Graphics 10, OOO-000. Rosenfeld, A., and Kak, A. C. (1982). Digital Picture Processing, vols 1 and 2. Academic Press, New York. Serra, J. (1982). Image Analysis and Mathematical Morphology. Academic Press, New York. Serra, J., and Vincent, L. (1992). “An Overview of Morphological Filtering,” Circuits. Systems and Signal Processing 11(1), 47-108. Stein, M. C. (1987). “Fractal Image Models and Object Detection.” In Visual Communications and Image Processing I1 (T. R. Hsing, ed.), Proc. SPIE, Vol. 845. Sternberg, S . R. (1986). “Grayscale Morphology,“ Comput. Vision, Graph., Image Proc. 35, 333-355. Super, B. J., and Bovik. A. C. (1991). “Localized Measurement of Image Fractal Dimension Using Gabor Filters,” J. Visual Commun. and Image Represent. 2 , 114-128. Teager, H. M., and Teager, S. M. (1989). “Evidence for Nonlinear Production Mechanisms in the Vocal Tract,” Proc. NATO ASI on Speech Production and Speech Modelling, France. Tewfik, A. H., and Deriche, M. (1991). “Maximum Likelihood Estimation of the Fractal Dimensions of Stochastic Fractals and Cramer-Rao Bounds.” In Proc. IEEE Int’l Conf. Acoust., Speech, and Signal Processing, Toronto, May.

246

PETROS MARAGOS

Thomas, T. J . (1986). “A Finite Element Model of Fluid Flow in the Vocal Tract,” Comput. Speech &Language 1, 131-151. Tricot, C., Quiniou, J. F., Wehbi, D., Roques-Carmes, C., and Dubuc, B. (1988). “Evaluation de la dimension fractale d’un graphe,” Revue Phys. Appl. 23, 111-124. Tritton, D. J. (1988). Physical Fluid Dynamics, Oxford University Press, Oxford. Voss, R. F. (1989). “Fractals in Nature: From Characterization to Simulation.” In The Science of Fractual Images (H.-0. Peitgen and D. Saupe, eds.), Springer-Verlag. Wornell, G. W., and Oppenheim, A. V. (1990). “Fractal Signal Modeling and Processing Using Wavelets.” In Proc. 1990 Digital Signal Processing Workshop. Mohonk, New Paltz, New York.

ADVANCES IN ELECTRONICS AND ELECTRON PHYSICS, VOL. 88

Fuzzy Set Theoretic Tools for Image Analysis SANKAR K. PAL Machine Intelligence Unit, Indian Statistical Institute, Calcutta, India

I. Introduction . . . . . . . . . . . . . . . . . . . . 11. Uncertainties in a Recognition System and Relevance of Fuzzy Set Theory 111. Image Ambiguity and Uncertainty Measures . . . . . . . . . . A. Grayness Ambiguity Measures . . . . . . . . . . . . . B. Spatial Ambiguity Measures Based on Fuzzy Geometry of Image . . IV. Flexibility in Membership Functions . . . . . . . . . . . . A. Bound Functions . . . . . . . . . . . . . . . . . B. Spectral Fuzzy Sets . . . . . . . . . . . . . . . . . V. Some Examples of Fuzzy Image-Processing Operations . . . . . . A. Threshold Selection (Fuzzy Segmentation) . . . . . . . . . B. Contour Detection . . . . . . . . . . . . . . . . . C. Optimum Enhancement Operator Selection . . . . . . . . . D. Fuzzy Skeleton Extraction and FMAT . . . . . . . . . . . V1. Feature/Knowledge Acquisition, Matching, and Recognition . . . . VII. Fusion of Fuzzy Sets and Neural Networks: Neuro-Fuzzy Approach . . VIII. Use of Genetic Algorithms . . . . . . . . . . . . . . . IX. Discussion . . . . . . . . . . . . . . . . . . . . . Acknowledgments . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . .

. . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . .

. . . . . . . .

241 249 25 1 252 256 260 26 1 262 264 264 261 269 269 212 218 288 290 29 1 292

I. INTRODUCTION

Pattern recognition and machine learning form a major area of research and development activity that encompasses the processing of pictorial and other nonnumerical information obtained from interaction among science, technology, and society. The second motivation for this spurt of activity in this field is the need for the people to communicate with the computing machines in their natural mode of communication. The third and most important motivation is that the scientists are also concerned with the idea of designing and making automata that can carry out certain tasks as we human beings do. The most salient outcome of these is the concept of fifth-generation computing systems. Machine recognition of patterns (Tou and Gonzalez, 1974; Duda and Hart, 1973) can be viewed as a two-fold task, consisting of learning the invariant and common properties of a set of samples characterizing a class and of deciding that a new sample is a possible member of the class by 241

Copyright 0 1994 by Academic Press, Inc. All rights of reproduction in any form reserved. ISBN 0-12-014730-0

248

SANKAR K. PAL

noting that it has properties common to those of the set of samples. Therefore, the task of pattern recognition by a computer can be described as a transformation from the measurement space M to the feature space F and finally to the decision space D. When the input pattern is a gray-tone image, the measurement space involves some processing tasks such as enhancement, filtering, noise reduction, segmentation, contour extraction, and skeleton extraction, in order to extract salient features from the image pattern. This is what is basically known as image processing. The ultimate aim is to make its understanding, recognition, and interpretation from the processed information available from the image pattern. Such a complete image recognition/ interpretation system is called a vision system, which may be viewed as consisting of low, mid, and high levels. In a pattern-recognition or vision system, the uncertainty can arise at any phase of the aforesaid tasks resulting from the incomplete or imprecise input information, the ambiguity/vagueness in input image, the ill-defined and/or overlapping boundaries among the classes or regions, and the indefiniteness in defining/extracting features and relations among them. Any decision taken at a particular level will have an impact on all higherlevel activities. It is therefore required for a recognition system to have sufficient provision for representing these uncertainties involved at every stage, so that the ultimate output (results) of the system can be associated with the least uncertainty (and not be affected or biased very much by the earlier or lower-level decisions). This chapter describes various fuzzy set theoretic tools and explores their effectiveness in representing/describing various uncertainties that might arise in an image-recognition system and the ways these can be managed in making a decision. Some examples of uncertainties that arise often in the process of recognizing a pattern are given in Section 11. Section 111 provides a definition of image and describes various fuzzy set theoretic tools for measuring information on grayness ambiguity and spatial ambiguity in an image. Concepts of bound functions and spectral fuzzy sets for handling uncertainties in membership functions are also discussed in Section IV. Their applications to low-level vision operations (e.g., segmentation, skeleton extraction, and edge detection), whose outputs are crucial and responsible for the overall performance of a vision system, are then presented in Section V for demonstrating the effectiveness of these tools in managing uncertainties by providing both soft and hard decisions. Their usefulness in providing quantitative indices for autonomous operations is also explained. Section VI describes the issues of feature/primitive extraction, knowledge acquisition and syntactic classification, and the features of DempsterShafer theory and rough set theory in this context. An application of the

FUZZY SET THEORETIC TOOLS FOR IMAGE ANALYSIS

249

multivalued recognition system for detecting curved structures from remotely sensed image is also described. Some of the recent attempts on fusion of the theories of fuzzy sets and neural networks for efficient handling of uncertainty (in the sense of parallel processing, robustness, and overall performance) are described in Section VII. The concept of genetic algorithms and its possible use are explained in Section VIII. 11. UNCERTAINTIES IN A RECOGNITION SYSTEM AND

RELEVANCE OF FUZZYSETTHEORY Some of the uncertainties that one encounters often while designing a pattern-recognition or vision (Gonzalez and Wintz, 1987; Rosenfeld and Kak, 1982) system will be explained in this section. Let us consider, first of all, the problem of processing and analyzing a gray-tone image pattern. A gray-tone image possesses some ambiguity within the pixels due to the possible multivalued levels of brightness. This pattern indeterminacy is due to inherent vagueness rather than randomness. The conventional approach to image analysis and recognition consists of segmenting (hard partitioning) the image space into meaningful regions, extracting its different features (e.g., edges, skeletons, centroid of an object), computing the various properties of and relationships among the regions, and interpreting and/or classifying the image. Since the regions in an image are not always crisply defined, uncertainty can arise at every phase of the aforesaid tasks. Any decision taken at a particular level will have an impact on all higher-level activities. Therefore, a recognition system (or vision system) should have sufficient provision for representing the uncertainties involved at every stage, i.e., in defining image regions, its features, and relations among them, and in their matching, so that it retains as much as possible the information content of the original input image for making a decision at the highest level. The ultimate output (result) of the system will then be associated with least uncertainty (and unlike conventional systems it will not be biased or affected very much by the lower level decisions), For example, consider the problem of object extraction from a scene. Now, the question is, how can someone define exactly the target or object region in a scene when its boundary is ill-defined? Any hard thresholding made for its extraction will propagate the associated uncertainty to the following stages, and this might affect its feature analysis and recognition. Similar is the case with the tasks of contour extraction and skeleton extraction of a region. From the aforesaid discussion, it becomes therefore convenient, natural, and appropriate to avoid committing ourselves to a specific (hard) decision

250

SANKAR K. PAL

(e.g., segmentation/thresholding, edge detection, and skeletonization) by allowing the segments or skeletons or contours to be fuzzy subsets of the image; the subsets being characterized by the possibility (degree) of a pixel belonging to them. Similarly, for describing and interpreting ill-defined structural information in a pattern, it is natural to define primitives (such as line, corner, curve) and relations among them using labels of fuzzy sets. For example, primitives that d o not lend themselves to precise definition may be defined in terms of arcs with varying grades of membership from 0 to 1 representing its belonging to more than one class. The production rules of a grammar may similarly be fuzzified to account for the fuzziness in physical relation among the primitives, thereby increasing the generative power of a grammar for syntactic recognition (Fu, 1982) of a pattern. The incertitude in an image pattern may be explained in terms of grayness ambiguity or spatial (geometrical) ambiguity or both. Grayness ambiguity means indefiniteness in deciding a pixel as white or black. Spatial ambiguity refers to indefiniteness in shape and geometry (e.g., in defining centroid, sharp edge, perfect focussing, and so on) of a region. There is another kind of uncertainty that may arise from the subjective judgment of an operator in defining the grades of membership of the object regions. This has been explained in Section IV in terms of uncertainty in membership function. Let us now consider the case of a decision theoretic approach to pattern classification. With the conventional probabilistic and deterministic classifiers (Duda and Hart, 1973; Tou and Gonzalez, 1974), the features characterizing the input patterns are considered to be quantitative (numerals) in nature. The patterns having imprecise or incomplete information are usually ignored or discarded from their designing and testing processes. The impreciseness (or ambiguity) may arise from various reasons. For example, instrumental error or noise corruption in the experiment may lead to partial or partially reliable information available on a feature measurement F,such as, F is about 500, say, or F is between 400 and 500, say. Again, in some cases the expense incurred in extracting the exact value of a feature may be high, or it may be difficult to decide on the actual salient features to be extracted. On the other hand, it may become convenient to use the linguistic variables and hedges, e.g., small, medium, high, very, more or less, and the like, in order to describe the feature information (e.g., F is very small). In such cases, it is not appropriate to give exact representation to uncertain feature data. Rather, it is reasonable to represent uncertain feature information by fuzzy subsets. Again, the uncertainty in classification or clustering of patterns may arise from the overlapping nature of the various classes. This overlapping may result from fuzziness or randomness. In the conventional classification

FUZZY SET THEORETIC TOOLS FOR IMAGE ANALYSIS

25 1

technique, it is usually assumed that a pattern may belong to only one class, which is not necessarily true. A pattern may have degrees of membership in more than one class. It is therefore necessary to convey this information while classifying a pattern or clustering a data set. Similarly, consider the problem of determining the boundary or shape of a class from its sampled points or prototypes. There are various approaches (Murthy, 1988; Edelsbrunner er al., 1983; Tousant, 1980) described in the literature that attempt to provide an exact shape of the pattern class by determining the boundary such that it contains (passes through) some of the sample points. This need not be true. It is necessary to extend the boundaries to some extent to represent the possible uncovered portions by the sampled points. The extended portion should have lower possibility to be in the class than the portions explicitly highlighted by the sample points. The size of the extended regions should also decrease with the increase of the number of sample points. This leads one to define a multivalued or fuzzy (with continuum grade of belonging) boundary of a pattern class (Mandal er al., 1992b). In the following section we will be explaining various fuzzy-set theoretical tools for image analysis (which were developed based on the realization that many of the basic concepts in pattern analysis, e.g., the concept of an edge or a corner, do not lend themselves to precise definition) and the way of using them for handling uncertainties in the process of recognizing an image pattern. 111. IMAGE AMBIGUITY AND UNCERTAINTY MEASURES

An L level image X(M x N ) can be considered as an array of fuzzy singletons, each having a value of membership denoting its degree of possessing some property (e.g., brightness, darkness, edginess, blurredness, texture). In the notation of fuzzy sets one may therefore write that

x = (px(x,,):rn

= 1 , 2 ,..., M ; n = 1 , 2,..., N )

(1)

where px(x,,,) denotes the grade of possessing such a property p by the ( m , n)th pixel. This property p of an image may be defined using global information, local information, positional information, or a combination of them depending on the problem. Again, the aforesaid information can be used in a number of ways (in their various functional forms), depending on individual’s opinion and/or the problem to hand, to define a requisite membership function for an image property. Basic principles and operations of image processing and pattern recognition in the light of fuzzy set theory are available in (Pal and Dutta Majumder, 1986).

252

SANKAR K . PAL

We will be explaining in this section the various image information measures (arising from both fuzziness and randomness) and tools, and their relevance for the management of uncertainty in different operations for processing and analysis. These are classified mainly in two groups, namely grayness ambiguity/uncertainty and spatial arnbiguityhncertainty, A , Grayness Ambiguity Measures

The definitions of some of the measures that were formulated to represent grayness ambiguity in an image X with dimension M x N and levels L (based on individual pixel as well as a collection of pixels) are listed below. 1. rth Order Fuzzy Entropy

H'(X)

=

(-1jk)

c Icl(sf)loglP(sf)l + (1

- P(Sf)I log(1 - fl(sf)II

i

i

= 1,2,

...,k

(2)

where sf denotes the ith combination (sequence) of r pixels in X ;k is the number of such sequences; and p(sf) denotes the degree to which the combination si, as a whole, possesses some image property p . 2. Hybrid Entropy

Hhy(x) = -Pwlog E, - Pb log Eb

(3)

with

Ew = (1/MM C m

Eb

= (l/MN)

Cn Pmnex~(1- p m n )

Cm Cn (1 - prnn)eXP(pmn)

(4)

m = 1 , 2,..., M ; n = 1 , 2,..., N Here pmn denotes the degree of whiteness of the (m,n)th pixel. Pw and Pb denote probability of occurrences of white (p,, = 1) and black (pmn= 0) pixels respectively; and E, and Ebdenote the average likeliness (possibility) of interpreting a pixel as white and black respectively. 3. Correlation

= 1

ifX,+X2=0

FUZZY SET THEORETIC TOOLS FOR IMAGE ANALYSIS

253

m = 1,2 ,..., M ; n = 1,2 ,..., N Here pYmn and pUlrnn denote the degree of possessing the properties pl and p2 respectively by the (m,n)th pixel and C(pl,p2) denotes the correlation between two such properties pl and p2 (defined over the same domain). These expressions (Eqs. 2 through 6) are the versions extended to the two-dimensional image plane from those defined (Murthy et al., 1985; Pal and Pal, 1992a) for a fuzzy set. H‘(X) gives a measure of the average amount of difficulty in taking a decision whether any subset of pixels of size r possesses an image property or not. Note that, no probabilistic concept is needed to define it. If r = 1, H‘(X) reduces to (nonnormalized) entropy as defined by De Luca and Termini (1972). Hhy(X),on the other hand, represents an amount of difficulty in deciding whether a pixel possesses a certain property pmn or not by making a prevision on its probability of occurrence. (It is assumed here that the fuzziness occurs because of the transformation of the complete white (0) and black pixels (1) through a degradation process; thereby modifying their values to lie in the intervals [0,0.5] and [0.5,1] respectively). Therefore, if pmn denotes the fuzzy set “object region” then the amount of ambiguity in deciding x,,,, a member of object region is conveyed by the term hybrid entropy depending on its probability of occurrence. In the absence of fuzziness (i.e., with exact defuzzification of the gray pixels to their respective black or white version), Hhy reduces to the two-state classical entropy of Shannon (1948), the states being black and white. Since a fuzzy set is a generalized version of an ordinary set, the entropy of a fuzzy set deserves to be a generalized version of classical entropy by taking into account not only the fuzziness of the set but also the underlying probability structure. In that respect, Hhy can be regarded as a generalized entropy such that classical entropy becomes its special case when fuzziness is properly removed. Note that the Eqs. (2) and (3) are defined using the concept of logarithmic gain function. Similar expressions using exponential gain function, i.e., defining the entropy of an n-state system have been given by Pal and Pal (1989a, 1991a,b; 1992a,b).

H=

pie’-pi, i

i = 1,2,

..., n

(7)

all these terms, which given an idea of indefiniteness or fuzziness of an image, may be regarded as the measures of average intrinsic information

254

SANKAR K. PAL

that is received when one has to make a decision (as in pattern analysis) in order to classify the ensembles of patterns described by a fuzzy set. H'(X) has the following properties: Pr 1: H' attains a maximum if pi = 0.5 for all i. Pr 2: H' attains a minimum if pi = 0 or 1 for all i. Pr 3: H' 2 H*', where H*' is the rth-order entropy of a sharpened version of the fuzzy set (or an image). Pr 4: H' is, in general, not equal to H',where H' is the rth-order entropy of the complement set, Pr 5: H' I Hr+' when all pi E [0.5, 11. H' 2 H'+' when all pi E [0,0.5]. The sharpened or intensified version of X is such that ~ . r * ( x m n )2

p.r(xrnn)

if~x(xrnn12 0.5

and

(8) ~ x 4 x m n )5 px(xrnn)

if M X r n n )

5

0.5

When r = 1, the property 4 is valid only with the equal sign. Property 5 (which does not arise for r = 1) implies that H' is a monotonically nonincreasing function of r for pi E [0,0.5] and a monotonically nondecreasing function of r for pi E [ O S , 11 (when the "min" operator has been used to get the group membership value). When all pi values are the same, H ' ( X ) = H 2 ( X ) = = H'(X). This is because the difficulty in taking a decision regarding possession of a property on an individual is the same as that of a group selected therefrom. The value of H' would, of course, be dependent on the pi values. Again, the higher the similarity among singletons (supports) the quicker is the convergence to the limiting value of H'. Based on this observation, an index of similarity of supports of a fuzzy set may be defined as S = H 1 / H 2 (when H 2 = 0, H' is also zero and S is taken as 1). Obviously, when p i E I0.5, I] and the min operator is used to assign the degree of possession of the property by a collection of supports, S will lie in [0, I] as H' s H'". Similarly, when pi E [0,0.5], S may be defined as H 2 / H ' so that S lies in [O, 11. The higher the value of S, the more alike (similar) are the supports of the fuzzy set with respect to the fuzzy property p . This index of similarity can therefore be regarded as a measure of the degree to which the members of a fuzzy set are alike. Therefore, the value of first-order fuzzy entropy (H') can only indicate whether the fuzziness in a set is low or high. In addition to this, the value of H', r > 1 also enables one to infer whether the fuzzy set contains similar

-

FUZZY SET THEORETIC TOOLS FOR IMAGE ANALYSIS

255

supports (or elements) or not. The similarity index thus defined can be successfully used for measuring interclass and intraclass ambiguity (i.e., class homogeneity and contrast) in pattern recognition and image processing problems. H ' ( X ) is regarded as a measure of the average amount of information (about the gray levels of pixels) that has been lost by transforming the classical pattern (two-tone) into a fuzzy (gray) pattern X.Further details on this measure with respect t o image processing problems are available in Pal and King (1981a, b), Pal (1982), and Pal and Dutta Majumder (1986). It is to be noted that H ' ( X ) reduces to zero whenever ,urn,,is made 0 or 1 for all (m,n), no matter whether the resulting defuzzification (or transforming process) is correct or not. In the following discussion it will be clear how Hhy takes care of this situation. Let us now discuss some of the properties of Hhy(X).In the absence of fuzziness when MNPb pixels become completely black (pmn= 0) and MNP, pixels become completely white ( P , ~ , ,= l), then E , = P,,,,Eb = Pb and Hhy boils down to the two state classical entropy Hc

=

- P w log P, - Pb log Pb,

(9)

the states being black and white. Thus Hhyreduces to H, only when a proper defuzzification process is applied to detect (restore) the pixels. IH,, - H,I can therefore be treated as an objective function for enhancement and noise reduction. The lower the difference, the lesser is the fuzziness associated with the individual symbol and the higher will be the accuracy in classifying them as their original value (white or black). (This property is lacking with the H ' ( X ) measure and the measure of Xie and Bedrosian (1984), which always reduces to zero or some constant value irrespective of the defuzzification process). In other words, IHhy - H,l represents an amount of information that was lost by transforming a two-tone image to a gray tone. For a given P, and Pb, (P, + Pb = 1 , 0 IP,, Pb Il), of all possible defuzzifications, the proper defuzzification of the image is the one for which Hh,, is minimum. If p,,,,, = 0.5 for all (m,n) then E, = Eb and

(10)

Hhy = -log(0.5 exp 0.5) i.e., H,,,, takes a constant value and becomes independent of P, and pb. This is logical in the sense that the machine is unable to make a decision on the pixels since all p,, values are 0.5. Let us now consider the measure correlation C ( p I ,p2) of Eq. (5). This has the following properties.

256

SANKAR K. PAL

(a) If for higher values of pl(x), p2(x) takes higher values and the converse is also true, then C(pl ,p2)must be very high. (b) If with increase of x, both p1and p2 increase, then C ( p l ,p2) > 0. (c) If with increase of x, p1 increases and p2 decreases or vice versa then C ( P ~ , CcC0. ~)

(4 C(PIYP1) =

1.

(el C(P1, PI) 2 C(Pl P2). (f) C(P1 1 - PI) = -1. (g) C(p1 P2) = C(P2 Pl). (h) - 1 5 C(PI,P2) 1. (i) C(pl ,p2) = -C(1 - p1,~ 2 ) . ci) C(PIYP2) = C(1 - P l , 1 - P2). Correlation of an image indicates the characteristics of relative variation between its two properties p l and p 2 , Based on these characteristics, bound functions are defined as shown in Section 1V.A.If one of these properties is considered to be the nearest crisp (two-tone) property of the other (say, p1 = 1 if p2 > 0.5, and p1 = 0 if p2 5 0.5), then C(pl,p2)lies in [0, 11. In other words, if p2 denotes a bright-image plane of an image X having crossover point at s, say, and is dependent only on gray level, then p1 represents its closest two-tone version threshold at s. Therefore, by varying s of the p2 plane, an optimum version of p2 (i.e., optimum fuzzy segmented version of the image) can be obtained for which correlation is maximum. Various segmentation algorithms based on transitional correlation and within class correlation have been derived (Pal and Ghosh, 1992a) using the co-occurrence matrix, Recently fuzzy divergence has been introduced by Bhandari et al. (1992, 1993) for measuring grayness ambiguity. Before leaving this section, it should be mentioned that there have been several attempts recently made on image information and uncertainty measures (Pal, 1992a) based on classical entropy and gray-level statistics. These include conditional entropy, hybrid entropy, higher-order entropy, and positional entropy (Pal and Pal, 1989b, 1991a, 1992a,b). Y

Y

Y

Y

=

B. Spatial Ambiguity Measures Based on Fuzzy Geometry of Image

Many of the basic geometric properties of and relationships among regions have been generalized to fuzzy subsets. Such an extension, called fuzzy geometry (Rosenfeld, 1984; Pal and Rosenfeld, 1988, 1991; Pal and Ghosh, 1990, 1992b), includes the topological concept of connectedness, adjacency and surroundedness, convexity, area, perimeter, compactness, height, width, length, breadth, index of area coverage, major axis, minor axis, diameter, extent, elongatedness, adjacency, and degree of adjacency. Some

FUZZY SET THEORETIC TOOLS FOR IMAGE ANALYSIS

257

of these geometrical properties of a fuzzy digital image subset (characterized by piecewise constant membership function p x (xmJ or simply p ) are listed below with illustrations. These may be viewed as providing measures of ambiguity in the geometry (spatial domain) of an image. 1. Compactness (Rosenfeld, 1984)

where = &&

(12)

and P(P) =

Ci,j,k

I&) - W)llA(i,j, k)l.

Here, a ( p ) denotes area of p , and p ( p ) , the perimeter of p, is just the weighted sum of the lengths of the arcs A ( i , j , k)along which the region p ( i ) and p ( j ) meet, weighted by the absolute difference of these values. Physically, compactness means the fraction of maximum area (that can be encircled by the perimeter) actually occupied by the object. In the nonfuzzy case, the value of compactness is maximum for a circle and is equal to 1/4n. In the case of the fuzzy disk, where the membership value is only dependent on its distance from the center, this compactness value is 2 1/4z. Of all possible fuzzy disks, compactness is therefore minimum for its crisp version.

Example 1. Let p be of the form 0.2 0.4 0.3 0.2 0.7 0.6 0.6 0.5 0.6

1

Then area a(p) = 4.1, perimeter p ( p ) = 2.3, and comp(p) = 0.775. 2 . Height and Width (Rosenfeld, 1984)

and

So, height/width of a digital picture is the sum of the maximum membership values of each row/column. For the fuzzy subset p of Example 1, height is h ( p ) = 0.4 + 0.7 + 0.6 = 1.7, and width is w(p) = 0.6 + 0.7 + 0.6 = 1.9.

258

SANKAR K. PAL

3 . Length and Breadth (Pal and Ghosh, 1990, 1992b)

and

The length/breadth of an image fuzzy subset gives its longest expansion in the column/row direction. If p is crisp, pmn = 0 or 1; then lengthjbreadth is the maximum number of pixels in a column/row. Comparing Eqs, ( 1 5 ) and (16) with (13) and (14), we notice that the lengthlbreadth takes the summation of the entries in a column/row first and then maximizes over different columnshows, whereas the height/width first maximizes the entries in a column/row and then sums over different columnshows. For the fuzzy subset p in Example 1, / ( p ) = 0.4 + 0.7 + 0.5 = 1.6, and breadth is b ( p ) = 0.6 + 0.5 + 0.6 = 1.7. 4. Index ofArea Coverage (Pal and Ghosh, 1990, 1992b)

In the nonfuzzy case, the index of area coverage (IOAC) has value of one for a rectangle (placed along the axes of measurement). For a circle this value is nr2/(2r* 2r) = d 4 . ZOA C of a fuzzy image represents the fraction (which may be improper also) of the maximum area (that can be covered by the length and breadth of the image) actually covered by the image. For the fuzzy subset p of example 1, the maximum area that can be covered by its length and breadth is 1.6 x 1.7 = 2.72, whereas the actual area is 4.1, so the IOAC = 4.1/2.72 = 1.51. Note the difference between IOAC(p) and comp(p). Again, note the following relationships

I(X)/h(X)I1 and

b ( X ) / w ( X )I1. When equality holds for Eq. (18) the object is either vertically or horizontally oriented.

5 . Major Axis (Pal and Ghosh, 1992b) Find the length of the object. Now rotate the axes through an angle 8, 8 varying between 0" and 90". The angle for which length is maximum

FUZZY SET THEORETIC TOOLS FOR IMAGE ANALYSIS

259

is said to be the angle of inclination of the object (with the vertical). The corresponding axis along which the length is maximum is said to be the major axis. The length along the major axis denotes the expansion of the object. 6 . Minor Axis (Pal and Ghosh, 1992b)

The axis perpendicular to major axis, for which breadth is maximum, is defined as the minor axis of the object. 7. Center of Gravity (Pal and Ghosh, 1992b) The center of gravity (CG) of an object can be defined in various ways. Two such definitions are given here. (a) CG of an object can be defined as the point of intersection of the major and the minor axes. (b) Take any pixel as the center. Take a neighborhood of radius r. Find the energy (area) of the circle. Now shift the center of the circle over all the pixels of the object. The center for which the energy is maximum is defined as the CG. If there is any tie, then increase the radius and obtain the CG.

For the fuzzy subset p of Example 1, length is [ ( p ) = 1.6, and breadth is b ( p ) = 1.7. Now if we rotate the object by 45" then its length is 41) = 0.6 + 0.7 + 0.6 = 1.9. Hence the object is inclined at an angle of 45" with vertical axis. So by major axis of this image we mean the axis inclined at an angle of 45" with the vertical. Similarly the minor axis of this object is inclined at an angle of 45" with horizontal. Trivially the CG of this object is through the pixel having membership 0.7.

8. Density (Pal and Ghosh, 1992b)

where N denotes the number of supports of p (i.e., summation is taken over pixels for which p is nonzero). The maximum value of density is one, and this value occurs only for a nonfuzzy case. Density can be used for finding the CG of an image. If we break the image into different regions, then the region having the maximum density may be regarded as containing the CG.

260

SANKAR K. PAL

9. Degree ofAdjacency (Pal and Ghosh, 1992b)

The degree to which two regions S and T of an image are adjacent is defined as

Here d(p) is the shortest distance between p and q, 4 is a border pixel (BP) of T, and p is a border pixel of S. The other symbols have the same meaning as in the previous discussion. The degree of adjacency of two regions is maximum (= 1) only when they are physically adjacent, i.e., d ( p ) = 0, and their membership values are also equal, i.e., p ( p ) = dq). If two regions are physically adjacent then their degree of adjacency is determined only by the difference of their membership values. Similarly, if the membership values of two regions are equal their degree of adjacency is determined by their physical distance only. The readers may note the difference between Eq. (20) and the adjacency definition given in Rosenfeld (1984).

IV. FLEXIBILITY IN MEMBERSHIP FUNCTIONS Since the theory of fuzzy sets is a generalization of the classical set theory, it has greater flexibility to capture faithfully the various aspects of incompleteness or imperfection (i.e., deficiencies) in information of a situation. The flexibility of fuzzy-set theory is associated with the elasticity property of the concept of its membership function. The grade of membership is a measure of the compatibility of an object with the concept represented by a fuzzy set. The higher the value of membership, the lesser will be the amount (or extent) to which the concept represented by a set needs to be stretched to fit an object. Since the grade of membership is both subjective and dependent on context, some difficulty of adjudging the membership value still remains. In other words, the problem is how to assess the membership of an element to a set. This is an issue where opinions vary, giving rise to uncertainties. Two operators, namely bound functions (Murthy and Pal, 1992) and spectralfuzzy sets (Pal and Das Gupta, 1992) have recently been defined to analyze the flexibility and uncertainty in membership function evaluation. These are explained below along with their significance in image analysis and pattern-recognition problems.

FUZZY SET THEORETIC TOOLS FOR IMAGE ANALYSIS

26 1

A . Bound Functions

Consider, for example, a fuzzy set “tall.” This is represented by an S-type function that is a nondecreasing function of height. Now, the question is, can any such nondecreasing function be taken to represent the above fuzzy set? Intuitively, the answer is “no.” Bounds for such an S-type membership function p have recently been reported (Murthy and Pal, 1992) based on the properties of fuzzy correlation (Murthy et al., 1985). The correlation measure between two membership functions pl and p, relates the variation in their functional values. The main properties on which the correlation was formulated are as follows:

P I : If for higher values of p l , pz takes higher values, and for lower values of p , , p2 also takes lower values, then C ( p l ,p2) > 0. P2: If p , increases and p2 increases then C ( p , ,p2) > 0. P3: If p1 increases and p2 decreases then C(pl,p2) < 0. P2 and P3 should not be considered in isolation of P , . Had this been the case, one can cite several examples when both pl and p2 increase, but C(pl,pz) < 0; and p , increases and p, decreases but C(pl, p2) > 0. Subsequently, the types of membership functions that should preferably be avoided in representing fuzzy sets are categorized with the help of correlation. Bound functions h , and h, are accordingly derived in order to restrict the variation in the p function. They are

= 1,

where

E

l - & I X l l

. ,

= 0.25. The bounds for membership function p are such that

h,(x) 5 p(x) Ih,(x)

for x E [O, 11

For x belonging to any arbitrary interval, the bound functions will be changed proportionately. For h , I p 5 h 2 , C(h,, h,) 1 0 , C(h,,p ) 2 0 and C(h,,p ) 2 0. The function p lying in between k, and h2 does not have most of its variation concentrated (1) in a very small interval, (2) toward one of the end points of the interval under consideration, and (3) toward both the end points of the interval under consideration. In other words, the membership function p of a fuzzy set should not have, in any interval of the domain, an abrupt change from, nonmembership to membership or viceversa, because this can make the representation of a fuzzy set crisp.

262

SANKAR K. PAL

It is to be noted that Zadeh’s standard S function (Pal and Dutta Majumder, 1986; Zadeh et al., 1975) satisfies these bounds. The significance of the bound functions in selecting an S-type function p for an image-segmentation problem has been reported in detail in (Murthy and Pal, 1990). It has been shown that for detecting a minimum in the valley region of a histogram, the window length w of the function p : [0, w] -, [0, I ] should be less than the distance between two peaks around that valley region. The ability to make the fuzzy set theoretic approach flexible and robust will be demonstrated further in Section V.

B. Spectral Fuzw Sets The concept of spectral fuzzy sets is used where, instead of a single unique membership function, a set of functions reflecting various opinions on membership elements is available, so that each membership grade is attached to one of these functions. By giving due respect to all the opinions available for further processing, it reduces the difficulty (ambiguity) in selecting a single function. A spectral fuzzy subset F having n supports is characterized by a set or a band (spectrum) of r membership functions (reflecting r opinions) and may be represented as F =

u j

I

Upb(xj)/xj [

i

,

xj E Y, i = 1,2, ..., r ; j = 1 , 2 , ..., n

(23)

where r, the number of membership functions, may be called the cardinality of the opinion set. &(xj) denotes the degree of belonging of xi to the set F according to the ith membership function. The various properties and operations related to it have been defined by Pal and Das Gupta (1992). The incertitude or ambiguity associated with this set is two-fold, namely ambiguity in assessing a membership value to an element (d,) and ambiguity in deciding whether an element can be considered to be a member of the set or not (d2).Obviously, d2 is related to a fuzzy set, and its functional nature is the same as H’ (Eq. 2 with r = 1). On the other hand, d , reflects the amount of disparity (disagreement) within opinions because of the spectral nature. Regarding d , , it has been observed that human beings do not find it very difficult to assign memberships t o elements that have either very low or very high possibility of belonging to that set. In other words, the difference in opinions is low for those elements (supports) whose degree of inclusion or possibilities of belonging to a set is subjectively very low or very high. The variation in opinion, on the other hand, is high for those supports whose degree of belonging is fairly medium. For example, consider a spectral fuzzy set labeled “tall men” over the range 5 ft to 7 ft.

FUZZY SET THEORETIC TOOLS FOR IMAGE ANALYSIS

263

The difference in opinions (or difficulty in assessing a membership) as expressed would be higher around 5 ft 10 in. than around 5 ft or 7 ft. Similarly, if someone is asked to bring a full glass of water several times with the same glass, the variation in the amount of water will be less compared to the case when half a glass of water is asked for. Similar observations may be made when the task is performed by several people. It therefore appears that the variation in membership function assignment is high for the elements having fairly medium belonging. Based on this concept, spectral index 8 is defined as (Pal and Das Gupta, 1992) 1 O(F) = d,(F) = n

eli,

j = 1,2, ..., n

(24)

where S= -

1

r/2(r - I ) ’ 1 (r + 1)/2r’

if r is even

if r is odd

8 provides, in a global sense, a quantitative measure of the average differences (or disagreement) or opinion, in assigning a membership value to a supporting element. The (dis)similarity between the concept of spectral fuzzy sets and those of the other tools such as probabilistic fuzzy set, interval-valued fuzzy set, fuzzy set of type 2 or ultrafuzzy set (Klir and Folger, 1988; Hirota, 1981; Turksen, 1986; Mizumoto and Tanaka, 1976; Zadeh, 1984) (which have also considered the difficulty in settling a definite degree of fuzziness or ambiguity) has been explained in Pal and Das Gupta (1992). The concept has been found to be significantly useful (Pal and Das Gupta, 1992) in segmentation of ill-defined regions where the selection of a particular threshold becomes questionable as far as its certainty is concerned. In other words, questions may arise such as “where is the boundary” or “what is the certainty that a level 1, say, is a boundary between object and background?” The opinions on these queries may vary from individual to individual because of the differences in opinion in assigning membership values to the various levels. In handling this uncertainty, the algorithm gives due respect to various opinions on membership of gray levels for object region, minimizes the image ambiguity d( = d, + d,) over the resulting band of membership functions and then makes a soft decision by providing a set of thresholds (instead of a single

264

SANKAR K. PAL

one) along with their certainty values. A hard (crisp) decision obviously corresponds to one with maximum d value, i.e., the level at which opinions differ most. The problems of edge detection and skeleton extraction (where incertitude arises from ill-defined regions and various opinions on membership values), and any expert system type application (where differences in experts' opinions leads to an uncertainty) may also be similarly handled within this framework. V. SOMEEXAMPLES OF FUZZYIMAGE PROCESSING OPERATIONS

Let us now describe some algorithms to show how the aforesaid information measures and geometrical properties (which reflect grayness ambiguity and spatial ambiguity in an image) can be incorporated in handling uncertainties in various operations, e.g., gray-level thresholding, enhancement, contour detection, and skeletonization by avoiding hard decisions and providing output in both fuzzy and nonfuzzy (as a special case) versions. It is t o be noted that these low-level operations (particularly image segmentation and object extraction) play a major role in an image-recognition system. As mentioned in Section 11, any error made in this process might propagate to feature extraction and classification. A . Threshold Selection (Fuzzy Segmentation) Given an L-level image X of dimension M x N with minimum and maximum gray values lminand ,,,/ respectively, the algorithm for its fuzzy segmentation into object and background may be described as follows:

Step I : Construct the membership plane using the standard S function of Zadeh (Zadeh, 1965; Pal and Dutta Majumder, 1986) as Pmn

= ~ ( 1= ) S(I; a, 6, C)

(28)

(called bright-image plane if the object regions possess higher gray values) or pmn= p ( l ) = 1 - S(I; a, b, C) (29) (called dark-image plane if the object regions possess lower gray values) with crossover point b and a band width A b = b - a = c - b. Step 2: Compute the parameter I ( X ) , where I(X) represents either grayness ambiguity or spatial ambiguity (as designated by H', correlation, compactness, IOAC, and adjacency, say) or both (i.e., product of grayness and spatial ambiguities).

265

FUZZY SET THEORETIC TOOLS FOR IMAGE ANALYSIS

Step 3: Vary b between lminand I,,,,, and select those b for which I ( X ) has local minima or maxima depending on I ( X ) .(Maxima correspond for the correlation measure only.) Among the local minima/maxima, let the global one have crossover point at s. The level s, therefore, denotes the crossover point of the fuzzy image plane pmn, which has minimum grayness and/or geometrical ambiguity. The

pmnplane then can be viewed as a fuzzy segmented version of the image X.

For the purpose of nonfuzzy segmentation, we can take s as the threshold (or boundary) for classifying or segmenting an image into object and background. Faster methods of computation of the fuzzy parameters are explained in Pal and Ghosh (1992b). Note that w = 2 A b is the length of the window (such that [0, w ] [0, 11) that was shifted over the entire dynamic range. As w decreases, the p(xmn) plane tends to have more intensified contrast around the crossover point, thus resulting in a decrease of ambiguity in X.As a result, the possibility of detecting some undesirable thresholds (spurious minima) increases because of the smaller value of A b . On the other hand, an increase in w results in a higher value of fuzziness and thus leads toward the possibility of losing some of the weak minima. The criteria regarding the selection of membership functions and the length of window (i.e., w) have been reported recently in Murthy and Pal (1990, 1992), assuming continuous functions for both histogram and membership function. For a fuzzy set “bright image plane,” the membership function p : [0, w] + [0, 11 should be such that +

1. p is continuous, p(0) = 0, p ( w ) = 1 2. p is monotonically nondecreasing, and 3. p(x) = 1 - p(w - x ) for all x E [0, w ] . Furthermore, p, should satisfy the bound criteria derived based on the correlation (Section 1V.A). If, instead of a single membership function, we have a set of monotonically nondecreasing functions to represent a collection of various opinions on the bright membership plane px(x,,,,) and we wish to give due respect to all of these opinions, then the concept of spectral fuzzy sets (Section 1V.B) can be used to minimize the parameter spectral index (Eq. 24) in addition to one of those represented by Z(X) to manage this uncertainty. Consequently, it will make a soft decision by providing a set of thresholds associated with their respective certainty values. Details on this issue are available in Pal and Das Gupta (1992). Let us now describe another way of extracting an object by minimizing higher-order entropy (Eq. 2) of both object and background regions using

266

SANKAR K . PAL

1

0.5+------, 1

* e c

F

r 4

0

_ . * -

.

I> ‘ \

.. \

t

FIGURE1. Inverse z function (solid line) for computing object and background entropy.

an inverse n function as shown by the solid line in Fig. 1. Unlike the previous algorithm, the membership function does not need any parameter selection to control the output. Suppose s is the assumed threshold so that the gray level ranges [l, s] and [s + 1, L ] denote, respectively, the object and background of the image X. The inverse n-type function to obtain p,,, values of X is generated by taking unionofS[x;(s-(L-s)),s,L]and 1 - S [ x ; l , s , ( s + s - l)],whereS denotes the standard S function. The resulting function, as shown by the solid line, makes p lie in [ O S , 11. Since the ambiguity (difficulty) in deciding a level as a member of the object or the background is maximum for the boundary level s, it has been assigned a membership value of 0.5 (i.e., crossover point). Ambiguity decreases (i.e., degree of belonging to either object or background increases) as the gray value moves away from s on either side. The p,, thus obtained denotes the degree of belonging of a pixel x,,, to either object or background. Since s is not necessarily the midpoint of the entire gray scale, the membership function (solid line of Fig. 1) may not be a symmetric one. It is further to be noted that one may use any linear or nonlinear equation (instead of the standard S function) to represent the membership function in Fig. 1. Therefore, the task of object extraction is to: Step 1: Compute the rth order fuzzy entropy of the object Hh and the background H;1 considering only the spatially adjacent sequences of pixels present within the object and background respectively. Use the “min” operator to get the membership value of a sequence of pixels. Step 2: Compute the total rth order fuzzy entropy of the partitioned image as H,’ = Hh + HL. Step 3: Minimize H,’ with respect to s to get the threshold for object background classification.

267

FUZZY SET THEORETIC TOOLS FOR IMAGE ANALYSIS

Referring back to Section III.A, it is seen that H 2 reflects the homogeneity among the supports in a set in a better way than H ' does. The higher the value of r, the stronger is the validity of this fact. Thus, considering the problem of object-background classification, H' seems to be more sensitive (as r increases) to the selection of the appropriate threshold; i.e., the improper selection of the threshold is more strongly reflected by H' than Hr- 1 . For example, the thresholds obtained by the H Zmeasure have more validity than those by H' (which only takes into account the histogram information). Similar arguments hold good for even higher order (r > 2) entropy. The methods of object extraction (or segmentation) described above are all based on gray-level thresholding. Another way of doing this task is by pixel classification. The details on this technique using fuzzy c-means, fuzzy isodata, fuzzy dynamic clustering and fuzzy relaxation are available in Pal and Dutta Majumder (1986), Bezdek (1981), Kandel(1982), Pedrycz (1990), Lim and Lee (1990), Pal and Mitra (1990), Rosenfeld and Kak (1982), Dave and Bhaswan (1991), and Gonzalez and Wintz (1977). B. Contour Detection

Edge detection is also an image-segmentation technique where the contours/ boundaries of various regions are extracted based on the detection of discontinuity in grayness. The key factors of this approach are: 1. Most of the information of an image lies on the boundaries between different regions where there is a more or less abrupt change in gray levels, and 2. The human visual systems seem to make use of edge detection, but not of thresholding .

To formulate a fuzzy edge-detection algorithm, let us describe an edginess measure based on H' (Eq. 2 ) that denotes an amount of difficulty in deciding whether a pixel can be called an edge or not (Pal and Pal, 1990). Let N& be a 3 x 3 neighborhood of a pixel at ( x , y ) such that N& = ((X,Y), (x - 1 , Y h (x + 1,Yh (X,Y - 11, (X,Y

+ I),

( x - 1,Y - l ) , ( x - l , y + l ) , ( X + 1,Y - l ) , ( X + 1,Y

+ 1)1

(30)

The edge-entropy HZ of the pixel ( x , y ) , giving a measure of edginess at ( x , y ) , may be computed as follows. For every pixel ( x , y ) , compute the average, maximum, and minimum values of gray levels over N;. Let us denote the average, maximum, and minimum values by A v g , M a x , Min

268

SANKAR K. PAL

A

FIGURE2.

R

B

C

function for computing edge entropy.

respectively. Now define the following parameters. D = mux(Mux - Aug, Aug - Min]

(31)

B = Aug

(32)

A = B - D

(33)

C=B+D

(34)

A n-type membership function (Fig. 2) is then used to compute pxy for all ( x , y) E N A , such that p(A) = p(C) = 0.5 and p ( B ) = 1. It is to be noted that pv 2 0.5. Such a p v , therefore, gives the degree to which a gray level is close to the average value computed over Nx,y. In other words, it represents a fuzzy set pixel intensity close to its average value, averaged over N l , y . When all pixel values over Nx,y are either equal or close to each other (i.e., they are within the same region), such a transformation will make all pxy = 1 or close to 1. In other words, if there is no edge, pixel values will be close to each other, and the p values will be close to one, thus resulting in a low value of H I . On the other hand, if there is an edge (dissimilarity in gray values over N,,,), then thep values will be more away from unity, thus resulting in a high value of HI. Therefore, the entropy H' over Nx,,ycan be viewed as a measure of edginess (H&) at the point ( x , y ) . The higher the value of H:y, the stronger is the edge intensity and the easier is its detection. Such an entropy plane will represent the fuzzy edge-detected version of the image. As mentioned before, there are several ways in which one can define a n-type function as shown in Fig. 2. The proposed entropic measure is less sensitive to noise because of the use of a dynamic membership function based on a local neighborhood. The method is also not sensitive to the direction of edges. Other edginess measures and algorithms based on fuzzy set theory are available in Pal and Dutta Majumder (1986), PalandKing(l98la), PalandKing(1983), andPal(1986).

FUZZY SET THEORETIC TOOLS FOR IMAGE ANALYSIS

269

C. Optimum Enhancement Operator Selection

When an image is processed for visual interpretation, it is ultimately up to the viewers to judge its quality for a specific application and how well a particular method works. The process of evaluation of image quality therefore becomes subjective, which makes the definition of a well-processed image an elusive standard for comparison of algorithm performance. Again, it is customary to have an iterative process with human interaction in order to select an appropriate operator for obtaining the desired processed output. For example, consider the case of contrast enhancement using a nonlinear functional mapping. Not every kind of nonlinear function will produce a desired (meaningful) enhanced version. The questions that automatically arise are “Given an arbitrary image, which type of nonlinear functional form will be best suited without prior knowledge on image statistics (e.g., in remote applications like space autonomous operations where frequent human interaction is not possible) for highlighting its object?” and “Knowing the enhancement function, how can one quantify the enhancement quality for obtaining the optimal one?” Regarding the first question, even if the image statistics are given, it is possible only to estimate approximately the function required for enhancement, and the selection of the exact fucctional form still needs human interaction in an iterative process. The second question, on the other hand, needs individual judgment, which makes the optimum decision subjective. The method of optimization of the fuzzy geometrical properties and entropy has been found recently (Kundu and Pal, 1990) to be successful in providing quantitative indices in order to avoid such human iterative interaction in selecting an appropriate nonlinear function and to make the task of subjective evaluation objective. The use of fuzzy enhancement in hybrid coding of an image is described in Nasrabadi et al. (1983). Further discussion on this issue is made in Section VIII.

D. Fuzzy Skeleton Extraction and FMA T

Let us now explain two methods for extracting the fuzzy skeleton of an object from a gray-tone image without getting involved in its (questionable) hard thresholding. The first one is based on minimization of the parameter ZOAC (Eq. 17) or compactness (Eq. 1 1 ) with respect to a-cuts (a-cut of a fuzzy set A comprises all elements of X whose membership value is greater than or equal to a, 0 < a 5 1) over a fuzzy core line (or skeleton) plane. The membership value of a pixel to the core line plane depends on its

270

SANKAR K. PAL

property of possessing maximum intensity, and property of occupying vertically and horizontally middle positions from the &-edge(pixels beyond which the membership value in the fuzzy segmented image becomes less than or equal to E , E > 0) of the object (Pal, 1989). If a nonfuzzy (or crisp) single pixel width skeleton is deserved, it can be obtained by a contour tracing algorithm (Pal et al., 1983) which takes into account the direction of contour. Note that the original image can not be reconstructed, like the other conventional techniques of gray skeleton extraction (Rosenfeld, and Kak, 1982; Levi and Montanari, 1970; Peleg and Rosenfeld, 1981; Salari and Siy, 1984) from the fuzzy skeleton obtained here. The second method is based on fuzzy medial axis transformation (FMAT) (Pal and Rosenfeld, 1991) using the concept of fuzzy disks. A fuzzy disk with center P is a fuzzy set in which membership depends only on the distance from P. For any fuzzy set f , there is a maximal fuzzy disk gPf 5 f centered at every point P , and f is the sup of the gPf ’s. (Moreover, iff is fuzzy convex, so is every gPf, but not conversely.) Let us call a set Sf of points f-sufficient if every gPf 5 gQf for some set of Q in S,; evidently f is then the sup of the g@’s. In particular, in a digital image, the set of Q’s at which gf is a (nonstrict) local maximum is f-sufficient. This set is called the fuzzy medial axis off, and the set of g@’s is called the fuzzy medial axis transformation (FMAT) off. These definitions reduce to the standard one i f f is a crisp set. For a gray-tone image X (denoting the nonnormalized fuzzy bright image plane), the FMAT algorithm computes, first of all, various fuzzy disks centered at the pixels and then retains a few (as small as possible) of them, as designated by gQs, so that their union can represent the entire image X. That is, the pixel value at any point t can be obtained from a union operation, as t has membership value equal to its own gray value (i.e., equal to its nonnormalized membership value to the bright image plane) in one of those retained disks. For example, consider a 5 x 5 image X as shown in Fig. 3. The lower-left pixel of intensity 4 has coordinate (1, 1). Fuzzy disks (upright square of odd side length) for all the border pixels have values 15,Oj except the one at position (1, 1) for which gP = (4,O).The pixels having intensity 6 have disk values of ( 6 , 5 , 0 )except the one at (2,2), for which it is ( 6 , 4 , 0 ) .The center pixel has gP = (7,6,41. In these sets of disk values, the first entry denotes the nonnormalized membership value of the pixel itself to that disk (i.e., membership value of the disk at r = 0). The consecutive entries denote similarly the memberships at r = 1,2, ... . The pixels constituting the fuzzy medial axis are marked bold. (Note that if we had the pixel intensity 4 of X replaced by 5 , the FMAT would have been reduced to only one disk with g(3,3) = (736,511.

FUZZY SET THEORETIC TOOLS FOR IMAGE ANALYSIS

27 1

1

I

( 5

5

5

5

51

5

6

6

6

5

5

6

2

6

5

5

6

6

6

5

4

5

5

5

5

FIGURE3. A 5 x 5 digital image. Pixels belonging to fuzzy medial axis (FMA) are marked bold. Pixels belonging to reduced fuzzy medial axis (RFMA) are underlined.

In order to restore the deleted pixels, simply put all the disk values of FMA pixels at those locations back. In case a location has more than one such value, select the largest one. It is to be noted that the representation in Fig. 3 is redundant, i.e., some more disks can further be deleted without affecting the reconstruction. The redundancy in pixels (fuzzy disks) from the fuzzy medial axis output can be reduced by considering the criterion gPf(t) 5 sup g&(t), i = 1,2, ... instead of gPf(t) Ig@(t). In other words, eliminate many other gPf's for which there exists a set of gQ"s whose sup is greater than or equal to gP'. For example, the point at location (3,4) in Fig. 3 can be removed because it is contained in the union of the fuzzy disks around (3,3) and (2,4) for (4,411, i.e., g(3,4) 5: supIg(3,3), g(2,4)1 (or 5: S U P W ,31, g(4,4)1) for all pixels in X.Similar is the case with the pixel at location (4,3), which can also be removed. The pixels representing the final reduced MA are underlined in Fig. 3. Let RFMAT denote the FMAT after reducing its redundancy. To demonstrate its applicability on a real image let us consider Fig. 4(a) as input. Figure 4(b) denotes its RFMAT output. Therefore, the fuzzy medial axis provides a good skeleton of the darker (higher-intensity) pixels in an image apart from its exact representation. FMAT of an image can be considered as its core (prototype) version for the purpose of image matching. It is to be mentioned here that such a representation may not be economical in a practical situation. The details on this feature and the possible approximation in order to make it practically feasible are available in Pal and Wang (1991; 1992).

272

SANKAR K. PAL

FIGURE4. (a) 36 x 60 “S” image. (b) RFMAT output of “S” image.

Note that the membership values of the disks contain the information of image statistics. For example, if the image is smooth, the disk will not have abrupt change in its values. On the other hand, it will have abrupt change in case the image has salt-and-pepper noise or edginess. The concept of fuzzy MAT can therefore be used as spatial filtering (both high-pass and low-pass) of an image by manipulating the disk values to the extent desired and then putting them back while reconstructing the processed image.

VI. FEATURE/KNOWLEDGE ACQUISITION, MATCHING, AND RECOGNITION

In the previous sections, we have discussed, in detail, various measures (both fuzzy set theoretic and classical) for ambiguity in an image and their applications in representing and handling the various uncertainties that might arise in some of the important operations in image processing and analysis. The processed output can be obtained in both fuzzy and crisp

FUZZY SET THEORETIC TOOLS FOR IMAGE ANALYSIS

213

(as a special case) forms. As mentioned before, these operations (particularly image segmentation and object extraction) play major roles in an imagerecognition system. Any error made in this process might propagate to the higher level tasks, Le., in feature extraction, description, and classification/ analysis. Let us now explain, in brief, some of the approaches to show how the uncertainty in the tasks of feature extraction, boundary/shape detection of classes, learning and matching in a pattern recognition system can, in general, be represented and managed with the notion of fuzzy set theory. In picture-recognition and scene-analysis problems, the structural information is very abundant and important, and the recognition process includes not only the capability to assign the input pattern to a pattern class, but also the capacity to describe the characteristics of the pattern that make it ineligible for assignment to another class. In such cases complex patterns are described as hierarchical or treelike structures of simpler subpatterns, each simpler subpattern is again described in terms of even simpler subpatterns, and so on. Evidently, for this approach to be advantageous, the simplest subpatterns, called pattern primitives, are to be selected. Another activity that needs attention in this connection is the subject of shape analysis that has become an important subject in its own right. Shape analysis is of primal importance in feature/primitive selection and extraction problems. Description of shape can be done in two ways, e.g., in terms of scalar measurements and through structural descriptions. In this connection, it needs to be mentioned that shape description algorithms should be information-preserving in the sense that it is possible to reconstruct the shapes with some reasonable approximation from the descriptors. As described in Section 111.B, the fuzzy geometrical parameters also provide scalar measurements of shape of a gray image. Having extracted these fuzzy geometrical properties of an image, one can go by the decision theoretic approaches for its recognition. The fuzzy measures have recently been used by Leigh and Pal (1992) for motion-frame analysis. The way uncertainty arising from impreciseness and incompleteness in input pattern information can be handled heuristically has been reported recently in Pal and Mandal, (1992) by developing a linguistic recognition system based on approximate reasoning. The system can take input features either in linguistic form (F is very small, say) or in quantitative form (i.e., F is 500) or mixed form (Fis about 500) or set form (e.g., F is between 400 and 500). An input pattern has been viewed as consisting of various combinations of the three primary properties, e.g., small, medium, and high, possessed by its different features to some degree. The system provides a natural output decision in linguistic form, which is associated with a confidence factor denoting the degree of certainty of the decision. There have also been some attempts (Nath and Lee, 1983; Yager, 1981) to provide the design concept

274

SANKAR K. PAL

of a classifier that needs a priori knowledge from the experts in linguistic form, It is to be noted that the patterns having imprecise or incomplete information are usually ignored or discarded from the designing and testing processes of the conventional decision theoretic or syntactic classifiers (Duda and Hart, 1973; Tou and Gonzalez, 1974). For feature selection and extraction problems using fuzzy set theory, the readers may refer to the papers by Pal and Chakraborty (1986), Pal (1992b), Dave and Bhaswan (1991), Bezdek and Anderson, 1985), Bezdek and Castelaz (1977) and Di Gesu and Maccarone (1986). A multivalued recognition system (Mandal et al., 1992a) based on the concept of fuzzy sets has been formulated recently. This system is capable of handling various imprecise inputs and in providing multiple class choices corresponding to any input. Depending on the geometric complexity (Mandal et al., 1992b) and the relative positions of the pattern classes in the feature space, whole feature space is decomposed into some overlapping regions. The system uses Zadeh’s compositionai rule of inference (Zadeh, 1977) in order to recognize the samples. Application of this system to IRS (Indian remote sensing) imagery for detecting curved structures has been reported by Mandal (1 992). In a remotely sensed image, the regions (objects) are usually ill-defined because of both grayness and spatial ambiguities. Moreover, the gray value assigned to a particular pixel of a remotely sensed image is the average reflectance of different types of ground covers present in the corresponding pixel area (36.25rn x 36.2% for the Indian remote sensing [IRS] imagery). Therefore, a pixel may represent more than one class with a varying degree of belonging. For detecting the curved structures, the recognition system (Mandal et al., 1992a) is initially applied on an IRS image to classify (based on the spectral knowledge of the image) its pixels into six classes corresponding to six land cover types namely, pond water, turbid water, concrete structure, habitation, vegetation, and open space. The green and infrared band information, being sensitive than other band images to discriminate various land cover types, are used for the classification. The clustered images are then processed for detecting the narrow concrete structure curves. These curves include, basically, the roads and railway tracks. The width of such attributes has an upper bound, which was considered there to be three pixels for practical reasons. So all the pixels lying on the concrete structure curves with width not more than three pixels were initially considered as the candidate set for the narrow curves. Because of the low-pixel resolutions (36.2% x 36.25m for IRS imagery) of the remotely sensed images, all existing portions of such real curve segments may not be reflected as concrete structures, and, as a result, the candidate

FUZZY SET THEORETIC TOOLS FOR IMAGE ANALYSIS

275

pixel set may constitute some broken curve segments. In order to identify the curves in a better extent, a traversal through the candidate pixels was used. Before traversing process, one also needs to thin the candidate curve patterns so that a unique traversal can be made through the existing curve segments with candidate pixels. Thus, the total procedure to find the narrow concrete structure curves consists of three parts: (1) selecting the candidate pixels for such curves, (2) thinning the candidate curve patterns, and (3) traversing the thinned patterns to make some obvious connections between different isolated curve segments. The multiple choices provided by the classifier in making a decision are utilized to a great extent in the traversal algorithm. Some of the movements are governed by only the second and combined choices. After the traversal, the noisy curve segments (i.e., with insignificant lengths) are discarded from the curve patterns. The residual curve segments represent the skeleton version of the curve patterns. To complete the curve pattern, the concrete structure pixels lying in the eight neighboring positions corresponding to the pixels on the above-obtained narrow curve patterns are now put back. This resultant image represents the narrow concrete structure curves corresponding to an image frame. The effectiveness of the methods has been demonstrated on an IRS image frame representing a part of the city Calcutta. Figures 5a and 5b show the input of the image in Green and infrared bands respectively. Figure 5c shows the clustered version into six regions, and Fig. 5d demonstrates the detected narrow concrete structure curves. The results are found to agree well with the ground truths. The classification accuracy of the recognition system (Mandal et a / . , 1992a) is not only found to be good, but its stability of providing multiple choices in making decisions is also found t o be very effective in detecting the roadlike structures from IRS images. Let us now consider the syntactic approach of description and recognition of an image based on the primitives extracted from the structural information of its shape. The syntactic approach t o pattern recognition involves the representation of a pattern by a string of concatenated subpatterns called primitives. These primitives are considered to be the terminal alphabets of a formal grammar whose language is the set of patterns belonging to the same class. The task of recognition therefore involves a parsing of the string. Because of the ill-defined character of the structural information, the uncertainty may arise both in defining primitives and in relations among them. In order to handle them, the syntactic approach has incorporated the concept of fuzzy sets at two levels. First, the pattern primitives are themselves considered to be labels of fuzzy sets, i.e., such subpatterns as “almost circular arcs,” “gentle,” “fair,” and “sharp” curves are considered. Secondly, the structural relations among the subpatterns may be fuzzy, so

276

SANKAR K . PAL

FIOURE 5. IRS Calcutta images in (a) green band, (b) infrared band, (c) clustered image in six classes, and (d) detected roads along with a bridge (enclosed by dark lines).

that the formal grammar is fuzzified by the weighted production rules, and the grade of membership of a string is obtained by min-max composition of the grades of the production used in the derivations. For example, the primitives like line and curve may be viewed in terms of arcs with varying grades of membership from 0 to 1; 0 representing a straight line and 1 representing a sharp arc. Based on this concept, an algorithm was developed (Pal and Dutta Majumder, 1986; Pal et al., 1983) for automatic

FUZZY SET THEORETIC TOOLS FOR IMAGE ANALYSIS

277

extraction of primitives from gray-tone edge-detected images by defining membership functions for vertical line, horizontal line, and oblique line from the angle of inclination, and the degree of arcness of a line segment from the coordinates of its end points. Its effectiveness in recognizing x-ray images, hand and wrist bones, and nuclear patterns of brain neurosecretory cells is demonstrated in (Pathak and Pal, 1986b; Kwabwe et a / . , 1985; Azimi et at., 1982; Pal and Bhattacharyya, 1990). A similar interpretation of the shape parameters of triangle, rectangle and quadrangle in terms of membership for “approximate isoceles triangles,” “approximate equilateral triangles,” “approximate right triangle,’’ and so on has also been made (Huntsberger et at., 1986) for their classification in a color image. In order to represent the uncertainty in physical relations among the primitives, the production rules of a formal grammar are fuzzified to account for the fuzziness in relation among the primitives, thereby increasing the generative power of a grammar. Such a grammar is called fuzzy grammar (Lee and Zadeh, 1969; Thomason, 1973; DePalma and Yau, 1975). A concept of fractionally fuzzy grammars (Pathak et al., 1984) has also been introduced with a view to improving the effectiveness of a syntactic recognition system. It has been observed (Pathak and Pal, 1986b; Pathak et at., 1984) that the incorporation of the element of fuzziness in defining sharp, fair, and gentle curves in the grammars enables one to work with a much smaller number of primitives. By introducing fuzziness in the physical relations among the primitives, it was also possible to use the same set of production rules and nonterminals at each stage. This is expected to reduce, to some extent, the time required for parsing in the sense that parsing needs to be done only once at each stage, unlike the case of the nonfuzzy approach, where each string has to be parsed more than once, in general, at each stage. However, this merit has to be balanced against the fact that the fuzzy grammars are not as simple as the corresponding nonfuzzy grammars. Recently, rule-based systems have gained popularity in pattern recognition and high-level vision activities. By modeling the rules and facts in terms of fuzzy sets, it is possible to make inferences using the concept of approximate reasoning. Such a system has been designed recently (Nafarieh and Keller, 1991) for automatic target recognition using about 40 rules. A knowledge-based approach using Dempster-Shafer theory of evidence (Shafer, 1976) has also been formulated (Korvin et al., 1990) for managing uncertainty in object-recognition problems when features fail to be homogeneous. Meaningful pay-offs are defined in this context. The problem is tackled by considering masses with fuzzy focal elements. An evidential approach to problem solving was also developed when a large number of knowledge systems (which might give contradictory or inconsistent

278

SANKAR K. PAL

information) is available (Korvin et al., 1990). It is to be mentioned in this connection that the definitions of credibility and plausibility of DempsterShafer theory of evidence when the evidence and propositions are both fuzzy in nature are available in Yen (1990) and Pal and Das Gupta (1990). Another way of handling uncertainty in knowledge acquistition based on the theory of rough sets (Pawlak, 1982) is reported in Grzymala-Busse (1988). The approach considered uncertainties arising from the inconsistencies in different actions of different experts for the same object, or from the different actions of the same expert for different objects described by the same values of conditions. The method involves learning from examples. For a set of conditions of the information systems, and a given action of an expert, lower and upper approximations of a classification, generated by the action, have been computed with the help of rough set theory. Based on these approximations, the rules produced from the information stored in a data base are categorized as certain and possible. The certain rules may be propagated separately during the inference process, producing new certain rules. Similarly, the possible rules may be propagated in a parallel way. Fuzzy set theory and rough set theory are independent and offer alternative approaches to deal with uncertainty. However, there is a connection between rough set theory and Dempster-Shafer theory, though they have been developed separately. Dempster-Shafer theory uses the belief function as a main tool, whereas the rough set theory makes use of the family of all sets with common lower and upper approximations (Pawlak, 1985; Grzymala-Busse, 1988). VII. FUSIONOF FUZZYSETSAND NEURAL NETWORKS: NEURO-FUZZY APPROACH Artificial neural networks are signal-processing systems that emulate the human brain, i.e., the behavior of biological nervous systems, by providing a mathematical model of combination of numerous neurons connected in a network. Human intelligenceand discriminating power are mainly attributed to the massively connected network of biological neurons in the human brain. The collective computational abilities of the densely interconnected nodes or processors may provide a material technique, at least to a great extent, for solving complex real-life problems in a manner a human being does (Pao, 1989; Kosko, 1992; Bezdek and Pal, 1992; Lippman, 1989; Ghosh et al., 1991, 1992a,b, 1993; Pal and Mitra, 1992; Burr, 1988; Proc. 1st Int. Conf. on Fuzzy Logic and Neural Networks, 1990b; Proc. 1st IEEE Int. Conf. on Fuzzy Systems, 1992a; Proc. 2nd Int. Conf. on Fuzzy Logic and Neural Networks, 1992b).

FUZZY SET THEORETIC TOOLS FOR IMAGE ANALYSIS

279

We see that the fuzzy set theoretic models try to mimic human reasoning and the capability of handling uncertainty, whereas the neural network models attempt to emulate the architecture and information representation schemes of human brain. The fusion of these two new technologies therefore promise enormous intellectual and material gains in the field of computer and system science by incorporating the similarity in their logical operations and learning processes, and combining their individual merits. The fusion or integration is mainly tried out in the following ways or in any combination of them. 1 . Incorporating fuzziness into neural network frameworks. This includes assigning fuzzy labels to training samples, fuzzifying the input data, and obtaining output in terms of fuzzy sets (Fig. 6).

Neural labels

network

FIGURE 6 . Neural network implementing fuzzy classifier.

2. Making the individual neuron fuzzy (input to such a neuron is a fuzzy set and the output also is a fuzzy set). Activity of the networks involving fuzzy neurons is a fuzzy process (Fig. 7).

FIGURE7 . Block diagram of fuzzy neuron.

3. Designing neural networks guided by fuzzy logic formalism (i.e., designing neural networks to implement fuzzy logic) and realization of membership functions representing fuzzy sets by neural networks (Fig. 8).

AnteccdeiiL

clauses

Neural network

Error

Consequelit clauses

FIGURE 8. Neural network implementing fuzzy logic.

280

SANKAR K. PAL

4. Changing the basic characteristics of the neurons so that they perform

the operations used in fuzzy set theory (like fuzzy union, intersection, aggregation) instead of doing the standard multiplication and addition operations (Fig. 9).

FIGURE9. Neural network implementing fuzzy connectives.

5 . Modeling the error or instability or energy function of a neural

network based systems using measures of fuzzineduncertainty of a set (Fig. 10).

Puzzy

Neural network

Nonfuzzy output

FIGURE10. Layered network implementing self-organization.

The first way of integration is to incorporate the concept of fuzziness into a neural network framework, i.e., to build fuzzy neural networks. For example, the target output of the neurons in the output layer during training can be fuzzy label vectors. In this case the network itself is functioning as a fuzzy classifier. Keller and Hunt (1985) first suggested the incorporation of the concept of fuzzy pattern recognition into perceptron (single layer). They described a method for fuzzifying the labeled target data used for training the perceptron. Instead of giving hard labels to the training samples, membership functions denoting their degrees of belonging to the classes were used as labels. Instead of using the weight updation as

w+ w + ex, (c is a constant and

(35)

X, is the input data) they used

w w + JUlk - U*klmCX, 4-

(36)

FUZZY SET THEORETIC TOOLS FOR IMAGE ANALYSIS

28 1

where m is a constant and uik denotes the degree of belonging of xk to the ith class. Incorporation of membership function in the label vectors also acted as a good stopping criterion for linearly nonseparable classes (where the classical perceptron oscillates). The concept of fuzzy sets has also been introduced by Pal and Mitra (1992) in designing classifiers (both supervised and unsupervised) for uncertainty analysis and recognition of patterns using Kohonen’s model and the multilayer perceptron. A self-organizing artificial neural network capable of fuzzy partitioning of patterns has been developed that takes membership values to linguistic properties (e.g., low, medium, and high) along with some contextual class information to constitute the input vector. An index of disorder based on mean square distance between input and weight vectors has been defined in order to provide a quantitative measure for the ordering of the output space. The method based on the multilayer perceptron, on the other hand, involves assignment of appropriate weights to the backpropagated errors depending on the membership values at the corresponding outputs. During training, the learning rate is gradually decreased until the network converges to a minimum error solution. The performance is compared with that of the conventional model and Bayes’s classifier. It has been shown that these modified versions provide better performance for certain nonconvex decision regions (Pal and Mitra, 1993) as compared to the conventional ones. Though the effectiveness of the classifiers is demonstrated on some artificial data and speech data, the problem of image recognition under uncertainty can easily be dealt with within this framework. For example, the fuzzy geometrical properties (Section 1II.B) of a pattern can be used as features for learning the network parameters. The fuzzy segmented version, fuzzy edge-detected version, or fuzzy skeleton of an image may also be used along with their degrees (values) of ambiguities for the purpose of network training and its recognition. Sanchez (1990) has pointed out the noticeable similarities (like training by example, dynamic adjustment of changes in the environment, ability to generalize, tolerance to noise, graceful degradation at the border of the domain of expertise, and ability to discover new relations between variables) between neural networks and expert systems. He developed a fuzzy version of the connectionist expert system of Gallant (1988). In such an expert classification system knowledge base is generated from a training set of examples and is stored as connection strengths. The weight ( w i j ) between the input and the hidden layers are linguistic labels of fuzzy sets (identified by membership function) characterizing the variation of the input neurons and are assumed to be known. The weights between the hidden layer and the output layers (bij)are determined by training. The output of the neurons are computed by combining the weights for

282

SANKAR K. PAL

inferencing as q ( t ) = min(w@,b, J

(37)

A possible utility of such an expert system in biomedical domain is also

stated. The second way to incorporate fuzziness into the standard neural network is by making the individual neurons fuzzy (Lee and Lee, 1975; Yamakawa and Tomada, 1989). The idea is originally introduced by Lee and Lee (1975). The classical model of a neuron (McCulloch and Pitts, 1943) assumes its activity as an all-or-none process. It fails to model a type of imprecision that is associated with the lack of sharp transition from the occurrence to nonoccurrence of an event. Some of the concepts of fuzzy set theory are employed by Lee and Lee to define a fuzzy neuron, which is a generalization of the classical neuron. The activity of a fuzzy neuron is a fuzzy process (Lee and Lee, 1975). The input to such a neuron is a fuzzy set, and the outputs are equal to some positive numbers pj’s (0 < pj Il), if it is firing, and zero if it is quiet, pj denoting the degree to which thejth output is fired, the output is, therefore, also a fuzzy set. Unlike conventional neurons, such a neuron has multiple outputs. A fuzzy neural network is defined as a collection of interconnected fuzzy neurons. The utility of fuzzy neural networks to synthesize fuzzy automata is also investigated by them. The third fusion methodology is to use neural networks for a variety of computational tasks within the framework of a preexisting fuzzy model (i.e., implementation of fuzzy-logic formalism using neural networks). The use of multilayer feed forward neural network for implementing fuzzy logic rules (if-then rules) in introduced by Keller and coworkers (Keller and Tahani, 1991; Keller and Tahani, 1992; Keller et a/., 1992; Keller and Krishnapuram, 1992). It has been shown that the networks designed for implementing fuzzy rules can learn and extrapolate complex relationships between antecedents and consequent clauses for rules containing single, conjunctive, and disjunctive antecedent clauses. For rules having conjunctive clauses, the architecture has a fixed number of neurons in the input layer for each antecedent clause, a set of neurons in the hidden layer connected only to the neurons in the input layer associated with each antecedent clause. The neurons in the output layer are connected to all the neurons in the hidden layer. For implementing rules with disjunctive antecedent clauses, one more hidden layer was necessary. In Keller et a/. (1992), attempts are made to embed apriori knowledge of each rule directly into the weights of the network. In other networks the standard back-propagation learning algorithm is applied for learning weights. An attempt is also made by Takagi et al. (1992) to design structured neural networks to perform if-then fuzzy inference rules.

FUZZY SET THEORETIC TOOLS FOR IMAGE ANALYSIS

283

Attempts are also made to have membership function representation by neural networks (Ishibuchi and Tanaka, 1990; Takagi and Hayashi, 1991). A method has been suggested by Ishibuchi and Tanaka (1990) to identify real-valued and interval-valued membership functions from a set of given input-output data using a feed-forward layered neural network and backpropagation of error. Suggestions are also given to design membership functions of fuzzy neurons in Yamakawa and Furukawa (1992). A great deal of effort has also been given to design neural networkdriven optimal decision rules for fuzzy controllers (Gupta et al., 1989; Yager, 1992; Hayashi et al., 1992). A system for implementing fuzzy logic controllers using a neural network is designed in Yager (1992). The linguistic values associated with the fuzzy control rules are realized by separate neural-network blocks. The network is crafted depending on the inference structure provided by fuzzy logic involving intersection operations of fuzzy sets. The importance of different rules in the system is learned by operating the whole system and employing a rule that is of the form of the generalized delta rule. Suggestions are also given (Berenji, 1992; Berenji and Khedkar, 1992) for learning and tuning of fuzzy logic controllers based on reinforcement learning. Emphasis is mainly to adjust membership functions of the linguistic labels used in control rules. A fuzzy version of Kohonen’s self-organizing feature map algorithm is developed by Huntsberger and Ajmarangsee (1990) in order to generate continuous valued outputs (representing the degree of belonging) by adding one layer to the original Kohonen network. Fuzziness is also incorporated in the learning rate by replacing the learning rate usually found in Kohonentype update rules for the weight vectors with fuzzy membership of the nodes in each class. The proposed update rule is

where &k is the fuzzy membership of input standard updating rule

?(t

xk

in class i instead of the

+ 1) = Y(t)+ a[& - Y(tN

(39)

with a as a constant. They also have shown that the results produced by this fuzzy version of Kohonen’s algorithm are similar to those obtained by fuzzy c-means algorithms (Bezdek, 1981). Parallel implementations of this technique are also suggested. Further modification on the rate of learning is done by Bezdek et a/. (1992), and a relationship between the fuzzy version of Kohonen’s algorithm and the fuzzy c-means algorithm is established. Another way of fusion is to change the integration/transformation operation performed at each node so that they perform some sort of fuzzy aggregation (i.e., fuzzy union, intersection, aggregation). In Krishnapuram

284

SANKAR K. PAL

and Lee (1992a,b) fuzzy set connectives are used in multilayer network structures suitable for pattern recognition and other decision-making systems. Various union, intersection, generalized mean, and multiplicative hybrid operators (which are used by fuzzy sets to aggregate imprecise information in order to arrive at a decision in uncertain environments) are implemented by layered networks. A generalized mean operator was introduced by Yager (1978). Its form is

where wis are the relative importance given to corresponding inputs xis and C;= wi = 1. The hybrid (compensatory) model used was the y-model of Zimmerman and Zysno (1983) and is expressed as y = ni(xiG;)'-Y(l - n,(l - Xi)6i)Y

(41)

where 6;= rn and 0 5 y i 1; xi E [0, 11 are the inputs, bi is the weight associated with xi and y controls the degree of compensation. The hybrid operator can behave as union, intersection, or mean operator with different sets of parameters, which can be learned through training procedure. An iterative algorithm to determine the type of aggregation functions and its parameters at each node in the network is also provided, thereby making the network more flexible. The learning procedure involved is the same as that of the MLP. The training procedure of the multiplicative y-model is slow. To achieve faster convergence the additive y-model is studied, under the above framework, by Keller and Chen (1992) as an alternative connective in such networks. Gupta (1992) suggested the use of generalized AND (which can be expressed using the notation of triangular norms) and OR (represented by triangular conorm) operations for fuzzy signals (signals bounded by the graded membership function over the unit interval [0, 11) instead of multiplication and summation operations as used in standard neural networks. Thus for fuzzy inputs, x ( t ) E [0, 11" and synaptic strengths w ( t ) E [0, 11" the weighted synaptic signal z ( t ) E [0, 11" is defined as zi(t) = w i ( t ) A N D x i ( t ) ,

i = 1,2,

..., n,

(42)

and the aggregated input to a neuron is ui(t) = O R z j ( t ) . i

(43)

The nonlinear mapping with threshold wo E [0, 11 is then defined as

~ ; ( t=) [ui(t) OR wo(t)]"

(44)

FUZZY SET THEORETIC TOOLS FOR IMAGE ANALYSIS

285

where a, is a positive quantity. For 0 < a I1 the operation corresponds to dilation operation of a fuzzy set, and for CY > 1 it corresponds to concentration operation. Pedrycz (1991) tried to introduce fuzziness in neural networks in a different way. He pointed out the analogies between structures involving composite operators and a certain class of neural networks. Links are established between neural network architectures and relational systems in terms of fuzzy relational equations. The proposed architecture is based exclusively on set theoretic operations. The individual neurons perform logical operations (like max, min) which are mainly used in set theory instead of arithmetic operations. The problem of learning of connection strengths or weights was also studied, and relevant learning rules were proposed. A performance index, called equality index, is also introduced keeping track of these logical operations. Pedrycz has also suggested (1992) a design of neural networks to implement logic operations used in fuzzy set theory. The fifth way to integrate the concepts of fuzzy sets and neural networks is to use the fuzziness measures/uncertainty measures of a fuzzy set to model the error in neural networks. An attempt is made in this context by Ghosh el a/. (1993) to incorporate various fuzziness measures in a multilayer network for performing (unsupervised) self-organizing tasks in image processing, in general, and object extraction in particular. The network architecture is basically a feed forward one with back propagation of error (Fig. l l ) , but unlike conventional MLP it does not require any supervised learning. Each layer has M x N neurons for an M x N image (each neuron corresponding to an image pixel). Each neuron is connected to the corresponding neuron in the previous layer and its neighbors. Another structural difference from the standard MLP is that there exists a feedback

FIGURE1 1. Schematic representation of self-organizing rnultilayer neural network.

286

SANKAR

K. PAL

path from the output to the input layer. The status of neurons in the output layer is described as a fuzzy set. A fuzziness measure (e.g., index of fuzziness and entropy as mentioned in Section 1II.A) of this set is used as a measure of error in the system (instability of the network) and is back-propagated to correct weights. The input value (Ui)to the ith neuron in any layer (except the input layer) is calculated using the formula

where Wo is the connection strength between the ith neuron of one layer andjth neuron of the previous layer, and oj is the output status of the j t h neuron of the previous layer. j can either belong to the neighborhood of i, or j = i of the previous layer. The output is then obtained as 1

Starting from the input layer, this way the input pattern is passed on to the output layer and the corresponding output states are calculated. The output value of each neuron lies in [0, 11. After the weights have been adjusted by back-propagating the fuzziness measure of the output status of the neurons (which is treated as a fuzzy set) properly, the output of the neurons in the output layer is fed back to the corresponding neurons in the input layer. The second pass is then continued with this as input. The iteration (updating of weights) is continued as in the previous case until the network stabilizes, i.e., the error value (measure of fuzziness) becomes negligible. For example, the expression for weight updating for quadratic index of fuzziness (Kauffmann, 1980) is AY; =

tt( - oj)f'(lj)oi q(l - oj)f'(lj)oi

if 0 5 oj I0.5 if 0.5 < oj 5 1.0

(47)

for the output layer and

for hidden layers; where 6 k = -aE/aZk, and q is a proportionality constant. In the converged state the ON neurons constitute one class, and the OFF neurons another. Figure 12 demonstrates the variation of the learning rate for different fuzziness measures. In Kios and Liu (1992) an approach is provided to design optimal network architecture by optimization of fuzziness of a set.

FUZZY SET THEORETIC TOOLS FOR IMAGE ANALYSIS

287

I I

I

I

I I I

I I

I

I I I

I

I I I

I

I

1

1-

Logarithmic entropy

II

I

~

/

‘\iF

Linear index of fuzziness

\

/

\

Quadratic index

0 0

0.2

0.4 0.6 Status of a neuron

0.8

1.0

FIGURE 12. Rate of learning with variation of output status for different error measures,

288

SANKAR K. PAL

Note that these attempts of integration are mainly in the field of pattern recognition and to some extent in fuzzy logic control. Literature on neurofuzzy image-processing is not adequate at this moment. For further references on this issue one can refer to Kosko (1992); Takagi (1990); Bezdek and Pal (1992); Werbos (1992); Proc. Int. Joint Conf. on Neural Networks (1989); Archer and Wang (1991); Mitra and Pal (1992); Carpenter et al. (1991); Proc. 2nd Joint Tech. Workshop on Neural Networks and Fuzzy Logic (1990a); Proc. 1st IEEE Int. Conf. on Fuzzy Systems, Znt. J. of Approximate Reasoning, vol. 6, no. 2 (1992); IEEE Tr. on Neural Networks, vol. 3, no. 5 (1992); Int. J . of Pattern Recognition and AI, vol. 6, no. 1 (1992). VIII.

U S E OF

GENETIC ALGORITHMS

Genetic algorithms (GAS) (Goldberg, 1989; Davis, 1991) are highly parallel, mathematical, adaptive search procedures (i.e., problem-solving methods) based loosely on the processes or mechanics of natural genetics and Darwinian survival of the fittest. They model operations found in nature to form an efficient search that is effective across a broad spectrum of problems. These algorithms apply genetically inspired operators to populations of potential solutions in an iterative fashion, creating new populations while searching for an optimal (or near-optimal) solution to the problem at hand. Population is a key word here: the fact that many points in the space are searched in parallel sets genetic algorithms apart from other search operators. Another important characteristic of genetic algorithms is that they are very effective when searching (e.g., optimizing) function spaces that are not smooth or continuous functions that are very difficult (or impossible) to search using calculus based methods. Genetic algorithms are also blind; that is, they know nothing of the problem being solved other than payoff or penalty information, GAS differ from many conventional search algorithms in the following ways. They consider many points in the search space simultaneously, not a single point, and therefore have less chance of converging to local optima. They deal directly with strings of characters representing the parameter sets, not the parameters themselves. They use probabilistic rules to guide their searching process instead of deterministic rules. GAS find out the global near-optimal solution employing three basic operations-reproduction/selection, crossover, and mutation-over a limited number of strings (chromosomes) called population. A string is a coded version of the parameter set. For example, a binary string of length p q can be considered as a chromosomal (string) representation of the

FUZZY SET THEORETIC TOOLS FOR IMAGE ANALYSIS

289

parameter set @arjI i = 1,2, ...,p ] , where each substring of length q is assumed to be the representative of each parameter. Reproduction is a process in which individual strings are copied according to their objective function values, f,called the fitness function. These strings are then entered into a mating pool, a tentative new population, for further genetic operator action. The crossover generates offspring for the new generation using the highly fitted strings (parents) selected randomly from the mating pool. Each pair of strings undergoes crossing over as follows: An integer position k is selected uniformly at random between 1 and I - 1, where I is the string length greater than 1. Two new strings are created by swapping all characters from position k + 1 to 1. Mutation is the occasional (with small probability) random alteration of the value of a string position. A random bit position of a random string is selected and is replaced by another alphabet. In dealing with pattern analysis problems, GAS may be helpful in determining the appropriate membership functions, rules, and parameter space, and in providing a reasonably suitable solution. For this purpose, a suitable fuzzy fitness function needs to be defined depending on the problem. Fuzziness may also be incorporated in the encoding process by introducing a membership function representing the degree of similarity/ closeness between the chromosome parameters (strings). For example, consider a scene analysis problem where the relations among various segments (or objects) may be defined in terms of fuzzy labels such as close, around, partially behind, or occluded. Given a labelling of each of the segments, the degrees to which each relationship fits each pair of segments can be measured. These measures can be combined to define an overall fuzzy fitness function. Given this fitness function, the relations among objects, and the relations among classes to which the objects belong, a genetic algorithm searches the space to find the best solution in determining a class to be associated most appropriately to each object. An approach based on genetic algorithm for scene labeling is reported in Ankenbrandt et al. (1990). Let us now consider the problem of contrast enhancement of an image by gray-level modification. Given an image it is difficult t o select a functional form that will be best suited without prior knowledge of image statistics. Even if we are given the image statistics it is possible only to estimate approximately the function required for enhancement, and the selection of the exact functional form still needs human interaction in an iterative process. Bhandari el al. (1993) attempted to demonstrate the suitability of GAS in automatically selecting an optimum set of 12 parameter values of a generalized enhancement function that maximizes some fitness function.

290

SANKAR K. PAL

The algorithm used both spatial and grayness ambiguity measures (as mentioned in Section 1II.A) as the fitness value. The algorithm was implemented on images having compact and elongated (noncompact) objects and found to produce satisfactory results. The algorithm does not need iterative visual interaction and prior knowledge of image statistics in order to select the appropriate enhancement function. Convergence of the algorithm is experimentally verified. Since the domains of the parameters here are continuous, one needs to increase the length of the strings to obtain a more accurate solution. Some attempts in applying the GAS for classification, segmentation, primitive extraction and vision problems are reported in Belew and Brooker (1991). The basic idea is to use the GA to search efficiently the hyper-space of parameters in order to maximize some desirable criteria. In Section V.D, we have seen that the task of extracting fuzzy medial axis transformation (FMAT) of an image involves enormous computation, and it is not guaranteed even if the resulting output provides a compact minimal set for image representation. Searching based on GAS may be helpful in this case. Primitive extraction and aggregation is another area where GAS may be useful. Some recent applications in determining optimal set of weights for neural networks are available in Whitley et al. (1990), Bornholdt and Graugenz (1992), and Machado and Rocha (1992). It has been found that the backpropagation technique of multilayer perceptron may be avoided, thereby improving its computational time and the possibility of getting stuck to local minima. It is to be mentioned here that the GAS are computationally expensive. Moreover, one should be careful in selecting the initial population and the recombination operators. IX. DISCUSSION The problem of pattern analysis and image recognition under fuzziness and uncertainty has been considered. The role of fuzzy logic in representing and managing the uncertainties (which might arise in a recognition system) was explained. Various fuzzy set theoretic tools for measuring information on grayness ambiguity and spatial ambiguity in an image were listed along with their characteristics. Some examples of image-processing operations (e.g., segmentation, skeleton extraction, and edge detection), whose outputs are responsible for the overall performance of a recognition (vision) system, were considered in order to demonstrate the effectiveness of these tools in managing uncertainties by providing both soft and hard decisions. Uncertainty in determining a membership function in this regard and the tools for its management were also explained. Apart from representing and managing

FUZZY SET THEORETIC TOOLS FOR IMAGE ANALYSIS

29 1

uncertainties, the tools based on fuzzy set theory can also be used for providing quantitative measures in order to avoid the subjectivejudgment on the quality of processed output and to avoid human intervention in autonomous operations. Most of the algorithms and tools described here were developed recently by the author with his colleagues. Some of the illustrations were taken from the existing literature and put here together in a unified framework. Processing of color images has not been considered here. Some recent results on color image information and processing in the notion of fuzzy logic are available in Lim and Lee (1990), Xie (1990), and Pal (1991). Uncertainties involved in other parts of a recognition system, such as primitive extraction/analysis and syntactic classification and knowledge acquisition, were discussed. An application of multivalued approach to IRS image analysis has been demonstrated. Recent attempts of researchers on fusion of fuzzy set theory and neural networks for better handling of uncertainty (in the sense of robustness, performance and parallel processing) in pattern-analysis problems have been mentioned. It may be mentioned here that neuro-fuzzy processing should continue to be a thrust research area at least for the next decade. Finally, the key features of genetic algorithms along with the possibility of successful use in this context were explained. Research is in progress at NASA’s Johnson Space center in making application of the aforesaid tolls and the recognition algorithm in space autonomous operations (e.g., camera tracking system and collision avoidance in Mars rover control [Lea et ai., 1989, 1990a,b,c, 1991, 1992) for supporting an unmanned mission. Various expert system shells based on fuzzy logic are now commercially available. Fuzzy logic chips developed by Togai and Watanabe at Bell Laboratories can be used in fuzzy-rule-based expert systems that do not require a high degree of precision. The fuzzy computer developed by Y amakawa of Kumamoto University has shown great promise in processing linguistic data at high speed and with remarkable robustness (Rogers and Hosiai, 1990; Proc. 2nd Congress ofthe Int. Fuzzy Systems Assoc., 1987). This may be an important step toward the development of a sixth-generation computer capable of processing common-sense knowledge. This capability is a prerequisite for solving many A1 problems, e.g., recognition of handwritten text and speech, machine translation, summarization, and image understanding that do not lend themselves to cost-effective solution within the bounds (limitations) of conventional technology. ACKNOWLEDGMENTS

The author acknowledges Mr. A. Ghosh, D. Bhandari, and D. P. Mandal for their assistance in preparing the manuscript.

292

SANKAR K. PAL

REFERENCES Ankenbrandt, C., Buckles, B., and Petry, F. (1990). Putt. Recog. Lett. 11, 285-293. Archer, N. P., and Wang, S. (1991). IEEE Transactions on Systems, Man, and Cybernetics 21, 735-742. Azimi Sadjadi, M. R., King, R. A., and Pal, S. K. (1982). IEE Proceedings, Pt.-F 129, 373. Belew, R. K., and Booker, L. B. (eds.) (1991). Proc. Fourth Int. Conf. on Genetic Algorithms. University of California Press, San Diego. Berenji, H. R. (1992). In?. J. of Approximate Reasoning 6, 267-292. Berenji, H. R., and Khedkar, P. (1992). IEEE Transactions on Neural Networks 3, 724-740. Bezdek, J . C. (1981). Pattern Recognition with Fuzzy Objective Function Algorithms. Plenum Press, New York. Bezdek, J. C., and Anderson, 1. M. (1985). IEEE Trans. Syst., Man and Cyberns. 15,637-641. Bezdek, J. C., and Castelaz, P. F. (1977). IEEE Trans. Syst., Man and Cyberns. SMC-7, 87-92. Bezdek, J . C., and Pal, S. K. (eds.) (1992). Fuzzy Models for Pattern Recognition: Methods that Search for Structures in Data. IEEE Press, New York. Bezdek, J . C., Tsao, E. C. K., and Pal, N. R. (1992). Proc. First IEEE Int. Conf. on Fuzzy Systems. San Diego, pp. 1035-1046. Bhandari, D., and Pal, N. R. (1993). Information Sciences 67, 209-228. Bhandari, D., Pal, N. R., and Dutta Majumder, D. (1992). Pattern Recognition Letters 13, 851-867. Bhandari, D., Pal, S. K., and Kundu, M. K. (1993). Proc. IEEE Int. Con5 on Fuzzy Systems. California, 1408- I41 3. Bornholdt, S., and Graugenz, D. (1992). Neural Networks 5, 327-334. Burr, D. J. (1988). IEEE Transactions on Acoustic, Speech, and Signal Processing 36, 1162-1 168. Carpenter, C. A., Crossberg, S., and Rosen, D. B. (1991). Neural Networks 4. Davis, L. (1991). Hand Book of Genetic Algorithms. Van Nostrand and Reinhold, New York. Dave, R., and Bhaswan, K. (1991). Proc. NAFIPS’ 91, University of Missouri-Columbia, pp. 195-199. De Luca, A., and Termini, S. (1972). Inform and Control 20, 301-312. DePalma, G. F., and Yau, S. S. (1975). “Fractionally Fuzzy Grammars with Applications t o Pattern Recognition.” In Fuzzy Sets and Their Applications to Cognitive and Decision Processes (Zadeh, L. A. el al., eds.). Academic Press, London. Di Gesu, V., and Maccarone, M. C. (1986). Pattern Recognition 19, 63-72. Duda, R., and Hart, P. (1973). Pattern Classification and Scene Analysis. John Wiley & Sons, New York. Edelsbrunner, H., Kirkpatrick, D. G., and Seidel, R. (1983). IEEE Trans. Inform. Theory 29. 551. Fu, K . S . (1982). Syntactic Pattern Recognition and Applications. Academic Press, London. Gallant, S . 1. (1988). Communication of the Association for Computing Machinery 31, 152-169. Ghosh, A,, and Pal, S. K. (1992a). Putt. Recog. Lett. 11, 387-397. Ghosh, A., Pal, N. R., and Pal, S. K. (1991). Biological Cybernetics 66, 151-158. Ghosh, A., Pal, N. R., and Pal, S. K. (1992b). Int. J . Putt. Recog. and Artificial Intell. 6, 989-1008. Ghosh, A., Pal, N. R., and Pal, S. K . (1993). IEEE Transactions on Fuzzy Systems 1, 54-68.

FUZZY SET THEORETIC TOOLS FOR IMAGE ANALYSIS

293

Goldberg, D. E. (1989). Genetic Algorithms in Search, Optimization, and Machine Learning. Addison-Wesley, Reading, Massachusetts. Gonzalez, R. C., and Wintz, P. (1987). Digital Image Processing, 2nd ed. Addison-Wesley, Reading, Massuchusetts. Grzyrnala-Busse, J. W. (1988). J. Intelligent and Robotic Systs. 1, 3-16. Gupta, M. M. (1992). Proc. Second In/. Conj. on Fuzzy Logic and Neural Networks. Iizuka, Japan, 157-160. Gupta, M. M., Pedrycz, W., and Kiszka, J . (1989). Proc. Third IFSA Congress. Seattle, 258-261. Hayashi, Y., Czogala, E., and Buckley, J . J . (1992). Proc. First IEEE Int. Conj. on Fuzzy Systems. San Diego, USA, 197-202. Hirota, K. (1981). Fuzzy Sets and Systs, 5, 31-46. Huntsberger, T. L., and Ajjimerangsee, P. (1990). Int. J . General Sysfems 16, 357-372. Huntsberger, T. L., Rangarajan, C., and Jayaramamurthy, S. N. (1986). IEEE Trans. Comp. C-35, 145-156. Ishibuchi, H . , and Tanaka, H. (1990). Proc. Firs( In/. Conf. on Fuzzy Logic and Neural Networks. Iizuka, Japan, 179-182. Kandel, A, (1982). Fuzzy Techniques in Pattern Recognition. Wiley Interscience, New York. Kaufrnann, A. (1980). Introduction to the Theory oJFuzzy Subsets-Fundamental Theoretical Elements, Vol. 1 . Academic Press, New York. Keller, J . M., and Chen, Z. (1992). Proc. Second Int. Conf. on Fuzzy Logic and Neural Networks. Iizuka, Japan, 85-87. Keller, J . M . , and Hunt, D. J. (1985). IEEE Transactions an Paltern Analysis and Machine Intelligence 1,693-699. Keller, J. M., and Krishnapuram, R. (1992). IEEE Transactions on Neural Networks 3 , 761-769. Keller, 3 . M., and Tahani. H. (1991). Int. J . Approximate Reasoning 6 , 221-240. Keller, J. M., and Tahani, H. (1992). Information Sciences 62, 205-221. Keller, J . M., Yager, R. R., and Tahani, H. (1992). Fuzzy Sets and Systems 45, 1-12. Kios, K. J., and Liu, N. (1992). IEEE Transactions on Neural Nefworks 3, 280-291. Klir, G. J., and Folger, T. A. (1988). Fuzzy Sets, Uncertainty, and Information. Prentice Hall, Englewood Cliffs, New Jersey. Korvin, A. de, Kleyle, R., and Lea, R. (1990). Ini. J. Intell. Systs. 5, 293-306. Kosko, B. (1992). Neural Networks and Fuzzy Systems. Prentice-Hall, Englewood Cliffs, New Jersey. Krishnapurarn, R., and Lee, J . (1992a). Fuzzy Sets and Systems 46, 11-27. Krishnapuram, R., and Lee, J. (1992b). Neural Networks 5, 335-350. Kundu, M. K., and Pal, S. K. (1990). Part. Recog. Lett. 11, 811-829. Kwabwe, S. A,, Pal, S. K., and King, R. A . (1985). In/. J . Syst. Sci. 16, 403-413. Lea, R. N., Giarratano, J., Fritz, R. H., and Jani, Y. K. (1990a). Proc. 8th International Congress of Cybernetics and Systems. New York. Lea, R. N., and Jani, Y. (1992). In/. J. Approximate Reasoning 6 , 151-184. Lea, R. N., Jani, Y. K., and Berenji, H. (1990b). Proc. 5th IEEEInternationalSymposiumon Intelligent Control 2, 903. Lea, R. N., Murphy, M. G., and Walters, L. (1991). Proc. NAFIPS 91, University o f Missouri-Columbia, 154. Lea, R. N., Togai, M.. Teichrow, J., and Jani, Y. (1989). Proc. IFSA 89 Congress., 23-29. Lea, R. N., Walters, L., and Jani, Y. K. (1990~).Proc. 1st International Symposium on Measurement and Control in Robotics, D.3.1 , I . Lee, E . T., and Zadeh, L. A. (1969). Inform. Sci. 1, 421-434.

294

SANKAR K. PAL

Lee, S. C., and Lee, E. T. (1975). Mathematical Biosciences 23, 151-177. Leigh, A. B., and Pal. S. K. (1992). Proc. NAFIPS-92, 69-80. Levi, G., and Montanari, U. (1970). Inform. and Control 17, 62-91. Lim, Y. W., and Lee, S. U. (1990). Pattern Recognition 23, 935-952. Lippmann, R. P. (1989). IEEE Communications Magazine, 47-64. Machado, R. J., and Rocha, A. F. D. (1992). Proc. IEEE 1st Int. Conf. on Fuzzy Systems, FUZZ-IEEE’92. San Diego, 493-500. Mandal, D. P. (1992). “A Multivalued Approach for Uncertainty Management in Pattern Recognition Problems Using Fuzzy Sets.” Ph.D. Thesis, Indian Statistical Institute, Calcutta. Mandal, D.P., Murthy, C. A., and Pal, S. K. (1992a). IEEE Trans. System, Man & Cybern. SMC-22, 307-320. Mandal, D. P., Murthy, C. A,, and Pal, S. K. (1992b). Int. J. General Systems 20, 307-339. McCulloch, W. S., and Pitts, W. (1943). Bulletin ojMarhematica1 Biophysics 5. 115-133. Mitra, S., and Pal, S. K. (1994). IEE Trans. Syst.. Man and Cyberns. 24(2), in press. Mizumoto, M., and Tanaka, K. (1976). Inform. and Control 31, 312-340. Murthy, C. A. (1988). “On Consistent Estimation of Classes in R2 in the Context of Cluster Analysis.” Ph.D. Thesis, lndian Statistical Institute, Calcutta. Murthy, C. A., and Pal, S. K. (1990). Pal?. Recog. Lett. 11, 197-206. Murthy, C. A., and Pal, S. K. (1992). Inform. Sci. 60. 107-135. Murthy, C. A., Pal, S. K., and Dutta Majumder, D. (1985). Fuzzy Sets and Systs. 7, 23-38. Nafarieh, A., and Keller, J. (1991). I n f . J. Intell. Systs. 6, 295-312. Nasrabadi, N. M., Pal, S. K., and King, R. A. (1983). EIectronics Lett. 19, 63. Nath, A. K., and Lee, T. T. (1983). Fuzzy Sets and Systs. 11, 265. Pal, N. R., and Pal, S. K. (1989a). fEE Proc. 136, Pt. E , 284-295. Pal, N. R., and Pal, S. K. (1989b). Signal Processing 16, 97-108. Pal, N. R., and Pal, S. K. (1991a). IEEE Trans. Syst.. Man and Cyberns. SMC-21, 1260-1270. Pal, N. R., and Pal, S. K. (1991b). Int. J. Pall. Recog. and Artvicial Intell. 5, 459-483. Pal, N. R., and Pal, S. K. (1992a). Inform. Sci. 61, 211-231. Pal, N. R., and Pal, S. K. (1992b). fnjorm. Sci. 66, 119-137. Pal, S. K. (1982). IEEE Trans. Patt. Anal. and Machine Intell. PAMI-4, 204-208. Pal, S. K. (1986). Patt. Recog. Lett. 4, 51-56. Pal, S. K. (1989). Part. Recog. Lett. 10, 17-23. Pal, S. K. (1991). Int. J. Syst. Sci. 22, 511-549. Pal, S. K. (1992a). J. ojScient$ic and Industrial Research 51, 71-98. Pal, S. K. (1992b). Inform. Sci. 64, 165-179, Pal, S. K., and Bhattacharyya, A. (1990). Patt. Recog. Lett. 11, 443-452. Pal, S. K., and Chakraborty, B. (1986). IEEE Trans. Syst., Man and Cyberns. 16, 754-760. Pal, S. K., and Das Gupta, A. (1990). Proc. I n t . Conj. Fuzzy Logic & Neural Nefworks (IIZUKA’90). Iizuka, Japan, vol. 1. 299-302. Pal, S. K., and Das Gupta, A. (1992). Inform. Sci. 65. 65-97. Pal, S. K., and Dutta Majumder, D. (1986). Fuzzy Mathemafical Approach to Pattern Recognition. John Wiley and Sons, (Halsted Press), New York. Pal, S. K., and Ghosh, A. (1990). Patt. Recog. Lett. 11, 831-841. Pal, S. K., and Ghosh, A. (1992a). Inform. Sci. 62, 223-250. Pal, S. K., and Ghosh, A. (1992b). Fuzzy Sets and Sysls. 48, 23-40. Pal, S. K., and King, R. A. (1981a). Electronics Letters. 17, 302-304. Pal, S. K.,and King, R. A. (1981b). IEEE Trans. Syst.. Man and Cyberns. SMC-11,494-501. Pal, S. K., and King, R. A. (1983). IEEE Trans. Pattern Anal. Machine Intell. PAMI-5, 69-77,

FUZZY SET THEORETIC TOOLS FOR IMAGE ANALYSIS

295

Pal, S. K., King, R. A., and Hashim, A. A. (1983). IEEE Trans. Syst., Man and Cyberns. SMC-13, 94-100. Pal, S. K., and Mandal, D. P. (1992). Inform. Sci. 61, 135-161. Pal, S. K., and Mitra, S. (1990). Putt. Recog. Lett. 11, 525-535. Pal, S. K., and Mitra, S. (1992). IEEE Transactions on Neural Networks 3, 683-697. Pal, S. K., and Mitra, S. (1993). Information Sciences (accepted). Pal, S. K., and Pal, N. R. (1990). Proc. INDO-US Workshop on Spectrum Analysis in One and Two Dimensions. New Delhi, NBH Oxford Publishing Co., New Delhi, 285-300. Pal, S. K., and Rosenfeld, A. (1988). Patr. Recog. Lett. 7 , 77-86. Pal, S. K., and Rosenfeld, A. (1991). Putt. Recog. Lett. 10, 585-590. Pal, S. K., and Wang, L. (1991). Proc. IFSA’91 Congress. Brussels, Belgium, 167-170. Pal, S. K., and Wang, L. (1992). Fuzzy Sets and Systems 50, 15-34. Pao, Y. H. (1989). Adaptive Pattern Recognition aird Neural Networks. Addison-Wesley, Reading, Massachussets. Parsi, B. K., and Parsi, B. K. (1990). Biological Cybernetics 62, 415-423. Pathak, A,, and Pal, S. K. (1986a). Putt. Recog. Lett. 4, 63-69. Pathak, A,, and Pal, S. K. (1986b). IEEE Trans. Syst., Man and Cyberns. SMC-16, 657-667. Pathak, A,, Pal, S. K., and King, R. A. (1984). Putt. Recog. Lett. 2, 193. Pawlak, Z. (1982). Int. J. Inform. Comp. Sci. 11, 341-356. Pawlak, Z. (1985). Fuzzy Sets and Sysrs. 17, 99-102. Pedrycz, W. (1990). Partern Recognition 23, 121-146. Pedrycz, W. (1991). IEEE Transactions on Pattern Analysis and Machine Intelligence 13, 289-297. Pedrycz, W . (1992). IEEE Transactions on Neural Networks 3, 770-775. Peleg, S . , and Rosenfeld, A. (1981). IEEE Trans. Putt. Anal. Mach. Intell. P A M I - 3 , 208-210. Proc. of the Second Congress of the lnternational Fuzzy Systems Association (1987). Tokyo, Japan. Proc. Int. Joint Conf. on Neural Networks (1989). Washington DC, USA. Proc. Second Joint Technology Workshop on Neural Networks and Fuzzy Logic (1990a). NASA Conference Publication 10061, Johnson Space Center, Houston, Texas, USA, April 10-13. Proc. Int. Conf. Fuzzy Logic and Neural Networks (1990b). IIZUKA’90, Kyusu Institute of Technology, Iizuka, Fukuoka, Japan, July 22-24. Proc. First IEEE International Conference on Fuzzy Systems (1992a). San Diego, USA. Proc. Second International Conference on Fuzzy Logic and Neural Networks (1992b). lizuka, Japan. Rogers, M., and Hoshiai, Y. (1990). Newsweek, May 28, 46. Rosenfeld, A. (1984). Putt. Recog. Lett. 2, 31 1-317. Rosenfeld, A., and Kak, A. C. (1982). Digital Picture Processing, vol. 2. Academic Press, New York. Salari, E., and Siy, P. (1984). IEEE Trans. Syst.. Man and Cyberns. SMC-14, 524-528. Sanchez, E. (1990). Proceedings First Int. Conf. on Fuzzy Logicand NeuralNetworks. lizuka, Japan, 31-35. Shafer, G. (1976). A Mathematical Theory of Evidence. Princeton University Press, Princeton, New Jersey. Shannon, C.E. (1948). Bell. Syst. 7ech. Jour. 27, 379. Special issue on fuzzy logic and neural networks (1992). IEEE Transactions on Neural Networks 3. Special issue on neural networks (1992). Int. J . of Pattern Recognition and Artificial Intelligence 6 .

296

SANKAR K. PAL

Special issue on fuzzy logic and neural networks (1992). Int. J. Approximate Reasoning 6. Takagi, H. (1990). In Proc. Int. Conf. Fuzzy Logic and Neural Networks, IIZUKA’90, Kyusu Institute of Technology, Iizuka, Fukuoka, Japan, July 22-24, 13-26. Takagi, H., Suzuki, N., Koda, T., and Kojima, Y. (1992). IEEE Transactions on Neural Networks 3, 752-760. Takagi, H., and Hayashi, I. (1991). Int. J. OfApproximate Reasoning 5, 191-212. Thomason, M. G. (1973). Patfern Recognition 5, 383-390. Tou, J. T., and Gonzalez, R. C. (1974). Pattern Recognition Principles. Addison-Wesley, London. Tousant, G. T. (1980). Proc. 5th Int. Conf, Pattern Recognition. Miami Beach, Florida, 1324- 1347.

Turksen, I. B. (1986). Fuzzy Sets and Systs. 20, 191-210. Werbos, P. J. (1992). Int. J. of Approximate Reasoning 6, 185-219. Whitley, D., Starkweather, T., and Bogart, C. (1990). Parallel Computing 14, 347-361. Xie, W. X. (1990). Fuzzy Sets and Systs. 36, 157-165. Xie, W . X., and Bedrosian, S. D. (1984). IEEE Trans. Syst., Man and Cyberns. 14, 151. Yager, R. R. (1978). Fuzzy Sets and Systems 1, 87-95. Yager, R. R. (1981). Int. J. Comp. Inf. Sci. 10, 141. Yager, R. R. (1992). Fuzzy Sers and Systems 48, 53-64. Yamakawa, T., and Furukawa, M. (1992). Proc. First IEEE Int. Conf. on Fuzzy Systems. San Diego, USA, 75-82. Yamakawa, T., and Tomada, S. (1989). Proceedings Third ZFSA Congress, Seattle, 30-38. Yen, J. (1990). IEEE Trans. Syst., Man and Cyberns. SMC-20,559-570. Zadeh, L. A. (1965). Inform. Control 8, 338-353. Zadeh, L. A. (1977). Synthese 30, 407-428. Zadeh, L. A. (1984). IEEE Spectrum August, 26-32. Zadeh, L. A., Fu, K. S., Tanaka, K., and Shimura, M. (1975). Fuzzy Sets and Their Applications to Cognitive and Decision Processes. Academic Press, London. Zimmerman, H. J., and Zysno, P. (1983). Fuzzy Sets and Systems 10, 243-260.

ADVANCES IN ELECTRONICS AND ELECTRON PHYSIC'S, VOL. 88

The Differentiating Filter Approach to Edge Detection Maria Petrou Department of Electronic and Electrical Engineering, University of Surrey, Guildford, United Kingdom

I. Introduction . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . Theory . . . . . . . . . . . . . . . . A. The Good Signal-to-Noise Ratio Requirement . . B. The Good Locality Requirement . . . . . . . C. The Suppression of False Maxima . . . . . . D. The Composite Performance Measure . . . . . E. The Optimal Smoothing Filter . . . . . . . F. Some Example Filters . . . . . . . . . . Theory Extensions . . . . . . . . . . . . . A. Extension to Two Dimensions . . . . . . . B. The Gaussian Approximation . . . . . . . . C. The Infinite Impulse-Response Filters . . . . . D. Multiple Edges . . . . . . . . . . . . E. A Note on the Zero-Crossing Approach . . . . Postprocessing . . . . . . . . . . . . . . Conclusions . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . .

11. Putting Things in Perspective

III.

IV.

V. VI.

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

291 301 309 312 315 319 320 322 323 324 325 326 329 331 332 333 339 343

I . INTRODUCTION The purpose of computer vision is to identify objects in images. The images are obtained by various image capture devices like CCD cameras and analogue film cameras. In general an image has to be represented in a way that computers can understand it. Computers understand numbers, and numbers have to be used. An image, therefore, is a two-dimensional array of elements, each of which carries a number that indicates how bright the corresponding analogue picture is at that location. The elements of the image array are called pixels, and the values they carry are usually restricted by convention to vary between 0 (for black) and 255 (for white). To be able to represent a scene or an analogue picture in adequate detail, we need to use many such picture elements, i.e., our image arrays must be pretty large. For example, to imitate the resolution of the human vision system, we probably need arrays of size 4000 x 4000, and to imitate the resolution of an ordinary television set, we must use arrays of size 1000 x 1000. To store 291

Copyright Ic 1Y94 hy Academic Pres,. Inc All rights of reproduction in any form reserved ISBN 0-12-014730-0

298

MARIA PETROU

a television-size image, therefore, we need about eight Mbytes of memory. And this is only for a black-and-white image, usually called a grey image to indicate that not only black and white tones are used but also all possible shades in between. If we want to represent a coloured picture, we need three times as many bits, because it has been shown that any colour can be reproduced by blending appropriate amounts of three basic colours only. This is known as the trichromatic theory of colour vision. So, a coloured image can be represented by a three-dimensional array of numbers, two of the dimensions being the spatial dimensions which span the image and the third dimension being the one used to store three numbers that correspond to each pixel, each giving the intensity of the image in one of the three basic colours used. In this chapter, we are going to talk only about grey images, so this is the last time we make any reference to colour. It is clear from the above discussion that an image contains an enormous amount of information, not all of which is useful, necessary, or wanted. For example, we all can recognize that the person depicted in Fig. l b is the same as the person in Fig. la, although Fig. l b is only a sketch. That image is a binary image, and thus each pixel requires only two bits to be represented. This is a factor of 4 reduction in the number of bits needed for the representation of the grey image and a factor of 12 reduction in the

FIGURE1. (a) An original image. (b) Edges detected by hand.

THE DIFFERENTIATING FILTER APPROACH TO EDGE DETECTION

299

number of bits needed for the representation of the corresponding colour image. And yet, for the purpose of recognition, such a representation is adequate. If we could make the computer produce sketches like this it would be very useful: first because in order to identify the shape of the object much less number crunching will have to take place and second because having found the outline of the object its properties can be computed more easily. A lot of vision problems would even stop at the point of the shape description, as many objects can be easily identified from their shape only. The task of making the computer produce a sketch like Fig. l b is called edge detection, and the algorithms that can do that are called edge detectors. Is edge detection a difficult task for a computer? Well, it has proven to be very difficult indeed, in spite of all the ingenuity and effort that has gone into it. Let us try to follow the steps I took when I drew the sketch of Fig. l b , starting from the image shown in Fig. la. I first looked at places where there was some changes in brightness and I followed them around. I did not bother with the changes in brightness that occur inside the boy’s shirt because I know that they do not matter in the recognition process. I did not bother with the shades that appear in the face, as they may be due to image reproduction problems or play no role in the representation of the basic characteristics of the face. I did bother with changes in brightness around the nose area, even though they were faint and gradual, and I did reproduce very faint outlines if they were straight, meaningful, and seemed to complete the shapes represented. If we read carefully again the previous statement, we will notice that a lot of thinking went into the process without even realising it. In particular, a lot of knowledge and experience was incorporated into it, knowledge that has been acquired over a lifetime! Well, most edge-detection effort so far has gone into attempting to reproduce the first small part of the description of the process, i.e., to make computers recognize the places where there is some change in brightness! And in spite of the hundreds of methods developed and the hundreds of papers published, a good edge detector today will not produce anything as good as what is shown in Fig. lb; instead, something like what is shown in Fig. 2 will be the result. The reason is that most of the effort has gone into the first part of the description, namely into identifying places where the brightness changes. In fact, this task seems relatively easy, but even that is difficult enough to have been the motivation of hundreds of publications. The rest of the description given is in fact extremely difficult. It is all about knowledge acquisition, representation, and incorporation and is part of the much wider field of research, including pattern recognition and artificial intelligence. This chapter will only deal with the first part of the problem. In the section on

300

MARIA PETROU

J \/---FIIXJRE2. The output of a good edge detector when applied to the image of Fig. la.

postprocessing we shall come the nearest we shall come to the incorporation of knowledge, but even that is going to be very elementary and nothing in comparison to the knowledge a human utilises when producing something like Fig. lb. It is my personal belief that the quest for the best edge detector has reached saturation point from the point of view of the image-processing approach and that any breakthough or significant improvement in the future will have to come from the integration of the edge-detection process

THE DIFFERENTIATING FILTER APPROACH TO EDGE DETECTION

301

into a vision system, where knowledge is used at, and information is transferred back and forth, between all the levels of understanding and image analysis. As I said earlier, at first glance the identification of points where the intensity changes seems to be very easy. In fact it seems that we can achieve it by just scanning along the signal and noting any difference in the greylevel value we see. Every time this local difference is a local maximum, we note an edge. Let us do this first for a one dimensional signal, namely one row of the image. In Fig. 3a we plot the grey values in the image along a certain row and at the vicinity of an edge. To identify places where the grey value changes, I scan the signal and find the difference in grey-level values between a pixel and its next neighbour. Formally this process is called “convolution by the mask ” Ideally, this difference represents the local derivative of the intensity function calculated at the point halfway between the two successive pixels. For the sake of simplicity, however, we may assign the difference to the pixel under consideration. This small discrepancy can be avoided if we use the next and the previous neighbour to estimate the local difference. Since these neighbours are two interpixel distances away from each other, we may say that “we convolve the signal with mask 1 -0.5 I 0 1 0.5 1 . ’ If I; is the grey value at pixel i, we may say that the difference AI; at the same pixel is given by:

1-11-1.

Figure 3b shows the result of this operation. An edge is clearly the point where this difference is a local maximum. The most noticeable thing about Fig. 3b is that if we identify all the local maxima in the output signal we shall have to mark an edge in several places along the signal, most of which are spurious. This is shown in Fig. 3c. As we can see, the edge points detected are so many, that they hardly contain any useful information. The obvious cause of the problem is that when we do the edge detection, we ignore small and insignificant changes in the intensity value. When the computer does it, it does not know that. Therefore, we have to tell it! The proper terminology for this is thresholding. Effectively we tell the computer to ignore any local maximum in the value of the derivative which is less than a certain number, the threshold. How we choose this number is another topic of research. It can be done automatically by an algorithm we give the computer, or it can be done manually, after we look at the values of the local maxima, or even more grossly, by trial and error, until the result looks good. Alternatively, one may try to stop all these spurious local maxima from arising in the first place. If we look carefully at the image in Fig. la, we shall see that although the wall in the background is expected to be of

302

MARIA PETROU

raw data

(a)

smoothed data

(d)

0

:

n X u

~

1.

=*

~ ....mD.

8

P4

.8

8

8 .

( v -

0

0

0

5

10

15

20

5

0

15

20

X

X

first difference

(b)

10

(el first difference of smoothed dato

v -

UP4 1 . 1

8

0

0

0

5

10

15

X

(C)

local maxima

X

20

0

5

10

15

20

X

(fl local maxima in smoothed dota

X

FIOURE 3. Top panels: A raw signal and its smoothed version. Middle panels: The first difference of the signals in the top panels. Bottom panels: The locations of the local maxima in the values of the first difference.

THE DIFFERENTIATING FILTER APPROACH TO EDGE DETECTION

303

uniform brightness, it seems to contain quite a variation in grey tones in the image. These variations are those that create all the spurious edges (see for example Torre and Poggio, 1986). The major reason of this lack of uniformity even for regions that in reality are very uniform, is the thermal noise of the imaging device. The best way to get rid of it is to smooth the signal before we apply any edge detection. This can be done, for example, by replacing the grey value at each pixel position by the average value over three successive pixels. The resultant signal then will look like Fig. 3d. Formally, we can say that the smoothed value Si at pixel i is given by:

We then apply the difference operation

to the smoothed signal and obtain the signal in Fig. 3e. If we keep only the local maxima, we obtain the signal in Fig. 3f. It is clear that some thresholding will stiIl be necessary, although fewer spurious edges are present in this signal than in the signal of Fig. 3c. There are a number of things to be noticed from the above operation: After the smoothing operation, the edge itself became very flat and shallow, so its exact location became rather ambiguous. In fact, the more smoothing is incorporated, i.e., the more pixels are involved in the calculation of Si by Eq. (2), the more blurred the edge becomes and the fewer the spurious edges that appear. This observation is known as the uncertainty principle in edge detection. In the next section we shall see how we can cope with it. We can substitute from Eq. 2 to Eq. 3 to obtain:

That is, we can perform the operations of smoothing and differencing in one go, by convolving the original signal with an appropriate mask, in this case with the mask - 1 -$ I 0 I I This is because both operations, namely smoothing and differencing, are linear. It is not always desirable for the two operations to be combined in that way, but sometimes it is convenient.

I 4

4 1.

We shall see in the next section how the two major observations above will be used in the process of designing edge-detection filters. However, first

304

MARIA PETROU

we shall see how the simple ideas above can be extended to the detection of edges in two dimensional signals, i.e., images. There are two major differences between the location of discontinuities in a one-dimensional and in a two-dimensional signal: First, sharp changes in the value of a two-dimensional function coincide with the local maxima of the first magnitude of the gradient of the function. For a two-dimensional signal the smoothing does not have to take place along the same direction as the local differencing. The gradient of a two-dimensional function I(x, y ) is a vector given by:

ar

az

gE-i+-j ax ay

where i and j are the unit vectors along the x and y directions respectively. Two things are obvious from the above expression. First, we must estimate the derivative of the intensity function in two directions instead of one; and second, an edge in a two-dimensional image is made up from elements, called edgels, each of which is characterized by two quantities, the magnitude of the gradient and its orientation. The orientation of an edge1 is useful for some applications, .but it is not always required. Clearly, an edge must coincide with places where lgl is a local maximum along the direction it points. In the rest of this section we shall combine all the above ideas to create our own first edge detector which in spite of all its simplicity seems to work quite well for a large number of images and has served the vision community for several years as a quick “dirty” solution, before, and even after, much more sophisticated algorithms became available. It is called the Sobel edge detector after Sobel, who first proposed it (see, for example, Duda and Hart, 1973). First we want to estimate the partial derivative of the brightness function along the x axis of the image. To reduce the effect of noise, we decide to smooth the image first by convolving it in the y direction by some smoothing mask. Such a mask is 1 1 1 2 1 1 I. We then convolve the smoothed image along the x axis with the mask W I T ]and estimate the local partial derivative aI/ax, which we call AZx. We follow a similar process in order to estimate the partial derivative of the brightness function, AI,, i.e., we smooth along the x axis by convolving with the smoothing mask ( 1 1 2 1 1 1and we difference along the y axis. We can then estimate the value of the magnitude of the gradient at each position by computing:

G

I

AI;

+ AIy”

(6)

THE DIFFERENTIATING FILTER APPROACH TO EDGE DETECTION

305

Notice that C is not the magnitude of the gradient but rather the square of it. Since only relative values matter, there is no point in adding to the computational burden by taking square roots. We thus create a new output that at each pixel position contains an estimate of the magnitude of the gradient at that particular position. We can also estimate the approximate orientation of the gradient at a given position by comparing the outputs of the differences along the horizontal and the vertical directions at each position. If the horizontal difference is the greatest of the two, then a mainly vertical edge is indicated and to check for that we check if the magnitude of the gradient is a local maximum when compared with the values of the gradient the two horizontal neighbours of the pixel have. If the vertical difference is the largest one, a horizontal edge is indicated, and to confirm that we check if the gradient is a local maximum in the vertical direction. If either of the two hypotheses is confirmed, we mark an edge at the pixel under consideration. Figure 4a shows the result of applying this algorithm, to the image of Fig. la. It is clear that lots of spurious edges have been detected, and some postprocessing is necessary. After some trial and error concerning the value of a suitable threshold, Fig. 4b was obtained. We summarize the basic steps of this algorithm in Box 1 .

Coiivolvc

iiil)ii[

a

iinage vertically w i t h mask 2

Box I . A simple edge-detection algorithm.

306

MARIA PETROU

(a) (b) FIGURE 4. (a) The output of the algorithm presented in Box 1 when applied to the image of Fig. la. (b) The same output after thresholding.

The results shown in Fig. 4 are very encouraging, and if all images exhibited the same level of noise as the image in Fig. la, there would not have been much point for further refinement. It is worth, however, experimenting with some more noisy images, notably an image like the one in Fig. 5a. Figure 5b shows the output of the above algorithm. This output

(a) (b) FIGURE5. (a) A synthetic image with 100% additive Gaussian noise. (b) The result of applying the algorithm of Box 1 plus thresholding to the previous image.

THE DIFFERENTIATING FILTER APPROACH TO EDGE DETECTION

307

is after a suitable threshold was chosen by trial and error! Clearly, such a result is very unsatisfactory, and the need is indicated for some more sophisticated approach to the problem.

11. PUTTING THINGSIN PERSPECTIVE

The approach we shall discuss in this chapter is only one way of dealing with the problem of edge detection. The reason it has been chosen is because it has prevailed over all other approaches, and it has become very popular in the recent years. In this section we shall review briefly the other approaches so that things are in perspective. Edge detection has attracted the attention, of researchers for a long time since the early days of computer vision. Quite often people interested in other aspects of vision bypassed the problem assuming that “a perfect line drawing of the scene is available.” As we mentioned in the introduction, a perfect line drawing has eluded us for a long time, and it has become increasingly obvious that it cannot be obtained in isolation of the other aspects of vision research. In spite of that, hundreds of papers have been published on the subject, and although it is impossible to review them all, we can at least record the basic trends in the field. We can divide the approaches into three very gross categories: The region approach. The template-matching approach. The filtering approach. The region approaches try to exploit the differences (often statistical) between regions which are separated by an edge. Examples of such approaches are the work of de Souza (1983), Bovic and Munson (1986), Pitas and Venetsanopoulos (1986), Kundu and Mitra (1987), and Kundu (1990), and they are often referred to as “nonlinear filtering approaches.” Such edge detectors are particularly successful when there is a prior hypothesis concerning the exact location and orientation of the edge, i.e., when the approach is model based and relies on hypothesis generation and testing (e.g., Graham and Taylor, 1988). An alternative type of approach is based on region segmentation that exploits the statistical dependence of pixel attributes on those of their neighbours. This statistical dependence of the attributes of pixels which make up a region may be discontinued, when a certain quantity concerning two neighbouring pixels exceeds some threshold. Such an approach is usually incorporated into a more general process of image segmentation or image restoration using Markov random fields, for example, and the proper term for it is “incorporating a line

308

MARIA PETROU

process in the system.’’ The “line process” is in fact the implicit acceptance of an edge between pixels which are sufficiently dissimilar. An example of such work is the work of Geman and Geman (1984). In general these methods tend to be slow. They also rely on estimates of the Markov parameters used, i.e., on image or at least region models, which are not usually available, and it is not easy to estimate. In the template-matching approaches, one can include the approach of Haralick (1980 and 1984) and Nalwa and Binford (1986), who model either the flat parts of the image function (facet model), or the edge itself. In the same category one should include the robust approach by Petrou and Kittler (1992) who tried to identify edges by fitting an edge template at each location which, however, did not minimize the sum of the squares of the residuals, but it rather relied on an elaborately derived kernel which weighed each grey value according to its difference from the corresponding value of the template. The process was very slow, and the results did not seem convincingly better than the results of the linear approaches. The problem with all model-based approaches (region-based and template-based included) is that one may tune the process very well according to the assumptions made, but the assumptions, i.e., the models adopted, do not apply at all edges in an image, so beautifully built theories fail because reality stubbornly prefers exceptions to the general rules! However, the last word has yet to be said about these lines of approach, and it is possible that in the future they may produce better results. Under the third category of edge detectors, we include all those which rely on some sort of filtering. Filters are often designed to identify locations of maximal image energy, like those by Shanmugam et al. (1979) and Granlund (1978), or to respond in a predetermined way when the first or the second derivative of the signal becomes maximal. In general, one understands filtering as a convolution process; this however is not always true and nonlinear filters which effectively adapt to the local edge orientation with the purpose of maximally enhancing it have been developed (for example see van Vliet et al., 1989). In the same category of nonlinear filtering one should include the morphological operator of Lee et al. (1987). A special type of filters was proposed by Morrone and Owens (1987). These were in quadrature with each other, designed to locate positions of energy maxima and classify the features detected by examining the phase of the filter outputs. The filters are chosen to form a Hilbert transform pair, and the sum of the squared outputs of the two convolutions is supposed to be the energy of the signal. Detailed experimentation on this claim has shown that this is not true exactly, unless one of the filters is matching the signal, something that is very difficult when the signal may be of varying profile. However, such filters have become reasonably popular recently and research

THE DIFFERENTIATING FILTER APPROACH TO EDGE DETECTION

309

in that direction is still under development (see for example Perona and Malik, 1992). The attraction of the approach relies on the simultaneous identification of step type edges and line type edges. In this chapter we shall concentrate on the filters that are designed to identify maxima of the first derivative of the signal. The reader is referred to the above-mentioned references for details of other approaches and the brief survey of recent trends in edge detection by Boyer and Sarkar (1992).

111. THEORY

In Section I we saw some of the fundamental problems of edge detection, we constructed our first edge detector, and we saw its inadequacy in coping with very noisy images. To be able to d o better than that, we must examine carefully what exactly we are trying to do, express the problem in a way that can be tackled by the tools an engineer and designer has at his or her disposal, and finally solve it. That is what we shall attempt to do in this section. It is not difficult to convince ourselves by looking at Fig. 5 that the problem we really try to solve is to detect a signal in a very noisy input we are given. We saw that the intuitive filters we used in Section I did not really work. To choose another filter, we really need to know something more about the nature of the signal we try to detect and the noise we are dealing with. So, we must start by modelling both signal and noise. Since the noise most of the time is caused by the thermal noise of the imaging device, the most plausible way to model it is to assume that it is additive, Gaussian and homogeneous white noise with zero mean and standard deviation 6. The word “additive” means that the input signal I ( x,y ) , can be written as:

I(x,Y ) = u(x,u) + N x , u),

(7)

where u(x,y ) is the signal we try to isolate and n(x, y ) is the noise. The word “Gaussian,” means that at every location (x,y ) , the noisy component n, say, of the grey value, is chosen at random, from a Gaussian distribution of the form

where p(x,y ) is the mean and a(x, y ) is the standard deviation of the noise. This expression implies that at each location the noise is of different level and standard deviation. This would make the noise inhomogeneous over the image, something which is both unlikely to occur and difficult to handle.

310

MARIA PETROU

That is why we assume that the noise is “homogeneous,y’ and that the quantities p(x, y) and a(x, y) are not really functions of position. Further, if p were different from zero, there would have been a biased component to the noise which could easily be detected and removed at a preprocessing stage, The word “white” means that if we consider an image which consists of noise only, its Fourier spectral density is flat, i.e., all frequencies contribute to it with the same amplitude. Another way of saying the same thing is to state that the noise is uncorrelated. This means that the noisy grey value added to the signal grey value at each location is not affected by and does not affect any other noisy grey value added anywhere else in the image. That is, if I consider any two pairs of grey noise values at a certain relative position r, and I average the product of all possible such pairs at the same relative position over the image, the result will tend to zero as the size of the image I consider gets larger and larger. When, however, I compute the average square grey value of the noise field, the result will tend to become equal to the standard deviation of the noise, as the size of the image we consider gets larger. We say then that the autocorrelation function Rnn(r)of the noise field is a delta function:

It is known that the Fourier transform of the autocorrelation function of a random field is the spectral density of the field and knowing that the Fourier transform of a delta function is a constant, we deduce that the spectral density of uncorrelated noise is white, i.e., constant. Having understood the noise we are dealing with, or at least that we assume we are dealing with, we turn next to the method we are prepared to use in order to identify edges. To keep matters simple and fast, we prefer to use linear filters. There are various reasons for that: The implementation of linear filters is easy. In fact, one can use the general framework for edge detection given in Box 1 and only replace the simple masks by some more sophisticated ones. Various attempts have been made to replace the linear process of edge detection with some nonlinear one, but they did not show convincingly enough that they could produce any better results than the linear approach. We understand exactly how the linear approach works, thus we feel more in control when we use it. Edge detection is only a preprocessing stage to a vision system, and we need some method that works fast and efficiently, while nonlinear methods tend to be rather slow.

THE DIFFERENTIATING FILTER APPROACH TO EDGE DETECTION

3 11

For these reasons, we shall restrict ourselves to the design of convolution filters. Just as we did in Section I, we shall start by considering one-dimensional signals only. Let us say, therefore, that the noisy signal we have can be expressed as: Z(x) = u(x) + n(x). (10) We are seeking to define a convolution filter f ( x ) which, when convolved with the above signal will produce an output with a well-defined maximum at the location of the edge (feature) we wish to detect. We can try to systematize the desirable properties of the filter we want to develop, as follows: We want to be able to detect the edge even at very high levels of noise, in other words, we want our filter to have high signal-to-noise ratio. We want the maximum of the output of the filter to be as close as possible to the true location of the edge/feature we want to identify. We want to have as few as possible spurious maxima in the output. These basic requirements from a good edge filter were first identified by Canny (1986), who set the foundations of the edge-filter theory. Although the above requirements as stated seem vague and general, one can translate them into quantitative expressions that can be used in the filter design. Before we do that, we must discuss first the properties of the filter function itself

0

Since the filter is assumed to be a convolution filter, we do not want to have to convolve with a filter of infinite size. We do not want to use a filter which goes abruptly to zero at some finite value, because sharp changes in a function can only be created by the superposition of strong high-order harmonics when Fourier analysis is performed. Since convolution of two functions corresponds to the multiplication of their spectra, the presence of significant high-frequency components in the spectrum of the filter will imply that the high-frequency components of the input signal will be acentuated. However, the noise is assumed white, and the signal is the product of an imagelsignal capturing device which naturally is having a band limited frequency of operation. Thus, the high frequencies in the input signal will be those that are dominated by the noise, while the low frequencies will be dominated by the spectrum of the true uncorrupted signal. Accentuation of the high frequencies, therefore, is equivalent to accentuation of noise, contrary to what we try to achieve. For this reason, we want the filter to go smoothly to zero at its end points.

312

MARIA PETROU

Another desirable property the filter should possess is that its output should be zero if the input signal does not contain any features, i.e., if the input signal is absolutely flat. This can be achieved if the filter has zero direct component. The above mentioned requirements can be summarized as follows:

f(* w ) = 0,

f ’ ( kw) = 0,

f ( x ) = 0 for 1x1 > w , (11)

Sr,f(X)dX = 0 wheref’(x) is the first derivative of the filter, and w is its finite half-width. A. The Good Signal-to-NoiseRatio Requirement

To be able to tell whether a filter has good signal-to-noise ratio or not, without trying it in practice, we must calculate expressions of the filter response to the signal and to the noise separately. Since the filter is assumed to be a convolution filter, its response to the signal can be written as:

~(2) = or equivalently,

L

. :S

~(2) =

u(x)f(i- X) dx

(12)

~ ( -2x)~(x) dx

(13)

given that the order by which two functions are convolved does not really matter. Similarly, the response of the signal to the noise component is:

L¶

=

S_. W

00

v(2)

n(x)f(2 - x) dx =

n(2 - x)f(x) dx

(14)

The noise is a random variable, and thus v(2) will be a random variable too. The only way we can characterise it, then, is through its statistical properties. One way to estimate its magnitude, is to compute its mean square value denoted by If we multiply both sides of Eq. (14) with v(2) and take the expectation value, we have:

m.

E ( [ v ( i ) ] 2=]

iw

f(x)E[v(Z)n(i - x)] dx

(15)

-W

where we have made use of the following facts: 1. The quantity v(2) does not depend on the variable of integration, so it

can be placed inside the integral sign on the right-hand side of the equation.

THE DIFFERENTIATING FILTER APPROACH TO EDGE DETECTION

3 13

2. We can exchange the order of integration and taking of the expectation value on the right-hand side of the equation, because the expectation value is taken over all possible outcomes of the random process that gives rise to the noise at the specific location, i.e., by definition:

where N(n(i))is any function of the noise. 3. The expectation value integration affects only the random variables, i.e. quantities that are functions of the noise and not the deterministic filter function f ( x ) . The autocorrelation function of a random variable and the cross-correlation function between two random variables are respectively defined as: R""(7)= E(n(x)n(x+

T)]

&(7)

= E(v(x)n(x + 7))

(17)

Making use of this definition, Eq. (15) can be rewritten as:

mw)2) =

j

W

f(x)R,(-x) dx

(1 8)

-W

It is clear from the above expression that we need an expression for R,,(x). We start from Eq. (14) as before, only that now we multiply both sides with n ( i ) . Following the same steps we obtain:

s_. W

E { v ( i ) n ( 2 ) )=

f ( ~ ) E ( t ~-(xf) n ( i ) ] dx

(19)

Expressed in terms of the autocorrelation and cross-correlation functions, the above result can be restated as:

I_. W

Rvn(2- 2) =

f(x)Rfln(2- 2

+ X ) dx

(20)

However, the autocorrelation function of the noise is supposed to be given by Eq. (9). If we make use of that expression, we find that:

RJi

- 2) =

aZF(2 - i )

(21)

The above equation equivalent can be written as:

R,"W =

&-(-XI

(22)

Finally, substituting into Eq. (18). we obtain: W

(23) -W

314

MARIA PETROU

Having computed the response of the filter to the signal and estimated the magnitude of its response to noise, we are ready now to define the signal-tonoise ratio of the filter output:

We can simplify this expression, by saying that we redefine the origin of the x (or 2)axis to be the location of the edge we wish to detect. Then, at the location of the edge, the signal-to-noise ratio will be given by the above expression calculated at 2 = 0. Further, we do not need to carry around constants that do not affect the choice of functionf(x). Such a constant is the standard deviation of noise. We can define, therefore, a measure of the signal-to-noise ratio, as follows:

The filter functionf(x) should be chosen in such a way that this expression is as large as possible. There are some interesting observations we can make by just looking at expressions (24) and (25): It is known that any function can be written as the sum of a symmetric and an antisymmetric part. Let us say that our filter functionf(x) can be written as: f(4 = + f,(x) (26)

m)

where f,(x) is its symmetric part and f,(x) is its antisymmetric part. On substitution in Eq. (25) we obtain: S=

,!I 4-x)f,(x)dx + r w u(- x)f,(x) dx dSrwfs2(x)dx +, ! S fa2(x)dx + 2 !yw f,(x)f,W dx

(27)

So far, we have not made any assumption concerning function u(x) with which we model the feature we wish to detect. Since our purpose is to detect sharp changes in the signal, centered at x = 0, the signal must be modelled by an appropriate function, like a sigmoid, or a step function. Further, since the filter is made to give zero response to a constant background, such a function should only model the signal without its direct component. Therefore, any function which models an edge reasonably will be an antisymmetric function. Given that the product of a symmetric and an antisymmetric function is antisymmetric, and given that we integrate over a symmetric interval, the

THE DIFFERENTIATING FILTER APPROACH TO EDGE DETECTION

3 15

implication is that j!, u(- x)&(x) dx = 0 and I !, f,(x)&(x) dx = 0. The symmetric part of the filter function, therefore, does not contribute at all to the magnitude of the signal. On the contrary, it contributes to the magnitude of the filter’s response to noise as it can be seen from the extra integral that remains in the denominator of the above expression. We conclude, therefore, that the filter for the detection of edges should be an antisymmetric function. If we decide to model the edge we want to detect by a step function, the “strength” of the signal will be the amplitude of the step at x = 0, call it A . Then it is clear from expression 24 that this amplitude can come out of the integral in the numerator, and thus the signal-to-noise ratio we measure will be proportional to the true signal-to-noise ratio M a . If instead of using filterf(x) we use filter af(x), the signal-to-noise ratio for the filter response is not going to change, i.e., it is independent of the filter amplitude. If on the other hand, we scale the size of the filter and make it go to zero at x = +pw, say (with /3 > l), instead of w , the signal-to-noise ratio will be scaled up accordingly by @. We can see that as follows: The scaled filter would bef(x/P) and obviously would go to zero when x = +pw. If we substitute this filter expression in (25) and adjust the limits of integration appropriately, we shall have a measure of the signal-to-noise ratio of this particular filter. To relate it to the signalto-noise ratio of the original filter, we must change the variable of integration to y = x//3, say. Then it is trivial to see that the signal-tonoise ratio of the new filter is @ times the signal-to-noise ratio of the old filter. Thus, using larger filters we improve upon the signal-tonoise ratio performance.

B. The Good Locality Requirement We can turn our attention now to the problem of good locality. The edge we wish to detect will be marked at the location of an extremum of the output, i.e., at the point where

Using Eq. (12) we can compute as(X)/&f as: as(2) = -

a2

1

OD

-m

u(x)f’(Z - x ) d x

316

MARIA PETROU

Similarly, from Eq. (14) we obtain:

av(9 = n(xlf'(2 - x ) dx a2 -m It will be convenient later if we exchange the order of convolution in the above expression and without any loss of accuracy we rewrite it as: ~

In the absence of noise, the extremum in the filter output would coincide with the true location of the edge that is assumed to be at x = 0. This is very easy to see. Indeed, in the absence of noise, Eq. (28) becomes:

At the point 2 = 0 this expression is: OD

(33)

We have argued earlier that the filter should be an antisymmetric function, just like the function u(x) with which we model the signal. The first derivative of an antisymmetric function is a symmetric function, and the product of a symmetric and an antisymmetric function vanishes when integrated over a symmetric interval. The implication is that in the absence of noise

which means that the output is an extremum at the exact location of the edge. Because of the noise, however, the location of the extremum of the output will be misplaced, as computed from Eq. (28). The amount by which it will be misplaced is a random variable, and we can compute its mean-square value. Indeed, the misplacement is not expected to be a very large number, so we may expand the function f'(2 - x), which appears in Eq.(29), as a Taylor series about the point x' = 0:

f ' ( 2- x ) = f ' ( - x )

+ 2f"(- x ) +

(3 5 ) On keeping only the first two terms of the expansion and by substituting in Eq. (29) and remembering that f'(- x) is a symmetric function, we obtain: * . a

THE DIFFERENTIATING FILTER APPROACH TO EDGE DETECTION

3 17

We could use the above result to substitute directly in Eq. (28); however, it will give us more insight to put it in a different form. It is obvious from the properties of convolution that the two expressions below are identical:

If we compute both sides we obtain:

I-00

uy-x)f'(x) dx =

j

u(x)f"(-x) dx

(38)

m:

On the grounds of this result, Eq. (36) can be written as: as(.q

W

-= x ' s a2

u'(-x)f'(x) dx

(39)

-W

If we use this result and that of Eq. (31) into (28) we obtain:

2

jw

U'(-X)f'(X)

-W

dx =

SIW

n(2 - x)f'(x) dx

(40)

Both sides of the above expression contain random variables and we can compute their square expectation values as follows:

Notice that the expectation integral operates only on the random variables and not on the deterministic factors. The expectaction value on the righthand side of this equation is effectively the expectation value of the square output of the convolution of filter f ' ( x ) with pure noise. Equation (23) above tells us that this is equal to d j Y w f ' ( x l f '(x) dx. Thus, the expectation value of the square misplacement of the location of the maximum in the output away from the true edge location is:

Clearly, the smaller this expectation value is, the more closely the output maximum is to the true edge location. Thus, we define a good locality measure by an expression proportional to the inverse of the right-hand side of the above equation and without any unecessary constants involved:

318

MARIA PETROU

We can make some interesting observations by looking at this expression: The good locality measure is independent of the filter amplitude. If we scale the filter as we did in the case of signal-to-noise ratio, the good locality measure of the scaled filter will turn out to be l/@ the good locality measured of the unscaled filter. Thus, the larger the filter is, the more ambiguity is introduced into the exact location of the detected feature. This is exactly the inverse of what we concluded about the signal-to-noise ratio, and the two conclusions together are known as the “uncertainty principle in edge detection.” For any two functionsfi(x) and fi(x), Schwarz’s inequality states that

with the equality holding when one function is the complex conjugate of the other. If we apply it to the expressions for S and L as given by Eqs. (25) and (43) respectively, we shall find that the filter that maximizes the signal-to-noise ratio is given byf(x) = u(-x) and that the filter that maximizes the good locality measure must satisfy f ’ ( x ) = u’(-x). This means that both measures can be maximized by the same function, i.e. , the “matched filter” for the particular signal. The last observation led Boie et al. (1986) to dispute the validity of the uncertainty principle and advocate the use of matched filters for edge detection. The uncertainty principle, however, is referred to the size of the filter and not its functional form. The question Canny (1986) and other people who followed this line of research tried to answer was: If I fix the size of the filter, how can I choose its shape so that I compromise between maximizing its signal-to-noise ratio and its good locality performance? For an isolated edge modeled by a step function, the matched filter is a truncated step of the opposite sign. This is the well-known difference-ofboxes operator (see, for example, Rosenfeld and Thurston, 1971), which due to its sharp ends creates an output with multiple extrema, something we wish to avoid. Boie et al. (1986) avoided this problem by not making the assumption of white Gaussian noise. Instead they analysed the physical causes of noise in the imaging device and came up with a nonflat noise spectrum. It is not clear from their work whether this by itself was adequate to make their filters go smoothly to zero or not. Their matched filters do go to zero smoothly, but some of them seem to be more than 100 pixels long! Further, instead of modelling the edge itself, they modelled its first derivative by a Gaussian function. If an edge were an ideal step edge, its derivative would have been a delta function. Clearly, the band limited

THE DIFFERENTIATING FILTER APPROACH TO EDGE DETECTION

3 19

range of operation of the imaging devices converts any such derivative to something that is better approximated by a Gaussian. In the case of white noise expressionf'(x) = u'(-x) would have implied a filter made up of the integral of a Gaussian that has sharp ends, unless one forces it in some way to go to zero, perhaps making some extra assumptions concerning the proximity of neighboring edges. In general, the matched filters by Boie et al. (1986) have not gained much popularity, perhaps because they do not seem very practical,

C. The Suppression of False Maxima Since we consider filters that go smoothly to zero at their end points, the only source of false maxima in the output is the response of the filters to noise. Rice (1945) has shown that if we convolve a function with Gaussian noise, the output will oscillate about zero with average distance between zero crossings given by:

where R,(r) is the spatial autocorrelation function of function g(x),defined by: (46)

Upon differentiation, we obtain: (47)

We can define a new variable of integration in the integral of the right-hand side; i E x + r. Then:

I-I-. m

R&(r) =

g(R - r)g'(Z)di

(48)

Upon one more differentiation, we obtain: a0

R;:(T) =

-

g ' ( i - r)g'(Z)dx'

(49)

Thus, the expressions that appear in Eq (45) can be written in terms of the convolving function and its derivative as:

1

m

R,(O) =

-00

[s(x)12dx,

R;;(O) =

-

J

00

[s'(x)I2dx -m

(50)

320

MARIA PETROU

It is clear that the false alarms in our case are the extrema in the output of the convolution of the filter with the noise, which coincide with the zeros in the first derivative of the output. These are the same as the zero crossings that will arise if we convolve the noise with the first derivative of the filter. That is, the false alarms in the output arise from the notional convolution: I!’, f’(x)n(Z - x)dx. The role of function g(x), therefore, in our case is played byf’(x) and thus, a measure of the average distance between the extrema in the output of our filter when we convolve it with noise can be defined as:

where we divided by w to make the expression scale-independent. We can use this expression as a measure of reduced number of spurious edges in the output. Clearly, the larger the average distance between the extrema in the output due to noise, the smoother the output will look and thus the easier it will be to isolate the true edges from the spurious ones. D. The Composite Performance Measure

We have derived in the previous three subsections quantitative expressions for the qualities we would like our filter to possess. The way these expressions have been defined implies that a good filter should maximize the values of all three of them. It is clear, however, just by looking at Eqs. (25) and (43) on one hand and (51) on the other, that it is impossible to maximize all three quantities simultaneously, since the integral,!j [f”(x)I2dx appears in the numerator in (43) and in the denominator in (51). There is a need, therefore, for some sort of compromise, where we try to satisfy all three criteria as well as possible. This can be done by forming a composite criterion, call it P, by combining the three criteria above. We then have to choose functionf(x) in such a way that this composite criterion is maximal. Such a function will probably depend on certain parameters that will have to be chosen so that the boundary constraints are satisfied and the composite criterion does take a maximal value. The way various researchers proceeded from this point onwards diverges and has led to a variety of filters admittedly not very different from each other. The exact details of the optimization process used are not of particular interest and can be found in the respective references given. We shall outline here only the basic assumptions of each approach. Canny’ composite criterion was formed by multiplying the first two quantities only, S and L: P, I SL. He then chose the filter function by maximizing P, subject to the extra condition that C is constant. Canny’s

THE DIFFERENTIATING FILTER APPROACH TO EDGE DETECTION

321

model for an edge was a pure step function defined by: u(x) =

1 0

forx? 0 for x I0

Such a function is entirely self-similar at all scales, i.e., it does not introduce to the problem an intrinsic length scale, and thus the filter derived can be scaled up and down to fit the user’s size requirements. This way Canny derived the following filter:

f(x) =

I

eax[K1 sin(&) + K2 cos(&)] + e - Y K 3 sin(Qx) + K4 cos(Qx)] + K , -f(-x)

for ’ w 5 x I0 (53) forOsxsw

This filter depends on seven parameters K , , . . .,K , , a, and a, which can be chosen so that the boundary conditions expressed by the first two Eqs. (1 1) and the antisymmetry implication thatf(0) = 0 are satisfied. Further, as we mentioned earlier, the scaling of the filter does not affect its performance. Thus, one of the filter coefficients can arbitrarily be set to one so that the number of unknown parameters reduces to six. The problem is still underconstrained as the three boundary conditions are not enough to specify all six parameters, which have to be chosen so that C and P, take maximal values. Canny argued that it was not the exact value of C that mattered, but the error created to the output due to the presence of false maxima in relation to the error introduced by thresholding at the end. Thus, he tried to choose the values of the remaining three parameters (after the boundary conditions had been used) to maximize P, and at the same time minimize the error caused by false maxima expressed as a fraction of the error caused by thresholding. He used stochastic optimization to scan the 3D parameter space since the function he had to optimize was too complicated for analytic or deterministic approaches. Spacek (1986), in order to reduce the ambiguity, created a composite performance measure by multiplying all three criteria to form a composite one. Spacek’s composite criterion, therefore, is: P, E (SLC)2. He also modelled the edge by a step function. The best filter then appears to be one which is given by the same equation as Canny’s filter 53, but with Q = a. Thus, the number of independent parameters on which the filter depended was reduced to five, After using the boundary conditions, Spacek fixed parameter a to one, as the filter seemed to be insensitive to it, and chose the remaining parameters so that the composite performance measure took maximal value. Petrou and Kittler (1991) followed similar to Spacek’s approach but argued that the best model for an edge is a ramp, since any image processing

3 22

MARIA PETROU

device will smooth out all sharp changes in an image due to its finite band width of operation. The edge model they assumed was: 1 - 0.5e-" u(x) = [0.5esx

for x 2 0 for x 5 0

(54)

-

-

where s is some positive constant possibly in the range 0.5 to 3 which is intrinsic to the imaging device and thus identical for all scene step edges (and thus, image ramp edges) in images that were captured by the same device. The filter they derived is given by:

+

eax[KIsin(ux) + K2 cos(ux)] e-ax[K3sin(ux) + K4 cos(ux)] for - w Ix I0 + K5+ K,e" (55) -f(-x) for 0 s x Iw By a semiexhaustive search of the 2D parameter space they were dealing with (after the boundary conditions were used) they determined the values of the parameters which appear in the above expression so that the combined performance measure was maximized. They tabulated the parameter values for s = 1 and various filter sizes and explained how they should be scaled for different values of s. Finally, they derived the filter for step edges as a limiting case of the filter for ramps.

E. The Optimal Smoothing Filter The filters we discussed in the previous sections were meant to be filters that estimate the first derivative of the signal when it is immersed in white Gaussian noise. It is clear, however, from the definition of the convolution integral, that the first derivative of the output of a convolution is the same as the convolution of the initial signal with the derivative of the filter. We can turn this conjecture upside down and state that the result of convolving a signal with a differentiating filter can be obtained by convolving the signal with the integral of the filter first and then differentiating the output. The integral of the differentiating filter, however, is going to be a symmetric bell-shaped function that will act as a smoothing filter. Thus, we can separate the process of smoothing and differentiation so that we can perform one at a time along the directions we choose in a two-dimensional image, just as we did at the end of Section I. The integral of filter 55 is given by: h(x) =

ea[L1 sin(ux) + L2cos(ax)]

+ e-IIX[L3sin(@ + L4 cos(ax)]

+LSx+L,eSx+L, N-x)

for - w s x s O for 0 I x Iw

(56)

THE DIFFERENTIATING FILTER APPROACH TO EDGE DETECTION

323

.

The parameters L , , . .,L, can be expressed as functions of parameters K , , ...,Ks. Clearly, the extra constant of integration L, has to be chosen in such a way that h(0) = 0. Further, the filter should be scaled so that when it acts upon a signal with no features, i.e., constant, it will not alter it. In other words, the direct component of the filter should be 1. This is equivalent to saying that the sum of its weights must be 1. Petrou and Kittler (1991) have tabulated the values of the parameters of the above filter for s = 1 and for various filter sizes and explained how these parameters should be scaled for filters of different values of s and different sizes.

F. Some Example Filters To demonstrate the results of the theory we developed so far, we can use some filters in the scheme proposed in Box 1. For a start, we have to choose an appropriate value of the parameter s. This can be done by modelling a couple of representative edges in one of the images we plan to process, but in general the value s = 1 is a quite representative value. So, we shall use, for simplicity, s = 1 in the filters that we shall implement. Making use of the information given in Petrou and Kittler (1991), we can derive the differencing and smoothing filters of sizes 5 to 13, given in Box 2. Filters smaller than that are not worth considering because they tend to be badly subsampled and therefore loose any optimality property. Filters larger than that sometimes may be useful in particularly noisy images, but we shall not consider them here. Difleereiitiatiori filters -.

I I

J

Smootlrirrc filters

Box 2. Differentiation and smoothing filters of various sizes for ramp edges computed for slope parameters s = 1 . Incomplete filters are supposed to be completed using the antisymmetry and the symmetry property of the differentiation and smoothing filters respectively.

324

MARIA PETROU

I:

f-

FIOURE6. The ouput of applying the algorithm of Box 1, with the filters of size 13 of Box 2 plus thresholding, to the image of Fig. 5a.

We used the ramp filters of size 13 from the above table in the scheme defined in Box 1 to derive the results shown in Fig. 6. The improvement over the simple filters used in Section I is very noticeable, and it certainly justifies the effort. IV. THEORY EXTENSIONS

The work we discussed in the previous section forms only the bare bones of the line of approach reviewed in this article and has sparked off several papers concerned with improvements and variations of the basic theory. For example, the three criteria derived can be modified to apply to the design of filters appropriate for the detection of features with symmetric profiles, like roads and hedges in an aerial photograph (see for example, Petrou and Kittler, 1989; Petrou and Kolomvas, 1992; Petrou, 1993; and Ziou, 1991). However, linear feature detection is not the subject of this article, and we are not going to discuss it here. The major issues which merit discussion are the extension of the theory to two dimensions, the approximation of filters by simplified versions, their modification for more

THE DIFFERENTIATING FILTER APPROACH TO EDGE DETECTION

325

efficient implementation as well as considerations of interference from other features in the signal. We shall discuss all these points one at a time but first we must see why the need for extensions and modifications arose. There are three drawbacks of the theory we discussed so far: The whole theory was developed in the context of one-dimensional signals. Images, however, are two-dimensional signals, and edges in them appear in all sorts of orientations. The filters seem to be cumbersome and not very efficient in their implementation. Edges were considered as isolated features, and no thought was given to the influence of one edge to the detection of a neighbouring one. In the subsections that follow we shall discuss how various researchers dealt with the above-mentioned problems.

A. Extension to Two Dimensions The optimal filters derived in the previous section concern edges in one-dimensional signals. To determine the gradient of the image function we only need to convolve the image in two orthogonal directions with onedimensional masks, and that is what the filters are supposed to be doing. In a two-dimensional signal, however, an edge can have any orientation, not necessarily orthogonal to the direction of convolution. If we assume pure step edges, the differentiation filter should not be affected by that: A step edge remains a step edge even when it is viewed at an angle by the convolution filter. The problem arises when one wants to make the filters more robust to noise and thus propose to smooth first along the direction orthogonal to the direction of differentiation, just as we did in Box 1. Then the true orientation of the edge matters, since any smoothing in a direction that does not coincide with the direction of the edge will result in blurring it, and a blurred edge has no longer an ideal step function profile irrespective of orientation; in fact, the more slanted the edge is to the direction of convolution, the more blurred it will become. Canny (1986) solved the problem of edge orientation by convolving the image in more than two directions and combining the results. He used as smoothing filter the Gaussian to smooth the image first in the orthogonal direction to that of convolution. Spacek’s (1986) approach to the problem was different. On the grounds that the differentiating filter is antisymmetric and cannot possibly have a two-dimensional counterpart, he concentrated on the smoothing filter,

326

MARIA PETROU

which is symmetric and thus does have a two-dimensional version. To produce this two-dimensional version, Spacek (and Petrou and Kittler, 1991, as well) simply replaced the x variable in the definition of h(x) by the polar radius r. This two-dimensional filter was then used to smooth the image first, before differentiating it with a very simple differencing mask, for example like the one we used in Section I. There are various drawbacks of this approach: 0

The spectral properties of a one-dimensional filter are different from the spectral properties of its circularly symmetric version. For example, the Fourier transform of a finite width pulse is a sinc function, while the Fourier transform of its circularly symmetric version is expressed in terms of another Bessel function. Both transforms “look” similar, but the exact locations of their zeros, maxima, and the like are different. However, having said that, the method used by the above researchers is often used in practice for the extension of filters to two dimensions, because in general, a good one-dimensional filter when circularized gives rise to a pretty good two-dimensional one. The circularly symmetric smoothing filter is not separable, so that a full two-dimensional convolution has to be performed before differencing takes place. This tends to be slow and cumbersome.

There have been some unsuccessful attempts to modify the theory so that optimal two-dimensional filters can be directly developed. The attempts concentrated mainly in the development of filters that detect edges as zero crossing points, i.e., filters that estimate the locations where the second derivative of the image function becomes zero, which are obviously the locations where the first derivativeattains maximum. Such filters correspond to the Laplacian of a Gaussian filters of Marr and Hilderth (1980). However, attempts to derive such filters did not go very far, mainly due to the lack of a simple two-dimensional formula that corresponds to Rice’s onedimensional result concerning the density of zeros of the filter response to noise. Such a formula would give the average distance between zeros in the response of a two-dimensional filter to a noise field. Apart from the calculational difficulty in deriving such a formula, it is not even clear how to define what we mean by density of false zero crossings in two dimensions.

B. The Gaussian Approximation

We saw that the extension of the optimal smoothing filter to two dimensions led to filters that involve cumbersome two-dimensional convolutions. This is because the circularized filter h(r) is not separable. A two-dimensional

THE DIFFERENTIATING FILTER APPROACH TO EDGE DETECTION

327

Gaussian, however, is the product of two one-dimensional ones, and a two-dimensional convolution with it can be done by two cascaded onedimensional convolutions. For a filter of size 7 x 7, say, this implies 2 x 7 multiplications per pixel as opposed to 7 x 7 multiplications. This is one of the main reasons that the optimal filters have hardly been used in practice, and instead Gaussian filters have been preferred. The other reason is that Canny himself, when he derived his differentiating filters, proposed that they can be approximated well by the derivative of a Gaussian. In fact this statement was taken so literally that most people when they say they use the “Canny filter” actually mean the derivative of a Gaussian! In fact, a Gaussian filter can be made to look as similar or as dissimilar as one wishes to the optimal filter, according to the choice of the standard deviation used! Figure 7 shows two Gaussian filters that have been chosen to correspond to the optimal filter. The first one was chosen so that the maxima of the two filters match. If we look at the tails of these filters, we shall see that the Gaussian filter has a significantly sharp edge, which implies that the noise characteristics of this filter will be different from the noise characteristics of the optimal filter. Canny

-2.0 -1 0.0

I

-5.0

I

I

0.0

5.0

10.0

FIG.7 . Two Gaussian “approximations” to the optimal filter of size 13 given in Box 2.

328

MARIA PETROU

(and several other researchers as well) computed the performance measure of a Gaussian filter by simply substituting the Gaussian function into the formula of the composite performance measure and allowing the limits to go to infinity. By doing such a calculation Canny concluded that the performance measure of the Gaussian approximation is 80% of the performance measure of the optimal filter. I consider this calculation meaningless. The Gaussian filter is infinite in extent, and, when used in practice, it is bound to be truncated. Truncation will cause noise accentuation and false responses. These false responses, however, are not accounted for by the performance measure, which considers only the false responses caused by the random noise field within the finite boundaries of the filter. Thus, composite performance measures computed for Gaussian filters using the formulae derived in the previous section are meaningless; either one uses infinite filter limits for their computation or truncated ones. It seems more reasonable to fix the noise characteristics of the filters one tries to associate, in order to make any meaningful comparisons. We can do that as follows: Suppose that we digitize the optimal filter as we did in Section 1II.F. The continuous filter function then is represented by seven or thirteen, say, numbers. Thus, some of its properties are lost. In effect we band limit it that way, and, by doing so, we make it to be of infinite extent in the image domain. We can use this fact to compute the discontinuity we introduce by truncating the filter now to its finite size and choose the standard deviation of the Gaussian filter so that it has the same discontinuity at the point of truncation. Further, we scale the Gaussian filter so that the sum of the squares of the weights of the two filters are the same, since this is the quantity that enters into computing the response of the filter to noise by Eq. (23). Then we can claim that we have defined the Gaussian filter that best corresponds to the optimal filter as we chose it by fixing the two filters’ responses to noise. This Gaussian approximation to the optimal filter is also shown in Fig. 7. We can see that it is very different from the other Gaussian approximation, and, as expected, it produces different results. In general, anything that looks like the derivative of a bell-shaped function can be approximated by the derivative of a Gaussian, but what matters is what parameter values we choose for the approximating Gaussian, and this is something to which there is no easy guidance. In conclusion, Gaussian filters are clearly easier to compute and more efficient to implement, but one should have in mind that they are not the product of any optimality theory, and since they can be made to look and behave as dissimilar as one likes to the filters that resulted from the theory developed in the previous section, they should not be associated with them.

THE DIFFERENTIATING FILTER APPROACH TO EDGE DETECTION

329

C. The Infinite Impulse-Response Filters

Concerned with the efficient implementation of the optimal filters, Deriche (1987) had the idea of allowing them to be of infinite extent and imple-

menting them recursively. The idea of edge detection by infinite impulseresponse filters has been around for some time (see, for example, Modestino and Fries, 1977). Recursive implementation implies that the same number of operations are performed per pixel, irrespective of the actual filter effective size and shape. Deriche allowed the limits in the Canny’s performance criteria to go to infinity and thus derived a filter of the form: f(x) =

-

ce-alxl sin(&)

(57)

where c is a scaling constant, and a and SZ are the filter parameters, to be chosen by the user by experimentation. Deriche’s filter can be derived from Canny’s filter if we take the limit w CQ. Indeed, by just looking at Formula (53), which holds for x I0, it becomes obvious that for large w the term multiplied by e-ax will explode unless K3 = K4 = 0. Further, the only way to make the filter go smoothly to zero at infinity is to choose also K2 = K5 = 0. Thus, filter 57 arises. Taking this limit, however, is wrong, because although one can do it if one considers the function in isolation, the theory that gave rise to the derivation of this function does not apply for the limiting case. Indeed, the criterion C , Canny derived, measures the average distance between zero crossings as a fraction of the filter width. When the filter width becomes infinite, the C criterion is undefined or becomes zero. Deriche in his paper claims that he used this criterion measure in his optimization process (what he and Canny call k), but a careful examination of his equations shows that he actually used the same measure as Canny, i.e., the percentage of error caused by the presence of false maxima, as a fraction of the error due to thresholding, a quantity Canny calls r. Apart from the fact that maximization of this quantity r is not the same as maximization of k (or C in our terminology), the derivation of r is based on the definition of k and that is besieged by the fact that k is badly defined for infinite filters. Sarkar and Boyer (1991a) spotted the inadequacy of the theory behind the above filters and reworked out the correct performance criteria appropriate for infinite impulse response filters. In particular they redefined the criterion concerning the average density of false maxima. To do that, they defined an effective filter width as: -+

330

MARIA PETROU

Thus, the measure of average distance of false responses now becomes:

Sarkar and Boyer subsequently optimized the composite performance measure Canny had defined P, = SL subject to the condition that is constant. The equations they had, however, were too complicated and no analytic form of the filters could be found. They derived their filters numerically and approximated them by a function of the form:

f ( x ) = Ae-"X(cos(+) - cos(/3aor + 4))

for x > 0,

(60)

where /3 = 1.201, = 0.771, and A and CY > 0 are scaling constants that do not affect the shape of the filter. The recursive implementation of this filter can be achieved by scanning the image one line at a time from left to right, to form the input signal sequence x(n). Its reverse version, x,(n), is formed when we scan the line from right to left. If there are N pixels in a line, the input sequence and its reverse are related by x,(n) = x(N - n + 1). The double scanning of each line is necessary because the filter is defined for both positive and negative values of x , and thus consists of a causal and anticausal part. These two sequences are used to form the corresponding output sequences given by:

~ + ( n=) bl.~+(n- 1) + b 2 ~ + ( n 2) + b3Y+(n - 3) + u,x(n - 1) + u*x(n - 2),

(61)

y-(n) = bly-(n - 1) + bzy-(n - 2) + b,y-(n - 3) + u,x,(n - 1) + u2x,(n - 2)

(62)

The total filter output sequence will be:

u(n) = r+(n)- y-,(n),

vn,

(63) where [y-,(n)) is the inverse sequence of [ y - ( n ) ) .The parameters that appear in the above expressions are given by the following equations in terms of the filter parameters: b , = e-"(l

+ 2cos(pa)

b2 = -ble-a b - e-3U 3 -

+ 4))

a, = Ae-"(cos(4) - cos(/k~

u2 = Ae-2a(cos($) - cos(2Pa

+ 4)) - b,ul

THE DIFFERENTIATING FILTER APPROACH TO EDGE DETECTION

33 1

Sarkar and Boyer derived also the integral version of the above filter to be used for smoothing in the orthogonal direction to that of differentiation. The recursive implementation of these filters means that only 40 multiplications per pixel are performed irrespective of the values of the filter parameters. This number should be compared with the number of multiplications needed when the convolution filters derived in the previous section are used, which are 4 xfirter size. So, for filters of size 5, 7, or 9, convolution is more economical, while for filters of size larger than 10 the recursive implementation may give considerable computational gains. There are two drawback in the infinite impulse-response filters:

As it can be seen from Eqs. (61) and (62) above, the filter output is given by recursive relations that need initial values. Whatever initial values we use, their effect will propagate to all subsequent values of the sequence, that is, the recursive implementation of the filter introduces infinite boundary.effect! The infinite size of the filters in effect allows the interference of neighbouring edges. Indeed, the whole filter theory is based on the assumption that we wish to identify an isolated edge in the signal. The effect of several edges near each other on the output of the filter was not considered. How the theory can be extended to cope with this effect will be discussed in the next section. D. Multiple Edges

Shen and Castan (1986) also worked on the idea of infinite impulse response edge detection filters. In addition, they were concerned with the multipleedges problem and discussed how to avoid it. They used criteria similar to the criteria we developed in Section 111, but they appropriately modified them so that filters could be discontinuous at the centre. That way they derived an optimal smoothing filter, the first derivative of which can be used for the detection of edges as extrema of the first derivative of the signal and its second derivative for the detection of edges as zero crossings of the second derivative of the signal. Their filter has the form:

where c is a scaling constant and p is a positive filter parameter. The parameters of the filter should be chosen in such a way that the interference from neighbouring edges is minimized. The interference effect was studied by considering a series of crenellated edges so that the signal jumps from - A to + A at some irregular intervals. Their analysis was based on the

332

MARIA PETROU

following assumptions: 0

In any arbitrary space interval (xo, xo + Ax), the probability of having one step edge is independent of xo.

a The number of step edges in an interval (xl, x2) is independent

of that

in another interval (x3,x4) if the two intervals do not overlap. 0

If &(Ax) is the probability of having k edges in an interval Ax, limAx+o(P2(Ax)/Ax)= 0.

The above assumptions can be used to show that:

where A > 0 is the average density of edge points in the signal. If Q is the standard deviation of the noise, there researchers showed that the filter parameters should be: c=-

AZA d2Dl

p = 2a,

wherea =

J7A

+27t

(67)

It is obvious from the above expressions that when the average distance between edges decreases, i.e., k increases, p increases too so that the filter becomes sharper and the danger of interference from nearby edges decreases. These filters were shown by Castan et al. (1990) to satisfy the criterion of maximal signal-to-signalratio and modified versions of the other two criteria of optimality: The good locality criterion was replaced by a modified version to allow for the filter discontinuity at the centre and thus permitting zero error in the edge locality (something which Canny’s criteria do not allow). The multiple responses criterion was replaced by the requirement that there should be one extremum only at the vicinity of the edge, but small dense extrema away from the edge are allowed. According to these criteria the filter given by 65 is optimal in its own right. The same authors proceeded to implement this filter recursively, as well as its first and second derivatives to be used for the actual differentiation of the signal.

E. A Note on the Zero-Crossing Approach Marr and Hildreth (1980) proposed to detect edges as the zero crossings of the second derivative of the signal. Combining this with the Gaussian filter for smoothing led to the Laplacian of a Gaussian filters, which were quite popular in the early 1980s. The theory we are reviewing in this article was

THE DIFFERENTIATING FILTER APPROACH TO EDGE DETECTION

333

appropriately modified by Sarkar and Boyer (1991b), so that optimal filters that detect edges as zero crossings can be defined and implemented recursively. As the philosophy of such filters is slightly different from the philosophy of our approach so far, no more details will be given concerning them. One point, however, is worth mentioning, and that concerns the derivation of the optimal differentiating filter and its subsequent integration to form the “optimal” smoothing filter, as we did in Section III.E, or the derivation of the optimal smoothing filter and its differentiation to form the “optimal” diferencing filter, as Castan et al. (1990) did, as we discussed in 1V.D. Sarkar and Boyer showed that the optimal filter for detecting edges as zero crossings (i.e., effectively locating the zero crossings of the second derivative of the signal) is not the derivative of the optimal filter for detecting edges as extrema of the first derivative of the signal. The implication of this is that the derivative of the optimal smoothing filter is not necessarily the optimal differencing filter and vice versa. In other words, if one wants to derive the optimal smoothing filter one should start from the beginning, rather than integrate the optimal differentiating filter and so on. So, one should have this in mind and probably put the word in quotation marks when the filter referred to was not directly derived by optimizing a set of criteria but was rather the result of integratioddifferentiation of an optimal filter. V . POSTPROCESSING

All the theory we developed and discussed so far concerns the design of convolution filters that will effectively enhance the locations of the edges. Figure 8a shows the output of filtering the image of Fig. l a with the filters of size 9 given at the end of Section 111. The outputs of the directional convolutions have been combined to form the gradient magnitude output. For displaying purposes, the output has been linearly scaled to range between 0 and 255. We see that the edges of the image stick out quite nicely, and, therefore, we may think that if we simply threshold this output, we may identify them, provided that the threshold has been chosen carefully. However, before we do that, we must isolate the local maxima of the gradient because that is where edges are located. Figure 9 shows schematically the shape of the output surface near the location of an edge. The curves are the contours of constant gradient magnitude, and the thicker the line, the higher the value. The direction of the gradient is orthogonal to the contours. Clearly, we would like the edge to be marked along the thickest line. So, we must look for maxima of the gradient magnitude in a direction orthogonal to the edge direction, i.e., in a direction along the gradient

334

MARIA PETROU

40000.0 ,

20000.0

!

10000.0

-

0.0

0.0

100.0

200.0

300.0

400.0

5 1.0

(4 FIGURE8. (a) This image shows the magnitude of the gradient value at each location computed using the differencing filter of size 13 of Box 2. The values have been scaled to vary between 0 and 255. (b) The local maxima of the gradient image shown in a. (c) The histogram of the values of the gradient image shown in a. The arrow indicates the threshold used for the edge map shown in d. (d) The edge map of b after thresholding with threshold 56.

THE DIFFERENTIATING FILTER APPROACH TO EDGE DETECTION

335

vector. In our elementary edge detector the direction along which we examined whether the output is a local maximum or not is determined grossly. It is allowed to be either vertical or horizontal. In more sophisticated versions of edge detectors, the direction of the edge is detected by taking the inverse tangent of the ratio of the output in the y direction over the output in the x direction. The angle determined that way would, in general, define a direction pointing in between the neighbouring pixels, since it can take continuous values. The values of the gradient along this direction can be calculated by linear interpolation using the values of the neighbouring pixels. The value of the gradient at the pixel under consideration is then compared to the estimated values of the gradient on either side along the gradient direction, and if it is found to be a local maximum, the presence of a possible edge is marked in that location. In even more sophisticated versions of the algorithm the gradient values are fitted locally by a second-order surface, and the exact location of the local maximum is computed from this analytic fitting (see, for example, Huertas and Medioni, 1986). Such an approach results in subpixel accuracy in the location of the edges. Having isolated the local maxima, one might think that the task is over. Figure 8b actually shows the edges we find from the output in Fig. 8a if we keep the local maxima of the gradient. We see that there are lots of unwanted edges which somehow have to be weeded out. One would expect that all edges which are due to texture or noise will probably have very low magnitude, while edges which are genuine will have much higher values.

336

MARIA PETROU

of

FIGURE9. The image brightness at the vicinity of an edge. The thick lines correspond to locations of constant gradient magnitude (the thicker the line, the higher the value of the gradient magnitude). The thin lines correspond to locations of constant gradient direction. Ideally, we would like the edge to be marked along the thickest line.

If we plot therefore the number of pixels versus the gradient magnitude value, we expect to find two peaks, one representing the unwanted edgels and one the genuine ones. Unfortunately, this is not the case, as can be seen from Fig. 8c. The histogram of the edge magnitudes is monomodal; no clear differentiation can be made as to which pixels are edges and which are background on the basis of magnitude only. Even so, people often experiment by thresholding the gradient values, choosing a threshold more or less at random and adjusting it until the result looks acceptable. That is how Fig. 8d was produced. The arrow on the histogram in Fig. 8c shows the exact location of the threshold used. It is obvious that some correct edges have been missed out simply because the contrast across them is rather low, while other edges with no physical significance were kept. Simple thresholding according to gradient magnitude entirely ignores the actual location of the edgels. We must therefore, take into consideration the spatial arrangement of the edgels before we discard or accept them. This is called hysteresis thresholding. Canny incorporated hysteresis thresholding in his edge-detection algorithm, and, as experience has shown, it turned out

THE DIFFERENTIATING FILTER APPROACH TO EDGE DETECTION

337

to be an even more significant factor in the quality of the output than the good filtering itself. It is often said that Sobel filtering followed by hysteresis thresholding is as good an edge detector as one can get. This is not exactly true, but it shows how important this stage of processing is in comparison to the filtering stage. Hysteresis thresholding consists of the following steps: Define two thresholds, one low and one high. Remove all edgels with gradient magnitude below the low threshold from the edge map. Identify junctions, and remove them from the edge map. A junction is any pixel which has more than two neighbouring edge pixels. Of the remaining edgels in the map create strings of connected edgels. If at least one of the edgels of a string has magnitude above the high threshold, accept all the edgels in the string as genuine. If none of the edgels in the string has magnitude above the high threshold, remove the whole string from the edge map. You may or may not wish to put back the junction points removed from the edge map at the beginning. If the removed junction points are to be put back, we accept only those that are attached to retained strings of edgels. Usually there are very few junction points in the filtered output, due to the way the edgels are picked. In fact, a serious criticism of this approach is that the filters respond badly to corners, and the process of nonmaxima suppression eliminates corners or junctions in general. The identification of junctions in an image is another big topic of research (for a review, see Eryurtlu and Kittler, 1992). People have either attempted to do it as an extra stage in the image processing chain (see, for example, Rangarajan et al., 1989; and Mehrotra and Nichani, 1990), or as part of the edge-detection process (see, for example, Harris and Stephens, 1988). We shall not go into the details here, as it is beyond the scope of this article. We must note, however, that the described approach is not designed to respond correctly to junctions and that in the output edge map most of the junctions will probably be missing. One issue that is of paramount importance is the choice of the two thresholds. Various researchers have carefully analysed the sources of the unwanted edgels and have come with various formulae concerning the choice of the thresholds (e.g., Voorhees and Poggio, 1987; and Hancock and Kittler, 1991). Unfortunately, these formulae depend on the exact filters used for smoothing and differentiation, and they certainly require an estimate of the level of noise in the image. On the other hand, most of the

338

MARIA PETROU

people who have used edge detectors have formed their own opinions as to what is a good set of thresholds. Although it is impossible to give a recipe that will work for all filters and all images, we can summarize the general consensus here, which is based on the collective experience of a large number of people and is backed by theoretical results derived by the abovementioned workers:

-

The high threshold should be a number in the range f x mean, -mean of the gradient value calculated before any nonmaxima suppression takes place. The small threshold should be between 4 to of that. The values used for the production of the output shown in Fig. 10a were the mean and two-thirds of that (i.e., 30 and 20, respectively). Another rule of thumb, not based on any theoretical work, is as follows: If one computes the statistics of the gradient magnitude after the nonmaxima suppression, a good set of thresholds is the mean and a tenth of the mean of the distribution of the gradient magnitudes.

FIGURE10. (a) The result of applying the algorithm of Box 1 with filters of size 9 from Box 2 and hysteresis thresholding with maximum and minimum thresholds the mean and two-thirds of the mean gradient value, respectively, computed before non-maxima suppression, to the image of Fig. la. (b) The result of applying the algorithm of Box 1 with filters of size 9 from Box 2 and hysteresis thresholding with maximum and minimum thresholds the mean and a tenth of the mean gradient value, respectively, computed after nonmaxima suppression to the image of Fig. la.

THE DIFFERENTIATING FILTER APPROACH TO EDGE DETECTION

339

Figure 10b shows the result when using this rule (thresholds used 33 and 3.3). There is not much difference between the two results, so the two rules seem to be reasonably equivalent. We may wish to compare these results with Fig. 2, which was produced using thresholds 65 and 40, i.e., twice the mean and the small threshold about two-thirds of the high threshold. The single threshold result of Fig. 8d was produced with an in-between value of the last two, namely 56. Which of these results is preferable is very much a matter of application. The two rules of thumb mentioned above allow the preservation of much detail, while the thresholds used in Fig. 2 and 8d were chosen by trial and error to produce a “clean” picture, personally judged as “good” for presentation purposes. VI. CONCLUSIONS The work we presented in the previous sections focused on a small but significant part of the research effort in edge detection, namely that of convolution filters which respond with an extremum when the first derivative of a signal function is an extremum. Very elaborate filters were developed and shown to perform quite well when applied to difficult images. These filters are optimal within the restrictions of the approach adopted and the criteria used. However, there were some disquieting results. Deriche’s filters were developed using inconsistent criteria, i.e., they were allowed to be of infinite extent while the criteria used to justify them were ill-defined for infinite boundaries. And yet, those filters have become reasonably popular, and most of the users will tell you that they perform well enough. One has to wonder then how much the optimality criteria matter and how much the restrictions we impose define the rules of the game. Spacek (1986) had the idea to ignore any optimality and simply define a filter that is a cubic spline that simply fits the boundary conditions and nothing else. This filter is given by the following equation: f(x) = A [

(:>’

+ 2(;7

+

(t)]

for - w

I

x I0,

(68)

where A is an amlitude parameter. Spacek calculated the value of the composite performance measure of this filter for step edges and found it less than the value of the performance measure of the optimal filter. Petrou and Kittler (1991) showed that the difference becomes more significant when ramp edges are assumed and increases as the slope of the ramp edges decreases, i.e., as the ramps become shallower. However, these calculations are theoretical assessments of the filters, and we do not know how they translate to practical filter performance. To test correctly the performance

340

MARIA PETROU

of an edge detector we must have an image and its perfect edge map as drawn by hand, say, and compare the output of the edge detector against the desirable edge map, The fraction of the edge pixels that were not detected will form the underdetection error, and the fraction of spurious edge pixels will form the overdetection error. Then we could say that we have a measure of the true performance of the edge detector. It would be useful to know what the correspondence is between the value of a theoretical performance criterion and the underdetection and overdetection errors of the filter when some standard images are used. This, however, does not seem to be easy or even possible. The problem starts from the fact that it is very difficult to find or even develop such standard images. The reason is the example in hand, the one given in the introduction: A lot of knowledge creeps in when we produce the hand segmentation of an image, and any comparison against it is heavily biased in favor of the handproduced edge map. Even so, one may argue that we are prepared to allow for this factor and that we do not even hope to establish filters with zero overdetection and zero underdetection error. What matters really is the relative performance of the various filters when applied to the same image and their outputs are compared with the same hand-produced edge map. However, notice that I used the word “edge detector” and not “edge filter” when I talked about comparisons with the hand-drawn edge map. This is because edge filters simply enhance the edges, they do not identify them. It is the nonlinear postprocessing that does the identification and that relies on thresholds that could be chosen almost arbitrarily and that clearly should be different for different filters as detailed analysis has shown (see, for example, Hancock and Kittler, 1991). Furhter, the best edge detector should be one that requires the least adjustment from one image to the other, or for which the adjustment happens automatically. To assess the performance of an edge detector taking this into consideration, one certainly needs a set of images with known edge maps. And then the question arises as to what is a representative set of images! For the above reasons, it is very difficult to have absolutely objective criteria about the performance of edge detectors. This is also the reason why everybody who published anything on edge detection was able to show that his or her edge detector performs better than other edge detectors! In view of the above discussion, it seems that it is reasonable to compare filter outputs by applying them to the same image and for each filter playing with the parameters until a “good” result is achieved. Hardly a satisfactory process, but probably the fairest one under the circumstances. The cubic spline filter given by Eq. (68) was used to produce the result in Fig. 11. Visually, it is difficult to see much difference between this output and that of Fig. 2! Compare, however, Figs. 12a and 12b. Figure 12a was produced

THE DIFFERENTIATING FILTER APPROACH TO EDGE DETECTION

341

FIGURE1 1 . The result of applying the algorithm of Box 1, with the spline filter of size 9 and hysteresis thresholding, to the image of Fig. la.

by the spline filter and 12b by the optimal filter of the same size. Both results were obtained using hysteresis thresholding with thresholds in ratio 2 : 3 and the high threshold chosen to be twice the mean of the gradient computed before the nonmaxima suppression. It is clear that the result of the optimal filter is superior as the circle and the straight lines were better detected. Both filters did a rather bad job at the perforations, partly because of their proximity and partly because of the junctions involved.

342

MARIA PETROU

FIGURE 12. (a) The result of applying the algorithm of Box 1, with the spline filter of size 13 and hysteresis thresholding, to the image of Fig. 5a. (b) The result of applying the algorithm of Box 1 , with the filter of size 13 from Box 2 and hysteresis thresholding, to the image of Fig. Sa.

From this example and from my experience with edge detectors, I would like to summarize the conclusions of this chapter in the form of the message to take home: 0

0

There is no filter or edge detector which is appropriate for every image. The most important parameter of a filter is its size. The noisier the image, the larger the filter should be. The sharper the filter at the centre, i.e., the more it resembles the difference of boxes operator, the more accurately it will locate the edges and the more sensitive to noise it will be. The general shape of the filter should be something like the filters presented in Fig. 7. The filter should go smoothly to zero if it is of finite size, and its value should drop to insignificant values within a small distance from its centre if it is of infinite size, to avoid interference from other features. The post processing stage is of paramount importance. Contextual postprocessing like probabilistic relaxation (e.g., Hancock and Kittler, 1990), salient feature selection (e.g., Sha’ashuna and Ullman, 1988) or at least hysteresis thresholding is recommended. For images with low levels of noise the Sobel or even simpler masks should be used. (The optimal filters we discussed are designed to cope with high levels of noise, and they will work badly due to the

THE DIFFERENTIATING FILTER APPROACH TO EDGE DETECTION

0

343

overblurring of the true edges and the rounding of the corners if applied to noise-free images like those created by software graphics packages.) The noisier the image, the more is to be gained by using an optimal filter. Know thy edge detector. Avoid default values for the thresholds of the postprocessing stage or the filter size; instead, check the role of each parameter, particularly for filters whose shape changes with the value of the parameter, and adjust them accordingly.

Given that the exact filter shape seems to make little difference to the final outcome for images of low to intermediate levels of noise, is one to conclude then that all the elaborate theory we developed was useless? I would say no. For a start, such a conclusion would be a hindsight view. We would never have known unless lots of people had toiled developing the theory and the filters in the first palce. Besides, the optimal filters do make a difference for images of high levels of noise. In particular, the filters presented in Box 2 require only the specification of size to guarantee a good result, as opposed to the Gaussian-type filters for which the user has to play with two parameters, namely size and standard deviation, to achieve an acceptable result. Finally, even for images of low to intermediate levels of noise, if one is to use a filter, one might as well use something that is the result of careful consideration even though the difference it makes might be disproportionate to the effort put in developing it!

REFERENCES Boie, R. A., Cox, I. J., and Rehak, P. (1986). Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition 100-108. Bovic, A. C., and Munson, D. C., Jr. (1986). Computer Vision, Graphics and Image Processing 33, 377-389. Boyer, K. L., and Sarkar, S. (1992). Applications of Art. Intelligence X : Machine Vision and Robotics SPIE-1708, 353-362. Canny, J . (1986). IEEE Trans. Pattern Analysis and Machine Intelligence PAMI-8, 679-698. Castan, S., Zhao, J., and Shen, J. (1990). Proc. 1st European Conf. on Computer Vision (0.Faugeras, ed.) ECCV-90, 13-17. Deriche, R. (1987). International Journal of Computer Vision 1, 167-187. Duda, R. 0.. and Hart, P. E. (1973). Pattern Classification and Scene Analysis. John Wiley, New York. Eryurtlu, F., and Kittler, J. (1992). In: Signal Processing VI, Theories and Applications (J. Vandewalle, R. Boite, M. Moonen, A. Oosterlinck, eds.). Elsevier, 1, 591-594. Geman, S . , and Geman, D. (1984). IEEE Trans. Pattern Analysis and Machine Intelligence PAMI-6, 721-741.

344

MARIA PETROU

Graham, J., and Taylor, C. J. (1988). Proc. of the 4th Alvey Vision Conf., Manchester, UK, 59-64. Granlund, G. H. (1978). Computer Graphics and Image Processing 8, 155-178. Hancock, E. R., and Kittler, J. (1990). IEEE Trans. Pattern Analysis and Machine Intelligence PAMI-12, 165-181. Hancock, E. R., and Kittler, J. (1991). Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition, 196-201. Haralick, R. M. (1980). Computer Graphics Image Processing 12, 60-73. Haralick, R. M. (1984). IEEE Trans. Pattern Analysis and Machine Intelligence PAMI-6, 58-68. Harris, C., and Stephens, M. (1988). Proc. 4thAlvey Vision Conf., Manchester, UK, 189-192. Huertas, A., and Medioni, G. (1986). IEEE Trans. Pattern Analysis and Machine Intelligence PAMI-8, 651-664. Kundu, A., and Mitra, S. K. (1987). IEEE Trans. Pattern Analysis and Machine Intelligence PAMI-9, 569-575. Kundu, A. (1990). Pattern Recognition 23, 423-440. Lee, J. S. J., Haralick, R. M., and Shapiro, L. G. (1987). IEEE J. of Robotics and Automation RA-3, 142-156. Marr, D., and Hildreth, E. (1980). Proc. R. SOC.Lond. B-207, 187-217. Mehrotra, R., and Nichani, S. (1990). Pattern Recognition 23, 1223-1233. Modestino, J. W., and Fries, R. W. (1977). Computer Graphics Image Processing 6 , 409-433. Morrone, M. C., and Owens, R. (1987). Pattern Recognition Letters 6, 303-313. Nalwa, V. S., and Binford, T. 0. (1986). IEEE Trans. Pattern Analysis and Machine Intelligence PAMI-8, 699-7 14. Perona, P., and Malik, J. (1992). Applications of Art. Intelligence X : Machine Vision and Robotics SPIE-1708, 326-340. Petrou, M. (1993). IEE Proceedings-I Communications, Speech and Vision 140, 331-339. Petrou, M., and Kittler, J. (1989). Proc. 6th Scandinavian Conference on Image Analysis, SCIA '89. Oulu, Finland (M. Pietikainen and J. Roning, eds.), 816-819. Petrou, M., and Kittler, J. (1991). IEEE Pattern Analysis and Machine Intelligence PAMI-13, 483-491. Petrou, M . , and Kittler, J. (1992). Applications of Art. Intelligence X: Machine Vision and Robotics SPIE-1708, 267-281. Petrou, M., and Kolomvas, A. (1992). In: Signal Processing VI, Theories and Applications ( J . Vandewalle, R. Boite, M. Moonen, A. Oosterlinck, eds.). Elsevier 3, 1489-1492. Pitas, I., and Venetsanopoulos, A. N. (1986). IEEE Pattern Analysis andMachine Intelligence PAMI-8, 538-550. Rangarajan, K., Shah, M., and van Brackle, D. (1989). Computer Vision Graphics and Image Processing 48, 230-245. Rice, S . 0. (1945). Bell Syst. Tech. J. 24, 46-156. Rosenfeld, A., and Thurston, M. (1971). IEEE Trans. Comput C-20, 562-569. Sarkar, S., and Boyer, K. L. (1991a). IEEE Pattern Analysis and Machine Intelligence PAMI-13, 1 154-1 171. Sarkar, S., and Boyer, K. L. (1991b). CVGIP: Image Understanding 54, 224-243. Sha'ashua, A . , and Ullman, S. (1988). 2nd Intern. Conf. Comp. Vision ICCV-88, 321-326. Shanmugam, K. S., Dickey, F. M., and Green, J. A. (1979). IEEE Trans. Pattern Analysisand Machine Intelligence PAMI-1, 37-49. Shen, J., and Castan, S. (1986). Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition, 109-1 14. de Souza, P. (1983). Computer Vision, Graphics and Image Processing 23, 1-14.

THE DIFFERENTIATING FILTER APPROACH TO EDGE DETECTION

345

Spacek, L . A. (1986). Image and Vision Comput. 4, 43-00. Torre, V., and Poggio, T. A. (1986). IEEE Pattern Analysis and Machine Intelligence PAMI-8, 147-163. van Vliet, L., Young, I . T . , and Beckers, G. L. (1989). Computer Vision, Graphic and Image Processing 45, 167-195. Voorhees, H., and Poggio, T. A. (1987). Proc. 1st Intern. Conf. Computer Vision. London, 250-258.

Ziou, D. (1991). Pattern Recognition 24, 465-478.

This Page Intentionally Left Blank

Index

A

phase transitions cholesteryl esters, 177 fluoroalkanes, 178 paraffins, 176 porin, 178 Crystal growth epitaxial orientation on inorganic substrates, 119 on organic substrates, 120 Langmuir-Blodgett films, 118 reconstitution (proteins), 120 solution crystallization evaporation of solvent, 118 self-seeding, I18 Crystal structures of chitosan, 137, 138, 165 of copper perchlorophthalocyanine,161 of diketopiperazine, 157 early structure determinations, 1 I5 methylene subcells, 155 membrane proteins bacteriorhodopsin, 168 porins, 168-169 of paraffins, 162 of phospholipids, 163 of poly(butene-I), 167 of poly(&-caprolactone), 166 of polyethylene, 166 polymer structures, 133 solved by direct methods, 139 Crystallographic phase determination direct methods crystal bending, effect of, 153 density modification, 141

Access function, 69 Ada, 92

B Basis restriction MSE, 47-48 Bond distances and angles, calculation of, 145 C

C + + , 70, 77, 92 Cascaded function call, 77 C language, 73, 106 data types, I08 C library, 107 CMT, 41 Collage, 237 Common dyadic symmetry, 17 Common imaging architecture, 72 Convolution, 301, 3 1 1 Cover Minkowski, 207 morphological, 218, 230 Crystal bending diffraction incoherence from, 152 effect on structure analysis, 153 Crystal disorder binary solids lipids, 181, 183 paraffins, 180 effect on diffraction intensities, 174 347

348

INDEX

dynamical scattering, effect of, 149 electron microscope images, 130 examples of solved structures, 139 maximum entropy, 142, 171 phase invariant sums, 134 Sayre equation, 141 secondary scattering, effect of, 151 Patterson function, 132 trial and error, 131 Crystallographic residual definition, 131 significance, 131

D Data structure, 63 design in C, 73 implementation, 81 BOUNDARY Structure, 85 DEVICE Structure, 89 IMAOE structure, 82 K-STATE structure, 88 POLYLINE structure, 89 scalar types, 90 SEQUENCE Structure, 86 TRACE structure, 87 VIDEODISC structure, 87 WINDOW structure, 84 DCT, see Discrete cosine transform DFT, 6 Dilation function, 203 set, 202 support-limited, 219, 231 Dimension box-counting, 21 1 entropy, 212 Hausdorff, 204 Minkowski-Bouligand, 206, 208,209 similarity, 206 Discrete cosine transform, 2, 7, 10 integer, see ICT the four versions, 12 weighted, 9 Discrete sine transform, 8, 28 integer, 27 the four versions, 12 DST, see Discrete sine transform Dyadic symmetry decomposition, 23-25 definition, 17

dependence, 20 type, 18, 20 Dynamical scattering, 147-149 multiple beam theory, 148 phase grating approximation, 147 slice methods, 147 two-beam theory, 147

E Edge, modeling of ramp, 322 step, 321 Edge detection approaches to adaptive filters, 308 Hilbert transform pair, 308 linear filtering, 310 model-based, 308 nonlinear filtering, 307 quadrature filters, 308 region, 307 robust, 308 template matching, 308 zero crossing, 332 definition of, 299 performance assessment of, 340 Edge detection filters Canny, 321,327 desirable properties of, 31 1 difference of boxes, 3 11 Gaussian approximations, 326 infinite impulse response, 329 Deriche, 329 drawbacks of, 331 recursive implementation of, 330, 332 Sarkar and Boyer, 329 Shen and Castan, 331 matched, 318 Petrou and Kittler, 322 quality measure of composite performance, 320-321 false maxima, 319 good locality of, 315, 332 signal-to-noise ratio, 312, 332 scaling of, 315, 318 Sobel, 304 Spacek, 321 spline, 339 zero crossing, 332

349

INDEX Edgel, 304 Electron diffraction camera length, 123 diffraction geometries, 123 goniometry, 124 illumination of sample, 122 intensity data, 125 recording diffraction patterns, 122 Electron microscopy effect of radiation damage, 156 high-resolution, low dosage, 127 lattice images, examples, 158 low-magnification, diffraction-contrast, 126 phase contrast transfer function, 129 Erosion function, 203 set, 202 support-limited, 219, 231 Error handling in imaging, 78, 90

F Fast computation algorithm of ICT, 57-58 of Walsh transforms, 25 Filters, two-dimensional, 326 Fourier analysis, 6 Fourier-transform pairs, 172ff Fractal dimension, 204 set, 204 Cantor, 205 signal, 213 fractional Brownian motion, 216 fractal interpolation function, 214 Weierstrass cosine function, 213 Function implementation, 91 color conversion, 97 convolution, 98 differencing, 94 efficiency, 104 neighborhood operation, 101 portability, 105 sobel operator, 101 thresholding, 96 Function interface design, 74 Fuzzy entropy conditional, 256 higher-order, 256

hybrid, 252-255 positional, 256 rth-order, 252-255 Fuzzy geometry, 256-260 breadth, 258 center of gravity, 259 compactness, 257 degree of adjacency, 260 density, 259 height, 257 index, area coverage, 258 length, 258 major axis, 258-259 minor axis, 259 width, 257

G Genetic algorithms, 288-290 crossover, 288-289 enhancement, 289-290 fitness function, 289 mutation, 288-289 parameter selection, 289-290 reproduction, 288-289 Gradient, definition of, 304 Graphical user interface, 107

H High-definition television (HDTV), 2 Hysteresis thresholding, 336

I ICT, 10 derivation order-8, 31 order-16, 42-44 fast computation algorithm, 57-58 fixed-point error performance, 52-56 implementation, 49-52 performance, 44-49 Image binary, 298 color, 298 definition of, 297 fractal, 237 gray, 298 size of, 297

350

INDEX

Image ambiguity, uncertainty measures, 25 1-260 grayness, 252-256 correlation, 252-253, 255-256 fuzzy entropy, 252-256 spatial, 256-260 Image coding standards, 2 Image interchange format, 72 Image processing analysis, 247-296 contour detection, 267 enhancement, 269 FMAT, 269-272, 290 fuzzy disks, 270 fuzzy segmentation, 264-267 fuzzy skeleton, 269-272 pixel classification, 267 threshold selection, 264-267 design, 67 for flexibility, 69 object-oriented, 68 for portability, 71 for speed, 67 error handling, 78, 90 international standard, 72 Image representation, 65 oct-tree, 66 pryamid structure, 66 quad-tree, 65 symbolic, 66 Inheritance, 69 Integer cosine transform, see ICT J

Junction finding, 337

M Markov random fields, 307 Markov process, 7, 45,47 Matching, recognition, 272-278 Dempster-Shafer theory of evidence, 278 feature selection, 274 multivalued recognition, 274 remote sensing, 274-276 rough sets, 278 rule-based systems, 277 syntactic classification, 275-277 Morphological operation function, 202 set, 202 skeleton, 239

N Neural networks, 278-288 neuro-fuzzy approach, 278-288 back propagation, 282-286 connectionist expert system, 281 fuzzy neurons, 279 Kohonen’s algorithm, 283 perception, 280-281 self-organizing network, 281 Noise additive, 309 autocorrelation function of, 310 filter response to, 312, 319 Gaussian, 309 homogeneous, 310 thermal, 303, 309 uncorrelated, 310 white, 310 Nonmaxima suppression, 301, 335, 338

K KLT (Karhunen-Loeve transform), 7 Knowledge, in edge detection, 299

L Line process, 307 Lipids binary solids, 183 rnethylene subcells, 155 phase transitions, 176 Local maxima, 301, 335

0

Opening, 202, 239

P Paraffins binary solids, 183 crystal structure analysis, 162 methylene subcells, 155 phase transitions, 176 Pixel. 297

INDEX Polymers chitosan, 137, 138, 165 early structure analyses, 133 lattice images, 158 poly(butene-I), 167 poly(&-caprolactone), 166 polyethylene, 166 poly(truns-cyclohexanediyldimethylene succinate), 165 structures by direct methods, 139 Polymorphism, 69, 74 implementation in C, 92 Porins, crystal structures of, 168-169 Omp C, 169 Omp F, 168 Pho E, 169 VDAC, 169 POSIX, 71 Proteins bacteriorhodopsin, 168 porins, 168ff

S Secondary scattering, 151 Shape estimation, 251 Signal, modeling of, 31 1 Small molecules early structure analyses, 1I5 lattice images, 158 structure analyses, examples of, 157ff structures by direct methods, 139 Smalltalk, 68, 77 Smoothing, 303, 304 filter, 322, 323 Sobel edge detector, 304 Speech, 226 Structure refinement density flattening, 146 effect of dynamical scattering, 150 Fourier refinement, 145 least squares refinement, 143

35 I T

Threshold, choice of, 337 Thresholding, 301, 336 Transform C-matrix (CMT), 41 cosine, see DCT Fourier, 6 integer, 25 Karhunen-Loeve, 7 optimal, 5 orthogonal, 3 for image coding, 5-10 sine, see DST sinusoidal, 10-15 integer, 24-40 integer, derivation, 30 symmetry cosine, 9 Walsh, 15, 25 (see also Walsh matrix) Transform coding bit allocation, 4 block diagram, 3 quantization error, 4 Transform efficiency E d , 45 Traversing image data, 94, 104

U Uncertainty, 248 measures, 251-260 principal, 303, 318

W Walsh matrix, 15 binary, 20 conversion between orderings, 22 dyadic-ordered, 16, 22 natural-ordered, 17, 21 sequency-ordered, 16, 21 Walsh transform, 15 fast computation algorithm, 25