The 1998 IEEE International Joint Conference on Neural Network Proceedings: IEEE World Congress on Computational Intelligence: May 4-May 9, 1998 Anchorage, Alaska, USA

ms for a Neural ally I ~ ~ i b ~ ~ e ~ Qiang Gan*#,Jun Yao' and K.R.Subramanian*

* School of Electrical and ElecfronicEngineering Nanyang Technological University Nanyang Avenue, Singapore 639798 E-mail: eqgan@tu. edusg Abstract This paper presents a neural network with its output layer as a classijier and its hidden layer constrained by laterally inhibited receptive Pel& asfeature extractor, in which the idea that wavelet transforms are very suitable for modeling the primary visual informationprocessing is reflected. Two learning algorithms for designing the receptive fields are proposed. The problem associated with local minima caused by the inherent oscillatory property in laterally inhibited receptive fields is combated in the algorithm using discrete wavelets. Good performance is obtained in the experiment of ECG signal classification using the neural network.

1. Introduction Neural networks have been established as a general tool for approximation and classification by fitting inputloutput data effectively into nonlinear models. The multilayer perceptron, in which a neuron receives inputs from all the neurons in the adjacent pre-layer, is widely used for function approximation and signal classification. On most occasions it performs quite satisfactorily. However, when the network input is characterized by time-frequency localized features, the generally used multilayer perceptron with unconstrained global connections between the adjacent layers does not work well. In the human visual system there exist example models for dealing with this problem [l]. That is the conception of receptive field, the shape of which can be adapted with the visual input under certain constraints. In the human visual system there are various receptive fields of different shapes. For example, there are Gauss function shaped receptive fields for local smoothing, and Gabor

0-7803-4859- 1/98 $ I O.oOO1998 EEE

1156

#

Department of Biomedical Engineering Southeast University Nanjing 2 I0096 P.R.China

function shaped receptive fields for combining local smoothing and sharpening which provides the function of lateral inhibition. We find that this kind of receptive fields can not be formed automatically by learning without any constraints on the weights in a multilayer perceptron. Wavelet transform is a good model for the receptive fields in the human visual system [2]. Because a wavelet function satisfies the admissibility condition [3], it must be oscillatory across its zero points. Hence, wavelet hctions provide natural models for laterally inhibited receptive fields, which are good at extracting time-frequency localized features. Actually, Gabor function has been widely used in theoretical studies of the primary visual information processing such as lateral inhibition and it can be regarded as a mother wavelet of good time-frequency localization properties. Through dilation and translation a wavelet filter bank can be formed as a group of receptive fields which approximately perform wavelet transforms. Hence, in the design of receptivefields we can benefit from the advanced theory of wavelet transforms. There have been several pieces of work done on combining neural networks with wavelet transforms which perform as the receptive fields of hidden neurons. Szu [4] developed neural network adaptive wavelets for signal representation and classification, and tentatively applied them in phoneme recognition and image compression. Gan [5] proposed a wavelet neural network architecture and applied it to ECG signal classification. Dickhaus [6] and Akay [7]have also studied biomedical signal detection

and classification using different wavelet network structures. The key issue in the design of this kind of neural networks is how to obtain optimal sets of dilation and translation parameters (or wavelet parameters). In all the networks mentioned above, continuous wavelet parameters are used and trained by the gradient-descent learning algorithm, or preset and fixed wavelets are

applied. Because of the inherent oscillatoryproperty of the wavelet function, learning wavelet parameters is easy to sink into local minima and it is difficult to get the optimal result. In this paper, we use Gabor function to constrain the receptive fields of the hidden neurons so that the lateral inhibition is introduced into the network. Furthermore, discrete wavelets are used and a method for calculating wavelet parameters is proposed to combat the problem of unconvergence in the learning process. The remainder of this paper is organized as follows. The neural network formulation is put forward in section 2. Two learning algorithmsare proposed in section 3. Simulation studies on ECG signal classification are carried out in section 4, followed by discussions and conclusions in section 5 .

2. Neural network formulation The neural network under consideration, with laterally inhibited receptive fields, can be described as follows:

i = 1,2,......, NOL.

s( t ) = (1 - e-') /

where, y f L and

X,

(1 + e-.' )

denote the output and the input of the

network respectively, represent the weights between output and hidden layers, h ( ( j - b 1 ) / u , ) produce the weights between hidden and input layers, and NOL, NHL, NIL are respectively the number of nodes in output, hidden, and input layers.

h(4 and can be adjusted by dilation and translation parameters a, and b, . If we regard the Gabor function as a mother wavelet, the hidden layer is actually composed of a wavelet filter bank which plays the role of feature extraction. In connection with the human visual system, the weights connected to the hidden layer performjust like receptive fields with lateral inhibition because the Gabor function shapes like a Mexican hat as shown in Fig.1. Compared to a general feedforward neural network, the neural network described by (1)-(3) has a similar structure, but the weights connected to the hidden layer are constrained and adapted indirectly by learning dilation and translation parameters. As we know, Gabor wavelets are nonorthogonal. Gabor function is selected here as the mother wavelet based on the following reasons. First, although there are many orthogonal wavelet bases, the isotropic (or symmetric), compactly supported and orthogonal wavelets do not exist. Second, according to the uncertainty principle about the time-frequency resolution, a Gaussian or modulated Gaussian provides the optimal tradeoff between time localization and frequency resolution. However, Gauss function does not satisfy the admissibility condition and can not provide lateral inhibition. Gabor function is a modulated Gaussian, which not only has optimal joint time-frequency resolution [ti],but also introduces lateral inhibition. Third, Gabor function has been proved to be a very good model for the primary visual information processing. Furthermore, our purpose of using the wavelets is to extract useful features, but not to reconstruct signals. It is well known that wavelet transforms are good at representing signals with time-frequency localized features. The neural network proposed here would be suitable for the classification of signals or patterns with time-frequency localized characteristics.

3. Learning algorithms

-0.5 -5

0

Learning algorithms are developed in order to obtain optimal weights for classification and wavelets for extracting features from a particular type of signals. In the case when continuous parameters are used, a learning algorithm can be derived out based on the gradient-descent method. This kind of learning algorithm may not converge when there exist oscillatory functions in the neural network formulation. We try to combat this problem by using discrete wavelets and learning them by a constructive algorithm.

5

Figure I.Gabor function and lateral inhibition

Note that the weights connected to the hidden layer are generated by dilation and translation of a Gabor function

3.1 For receptive fields with continuous

1157

parameters

If the wavelet parameters a, and b, take continuous values, a back-propagation base algorithm can be used to train the neural network defined by (1)-(3), although trivial modification on the rules for updating weights and parameters should be made. We can derive the updating rules as follows:

3.2 For receptive fields with discrete parameters Firstly, the wavelet parameters are constrained to take discrete values by the following equations: a, =ao6, = n,b,a,"

(14) =nlboal

In this way, the receptive fields are reformed as follows:

where m, ,n, E Z . Instead of training a, and b, directly by error back-propagation, they can be calculated according to (14) if we can determine the values of a,,b,,m,, andn,. The formula for determining these parameters are derived in the following. h,, (t) will define sets of windows in a wavelet filter bank. For brevity, we just consider a particular set of windows defined by wavelet parameters a and b, corresponding to a,,b,,m, andn. In the discrete domain, the windows indexed by m and n and defined by h,, (t) in the time-frequency plane can be written as follows:

[nb,af + aft* - afAh,nb,a," +aft*+ afAh) a* A, a' A, x[---+--) af a f ' a r a,

In the above equations, d, represents the desired output, y p is the output of the Ith node in the hidden layer, andE denotes the error used in the back-propagation. To keep the receptive fields laterally inhibited, the

mother wavelet function has to be oscillatory across its zero points, as shown in Fig. 1. This will make the error function E highly nonconvex. Therefore, local minima are expected in the learning process, or the learning process may not converge. This is the major shortage to be overcome in training this kind of neural networks. We will resolve this problem by using discrete wavelets in the following subsection.

1158

where, f* and A,, are the centre and the radius of the mother wavelet h(t), respectively, while "and A, are those of the Fourier transform i ( w ) , respectively. Actually, t * , A h , w ' , a d Ah can be determined by w, and CT which are parameters in the Gabor function. Selecting suitable values for parameters a, and bo is important. From the point of view of function reconstruction, we hope that suitable selection of a, and bo can make h,(t) constitute a tight frame [3]. On the other hand, from the point of view of filter banks, we hope that the windows defined by h,, (t) can properly cover the interesting areas in the time-frequency plane. Because we are interested in extracting useful time-frequencylocalized features, we will select the values of a, and bo from the point of view of filter banks. If a, and bo are not properly selected, the windows will be too sparse to cover the whole

time-frequency plane and the information in signals will be lost; or the windows will be overlapped and the information extracted is redundant. From (16) we note that the windows in the frequency domain are not influencedby the translation parameter. Let us consider two adjacent windows corresponding to m and m+l. In order that the windows can cover the whole frequency domain, while having no window overlapped, the following equality should be satisfied:

In the time domain, the situation is more complicated, because the windows are affected by both dilation and translation parameters and the number of adjacent windows are more than two. However, we only pay attention to the selection of bo in the time domain. We consider two adjacent windows with index m unchanged. In this case, the windows in time domain should satisfy:

( n + l)boar+ a r t * - arA, = nboa," + art" + a,"Ah , Le., bo = 2Ah

(18)

Although (18) does not give an ideal selection of bo, it provides a simple and satisfactory solution. Now let us consider how to determine m and n. As we know, the centre of the window defined by h,(t) in the frequency domain is w' / (~zu,"),and that in the time domain is nb,a,"+a,"t'. If the maximum and the minimum frequencies of the input signals are f,, and f,, (f,, f 0) ,respectively, and the maximum length of input signals is T,, ,then the following inequalitiesshould be satisfied

w* fmin

5%

5

or

t8 xa'* -- >n>---

bo

a30

t*

bo

(if bo c0) (23)

Because the values of m and n are limited by (2 1)-(23), we can even take into account all the integer values of m and n in the whole ranges given above and calculate the output values of the corresponding wavelet filters. Those m and n which result in output values larger than the given thresholds will be selected. Note that the number of wavelet filters, or the number of hidden nodes, are automatically determined by learning. Given a set of training signals, we can get satisfactory wavelet parameters using the above algorithm. It is able to avoid the problem of unconvergence which exists in the continuous wavelet parameter adjustment as described in (4)-( 13). It should be noted that the weights connected to the output layer are still trained using the backpropagation algorithm.

4. Simulation studies To test the classification performance of the neural network with laterally inhibited receptive fields and the corresponding learning algorithms, we apply the network to ECG signal classification. The hidden layer consists of three subnetworks, each receiving one period of the ECG signal respectively. All the outputs of the subnetworks are connected to an output layer where the final classification is accomplished. The network size is set as follows: NOL=4, NIL=460x3, and the size of the hidden layer is determined by the learning algorithm. Three subnetworks are needed because generally at least three periods of ECG signals are required for diagnosing some cardiac disease [5]. Because the page length of the paper is limited, the details of the network architecture are not fully discussed here. The input-output function of the network is given as follows:

f max

0 5 nb,a," + a r t * 5 T,, Hence, the ranges of m and follows:

n

(20) should be constrained as

1159

i = 42,. .....,NOL where, k is the subnetwork index. In the case of applying continuous wavelets, a, and b, are directly trained according to (4)-(13). If discrete wavelets are used, a, and b, are calculated according to (14), with ao,bo,m, undn, determined by (17)-(18) and (21)-(23).

The ECG signals used in our simulation are from the MIT/BIH arrhythrma database. We have selected Normal, Bundle Branch Block, Infarct, and Premature Ventricular Contraction waves, each consistingof 200x3 periods. They are divided into a training set and a testing set. Some typical examples of ECG signals are shown in Fig. 2. We note that the ECG signals are mostly flat and their high frequency components are localized to short time intervals. For this kind of signals, time-frequency localized representationssuch as wavelet transforms are very useful.

Normal

Bundle Branch Block

Infarct

training phase. The differences between the ECG wave and the centres can be used as inputs to the network. With discrete wavelets, in order to make the Gabor function satisfy the admissibility condition of wavelet transforms, its parameters are set as follows: o2= 8, and W , =5.3. According to (17) and (18), uo=1.14 and bo = 3.28. We select integer values of mkl and nk, from the ranges given by (2 1)-(23), only those corresponding to the wavelet filters that obtain output values larger than the given thresholds are maintained to form the receptive fields, which extract useful time-frequency localized features of the input signal for classification. The number of wavelet filters selected in each subnetwork by the learning procedure is as follows: NHL,= 15, NHL, = 16, M L , = 15. After learning, the classification performance of the network trained with discrete wavelets is tested. The simulation result is given in the row marked as NN-LI2 in Table 1. In order to make a comparison, the network trained with continuous wavelets is also investigated in the simulation studies. The corresponding classification performance is given in the row marked as NN-LI1 in Table 1. We note that it often happens for the network with continuous wavelets to sink into local minima or not to converge. The performance of a standard BP network with a similar structure is also given in Table 1. It is shown that the neural networks with laterally inhibited receptive fields achieve better recognition rate than the BP network. We note that the neural network proposed in this paper and the BP network have a similar structure and operate in the same way in the testing phase, although their weights are trained by different algorithms. Therefore, the comparison here is reasonable. Although the ECG signals in the testing set are corrupted by noise and fake peaks, and have never been seen by the network before testing, the recognition rate is considerably high.

Premature Ventricular Contraction Figure 2. Typical ECG signals

Before entering the network, the ECG waves are preprocessed. The preprocessing includes the detection of peaks of R-waves and the normalization which makes the amplitudes of ECG signals minus their mean values change from -1 to +l. Three periods of ECG waves are directly inputted into the network at the same time. The peaks of the R-waves are located at the centres of the three subnetworks respectively. To further improve the classification performance, a centre for each class of ECG waves can be obtained by a clustering algorithm in the

1160

Net type I Normal B.B.B I Infarct I P.V.C BP I 85% 88.75% I 99% I 98% NN LI1 I 92.5% 1 95% 198.75% 90.5% NN L12 I 100% 96.6% I 98.25% 98.25%

~

I

5. Discussions and conclusions In the simulation studies, Gauss function shaped receptive fields have also been tested. It is illustrated that Gabor function shaped receptive fields perform better in the ECG signal classification. The major difference between Gauss and Gabor functions lies in that Gabor

function is oscillatory across its zero points and thus provides lateral inhibition. The smoothing effect of Gabor function as a receptive field plays a role of removing noise, while its sharpening ability resulting from the lateral inhibition can enhance the localized features in signals. Obviously, the lateral inhibition provided by Gabor function shaped receptive fields improves the feature extraction ability of the neural network. We note that lateral inhibition is important , but it can not be learnt using BP-type algorithms without any constraints on the weights in advance. The oscillatory behavior of the Gabor function makes it satisfy the admissibility condition of wavelet functions. The Gabor function shaped receptive fields form a wavelet filter bank and the hidden layer of the neural network plays a rule similar to a group of wavelet transforms. What is more, by learning the wavelet parameters it is possible to achieve the optimal or sub-optimal result. From the point of view of the generalization ability of neural networks, we should be able to figure out theoretically how the lateral inhibition influences the performance surface of the neural network. This would be one of our future research directions. In this paper, Gabor function shaped receptive fields are successfully introduced into a neural network for signal classification. By using discrete wavelets a learning algorithm is derived such that the problem of unconvergence caused by the oscillatory property of the receptivefields can be eliminated. Simulationstudies show that introducing lateral inhibition into the neural network is useful to improve its classificationperformance.

References

1161

R.B.Pinter and B.Nabet, ed., Nonlinear Vision: Determination of Neural Receptive Fields, Function, and Networks, CRC Press, 1992. L.Gaudart, J.Crebassa, and J.P.Petrakian, “Wavelet transform in human visual channels”, Applied Optics,vo1.32, no.22, 1993,pp.4119-4127. LDaubechies, Ten Lectures on Wavelets, Society for Industrial and Applied Mathematics, Philadelphia, 1992. H.Szu, B.Telfer, and S.Kadambe, “Neural network adaptive wavelets for signal representation and classification”, Optical Engineering, vo1.3 1, 110.9, 1992, pp.1907-1916. [51 Q.Gan, J.Yao, H.-C.Peng, and J.-Z.Zhou, “Wavelet neural network for ECG signal classification”, Proc. of Int. Conf. on Biomedical Engineering, Hong Kong, 1996. H.Dickhaus and H.Heinrich, "Classifying biosignals with wavelet networks”, IEEE Engineering in Medicine and Biology Magazine, vo1.15, no.5, 1996, pp. 103-111. 171 Y.M.Akay, M.Akay, W.Welkowitz, and J.Kostis, “Noninvasive detection of coronary artery disease using wavelet-based fuzzy neural networks”, IEEE Engineering in Medicine and Biology Magazine, vo1.13, 110.5, 1994, pp.761-764. J.Ben-Arie and K.R.Rao, “Nonorthogonal signal representation by Gaussians and Gabor functions”, E E E Trans. on Circuits and Systems, 11, vo1.42, no.6, 1995, pp.402-413.