Machine Intelligence: Quo Vadis? (Advances in Fuzzy Systems- Applications and Theory, Volume 21)

-

and consistent with its theoretical covariance, is accepted if v P (k ) E [h,, b 2 ] . The acceptance interval [ b l ,b2 1 is two-sided (1 - a ) . 100% probability concentration region for v p ( k ) ,that is, P{vp(k) E [h,h ] / H o } = 1 - a . The hidden neuron whose input and output parameters have accumulated certain level of knowledge, and new observations cannot significantly improve it, will be referred as ,,specialized". The moment of neuron specialization is determined based on the number of samples that have activated the neuron above certain threshold. A new hidden neuron will be added if the current input to the RRBF network activates only specialized hidden neurons and criterion v p ( k )> b2 is satisfied. 4.2

RRBF network pruning

During adaptation to time-varying environment some connections or hidden neurons may become insignificant and should be pruned. A connection is insignificant if its parameter and the parameter change are both insignificant. In RRBF network the significance of the input connection is determined based on

120 B. Todorovid,

M.Stankovid and C. Moraga

width c g r the , significance of the recurrent connections based on the width uil, and the significance of the output connection based on weight ai . The well-known pruning method OBS [I], ranks parameters according to the saliency, which is defined as the change in the training error when the particular parameter is eliminated. The parameter with the smallest saliency is pruned. The similar idea is used in [4,5] to derive the on-line pruning method for parameters of the feed forward RBF network, estimated by extended Kalman filter. The same pruning method is applied to the RRBF network [6]. The saliency coefficient of the p-th parameter S,, defined as the minimal increase of the cost (10) when the p-th parameter is pruned, and the corresponding parameter update &(k/k) are given by:

where i ( k / k ) is the parameter estimate and P ( k / k ) is the state estimation error covariance in time step k; u p is p-th unit vector. The hypothesis H o that the parameter is statistically insignificant, is accepted if [6]:

I i ; ( k / k ) I< y, i ; ( k / k ) = Pp,(k/k)-*’*ip(k/k) where y is obtained from P(I $ ( k / k )

i ; ( k / k ) we conclude that I i ; ( k / k ) I=

(15)

I< y/Ho) = 1- a . Comparing

S , ( k ) and

J2s,(k), therefore parameter insignificance

test can be rewritten as: J2Sp(k) < y . In order to test significance of the parameter change, the following fading memory sum forp-th parameter is considered [6]: 6,W

= ~ ~ - ~ ~ ~ ~ ~ ~ ~ ~ , ~ ~ - ~ ~ + ( ~(16) , 7

where 0 < p < 1 and &,,(k - 1) is the p-th parameter’s process noise variance and $,(k) is the activity of the i-th hidden neuron, to whom parameter belongs. The distribution of (16) is approximately the scaled chi-square 6,(k)

- qf, where

c = 1/(2 - p) represents the scaling factor and n’ = (2 - p ) / p is the number of degrees of freedom. The hypothesis H o , that the estimated parameter change is consistent with the process noise variance, is accepted if S P ( k )~ [ b , b zwhere ] , the acceptance interval is chosen to be (1 -a).100% probability concentration region

for 6,(k).

Extended Kalman Falter Based Adaptation

121

The specialized or rarely activated hidden neuron should be pruned if all of its output connections have statistically insignificant parameters or at least one of its input connections has insignificant, small width. Hidden neuron is significant if all of its input connections have significant width and at least one of the output connections has significant weight. 5

Experiments

We have developed the NARX RRBF network to be applied in modeling and control of nonlinear dynamic systems with time-varying dynamics. In our preliminary experiments the network was reduced to the recurrent NAR model (no exogenous inputs) for non-stationary time series prediction. The following examples show that learning algorithm produces very compact networks with small number of hidden neurons and adaptable parameters.

5.1

Non-stationary Mackey-Glass time series prediction

EKF trained RRJ3F was applied to prediction of non-stationary Mackey-Glass time series, defined by differential delay equation:

i ( t ) = -bs(t)

+ as(t - r)/(l

- s(t - z)")

,

(17)

with a = 0.2, b = 0.1. We integrated the equation (17) for 0 < t < 3 100s using forth order Runge Kutta with step size 0.1, and the history initialized to 1.2. During integration delay z was varied according to: t

= 23.5 + 6.5(0.7sin(2xt/3100)+0.3sin(5nt/3100)),

(18)

Every tenth sample was used, and samples from first 100s were discarded, leaving s(k),k = 1,..,3000 samples for training. Observation were obtained according to y ( k ) = s ( k ) - cos(3d/3000) + v(k) , where the v(k) was Gaussian white noise with a variance which gave the signal to noise ratio SNR=40dB. The RRBF network was trained to predict x(k + 6) from x(k) , x(k - 6), x(k - 12), x(k - 18), x ( k - 24), where x(k) was the output of the RREiF network at time step

k, and observations y ( k ) ,k = 30,...,3000 were used for training. At the end of data sequence learning algorithm produced the RRBF network with 5 recurrent connections and 4 hidden neurons (3 significant), i.e. 45 adaptable parameters. The root mean square error, measured on the last 1000 data samples was FWSE=0.0209. Note that learning was sequential with each training sample presented only once to the network. Network was always asked for prediction before the parameter or structure adaptation. Therefore we can consider that the given RMSE is a test error measure.

122

B. TodoroviC. M . StankoviC and C. Moraga 2.5

---"

2.5,

y(k)

- RRBF output

2

2

2 G

'c

$ 1.5

-

1

y

0.5

I

f j

1.5

1

V

0.5 -0.5

I

I 2600

2700 2800 Time steps: k

2900

500

3000

1000

1500

2000

2500

3000

T h e steps: k

a) Comparison

b) Comparison

21.5-

s

10.5-

P

0

500

1000

1500

2000

2500

3000

Time steps: k

c) Innovation

Time steps. k

d) Growth and pruning pattern

Fig. 2. Results of non-stationary Mackey-Glass time series prediction

5.2

Non-stationary Lorenz time series prediction

We have applied recurrent radial basis function network with adaptive structure to model the chaotic system described by Lorenz equations [3]:

where p , o and p are adjustable parameters. Equations (19) were integrated for 0 < t < 75s using forth order Runge Kutta method with step size 0.01. Parameters p , o and p were varied according to:

Extended Kalman Filter Based Adaptation

123

The time series from the variable s2 (sampled at a period of 0.05 seconds) was scaled between [-1,1] and Gaussian white noise v(k) was added to obtain the observations y(k),k = 1,...,1500 with SNR=40&. The FW3F network was trained

to predict x(k + 1) from x(k) , x ( k - 1) , x ( k - 2 ) , x(k - 3), where x ( k ) was the output of the RRBF network at time step k.

i

1

8

0.5

tm 2

-

0

5 -0.5 1 I

I

I100

1200 1300 Time steps: k

1400

1500

a) Comparison

l

-

1000

500

200

400

600 800 1000 1200 1400 Time steps k

b) Companson

1500

Time steps: k

c) Innovation

d) Growth and pruning pattern

Fig. 3. Results of non-stationary Lorenz time series prediction

Each training sample was presented only once to the network. At the end of data sequence the RRBF network with 4 recurrent connections had 3 hidden neurons (2 significant), i.e. 28 adaptable parameters. The root mean square error, measured on the last 1000 data samples was RMSE=0.0147. 6

Conclusions

Extended Kalman Filter is applied to the parameter, state and structure estimation of recurrent radial basis fiinction network. The architecture of recurrent radial basis function network is based upon nonlinear autoregressive model with exogenous

124

B. TodoroviC, M. StanlcoviC and

C. Moraga

inputs. Augmented state of the recurrent radial basis function network is the stacked vector consisting of parameters and outputs of the network. This state is estimated using extended Kalman filter. The on-line structure adaptation of the network is achieved by combining growing and pruning of the hidden units and connections of the network. Statistical criteria for growing and pruning were derived using the state estimation error and innovation statistics. Recurrent network trained by extended Kalman filter is applied in non-stationary time series prediction. 7

References 1. Hassibi, B., Stork, D., Wolff, G. J.: Optimal Brain Surgeon and General Network Pruning, IEEE Int. Conf: Neural Networks, San Francisco (1993) 293299. 2. Kadirkamanathan, V.: A statistical inference based growth criterion for the RBF network, In Proc. IEEE Workshop on Neural Networks for Signal Processing (1 994) 3. Lorenz, E. N.: Deterministic non-periodic flow, J. Atm. Science, vol. 20, pp. 130-141, 1963 4. TodoroviC, B.: Incremental adaptation of RBF network structure, Master Thesis, University of NiS (2000) 5. TodoroviC, B., StankoviC, M, Todorovic-Zarkula, S.: Structurally adaptive RBF network in non-stationary time series prediction, In Proc.IEEE AS-SPCC, Lake Louise, Alberta, Oct. 1-4 (2000) 224-229 6 . TodoroviC, B., StankoviC, M : Training recurrent radial basis function network using extended Kalman filter: parameter, state and structure estimation, In Proc. South-Eastern Europe Workshop on Computational Intelligence and Information Technology, Press T. University of Nis, Yugoslavia (200 1) 7. Williams, R.J.: Some observations on the use of the extended Kalman filter as a recurrent network learning algorithm, Technical Report NU-CCS-92-1. Boston: Northeastern University, College of Computer Science (1992)

A MULTI-NF APPROACH WITH A HYBRID LEARNING ALGORITHM FOR CLASSIFICATION DANUTA RUTKOWSKA AND ARTUR STARCZEWSKI Technical University of Czestochowa, Department of Computer Engineering, Czesfochowa, Poland E-maii:[email protected]

The paper presents an approach to classification based on neuro-fuzzy systems and hybrid learning algorithms. A new method of rule generation is proposed. The rules are used in order to create a connectionist neuro-fuzzy architecture of the multi-NF system. Parameters of the rules are then adjusted by a gradient algorithm. Thus, the system can be employed to solve multi-class classification problems. Some examples are depicted. Keywords: neuro-jizq systems, connectionist networks, learning methods, rule generation, classificationproblems, intelligent systems

1

Introduction

Many different systems have been applied to classification problems. In the area of Computational Intelligence, neural networks, fuzzy systems and neuro-fuzzy systems are widely employed as classifiers; see e.g. [5], [S], [17]. In Section 2 , a neuro-fuzzy system that solves the well-known IRIS classification task [3] is presented. This kind of system can be used successfully in many problems concerning classification, as well as control or function approximation; for details, see [9]. However, in order to achieve better performance of the classification in the case of many classes, the multi-NF system described in Section 3 has been proposed. This system is composed of connectionist neuro-fuzzy (NF) networks, similar to the NF network illustrated in Fig. 1. In Section 4, rule generation algorithms are depicted and a new method is proposed in order to create the NF networks. The number of these rules determines the number of elements (nodes) of the networks. Hybrid learning methods of the NF networks are described in Section 5. A gradient algorithm is applied to parameter tuning. Thus, the parameters of membership functions of the fuzzy IF-THEN rules can be adjusted. Examples of classification problems, solved by means of the multi-NF approach, are depicted in Section 6. Apart from the IRIS classification task, medical diagnosis applications have been considered. The medical data are available on the Internet [7]. Conclusions and final remarks are presented in Section 7. It should be mentioned that the multi-NF approach was introduced in [ 151, also presented in [13], [16], as well as [9], and referred to as the multi-segment or

125

126

D. Rutkowska and A . Starczewski

hierarchical system. However, the learning methods differed from the new algorithm proposed in this paper. 2

Neuro-fuzzy systems for classification

As mentioned in Section 1, classification problems can be solved by means of neural networks, fuzzy systems, and neuro-fuzzy systems. In this paper, neuro-fuzzy systems in the form of connectionist multi-layer networks are considered as classifiers. Fig.1 illustrates such a network that can be used in order to solve the well-known IRIS classification task. The neuro-fuzzy system (network), which is also called the fuzzy inference neural network, is a multi-layer architecture, similar to classical neural networks. However, the elements (nodes, neurons) perform different functions than neurons of neural networks, except of two classical (linear) neurons that realize the sum operations. The neuro-fuzzy network shown in Fig.1 represents a fuzzy system that employs the singleton fuzzifier, Mamdani approach to fuzzy inference with product operation as the Cartesian product, and center-average defuzzification method; see e.g. [ 181, [9], for details. The system portrayed in Fig.1 performs the fuzzy inference using 3 fuzzy IF-THEN rules in the following form: R' : IF x, is A / AND ... AND x,,is A: THEN y is B J

(1)

where x1,.. . ,x, and y are linguistic variables, A,', ...,A,' and B are fuzzy sets; j = 1,. . . ,N . The linguistic variables correspond to the inputs and output of the system. In Fig. 1, there are 3 rules, and 4 inputs, which means that N = 3 and n = 4 . Values of the linguistic variables can be crisp or fuzzy; XI, X2,X 3 , X4 and j j denote crisp input and output values, respectively. The fuzzy sets A : , . . . ,AnJand B are defined in the universes of discourse

XI,..., X , , , Y c R , w h e r e x,,?, E X , and y , y ~ Y , f o ri = l , ..., n . The membership functions of the fuzzy sets A { , . . . ,A,/ and B are usually chosen as Gaussian or triangular functions. In this paper, the Gaussian functions are employed. These functions are expressed as follows:

and

A Multi-NF Approach f o r Classajication 127

vJ,

where T,',o;' and oJare center and width parameters of these membership functions, respectively. In the neuro-fuzzy network illustrated in Fig.1, elements of the first layer, denoted as A,' , for i = 1, ... ,n , j = 1,. ..,N , realize the membership functions (2). Elements of the second layer perform the Cartesian product of the fuzzy sets A / , ...,A,' , using the product operation of the membership functions of these fuzzy sets. The output values of these elements represent the antecedent matching degree. This part of the network corresponds to the IF part (antecedent part) of the rules (1). The next part of the network shown in Fig.1 realizes the center-average defuzzification. The two linear neurons and the element that performs the division operation constitute the defuzzification layer. The weights of the first linear neuron, denoted as v',v2,v3, have the interpretation of the centers of the membership functions of the consequent fuzzy sets B / , for j = 1,2,3. Hence, v J = . These fuzzy sets can also be chosen as singletons, which means that their membership functions equal 1 for y = for y

f

vJ.

Fig. 1. Neuro-fuzzy system for IRIS classification

v J and

0

128

D. Rutkowska and A . Starczewski

As we see in Fig.1, the elements of the second layer, which produce the antecedent matching degree, also called the degree of activation of the rule (or rule firing level) are connected with the appropriate v J , for j = 1,2,3, corresponding to the conclusion part (THEN part) of the rule. The neuro-fuzzy network presented in Fig. 1, as mentioned earlier, can be used in order to solve the IRIS classification task. The problem is to classify the vectors of features of the iris flowers to the proper class of iris species. There are three species of iris: Sestosa, Versicolor, and Virginica. Thus, 3 classes are distinguished. Four features of the iris flowers are measured: sepal length, sepal width, petal length, petal width. Hence, the data vectors that contain the flower measurements include 4 components. The well-known Fisher’s iris data set [3] is composed of 150 data items (vectors), 50 for each of the iris species. The IRIS classification task can be solved using the neuro-fuzzy system shown in Fig.1, based on 3 rules that correspond to the iris species. Thus, the consequent parts of these rules represent the classes of the Sestosa, Versicolor, and Virginica. Values of the weights v’ ,v 2 ,v 3 can be chosen as 1,2, 3, respectively, which means the first (Sestosa), second (Versicolor), and third (Virginica) class. The inputs X,,X,,X3,X, are components of the data vectors. Now, the problem is to choose proper values of the parameters (center and width) of the Gaussian membership functions (2). Let us analyze values of the data of iris flowers. The values of the first feature (sepal length) range from 4.3 to 5.8 for the Sestosa, from 4.9 to 7.0 for the Versicolor, and from 4.9 to 7.9 for the Virginica. The values of the second feature (sepal width) range from 2.3 to 4.4 for the Sestosa, from 2.0 to 3.4 for the Versicolor, and from 2.2 to 3.8 for the Virginica. The values of the third feature (petal length) range from 1.0 to 1.9 for the Sestosa, from 3.0 to 5.1 for the Versicolor, and from 4.5 to 6.9 for the Virginica. The values of the fourth feature (petal width) range from 0.1 to 0.6 for the Sestosa, from 1.0 to 1.8 for the Versicolor, and from 1.4 to 2.5 for the Versicolor. It is easy to notice that the ranges of the sepal length and sepal width features overlap for every class of the iris species. The ranges of the petal length and petal width for the Sestosa are separated from the ranges of these features for the Versicolor and Virginica, which overlap. Thus, it is more difficult to correctly classify the data vectors that belong to the Versicolor and Virginica than to the Sestosa class. In the case of the iris classification, we can determine the parameters of the antecedent membership functions based on the ranges of the feature values, portrayed above. It seems reasonable to choose the center parameters as the centers of these ranges, and the width parameters as the half of these ranges. Hence, we obtain the following values of the center and width parameters of the membership functions (2), for i = 1,2,3,4, and j = 1,2,3 :

A Multi-NF Approach for Classijication 129

A,' : i?: = 5.05,

0:= 0.75

; A: : i?; = 5.95 , 0;= 1.05 ; A: :

Y:

= 6.4, oI = 1.5 3

A : : Z : = 3 . 3 5 , 0:=1.05 ; A ~ : ? ~ = 2 . 7 0 , 0 ~ = 0 . 7A; ~ : Z ~ = 3 . 0 , ~ ~ : A::Zi =1.45, 0: =0.45; A,' : Z,'=4.05, 0: =1.05; A: : ? .: =5.7, o3 3 = 1.2

A::?: =0.35,

D',

=0.25;

A,' : Y,' =1.4,

0: =0.4;

A: : 2; =1.95,

0: ~

0.55

The neuro-fuzzy system shown in Fig.1, with the antecedent membership functions presented above, performs very well solving the iris classification problem. However, several mistakes occur with regards to the data vectors that belong to the overlapping region of the Versicolor and Virginica classes; even after the gradient learning. In order to eliminate these misclassifications, more rules should be applied. The multi-NF approach, described in the next section, when employed to the iris classification task, uses more rules but allows to perform the classification without mistakes, also concerning the overlapping region. 3

Multi-NF approach

The neuro-fuzzy system presented in Section 2 can be employed to various classification tasks. Of course, the number of inputs as well as elements of the first and second layers differ depending on the problem to be solved. The systems of this kind usually perform very well in the case of two distinct classes. It is more difficult to create a similar system in order to classify data to many classes, especially in the case when the regions of the data vectors - associated with the particular classes - overlap in the high dimensional space. Therefore, the multi-NF system has been proposed in [ 151. This system is composed of single NF networks corresponding to the separate classes; see Fig.2. The output module (OM) produces the output value of the system that informs about the overall classification result that is determined based on the outputs of the NF networks.

130

D. Rutkowska and A . Starczewski

lnput vector

_I, NF1 Classification result

b NF2

NFM

OM

b

'

Fig.2. Multi-NF system for classification

The idea of the multi-NF system is to decompose a multi-class classification problem to simpler tasks when data vectors that belong to one class must be separated from others. Each NF network is associated with one individual class and is responsible for recognizing data vectors belonging to this class. The number of the networks in the multi-NF system, M, equals to the number of classes. However, it can equal to the number of classes minus one, since it is obvious that if a data vector is not accepted by every NF network of the system, it belongs to the one remain class which is not associated with the NF networks. There are two possible ways of realizing the multi-NF system. The input vectors can enter into every NF network in parallel or only the first network receives all the input data vectors entered into the system. In the latter case, an input vector is entered into the second NF network if it is discarded by the first one as not belonging to the first class. Thus, the second network as well as the next ones do not receive the input data vectors recognized as members of the first class which is associated with the first NF network. Similarly, the data vectors accepted by the second NF network as belonging to the second class (associated with this network) do not need to be entered into the input of the third and subsequent NF networks. The output values of both systems, for a given input data vector, equal to the output of the NF network which has recognized this input vector as belonging to its class. Thus, the output module transmits the proper NF network output to the output of the system. The output value of the NF network which accepts an input data vector as a member of its class differs from zero, while the output values of the preceding networks of the system equal zero. This kind of performance is resulted from the NF network architecture. The connectionist multi-layer architecture of the network is illustrated in Fig.3. The first NF network, in the multi-NF system, with non-zero output value is taken into account by the OM unit to produce the output value of the system.

A Multi-NF Approach for Classification

131

The NF network is similar to the well-known multi-layer network that represents the fuzzy system based on the Mamdani approach to fuzzy inference, singleton fuzzifier, center-average defuzzification method, Gaussian membership functions, and product operation as the Cartesian product; see [ 181, [9], for details. The first layer refers to antecedent fuzzy sets. The elements of this layer realize Gaussian membership functions of the fuzzy sets in the antecedent part of the IFTHEN rules. The second layer consists of the elements that perform the product operation. Both layers correspond to the first layer of the RBF network which is equivalent to the fuzzy system (in the case when width parameters of the Gaussian functions are equal); for details see [4]. The next layer contains the elements that realize sigmoidal functions. This is the additional layer, introduced in [15] especially for the NF network of the multiNF system in application to classification problems.

Fig.3. NF network of the multi-NF system

The last layers, which include two classical linear neurons and the element that perform the division operation, realize the center-average defuzzification. In addition, the constant R that represents the so-called zero rule was introduced in [ 151for the NF network of the multi-NF system. It is worth emphasizing that the neuro-fuzzy systems for classification do not need the defuzzification layers since the classification result reflects the outputs of the second layer. The elements of the second layer produce values of the so-called antecedent matching degree, that is the degree of activation of the rule (or rule firing level). It represents the membership of the input vector in the Cartesian product of the antecedent fuzzy sets. In classification tasks, each rule is associated with a corresponding class, so the maximal value of the antecedent matching degree indicates the class to which the input data vector belongs. The additional sigmoidal

132

D . Rutkowska and A . Starczewski

layer, as well as the zero rule, can be introduced in order to produce the proper classification decision at the output of the system. Of course, the system output gives the same classification results as concluded from the outputs of the second layer. The sigmoidal layer and the constant R play an important role in this network since every rule corresponds to the same class. As mentioned earlier, each NF network is responsible for recognizing data vectors that belong to one class, associated with this network. The weight value v of the classical (linear) neuron represents this class in the consequent part of the IF-THEN rules. The NF networks can be trained in the similar way to classical neural networks, using the gradient algorithm based on the steepest descent optimization method. This kind of algorithm is analogous to the back-propagation algorithm which is commonly applied to train neural networks [17]. The gradient method for learning classical neuro-fuzzy networks is presented in [ 181, [9]. Learning methods of the NF networks are described in Section 5. 4

Rule generation algorithms

The multi-NF system for classification, illustrated in Fig.2, is composed of M networks shown in Fig.3. Each NF network consists of different number of elements (nodes) in the first three layers. The number of these nodes depends on the number of fuzzy IF-THEN rules. Each NF network is responsible for one class, associated with this network. Different number of the fuzzy rules can be applied to recognize data vectors that belong to an individual class. Each NF network of the multi-NF system represents a fuzzy system that should decide whether an input data vector belongs or does not belong to the class associated with this network. The decision is inferred based on the collection of fuzzy IF-THEN rules that have the same conclusion part, that is “THEN class ck “ for the NF k network, where k = 1, ... ,M . The problem is how to generate this collection of the rules for each NF k network. In this paper, the following algorithm is proposed: 1. For k = 1,...,M , collect the labeled data vectors z I , for 1 = 1,.. . qk , associated with class k , where qk is the number of the data vectors labeled to the class k. 2. Set k = l . 3. Fix small values of D and z . 4. Set c = l , a n d i = l , Z=1. 5. Create cluster Vi with the prototype (cluster center) v j = z I . 6. For 1 = 2,. . .qk , check the Euclidean distance between data vector z I and the prototype vi ,according to the condition:

A Multi-NF Approach for Classijcation 133

(4)

J ~ z-/vi))5 D

If inequality (4) is satisfied, include data vector zI into cluster Vi , and set w J. = z I , for j = 1,. ..,oi , where oiis the number of the data vectors in the cluster Vi . 7 . Update the cluster center v i using the following formula: Wi 1 v; := - v; + x w j I+.;( j=,

)

8. For I = 27...qk,and i = 1,. ..,c , if z I E Vi ,create new clusters by returning to step 5 with c := c + 1 . 9. If stopping criteria (for example, a desired number of clusters) are not met, increase the value of D as follows:

D:=D+z

(6)

and replace the set of the labeled data vectors zI for I = 1,. . .qk ,by the set of the prototypes vi ,for i = 1,... ,c . Then, return to step 4. 10. If the stopping criteria are met or formula (5) does not change values of the prototypes, use the clusters obtained Vi , i = 1,. . . ,c , in order to formulate the fuzzy IF-THEN rules. The number of the rules equals to the number of the clusters, that is c . The membership functions of the antecedent fuzzy sets are Gaussian functions with center and width parameters determined as components of the prototype vectors and the value of D , respectively. 11. If k < M ,then k := k + 1 ,and go to step 3 to generate fuzzy IF-THEN rules for the next class. Otherwise, stop. The idea of the algorithm, proposed above, is to decrease the amount of the initial data set - replacing them by the prototypes. Then, the number of the clusters obtained are decreased in the similar way, when the prototypes are treated as the data vectors and new (bigger) clusters are created. The fuzzy IF-THEN rules are formulated based on the final clusters and their prototypes, determined by this clustering algorithm. Other methods for rule generation can also be applied. The algorithm presented in this section is similar to that introduced in [15] and described in [9]-[12]. Those algorithms allow to obtain clusters and generate fuzzy rules, based on the cluster centers, but use different method of updating the prototypes. The well-known h z z y clustering algorithm, called the fuzzy c-means [2], can also be employed. However, it is worth emphasizing that in this case the number of clusters, c, is not determined by this method and has to be fixed. This means that we should know the number of rules before using this algorithm.

134

5

D. Rutkowska and A . Starczewski

Hybrid learning methods

The rule generation method proposed in Section 4 is employed in order to construct the connectionist multi-layer architectures of the NF-networks presented in Fig.3. As mentioned earlier, the number of elements in the first three layers of these networks depends on the number of the fuzzy IF-THEN rules. The clustering algorithm described in Section 4 determines the number of rules, resulting in the information about the elements in the particular layers. Thus we know the architecture of the networks as well as the form of the rules (1). For each class, the number of rules, N ,equals to the number of clusters, c . The clustering algorithm not only allows to construct the NF architectures but also provides initial values of the parameters of the membership functions. This means that this algorithm can be treated as a learning method that finds values of the parameters (centers and widths of the membership functions) based on the labeled data vectors. However, these values are not optimal and should be tuned using another learning method, for example, a gradient algorithm. Thus, the hybrid approach (see Fig.4) is recommended in order to determine optimal values of the parameters. The clustering algorithm is depicted in Section 4. The gradient algorithm for the NF network shown in Fig.3 is presented in [ 151. The sigmoidal functions, realized by the elements in the third layer of the NF network portrayed in Fig.3, are expressed by the following formula:

s(9)=

1 1+exp[-p(~-h)]

(7)

where h E (0,l) ,p > 0 . It is easy to notice that the network illustrated in Fig.3 is described by the function:

,=I

where

is the antecedent matching degree (rule firing level, degree of activation of the rule), and p is defined by Equation (2). Ai

The constant h , in formula (7), is interpreted as a threshold membership value that refers to the rule firing level. The constant p defines the slope of the sigmoidal

A Multi-NF Approach for Classification

135

function. A large value of p is better for the system performance (for a testing phase) but the lower value is used by the learning method presented below. The gradient algorithm, introduced in [15], adjusts parameters Y/,o: of the membership functions defined by Equation (2), for i = 1,.. .,n , and j = 1,. ..,N , and realized by the elements of the first layer of the NF network shown in Fig.3. This algorithm is based on the steepest descent optimization method, and formulated analogously to the gradient learning procedure that is proposed in [18] for classical neuro-fuzzy systems (Fig. 1). The formulas that represent the algorithm for tuning the parameters , !F o: are expressed as follows:

where J' is the desired output value of the NF network, t = 0,1,2,. .., and a = 2qp, while y~E (0,l) is the stepsize constant of the steepest descent algorithm, also called the learning rate; usually, a = 0.7, while p = 10, and h = 0.36. The parameter tuning by means of formulas (lo), (1 1) is performed based on the learning sequence, which is composed of the pairs of the input data vectors with the desired output values. Although every labeled data vector can be included to the learning sequence, the following procedure is recommended for the multi-NF system. The first NF network, constructed using fizzy rules associated with the first class, is trained based on the whole learning data set. The second NF network employs the learning sequence that is composed of less number of the data vectors, because the vectors labeled to the first class are not included. Similarly, the learning sequence applied to the third NF network does not contain the data vectors labeled to the first and second classes, and so on. Thus, the last NF network uses the shortest learning sequence, which does not contain the data vectors that are labeled to all the preceding classes. It is better to apply scaled data vectors. Initial values of the parameters F,',of , for t = 0 , are derived from the clustering algorithm described in Section 4. The scaled data should be employed.

136

D. Rutkowska and A . Starczewski

Optimal values of these parameters are thus obtained by use of the hybrid learning approach illustrated in Fig.4.

Hybrid

% '

#'

Fig.4. Hybrid learning approach

6

Classification examples

In Section 2 a neuro-fuzzy system for the IRIS classification problem is presented. This system solves the task very well (except the overlapping region) based on the fuzzy IF-THEN rules which have been formulated by analyzing the data set. It is worth emphasizing that this method of rule generation does not work for many other examples, especially when the ranges of feature values overlap. Therefore, the multi-NF system, depicted in Section 3, and the algorithms presented in Sections 4 and 5 are useful. To achieve better classification performance usually more rules than the number of classes should be employed. The multi-NF system, when applied to the iris classification task, solves this problem without any mistakes. However, this system is composed of two NF networks, and one of them is constructed using 8 rules. The network associated with the Sestosa class does not need more than one rule but, as explained in Section 2, this class is much easier to separate from the others. The multi-NF system has been used in order to solve some medical diagnosis tasks. The medical data sets are available on the Internet [7]. One of them is the breast cancer database, which was obtained from the University of Wisconsin Hospitals, Madison, Wisconsin, USA; see also [6], [11. Another medical database, supplied by the Cleveland Clinic Foundation, concerns the heart disease diagnosis. The data set that contains items representing the thyroid disease has also been classified by this system. The breast cancer database has been applied to illustrate how the multi-NF system classifies the data vectors in the case when two classes are distinguished. From this database, we have got 683 data items that contain values of 9 attributes: clump thickness, uniformity of cell size, uniformity of cell shape, marginal adhesion, single epithelial cell size, bare nuclei, bland chromatin, normal nucleoli, mitoses. The values of these attributes are integers that range from 1 to 10. In addition, each data item includes the diagnosis that corresponds to the instance represented by the attribute values. Each instance belongs to one of 2 possible

A Multi-NF Approach for Classzfication

137

classes of the diagnosis: benign or malignant. The former is expressed in the database by integer 2, the latter by 4. Distribution of both class instances in this set of medical data is: benign - 65.5%, malignant - 34.5%. The two classes of the breast cancer diagnosis are linearly inseparable [ 11. The multi-NF system, applied to the breast cancer problem, reduces to only one NF network. However, as mentioned in Section 3, the system composed of two NF networks (one for each class) can also be employed; see [16]. Since the only one is sufficient, the network corresponding to the malignant class was constructed. The system with 9 inputs (for 9-component data vectors) and loutput (for the diagnosis ) was created based on 7 rules. The rules were determined by the clustering algorithm described in Section 4. Then the gradient learning method, depicted in Section 5, tuned parameters of the membership functions, adjusting the rules to the breast cancer task. Using this NF network, we obtained the result of about 98% correct classification decisions inferred by the system. The heart disease database, available on the Internet, contains data items that represent instances expressed by values of 13 attributes. The features taken into account as the attributes are e.g.: age (in years), sex (1 - male, 0 - female), chest pain type (four different types - values 1, 2, 3 or 4), resting blood pressure (in mm Hg), serum cholestoral (in mg/dl), fasting blood sugar (1 - if greater than 120 mg/dl, and 0 - otherwise), resting electrocardiographicresults (three states - values 0, 1, 2), maximum heart rate achieved, exercise induced angina (1 - yes, 0 - no). Each of the data items representing instances of the heart disease is associated with one of 5 classes, i.e. diagnosis, expressed by integers 0, 1, 2, 3, 4. However, almost all published experiments with this database distinguish only two types of diagnosis: presence or absence of the disease. Let us notice that the data set of 297 items includes 160, 54, 35, 35, and 13 items which belong to class 0, 1, 2, 3, and 4, respectively. It is much easier to classify the heart disease data vectors to one of two classes: negative diagnosis (class 0) and positive diagnosis (classes 1,2,3,4 treated as one class). In this case, the former contains 160 and the latter 137 data vectors. The heart disease problem with two classes of the diagnosis has been solved by means of classical neuro-fuzzy systems (in the form of the network shown in Fig. 1) as well as the multi-NF system. In [ l I], [12] some results concerning the use of fuzzy inference neural networks, in application to this task, are presented. Those systems employ fuzzy IF-THEN rules generated by clustering algorithms that differ from that proposed in this paper. The multi-NF system, applied to solve this problem, reduces to only one NF network, similarly to the breast cancer diagnosis task. This network was created based on 18 rules obtained using the algorithm depicted in Section 4. Then, the network was trained according to the hybrid approach described in Section 5. This system inferred about 95% correct diagnosis answers when the heart disease data vectors were entered into its inputs. The multi-NF system is especially useful for classification tasks when more than two classes are distinguished. Therefore, we applied this system in order to solve the heart disease problem with 5 classes, i.e. one class of negative diagnosis

138

D. Rutkowska and A . Starczewski

and four classes of positive diagnosis. The multi-NF system composed of four NF networks, associated with the positive diagnosis, was employed. The networks NF 1, NF 2, NF 3, and NF 4 correspond to classes 4, 3, 2, and 1, respectively. Each network was created based on fuzzy rules generated by the algorithm proposed in Section 4. Thus, these networks incorporate 8, 18, 13, and 16 rules, respectively. The classification result is at least 90% of correct diagnosis and can be better if more rules are used. It is worth emphasizing that there is a very small number of the heart disease data vectors assigned to classes 3, 4, and especially class 5. Therefore, it is almost impossible to split the data sets into learning and testing vectors. When attempted to separate learning and testing sets, the system performed even better working on the learning data vectors (96.7% correct answers) but much worse on the testing data. Of course, this system incorporated less number of rules (6, 12, 11, 14, for the particular NF networks). In this case, the learning and testing sets are composed of 212 and 85, respectively, data vectors randomly chosen from the database containing 297 data items. The multi-NF system composed of the NF networks constructed based on 2, 7, 3, and 9 rules, respectively, performed better on the testing set than the previous system (with the larger number of rules) but worse for the input vectors that belong to the training sequence. The thyroid medical data, available on the Internet, have a learning set and a testing set already prepared. The former contains 3772 instances and the latter 3428 data vectors. Each instance is represented by values of 21 attributes, and belongs to one of 3 classes of diagnosis: one class of negative diagnosis and two classes of positive diagnosis. More precisely, the first class refers to normal (not hypothyroid), the next ones mean hyperfunction and subnormal functioning, respectively. These classes are expressed in the database by integers 1, 2, 3. The feature values, similarly to the heart disease data, are in the form of integers or real numbers. There are 15 attributes having binary values, and 6 attributes characterized by continuous values ranging from 0 to 1. Since the thyroid data vectors include more components, corresponding to the features (attributes), than the heart disease database, the problem is more difficult to classify. On the other hand, this data set contains much more data items, and in addition - the learning and testing sets. Thus, performance of the system designed to solve the thyroid problem can be checked more precisely. The multi-NF system, used in order to classify the thyroid data vectors, consists of two NF networks that correspond to the positive diagnosis (hyperfunction and subnormal functioning classes). The first network was created based on 25 rules, while the second one incorporates 53 rules, generated by the clustering algorithm proposed in Section 4. The system was trained using the gradient method described in Section 5 . When the learning vectors were employed to test the system performance, we observed 94.2% diagnosis correctly inferred. Using the testing vectors, the percentage of correct answers was 93%. It is explained in [7] that a good classifier, applied to the thyroid problem, should produce more than 92%

A Multi-NF Approach for Classifkation

139

correct diagnosis answers. Thus, the classifier proposed in this paper fulfils this requirement.

7

Conclusions

From the simulation results, presented in Section 6, we conclude that the multi-NF system, although introduced to solve classification problems with many classes, is also very useful when only two classes are distinguished. The idea of this system is to accept data vectors that belong to the class represented by the NF network and to reject those which are members of the another class. Thus, if the data vector activates the rules incorporated by the NF network, a non-zero output value is produced by the system, otherwise the output value equals zero. In the case of many classes, the multi-NF system composed of many NF networks, realizes the idea of accepting an input data vector by a particular network as a member of its class and rejecting this vector if it does not belong to this class. Thus the system works according to the main idea of classification that is to recognize similar objects as members of one class and distinguish them from other classes. The clustering method proposed in this paper generates h z z y IF-THEN rules that are employed to construct the NF networks - components of the multi-NF system. This method is a part of the hybrid approach that also includes the gradient algorithm to tune parameters of the membership functions. The multi-NF system with the hybrid learning method can be treated as an intelligent system [9]. Such a system can create its architecture and adjust to the classification problem. This kind of system differs from those which use a rule base prepared by experts. It is worth mentioning that the idea of using a multi-network architecture is also employed by other authors, for example, applying ART-map networks [ 141.

References 1. Bennett K.P. , Mangasarian O.L.: Robust linear programming discrimination of two linearly inseparable sets, Optimization Methods and Software 1, Gordon &

Breach Science Publishers, 1992, pp.23-34. 2. Bezdek J.C.: Pattern Recognition with Fuzzy Objective Function Algorithms, Plenum Press, New York, 1981. 3. Fisher R.A.: The use of multiple measurements in taxonomic problems, Ann. Eugenics, Vo1.7, 1936, pp.179-188. 4. Jang J.-S.R., Sun C.-T.: Functional equivalence between radial basis function networks and fuzzy inference systems, IEEE Transactions on Neural Networks, V01.4, No.1, 1993, pp.156-159. 5 . Kuncheva L.I.: Fuzzy Classifier Design, Physica-Verlag, A Springer-Verlag Company, Heidelberg, New York, 2000.

140 D. Rutkowska and A . Starczewski

6. Mangasarian O.L., Wolberg W.H.: Cancer diagnosis via linear programming, SIAM News, Vo1.23, No.5, 1990, pp.1-18. 7. Mertez C.J., Murphy P.M.: UCI repository of machine learning databases, ht~://~w.ics.uci.ed~vublmachine-learnin~-databases. 8. Nauck D., Klawonn F., Kruse R.: Foundations of Neuro-Fuzzy Systems, John Wiley & Sons, 1997. 9. Rutkowska D.: Neuro-Fuzzy Architectures and Hybrid Learning, PhysicaVerlag, A Springer Verlag Company, Heidelberg, 2002. 10. Rutkowska D., Starczewski A.: Two-stage clustering algorithm, Proc. 4* Conference on Neural Networks and Their Applications, Zakopane, Poland, 1999, pp.220-22 1. 11. Rutkowska D., Starczewski A.: Neuro-fuzzy system with clustering algorithm in application to medical diagnosis, Proc. 4* Conference on Neural Networks and Their Applications, Zakopane, Poland, 1999, pp.533-538. 12. Rutkowska D., Starczewski A.: Fuzzy inference neural networks and their applications to medical diagnosis, In: Fuzzy Systems in Medicine, Szczepaniak P.S., Lisboa P.J.G., Kacprzyk J. (eds.), Physica-Verlag, A Springer Verlag Company, Heidelberg, 2000, pp.503-518. 13. Rutkowska D., Starczewski A.: A neuro-fuzzy classifier that can learn from mistakes, Proceedings of the 1Oth International Conference on System Modelling Control ,Zakopane, 2001, pp. 189-194. 14. Sincak P., Hric M., Sarnovsky J., Kopco N.: Fuzzy cluster identification in the feature space using neural networks, Proc. 6'h International Conference on Soft Computing (IIZUKA 2000), Iizuka, Japan, 2000, pp.849-854. 15. Starczewski A,: Hybrid Learning of Neuro-Fuzzy Systems Using Clustering Algorithms, Ph.D. Thesis, Technical University of Czestochowa, Poland, (1999); in Polish. 16. Starczewski A., Rutkowska D.: New hierarchical structure of neuro-fuzzy systems, Proc. 5'h Conference on Neural Networks and Soft Computing, Zakopane, Poland, 2000, pp.383-388. 17. Zurada J.M: Introduction to Artificial Neural Systems, West Publishing Company, 1992.

A NEURAL FUZZY CLASSIFIER BASED ON MF-ARTMAPa PETER SINck2.’, MARCEL HRIC’ , RICHARD VALO’

PAVOL HORANSKY3, PAVEL KAREL3 ‘Centerfor Intelligent Technologies, Department of Cybernetics and A1 Faculty of EE and Infornatics, Technical University of Kosice, Slovakia ’Siemens PSE& I, AG Vienna, ECANSE Group, Austria, 3Tatrabanka a.s. Bratislava, Slovakia Abstract : The project deals with the further progress in MF-ARTMAP (Membership Function ARTMAP ) approach for classification purposes. The research was accomplish to extend the method with various membership functions utilization in classification procedure. The MF-ARTMAP is similar to ARTMAP family neural networks but provide additional information about the degree of membership of input to the detected fuzzy class in the feature space. The ability to have various types of membership functions had impact in reducing the number of nodes in the recognition layer and having better generalization of the system. Experiments were accomplished on image data and also on financial data. Results are encouraging to continue the research in direction of rule extraction using these technologies. Keywords: Neural Networks, ART neural networks, fuzzy logic, feature space, classification procedures, fuzzy cluster,fuzzy class, classification accuracy assessment

1

Introduction

Intelligent technologies play important role in decision systems. Decision procedure is very important part of any complex decision support system. Decision and pattern recognition are very close procedure and could be viewed as similar processes. Both are working with feature space and in both approaches is the final procedure a classification which makes the final decision or categorization. Even considering prediction could be viewed as trends classification in selected feature space where the final step is again classification with time series-like input and output value. In all cases the classification is considered as unknown function approximation and if it is a function we can make decision, categorization or even prediction. Classification is a very important procedure in the feature space with very high application potential. It is in fact clusters identification in the feature space and their association with classes of interest. The main problem is in definition of the cluster in the feature space which could be very complex and is leading to fuzzy approach to cluster identification and also frequent situation when two or more different clusters in the feature space are associated with the same class. a This project is supported by Vega Project from Ministry of Education of Slovak Republic “Intelligent Technologies in Modeling Intelligent Systems”, partially by EU Maria-Curie Individual Fellowship and also by Tatrabanka a s . Slovakia.

141

142

P. SinEdk et al.

This fist problem is associated with a cluster shape determination that is in fact already given on training data and the problem is the overlap of two or more clusters in the feature space. Therefore the knowledge about fuzzy categorization of the unknown input could be a solution to increase accuracy of the classification system. The later problem is associated with the data nature and the fact that classes determined by experts could be represented by various clusters in the feature space in completely different locations of the feature space. The identification of such a classes is rather difficult with approaches like neural network based on BP approach which must design the approximation of rather complicated discrimination function. The approaches based on ART or ARTMAP technology are rather flexible and much more suitable for these situations when class is consisted of various clusters in the feature space. These two considerations let us to design a MF ARTMAP approach and work on its further development mainly for classification purposes. 2

Motivation of the project

Classification is mapping from feature space into space of classes. Considering supervised mode a determination of training sets is extremely important. Many times is a situation that the training sites produce non-homogeneous training data which are represented in the feature space as union of clusters in different locations in the feature space.

p - adJustment (match tracking)

Figure 1 : The basic concept of ARTMAP neural network

A Neural Fuzzy Classifier Based o n M F - A R T M A P

143

From this perspective the classification approach which uses labeled clustering is the most appropriate way of handling classification task. In Figure 1 is the basic concept of ARTMAP neural network which is approaching a problem as labeled clustering task. The role of the Mapfield neural layer is to associate the various clusters into the desired class. The coefficient p is adjusted in the training phase and reflects the plasticity of the system. Number of nodes in the recognition layer reflects number of identified clusters in the feature space. It very often happens that it is very difficult to decide if a certain point in feature space belongs to a certain class. Therefore an approach based on fuzzy sets has many advantages to reduce misclassification results. Sometimes it is more convenient to have results in the form of transparent information concerning relations of the observed point in the feature space to all classes of interest. Instead of the crisp classifier output we can be more satisfied with outputs based on fuzzy sets and namely on values of membership functions of observed input to fuzzy cluster and fuzzy classes. The notions of fuzzy clusters and fuzzy class are described in the next part of this paper. The motivation is to provide for the end-user a smaller number of misclassifications and higher readability of the classification results. The output of these classification results is a vector of values describing relation of the input to each class of interest. The desire is to have a highly parallel tool with incremental learning ability similar to ARTMAP family neural network. 3

Description of the Method

The project is based on the assumption that data in feature space are organized in fuzzy clusters. Fuzzy cluster is considered as a fuzzy relation A in multidimensional feature space.

=I

1

Where A is a fuzzy relation, Y is feature space, x is point in feature space and Xs, E, F are parameters of fuzzy relation. Each fuzzy relation A is defined by equation [2] as combination of partial functions fi for each dimension.

144 P. SznEdlc et al.

Partial funcion for each dimension is defined by equation [3].

There are many fuzzy clusters in feature space and a certain set of fuzzy clusters create a fuzzy class. Fuzzy class is the union of fuzzy clusters belonging to a considered class defined by training set e.g. n

CL = { U A , } i=l

(4)

Generally we can consider fuzzy class as a set of fuzzy clusters Ai representing the variety of the numerical representation of the class. Relation between pcL (x) and pA(x)must be as follows: kL(X= ) max, = l , n ( p , ( X ) ) (5) where Ai is a fuzzy cluster which belongs to class CL and n is a number of fuzzy clusters creating class CL. The MF-ARTMAP is intended to be such a tool to calculate values of membership functions of X to each class of interest in feature space. In Figure 2 we can find an example of 2-dimensional fuzzy relation with various parameters in single-dimensional directions in the space. The space is n+l dimensional where n is number of features and (n+l)-st dimension is fuzzy relation value. As it is clear from the basic equation of the membership function the influence of the shape is done by variables E and F. Basically these 2 variables influence the shape of the membership function of the cluster in the feature space. The membership function should be adapted to the cluster as it exists in the feature space, which means each cluster should have completely different shape of the membership function with ability to describe the non-linearity of the clusters in the feature space. Variable E and F are different in each cluster and even each dimension of the feature space could lead to the fitted shape of the membership

A Neural Fuzzy Classafier Based o n M F - A R T M A P

145

1 0.8 0.6 0.4 0.2 0 1

0

0

Figure 2 : Two dimensional membership value with various parameters

'1

....T'

..... ........ .: . .. . . .

,.....,,.. ::.... .' .(...,

...:

1

\

....i,. ....,,, ; . .. ..... . . . . ,

05

05

0 1

1

0 1

0 0

'1

.. ..,..

....,...

0 0

,..... ...: .

i

1

.

...

...: ..

:

. ..,,,..A*: ....., . .. .

05

0 1

05

1 0 0

0 1

1 0 0

Figure 3 : The shapes of the membership functions when F=l, E=O. 1, E=0.4 (top) and E=0.2, F=l, F=2 (bottom) function to each detected cluster. In Figure 3 are examples of the influence of E and F to the shape of the membership function. The representation in more than 2 dimensional space is rather complex and could lead to efficient description of the

146 P. SznEdk et al.

membership to the clusters which mean ability to reduce the number of nodes in the recognition layer of MF-ARTMAF’ neural network. The learning procedure on the training data is searching the proper values of E and F in every direction of the feature space and every cluster that is detected by MF-ARTMAP. The procedure is efficient and only few iterations are need to achieve stability and able to identify inputs from the training set into different fuzzy clusters and consequently to fuzzy classes. This gives possibility to have more complex view about categorization and relation of categories in feature space what is a non trivial information if we do have high dimensional feature space defined by interpreter for pattern recognition purposes. 3.1

Description of the neural network topology

Topology of MF ARTMAP is based on a similar architecture to ARTMAP. In Figure 4 can be seen general topology of MF-ARTMAP neural networks with 4 neural layers. The input layer is mapping the input into the comparison layer where partial function fi for each dimension is calculated by equation [3], then the final value of fuzzy relation for each cluster is calculated by equation [4] and the comparison between A values and threshold value are done. The input pattern is tested if it does not belong to one of the clusters. In this case the second layer is dynamically changing according to the number of clusters in the 3-rd layer. So the 2-nd and 3-rd layers are extending according to the number of clusters found in the feature space. The recurrent connection between the 2-nd and 3-rd layer is encoding Xs, E and F parameters associated with a particular membership function. The 4-th layer is representing the mapfield like part of a neural network whose role is to integrate the clusters into a resulting class. The input to the mapfield is from the 3rd layer and also from outside of the neural network as associated output into the overall MF-Artmap neural networks. Basically it can be listed the number of neural layers in the MF-Artmap can be listed as follows: Layer #1 - input mapping layer, number of neurons equal to “n”, where n is dimensionality of the feature space, Layer #2 - comparison layer, number of neurons equal to “n x nc”, where nc is the number of clusters identified in the recognition layer, Layer #3 - recognition layer, number of neurons equal to “nc”, where nc is the number of clusters, Layer #4 - mapfield layer, number of neurons equal to “M”, where M is the number of classes for classification procedure. More info can be found in [ 141.

A Neural Fuzzy Classzjier Based o n M F - A R T M A P 147

40utput 4

G

Figure 4 : General MF-ARTMAP topology with a dynamic number of neurons in the 2-nd and 3-rd neural network layer. 3.2

Parallel MF-ARTMP

The notion of modular neural network has been known for many years. It is a very promising idea of solving complex problems by its distribution into more subproblems that are easy to solve. Basically the “divide and conquer” principles are usually used in modular neural networks. There are some difficult questions about the separability problems and discrimination hyper-planes determination. The key problem is in making an answer to the following problem: “Is it easier to separate one particular class from the feature space or to identify more classes among each other? So in fact the question is about difficulties of dichotomous classification comparing the multi-class approach. The first impression could be that dichotomous classification is always easier than the multi-class approach, but it is very difficult to conclude in general. For investigating these ideas, a parallel ARTMAP approach was designed and tested. The basic philosophy of this approach is illustrated in the following Figure 5. The MF-ARTMAP is suitable for solving the conflict in this approach because the values of membership hnctions to fuzzy clusters are good I’

148

P. SinEdk et al.

indicators of conflict solution among more experts as it is indicated in the Figure 5. The basic advantages of Parallel MF-ARTMAP are as follows: 1. Ultra-fast learning abilities on highly parallel systems e.g. PC-farms. This feature can be very useful in cases of large databases with easier handling of larger amount of data. 2. Easy and comfortable extension of classes of interest by adding a new expert network and train on the new class training data. This is very interesting in case of frequent adding the new classes to list of classes of interest. 3. Easy identification of class “unknown” by measuring the membership function value of the unknown input to the fuzzy classes. If the value is lower than the given value - then it is rejected and proclaimed for class “unknown”. This feature is very important when large data is considered with many classes. 4. Easy readability of fuzzy classes as unions of fuzzy clusters and identification of their basic parameters.

On the other side the basic disadvantage of Parallel MF-ARTMAP is the necessity of more parameters determination considering starting values of vigilance parameters for each expert network separately. output

t

t

Input

Figure 5 The Basic philosophy of Parallel MF ARTMAP

A Neural Fuzzy Classifier Based o n M F - A R T M A P

149

This can be avoided by designing the meta-expert controller setting up a1 the parameters for the overall complex of experts e.g. fuzzy system. Also the final decision on the output of the modular system could be adapted according to the a priory knowledge or any external information.

4

Experimental results

Experiments using benchmark and real-world data were done during this project. The aim was comparative analysis of known CI systems with those modified or developed during the research namely MF-Artmap, and MF-Artmap with adaptation F parameter. Basically the same real-word data were used and so the comparison can be done assuming the same training and testing data, which means the same amount of knowledge was used for classification procedure. 4. I

Accuracy assessment

Accuracy assessment was evaluated using contingency table analysis. A contingency table was used in a basic comparison study between all methods, which were investigated and developed. Some details about contingency table analysis for accuracy assessment of classification results can be found in [8]. 4.2

Experiments on benchmark data

There were 2 benchmark data used for testing classification results for comparative purposes. Circle in the rectangle and double spiral problem was used for dichotomous classification testing. These two benchmark data are used the most for estimating the classifier level of sophistication. If results on these benchmark data are good there, is a good assumption that the classifier will be successful in main applications. On the following tables are the results from the selected benchmark classification procedure. Results are on testing data.

150

P. SinEdk et al.

Actual class

MF Artmap A A’ B’ -

B

MF Artmap with F adaptation

Parallel MF Artmap A

B

A

B

97,62 2,92

98,14

1,46

98,32

1,82

2,38 /97,08

1,86

98,54

1,68

98,18

Parallel MF Artmap with F adaptation

2,1

I

97,66

Table 1 : Results on the “circle in the square” dichotomies classification

1

1

05

05

1 IUI

0 111 lrn

1Jo

o n

Figure 6 : Classification of “circlc in the squarc” without and with F adaptation

A Neural Fuzzy Classifier Based on M F - A R T M A P

151

Actual class Parallel MF Artmap

MF Artmap ..........................................................

A

:

B

....................................

/

1

87,54 10,76 87,96 ’

............ .....

... ................................

12,46 ‘89,24 12,04

,

!

MF Artmap with F adaptation

...,.............

~

Parallel MF Artmap with F adaptation

.............................

B

1

7,86

...............................

I 92,14

88,72

9,95 11,06

1

89,72

Table 2 : Results on the “double spiral” dichotomous classification. As it is clear in the Figure 6 the left picture shows nodes, which are more complex, and able to fit more nonlinear data as approach without F adaptation. The F parameter adaptation shows and positive response in decreasing number of nodes in

Figure 7 : Classification of “double spiral” the recognition layer in MF ARTMAP which brings less computational demands and faster computation. This shows better generalization of the system over the data in training and testing sets.

152

4.3 4.3.1

P. SinEdk et al.

Experiments on red-world data Experiments on multi-spectral image data

Basically the behaviors of the methods were observed on multi-spectral image data with the aim to obtain the best classification accuracy on the test data subset. The Kogice data consists of a training set of 3 164 points in the feature space and of a test set of 3 167 points of the feature space. A point in the feature space has 7 real-valued coordinates of the feature space normalized into the interval (0,l) and 7 binary output values. The class of a fact is determined by the output which has a value of one; the other six output values are zero. The data represents 7 attributes of the color spectrum sensed from Landsat satellite. The representation set was determined by a geographer and was supported by ground verification procedure. The main goal was land use identification using the most precise classification procedure for achieving accurate results. The image was taken over the eastern Slovakia region particularly from the City of Kosice region. There were seven classes of interest picked up for classification procedure . The results with classification with F adaptation are on the table 3 and results show slide improvement to results presented in [ 151.

95.51

0.21

OLOO-

0.00

83.16

0.00

0.00

_ _0.00 -3.85

4.23

0.00

,

2.76

0.00

3.01

.

7.18

100

0.00

1.66

0.00 ,O.O&

11.91 -0.00

96.66

1.10

0.00

0.00

0.22

87.29

0.17

’

7

4.72

~

I

0.00

0.00

/ 0.00 0.00 .

7.04

0.00

0.00

0.00

--0.00

0.00

99.49

5.36

0.64

0.00 j 0.00

0.11

0.00

0.34

83.10

i

Table 3 : Contingency table of Landsat TM classification on the test sites for MFARTMAP

A Neural Fuzzy Classafier Bused o n MF-ARTMAP

153

~

z*

................................

~

................................

.............................

Actual

.... l

~

.

~~

0

. 3

A / B / C ] D I E I F / G

a" B'

C'

D'

1

I 3,55

4,12

5,26

0,OO 1............................... 0,OO 99,03 0,OO

0,OO

0,OO

...............................................................

...............................

1

0,OO

A' 94,92 0,23 .............................

.................................

0,64

...............................................................

'

0,OO .........................

87,50 0,OO

7,71

1 0,OO

! 1

0,OO

I 3,41

0,OO

0,OO

__________

95,51 2,75

I 0,48 I '

""

+

0,9

Table 4 :Contingency table of Landsat TM classification on the test sites for MFARTMAP with F adaptation

Figure 8 : Original image. Highlighted areas were classified by expert (A - urban area, B - barren fields, C - bushes, D - agricultural fields, E - meadows, F forests, G - water)

P. SinEdk et al.

154

Figure 9 : Classification results in the form of thematic map. The results are corresponding with the contingency Table 4. 4.3.2

Experiments on financial fraud data

The application of MF-ARTMAP in financial fraud transaction data is an interesting challenge of this approach. The theoretical advantage of this approach is to reveal relations between types of frauds and indication and recommendations to change a feature space with the aim to be able to discriminate the type of the fraud. The financial fraud data consists of a training set of 9000 points in the feature space and of a test set of 9000 points of the feature space. A point in the feature space has 22 attributes, 10 real and 12 nominal, and 7 binary output values. The class of a fact is determined by the output which has a value of one; the other six output values are zero. Class “0” represents clear financial transaction, classes “1” “6” represent some kinds of fraud financial transaction. The data represents 22 attributes of the financial transaction. The representation set was determined by financial fraud experts. The pre-analysis of the data for processing was made in the following aspects: 1.

2.

analysis of the non-contradictory labeling of the data by expert means the expert should not label similar data with different labels analysis of statistical occurrences of the fraud types - if there is no sufficient data representation of the fraud type - there is difficult to make decision about fraud identification

A Neural Fuzzy Classifier Based on MF-ARTMAP 155

3. way how to encode the linguistic features into the numerical form to be able to process the complex fraud description in 22 dimensional feature space.

The analysis of point 1 shows some problems which has been cleaned and some correction in representing set of fraud has been made. Concerning to the point 2 analysis shows that fraud type 2 and 6 had only 4 cases in representative set - that means we have removed them from the classification analysis and fiuther processing. So for classification processing we have used classes 0 - no fraud, 1, 3, 4, 5 - these are selected frauds. The frauds type 2 and 6 has been not included into the classification because of non sufficient representation in representing set. The classification has been made and tables 5 and 6 representing the classification results on financial fraud data with 2 randomly selected subsets of the representing sets. It is clear that both results are similar and if the representative set per fraud are sufficient is it possible to used these approaches for these type of data. In case if the representation of the fraud is poor - there are only few examples then the identification will be done using rule-base approach implemented into the overall classification approach. This will be a matter of hrther research to integrate the rule-base approach and neural technology into final decision support system. The complex system must also take into consideration the type of error so we have to take extreme care if the fraud is classified as non-fraud - which is more important problem if the non-fraud is classified as a fraud. These considerations will be included into the hture research. The number of clusters associated with each class are interesting view of ART technology can easily handle non-uniform classes consisting of more different clusters and belonging according to the expert into one class. The table 7 shows number of clusters associated which each type of class after training procedure. The MF-ARTMAF' reveals also relation of these frauds to the other types of the fraud and give more complex view to the classification procedure what is important for improvement of the overall process.

156 P. Sincák et al.

Table 5 : Financial fraud transaction classification on test data

Table 6 : Financial fraud transaction classifications with changing training and test data ( results on test data)

A Neural Fuzzy Classijer Based on MF-ARTMAP 157

Type of the class Number of clusters in the class

0

1

3

4

5

763

30

16

5

2

Table 7 : Number of clusters in the feature space associated with type of fraud type Regarding to the results presented in the tables 5 and 6 the values of the membership function of misclassification results belonging to the class 0 - but in fact (according to the expert identification) these frauds were not type 0 (non-fraud) which is a very serious problem. Therefore we did an analysis of the misclassifications patterns and values of the membership functions produced with the MF ARTMAP. There are 2 major results of this analysis to prevent an misclassifications as follows: 1. to give a feedback to the expert to re-evaluate the misclassified patterns and if pattern with high membership function to the non-fraud and is still a fraud according to the expert some further analysis must be done. 2. if the membership hnction of the misclassified patterns to class ‘nonfraud’ is the highest but not exceeding a certain thresholds e.g. 0.7 is probably a new type of fraud which does not belong to any of selected examples and should be classified as another type of cluster (class). This information should be discussed with the expert to confirm these considerations and implement it into the system.

The above consideration can be done only by MF-ARTMAP because of providing the values of membership function of the patterns to the fuzzy clusters or hzzy classes. The ARTMAP technology does not provide this information and is not possible to make and feedback to the expect in this extend. The following values in the Table 8 shows the values of membership functions of the patterns which were misclassified from fraud type 4 to class 0 (non-fraud). The values are in interval and could be a solid base for further discussion with the expert to improve the classification results and decrease the misclassification of fraud 4 to class 0.

158

P. SanEdk et al.

Class

misclassified patterns to the class (interval) originally labeled by

Class 0

From 0.9801 To 0.9993

Class 4

I

From 0.9598 To 0.9890

Table 8: The table shows membership values of the patters to classes 0 and 4 which have been identified by expert as class 4. This table could inspiration for discussion with the expert to improve the classification results and decrease the misclassification of the frauds.

5

Conclusion

The paper presents further research on the MF-ARTMAP with F adaptation approach to the classification procedure. The advantage of this approach is higher readability of the neural network in providing the values of membership functions of input to fuzzy clusters or fuzzy classes identified by this approach. Results of these neural networks are comparable with MF ARTMAE’ [14] on benchmark and real-world data and in addition, provide more useful information about the measure of membership into all fuzzy classes and fuzzy clusters discovered in the feature space. Adaptation of F parameter seems to be a useful tool to investigate in the future. This seems to be an interesting advantage of this approach. The experiments on satellite image and financial fraud data has been presented and MF ATRMAP is able to provide an useful feedback information to the expert in fraud identification to reconsider the fraud labeling. The expert should also provide explanation why the misclassified pattern have close metric distance to class -“nonfraud“ and in this case some additional features should be under consideration to distinguish fraud from non-fraud transaction. In the future research we will provide further development of MF-ARTMAP to be able to retrieve some explicit knowledge from the network topology as well as to be able to implement some apriory knowledge into the system. Also beside parallel MF-ARTMAP we will focus our attention to hierarchical classification system to achieve the highest classification accuracy in the form of classification modular complex. Also some implementation the MF-ARTMAP as an autonomous agent and a part of Intelligent Multi-Agent System.

A Neural Fuzzy Classzfier Based o n M F - A R T M A P

159

References : [l] P.M. Atkinson and A.R.L. Tatnall, “Neural networks in remote sensing,” Int. J. Remote Sensing, vol. 18,4, 1997, 7 11-725. [2] J. Richards, Remote sensing digital image analysis: An introduction. Springer Verlag: Berlin, 1993. [3] B.G. Lees and K. Ritman, ”Decision-tree and rule-induction approach to integration of remotely sensed and GIS data in mapping vegetation in disturbed or hilly environments, ” Environmental Management, vol. 15, 1991, pp. 823-83 1. [4] C.H. Chen, “Trends on information processing for remote sensing,” in Proceedings of the International Geoscience and Remote Sensing Symposium (IGARSS), V O ~ .3, Aug. 3-8 1997, pp. 1190-1192. [5] P. Werbos, Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences, Ph.D. thesis, Harvard University, 1974. [6] G.A. Carpenter, M.N. Gjaja, S. Gopal, and C.E. Woodcock, ”ART neural networks for remote sensing” Vegetation classification from Landsat TM and terrain data,“ IEEE Transactions on Geoscience and Remote Sensing, vol. 35, no. 2, 1997, pp. 308-325. [7] D. Rutkowska, R. Nowicki, ,,Implication-based neuro-fuzzy architectures” in International journal of applied mathematics and computer science, vol. 10, no. 4, 2000, pp. 675-701 [8] P. SinEak, H. Veregin, and N. KopEo, “Conflation techniques in multispectral image processing, ” Geocarto Int, March, pp. 11- 19. 2000. [9] S. Grossberg, “Adaptive pattern classification and universal recoding, I: Feedback, expectation, olfaction, and illusions, Biological Cybernetics, vol. 23, 1976, pp. 187-202. [lo] G.A. Carpenter, B.L. Milenova, and B.W. Noeske, “Distributed ARTMAP: a neural network for fast distributed supervised learning,” Neural Networks, vol. 1 1, no. 5, Jul. 1998, pp. 793-813. [ 111 J.R. Williamson, “Gaussian ARTMAP: A neural network for fast incremental learning of noisy multidimensional maps,” Neural Networks, vol. 9, 1996, pp. 881897. [ 121 R.K. Cunningham, Learning and recognizing patterns of visual motion, color, and form, Unpublished Ph.D. thesis, Boston University, Boston, MA: 1998. [ 131 R. Duda and P. Hart, Pattern Classification and Scene Analysis, Wiley, New York: 1973. [14] SinCak, KopEo, Hric, Veregin: MF-ARTMAP to identify fuzzy clusters in feature space, accepted to IIZUKA 2000, 6-th International Conference on Computational Intelligence, IIZUKA - Japan, October 1-4, 2000 (1) [15] SinEik, Hric, VaSEBk : Pattern Recognition with MF-ARTMAP Neural Networks, accepted to INTECH 200 1, 2-nd International Conference on Intelligent Technologies, Bangkok - Thailand, November 27-29,2001 If

160

P. SinEa'k et al.

[ 161 Ocelikova, E.-Nguyen Hong, T.: Diferential Equations for maximum entropy Image Restoration. In: Proc. of the 31d International Conference ,, Informatics and Algorithms '99 ,,, PreSov, September 9-10, 1999, pp. 214-218. . ISBN 80-88941-059 [I71 Ocelikova, E.-Klimesova, D.: Clustering by Boundary Detection. In: Proc. of the 4" International Scientific Technical Conference "Process control - a P 2000", Pardubice ,June 2000, pp. 108, ISBN 80-7 194-271-5. [18] D. Klimesova , E. Ocelikova,: GIS and Spatial Data Network. In: Proc. of International Conference "Agrarian Perspectives X-Sources of Sutainable Economic Growth in the Third Millenium. Globalisation versus Regionalism. sept. 18-19, 2001, Prague, Czech republic, ISBN 80-2 13-0799-4

MATHEMATICAL PROPERTIES OF VARIOUS FUZZY FLIP-FLOPS AS A BASIS OF FUZZY MEMORY MODULES KAORU HIROTA, SHINICHI YOSHIDA Deptartment of Computational Intelligence and Systems Science, Interdisciplinary Graduate School of Science and Engineering, Tokyo Institute of Technology, 4259 Nagatsuta-cho, Midori-ku, Yokohama 226-8502, Japan

E-mail: he'[email protected] D, T, SR, and J K fuzzy flip-flops are proposed and their characteristics are graphically shown in foiir-max-min, algebraic, bounded, drastic-logical operation systems. Some properties of there logical forms are analytically shown. The circuits of the proposed flip-flops are designed and simulated on VHDL circuit simulator. The result of synthesis shows that the areas of D, T, SR fuzzy flip-flops are nearly 0, 2/3 1 / 2 of that of JK fuzzy flip-flop and the delay times of D, T, SR fuzzy flip-flops are nearly 0, 2 / 3 , 2/3 of that of J K type, respectively.

Keywords: fuzzy flip-flop, memory element, fuzzy logic, FPGA, logic circuit

1

Introduction

In binary logic, flip-flop circuits are used as basic memory modules in sequential circuits and computers. Concept of fuzzy flip-flop, which is a fundamental circuit of fuzzy memory element, is proposed in 1989[1] and implemented with analog transistor and TTL digital circuit[2]. Although their theoretical consideration from a viewpoint of max-min fuzzy logic has been studied, they deal with only JK flip-flop and mainly use under max-min logical operation system (1 - ., A, V). So D, T, and SR fuzzy flip-flops, that are less functional but more simple and fast compared with JK fuzzy flip-flops, have been proposed[6]. This paper surveys D, T, SR, and JK fuzzy flip-flops. Their FPGA circuits are also designed on Synopsys Design Compiler and its circuit areas and delay times are measured. 2

JK Fuzzy Flip-Flop

The min-term expression of a chatacteristic equation of JK flip-flop is

Q(t

+ 1) = JKQ + JKQ + J K Q + J K Q ,

161

(1)

K. Hirota and S. Yoshida

162

simplified as

+

~ ( 1) t = JQ

+KQ,

(2)

where time variable ( t )is omitted in the right hand side for the simplicity. On the other hand, the max-term expression is

+

Q (t 1) = ( J

+ K + Q ) . ( J + + Q ) . ( J + + Q ) . ( J + + K),

(3)

which can be simplified as

+

Q(t 1) = ( J

+ Q ) . (K+ Q )

(4)

Of course, these two equations, Eq. (2) and Eq. (4), are equivalent in binary logic(Boo1ean algebra). Their fuzzy extension, however, are not generally equivalent. f i z z y extensions of (2) and (4) are QR(~

+ 1) = (J@Q@)O(K@OQ),

(5)

Qs(t + 1) = (JOQ)@(K@OQ@),

(6)

and

where (0,0, .@) indicate t-norm, s-norm, and fuzzy negation, which are fuzzy extensions of AND, OR, and NOT, respectively. Their characteristics are graphically illustrated in Fig. 1 and Fig. 2, respectively. When Q(t)=0.5, Fig. 1 (b) shows that it cannot be input the value larger than 0.5 by any input values J(t) and K(t). On the other hand, it can always be input 0.5 and less than 0.5, i.e. it can always be reset. Because of this reason, Eq. 5 is called reset-type J K fuzzy flip-flop. Similar Eq. 6 is called set-type JK fuzzy flip-flop.

Proposition 1 If the following relation concerning fuzzy negation, tnorm, and s-norm is valid

then

will be obtained.

Fuzzy Flip-flops as a Basis of Fuzzy Memory Modules 163

01

a(

0

1

1

1

0.5

0.5

0.5

O0

B

B

(a) Q(t)=O

(b) Q(t)=0.5

0

(c)

Q(t)=1

Figure 1. Characteristics of min-max set-type JK fuzzy flip-flop

(a> Q(t)=o

(b) Q(t)=0.5

( c ) Q(t)=1

Figure 2. Characteristics of min-max reset-type J K fuzzy flip-flop

Proof

164 K. Hirota and S. Yoshida

2.1

Max-Man operation system

Eq. (5) and Eq. ( 6 ) using logical operation system (1 - .,A, V) as (.@,@,@) are QR(~

+ 1) = { J A (1 - Q ) ) V ((1- K ) A Q),

(10)

Qs(t + 1) = ( J V Q) A ((1 - K ) V (1- Q ) ) .

(11)

and

Eq. (10) and Eq. (11) cannot used as an element of memory module, because they cannot memorize any value [0,1] at any time. In order to avoid this problem, another J K fuzzy flip-flop was proposed by combining these two J K fuzzy flip-flops (Eq. (12)).

Eq. (10) and Eq. (11) are continuously connected on the line segment J = K . This can be analytically shown as follows. 1) In the case J = K < Q, Eq. (10) is expressed as { J A (1 - Q ) ) V ((1 - J ) A

Q>

(13)

so

Q>J 2J

A (1 - Q)

(1-J) > ( 1 - Q ) 2 J A ( 1 - Q )

(14) (15)

should hold, and (1 - J ) A &

>J

A (I - & )

(16)

&.

(17)

is obtained. Thus Eq. (10) is equal to

(1 - J ) A

On the other hand, Eq. (11) is clearly equal to

Q A (1 - J ) ,

(18)

thus both Eq. (10) and Eq. (11) are the same. 2) In the case J = K = Q, Eq. (10) and Eq. (11) are clearly equal t o the same value as

J A (1 - J ) .

(19)

Fuzzy Flip-flops as a Basis of Fuzzy Memory Modules

> Q , Eq. (10) is expressed as { J A (1- Q ) ) V ((1- J ) A Q>.

165

3) In the case J = K

(20)

Here

J > Q 2 (1- J ) A Q (1- Q )

(21)

> (1 - J ) >_ (1 - J ) A Q

and

J A (1 - Q)

> (1 - J ) A Q

thus, Eq. (10) is equal to

J A (1- Q ) . On the other hand, Eq. (11) is clearly equal to

J A (1- Q). So both are the same. Therefore the combined JK fuzzy flip-flop Eq. (12) is continuous. The combined J K fuzzy flip-flop can also be expressed as unified form(Eq. (26), Eq. (27)).

Q ( ~ + ~ ) = { ( J A ( ~ - Q ) } ~ { ( ~ - ~ ) ~ Q } v ( J (26) A ( ~ - ~ )

+ 1) = { J V (1- K ) } A ( J V Q) A ((1- K) V (1

Q)}

(27) Its characteristics are shown in Fig. 3, Fig. 4, Fig. 5, and Fig. 6 . They are under logical, algebraic, bounded, and drastic operation system, respectively. Any value [0,1] can always be input into this fuzzy flip-flop. This type of J K fuzzy flip-flop has same ability as binary JK flip-flop, i.e. set and reset operation.

Q(t

-

2.2 Alegebraic operation system Under algebraic operation system (., i, 1 - .) as (@,@, Eq. (5) and Eq. (6) are expressed as

.a), characteristics

QR(~

+ 1) = { J . (1 - Q > ) i { ( l -K ) . Q}

(28)

Q s (+ ~ 1) = ( J + Q > .((1- W i ( 1 - Q)},

(29)

which can be transformed as

QR(t

+ 1) = J + Q - 2 J Q - K Q + J Q 2 + J Q K - J Q K 2 ,

(30)

166

K. Hirota and S. Yoshida

1

0.5

0

0

0

0

(4 Q(t)=O

(b) Q(t)=0.5

(c) Q(t)=l

Figure 3. Characteristics of JK fuzzy flip-flop (Max-Min)

011 1

0.5 0

0

(4 Q(t)=O

(b) Q(t)=0.5

(c) Q(t)=l

Figure 4. Characteristics of J K fuzzy flip-flop (Algebraic)

Qs(t + 1) = J

+Q

-

J Q - J K Q - KQ2 + JKQ’.

+

The difference between Q R ( ~ 1) and Qs(t + 1) is

therefore

(31)

Fuzzy Flip-flops as a Basis of h z z y Memory Modules

167

o(

1

05 0

B

(4 Q(t)=O

(b) Q(t)=0.5

(c) Q(t)=l

Figure 5. Characteristics of J K fuzzy flip-flop(Bounded)

(4 Q(t)=O

(b) Q(t)=0.5

(c) Q(t>=1

Figure 6. Characteristics of J K fuzzy flip-flop(Drastic)

2.3 Bounded operation system

Under bounded operation system (0, @, 1- .), characteristics Eq. (5) and Eq. ( 6 ) are expressed as

168 K. Harota and S. Yoshida

(J-K J>K,KLQ J-K 5 2 K,K 2 Q Q-K JQ

(37)

1 J>K,K
(38)

+

>

hence

+ 1) 5 Qs(t + 1).

QR(~

2.4

(39)

Drastic operation system

Under drastic operation system (A, V, 1 - .), characteristics Eq. ( 5 ) and Eq. (6) are expressed as

Qs(t

+ 1) = (JVQ)A{(l- K)V(1- Q)}.

(41)

Also the following relation is valid.

+ 1) 5 Qs(t + 1).

QR(~

(42)

We omit the proof, but it is easily shown by calculating &(t+ 1) and Qs(t+1) in all cases that each variable of J ( t ) , K ( t ) ,and Q(t) take one of 0, 1, and otherwise.

3

D and T fuzzy flip-flops

In this section we define D and T fuzzy flip-flops as fuzzification of their simplest logical form in binary logic 9.1

D fuzzy flip-flop

Characteristic equation of D flip-flop is

Q(t + 1) = D ( t ) .

(43)

Fuzzy Flip-flops as a Basis of Fuzzy Memory Modules

169

Figure 7. Characteristic of D fuzzy flip-flop

Since there is no operation in Eq.(43), the characteristic equation is naturally extended to the form of D fuzzy flip-flop. In the case of D fuzzy flip-flop, the output Q ( t + l ) is the input D ( t ) E [0,1]. It is the one clock delay element. Figure 7 shows the characteristic of Eq.(43). The characteristic of Eq.(43) is always the same under any fuzzy logical operation systems because it has no fuzzy operation in its right hand side.

3.2

T fuzzy flip-flop

characteristic equations of T fuzzy flip-flop are shown as Eq. (44) and Eq. (45), which are the extensions of T flip-flops of minterm and maxterm expression of in binary logic, respectively.

Q(t+ ~ ) M I N= (TOQ@)O(T@OQ)

(44)

(45) Q(t+ M MAX = (TOQ)O(T@OQ@) In the case of T flip-flop, minterm expression and maxterm expression (generally, they are redundant forms in boolean logic) are irredundant forms and they are not unique. Although Eq. (44) and Eq. (45) are equivalent in binary logic, they are not in fuzzy logic. Now the difference of them are shown under max-min, algebraic, bounded, drastic fuzzy logical operation systems.

Proposition 2 When the fuzzy logical operation system (.@, 0,0) is

170

K. Hirota and S. Yoshida

m u - m i n (1 - .,A, V), algebraic (1 - ., .,i), bounded (1 - *, 0, @), or drastic (1 - .,A, V), then the inequality QMIN(~

+ 1) 5 Q M A X ( ~+ 11,

(46)

holds. Equality always holds if and only if (.@, 0, 0) = (1- .,A, V).

Proof (i)Case of (1- .,A, V). + 1) = ( T ( t )v Q(t))A ( T @ ( tV) Q@(t>> = (T A Q @ )v (T@A Q ) v ( T A T @ ) v (Q A Q @ )

QzAX(t

(47)

Since the value of third and fourth term in the right hand side are at most 1/2, minimum operations between them and the other terms whose values are greater than or equal to 1/2 keeps the same value (Kleene's equality). (47) = (T A Q @ )v (T@A Q ) v { ( T A T @ ) A (Q v Q @ ) }v {(Q A Q @ )A (T v T

(11) Case of

(11i) Case of

Since

is less that or equal to

in all cases,

Fuzzy Flip-flops as a Basis of Fuzzy M e m o r y Modules

171

From the above results,

More general results, discussed below, are also obtained. Proposition 3 If (.@,0 , ~ always ) satisfies following relation

AO(BQC) 2 (AOB)O(AOC), A@(BOC)I: (AOB)O(AOC),

(56)

then the inequality

+ 1) I Q M A X ( ~+ 11,

QMIN(~

(57)

holds. Proof

Figure 8 shows the characteristics of the equation (44) using logical, algebraic, bounded, and drastic operation systems, respectively, while Figure 9 shows those of equation (45).

172

K. Hirota and S. Yoshida

(a) logical

(b) algebraic

(c) bounded

(d) drastic

Figure 8. Characteristics of T fuzzy flip-flop (minterm)

From these figures, we can see that

holds.

3.3 SR fuzzy flip-flop Binary SR flip-flop which has three functions-bit set, bit reset, and hold-is is a basic element of a memory module. If the set input S = 1, then the next state Q(t 1) = 1. If the reset input R = 1, then the next state Q(t 1) = 0. If S = R = 0, then the next state holds current state, i.e., Q ( t + 1) = Q ( t ) . The input S = R = 1is forbidden. But in order to construct the characteristic equation of SR flip-flop, there exist two types of SR fuzzy flip-flop. One is set-type, whose Q ( t + 1) = 1 when S = R = 1, and another is reset-type, whose Q ( t + 1) = 0 when S = R = 1.

+

+

Fuzzy Flip-flops as a Basis of Fuzzy M e m o r y Modules

(a) logical

(b) algebraic

(c) bounded

(d) drastic

173

Figure 9. Characteristics of T fuzzy flip-flop (maxterm)

Eq. (60) and Eq.(61) are the characteristic equations of set-type and reset-type SR fuzzy flip-flop, respectively.

Qs(t + 1) = SO(R"@Q)

+ 1) = @@(SO&)

QR(~

(60)

(61)

Now we show an order relation between set-type and reset-type SR fuzzy flip-flop in some case. Namely under max-min, algebraic, and bounded operation systems, fuzzy truth value of set-type SR fuzzy flip-flop is always greater than or equal t o that of reset-type. Needless t o say, such a relation stands in binary logic. Proposition 4 If a operation system (.@, @,@) satisfies

A@(BOC)L (A@B)O(A@C), AO(BOC) 2 (AOB)O(AOC), then it satisfies

Q s ( t + 1) 2

+ 1).

QR(~

174 K. Hirota and S. Yoshida

Proof Qs(t + 1) = S(t)O(R@(t)OQ(t))

2 (R@OS)O(R@OQ) 2 R@O(S@&)

+ 1)

(64)

QR(~

Q.E.D. Corollary 1 If (.a,@,@) is logical operation system (1 - A , V) or algebraic operation system (1 - .,.,-l), characteristic of SR fuzzy flip-flop satisfies Eq. (63). Proposition 5 If the operation system (.@,0,0) is bounded operation system (1 - ., 0,e),Eq.(63) holds.

Proof

Qs(t + 1) = s(t)@ (R@(t)0Q(t)) S + Q - R (0 5 Q - R 5 1- S ) (SA) (0 5 1 - S 5 Q - R ) (SB) (Q - R 5 0) (SC)

={I

+

Q R ( ~ 1) = R@(t) o ( S ( t )e Q ( t ) ) S Q - R ( R 5 S Q 5 1) (RA) ={0 ( S + Q5 R 51) (RBI 1-R (15S+Q) (RC)

+

(65)

+

(66)

(SB) of Eq.(65) is greater than or equal to all cases of (RA), (RB), and (RC) of Eq.(66), and (RB) of Eq.(66) is smaller than or equal to all cases of

(SA), (SB), and (SC) of Eq.(65), and (SA)-(RA) is 0. Therefore] (SA)-(RC) = S

+ Q - R - (1 - R) = S + Q - 1

2 0 (.: 1 5 S + Q ) (SC)-(RA) = S - ( S Q - R) = -(Q

+

(67) -

R)

2 0 (... Q - R 5 0) (SC)-(RC) = S - (1 - R ) = R >O -

-

(68)

(1 - S )

(..'Q-R
Hence we obtain Q s ( t + 1) 2 Q R (+ ~ 1).

(69)

Fuzzy Flip-flops as a Basis of f i z z y Memory Modules

175

Figure 10. Characteristic of SR fuzzy flip-flop(set type)

Figure 11. Characteristic of SR fuzzy flip-flop (reset type)

Q.E.D. For example, we show the characteristics of SR fuzzy flip-flop when the operation system is operation system (1 - ., A, V). Figure 10 shows the characteristic of set-type SR fuzzy flip-flop, and Figure 11shows that of the resettype. These figures show the value of Q ( t 1) when Q ( t )= 0, Q ( t ) = 0.5, Q ( t )= 1, respectively. In the drastic operation system (1 - ., A, V), there is no uniform inequality between set-type and reset-type, which can be shown as follows. When S = 0.5, R = 0.1, Q(t)= 0.9,

+

Q s ( t + 1) = SV((1 - R ) A Q ( t ) )= 0.5,

+ 1) = (1- R ) A ( S V Q ( t ) ) = 0.9.

QR(~

(70) (71)

. Therefore Q s ( t + 1) < Q R ( ~+ 1)

(72)

176

K. Harota and S. Yoshida Table 1. Comparison of functions

Negation

Input

Hold

0

X

X

X

0

X

D-FFF T-FFF

SR-FFF JK-FFF [ 11 JK-FFF [2]

0

0

X

n

0

0

0

0

0

-

0 200 150

200

hlsounded

9 100

EdBounded 100

50

50

”

0

Figure 12. Circuit Area (gate)

Figure 13. Delay Time (ns)

When S = 0.5, R = 0.9, Q(t)= 0.9,

+ +

Q s ( t 1) = Stu((1- R ) A Q ( t ) )= 0.5, & ~ ( t 1) = (1 - R ) A ( s V Q ( t )= ) 0.1.

(73) (74)

Q s ( t + 1) > Q R ( ~+ 1).

(75)

. Therefore

4

Performance of fuzzy flip-flops

Table 1 shows the comparison of functions that can be performed by various fuzzy flip-flops. The symbol “A”in JK-FFF[l] means that it can input arbitrary value only if both set-type and reset-type are used together and are configured appropriately. Figure 12 shows circuit areas of D, T, and SR fuzzy flip-flops and that of unified form of JK fuzzy flip-flops[2] using logical, algebraic, bounded, and

Fuzzy Flip-flops as a Basis of Fuzzy Memory Modules

177

drastic operation systems, respectively. Figure 13 shows their delay times. “Area” in Figure 12 indicates the number of gates, while “Delay” in Figure 13 indicates the time(ns) that the signal runs from inputs t o outputs. Compared with J K fuzzy flip-flops, T and SR fuzzy flip-flops use 213 and 1/2 area of circuit resources respectively, and their delay times are improved t o 213 of that of JK’s in every operation system. This fact shows that the circuit area of fuzzy flip-flops is proportional to the number of t-norm, s-norm, and fuzzy negation. As D fuzzy flip-flop is composed of output latches only, both its circuit area and delay time are negligible. 5

Summary

We define JK, D, T, and SR fuzzy flip-flop as a basic element of fuzzy memory module. Their characteristics are shown under four operation systems: maxmin logical (l-.,A,V), algebraic (l-.,.,$), bounded (l-.,a,@), anddrastic (1 - ., A, V) operation systems. And then the inequalities between set-tyep and reset-type JK fuzzy flip-flops, between maxterm-expressed and mintermexpressed T fuzzy flip-flops, and between set-type and reset-type SR fuzzy flip-flops are analytically shown. Their circuits are designed using VHDL for digital programmable devices FPGA or CPLD. The result of circuit areas and delay times shows that the areas of D, T, and SR fuzzy flip-flops decrease 213 t o 113 of JK’s, and delay times of them decrease 213 t o 1/2 of JK’s. Concept of these fuzzy flip-flops gives the foundation of the realization of fuzzy sequential circuits, multi-stage fuzzy inference processors, and fuzzy computers.

References

1. K.Hirota, K.Ozawa: “Concept of Fuzzy Flip-Flop” ,IEEE Transactions on Systems, Man, and Cybernetics, Vo1.19 No.5, pp.980-997 (1989) 2. K.Hirota, K.Ozawa: “Fuzzy Flip-Flop and Fuzzy Registers” ,Fuzzy Sets and Systems (North-Holland) , Vo1.32 No.2, pp.139-148 (1989) 3. K.Hirota, W.Pedrycz: “Designing sequential systems with fuzzy J-K flipflops”, Fuzzy Sets and Systems (North-Holland), Vo1.39 No.3,pp.261-278 (1991) 4. J.Diamond, W.Pedrycz, D.McLeod: “Fuzzy J K Flip-Flop as Computational Structures Design and Implemantation”, IEEE Transactions on Circuits and Systems 1I:Analog and Digital Signal Processing), Vo1.41 No.3, pp.215-226 (1994) 5. K.Hirota, W.Pedrycz: “Design of Fuzzy Systems With Fuzzy Flip-Flops”,

178 K . Hirota and S. Yoshida

IEEE Transactions on Systems, Man, and Cybernetics, Vo1.25 No.1, pp.169-176 (1995) 6. S.Yoshida, Y.Takama, K.Hirota: “Fuzzy Flip-Flops and their Applications to Fuzzy Memory Element and Circuit Design using FPGA”, Jounral of Advanced Computational Intelligence, vo1.4 No.5, pp.380-386 (2000)

GENERALIZED T-OPERATORS IMRE J . RUDAS Budapest Polytechnic H-1081 Budapest Nbpszinhaz u. 8. Hungary E-mail: rudas@bmJhu Fuzzy set theory provides a host of attractive aggregation operators for integrating the membership values representing uncertain information. The variety of these operators might be confusing and make it difficult to decide which one to use in a specific model or situation. The tutorial gives a survey of the existing aggregation connectives starting from the classical Zadehian-operators, through the theory of t-operators, till the most up-to-date operators, containing the results of the author and his colleagues on entropy and evolutionary operators.

Keywords: f-operators, evolutionary operators, distance-based operators, entropy-based firzzy connectives, generalized operations.

1

Introduction

Many applications of fuzzy set theory involve the use of a fuzzy rule base to model complex and approximately or not well known systems. The most typical applications are fuzzy logic control, fuzzy expert systems and fuzzy systems modeling. The rule base consists of a set of n IF-THEN rules. In case of two-inputsingle output system the rules are of the form [14]: %, : if x is A, and y is B, then z is C,

also

%, : if x is A, and y is B, then z is C,

also

............ also

1. 2. 3. 4.

:if x is A, and y is B, then z is C,

The hzzy inference process consists of the following four step algorithm [14]: Determination of the relevance or matching of each rule to the current input value. Determination of the output of each rule as hzzy subset of the output space. These individual rule outputs will be denoted by R j , Aggregation of the individual rule outputs to obtain the overall hzzy system output as hzzy subset of the output space. We shall denote this overall output by R. Selection of some actions based upon the output set.

179

180 I. J . Rudas

Our purpose here is to investigate the requirements for the operations that can be used to implement this reasoning process. 2

T-operators, negation and some basic properties

Original fuzzy set theory was formulated in terms of Zadeh’s standard operations of mimimum, maximum and complement. Since 1965 for each of these operations several classes of operators, satisfying appropriate axioms, have been introduced. By accepting some basic conditions, a broad class of set of operations for union and intersection is formed by t-operators. Definition 1 A mapping T :[0,1] x [0,1] + [0,1] is a t-norm if it is commutative, associative, non-decreasing and T ( x , l ) = x , for all x E [0,1]. Definition 2 A mapping S: [O,l]x[O,l] + [0,1] is a t-conorm it is commutative, associative, non-decreasing and S(x,O) = x , for all x E [0,1]. Definition 3 A mapping N :[0,1] -+ [0,1] N is a negation, if non-increasing andN(O)= 1 andN(1)=0. N is a strict negation if N is strictly decreasing and N is a continuous function. N is a strong negation if N is strict and N(N(a))= a, that is, N is involutive. Further it is assumed that T is a t-norm, S is a t-conorm and N is a strict negation. 3

Uninorms

Uninorms are such kind of generations of t-norms and t-conorms where the neutral element can be any number from the unit interval. The class of uninorms seems to play an important role both in theory and application [ 151. Definition 4 [13] A uninorm U is a commutative, associative and increasing binary operator with a neutral element e E [0,1], i.e. U (x, e)= x, Vx E [0,1]. The neutral element e is clearly unique. The case e = 1 leads to t-conorm and the case e = 0 leads to t-norm. The first uninorms were given by Yager and Rybalov [27]

and

Generalized T-operators 181

U , is a conjunctive right-continuous uninorm and U , is a disjunctive leftcontinuous uninorm. Regarding the duality of uninorms Yager and Rybalov have proved the following theorem [27]. Theorem 1 Assume U is a uninorm with identity element e, then U(x, y ) = 1- U(1- x,l - y ) is also a uninorm with neutral element 1- e .

4

Nullnorms

Definition 5 [5] A mapping V :[0,1] x [0,1] + [0, I]. nullnorm, if there exists an absorbing element a E [0,1], i.e., ~ ( x , a=)a , ~x E [oJ], v is commutative, v is associative, non-decreasing and satisfies V(X,O)= x for a11 x E [0, a ]

V(X,I)= x for all x E [a,11

(3) (4)

The Frank equation was studied by Calvo, De Baets, and Fodor in case of uninorms and nullnorms, and they found the followings. Theorem 1 Consider a uninorm U with neutral element e E [0,1], then there exists no nullnorm V with absorbing element e such that the pair (U,V) is a solution of the Frank equation ~ ( xy ), + ~ ( xy ), = x + y for a11 (x,y ) E [0, 11x [0, I]. ( 5 ) 2. Consider a nullnorm V with absorbing element e E [0,1], then there exists no uninorm U with neutral element e such that the pair (U,V) is a solution of the Frank equation.

5

Compensative operations

We have seen that there are no t-operators lying between the minimum and maximum operators. This could be a disadvantage of the application of t-operators as aggregation operators in several intelligent systems where hzzy set theory is used to handle uncertain information. A union operator produces a high output whenever at least one of the input values representing degrees of satisfaction of different features or criteria is high. An intersection operator produces a high output only when all of the inputs are high. In real applications, for example at decision making it would be required that a higher degree of satisfaction of one of the criteria could be compensated for a lower degree of satisfaction of another criteria to a certain extent. In this sense, union

182

I. J . Rudas

provides h l l compensation, while in case of intersection there is no compensation at all. To handle the problem Zimmermann and Zysno [30] has introduced the socalled y-operator as the first compensatory operator. Since than compensative operators have been studied by several authors. Definition 6 An operator M is said to be a compensative if and only if

6

Averaging operators

Averaging operators represent a wide range of aggregation operators [ 141 Definition 7 An averaging operator Mis a mapping

M :[0,1]x [0,1] + [0,1]

(7)

satisfies the following properties: A4.a M ( x ,x) = x, Vx E [0,1] ; idempotency, A4.b M ( x ,y ) = M ( y ,x), Vx,y E [0,1]; commutativity, A4.c M(0,O) = 0, M(1,l) = 1 ,; boundary conditions, A4.d M ( x ,y ) I M ( z , w), if x I z and y I w , monotony, A4.e M is continuous. The next proposition shows that for any averaging operator M, the global evaluation of an action will lie between the worst and the best local rating [ 151. Proposition 8[14] If M is an averaging operator, then

7

Absorbing-norms

Definition 8 Let A be a mapping

A:[071]x[0,1]+[0,1]. A is an absorbing-norm, if for all

y , z E [0,1] satisfies the following axioms: A1 .a There exists an absorbing element a E [0,1], i.e., A(x,a) = a, VX E [0,1]. X,

A1 .b A(x,y) = A b , x ) that is, A is commutative, A1 .c A(A(x,y),z)= A(x,A(y,z))that is, A is associative,

Generalized T-operators

183

It is clear that a is an idempotent element A(a,a)= a, hence the absorbing element is unique. If there would exist at least two absorbing elements a,, a 2 ,a,, f a2 for which A(a, , a 2 )= a,, and A(a,,a,) = a2 ,so thus a, ,= a2 T-operators are special absorbing-operators, namely for any t-norm T, T(0,x)= 0, b'x E [0,1] and for any t-conom S, S(1,x) = 1, b'x E [0,1]. As a direct consequence of the definition we have if x 5 a then A(x,a) = a = max(x,a), if x 2 u then A(x,a) = a = min(x,a). These properties provide the background to define some simple absorbingnorms. The trivial absorbing-norm A, : [O, I] x [0,I] + [O, I] with absorbing element a is A, :(x, y) + a, v(x, Y) E

[w]x [OJ]

(9)

Theorem 2. (Rudas [22] ) The mapping

Amin: [O, 11x [0, I] -+ [0,1] defined as

and the mapping A,,,,,: [0,1] x [0,1] -+ [0,1] defined as mi&, y), if (x,y) E [a,11x [a,11 max(x, y ),elsewhere

are absorbing-norms with absorbing element a. The structures of these absorbing-operatorsare shown in Figure 1,2.

Corollary 1. From the structure of Amin and A,,, the following properties can be concluded:

(w) =

(OJ) = Amin(40) = 0 , Amin(0) =0> 4,, (1J) = A,,(OJ) = AJW) = 1 > A,,, (L1) = 1 .

4 i "

Amin

With the combination of Amin,A,, .and A, fkther absorbing-norms can be defined.

184 I. J . Rudas

a

I

min

min

I

a Figure 1. The structure of

Ain

a Figure 2. The structure of

A,, .

1

4

I

1

Theorem 3. (Rudas, [22]) The mapping A& : [O,l]x [0,1]-+ [0,1] dejined as

and the mapping

g,, : [0,1] x [O, 11+ [0,1] defined as

are absorbing-norms with absorbing element a. The structures are illustrated in Figure 3.

Generalized T-operators

I

a

Figure 3 The structure of

xi,,and A:=

185

1

Theorem 4 (Rudas [22]). Assume that A is an absorbing-norm with absorbing element a. The dual operator of A denoted by 2 A(x,y) = 1 - A(1- x,l - y). is an absorbing-norm with absorbing element 1-a. Let us define a kind of complements of A,,,in and A,,,,, .replacing the operator min with max and the max with min as follows.

Definition 9

We have received the first uninorms given by Yager and Rybalov [27] Because of the constructions of these operators for the pairs (Amin,Ud)and

(Aa,,U , ) the laws of absorption and distributivity are fulfilled. Theorem 5. For the pairs (Amin,U,) and (A,,,, ,U,) the following hold 1. Absorption laws A,~,(u,(x,~),x) = x for all x E [OJ] , u,(A,~,(x,~),x)= x for all x E [OJ] ,

186

I. J . Rudas A,,,= (u,( x , z ) , x ) = x for all x E [OJ] , ~~(~,(x,z),x)= x for all x E [OJ] .

2.

Laws of distributivity For all x E [0,1]

8

distance-based evolutionary operators

Let e be an arbitrary element of the closed unit interval [0,1] and denote by d(x,y) the distance of two elements x and y of [0,1]. The idea of definitions of distancebased operators is generated from the reformulation of the definition of the min and max operators as follows min(x, y ) =

x,if d ( x , ~I)d ( y , ~ ) Y j f d(X?O)'

44)

Definition 10 The maximum distance minimum operator with respect to e E [OJ] is defined as

1' 1'

if d(x,e ) > d ( y ,e ) if d(x, e ) < d ( y ,e) . maxr(x,y) = y , min(n, y)jf d(x,e)= d ( y 7e)

(24)

Definition 11 The maximum distance maximum operator with respect to e E [0,1] is defined as

if d(x,e) > d ( y ,e ) if d(x, e) < d ( y ,e ) .

m a x y (x,y ) = y , ma+, y)jf d(x,e ) = d ( y ,e )

(25)

Definition 12 The minimum distance minimum operator with respect to e E [OJ] is defined as

Generalized T-operators 187

1'

if d(x,e)< d ( y ,e )

if d(x,e ) > d ( y , e ) . miny(x,y) = y, min(x, y)jf d(x,e ) = d(y,e )

(26)

Definition 13 The minimum distance maximum operator with respect to e E [OJ] is defined as minTa (x,y ) =

9

The structure of evolutionary operators

It can be proved by simple computation that the distance-based evolutionary operators can be expressed by means of the min and max operators as follows. max?

=

1 1 1 1

max(x, y)jf y > 2e - x min(x, y), if y < 2e - x min(x, y),if y

= 2e-x

min(x, y), if y > 2e - x

minp

=

maxy

=

ma+, y)jf y < 2e - x min(x, y ), if y = 2e-x max(x, y)jf y > 2e - x min(x, y),if y < 2e - x max(x, y)jf y = 2e-x

mi&, y),if y > 2e - x

miny

=

max(x, y)jf y < 2e - x max(x, y)jf y = 2e-x

The structures of the m a x p and the m i n y operators are illustrated in Fig. 4-5.

188 I. J . Rudas

Figure 4 Maximum distance minimum ( max

)

Figure 5 Minimum distance minimum ( m i n p )

10 Properties of distance-based operators

Theorem 6 The distance-based operators have the following properties (Rudas 1221) maxy 0

max? (x, x) = x, Vx E [O, 11 ,that is m a x y is idempotent, m a x r (e, x ) = x that is, e is the neutral element, m a x y is commutative and associative, m a x r is left continuous,

Generalized T-operators 189

max;’” is increasing on each place of [0,1] x [0,1] maxy maxr” (x,x) = x, ‘dx E [0,1] ,that is maxr” is idempotent, maxra (e, x ) = x that is, e is the neutral element, m a x r is commutative and associative, maxfaax is right continuous, m a x y is increasing on each place of [OJ] x [O,l] . min y minry(x,x) = x, ‘dx E [0,1] ,that is m i n T T is idempotent, m i n y (e,x) = e that is, e is an absorbing element, m i n r is right continuous, m i n y is commutative and associative. minr” minfaax(x, x) = x, ‘dx E [0,1] ,that is m i n y is idempotent, m i n y (e,x)

=e

that is, e is the absorbing element,

m i n y is left continuous, m i n y is commutative and associative.

..

Corollary 2 max? and m a x y are uninorms, both of the operators are compensative ones. Regarding duality of uninorms Yager and Rybalov have proved the following proposition.

Proposition 2 [27] Assume U is a uninorm with identity element e, then f i ( x , y ) = 1 - U(1- x,l - y ) is also a uninorm with identity i? = 1 - e .

Corollary 3 a) The dual operators of the uninorms m a x y is maxyex,and

190

I. J . Rudas

b) the dual operators of the uninorms max? If e = 0 then m a x g = m a x t y . Proposition 3 The Pairs the absorption laws

is max?:

and

satisfy

11 Distance-based operators as parametric evolutionary operators The min and max operators As special cases of distance-based operators can be obtained depending on e as follows: a) i f e = 0 then maxy'" (x,y ) = max(x, y ) , max y (x,y ) = ma&, y ) , min,"'" (x,y >= min(x, y ) , m i n y ( x , y ) = mi&, y ) ,

b) i f e = 1 then m a x p (x,y ) = mi+, y ) , max ;lax (x,y ) = min(x, y ) , m i n y (x,y ) = max(x, y ) , m i n y (x,y ) = max(x, y ) . This means that the distance-based operators form a parametric farnib with parameter e. They are also evolutionaiy types in the sense that if for example in case of m a x r while e is increasing starting from zero till e = 1 the max operator is developing into the min operator.

Generalized T-operators 191

12 Entropy-based fuzzy connectives

The special case of distance-based operators when e = 0.5 leads to the entropy based-operators, namely the maximum fuzziness maximum and the minimum fuzziness minimum operations, introduced by Rudas and Kaynak [20]. The original definitions as fuzzy connectives are summarized in this section with an extension to the minimum fuzziness maximum and the maximum fuzziness minimum operations. The definitions are based on the concept of elementary entropy function, which assigns a value to each element of a fuzzy subset that characterizes its degree of fuzziness. The operations are defined on the basis of selecting the more or the less k z y membership degree as the output of the connectives. Definition 14 Let A be a fuzzy subset of X and A is its conventional fuzzy complement, i.e.

The elementary entropyfunction o f A is [18], [20]

qA:xa

I

if

'1 1

- PA ().,

if

(44)

5

Definition 15 Let A and B be two fuzzy subsets of the universe of discourse X and denote pAand qBtheir elementary entropy functions, respectively. The minimum fuzziness generalized minimum is defined as

The minimum fuzziness generalized maximum is defined as

192

I. J . Rudas

Definition 16 Let A and B be two fuzzy subsets of the universe of discourse X and denote pAand pBtheir elementary entropy functions, respectively. The maximumfuzziness generalized maximum is defined as

u r = u r ( A , B ) = {( X ' P u . " " " ( X ) ) / X E

1

PA

Puy :

'a

XPUF(X)E

('19

Pug('),

maxba

(xX

PB

(x)),

[0,11}> where

if PA (')

V)B

if

P B

' (1' ('> > (1' p.4

if

PA

(x) = PB (x)

'

(47)

The maximumfuzziness generalized minimum, is defined as

,y ='_"(A,B)={

(X,PC"(X))jXEX.PF"(X)t

Ply : a

1

PA PB('), min(PA

[0,11}, where if FA

7 )'(

('),

PB

('))9

(1' (1'

> 0 ) B (')

if P B > PA (') if PA (') = V ) B (')

(48)

Theorem 7 The membership functions of these operators can be expressed in terms of conventional min and max operations as follows [ 181, [20]:

It can be verified easily that the entropy-based operations correspond to the distance-based operations with respect to e as follows:

Generalized T-operators 193

13 Some other families of generalized operations Some methods of generation of generalized operations were introduced by Batyrshin et.al. [1,2,3].

Definition 17 Let T be a mapping T :[0,1] x [0,1] + [0,1] .

(57)

T is a conjunction operation if Definition 18 Let S be a mapping

s:[0,1] x [0,1] + [0,1] S is a disjunction operation

if

=a

(58)

for all a

It can be seen easily, that conjunction and disjunction operations satisfy the following properties:

T(0,O)= T(0,l) = T(1,O) = 0, T(1,l) = 1, T(0,a) = T(a,O) = 0, S(0,l) = S(1,O)= S(1,l) = 1 S(0,O) = 0. S(1,a) = S(a,l) = 1 Including these properties in the axiom systems, the concepts of quasioperations are introduced. These axioms together with commutativity and associativity the axiom skeleton of generalized conjunction and disjunction operations given by Klir and Folger are obtained [ 171.

Definition 19 Let T be a mapping

T : [0,1] x [0,1] -+ [0,1] T is a quasi-conjunction operation if

(65)

194

I. J . Rudas

T(0,O) = T(0,l) = T( 1,O) = 0, T( 1,l) = 1. Definition 20 Let S be a mapping

s:[0,1] x [0,1] + [0,1] S is a quasi-disjunction operation if S(O,1)

= S(1,O) = S(1,l) =

1, S(0,O) = 0.

Definition 21 Let T be a mapping T :[0,1] x [OJ]

+ [0,1] .

T is a pseudo-conjunction operation if T(0,l) = T(1,O) = 0, Definition 22 Let S be a mapping

s:[O,l] x [OJ] -+ [0,1] S is a pseudo-disjunction operation if S(0,l) = S(1,O) = 1.

Easy to see that the axioms imply T(O,O)=OandS(l,l)= 1, but T( 1,l) = 1 and S(0,O)= 0 do not follow. Theorem 8

Suppose T1, T2, are conjunctions, S1, and S2 are pseudo-

disjunctions, g1, g2:[0,1] [0,1] are non-decreasing functions such that g1(1)= g2(1)=1; then the following functions

are conjunction operations. As basic conjunctions T2 and Tl and basic disjunction S1 one can use, for example, the simplest T-norms and T-conorms. Disjunction operations may be generated dually or obtained from conjunctions by means of negation operation. Examples of simple parametric conjunctions obtained in this way are the following [1,31:

Generalized T-operators 195

T(x,y) = min(x,y)max{l-p(1-$,I-q(1-y), 0}, T(.,y) = min(x,y)mm(2’,yq), T(x,y) = min(x,y)min(l,2”+yq), T(X,Y) = xy(x + y - xy) p , T(x,y) = min(x,y)(2’ + yq - 2’ yq)

where p,q are positive numbers.

Theorem 9.[3]. Suppose T is a quasi-conjunction, S is a quasi-disjunction andf; g, h:[O,1]+[0, I ] are non-decreasing functions such thatAO)= g(0) = h(0) =0, A1)= g( 1) = h( 1) = 1 ; then thefunctions Tl(X,Y)=AT(g(x),hO” s1(XJY)=AS(g(x),h O ) ) ,

are a quasi-conjunction and a quasi-disjunction respectively.

Theorem 10. Suppose TI, T, are quasi-conjunctions, Sl and Sz are pseudodisjunctions, h, gl, g2:[0,1] +[O, 11 are non-decreasing functions such that gl(l) = gZ(1) = I ; then the following functions T h y ) = TZ(Tl(X,Y),&(gl(x)>g2O>)>, T ( a 4 = TZ(Tl(X,Y),g1 (SI(4Y)N, T(X,Y)= T2(Tl(X>Y), s2(h(4mx,Y))),

are quasi-conjunctions. From Theorems 9 and 10 we can obtain recursively the following simplest parametric quasi-conjunction operations: T(x,y) = min(xP,yq), T(X,Y) =xpyq, T(X,Y) =(xY)p(x + Y - XY) q, where p,q are positive real numbers. We see that the new definition of conjunction and disjunction operations provides the possibility to build the simplest parametric classes of conjunction and disjunction operations. In the following sections we discuss some applications of the new conjunctions in fuzzy modeling.

Acknowledgement The authors gratefully acknowledge the support by the FANUC‘s “Financial Assistance to Research and Development Activities in the Field of Advanced

196 I. J . Rudas

Automation Technology Fund” for 200 1, and the support by the Hungarian National Research Fund OTKA in the projects T 34651 and T 034212.

References 1. I. Batyrshin, 0. Kaynak, “Generation of generalized conjunction and disjunction operations,” in: IPMU-98, Paris, La Sorbonne, pp. 1762-1768, 1998. 2. I. Batyrshin, 0. Kaynak, I. Rudas. Generalized conjunction and disjunction operations for fuzzy control, in: EUFIT’98, Aachen, Germany, 1998, vol. 1, pp. 52-57. 3. I. Batyrshin and 0. Kaynak, “Parametric classes of generalized conjunction and disjunction operations for fuzzy modeling,” IEEE Trans. Fuzzy Syst., vol. 7, N 5, pp. 586-596, 1999. 4. I. Batyrshin and M. Wagenknecht, “Noninvolutive negations on [O,l],” J. Fuzzy Math., vol. 5, N 4, pp. 997-1010, 1997. 5. 0. Cervinka, “Automatic tuning of parametric T-norms and T-conorms in fuzzy modeling,” in Proc. 7th IFSA World Congress. Prague: ACADEMIA, 1997, V O ~1,. pp. 416-421. 6. De Baets, B.: Uninorms: the known classes. Fuzzy Logic ans Intelligent Technologies for Nuclear Science and Industry. (D. Ruan, H.A. Abderrahim, P.D’hondt and E. Kerre, eds). Proc. Third Int. FLINS Workshop (Antwerp. Belgium), World Scientific Publishing, Singapore, 1988, pp. 2 1-28. 7. G. De Cooman and E.E. Kerre, “Order norms on bounded partially ordered sets,” J. Fuzzy Math., vol. 2, pp. 281-3 10, 1994. 8. D. Dubois and H. Prade, “A review of fuzzy set aggregation connectives,” Inform. Sci.,vol. 36, pp. 85-121, 1985. 9. J.C. Fodor, “Strict preference relations based on weak t-norms,” Fuzzy Sets Syst., V O ~ 43, . pp. 327-336, 1991. 10. J.C. Fodor, “A new look at fuzzy connectives,” Fuzzy Sets and Syst., vol. 57, pp. 141 - 148, 1993. 11. J. Fodor, M. Roubens: Fuzzy Preference Modelling and Multicriteria Decision Support. Kluwer Academic Publishers, 1994, The Netherlands. 12. Fodor, J., Yager, R., Rybalov, A.: Structure of uninorms. International Journal Uncertuinq, Fuzziness, and Knowledge Based Systems. 5 (1997) pp. 41 1-427. 13. M. M. Gupta and J. Qi, “Theory of T-norms and fuzzy inference methods,” Fuzzy Sets Syst., vol. 40, pp. 431-450, 1991. 14. R. Fulltr: Introduction to Neuro-Fuzzy Systems. Physica-Verlag. , Heidelberg, 2000. 15. R. Fuller: Fuzzy Reasoning and Fuzzy Optimization. Turku Center for Computer Sciences. No 9. September 1998.

Generalized T-operators 197

16. E. P: Klement, “Construction of fuzzy o-algebras using triangular norms,” J. Math. Anal. Appl., vol. 85, pp. 543-565, 1982. 17. G. J. Klir and T.A: Folger, Fuzzy Sets, Uncertainty, and Information. PrenticeHall International, 1988. 18. I. J. Rudas, 0. Kaynak: New Types of Generalized Operations. Computational Intelligence: Soji Computing and Fuzzy-Neuro Integraiion with Applications. Springer NATO ASI Series. Series F: Computer and Systems Sciences, Vol. 192. 1998. (0.Kaynak, L. A. Zadeh, B. Tiirksen, I. J. Rudas editors), pp. 128156. 19. I. J. Rudas, A. Szeghegyi, J. F: Bitb, G. Geary:. Non Monotone Generalized Fuzzy Operations for Fuzzy Logic Controllers. 5th IEEE International Workshop on Robotics in Alpe-Adria-Danube Region. June, 1996, Budapest, pp. 529-533. 20. I. J. Rudas, M. 0. Kaynak: Minimum and maximum fizziness generalized operators Fuzzy Sets and Systems 98 (1998) 83-94. 21. Rudas, I.J., Kaynak, M.O.: Entropy-Based Operations on Fuzzy Sets. IEEE Transactions on Fuzzy Systems, ~01.6,no. 1, February 1998. pp. 33-40. 22. I.J. Rudas: Evolutionary operators new parametric type operator families The International Journal of Fuzzy Systems. . 1999. Vo1.23. No.2. pp. 149-166. 23. M. Sugeno, “An introductory survey offuzzy control,” Inform. Sci., vol. 36, pp. 59-83, 1985. 24. I. B. Turksen, “Intelligent hzzy system modeling,” in Kaynak et al. (eds), Computational Intelligence - Soft Computing and Fuzzy-Neuro Integration with applications, Springer-Verlag: 1998, pp. 157 - 176. 25. L.-X. Wang, A Course in Fuzzy Systems and Control. Prentice Hall PTR. Upper Saddle River, NJ, 1997. 26. S . Weber, “A general concept of fuzzy connectives, negations and implications based on t-norms and t-conorms,”Fuzzy Sets Syst., vol. 11, pp. 115-134, 1983. 27. Yager, R., Rybalov, A.: Uninorm aggregation operators. Fuzzy Sets and Systems 80 (1998), pp. 105-136. 28. Yager, R., Aggregation operators and fuzzy system modeling. Fuzzy Sets and Systems 67 (1994), pp. 129-145. 29. L.A. Zadeh, “Fuzzy sets,” Inform. Contr., vol. 8, pp. 338-353, 1965. 30. Zimmermann, H.-J., Zysno, P. (1980), “Latent connectives in human decision making. 31. Zimmermann, H.Fuzzy set theory and its applications. Kluwer Academic Publishers., 1981.

This page intentionally left blank

FUZZY RULE EXTRACTION FROM INPUT/OUTPUT DATA L. T. KOCZY*, J. BOTZHEIM*, A. B. RUANO**, A. CHONG ***, T. D. GEDEON*** *Department of Telecommunication and Telematics, Budapest University of Technology and Economics, H-l I 1 7 Budapest, Pcizrndny P. sktany Ud,Hungavy [email protected] botzheim@,alpha.tit.bme.hu

** Department of Electronic Engineering and Computing, Faculty of Sciences and Technology, University of Algarve, 8000 Faro, Portugal antano@,ualg.ut * ** Department of Information Technology Murdoch University, Australia South Street, Murdoch 6150 WA cchong@,murdoch.edu.au taedeon@,central.murdoch.edu.au This paper discusses the question how the membership functions in a fuzzy rule based system can be extracted without human interference. There are several training algorithms, which have been developed initially for neural networks and can be adapted to fuzzy systems. Other algorithms for the extraction of fuzzy rules are inspired by biological evolution. In this paper one of the most successful neural networks training algorithm, the Levenberg-Marquardt algorithm, is discussed, and a very novel evolutionary method, the so-called “bacterial algorithm”, are introduced. The class of membership functions investigated is restricted to the trapezoidal one as it is general enough for practical applications and is anyway the most widely used one. The method can be easily extended to arbitrary piecewise linear functions as well. Apart from the neural networks and evolutional algorithms, fuzzy clustering has also been used for rule extraction. One of the clustering-based rule extraction algorithms that works on the projection of data is also reported in the paper.

Keywords: fuzzy systems, fuzzy rules extraction, Levenberg-Marquardt algorithm, bacterial algorithm.

1

Introduction

In the application of fuzzy systems to modelling and control one of the most important tasks is to find the optimal rule base. This might be given by a human expert or might be given a priori by the linguistic description of the modelled system. If, however, neither a suitable expert, nor the necessary linguistic descriptions are available, the system has to be designed by other methods based on numerical data. In training, the objective is to tune the membership functions in the

199

200 L. T. Koczy et al.

fuzzy system such that the system performs a desired mapping of input to output. The mapping is given by a set of examples of this function, the so-called pattern set. Each pattern pairp of the pattern set consists of an input activation vector x@)and its target activation vector fi’.After training the membership functions, when an input activation x@’ is presented, the resulting output vector y@)of the fuzzy system should equal the target vector fi’. The distance between the target and the actual output vector must be minimised for each pattern. There are several methods to minimise these distances. These methods can be adopted from the field of neural networks or from the evolution phenomenon of living beings. The paper is organised as follows. Section 2 describes the basics of fuzzy systems. The neural network algorithm is shown in the Section 3. In Section 4 the bacterial algorithm is described. In section 5, 6, 7, 8 and 9, we complete our discussion by introducing a fuzzy clustering based rule extraction technique. Section 10 is the conclusion of the paper.

2

Fuzzysystems

The theory of fuzzy logic was developed by Zadeh in the early 1960s. His theory was essentially the rediscovering the multivalued logic created by Lukasiewicz, however, with going much further in some application related aspects. In 1973 he pointed out that the new fuzzy concept could be excellently used for describing very complex problems with a system of fuzzy relations represented by a fuzzy rule base [l]. A fuzzy rule base contains fuzzy rules Ri:

Ri: IF (XI is A i l ) AND (x2 is Ai2) AND ... AND (xn is Ain) THEN (y is Bi), (1) where A , and Bi are fuzzy sets, xi and y are fuzzy inputs and output. The meaning of the structure of a rule is the following:

IF Premise THEN Conclusion (2) where the premise consists of antecedents linked by fuzzy AND operators. The Centre of Gravity (COG) defuzzification method is used here because it is general and easy to compute. This method calculates the crisp output by the sums of the centre of gravities of the conclusions. Thus, a fuzzy inference system can compute output y of an input vector x. The main purpose is to make the best solution possible for each input vector, therefore the optimum rule base need to be found. 3

The Levenberg-Marquardt algorithm

Our goal is to find the optimal rule base. This means that the distances between the targets and the corresponding output vector gives smaller error than in the case of another rule base. The error is measured by the following function:

Fuzzy Rule Extraction from Input/Output Data

& ,

201

p=l

where P is the number of patterns in the pattern set, to is thepthtarget vector, yo is the prhoutput vector. The most used method to minimise (3) is the Error-Back-Propagation (BP) algorithm, [7] which is a steepest descent algorithm. A newer method is the Levenberg-Marquardt algorithm. Denoting the parameter vector by L, and the Jacobean matrix by - :

~ [ k=]~ [ k-] ~ [ -k11 the LM update, is given as the solution of (5) In (5), a is a regularization parameter, which controls the both the search direction and the magnitude of the update. ( 5 ) can be recast as:

The complexity of this operation is of U ( n ’ ) , where n is the number of

J- . If we apply this algorithm in fuzzy systems then the parameters z columns of must be found. The structure of the fuzzy system is the following:

A grid in the input space is defined. Vectors of knots must be defined, one for each input dimension. These vectors determine the place of the membership functions. In each ithaxis Aj,j will be defined, where

j = OJ, . . . , q .They are arranged in such a way that

is the minimal and

Ai,
the maximal value of the

it’ input. The

membership functions can be interpreted as in Fig. 1 (it is a quasi Ruspini-partition, namely it is a Ruspini-partition between Ai,, and ).

202 L.T..'2 Koczy et al.

Fig. 1. The positions of the membership functions

Thejlh interval of the iih input is denoted by I , , and is defined as follows:

I.. = 1,J

for j

= l,...,ri - I

[A;,;-,A;,,]if j = rj

Thej i hmembership function in the ithdimension:

L

~ 0 7

otherwise

or in another form:

where:

N;,;W =

1,

if x E Ii,;

0,

If x E I;,,

i

The number of membership function in the ithdimension is:

where Lx

means the lower integer part of real x. The number of fuzzy rules:

Fuzzy Rule Extraction from Input/Output Data 203 n

R=JJWZi i=l

where n is the number of input dimensions. Each output can be defined also by four parameters: v , . ,v~,, , ~ vr,3, , v , ,in~the yth rule. The task is to find the location of each 2 and v parameter. We suppose that the number of knots is given in each input dimension, so the number of rules is also given. The inference method must be discussed. The activation degree of the

rfhrule (if the t-norm is the product):

n

wris the importance of the ithmembership

Y th rule

if the input vector is x and pi,r( x i ) is the

fimction in the rth rule. The

y f houtput

is being cut in the height

w,. Then with the COG defuzzification method the output is calculated:.

If this defuzzification method is used, the integrals can be easily computed. In

(15) youtwill be the following: R

r=l

c, =3w,((, -(,N-w,) 0,=32(vr,3vr,4 -Y,1Vr,J

(16)

4 =~((v,,-~,4+v,,I-v,,2)(Vr~-~,4-~,I+~,2) The algorithm needs to be modified, so that the ordering relation between the knots for each input dimension is maintained. In order to maintain the same search direction, the update vector, when violation of (7) is detected ( I,+, [k]< I,[ k]), is

204 L. T.Kdczy et al.

reduced by a factor g so that the position of the (i+l)Ih knot is located half-way between the previous distance of the two corresponding knots:

Jacobean computation: (4) can be written as follows:

where

@ "

'Y &r

h

r

(19)

"r,i,k

where hr,i,k is the kthbreakpoint of the ithmembership function in the rthrule. So:

From (14) can be seen that wr depends on the membership functions , and each membership function depends only on four A parameters. So, the derivatives of wr will be:

The derivatives of the membership functions will be calculated as follows:

Fuzzy Rule Extraction from Input/Output Data

205

ay and ay have to be also computed. From (16) the following can be written: h

r

3 Vr,k

ay dpar.

-

1

3

den:

dF * dG . -n u m 2 dParr* dparr. (den)

where par, = w,,v,,,,v , , ~ vr,3, , v , , ~den , is the denominator and num is the numerator of (16) , resp.

Fr, is the < member of the sum in the numerator and

G . is the r* member in the denominator, v,k is the k*hbreakpoint of the rthoutput membership function. The derivatives will be as follows:

206

L. 2'. Kdczy et al. n

The number of columns of J will be n, -

A parameters and n, 4

+ n, . nA= 4 c m,.is the number of i=l

= 4R is the number of v parameters.

The bacterial algorithm

Nature inspired some evolutionary optimisation algorithms suitable for global optimisation of even non-linear, high-dimensional, multimodal, and discontinuous problems. The original genetic algorithm (GA) was developed by Holland [2] and was based on the process of evolution of biological organisms. A more recent approach is the bacterial evolutionary algorithm (BEA). This gives an alternative to other algorithms, because it is simpler, and it is possible to reach lower error within a short time. This method includes not only the bacterial mutation operator - as in the pseudo-bacterial genetic algorithm (PBGA) - but also the gene transfer operation. These operations were inspired by the microbial evolution phenomenon. The bacterial mutation operation optimises the chromosome of one bacterium, the gene transfer operation allows to transfer information between the bacteria in the population. 4.I

The encoding method:

The membership functions are described by four parameters with the four breakpoints of the trapezium. Moreover the membership functions are identified by the two indices i and j . So, the membership function A,(a,,b,,c,,dJ belongs to the ith rule and the jth input variable. B,(a,,b,,c,,dJ is the output membership function of the ithrule. The relative importance of the jth fuzzy variable in the ithrule:

, if u, < x i
I

0,

otherwise

where avs3v*v-5iv must hold. So the encoding method of a fuzzy system with two inputs and one output, see in Fig. 2.

Fuzzy Rule Extraction from Input/Output Data 207

lRule1 IRule2 )Rule3 IRuIe4

I

. . . . . ., , . . . ., . , ,

IRulen

1

Fig. 2. Fuzzy rules encoded in a chromosome

For example, Rule 3 in Fig. 2 means:

ifxl is A31(4.3,5.I,5.4,6.3)and x2 is A32(1.2,I.3,2.7,3.1) then y is B3(2.6,3.8,4.I , 4.4)

(26)

where A , and Bi mean the trapezoidal membership function with the four breakpoints.

4.2 4.2.1

The bacterial evolutionary algorithm: Generating the initial population

First the initial (random) bacteria population is created. The population consists of n chromosomes (bacteria). This means that all membership functions in the chromosomes must be randomly initialised. The initial number of rules in one chromosome is Nmm.So, n(k+I)NmIIX membership functions are created, where k is the number of input variables in the given problem, and each membership function has four parameters. 4.2.2

Bacterial mutation

The bacterial mutation is applied to each chromosome one by one [5]. First, m -1 copies (clones) of the rule base are generated [4]. Then a certain part of the chromosome [5] is randomly selected and the parameters of this selected part are randomly changed in each clone (mutation). Next all the clones and the original bacterium are evaluated by the error criterion. The best individual transfers the mutated part into the other individuals. This cycle is repeated for the remaining parts, until all parts of the chromosome have been mutated and tested. At the end the best rule base is kept and the remaining m -1 are discharged. The above procedure is repeated once more, however, with different parameters [4]. It is an important question how long is one part suffering mutation and what is the degree of the mutation (expressed as the relative size in terms of the interval). This approach allows both selecting more than one membership function and fine-tuning. The number of mutated membership functions and the mutation degree are external parameters of the bacterial mutation. If selecting more than one membership

208

L. T.Kdczy et a1

function is allowed then the local minima in the optimisation process can be avoided. Application of changing two or more membership functions at a time with fine-tuning in the first bacterial mutation and in the second step changing one membership function in the whole interval of the given variable was proposed in ~41. 4.3

Gene transfer

The gene transfer operation allows the recombination of genetic information between two bacteria. (see Fig.3) 1. First the population must be divided into two halves. The better bacteria are called superior half, the other bacteria are called inferior half. [5] 2. One bacterium is randomly chosen from the superior half, this will be the source bacierium, and another is randomly chosen from the inferior half, this will be the destination bacterium.

L

I

lILmmIm Fig .3. Gene transfer

3. A “good” part from the source bacterium is chosen and this part can overwrite a not-so-good part of the destination bacterium or simple be added. [ 5 ] A good part can be a fuzzy rule with a high degree of activation value. The activation value of a fuzzy rule can be calculated for example:

-

where Wi is the mean activation value of the rule ifhrule,

w(’)is the activation

value of the i‘hrule for the j f hpattern, P is the number of patterns. So the best

Fuzzy Rule Extraction from Input/Output Data

4.

4.4

209

part of the source bacterium is the rule which has the greatest mean activation value. 1, 2, and 3 are repeated for Ninftimes, where Ninf is the number of “infections” per generation. [5]. Stop condition

If the population satisfies a stop condition or the maximum generation number is reached then the algorithm ends, otherwise it returns to the bacterial mutation step. Between the bacterial mutation and the gene transfer step further operators can be applied, which optimise the rule number in the rule base. [6]

5

Clustering-Based Rule Extraction Technique

Recently, clustering-based approaches have been proposed for rule extraction [ 10, 111. Most of the techniques use the idea of partitioning the input space into fixed regions to form the antecedents of the fuzzy rules. Although these techniques have the advantage of efficiency, they may lead to the creation of a dense rule-base that suffers fiom rule explosion. In general, the number of rules generated is P where d is the number of input dimensions and t is the terms per input. In this case, the number of rules generated grows exponentially with the increase of input dimensions. Due to this reason, the techniques are not suited for generating fuzzy rule-bases that have a large number of input dimensions. Among the rule extraction techniques proposed in the literature, Sugeno and Yasukawa’s [12] technique (abbreviated as SY method hereafter) is one of the earliest works that emphasize the generation of a sparse rule-base. The SY approach clusters only the output data and induces the rules by computing the projections to the input domains of the cylindrical extensions of the fuzzy clusters. This way, the method produces only the necessary number of rules for the inputoutput sample data (more details later). The paper [12] discusses the proposed technique at the methodological level leaving out some implementation details. The SY technique was further examined in [ 16,171 where additional readily implementable techniques are propose to complete the modeling methodology. In this paper, a novel rule extraction technique that works on the projection of data and fuzzy clustering is introduced. The technique, known as projection-based fuzzy rule extraction (PB) , extends the idea of SY approach for fuzzy modeling. The goal of the rule extraction technique is to automate the construction of a fuzzy rule base from a set of input-output sample data. In the next paragraphs, a brief description of the original SY method is presented. In the first step of SY modelling, the Regularity criterion [13] is used to assist in the identification of ‘true’ input variables that have significant influence on the output. The input variables that have less or no influence on the output are ignored

210

L. 2'. Kdczy et al.

for the rest of the process. The true input variables are then used in the actual rule extraction process. The rule extraction process starts with the determinition of the partition of the output space. This is done by using fuzzy c-means clustering [I41 (see section 6). For each output fuzzy cluster B, resulting from the fuzzy c-means clustering, a cluster in the input space A, can be induced. The input cluster can be projected onto the various input dimensions to produce rules of the form: If xl is A,, and x2 is A12and . . . x, is A,,, then y is B, However, it is remarked in the paper [ 121 that there can be more than one fuzzy cluster in the input space which corresponds to the same fuzzy cluster B,. In this case more than one rule is formed with the same consequent. Suppose that two input clusters (A, and A,) are induced from the output cluster B,, we obtain the following two rules: If xl is All and x2 is A,* and . . . x, is A,, then y is B, If x l is A,] and x2 is AJ2and . . . x, is A,,, then y is B, Unfortunately, no concrete procedures for determining the number of input clusters to be induced from an output cluster is discussed in the paper. 6

Fuzzy C-Means Clustering

Given a set of data, Fuzzy c-Means clustering (FCMC) performs clustering by iteratively searching for a set of fuzzy partitions and the associated cluster centers that represent the structure of the data as best as possible. The FCMC algorithm relies on the user to specify the number of clusters present in the set of data to be clustered. Given the number of cluster c, FCMC partitions the data X = {xI,x2,.. .,x,,} into c fuzzy partitions by minimizing the within group sum of squared error objective function as follows (eqn 28).

where J,(U,V) is the sum of squared error for the set of fuzzy clusters represented by the membership matrix U, and the associated set of cluster centers V. 11. 1 1 is some inner product-induced norm. In the formula, llxk - vi1I2 represents the distance between the data xk and the cluster center vi. The squared error is used as a performance index that measures the weighted sum of distances between cluster centers and elements in the corresponding fuzzy clusters. The number rn governs the influence of membership grades in the performance index. The partition becomes fuzzier with increasing m and it is proven that the FCMC algorithm

Fuzzy Rule Extraction from Input/Output Data

converges for any m minimum are

E

211

( 1 , ~ [l]. ) The necessary conditions for eqn 28 to reach its

Vi, V k and

(Uik k=i

>"

In each iteration of the FCMC algorithm, matrix U is computed using eqn 29 and the associated cluster centers are computed as eqn 30. This is followed by computing the square error in eqn 28. The algorithm stops when either the error is below a certain tolerance value or its improvement over the previous iteration is below a certain threshold. The FCMC algorithm cannot be used in situations where the number of clusters in a set of data is not known in advance. Since the introduction of FCMC, a reasonable amount of work has been done on finding the optimal number of cluster in a set of data. This is referred to as the cluster validity problem. The optimal number of clusters are determined by means of a criterion, known as the cluster validity index. Sugeno and Fukuyama proposes the following cluster validitiy index [ 151.

where n is the number of data points to be clustered; c is the number of clusters; xk is the Ph data, x i s the average of data; v, is the ithcluster center; Uikis the membership degree of the Ph data with respect to the ith cluster and in is the fuzzy exponent. The number of clusters, c, is determined so that S(c) reaches a local minimum as c increases. The terms 11 xk - vi 11 and 11 vi- 5 11 represent the variance in each cluster and variance between clusters respectively. Therefore, the optimal number of clusters is found by minimizing the distance between data to the corresponding cluster center and maximizes the distance between data in different clusters. Other cluster validity indexes can be found in [ 181

212

7

L. T. K6czv et a1

Projection-Based Rule Extraction Technique

The main idea of the proposed PB rule extraction technique is similar to the SY approach [ 121. Below, our novel technique will be discussed. The algorithm of the approach consists of 5 steps.

1. Perfom fuzzy clustering on the output space. We suggest the use of Fuzzy-c

Means Clustering with a suitable cluster validity index but remark that our algorithm does not place any restriction on the clustering technique used. 2. For each fuzzy output cluster Bi approximated, all the points belonging to the cluster are projected back to each of the input dimensions. For each dimension, fuzzy clustering is again applied to the projection of the points. 3. The previous step results in multiple 1D fuzzy clusters in each input dimension. For each fuzzy cluster, a trapezoidal cluster is approximated using the technique discussed in section 8. 4. Each of the n clusters (cd,- cd,,in the input dimension d, is a projection of the multi-dimensional input cluster to that input dimension. Next the clusters from individual dimensions are combined to form the multi-dimensional input cluster. The merging process is described in section 9. 5. For each of the multi-dimensional cluster identified, a rule can be formed. For example, if a multi-dimensional cluster is formed with [CI1,CZ3, for the points projected from output cluster Bi, we obtain the following rule: If x l is C11 and x2 is Cz3and x3 is C34 then y is Bi Where c,,is the nthcluster identified at input dimension d. 8

Trapezoidal Approximation Technique

A straightforward trapezoidal approximation technique from fuzzy clusters has been proposed in [16]. The trapeze approximation of the clustered raw data is proceeded in two steps. First, the convex hull of the original data set is determined, then the convex hull is approximated by a trapezoidal membership function. Figure 4 depicts the idea of the construction of trapezoidal membership function.

Fig.4. The positions of the membership functions

Fuzzy Rule Extraction f r o m Input/Output Data

213

The algorithm is the following: 1. Determine pminand pm, the minimudmaximum of the membership degree of the data point in the given cluster, x, the fist point where the maximum pmax is attained, further dminand d,, the minimudmaximum of all the data values in the given cluster domain. 2. Set the boundaries of investigated interval to

where r >1 is a given parameter, usually between 2 and 4. 3. Determine the parameters of the left slope, x1 and x2, as: a) Let us initialize x, as the last data point which has smaller membership degree than m and xi be the next point of the convex hull: p(xj+l)= p(xi) > m. A X j 1 < m; (If there is no such point x in the convex hull which satisfies p(x) < m then x, and xi is the first two leftmost point of the convex hull). Further the parameters of the left X I = dmin and xz = dmin. b) Let xI1 and x * be ~ the location of the intersection made by the support and the core, respectively, with the line passing through the points (xj ; p(xj)) and (xi; p(xi)) (see Figure 5). c) If > x1 then x1 := xI1 d) If x*z < x2 or i = j + 1 then x2 := xt2 e) If xi+l 5 x,, then let i := i + 1 and go to step (3b), otherwise continue 4. Determine the parameters of the left slope, x3 and x4, analogously as in the previous step. 5. Order the parameters according to x1 5 x2 5 x3 5 x4. For the convenience of later steps, we convert the trapezoidal clusters to ruspini partition as illustrated in f i g y 1. M = 213 ,a!&

____________________________ / Fig.5. Determination of X’, and X;

m = 113 range

214

9

L. T. K d c t y et al.

Merging Scheme

The reconstruction of multi-dimensional clusters by combining the 1D clusters identified at each dimension can be problematic. Let t be the average number of clusters identified at each dimension. The total number of possible combination is k. Since the number of combination grows exponentially with the increase of dimensions, examining every combination of the 1D clusters is computationally infeasible. In this section, we propose a fast merging technique. The merging process involves the use of a threshold t. The cluster in the multidimensional space is determined to be the region where the number of projected points in the region exceeds t. A point p is contained in the cluster Ciif p,&) > p&) for all jgi. The 4-step algorithm is presented. I . Find one of the multi-dimensional clusters C where the number of points that falls into all its projection exceeds the threshold t. 2 . Remove all data points that are contained in the cluster C approximated. 3. Repeat steps 1 - 2 until no more cluster can be found. The pseudo-C-code for step 1 of the algorithm is presented as follows. 10 PROCEDURE find-MD-cluster

Let Ui be the set of one-dimensional clusters in dimension i Let mdCluster = [ ] for i = 1 to k for each unit u E Ui utemp = mdCluster x u if utemp is dense denseunit = utemp break end if end for end for For the convenience of discussion, we define [ ] as the zero-dimensional (empty) cluster where [ 3 x C, = C,. The algorithm scans through each of the k dimensions to find one of the multi-dimensional clusters in the data giving the complexity O(k). Having identified a cluster, all the data points that is contained within the cluster is removed. The process is repeated until no more cluster can be found. The overall complexity is O(ck) where c is the total number of clusters in the data. Since the complexity of the algorithm is linear, it is computational feasible to deal with data with very large number of dimensions.

Fuzzy Rule Extraction from Input/Output Data

215

11 Conclusions The Leveneberg-Marquardt and the bacterial evolutionary algorithm were described in this paper. The bacterial algorithm seems to be simpler and robust. In this method, the fuzzy systems can be described easier, and the algorithm allows using not only Ruspini-partition. Apart from the neural and evolutionary algorithm, a clustering based rule extraction technique has also been reported. The technique has the advatage of being computationally efficient.

References 1. L.A.Zadeh: Outline of a new approach to the analysis of complex systems and decision processes, IEEE Tr. Systems, Man and Cybernetics 3 (1973), pp. 2844. 2. J.H.Holland: Adaptation in Nature and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence, MIT Press, Cambridge, 1992. 3. L.J. Fogel, A.J.Owens, and M.J.Walsh: Artificial Intelligence through Simulated Evolution, Wiley, New York, 1966. 4. M.Salmeri, M.Re, E. Petrongari, and G.C.Cardarilli: A Novel Bacterial Algorithm to Extract the Rule Base from a Training Set, Dept. of Electronic Engineering, University of Rome, 1999. 5. N.E.Nawa, and T.Furuhashi: Fuzzy System Parameters Discovery by Bacterial Evolutionary Algorithm, IEEE Tr. Fuzzy Systems 7 (1999), pp. 608-616. 6. J.Botzheim, B.Hhmori, and L.T.K6czy: Extracting trapezoidal membership functions of a hzzy rule system by bacterial algorithm, 7'h Fuzzy Days, Dortmund 2001,Springer-Verlag, pp. 2 18-227. 7. Werbos, P., Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences, PhD. Dissertation, Appl. . Math., Harvard University, USA, 1974 8. Marquardt, D., An Algorithm for Least-Squares Estimation of Nonlinear Parameters, SIAM J. Appl. Math., 1 1, 1963, pp. 43 1-441 9. A.E.Ruano, C. Cabrita, J.V.Oliveira, L.T.K6czy, D. Tikk: Supervised Training Algorithms for B-Spline Neural Networks and Fuzzy Systems, Joint gth IFSA World Congress and 20th NAFIPS International Conference, Vancouver, Canada, 200 1. 10. Wong, K.W., Fung, C.C., and Wong, P.M. A self-generating fuzzy rules inference systems for petrophysical properties prediction. in Proceedings of IEEE International Conference on Intelligent Processing Systems. 1997. Beijing. 11. Wang, L.X. and Mendel, J.M., Generating fuzzy rules by learning from examples. IEEE Transactions on Systems, Man, and Cybernetics, 1992. 22(6): p. 1414-1427.

216

L. T.Kdczy et

al.

12. Sugeno, M. and Yasukawa, T., A fuzzy-logic-based approach to qualitative modeling. IEEE Transactions on Fuzzy Systems, 1993. l(1): p. 7-31. 13. Ihara, J., Group method of data handling towards a modelling of complex systms - IV. Systems and Control (in Japanese), 1980.24: p. 158-168. 14. Bezdek, J.C., Pattern Reconition with Fuzzy Objective Function Algorithms. 1981, New York: Plenum Press. 15. Fukuyama, Y. and Sugeno, M. A new method of choosing the number of clusters for fuzzy c-means method. in Proceedings of the 5‘h Fuzzy System Symposium. 1989. 16. Tikk, D., Gedeon, T. D., Koczy, L. T., and Biro, G. Implementation details of problems in Sugeno and Yasukawa’s qualitative modelling. Research Working Paper RWP-IT-02-2001, School of Information Technology, Murdoch University, Perth, W.A., 2001. P. 17. 17. Wong, K.W., Kbczy, L.T., Gedeon, T.D., Chong, A., Tikk, D. (2001) “Improvement of the Clusters Searching Algorithm in Sugeno and Yasukawa’s Qualitative Modeling Approach” in Reusch, B. (Ed), Computational Intelligence: Theory and Applications, Springer-Verlag, Berlin, Proceedings of 7th Fuzzy Days in Dortmund - International Conference on Computational Intelligence, October 200 1, Dortmund, pp. 536-549. 18. Yang, M.S., Wu, K.L., A New Validity Index For Fuzzy Clustering, in Proceedings of IEEE International Conference on Fuzzy Systems, December, Melbourne, 4 pages.

KNOWLEDGE DISCOVERY FROM CONTINUOUS DATA USING ARTIFICIAL NEURAL NETWORKS RUDY SETIONO, JACEK ZURADA National University of Singapore, 3 Science Drive 2, Singapore 11 7543 University of Louisville, Computational Intelligence Lab, Louisville, K Y 40208, USA We describe a method for knowledge discovery from continuous data using neural networks. The method extracts linear regression rules from trained neural networks. Each rule in the extracted rule set corresponds to a subregion of the input space and a linear function involving the relevant input attributes of the data approximates the network output for all data samples in this subregion. Having such linear regression rules is desirable in some application as the output from neural networks are highly nonlinear and generally difficult to comprehend. In contrast, better insights may be obtained from the data if the relationship between the input and the output is expressed as linear functions. The method that we proposed approximates the nonlinear output of a trained neural network by a small number of linear functions while at the same time maintains its predictive accuracy. Illustrations on how the method works on real world data sets are given.

Keywords: knowledge discovery, neural networks, linear regression rules. 1

Introduction

The problem of knowledge discovery from data with continuous attributes has been attracting interests from both the machine learning and statistics communities. Many methods that have been developed for discovering interesting relationship among the data attributes generate decision trees. For example, the model-tree predictor method M5’ generates binary decision trees with linear regression functions at the leaf nodes [l]. This method is an improved re-implementation of Quinlan’s M5 [2]. Tree generation is achieved by splitting each non-leaf node so as to minimize the intra-subset variation in the function values down each branch of the tree. Nodes in the tree are pruned by estimating the expected error at the nodes for the test data. A linear regression model is computed for each non-leaf node by using only the attributes that appear in nodes below this non-leaf node. The model is simplified by dropping terms that would increase the predictive accuracy of the tree. Several tree generating methods first convert the regression problem into a classification problem by discretizing the target values. Once the decision tree

217

218

R. Setiono and J . Zurada

for classification has been generated, a regression tree where the predictions at the leaf nodes are computed by different models can be obtained. The simplest model predicts all samples in a leaf node as constant value, which is the average of the target values of all training samples that fall into this node. A more complex model such as linear regression function of the input attributes can also be built to improve accuracy. The crucial question is then how t o divide the range of the continuous target values into a number of subintervals. Too many subintervals will introduce many classes, which in turn will generate large decision trees. The leaf nodes of such trees represent small regions of the input space with only a small number of samples in each region. The generated linear models are likely t o overfit the data as a result. On the other hand, if there are too few subintervals, the overall predictive accuracy of the regression tree may be poor due t o inadequacy of the linear models to fit the data. The RECLA system [3] resolves the problem of determining the number of subintervals by employing the wrapper approach [4]. Using this approach, a decision tree generating algorithm is wrapped around a method which discretizes the continuous target values. The possible ways for discretization include computing: (1) equally probable intervals, where each subinterval contains the same number of data samples; (2) equal width intervals; and (3) k-means clustering, where the sum of the distances of all data samples in an interval t o its center-of-gravity (centroid) is minimized. The best set of subintervals is determined according to the accuracy of the generated decision trees on the cross-validation set. The Relative Unsupervised Discretization (RUDE) algorithm [5] discretizes not only the continuous target values but also all of the continuous input variables in the data. A key component of this algorithm is a clustering algorithm which groups values of the target feature into subintervals that are characterized by similar values of some input attributes. Once the variables have been discretized, C4.5 [6] is applied for solving the original regression problem. The experimental results on five benchmark data sets show that when compared to the trees from the data sets discretized using the equal width interval and the k-means clustering methods, the decision trees generated from RUDE-discretized data sets have fewer nodes but lower predictive accuracy. CART [7] is a method for regression (as well as classification) that does not require the user to discretize the target values. An optimal regression tree is generated in two steps. In the first step, a tree that overfits the data is generated. All the data samples are assigned to the root node. If the prediction errors are too large, the samples in the node are partitioned into

Knowledge Discovery from Continuous Data 219

two groups by selecting an attribute value that would result in a maximum reduction in errors. Two leaf nodes are generated for the partitioned data set. By recursively splitting the nodes, an overfitting tree is constructed. In order t o improve the prediction accuracy, the second step of CART prunes the tree by combining nodes along the tree branches upward. Cross-validation samples can be used t o determine which nodes are t o be combined. Once an optimal tree has been constructed, prediction for a new sample is usually computed by simply averaging the target values of all samples that fall in the corresponding leaf node. Neural networks, which have been shown t o be universal function approximators, would be an excellent choice for knowledge discovery from data with continuous attributes. The main drawback, however, is the complex nature of the input-output relation of the data as represented by a trained network. The simplest network architecture commonly applied for function approximation is the feedforward neural network with one layer of hidden units. As the computation of the hidden unit activation involves nonlinear function such as the hyperbolic or the sigmoidal tangent functions, it is not easy to explain the predictions from the network as meaningful rules that may be useful for knowledge discovery. In this paper, we present the method REFANN (Rule Extraction from Function Approximating Neural Networks). The key component of this method is the approximation of the hidden unit activation function as a piecewise linear function. Having a piece-wise linear hidden unit activation function enables us to represent the network's prediction as a set of regression rules that is simple enough for users t o understand. Rules for function approximation normally take the form: if (condition is satasfied), then predict y = f (x), where f (x)is either a constant or a linear function of x,the input attributes of the data. This type of rules is suitable because of their similarity to the traditional statistical approach of parametric regression and to those generated by decision tree methods. More than one rule is usually needed to approximate the nonlinear inputoutput mapping of the network well. The approximation of the nonlinear hidden unit activation function divides the input space into smaller subregions. The method presented here predicts the target values of all samples that fall in the same subregion by a single rule consequent in the form of a linear equation whose coefficients are determined by the weights of the network connections. The rules generated are almost as accurate as the original networks from which the rules are extracted. For many data sets that we tested, the number of rules is sufficiently small that useful knowledge about the problem domain can be discovered.

220

2

R. Setiono and J. Zumda

Network training and pruning algorithm

The available data samples (I,, y p ) , p = 1 , 2 , . . . , K where input I, E IRN and target yp E IR,are first randomly divided into 3 subsets: the training, the cross-validation and the test sets. Using the training data set, a network with H hidden units is trained, so as t o minimize the sum of squared errors E(w,v) augmented with a penalty term O(w,v):

where €1 , € 2 , ,5 are positive penalty parameters, w i j is the weight of the connections from input unit j to hidden unit i and wi is the weight of the connection from hidden unit i t o the output unit. The penalty term O(w,v) when minimized pushes the weight values towards the origin of the weight space, and in practice results in many final weights taking values near or at 0. Network connections with such weights may be removed from the network without sacrificing the network accuracy [8]. The hidden unit activation value Ai, for input I, and its predicted function value Y p are computed as follows:

c H

~p

=

ViAip,

(3)

i=I

Ijp is the value of input j for pattern p . The function h ( z ) is the hidden unit activation function. This function is normally the sigmoid function or the hyperbolic tangent function. We have used the hyperbolic tangent function tanh(c) = (et - e-c)/(et + e d ) . Once the network has been trained, its hidden and input units are inspected as candidates for possible removal by a network pruning algorithm. A pruning algorithm called N2PFA (Neural Network Pruning for Function Approximation) [9] has been developed. This algorithm removes redundant and irrelevant units by computing the mean absolute error (MAE) of the network’s prediction. In particular, ET and E X , respectively the MAE’S on the training set 7 and the cross-validation set X , are used to detcrmine when

Knowledge Discovery from Continuous Data 221

pruning should be terminated:

where 1 7 1 and (XI are the cardinality of the training and cross-validation sets, respectively.

Algorithm NSPFA Given: Dataset ( I ,,yp),p= 1 , 2 , . . . , K . Objective: Find a neural network with reduced number of hidden and input units that fits the data and generalizes well. Step 1. Split the data into 3 subsets: training, cross-validation, and test sets. Step 2. Train a network with a sufficiently large number of hidden units to minimize the error function (1). Step 3. Compute E T and EX, and set ETbest = ET, EXbest = EX, Emax = max{ ETbest, EXbest}. Step 4. Remove redundant hidden units:

1. For each i = 1 , 2 , . . . , HI set vi = 0 and compute the prediction errors ETi. 2. Retrain the network with vh = 0 where ETh = mini ETi, and compute ET and EX of the retrained network.

+

3. If E T 5 (1 a)Emax and EX 5 (1+ a)Emax, then Remove hidden unit h. Set ETbest = min{ET, ETbest}, EXbest = min{EX, EXbest} and Emax = max{ ETbest, EXbest}. 0 Set H = H - 1 and go to Step 4.1. Else use the previous setting of network weights. 0

Step 5 . Remove irrelevant inputs: 1. For each j = 1 , 2 , . . . ,N , set wij = 0 for all i and compute the prediction errors E Tj . 2. Retrain the network with win = 0 for all i where ET, = minj E T j , and compute ET and EX of the retrained network.

222

R. Setiono and J . Zurada

+

3. If E T 5 (1 a)Emaz and EX 5 (1

+ a)Emaz, then

Remove input unit n. Set ETbest = min{ET, ETbest}, EXbest = min{EX, EXbest} and Emaa: = max{ETbest, EXbest}. Set N = N - 1 and go to Step 5.1. Else use the previous setting of network weights.

Step 6. Report the accuracy of the network on the test data set. The value of Emaa: is used t o determine if a network unit can be removed. Typically, at the beginning of the algorithm when there are many hidden units in the network, the training mean absolute error E T will be much smaller than the cross-validation mean absolute error EX. The value of E T increases as more and more units are removed. As the network approaches its optimal structure] we expect EX to decrease. As a result, if only ETbest is used to determine whether a unit can be removed, many redundant units can be expected to remain in the network when the algorithm terminates because ETbest tends to be small initially. On the other hand, if only EXbest is used, then the network would perform well on the cross-validation set but may not necessarily generalize well on the test set. This could be caused by the small number of samples available for cross-validation or an uneven distribution of the data in the training and cross-validation sets. Therefore, Emax is assigned the larger of ETbest and EXbest so as t o remove as many redundant units as possible without sacrificing the generalization accuracy. The parameter 0: > 0 is introduced to control the chances that a unit will be removed. With a larger value of a , more units can be removed. However, the accuracy of the resulting network on the test data set may deteriorate. We have conducted extensive experiments to find a value for this parameter that works well for the majority of our test problems.

3

Approximating Hidden Unit Activation Function

Having produced the pruned networks, we can now proceed to extract rules that explain the network outputs as a collection of linear functions. The first step in our rule extraction method is to approximate the hidden unit activation function h ( z ) = tanh(z) by a 3-piece linear function. It is sufficient to illustrate the approximation just for values of z 2 0. Suppose that the input z ranges from 0 to .z, A simple approximation of h ( z ) is to over-estimate it by the piecewise linear function L(x) as shown in

Knowledge Discovery from Continuous Data

223

X

0

xm

XO

Figure 1. The tanh(z) function (solid curve) for z E [0,]z, linear function (dashed lines).

is approximated by a piece-wise

Fig. 1. To ensure that L ( z ) is larger than h ( z ) everywhere between 0 to ,z, the line on the left should intersect the origin with a gradient of h‘(0) = 1, and the line on the right should intersect the coordinate (z, h ( z m ) )with a gradient of h’(z,) = 1 - h2(z,). Thus, L ( z ) can be written as

L ( z )=

{ h‘(z,)(z ”

- x,)

+ h(z,)

if 0 5 z 5 zo if 3: > 20

(5)

The point of intersection z o of the two line segments is given by

The total error EA of estimating h ( z ) by L ( x ) is given by E A = l r m ( L ( ~-)h ( z ) )dz =

51[ ~ g+

(2,

+ --12 -1n0.5

+ h ( z m ) )-] lncoshz,

- zO)(zO

as z ,

(7)

+ cx) .

That is, the total error is bounded by a constant value. Another simple linearization method of approximating h ( z ) is to underestimate it by a 3-piece linear function. It can be shown that the total error

224

R. Setiono and J . Zumda

of the under-estimation method is unbounded and is larger than that of the over-estimation method for 2, > 2.96.

4

Rule Generation

REFANN generates rules from a pruned neural network according to the following steps:

Algorithm REFANN Given: Data set (Ip,g p ) , p = 1 , 2 , .. . , K and a pruned network with H hidden units. Objective: Generate linear regression rules from the network. Step 1. For each hidden unit i = 1 , 2 , .. . , H : 1. Determine xirn from the training samples. 2 . Approximate the hidden unit activation function as follows: rn 0

Compute zio (Eqn. 6). Define the function Li(z):

Lz(x)=

{

+

(z zim)h’(zi,) - h(zi,) if z < -xi0 z if - xi0 5 z (z - zim)h’(zirn) h(zi,) if z > zio

+

5 zio

3 . Using the pair of points -xi0 and zio of function L i ( z ) ,divide the input space into 3H subregions.

Step 2. For each non-empty subregion, generate a rule as follows: 1. Define a linear equation that approximates the network’s output yp for input sample p in this subregion as the consequent of the extracted rule: H

y p = ~ v i L i ( s i p )where ,

(8)

i=l N

sip = c w i j I j p

(9)

j=1

2. Generate the rule condition: (C, and C2 and . . . C H ) , where Ci is either sip < -zio1 -xi0 5 sip 5 zio, or sip > zio.

Step 3. (Optional) Apply C4.5 [6] to simplify the rule conditions.

Knowledge Discovery from Continuous Data

225

Table 1. Attributes of the pollution data set.

Attribute PRE JAT JUT OVR65 POPN EDUC HOU DENS NOW WWDRK POOR HC NOX

so2

HUMID

description average annual precipitation in inches average January temperature in degrees F average July temperature in degrees F percentage of population aged 65 or older average household size median school years completed by those over 22 percentage of housing units which are sound and with all facilities population per square mile in urbanized areas in 1960 percentage of non-white population in urbanized areas in 1960 percentage of those employed in white collar occupations percentage of families with income less than $3000 relative hydrocarbon pollution potential relative nitric oxides pollution potential relative sulphur dioxide pollution potential annual average % relative humidity at lp m

In general, a rule condition Ci is defined in terms of the weighted sum of the inputs sip (Eqn. 9) which corresponds to an oblique hyperplane in the input space. This type of rule condition can be difficult for the users to interpret. In some cases, the oblique hyperplanes can be replaced by hyperplanes that are parallel to the axes without affecting the prediction accuracy of the rules on the data set. Consequently, the hyperplanes can be defined in terms of the isolated inputs, and are easier for the users to understand. In some cases of real-life data, this enhanced interpretability would come a t a possible cost of reduced accuracy. If the replacement of rule conditions is still desired, it can be achieved by employing a classification method such as C4.5 in the optional Step 3. 5

Illustrative Examples

The following examples of applying REFANN on two different data sets illustrate the algorithm in more details The input attributes of the first data set are continuous, while those of the second data set are mixed.

Example 1. Pollution data set

226

R. Setiono and J . Zurada

The data set has 15 continuous attributes as listed in Table 1. The goal is to predict the total age-adjusted mortality rate per 100,000 (MORT). The values of all 15 input attributes were linearly scaled t o the interval [0, 11,while the target MORT was scaled so that it ranged in the interval [0,4]. One of the networks that had been trained for this data set was selected to illustrate in details how the rules were extracted by REFANN. This network originally had eight hidden units, but only one hidden unit remained after pruning. The numbers of training, cross-validation and test samples were 48, 6, and 6, respectively. Many input units were also removed from the network. Only the connections from six inputs: PRE, JAT, JUT, HOU, NOW, and SO2 were still present after the network had been pruned. The weighted input value with the largest magnitude was taken as x, and the value of xo was computed according to Eqn. 6 to be 0.6393. Therefore, the hyperbolic tangent function was approximated by Ll(S1,)

=

{

-0.4155

+ 0.3501 sip

Slp

0.4155

+ 0.3501 sip

if slP < -0.6393 if - 0.6393 5 slP 5 0.6393

if slP > 0.6393

The three subsets of the input space were defined by the following inequalities: 0

0

0

Region 1: slP < -0.6393 H -0.76 PRE + 0.48 JAT 0.24 HOU - 0.98 NOW - 0.79 SO2 + 0.24 < -0.6393 Region 2: -0.6393 5 s l P 5 0.6393 0.24 HOU - 0.98 NOW - 0.79 SO2

+ 0.29

JUT

+

-0.76 PRE+0.48 JAT1-0.29 JUT+

+ 0.24 5 0.6393 Region 3: slP > 0.6393 H -0.76 PRE + 0.48 JAT + 0.29 0.24 HOU - 0.98 NOW 0.79 SO2 + 0.24 > 0.6393

JUT

+

-

It should be noted that the coefficients of the two parallel hyperplanes that divide the input space into the three regions are equal to the weights w l j from the j - t h input unit to the hidden unit. Upon multiplying the coefficients of L1 (sip) by the connection weight value from the hidden unit t o the output unit (Eqn. 8) and re-scaling the input and output data back into their original values, we obtain the following rules: Rule Set 1: Rule 1: if Region 1, then ij = Y1. Rule 2: if Region 2, then ij = Y,. Rule 3: if Region 3, then y = Y3.

Knowledge Discovery from Continuous Data

227

Table 2. The errors of a pruned network, two rule sets extracted from it, and multiple linear regression for the pollution data on the test set. The error rates are computed in terms of the Root Mean Squared Error (RMSE), the Relative Root Mean Squared Error (RRMSE Eqn. 17), the Mean Absolute Error (MAE), and the Relative Mean Absolute Error (RMAE - Eqn. 18).

Pruned network Rule set 1 Rule set l a Linear regression

RMSE 21.62 20.63 20.64 69.61

RRMSE 48.45 46.22 46.25 155.96

MAE 18.50 17.12 17.17 53.42

RMAE 49.35 45.67 45.81 142.51

The predicted value y is given by one of the following 3 equations: Yl = 1030.51 0.95 PRE - 0.59 JAT - 0.70 JUT - 0.55 HOU 1.39 NOW 0.16 SO2 Y2 = 1075.68 2.28 PRE - 1.43 JAT - 1.68 JUT - 1.31 HOU + 3.34 NOW 0.37 SO2 Y3 = 940.45 0.95 PRE - 0.59 JAT - 0.70 JUT - 0.55 HOU + 1.39 NOW 0.16 SO2 An optional final step of REFANN is to describe the 3 input subspaces by rule conditions generated by C4.5. All training samples p in Region 1 were given a target value of 1, in Region 2 a target value of 2, and in Region 3 a target value of 3. The following rules were generated by C4.5: Rule Set la:

+ + +

+

+ + +

Rule 1: if SO2 > 146, then ij = Yl. Rule 2: if NOW > 27.1, then ij = Yl. Rule 3: if PRE > 27.1 and NOW 5 27.1, then ij = Y,. Rule 4: if PRE 5 13, then y = Y,. Default Rule:

5 = Y3.

The errors of the pruned neural network and the rule sets are shown in Table 2. We combined the training and cross-validation sets and obtained the coefficients of the linear regression that fit the samples. Using the backward regression option of SAS [lo], none of the input attributes was found to be significant a t the default significance level of 0.10. The errors from linear regression on the test set are included in Table 2 for comparison. In addition to the mean absolute error (MAE) and the root mean squared error (RMSE),

228

R. Setiono and J . Zumda

we also show the relative root mean squared error (RRMSE) and the relative mean absolute error (RMAE): RRMSE = 100 x

d.C(Vp

-Y

C(& - yp)2

~)~/ P

v

where the summation is computed over the samples in the test set and is the average value of yp in the test set. These relative errors are sometimes preferred over the usual sum of squared errors or the mean absolute error because they normalize the differences in the output ranges of different data sets. An RRMSE or an RMAE that is greater than 100 indicates that the method performs worse than the method that simply predicts the output using the average value of the samples. The result highlights the effectiveness of the REFANN algorithm in splitting the input space into subspaces, where in each subspace a linear equation is generated. By applying different equations depending on the input, the accuracy of the predictions is improved. The error statistics also indicate that the prediction quality of the rules is very close to those of the pruned network.

Example 2. AutoMpg data set The target to be predicted in this problem is the city-cycle fuel consumption of different car models in miles per gallon. The 3 discrete attributes of the data are (1) cylinders with possible values of 3, 4, 5, 6, and 8; (2) model with possible values of 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, and 82; and (3) origin with possible values of 1, 2, and 3. The 4 continuous attributes are (1) displacement, (2) horsepower, (3) weight, and (4) acceleration. The training set contained 318 samples, while the cross validation and test sets contained 40 samples each. The unary-coded data required the neural network t o have 26 input units. One pruned network has 1 hidden and 7 input units left. The relevant network inputs are the following: (1) 14 = 0.2 iff cylinders a is greater than 3, (2) 19 = 0.2 iff model is later than 78, (3) 111 = 0.2 iff model is later than 76, (4) 1 1 4 = 0.2 iff model is later than 73, (5) 1 2 1 = 0.2 iff origin is 1, (6) 123 is horsepower, and (7) 1 2 4 is weight. A rule set consisting of just 2 rules was obtained: Rule Set 2: Rule 1: if Region 1, then jj = Yl. have used the 0/0.2 instead of 0/1 encoding scheme.

Knowledge Discovery from Continuous Data 229

Rule 2: if Region 2, then jj = Y 2 . The two subregions of the input space are defined as follows:

+

+ 0.67 1 1 1 + 0.52 1 1 4 -

+

+ 0.67 1 1 1 + 0.52 1 1 4

Region 1: slP < -0.8873 @ 0.41 1 4 1.07 19 0.41 1 2 1 - 0.003 1 2 3 - 0.0004 1 2 4 < -1.7198 0

Region 2: slP 2 -0.8873 H 0.41 1 4 1.07 19 0.41 1 2 1 - 0.003 1 2 3 - 0.0004 1 2 4 2 -1.7198

-

and the two corresponding linear equations are Y 1 = 16.26 0.54 1 4 1.40 1 9 0.88 1 1 1 0.69 1 1 4 - 0.53 1 2 1 - 0.0046 1 2 3 -

+

+

+

+

0.0005 1 2 4 Y 2 = 46.73 + 7.85 1 4 + 20.30 1 g + 12.72 1 1 1 + 9.96 1 1 4 - 7.73 1 2 1 - 0.0661 1 2 3 0.0079 1 2 4 We obtained the following rule set from C4.5 after executing the optional Step 3 of the algorithm REFANN: Rule Set 2a: Rule 1: if (19 = 0) and

(123

> 115) and

(124

> 3432), then 6 = Yl.

Rule 2: if

(111

= 0) and

(124

> 3574), then 6 = Y I .

Rule 3: if

(111

= 0) and

(123

> 130), then jj = Yl.

Rule 4: if

(123

5 98), then 6 = Y 2 .

Rule 5 : if

(123

5 130) and ( 1 2 4 5 3432), then 6 = Y 2 .

Rule 6: if

(111

= 0.2) and

(124

5 3432), then y = Y 2 .

Rule 7 : if

(111

= 0.2) and

(123

5 115), then y = Y 2 .

Rule 8: if

(19

= 0.2), then y = Yz.

Default rule: y = Yz. We compare the predictive accuracy of the extracted rules with the neural network and multiple linear regression in Table 3. The multiple linear regression model has 14 parameters that are significant at Q: = 0.10. Fitting the data with more input attributes, however, does not give a better model as shown by the RMSE and MAE of this model. By using a pruned neural network to divide the input space into two regions and having a linear equation in each of these regions for prediction, the RMSE and MAE are reduced by 18% and 27%, respectively.

230

R. Setiono and J . Zurada

Table 3. The errors of a pruned network, two rule sets extracted from it, and multiple linear regression for the autoMpg data on the test set.

Pruned network Rule set 2 Rule set 2a Linear regression

6

RMSE 2.91 3.00 3.00 3.65

RRMSE 35.31 36.33 36.36 44.25

MAE 2.03 2.07 2.07 2.85

RMAE 29.62 30.10 30.18 41.53

Conclusion

We have presented a method for discovering knowledge from data sets where the underlying problem of interest is that of predicting a continuous-valued target variable. The relationship between the input variables and the target variable for many practical data sets can be expected t o be nonlinear. The task of knowledge discovery from such a data set involves generating rules that are meaningful to humans and it is a challenging one. Our proposed method represents the nonlinear multivariable relationship as a set of rules where each of the rule consequent is a linear function. The conditions of the rules divide the input space of the data into smaller disjoint subspaces. The key components of the method are a feedforward backpropagation neural network and a simple linearization of the hidden unit activation function of the network. Neural networks are particularly suitable for learning nonlinear relationship among data variables. By approximating the nonlinear hidden unit activation function of the network as piece-wise linear functions, the network's outputs can also be expressed as linear functions of the input variables. In this paper, we have described the algorithm N2PFA for pruning neural networks that have been trained for regression and the algorithm REFANN for extracting linear regression rules from the pruned networks. The algorithm N2PFA produces pruned networks whose predictions are as accurate as those of other regression methods for many of the problems that we have tested [9]. The algorithm REFANN attempts t o provide an explanation for the network outputs by replacing the nonlinear mapping of a pruned network with a set of linear regression equations. Using the weights of a trained network, REFANN divides the input space of the data into a small number of subregions such that the prediction for the samples in the same subregion can be computed by a single linear equation. REFANN approximates the nonlinear hyperbolic tangent activation function of the hidden units using a simple 3-piece linear

Knowledge Discovery from Continuous Data 231

function. It then generates rules in the form of linear equations from the trained network. The conditions in these rules divide the input space into one or more subregions. For each subregion, a linear equation that approximates the network output is generated. We have conducted extensive experiments using a wide range of real world data sets and the results confirm the effectiveness of the algorithm. An extended version of this paper reports our findings in detail and it will be in print in a journal [ll]. References 1. Wang, Y. and Witten, I.H.(1997) Induction of model trees for predicting continuous classes. In Proc. of the Poster Papers of the European Conference on Machine Learning. Prague: University of Economics, Faculty of Informatics and Statistics, 128-137. 2. Quinlan, R. (1992) Learning with continuous classes. In Proc. of the Australian Joint Conference on Artificial Intelligence, Singapore, 343348. 3. Torgo, L. and Gama, J. (1997) Search-based class discretization. In Proc. of the 9th European Conference on Machine Learning, ECML-97, Lecture Notes in A1 1224, Springer, M. van Someren and G. Widmer (Eds), Prague, 266-273. 4. John, G.H., Kohavi, R.,and Pfleger, K. (1994) Irrelevant feature and the subset selection problem. In Proc. of the 11th International Conference on Machine Learning, Morgan Kaufmann Publisher, 121-129. 5. Ludl, M-C. and Widmer, G. (2000) Relative unsupervised discretization for regression problems. In Proc. of the 11th ECML, ECML 2000, Lecture Notes in A1 1810, Springer, R.A.Mantaras and E. Plaza (Eds), Barcelona, 246-253. 6. Quinlan, R. (1993) C4.5: Programs for Machine Learning. Morgan Kaufman, San Mateo, California. 7. Breiman, L., F'riedman, J.H., Olshen, R.A., and Stone, C.J. (1984) Classification and Regression Trees. Wadsworth International Group, Belmont , California. 8. Setiono, R. (1997) A penalty function approach for pruning feedforward neural networks. Neural Computation 9(1), 185-204. 9. Setiono, R. and Leow, W.K. (2000) Pruned neural networks for regression. In Proc. of the 6th Pacific Rim Conference on Artificial Intelligence, Lecture Notes in A1 1886, Springer, R. Mizoguchi and J. Slaney (Eds), Melbourne, 500-509. 10. SAS Technical Report A-102, SAS Regression Applications, SAS Institute

232

R. Setiono and J . Z u m d a

Inc. Cary, NC, USA. 11. Setiono, R., W.K. Leow, and Zurada, J. (2002) Extraction of rules from artificial neural networks for nonlinear regression. IEEE Transactions on Neural Network, forthcoming.

Advanced Applications with Machine Intelligence

This page intentionally left blank

REVIEW OF FUZZY LOGIC IN THE GEOLOGICAL SCIENCES: WHERE WE HAVE BEEN AND WHERE WE ARE GOING ROBERT V. DEMICCO Department of Geological Science And Centerfor Intelligent Systems Binghamton University Binghamton, New York 13902-6000 E-mail: [email protected] Geology is the application of physics, chemistry and biology to the study of the Earth. This report is a survey of approximately 70 recent papers in major geological science journals where fuzzy logic has been used to tackle Earth Science problems. These papers are briefly reviewed and grouped into nine categories: 1) geotechnical engineering; 2) surface hydrology; 3) subsurface hydrology; 4) hydrocarbon exploration; 5) ground-water risk assessment; 6) seismology; 7) soil science and landscape development; 8) deposition of sediments; and 9) a miscellaneous group. Papers cited in each category are intended to give an interested worker access to recent refereed papers in major geologically oriented journals that should be available in even modest university libraries. These papers should serve as an introduction into the literature of applications of fuzzy logic in the geological sciences by Earth Scientists. Keywords: fuzzy logic, geotechnical engineering, hydrology, hydrocarbon exploration. seismology, soil science.

1

INTRODUCTION

Many fields in the geological sciences are beginning to exploit the potential of machine intelligence. This report focuses 011 recent literature in geologically oriented journals on what is one of the most commonly employed “soft computing” techniques in the geological sciences: fuzzy logic. Since the mid 1980s, when fuzzy logic was initially developed for industrial control [11, many successful applications of fuzzy logic have been developed in other areas of engineering [2], as well as in decision making, business management, operations research and other professional areas [3], [4]. Although applications of fuzzy logic in the sciences are comparatively less developed, the utility of fuzzy logic has been demonstrated in chemistry [5], quantum physics [6], [7], [8], economics [9], ecology [lo], [l I], and geography [ 121. The trend in the Earth Sciences over the last ten years has been to view the Earth as a system and treat the hydrosphere, atmosphere, biosphere and lithosphere as interconnected subsystems. This approach is interdisciplinary and has been largely fueled by concern about the Earth’s present and past environments, with a growing realization that what happens in any one of the Earth’s “spheres” has impact on the others. Geology is the application of chemical, physical, and biological principles to the study of the lithosphere. Most problems in geology involve systems with large

235

236

R. V . Demicco

numbers of components and rich interactions among the components that are usually nonlinear and non-random. Such problems of organized complexity typify geologic systems and are exemplified by the geological systems that operate at the surface of the Earth. In response to the recognition that geology deals with the realm of organized complexity, there has been a recent explosive growth in the theory and application of fuzzy logic and other related “soft” computing techniques in the Earth Sciences. These techniques are now opening new ways of geologic modeling based on knowledge expressed in natural language. Geology, of all of the natural sciences, most readily lends itself to analysis and modeling by the emerging “soft computing” techniques being explored by this conference. In particular, fuzzy logic, in the broad sense, has at least a twenty-year history (and a growing body) of refereed works where it has been successfully applied to many areas of geological research. There are a number of reasons for this. First, geology is primarily a field science that began as an outgrowth of the mineral extraction industry. As such, the variables that geologists have routinely measured for hundreds of years are continua that commonly vary over many orders of magnitude. For example, the size of sedimentary particles ranges at least from mm through lo4 mm. Hydraulic conductivity, the constant k in Darcy’s Law

6h

Q=-k-* A , 61 varies from approximately lo-’’ to 10’ m/s2 for water and Earth materials’ [where Q is directional volume flow rate (m3/s), h is the hydraulic head (m) (a proxy for a fluid potential field made up of potential energy and pressure energy terms), I is a length (m) over which the potential change is measured, and A is the cross-sectional area of the fluid flow]. Currently, most of these naturally continuous variables are, more often than not, broken up into arbitrary “pigeon holes” by geologist seeking to “classify”. Second, because geological research is field based, it is commonly carried out over fairly broad regions of tens to hundreds of square kilometers in size. And, where subsurface data are added, the volume of a typical rock body being studied even on a modest scale, for minerals, gas, ground-water, or information about conditions on the ancient Earth, ranges up to1000 km3. Direct sampling of the entire rock or sediment body is clearly prohibitive, and so much of the threedimensional distribution of rock properties is measured over a relatively tiny percentage of the study area and inferred over the entire volume. The inference is commonly based on “expert knowledge” from a variety of sources to model the disposition of rock properties. 1

A closely related, curious unit that takes into account the viscosity and density of different fluids as well as the cross-sectional area of flow is known as a darcy. A darcy has the dimensions of [L2] and 1000 millidarcies = 1 darcy =: 10.’ m/s.

Review of Fuzzy Logic in the Geological Sciences 237

Third, remote sensing has always been used in geological data gathering. The

two most common remote sensing techniques are exploration seismic surveys and

geophysical measurements of boreholes. In the former, the differential impedance of rocks to artificial vibrational energy introduced into the subsurface produces what amounts to an “ultra sound” of subsurface rock disposition. In borehole geophysical surveys, the gamma ray output of rocks and the electrical resistivity of rocks to an induced potential are measured as proxies for rock type, permeability, etc. These techniques require correspondencesbetween the proxy measures and the desired rock properties to be established. These correspondences are commonly not one-to-one. In addition to these subsurface methods, there is growing reliance on Geographical Informational Systems (GIS) and satellite remote sensing in a number of geological fields interested in the land’s surface and it’s development. In spite of the above considerations, which highlight the imprecise nature of much geological information, as of this writing, most geophysicists and many geologists are wedded to the application of Newtonian mechanics, and particularly, partial differential equations, to modeling the Earth. The purpose of this paper is to briefly review the range of uses of fuzzy logic in the geological sciences literature to allow an interested worker access to the literature. The sources are major geological sciencesjournals that will be available even in a modest research library. 2

REVIEW OF AREAS OF GEOLOGY WHERE FUZZY LOGIC HAS BEEN EMPLOYED: (Where We Have Been)

The following synopsis divides the current literature of fuzzy logic into eight specific categories and one miscellaneous category. The papers cited in each category do not comprise a complete bibliography of materials. Instead, they represent recent refereed papers in major geological journals that will serve as an introduction into the literature. The categories chosen are briefly described here. It should come as no surprise that geotechnical engineering and areas closely allied to engineering, surface hydrology, subsurface hydrology, and hydrocarbon exploration, have seen early and extensive use of fuzzy logic. There have been a number of management-oriented models for ground-water risk assessment based on fuzzy logic. These models incorporate surface and subsurface hydrologic data. Exploration geophysicists in hydrocarbon exploration have adapted fuzzy logic into seismic processing and evaluation. Earthquake seismology is a discipline still very closely wedded to Newtonian mechanics. However, a growing number of geophysicists have recently adapted fuzzy logic and there are scattered uses mentioned below. Soil science and landscape development have seen extensive use of fuzzy logic and models of modern and ancient deposition of sediments have also seen modest use of fuzzy logic to simulate sediment production, sediment erosion, sediment transportation and sediment deposition. Finally, there is scattered miscellaneous literature in the “core” geological sciences outlining potential uses of fuzzy logic in specific areas not included in the categories outlined here.

238

3

R. V. Demicco

GEOTECHNICAL ENGINEERING

There are a number of models for river discharge control and flood management using fuzzy control systems. These include both simple, one dam systems [ 131 and an optimization model for simultaneous flood gate control of the Yangtze River involving the Three Gorges Dam, flood basin areas along the river, and eight flood control dams on tributaries [ 141. Optimized watershed management plans have also been developed for the Lake Erhai basin in southern China using a fuzzy multiobjective management program [15]. The Lake Erhai watershed is under intense developmental pressure from a variety of competing land uses (agricultural, scenic, light industry, etc). A number of applications of fuzzy logic have been developed for excavation and mining operations. A fuzzy expert system developed to evaluate the failure potential of road-cut slopes and embankments was applied along a highway in a landslide prone area of Jordan [ 161. Rock trencher performance was modeled with fuzzy logic [ 171 and fuzzy clustering algorithms have been developed to identify fractures sets encountered in exploration dnlling [181. 4

SURFACE HYDROLOGY

The discharge of a stream is the volume flow per unit time through a cross section of the stream at a point along the stream’s course. Obviously, reliable prediction of discharge, especially low flows during droughts and high flows that lead to flooding is important for a variety of reasons. The discharge of a river represents a complicated, highly non-linear response of a watershed area to precipitation. Complicating factors include: whether the soils are frozen or thawed, and, if thawed, their moisture content; the topography of the watershed; the plant cover of the watershed, its growth cycle, and whether the leaves are wet or dry; the intensity, duration and location of precipitation; etc. [ 191. A number of deterministic models have been developed to try to predict discharge from precipitation records or forecasts, with real-time predictions as the goal. These models vary in sophistication, but, as most were developed for a specific geographic area, they are difficult to apply globally and are highly parameterized. One of the first uses of fuzzy logic was to help refine the parameters input to these models. One well-known, deterministic model employed to study a small watershed in Brittany, France, was augmented by incorporation of fuzzy sets [20]. In this area, fuzzy slope measurements were combined with a fuzzy image analysis of the terrain (as measured by synthetic aperture radar) to evaluate the saturation state of the catchment area. This information, in turn, fed into the standard deterministic model. Another pre-existing deterministic model of rainfalVrunoff for a small catchment in Taiwan [21] was modified to use a fuzzy multi-objective function to calibrate the model parameters. Deterministic watershed response models on experimental, well instrumented and well-studied watersheds in

Review of f i z z y Logic i n the Geological Sciences

239

the western U.S. have also incorporated fuzzy logic based inferences from soils maps to provide parameters into the model [22]. Finally, the results of 5 deterministic watershed response models applied to 11 catchment areas in Taiwan were combined into a single response model using an affine Takagi-Sugeno model ~31. There are also a number of recent watershed response models that are entirely based on fuzzy logic. An adaptive neural fuzzy inference system (ANFIS) analysis of long-term data from a watershed in Tuscany, Italy, extracted fuzzy rules that were used in the development of a rainfalVresponse model [24]. Back propagation fuzzy-neural networks have been applied on long-term data from a watershed in Taiwan for the same purpose [25]. Somewhat closely related to these watershed response models are studies that have attempted to model and predict precipitation input onto watersheds. Pongracz and colleagues [26] tried to develop a drought prediction model for the Great Plains of the U.S. They developed a set of fuzzy rules that related two inputs: 1) the Southern Ocean Index (SOI) as a proxy for El Nino Southern Oscillation (ENSO); and 2) the geopotential height field of the SOOhPa level over a large area of the western hemisphere; to a long term record of droughts in 8 regions of Nebraska. This approach was extended to the general stochastic prediction of a time series for precipitation over Europe [27]. In the general European model, a fuzzy classification of point measurements of geopotential atmospheric pressure surfaces over a large-scale grid of Europe served as an input into a more conventional stochastic model.

5

Subsurface Hydrology

Ground-water flow has been most commonly modeled by the piece-wise solution of the diffusion equation either by finite difference approximations [28] or finite elements [29]. Solute transport in ground-water systems has likewise been modeled by finite difference or finite element approximations of the advective-dispersive equation [30]. These types of models are built around the empirical Darcy’s law (equation 1) and are very commonly applied to problems of ground-water well-field development or, where pollutants are dissolved in the ground-water, remediation plans. There is a growing recognition that such models (although quite common) may be inadequate [31] due to the inherent impression of knowledge of the threedimensional distribution of hydraulic conductivity. More fundamentally, the general fuzzy nature of the variables hydraulic conductivity, hydraulic head and storativity (specific yield) themselves have been incorporated in both steady-state [32] and transient [33] ground-water flow models. These models used fuzzy numbers in the differential equations and a fuzzy-technique for solution of the equations. A similar model for flow in the unsaturated zone (the surface-most soil zone where hydraulic conductivities vary with state of pore saturation) has also been developed [34]. Recently, fuzzy rule-based models for solute transport in the

240

R. V. Demicco

unsaturated zone have developed [35]. This change from using fuzzy numbers to represent imprecise variable in traditional finite difference solutions to models entirely based on fuzzy rules has mirrored a similar development in surface flow models. Another type of flow modeling seeks to understand ground-water flow from first principles instead of the empirically derived Darcy flow equation. In one such model, the degree of interconnectiveness among pores in a porous medium was modeled with fuzzy sets [36]. 6

Ground Water Risk Assessment

Assessing the risk of contaminant pollution of ground-water from various anthropogenic sources is an important element of municipal and agricultural wellfield planning. Aquifer vulnerability to contamination depends on soil properties, precipitation, topography, etc., all of which can be modeled with fuzzy sets. Models have been developed to assess non-point source pesticide pollution of aquifers [37] and to assess the potential of industrial ground-water contamination [38], [39]. 7

Hydrocarbon Exploration

Not surprisingly, this area has seen the most development of all so-called “soft computing” technologies including neural networks, fuzzy logic, genetic algorithms, and data analysis. There have been a number of recent journal issues solely dedicated to this area and they are an excellent place to gain access to the literature. Computers and Geosciences (volume 26, number 8, October 2000) produced an issue entitled “Applications of Virtual Intelligence to Petroleum Engineering”. This issue [40] was edited by Shahab Mohagheg and contained nine technical papers in addition to an introductory note. Most of the papers are dedicated to neural networks many of which rely heavily on fuzzy logic. In addition to the special issue cited above, numerous single papers on applications of soft computing to the Earth Sciences are found in Computers and Geosciences. The Journal of Petroleum Geology (volume 24, number 4, October 2001) published a thematic issue entitled: “Field applications of intelligent computing techniques”. This issue was edited by Wong and Nikravesh [41] and included five technical papers and an introduction by the editors. Most of the papers were applications using neural networks. However, there was specific application of fuzzy logic to the biostratigraphic interpretation of mudstones in a North Sea oil field [42] and specific application of fuzzy partitioning to the classification and interpretation of remotely sensed resistivity and spontaneous potential wire-line well logs [43].

Review of Fuzzy Logic in the Geological Sciences

241

The Journal of Petroleum Science and Engineering produced two special issues dedicated to “Soft computing and Earth Sciences” edited by Nikravesh, Aminzadeh and Zadeh. Part one [44] (volume 29, numbers 3-4, May 2001) contained seven technical papers plus an introduction by the editors. Part two [45] (volume 31, numbers 2-4, January 2001) carried nine papers. These two special issues have a bit more variety of applications of soft computing to the Earth Sciences but are still mostly dealing with applications of neural networks to data analysis. An extended version of these two theme issues is currently in press. 8

Earthquake Seismology

Deyi and Xihui [46] presented the result of an international symposium of fuzzy mathematics in earthquake research. Since the publication of this volume, application of fuzzy logic to seismology has focused on earthquake prediction, including: assessment of the magnitude of an earthquake (absolute amount of energy a seismic event puts into the ground) [47], [48], and the pattern of surface disruption. The surface effects of passage of seismic waves can vary dramatically in an urban area depending on the material properties of the area, the type and depth to bedrock in the area, and the amplification effects of earthquake waves by the shape of the resonating deposits. Prediction of the ground motion at a site depends not only on these properties, but also on the properties of the incident waves, including their orientation. Fuzzy based neural-network approaches have developed to integrate the properties of the waves and the properties of the ground to predict ground motion [49], [50], [51]. Locating the source site of earthquake waves arriving at a distant site has been one of the staples of seismology since it’s beginning. A number of deterministic models have developed to make these calculations. Recently, Lin and Sanford [52] described an inversion technique wherein deviations between theoretical and observed arrival times are assessed with fuzzy logic. 9

Soil science and landscape development

Maps that depict the distribution of different soil types are a standard product of geological and agricultural surveys throughout the world and have a variety of critical uses. Such uses include agricultural planning, conservation, input into watershed models, input into ground-water models, and the legal definition of wetlands among others. Soils are three-dimensional mantles of weathered Earth materials with a complex biogeochemical evolutionary history depending on parent material, precipitation, temperature, topography, ground-water levels, etc. [53]. Traditional soil classifications and maps produced from them ignored intergradations of soil types, both horizontally and vertically, and were based on widely spaced sampling pits. Although the actual microscopic structure of soils is

242

R. V . Demicco

difficult to assess, a two-dimensional, fuzzy model of soil element disposition has been developed [54]. In recent years soil science has been revolutionized by the development of Geographical Information Systems (GIs), improvements of remote sensing capabilities (to patches 10 meters or so on a side), and a fuzzy approach toward soil classifications and mapping [55], [56], and [57]. Examples of this approach have been applied to different geographical areas including: an alluvial flood plain in western Greece [58]; an area in New South Wales, eastern Australia [59]; and areas in Wisconsin and Montana [60]. In each of these studies, horizontal and vertical measurements of soil properties in test pits are employed to devise fuzzy soil classification systems, i.e. systems wherein a point can belong to more than one soil type. In this way, intergradations among soil types are naturally handled and small areas of slightly different soil types within larger areas can be identified. These soil types are then mapped from remotely sensed images of an area at a resolution of blocks that are approximately a few tens of meters on a side. Remote sensing commonly measures intensity of various wavelengths of radiation reflected off the Earth’s surface or produced at the surface. A transfer function (commonly fuzzy) is then developed to related wavelength and intensity of radiation to a soil type. These techniques have also been used to interpret glacial features from the eastern Italian Alps [61] and Greece [62]. Finally, fuzzy logic has been employed to understand geochemical aspects of soils and recent stream sediments. Fuzzy c-means clustering of elements measured in stream samples from the Alps of Austria established 4 categories of background levels of various elements [63]. These categories were better able to screen out non-anomalous concentrations of metals. Similarly, fuzzy clustering established relationships among glacial tills, bedrock and elements measured in soils of Finland [64]. 10 Deposition of sediments

Modern sedimentary systems comprise volumes of the uppermost tens of meters of lithosphere, and overlying hydrosphere and atmosphere where sediments accumulate as, for example, on the delta of the Mississippi River or on the floor Death Valley, California. Earth scientists have limited knowledge of the physical, chemical and biological processes that control sediment accumulation in modem sedimentary systems. This presents a difficulty insofar as analogous ancient sedimentary systems are where the sediments and sedimentary rocks, which hold much of the direct, long-term evidence of the history of the biosphere, atmosphere, hydrosphere and lithosphere of this planet, accumulated. In order to infer such important information as long and short term variations in geochemical cycles, ecosystems, and climate, we seek to recover from ancient sedimentary deposits just these records of the physical, chemical and biological processes that operated on those ancient surfaces. Our poor understanding of modem depositional processes is

Review of Fuzzy Logic in the Geological Sciences

243

due to: the large sizes of modern sedimentary systems (102-105km2);difficulties in instrumentation (especially during rare events such as hurricanes and floods); restrictions of observations on active processes to a few hundred years; and laborintensive data gathering. The thorniest problem faced by molders is simulating sediment erosion, sediment transportation, and sediment accumulation within a model. In coastal and shallow marine systems, waves, wave-induced currents, tidal currents and storminduced (i.e. ‘event’) waves and currents lead to ever-changing patterns of sediment erosion, transportation, and accumulation. Modeling such events entails handling physical laws and empirically derived relationships. These physical laws and empirical relationships are generally described by nonlinear, complex sets of partial differential equations [65], [66], [67]. Moreover, these equations must be coupled during solution. Furthermore, some parameters that cannot be easily formalized, such as antecedent topography and changing boundary conditions, and incorporation of “rare” events need to be taken into account. When we consider carbonate depositional systems, we are also confronted by the in situ formation of the sediments themselves both as reefs and bank-interior sediments. Coastal oceanographic modelers have made great strides in dealing with the complexities of coupled solutions as well as wave dynamics, current dynamics and sediment transport. However, finite difference and finite element numerical simulations have two drawbacks when applied to stratigraphic models. First, they are site specific and depend on rigorous application of boundary conditions, initial conditions, and wave and tidal forcing functions over a discrete domain. Secondly, these processresponse models operate at tens to hundreds of year time scales, which are very short in comparison to basin-filling models. As a result, the effects of large, complex storm events, which are suspected of being important agents in ancient depositional systems, are only rarely included in coastal models. Indeed, such complexities lead to questioning the applicability of even short-term coastal models built around dynamic sedimentary process simulators [68]. Sedimentary models that wed GIS-based observations of processes and sedimentary product through neural networks [69] offer the potential of a new generation of models to assess sedimentation in harbors and coastal areas. As a result of the difficulties outlined above, much of what we know about the evolution of sedimentary systems on geological time scales (lo5 - lo8 y) is increasingly based on quantitative models. Traditionally, these models have relied on the piece-wise solution of partial differential equations for fluid flow and sediment transport over grids of points using highly parameterized initial, internal, and boundary conditions. The disadvantages of such models are that they are quite computationally intensive and require a large degree of user judgment in the coefficients and boundary conditions employed. Fuzzy logic was initially introduced into stratigraphic models to overcome the computational difficulties of sedimentary process modeling [70], [71]. FUZZIM [71] is a share-ware program that replaces sedimentary physics with common-sense rules based on hard and soft

244

R. V. Demicco

information developed by sedimentologists over the past 100 years. Other modelers have incorporated fuzzy logic into long-term stratigraphic models of various sedimentary environments [72], [73], [74]. There are very few models wherein fuzzy logic has been utilized to describe the diagenetic processes that transform deposited sediment into rock. However, fuzzy clustering was used to develop a classification of geochemical and rock magnetic effects in deep-sea sediments due to flux of hydrothermal fluids [75]. 11 Miscellaneous

There have been a number of general papers advocating the use of fuzzy logic in the geological sciences [76], [77]. Bardossy and Duckstein, [78] authored a useful introductory text where various applications of fuzzy rule-based modeling in biological and engineering applications as well as geosciences are developed. In addition to these general references there have been the following specific applications of fuzzy logic to geological problems outside of the main areas listed above. These include the study of volcanoes [79] and a fuzzy rule-based model to simulate latent heat fluxes of coniferous forests [80]. Fuzzy set theory was also applied to thermodynamic parameters in aqueous chemical equilibrium calculations [8 11. Finally, fuzzy clustering of paleomagnetic measurements on deep-sea sediments produced populations that were a proxy €or estimating orbital forcing of climate over the last 276,000 years [82].

12 Conclusions The papers cited in each category above by no means comprise a complete bibliography of materials of use of fuzzy logic in the geological sciences. Instead, they are intended to give a potential worker access to recent refereed papers in major journals that should be available in even modest university libraries. These papers and their bibliographies should serve as an introduction into the literature. One final observation is in order. There seems to be a major trend in the use of fuzzy logic in the geological sciences. Initially, fuzzy sets were used to capture the continuous nature of geologic data and various techniques were developed to use fuzzy sets in previously developed deterministic models of geological phenomena. More and more, fuzzy rule-based models are beginning to supercede the older deterministic models. This trend will no doubt continue into the future. Acknowledgements The United States National Science Foundation Grant EAR9909336 to R.V.D. and G. Klir supported this research.

Review of Fuzzy Logic in the Geological Sciences

245

References 1. Sugeno, M., 1985, Industrial applications of fuzzy control. Elsevier, Amsterdam, New York, 269 p. 2. Ross, T. J., 1995, Fuzzy Logic with Engineering Applications. McGraw-Hill, New York. 3. Ruspini, E. H., Piero, P., Bonissone, P., Pedrycz, W., 1998, Handbook of Fuzzy Computation. Institute of Physics, Philadelphia. 4. Dubois, D., and Prade, H., (eds.), 1999-2000, The Handbooks of Fuzzy Sets Series. Kluwer, Boston. 5. Rouvray, D. H., 1997, Fuzzy Logic in Chemistry. Academic Press, San Diego. 6. Hsu, J. P., 1991, Theory of fuzzy transitions between quantum and classical mechanics and proposed experimental tests. Physical Review A. v. 43, p. 3227-323 1. 7. Pykacz, J., 1993, Fuzzy quantum logic I. International Journal of Theoretical Physics, v. 32, p. 1691. 8. Cattaneo, G., 1993, Fuzzy quantum logic 11. The logics of unsharp quantum mechanics. International Journal of Theoretical Physics, v. 32, p. 1709. 9. Billot, A., 1992, Economic Theory of Fuzzy Equilibria: An Axiomatic Analysis. Springer-Verlag, New York, 180 p. 10. Salski, A., 1992, Fuzzy knowledge-based models in ecological research. Ecological Modelling, v. 63, p.103-112. 1 1, Libelli, S. M., and Cianchi, P., 1996, Fuzzy Modelling: Paradigms and Practice. Kluwer, Boston. 12. Gale, S., 1972, Inexactness, fuzzy sets, and the foundation of behavioral geography. Geographical Analysis,v. 4, p. 5-1 6. 13. Teegavarapu. R. S. V., and Simonovic, S. P., 1999, Modeling uncertainty in reservoir loss functions using fuzzy sets. Water Resources Research, v., 35, p. 2815-2823. 14. Cheng, C., 1999, Fuzzy optimal model for the flood control system of the upper and middle reaches of the Yangtze River. Hydrological Sciences - Journal, v. 44, p. 573-582. 15. Huang, G. H., Liu, L., Chakma, A., Wu, S. M., Wang, X. H., and Yin, Y. Y., 1999, A hybrid GIs-supported watershed modeling system: application to the Lake Erhai basin, China. Hydrological Sciences - Journal, v. 44, p. 597-610. 16. Al-Homoud, A. S., and Al-Masri, G. A., 1999, CSEES: an expert system for analysis and design of cut slopes and embankments. Environmental Geology, V.39, p. 75-89. 17. Grima, A. M., and Verhorf, P. N. W., 1999, Forecasting rock trencher performance using fuzzy logic. International Journal of Rock Mechanics and Mining Sciences & Geomechanics Abstracts, v. 36, p. 413-432.

246

R. V. Demicco

18. Hammah, R. E., and Curran, J. H., 1998, Fuzzy cluster algorithm for the automatic identification of joint sets. International Journal of Rock Mechanics and Mining Sciences & Geomechanics Abstracts, v. 35, p. 889-905. 19. Dunne, T., and Leopold, L. B., 1978, Water in Environmental Planning. W. H. Freeman, San Francisco, 8 18 p. 20. Franks, S. W., Gineste, P., Beven, K. J., and Merot, P., 1998, On constraining the predictions of a distributed model: the incorporation of fuzzy estimates of saturated areas into the calibration process. Water Resources Research, v. 34, p. 787-797. 21. Yu, P. -S., and Yang, T. -C., 2000, Fuzzy multi-objective function for rainfallrunoff model calibration. Journal of Hydrology, v. 238, p. 1-14. 22. Zhu, A. X., and Mackay, D. S., 2001, Effects of spatial detail of soil information on watershed modeling. Journal of Hydrology, v. 248, p. 54-77. 23. Xiong, L., Shamseldin, A. Y., and O’Connor, K. M., 2001, A non-linear combination of the forecasts of rainfall-runoff models by the first-order TakagiSugeno fuzzy system. Journal of Hydrology, v. 245, p. 196-217. 24. Gautam, D. K., and Holz, K. P., 2001, Rainfall-runoff modeling using adaptive neuro-fuzzy systems. Journal of Hydroinformatics, v. 3, p. 3-10. 25. Chang, F. -J., and Chen, Y. -C., 2001, A counterpropagation fuzzy-neural network modeling approach to real time streamflow prediction. Journal of Hydrology, v. 245, p. 153-164. 26. Pongracz, R., Bogardi, I., and Duckstein, L., 1999, Application of fuzzy rulebased modeling technique to regional drought. Journal of Hydrology, v. 224, p. 100-1 14. 27. Stehlik, J., and Birdossy, A., 2002, Multivariate stochastic downscaling model for generating daily precipitation series based on atmospheric circulation. Journal of Hydrology, v. 256, p. 120-141. 28. McDonald, M. G., and Harbaugh, A. W., 1988, Chapter A l , A modular threedimensional finite-difference ground-water flow model. Techniques of WaterResources Investigations of the United States Geological Survey, Book 6. 29. Istok, J., 1989, Groundwater Modeling by the Finite Element Method. Water Resources Monograph 13, American Geophysical Society, 495 p. 30. Freeze, R. A., and Cherry, J. A., 1979, Groundwater. Prentice-Hall, Inc., Englewood Cliffs, New Jersey, 604 p. 31. Konikow, L. F., and Ewing, R. C., 1999, Is a probabilistic performance assessment enough? Ground Water, v. 37, p. 48 1. 32. Dou, C., Woldt, W., Bogardi, I., Dahab, M., 1995, Steady state groundwater flow simulation with imprecise parameters. Water Resources Research, v. 3 1, p. 2709-2719. 33. Dou, C., Woldt, W., Dahab, M., and Bogardi, I., 1997, Transient ground-water flow simulation using a fuzzy set approach. Ground Water, v. 35, p. 205-215.

Review of Fuzzy Logic an the Geological Sciences

247

34. Schulz, K., and Huwe, B., 1997, Water flow modeling in the unsaturated zone with imprecise parameters using a fuzzy approach. Journal of Hydrology, v. 201, p. 21 1-229. 35. Dou, C., Woldt, W., and Bogardi, I., 1999, Fuzzy rule-based approach to describe solute transport in the unsaturated zone. Journal of Hydrology, v. 220, p. 74-85. 36. Zeng, X., Vasseur, C., and Fayala, F., 2000, Modeling microgeometric structures of porous media with a predominant axis for predicting diffusive flow in capillaries. Applied Mathematical Modelling, v. 24, p. 969-986. 37. Freissinet, C., Vauclin, M., and Erlich, M., 1999, Comparison of first-order analysis and fuzzy set approach for the evaluation of imprecision in a pesticide groundwater pollution screening model. Journal of Contaminant Transport, v. 37, p. 21-43. 38. Zhou, H., Wang, G., and Yang, Q., 1999, A multi-objective fuzzy pattern recognition model for assessing groundwater vulnerability based on the DRASTIC system. Hydrological Sciences - Journal, v. 44, p. 61 1-618. 39. Ozdamar, L., Demirhan, M., Ozpinar, A., and Kilanc, B., 2000, A fuzzy areal assessment approach for potentially contaminated sites. Computers & Geosciences, v. 26, p. 309-3 18. 40. Mohagheg, S., (ed.), 2000, Applications of Virtual Intelligence to Petroleum Engineering. Computers & Geosciences, v. 26. 41. Wong, P. M., and Nikravesh, M., (eds.), 2001, Field Applications of Intelligent Computing Techniques. Journal of Petroleum Geology, v. 24, n. 4. 42. Wakefield, M. I., Cook, R. J., Jackson, H., and Thompson, P., 2001, Interpreting biostratigraphical data using fuzzy logic; the identification of regional mudstones within the Fleming Field, UK North Sea. Journal of Petroleum Geology, v. 24, p. 417-440. 43. Finol, J. J., Guo, Y. K., and Jing, X. D., 2001, Fuzzy partitioning systems for electrofacies classification; a case study for the Maricaibo Basin. Journal of Petroleum Geology, v. 24, p.441-458. 44. Nikravesh, M., Aminzadeh, F., and Zadeh, L. A., (eds)., 2001, Soft computing and Earth Sciences: Part 1. Journal of Petroleum Science and Engineering, v. 29, n. 3-4. 45. Nikravesh, M., Aminzadeh, F., and Zadeh, L. A,, (eds)., 2001, Soft computing and Earth Sciences: Part 2. Journal of Petroleum Science and Engineering, v. 31, n. 2-4. 46. Deyi, F., and Xihui, L., (eds.), 1985, Fuzzy mathematics in earthquake researches, Proceedings of International Symposium on Fuzzy Mathematics in Earthquake Researches, Seismological Press, Beijing, 646 p. 47. Wang, W., Wu, G., Huang, B., Zhuang, K., Zhou, P., Jiang, C., Li, D., and Zhou, Y., 1997, The FAM (fuzzy associative memory) neural network model and its application in earthquake prediction. Acta Seismologica Sinica, v. 10, p. 321-328.

248 R. V. Demicco

48. Wang, X. -Q., Zheng, Z., Qian, J., Yu, H. -Y., and Huang, X. -L., 1999, Research on the fuzzy relationship between the precursory anomalous elements and earthquake elements. Acta Seismologica Sinica, v. 12, p. 676-683. 49. Muller, S., Legrand, J. -F., Muller, J. -D., Cansi, Y., and Crusem, R., 1998, Seismic events discrimination by neuro-fizzy-based data merging. Geophysical Research Letters, v. 25, p. 3449-3452. 50. Muller, S., Garda, P., Muller, J. -D., and Cansi, Y., 1999, Seismic events discrimination by neuro-fuzzy merging of signal and catalogue features. Physics and Chemistry of the Earth (A), v. 24, p. 201-206. 51. Huang, C., and Leung, Y., 1999, Estimating the relationship between isoseismal area and earthquake magnitude by a hybrid fuzzy-neural-network method. Fuzzy Sets and Systems, v. 107, p. 131-146. 52. Lin, K., and Sanford, A. R., 2001, Improving regional earthquake locations using a modified G matrix and fuzzy logic. Bulletin of the Seismological Society of America, v. 91, p. 82-93. 53. Birkeland, P. W., 1974, Pedology, Weathering, and Geomorphological Research. Oxford University Press, New York, 285 p. 54. Moran, C. J., and McBratney, A. B., 1997, A two-dimensional fuzzy random model of soil pore structure. Mathematical Geology, v. 29, p. 755-777. 55. Galbraith, J. M., Bryant, R. B., and Ahrens, R. J., 1998, An expert system of soil taxonomy. Soil Science, v. 163, p. 748. 56. Galbraith, J. M., and Bryant, R. B., 1998, A functional analysis of soil tzxonomy in relation to expert system techniques. Soil Science, v. 163, p. 739747. 57. Wilson, J. P., and Burrough, P. A., 1999, Dynamic modeling, geostatistics, and fuzzy classification: new sneakers for a new geography? Annals of the Association of American Geographers, v. 89, p. 736-746. 58. Kollias, V. J., Kalivas, D. P., and Yassoglou, N. J., 1999, Mapping the soil resources of a recent alluvial plain in Greece using fuzzy sets in a CIS environment. European Journal of Soil Science, v. 50, p. 261-273. 59. Triantafilis, J., Ward, W. T., Odeh, I. 0. A., and McBratney, A. B., 2001, Creation and interpolation of continuous soil layer classes in the Lower Naomi Valley. Soil Science Society of America Journal, v. 65, p. 403-413. 60. Zhu, A. X., Hudson, B., Burt, J., Lubich, K., and Simonson, D., 2001, Soil mapping using GIS, expert knowledge, and fuzzy logic. Soil Science Society of America Journal, v. 65, p. 1463-1472. 61. Serandrei-Barbero, R., Rabagliati, R., Binaghi, E., and Rampini, A., 1999, Glacial retreat in the 1980s in the Breonie, Aurine and Pusteresi groups (eastern Alps, Italy) in Landsat TM images. Hydrological Sciences -Journal, v. 44, p. 279-296. 62. Smith, G. R., Woodward, J. C., Heywood, D. I., and Gibbard, P. L., 2000, Interpreting Pleistocene glacial features from SPOT HRV data using fuzzy techniques. Computers & Geosciences, v. 26, p. 479-490.

Review of Fuzzy Logic in the Geological Sciences

249

63. Rantitsch, G., Application of fuzzy clusters to quantify lithological background concentrations in stream-sediment geochemistry. Journal of Geochemical Exploration, v. 71, p. 73-82. 64. Lahdenpera, A. -M., Tamminen, P., and Tarvainen, T., 2001, Relationships between geochemistry of basal till and chemistry of surface soil at forested sites in Finland. Applied Geochemistry, v. 16, p. 123-136. 65. Li, M. Z., and Amos, C. L., 1995; SEDTRANS92: a sediment transport model of continental shelves. Computers & Geosciences, v. 21, p. 553-554. 66. Acinas, J. R., and Brebbia, C. A., (eds.), 1997, Computer Modelling of Seas and Coastal Regions. Computational Mechanics, Inc., Billerica, MA, 442 p. 67. Harff, J., Lemke, W., and Stattegger, K., (eds.), 1999, Computerized Modeling of Sedimentary Systems. Springer, New York, 452 p. 68. Pilkey, 0. H., and Thieler, E. R., 1996, Mathematical modeling in coastal geology. Geotimes, v. 41, p. 5. 69. Yang, Y., and Rosenbaum, M. S., 2001, Artificial neural networks linked to GIS for determining sedimentology in harbours. Journal of Petroleum Science and Engineering, v. 29, p. 213-220. 70. Nordlund, U., 1996, Formalizing geological knowledge - with and example of modeling stratigraphy using fuzzy logic: Journal of Sedimentary Research, v. 66, p. 689-712. 71. Nordlund, U., 1999, FUZZIM: forward stratigraphic modeling made simple. Computers & Geosciences, v. 25, p. 449-456. 72. Edington, D. H., Poeter, E. P., and Cross, T. A., 1998, FLUVSIM; a fuzzylogic forward model of fluvial systems. Abstracts with Programs - Geological Society of America Annual Meeting, v. 30, p. A105. 73. Parcell, W. C., Mancini, E. A,, Benson, D. J., Chen, H., and Yang, W., 1998, Geological and computer modeling of 2-D and 3-D carbonate lithofacies trends in the Upper Jurassic (Oxfordian), Smackover Formation, Northeastern Gulf Coast. Abstracts with Programs - Geological Society of America Annual Meeting, v. 30, p. A338. 74. Demicco, R. V., and Klir, G. J., 2001, Stratigraphic simulations using fuzzy logic to model sediment dispersal. Journal of Petroleum Science and Engineering, v. 31, p. 135-155. 75. Urbat, M., Dekkers, M. J., and Krumsiek, K., 2000, Discharge of hydrothermal fluids through sediment at the Escanaba Trough, Gorda Ridge (ODP Leg 169): assessing the effects on the rock magnetic signal. Earth and Planetary Science Letters, v. 176, p. 481-494. 76. Fang, J. H., 1997, Fuzzy logic and geology. Geotimes, v. 42, p. 23-26. 77. Fang, J. H., and Chen, H. C., 1990, Uncertainties are better handled by fuzzy arithmetic. American Association of Petroleum Geologists Bulletin, v. 74, p. 1228- 1233.

250

R. V . Demicco

78. Bardossy, A., and Duckstein, L., 1995, Fuzzy Rule-Based Modeling with Applications to Geophysical, Biological and Engineering Systems. CRC Press, Boca Raton, 323 p 79. Cagnoli, B., 1998, Fuzzy logic in volcanology. Episodes, v. 21, p. 94-96. 80. Van Wijk, M. T., and Bouten, W., 2000, Analyzing latent heat fluxes of coniferous forests with fuzzy logic. Water Resources Research, v. 36, p. 18651872. 81. Schulz, K., Huwe, B., and Peiffer, S., 1999, Parameter uncertainty in chemical equilibrium calculations using fuzzy set theory. Journal of Hydrology, v. 2 17, p. 119-134. 82. Kruiver, P. P., Kok, Y. S., Dekkers, M. J., Langereis, C. G., and Laj, C., 1999, A psuedo-Thellier relative palaeointensity record, and rock magnetic and geochemical parameters in relation to climate during the last 276 kyr in the Azores region. Geophysical Journal International, v. 136, p. 757-770.

BAYESIAN NEURAL NETWORKS IN PREDICTION OF GEOMAGNETIC STORMS GABRIELA ANDREJKOVA Department of Computer Science, Faculty of Science P. J. Safkrik University Jesennd 5, 041 54 KoSice, Slovakia e-mail: [email protected] Bayesian probability theory provides a framework for data modelling. In this framework it is possible to find models that are well-matched to the data, and to use these models to make nearly optimal predictions. In connection to neural networks and especially to a neural network learning the theory is interpreted as an inference of the most probable parameters for the model and the given training data. This article describes an application of the Bayesian probability theory to the physical problem "Prediction of Geomagnetic Storms".

Keywords: Bayesian probability theory; Neural Network; Geomagnetic Storm; Prediction. 1

Introduction

Neural networks continue to offer an attractive paradigm for the design and analysis of adaptive, intelligent systems for many applications in artificial intelligence 4 , 5 , This is true for a number of reasons: for example, amenability to adaptation and learning, robustness in the presence of noise, potential for massively parallel computation. Predictions of the hourly D,t index from the interplanetary magnetic field and solar plasma data, based on Artificial Neural Networks (ANN), were made and analysed by Lundstedt and Wintoft (1994) (feedforward networks) and Andrejkov6 et al. (1996, 1999) (recurrent networks, fuzzy neural networks) ', '. Recent results have shown that it is possible to use dynamic neural networks for GMS prediction and modelling of the solar wind-magnetosphere coupling. In this study we are reporting preliminary results using a Bayesian neural network model.

'

There has been increased interest in combining artificial neural networks with Bayesian probability theory '. The Bayesian probability theory have been

251

252

G. Andrejkovd.

proved to be very successful in a variety of applications, for example D. J. C. MacKay (1995), 8, 9, M. I. Schlessinger and V. Hlavae, l 3 and P. Miiller and D. R. Insua, l o . The effectiveness of the models representing nonlinear input-output relationships depends on the representation of the input-output space.

A designed neuro-Bayesian model will predict the occurrence of geomagnetic storms on the base of input parameters n,v, (TB,and B,: n . . . the plasma density of solar wind, w . . . the bulk velocity of solar wind, B,, (TB=. . . zcomponent of the interplanetary magnetic field and its fluctuation.

To follow the changes of the geomagnetic field values we use the quantity D,t index. Its values are in interval f l O n T during normal situation but during the geomagnetic storm they can decrease as much as some hundreds nT in a few hours. In Section 2, we describe some basic definitions and properties of the Bayesian probability theory. In Section 3, we briefly describe the neural networks as a probabilistic models. Section 4 contains the starting point to the finding weights of neural networks. Some interesting results for GMS prediction are described in Section 5.

Bayesian probability theory

2

A Bayesian data-modeller's aim is to develop probabilistic model that is well matched to the data and makes optimal predictions using that model. Bayesian inference satisfies the likelihood principle: Inferences depend only on the probabilities assigned to the data that were received, not on properties of the data which might have occured. We will use the following notation for conditional probabilities: 0

a,0 # 0 - the space of elementary events;

0

3-1 -

0

A, B - events, P ( A ) ,P ( B ) - a probability of the events A, B ,

0

(Cl,'H,'P) - a probability space,

0

- algebra of some nonempty subset

of R (a model of computation),

P(AIB,'U) is pronounced "the probability of A, given B and 3-1" and it explains the conditional probability;

Bayesian Neural Networks i n Prediction of Geomagnetic Storms 253 0

the statements B and mean the conditional assumptions on which this measure of plausibility is based;

The Bayesian approach require: 0

0

specifying a set of prior distributions for all of weights in the network (and variance of the error) and computing the posterior distributions for the weights using Bayes’ Theorem.

Prior distribution is a probability distribution on the unknown parameter vector w E R in the probability model, typically described by its density function P(w) which encapsulates the available information about the unknown value of w . In our case, the vector of weights w has not some know prior distribution and it means the prior distribution will be replaced by a reference prior function. Posterior distribution is a probability distribution on the unknown parameter vector w E R in the probability model, typically described by its density function P(wID), conditionally on the model, encapsulates the available information about the unknown value of w, given the observed data D and the knowledge about w , which the prior distribution P ( w ) might contain. It is obtained by Bayes’ Theorem. Bayes’ Theorem: Given data D { X ( ~Y(~)} ) , generated by the probability model {P(DIA),AE R} and a prior distribution P ( A ) ,the posterior distribution of A is P(AID) cx P(DIA)* P ( A ) . The proportionality constant is

{J, P(DIA) * P(A)dA}-l.

Two approaches have been tried in the finding of the posterior probability: 0

0

to find the most probable parameters (weights) using methods similar to the conventional training and then approximate the distribution over weights using information available at this maximum. t o use Monte Carlo method to sample from the distribution over weights. We applied the method and we use Markov chaines.

There are two rules of probability which can be used: 0

The product rule relates to joint probability of A and B , P ( A ,B ( R )to the conditional probability:

P ( A ,BlR) * P(BIR) = P(AIB,X)

(1)

254 0

G. Andrejkovci

The sum rule relates the marginal probability distribution of A, P(AI'?f), to the joint and conditional distributions: R

B

Having specified the joint probability of all variables as in equation, we can use the rules of probability to evaluate how our beliefs and predictions should change when we get new information.

3

Neural Networks as probabilistic models

A supervised neural network is a non-linear parametrized mapping from an input x to an output y^ = f(x,w;A). The output is a continuous function of the parameters w, which are called weights and A is an architecture of the network. 1 , The network is trained in the classical way using a data set D = ( ~ ( ~ y'")} by the backpropagation algorithm. It means the following sum squared error is minimized

The weight decay is often t o include to the objective function for the minimization. It means

M(w) = BED where Ew =

+ aEw,

(4)

xiwl.

The learning process above can have the following probabilistic interpretation. The error function is interpreted as minus the log likelihood for a noise model:

parameter /3 here defines a noise level

where rs& =

k.

rsi = i .

Bayesian Neural Networks in Prediction of Geomagnetic Storms 255

The function E corresponds to the deduction of parameters w according t o data D . It means

Bayesian inference for modelling problems may be implemented by analytical methods, by Monte Carlo sampling, or by deterministic methods using Gaussian approximations.

4

Starting points to the application

We deal only with neural networks used for regression. Assuming a Gaussian noise model, the conditional distribution for the output vector given the input vector based on this mapping will be as follows:

where d is the dimension of the output vector and in the outputs.

0

is the level of the noise

In the Bayesian approach t o the statistical prediction, one does not use a single ”best” vector of weights, but rather integrates the prediction from all possible weight vectors over the posterior weight distribution which combines the data with prior computed weights. The best prediction for the given input from the testing data x;+1 can be expressed

where d is the dimension of the weight vector. Posterior probabilities of weight vectors are the following:

256

G. Andrejkovd

For the full formulation of Bayesian problem it is necessary to add the prior distribution of weights. One of the possibilities is the following:

To compute the previous integrals it is very time consuming problem. It is possible to use Metropolis algorithm and it is the base for Monte Carlo method that we used in the prediction of GMS. We used the Monte Carlo method for a construction of our models. The algorithm was applied according the construction of R. M. Neal ll. 5

Results of GMS predictions

We have t o discuss various implementation issues which is necessary to do for the real prediction. The data are available from the NASA "OMNI tape" and are distributed by National Space Science Data Center and WDC-A for Rockets&Satellites. In the period 1963 - 1999 at each hour are measured and saved the next quantities: B,, UB=,n, u and D S t . Some data are not complete and we use liner interpolation to fill them but only in the case if the gap has less then 30 hours. The reconstructed data are used for a choose of the samples t o the training set according t o the following criteria: if the value D,t decreases at least 40 n T during two hours then the training sample (the storm) is created from the measured values 36 hours before the decreasing, 2 hours of the identification of decreasing and 108 hours after the decreasing. The file of the values have t o fulfill requirement of completeness of measurements. It means 144 hours describe one event GMS. One storm is used for the learning of the neural network by the moving of 8 hours window. We have prepared the training data set two data testing sets A and B. To prepare the A and B sets we used the data from years 1980 - 1984 and 1989 - 1999 because we had the continued values of parameters n , Y,B,, n B Z and D,t. The prepared data were represented by a sequence of

where pt can be applied as time series.

Bayesian Neural Networks i n Prediction of Geomagnetic Storms 257 Table 1. Experimental Results

Data

#Iteration

#Good

#Bad

Average

Success

Predictions

Predictions

4000

49

87

2.78377

36,03 %

4000

84

52

1.86015

61,76 %

6000

62

74

1.90585

45,59 %

6000

101

35

0.48040

74,26 %

12000

76

60

1.19665

55,88 %

12000

113

23

0.23863

83,09 %

18000

86

50

0.77771

63,24 %

18000

109

27

0.23801

80,15 %

Error

The software of M. Levickjr described in was modified and used in the present application. The algorithm based on the works of Neal and McKay was written in Delphi 5. The first computed results are in the following Table 1. The models are just tested. We present results computed with two data sets A and B. Prediction performance is measured by #Good Predictions, #Bad Predictions, AverageError and % of Success. Total test samples in testing sets A and B: 272, the number of input neurons in the neural network: 32, the number of hidden neurons in the neural networks: 28, the number of output neurons: 1. The computed results are interesting from the following points of view: 0

0

0

With the higher number of iteration the average error decreases. It is one of criteria t o the evaluation of the model. After 18000 iterations the success grows very slowly in the case of the testing data set A and decrease in the case B. Bayesian neural networks that we used in the prediction of geomagnetic storms look like very good model. They move the weight vector to the most probable part of the weight space.

258

G. Andrejkova'

References

1. AndrejkovB, G., AzorovB, J., Kudela, K.: Artificial Neural Networks an Prediction D,t Index. Proceedings of the 1st Slovak Neural Network Symp., ELFA, KoSice, 1996, pp. 51-59. 2. AndrejkovB, G., T6th, H., K., Kudela, K.: Fuzzy Neural Networks in the Prediction of Geomagnetic Storms. Proceedings of " Artificial Intelligence in Solar-Terrestrial Physics" , Publisher European Space Agency, Lund, 1997, p. 173-179. 3. Bernardo, J. M.: Bayesian Reference Analysis, A Postgraduate Tutorial Course, Facultat de Matematiques, Valencia, 1998. 4. Hassoun, M. H.: Fundamentals of artificial neural networks, MIT Press, Cambridge, 1995. 5. Hertz, J., Krogh, A., Palmer, R.G.: Introduction t o the theory of neural computation, LN Vol. 1, Santa Fe Institute Studies in the science of complexity, Addison-Wesley. 1991. 6. M. LevickL: Neural Networks in the analysis and the document classification, Diploma Thesis, P. J. SafArik University, KoSice, 2002. 7. Lundstedt, H., Wintoft, P.: Prediction of geomagnetic storms from solar wind data with the use of a neural network. Ann. Geophysicae 12, EGSSpringer-Verlag, 1994, p.19-24. 8. MacKay, D.J.C.: Bayesian Methods for Neural Networks: Theory and Applications. Neural Network Summer School, 1995. 9. MacKay, D.J.C.: A practical Bayesian Framework for Backprop Networks. Neural Computation 4, p. 448-472. 10. Muller, P., Insua, D.R.: Issues in Bayesian Analysis of Neural Network Model. Neural Computation 10, p. 749-770. 11. Neal, R. M.: Probabilistic Inference Using Markov Chain Monte Carlo Methods. Technical report CRG-TR-93-1, University of Toronto, 1993. 12. Neal, R. M.: Bayesian Training of Backpropagation Networks by the Hybrid Monte Carlo Method Technical report CRG-TR-92-1, University of Toronto, 1992. 13. Schlesinger, M. I., HlavBE, V. : Deset pfedna'dek z teorie statistickkho a strukturnzho rozpozna'vani. CVUT, Praha 1999.

ADAPTATION IN INTELLIGENT SYSTEMS - CASE STUDIES FROM PROCESS INDUSTRY KAUKO L E I V I S U University of Oulu, Control Engineering Laboratoly, P.O. Box 4300, FIN-90014 Oulun yliopisto, Finland E-mail:kauko. [email protected]? Intelligent methods are more and more applied to several tasks also in process industry. In industrial applications, the changing production environment causes the urgent need for adaptation. This paper concerns with applications of Smart Adaptive System to industrial problems using some case studies as examples.

Keywords: Intelligent systems,fuzzy logic, neural networks,paper mills,steel mills

1

Introduction

Many industrial applications of intelligent methods, including fuzzy logic, neural networks, methods from machine learning, and evolutionary computing, have recently launched especially in cases were an explicit analytical model is difficult. Examples come from the domains of measurements, monitoring, process control, diagnostics, quality control, forecasting, data mining, and many others. While being powerful and contributing to increased efficiency and economy of industrial processes, most solutions using intelligent methods lack one important property: they are not adaptive (or not adaptive enough), when the production environment changes. In other words, most such systems have to be redesigned - in many cases from scratch - when the production settings or process parameters change significantly. This may look strange, because at the first sight adaptivity seems to be the central issue for intelligent technology. The typical "learning by example" concept combined with techniques inspired by biological concepts is supposed to provide enhanced adaptivity. However, after looking more carefully, it becomes clear that most techniques for "learning" are used for one-time estimation of models from data, which remain fixed during routine application. "True" online learning, in most cases, has not reached a level mature enough for industrial applications. The target setting for EUNITE, the European Network on Intelligent Technologies for Smart Adaptive Systems, was originally written in the following way: "to join forces within the area of Intelligent Technologies for better understanding of the potential of hybrid systems and to provide guidelines for exploiting their practical implementations and particularly, 0 to foster synergies that contribute towards building Smart Adaptive Systems implemented in industry as well as in other sectors of the economy."

259

260

K. Leaviska

These targets are to be gained by various means: Roadmap activities, collecting case studies, writing best practice guidelines, concentrating on training and technology transfer, etc.

2

Smart Adaptive Systems

Early definitions of adaptation are from the 1950’es to 1970’es, but they are still valid today (see for instance Sagasti, 1970, Ashby, 1972). Control Engineering offers also a starting point in defining adaptive systems, when it defines adaptive controller simply as follows: “An adaptive controller is a controller with adjustable parameters and a mechanism for adjusting the parameters”. Further on, Control Engineering applications are defined as: Gain scheduling, that adapts to known changes in the environment Model Reference Adaptive Control that adapts the system parameters so that it follows the model behaviour Self-Tuning Control that adapts the controller so that the system response remains optimal in the sense of an objective function. We can carry knowledge about adaptive systems from control to other areas by analogy. An adaptive system is a well-defined concept within the automatic control community. Breakthroughs could occur by applying definitions, structures and theory to other areas such as management, transportation or healthcare. However, it should be remembered that Control Engineering methods are mostly concerned with parameter adaptation, which is only one alternative when speaking about Intelligent Systems. “Smart system” can be interpreted as a synonym to “Intelligent system”. A short definition is proposed in the first version of EUNITE Roadmap:

“Smartsystem is aware of its state and operation and can predict what will happen to it. This knowledge can also lead to adaptation. ”

A short definition for the Smart Adaptive System (SAS) comes from Anguita (2001): “A SAS can: Adapt to a changing environment Adapt to a similar settings without explicitly being ported to them Adapt to a newlunknown application.” The first level is the easiest form of adaptation and it concerns with systems that adapt their operation in changing environment by using their intelligence to recognise the changes and to react accordingly. The second level considers the change in the whole environment and the system’s ability to respond to it. The third level is the most demanding one and it requires tools to learn the system’s behaviour

Adaptation i n Intelligent Systems

261

from very modest initial knowledge. Some examples of it exist on Machine Learning field (Anguita, 2001). Both process and production industries face considerable control challenges, especially in the consistent production of high quality products, more efficient use of energy and raw materials, and stable operation in varying conditions. The processes are nonlinear, complex, multivariable and highly interactive. Usually, the important quality variables can be estimated only from other measured variables. Constraints, e.g. physical limitations of actuators must be taken into account. Significant interactions between process variables cause interactions between the controllers. Various time-delays depend strongly on operating conditions and can dramatically limit the performance and even destabilise the closed loop system. Uncertainty is an unavoidable part of the process control in real world applications. Systems for quality, environment and safety seem to be integrating and different kind of diagnostic functions play a more and more important role in the future. Production scheduling and resource allocation also present methodological challenges. Following chapters refer to potential SAS applications in process industry based on some case studies. Chapters are arranged according to the usual control hierarchy starting from measurements (Software Sensors) and control systems, and proceeding to &agnostics and quality control. Case studies come from the applications developed in Control Engineering Laboratory, University of Oulu. 3

3.1

Software Sensors

Basic Principles

Software Sensors are used in making the existing measurements more efficient or in replacing the non-existing measurements with software systems that form the measurements signals e.g. from other, existing measurements, laboratory analyses and a priori expert knowledge. Good example is the combining of information from several temperature or concentration measurements, e.g. from the blast furnace, to form a single measurement or the indication of the process state (Alaraasakka et al. 1998). Another possibility is to use process information to construct a Software Sensor for the non-existing quality measurement, e.g. in paper or biotechnical processes where on-line analysers are difficult to develop and expensive to install and maintain. Adaptation needs come from the necessity to correctly react to changing raw material quality, to different product specifications, or even to different processing alternatives. Portability is important from the commercial point of view: generic applications that guarantee long range of applications are needed. Software sensors can be developed with various modelling techniques. In early fermentation applications, fuzzy systems were used as a conversion estimator

262 K. Leaviska

(Kivikunnas, Bergonzini Corradini and Juuso, 1996) and as a trend analyser (Kivikunnas, Ibatici and Juuso, 1996). Some comments on using neural networks and adaptive algorithms are given in de Assis et al. (2000).

3.2

Case -Kappa Number Prediction in Pulp Cooking

Kappa number describes the extent of cooking wooden chips in the first stage of fibre production for paper industry. Kappa number cannot be measured directly during processing, but some measurement possibilities exist after it. However, the knowledge of Kappa number development during cooking is crucial to control the product quality. Kappa number prediction with measurements from ABB’s Cooking Liquor Analyser (CLA 2000) has been used as a comparison case for different methodologies for building Software Sensors (Leiviska et al. 2001, Isokangas and Juuso, 2000, Murtovaara et al., 1999): fuzzy logic (FL), partial least squares method (MLR), artificial neural networks (ANN) and linguistic equations (LE). While the cooking progresses, some wood comFonents start to dissolve into the cooking liquor. Usually, alkali components in the liquor are analysed by measuring the conductivity of the liquor sample, solid contents by measuring the refractive index and dissolved lignin by measuring the UV-absorbency of the sample. These measurements form the core for Kappa number prediction. Neural network models and linguistic equation models seem to learn the process behaviour in a similar manner. Differences are met in using the models in process environment (Murtovaara et al., 1999). Neural network models are suitable for processes where process conditions are stable and there is a lot of data available. The linguistic equation models are not so sensitive for changes in process conditions. The performance of fuzzy models depends on the shape and number of membership functions. The first level of adaptivity is easily gained with linguistic equations that adapt to changes in, e,g. production level using multiple models (Murtovaara et al. 1999). The second level corresponds to changing from continuous to batch process that, at this moment, requires manual tuning and requires also dynamic modelling. The third level could mean the porting of the analyser to a totally different process, but there are no experiences available. 3.3

Case -Adaptive Modelling of Carbon Dioxide in a Burning Process

The amount of carbon dioxide (C02) in combustion gases is one of the main variables that should be monitored in the burning process. Concentration of C 0 2 is usually measured with different types of gas analysers, but unfortunately, present methods are available only for large power plants. Thus, the need for alternative methods is evident in smaller wood burning processes. One possibility is to model the COZ concentration using existing process measurements.

Adaptation in Intelligent Systems 263

A fuzzy, adaptive modelling approach for estimating the carbon dioxide concentration of a burning process is presented in Ruusunen and Leiviska (2002). Adaptation mechanism to changes in operation conditions is based on recognition of three different combustion regimes: ignition, burning and charring. These regimes are described by three fuzzy membership functions. For each burning phase, two TS-type fuzzy models are constructed. The selection of model input variables is based on combustion theory and it is an important part of the model development. A gradient method and supervised learning is used for identification of the consequent parameters. Modelling results showed that the present model has a good capacity to approximate COz concentration of highly changing burning process. In comparison with other models, the performance of the fuzzy model was superior due to an adaptation mechanism that also enabled a robust and simple structure. Small size of training data and the nonlinear process limited performance of other models, while local learning and generalisation capabilities of the fuzzy model compensated for these disturbing factors. This system fulfils, by nature, the first level of SAS definition. It also makes it easy to convert the model to another burning process, according to the second level definition. Methodology itself is robust enough to allow porting to totally different areas where only limited data is available. 4

4.I

Adaptive Intelligent Control

Basic Principles

Adaptive controllers must include two elements that make adaptation possible: detection system that reveals changes in process characteristics and the adaptation mechanism, which updates the controller parameters. Changes in process characteristics can be detected through on-line identification of the process model, or by assessment of the control response. Adaptation mechanisms rely on parameter estimates of the process model, e.g. gain, dead time and time constant. The choice of performance measures depends on the type of response the control system designer wishes to achieve. Alternative measures include overshoot, rise time, settling time, delay ratio, frequency of oscillations, gain and phase margins and various error signals. Fuzzy logic controllers (FLC) can cope with a fairly large amount of nonlinearities in the process behaviour, but applying the following mechanisms extend their operating area: The changes in the amount of the nonlinearity can be taken into account by changing scaling fectors for appropriate variables. Altering the scaling factor is

264

K. Leiviska

similar to gain tuning in standard PID controllers as it changes the sensitivity of the controller to the input Changing the membership functions will alter the shapes of the fuzzy sets, and thus the gain can be modified within a specific region. Altering rules can change drastically the controller operation, and thus should be the last alternative in the adaptation procedure. These types of controllers are called self-organizing controllers. A vast literature is available (see references). The multilevel linguistic equation (LE) controllers correspond to adaptive fuzzy controllers, but also here the adaptation can be firther improved by the following mechanisms (Juuso et al. 1997, 1998): Altering the strengths of braking and asymmetrical action will fine-tune the controller operation for larger disturbances and remove the steady-state error. The changes in the strengths of the nonlinearity can be taken into account by scaling the membership definitions for appropriate variables. This mechanism, similar to the scaling in fuzzy controllers, is the basic technique in the operating point adaptation. Asymmetrical scaling is the main technique in braking and asymmetrical actions. Changing membership definitions will alter the gain within a specific region. This mechanism is used if the adaptation of the multilevel LE controller is not sufficient. Changing equations, which means that the control strategy is changed, is seldom used. The first direct linguistic equation controller was implemented in 1996 for a solar power plant in Spain (Juuso et al. 1997, 1998), and later the multilevel LE controller was installed for a lime kiln at UPM-Kymmene Pietarsaari mills (Juuso 1999a; Juuso 1999b). Both implementations are very compact and easier to tune than corresponding fuzzy controllers were. Modularity is beneficial for the tuning of the controller to various operating conditions, and most important is that the same controller can operate on the whole working area. 4.2

Case - Rotary Dryer Control

Rotary dryers are extensively used in chemical industry and minerals processing. To control a rotary dryer is difficult due to the long, and also varying, time delays involved, and also the changes in raw materials. Variations in the input variables as in the moisture content, temperature or flow of the solids will disturb the process for long periods of time, until they are observed in the output variables. Therefore, pure feedback control is inadequate for keeping the most important variable to be controlled, the output moisture content of the solids, at its target value with

Adaptation i n Intelligent S y s t e m s 265

.

acceptable variations. Also the development of model-based control systems has proved difficult and time consuming due to the complexity of the process. Increasing demands for uniform product quality and for economic and environmental aspects have necessitated improvements in dryer control. In Control Engineering Laboratory, University of Oulu, several methods for the rotary dryer control have been studied: conventional PI control, model-based cascade control, fuzzy control, and neural network control. Also two types of hybrid controllers where fuzzy logic and neural nets have been connected with PI control have been introduced (Yliniemi 2001, 1999, Koskinen et al. 1998, Yliniemi et al. 1998). Lately, three different fuzzy adaptive controllers were tested for the rotary dryer control. The first method was based on the previously applied hydrid PI-FLC controller with a simple gain adjusting approach. The output scaling factor is adjusted based on the experimentally defined rule base and membership functions. Good results were achieved and the controller performance was clearly improved. The second method introduced a fuzzy PI-controller and adjusting all scaling factors; for both inputs and the output. Results were not so encouraging, probably because of the fact that both the controller and the tuning part utilised rule bases and membership functions taken from the literature. The third approach was a simple procedure for tuning normal PI-controller with a fuzzy tuner. Once again, the adaptive controller performed better than the original one. All these controllers fulfil the first level definition of SAS; they can adapt to changes in the operation point of the dryer. They can also be transferred to another similar application with ease. The third level, porting them to a totally new application area must be approached with some hesitation. Methods itself are generic, but being rule-based they need process knowledge to be successfully applied. The results gained with the second method above seem to show this fact very clearly. 5

5.I

Diagnostics

Basic Principles

Fault diagnosis has been an area of strong theoretical development and industrial applications for several years. Various approaches have been studied and applied and they differ mostly in the following aspects: What sources of faults have been studied; sensors, actuators, controllers, control loops, process equipment, process parameters, operator-induced faults, How the faults have been described; Boolean, quantitative, qualitative; static or dynamic; deterministic or stochastic and What methods have been used in fault detection; testing against thresholds, pattern recognition, rule-based reasoning, fuzzy logic, neural networks, etc.

266

K. Leiviska

Fault diagnosis is already a classical area for fuzzy logic applications (Isermann 1997). Compared with algorithmic-based fault diagnosis the biggest advantage is that fuzzy logic gives possibilities to follow human’s way of fault diagnosing and to handle different information and knowledge in a more efficient way. Applications vary from troubleshooting of hydraulic and electronic systems using vibration analysis to revealing the slowly proceeding changes in process operation. The essential feature in the future is the combining of fuzzy logic with model-based diagnosis and other conventional tools and finally their integration within condition monitoring systems. Fault diagnosis is usually divided in three separate stages or tasks: Fault detection that generates symptoms based on the monitoring of the object system. Fault localisation that isolates fault causes based on the generated symptoms, and Fault recovery that decides upon the actions required to bring the operation of the object system back to the satisfactory level. The fault detection problem is, in principle, a classification problem that classifies the state of the process to normal or faulty. This is based on thresholds, feature selection and classification and trend analysis. Fault localisation starts from the symptoms generated in the previous stage and it either reports fault causes to the operator or starts automatically the corrective actions. In the first case, the importance of the madmachine interface must be emphasised. Actually this stage includes the defining of size, type, location and detection time of the fault in question. The frequency range of phenomena studied in fault detection varies tremendously; from milliseconds in vibration analysis to hours and days in analysing the slowly developing process faults. This sets different requirements for the system performance and computational efficiency in real-time situations. Also different methods are needed in data pre-processing. Vibration analysis uses spectral and correlation analysis to recognise the “fingerprints” of faults whereas regression analysis, moving averages or various trend indicators are used in revealing slowly developing faults. Especially for slow processes met in chemical, pulp and paper and biotechnical industries, temporal reasoning and trend analysis are very valuable tools to diagnose and control the process. Environmental changes together with changes in quality and production requirements lead to the need for adaptation. For instance, in surface quality testing of steel, change in the steel grade changes the patterns of single faults and it can also introduce totally new faults. This might require re-calibration of the camera system and training new patterns to the fault detection system. Reasons for surface faults can be traced back to earlier stages of production, to dosing of different chemicals, to inadequate control of process operations, and also to different routing of customer orders through the production line. Here, the steel grade change would

Adaptation in Intelligent Systems 267

require new models to be taken into the use, because of other variables affecting the surface quality. Fault diagnosis is in many cases essential to quality control and optimising the performance of the machinery. For example the overall performance of paper machines can be rated to the operating efficiency, which varies depending on breaks and the duration of the operations caused by the web break. Most of the indistinct breaks are due to dynamical changes in the nonlinear chemical process with many and long time varying delays. There are also process feedbacks on several levels, closed control loops, factors that exist and cannot be measured and interactions between physical and chemical factors. Combining expertise with data driven methods is necessary in complex industrial applications.

5.2

Nozzle Clogging in Continuous Casting of Steel

Surface faults of steel originate from different sources along the long production line. One important source is the clogging of the submerged entry nozzle that connects the tundish and the mold in the continuous casting of steel. It is a tube with a diameter of about 15 cm and it divides the molten steel into two directions on the both sides of the nozzle. The nozzle transfers molten steel from the tundish to the mold and separates steel from the atmosphere. Nozzle is clogged when some material is collected on the inner surface of the nozzle and stops the flow of molten steel. It causes production losses and quality impairment in continuous casting, especially with aluminium-killed steels. Variations in stopper rod position and casting speed provide the operator with the first information on increased risk of nozzle clogging. This cannot, however, answer the question, how long time the casting can continue and when the nozzle should be changed. Neural networks are used to predict of the amount of steel that can be cast without changing the nozzle (Leiviska et al. 2002). These models are based on data collected from Rautaruukki Steel Mill’s converter plant. Nozzle clogging has been modelled on two casters; numbers 5 and 6. Feedfonvard networks with backpropagation were used. Data from 5800 heats was available, but only a small part of it was used. First of all, only clogging cases were used in training and they were divided first according to casters and secondly based on steel grades. Practice showed that different variables dominate with different grades and no general models could be done. Results seem promising; in several cases the cast tons are estimated with the accuracy of *6O tons (one heat) in more than 80% of cases. Cross-testing considered using the model developed for caster 5 to the data from caster 6. It showed that different variables affect the nozzle clogging in different casters. When testing the models with successful castings, it was found that the models never gave too high predictions.

268

K . Leiviska

This is an example of a hybrid system. The risk of clogging is determined based on the stopper rod position and casting speed variations. Neural network models are used when the risk increases. The system itself is not adaptive at all. Adaptivity is gained by changing the model according to the caster and the grade. Basically, this corresponds to SAS second level adaptation, but no automatic operations are connected to it in this application. 5.3

Web Break Indicator for a Paper Machine

Paper web breaks can be divided into three classes: process related, mechanics related and automation related. Process related breaks are considered the most important ones especially in the short circulation and the wet end of paper machine, starting from the mixing chest to the first drying group. This area includes nonlinear processes and there are many long delays that change with time and with process conditions. It includes also closed control loops and there exist factors that cannot be measured. There are also interactions between physical and chemical factors. A prototype web break sensitivity indicator was developed together with Metso Corp. to give the process operators a continuous indication of web break sensitivity in an easily understandable way (Juuso et al. 1998). The prototype of web break sensitivity indicator has been implemented as a Case-Based Reasoning -type application. The application is based on recorded historical data of main process measurements. For modelling the break sensitivity of the paper machine was divided into five classes starting from “no breaks”- class and ending at “a lot of breaks”-class. Models were trained using Linguistic Equations approach. The indicator contains 25 separate LE models; five models for each class. The indicator compares on-line data to these models in database and estimates how well the present measurements fit to each of the break sensitivity classes. The final result is deduced with fuzzy logic and presented as a “break sensitivity index”. The system has been tested at two UPM Kymmene paper mills. Tests showed that the prototype indicator operates as planned. The existing system is not adaptive. LE approach makes it easy to add the first level adaptation, but in this application it is not so essential. More important is the second level adaptation; i.e. to be able to deal with different grades and actually with different paper machines. This could, in theory, also be done using LE and automatic generation of equations, but in practice a lot of experience and knowledge is required in selecting variables and equations. Same agrees with the third level, for instance porting the method to a steel rolling mill application.

A d a p t a t z o n zn Intellzgent Systems 269

6

6.I

Quality Control

Basic Principles

Quality control within process industries is used both for registration of product quality as well as for direct feedback to the process control system. When product quality is used directly in the control of the process it is very important to have a measurement system that makes the quality figure available as fast as possible. Many quality control systems are based on computer vision. It is very interesting to note that Intelligent Methods are used extensively in almost all stages of computer vision process from sensing to scene interpretation. If the time delay is too long between taking a sample of the product and until the quality analysis is available, it is often necessary to base quality control on measurements, which are related to the product quality, i.e. Software Sensors. Also diagnostic information is used in many cases. Quality control is very often based on modelling of relationships between continuous measurements and quality analyses. For processes, which are difficult to model mathematically, the techniques of neural nets, fuzzy logic and genetic algorithms are methods, which are likely to produce good results. Especially a combination of methods is a promising way to as good a quality control basis as possible. Quality data origins from several sources; it is partly connected with raw materials, the production processes and final or intermediate products. There are two types of data: discrete data (attributes) describing for instance the amount of defective products in a sample or the number of defects in a product and continuous data (variables) describing for instance analysed or measured values. The processing of continuous quality variables (e.g. paper quality, results from quality analysers in chemical processes) resembles the problem setting in process control and fault diagnosis. The analysis of data on attributes proceeds in three stages: feature extraction, classification and decision. The applications of quality control vary both in terms of on-line processing and madmachine interface requirements. Advisory systems can only tell the user what kinds of defects have been found and record the defect data e.g. for possible customer reclamations. They can also give advises for product rejecting. On-line systems can automatically take the product that does not fulfil quality standards out from the production line and reject it or return it to production and repairing. 6.2

Case - Quality Control of a TMP Plant

TMP is a common process to produce wood fibres to paper production. The quality control in a TMP plant (maintaining freeness and fibre length at optimum levels) has to cope with two different kinds of disturbances: slowly proceeding wearing of refiner plates and faster variations in wood quality (Myllyneva et al. 2001). Pulp

270

K . Leiviska

quality is usually obtained from laboratory testing of pulp or finished paper samples, so this information is a few hours old and therefore of no use for real-time control. Also, delays in on-line freeness and fibre length measurements have limited the use of automatic on-line quality control. Generally, refiner control systems are based on the principle that freeness is directly related to specific energy consumption in the process. Motor load usually controls this. The strategy presented in Myllyneva et al. (2001) is based on the fact that the refining consistency is another key variable in the TMP-plant control because it affects on how energy is transferred to fibres. When the consistency and motor load are stabilised there exists a good basis for the actual quality control of the TMP-plant. In this case this stabilisation is done using conventional adaptive controllers. The fuzzy quality control is based on on-line measurements of freeness, fibre length and consistency. The fuzzy quality controller works as a master controller adjusting the set points of the motor loads and primary stage consistency. Inputs are the difference between the freeness set point and measurement, consistency set point and mean fibre length measurement. The consistency set point is used as a control input instead of the measurement, because the actual consistency always includes short-term fluctuations around the set point and therefore may cause some unnecessary output changes. The fuzzy quality controller has been implemented in the mill’s automation system using standard fuzzy controller toolbox available in the system. Minimummaximum method was used in inference and singletons were used for outputs. The controller has been tuned so that the main controlled variable is the fibre length. Therefore, the freeness level is allowed to increase, when the motor load control opens the plate position more than the consistency controller requests. The fuzzy quality controller was implemented as a part of AutoTMP control system at four refiner lines at Holmen Paper, Hallstavik mill in Sweden (Myllyneva et al. 2001). The system features the TCA consistency measurement in the blow line. Based on it, the refining consistency and motor load are controlled using adaptive control algorithms. The fuzzy control strategy maintains the fibre length at an acceptable level and the freeness is controlled to be on a target level. Operator confidence in the control system has been very high; 95-100 % uptime for the consistency and motor load controls and over 90 % uptime for the fuzzy quality controls have been experienced. The six months follow-up period proved a 20-50 % decrease in the standard deviations of the key quality and process variables. In this case the quality controller is a fuzzy supervisory controller without adaptive features. The system is a hybrid one, where adaptivity is present at the direct process control level. It more or less represents the first level SAS. The second level SAS (porting to similar settings) is achieved easily for process control level, but requires tuning of the fuzzy part. The system is based on special

A d a p t a t i o n in Intelligent Systems

271

measurements and deep process knowledge, so the third level SAS is not applicable here.

7

Conclusions

This paper has introduced the applications of Smart Adaptive Systems in process industries using some case studies as examples. The applications are high in numbers and the future development seems to favour adaptive systems. Needs for adaptivity is coming from changes in process conditions, but also from the facts that generic easily tuneable systems are desirable from the commercial point of view. EUNITE is the European Network on Intelligent Technologies for Smart Adaptive Systems that started operations at the beginning of 200 1. Inside EUNITE, Smart Adaptive Systems have been defined to cover three levels of adaptation: adaptation to changes in environment, adaptation to a new application in a new, but similar environment and adaptation to a totally new/unknown application in different environment. The findings in this paper show that this definition is valid in process applications and can facilitate the analysis of systems applicability. References 1. Alaraasakka, M., Leiviska, K., Seppken, M. (1998). Neural Network based Classifier of Blast Furnace Gas Distribution. Proceedings of TOOLMET '98 Symposium. Oulu, Finland, 40-44 2. Anguita, D., (2001): Smart Adaptive Systems - State of the Art and Future Directions for Research. First EUNITE Annual Symposium, Tenerife, Spain, Dec. 13-14,2001 3. Ashby, W.R. (1972): Design for a Brain. Science Paperbacks, Chapman and Hall, London 4. Chung, H.Y., Chen, B.C., Lin, C.C. (1998): A PI-type fuzzy controller with self-tuning scaling factors. Fuzzy Sets and Systems 93, 23-28 5. Daugherity, W.C., Rathakrishnan, B., Yen, Y. (1992): Performance evaluation of a self-tuning fuzzy controller. In: Proc. IEEE International Conference on Fuzzy Systems 389-397 6. de Assis A.J., R.M. Filho (2000): Soft sensors development for on-line bioreactor state estimation. Computers / Chemical Engineering 24, 1099- 1103 7. He, S.Z., Tan, S., Xu, F.L. (1993): Fuzzy self-tuning of PID controllers. Fuzzy Sets and Systems 5 6 , 3 7 4 6 8. Ikaheimonen, J., Leiviska, K., Matkala, J. (2002). Nozzle clogging prediction in continuous casting of steel. Accepted to 2002 IFAC World Conference, Barcelona 9. Isermann R. (1997). Supervision, fault-detection and fault-diagnosis methods. XIV Imeko World Congress, 1-6 June 1997, Tampere, Finland

272

K. Leiviska

10. Isokangas, A., Juuso E.K. (2000): Development of Fuzzy Systems from Linguistic Equations for Kappa Number Prediction in Continuous Cooking. In: L. Yliniemi and E. Juuso (eds.), Proceedings of TOOLMET 2000 Symposium, Oulu, April 13-14, pp. 63-77 11. Jung, C.H., Ham, C.S. Lee, K.I. (1995): A real-time self-tuning fuzzy controller through scaling factor adjustment for the steam generator of NNP. Fuzzy Sets and Systems 7,: 53-60 12. Juuso, E.K. (1999a). Fuzzy Control in Process Industry: The Linguistic Equation Approach. In: H. B. Verbruggen, H.-J. Zimmermann, R. Babuska (eds.) Fuzzy Algorithms for Control, International Series in Intelligent Technologies, pp. 243-300, Kluwer, Boston 13. Juuso, E.K. (1999b). Intelligent Dynamic Simulation of a Lime Kiln with Linguistic Equations. In: H. Szczerbicka (ed.), ESM’99: Modelling and Simulation: A Tool for the Next.Millenium, 13th European Simulation Multiconference, SCS, Delft, The Netherlands, 1999. Volume 2, pp. 395-400, Delft, The Netherlands, SCS 14. Juuso, E. K., Balsa, P, Valenzuela, L., Leiviska, K. (1998). Robust Intelligent Control of a Distributed Solar Collector Field. 3rd Portuguese Conference on Automatic Control, Sept 9-1 1, 1998, Coimbra, Portugal. 6 pp. 15. Juuso, E. K., Balsa, P., Leiviska, K. (1997). Linguistic Equation Controller Applied to a Solar Collector Field, Proceedings of ECC’97, Paper 267,6 pp. 16. Juuso, E., Ahola,T., Oinonen, K., Leiviska, K. (1998). Web Break Sensitivity Indicator for a Paper Machine. Proceedings of 1998 EUFIT Symposium, Aachen, Germany 17. Kivikunnas, S., Bergonzini Corradini, M., Juuso, E.K. (1996): Fuzzy conversion estimation in fermentation control. In: L. Yliniemi, E.Juuso (eds.), Proceedings of TOOLMET’96 (Oulu, April 1-2, 1996), University of Oulu, Control Engineering Laboratory, Report A No 4, pp. 177-186 18. Kivikunnas, S., Ibatici, K., Juuso, E. (1996): Process trend analysis and fuzzy reasoning in fermentation control. In: B. G. Mertzios, P. Liatsis (eds.), Proceeding of IWISP’96 (Manchester, Nov. 4-7, 1996), p. 137-140 19. Koskinen, J., Yliniemi, L., Leiviska, K. (1998): Fuzzy modelling of a pilot plant rotary dryer. In: Proc.UKACC International Conference on CONTROL’98. Swansea, 1 , 515-5 18 20. Leiviska, K., Juuso, E., Isokangas, A. (2001). Intelligent Modelling of Continuous Pulp Cooking. In: Industrial Applications of Soft Computing (K. Leiviska, Ed.). Studies in Fuzziness and Soft Computing, Vol. 71, Springer Verlag 21. Lui, H.C., Gun, M.K., Goh, T.H. Wang, P.Z. (1994): A self-tuning adaptive resolution (STAR) fuzzy control algorithms. In: Proc. 3rd IEEE World Congress on Computational Intelligence. 3, 1508-1 5 13 22. Mudi, R.K. and N.R. Pal (1999). A Robust Self-Tuning Scheme for PI- and PID Type Fuzzy Controllers. IEEE Transactions on Fuzzy Systems, 7,2-16

A d a p t a t i o n in Intelligent S y s t e m s 273

23. Mudi, R.K. and N.R. Pal (2000). A self-tuning fuzzy PI controller. Fuzzy Sets and Systems, 115,327-338 24. Murtovaara, S., Leiviska, K., Juuso, E., Sutinen R. (1999): Modelling of Pulp Characteristic in Kraft Cooking. Oulu, December 1999, University of Oulu, Control Engineering Laboratory, Report A No 9. 20 p 25. Myllyneva, J., Karlsson, L., Joensuu I. (2001): Fuzzy Quality Control of a TMP Plant. In: Industrial Applications of Soft Computing (K. Leiviska, Ed.). Studies in Fuzziness and Soft Computing, Vol. 7 1, Springer Verlag 26. Rarnkumar, K.B., Chidambaram, M. (1 995): Fuzzy self-tuning PI controller for bioreactors. Bioprocess Engineering 12(5), 263-267 27. Ruusunen, M., Leiviska, K. (2002): Fuzzy Modelling of Carbon Dioxide in a Burning Process. Accepted for IFAC World Congress, Barcelona, July 28. Sagasti, F. (1970): A Conceptual and Taxonomic Framework for the Analysis of Adaptive behaviour, General Systems, Vol. XV, 1970 29. Yliniemi, L. (1999): Advanced control of a rotary dryer. Dissertation, University of Oulu, Department of Process Engineering 30. Yliniemi, L. (2001); Adaptive Fuzzy Control of a Rotary Dryer, In: Industrial Applications of Soft Computing (K. Leiviska, Ed.). Studies in Fuzziness and Soft Computing, Vol. 7 1, Springer Verlag 31. Yliniemi, L., Koskinen, J., Leiviska, K. (1998): Advanced control of a rotary dryer. In: Heidepriem, J. (ed.): Automation in Mining, Mineral and Metal Processing Preprints. Elsevier Science, NewYork, pp. 127-132

This page intentionally left blank

ESTIMATION AND CONTROL OF NON-LINEAR PROCESS USING NEURAL NETWORKS ANNA JADLOVSKA Department of Cybernetics and Artificial Intelligence, Technical University, Letncj 9, 040 01 KoSice, Slovak Republic E -mail:[email protected] A neural network application for system identification and control of non-linear process is described in this paper. The non-linear identification is mostly using feed-forward neural networks as useful mathematical tool to build non-parametric model between the input and output of a real non-linear process. The possibility of on-line estimation of the actual parameters from off-line trained neural model of the non-linear process using the gain matrix is considered in this paper. This linearization technique is used in the algorithm on-line tuning of the controller parameters based on pole placement control design for non-linear SISO process. Keywords: neural network, non-linear parameter instantaneous linearization

1

estimation, non-linear control,

Introduction

The purpose of this paper is to show how feed-forward neural network (Multi Layer Perceptron - MLP) can be used for modeling and control of the non-linear process. When the mathematical model of the process cannot be derived with an analytical method, then the only way is using the relationship between the input and output of the process. Fitting the model from the data is known as an identification of the process. For linear processes this technique is generally well known [7]. For processes, which are complex or difficult to model the non-linear identification can use feed-forward neural network - MLP as useful mathematical tool to build the non-parametric model between the input and the output of the real non-linear process [ 2 ] , [3], [ 5 ] , [9]. We will consider the possibility of an on-line estimation of the actual parameters from off-line trained neural model of the nonlinear process using the gain matrix, introduced later. This linearization technique is used to perform an on-line tuning of the controller parameters based on pole placement control design in control structure, whwhich the functional behaviour is similar to a gain sheduling control [4], [8]. The advantage using instantaneous linearization is that the controller parameters can be changed in response to process changes.

275

276

2

A . Jadlovskd

Non - linear System Identification

In this part we will discuss some basic aspects of non-linear system identification using, among numerous neural networks structures, only Multi-Layer Perceptron MLP (a feed-forward neural network) with respect to model based neural control, where the control law is based upon the neural model. In this paper we will use feed-forward neural network MLP with a single hidden layer. This structure is shown in matrix notation in Fig. l., [8].

II

q - I

4 - 1I

I

I

I L___-_____________-----____J Fig. 0. Matrix block diagram of an MLP

The matrix W, represents the input weights, the matrix W2, represents the output weights, F represents vector function containing the non-linear (tanh) neuron functions. The ‘I ‘ shown in Fig.1 together with the last column in W, giving the offset in the network. The net input is represented by the vector Zjn and the net output is represented by the vector Zout . The mismatch between the desired output Zour and Zout is the prediction error E.

The output from MLP can be written as:

From a trained MLP (by Back-Propagation-Error algorithm - first-order gradient method, Gauss-Newton algorithm - second-order gradient method) which has m o inputs and rn2 outputs a gain matrix N can be found by differentiating with respect to the input vector of the network. The gain matrix N can be calculated from (1)

Estimation and Control of Non-linear Process

277

where W; W, (excl. last column). The above mentioned gain matrix N allows an on-line estimation of the actual model parameters from off-line trained neural model - MLP of the non-linear process. In our paper we will apply an idea using input-output parametric non-linear models ARMAX (NARMAX) in the non-linear system identification by neural networks, [2], [ 6 ] . The non-linear ARMAX (NARMAX) model can be defined Y(k)= F(Y(k - I),. ..,Y(k - p ) ,U(k - r),. ..,U(k - rn),E(k - I ),...,E(k - p),e)

Y ( k )= G(k)+ E ( k )

(3)

where F is non-linear vector function, 6' represents the parameters and E ( k ) is the prediction error. Here p and rn denote the number of delayed outputs and inputs. The neural NARMAX model with input and output vectors

~ ( -kI), ..., ~ ( -kp ),..., ~ ( -z),...,u(~ k - rn), E(k -I),..& -p )

zou, (k)= f

(4

(4)

is shown in Fig.2, which is a recurrent network After training neural network MLP the actual gain matrix N ( k ) can be on-line estimated, and calculated by (2) and for NARMAX model by (5): d f(k) N ( k ) = dZ&) d Z l ( k ) d { Y ( k - 1 ) ............ E ( k - p ) } ~

=

[hI (k)...

-

hp( k ) i, ( k ) . .. b, ( k )

( k ) .. . Ep ( k ) }

(5)

where a , ( k ) for i = 1,.. . , p , b,(k) for i = l,..., rn, l , ( k ) for i = 1,.. . , p are estimated parameters of neural NARh4AX model for step k . Because neural NARMAX model in Fig.2 contains feedback loops around MLP we will apply for training this recurrent network a second order Recursive Prediction Error Method (RPEM) using Gauss-Newton search direction, [2], [4], ~71.

278

A . Jadlovskd

Fig. 0. Input - output neural NARMAX model

3

Non-linear Control

The model of non-linear process and the training method has been considered generally for the multivariable case in part 2. Next we will think about control design for non-linear SISO process using neural NARMAX model. A trained neural NARMAX model representing a model of the non-linear process is ussed for an on-line estimation actual process parameters by gain matrix N ( k ). This linearization technique called instantaneous linearization allows an online tuning of the controller parameters using pole-placement control strategy. This control concept, which is well known from the linear control theory will be implemented with RST - controller, [ 11. An example of the control structure using the estimation process parameters from neural NARMAX model, which is applied for an on-line tuning parameters of RST-controller by pole-placement design is illustrated in Fig.3. 3.1

Simulation Results of Non-linear Control Using Parameter Estimation

The idea and results of the estimation process parameters from an off-line trained neural NARMAX model and its using for tuning parameters of RST - controller

Estimation and Control of Non-linear Process 279

designed by pole-placement strategy (non-linear system control) are presented for non-linear test SISO process, [4] : y(k+I)=

0,95. y ( k ) + 0 , 2 5 4 k )+ 0,58 . u ( k ) . y ( k )

1+ Y ( U 2

We consider NARMAX model with 6 inputs ( p = m = 2 ) and 8 neurons in the hidden layer. The activation hnction in the hidden layer is tunh function and in the output layer alinear function is selected. The actual gain matrix N ( k ) can be calculated by (2) and the actual values of the estimated 6 - and b - parameters can be obtained from N ( k ) by (5). These parameters can be used to calculate the RSTcontroller parameters.

Fig. 1. Control scheme using a trained neural network for parameter estimation

In Fig.4 there are plotted the input and the output signals, which are used to train the neural net in the NARMAX structure by simulation scheme in Fig.5. The neural NARMAX model is trained with Gauss-Newton algorithm based on RPEM. Model is validated using time validation test in Fig.6 [2]. It is clear that neural model can be accepted by this test. The presentation of results of a non-linear control, a control with a RSTcontroller designed by a pole-placement design using an on-line parameter estimation from an off-line trained neural model is illustrated in Fig.7 at the change of one parameter of the non-linear SISO process (6). This example shows real power of the neural modeling using the structure ARMAX known from the theory of linear identification and the possibility to apply

A . Jadlovskd

280

the pole-piacement method known from the linear control theory for control of nonlinear SISO process in the control structure, which is similar with the self-tuning control. 2.5

2 1.5 1

0.5

n 0

200

400

800

600

1000

1200

5,

1

4 -

3 2 -

-1

'

0

200

400 600 800 Fig. 2. The input and output training signals

+a

Gain

I

1200

1000

cia&

To W o k p a c e 3

Y T a W o k p a ce2

ZOH

To Wokpacel Scope

Scope1

Fig. 3. The simulation scheme for collection of the training data

Estimation and Control of Non-linear Process

281

2.5 2 1.5

1 0.5

0 -0.51

0

100

50

150

200

250

300

Fig. 4. Validation of the neural model

0.6

0.5

0.4 0.3 0.2 0.1 0 0

50

100

150

200

250

300

350

0.2 -

0.15

-

0.1 -

-0.05 -0.1

r

-

-

Fig. 5. The RST controller based on actual parameter estimation from neural NARMAX model

7

282

4

A . Jadlovskd

Conclusions

In this paper the neural NARMAX is trained model as one-step predictor for nonlinear SISO process. After training this NARMAX model can be used in closed control loop to an on-line estimation of the process parameters, which allow tuning of the controller parameters by the pole-placement method. A practical simulations by language MATLAB/SIMULINK and Neural Toolbox ilustrate, that this control strategy using linearization technique by the gain matrix from neural NARMAX model produces excelent performance for control of non-linear SISO process. But this controller design can be applied for only non-linear processes, which does not contain hard nonlinearities. References 1. Astrom, K., J. and B. Wittenmark (1990). Computer Controlled Systems, Theory and design, Prentice-Hall, second edition 2. Chen, S., S. Billings and P. Grant (1990). Non-linear system identification using neural networks. In: Znternational Journal of control, Vo1.5 1, No.6, pp.1191-1214 3. Jadlovska A. (2000). An Optimal Tracking Neuro-Controller for Nonlinear Dynamic Systems, In: Control System Design, A Proceedings volume from P A C Conference Bratislava, Slovak Republic, 18-20 June, Published for IFAC by Pergamon - an Imprint of Elsiever Science, pp.493-499, ISBN 00-08 043 546 7 4. Jadlovska A. (2001). Modeling and Control of Nonlinear Processes Using Neural Networks, Ph.D.thesis, FEI - TU, KoSice, 156 pages 5. Jadlovska A. and J. Sarnovsky (2001). Neural Predictive Control of Non-linear System. In: Proceedings of the 13-th Znternational Scient$c-Technical Conference PROCESS CONTROL ‘01, StrbskC Pleso, High Tatras, Slovak Republic, CD-disk, 5 pages, ISBN 80-227- 1542-5 6. Leontaritis I., J. (1985). Input-output parametric models for non-linear systems, part 1 and 2., In: International Journal of Control, Vo1.41, No.2, pp.303-344 7. Ljung L. (1987). System Identification, Theory for User, Prentice Hall, first edition 8. Najvarek, J. (1996). Matlab and neural networks, FEI VUT, Brno 9. Narendra, K. S. and A.U. Levin (1990). Identification and control of dynamical systems using neural networks, In: IEEE Transaction on Neural Networks, Vol. 1, NO.1, pp.4-27

THE USE OF NON LINEAR PARTIAL LEAST SQUARE METHODS FOR ON-LINE PROCESS M O N I T O R " AS AN ALTERNATIVE TO ARTIFICIAL NEURAL NETWORKS PAOLO F. FANTONI, MARIO HOFFMANI? IFE OECD Halden Reactor Project, Norway E-rnai1:[email protected]

WESLEY HTNES, BRANDON RASMUSSEN The University of Tennessee, W, USA ANDREAS KIRSCHNER IPM, University of Applied Sciences Zittau/Gorlitz (FH) On-Line monitoring evaluates instrument channel performance by assessing its consistency with other plant indications. Industry and EPRI experience at several plants has shown this overall approach to be very effective in identifying instrument channels that are exhibiting degrading or inconsistent performance characteristics. On-Line monitoring of instrument channels provides information about the condition of the monitored channels through accurate, more frequent monitoring of each channel's performance over time. This type of performance monitoring is a methodology that offers an alternate approach to traditional timedirected calibration. On-line monitoring of these channels can provide an assessment of instrument performance and provide a basis for determination if adjustments are necessary. Elimination or reduction of unnecessary field calibrations can reduce associated labour costs, reduce personnel radiation exposure and reduce the potential for miscalibration. PEANO is a system for on-line calibration monitoring developed in the years 1995-2000 at the Institutt for energiteknikk (IFE), Norway, which makes use of Artificial Intelligence techniques for its purpose. The system has been tested successfully in Europe in off-line tests with EDF (France), Tecnatom (Spain) and ENEA (Italy). PEANO is currently installed and used for online monitoring at the HBWR reactor in Halden. A major problem in the use of Artificial Neural Networks, as in PEANO, is its limited retraining capability (which is necessary whenever process component changes occur) and its exponential complexity increase with the number of monitored signals. To overcome these limitations, an approach based on Non Linear Partial Least Square, an extension of the well-known LPS method, is proposed. In this work the NLPLS algoritm will be implemented in the PEANO architecture and its performance will be compared with the current PEANO version, based on ANN. For this purpose, real data from an operating PWR will be used for testing both systems.

Keywords: Signal Validation, Process Monitoring, Pattern Recognition, Artz/icial Neural Networks. NLPLS

1

Introduction

Signal validation models are often constructed using sets of highly correlated variables. Variable matrices characterized by linear dependencies between columns are said to be collinear and present an ill-posed problem to empirical modeling

283

284

P. F. Fantoni et al.

tasks. This causes an increase in the noise level of predictions and causes unstable and unrepeatable results. The problem arises due to the numerical complexities of inverting collinear matrices. The data sets are often highly correlated due to the number of sensors used to monitor large-scale processes, such as a nuclear power plant or other type of electrical generation facility. These sensors are located throughout the system to monitor parameters that are physically related and thus, their measurements are also related. The use of redundant sensors to monitor safety critical parameters exacerbates the problem by increasing the average correlations of the data set. One class of methods well suited for empirical modeling with collinear data is projection based techniques. Partial least squares (PLS) is a member of this class whereby, the predictor variables are transformed to an equivalent set of orthogonal variables. Since orthogonal vectors are independent, this eliminates the problems associated with collinearity. Though other projection based methods, such as principal component regression (PCR) are available, PLS poses an advantage in that its projections to latent variables are supervised by the desired response, whereas in other methods the projections are unsupervised with respect to the desired response. PLS has many attractive features making it an important model to consider for signal validation purposes. The creation of orthogonal bases from which univariate regression models can be derived, eliminates the numerical problems associated with collinearity. An additional feature is the inherent regularization of the method, which provides reproducible, stable solutions.

2

2.1

TheMethod

The PLS Method

First introduced by H. Wold in the field of econometrics [I], PLS has become an important technique in many areas, including psychology, economics, chemistry, medicine, pharmaceutical science, and process modeling [2]. PLS is a class of techniques for modeling the associations between blocks of observed variables by means of latent variables [3]. These latent variables are created through a supervised transformation, whereby an orthogonal basis results. The PLS algorithm used in this work is a special case of the standard technique, where the Y block consists of a single column or response variable. This case of PLS modeling is referred to as PLS-I [4] and is employed here due to its regularization properties, putting it in the same class as the methods of Ridge Regression (RR) and Principal Component Regression (PCR) [3]. Extensive discussions of PLS methods for the general case of more than one response variable are available in the literature [5-81. All discussions presented here refer to the PLS-1 technique. The PLS-1 technique is an inferential modeling method. The inferential

Non-linear Partial Least Square Methods for On-line Process Monitoring

285

structure provides the additional benefit of eliminating the ability for perturbations in a specific variable to directly propagate to the prediction of that variable. This is due to the architecture of inferential models, in which a variable's response is inferred through the use of other variables correlated with it. PLS-1 is an inferential modeling technique of the class of regularization techniques. It has been directly compared to RR and truncated singular value decomposition (TSVD) in a signal validation study [9], as well as to RR and PCR in a chemical application [ 101. In both cases, favorable results were reported for the PLS algorithm comparisons, and stable solutions were obtained. The chemical application resulted in a PLS model exhibiting a solution which was stabilized in comparison to the OLS solution, and comparable in prediction error to RR. The signal validation study was based on a predictor variable block plagued with collinearity, and reported a highly stable PLS solution in comparison to unregularized neural networks, and OLS. This stability results from the orthogonalization of the predictor variable block, and the reduced variance of the estimates due to the elimination of higher ordered latent variable pairs. These higher ordered latent variable pairs contain the least amount of variation related to the response variable, and are generally considered to be noise. The inner relationships of the standard PLS algorithm are constructed using univariate regressions on the latent vectors. In attempts to enhance the technique, quadratic functions have been introduced into the inner relationship [l l-141. However, quadratics are still linear in their parameters and do not guarantee a proper solution [ 151. The use of single hidden layer feedforward neural networks (NN) has been suggested [16], and applied in the field of chemistry [17-211. These methods have been under study at the University of Tennessee, for the purposes of signal validation in large-scale processes, beginning in 1999. The method used herein, uses hlly connected feedforward neural networks in the inner relationships and maintains the linear orthogonalization in the outer relationships. This method will be referred to as neural network partial least squares ("PLS), to avoid confusion with the various nonlinear methods utilizing quadratics or radial basis functions [22]. A NNPLS signal validation system has been implemented, on a trial basis, at the gth unit of Tennessee Valley Authority's Kingston fossil plant, in Kingston, Tennessee, USA [23-261. Two features of the PLS algorithm provide extensive benefits for designing empirical models for signal validation tasks. The first is its elimination of numerical problems associated with collinearity. A matrix is said to be collinear if the columns in X are approximately or exactly linearly dependent. Collinearity means that the matrix X will have some dominating types of variability that carry most of the available information [8]. A square matrix is said to be singular if there is at least one linear dependency among the rows (or columns) of the matrix. For the case of a singular matrix the determinant is zero: I X = 0 . The rank of a matrix is defined as the maximum number of linearly independent columns (or rows). Based on this, matrices containing linearly dependent columns are said to be rank-

x

I

286

P. F. Fantoni

et al.

deficient, or ill-conditioned. In relation to empirical modeling, the matrix of concern

XTX. Any linear dependence that exists in a matrix x will be preserved in the new matrix XTX. Situations where the determinant of XTX is near zero lead to is

unstable estimates of the regression coefficients, in multiple linear regression (MLR), which may be unreasonably large or have the wrong sign [27]. When the determinant of a matrix is zero its inverse does not exist, and when it is near zero, its inverse contains values of extremely large magnitude. Collinearity in a data set leads to an ill-posed problem that causes inconsistent results when data based models such as MLR or neural networks are used [4]. Data sets from large-scale processes, such as an electrical generating facility, are often plagued with collinearity. Thus, MLR is not a suitable method of empirical modeling for these situations, due to the required matrix inversion to obtain the solution. The use of autoassociative artificial neural networks (AANN), used extensively for signal validation [28-3 13, is also inhibited by collinearity for similar reasons [32]. Some type of regularization method should be utilized when dealing with collinear data. Regularization can be integrated into neural network development through the use of cross validation training and robust training techniques however, these methods are time consuming and require significant oversight and knowledge [33-341. Other methods of regularization include truncated singular value decomposition (TSVD), and ridge regression (RR) [35-361. The utility of the PLS algorithm in eliminating the problems associated with collinearity is an indispensable feature of the method. Though this feature is also available in PCR, the orthogonal projections to latent structures in PCR is unsupervised with regard to the response variables, focusing only on the variability contained in the predictor variable set. PLS on the other hand is supervised in that its orthogonal projections are performed to capture the maximum covariance between the predictor variables and the desired response, in its latent variables. The first latent variable contains the maximum covariance in a given direction and subsequent latent variables contain the maximum covariance remaining, in a direction orthogonal to all of the previous latent variables. The second beneficial feature of PLS is its elimination of the higher ordered latent vectors from passing through to the inner relationships of the model. These higher order latent variables contain decreasing amounts of variance related to the response variable block, Y . This inherent regularization, via supervised dimensionality reduction, provides reproducible and stable solutions. The benefits associated with regularization, to stabilize the solutions of ill-posed problems, are making its incorporation into the design of signal validation systems essential [37]. There are two sets of relationships involved in the PLS technique, the outer relationships and the inner relationships. Consider an rn x n block of predictor variables, X , and an m X 1 response variable, Y . Note that the restriction of the response variable matrix to a single variable is the case for PLS-1. A schematic of the PLS-I algorithm is diagram of the mapping structure of the PLS-1 model is

Non-linear Partial Least Square Methods for On-line Process Monitoring

287

x

The outer relationships are linear relationships between the predictor variables, , and the latent variables for the same block. The creation of latent variable pairs is an iterative process. The diagram of the mapping performed by a single iteration of the PLS-1 algorithm provides further illustration.

Figure 1. Structure of the PLS Method

Following the creation of a latent variable pair, the predictor variables are successively deflated by the information extracted by the current latent variable pair. For this reason, the observed variable block is often referred to as a residual matrix, and herein will be referred to as the input residual matrix. Similarly there is an output residual matrix created by the deflation of the response variable matrix at the end of each iteration. The deflation operations are not an essential step in the PLS algorithm [38] however, the residual matrices provide information regarding the amount of variance contained in each successive set of latent variable pairs [19]. The outer relationships for the predictor variables and the subsequent deflation of the input residual matrix are given below. Define: E = input residual matrix f = output residual matrix W= predictor variable transformation weights t= input latent variable b = regression coefficient P= input loading vector

a = 1,..., R (iteration index) j = 1,...,n (column index) The outer relationship can be explicitly written as:

The deflation of the input residual matrix is given by:

288

P. F. Fantoni et al. 0

x,

Note that E = subscripts indicate the matrix indices, and superscripts indicate the iteration index. The number of iterations of the outer relationship, a , dictates the number of pairs of latent variables, and constitutes selection of the model. The number of latent variable pairs computed is the rank of the PLS model, R , and is often determined via a cross validation technique [9, 401. The PLS-1 algorithm does not require the transformation of the response variable, since it is a rank one matrix. Hence, there is no corresponding outer relationship for the response variable. The optional task of deflation of the output residual vector may still be carried out if desired, to provide a quantification of the variability explained by each consecutive input latent variable. The deflation of the output residual is given by:

Note that

f

= y , and

b" is the coefficient from the univariate regression of

t" onto f ". All of the discussions regarding the input outer relationships of the PLS algorithm also apply to the NNPLS algorithm. Differences between the methods occur in the inner relationships, and slightly modify the equation for the deflation of the output residual. The inner relationships of the PLS-1 algorithm are a set of univariate regressions between each latent variable pair up to the rank of the model. Substituting these univariate regressions with feedforward neural networks provides the algorithm with nonlinear mapping capabilities, and will often produce a more parsimonious model [19]. Following the method of S.J. Qin and T.J. McAvoy, each latent variable pair is mapped with a single input single output (SISO) neural network [20]. The use of very simple neural networks circumvents the overparameterized problem of the direct neural network approach. The combination of the PLS-1 linear outer relationships and the neural network inner relationships will be referred to as NNPLS- 1. 2.2

The NNPLS Approach

The neural networks used in the inner relation contain a single hidden layer with two neurons containing hyperbolic tangent activation functions, and a single output neuron with a linear activation function. The neural networks were trained with the scaled conjugate gradient (SCG) training algorithm [40]. Cross validation training was used for early stopping of the SCG algorithm. Initialization of the neural network weight and bias values can be based on the univariate regression solution. This method of initialization will decrease the time required to train the neural networks, and due to the SCG training algorithm finding the nearest local minimum from the initial point, the solution will be better than the initial linear model [20].

Non-linear Partial Least Square Methods for On-line Process Monitoring 289

Each latent variable pair requires a SISO neural network. The neural network weights and biases for each set of score vectors are determined during the current iteration and the explained variation is subtracted from the output residual via network simulation. The decision to use two hidden neurons was to provide the neural networks with a significant number of degrees of freedom without increasing the complexity of the model beyond that which is required. Through experimentation, it was noted that two hidden neurons often provided a better solution than a single hidden neuron, and greater than two hidden neurons did not result in improvements great enough to warrant their inclusion. For the purposes of a large-scale signal validation system the required architecture is autoassociative such that for a given set of input variables, the model provides a prediction at the output for each variable in the input set. For each variable a separate inferential NNPLS-1 model is required. The combination of a full set of inferential models mimics the standard autoassociative architecture.

2.3

The PEANO System

Artificial Neural Networks (ANN) and Fuzzy Logic can be combined to exploit learning and generalization capability of the first technique with the approximate reasoning embedded in the second approach [41]. Real-time process signal validation is an application field where the use of this technique can improve the diagnosis of faulty sensors and the identification of outliers in a robust and reliable way. PEANO [42-481 implements a fuzzy-possibilistic clustering algorithm [49-521 to classify the operating region in which the validation process has to be performed. The possibilistic approach (rather than probabilistic) allows a "don't know" classification that results in a fast detection of unforeseen plant conditions or outliers. The fuzzy classifier identifies the incoming signal pattern (a set of reactor process signals) as a member of one of the clusters covering the entire universe of discourse represented by the possible combinations of steady-state and transient values of the input set in the n-dimensional input world. Each cluster is associated with one ANN that was previously trained only with data belonging to that cluster, for the input set validation process. During the operation, while the input point moved in an n-dimensional world (because of process state changes or transients), the classifier provides an automatic switching mechanism to allow the best tuned ANN to do the job. There are two main advantages in using this architecture: the accuracy and generalization capability is increased compared to the case of a single network working in the entire operating region and the ability to identify abnormal conditions, where the system is not capable of operating with a satisfactory accuracy, is improved.

290

P. F. Fantoni et al.

The structure of PEANO is shown in Figure 2. PEANO has a Client-Server architecture. The server is connected to the process through a TCPhP communication protocol and the results of the validation activity are transferred to the client programs, also using TCP/IP.

I

I

I

I

Figure 2. The structure of PEANO

Figure 3 shows the display of a PEANO client during an on-line validation test. The error bands in the mismatch plots are calculated by PEANO during the training according to the expected error of prediction for each particular cluster and signal [53].The error bands should be interpreted as follows: Narrow error band : It is normally set at 2 standard deviations of the expected error. Exceeding this band is considered as the first warning, especially if the situation persists. Wide error band : It is normally set at 3 standard deviations of the expected error. Exceeding this band is considered a definite alarm condition. 2.4

Tests using ANN and NLPLS inside PEANO

Figure 4 to Figure 6 describe some results achieved with PEANO-ANN, using real data from a US PWR nuclear plant (data provided by EPRI). PEANO was used to monitor 55 process signals up to the secondary side of the steam generators. The data was taken from 3 months of operation (March to May 2000) at different operation conditions.

Non-linear Partial Least Square Methods for On-line Process Monitoring

291

Figure 3. The Client display of PEANO

Figure 4 shows the re-calibration of a steam flow sensor while the plant was in operation. It is visible that after this intervention, the mismatch was well within the 2 standard deviation limit. Figure shows the monitoring of a steam line pressure sensor during steady-state operation at full power. Note how the system acts like a low-pass filter, removing process and instrument noise. The error is always at the center of the tolerance band. Figure 6 is an interesting example of span drift of a steam flow sensor. Span drifts are difficult to detect because they show up only at some location of the instrument range. In this real-life example, the instrument was perfectly inside the calibration range until the power level came close to the rated level (high end of the instrument range). At this point the instrument started to drift and eventually finished outside the allowed tolerance band. At the plant, this drift was discovered only one month later. In the PEANO-NLPLS, the bank of ANN’S has been modified to use the NLPLS algorithm described in this paper. The total development time for this application, using the same EPRI data, was 0.5 hours, while for the PEANO-ANN it was 3 days. This dramatic improvement in the training time makes any retraining need possible and easy to perform. Figure 1 and Figure 2 the same tests as in Figure 4 and Figure 5, with the NLPLS algorithm in PEANO. The performance and accuracy is in the same range and in both cases the miscalibrated instruments were detected correctly.

292

3

P. F. Fantoni et al.

Conclusions

This paper shows how an approach based on NLPLS techniques can estimate and predict efficiently the behavior of a non-linear process, in comparison with a traditional approach based on ANN. The use of Non Linear Partial Square Methods is expected to improve the performance of a signal validation and estimation system, solving some of the problems embedded in any ANN approach, like retraining capability, training speed and large scale applications. Steam flow FT-1-28A. April 2 0 0 0

time (min) Mismatch

0

200

400

600 time (min)

aoo

1000

Figure 4. Drift detection with PEANO-ANN

1200

Non-linear Partial Least Square Methods f o r On-line Process Monitoring 293 Steam line pressure PT-1-2B, April 2 0 0 1 845,

I

830 1

I

I

200

400

I

I

200

400

,

I

I

1

800

1000

1200

I

I

I

I

600

800

1000

I200

600 t i m e ( m in )

Mismatch 4

-4

time (min)

Figure 5 . Noise removal with PEANO-ANN

Steam flow, end of March 2000

80

I

0

I I

I I

I

I

I

I

I

,

I

I

I

I

2000

4000

6000

8000

10000

12000

14000

10000

12000

14000

time (min) Mismatch

0

2000

4000

6000 8000 time (min)

Figure 6 . Span drift detection with PEANO-ANN

294

P. F. Fantoni et al.

Figure 1. NLPLS drift detection (compare with Fig.3)

Steam Line Pressure, April 2001, PT-1-28

845 i

'

-10 0

1

200

400

600

800

1000

Figure 2. NLPLS noise removal (compare with Fig. 4)

I

1200

Non-linear Partial Least Square Methods for On-line Process Monitoring

295

References 1. Wold H. (1966), "Non-linear Estimation by Iterative Least Squares Procedures," In: Research Papers in Statistics, David, F. (Ed.), Wiley, NY. 2. Ranpar, S., Lindgren, F., Geladi, P., and S. Wold (1994), "A PLS Kernel Algorithm for Data sets with Many Variables and Fewer Objects, Part 1: Theory and Algorithm," Journal of Chemometrics, 8, 111-125. 3. Wegelin, J.A. (2000), "A Survey of PLS Methods, with Emphasis on the TwoBlock Case," University of Washington Technical Report No. 371, Seattle. 4. Gil, J.A., and R. Romera (1998), "On Robust Partial Least Square (PLS) Methods," Journal of Chemometrics, 12,365-378. 5. Geladi, P., and B. R. Kowalski (1986), "Partial Least Squares Regression: A Tutorial," AnaIytica Chimica Acta, 185, 1-17. 6 . Hoskuldsson, A.( 1988), "PLS Regression Methods," Journal of Chemometrics, 2,211-228. 7. Hoskuldsson , A. (1995), "A Combined Theory for PCA and PLS," Journal of Chemometrics, 9, 91-123. 8. Martens, H., and T. Naes (1989), Multivariate Calibration, John Wiley and Sons, Chichester. 9. Hines, J.W., A.V. Gribok, I. Attieh, and R.E. Uhrig (1999), "Regularization Methods for Inferential Sensing in Nuclear Power Plants," Fuzzy Systems and Soft Computing in Nuclear Engineering, Ed. Da Ruan, Springer. 10. Wold, S., Ruhe, A., Wold, H., Dunn, W.J. 111(1984), "The collinearity problem in linear regression: The partial least squares approach to generalized inverses," SIAM J. Sci. Stat. Comput., vol. 5, pp. 735-743. 11. Baffi, G., Martin, E.B., Morris, A.J.(1999a), "Non-linear projection to latent structures revisited: the quadratic PLS algorithm," Computers in Chemical Engineering vol. 23, pp. 395-41 1. 12. Ni, Y., and Z. Peng (1995), "Determination of Mixed Metal Ions by Complexometric Titration and Nonlinear Partial Least Squares Calibration," Analytica Chemica Acta, 304,217-222. 13. Wold, S., N.K. Wold, and B. Skagerberg (1989), "Nonlinear PLS Modeling," Chemometrics and Intelligent Laboratory Systems, 7, 53-65. 14. Wold, S. (1992), "Nonlinear Partial Least Squares Modeling 11. Spline Inner Relation," Chemometrics and Intelligent Laborato y Systems, 14, 7 1-84. 15. Bro, R. (1995), "Algorithm for Finding an Interpretable Simple Neural Network Solution Using PLS," Journal of Chemometrics, 9,423-430. 16. Gemperline, P.J., Long, J.R., and V.G.Gregoriou (199 l), Anal Chem., 63,23 13. 17. Baffi, G., Martin, E.B., Morris, A.J. (1999), Non-linear projection to latent structures revisited: the neural network PLS algorithm," Computers in Chemical Engineering vol. 23, pp. 1293-1307. 18. Holcomb, T.R., and M. Morari (1992), "PLS/Neural Networks," Computers Chem. Engng., 16,393-41 1. 'I

296

P. F. Fantoni et al.

19. Malthouse, E.C., A.C. Tamhane, and R.S.H. Mah (1997), "Nonlinear Partial Least Squares," Computers Chem. Engng., 21, 875-890. 20. Qin, S.J., and T.J. McAvoy (1992), "Nonlinear PLS Modeling Using Neural Networks," Computers in Chemical Engineering, vol. 16, n. 4, pp. 379-391. 21. Qin, S.J., and T.J. McAvoy (1996), "Nonlinear FIR Modeling via Neural Net PLS Approach," Computers chem. Engng., 20, 147-159. 22. Wilson, D.J.H., G.W. Irwin, and G. Lightbody (1997), "Nonlinear PLS Modeling Using Radial Basis Functions," Proceedings of the American Control Conference, Albuquerque, NM, pp. 3275-3276. 23. Hines, J.W., B. Rasmussen, and R.E. Uhrig (2000), "An On-line Sensor Calibration System," Presented at the 13'h International Congress and Exhibition on Condition Monitoring and Diagnostic Engineering Management, Houston Texas. 24. Hines, J.W., and B. Rasmussen (2001), "Continuous Calibration Verification at a Coal Fired Power Plant," Presented at ICOMS, The International Conference of Maintenance Societies, Melbourne, Australia, June. 25. Rasmussen, B., J.W. Hines, and R.E. Uhrig (2000a), "Nonlinear Partial Least Squares Modeling for Instrument Surveillance and Calibration Verification," by published in the proceedings of the Maintenance and Reliability Conference (MARCON), Knoxville, TN, May 7-10. 26. Rasmussen, B., J.W. Hines, and R.E. Uhrig (2000), "A Novel Approach to Process Modeling for Instrument Surveillance and Calibration Verification," Proceedings of The Third American Nuclear Society International Topical Meeting on Nuclear Plant Instrumentation and Control and Human-Machine Interface Technologies, Washington DC, November 13-17. 27. Massart, D.L., B.G.M. Vandeginste, S.N. Deming, Y. Michotte, and L. Kaufinan (1988), Chemometrics: A textbook, Elsevier Science Publishers, Amsterdam. 28. Fantoni, P., S. Figedy, A. Racz, (1998), "A Neuro-Fuzzy Model Applied to Full Range Signal Validation of PWR Nuclear Power Plant Data," FLINS-98, Antwerpen, Belgium. 29. Hines, J.W., A.V. Gribok, 1. Attieh, and R.E. Uhrig (2000), "Neural Network Regularization Techniques for a Sensor Validation System," American Nuclear Society Annual Meeting, San Diego, California, June 4-8. 30. Upadhyaya, B.R., and E. Eryurek (1992), "Application of Neural Networks for Sensor Validation and Plant Monitoring," Nuclear Technology, vol. 97, pp. 170-176. 31. Xu, X., J.W. Hines, and R.E. Uhrig (1999), "Sensor Validation and Fault Detection Using Neural Networks," Proceedings of the Maintenance and Reliability Conference (MARCON), Gatlinburg, TN, May 10-12. 32. Qin, S. (1997), "Neural networks for intelligent sensors and control - Practical issues and some solutions," In Neural Systems f o r Control, Chapter 8, Edited by 0. Omidvar and D. L. Elliott, Academic Press.

N o n - h e a r Partial Least Square Methods for On-line Process Monitoring

297

33. Xu, X. (2000), PhD dissertation, “Automated Neural Network-Based Instrument Validation System,” The University of Tennessee, Nuclear Engineering Department, Knoxville, TN. 34. Hines, J.W., A.V. Gribok, I. Attieh, and R.E. Uhrig (1999), “The Use of Regularization in Inferential Measurements,” Presented at the Enlarged Halden Programme Group (EHPG) Meeting, Loen, Norway, May 24-29. 35. Hoerl, A.E., and R.W. Kennard (1970), “Ridge Regression: Biased Estimation for Nonorthogonal Problems,” Technometrics, 12, 55-67. 36. Hines, J.W., A.V. Gribok, I. Attieh, and R.E. Uhrig (2000), “Improved Methods for On-Line Calibration,” Proceedings of the 8th International Conference on Nuclaear Engineering, Baltimore, MD, April 2-6. 37. Dayal, B.S., and J.F. MacGregor, “Improved PLS Algorithms (1997), Journal of Chemometrics, 11, 73-85. 38. Hoskuldsson, A. (1996), “Experimental Design and Priority PLS Regression, Journal of Chemometrics, 10,637-668. 39. Wold, H. (1982), “Soft Modeling, The Basic Design and Some Extensions,” In Systems Under Indirect Observation, 1-11, K.G. Joreskog and H. Wold Eds., North-Holland, Amsterdam 40. Moller, M. F. (1993), “A scaled conjugate gradient algorithm for fast supervised learning,” Neural Networks, vol. 6, pp. 525-533. 4 1. Fantoni P, Mazzola A (1994) Applications of Autoassociative Neural Networks for Signal Validation in Accident Management. In: Proceedings of the IAEA Specialist Meeting on Advanced Information Methods and Artificial Intelligence in Nuclear Power Plant Control Rooms, Halden, Norway. 42. Fantoni p, Mazzola A (1995) Transient and Steady State Signal Validation in Nuclear Power Plants using Autoassociative Neural Networks and Pattern Recognition. In: Proceedings of SMORN VII, Avignon, France. 43. Fantoni P, Mazzola A (1996) A Pattern Recognition-ArtificialNeural Networks Based Model for Signal Validation in Nuclear Power Plants. In: Annals of Nuclear Energy, Vol. 23, No.13, pp 1069-1076. 44. Fantoni P, Mazzola A (1996) Multiple Failure Signal Validation in Nuclear Power Plants using Artificial Neural Networks. In: Nuclear Technology, Vol. 113, NO. 3, pp 368-374. 45. Fantoni P (1996) Neuro-Fuzzy Models Applied to Full Range Signal Validation in NPP. In: Proceedings of NPIClkHMIT’96, The Pennsylvania State Univ., PA. 46. Fantoni P, Figedy S, Racz A (1998) PEANO, A Toolbox for Real-Time Process Signal Validation and Estimation. HWR-5 15, OECD Halden Reactor Project (restricted). 47. Fantoni P, Renders J (1998) On-Line Performance Estimation and Condition Monitoring using Neuro-Fuzzy Techniques. In: Proceedings of the Workshop on On-Line Fault Detection and Supervision in the Chemical Process Industries, Lyon, France.

298

P. F. Fantoni et al.

48. Fantoni P (2000) A Neuro-Fuzzy Model Applied to Full Range Signal Validation of PWR Nuclear Power Plant Data. In: International Journal of General Systems, Vol. 29(2), pp 305-320. 49. Tou JT, Gonzalez R C (1974) Pattern Recognition Principles. Addison Wesley Publishing Company, Reading, MA. 50. Bezdek JC (1981) Pattern Recognition with Fuzzy Objective Function Algorithms. Plenum Press, New York, NY. 5 1. Gustafson DE, Kessel W C (1979) Fuzzy Clustering with a Fuzzy Covariance Matrix. In: Proceedings of IEEE CDC, San Diego, CA. 52. Krishnapuram R, Keller J (1993) A Possibilistic Approach to Clustering. 1n:IEEE Transactions on Fuzzy Systems, Vol. 1, No. 2, pp 98-100. 53. Fantoni P, Mazzola A (1996) Accuracy Estimate of Artificial Neural Networks Based Models for Industrial Applications. In: Artificial Intelligence in the Petroleum Industry. Symbolic and Computational Applications 2, Chapter 17, B Braunschweig & B Bremdal Editions.

RECURRENT NEURAL NETWORKS FOR REAL-TIME COMPUTATION OF INVERSE KINEMATICS OF REDUNDANT MANIPULATORS JUN WANG AND YUNONG ZHANG DEPARTMENT OF AUTOMATION AND COMPUTER-AIDED ENGINEERING T H E CHINESE UNIVERSITY OF HONG KONG SHATIN, NEW TERRITORIES, HONG KONG EMAIL: [email protected] Recurrent neural networks are discussed for real-time inverse kinematic control of redundant manipulators. Three recurrent neural network models, the Lagrangian neural network, the primal-dual neural network, and the dual neural network. We begin with the Lagrangian neural network for the inverse kinematics computation based on the Euclidean norm of the joint velocities t o show the feasibility. Next, we present the primal-dual neural network for minimum infinity norm kinematic control of redundant manipulators. To reduce the model complexity and increase the computational efficiency, the dual neural network is developed with the advantages of simple architecture and exponential convergence. Simulation results based on the PA10 robot manipulator substantiate the effectiveness of the present recurrent neural network approach.

Keywords: kinematically redundant manipulators, recurrent neural networks, inverse kinematics. 1

Introduction

Kinematically redundant manipulators are those having more degrees of freedom than required to perform manipulator given tasks. The redundancy of such manipulators includes intrinsical redundancy and functional redundancy [ 11. The use of kinematically redundant manipulators is expected to increase dramatically in the future because of their ability to avoid obstacles, joint limits, and singularities, and to optimize various performance criteria, while conducting the end-effector motion task. The real-time computation of inverse kinematics solutions is very timeconsuming in high degree-of-freedom sensor-based robotic systems, especially for the case when multiple performance criteria and/or dynamic physical constraints are considered, due to the time-varying nature and the real-time calculation requirement. Parallel computation methods such as neural network approaches are effective and efficient alternatives for inverse kinematics solu-

299

300

J . W a n g and

Y.Zhang

tion computation [10]-[12]. In recent years, new interests in neural network research have been generated to reduce the computational complexity and t o improve the computational efficiency for kinematic control of redundant manipulators. Various neural network models have thus been developed including feedforward networks and recurrent neural networks [13]-[28].Unlike feedforward neural networks, most recurrent neural networks do not need off-line supervised learning and thus are more suitable for real-time robot control in uncertain environments. For example, Wang et aZ[19] developed a recurrent neural network, called the Lagrangian network, t o resolve manipulator redundancy at the velocity level. The Lagrangian network generates real-time solution t o the inverse kinematics problem formulated as a time-varying quadratic optimization problem. In contrast t o the minimum two-norm kinematic control scheme present in [18][19], a primal-dual neural network together with a pseudo-inverse neural network was developed in [18] and a primal-dual neural network with reduced architecture complexity was developed in [25] by minimizing the infinity-norm of robot joint velocities. To reduce model complexity, a single-layer dual neural network model was first presented in [26] for the inverse kinematics of redundant manipulators. The number of neurons of the dual network is just equal to the dimensionality of the robot joint, and moreover the dual neural network was extended for bi-criteria kinematic control of redundant manipulators in [28]. The dual neural network was proved t o be globally exponentially stable [26]-[28]. The remainder of this chapter is organized in four sections. The problem formulation and preliminaries are presented in Section 2. The Lagrangian neural network, the prime-dual neural network and the dual neural network are reviewed respectively in Subsections 3.1, 3.2 and 3.3 of Section 3. Simulation results based on the PA10 intelligent robot arm are illustrated in Section 4. Section 5 concludes this chapter with final remarks. 2

Problem formulation

The forward kinematics equation in robotics is concerned with the transformation of position and orientation information in a joint space to a Cartesian space described as:

where O ( t ) is an n-vector of joint variables, r ( t ) is an m-vector of Cartesian position and orientation variables, and I ( . ) is a continuous nonlinear function with a known structure and parameters for a given manipulator.

Real-time Computation of Inverse Kinematics of Redundant Manipulators 301

The inverse kinematics problem is t o find the joint variables given the desired positions and orientations of the robot end-effector through the inverse mapping of ( 1):

e ( t ) = f-l(T(t)). Solving the inverse kinematics problem of redundant manipulators is of vital importance in robotics [1]-[9]. The inverse kinematics problem involves the existence and uniqueness of a solution, and effectiveness and efficiency of solution methods. The inverse kinematics problem is thus much more difficult to solve than the forward kinematics problem for serial-link manipulators. The difficulties are compounded by the requirement of real-time solution in sensorbased robotic operations. Therefore, real-time solution procedures to the inverse kinematics problem of redundant manipulators are of very importance in robotics. The most direct way to solve (2) is t o derive a closed-form solution from (1). Unfortunately, obtaining a closed-form solution is difficult for most manipulators due to the nonlinearity of f(.). Moreover, the solution is often not unique for redundant manipulators due to the extra degrees of freedom. Making use of the relation between joint velocity 6 ( t ) and Cartesian velocity +(t)is a common indirect approach t o the inverse kinematics problem; i.e.,

where J ( e ) E R m x nis the Jacobian matrix defined as J ( 0 ) = af(0)/%. In a redundant manipulator, (1) and (3) are underdetermined since m < n, and hence they may admit infinite number of solutions. The indirect approach begins with the desired velocity of the end-effector +(t )based on a planned trajectory and required completion time T . The corresponding joint vector O(t) is obtained by integration of e(t)for a given e(0). The resulting e ( t ) then is used t o control the manipulator. Much effort has been devoted to numerical solutions of the inverse kinematics problem. For example, in the table lookup method the inverse Jacobian matrix is stored in memory a priori instead of computing in real time. In the pivot method, the inverse Jacobian is broken down into manageable submatrices. In the extended pivot method, the joint velocity is obtained by directly computing the joint velocities. In the residue arithmetic method, the pseudoinverse of the Jacobian is computed in a parallel manner. In the leastsquares method, e(t) is computed directly without solving the pseudoinverse of Jacobian explicitly. Other numerical methods such as the Newton's method are also developed for solving the inverse kinematics problem. For a review and an in-depth discussion, see [4][5].

302

J . Wang and Y. Zhang

Since 6 is underdetermined in a kinematically redundant manipulator, another way to determine e(t) without computing the pseudoinverse is to online solve the following time-varying optimization problem with equality constraints: minimize h ( e ( t ) ) , subject to J ( f ? ) e ( t= ) T(t),

(4)

where h(b(t))is a general performance measure and can be selected as eTWe/2 with W E ?JPxnbeing a symmetric positive-definite weighting matrix or := maxi l&l. The superscript denotes the transpose operator. If W is the identity matrix I, then the objective function to be minimized is equivalent t o the Euclidean-norm of joint velocity Ile(t)II$.If W is the inertia matrix, then the objective function to be minimized is the local kinetic energy. The problem formulation (4)is only a basic and common form for the involved inverse kinematics problems in the chapter. The formulation may become more realistic when considering more performance criteria or adding more physical constraints such as joint limits and joint velocity limits.

llelloo

Figure 1. Block diagram for the neural network based kinematic control process (Tang and Wang, A recurrent neural network for minimum infinity-norm kinematic control of redundant manipulators with an improved problem formulation and reduced architectural complexity, IEEE Trans. Syst., Man, Cybern., 0 2 0 0 1 IEEE).

Various neural network models have been developed in the past decade for inverse kinematics of redundant manipulators. In the early 1 9 9 0 ’ ~the ~ research focused on feedforward neural networks such as the multilayered Perceptron trained via supervised learning using the backpropagation algorithm or its variants, which need off-line training and may violate the real-time computation requirement of industrial robots. Since the rnid-l990’s, recurrent neural networks with feedback connections have also been applied to kinematic control. In particular, three recurrent neural network models are discussed successively in the following section; i.e. the Lagrangian network, the primal-dual network and the dual neural network in [19][25]-[28].Fig. 1 delineates the kinematic control process based on recurrent neural networks.

Real-time Computation of Inverse Kinematics of Redundant Manipulators

3 3.1

303

Recurrent Neural network models The Lagrangian neural network

In this subsection, the performance measure of (4)is eTWe/2 t o show the development and stability property of a Lagrangian neural network for online manipulator redundancy resolution. The Lagrangian of the time-varying equality-constrained quadratic program (4)is defined as:

where A ( t ) is an rn-dimensional column vector of Lagrangian multipliers at time t. By setting the partial derivatives of L(6,A) t o zero, the Lagrange necessary condition gives rise to the following time-varying algebraic equations:

It can be shown that the optimality condition is also sufficient. Multiplying both sides of (6) by -1, then rewriting (6) and (7) in a combined matrix form, we have

Let the state vectors of output neurons and hidden neurons be denoted by v ( t ) and u ( t ) , an n-vector representing estimated e ( t ) and an rn-vector representing estimated A ( t ) , respectively. The dynamic equations of the proposed Lagrangian network can be expressed by the following time-varying linear differential equations:

ClW(t)= -Wv(t) - J(O)Tu(t), CZiL(t)= J ( Q ) v ( t ) i(t), where C1 E X n x n and C2 E X m x m are positive diagonal capacitive matrices. The positive diagonal capacitive matrices C1 and C2 are used to precondition the system matrix and scale the convergence rate of the Lagrangian network. Fig. 2 illustrates the kinematic control process based on the Lagrangian network. In this context, the desired velocity vector T ( t ) is input into the Lagrangian network, and a t the same time the Lagrangian network outputs the computed joint velocity vector e(t). In details, (9) shows that the symmetric connection weight matrix among the neurons represented by v ( t ) is -W, and that the time-varying connection weight matrix from the neurons represented

304 J . Wang and Y. Zhang

by u(t) t o the neurons represented by w ( t ) is - J ( 0 ) T . (10) shows that the time-varying connection weight matrix from the neurons represented by v ( t ) to the neurons represented by u ( t ) is J(O), and that the external input vector to the hidden layer is -+(t).

Figure 2. Block diagram of the Lagrangian network for robot kinematic control (Wang, Hu, and Jiang, A Lagrangian network for kinematic control of redundant manipulators, IEEE Trans. Neural N e t . , 0 1 9 9 9 IEEE).

Before obtaining the global stability results summarized below, the Lagrangian network is written as the following time-varying linear dynamic system:

Given an initial point (0, we say the solution ( ( t )of the system starting from &, is stable if for any positive real number S > 0, there exists a positive real number E > 0, such that for any initial point ((0) in the &-neighborhoodof (0, the corresponding solution of (11) remains in the S-neighborhood of ( ( t )for t E [0, +m). We also say that the v state of the solution ( ( t )is asymptotically stable if the v part of components of the corresponding solution converges to v ( t ) as t + +co. As the system defined in (11) is linear, these types of stability is equivalent to the stability of zero solution to the corresponding homogeneous system, namely, the system without b ( t ) term. The Lagrangian network defined in (11) is shown t o be globally stable. Furthermore, 21 part of the state vector is globally asymptotically stable [19]. The above analytical results show that no pole of the neural system is located on the right-hand side or on t,he imaginary axis of the complex plane, and only one pole is located at the origin if the rank of A(t) is n m - 1. Therefore, if the redundant manipulator loses a t most one degree of freedom

+

Real-time Computation of Inverse Kinematics of Redundant Manipulators

305

+

at any time (i.e., rank(A(t)) 2 n m - l), the Lagrangian network with sufficiently large gains (i.e., sufficiently small values of C1 and 6 ' 2 ) has the asymptotic tracking capability. At the limiting state, limt-tco tj(t) = limt+oo G ( t ) = 0,

which satisfies the Lagrange optimality condition (8).

3.2

The primal-dual neural network

The minimum two-norm of joint velocity vector, as discussed in Subsection 3.1, is widely used as an optimization criterion in many robotics applications. The two-norm optimization scheme minimizes the squared sum of joint velocities, which does not necessarily minimize the magnitudes of individual joint velocity. It is used as the optimization criterion by researchers more because it is mathematically tractable than physically desirable [4]. The minimum infinity-norm of joint velocities (i.e., I18(t)llm),however, minimizes the largest component of the joint velocity vector in magnitude and is consistent with the physical velocity limits. Moreover, the minimization of infinity-norm of joint velocity vector enables a better direct monitoring and control of the magnitude of individual joint velocities than that of 2-norm of joint velocities [9]. It is therefore more desirable in the situation where low individual joint velocity is of primary concern; e.g., robotic surgery. In this subsection, we first convert the infinity-norm minimization problem (4) into a linear program which can be solved by the primal-dual neural network. For 8 = [el,8 2 , . . . , E !Rn, its infinity-norm is defined as

where I . I denotes the absolute value of the component, and e j E Rn is the j t h column of the identity matrix I . Let the objective function I16(t)llcoin (4) be s ( t ) ;i.e., s ( t ) = m a x l s j s n Ie,Te(t)I. The minimum infinity-norm inverse-kinematics problem (4) is then equivalent to minimize s ( t ) , subject to leTe(t)I

5 s(t) ,

J ( e ) e ( t )= i.(t),

306

J . W a n g and Y . Zhang

which can be re-formulated as minimize s ( t ) ,

where i := [l,1,.. . , 1IT and 6 := [O,O,. . . ,0IT denote respectively the one and null vectors with appropriate dimensions. Rewriting (12) in a standard matrix form, we have minimize c T y , subject to A l y 2 b l , A2Y = b2,

where

A2

= [J(e),O] E SRmx(n+l),

b2 = f ( t ) E SR”,

c = [O,O,.. .,O,1IT E

?Rn+l,

Now, by the dual theory [29] the dual linear program corresponding t o the linear program (13) is

maximize bTz2, subject to ATzl ATz2 = c, z2 unrestricted,

+

z1

> 0,

(14)

where z1 E SRZn and 22 E !Rm are the dual decision variables. In view of the primal and dual programs (13) and (14), define the following energy function:

1 1 (A1y)T (Aly - IAlyI) + ,zT(zl - 1211). (15) 4 The first term in (15) is the squared duality gap. The second and third terms are for the equality constraints respectively in (13) and (14). The fourth and last terms are for the nonnegativity constraints respectively in (13) and (14). Clearly, the energy function (15) is convex and continuously differentiable. It can be seen that E ( y * , z ; , z;) = 0 if and only if (y*, z ; , z;) is the optimal solution of the primal and dual linear programs defined in (13) and (14). -

Real-time Computation of Inverse Kinematics of Redundant Manipulators 307

Figure 3. Block diagram of the primal-dual neural network for robot kinematic control (Tang and Wang, A recurrent neural network for minimum infinity-norm kinematic control of redundant manipulators with an improved problem formulation and reduced architectural complexity, IEEE Trans. Syst., Man, Cybern., 0 2 0 0 1 IEEE).

In view of the energy function (15), the dynamics for the neural network solving (13) and (14) can be defined by the negative gradient systems. Note that 21 := [yT,zT,zT]T, and that for any column vector 2, z - 1x1 = 2 2 where 2- = [x,, 2 2 , . . . , and xi = min(0, xi}. The network dynamical equations can be thus expressed as:

where p E 8 is a strictly positive parameter used to scale the convergence rate of the network and should be selected as large as possible. Fig. 3 shows the block diagram for the architecture of the primal-dual neural network. In this context, the desired velocity vector of the end-effector i ( t ) are fed into the neural network, and the neural network generates the command signal y at the same time which contains the minimum infinity-norm joint velocity vector 4. The stability property of the primal-dual neural network is as follows [ 2 5 ] :

308 J . Wang and Y . Zhang

the primal-dual neural network defined in (16)-(18) is proved to be globally stable and convergent to the optimal solutions of the linear programs (13) and (14).

3.3

The dual neural network

Though the Lagrangian neural network is effective to solve equalityconstrained quadratic programs, it is known that when solving inequalityconstrained quadratic programs, the Lagrangian network may exhibit the premature defect, and that the dimensionality of Lagrangian network will be much larger than that of original problems due to the introduction of slack and surplus variables [12][19]. The primal-dual neural network present in Subsection 3.2 handles the primal program and its dual problem simultaneously by minimizing the duality gap with Kuhn-Tacker conditions. Unfortunately, because of using gradient descent method, the dynamic equations of the primal-dual neural network are usually very complicated and contain high-order nonlinear terms; e.g., (16)-(18). To reduce scheme complexity and increase computational efficiency, a dual neural network model has been developed for inverse kinematics control. Though the dual network first in [26] is to solve equality-constrained programs only, its design method can be generalized to handle the quadratic programs under equality and bound constraints [27][28]. A practical problem arising in robotics is the joint limit avoidance. Among the previous studies on recurrent neural network based inverse-kinematics [18]-[26],it is always assumed implicitly that there exists no joint limit when solving such inverse kinematic problems. However, joint limits are physical constraints of the work space of a robot and they do exist for all kinds of robot. If a solution exceeds the mechanical joint rotation limit, the desired path becomes impossible to follow and the solution is then inapplicable. Therefore, in this section the feature of joint limit avoidance, as a basic requirement, is considered into the redundancy resolution scheme (4); i.e., the following time-varying optimization problem 1. minimize - ~ ( t > ~ ~ b ( t > , 2 subject to J ( 0 ) b ( t )= T ( t ) , 0- 5 0 ( t ) 5 o+,

(19)

The limited joint range [0-, 0+] in (19) has t o be converted to some kind of dynamically-updated bound constraints on joint velocity variables theta.

Real-time Computation of Inverse Kinematics of Redundant Manipulators

The proposed effective conversion scheme is t o replace 8C ( t ) 5 &t) I

5 O(t) 5 O+

r+w

309

with

(20)

where r - ( t ) := a(pO- - O ( t ) ) and [+(t) := (.(PO+ - O(t)). The coefficient 0 < p < 1usually being 0.99 defines the critical areas [O-,pO-] and [pO+,O+] t o prevent the robot arm from hitting its joint limit. The coefficient a > 0 usually being 1 for the ensuing PA10 simulations, of which the maximal allowable value is limited by bounds on joint velocity or actuator parameters, determines the deceleration magnitude when robot entering such critical areas. The repelling mechanism at the joint limits is based on the existence of critical areas and the dynamical updating of joint velocity bound constraint. For any i, if the ith joint variable Oi(t) is within its feasible range (pO;,pO:), then & : 5 0, [+ 2 0, and with an appropriate value of a , it follows from (20) that e i might be positive or negative solution to the inverse kinematic problem (19) if existent. But if O i ( t ) arrives at the critical area [PO:, O:], then 50 and 5 0; that is, the bound constraint (20) becomes &' 5 0, which implies a deceleration opposite to the original joint movement and drives the i t h joint away from its physical limit. So does the other critical area [O;, PO;]. In summary, the inverse-kinematic problem for limited-joint-range redundant manipulators is re-formulated as 1. minimize -e(t)TwO(t), 2 subject to J(O)e(t)= i.(t),

<-

5 4 t ) 5 r+,

which is re-written in the following quadratic programming form: 1 minimize - 0 ' ( t ) ~ B ( t ) , 2 subject to J ~ e ( tE) 0, where y- = [i.', (<-)' I T, y+ = [i.', (<+)'IT, thus := {u E Rm+nly- 5 u 5 y+} and J E := [ J T , I T ] ' . Now we consider the equivalent formulation about the optimal solution to (21). By the Karush-Kuhn-Tucker condition [30], we know that 0 is a solution to (21) if and only if there exists the dual decision vector u E Rm+nsuch that W e - Jgu = 0 and

ri, + [ J d l i = ~i , [ ~ , d ]=i

7%:5 [J&i 5,:y

if ui > 0, if ui < 0, if ui = 0,

310

J . Wung and Y. Zhang

Figure 4. Block diagram of the dual neural network for robot kinematic control.

which is equivalent to the system of piecewise equations J E = ~ g(JE8 - u) [11][31][32],where the projection operator g(u) = [gl(u1), . . . ,gm+n(um+n)]T is defined as -

gi(ui) =

{

yi

if ui < ~ ~ 7 ,

ui 7:

if 7%: 5 ui 5 7+, i = I , 2 , . . . ,m if ui > 7+,

+ n.

Therefore, 6 is a solution t o (21) if and only if there exists u E Rm+nsuch that W e - Jgu = 0 and J E =~ g(JE8 - u);that is,

{

8 = W-l J

~u, g(JEW-l Jgu - u)= JEW-' Jg'u,

which gives rise to the dual neural network for solving (21) with the following dynamics

zi = p(g(JEW-1Jgu - u)- JEW-1J;u)

(22)

where p E R ! is a positive design parameter used to scale the convergence rate of the network. The dynamic equation described in (22) shows that the dual neural network, even with the extra n joint bound constraints, is composed of only one layer of m n neurons and without any multiplier, penalty parameter or high-order terms. Fig. 4 shows the block diagram of the dual neural network for inverse kinematics. A special case of the proposed dual neural network (22) was first developed in [26] for solving the time-varying quadratic program with equality constraints only. That is, without considering the bound constraint (20), we have the following dynamic equation

+

zi = -pJW-'JTu

+ p?,

Real-time Computation of Inverse Kinematics of Redundant Manipulators

311

since R := {+}, JE := J , u E Rm, and CJ(JEW-'J;U - u) = +. Clearly, the size of the simplified dual neural network is only m. Compared with the Lagrangian network, the architecture of the dual network is reduced to m / ( n m). For example, if n = 7 and m = 6 , then the complexity reduction is about 54%. Some related definitions are first presented. A neural network is said t o be globally convergent if starting from any initial point taken in a given domain of the whole associated Euclidean space, every trajectory of the corresponding dynamic system converges t o an equilibrium point that depends on the initial state of the trajectory. Furthermore, the system is said to be globally exponentially convergent if every trajectory starting from any initial point z ( t 0 ) in a given domain satisfies Ilz(t)- z*ll 6 K I l z ( t O ) - z*ll exp(-v(t - t o ) ) , Vt 2 to 2 0 where K. and r] are strictly positive constants, z* is an equilibrium point depending on initial states. Under the above definitions, the convergence results of the dual network model (22) are obtained as follows

+

~71.

Starting from any initial state, the dual neural network (22) is exponentially convergent to an equilibrium point u*,which together with 8* = W-'J;u* is an optimal solution (u*,8*)t o the constrained inverse kinematic problem (21).

TABLE I JOINTr

6 7

. 3 ~ ~ OF ~ 5T H E

wrist 1 wrist 2

PAL0 M&NIPIILATOX

pivoting rotating

firad iZnrad

Figure 5. Mechanical configuration of the PA10 robot arm (Wang, Hu, and Jiang, A Lagrangian network for kinematic control of redundant manipulators, IEEE Trans. Neural Net., 0 1 9 9 9 IEEE).

312

4

J . Wang and Y. Zhang

Simulation results

Simulations of the recurrent neural networks have been performed for motion control of the PA10 manipulator (a Portable General Purpose Intelligent Arm) made by Mitsubishi [19]. The PA-10 manipulator has seven degreesof-freedom (three rotation axes and four pivot axes) as illustrated in Fig. 5 with joint limits O* shown in Table 1. The homogeneous transformation matrices and the Jacobian are shown in the Appendices. The key specifications of the PA-10 manipulator are as follows: the full length of the manipulator is 1345mm, the weight of the manipulator is 35kgf, the maximum combined speed with all axes is 1.55m/sec, the payload weight is lOkgf, and the output torque is 9.8Nm. 4.1

Simulation results of the Lagrangian network and the prima-dual neural network

This simulation is conducted to compare the primal-dual neural network with the Lagrangian neural network for the minimum norm inverse kinematic control of redundant manipulators (4).

Figure 6 . Desired end-effector velocity profiles of the PA10 manipulator as an input to the primal-dual neural network and the Lagrangian network (Tang and Wang, A recurrent neural network for minimum infinity-norm kinematic control of redundant manipulators with an improved problem formulation and reduced architectural complexity, IEEE Trans. S y s t . , M a n , Cybern., 0 2 0 0 1 IEEE).

Figure 7. Motion trajectories of the PA10 manipulator synthesized by the primal-dual network under infinity-norm joint velocity minimization (Tang and Wang, A recurrent neural network for minimum infinity-norm kinematic control of redundant manipulators with an improved problem formulation and reduced architectural complexity, IEEE Trans. S y s t . , Man, Cybern., 0 2 0 0 1 IEEE) .

Real-time Computation of Inverse Kinematics of Redundant Manipulators

313

The end-effector of PAlO arm is t o follow an inclined circular path with radius of 0.2m and its orientation is kept unchanged throughout the motion with velocity profiles as shown in Fig. 6. The initial configuration of the manipulator is e(0) = [0, -n/4,0, n / 2 , 0 , -n/4, OITrad. The prime-dual neural network (16)-(18) and the Lagrangian network (9)-(10) are simulated by using the fourth-order Runge-Kutta method with p = 5 x lo5. Fig. 7 shows the joint motion trajectories of PAlO synthesized by the primal-dual neural network. Subplots (a) and (b) of Fig. 8 delineate transients of the PAlO joint velocities computed respectively by the primal-dual neural network (infinitynorm minimization scheme) and the Lagrangian neural network (two-norm minimization scheme). The comparative results in Fig. 8(c) show that in the infinity-norm simulation the largest joint velocity in magnitude has a reduction of 30% on average compared t o that of the minimum two-norm simulation. Fig. 8(d) depicts the transient of the energy function (15) for the primal-dual neural network during the manipulator operation. The energy function of the primal-dual network is converged to almost zero in less than l p s and being nearly zero thereafter. This substantiates the suitability of recurrent neural networks for real-time computation of the inverse kinematic solution for redundant manipulators.

4.2 Simulation results of the dual neural network Compared to joint velocity limits, in robotic simulation and experiments, the joint limits are more crucial and significant, since velocity limits can be generally escaped by increasing the task time (i.e., the duration of the task execution), but joint limits can not be avoided by such a simple idea. When using the Lagrangian neural network or the primal-dual network for PAlO kinematic control, violations of joint limits may occur; e.g., in the case of the initial joint state O(0) = [ 0 , 7 r / 4 , 0 , 2 ~ / 3 , 0-7r/4, , OITrad in the previous example now shown in Fig. 9(a). Although the maximal Cartesian position and velocity tracking errors computed by the Lagrangian network are less than 1.5 x lO-*mm and 6.0 x 10-5mm/sec respectively, the joint variable 84 has exceeded its mechanical range [-2.6831,2.6831] during the time period [3.8,5.5]s and thus the solution becomes inapplicable. If such a solution is directly applied to the PAlO manipulator with 84 finally locked at position -2.6831rad, the tracking error increases considerably, in addition to the physical damage possibly caused. Hence the path-following task fails. Now let us consider the joint limits in Table 1. Specifically, the developed dual neural network (22), with a = 1, p = 0.99 and p = lops, is applied to the PAlO robot arm for the same circular-path following task. Fig. 9(b)

314

J . Wang and Y. Zhang

(a) under infinity-norm minimization

(b) under two-norm minimization

(c) comparison of joint velocity vectors

(d) the energy function of the primal-dual network

Figure 8. Transients of the PA10 manipulator synthesized respectively by the primal-dual neural network under infinity-norm joint velocity minimization and by the Lagrangian neural network under two-norm joint velocity minimization (Tang and Wang, A recurrent neural network for minimum infinity-norm kinematic control of redundant manipulators with an improved problem formulation and reduced architectural complexity, IEEE Trans. Syst., M a n , Cybern., 0 2 0 0 1 IEEE).

shows joint variables of the PAlO robot arm synthesized by the dual network. Fig. 10 illustrates the motion trajectories of the PAlO manipulator and the corresponding transients of joint velocity variables and the Cartesian errors.

Real-time Computation of Inverse Kinematics of Redundant Manipulators 31

’

315

’ + 8.

(a) no joint limits

(b) a = 1,

= 0.99

Figure 9. Joint variables of the PAlO manipulator synthesized respectively by the Lagrangian neural network without considering joint limits and by the dual neural network with a = 1 and p = 0.99.

As seen in Fig. 9(b), the joint variable 04 has never exceeded the mechanical range [-2.6831,2.6831], and other joint velocity variables are all kept within the limits in Table 1. In addition, the maximal Cartesian position and velocity errors are respectively less than 5 x 10-4mm/sec and 5~ 10-3mm/sec. Namely, in Fig. 10, e,, ey and e, denote the components of tracking position error e respectively along the x, y and z axes of the base frame, and similarly e , , ey and ez denote respectively the x,y and z-axis components of tracking velocity error e at the end-effector of the PAlO robot arm. The above circular-path following experiments demonstrate the effectiveness of the proposed neural network approaches for online resolving redundancy of physically constrained manipulators. 5

Concluding remarks

Three recurrent neural network models for real-time kinematic control have been reviewed in this chapter, which are respectively the Lagrangian neural network, the primal-dual neural network and the dual neural network. Compared with the supervised learning neural networks for robot control, the present recurrent neural network approach eliminates the need for off-line training and produces much smaller position and velocity errors. The Lagrangian neural network first shown in [19] is the starting point to the inverse

316

J. Wang and Y. Zhang

Figure 10. Motion trajectories, joint velocities and Cartesian errors of the PA10 manipulator synthesized by the dual neural network.

kinematics studies based on recurrent neural networks, whereas the primaldual neural network is much flexible by considering the primal program and its dual problem simultaneously and using gradient decent methods. To reduce the scheme complexity and increase the computational efficiency of the primal-dual network, the dual neural network model has been developed with the advantages of piecewise linearity, simple architecture and exponential convergence. Computer simulations based on the PA10 robot arm substantiate the effectiveness of the present recurrent neural network approach.

Real-time Computation of Inverse K i n e m a t i c s of Redundant Manipulators

317

References

1. L. Sciavicco and B. Siciliano, Modelling and Control of Robot Manipulators, Great Britain: Springer-Verlag London, 2000. 2. D. E. Whitney, “Resolved motion rate control of manipulator and human prostheses,” IEEE Trans. Man-Machine Systems, vol. 10, pp. 47-53, 1969. 3. A. Liegeois, “Automatic supervisory control of the configuration and behavior of multi-body mechanisms,” IEEE Trans. Systems, Man, and Cybernetics, vol. 7, pp. 868-871, 1977. 4. C. A. Klein and C. H. Huang, “Review of pseudoinverse control for use with kinematically redundant manipulators,” IEEE Trans. Syst. , Man, and Cybern., vol. 13, pp. 245-250, 1983. 5. D. R. Baker and C. W. Wampler 11, “On the inverse kinematics of redundant manipulators,” Intl. J . of Robot. Research, vol. 7, pp. 3-21, 1988. 6. M. Vukobratoric and M. Kircanski, “A dynamic approach to nominal trajectory synthesis for redundant manipulator,” IEEE Trans. Syst., Man, Cybern., vol. 14, pp. 580-586, 1984. 7. A. A. Maciejemski and C. A. Klein, “Obstacle avoidance for kinematically redundant manipulators in dynamically varying environments,” Intl. J. Robot. Res., vol. 4, no. 3, pp. 109-117, 1985. 8. T. Yoshikawa, “Manipulability of robot mechanisms,” Intl. J . Robot. Res., vol. 4,no. 2, pp. 3-9, 1985. 9. A. S. Deo and I. D. Walker, “Minimum effort inverse kinematics for redundant manipulators,” IEEE Trans. Robotics Automat., vol. 13, no. 5, pp. 767-775, 1997. 10. D. W. Tank and J. J. Hopfield, “Simple neural optimization networks: An A/D converter, signal decision circuit, and a linear programming circuit,” IEEE Trans. Circuits Syst., vol. 33, pp. 533-541, 1986. 11. D. P. Bertsekas, , Parallel and Distributed Computation: Numerical Methods. Englewood Cliffs, NJ: Prentice-Hall, 1989. 12. S. Zhang and A. G. Constantinides, “Lagrange programming neural netw or k~, ’IEEE ~ Trans. Circuits Syst., vol. 39, no. 7, pp. 441-452, 1992. 13. S. Y. Kung and J. N. Hwang, “Neural network architectures for robotic application,” IEEE Trans. Robot. Automat., vol. 5, pp. 641-657, 1989. 14. Y. Li and N. Zeng, “A neural network based inverse kinematics solution in robotics,” in Neural Networks in Robotics, G. A. Bekey and K. Y. Goldberg (eds.), pp. 97-111, Kluwer, Boston, MA, 1993. 15. J . F. Gardner et al., “Applications of neural networks for coordinate

318 J . W a n g and Y.Zhang

16. 17. 18. 19. 20.

21.

22.

23.

24. 25.

26. 27.

transformations in robotics,” J. Intell. Robot. Syst., vol. 8 , pp. 361-373, 1993. S. Lee and R. M. Kill “Redundant arm kinematic control with recurrent loop,” Neural Networks, vol. 7, pp. 643-655, 1994. Z. Mao and T. C. Hsia, “Obstacle avoidance inverse kinematics solution of redundant robots by neural networks,” Robotica, vol. 15, pp. 3-10, 1997. H. Ding and J. Wang, “Recurrent neural networks for minimum infinitynorm kinematic control of redundant manipulators,” IEEE Trans. Syst. , Man, Cybern., part A , vol. 29, no. 3, pp. 269-276, 1999. J. Wang, Q. Hu, and D. Jiang, “A Lagrangian network for kinematic control of redundant manipulators,” IEEE Trans. Neural Net., vol. 10, no. 5, pp. 1123-1132, 1999. W . S. Tang and J. Wang, “A primal-dual neural network for joint torque optimization of redundant manipulators subject to torque limit constraints,” Proc. IEEE Intl. Conf. Syst., Man, and Cybern., Tokyo, Japan, vol. 4, pp. 782-787, 1999. W. S. Tang and J. Wang, “A primal-dual neural network for kinematic control of redundant manipulators subject to joint velocity constraints,” Proc. Intl. Conf. Neural Inform. Proc., Perth, Australia, pp. 801-806, 1999. W. S. Tang, J. Wang, and Y. S. Xu, “Infinity-norm torque minimization for redundant manipulators using a recurrent neural network,” Proc. IEEE Conf. Decision and Contr., Phoenix, AZ, USA, pp. 2168-2173, 1999. W. S. Tang, M. Lam, and J . Wang, “Kinematic control and obstacle avoidance for redundant manipulators using a recurrent neural network,” Artificial Neural Net. - Proc. Intl. Conf. Artificial Neural Net., Vienna, Austria, pp.922-929, 2001. W. S. Tang and J. Wang, “TWO recurrent neural networks for local joint torque optimization of kinematically redundant manipulators,” IEEE Trans. Syst., Man, Cybern., part B, vol. 30, no. 1, pp. 120-128, 2001. W. S. Tang and J. Wang, “A recurrent neural network for minimum infinity-norm kinematic control of redundant manipulators with an improved problem formulation and reduced architectural complexity,” IEEE Trans. Syst., Man, Cybern., part B, vol. 31, no. 1, pp. 98-105, 2001. Y. Xia and J . Wang, “A dual neural network for kinematic control of redundant robot manipulators,” IEEE Trans. Syst., Man, Cybern., part B, vol. 31, no. 1, pp. 147-154, 2001. Y. Zhang and J. Wang, “A dual neural network for constrained joint

Real-time Computation of Inverse Kinematics of Redundant Manipulators

28. 29. 30. 31. 32.

319

torque optimization of kinematically redundant manipulators,” IEEE Fans. Syst., Man, Cybern., part B, accepted, 2002. Y. Zhang and J. Wang, “Bi-criteria kinematic control of redundant manipulators using a dual neural network,” Proc. Intl. Joint Conf. Neural Net., Honolulu, HI, USA, May 2002. D. G. Luenberger, Linear and Nonlinear Programming, Addison-Wesley, Reading, MA, 1984. M. S. Bazaraa, H. D. Sherali, and C . M. Shetty, Nonlinear Programming - Theory and Algorithms, New York: Wiley, 1993. 0. L. Mangasarian, “Solution of symmetric linear complementarity problems by iterative methods,” J. Optim. Theory Applicat., vol. 22, no. 2, pp. 465-485, 1979. W. Li and J. Swetits, “A new algorithm for solving strictly convex quadratic programs,” SIAM J. Optim., vol. 7, no. 3, pp. 595-619, 1997.

This page intentionally left blank

TOWARDS PERCEPTION-BASED FUZZY MODELING: AN EXTENDED MULTISTAGE FUZZY CONTROL MODEL AND ITS USE IN SUSTAINABLE REGIONAL DEVELOPMENT PLANNING JANUSZ KACPRZYK SYSTEMS RESEARCH INSTITUTE, POLISH ACADEMY OF SCIENCES UL. NEWELSKA 6,Ol-447 WARSAW, POLAND E-MAIL: [email protected] We show how the basic Bellman and Zadeh’s (1970) model of multistage decision making (control) in a fuzzy environment can be extended to account for human perceptions concerning its basic elements, i.e. the fuzzy constraints and fuzzy goals, by introducing objective (related more to measurements) and subjective (related more to perceptions) fuzzy constraints and fuzzy goals. To illustrate the extended, perception based model, we present a fuzzy socioeconomic sustainable regional development model initiated by Kacprzyk and Straszak (1984), and further developed by Kacprzyk (1997), Kacprzyk, Romero and Gomide (1999), etc. Life quality indicators are crucial elements of this model, and (the satisfaction of) their attainment is strongly related to human perception. The model may be viewed as as example of how fuzzy logic, or - more generally - the computing with words paradigm, can help devise new more human consistent, perception based models.

Keywords: multistage decision making (control) under fuzziness, fuzzy dynamic programming, computing with words, perceptions, socio-economic planning, regional development. 1

Introduction

Traditionally, computing involves mainly the manipulation of numbers that are, in a natural way, supplied by (objective and precise) measurements. Humans, however, evaluate virtually all aspects chracterizing reality around them not by measurements but employing perceptions. Though a pivital role of perceptions has been recognized for a long time, no formal, “computational” approach to deal with perceptions has been proposed. Computing with words seems to be the first constructive yet simple enough attempt to devise a formal apparatus to calculate with perception. Its point of departure is natural: humans employ mostly words in computing and reasoning, arriving at conclusions expressed as words from premises expressed in a natural language or having the form of mental perceptions. As used by humans, words have fuzzy denotations, and the same applies to the role of words in computing with words.

32 I

322

J . Kacprzyk

Computing with words, as a general paradigm, has been proved to be successful in many areas, and the the best source of infromation is here Zadeh and Kcprzyk’s (1999) volumes. In this paper, we present its use in extending - by making it possible to reflect and express perceptions - Bellman and Zadeh’s (1970) general approach to decision making in a fuzzy environment that is a framework for all fuzzy decision making, optimization, control, etc. models, and presumably the most widely used general fuzzy approach [cf. Kacprzyk’s (1983, 1997) books]. Then, we will show as an example its use in a sustainable regional development planning model developed over the years by Kacprzyk and Straszak (1984)’ Kacprzyk (1997), Kacprzyk, Romero and Gomide (1999), etc. This model includes as a crucial element life quality indicators, and the (satisfation of) their attainement is strongly related to human perception. 2

Extending Bellman and Zadeh’s approach to decision making and control under fuzziness

In Bellman and Zadeh’s (1970) model, if X = { x } is some set of possible options (alternatives, variants, choices, decisions, . . . ), then the fuzzy goal is defined as a fuzzy set G in X , characterized by its membership function p~ : X -+ [0,1] such that ~ G ( z )E [0,1] specifies the grade of membership of a particular option x E X in the fuzzy goal G, and the fuzzy constraint is similarly defined as a fuzzy set C in the set of options X , characterized by p c : X -+ [0,1] such that p c ( x ) E [0,1] specifies the grade of membership of a particular option x E X in the fuzzy constraint C. The general problem formulation is: “Attain G & satisfy C” which leads to the fuzzy decision p D ( x ) = PC ( x ) A pC ( x ),

for each x E X

(1)

where “A” is the minimum that may be replaced by another appropriate operation (e.g., a t-norm). The maximizing decision is defined as an x’ E X such that PD(X*)

= maxpD(x) X€X

(2)

The human factor is crucial in reality, and this implies that the satisfaction of constraints and attainment of goals have both an objective and subjective aspect. The Bellman and Zadeh’s (1970) framework can therefore be extended by introducing: an objective fuzzy goal p ~ , ( x ) a, subjective fuzzy goal p ~ * ( x an objective fuzzy constraint pc, (x), and a subjective fuzzy constraint pc, ( x ) .

Towards Perception-based Fuzzy Modeling

323

We wish to “Attain [G, and G,] & satisfy [C, and C,]” which leads to P D ( z ) = [PG,).(

A PG,

(.)I

A [PC,

(.)I

A PCa

for each x E X

I

(3)

and the maximizing, or optimal decision is defined as in (2). This framework can be extended to handle multiple fuzzy constraints and fuzzy goals, and also fuzzy constraints and fuzzy goals defined in different spaces [cf. Kacprzyk (1997)l. Namely, if we have: no > 1 objective fuzzy goals - GL, . . . , defined in Y , n, > 1 subjective fuzzy goals - G i , . . . , GYs defined in Y , m, > 1 objective fuzzy constraints C: , . . . , Corn0 defined in X , m, > 1 subjective fuzzy constraints - Gb, . . . ,Crs defined in X , and a function f : X tY , y = f(x), then G’;O

~

PD(x) =

= (pGA[f(x)] A A [Pc; A [Pc:

’ ’’

A PGgo [f(X)l> A (PG; [f(.)1 A ‘ ’ ’ A PG:. [f(x)1) A A . . . A PCFS A [Pc;

. . . A PUC,”. (.)I A A . . . A l l c g m s (x)1, A

(.)I

for each x E X

and the maximizing decision is defined as (2), i.e.

3

(4)

~ D ( x *= ) max,cx ~ D ( z ) .

Extending multistage decision making (control) in Bellman and Zadeh’s setting

The control process proceeds basically as follows. The decision (control) space is U = {u} = { c l , ..., ern}, the state (output) space is X = {x} = {sl,. . . , s,}, and both are finite. We start from an initial state xo E X , apply a decision (control) uo E U , which is subjected t o a fuzzy constraint p ~ o ( u o and attain a state x1 E X via a known state transition equation of the system under control S; a fuzzy goal ~ G(x1) I is imposed on x1. Next, we apply u1, subjected to p p ( u l ) , and attain X Z , subjected to ~ G Z ( ~ Z )etc. , The state transitions of the (deterministic) system under control is given by Zt+l

= f(xt,ut),

t = 0,1,...

(5)

where xt,xt+l E X = { s l , .. . , s,} are the states at t and t + 1, respectively, and ut E U = { Q , . . . , cm} is the decision (control) at t . At t , t = 0 , 1,. . ., ut E U is subjected t o a fuzzy constraint p ~ (ut), t and on xt+l E X a fuzzy goal is imposed, PGt+l (xt+l). The fixed and specified in advance initial state is xo E X I and the termination time (planning horizon), N E {1,2, . . .}, is finite, and fixed and specified in advance.

324

J . Kacprzyk

System

under control

... control stage t

a6

PG, ( x t )

PG, (.t )

Figure 1. Evaluation of (extended) performance of stage t

The performance of the particular decision making (control) stage t , t = 0,1,. . . , N - 1, is given by Vt

= PCt ( u t ) A PGt+' ( Z t + l ) = PCt (.t)

A PGt+l [f ( Z t , u t ) ]

(6)

while the performance of the whole multistage decision making (control) process is given by the fuzzy decision

.

I

p ~ ( U 0 ,.. , U N - ~ Zo)

= Vo A V l A . . . A V

= [ k Y ( u O )A P G 1 ( Z l ) ]A . . . A

N - ~=

[PCN-'(uN-l) A P G N ( Z N ) ]

(7)

The problem is to find an optimal sequence of decisions (controls) u:, . . . , u & - ~such that

Kacprzyk's (1997a) book provides and wide coverage of various aspects and extensions to this basic formulation. In the case considered here, we have, at each t = 0 , 1 , . . . , N - 1: an objective fuzzy constraint pc; ( u t ) and a subjective fuzzy constraint pc: ( u t ) , and an objective fuzzy goal p G k + 1 ( u t + l ) and a subjective fuzzy constraint PG:+'

1.

The (extended) performance of stage t , t = 0 , 1 , . . . , N -

Vt

= [PC; ( u t )A PG': (' % )I

-

1, is then

A [PG; ( Z t ) A PGf, ( Z t ) ]

which can be schematically shown as in Figure 1.

(9)

Towards Perception-based Fuzzy Modeling

325

The (extended) performance of the whole process is then

and we seek again an pD(u:, . . . , U L - ~ I XO) =

such that max

UO,

..., U N - l € U

p ~ ( u 0 ., .. , U N - I 1 x 0 )

(11)

There is an extremely relevant aspect related t o the subjective fuzzy constraints and fuzzy goals. Consider subjective fuzzy goals in which this is presumably much more pronounced than in subjective fuzzy constraints. Namely, it often happens that the (subjective) human satisfaction resulting from the attainment of some level of xt+l - exemplified by a value of a life quality index in Section 4 - depends not only on the "objectively attained" value but on how this value looks like in comparision with the past, future prospects, etc. For simplicity, let us concentrate on he past only. The trajectory of the multistage decision making (control) process from t = 0 to a currect stage t = k is Hk

= ( X ~ , U O , C ~ O , C ~ O , X ~. ., .G, U~k,-G 1 , ~C, : - l , C f - l , x k , G t , G t )

(12)

that is, it involves all aspects of what has happended in terms of decisions aplied, states attained, and objective and subjective opinions of how well the fuzzy constraints have been satisfied and fuzzy goals attained. However, it is often sufficient to take into account the reduced trajectory

which only takes into account the current, t = k, and previous stage, t = k - 1. Let us assume this reduced trajectory. A further simplification is that with a trajectory, or reduced trajectory, an evaluation function is associated, E : s ( H k ) -+ [O,1]or e : S(hk) + [0, I], where s ( H k ) and S(hk) are the sets of trajectories and reduced trajectories, respectively, such that E ( H k ) E [o, 11 and e ( h k ) E [o, 11denote the satisfaction of the past development, from 1 for full satisfaction to 0 for full dissatisfaction, through all intermediate values. The subjective fuzzy constraints and huzzy goals are now: 0

when the (reduced) trajectory is accounted for

326

J . Kacprzyk

when the evaluation of the (reduced) trajectory is accounted for

Problem (8) can be solved using the following two basic traditional techniques: dynamic programming, and branch-and-bound, and also using the two new ones: a neural network, and a genetic algorithm. We will only briefly show the use of dynamic programming, and refer the redear for an extensive coverage on this and other solution techniques to Kacprzyk's (1997a) book. such that First, we rewrite (8) as t o find u;, . . . , 1ZO)=

pD(U:,...,uN-l

...

A PCN-I

A PGN (f(zN-17uN-I))I

(UN-1)

(16)

and then, since P C N - 1 ( u N - ~ )A ~ G ( N ~ ( z N - I ,U N - ~ ) ) depends only on U N - 1 , then the maximization with respect to uo, . . . ,U N - ~in (16) can be split into:

. . . ,U N - 2 , and

the maximization with respect to

UO,

the maximization with respect to

UN-1,

written as

which may be continued for U N - 2 , U N - 3 , etc. This backward iteration leads t o the following set of fuzzy dynamic programming recurrence equations: P-N-t

N-i+l { ZG

= - maxuN-,[

(2N-i)

= f(zN-il

~ c N -( u ~

N - ~ )A

~ G N - - (I

z N - ~ )A

p E ~ - i + (l x ~ - i + l ) ]

i=O,l ,...,N

UN-i);

(18) where p G ~ -(i X N - i ) is viewed as a fuzzy goal a t control stage t = N - i induced by the fuzzy goal at t = N - i 1, i = O , l , . . . , N ; p E (~Z N ) = ~ G (zN). N

+

Towards Perception-based Fuzzy Modeling

327

The uo, . . . ,U N - ~sought is given by the successive maximizing values of = 1,.. . ,N in (18) which are obtained as functions of X N - ~ ,i.e. as an optimal policy, a N - i : + u, such that U N - ~= U N - i ( X N - i ) . It easy t o notice that if we use the subjective fuzzy constraints and fuzzy goals t o extend the above fuzzy dynamic programming model, then the very idea of dynamic programming, i.e. the use of