Asli Celikyilmaz and I. Burhan Türksen Modeling Uncertainty with Fuzzy Logic
Studies in Fuzziness and Soft Computing, Volume 240 Editor-in-Chief Prof. Janusz Kacprzyk Systems Research Institute Polish Academy of Sciences ul. Newelska 6 01-447 Warsaw Poland E-mail:
[email protected] Further volumes of this series can be found on our homepage: springer.com Vol. 224. Rafael Bello, Rafael Falcón, Witold Pedrycz, Janusz Kacprzyk (Eds.) Contributions to Fuzzy and Rough Sets Theories and Their Applications, 2008 ISBN 978-3-540-76972-9 Vol. 225. Terry D. Clark, Jennifer M. Larson, John N. Mordeson, Joshua D. Potter, Mark J. Wierman Applying Fuzzy Mathematics to Formal Models in Comparative Politics, 2008 ISBN 978-3-540-77460-0 Vol. 226. Bhanu Prasad (Ed.) Soft Computing Applications in Industry, 2008 ISBN 978-3-540-77464-8 Vol. 227. Eugene Roventa, Tiberiu Spircu Management of Knowledge Imperfection in Building Intelligent Systems, 2008 ISBN 978-3-540-77462-4 Vol. 228. Adam Kasperski Discrete Optimization with Interval Data, 2008 ISBN 978-3-540-78483-8 Vol. 229. Sadaaki Miyamoto, Hidetomo Ichihashi, Katsuhiro Honda Algorithms for Fuzzy Clustering, 2008 ISBN 978-3-540-78736-5 Vol. 230. Bhanu Prasad (Ed.) Soft Computing Applications in Business, 2008 ISBN 978-3-540-79004-4 Vol. 231. Michal Baczynski, Balasubramaniam Jayaram Soft Fuzzy Implications, 2008 ISBN 978-3-540-69080-1
Vol. 232. Eduardo Massad, Neli Regina Siqueira Ortega, Laécio Carvalho de Barros, Claudio José Struchiner Fuzzy Logic in Action: Applications in Epidemiology and Beyond, 2008 ISBN 978-3-540-69092-4 Vol. 233. Cengiz Kahraman (Ed.) Fuzzy Engineering Economics with Applications, 2008 ISBN 978-3-540-70809-4 Vol. 234. Eyal Kolman, Michael Margaliot Knowledge-Based Neurocomputing: A Fuzzy Logic Approach, 2009 ISBN 978-3-540-88076-9 Vol. 235. Kofi Kissi Dompere Fuzzy Rationality, 2009 ISBN 978-3-540-88082-0 Vol. 236. Kofi Kissi Dompere Epistemic Foundations of Fuzziness, 2009 ISBN 978-3-540-88084-4 Vol. 237. Kofi Kissi Dompere Fuzziness and Approximate Reasoning, 2009 ISBN 978-3-540-88086-8 Vol. 238. Atanu Sengupta, Tapan Kumar Pal Fuzzy Preference Ordering of Interval Numbers in Decision Problems, 2009 ISBN 978-3-540-89914-3 Vol. 239. Baoding Liu Theory and Practice of Uncertain Programming, 2009 ISBN 978-3-540-89483-4 Vol. 240. Asli Celikyilmaz, I. Burhan Türksen Modeling Uncertainty with Fuzzy Logic, 2009 ISBN 978-3-540-89923-5
Asli Celikyilmaz and I. Burhan Türksen
Modeling Uncertainty with Fuzzy Logic With Recent Theory and Applications
ABC
Authors Prof. I. Burhan Türksen University of Toronto Mechanical & Industrial Engineering 170 College St., Haultain Building Toronto, Ontario, M5S 3G8 Canada E-mail:
[email protected] and TOBB Economy and Technology University Mühendislik Fakültesi Endüstri Mühendisligi Bölümü Sögütözü Caddesi No.43 06560 Ankara Turkey E-mail:
[email protected]
Dr. Asli Celikyilmaz University of California, Berkeley BISC - The Berkeley Initiative in Soft Computing Electrical Eng. and Computer Sciences Department 415 Soda Hall Berkeley, CA, 94709-7886 USA E-mail:
[email protected],
[email protected],
[email protected]
ISBN 978-3-540-89923-5
e-ISBN 978-3-540-89924-2
DOI 10.1007/978-3-540-89924-2 Studies in Fuzziness and Soft Computing
ISSN 1434-9922
Library of Congress Control Number: 2008941674 c 2009 Springer-Verlag Berlin Heidelberg This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilm or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. The use of general descriptive names, registered names, trademarks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. Typeset & Cover Design: Scientific Publishing Services Pvt. Ltd., Chennai, India. Printed in acid-free paper 987654321 springer.com
To: Fuzzy Logic Research Fuzzy Data Mining Society and Prof. L.A. Zadeh
Foreword
Preface
The world we live in is pervaded with uncertainty and imprecision. Is it likely to rain this afternoon? Should I take an umbrella with me? Will I be able to find parking near the campus? Should I go by bus? Such simple questions are a common occurrence in our daily lives. Less simple examples: What is the probability that the price of oil will rise sharply in the near future? Should I buy Chevron stock? What are the chances that a bailout of GM, Ford and Chrysler will not succeed? What will be the consequences? Note that the examples in question involve both uncertainty and imprecision. In the real world, this is the norm rather than exception. There is a deep-seated tradition in science of employing probability theory, and only probability theory, to deal with uncertainty and imprecision. The monopoly of probability theory came to an end when fuzzy logic made its debut. However, this is by no means a widely accepted view. The belief persists, especially within the probability community, that probability theory is all that is needed to deal with uncertainty. To quote a prominent Bayesian, Professor Dennis Lindley, “The only satisfactory description of uncertainty is probability. By this I mean that every uncertainty statement must be in the form of a probability; that several uncertainties must be combined using the rules of probability; and that the calculus of probabilities is adequate to handle all situations involving uncertainty…probability is the only sensible description of uncertainty and is adequate for all problems involving uncertainty. All other methods are inadequate…anything that can be done with fuzzy logic, belief functions, upper and lower probabilities, or any other alternative to probability can better be done with probability.” What can be said about such views is that they reflect unfamiliarity with fuzzy logic. The book “Modeling Uncertainty with Fuzzy Logic,” co-authored by Dr. A. Celikyilmaz and Professor I.B. Turksen, may be viewed as a convincing argument to the contrary. In effect, what this book documents is that in the realm of uncertainty and imprecision fuzzy logic has much to offer. There are many misconceptions about fuzzy logic. Fuzzy logic is not fuzzy. Like traditional logical systems and probability theory, fuzzy logic is precise. However, there is an important difference. In fuzzy logic, the objects of discourse are allowed to be much more general and much more complex than the objects of
VIII
Foreword
discourse in traditional logical systems and probability theory. In particular, fuzzy logic provides many more tools for dealing with second-order uncertainty, that is, uncertainty about uncertainty, than those provided by probability theory. Imprecise probabilities, fuzzy sets of Type 2 and vagueness are instances of secondorder uncertainty. In many real-world settings, and especially in the context of decision analysis, the complex issue of second-order uncertainty has to be addressed. At this juncture, decision-making under second-order uncertainty is far from being well understood. “Modeling Uncertainty with Fuzzy Logic” begins with an exposition of the basics of fuzzy set theory and fuzzy logic. In this part of the book as well as in other parts, there is much that is new and unconventional. Particularly worthy of note is the authors' extensive use of the formalism of so-called fuzzy functions as an alternative to the familiar formalism of fuzzy if-then rules. The formalism of fuzzy functions was introduced by Professor M. Demirci about a decade ago, and in recent years was substantially extended by the authors. The authors employ their version of the formalism to deal with fuzzy sets of Type 2, that is, fuzzy sets with fuzzy grades of membership. To understand the authors' approach, it is helpful to introduce what I call the concept of cointension. Informally, cointension is a measure of the closeness of fit of a model to the object of modeling. A model is cointensive if its degree of cointension is high. In large measure, scientific progress is driven by a quest for cointensive models of reality. In the context of modeling with fuzzy logic, the use of fuzzy sets of Type 2 makes it possible to achieve higher levels of cointension. The price is higher computational complexity. On balance, the advantages of using fuzzy sets of Type 2 outweigh the disadvantages. For this reason, modeling with fuzzy sets of Type 2 is growing in visibility and importance. A key problem in applications of fuzzy logic is that of construction of the membership function of a fuzzy set. There are three principal approaches. In the declarative approach, membership functions are specified by the designer of a system. This is the approach that is employed in most of the applications of fuzzy logic in the realms of industrial control and consumer products. In the computational approach, the membership function of a fuzzy set is expressed as a function of the membership functions of one or more fuzzy sets with specified membership functions. In the modelization/elicitation approach, membership functions are computed through the use of cointension-enhancement techniques. In using such techniques, successive outputs of a model are compared with desired outputs, and parameters in membership functions are adjusted to maximize cointension. For this purpose, the authors make skillful use of a wide variety of techniques centering on cluster analysis, pattern classification and evolutionary algorithms. They employ simulation to validate their results. In sum, the authors develop an effective approach to modeling of uncertainty using fuzzy sets of Type 2 employing various parameter-identification formalisms. “Modeling Uncertainty with Fuzzy Logic” is an important contribution to the development of a better understanding of how to deal with second-order uncertainty.
Foreword
IX
The issue of second-order uncertainty has received relatively little attention so far, but its intrinsic importance is certain to make it an object of growing attention in coming years. Through their work, the authors have opened a door to wide-ranging applications. They deserve our compliments and congratulations.
Berkeley, CA November 30, 2008
Lotfi A. Zadeh
Preface
Preface
A representation of a system with a model and an implementation of that model to reason with and provide solutions are central to many disciplines of science. An essential concern of system models is to establish the relationships between system input variables and output variables. In cases of complex systems, conventional modeling approaches usually do not perform well because it is difficult to find a global function or analytic structure for such systems. In this regard, fuzzy system modeling (FSM) – meaning the construction of representations of fuzzy systems models – with fuzzy techniques and theories provide an efficient alternative which has proven to be quite successful. In one of his latest works1, Professor Lotfi A. Zadeh – describes remarkable capabilities of fuzzy logic as follows: “..Fuzzy logic may be viewed as an attempt at formalization/mechanization of two remarkable human capabilities. First, the capability to converse, reason and make rational decisions in an environment of imprecision, uncertainty, incompleteness of information, conflicting information, partiality of truth and partiality of possibility – in short, in an environment of imperfect information. And second, the capability to perform a wide variety of physical and mental tasks without any measurements and any computations.”. The capabilities of the fuzzy logic open possibilities for a wide range of theoretical and application problem domains. In spite of its success, implementation of fuzzy techniques and theories has been a difficult task in representing complex systems and building fuzzy system models. It requires identification of many parameters. For instance, an important problem in development of fuzzy system models is to generate fuzzy if-then rules. These rules may be constructed by an extraction of knowledge from human experts and to construct suitable membership functions. However, information supplied by humans suffers from certain serious problems. Firstly, human knowledge is usually incomplete or not organized since different experts usually make different decisions. Even the same expert may have 1
L.A. Zadeh, “Is there a need for fuzzy logic”, Information Sciences, vol. 178, issue 13, July 2008.
XII
Preface
different interpretations of the same observation on different times. Furthermore, knowledge acquisition from experts is not systematic or efficient. These problems have let researchers to build automated algorithms for modeling systems using fuzzy theories via machine learning and data mining techniques. With the above problems in hand, in this book, we propose novel and unique fuzzy-modeling approaches such deficiencies. We mainly focus on algorithms based on the novel fuzzy functions method. The new fuzzy functions approach employs membership values differently than any other fuzzy system models are implemented to date. The membership values can be thought of as ‘atoms’ that hold potential information about a system behaviour to be activated and release its power. Such potential information obtained from membership values are captured in local fuzzy functions as predictors of the system behaviour. Instead and in place of fuzzy if-then rule base structures, Fuzzy functions are implemented to build models of a given system. This book presents essential alternatives of the fuzzy functions approaches and their advancements. The aim is to build more powerful fuzzy models via autonomous approaches that can identify hidden structures via the optimization of their system parameters. The term “fuzzy functions” has been used by researchers to define different things. The building blocks of the fuzzy set theory is proposed by Professor L.A. Zadeh in 1965 especially the fuzzy extensions of classical basic notations such as logical connectives, quantifiers, inference rules, relations, arithmetic operations, etc. Hence, these constitute the initial definitions of fuzzy functions. Later different forms of fuzzy functions have been presented. In 1969, Marinos introduced the concept of well-known conventional switching theory techniques into the design of fuzzy logic systems, based on fuzzy set theory and fuzzy operations of Professor L.A. Zadeh. In their 1972 paper, Siy and Chen explored simplification of fuzzy functions and they defined fuzzy functions as the polynomials, which are formed by possible combinations of operations on fuzzy sets. Hence, fuzzy functions are defined as relations between fuzzy variables. Other researchers have also defined fuzzy functions as being a special case of fuzzy relations. Sasaki in 1993 and later in 1999 Demirci defined fuzzy functions as a special case of fuzzy relations and explored their mathematical representations. Thus, the fuzzy functions we introduce in this paper are different than the latter fuzzy functions. The idea of the fuzzy functions of this study has emerged from the idea of representing each unique rule of Fuzzy Rule Bases in terms of functions. The main structure identification tools of this book capture the hidden structures via pattern recognition methods. As well we present a new improved fuzzy clustering algorithm that would help identify the relationships between the input variables and the output variable in local models mainly by regression type system development techniques. We also focused on classification problem domains by building multiple fuzzy classifiers for each hidden pattern identified with the improved fuzzy clustering method. We present a new fuzzy cluster validity method to demonstrate how the methodologies presented fit to the estimation approaches.
Preface
XIII
Later in the book, we incorporate one of the soft computing tools to optimize the parameters of the fuzzy function models. We focus on a novel evolutionary fuzzy functions approach–the design of “improved fuzzy functions” system models with the use of evolutionary algorithms. Evolutionary algorithms are a broad class of stochastic optimization tools, inspired by biology. An evolutionary algorithm maintains a population of candidate solutions for the problem at hand, and makes it evolve by iteratively applying a set of stochastic operators; know as mutation, recombination, and selection. The resulting process tends to find, given enough time, globally optimal solutions to the problem much in the same way as in nature populations of organisms tend to adapt to their surrounding environment. Hence, using the evolutionary algorithms as the optimization tool, the local structures of the given database are identified with a new improved fuzzy clustering method and represented with novel “fuzzy functions”. Although it has been investigated for many years now, the problem of uncertainty modeling is yet to be satisfactorily resolved in system modeling communities. In engineering problems, building reliable models depends on the identification of important values of variables of model equations. However, in real life cases, these important values may not be obtained due to imprecise, noisy, vague, or incomplete nature of current information. The goal of this book is to build an uncertainty modeling architecture to represent and handle the uncertainty in parameters and structure of the fuzzy functions to capture the most available information. The uncertainty in systems can be captured with higher order fuzzy sets, viz. interval valued type-2 fuzzy sets, which was first introduced by Professor Lotfi A. Zadeh. Type-2 fuzzy systems implement type-2 fuzzy sets to capture the higher order imprecision inherited in systems. In particular, this book introduces the formation of type-2 fuzzy functions to capture uncertainties associated with system behaviour. The central contributions of this work is to expose the fuzzy functions approach and enhance it to capture imprecision in setting system model parameters by constructing a new uncertainty modeling tool. To replace the standard fuzzy rule bases with the new improved fuzzy functions succeeds in capturing essential relationships in structure identification processes and overcomes limitations exhibited by earlier fuzzy inference systems based on if-then rule base methods because there is an abundance of fuzzy operations and hence the difficulty of the choice amongst the t-norms and co-norms, methods of fuzzification and defuzzification. Designing an autonomous and robust fuzzy system model and reasoning with it is the prime goal of this approach. This new fuzzy system modeling (FSM) approach implements higher-level fuzzy sets to identify the uncertainties in: (1) the system parameters, and (2) the structure of fuzzy functions. With the identification of these parameters, interval valued fuzzy sets and fuzzy functions are identified. Finally, an evolutionary computing approach with the proposed uncertainty identification strategy is combined to build fuzzy system models that can automatically identify these uncertainty intervals.
XIV
Preface
After testing the new fuzzy functions tools on various benchmark problems, the algorithms are successfully applied to model decision processes in two real problem domains: the desulphurization process in steel making and stock price prediction activities. For both of these problems, the proposed methods produce robust and high performance results, which are comparable (if not better) than the best system modeling approaches known in the current literature. Several aspects of the proposed methodologies are thoroughly analyzed to provide a deeper understanding. These analyses show consistency of the results. As a final note, the fuzzy modeling approaches demonstrated in this book may supply suitable framework for engineers, and practitioners for a design of complex systems with fuzzy functions instead of crisp functions, that is to say, for a design of artificial systems based on well known functions, viz. regression, classification, etc. Although in this book we only show examples from economy and production industry, we believe that fuzzy modeling can and should be utilized in many fields of science, including biology, economics, psychology, sociology, history and more. In such fields, many models exist in current research literature and they can be directly transformed into mathematical models using the methodologies presented herein.
September 2008
Asli Celikyilmaz University of California, Berkeley, USA I. Burhan Turksen University of Toronto, Canada TOBB Economy and Technology University, Turkey
Contents Contents
1
Introduction………………………………………………… 1.1 Motivation………………………………………………………... 1.2 Contents of the Book……………………………………………... 1.3 Outline of the Book……..................................................................
1 1 3 9
2
Fuzzy Sets and Systems………………………………………………. 2.1 Introduction ……...……...……...……............................................ 2.2 Type-1 Fuzzy Sets and Fuzzy Logic……………………………... 2.2.1 Characteristics of Fuzzy Sets……………………………… 2.2.2 Operations on Fuzzy Sets………………………………….. 2.3 Fuzzy Logic……………………………………………………….. 2.3.1 Structure of Classical Logic Theory.………………………. 2.3.2 Relation of Set and Logic Theory………………………….. 2.3.3 Structure of Fuzzy Logic………………………………….. 2.3.4 Approximate Reasoning……………………………………. 2.4 Fuzzy Relations…………………………………………………… 2.4.1 Operations on Fuzzy Relations…………………………….. 2.4.2 Extension Principle………………………………………… 2.5 Type-2 Fuzzy Sets…………………………………………………. 2.5.1 Type-2 Fuzzy Sets………………………………………….. 2.5.2 Interval Valued Type-2 Fuzzy Sets………………………… 2.5.3 Type-2 Fuzzy Set Operations………………………………. 2.6 Fuzzy Functions…………………………………………………… 2.7 Fuzzy Systems………………………………………………….… 2.8 Extensions of Takagi-Sugeno Fuzzy Inference Systems…………. 2.8.1 Adaptive-Network-Based Fuzzy Inference System (ANFIS)…………………………………………………... 2.8.2 Dynamically Evolving Neuro-Fuzzy Inference Method (DENFIS)………………… ……………………………... 2.8.3 Genetic Fuzzy Systems (GFS)…………………………….. 2.9 Summary………………………………………………………..…
11 11 12 13 14 18 18 19 19 21 22 25 25 28 29 31 32 33 36 40
Improved Fuzzy Clustering………………………………………….. 3.1 Introduction………………………………………………………. 3.2 Fuzzy Clustering Algorithms……………………………………...
51 51 52
3
41 44 46 50
XVI
Contents
3.2.1 Fuzzy C-Means Clustering Algorithm ……………………. 53 3.2.2 Classification of Objective Based Fuzzy Clustering Algorithms………………………………………………… 58 3.2.3 Fuzzy C-Regression Model (FCRM) Clustering Algorithm………………………………………………….. 58 3.2.4 Variations of Combined Fuzzy Clustering Algorithms…….. 61 3.3 Improved Fuzzy Clustering Algorithm (IFC)……………………… 64 3.3.1 Motivation …………………………………………………. 64 3.3.2 Improved Fuzzy Clustering Algorithm for Regression Models ( IFC )……………………………………………… 69 3.3.3 Improved Fuzzy Clustering Algorithm for Classification Models (IFC-C) ……………………………………..……. 73 3.3.4 Justification of Membership Values of the IFC Algorithm….. 77 3.4 Two New Cluster Validity Indices for IFC and IFC-C…………….. 85 3.4.1 Overview of Well-Known Cluster Validity Indices ………... 86 3.4.2 The New Cluster Validity Indices…………………………… 90 3.4.3 Simulation Experiments [Celikyilmaz and Turksen, 2007i;2008c]……………………………………………..…. 94 3.4.4 Discussions on Performances of New Cluster Validity Indices Using Simulation Experiments……………………... 100 3.5 Summary…………………………….…………………………….. 103 4
Fuzzy Functions Approach……………………….………….……….. 4.1 Introduction ……………………………………………………….. 4.2 Motivation…………………………………………………………. 4.3 Proposed Type-1 Fuzzy Functions Approach Using FCM – T1FF…..…………………………………………………… 4.3.1 Structure Identification of FF for Regression Models (T1FF )……………………………………………………… 4.3.2 Structure Identification of the Fuzzy Functions for Classification Models (T1FF-C )……………………......... 4.3.3 Inference Mechanism of T1FF for Regression Models……………………………...……………….…….. 4.3.4 Inference Mechanism of T1FF for Classification Models.... 4.4 Proposed Type-1 Improved Fuzzy Functions with IFC – T1IFF....……………………………………………..……… 4.4.1 Structure Identification of T1IFF for Regression Models.... 4.4.2 Structure Identification of T1IFF-C for Classification Models……..………………………………………………. 4.4.3 Inference Mechanism of T1IFF for Regression Problems...……………………………………………….... 4.4.4 Inference with T1IFF-C for Classification Problems........... 4.5 Proposed Evolutionary Type-1 Improved Fuzzy Function Systems…………………………………………………………….. 4.5.1 Genetic Learning Process: Genetic Tuning of Improved Membership Functions and Improved Fuzzy Functions…...
105 105 107 112 112 119 121 122 125 125 131 132 135 136 139
Contents
5
6
XVII
4.5.2 Inference Method for ET1IFF and ET1IFF-C……….......... 4.5.3 Reduction of Structure Identification Steps of T1IFF Using the Proposed ET1IFF Method…………………........ 4.6 Summary………………………………………………………….
146 147
Modeling Uncertainty with Improved Fuzzy Functions…………..
149
5.1 Motivation ……...……...……...……........................................... 5.2 Uncertainty ………………………………………………………. 5.3 Conventional Type-2 Fuzzy Systems………….…………………. 5.3.1 Generalized Type-2 Fuzzy Rule Bases Systems (GT2FRB)…........................................................................ 5.3.2 Interval Valued Type-2 Fuzzy Rule Bases Systems (IT2FRB)…………………………………………….….. 5.3.3 Most Common Type-Reduction Methods……………….. 5.3.4 Discrete Interval Type-2 Fuzzy Rule Bases (DIT2FRB)..... 5.4 Discrete Interval Type-2 Improved Fuzzy Functions…………….. 5.4.1 Background of Type-2 Improved Fuzzy Functions Approaches………………………………………………… 5.4.2 Discrete Interval Type-2 Improved Fuzzy Functions System (DIT2IFF)…………………………………..…..... 5.5 The Advantages of Uncertainty Modeling…………………..…... 5.6 Discrete Interval Type-2 Improved Fuzzy Functions with Evolutionary Algrithms…………..………………………………. 5.6.1 Motivation………………………...……………………… 5.6.2 Architecture of the Evolutionary Type-2 Improved Fuzzy Functions……..…..……………………………….. 5.6.3 Reduction of Structure Identification Steps of DIT2IFF Using New EDIT2IFF Method……...………………...…. 5.7 Summary…………………………………………………………
149 154 157
Experiments…….…….…….…….…….…….…….…….…….……. 6.1 Experimental Setup ……...……...……...……............................... 6.1.1 Overview of Experiments…………………………………. 6.1.2 Three-Way Sub-sampling Cross Validation Method……… 6.1.3 Measuring Models’ Prediction Performance……………… 6.1.3.1 Performance Evaluations of Regression Experiments…………………………..…………. 6.1.3.2 Performance Evaluations of Classification Experiments……………………..…………….... 6.2 Parameters of Benchmark Algorithms…………………………… 6.2.1 Support Vector Machines (SVM)…………………………. 6.2.2 Artificial Neural Networks (NN)………………………….. 6.2.3 Adaptive-Network-Based Fuzzy Inference System (ANFIS)……………………………………………………
145
157 160 162 164 167 168 179 193 196 196 197 213 215 217 217 217 219 221 221 223 227 228 229 229
XVIII
6.2.4 Dynamically Evolving Neuro-Fuzzy Inference Method (DENFIS)…………………………………………………. 6.2.5 Discrete Interval Valued Type-2 Fuzzy Rule Base (DIT2FRB)……………………………………………..... 6.2.6 Genetic Fuzzy System (GFS)…………………………..... 6.2.7 Logistic Regression, LR, Fuzzy K-Nearest Neighbor, FKNN…………………………………………................. Parameters of Proposed Fuzzy Functions Algorithms………….. 6.3.1 Fuzzy Functions Methods……………………………….. 6.3.2 Imporoved Fuzzy Functions Methods……………………. Analysis of Experiments – Regression Domain………………… 6.4.1 Friedman’s Artificial Domain…………………………… 6.4.2 Auto-mileage Dataset……………………………..……... 6.4.3 Desulphurization Process Dataset……………………….. 6.4.4 Stock Price Analysis……………………………………... 6.4.5 Proposed Fuzzy Cluster Validity Index Analysis for Regression……………………………………………….. Analysis of Experiments - Classification (Pattern Recognition) Domains…………………………………………………………. 6.5.1 Classification Datasets from UCI Repository…………… 6.5.2 Classification Dataset from StatLib……………………… 6.5.3 Results from Classification Datasets…………………….. 6.5.4 Proposed Fuzzy Cluster Validity Index Analysis for Classification………………………………………..... 6.5.5 Performance Comparison Based on Elapsed Times……... Overall Discussions on Experiments……………………………. 6.6.1 Overall Comparison of System Modeling Methods on Regression Datasets...……….…………………………… 6.6.2 Overall Comparison of System Modeling Methods on Classification Datasets…………………...………………. Summary of Results and Discussions……………...…………….
297 300
Conclusions and Future Work……………………………………… 7.1 General Conclusions...……...……...……..................................... 7.2 Future Work……………………………………………………… References……………………………………………………………..
305 305 310 313
Appendix……………………………………………………………… A.1 Set and Logic Theory – Additional Information………………… A.2 Fuzzy Relations (Composition) – An Example………………….. B.1 Proof of Fuzzy c-Means Clustering Algorithm………………….. B.2 Proof of Improved Fuzzy Clustering Algorithm………………… C.1 Artificial Neural Networks ANNs)……………………………… C.2 Support Vector Machines………………………………………... C.3 Genetic Algorithms………………………………………………
321 321 322 323 326 327 329 338
6.3
6.4
6.5
6.6
6.7 7
Contents
231 231 232 234 234 234 236 238 238 245 251 262 276 278 279 281 281 283 284 289 290
Contents
C.4 Multiple Linear Regression Algorithms with Least Squares Estimation……………………………………….…….….…….. C.5 Logistic Regression…………………………………………….. C.6 Fuzzy K-Nearest Neighbor Approach……………………….…. D.1 T-Test Formula……………………………………………….… D.2 Friedman’s Artificial Dataset: Summary of Results…………… D.3 Auto-mileage Dataset: Summary of Results…………………… D.4 Desulphurization Dataset: Summary of Results………………... D.5 Stock Price Datasets: Summary of Results…………………….. D.6 Classification Datasets: Summary of Results…………………... D.7 Cluster Validity Index Graphs………………………………...... D.8 Classification Datasets – ROC Graphs………………………….
XIX
340 341 343 344 345 354 363 367 388 397 398
List of Tables Contents
Table 2.1 Table 2.2
Some well known t-norms and t-conorms...……………….. The AGE and SALARY attributes of Employees...………..
18 26
Table 3.1 Table 3.2
Distance Measures...………………...................................... Membership values as input variables in Fuzzy Function parameter estimations…………………………………......... Correlation Analysis of FCM clustering and IFC membership values with the output variable……………..… Significance Test Results of fuzzy functions using membership values obtained from FCM clustering and IFC fuzzy functions………………………………………… Functions used to generate Artificial Datasets……………... Optimum number of clusters of artificial datasets for m {1.3,2.0}……………………………………………….. Optimum Number of Clusters of IFC models of stock price dataset identified by different validity indices……….. Optimum Number of Clusters of IFC models of Ionosphere dataset indicated by different validity indices……................
55 79
Table 3.3 Table 3.4
Table 3.5 Table 3.6 Table 3.7 Table 3.8
Table 4.1 Table 4.2
Table 5.1 Table 5.2 Table 5.3 Table 5.4 Table 5.5
∈
Number of parameters of a Type-1 Improved Fuzzy Functions – T1IFF experiment Parameter………….…….... The number of parameters of an Evolutionary Type-1 Improved Fuzzy Function experiments……………………. Improved Fuzzy Functions Parameter Representation……………………………………………… Differences between the and Earlier Type-2 Fuzzy System Modeling Approach………………………………...……... The steps of the Genetic Learning Process of EDIT2IFF… Number of parameters of Discrete Interval Type-2 Improved Fuzzy Functions (DIT2IFF) …………...………. The number of parameters of Evolutionary Discrete Interval Type-2 Improved Fuzzy Functions. (EDIT2IFF)...
81
85 95 101 102 103
146 147 177
180 204 214 215
XXII
Table 6.1 Table 6.2
Table 6.3 Table 6.4 Table 6.5 Table 6.6
Table 6.7
Table 6.8 Table 6.9 Table 6.10 Table 6.11 Table 6.12 Table 6.13 Table 6.14
Table 6.15 Table 6.16
Table 6.17
Table 6.18 Table 6. 19
List of Tables
Overview of Datasets used in the experiments…………… Calculation of overall performance of a method based on three way cross validation results. The overall performance is represented with tuple of 〈 PM , stPM 〉…… Contingency Table to calculate accuracy…………………. Learning parameters of Support Vector Machines for classification and regression methods……………………... Learning parameters of 1-Layer Neural Networks………... Learning parameters of Adaptive Network Fuzzy Inference Systems – ANFIS (Takagi-Sugeno) Subtractive Clusteing Method…………………………...... Learning parameters of Dynamically Evolving Neuro-Fuzzy Inference System - DENFIS Online Learning with Higher order Takagi-Sugeno (TS) inference……….………………………………………….. Learning parameters of Type-2 Fuzzy Rule Base Approach - DIT2FRB…..………………………………..... Initial Parameters of Genetic Fuzzy System…………......... The Parameters of Type-1 and Type-2 Fuzzy Functions Methods for Regression Problems………..……………….. The Parameters of Type-1 and Type-2 Fuzzy Functions Methods for Classification Problems………..………..…… The Parameters of Type-1 and Type-2 Improved Fuzzy Functions Methods for Regression………..……………..... The Parameters of Type-1 and Type-2 Improved Fuzzy Functions Methods forClassification Problems………..….. R2 values obtained from the application Type-1 Fuzzy Functions Approaches and its variations on Training-Validation-Testing Datasets of Freidman’s Artificial Dataset.…………..…………………………….... Optimum Parameters of Variations of Type-1 Fuzzy Functions Approach variations………..……….…..…........ R2 values obtained from the application of Benchmark Approaches on Training-Validation-Testing Datasets of Freidman’s Artificial Dataset and their optimum Mode parameters…………………………………………... R2 values obtained from the application Type-2 Fuzzy Functions Approaches and its variations on Training-Validation-Testing Datasets of Freidman’s Artificial Data set…………..…………………………….... Optimum Parameters of Variations of Type-2 Fuzzy Functions Approach……..…...……….………………….... R2 values obtained from the application Earlier Type 2 Fuzzy Rule Base –DIT2FRB Approach on TrainingValidation-Testing Datasets of Freidman’s Artificial Dataset…………………………………………………….
218
221 223 228 229
230
231 232 233 235 235 236 237
239 239
240
240 240
241
List of Tables
Table 6.20 Two sample left-tailed t-test results (p<0.05) for Fried Artificial Dataset.FR: Fail to Reject the Null Hypothesis, R: Reject the Null-Hypothesis. The numbers below each decision indicate the probability of observing the decision (FR/R)……................................... Table 6.21 Auto-Mileage Dataset Variables…………………………... Table 6.22 R2 values obtained from the application Type-1 Fuzzy Functions Approaches and its variations on TrainingValidation-Testing Datasets of Auto Mileage Data set........ Table 6.23 Optimum Parameters of the variations of Type-1 Fuzzy Functions Approach……..…………..…………………….. Table 6.24 R2 values obtained from the application of Benchmark Approaches on Training-Validation-Testing Datasets of Auto-Mileage Dataset and their optimum model parameters………..…………………………………........... Table 6.25 R2 values obtained from the application variations of Type-2 Fuzzy Functions Approaches on TrainingValidation-Testing Datasets of Auto-Mileage Dataset......... Table 6.26 Optimum Parameters of variations of Type-2 Fuzzy Functions Approach……..………………………………… Table 6.27 R2 values obtained from the application Earlier Type-2 Fuzzy Rule Base –DIT2FRB Approach on TrainingValidation-Testing Datasets of Auto-Mileage Dataset......... Table 6.28 Two-sample left tailed t-test results (p<0.05) for Auto-Mileage Dataset. FR: Fail to Reject the Null Hypothesis, R: Reject the Null-Hypothesis. The numbers below each decision indicate the probability of observing the decision (FR/R)..………..……………….. Table 6.29 Desulphurization Dataset variables……………………...... Table 6.30 R2 values obtained from the application Type-1 Fuzzy Functions Approaches and its variations on TrainingValidation-Testing Datasets of Reagent1 and Reagent2 Datasets of Desulphurization Process…………..…………. Table 6.31 Optimum Parameters of variations of Type-1 Fuzzy Functions Approaches…..…………………………….…... Table 6.32 R2 values obtained from the application of Benchmark Methods on Training-Validation-Testing Datasets for Reagent1 and Reagent2 of Desulphurization Process and their optimum model parameters………..……………. Table 6.33 R2 values obtained from the application of variations of Type-2 Fuzzy Functions Approach on TrainingValidation-Testing Datasets for Reagent1 and Reagent2 Datasets of Desulphurization Process……………………... Table 6.34 Optimum Parameters of variations of Type-2 Fuzzy Functions Approach……..…………………..……………..
XXIII
243 245
246 246
247
247 247
248
249 254
256 257
257
258 258
XXIV
List of Tables
Table 6.35 R2 values obtained from the application Earlier Type-2 Fuzzy Rule Base –DIT2FRB Approach on TrainingValidation-Testing Datasets of Desulphurization Process… Table 6.36 Two-sample left tailed t-test results (p<0.05) for Reagent 1 Dataset. FR: Fail to Reject the Null Hypothesis, R: Reject the Null-Hypothesis. The numbers below each decision indicate the probability of observing the decision (FR/R).…………….……………… Table 6.37 Two-sample left tailed t-test results (p<0.05) for Reagent 2 Dataset. FR: Fail to Reject the Null Hypothesis, R: Reject the Null-Hypothesis. The numbers below each decision indicate the probability of observing the decision (FR/R)……..……………................ Table 6.38 List of variables used in Stock Price Estimation Models..... Table 6.39 Descriptions of Stock Datasets that are used in the experiments………..……………...………………..…….... Table 6.40 STB Values obtained from two different Hypothetical Models of an Artificial Stock Data. Both hypothetical models start trading with $100……….……………………. Table 6.41 RSTB Prediction errors of two Hypothetical Models* of Artificial Stock Dataset. Both hypothetical models start trading with $100.………..………...……………..….. Table 6.42 Predictions obtained from two Hypothetical Models of Artificial Stock Data and the RSTB Values. Both hypothetical models start trading with $100………............. Table 6.43 Average performance measures of models of five Real Stock Prices based on MAPE of Testing datasets. The standard deviations over five different testing dataset is shown in parenthesis………….……………………........... Table 6.44 Average performance measures of models of five Real Stock Prices based on RSTB of Testing datasets. The standard deviations over five different testing dataset Is shown in parenthesis…………..………………............... Table 6.45 Summary of performances of models on five different real stocks based on profitability values of best models of proposed, benchmark and ‘buy&hold’ methods……..… Table 6.46 The effects of broker commission on the average profit obtained from benchmark models of TD Bank stock prices. Different investment amounts are shown..…........... Table 6.47 Description of Classification Datasets from UCI Repository that are used in the experiments……….…….... Table 6.48 Performance Measure in AUC values of proposed and benchmark classification methods on six different classification datasets………….………………………….. Table 6.49 Optimum Parameters of the best classification models of five different classification datasets…………………….....
259
260
261 264 266
268
270
271
272
273
274
275 279
282 282
List of Tables
Table 6.50 Parameters of four different models that are build using four different methods to compare their elapsed times on Desulphurization data set……...…..…………………... Table 6.51 Parameters of different models that are build to measure their elapsed times. Four different methods are paired based on their optimization methods……………………… Table 6.52 Average Profit based on the RSTB Performance Measure and Ranking of system modeling techniques on five Real Stock Price Datasets…...……..………………………….... Table 6.53 Overall two-sample left tailed t-test results (p<0.05) for Stock Price Datasets. FR: Fail to Reject the Null Hypothesis, R: Reject the Null-Hypothesis. The numbers below each decision indicate the probability of observing the decision (FR/R)…….………………………………..... Table 6.54 TD Canada Trust Buy–Sell-Hold trend and Current Market values between 73- 82th days obtained from SVM and proposed Evolutionary Type-2 Improved Fuzzy Functions model (EDIT2IFF).…………………………...... Table 6.55 Average R2 and Rankings of system modeling techniques on Regression Problems. The values in parenthesis are standard deviation obtained from cross validation iterations. The values in bold indicate the optimum method of the corresponding dataset of each column………...…..... Table 6.56 Overall two-sample left tailed t-test results (p<0.05) for Regression Datasets. R: Fail to Reject the Null Hypothesis, R: Reject the Null-Hypothesis. The numbers below ach decision indicate the probability of observing the decision (FR/R).………………………………………... Table 6.57 Average AUC values and Rankings of system modeling techniques on Classification Problems. The values in parenthesis indicate the rank of each methodology for the corresponding dataset in each column...……..…………..... Table 6.58 Overall two-sample left tailed t-test results (p<0.05) for the Classification Datasets: FR: Fail to Reject the Null Hypothesis, R: Reject the Null-Hypothesis.……….…..…. Table 6.59 Overall Ranking Results of Classification Datasets…….… Table 6.60 Summary of Results………………………….…………… App. Table D.1 Friedman’s Artificial R2 results on Training Data set from Cross Validation (CV) Trials……………….. App. Table D.2 Friedman’s Artificial R2 results on Validation Dataset from Cross Validation (CV) Trials………….. App. Table D.3 Friedman’s Artificial R2 results on Testing Dataset from Cross Validation (CV) Trials…………………...
XXV
285
287
291
292
295
296
297
298
299 299 302
345 346 346
XXVI
List of Tables
App. Table D.4 Friedman’s Artificial MAPE results on Training Dataset from Cross Validation (CV) Trials………….. App. Table D.5 Friedman’s Artificial MAPE results on Validation Dataset from Cross Validation (CV)Trials…………... App. Table D.6 Friedman’s Artificial MAPE results on Testing Dataset from Cross Validation (CV) Trials………….. App. Table D.7 Friedman’s Artificial RMSE results on Training Dataset from Cross Validation (CV) Trials………….. App. Table D.8 Friedman’s Artificial RMSE results on Validation Dataset from Cross Validation (CV) Trials…………. App. Table D.9 Friedman’s Artificial RMSE results on Testing Dataset from Cross Validation (CV) Trials………….. App. Table D.10 Auto Mileage Dataset R2 values on Training Datasets of Cross Validation (CV) Trials…………… App. Table D.11 Auto Mileage Dataset R2 values on Validation Datasets of Cross Validation (CV)…………………. App. Table D.12 Auto Mileage Dataset R2 values on Testing Datasets of Cross Validation (CV)……………….…. App. Table D.13 Auto Mileage Dataset MAPE values on Training Datasets of Cross Validation (CV)…………….……. App. Table D.14 Auto Mileage Dataset MAPE values on Validation Datasets of Cross Validation (CV)………………….. App. Table D.15 Auto Mileage Dataset MAPE values on Testing Datasets of Cross Validation (CV)…………….……. App. Table D.16 Auto Mileage Dataset RMSE values on Training Datasets of Cross Validation (CV)……………….…. App. Table D.17 Auto Mileage Dataset RMSE values on Validation Datasets of Cross Validation (CV)………………….. App. Table D.18 Auto Mileage Dataset RMSE values on Testing Datasets of Cross Validation (CV)……………….… App. Table D.19 Desulphurization Reagent1 R2 values on Training Dataset of Cross Validation (CV)Trials………...….. App. Table D.20 Desulphurization Reagent1 R2 values on Valida tion Dataset of Cross Validation (CV)Trials………. App. Table D.21 Desulphurization Reagent1 R2 values on Testing Dataset of Cross Validation (CV)Trials…………… App. Table D.22 Desulphurization Reagent2 R2 values on Train ing Dataset of Cross Validation (CV)Trials……….. App. Table D.23 Desulphurization Reagent2 R2 values on Valida tion Dataset of Cross Validation (CV)Trials………. App. Table D.24 Desulphurization Reagent2 R2 values on Testing Dataset of Cross Validation (CV) Trials………....... App. Table D.25 TD Canada Trust Cross Validation (CV) RSTB values for Testing Dataset Buy_and_Hold profit =$102.28……………………………………………
347 347 348 348 349 349 354 354 355 355 356 356 357 357 358 364 365 365 366 366 367
369
List of Tables
App. Table D.26 TD Canada Trust Cross Validation (CV) MAPE% for Training Dataset…………………....... App. Table D.27 Canada Trust Cross Validation (CV) MAPE% for Validation Dataset…………………… App. Table D.28 D Canada Trust Cross Validation (CV) MAPE% for Testing Dataset………………………. App. Table D.29 Canada Trust – Market Value estimation of two prediction algorithms. The shaded region indi cates that from that day forward the two method ologies starts to predict different values for the next day stock prices………………………………. App. Table D.30 Bank of Montréal Cross Validation (CV) RSTB values for Testing Dataset…………………………. App. Table D.31 Bank of Montréal Cross Validation (CV) MAPE values for Training Dataset………………………... App. Table D.32 Bank of Montréal Trust Cross Validation (CV) MAPE values for Validation Dataset……………… App. Table D.33 Bank of Montréal Trust Cross Validation (CV) MAPE values for Testing Dataset…………………. App. Table D.34 Sun Life Cross Validation (CV) RSTB values for Testing Dataset…………………………………….. App. Table D.35 Sun Life Cross Validation MAPE for Training Dataset………………………………………........... App. Table D.36 Sun Life Cross Validation MAPE % for Valida tion Dataset………………………………………… App. Table D.37 Sun Life Cross Validation MAPE % for Testing Dataset……………………………………………... App. Table D.38 Enbridge Cross Validation (CV) RSTB values for Testing Dataset………………………………… App. Table D.39 Enbridge Cross Validation (CV) MAPE % for Training Dataset…………………………………… App. Table D.40 Enbridge Cross Validation (CV) MAPE % for Validation Dataset…………………………............. App. Table D.41 Enbridge Cross Validation (CV) MAPE % for Testing Dataset…………………………………….. App. Table D.42 Loblaw Cross Validation (CV) RSTB values for Testing Dataset…………………………………….. App. Table D.43 Loblaw Cross Validation (CV) MAPE % for Training Dataset…………………………………… App. Table D.44 Loblaw Cross Validation (CV) MAPE % for Validation Dataset………………………………… App. Table D.45 Loblaw Cross Validation MAPE % for Testing Dataset…………………………………………....... App. Table D.46 Cancer Dataset AUC values from testing dataset………………………………………………
XXVII
369 370 370
371 374 374 375 375 376 376 377 377 378 378 379 379 380 380 381 381 388
XXVIII
List of Tables
App. Table D.47 Cancer Dataset Accuracy values from testing Dataset…………………………………………….. App. Table D.48 Diabetes Dataset AUC valued from testing Dataset…………………………………………....... App. Table D.49 Diabetes Dataset Accuracy values from testing Dataset…………………………………………....... App. Table D.50 Liver Dataset AUC values from testing dataset…… App. Table D.51 Liver Dataset Accuracy values from testing Dataset…………………………………………....... App. Table D.52 Ion Dataset AUC values from testing dataset……… App. Table D.53 Ion Dataset Accuracy values from testing dataset……………………………………………… App. Table D.54 Credit Dataset AUC values from testing dataset……………………………………………… App. Table D.55 Ion Dataset Accuracy values from testing Dataset…………………………………………....... App. Table D.56 California Dataset AUC values from testing Dataset……………………………………………... App. Table D.57 California Dataset Accuracy values from testing Dataset……………………………………………...
388 389 389 390 390 391 391 392 392 393 393
List of Figures Contents
Fig. 1.1 Contents of the book at a glance………………………………. Fig. 2.1 Membership functions of fuzzy sets “young” (green), “middle age” (red) and “old” (blue)…………………………… Fig. 2.2 Fuzzy Set Properties on a “Trapezoidal” membership function example…………………………………………........ Fig. 2.3 Union and Intersection………………………………………... Fig. 2.4 Complement of Fuzzy Set A,Ac. (--- line) ………………….... Fig. 2.5 Crisp Binary Relation (Equalityrelation, E, between x and y) ……………………………………………………...…. Fig. 2.6 Cartesian product of two fuzzy sets (A and B)……………….. Fig. 2.7 Fuzzy set of young employees (A(x)) ………………………... Fig. 2.8 Fuzzy set of salary of young employees (B(y))………………. Fig. 2.9 Comparison of Type-1 and Type-2 Fuzzy sets. The shaded region on the right graph is the secondary membership function of x…………………………………………………… Fig. 2.10 Example of a full type 2 membership function. The shaded area is the ‘Footprint of Uncertainty’ (FOU). The amplitude of the sticks is the secondary membership values……………….. Fig. 2.11 Interval Valued Type-2 Fuzzy Set…………………………….. Fig. 2.12 Generalized Type 1 Fuzzy Inference Systems (Fuzzy Logic System)……………………………………………………….. Fig. 2.13 TSK Fuzzy Reasoning…………………………..……….……. Fig. 2.14 A Sample ANFIS Structure……………………..…………….. Fig. 2.15 Chromosome structure of a Generic Fuzzy System. Each ai j and bi j indicates the triangular membership function parameters….………………………………………………… Fig. 2.16 The effect of changing parameters on membership function……………………………………………………......
Fig. 3.1 Scatter Plot of artificial data with two different random funtions.f1(x)=y+radom_noise,f2(x)=0.3+random_Noise….... Fig. 3.2 Types of Membership value calculation equations…………... Fig. 3.3 Scatter plot of the artificial dataset………………………........
3
13 14 16 16 23 24 27 27 29
30 31 39 42 42
48 49
59 78 80
XXX
List of Figures
Fig. 3.4 (White Surface) Actual Decision Surface using Member Ship values (from the FCM clustering), (Black Linear Surface) the estimated linear decision surface. ‘u’ indicates the membership values.……………………………. Fig. 3.5 (White Surface) Actual Decision Surface using Member ship values (from the IFC), (Black Linear Surface) the estimated linear decision surface. ‘u’ indicates the improved membership values..………………………............. Fig. 3.6 Compactness and Separation concepts of ratio-type CVI index.……………………………………………………......... Fig. 3.7 Four Different Functions with Varying Separability Relations……………………………………………………… Fig. 3.8 Dataset 1 - Cluster validity measures, XB, XB*, Kung-Lin and cviIFC, versus c for two m values;m=1.3 (.-) and m=2.0(*-) using 4-patterned dataset………………………….. Fig. 3.9 Dataset 2- Cluster validity measures, XB, XB*, Kung-Lin and cviIFC, versus c fortwo m values; m=1.3 (.-) and m=2.0(*-) using 5-patterned dataset………………………….. Fig. 3.10 Dataset 3- Cluster validity measures, XB, XB*, Kung-Lin and cviIFC, versus c for two m values; m=1.3 (.-) and m=2.0(*-) using 7-patterned dataset.…………………………. Fig. 3.11 Dataset 4-Cluster validity measures, XB, XB*, Kung-Lin and cviIFC, versus c for two m values; m=1.3 (.-) and m=2.0(*-) using 9-patterned dataset…………...………..……. Fig. 3.12 Density graph of stock price dataset using two financial indicators. Two components that are well separated are indicated…………...…………………………...……….…….. Fig. 3.13 Stock Price Dataset - Cluster validity measures, XB, XB*, Kung-Lin and cviIFC, versus c for two m values; m=1.3 (.-) and m=2.0(*-)……………...……………….…….. Fig. 3.14 Ionosphere Dataset- Cluster validity measures, XB, XB*, Kung-Lin and cviIFCC, versus c for two m values; m=1.3 (.-) and m=2.0(*-)………...………………….….……. Fig. 4.1 Framework of fuzzy system models with “Fuzzy Functions” approach…………………………………………... Fig. 4.2 Evolution of Type-1 Fuzzy Functions Strategies …...………... Fig. 4.3 Modeling with Fuzzy Functions Systems …………................. Fig. 4.4 Mapping the Input-Output space onto individual clusters using a one dimensional Toy Example. Support Vector Regression is used to find the decision surfaces for two clusters in (1+1) space, i.e., only membership values are used as additional dimensions. ………………..……………… Fig. 4.5 Type-1 Improved Fuzzy Functions Framework based on sub-sampling cross validation method………………………...
83
84 88 93
95
96
96
97
97
99
100
110 112 113
116 125
List of Figures
Fig. 4.6 Structure of the T1IFF approach. (Top) Structure Identification with Improved Fuzzy Functions, (Bottom) Improved fuzzy function i.………………………………....... Fig. 4.7 Implementation of Improved Fuzzy Clustering in T1IFF………………………………………………………….. Fig. 4.8 Topology of Type-1 Improved Fuzzy Functions Systems for Classification Models…..…………………………………. Fig. 4.9 Proposed Inference Mechanism of T1IFF..…………..…… Fig. 4.10 Evolutionary Improved Fuzzy Functions (ET1IFF) System Modeling Framework –Extension of the Improved Fuzzy Functions (T1IFF) Approach to Genetic Fuzzy Systems. i=1…c………………………………………..……… Fig. 4.11 Genome Encoding for ET1IFF systems. (Top) Hierarchi cal structure genome encoding when linear regression is used (Middle) Encoding when SVM is used (Bottom) Example chromosome structure. ……………………………... Fig. 4.12 Decision surfaces of one chromosomes of ET1IFF model using two different fuzzy function structures. (m,c, Creg, )={1.75,3,54.5,0.115}. uci represents membership values of corresponding cluster. ……………………………… Fig. 4.13 Genome Two different chromosomes from the GLP algorithm of ET1IFF modeling approach applied on the Artificial Dataset. The dark colored token is the only difference between the two chromosomes. ‘1’:Linear Kernel Model, ‘2’:Non-Linear Kernel Model. ……...………... Fig. 4.14 Genome Encoding for ET1IFF-C systems for Classification Domains. (Top) Hierarchical structure genome encoding for linear regression (Middle) Encoding when SVM is used (Bottom) Example chromosome structure...................................................................................... Fig. 5.1 Type-1 Improved Fuzzy Functions Systems. ‘*’ are Mendel’s terminology which correspond to terminology used for fuzzy function strategies in this book..….................. Fig. 5.2 Type-2 Improved Fuzzy Functions System. * indicate difference between Mendel’s terminology and Fuzzy Function terminology.…........................................................... Fig. 5.3 Different Types of Uncertainty................................................ Fig. 5.4 Example of a full type 2 membership function. The shaded area is the ‘Footprint of Uncertainty’ (FOU). The amplitude of the sticks is secondary membership values......... Fig. 5.5 The structure of Generalized Type-2 Fuzzy Rule Base Systems (GT2FRB)…………………………………….……. Fig. 5.6 DIT2FRB inference strategy by [Uncu, 2003; Uncu and Turksen, 2007]…………………………………………...
XXXI
127 127 131 134
138
140
143
144
145
151
152 155
158 158 165
XXXII
List of Figures
Fig. 5.7 Interval Type-2 membership function with fixed mean and uncertain standard deviation. The shaded area in cludes infinite number of membership values……………….. Fig. 5.8 Artificial dataset with one input variable (x1) and a single output (y) ………………………………………………......... Fig. 5.9 Membership values for cluster 1 using the artificial data set determined by changing m values………………………... Fig. 5.10 Changing cluster center locations of cluster 1 based on changing m values. The square dots in bold indicate the cluster centers for changing m values………………………... Fig. 5.11 Uncertainty interval of a cluster represented with membership functions of 3-tuples, < c,mr,τs>. c is the number of clusters, mr defines the level of fuzziness, τs represents the interim fuzzy function structures of the Improved Fuzzy Clustering (IFC).…………………………... Fig. 5.12 A general topology of the DIT2IFF structure identification. The shaded fuzzy functions represent the fuzzy functions of the corresponding cluster. The output value of kth data point, yk, is calculated using each output obtained from these fuzzy functions. …………………….….. Fig. 5.13 DIT2IFF Inference Framework. Grey boxes depict enhanced/newly introduced operations compared to earlier DIT2FRB approach.……………………………….…. Fig. 5.14 Cased Based Type reduction for inference method of DIT2IFF system. The dark dots (●) represent the opti mum membership values and fuzzy functions determined for each testing vector, l, l=1nte……………………………... Fig. 5.15 RMSE values of improved fuzzy functions for a specific cluster i. Each line represents one type of fuzzy functions denoted in the legend of the figure. u represents membership values.………………………………………….. Fig. 5.16 The architecture of Evolutionary Discrete Interval Type-2 Improved Fuzzy Functions…………………………….…….. Fig. 5.17 EDIT2IFF design – Phase 1. The dark dot (●) represent the tokens (each gene) corresponding to the parameters specific to the function type, e.g., if SVM is chosen, these parameters would include the regularization (Creg), epsilon (ε), Kernel Type (K(.)). Darker colored tokens represent control genes………………………………….… Fig. 5.18 Genetic Learning Process – Phase 1 of Evolutionary Interval Type-1 Fuzzy Functions models, identifying (TOP) interval valued membership functions, (BOTTOM) interval for fuzzy functions……………..…..
172 174 175
175
181
183
188
190
195 197
199
201
List of Figures
Fig. 5.19 Decision surfaces obtained from GLP for changing fuzzy function structures of chromosomes. K(.)={Linear or Non-linear} and (mlow,mup,Creg,ε)= {1.75,2.00,54.5,0.115},c*=3. uclusi represents improved membership values of corresponding cluster i.…………........ Fig. 5.20 Two different chromosomes from the GLP algorithm of EDIT2IFF modeling approach applied on the Artificial Dataset. The dark colored token is the only difference between the two chromosomes. ‘1’:Linear Kernel Model, ‘2’:Non-Linear Kernel Model.……………………………… Fig. 5.21 Interval valued membership values of evolutionary type2 fuzzy functions strategy. Uncertainty interval repre sented with membership values dispersion induced by each tuple <mr,τs> (a) Optimized uncertainty interval from GLP-Phase 1 of EDIT2IFF, (b) Discrete Improved Membership Values- Phase2 EDIT2IFF (c) magnified view of different discrete membership values for any x′…….. Fig. 5.22 Uncertainty interval of Fuzzy Function Structures. (a) Uncertainty interval represented with different output values obtained from list of fuzzy function structures induced by each tuple <mr,τs,Φp> (b) Optimized uncertainty interval from GLP-Phase 1 of EDIT2IFF, (c) magnified view of different output values, yik r,s,ψ, for a specific x′ vector obtained form optimized list of fuzzy functions. ……………………….…………………………… Fig. 6.1 General Framework of the Three-Way Cross Validation Method……………………………………………………….. Fig. 6.2 Three-way cross validation used in Fuzzy Functions approaches…………………………………………………… Fig. 6.3 Schematic view of the Three-Way Cross Validation Process that is used for stock price estimation models………. Fig. 6.4 A sample Receiver Operating Characteristic – ROC curve………………………………………………………….. Fig. 6.5 The ROC “curve” created by varying threshold val ues.The table at right shows 20 data points and the score (probabilities) assigned to each of them. The graph on the left showsthe corresponding ROC curve with each point labeled by the threshold that produces it………………. Fig. 6.6 Scatter diagram of selected variables from Friedman Artificial dataset………………………..…………………….. Fig. 6.7 Average Cross Validation Test R2 values of Optimum Models –Freidman’s Artificial Dataset. Standard errors of five repetitions are shown on the curve………………………
XXXIII
205
207
210
211
219 220 220 224
225 238
241
XXXIV
List of Figures
Fig. 6.8 Cross Validation Test R2 values of Optimum Models – Friedman’s Artificial Dataset. Standard errors of five repetitions are shown on the curve…………………..………. Fig. 6.9 Correlation between Performance Measures of each Friedman’s Training, Validation and Testing Dataset for five cross validation models.cviXXX represent the cross validation models, TR: training, VR: validation, TE: testing datasets……………………………………………….. Fig. 6.10 Average Cross Validation Test R2 values of Optimum Models – Auto Mileage Dataset. Standard errors of five repetitions are shown on the curve……………………..……. Fig. 6.11 Auto-Mileage Dataset Cross Validation Test Performance measures of Optimum Models using R2, MAPE, and RMSE measures. Standard errors of five repetitions are shown on the curve…………………….…….. Fig. 6.12 Correlation between Performance Measures of AutoMileage Dataset Training, Validation and Testing Data sets for five cross validation models. cviXXX represent the cross validation models, TR: training, VR: validation, TE: testing datasets.………………………………………..… Fig. 6.13 Process flow of Hot Metal Pretreatment…………………….. Fig. 6.14 A Blast Furnace at a Hot Metal Pre-treatment plant……….... Fig. 6.15 Torpedo Car Desulphurization Process…………………….... Fig. 6.16 Distribution of Reagent1 versus Reagent2 variables of Desulphurization Dataset…………………………………….. Fig. 6.17 Kernel density estimation of Reagent 1 for four grades of Aim-sulphur (discrete variable) …………………………...… Fig. 6.18 Average cross validation R2 values of optimum models of Desulphurization Reagent 1 Dataset………………........… Fig. 6.19 Average cross validation R2 values of optimum models of Desulphurization Reagent 2 Dataset…………………....… Fig. 6.20 Two financial indicators of a selected Stock’s Price versus Closing Stock Price………………………………...… Fig. 6.21 TD Canada Trust – 400 days historical price (23 Nov 2005-8 June 2007). First 300 days are used for training and validation to optimize the model parameters, and the last 100 days are used for testing the performance of the optimum model. ……………………………………….…….. Fig. 6.22 Loblaws – 444 days historical price (Nov. 14 2005Aug.21, 2007). First 344 days are used for training and Validation to optimize the model parameters, and the last 100 days are used for testing the performance of the optimum model.………………………………………………
244
244
248
250
251 251 252 252 254 255 259 259 263
265
265
List of Figures
Fig. 6.23 Bank of Montreal – 400 days historical price (Nov 11 2005-Aug 20 2007). First 200 days are used for training and validation o optimize the model parameters, and the last 100 days are used for testing the Performance of the optimum model………………………………………………. Fig. 6.24 Enbridge – 400 days historical price (Nov 14 2005-Aug 21 2007). First 200 days are used for training and validation to optimize the model parameters, and the last 100 days are used for testing the performance of the optimum model……………………………………………..... Fig. 6.25 Sun Life – 400 days historical price Nov. 11, 2005-Aug 20, 2007). First 345 days are used for training and validation to optimize the model parameters, and the last 100 days are used for testing the performance of the optimum mode………………………………………………………….. Fig. 6.26 Actual and Predicted Stock Prices of an artificial Dataset of Stock Prices for a 10 days period. Predicted stocks from Hypothetical Model 1, Hypothetical Model2 and Actual Prices are shown. Model 2 is not a good model of the actual stock prices……………………………………...… Fig. 6.27 The effect of broker commission on profit for different values of investments………………………………………… Fig. 6.28 Proposed cviIFC cluster validity values of proposed IFC results for Friedman’s Artificial Dataset for two different fuzziness values; m={1.3,2.0}…………………………..…… Fig. 6.29 Automobile Mileage Dataset cviIFC graphs for two different fuzziness values; m={1.3,2.0}…………………. Fig. 6.30 The Elapsed times (min) of structure identification (learning) and reasoning algorithms of optimum methodologies using different samples of training and testing datasets, respectively. (left) the training (learning) periods, (right) the inference periods……………………….... Fig. 6.31 Elapsed times in minutes of structure identification of different fuzzy function systems……………………………... Fig. 6.32 TD Canada Trust Stock Prices of Testing dataset (100 day trading period) – (Top) Actual and Predicted using two different algorithms, SVM and Proposed EDIT2IFF, (bottom) current-market-values at each t – trading day obtained from estimated models……………... Fig. 6.33 Magnified View of TD Canada Trust Stock Prices in Figure 6.32- Actual versus Predicted and Current Market Values………………………….…………………………… Fig. 6.34 Comparison of System modeling techniques based on classification Ranking Results………………….…………….
XXXV
265
265
265
270 276
277 278
286 288
293
294 300
XXXVI
List of Figures
App. Fig. A.1 Composition of binary fuzzy relations R and S……….. App. Fig. C.1 A Neural Network with an input layer, hidden layer and one-output neurons………………………………... App. Fig. C.2 Separating Hyperplane with relaxed constraints…......... App. Fig. C.3 Soft Margin Loss Function……………………….…… App. Fig. C.4 One-point Crossover………………………………...… App. Fig. C.5 Uniform Crossover…………………………………….. App. Fig. D.1 Idealized distributions for treated and comparison group values……………………………………….. App. Fig. D.2 Diabetes Dataset CVI Graphs of IFC-C method for two different fuzziness values; m={1.3,2.0}……..…… App. Fig. D.3 Ion Dataset CVI Graphs of IFC-C method for two different fuzziness values; m={1.3,2.0}……………..... App. Fig. D.4 Liver Dataset CVI Graphs of IFC-C method for two different fuzziness values; m={1.3,2.0}………...... App. Fig. D.5 Credit Dataset CVI Graphs of IFC-C method for two different fuzziness values; m={1.3,2.0}………….. App. Fig. D.6 Cancer Dataset CVI Graphs of IFC-C method for two different fuzziness values; m={1.3,2.0}…………. App. Fig. D.7 Ionosphere Dataset– Receiver Operating Characteristic (ROC) Graphs of Optimum model DIT2FF versus (a) Logistic Regression, (b) ANFIS, (c) Neural Networks, (d) Support Vector Machines…………………………………………..…… App. Fig. D.8 Credit Dataset – Receiver Operating Characteristic (ROC) Graphs of Optimum model - DIT2IFF versus (a) Logistic Regression, (b) ANFIS, (c) Neural Networks, (d) Support Vector Machines………. App. Fig. D.9 Cancer Dataset – Receiver Operating Characteristic (ROC) Graphs of Optimum model - DIT2FF versus (a) Logistic Regression, (b) ANFIS, (c) Neural Networks, (d) Support Vector Machines…….... App. Fig. D.10 Liver Dataset – Receiver Operating Characteristic (ROC) Graphs of Proposed DIT2FF versus (a) Logistic Regression, (b) ANFIS, (c) Neural Networks, (d) Support Vector Machines…………….... App. Fig. D.11 Diabetes Dataset- Receiver Operating Characteristic (ROC) Graphs of Proposed DIT2IF versus (a) Logistic Regression, (b) ANFIS, (c) Neural Netorks, (d) Support Vector Machines……..….
323 328 330 336 339 340 344 397 397 397 398 398
398
399
399
400
400
Notation
Preface
A
norm matrix A
A,B,C
type-1 fuzzy sets
A, B , C
complement of type-1 fuzzy sets.
A , B , C
type-2 fuzzy sets
Ai
aggregated input fuzzy set characterizing the antecedent of the ith rule
Aij
fuzzy set associated with the jth input variable in ith rule.
Ai
aggregated type-2 fuzzy set characterizing the antecedent of the ith rule.
Aij
type-2 fuzzy set associated with the jth input variable in ith rule.
Air
aggregated fuzzy set characterizing the antecedent of the ith rule identified by level of fuzziness mr
α
A, α+A
The alpha-cut and strong alpha-cut of fuzzy set A
ANFIS
Adaptive Network Based Fuzzy Inference System
a,b,c
membership values
B*
model output fuzzy set
Bi
output fuzzy set in ith rule
Bi
type-2 output fuzzy set in ith rule
c
number of clusters
c*, c-optimal
optimum number of clusters
XXXVIII
Notation
C-reg, C-regularization
regularization parameter of support vector machines methodology
Core(A)
The core of fuzzy set A
cviIFC
proposed cluster validity index, i.e., formula, for improved fuzzy clustering
cviIFC-C
proposed cluster validity index, i.e., formula, for improved fuzzy clustering for classification problem domains
CNF
Conjunctive Normal Forms
Dr
column matrix of distance values characterizing the nearest κ data-points to the rth testing data vector
DENFIS
Dynamically Evolving Neural Fuzzy Inference System
DNF
Disjunctive Normal Forms
DIT2FRB
Discrete Interval Valued Fuzzy Rule Base
DIT2FF
Discrete Interval Type-2 Fuzzy Functions
DIT2IFF
Discrete Interval Type-2 Improved Fuzzy Functions
drq
distance between the rth testing vector and qth training vector
d=||xk-vi(x)||, d(xk,vi(x)), dik(x)
the distance value between the kth data vector and ith cluster center vector, vi(x).
di(xy)
the distance value between the kth input-output data vector and ith cluster center vector, vi(x|y).
det
determinant
ET1FF
evolutionary type-1 fuzzy functions
ET1IFF
evolutionary type-1 improved fuzzy functions
ET1FF-C
evolutionary type-1 fuzzy functions for classification
ET1IFF-C
evolutionary type-1 improved fuzzy functions for classification
EDIT2FF
evolutionary discrete interval type-2 fuzzy functions
EDIT2IFF
evolutionary discrete interval type-2 improved fuzzy functions
Notation
XXXIX
EDIT2FF-C
evolutionary discrete interval type-2 fuzzy functions for classification
EDIT2IFF-C
evolutionary discrete interval type-2 improved fuzzy functions for classification
FCM
fuzzy c-means clustering algorithm
FCNF
Fuzzy conjunctive normal forms
FCRM
fuzzy c-regression clustering method, i.e., fuzzy switching regression
FDNF
Fuzzy disjunctive normal forms
FIS
Fuzzy Inference System
FRB
fuzzy rule bases
fi(Φi)
type-1 fuzzy function identified by the input matrix of ith cluster in feature space
fi(Φi,Ŵi)
type-1 fuzzy function identified by the input matrix of ith cluster in feature space and the regression coefficients represented with Ŵi.
fir,s,ψ
type-1 fuzzy function of cluster i characterizing the regression function of ith rule identified by using tuples of 〈mr,τis,Φiψ〉
fip(Φip Ŵip)
Fuzzy functions of cluster i identified by the parameters of the pth chromosome of evolutionary algorithm model.
GFS
Genetic Fuzzy System
GT2FRB
General Type-2 Fuzzy Rule Bases
gi(τi), gi(τi,ŵi),
interim fuzzy classifier functions identified by improved fuzzy clustering algorithm for classification problem domains, whose coefficients are represented with ŵi
hi(τi), hi(τi ⎜ŵi), hi(⋅)
interim fuzzy functions identified by improved fuzzy clustering algorithm, characterized by coefficients of ŵi
H(A)
Height of fuzzy set A
IFC
improved fuzzy clustering algorithm for regression problem domains
IFC-C
improved fuzzy clustering algorithm for classification problem domains
XL
Notation
IFF
improved fuzzy functions
IT2FRB
interval type-2 fuzzy rule base model
J
objective function value of FCM clustering algorithm
J(t)
objective function value of FCM clustering algorithm at tth iteration
JmIFC
objective function value of the improved fuzzy clustering (ifc) given a level of fuzziness (m)
K(.)
kernel function
K(xk,xj)
m* r
kernel function between two vectors, i.e., vector i and j optimal level of fuzziness
m
rth level of fuzziness value in interval type-2 fuzzy membership values
mk*
the value for which the corresponding type-2 fuzzy system model yields the least prediction error for kth input data vector in training dataset.
m-Col
collection table that holds the optimum level of fuzziness values of each input data vector
NN
neural networks
NNC
neural networks for classification problems
NUP
Number of uncertainty parameters to identify the uncertainty interval of membership values and fuzzy functions
n
number of data points (objects or observations or patterns) in training dataset.
niα
reduced number of data points of an input matrix of ith cluster characterized by a threshold, α, based on membership values
ndv
number of data points in validation dataset
nf
total number of discrete values of fuzzy function structures (Φψ), ψ=1…nf
nif
total number of discrete values of interim fuzzy function structures (τs), s=1…nif, of improved fuzzy clustering.
Notation
XLI
nr
total number of discrete values of level of fuzziness (mr), r=1…nr
nte
number of objects in testing dataset
nv
number of input variables in the dataset
OTV
optimum threshold value
Pi(yk=1|Φik)
predicted posterior probability of the kth data vector in feature space of ith cluster, Φik, that its output will be 1
Pμ
indicates the domain of parameters of membership function
PL
domain of learning parameters of fuzzy system modeling approaches
PmL
domain of learning parameters identified by level of fuzziness
PτL
domain of interim fuzzy function structures identified by interim matrix structure of IFC
PΦL
domain of fuzzy function structures identified by input matrix structure in the feature space
Pˆi
posterior probability values of data vectors obtained from local fuzzy classifier functions of ith cluster
Pˆik (yk=1)
predicted posterior probabilities of the output value yk=1 for a vector k from local fuzzy classifier functions.
pˆ i
posterior probability values of data vectors obtained from interim fuzzy classifier functions of ith cluster
ˆpik (yk=1) or ˆpik (yk=1|τik)
predicted posterior probabilities of the output value yk=1 for a vector k from interim fuzzy classifier functions.
q
the number of different embedded type1 improved fuzzy function (T1IFF) models to be constructed using parameter set of {c*,mr,τs,Φp}, c*=2,..n, r=1…nr, s=1…nif, p=1,…(nf)c*,q=1,…,nif×nr×(nf)c*
r
the identifier for different level of fuzziness parameter, r=1…nr
RP
recognition performance
XLII
Notation
Ri
ith rule in type-1 fuzzy rule base
RSTB
Robust simulated trading benchmark
Ri
ith rule in type-2 fuzzy rule base
S
S-norm connective
SEiq
square error between the actual output and the predicted output of data vector q, using the local model or interim fuzzy function model of ith cluster.
Supp(A)
Support of fuzzy set A
SVC
support vector machines for classification models
SVR
support vector machines for regression models
s
indicator for support vector
s
the identifier for different interim fuzzy function structures, τs, s=1..nif
T1FF
type-1 fuzzy functions
T1IFF
type-1 improved fuzzy functions
T1FF-C
type-1 fuzzy functions for classification
T1IFF-C
type-1 improved fuzzy functions for classification
T
T-norm connective
T
transpose operation
TSK
Takagi-Sugeno-Kang fuzzy inference system
t
indicator for iteration number
t
the number of different embedded IFC models to be constructed using parameter set of {mr,τs}, r=1…nr, s=1…nif, t=1,…,nif×nr
te
testing dataset indicator
U
partition (membership value) matrix, u⊂[0,1]n×c
U*
optimal partition (improved membership value) matrix, u*⊂[0,1]n×c
u
membership value
Notation
XLIII
V
matrix of cluster centers of vectors of the same form as xk , V⊂ℜc×nv
v
indicator for validation matrix
vi(t)
cluster center vector, i.e., cluster prototype, of a cluster I identified at the tth iteration of fuzzy clustering algorithm
vi(x), vi
cluster center vector, i.e., cluster prototype, of a cluster i
W
objective function of FCM after Lagrange transformation
Ŵi
coefficients of a (system) fuzzy function of a particular cluster i obtained from any regression approximation function e.g., Ŵi = [Ŵ0i Ŵ1i … Ŵnv+nm,i], where there are nv+nm different variables including nv dimensional input variables and nm dimensional membership values and their transformations of a cluster i. Ŵip*
optimal fuzzy function parameters of cluster i identified with the pth chromosome
Ŵir,s,ψ
Parameters of local improved fuzzy functions of cluster i identified with mr, τs, Φψ
Wˆi*,k
optimum parameters of local fuzzy functions of cluster i identified for kth training data vector
Wˆ j*,i ,k
The optimum coefficient of fuzzy functions of cluster i characterized for the jth input variable of kth input data vector
ŵi
coefficients of an interim fuzzy functions obtained from improved fuzzy clustering algorithm, e.g., ŵi = [ŵ0i ŵ1i … ŵnm,i] where there are nm different membership values and their transformations used to estimate the interim fuzzy function of a cluster i, not including original input variables.
ŵi*
Optimum coefficients of an interim fuzzy functions obtained from improved fuzzy clustering algorithm, e.g., ŵi = [ŵ0i ŵ1i … ŵnm,i] where there are nm different membership values and their transformations used to estimate the interim fuzzy function of a cluster i, not including original input variables.
XLIV
Notation
ŵj,i
the interim fuzzy function coefficient of the jth input variable in cluster i
ŵ*j,k
The optimum interim fuzzy function coefficient for the jth input variable of kth data vector
Xv
The domain of validation dataset
X
test
The domain of testing dataset
XB
Xie-Beni cluster validity index
XB*
improved Xie-Beni cluster validity index
xk
kth input data vector of the training dataset, xk=(x1,k, …, xnv,k).
xkv
kth input data vector of the validation dataset, xkv=(x1kv, …, xnvkv).
xktest
kth input data vector of the testing dataset, xktest=(x1ktest, …, xnvktest).
xj,k
jth variable of kth data vector.
x'
singleton observation
xj'
The value of jth input variable in the given singleton observation
Y
column matrix of output variable of the given system.
yk
output variable of the kth input data vector.
ykv
output variable of the kth input data vector in validation dataset.
yktest
output variable of the kth input data vector in testing dataset.
r , s ,ψ
yˆik
predicted output value of kth input data vector using the fuzzy function of cluster i for tuples of 〈c*, mr,τis,Φiψ〉
Z+
positive integers
Z
empirical dataset of all system variables. z={x1, …, xnv, y}⊆ℜ(nv+1) in (nv+1) dimensions.
zk, , z(xk,yk)
kth data vector of inputs and the output, e.g., zk=[xk,yk]
Φi , Φi(x,μi)
phi: the training matrix of ith cluster in feature space, composed of original input vectors x, of nv different input variables, and nm different transformations of membership values (μi) to approximate local fuzzy functions.
Notation
XLV
Φik
phi: the kth vector from the training matrix of ith cluster in feature space.
Φψ
(ψ)th input matrix structure in feature space comprised of original input variables (x) membership values and their transformations to be used in type-2 fuzzy functions strategies.
Φiψ(x,mr,τs) or Φir,s,ψ
(ψ)th input matrix structure of cluster i in feature space characterized with parameters of mr and τs
Φ p = {Φψi =1 , Φψi = 2 ...Φψi =c*}
(p)th embedded T1IFF model identifies different local fuzzy function structures, ψ=1,…,nf, in each of its clusters, i=1…c*.
Φi(x,μi), Φi(x,τi)
the training input matrix of ith cluster in feature space to identify “fuzzy functions”
Φi′
Sample input matrix of cluster i in feature space
Φ-Col
The collection table that holds the optimum parameters and structures of the local fuzzy functions identified for each training data vector
α, α-cut
alpha:the threshold for membership values
(α
* ik
lagrange multiplier of kth data vector in ith cluster used in support vector machines algorithm.
(α
r ,s ,ψ ik
,α ik )
,α ik* r ,s ,ψ
)
lagrange multiplier of kth data vector in ith cluster used in support vector machines algorithm identified by the T1IFF model with parameters mr, τs, Φψ
βk
beta: Lagrage multipliers of k th data vector identified by SVM. IT also indicates the parameters of switching regression function of cluster k
βik
beta: Lagrage multipliers of k th data vector in cluster i identified by SVM.
δ
gamma: Gaussian radial basis kernel function parameter indicating the spread of the Gaussian distribution of similarities
ε, epsilon
epsilon parameter of an SVM model indicating the error between the actual and predicted output. It also indicates the error value of regression equation
XLVI
Notation
εi, epsilon
epsilon parameter of SVM model for ith cluster, viz., rule.
η
eta: indicates a user defined constant of membership values calculation function of an hybrid clustering methodology
κ
kappa: Number of nearest training vectors identified for each testing data vector during inference
μi
mu:The list of membership values of data points in cluster i
μiimp
the improved membership values scatter diagram of ith cluster
μik (t)
membership value of kth input vector in cluster i identified during tth iteration of FCM clustering method
μik , μik(x)
membership value of input vector xk in cluster i.
μikimp , μikimp(x)
improved membership value of input vector xk in cluster i.
μik(xy)
membership value of input-output vector xk in cluster i.
μikimp
improved membership value of kth input vector in cluster i obtained from ifc algorithm.
μikimp*
Optimum improved membership value of kth input vector in cluster i obtained from IFC algorithm.
μi c*,r , s ( x) or μr,s(x)
membership value scatter diagram of ith cluster identified by parameters of c*, mr and τs
μ A ( x)
Membership function of type-1 fuzzy set A
μ A ( x)
Membership function of type-2 fuzzy set Ã
μi ( x)
aggregated type-2 membership values of cluster i characterizing the antecedent of the ith rule
μi ( x j )
type-2 membership function of the jth antecedent variables in cluster i
μ R ( x, y )
membership function of type-2 fuzzy relation
i
R of rule i
Notation
XLVII
μir ( x)
Membership values of input matrix in ith rule identified by using mr assuming the number of clusters is fixed
μ A ( x, u )
Primary membership function of type-2 fuzzy
set A
μiL ( x j )
lower membership function of interval type-2 fuzzy set of jth antecedent variable
μiU ( x j )
upper membership function of interval type-2 fuzzy set of jth antecedent variable
μ i* ( y )
interval valued type-2 model output fuzzy set
μ i*L ( y )
lower membership function of interval valued type-2 model output fuzzy set
μ i*U ( y )
upper membership function of interval valued type-2 model output fuzzy set
θ
primary membership values of type-2 fuzzy sets
τs
interim matrix identified with (s)th interim fuzzy function structure to construct interval interim fuzzy functions
τ*i , τ *i ( μiimp* )
optimal interim matrix used by IFC method
τi′
a user defined interim matrix
τ-Col
the collection table that holds the interim fuzzy functions structure and parameters identified for each training vector
ψ
the identifier for different (system, primary) fuzzy functions, Φψ, ψ=1..nf
ℜ
real numbers
℘
Constant to calculate the maximum number of clusters of a fuzzy clustering algorithm
Æ
mapping or implication operator
Λ
Level set of a fuzzy set
°
meet operator join operator composition operator
XLVIII
★
Notation
general t-norm
∧ ∨
min operator
∪
union (disjunction) operator
∩
intersection (conjunction) operator
max operator
Chapter 1
Introduction 1 Introduction
This chapter presents the motivation of this book. The new methodologies are briefly presented and the contents of each chapter are given.
“I coined the word –fuzzy- because I felt it most accurately described what was going on in the theory. I could have chosen another term that would have been more "respectable" with less pejorative connotations. I had thought about "soft", but that really did not describe accurately what I had in mind. Nor did – unsharp-, -blurred-, or –elastic-. In the end, I couldn't think of anything more accurate so I settled on –fuzzy-.” Prof. Lotfi A. Zadeh
1.1 Motivation Uncertainty exists in almost every real world problem. In general, uncertainty is inseparable from measurement. It emerges from a combination of limits of measurement with instruments and unavoidable errors in measurement. In cognitive problems, uncertainty emerges from the vagueness and ambiguity inherent in natural languages. In social intercourse, it may emerge from shared meanings obtained from people in social interactions. Thus, uncertainty is essential to human beings at all levels of their interaction with the real world. It is believed that an important turning point in the evolution of the modern concept of uncertainty is the publication of Prof. Lotfi A. Zadeh’s seminal paper in 1965. In his paper, Zadeh introduced a theory of fuzzy sets, which are sets with imprecise boundaries. The unique property of fuzzy sets is that, memberships in a fuzzy set are not a matter of acceptance or denial, but rather a matter of degree. Fuzzy System Modeling has been studied to deal with complex, not clearly explained and uncertain systems, in which conventional mathematical models may fail to reveal satisfactory results. Well-known fuzzy systems are based on fuzzy sets and logic, which are especially useful for systems in which the concepts are vague. In most fuzzy models of fuzzy systems, fuzzy rule bases are used together A. Celikyilmaz and I.B. Türksen: Model. Uncertain. Fuzzy Logic, STUDFUZZ 240, pp. 1–10. springerlink.com © Springer-Verlag Berlin Heidelberg 2009
2
1 Introduction
with membership functions of fuzzy linguistic terms for input-output variables. In these models, the input and output relationships are represented by IF…THEN rules. In current literature, there are variations of fuzzy inference systems with different structure identification techniques, some of which will be reviewed in the following chapters. Fuzzy rule bases comprise a series of fuzzy operators to fuzzify, aggregate and then map the input membership values onto output domain and then aggregate and defuzzify the membership values of the output variables. Identification of fuzzy sets, and their parameters, the shape and number of membership functions, and their parameters, etc. have been the focus of most fuzzy researchers. Numerous fuzzy system-modeling approaches have been introduced to combine other soft computing approaches to optimize the parameters, or to use higher order fuzzy sets to capture uncertainties. Structurally, these approaches are based on fuzzy rule bases, which should not be the only structure to build fuzzy systems. We believe that these approaches have some difficulties, which will be addressed throughout the following chapters. Thus, the main aim of this book is to present an alternative approach to fuzzy rule bases, entitled, “Fuzzy Functions”. The new Fuzzy Functions approach employs membership values differently than any other fuzzy system models implemented to date. The membership values can be thought of as ‘atoms’ that hold potential information about a system behavior to be activated to release its power. Such potential information obtained from membership values are captured in local Fuzzy Functions as predictors of the system behavior. Instead of using Fuzzy Rule Base (FRB) structures, Fuzzy Functions are implemented to build models of a given system. This book presents essential alternatives of the Fuzzy Functions approaches and their advancements. The aim is to build more powerful fuzzy models via autonomous approaches that can identify hidden structures via the optimization of their system parameters. Although it has been investigated for many years now, the problem of uncertainty modeling is yet to be completely solved in system modeling communities. In engineering problems, building reliable models depends on the identification of important values of variables of model equations. However, in real life cases, these important values may not be obtained due to imprecise, noisy, vague, or incomplete nature of current information. The goal of this book is to build an uncertainty modeling architecture to capture the uncertainty in parameters and structure of the Fuzzy Functions to capture the most available information. The uncertainty in systems can be captured with higher order fuzzy sets, viz. interval valued type-2 fuzzy sets, which was first introduced by Zadeh [1975a,b]. Type-2 fuzzy systems implement type-2 fuzzy sets to capture the higher order imprecision inherited in systems. In particular, this book introduces the formation of “Type-2 Fuzzy Functions” to capture uncertainties associated with system behavior. Initially, “Type-1 Fuzzy Functions”, which is presented instead of type-1 fuzzy rule bases, will be thoroughly investigated. Then type-2 extensions of the “Fuzzy Functions” will be discussed in terms of structure identification and inference strategies. Furthermore, to capture uncertainty intervals of membership
1.2 Contents of the Book
3
values and fuzzy function structures and their type-2 fuzzy functions optimally, evolutionary algorithms will be discussed. One of the aims of this book is to introduce alternative methods to current fuzzy system modeling methodologies, in effective and versatile manner. They can be used both for regression and classification domains easily (with slight changes in the modeling structure). The underlying structure of these introduced methodologies will be thoroughly examined in the following chapters. To give a general view of the system modeling structures, the following items are considered in this book: ¾
Multi-input single output systems,
¾
Regression and binary classification system representation,
¾
Structure identification of a fuzzy system model that is based on a fuzzy clustering method and “Fuzzy Functions” approach,
¾
Small to medium size datasets,
1.2 Contents of the Book This book concentrates on new fuzzy system modeling approaches, which implement type-1 and type-2 fuzzy sets and use other soft computing methods such as genetic algorithms and support vector machines for parameter optimization, membership function identification, uncertainty interval capturing, and local function approximation. Two different versions of the new fuzzy clustering method along with their validity measures are presented. All these methods are depicted in Figure 1.1. The new methodologies presented in this book are listed in more detail as follows: Structure Identification With Improved Fuzzy Functions IFC*
Inference with Type 1 Improved Fuzzy Functions
New Cluster Validity functions for IFC*
Modeling Uncertainty with Type 2 Improved Fuzzy Functions
Parameter Optimization with Evolutionary Fuzzy Functions
Capturing Uncertainty Bounds with Evolutionary Type 2 Improved Fuzzy Functions
Case Based Type Reduction Method
Extensions of each new method to classification problem domains
Fig. 1.1 Contents of the book at a glance
¾ Type-1 Fuzzy Systems Based on Improved Fuzzy Functions Fuzzy systems based on Fuzzy Rule Bases (FRB) have been successfully used to model human problem-solving activities. A classical way to represent the human knowledge is using ‘IF…THEN’ fuzzy rules. It is mostly the interpretability property that makes fuzzy systems preferable to other soft computing approaches. There
4
1 Introduction
are various FRB systems, based on the way the antecedent and consequent parameters are formed. One should note that in these FRB systems, membership values of fuzzy sets may represent different attributes such as degree of belongingness, degree of fire, degree of compatibility, weight or strength of local functions, or individual objects. For instance in Fuzzy c-Regression methods [Hathaway and Bezdek, 1993], membership values are used as weights (strengths) assigned to each local function identified by the system model. Nonetheless, to formulize the structure of fuzzy function approaches, in this book a new representation of membership values is introduced. Fuzzy Functions approaches, initially presented by Turksen in 2005 [Turksen, 2008] and analyzed in [Celikyilmaz, 2005], have emerged from the idea of representing each unique rule of a FRB system in terms of ‘fuzzy functions’. One of their prominent features is that; the degree of belongingness of each sample vector in a fuzzy set has a direct affect on how the local fuzzy functions of the particular set are defined. The standard Type-1 Fuzzy Functions approaches are multi-variable crisp valued functions, which implement type-1 fuzzy sets. In a sense, membership values are used as additional dimensions in approximating hyper-surfaces for each cluster identified by any type of fuzzy clustering algorithm. In initial forms, we used well-known fuzzy c-means clustering algorithm of Bezdek [1981a]. Then a new fuzzy clustering method, to be introduced in chapter 3, is implemented into fuzzy functions approaches to enhance membership values to predict the relationship between independent and dependent variables in local structures. This new fuzzy clustering algorithm introduces two new membership functions for regression and classification (pattern analysis) models. We hypothesize that membership values obtained from improved clustering algorithms can increase the predictive power of fuzzy models of each cluster, when used together with the original input variables to explain the behavior of the input and output variables in local models. In this sense, the resulting fuzzy functions are referred to as “Improved Fuzzy Functions” and this approach is denoted as “Type-1 Improved Fuzzy Functions”. The novel type-1 improved fuzzy functions approach introduces a new structure identification (training) and inference (reasoning) methods, different from training and inference methods of standard type-1 fuzzy functions, which use Fuzzy c-Means (FCM) clustering method. It should be pointed out that fuzzy functions and improved fuzzy functions has been introduced instead of FRB methods to eliminate fuzzy operations all together and most importantly improve prediction power.
¾ Novel Fuzzy Clustering Methods Fuzzy clustering algorithms, especially the Fuzzy C-Means (FCM) Clustering algorithm of Bezdek [1981a] has been the most commonly used and extended method in fuzzy systems research area. In FCM clustering algorithm, the local structures are represented with fuzzy sets, which are defined by membership functions. In the initial fuzzy function practices, we used FCM clustering algorithm. Later, to enhance the power of membership values obtained from membership functions, a new fuzzy clustering algorithm is introduced. The new membership functions can produce enhanced membership values.
1.2 Contents of the Book
5
Since supervised multi-input single-output (MISO) models are the main interest of this book, we designed a clustering algorithm that can approximate local models to explain the relationship between inputs and outputs. The FCM clustering algorithm is designed to find local grouping of data vectors by leaving out any linear or non-linear relationships between them. However, data objects could be grouped by considering not only the local groupings of input-output data (clusters) but also their relationships. Here a new clustering algorithm, namely Improved Fuzzy Clustering (IFC), is introduced to find local fuzzy partitions with fuzzy functions based on combined clustering structures. The IFC structure is influenced by Höppner & Klawonn [2000, 2003], Chen et al. [1998] and Menard [2001]. The structure of these earlier versions of fuzzy clustering algorithms and introduced IFC algorithm are similar in the sense that they share a similar objective function, which combines standard fuzzy clustering and fuzzy c-regression methods. However, there are many structural differences between introduced IFC of this book and others. These differences are discussed in the following chapters. It should be noted that, new IFC is implemented in a system of Fuzzy Functions, where membership values are used as additional predictors in approximating the local inputoutput relationships. Earlier similar research [Höppner and Klawonn, 2003; Chen et al.,1998] use FRB structures to build fuzzy models. The new IFC carries out two objectives: (i) to find good representation of partition matrix, which captures multiple model structures of the given system by identifying hidden patterns, (ii) to shape membership functions, so that the membership values obtained from these functions can also help to explain the local input-output relationships. An extension of IFC, namely Improved Fuzzy Clustering for classification (IFC-C) systems is presented as an alternative clustering methodology. ¾
Cluster Validity Functions for the New Fuzzy Clustering Methods
In general, fuzzy clustering methods, including introduced Improved Fuzzy Clustering (IFC) algorithm, assumes that some initialization parameters are known prior to the model run. This is usually a problematic issue since different parameters may reveal different results, which could eventually affect system performance. Literature indicates that many cluster validity functions have been introduced to validate number of clusters of especially FCM [Bezdek, 1981a] clustering models. Among well-known validity functions, [Fukuyama and Sugeno, 1989; Xie and Beni, 1991; Pal and Bezdek, 1995; Bezdek, 1976] are the most commonly used FCM clustering validation functions. In later years, many variations of these functions, e.g., [Boguessa et al., 2006; Dave, 1996; Kim et al., 2003; Kim and Ramakrishna, 2005], are presented by modifying or extending earlier validity functions. These validity functions measure characteristics of a point-wise fuzzy clustering method, i.e., FCM clustering algorithm. Most validity indices are designed to validate FCM clustering algorithm. They use characteristics of FCM to indicate the optimum number of clusters. In this sense, earlier validity indices designed for FCM clustering may not be suitable for other variations of fuzzy clustering algorithms, which are designed for different purposes, e.g., Fuzzy C-Regression (switching regression) algorithm (FCRM)
6
1 Introduction
[Hathaway and Bezdek, 1993] or novel improved fuzzy clustering (IFC) approach. For these types of clustering algorithm variations, different validity measures have been introduced. For instance, in [Kung and Lin, 2004] a new validity index is introduced to measure optimum number of clusters of FCRM applications. Their validity function is a modification of Xie-Beni [1991] ratiotype validity function. The validity functions should be designed based on the objectives and structure of the corresponding fuzzy clustering method. In this book, two new cluster validity functions are introduced to measure the optimum number of clusters of the models of the new clustering algorithms, IFC and IFC-C. These validity functions are ratio type indices, which measure the compactness to separability ratio of clusters as well as performance of local fuzzy functions. ¾
An Evolutionary Algorithm to Optimize Novel Type-1 Fuzzy Functions
An important characteristic of fuzzy function systems is that they require a few key parameters that may affect the performance of a fuzzy system’ behavior. Optimization of each of these parameters usually requires analysis of different components. For instance, the number of clusters is analyzed based on cluster validity functions, whereas the optimum degree of fuzziness parameter should be analyzed by observing the behavior of membership functions or the performance of the overall model. In a system model, these parameters may affect each other; therefore, they should be optimized interactively, rather than separately. In addition, a procedure that learns the fuzzy function system automatically from data must respect the above properties. There are several optimization methods and yet there are no universal methods. The standard gradient based optimization methods might not be effective in the context of fuzzy systems given their non-linear character (in particular the form of the membership functions) and the modularity of the systems. This forces us to explore other optimization methods of more global optimization capabilities such as, e.g., genetic algorithms. In this book, we implement evolutionary algorithms using genetic algorithms, to encode system parameters as part of a hybrid genetic code. Then fuzzy function system models are evolved to capture optimum system parameters based on a stochastic search method. Hence, a new evolutionary type-1 fuzzy structure identification and inference method are introduced by combining evolutionary algorithms with improved fuzzy functions. The stochastic optimization model within fuzzy functions approach would reduce the amount of computation steps of parameter optimization compared to type-1 improved fuzzy functions that is based on exhaustive (grid) search method. ¾
A New Type-2 Improved Fuzzy Functions Approach to Capture Parameter Uncertainties
Type-1 improved fuzzy function systems and the evolutionary type-1 fuzzy functions are introduced to improve system model performance and eliminate fuzzy operators of fuzzy inference systems. However, they have limited capacity to identify uncertainties, because type-1 fuzzy sets are used in these systems. Quite often, the knowledge that we use to construct the fuzzy functions is uncertain.
1.2 Contents of the Book
7
Type-1 systems are not able to deal with fuzzy function uncertainties directly. Such uncertainties cause the generation of fuzzy functions whose membership values are uncertain. Type-2 fuzzy sets on the other hand can be very useful to identify uncertainty interval of membership values of a fuzzy set, which encapsulates the unknown exact membership function. In this respect, type-2 fuzzy sets can be used to handle fuzzy function uncertainties, while determining membership values. In addition, in fuzzy function systems, some of the essential system parameters (such as the number of clusters, degree of fuzziness, parameter and structure of fuzzy functions of each cluster) are provided by the designer before the model construction, which potentially introduces various sorts of uncertainties. Hence to capture some of the uncertainties of system models, a new type-2 improved fuzzy function system is introduced. ¾
A New Cased-Based Inference and Cased-Based Type Reduction Method for Type-2 Fuzzy Functions
The membership values of the new improved fuzzy clustering algorithm use a hybrid distance measure, which combines the distance between each data vector to each cluster with the error measure of the interim fuzzy functions. Thus, to compute the membership values of new observations, one needs to use the training cases to obtain the information on the error of these functions. A case based inference method is presented where one uses the nearest training vectors to fuzzify the new observations. This method is an example of semi-parametric inference algorithm. When higher order fuzzy sets are used in fuzzy systems, the type of the fuzzy sets should be reduced before defuzzification method. Then, defuzzification reduces the type from one to zero to get a crisp value. In type-2 improved fuzzy functions approach, after the fuzzification step, the type of the interval type-2 membership functions are reduced down to type-1 based on the new Case-Based Type-Reduction method. This method is based on a search method, which does not require complex fuzzy operators as the other type reduction methods of standard type-2 fuzzy logic systems that are based on fuzzy rule bases [Karnik et al. 1999; Mendel 2001]. After the type of fuzzy sets is reduced down to type-1, the rest of the steps of introduced type-2 improved fuzzy functions system are similar to type-1 improved fuzzy functions system. In the paragraph above, we stated expressions such as “type-1 or type-2 membership functions”. In fact one ought to state “type-1 or type-2 membership values” generated from a clustering algorithm. It should be noted that “membership functions” are idealized representations of membership value scatter clouds. Therefore, in this writing whenever “membership functions” are stated, the intention is the idealized, curve-fitted representation of scatter clouds of membership values. In particular, in this book, the aim is to point out that in earlier type-2 fuzzy logic system approaches usually an expert defines membership functions of fuzzy sets that form the type-2 fuzzy rule bases. Whereas in the introduced “Fuzzy Functions” approach, one does not curve-fit over the scatter clouds as extracted from clustering methods and one does not define idealized membership functions
8
1 Introduction
but one uses membership values and their transformations as new variables in generating the form of the fuzzy functions. ¾
An Evolutionary Algorithm to Optimize the Type-2 Improved Fuzzy Functions
The system parameters of type-2 improved fuzzy functions, e.g., the uncertainty interval of membership values, the number of hidden structures, the list of fuzzy functions structures, the type of the function estimation and their corresponding parameters, etc., are identified based on a grid search. It is possible to find an optimal model when a wider search space is used, yet it is computationally inefficient when there are many system parameters to optimize. An evolutionaryalgorithm method using genetic algorithms is implemented into type-2 improved fuzzy functions approach to identify such system parameters automatically. With the new method, the number of iterations of the type-2 fuzzy functions optimization, which is based on a grid search, is reduced and the optimum parameters are automatically determined based on a stochastic search algorithm. With the implementation of evolutionary search algorithms in uncertainty modeling of improved fuzzy functions approaches, we are trying to locate the uncertainty interval where the optimum model would be. Evolutionary model is capable to optimize this interval based on the given data and interval valued parameters. ¾
A New Performance Measure to Be Used in Evaluation of the Prediction Problems of Stock Prices
In any trading system modeling, the main goal is to improve the profitability. A profitable prediction is a better prediction even it has less accuracy based on different criteria, e.g., accuracy in predicting the next day directions of a stock. In [Deboeck, 1992] it was shown that a neural network that correctly predicted the next-day direction 85 percent of the time, consistently lost money. Although the system correctly predicted market direction, the prediction accuracy was low. Hence, one can conclude from such instances that evaluation of trading models should not just be based on predicted directions of stock price movements. In addition, as will be shown in the results of the prediction of stock prices, the accuracies of benchmarking methods are not always significantly different from one another. Thus, it can be difficult to identify the best model based on error reduction. Since the aim of stock trading models is to return profit, the profitability should be the performance measure. For these reasons, on top of the wellknown performance measures used in this book, here we introduce a new criterion – Robust Simulated Trading Benchmark (RSTB), based on profitability of models to predict the stock prices. The RSTB combines three different properties in one performance measure including the market directions, prediction accuracy and robustness of models. RSTB is driven by a conservative trading approach.
1.3 Outline of the Book
9
In summary, the new methodologies presented in this book can be listed as follows: • • • • • • • • • • •
Two novel improved fuzzy clustering methods, for regression and classification problem domains. A new type-1 improved fuzzy functions approach instead of fuzzy rule bases. A new type-2 improved fuzzy functions approach for uncertainty modeling. Extensions of type-1 and type-2 fuzzy functions to classification problem domains. Two new hybrid type-1 and type-2 fuzzy functions approaches using evolutionary computing methods. Extensions of hybrid methods to classification problem domains. Two novel cluster validity functions to evaluate the new improved fuzzy clustering methods, for regression and classification problem domains. A new semi-parametric cased based inference method for type-1 improved fuzzy functions. A new cased based type reduction for type-2 improved fuzzy functions. Extensions of the type-1 and type-2 improved fuzzy functions, and hybrid type-1 and type-2 improved fuzzy functions approaches, using standard FCM clustering method instead of improved fuzzy clustering method. A new performance measure specifically implemented for the stock price prediction systems.
1.3 Outline of the Book In chapter 2, the basic concepts of fuzzy set and logic theory, which are required to explain the introduced algorithms, are presented. In this book, a new fuzzy functions approach is introduced; therefore, details of the introduced fuzzy functions are given. Reviews of some of the well-known fuzzy rule base structures and their extensions that are used in the chapter of experiments for benchmarking purposes (Chapter 6) are presented. In chapter 3, a detailed literature review on fuzzy cluster algorithms are given, since fuzzy clustering is the core of the structure identification method of the new fuzzy functions method of this book. Next, the introduced improved fuzzy clustering algorithm for prediction models and its extension to classification problem domains are presented. Starting from its motivation, the mathematical structure and differences from similar fuzzy clustering algorithms and justification of its performance improvement as opposed to well-known fuzzy c-means clustering is thoroughly analyzed with simulation experiments. After reviewing literature on well-known cluster validity indices and discussing their strengths and weaknesses on different fuzzy clustering methods, two new cluster validity indices are presented for the two novel fuzzy clustering algorithms. Their strengths are justified
10
1 Introduction
through simulation experiments. Finally, a summary of the contributions are given at the end of the chapter. In chapter 4, the presented fuzzy system with fuzzy functions are explained comprehensively for the domains of prediction using standard fuzzy c-means clustering algorithm and improved fuzzy functions approach with the novel improved fuzzy clustering algorithm. The presented algorithms are explained at each step of the fuzzy system modeling in detail. In addition, extensions of the new algorithms are presented for classification problem domains. Then, the structure identification and inference methods of the new fuzzy modeling with improved fuzzy functions methodology are presented based on evolutionary algorithms at the end of this chapter. Reduction of the number of the structure identification steps is discussed with the presented evolutionary fuzzy functions as compared to the earlier fuzzy functions. Similar to other chapters, a summary is provided at the end of the chapter. In chapter 5, uncertainty modeling is presented with improved fuzzy functions. Three-phase type-2 fuzzy system architecture is introduced. Then a new type reduction method is introduced using evolutionary algorithms to identify the uncertainty interval of fuzzy sets. A summary is given at the end of the chapter. In chapter 6, the new algorithms are applied on ten different datasets of different structures, i.e., regression and classification problem domains. The results are compared with the results obtained from other system modeling techniques, such as adaptive network fuzzy inference system, neural networks, dynamically evolving network fuzzy inference system, support vector machines, and other fuzzy systems models of closest structures to the presented system modeling approaches. The performance of each modeling technique is discussed in terms of performance improvement, robustness, and statistical significance. In addition, the effectiveness of each presented algorithm is discussed separately based on the results. In the final chapter, chapter 7, the conclusions and remarks are given about the presented algorithms and possible extensions.
Chapter 2
Fuzzy Sets and Systems 2 Fuzzy Sets and Systems
This chapter reviews basic principles of the fuzzy sets and fuzzy logic as well as the inference methodology of the approximate reasoning and the extension principle theories that are fundamental parts of structure identification with traditional fuzzy rule base systems. Also presented is the “Fuzzy Functions” as defined in the literature and as presented in this work. Extensions of some well-known fuzzy inference systems including structures of well known hybrid fuzzy systems are also presented at the end of this chapter.
"No assertion is ever known with certainty... but that does not stop us making assertions." Carneades, 214-129 BCE
2.1 Introduction Professor Lofti A. Zadeh [1965] has introduced the concept of “Fuzzy Sets”, which are sets with imprecise amplitudes. He stated that the “membership” in a fuzzy set is not a matter of affirmation or denial, but rather a matter of degree. Over the last forty years, his proposal has gained recognition as an important point in the evolution of modern concept of imprecision and uncertainty and his innovation represent a paradigm shift from the classical sets or the crisp sets to “Fuzzy Sets”. Fuzzy set and fuzzy logic theory are renowned theories, with which one can capture the natural phenomenon of the imprecision and uncertainty. The characteristic of fuzzy set, namely the membership function, is a function whose range is an ordered membership set within the closed unit interval. Therefore, a fuzzy set is usually characterized as a function. This chapter displays a brief introduction to fuzzy sets and fuzzy logic theories. Firstly, traditional fuzzy sets, i.e., type-1 fuzzy sets and their fundamental properties A. Celikyilmaz and I.B. Türksen: Model. Uncertain. Fuzzy Logic, STUDFUZZ 240, pp. 11–50. springerlink.com © Springer-Verlag Berlin Heidelberg 2009
12
2 Fuzzy Sets and Systems
will be introduced. Operations on fuzzy sets and the connectives used to implement these operations are explained. Definitions of some basic concepts of conventional fuzzy system modeling approaches including the extension principle, fuzzy numbers, operations on fuzzy sets and fuzzy relations are presented. Then, higher order fuzzy sets, i.e., type-2 fuzzy sets, which were introduced by Zadeh [1975a], will be reviewed. Following, the definition of fuzzy functions, as recognized in the literature and as described in this work, is a new form of fuzzy system modeling, will be explained in detail. Finally, we give a brief summary of the structures of extensions of well-known fuzzy rule bases, and hybrid fuzzy systems that are used as benchmarking methodologies in the chapter of experiments we have carried out (Chapter 6).
2.2 Type-1 Fuzzy Sets and Fuzzy Logic A (crisp) set is a collection of distinct objects. A crisp set is defined in a way to partition objects in a given universe of discourse into two groups: members (those that do belong to a certain set) and non-members (those that do not belong to a certain set). In traditional set theory, there exists a sharp distinction between members and non-members of a set. Contrary to crisp sets, a fuzzy set can be formed by assigning a membership value to each object in the interval of [0,1]. Membership values represent the degree to which an object belongs to a fuzzy set. Let X denotes the universe of discourse, where x represents an element of the universe, X and A denote a fuzzy set. A fuzzy set is hence characterized by its membership function, μA(x) as:
μA(x) : X Æ [0, 1]
(2.1)
Membership function, as described in (2.1), states that values assigned to the elements of the universal set, X, fall within a specified range. In the meantime, it indicates the membership grade of these elements in fuzzy set A. Further details of membership functions can be found in [Turksen, 1991; 1999; 2001; 2002]. A fuzzy set A, on universe of discourse of X can also be defined as a set of ordered pairs as: A={(x, μA(x))| x ∈X }
(2.2)
Membership functions have been commonly formulized with straight lines, which are the idealized representations. In real life applications, membership values form scatter-clouds, which are then normalized into idealized shapes such as triangular, trapezoidal, or Gaussian membership functions. Figure 2.1 shows membership functions of fuzzy sets “young”, “middle age” and “old’. Horizontal axis of graph in Figure 2.1 represents “age” in years (the universal set X) and vertical axis represents the degree to which a person can be labeled “young”, “middle age” or “old”. Hence, membership graph in Figure 2.1 represents membership function of a fuzzy set, which represents “the group of people that can be considered to the young”, “the group of people that can be considered to the middle aged” and the “group of people that can be considered to the old aged”.
2.2 Type-1 Fuzzy Sets and Fuzzy Logic
13
μΑ
1
Membership function of Young
25
Membership function of Middle Age
50
Membership function of Old
75
100
AGE
Fig. 2.1 Membership functions of fuzzy sets “young” (green), “middle age” (red) and “old” (blue)
2.2.1 Characteristics of Fuzzy Sets Conventionally, there are two alternate notations used to represent fuzzy sets: discrete or continuous. A fuzzy set A on a discrete universe of discourse, X, is represented by; A = ∑x∈X µA(x)/x
(2.3)
In (2.3), ‘∑’ does not represent mathematical addition operation. It is an aggregation operator that represents collection of membership values that belong to a fuzzy set A on continuous universe of discourse, X, which is denoted by following equation, A = ∫x∈X µA(x)/x
(2.4)
Again ‘Û’ is not an integration operation but an aggregation operation. One of the most important concepts of fuzzy sets is the concept of an “α-level set (α-cut)” and “strong α-cut”. Given a fuzzy set A defined on X and any number α∈[0,1], an α-cut, αA, and the strong α-cut, α+A are defined as: αA={x | µ (x)≥α } A
(2.5)
α+A={x | µ (x) > α } A
(2.6)
“α-cut” and “strong α-cut” of a fuzzy set A is a crisp set that contains all the elements of a universal set X, whose membership grades in A are ‘greater than or equal to’ or ‘greater than’ the specified value of α, respectively. The set of all levels of α∈[0,1] that represent distinct α-cuts of a given fuzzy set A are called the level set of A. Any fuzzy sets can be represented using its level sets. This representation is called
14
2 Fuzzy Sets and Systems
the α-cut decomposition (representation) of the fuzzy set and is formulated as a theorem. The theorem states that for any fuzzy set A on U,
{
}
μ A ( u ) = sup min α , μ α A ( u ) , ∀ u ∈ U α ∈[0 ,1 ]
(2.7)
Notice that μ α A ( u ) is a membership function of a crisp set and assumes either 0 or 1. The “support” of a fuzzy set A within a universal set X is a crisp set that contains all the elements of X that have non-zero membership grades in A. It should be noted that, the support of A is as same as strong α-cut of A for α=0 and is defined as Supp(A)={x∈X | μA(x) > 0 }
(2.8)
‘1-cut’ of A is called the “core” of A and is denoted by Core(A)={x∈X | μA(x) = 1 }
(2.9)
The “height of” fuzzy set A, h(A), is the largest membership grade obtained by any element in the set and it is defined as follows: (2.10)
h(A) = supx∈X (μA(x))
A fuzzy set A is called “normal” when h(A)=1 and it is called “subnormal” when h(A)<1. The fuzzy set characteristics defined above is displayed with the aid of an example of the trapezoidal membership function as shown in Figure 2.2 below.
Core of fuzzy set A 1
Height of fuzzy set A
-cut
a1
a2
a3
a4
x
Support of fuzzy set A
Fig. 2.2 Fuzzy Set Properties on a “Trapezoidal” membership function example
2.2.2 Operations on Fuzzy Sets In classical set theory, four fundamental operations are defined on sets; complement, containment, intersection and union operations. These four operations
2.2 Type-1 Fuzzy Sets and Fuzzy Logic
15
are also defined for fuzzy sets as standard fuzzy set operations along with many other fuzzy set operators. Two of the most widely used operators are triangular norm (t-norm) and triangular conorm (t-conorm), which are widely accepted and has been widely used in fuzzy sets literature. Hence, in this section, a brief definition of t-norms and t-conorms is also presented. In addition, the following definitions of standard fuzzy sets are based on De Morgan Triplets with (Min, Max, Complement). Let entire set of finite elements denoted as S={x1,…,xn}. A fuzzy set A can be defined as A⊆S. Based on the membership function definition in (2.3), membership values of fuzzy set A can be expressed as:
A= μA(x1)/x1 + μA(x1)/x1 + …+ μA(xn)/xn = ∑k μA(xk)/xk
(2.11)
In (2.11), note that the ‘+’ symbol does not refer to the ordinary addition, thus it is a special representation of discrete fuzzy sets, symbolically. Definition 2.1. Equality of fuzzy sets: It can be defined by the equality of membership functions. Let A,B ⊆ S represent two fuzzy sets and equality of two fuzzy sets is defined as follows:
A=B ⇔ μA(x) = μB(x), ∀x∈S
(2.12)
Definition 2.2. Inclusion of fuzzy sets: Defined by the inequality of membership functions as follows:
A⊆B ⇔ μA(x) ≤ μB(x), ∀x∈S
(2.13)
It should be remarked that that ’A⊂ B’ is defined in fuzzy sets as follows:
A⊂B ⇔ μA(x) ≤ μB(x), ∀x∈S and ∃y∈S
μA(y) ≤ μB(y)
(2.14)
Definition 2.3. Union of fuzzy sets: Defined by the maximum of membership functions as:
A∪B : μA∪B(x) = max [μA(x), μB(x)]
(2.15)
Definition 2.4. Intersection of fuzzy sets: Defined by the minimum of membership functions as:
A∩B : μA∩B(x) = min [μA(x), μB(x)]
(2.16) c
Definition 2.5. Complement of fuzzy set: The complement of fuzzy set A, A ,is defined as follows: μAc(x)=1- μA(x)
(2.17)
16
2 Fuzzy Sets and Systems
Figure 2.3 displays graphical representation of union and intersection of fuzzy sets and Figure 2.4 displays the graphical representation of complement of fuzzy sets. Logical operations have been the focus of considerable discussions over the evolutionary history of fuzzy logic. Originally, “min” and “max” functions were presented to model logical conjunction and disjunction. These functions obviously generalize traditional Boolean operators, but it was immediately recognized that they were not the only possible functions. In the last three decades, many different operators including triangular norms (t-norms for short) and triangular conorms (tconorms for short) have been presented. The following definitions of t-norms and t-conorms can be found in Klir and Yuan [1995].
Fig. 2.3 Union and Intersection Fig. 2.4 Complement of Fuzzy Set A, Ac. (--- line) A
AC
Let a, b, c be membership values in the unit interval [0, 1]. Definition 2.6. Intersection / triangular norm (t-Norm) of Fuzzy Sets: A triangular norm (t-norm) is a function T on the unit interval [0,1], (i.e., z=T(a, b), 0
Axiom T1. T(a,1)=a (Boundary Condition) Axiom T2. T(a, b) = T(b, a) (Commutativity) Axiom T3. If b1≤b2 then T(a,b1)≤T(a,b2) (Monotonicity) Axiom T4. T(a, T(b, c)) = T(T(a, b), c) (Associativity) T-norms are generally accepted as equivalent to the class of fuzzy intersections. Therefore, these axioms are called the axiomatic skeleton for fuzzy intersections/t-norms.
2.2 Type-1 Fuzzy Sets and Fuzzy Logic
17
The boundary condition in axiom T1 ensures that when one argument of a t-norm is 1 (in other words membership value is 1), membership grade of the intersection is equal to the other argument. Commutativity ensures that the fuzzy intersection is symmetric. Monotonic property of t-norm ensures that a decrease in the value of membership in set A or B cannot increase the degree of membership in intersection. The axiom T4 states that intersection of any number of sets can be taken in any order. In order for the above four axioms of t-norm to hold true, the following three axioms must also hold true.
Axiom T5. T must be a continuous function (Continuity) Axiom T6. T(a, a) < a (Sub-idempotency) Axiom T7. a1≤ a2 and b1≤ b2 implies T(a1,b1)≤T(a2,b2) (Strict Monotonicity) The axiom of continuity prevents large changes in the membership value of A ∩ B when there is a very small change in the membership value of either set A or B. The axiom of sub-idempotency forces membership grade in A∩B not to exceed the value of a, in the case when both membership grades in A and B have the same value of a. Definition 2.7. Union / triangular conorm (t-Conorm) of Fuzzy Sets: A triangular conorm (t-conorm) is a function C on the unit interval [0, 1], (i.e., z=C(a, b), 0
Axiom C1. C(a,b) =C(b,a) (Commutativity) Axiom C2. C(a,C(b,c)) = C(C(a,b),c) (Associativity) Axiom C3. If b1≤ b2 then C(b1,c) ≤ C(b2,c) (Monotonicity) Axiom C4. S(a,0)=a (Boundary condition) Fuzzy union function returns membership grade of A∪B. Axioms T1-T4 differ from axioms C1-C4 in boundary condition. In order for axioms C1-C4 to hold true, below requirements must also be satisfied.
Axiom C5.C must be a continuous function (Continuity) Axiom C6.C(a, a) < a (Superidempotency) Axiom C7. a1≤ a2 and b1≤ b2 implies C(a1,b1)≤ C(a2,b2) (Strict Monotonicity) Definition 2.8. Combination of intersection and union (t-norms and t-conorms): According to classical set theory rules, intersection and union operation on sets are dual with respect to each other if they satisfy De Morgan laws such that:
A ∩ B = A ∪ B and A ∪ B = A ∩ B
(2.18)
18
2 Fuzzy Sets and Systems
However, this duality is satisfied for only some combinations of t-norms and tconorms and fuzzy complements. A t-norm, T, and a t-conorm, C, are dual with respect to fuzzy complement if and only if:
T(a,b) = 1-C(1-a,1-b), or
(2.19)
C(a,b) = 1-T(1-a,1-b)
(2.20)
is true. The triplet formed by a t-norm, its dual t-conorm, and standard complement is called a De Morgan triplet. De Morgan triplets were first introduced by Zadeh [1965] and they are one of the most widely used triplets. Fuzzy De Morgan Triplet can be explained as follows: Let A and B be two fuzzy sets on a universe of discourse X. Membership functions of fuzzy set A and fuzzy set B is defined as a=μA(x) and b=μB(x). According to Zadeh’s De Morgan Triplet formula the intersection, union and complement operations on these sets can be defined as follows: Intersection: A∩B: μA∩B(x)= a∧b =min(a,b), ∀x∈X Union: A∪B: μA∪B(x)= a∨b =max(a,b), ∀x∈X Complement: Ā: μĀ(x)= c(a) =1-a, ∀x∈X There are also other De Morgan Triplets which use different t-norm and t-conorm operators, some of which are listed in Table 2.1. Table 2.1 Some well known t-norms and t-conorms Names Zadeh Probabilistic Lukasiewicz Schweizer and Sklar Weber (λ > -1)
t-norms min(x, y) xy max(x+y -1, 0) max(0, (xp + yp - 1)1/p), p≠ 0 max((x + y – 1 + λxy) / (1 + λ), 0))
t-conorms max(x, y) x + y - xy min(x + y, 1) 1 - max((1 - x)p + (1 – y) p -1), 0) 1/p min(x + y + (λ/1-λ)xy, 1)
2.3 Fuzzy Logic 2.3.1 Structure of Classical Logic Theory Classical logic is the study of methods and principles of reasoning. It deals with the propositions that are required to be either true or false. One area of logic is propositional logic that deals with combinations of variables that stand for arbitrary propositions. The main concept in propositional logic is the study of rules by which new logic variables can be produced as functions of some given logic variables. Propositional logic is the expression of all the logic functions of n variables based on simple logic operations. Basic set operations are given in Appendix A.1. There are five main logic operations, which are defined by negation (!), conjunction (∧), disjunct ion (∨), implication (⇒) and equivalence (=). Using these
2.3 Fuzzy Logic
19
operations, one can define other operations. These five operations are the most commonly employed operations in formal logic. By combining different operators, for example, negations, conjunctions, and disjunctions in logic formulas, we can form another logic function. When a variable represented by a logic formula is always true, it is called tautology. Various forms of tautologies, which are called inference rules, can be constructed to deduce inferences. Let p and q be two propositions. The most frequently used basic inference form in classical logic is “Modus Ponens” inference form which is denoted as in the following,
Modus Ponens 1.p⇒q 2.p ∴q
(2.21)
The two propositions above the line in (2.21) are called “premises” and the proposition below the line is “conclusion”. If two premises are true than conclusion is also true. Modus Ponens can also be represented by [(pÆq)∧p]Æq ,
(2.22)
where p and q are two propositions, “Æ” implies implication operation and “∧” implies conjunction operation in logic.
2.3.2 Relation of Set and Logic Theory Study of logic goes back to more than two thousand years and up until now many symbols and diagrams have been devised. Around 300 BC Aristotle introduced letters as term-variables. Modern era of mathematical notation in logic began with George Boole (1815-1864), although none of his notation survived. Set theory came th th into being in the late 19 and early 20 centuries, largely a creation of Georg Cantor (1845-1918). It should be emphasized about set and logic theory that, although developed independent from each other, there is a relationship between them. Anything that can be expressed with set notations can also be expressed with linguistic terms in logic theory. In set theory, an element belongs to a particular set, or does not belong to a particular set and it is expressed linguistically as “x is element of A”; whereas in logic theory, a linguistic expression is assigned to be “true” or “false” and linguistically can be expressed as “x is element of A” is true or false. In order to express complex reasoning structures, simple atomic propositions (one or two premises at most) can be composed. The main idea that the conclusions are made from set of premises is to discover different ways in which the conclusion can be either found or formed by the given premises. (The most frequently used basic inference form is “Modus Ponens” inference form.)
2.3.3 Structure of Fuzzy Logic The fundamental difference between classical propositions and fuzzy propositions is in the range of their truth-values. Each classical proposition is required to be
20
2 Fuzzy Sets and Systems
true or false. However, the truth or falsity of fuzzy propositions is represented with a degree of truth, which is expressed by a number unit interval [0, 1]. In this sense, the term, “Fuzzy Logic” has emerged with the development of fuzzy set theory by Zadeh [1965] utilizing basic logic theory. In fuzzy sets, defined earlier, a fuzzy subset A of a crisp set X is characterized by assigning to each element x of X a degree of membership of x in A. If X is a set of propositions representing fuzzy logic propositions, then we can assign a degree of truth such that: “absolute true”, “absolute false”, or an “intermediate truth” like a proposition is truer than another proposition. The following definitions are adopted from [Klir and Yuan, 1995; Turksen, 2006]. Fuzzy logic is expressed as a system of concepts, principles and methods for dealing with approximate modes of reasoning rather than exact modes of reasoning. Hence, as in the analogy of fuzzy sets, there is the approximate truth concept in fuzzy logic. Therefore; fuzzy logic and fuzzy sets has a relation in the sense that, fuzzy logic is an application of fuzzy set theory. Fuzzy Logic utilizes the concepts, principles and methods of fuzzy set theory by formulating various forms of approximate reasoning. Hence, the concept of degree of membership in fuzzy sets must be utilized as a “degree of truth” in fuzzy logic. For a given fuzzy set A, the truth of a fuzzy proposition “x is a member of A” can be expressed depending on circumstances in two ways [Turksen, 2006]: (i) (ii)
In classical logic, it is the case that ‘x is A’ has an absolute degree of “truth”, i.e., t(A(x))=1, or alternatively, in fuzzy logic, it is the case that ‘x is A’ has a degree of truth, i.e., t(A(x))=tœ[0,1].
In the second case, we assign a fuzzy truth-hood to the expression “x is A”. In classical logic theory (namely the two-valued logic), all the propositions are either true or false [Klir and Yuan, 1995]. Fuzzy logic theory, on the other hand, allows inferences to include propositions of which truth-values are all real numbers in the unit interval [0.1]. This explains the uncertainty of truth-values. These values are interpreted as “degrees of truth”. The focus of fuzzy logic is on reasoning with fuzzy propositions involving imprecise concepts, which are expressed in natural language. This reasoning is called the approximate reasoning, which is to be explained in the next part. Fuzzy logic propositions, aside from classical logic propositions, utilize the assessment of truth-value of fuzzy propositions, in which the fuzziness of the propositions may vary from a combination of different linguistic components. Most commonly used propositions are the fuzzy implications, which contain simple fuzzy propositions as antecedent and consequent and they are expressed as: P: IF X is A, then Y is B,
(2.23)
where X and Y are variables that take values x and y from sets X and Y, respectively, A and B are relevant linguistic values, i.e., hot, warm, cold, etc, characterized by fuzzy sets. This conditional fuzzy proposition declares the truth-hood of a proposition p in two different ways.
2.3 Fuzzy Logic
(i)
21
The two truth-hood is declared as an absolute truth as “true”:
px ,y :“IF “X is A is true”, THEN ”Y is B is true,”
(2.24)
in which specific instances of “X is A is true” and “Y is B is true” are represented as A(x) and B(y) and the implication between them is symbolized as:
A(x)⇒ B(y) is true
(2.25)
A particular function must be chosen to find the degree to which a fuzzy proposition holds true. The most common implication function is the Lukasiewicz implication, denoted by I, and the degree to which propositions Px,y holds true can be determined by using the Lukasiewicz implication is as follows:
I(Px,y)=I[A(x),B(y)]=min[1,1-A(x)+ B(x)]
(2.26)
In words, (2.26) implies that; the degree of truth value of the conditional proposition Px,y is calculated by first finding the individual degree of truth values, A(x) and B(x), and then by applying the chosen fuzzy implication method (in this case Lukasiewicz implication) to the determined values of A(x) and B(y). The role of I is to provide with a bridge between fuzzy sets and fuzzy propositions. (ii) Alternatively we have the truthhood declared as fuzzy truthhood as a value tœ[0,1] as follows:
Px,y : IF “‘X is A’ is true to the degree t1”, THEN “‘Y is B’ is true to the (2.27) degree t2” Specific instances of ‘X is A’ is t1 and ‘Y is B’ is t2 are represented as (A(x), t1) and (B(y), t2) and the implication between them is symbolized as: (A(x),t1) ⇒ (B(y),t2) is true
(2.28)
With Lukasiewicz we get:
I(px,y)=I[(A(x) ,t1),(B(y),t2)]=[min[1,1-A(x)+ B(x)], min[1,1-t1+t2]]
(2.29)
Note that case (i) can also be represented in this form as follows:
I(px,y)=I[(A(x) ,1),(B(y),1)]=min[1,1-A(x)+ B(x)]
(2.30)
In conclusion, fuzzy set and logic theory are connected as in classical set and logic theory. In fuzzy theory, with the introduction of degrees of truth-values, the boundaries of classical sets and logic theory are more relaxed and the overlapping of fuzzy sets is allowed.
2.3.4 Approximate Reasoning Approximate reasoning is an important concept in fuzzy logic in the sense that, it represents the application area of fuzzy set theory. In classical logic theory
22
2 Fuzzy Sets and Systems
(two-valued logic), one of the most important tools of reasoning are the inference rules, which helps to derive new information from the given propositions. As mentioned earlier that Modus Ponens is the most commonly used inference method. Hence, Modus Ponens, as all other concepts of classical set and logic theory, is generalized to fuzzy set and logic theory. With the introduction of a calculus of fuzzy restrictions, Zadeh [1975b] paved the way towards a reasoning scheme called Generalized Modus Ponens (GMP). He defined a methodology known as Compositional Rule of Inference (CRI), which is used to infer fuzzy consequents utilizing GMP. Generally, GMP is defined as follows:
Premise 1: A→B Premise 2: A′ Deduction: B*
(2.31)
A and A′ are linguistic values defined on the universe of discourse of antecedent variable x with membership functions μA(x): x∈X Æ[0,1], and B and B* are linguistic values on the universe of discourse of consequent variable y with membership functions, μB(y):y∈YÆ[0,1]. “Æ” denotes implication relation operator, i.e., fuzzy implication and each premise is a relation and denoted as Ri:AÆB, i=1,..,number of relations. Note that, in this short hand version of GMP, each premise is assumed to be true and sets are defined as fuzzy sets. The short form of (2.31) is as follows: B =A' D(AÆB) *
(2.32)
In (2.32) “o” is the composition operator. Equation (2.32) can further be defined using the membership values, as presented by Zadeh [1965], as follows: µB*(y): supx∈X AND (μA'(x), μAÆB(x,y))
(2.33)
The proposition in (2.32) ○ is generally taken as t-norm connective. Therefore in (2.33) ‘AND’ operator is used to represent “MIN” operation in Zadeh rule base and sup is ‘MAX’ operator. Hence, this inference is also called max-min inference. Alternative representations of reasoning of (2.31) to (2.33) are more complicated that the one shown here. Since it is not the scope of this work, we will not discuss them any further. Readers are advised to read [Zadeh, 1965; 1975a; Klir and Yuan, 1995; Turksen, 2006] for further discussions on this topic.
2.4 Fuzzy Relations Fuzzy relations are first introduced by Zadeh [1965] in his seminal paper on fuzzy sets. They are defined as extensions of ordinary relations and they are applied to wide range of applications including clustering, pattern recognition, inference, systems and control. In order to explain the concept of the fuzzy relations, the crisp relations should be introduced first. A crisp relation defines the similarity between two or more crisp sets. The definitions of crisp relations are hence confined only to crisp binary
2.4 Fuzzy Relations
23
relations, which involve just two sets without loosing generality of elements. Crisp binary relations represent the presence or the absence of association, interaction, or any connection between elements of two sets. These two sets are considered as a set of pairs, the first one of which is called “domain” and second one is called the “range” set. The relation between these two pairs is explained as Cartesian product1, A×B. Let sets A and B be crisp sets on universe of discourse of X and Y, respectively. A crisp relation R can be expressed via a membership function, μR(x,y), where x∈X and y∈Y. Membership function of crisp relation R can be defined as follows: μR(x,y): X×YÆ{0,1}, ∀x∈X, ∀y∈Y
(2.34)
Namely, an element (x,y) in Cartesian product space X×Y, is assigned or is not assigned to the relation R. In other words, there are precisely two choices available, either 0 or 1. For instance, a crisp binary relation, an equality relation, E, can be defined between x and y, such that; x=y, and x∈[x1,x2] and y∈[y1, y2]. Equality relation can be denoted as E⊂[x1,x2]×[y1, y2]. Figure 2.5 shows the crisp relation between X and Y the product of two Cartesian. On the other hand, the fuzzy relations are crisp relations with additional capability of expressing the strength of this relation as well. Consequently, the fuzzy relations are fuzzy sets defined on the universal sets. Fig. 2.5 Crisp Binary Relation (Equality relation, E, between x and y)
[x1 , x2] X [y1 , y2] Y y1 R(x,y) y2
X1
x2
X
Similar to crisp binary relation, the fuzzy binary relation explains the relationship between two fuzzy sets such that each element of the domain of a fuzzy set is associated with one or more elements of the fuzzy set. It should be noted that, a fuzzy relation also defines a degree of association between elements of domain and range sets. Hence, let A and B be redefined as two fuzzy sets on the same universe of discourses, X and Y, respectively. The membership function of a fuzzy relation R, μR(x,y), is a mapping that can be defined as:
1
Cartesian Product of two sets A and B is defined to be the set of all points (a, b) where a ∈ A and b ∈ B It is denoted A×B, and is called the Cartesian product since it originated in Descartes formulation of analytic geometry. In the Cartesian view, points in the plane are specified by their vertical and horizontal coordinates, with points on a line being specified by just one coordinate. (definition obtained from www.mathworld.com).
24
2 Fuzzy Sets and Systems
μR(x,y): X×Y→[0,1], ∀x∈X, ∀y∈Y
(2.35)
and the binary fuzzy relation, R, can be defined as:
R = ∫X×Y µR(x,y)/(x,y)
(2.36)
Similarly, an example of a fuzzy binary relation can be defined as the following: Let E be the equality relation between variables x and y. A fuzzy relation the captures concepts such as “x is equal to y” or “x is close to y”. This concept can be captured by any membership function (in our case a continuous membership function), μR(x,y), defined over x and y. A fuzzy relation, R is defined as the Cartesian product R⊂[x1,x2]×[y1,y2] and the fuzzy relation can be shown graphically in Figure 2.6, by means of a “gray” level where A(x)œ[0,1], B(y)œ[0,1] and C(x,y)œ[0,1] and C(x,y)=Min(A(x), B(y)). Another way of expressing a fuzzy relation can be accomplished as shown in Appendix figure A.1 of the Appendix A.2, Fuzzy Relations. Binary fuzzy relation definition can now be extended to generalize n-dimensional fuzzy relation as follows: Let the Cartesian product of an universe of discourse in n-dimensional space be given as X1×X2×…×Xn. Then, the membership function of an n-dimensional fuzzy relation, R, defined over this space is shown as follows: Fig. 2.6 Cartesian product of two fuzzy sets (A and B)
C(x,y)
a' a '' a ''' a''''
µR(x1,x2,…,xn): X1×X2×…×XnÆ[0,1], ∀xi∈Xi, i=1,…,n
(2.37)
and similarly, an n-dimensional relation, R, can alternatively be designated symbolically as follows:
R = ∫ X1×X2×…×Xn µR(x1,x2,…,xn)/( x1,x2,…,xn)
(2.38)
2.4 Fuzzy Relations
25
2.4.1 Operations on Fuzzy Relations Since fuzzy relations are being built upon fuzzy sets, all operations (i.e., complements, intersections, or unions) on fuzzy sets are naturally applicable to fuzzy relations as well. It is important to note that some exceptional operations on binary fuzzy sets are not applicable to ordinary fuzzy sets. In this section, only those operations those are not applicable to ordinary fuzzy sets, especially the inverse operation and composition operation warrant special attention. Inverse operation of a fuzzy binary relation R on X×Y, which is denoted sym-1 bolically as R , is the relation between Y×X as shown in the following equation; -1
R (y,x)= R(x,y)
(2.39)
For all pairs of 〈y,x〉∈Y×X. The composition operation of two binary relations R and S, defined over X×Y∈ℜ and Y×Z∈S, respectively, is denoted as RDS. Since R and S are the binary fuzzy relations, there is an indirect link between each x and z and a matter of degree can then be defined accordingly. Therefore, a membership function that characterizes the composition operation, RDS, is denoted as μRDS(x,z) and in fact, it can be evaluated as in the following equation: μRDS(x,z) =supy∈Y (T(µR(x,y), µS(y,z))), ∀x∈X, ∀z∈Z
(2.40)
Similarly, the RDS may also symbolically be defined as follows:
RDS=∫X×Z [supy∈Y [T (μR(x,y), μS(y,z))] ]/(x,z)
(2.41)
In (2.41) the T represents a particular t-norm. For instance, if “MIN” t-norm is selected for T, then in (2.41), the membership degree of the chain 〈x,y,z〉 is determined by the degree of the weaker of the two links.
2.4.2 Extension Principle Extension principle, as introduced by Zadeh in 1975[1975a], is one of the most basic ideas of fuzzy set theory. Extension Principle helps to extend non-fuzzy mathematical concepts to deal with fuzzy quantities or variables. In order to develop the computation with fuzzy sets, we need to find a way to convert the crisp functions into fuzzy functions (i.e., fuzzification). Therefore, the concept of “fuzzifying” crisp functions is done with the aid of Extension Principle. There are other ways to fuzzify some crisp functions. However, extension principle is a wellknown and widely used principle among others. A crisp function f:X→Y is fuzzified when it is extended to act on fuzzy sets defined in X and Y. In other words, the fuzzified function, for which the same symbol f is usually used, has the form f: Ƒ(X)→Ƒ(Y). Thus, the crisp functions are fuzzified with the Extension Principle. Before reviewing the extension principle, a special case in which the extended functions are restricted to crisp power sets Ƥ(X) and Ƥ(Y) is reviewed first.
26
2 Fuzzy Sets and Systems
Given a crisp function from X to Y, its extended version is a function from P(X) to P(Y) that, for any A∈P(X), is defined by
f(A)={y | y=f(x), x ∈ A}
(2.42) -1
Additionally, the extended version of inverse of f, defined by f , is a function from P(Y) to P(X) that, for any B∈P(X), is defined by
f (B)={x | f(x) ∈ B} -1
(2.43)
-1
When the sets f(A) and f (B) can be expressed with their respective membership functions, we obtain [ f(A)](y)=supx|y=f(x) A(x)
(2.44)
-1
[ f (B)] (x)= B(f(x))
Allowing sets A and B in (2.44) to be fuzzy functions, we arrive at the following extension principle using membership functions as follows: Any given function f : X→Y induces two functions,
f: Ƒ(X)→Ƒ(Y).
(2.45)
f : Ƒ(Y)→Ƒ(X). -1
which are defined by ∀A∈ Ƒ(X), [ f(A)](y)=supx|y=f(x) A(x) or,
(2.46)
∀A∈ Ƒ(X), [ f(A)](y)=sup(x1,…,xnv)|z=f(x1,…,xnv) min (A1(x1),… An(xnv))
If we consider a Cartesian product of the universe X=X1×…× Xn , then the fuzzy mapping is defined between multiple inputs, and their fuzzy subsets are defined as Aj(xj), j=1,…,nv, then the t-norm of their membership functions are obtained first before the mapping is defined. Let X be the set of ages of the employees in a company and Y be the salaries of these employees. We would like to find out the salary of a “young employee”. It should be noted that there should be a function between the X and Y in order to give an answer to the previous question. In Table 2.2 suppose the Ages and the corresponding salaries are given. Table 2.2 The AGE and SALARY attributes of Employees
Age (in years)
20
25
30
35
40
45
50
55
60
65
Salary (in $K)
2.5
2.5
3.0
3.5
3.5
4.0
4.0
4.5
4.5
5.0
Let a fuzzy set A be the set of young people. Then the fuzzy set A with its membership values assigned can be defined as; A(“young employee”) =
A( x ) X
= 201 + 251 + 0.8 + 0.6 + 0.4 + 0.2 + 500 + 550 + 600 + 650 30 35 40 45
(2.47)
2.4 Fuzzy Relations
27
Using the young employees’ fuzzy set A, a fuzzy set B has to be defined to explain the young people’s salaries. Let function f define the relationship between the fuzzy sets A and B, which for each x in X assigns a particular y=f(x) in Y as shown in Table 2.2. It should be noted that there is a dependency between fuzzy sets A and B and it can be shown below as:
B= A(x) / f(x)
(2.48)
Therefore, from (2.48), the fuzzy set B can be explained as: B( x )
B= X
= 2.51 + 2.51 + 0.83 + 0.6 + 0.4 + 0.24 + 04 + 4.50 + 4.50 + 05 3.5 3.5
(2.49)
It should be noted from (2.49) that; some salaries are associated with more than one age (i.e., 3.5 K is assigned to ages 35 and 40, which have membership grades of 0.6 and 0.4 respectively). According to the disjunction nature of the association of salaries with ages, we take the maximum of the two membership grades as shown in (2.46). Thus, the final value of fuzzy set B, which defines the salaries of young employees, can be defined as: (2.50)
B = (1/2.5) + (0.8/3) + (0.6/3.5) + (0.2/4) + (0/4.5) + (0/5)
The graphical representation of fuzzy set A, which represents the young employees and fuzzy set B, which represents the salaries of young employees are given in Figure 2.7 and Figure 2.8. Fig. 2.7 Fuzzy set of young employees (A(x))
Fig. 2.8 Fuzzy set of salary of young employees (B(y))
1.2
Membership Value
1 0.8 0.6 0.4 0.2 0 2.5
3
3.5
4 4.5 Salary
5
5.5
28
2 Fuzzy Sets and Systems
2.5 Type-2 Fuzzy Sets The concept of type-2 fuzzy sets was introduced by Zadeh [1975a] as an extension of ordinary type-1 fuzzy sets, almost 10 years after he introduced type-1 fuzzy sets. Zadeh refers to fuzzy sets of type-2 as the fuzzy sets, whose membership functions are mappings from U to [0,1] classified as type-1. He proposes to use extension principle in order to do fuzzy operations, e.g., union, intersection, complementation, etc, on fuzzy sets of type-2. This can be done in two stages: first by defining type-1 fuzzy sets using interval valued membership functions and then to generalize from interval to fuzzy sets using the level set form of the extension principle. Type-2 fuzzy set properties described by Zadeh [1975a] will be summarized in the following parts. Type-1 fuzzy sets, whose membership functions are crisp sets (an example is shown on the left of Figure 2.9), do not provide sufficient support for many kinds of uncertainty necessary that appears in linguistic descriptions of numerical quantities or in subjectively expressed amounts, dimensions. In 1998, Mendel and Karnik [1998] have proposed to increase the number of degrees of freedom for fuzzy logic systems. Their argument was that by adding at least one higher degree to type-1 fuzzy sets may provide a measure of dispersion for totally certain membership functions of type-1. Hence, type-2 fuzzy sets capture the extensions of the type-1 fuzzy sets to a higher degree. Type-2 fuzzy sets have grades of membership that are themselves defined by type-1 membership functions, which are called secondary membership functions as being shown in the right graph of the Figure 2.9. At each value of a given variable, the membership is a function, whose domain, that is to say, the primary membership function, Jx is in interval [0,1], and whose range, secondary grades, also fall within [0,1] [Mendel, 2003] (right graph on Figure 2.9 and Figure 2.10). Interval valued fuzzy sets are just another form of type-2 fuzzy sets, whose secondary membership functions are uniform functions that only take on values of 1. Turksen’s [1986; 1995; 1989; 1999; 2006] approach is a pioneering representation of interval valued type-2 fuzzy sets. Turksen has shown that, normal forms, e.g., Disjunctive Normal Forms (DNF) and Conjunctive Normal Forms (CNF), which are equal in classical set theory, i.e., DNF≡CNF, are not equal in fuzzy set theory. In turn, he has shown that, Fuzzy Disjunctive Normal Forms (FDNF) is not equal to Fuzzy Conjunctive Normal Forms (FCNF) in fuzzy set theory, i.e., FDNF≠FCNF, which define the interval valued type-2 fuzzy sets. Mizumoto and Tanaka [1976], worked on the set theoretic operations on type-2 sets, and properties of membership grades of those sets accordingly. They also examined algebraic properties of fuzzy sets of type-2 under the operations of algebraic product and algebraic sum using the extension principle. Dubois and Parade [1979] has also worked on type-2 fuzzy sets and further proposed a formula for the composition of the fuzzy sets as an extension of type-1 sub-star composition, which only satisfies for the minimum t-norm. Mendel and Karnik [1998], Karnik et al. [1999], and Mendel and Liang [2000] have further
2.5 Type-2 Fuzzy Sets
29
extended this formula and named it as extended sub-star composition of type-2 fuzzy sets, to be briefly presented in Chapter 5. Karnik et al. [1999] have created a complete type-2 Fuzzy logic theory, which handles uncertainties in fuzzy logic system parameters. In this respect, in this work, we analyze the research of Karnik et al. [1999] on type-2 fuzziness as a separate category of the variations of type-2 fuzzy logic systems in Chapter 5.
Age_Person
Fig. 2.9 Comparison of Type-1 and Type-2 Fuzzy sets. The shaded region on the right graph is the secondary membership function of x.
In type-1 fuzzy logic approach, membership values are crisp values. The distinction between type-1 and type-2 fuzzy sets is shown in Figure 2.9. Type-2 fuzzy sets define an interval of primary membership values for each object, x, and they too have membership functions of type-1. In the following basic definitions of type-2 fuzzy sets will be introduced based on the definitions of Mizumoto and Tanaka [1976] and Mendel [2001]. Depending on the types of the membership functions of type-2 fuzzy sets, two different types of type-2 fuzzy sets have been defined: (i)
Type-2 fuzzy sets or full type-2 fuzzy sets [Karnik et al., 1999];
(ii)
Interval valued type-2 fuzzy sets [Liang and Mendel, 2000; Turksen, 1986].
2.5.1 Type-2 Fuzzy Sets A type-2 fuzzy set, denoted by Ã, is a fuzzy set, which is characterized by a type-2 membership function µÃ by (2.51) µÃ : X →[0,1] Mendel [2001] defines a type-2 fuzzy set denoted by Ã, which is characterized by a type-2 membership function, µÃ(x,u), as follows:
Ã={((x,u), µÃ(x,u)) | ∀x∈X, ∀u∈Jx ⊆[0,1]}
(2.52)
0≤ µÃ(x,u)≤1, Jx is the primary membership function in [0,1] interval and u are the primary membership values, as shown in Figure 2.10. à can also be expressed as
30
2 Fuzzy Sets and Systems
Ã=∫x∈X ∫u∈Jx µÃ(x,u) / (x,u) = ∫x∈X ∫u∈Jx fx(u) / (x,u)
(2.53)
where ∫∫ indicate the union over all admissible x and u, fx(u) is the secondary membership function. For discrete universe of discourse ∫ is replaced by ∑ as follows;
Ã=∑x∈X ∑u∈Jx µÃ(x,u) / (x,u) = ∑x∈X ∑u∈Jx fx(u) / (x,u)
(2.54)
Fig. 2.10 Example of a full type 2 membership function. The shaded area is the ‘Footprint of Uncertainty’ (FOU). The amplitude of the sticks is the secondary membership values. This figure is adopted and modified from Mendel[2003].
Secondary membership function is a vertical slice of µÃ(x,u), that is a function that maps membership values of universe of discourse X onto unit interval [0,1]. Type-2 fuzzy sets can be interpreted as the union of secondary membership functions, fx(u). Thus type-2 membership function can be represented with the secondary membership functions for continuous domains by μÃ(x,u) = ∫u∈Jx fx(u) / u
(2.55)
and for discrete universe of discourse by μÃ(x,u) = ∑u∈Jx fx(u) / u
(2.56)
In basic terms mapping of the type-2 membership function onto its secondary membership functions can be reinterpreted as: μÃ(x) : X → fx(u) / u, u∈Jx , Jx ⊆ [0,1]
(2.57)
Equation (2.57) states that, at each x there is a membership value assigned to u for each fuzzy set and it also has a secondary membership function, fx(u). u is the domain of the secondary membership function and it is called the primary membership function of x where Jx ⊆[0,1] and ∀x∈X.
2.5 Type-2 Fuzzy Sets
31
2.5.2 Interval Valued Type-2 Fuzzy Sets Although different representations exist, interval valued type-2 fuzzy sets derived by Karnik, Mendel, and Liang [1999], are commonly preferred methods because full type-2 fuzzy sets are computationally complex when the number of variables is large. Hence, interval valued type-2 fuzzy sets are a type of type-2 fuzzy sets where the membership function is defined by μÃ(x) : X → 1 / u, u∈Jx , Jx ⊆ [0,1]
(2.58)
which satisfies the following condition:
fx(u) = 1, ∀x ∈ X , u∈Jx , Jx ⊆ [0,1]
(2.59)
From (2.59) one can observe that, at each value of x, all the secondary membership values are equal to 1, as shown in Figure 2.11.
Fig. 2.11 Interval Valued Type-2 Fuzzy Set
However, there may be more than 1 membership function at any value of x. The membership functions, which form the boundary values, are defined as the lower and upper membership functions. The upper and lower membership functions of types in (2.59) define the upper and lower membership values at each value of x and using (2.59) they are defined by:
μ AL ( x) = min(u ), ∀x ∈ X
(2.60)
μ UA ( x) = max(u ), ∀x ∈ X
(2.61)
u∈J x
u∈J x
in which μ AL ( x) represents the lower membership function and μ UA ( x) defines the upper membership function. Consequently a type-2 fuzzy set is denoted by μÃ(x): X →1/u, u∈[ µ Ã(x), µ Ã(x)] L
U
(2.62)
32
2 Fuzzy Sets and Systems
2.5.3 Type-2 Fuzzy Set Operations Type-2 fuzzy set operations were first introduced in Zadeh’s paper [1975a]. In [Mendel, 2001] mathematical definitions of Type-2 fuzzy sets and operations are discussed in detail. This section presents basic operations necessary to conduct type-2 fuzzy logic systems, which is going to be reviewed in Chapter 5. Let two Type-2 fuzzy sets, à and B have membership functions, A A μÃ(x):X→fx(u)/u, u∈Jx , Jx ⊆[0,1] and μ B ( x) : X → f x ( w) / w, w ∈ J xB , J xB ⊆ [0,1].
Intersection of type-2 fuzzy sets (MEET OPERATOR): The membership function, μ A B ( x) obtained from the application of intersection
and operator, of two Type-2 fuzzy sets, A μ A B ( x) =
u
w
B , is denoted as follows:
min ( f x (u ), g x ( w) ) / T (u , w), ∀x ∈ X
(2.63)
In (2.63) T is the t-norm operator. Union of type-2 fuzzy sets (JOIN OPERATOR): The membership function, μ A B ( x) obtained from the application of union operator, of two type-2 fuzzy sets, Ã and B , is denoted with the following:
μ A B ( x) = ∫u ∫w min ( f x (u ), g x ( w) ) / S (u, w), ∀x ∈ X
(2.64)
In (2.64) S is the t-conorm operator. Complement of type-2 fuzzy sets: The complement of a type-2 fuzzy set à is denoted with A whose membership functions are denoted with μ A ( x) . The complement is computed as follows:
μ A ( x) = ∫u f x (u ) /(1 − u ), ∀x ∈ X
(2.65)
It should also be noted that, due to computational complexity of operations with type-2 fuzzy sets, interval type-2 fuzzy sets have been widely preferred to full type-2 fuzzy sets. Let à and B denote two interval valued type-2 fuzzy sets with membership functions, μÃ(x):X→1/u, u∈[ μ AL , μ UA ], where μ AL ( x) , μ UA ( x) , μ BL ( x) , and μ BU ( x) represent lower and upper membership functions respectively. The operators on interval valued type-2 fuzzy sets are briefly summarized next. Intersection of interval valued type-2 fuzzy sets: Membership function, μ A B ( x) obtained from the application of intersection operator, of two interval valued type-2 fuzzy sets, à and B , is denoted by:
2.6 Fuzzy Functions
33
μ A B ( x) : X → 1/ u , u ∈ ⎡⎣ μ AL B ( x), μ UA B ( x) ⎤⎦
(2.66)
In (2.66) the lower membership functions are defined as :
μ AL B ( x) = T ( μ AL ( x), μ BL ( x) ) ,
(2.67)
and upper membership functions are defined as:
μ UA B ( x) = T ( μ UA ( x), μ BU ( x) )
(2.68)
Union of interval valued type-2 fuzzy sets: Membership function, μ A B ( x) obtained from the application of intersection operator, of two interval valued type-2 fuzzy sets, Ã and B , is denoted by:
μ A B ( x) : X → 1/ u , u ∈ ⎡⎣ μ AL B ( x ), μ UA B ( x ) ⎤⎦
(2.69)
In (2.69) the lower membership functions are defined with:
μ AL B ( x) = C ( μ AL ( x), μ BL ( x) )
(2.70)
where C represents the t-conorm operation, and upper membership functions are defined with:
μ UA B ( x) = C ( μ UA ( x), μ BU ( x) )
(2.71)
Complement of interval type-2 fuzzy sets: The complement of an interval type-2 fuzzy set à is denoted with A , whose membership functions are denoted with μ A ( x) . The complement is computed as follows:
μ A ( x) : X → 1/ u , u ∈ ⎡⎣ μ AL ( x), μ UA ( x) ⎤⎦
(2.72)
In (2.72) lower membership function is expressed as follows:
μ AL ( x) := 1 − μ UA ( x) ,
(2.73)
and upper membership function is expressed as follows:
μ UA ( x) := 1 − μ AL ( x)
(2.74)
2.6 Fuzzy Functions “Fuzzy Functions” has been used by researchers to define different things. One of the many uses of the term refers to membership functions. It is impossible to classify
34
2 Fuzzy Sets and Systems
the researchers who use “Fuzzy Functions” to denote membership functions since most of the fuzzy theory researchers use them interchangeably. However, in literature there are other definitions of “Fuzzy Functions”. The building blocks of the fuzzy set theory is proposed by Zadeh [1965], especially the fuzzy extensions of classical basic notations such as logical connectives, quantifiers, inference rules, relations, arithmetic operations, etc. Hence, these constitute the initial definitions of fuzzy functions. Marinos [1969] introduced the concept of well-known conventional switching theory techniques into the design of fuzzy logic systems, based on fuzzy set theory and fuzzy operations of Zadeh [1965]. Marinos developed algebra for fuzzy sets, where the membership functions are interpreted as fuzzy numbers. Generally speaking, the processes with fuzzy attributes are represented with fuzzy logic functions. Hence, Marinos’s paper is one of the examples of the application of fuzzy inference mechanisms onto real world engineering problems based on multi-valued fuzzy functions. Later, Siy and Chen [1972] and Demirci [1999] has explored and presented many different arithmetic operators on complex fuzzy functions. These researches on fuzzy functions are the conceptual origin of the fuzzy functions as proposed in this work. Nonetheless, they did define the mathematical foundations of “Fuzzy Functions”. The following is the basic definition of fuzzy function with respect to fuzzy sets [Marinos, 1969]. Let X and Y be two fuzzy sets and x and y be the membership grades of an “object” with respect to sets X and Y, respectively. Definition 1. Two fuzzy sets X and Y are equal (X=Y) if, and only if, for every object i one has that its membership grade xi in X, i.e., µ(xi), is equal to its membership grade yi in Y, i.e., µ(yi). Definition 2. A fuzzy sets is a complement of another fuzzy set X and is denoted by X' if, and only if, for every object i one has that its membership grade xi' in X' is equal to (1-xi), where xi is the membership grade of the object i in X. Definition 3. A fuzzy set X is contained in another fuzzy set Y if, and only if, for every object i one has the xi≤ yi. Definition 4. Two fuzzy sets X and Y form a union, denoted by z=X+Y, if, and only if, for every object i one has that z=max(xi,yi). Definition 5. Two fuzzy sets X and Y form an intersection, denoted with Z=X⋅Y, if, and only if, for every object i one has the z=min(xi,yi).
Based on the above definitions, algebra for fuzzy sets is developed analogous to the Boolean algebra of two valued logic. In the sequel, the term membership grade will be replaced by more appropriate term “fuzzy variable” of a sample fuzzy function, which can be defined as follows: F(x,y)=x⋅y' + x'⋅y
(2.75)
f(x,y)=max[min(x,1-y), min(1-x,y)]
(2.76)
which implies that:
2.6 Fuzzy Functions
35
In other words, (2.76) is a function defined between two fuzzy sets identified with fuzzy numbers, i.e., membership values, and the membership values of this function is determined by the (⋅) and (+) operators among each membership values of its objects. Arithmetic operations on latter types of fuzzy functions get complicated as the number of variables and the number of operations increase. Additionally, there is infinite number of assignments to the grade of memberships, which may then increase the complexity. Kandel [1974; 1977] has worked on the minimization of the fuzzy functions to build simplified reasoning algorithms with fuzzy logic functions. Later in 1981, Ziwei examines the properties of fuzzy switching functions of n variables. It should be pointed that, these fuzzy functions are represented only via membership values. These types of fuzzy functions are used in proposed improved fuzzy clustering algorithm and denoted as “Interim Fuzzy Functions”, to be later presented in Chapter 3. The “Fuzzy Functions” have also been used to refer to fuzzy rules in fuzzy rule base systems, particularly to refer to the Takagi-Sugeno fuzzy inference systems, where the consequents are the linear or non-linear combinations, or functions between the input variables and the output variables. These systems will be briefly described in the next section. In these systems, each local model, i.e., fuzzy rule, is identified with a separate function. Different approximators, i.e., fuzzy function approximators, are used to identify fuzzy functions, such as linear regression functions [Takagi and Sugeno, 1985; Sugeno and Kang, 1988], multi-layered neural networks2, [Jang, 1993; Abe, 1999; Kasabov and Song, 2002] or genetic algorithms [Cordon et al., 2001,2004; Camaro et al., 2004]. The application of these types of “Fuzzy Functions” is the closest to the “Fuzzy Functions” strategies as being presented in this work. In 2005, Turksen introduced a novel representation and reasoning with “Fuzzy Functions”. These “Fuzzy Functions” are multi-variable crisp valued functions. The prominent feature of these functions, f(X,µ), are that they use the degree of membership, µ, of each object to the specified fuzzy set as an additional attribute just as the rest of the input variables, X. In a sense, the gradations (membership values) become the predictors. This type of “Fuzzy Functions” emerged from the idea of representing each unique fuzzy rule in terms of functions. One of the aims of formulizing this type of “Fuzzy Functions” is that they would require less fuzzy operators and only require the knowledge of how fuzzy sets of the given system and the formulation of “Fuzzy Functions” are determined. Any function approximation method, such as least squares or neural networks tool can be used to find the parameters of these “Fuzzy Functions”. Needless to say, empirical applications [Turksen and Celikyilmaz, 2006] of the system modeling with “Fuzzy Functions” using simple linear regression methods has shown promising results as compared to the conventional fuzzy rule base systems. Later, these functions are extended using machine learning algorithms, e.g., support vector machines [Celikyilmaz, 2
A brief background of Artificial Neural Network methods is presented in Appendix C.1.
36
2 Fuzzy Sets and Systems
2005; Turksen and Celikyilmaz, 2006] in performance improvement analysis. This work introduces a novel design of “Fuzzy Functions”, and presentes new improvements and the algorithms to explain the uncertainties in fuzzy system models in greater detail.
2.7 Fuzzy Systems Based on the type of fuzzy sets used, i.e., type-1 and type-2, the conventional fuzzy systems are denoted as type-1 or type-2 fuzzy systems accordingly. Here, we define conventional type-1 fuzzy system models based on fuzzy rule bases and their wellknown extensions that are used as benchmarking tools in chapter of experiments (Chapter 6). Type-2 extensions will be briefly presented in chapter 5 later as entitled Modeling Uncertainty with Improved Fuzzy Functions, where a novel type 2 fuzzy system modeling based on fuzzy functions approach is also presented. Fuzzy system models define relationships between input and output variables of a system by using linguistic labels (i.e., fuzzy sets) in a collection of IF-THEN rules. Among several fuzzy system model structures, the ones as proposed by Zadeh [1965], Takagi-Sugeno [1985], and Mizumoto [1989] are considered renowned and most commonly used approaches. Over the years, many variations of these fundamental fuzzy systems have been proposed. Nonetheless, in the next subsection, renowned fuzzy rule base structures proposed by Zadeh, TakagiSugeno and Mizumoto Fuzzy Rule Base structures will be briefly reviewed. Zadeh Fuzzy Rule Base Structure
Zadeh’s [1965] fuzzy rule base structure is formulated as: c nv ⎡ ⎤ R : ALSO ⎢ IF AND ( x j ∈ X j isr A ji ) THEN y ∈ Y isr Bi ⎥ i =1 j 1 = ⎣ ⎦
(2.77)
In (2.77); ¾ ¾ ¾ ¾ ¾ ¾ ¾ ¾ ¾
c is the number of rules in system model, xj is jth input variable, j=1,…,nv, nv is total number of input variables, Xj is the domain of xj, Aji is the linguistic label associated with input variable, xj in rule i with membership function μAji (xj):XjÆ[0,1], y is the output variable, Y is the domain of y, Bi is the linguistic label associated with the output variable in the ith rule with the membership function μBi (y):YÆ[0,1], AND is the logical connective used to aggregate membership values of input variables for a given observation in order to find the degree of fire of each rule, THEN (→,⇒) is the logical IMPLICATION connective,
2.7 Fuzzy Systems
37
¾
ALSO is the logical connective used to aggregate model outputs of fuzzy rules, ¾ ‘isr’ is introduced by Zadeh and it represents that the definition or assignment is not crisp, it is fuzzy. Inference parameters that one should identify before using fuzzy logic structure identification method are; c*, types of membership functions of each input variable, membership function of the output variable, and types of operators for AND, THEN, and ALSO. Mizumoto Fuzzy Rule Base Structure
Mizumoto fuzzy rule base structures [Mizumoto, 1989] is a simplified version of Zadeh fuzzy rule base structures, where the consequent of each rule is represented with a scalar, bi, instead of a fuzzy set Bi. These types of inference structures are defined as follows: c nv ⎡ R : ALSO ⎢ IF AND x j ∈ X j isr A ji i =1 ⎣ j =1
(
)
⎤ THEN yi = bi ⎥ ⎦
(2.78)
In (2.78) bi is the scalar associated with ith rule. Inference parameters to be determined by using Mizumoto fuzzy inference system are c, membership function of each input variable in each rule, scalars associated with each rule, bi, i=1,…,c, AND, THEN and ALSO connectives. The generalized fuzzy inference structure is explained as follows: Let an input vector x'=(x'1, x'2… x'nv) represent a testing vector, whose output value is unknown. Using a fuzzy rule base structure and an empirical data (xk,yk)= (xk,1, xk,2… xk,nv ,yk), k=1…n, we would like to apply fuzzy inference methods and obtain an approximate crisp output value for x'. Let Xj be the domain of xj, j=1…nv, Aij represent membership function associated with input variable xj in rule i denoted with membership function μi(xj):XjÆ[0,1], c be the number of rules in FRB, and nv be the number of attributes/input dimensions in defined system. Based on the general structure of fuzzy rule bases as shown in Figure 2.12 five common steps are followed: FUZZIFICATION: Fuzzification assigned a membership value to each new data vector, x', for each fuzzy set Aij in each rule i, and is represented with a membership value, μi(x'j), ∀i=1…c, ∀j=1…nv. AGGREGATION OF ANTECENDENTS: The membership values measured in Fuzzification step are aggregated to obtain a single degree of fire, viz. membership value. The aggregation operation is processed using fuzzy operators, the most common of which are t-norms using MIN connective, denoted with ∧. Hence, the degrees of fire of rule i for the new input vector, x', can be calculated as follows:
38
2 Fuzzy Sets and Systems
τ i ( x′ ) = AND μi ( x′j ) ,∀i = 1...c nv
j =1
(2.79)
IMPLICATION: Identification of Model Output Fuzzy Sets. As much as the degree of fire represents a measure of compatibility among the new data objects and the antecedents, it also defines the level of contribution of the output fuzzy set of each rule on identifying the final model output fuzzy set. In Mamdani type inference method [Mamdani and Assilian; 1974], AND connective is used as IMPLICATION connective, where MIN operation is used to obtain the membership function of the model output fuzzy set in rule i for a given observation, x' as follows: μi′ ( y ) = τ i ( x′ ) AND μi ( y ) = τ i ( x′) ∧ μi ( y ) ,∀i = 1...c
(2.80)
AGGREGATION OF CONSEQUENTS: One fuzzy set for each rule is identified for the model output in IMPLICATION step. Here ALSO operator is used to aggregate those output fuzzy sets to obtain one single fuzzy output set. In Mamdani Fuzzy Inference schema, ALSO is taken as OR operator – mainly used for MAX operator – and it is calculated as follows: c
μ ′ ( y ) = ∨ μi′ ( y )
(2.81)
i =1
DEFUZZIFICATION: The last step of the fuzzy inference method is to obtain a crisp output from Type 1 fuzzy inference. Here we define the most commonly used defuzzification method that is Center of Gravity (COG) by y′ =
y
y μ ′( y ) y
μ ′( y )
(2.82)
Takagi-Sugeno-Kang Fuzzy Rule Base Structure
One of the most commonly implemented and investigated fuzzy inference systems are Takagi-Sugeno-Kang (TSK) fuzzy rule base systems [Takagi and Sugeno, 1985; Sugeno and Kang, 1986], which is slightly different as compared with that of Zadeh’s fuzzy rule base. The only difference is the consequent part of the rule base structure. As Zadeh used fuzzy sets to represent both antecedent and consequent parts (arguments of fuzzy variable) of fuzzy rules; whereas in TSK fuzzy rule base, the antecedent part of a rule is characterized with an aggregation operator and the consequent is characterized with a regression line. Fuzzy rule base structure by TSK can be given as follows: c
nv
i =1
j =1
R : ALSO IF AND ( x j ∈ X j isr A ji ) THEN yi = ai xT + bi
(2.83)
2.7 Fuzzy Systems
39
In (2.83), ¾ ¾ ¾ ¾
ai and bi are regression line coefficients associated with ith rule, yi is the model output of ith rule, Ri, THEN is the connective, which weights yi for each rule by using corresponding degree of firing of a given observation in order to find the model output from each rule, ALSO is the connective, which takes the weighted average of the model output of each rule in order aggregate the model outputs of fuzzy rules.
The inference parameters of TSK fuzzy rule base method are c, membership function of each input variable in each rule, regression line coefficients for each rule, and AND, THEN, and ALSO connectives used in the inference method. Although the consequent part of the TSK fuzzy rule base in (2.83) is a linear function, any non-linear polynomial function with higher orders can be implemented. As mentioned in the earlier chapters, “Fuzzy Functions” have also been seen in the literature to denote approximators for the argument, usually the consequence as a part of the of fuzzy rules. The function can be linear or non-linear function. A well-known sample to such systems is the Takagi-Sugeno-Kang (TSK) [Takagi and Sugeno, 1985; Sugeno and Kang, 1986] type fuzzy inference system, where the consequent parts of fuzzy rules are represented with functions between inputs and output variables. The well-known fuzzy system modeling tools that are used as benchmarking methodologies in this work, implement Takagi-Sugeno type fuzzy rule bases. Before we review these systems, the structure of conventional Takagi-Sugeno type fuzzy inference systems will be explained next. In a fuzzy rule base structure each antecedent is assumed to be non-interactive, therefore separate fuzzy sets are identified for each antecedent input variable separately. The generalized structure of fuzzy inference systems or fuzzy rule base systems is shown in Figure 2.12. Identification of Degree of Fire Fuzzification + Aggregation of Antecendets
Implication
Fuzzy Input Sets
Rules + Aggregation of Deduced output Fuzzy sets
Defuzzification
Fuzzy Output Sets
Crisp Output Defuzzifier
Fig. 2.12 Generalized Type 1 Fuzzy Inference Systems (Fuzzy Logic System)
Fuzzy Inference is the mathematical procedure to deduce model output from a given set of fuzzy rules. One of the first real-life applications of fuzzy rule base structure on control systems is studied by Mamdani and Assilian [1974]. They used a fuzzy rule base in order to control a cement pilot plant. Today, there are many different applications of fuzzy inference systems including financial, health-care, robotics, web data mining, and many more. In the following, Takagi-Sugeno-Kang type fuzzy inference system structure will be explained in detail as an example.
40
2 Fuzzy Sets and Systems
Using a fuzzy rule base structure and an empirical data (xk,yk)= (xk,1, xk,2… xk,nv ,yk), k=1…n, we would like to apply fuzzy inference methods and obtain an approximate crisp output value for x'. Let Xj be the domain of xj, j=1…nv, Aij represent membership function associated with input variable xj in rule i as denoted with membership function μi(xj):XjÆ[0,1], where c is the number of rules in fuzzy rule base system, and nv be the number of attributes/input dimensions in defined system. FUZZIFICATION: A membership value is assigned to each new data vector, x', in each fuzzy set Aij of each rule i, and is represented with a membership value, μi(x'j), ∀i=1…c, ∀j=1…nv. AGGREGATION OF ANTECEDENTS: The membership values measured in Fuzzification step are aggregated to obtain a single degree of fire, viz. membership value. The aggregation operation is processed using fuzzy operators, the most common of which are t-norms using MIN connective, denoted with ∧. Hence, the degrees of fire of rule i for the new input vector, x', is calculated as follows:
μi ( x′ ) = AND μi ( x′j ) ,∀i = 1...c nv
j =1
(2.84)
IMPLICATION: Depends on how the consequent part of a fuzzy rule base structure is defined. For TSK type fuzzy rule bases, this step is merely used to weight th * the scalar model output of each rule. Thus, the model output of an i rule, yi in (2.83) is weighted as follows:
y*i = μi ( x′ ) × yi ,∀i = 1...c
(2.85)
AGGREGATE THE MODEL OUTPUT: This step also depends on the consequent structure of the rule base. In TSK fuzzy rule base structures, the model output of each rule is aggregated by taking the weighted average of the scalar output of each rule in the fuzzy rule base as follows:
∑ y* y = c i =1 i ,∀i = 1...c ∑ i =1 μi ( x′ ) c
*
(2.86)
DEFUZZIFICATION: It is used to reduce the fuzzy set into a crisp value in the universe of discourse of the output variable. This step is required for fuzzy inference systems, which define fuzzy sets in the consequent part of the rules. Since TSK rule bases consequent parts are identified with linear functions from which scalar values are obtained as in (2.83), defuzzification step is not required for TSK fuzzy systems.
2.8 Extensions of Takagi-Sugeno Fuzzy Inference Systems Classical system modeling based on two-valued logic, i.e., x∈ {0,1} has long been shifted to more sophisticated models, i.e., soft computing methods, where
2.8 Extensions of Takagi-Sugeno Fuzzy Inference Systems
41
boundaries of decisions are more relaxed to explain the concepts such as uncertainty and imprecision. Soft computing is an association of computing methods making the good use of the fuzzy logic, neural networks, rough sets, genetic algorithms, etc. In the last few decades, various synergetic combinations of soft computing methods to form hybrid models, such as neuro-fuzzy inference systems or genetic fuzzy systems, [Cordon et al, 2001;Deboeck, 1992; Tettamanzi and Tomassini, 2001; Roland, 2003; Wang et al. 2005; Oh and Pedrycz, 2006; Celikyilmaz and Turksen, 2007g], have been proposed to improve system modeling performances based on different aspects. Many benefits can be described, such as the increase of accuracy, the decrease of complexity or shortening of the computation time. In this section we briefly review structure identification methods of extensions of well known hybrid fuzzy inference systems, in particular adaptive network fuzzy inference system (ANFIS), dynamically evolving neural fuzzy inference system (DENFIS) and Genetic Fuzzy System (GFS)s. These hybrid fuzzy system modeling tools based on conventional fuzzy rule base structures are used as benchmarking tools in experiments to evaluate the performance of the presented hybrid fuzzy system modeling tool based on novel “Fuzzy Functions” strategies, to be discussed in Chapter 4.
2.8.1 Adaptive-Network-Based Fuzzy Inference System (ANFIS) In 1993, Jang [Jang, 1993] proposed the Adaptive-Network-Based Fuzzy Inference Method (ANFIS), which has a special place in fuzzy learning society; because it has been one of the most commonly used and investigated algorithms among the rest of the fuzzy learning algorithms in the last decades. ANFIS is a neuro-fuzzy technique that brings learning capabilities of neural networks to fuzzy inference systems. In ANFIS, the most commonly implemented fuzzy inference method is Takagi-Sugeno-Kang (TSK) Fuzzy Inference System (FIS) [Takagi and Sugeno, 1985; Sugeno and Kang, 1986]. The parameters of ANFIS are optimized in an Adaptive Neural Network framework. An adaptive network is a network structure consisting of nodes and directional links through which the nodes are connected. The adaptive network forces the output of each node to depend on the parameter(s) pertaining to the corresponding node, and the learning algorithm indicates how these parameters should be changed to minimize a defined error measure. Let us assume (for simplicity) that the fuzzy system under consideration has two inputs x1 and x2 and one output variable y and only two fuzzy rules are defined. The common structure of ANFIS is the TSK type fuzzy inference system [Takagi and Sugeno, 1985], which is defined as follows: Rule 1: IF x1 is A1 and x2 is B1 THEN y=p1x1 + q1x2 + r1 Rule 2: IF x1 is A2 and x2 is B2 THEN y=p2x1 + q2x2 + r 2
42
2 Fuzzy Sets and Systems
The architecture of the FIS defined above and its equivalent corresponding ANFIS structure are displayed in Figure 2.13 and Figure 2.14 respectively. Next, we will briefly explain each layer of ANFIS as shown in Figure 2.14. Layer 0: This layer corresponds to each individual input variable in the dataset, e.g., x1, x2. Layer 1 (Fuzzification): Each node in this layer (denoted with a square) represents a node function where Ai and Bi are linguistic labels associated with a node function, viz., membership function, μAi(x). In ANFIS usually bell-shaped or Gaussian membership functions are utilized as follows:
f1 = p1 x1 + q1 x2 + r1 f =
w1 f1 + w2 f 2 w1 + w2
= w1 f1 + w2 f 2
f 2 = p2 x1 + q2 x2 + r2
Fig. 2.13 TSK Fuzzy Reasoning
x1
A1
w1(x1,x2) ∏
A2
w1 =
w1 w1 + w2
⊗
f1(x,y)
w1 f1
∑ x2
B1 B2
Fuzzification
Layer
0
1
∏
w2(x1,x2) Aggregation of Antecedents
2
Fig. 2.14 A Sample ANFIS Structure
⊗
f2(x,y)
w2 f 2
w2 w2 = w1 + w2 Normalization of Degrees of Fire
3
Implication
4
Aggregation of Consequents
5
f(x,y)
2.8 Extensions of Takagi-Sugeno Fuzzy Inference Systems
μ Ai (x)=
43
1 1+ ⎡ ⎣⎢
( ) x − ci ai
2
⎤ ⎦⎥
bi
(2.87)
The parameter set {ai, bi, ci} in (2.87) represent the premise parameters. Layer 2 (Aggregation of Antecedents): The circle nodes represented with ∏ in Figure 2.14 denotes a fuzzy operator, e.g., product t-norm, which multiplies the incoming signals such as
wi = μ A ( x1 ) ∧ μ B ( x2 ), i = 1, 2 i
(2.88)
i
and sends the output to the next layer. Layer 3 (Normalization of Degrees of Fire): The nodes in this layer denoted with ⊗ calculate the ratio of the ith rule’s firing strength to the sum of firing strengths of all rules by
wi =
wi w1 + w2
(2.89)
Layer 4 (Implication): The nodes in this layer are denoted with a rectangle and they calculate the weighted output of each linear function as follows:
wi fi = wi ( pi x1 + qi x2 + ri )
(2.90)
The parameter set, {pi, qi, ri } in (2.90) will be denoted as consequent parameters. Layer 5 (Aggregation of Consequents): The single node denoted with a ∑ computes overall output as follows:
yˆ = ∑ i wi fi =
∑ i wi fi ∑i wi
(2.91)
In (2.91), ŷ is the estimated overall output value. Since ANFIS implements an adaptive learning algorithm, it uses either back propagation or a combination of least squares estimation and back-propagation for parameter estimation. Here, we describe the learning rules and their parameters as implemented in MATLABANFIS. Adaptive network structure, maps inputs through input membership functions and associated parameters, and then onto outputs through output membership functions and associated parameters. The parameters associated with the membership functions changes through the learning process. The computation of these parameters (or their adjustment) is facilitated by a gradient vector. This gradient vector provides a measure of how well the fuzzy inference system models the input/output data for a
44
2 Fuzzy Sets and Systems
given set of parameters. When the gradient vector is obtained, any of several optimization routines can be applied in order to adjust the parameters to reduce some error measure. This error measure is usually defined by the sum of the squared difference between actual and desired outputs. ANFIS in MATLAB implements two methods to identify the inference parameters, namely grid partitioning and subtractive clustering. Grid partitioning is a subjective approach, in which the number of clusters, into which each variable is segmented, is initially provided by the user. The number of rules is determined by multiplying the number of clusters of each input variable. This introduces dimensionality problem, in which as the number of variables increase the number of rules also increase. Subtractive clustering, proposed by Chiu [1994], is much more intuitive and does not suffer from dimensionality issues. In that, the rule extraction method first determines the number of rules and antecedent membership functions and then uses linear least squares estimation to determine each rule's consequent equations. Subtractive clustering method of ANFIS identifies inference parameters of TSK fuzzy inference system structure. It should be noted that, in subtractive clustering ANFIS method, one fuzzy set is identified for each input variable in each rule.
2.8.2 Dynamically Evolving Neuro-Fuzzy Inference Method (DENFIS) In 2002, Kasabov and Song proposed an alternate algorithm to ANFIS entitled Dynamically Evolving Neuro-Fuzzy Inference Method. Neuro-Fuzzy inference systems such as ANFIS consist of set of rules and an inference method that are embodied or combined with a connectionist structure for a better adaptation (the learning mechanism is explained in the latter section). On the other hand Evolving Neuro-Fuzzy Inference systems are such systems where both the knowledge and the inference mechanism evolve, change in time, with more examples presented to the systems. Hence, DENFIS has successfully applied on time series applications such as stock price estimation problem [Kasabov and Song, 2002]. In chapter of experiments we present stock price estimation models. Thus, we used DENFIS as a benchmarking hybrid fuzzy modeling tool. DENFIS has two modifications, online and offline learning using TSK type fuzzy inference method [Takagi and Sugeno, 1985; Sugeno and Kang, 1986]. In the online learning, linear functions in the consequence parts are created and updated through learning from data by using a linear least square estimator. In this work, we applied offline learning strategy because we use empirical data to build system models. In off-line model, the data is initially clustered to find c cluster centers by using on-line evolving clustering method (ECM) and off-line evolving clustering method with constraint optimization (ECMc). Then antecedent part of TSK style rules is created. Then c datasets are formed, each of them including one cluster center, and k learning pairs that are closest to the center in the input space. In the following we briefly explain each of these steps of DENFIS leaning algorithm.
2.8 Extensions of Takagi-Sugeno Fuzzy Inference Systems
45
Offline Evolving Clustering Method with Constraint Optimization –ECMc
The offline ECMc [Kasabov, Song, 2002] applies an optimization procedure to the cluster centers obtained from evolving clustering method (ECM). ECM implements a scatter partitioning of the input space (crisp partitioning) for the purpose of creating fuzzy inference rules. ECM dynamically estimates the number of clusters in a set of data, and finds their current cluster centers in the input data space. In a cluster the maximum distance, MaxDist, between an example point and the cluster center, is less than a threshold value, Dthr, that is set as a clustering parameter before the model run. In ECM, the distance between vectors is measured with Euclidean distance. At the start of the ECM process, the data examples come from a data stream and ECM process starts with an empty set of cluster centers. When a new cluster is created, the first cluster center, v1, is defined and its radius, R1, is initially set to zero. Then as examples are presented to the algorithm, cluster centers are updated through changing their centers’ positions and increasing their cluster radiuses. Depending on the position of the currently presented data sample, the cluster to be updated and how much it will be changed is determined. A cluster will not be updated any more when its cluster radius, Ri,i=1…c, reaches the value that is equal to a threshold value, Dthr. More detailed background of ECM can be found in [Kasabov, Song, 2002]. Next, cluster centers, vi, obtained from ECM are used as initial cluster centers for offline clustering methods, ECMc. The ECMc uses Euclidean distance measure between data vectors. In each cluster, cluster centers are calculated and an objective function is minimized based on the following distance measure: c ⎡ J ECMc = ∑ ⎢ ∑ xk − vi i =1 ⎣ x ∈v k
i
⎤ ⎥, k = 1,..., n ⎦
(2.92)
subject to
xk − vi ≤ Dthr , j = 1, 2,...c
(2.93)
Each cluster is defined by a membership matrix uik∈U∈{0,1} × , which is a binary matrix in which the element uik =1 implies that kth data object belongs to cluster i. Once the cluster centers vi are fixed, uik is obtained by n c
if xk − vi ≤ xk − v j , for each i ≠ j; uik = 1, else u ij = 0.
(2.94)
Using the iterative algorithm in ALGORITHM 2.1, the cluster centers and membership matrix are determined.
46
2 Fuzzy Sets and Systems
ALGORITHM 2.1 Evolving Clustering Method for constraint optimization of DENFIS Step 1: Initialize cluster centers with ECM clustering process, Step 2: Determine the membership matrix U by (2.94) Step 3: Apply ECMc clustering using objective function in (2.92) with constraints (2.93) to get new cluster centers. Step 4: If objective function J in (2.92) is below a certain value or certain number of iterations is over, then stop, otherwise return to Step 2.
DENFIS Leaning Mechanism
After the ECMc obtains cluster centers, DENFIS uses Takagi-Sugeno-Kang type fuzzy inference engine [Takagi and Sugeno, 1985; Sugeno and Kang, 1986]. The antecedent fuzzy sets are triangular fuzzy numbers, ⎧0 ⎪ x−a ⎪ μ ( x) = ⎨ bx−−aa ⎪ b−a ⎪⎩ 0
x≤a a≤ x≤b , b≤ x≤c
(2.95)
c≤x
where b is the value of the cluster center on the x domain, a=b-d*Dthr and c=b+d*Dthr, d=[1.2-2] is defined by the user. The threshold value, Dthr, is determined during clustering process. The DENFIS offline learning is implemented in the following way:
¾ The cluster centers obtained from ECMc are used to construct the antecedent part of the rules as in (2.95).
¾ c datasets are captured each of which include one cluster center and n data pairs that are closest to the center in the input space. Hence, data pairs may belong to several clusters. ¾ the consequent functions are learnt either using weighted least squares method using membership values as weight vector, or applying ANN learning method to find the functions of the consequents. Here, only a brief introduction to DENFIS is presented. More detailed information can be found in [Kasabov, Song, 2002].
2.8.3 Genetic Fuzzy Systems (GFS) In recent years, a great number of publications have explored the use of genetic algorithms as a tool for designing fuzzy systems. Genetic Fuzzy Systems apply genetic algorithm principles to optimize the parameters of fuzzy inference systems. In other words, fuzzy system generated or adapted by genetic algorithms is called Genetic Fuzzy Systems (GFS). Genetic algorithm (GA)3 is an optimization and search technique based on principles of genetics and natural selection. Genetic 3
Genetic Algorithms are briefly presented in Appendix C.3.
2.8 Extensions of Takagi-Sugeno Fuzzy Inference Systems
47
algorithms were introduced by John Holland in 1973 and later the methodology has been developed [Goldberg, 1989] and has been applied to various domains. Genetic algorithms have been used to perform different tasks such as generation of fuzzy rule base, optimization of fuzzy rule bases, generation of membership functions, and tuning of membership functions [Cordon et al., 2001]. All these tasks can be considered as optimization or search processes. The first step in designing GFS is to decide which parts of knowledge base are subject to optimization by the GA. As an example, the knowledge base of a descriptive TakagiSugeno-Kang (TSK) type fuzzy system [Takagi and Sugeno, 1985; Sugeno and Kang, 1988] is comprised of two components: a database containing the definitions of membership functions of the fuzzy sets associated with linguistic labels, and a fuzzy rule base, constituted by the collection of (non)linear functions. The GFS is utilized as a benchmarking tool in the chapter of experiments of this work, is based on a traditional TSK type fuzzy inference systems [Cordon et al., 2001]. In GFS, each chromosome encodes a complete fuzzy rule set. Parameters of triangular membership functions are optimized during the GA process. Here we will use triangular membership functions defined with two parameters as follows: 0 , if x ≤ a − b2 ⎧ ⎪ ⎪ 2 x−a μ ( x ) = ⎨1 − , if a − b2 < x < a + b2 (2.96) b ⎪ ⎪ 0 , if x ≥ a + b2 ⎩ The parameters a and b define the center and spread of triangular membership functions as shown in Figure 2.16. First-order TSK models, which identify linear local functions for each rule, are constructed. Using the rule base structure as shown in (2.83), TSK fuzzy model is described as follows:
Rule i: IF x1 is Ai and…and xnv is Anv THEN yi =βi x1 +…+βi xnv + βi 1
i
*
1
nv
0
(2.97)
where i=1,…,c, c is the number of IF-THEN rules, x=[x1,…,xnv] the premise input j variables, nv the number of input variable, βi , j=1,…,nv, the consequent paramej ters, yi is an output from the ith IF-THEN rule and Ai is a fuzzy variable. Given the input x, the final output of the fuzzy model is inferred by a weighted mean defuzzification as
∑ i =1 μi ( x ) yi* ,∀i = 1...c , c ∑ i =1 μi ( x ) c
y* =
(2.98)
where the weight implies the overall truth value of the premise of the jth implication for the input, and is calculated as
μi ( x ) = AND μi ( x j ) ,∀i = 1...c nv
j =1
(2.99)
48
2 Fuzzy Sets and Systems
AND is usually taken as t-norm function, so (2.99) is calculated using product t-norm as:
μi ( x ) = ∏ μi ( x j ) ,∀i = 1...c , nv
(2.100)
j =1
j
where µi(xj) is the grade of the membership function of Ai and is characterized by a Triangular function as in (2.95). Thus, design of fuzzy model consists of optimizing membership function parameters, i.e., a and b in (2.96), which represent the spread and center of the membership function. The affects of change in membership function parameters on the membership values are shown in Figure 2.16. The fitness of each chromosome is evaluated by the error between the fuzzy model output and the desired output, which is to be minimized. Using the GAs to design TSK model, one encodes the TSK model parameters into chromosome. To completely represent a fuzzy system, each chromosome must encode all the needed information in the rule set and membership functions. In order to represent a TSK fuzzy system, described above, each chromosome will encode the parameters of each antecedent membership function in each rule. When scalar valued gene structure is used, each token as shown in Figure 2.15, represents the value of each parameter. The Genetic Algorithm has to be set up by setting the upper and lower bounds for each parameter, through which the algorithm may search. In each membership function, there are three parameters to be optimised, that is the three vertex positions. For simplicity, each corner should have the same amount of freedom. Thus, for a TSK model with c number of rules, j nv number of antecedent fuzzy sets for each input variable, Ai , is defined with two parameters for each triangular membership function, i.e., spread and center, then j there will be c×nv×2 tokens in each chromosome. µi , i=1,…,c, j=1,…,nv represent th th parameters of j antecedent membership function in i rule.
j
j
Fig. 2.15 Chromosome structure of a Generic Fuzzy System. Each ai and bi indicates the triangular membership function parameters
The freedom afforded to each corner position of the triangular membership function can be calculated as follows: Corner variation = ± (abs(data_range) / 2*(no of membership functions)) The relationship means that the situation will never arise that the membership function will always maintain its triangular shape and a conflict will never arise between membership functions. Nonetheless, each membership function is independent of every other membership function so the optimum parameters for each MF will prevail as the algorithm evolves.
2.8 Extensions of Takagi-Sugeno Fuzzy Inference Systems
49
Fig. 2.16 The effect of changing parameters on membership function
Since the consequent parameters βi , j=1,…,nv, can be obtained by least square optimisation method, explicit inclusion of consequent parameters into a chromosome is unnecessary. It is only necessary to encode the parameter of the antecedent membership functions, i.e. centres and widths, which are necessary to represent a triangular membership function, into the chromosome. The initialization of a chromosome is usually done in the same way as the standard GAs using randomly generated numbers. The encoded chromosome is evolved by the conventional evolutionary operations (reproduction (crossover) and mutation). After each evolutionary step, the antecedent membership values of each data point are calculated using the parameters of each antecedent membership j function denoted by each chromosome. Then the consequent parameters βi using least square algorithm is calculated. The mean square error (MSE) for the training data or the validation data is regarded as the objective function (fitness function) of each chromosome and can be calculated as j
J mse =
2 1 n ∑ k =1 ( yk − y*k ) n
(2.101)
*
n is the data length, yk the predicted output and yk is the target output. Based on the fitness obtained with (2.101) the offspring is chosen for the next crossover and mutation steps by using the tournament selection approach4. Then crossover and mutation operation are performed on the current chromosomes of the model to generate new chromosomes in the search space. Elitist’s reinsertion approach guarantees that the best chromosome in the population always survives and is retained in the next generation. For each individual these fitness functions are evaluated until a fixed number of generations or when the search process converges with a given accuracy. The best chromosomes will be used to determine the optimal TSK fuzzy model. 4
Some of the characteristic functions of Generic Algorithms are reviewed in Appendix C.3.
50
2 Fuzzy Sets and Systems
2.9 Summary We briefly summarized characteristics and operations of Type-1 fuzzy sets, Type-2 fuzzy sets, fuzzy functions and extensions of well-known fuzzy structure identification and inference systems. This chapter presents foundations of the algorithms proposed in this work. Hence, this chapter can be considered as a reference for the reader. The following chapters will present the essentials of this work, including improved fuzzy clustering algorithm and structure and mathematical algorithms of the proposed fuzzy system modeling based on novel fuzzy functions.
Chapter 3
Improved Fuzzy Clustering 3 Improved Fuzzy Clustering
The new fuzzy system modeling approach based on fuzzy functions implements fuzzy clustering1 algorithm during structure identification of the given system. This chapter introduces foundations of fuzzy clustering algorithms and compares different types of well-known fuzzy clustering approaches. Then, a new improved fuzzy clustering approach is presented to be used for fuzzy functions approaches to re-shape membership values into powerful predictors. Lastly, two new cluster validity indices2 are introduced to be used to validate the improved fuzzy clustering algorithm results.
Everything is vague to a degree, you do not realize till you have tried to make it precise —Bertrand Russell
3.1 Introduction In 1965 and later in 1975, Zadeh [1965;1975a] introduced the concept of mathematical modeling with imprecise propositions, fuzzy sets and fuzzy logic. Since then, fuzzy sets and logic has been applied on many areas to control uncertain information and to simulate how inferences can be made with uncertain information. It is known that noises and uncertainties can never be totally eliminated in any given database system. Generally, a common way is to use fuzzy logic theory in clustering problems to further explain these types of uncertainties. Fuzzy clustering algorithms can identify overlapping clusters in a given dataset, while calculating membership values that specify to what degree each object belong to such clusters. In the next section, we present firstly the terminology of the fuzzy clustering methods and a general classification of fuzzy cluster analysis. The “Fuzzy 1
This section of this chapter is an extension of papers [Celikyilmaz, Turksen, 2007a,b].
2 This section of this chapter is an extension of papers [Celikyilmaz, Turksen, 2007c and
2008c]. A. Celikyilmaz and I.B. Türksen: Model. Uncertain. Fuzzy Logic, STUDFUZZ 240, pp. 51–104. springerlink.com © Springer-Verlag Berlin Heidelberg 2009
52
3 Improved Fuzzy Clustering
Functions” system modeling approach of this work (Chapter 4) utilizes the most commonly used type of fuzzy clustering algorithm, namely Fuzzy c-Means (FCM) clustering algorithm [Bezdek, 1981a] for structure identification. Therefore, mathematical background of FCM is presented in this chapter. Next, a novel Improved Fuzzy Clustering (IFC) algorithm instead of FCM clustering algorithm is presented to be utilized in “Fuzzy Functions” approaches for prediction problems (regression). Motivation and theory of the IFC algorithm is presented along with its mathematical transformation. An extension to the novel IFC, entitled Improved Fuzzy Clustering for classification – IFC-C - is also proposed to be applied on pattern recognition problem domains. Later, justification of the performance of IFC algorithm in comparison to FCM clustering algorithm is discussed with examples. Lastly, two new cluster validity index (CVI) are presented to be used to find the optimum number of clusters with IFC algorithm for classification and regression type system domains. Using artificial datasets, performance of the new CVIs is discussed and the results are compared to the results from other well known CVIs.
3.2 Fuzzy Clustering Algorithms System modeling with soft computing can be broadly separated into following two main parts: global and local system modeling [Babuška and Verbruggen, 1997]. In global modeling, overall system is analyzed as a whole to understand underlying relationships, if any. In local modeling, the system under study is first decomposed into meaningful parts and sub-models are built using linear or nonlinear methods. Hence, the class of fuzzy clustering algorithms is used to identify these local models. There are numerous ongoing research on enhancing fuzzy clustering algorithms and these can be classified as follows based on the clustering structure: ¾ ¾ ¾
Fuzzy clustering based on fuzzy relations, Fuzzy clustering based on objective function and covariance matrix, Non-parametric classifier, e.g., fuzzy generalized k-nearest neighbour rule, ¾ Neuro-fuzzy clustering, e.g., self organizing maps, fuzzy learning vector quantization, etc.
This work focuses on “objective” based fuzzy clustering algorithms, which assign an error or a quantitative measure to each possible cluster partition using an evaluation function and try to minimize/maximize total error/quality. Ideal solution is reached when cluster-partitions are assessed to obtain the best evaluation. Objective based clustering algorithms try to solve an optimization problem. Definition 3.1. (Objective Function) Objective function J(f) or J is an error or quantitative measure and our aim in fuzzy clustering algorithms is to find the global minimum or maximum of J depending on the structure of the clustering algorithm. J is used to compare different solutions usually for the same clustering problem.
3.2 Fuzzy Clustering Algorithms
53
Let X={x1,x2,…,xn} represent a set of n objects, each k object, k=1,…,n, is represented with nv dimensional vector, xk=[x1,k,…, xnv,k ]T ∈ ℜnv. A set of n vectors is then represented by n×nv data matrix
⎡ x1×1 X = ⎢⎢ " ⎢⎣ xn×1
x1×2 "
" xn×2
" "
x1×nv ⎤ " ⎥⎥ xn×nv ⎥⎦
(3.1)
A fuzzy clustering algorithm partitions given dataset X into c number of overlapping clusters, forming a fuzzy partition matrix, U. Definition 3.2. (Fuzzy Partition Matrix). Fuzzy partition matrix, U, is a matrix of degrees of memberships of every object xk, k=1,…,n in every cluster i, i=1,…,c. The value of the degree of membership of kth vector in cluster i is represented by μik∈U. The partition matrix is in the form:
⎡ μ1,1 ⎢ U = ⎢" ⎢⎣ μ1,n
μ2 ,1 " μc ,1 ⎤ ⎥ " " "⎥ μ 2 ,n " μc ,n ⎥⎦
(3.2)
In fuzzy clustering algorithm, each cluster is represented with a vector called vector of “cluster centers” or “cluster prototypes”, with which cluster structures in overall X can be represented. Definition 3.3. (Vector Cluster Centers/Prototype). In a dataset X of nv dimensional vectors, fuzzy clustering algorithm identifies c number of cluster center vectors, V={υ1,υ2,…,υc}∈ℜc×nv, where each cluster center ,υi∈ℜnv is also an nv dimensional vector. Each cluster center (υi) is usually represented as the centroids of nv objects, e.g., average of all the datum of the corresponding cluster. Among many different types of fuzzy clustering algorithms, this work deals with the objective function based and point-wise (distance-based) clustering algorithms. Since an extension of the presented system modeling approach with “Fuzzy Functions”, to be discussed in the following chapters, executes fuzzy c-means clustering (FCM) algorithm for a structure identification of the presented fuzzy system modeling; detailed explanations of this algorithm will be presented next.
3.2.1 Fuzzy C-Means Clustering Algorithm Fuzzy c-Means (FCM) [Bezdek, 1981a] clustering algorithm is a simple and yet the most commonly used and extended fuzzy clustering method of all fuzzy clustering approaches. In FCM clustering algorithm, it is assumed that number of clusters, c, is known or at least fixed, i.e., FCM algorithm partitions given data set X = {x1,...,xn} into c clusters. Since the assumption of a known or a previously fixed number of clusters is not realistic for many data analysis problems, there are
54
3 Improved Fuzzy Clustering
techniques such as cluster validity index (CVI) analysis to determine the number of clusters of FCM clustering algorithm. Some well known CVIs are discussed and two new cluster validity criteria are introduced in the later sub-sections of this chapter. Let each c cluster be represented with a cluster prototype, vi. FCM clustering algorithm, tries to minimize an objective function with two prior information, number of clusters, c, and fuzziness parameter, m, as follows: min J(X;U,V) =
∑ i =1 ∑ k =1 ( μik ) c
n
m
d 2 ( xk ,υi )
(3.3)
In (3.3), m∈(1,∞) represents the “degree of fuzziness”3 or “fuzzifier” of the fuzzy clustering algorithm and it determines the degree of overlapping of the clusters. “m=1” would mean no overlapping, which represents a crisp clustering structure. Here, d2(xk,υi) is a measure of the distance between kth object and ith cluster center. Squared distances satisfy that objective function is non-negative, J>0. The objective function will be 0 when all data objects are clusters at the centers, c=n. On the other hand, when data objects are farther away from cluster centers, υi, objective function will get larger. The location and the number of cluster centers affect the value of the objective function. Criterion of the objective function is minimum at the optimum solution, and one should search for the global minimum. In order to avoid trivial solutions, two constrains are imposed on the partition matrix U, as follows:
∑ i =1 μik = 1, c
∀k >0
0 < ∑ k =1 μik 0
(3.4)
n
(3.5)
The constraint in (3.4) implies that each row of partition matrix in (3.2) adds up to 14. The constraint in (3.5) implies that the column total of membership values cannot exceed the number of data vectors, n, nor it can be zero. This indicates that there is at least one member assigned to each cluster. However, none of these constraints force membership values of each cluster to have a certain distribution. General formula of the distance measure is given by the following formula:
d 2 ( xk ,υi ) = ( xk − υi ) Ai ( xk − υi ) ≥ 0 T
3
4
(3.6)
Fuzziness is a type of uncertainty (of imprecision) accepted in uncertainty theory [Zadeh, 1965, 1975a]. Various functions have been proposed to measure the degree of fuzziness measure. In fuzzy clustering algorithms, the overlapping constant, m, is used as the degree of fuzziness. In later chapters m will be used as a parameter to define uncertainty in proposed fuzzy functions approach, along with other measures. In some research such as Krishnapuram and Keller (1993), (3.4) is indexed in possibilistic approach to clustering.
3.2 Fuzzy Clustering Algorithms
55
In (3.6) the norm matrix Ai, i=1…c, is positive definite symmetric matrix. Other distance measures can also be used in fuzzy clustering algorithms. A short list of different distance measures are given in Table 3.1. FCM clustering algorithm uses Euclidean distance, therefore norm matrix, Ai, is equal to the identity matrix since input matrix is scaled to standard deviation 1 and mean equals 0, e.g., A=I. On the other hand, Gustafson and Kessel [1979] use Mahalanobis distance, in which case norm matrix of each cluster is equal to inverse of the covariance matrix of each cluster, e.g., Ai=Ci-1. Table 3.1 Distance Measures
Distance Measure Euclidean Distance
Function
Minkowski Distance
⎡ s ⎤ d p (a, b) = ⎢ ∑ | ai − bi | p ⎥ ⎣ i=1 ⎦
Maximum Distance
d∞ (a, b) = max | ai − bi |
1/ 2
⎡ nv ⎤ d2 ( a, b) = ⎢ ∑ ( ai − bi ) 2 ⎥ ⎣ i =1 ⎦
1/ p
,p>0
nv
i =1
Mahalanobis Distance
d A (a, b) = (a − b)T A (a − b)
From (3.3)-(3.6), one can imply that FCM clustering algorithm is a constraint optimization problem, which should be minimized in order to obtain optimum results. Therefore, FCM clustering algorithm can be written as a single optimization structure as follows:
min J ( X ;U ,V ) = ∑ i =1 ∑ k =1 ( μik ) d 2 ( xk ,υi ) A c
s.t.
n
m
0 ≤ μik ≤ 1, ∀i,k
∑ μ 0< ∑ c
i =1
ik
n
k =1
= 1, ∀k > 0
(3.7)
μik < n, ∀i > 0
Constraint optimization model in (3.7) [Bezdek, 1981a] can be solved using a well-known method in mathematics, namely Lagrange Multiplier method [Khuri, 2003], and the model is converted into an unconstraint optimization problem with one objective function. In order to get an equality constraint problem, primal constraint optimization problem is first converted into an equivalent unconstraint problem with the help of unspecified parameters known as Lagrange Multipliers, λ;
56
3 Improved Fuzzy Clustering
∑ i =1 ∑ k =1 ( μik ) c
max W(U,V) =
n
m
d 2 ( xk ,υi ) -λ A
(∑
c i =1
μik − 1
)
(3.8)
According to Lagrangian Method, the Lagrangian function must be minimized with respect to primal parameters and maximized with respect to dual parameters. According to the derivative of Lagrangian function in (3.8) with respect to the original model parameters, U and V should vanish. Hence, by taking the derivative of objective function in (3.8) with respect to cluster centers, V and membership values, U, optimum membership value calculation equation and clusters centers are formulized by:
μik(t )
⎛
n
2 ⎡ ⎤ c ⎛ d x ,υ ( t −1) ⎞ m−1 ( ) ⎢ ⎥ k i ⎟ ⎥ = ⎢∑ ⎜ ( − 1) t ⎜ ⎟ ⎢ j =1 ⎝ d ( xk ,υ j ) ⎠ ⎥ ⎣ ⎦
( )
υi(t ) = ⎜ ∑ μik(t ) ⎝ k =1
m
⎞ xk ⎟ ⎠
∑ ( μik(t ) ) n
m
−1
, ∀i = 1,..., c
(3.9)
(3.10)
k =1
In (3.9) υi(t-1) represent cluster center vector for cluster i obtained in (t-1)th iteration. Similarly, in (3.9) and (3.10), μik(t) denotes optimum membership values calculated at tth iteration. The proof of extracting the membership value calculation formula in (3.9) and the cluster center function in (3.10) can be found in Appendix B.1. The result from this operation shows that membership values and cluster centers are dependent on each other, so Bezdek [1981a] proposed an iterative algorithm to calculate membership values and cluster centers. Objective function Jt at each iteration, t, is measured by J(t) =
∑ ∑ (μ ) c
n
i =1
k =1
(t ) m ik
d 2 ( xk ,υi( t ) ) >0
(3.11)
FCM algorithm stops according to a termination criterion, e.g., either after certain number of iterations, or magnitude of separation of two nearest clusters is less than a pre-determined value (ε), etc. Iterative algorithm of FCM clustering algorithm is shown in ALGORITHM 3.1. The effect of the fuzziness value, m, can be analyzed by taking the limit of the membership value calculation equation in (3.9) at the boundaries as follows: −1
1 ⎡ ⎤ m−1 c ⎛ d 2 x ,υ ⎞ ( ⎢ ⎥ 1 k i ) ⎟ ⎥ = , ∀i, j = 1,..., c. lim μik ( x ) = lim ⎢∑ ⎜ 2 m→∞ m→∞ ⎜ ⎟ c ⎢ j =1 ⎝ d ( xk ,υ j ) ⎠ ⎥ ⎣ ⎦
(3.12)
3.2 Fuzzy Clustering Algorithms
57
ALGORITHM 3.1 Fuzzy C-Means Clustering Algorithm (FCM) Given data vectors, X={x1,..,xn}, number of clusters, c, degree of fuzziness, m, and termination constant, ε (maximum iteration number in this case). Initialize the partition matrix, U, randomly. Step 1: Find initial cluster centers using (3.10) using membership values of initial partition matrix as inputs. Step 2: Start iteration t=1…max-iteration value; Step 2.1. Calculate membership values of each input data object k in cluster i, μik(t ) , using the membership value calculation equation in (3.9), where xk are input data objects as vectors and υi( t −1) are cluster centers from (t-1)th iteration, Step 2.2. Calculate cluster center of each cluster i at iteration t, υi( t ) using the cluster center function in (3.10), where the inputs are the input data matrix, xk, and the membership values of iteration t, μik(t ) . Step 2.3. Stop if termination condition satisfied, e.g., | υi( t ) - υi( t −1) |≤ε. Otherwise go to step 1.
and under the assumption that no cluster centers are alike, we get;
⎧⎪1, if d 2 ( xk ,υi )
(3.13)
As the value of m is increased, μik will converge to 0, e.g., when m=6, any strong membership such as μik=0.85 will be decreased somewhere close to 1/c (see (3.12)). Since m parameter represents degree of overlap of clusters, as m gets larger, the fuzzier the results would be and overlapping will be wider. As m gets smaller, fuzzy clustering result will be more close to a crisp clustering model. m=1 is same as crisp clustering where there is no overlapping between clusters, and all the membership values are μik ∈ {0,1}. Earlier research [Turksen, 1999] indicates that, as a rule of thumb, m=2 should be used in system modeling analysis. In a more recent study [Ozkan and Turksen, 2007], the maximum and minimum values of m was mathematically proven to be m∈[1.4,2.6] based on Taylor expansion analysis of the membership value calculation function. Also in this work, the fuzziness interval, m∈[m-lower,mupper] will be investigated based on a new fuzzy modeling approach in uncertainty modeling chapter, in Chapter 5, entitled “Modeling uncertainty with Improved Fuzzy Functions”. FCM [Bezdek, 1981a] clustering algorithm can be regarded as the standard fuzzy clustering algorithm and many extensions of FCM clustering algorithm have been proposed for various different purposes. FCM clustering is also the core structure of the proposed improved clustering algorithm (IFC) of this work, to be explained in the following sub-sections. In the next sub-section, different classifications of fuzzy clustering algorithms are reviewed.
58
3 Improved Fuzzy Clustering
3.2.2 Classification of Objective Based Fuzzy Clustering Algorithms Objective function based fuzzy clustering algorithms other than FCM clustering algorithm can be categorized into three parts based on their purpose in system modeling as follows: 1. Algorithms that use adaptive distance measures [Gustafson and Kessel, 1979] or fuzzy maximum likelihood estimation methods [Gath and Geva, 1989], 2. Algorithms based on fuzzy linear variants and fuzzy c-elliptotypes [Bezdek et al., 1981b; Bezdek et al., 1981c], 3. Fuzzy c-regression algorithms using prototypes defined by regression functions [Hathaway and Bezdek, 1993]. Gustafson and Kessel [1979] proposed that for each cluster, different symmetric and positive semi definite matrix, A, should be used. Gath and Geva’s method [1989] is an extension of Gustafson and Kessel [1979] algorithm that also uses the size and density of the point-wise clusters. On the other hand, fuzzy c-varieties algorithm by Bezdek [1981a; 1981b, and 1981c] was developed for recognition of lines, planes and hyper-planes. Shell fuzzy cluster algorithms are also recently developed methods for recognizing circular, elliptical and parabolic shapes [Höppner et al, 1999]. Fuzzy c-regression model clustering algorithm, to be discussed in the following, falls into this category. Since the scope of the work only covers the third category of fuzzy clustering algorithms, we will leave out the definitions of the first two algorithms.
3.2.3 Fuzzy C-Regression Model (FCRM) Clustering Algorithm The objective of fuzzy c-regression model (FCRM) clustering algorithm [Hathaway and Bezdek, 1993], as in all clustering algorithms, is to classify objects into similar groups. FCRM clustering algorithm yields simultaneous estimates of parameters of c regression models, while fuzzy partitioning a given dataset. A prominent feature of this clustering algorithm that separates it from other pointwise clustering algorithms, e.g. FCM, is that, cluster prototypes are functions instead of geometrical objects. FCRM clustering algorithm can be used to separate linear patterns, where each pattern can be identified by a linear function. Hathaway and Bezdek [1993] displayed the domain of FCRM algorithm with a simpler example as shown in Figure 3.1. The artificial dataset used for this example consists of two regression models, e.g., linear functions. In [Hathaway and Bezdek, 1993] these models are called switching regression models and FCRM algorithm tries to identify these structures using fuzzy sets. Although FCRM clustering algorithms are based on standard FCM [Bezdek, 1981a] clustering algorithm, there are various differences between them. These differences can be briefly listed as follows:
3.2 Fuzzy Clustering Algorithms
•
59
In FCM [Bezdek, 1981a] clustering algorithm, clusters are hyper-sphere shaped, whereas in FCRM clustering algorithm [Hathaway and Bezdek, 1993], clusters are hyperplane-shaped. The representatives of clusters of FCM are cluster centers, υi, whereas the representatives of clusters in FCRM are hyper-planes, which are represented with nv-dimensional inputs and an output, e.g., for multiinput, single output model, by
•
yi = βi0 + βi1 x1 + ... + β inv xnv ,
(3.14)
where βi are the regression coefficients of each function i, i=1…c. •
FCM algorithm calculates cluster centers by averaging each data vector weighted with their membership values. FCRM calculates cluster representative functions by weighted least squares regression algorithm.
In order to demonstrate FCRM clustering algorithm, an artificial dataset in Figure 3.1 is composed using two different linear functions with random noise, ε, of mean=0 and standard-deviation=0.05, using MATLAB’s rand(x) function, where x∈[0,1] , i.e. f1(x)=rand(x)+ε, and f2(x)=ε+0.3. 0.7 f1(x) f2(x)
0.6 0.5
y
0.4 0.3 0.2 0.1 0
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
x
Fig. 3.1 Scatter Plot of artificial data with two different random functions. f1(x)=y+ random_noise, f2(x)=0.3+random_noise.
FCRM clustering algorithm yields simultaneous estimates of parameters of c regression models together with a fuzzy c partitioning of the data [Hathaway and Bezdek, 1993]. Each regression model is defined as; yk = fi (xk , βi)
(3.15)
60
3 Improved Fuzzy Clustering
In (3.17) xk=[x1,k,…,xnv,k]T∈ℜnv denotes kth data object and βi ∈ℜnv , i=1,…,c, are the parameters of functions fi and c represents total number of functions. Performance of captured functions is generally measured by Eik(βi) = (yk - fi (xk, βi))2
(3.16)
The objective function to minimize the total error of these approximated functions is calculated by
E (U,βi )= ∑ i =1 ∑ k =1 ( μik ) Eik ( βi ) c
n
m
(3.17)
where m ∈ (1,∞) is the “fuzzifier” exponent, which determines overlapping of the functions slightly different than FCM [Bezdek, 1981a] algorithm. In FCRM clustering algorithm as m approaches to ∞ each function will have almost the same parameters, i.e., coefficients (weights), β i, graphically speaking all the functions will be on top of each other. On the other hand, as m approaches 1, the functions will be as discrete from each other as possible. Therefore, choice of parameter m affects the performance of the FCRM clustering models as well. In FCRM modeling algorithm, membership values, μik, are interpreted as weights, i.e., coefficients of linear or polynomial regression functions. They represent to what extent values predicted by the model, fi (xk,βi), is close to yk. Therefore, based on membership value calculation equation of FCM in (3.10), membership value calculation equation of FCRM algorithm is re-formulated in [Hathaway and Bezdek, 1993] as follows: −1
1 ⎡ ⎤ m −1 c ⎛E ⎞ μik = ⎢ ∑ j =1 ⎜ ik E ⎟ ⎥ ,∀i, j = 1,...,c < n ⎢ ⎥ jk ⎠ ⎝ ⎣ ⎦
(3.18)
Minimization of the objective function in (3.17) yields an optimization model as shown in ALGORITHM 3.2. ALGORITHM 3.2 Fuzzy C-Roegression Models Clustering Algorithm (FCRM) Given the data vectors, X={x1,.., xn}, choose number of clusters, c, degree of fuzziness, m, and termination constant, ε, maximum iteration number, and the structure of regression models in (3.15). Initialize the partition matrix, U, randomly. Start iteration t=1,….max-iteration value; Step 1. Calculate the values of the model parameters βi that minimize the cost function in (3.17). Step 2. Update partition matrix, μik∈U, using (3.18). Terminate if |U(t)-U(t-1)|≤ε. Otherwise go to step 1.
3.2 Fuzzy Clustering Algorithms
61
Hathaway and Bezdek [1993] searches for the optimum parameters of the functions using weighted least-squares method. They use the membership degrees of the fuzzy partition matrix U as weights. In this specific situation, membership degrees, U input matrix, X, and output matrix, y are represented by
⎡ xiT,1 ⎤ ⎡ μi1 ⎡ y1 ⎤ ⎢ T ⎥ ⎢0 ⎢ ⎥ y x X i = ⎢ i ,2 ⎥ , y = ⎢ 2 ⎥ , U i = ⎢ ⎢ # ⎥ ⎢ # ⎢#⎥ ⎢ T ⎥ ⎢ ⎢ ⎥ ⎢⎣ 0 ⎢⎣ xi , n ⎥⎦ ⎣ yn ⎦
0
μi 2 # 0
" 0 ⎤ " 0 ⎥⎥ % # ⎥ ⎥ " μi , n ⎥⎦
Parameters,β i, of each function, fi, are calculated with weighted least squares algorithm: βi =[XTUiX]-1 XTUi y
(3.19)
In FCRM clustering method, linear functions are generally formulized to find hidden structures in a given dataset. Possible extensions of FCRM implement nonlinear functions to find hidden patterns. The new improved fuzzy clustering algorithm of this work, which utilizes FCRM type clustering algorithms, introduces additional non-linearity to the clustering algorithm by using non-linear algorithms to find functions in a novel way. For this purpose FCRM algorithm is one of the fundamental structures of the new clustering algorithm along with the standard FCM clustering algorithm. Next, we review different clustering algorithms, which combine regression and clustering methods in one structure.
3.2.4 Variations of Combined Fuzzy Clustering Algorithms In 1993, Sugeno and Yasukawa published a paper on a FL-based approach to qualitative modeling in which they proposed achieving structure identification by using a combination of FCM clustering algorithm [Bezdek, 1981a] and group method of data handling. In their approach, number of rules was captured by minimizing the variance in each output cluster and maximizing the variance between clusters. The main advantage of their method is that they were able to separate structure identification from parameter identification. These clusters constitute building blocks of fuzzy rule based approaches. Despite the popularity of FCM clustering algorithm, other variations of clustering algorithms are also used in literature to build fuzzy system models for different purposes. One of the prominent FCM clustering variation, fuzzy c-regression methods (FCRM), presented in the previous section, is commonly used to fuzzy partition a given dataset for local function approximation. In one of these studies, Chen et al. [1998] presents a new fuzzy clustering algorithm combining fuzzy c-functions clustering and alike fuzzy c-means clustering algorithm. Their fuzzy c-functions clustering based on linear fuzzy regression. In [Chen et al., 1998] Takagi-Sugeno-Kang (TSK) type system [Takagi and Sugeno, 1985; Sugeno and Kang, 1986] modeling structures are utilized. A nonlinear optimization algorithm is applied to identify the parameters of premises and consequences of each rule. In there, consequence and premise parameters are
62
3 Improved Fuzzy Clustering
optimized in turns with one fixed and the other optimized. In a sequence, they develop different criteria for optimization of model parameters of premise and consequents based on fuzzy c-partition space. Even though their algorithm cannot be considered as one united clustering algorithm, they still manage to combine fuzzy c-variants and c-means clustering algorithms in one system modeling and use them interchangeably to find parameters of premise and consequents of fuzzy rules, but again not in one optimization algorithm. On the other hand, Höppner and Klawonn [2003] combine FCM [Bezdek, 1981a] and FCRM [Hathaway and Bezdek, 1993] algorithms in one clustering schema, to build combined clustering structure. Their main goal was to update fuzzy clustering algorithm so that they can prevent the effect of harmonics by modifying the objective function. Their aim was to eliminate the counterintuitive membership values such that membership value of a linguistic term “young”, which is high for “17 years”, should not be higher for “23 years” than for “21 years”. They not only deal with point-wise clustering algorithms such as fuzzy c-means clustering (FCM) algorithm [Bezdek, 1981a], they also deal with fuzzy c-regression model clustering algorithms (FCRM) [Hathaway and Bezdek, 1993]. They modified the objective function of FCM clustering algorithm to yield the following membership value calculation equation based on their heuristic approaches:
( (
⎡ d ik2 − min dik2 − η c i = 1...c ⎢ μik = ⎢ ∑ j =1 2 d jk − min d ik2 − η ⎢⎣ i =1...c
) )
−1
⎤ ⎥ ,0 < η ⎥ ⎥⎦
(3.20)
η >0 is a user defined constant. The objective in [Höppner and Klawonn, 2003] was to find membership values that can eliminate harmonics by pushing larger membership values towards “1” and smaller ones towards “0” based on distances T of objects. In [Höppner and Klawonn, 2003], each function, ˆyi = βˆ i xi , is
interpreted as a rule in a T-S model. Combining FCM and FCRM algorithms in one clustering schema (since both algorithms are objective based), Höppner and Klawonn [2003] introduced a new combined distance function, which is the combination of both methods as follows: 2 d ik2 (( xk , yk ), (vi , βˆi )) = xk − υi ( x) + ( yk − βˆi T xˆ k ) 2
FCM distance
FCRM distance
(3.21)
In (3.23), (xk, yk) is a given input-output data sample, xk ∈ X, yk∈Y, k=1,..,n, n is the total number of training data vectors, d2 is distance function, υi is a cluster prototype of cluster i, which is the same as FCM clustering, cluster center function, i=1,…,c, c is the number of clusters. xˆ i represents a user defined polynomial, for instance a two dimensional polynomial can be formed with the following vector, xˆ (x1, x2)=(1, x1, x2, x1x2 , x12, x22)
(3.22)
3.2 Fuzzy Clustering Algorithms
63
and the coefficients of the polynomial are represented with βˆi ’s for each cluster i.
Hence, the first term of distance function in (3.21) is FCM distance measure and the second term is FCRM distance measure, which is equal to the error of estimated functions in (3.16). Based on the distance function in (3.21), they try to optimize partition matrix in their new fuzzy clustering algorithm, which uses the membership value calculation equation in (3.20). The coefficients βˆi are obtained in the same way as the cluster centers of FCM clustering are obtained. Therefore, the prototype update function of FCM clustering in (3.10) is replaced with;
⎛
n
( )
βˆi = ⎜ ∑ μik ⎝ k =1
m
⎞ ( yk xˆk ) ⎟ ⎠
∑ ( μik ) n
k =1
m
( xˆk xˆkT ), ∀i = 1,..., c
(3.23)
Their combined clustering algorithm is sketched in ALGORITHM 3.3. ALGORITHM 3.3 Fuzzy Model Algorithm of Höppner and Klawonn (2003) Step 1: Choose number of clusters, c, termination threshold, ε>0, η>0 , Step 2: Initialize the cluster prototypes, υi Step 3: Update membership matrix using (3.20) and distances (3.21) Step 4: Update prototypes using FCM’s prototype function in (3.10), and the coefficients using (3.23) Step 5: Iterate until |U(t)- U(t-1) |≤ε.
It should be noted that, the membership value calculation equation in (3.20) is not the result of a mathematical transformation obtained from combining two clustering algorithms, FCM and FCRM. They do not specifically imply how they chose polynomials to approximate local non-linear functions. Other variations of combined fuzzy clustering methods are also observed in the literature. For instance, in [Wang et al., 2002], FCRM algorithm is replaced with a gravity-based clustering algorithm, which is based on Newton’s gravity law. Their aim was to capture shell/curve like structures in a given data, which is suitable for image analysis. They combine gravity based clustering and fuzzy clustering algorithm to form an integrated clustering algorithm. Leski [2004]’s combined approach to fuzzy clustering algorithm is based on a local non-linear function estimation of local fuzzy models using FCRM [Hathaway and Bezdek, 1993] algorithm. Leski [2004] introduced an e-insensitive fuzzy c-regression algorithm, by converting optimization algorithm of FCRM clustering algorithm into a similar support vector regression optimization algorithm to capture the outliers. Using fuzzy partition space, Leski [2004] formed TSK type fuzzy system models. One drawback of this approach is that it is computationally complex as opposed to the rest of the fuzzy clustering variations mentioned earlier.
64
3 Improved Fuzzy Clustering
A major disadvantage of most of these combined fuzzy clustering approaches reviewed above is that; the algorithms are only defined for training datasets, where output variable is known. Learning algorithms in these earlier approaches utilize output values of training samples during training methods. Similarly, the distance measure and membership value calculation equations use output values during clustering algorithm. In most investigations, it would be preferred if membership value calculation equations of these fuzzy clustering methods would be used to find membership values of validation or testing data objects, whose output variable are not known. Almost none of these approaches explain explicitly how they calculate membership values of verification or testing samples using their proposed membership value calculation equations, nor do they define other membership value calculation equation. In this work, in chapter 4, a novel training and testing algorithm are explicitly defined to explain how to handle such problems. A probabilistic case-based inference is proposed for reasoning. Before moving on to explain the details of the proposed fuzzy clustering approach, it should be pointed out that, fuzzy clustering algorithms and their variations are designed for different purposes in system modeling. For instance, some fuzzy clustering methods are designed to capture noise in a given dataset, e.g., [Dave, 1991]; some try to eliminate harmonics [Kilic, 2002], others search for relationships through linear or non-linear functions, e.g., [Leski, 2004]. In this sense, the new improved fuzzy clustering algorithm has a novel goal. FCM and FCRM algorithms and their variations are the basis of the proposed improved fuzzy clustering algorithm of this work. Motivation and the underlying background of the new fuzzy clustering algorithm are presented in the next section.
3.3 Improved Fuzzy Clustering Algorithm (IFC) 3.3.1 Motivation Since, multi-input, single-output (MISO) models are the interest of this work; a clustering algorithm is designed to approximate local models to explain relationship between inputs and outputs. Most of the times, earlier clustering algorithms are designed to find local groupings of data vectors by leaving out any linear or non-linear relationships between them. However, data objects could be grouped by considering not only the local models of input-output data (clusters) but also relationships between them. Here a new clustering algorithm is proposed to find local fuzzy partitions to estimate local fuzzy models with fuzzy functions [Celikyilmaz and Turksen, 2007b] based on combined clustering structures such as in (3.21). “Fuzzy Functions” approach is initially proposed by Turksen in 2005 [Turksen, 2008]; the first implementation was published in [Celikyilmaz, 2005]. Later, Turksen and Celikyilmaz [Turksen and Celikyilmaz, 2006] published the idea as an alternate reasoning method to Fuzzy Rule Base structures. In simpler terms, instead of using fuzzy rule bases, each rule is converted into a “Fuzzy Function” as such ŷi=fi(Φ(μi,x)), i=1…c, to represent them with a linear or a non-linear function.
3.3 Improved Fuzzy Clustering Algorithm (IFC)
65
Feature space Φ(⋅) is similar to mapping original input space, x, onto a user defined higher dimensional space. The feature space dimension is determined by the user and it is formed by original input variables, membership values, μi, and or their mathematical transformations, for instance power transformations of membership values. In a sense, membership values are used as additional dimensions in approximating hyper-surfaces for each cluster identified by any type of fuzzy clustering algorithm. In the following chapters, we will analyze fuzzy system modeling with “Fuzzy Functions” approach in more detail. Here, only a brief introduction is given in order to explain the motivation of the proposed improved fuzzy clustering algorithm. Before introducing the details of the new clustering algorithm, its motivation are discussed in connection to earlier improved and combined fuzzy clustering algorithms such as combined clustering methods [Höppner and Klawonn, 2003]. Foundations of the Improved Fuzzy Clustering (IFC) approach of this work were introduced by Höppner & Klawonn [2000, 2003], Chen et al. [1998] and Menard [2001]. The structure of these earlier versions of fuzzy clustering algorithms and the IFC algorithm are similar in the sense that they share a similar objective function, which combines standard fuzzy clustering and fuzzy c-regression methods. However, there are many structural differences between the IFC of this work and earlier versions of FCM variations, viz. FCRM or improved clustering algorithms described in [Höppner & Klawonn, 2003; Chen et al., 1998; Menard, 2001]. These differences can be listed as follows (these earlier clustering types will be denoted as “earlier variations”):
•
•
•
•
Earlier variations use polynomials to estimate parameters of estimated functions. Novel IFC can use any type of function including simple functions such as linear regression, or non-linear functions such as support vector regression [Gunn, 1998]. Earlier versions and the IFC have different representations of membership values during structure and parameter identification. Earlier FCRM algorithms use membership values as weights in weighted regression algorithms, whereas the IFC method uses membership values as new predictors of functions in the sense of [Turksen, 2008] of each substructure (pattern) in the data. This is the unique property of “Fuzzy Function” methodologies proposed by Turksen in 2005 [2008]. An earlier combined fuzzy clustering algorithm [Höppner and Klawonn, 2003] as reviewed above, uses a slightly different version of cluster prototype function as in (3.23) to update estimated parameters of the calculated functions. Whereas new IFC updates coefficients of Fuzzy Functions use any type of function estimation method, e.g., multiple linear regression, support vector regression or even neural networks. Membership values of data points calculated with earlier membership value calculation equations produce higher values such as close to 1 when corresponding data objects can explain local input-output relationships. Similarly they estimate lower membership values if corresponding data vectors are outliers of that particular local structure. Whereas in IFC,
66
3 Improved Fuzzy Clustering
membership values and their transformations are considered as additional candidate input variables, not just as an indicator of data points. Structure of the IFC algorithm forces membership values to be better predictors of local models. • Earlier combined fuzzy clustering versions build Takagi and Sugeno [1985] based fuzzy system models, whereas IFC is proposed to be used in novel Fuzzy Functions systems [Celikyilmaz and Turksen, 2007b-k], to be presented in Chapter 4. • Earlier fuzzy clustering algorithms require observed output variable, yk, to estimate the error of each function of each local structure (cluster) identified, see (3.16). Given some data points, the algorithm identifies in which part the input space they fall and use the local model fi(x) assigned to it to predict the output. The error between the observed output and the estimated output from each local model is used as an input to the membership value calculation equation in (3.18). In short, during training, one needs to know what the output value of a particular data point is, before it can actually estimate its membership values. This could be problematic for real life datasets where the output values are generally not known. The improved fuzzy clustering algorithm (IFC) also uses a similar structure, where membership value calculation equations require output as inputs to calculate membership values of each data vector in each of these structures. But, in this work, a new inference engine that can calculate an approximate output value using case-based methods is proposed in order to calculate the membership values of vectors with unknown output values. Earlier system modeling approaches that utilize FCM variations stated above [Hathaway and Bezdek, 1993; Leski, 2004; Höppner and Klawonn, 2002] do not explain how membership values of testing data vectors are calculated. In this sense, IFC clustering is a unique approach and can be applied to any type of data structure, even when there are missing output values in the dataset. In this work, extending distance function in (3.21), a new Improved Fuzzy Clustering (IFC) is proposed. It should be noted that, new IFC is implemented in Fuzzy Functions (Fuzzy functions) systems [Celikyilmaz and Turksen, 2007b], where membership values and their user defined transformations are used as additional predictors in approximating the local input-output relationships. Earlier similar research [Höppner and Klawonn, 2003; Chen et al.,1998] use FRB structures, e.g., T-S models, where system modeling and membership values are used as weights of each local model. It should be emphasized again for the sake of the clarity and novelty of the IFC that, earlier variations of improved clustering methods, e.g., [Höppner and Klawonn, 2003; Chen et al., 1998], etc, approximate a polynomial function for each local partition using original input variables to estimate output variable and append this error term as a second term in the distance function of the clustering, as shown in equation (3.21). The IFC clustering algorithm is designed to find membership values to be used to model the system behavior. At each step of the
3.3 Improved Fuzzy Clustering Algorithm (IFC)
67
IFC optimization, a special regression function is estimated for each cluster. In approximating these functions, only the membership values and their user-defined transformations are used as input variables. These regression functions are called “Interim Fuzzy Functions” and are denoted as hi (τ i ) , where τi is the matrix of ith cluster, consisting of the membership values and their user defined transformations. One could use a simple model to estimate the parameters of these functions using linear regression functions such as least squares regression (LSE) or build non-linear models such as support vector regression [Gunn, 1998] or neural networks [Kosko, 1992]. The residual of the interim fuzzy functions, i.e., (yk-[ŷk=hik(τi)])2, is used as additional similarity information. Hence, they are added to the distance function as the second term. They are used as an additional term in the objective function of the the IFC algorithm. Since objective function based fuzzy clustering algorithms include a distance function as a similarity measure, the distance function should affect the behavior of the membership value calculation equation. In [Höppner and Klawonn, 2003], such an effect is not reflected to the membership value calculation equation in (3.20), instead the membership value calculation equation serves their particular purpose of finding more crisp membership values to eliminate harmonics. In this work, a new membership value calculation equation is introduced, which is structured as a result of the modified distance function. On the other hand, earlier fuzzy c-regression models and combined fuzzy clustering algorithms use membership values of each cluster as weights in local regression models. Since the unique aim of the IFC algorithm is to find membership values that would be better predictor arguments of Fuzzy Functions of each local partition (cluster), interim fuzzy functions of the IFC algorithm only uses calculated membership values and their transformations as input variables. Therefore, in the new IFC clustering optimization approach, we do not include the original scalar inputs while shaping the membership values. We just aim at finding the best membership values that can explain the output. In short, the new IFC clustering introduces a new membership value calculation equation as a result of the effect of the addition of the second term to the distance function, consequently to the objective function of the IFC algorithm. Whereas the earlier approaches use original input variables to find local linear models. Consequently, in the training algorithm of the Fuzzy Functions approach, after hidden structures are identified with IFC, we approximate regression functions for each of these fuzzy partitions (clusters) identified by IFC, using the membership values from IFC, their transformations and the original scalar input variables. We use linear regression methods, e.g., least squares estimation (LSE) as proposed by Turksen [2008] or non-linear regression methods e.g., support vector machines for regression (SVR) [Gunn, 1998] as proposed by Celikyilmaz [2005]. We call these local regression functions, the “Fuzzy Functions”. It is hypothesized that the modeling error of the proposed Fuzzy Function methods will be lower when the new IFC of this work is used instead of standard FCM [Bezdek, 1981a]. It should be emphasized that, in the earlier fuzzy system modeling algorithms, e.g., Zadeh fuzzy rule base structures [1975a], it is assumed that expert
68
3 Improved Fuzzy Clustering
information is available and linguistic description (the degree of memberships) are determined subjectively. This may cause some discrepancies since expert knowledge is subjective. Later, various fuzzy models have been developed implementing clustering algorithms into input-output data to find membership values. Among these studies are [Sugeno and Yasukawa, 1993; Delgado et al., 1997; Emami and Turksen, 1998] and others. In [Delgado et al., 1997] different approaches is presented for identification of fuzzy models including fuzzy clustering of the input-output (XY) domain, viz. Z={XY} input data space. They use FCM clustering to generate c fuzzy clusters in Z domain, i.e., Z = [X,Y] with centers denoted as υiZ=(υiX, υiY) for i=1,…,c. Hence, membership value calculation equation of a fuzzy relation associated with the ith cluster is defined as: −1
2 ⎛ ⎞ Z ⎜ c ⎛ d ( ( zk ),(υi ) ) ⎞ m −1 ⎟ ( μik ) = ⎜ ∑ ⎜⎜ d ( z ),(υ Z ) ⎟⎟ ⎟ , m > 1 5 j )⎠ ⎜ j =1 ⎝ ( k ⎟ ⎝ ⎠
(3.24)
Hence Delgado et al. [1997] identify local relationships between inputs and outputs around centroids, which are captured by the above fuzzy relation on XY domain and then they define membership value calculation equation of individual domains, e.g., X and Y. Identifying fuzzy cluster structure as two separate components of inputs and outputs does not imply that the membership value calculation equations are projections of the clusters, rather they are induced in X and Y spaces by the fuzzy clusters. In this way, c fuzzy sets are obtained in X and Y spaces denoted by υiX and υiY with membership value calculation equations defined as: −1
2 −1 2 ⎛ ⎞ ⎛ ⎞ X m −1 Y − 1 m ⎛ ⎞ c d x , υ ⎛ ⎞ ( k i ) ⎟ ⎟ , Y ⎜ c d ( yk ,υ i ) ⎟ ⎜ μikX = ⎜ ∑ ⎜ ⎟ ⎟ , m > 1 (3.25) ⎟ μ ik = ⎜ ∑ ⎜ X Y ⎜ j =1 ⎜⎝ d ( xk ,υ j ) ⎟⎠ ⎟ ⎜ j =1 ⎜⎝ d ( y k ,υ j ) ⎟⎠ ⎟ ⎝ ⎠ ⎝ ⎠
The assumption here is that the input variables are not independent. Instead of defining separate membership functions for each individual input variable, one multi-dimensional interactive membership function is formulated to represent the whole antecedent part of a fuzzy rule as defined in [Delgado, et al., 1997] by joint (interactive) membership values. This indicates that non-interactivity6 assumption between each component of the input space is not assumed as earlier fuzzy rule base approaches. Some examples of the applications of clustering input-output domain as well as the output domain can be found in [Emami and Turksen, 1998; Kemal, 2002; Uncu, 2003]. The new clustering algorithm is also based on the input-output clustering of the given system domain and also assumes interactivity between input variables. 5 6
t
Indicates any iteration. Non-interactivity assumption indicates that the antecedent part input variables are assumed to independent from each other and hence there would be no interaction what so ever between their antecedent fuzzy sets.
3.3 Improved Fuzzy Clustering Algorithm (IFC)
69
The IFC algorithm finds membership values which are candidate inputs for each local fuzzy model. These fuzzy models are regression functions, where output variable, y∈ℜ has a continuous domain. In this work, we also extended IFC method for classification domains, (IFC-C) where each object in the dataset belongs to one of the predefined classes. In chapter 4, a new fuzzy classifier design is introduced, which uses IFC-C clustering for fuzzy classifier models to solve classification problems. In there, the error i.e., (yk-ŷik)= (yk-h(τi)), is replaced with error between the actual binary output and the posterior probability, pˆ ik , of each object calculated by a chosen classification function, i.e., (yk- pˆ ik ). Next, the framework of the new improved fuzzy clustering (IFC) algorithm is presented for regression and classification problem domains.
3.3.2 Improved Fuzzy Clustering Algorithm for Regression Models ( IFC ) Standard Fuzzy c-Means (FCM) algorithm [Bezdek, 1981a] is used in fuzzy system models to find the membership values, which are assumed to represent optimum partitions of the given dataset. In Fuzzy Functions approaches these membership values are used as additional input variables to predict the parameters of regression models for each cluster. In this work, we propose a new fuzzy clustering method by modifying standard FCM algorithm. The new IFC is proposed to find membership values, which can improve the performance of the local models represented with fuzzy functions. The optimization approach of the new improved fuzzy clustering algorithm not only searches for the best partition of the data, but also aims at increasing the predictive power of the membership values to model the output variable with Fuzzy Functions. Hence, we call the new fuzzy clustering algorithm “Improved Fuzzy Clustering (IFC)” [Celikyilmaz, Turksen, 2007b]. For the given multi-input and single-output system, let the input matrix be represented as xy={(x1,y1),…,(xn,yn)}, where xk={x1…xnv} is nv dimensional kth data vector, k=1…n, n is the total number of data vectors and each yk is their corresponding output value. First, we introduce a new objective function, which carries out two purposes: (i) to find a good representation of the partition matrix; (ii) to find membership values to minimize the error of the Fuzzy Function models. To optimize the membership values, in the new IFC, we append the error we obtained from the regression functions to the objective function as follows:
J mIFC = ∑∑ ( μikimp ) d ik2 + ∑∑ ( μikimp ) c
n
m
i =1 k =1
In (3.26)
μikimp
c
n
i =1 k =1
m
( yk − hi (τ ik , wˆ i ) )
2
(3.26)
is the improved membership value of the kth input vector in ith
cluster, i=1…c, and m is the degree of fuzziness parameter which determines the overlapping of the clusters. Objective function to be minimized, JmIFC in (3.26), comprises of two separate terms. The first term is same as the standard FCM
70
3 Improved Fuzzy Clustering
algorithm. This term controls the precision of each input-output data vector to its cluster, d2=||xkyk-υi(xy)||2 and vanishes when each data sample is equal to the cluster center value. The second term is total squared error of Interim Fuzzy Functions hi(τi,ŵi) of its cluster, where τi is called the interim matrix and ŵi is the coefficient vector of cluster i. This term measures squared error of approximated user defined functions in the sense of which value transformations to be included in Fuzzy Functions that are built during the optimization of IFC algorithm. The only input variables are the membership values and/or their possible transformations, i.e., original scalar inputs are omitted. Therefore, input matrix, τ i ( μiimp ) , which is used to estimate these Interim Fuzzy Functions, hi(τi,ŵi), i=1,…c, utilize the membership values from the previous iteration step. Let the input matrix of ith cluster be composed of two-dimensional input vectors of the membership values and their log-odds transformations, τi=[μi log((1-μiimp)/μiimp)]. The set of planes in ℜ2 of each ith cluster is defined as hi=ŵ0i+ŵ1iμiimp+ŵ2ilog((1-μiimp)/μiimp), or hi=τiTŵi , where ŵiT = [ŵ0i ŵ1i ŵ2i] are the coefficients of the Interim Fuzzy Functions. A particular set of fuzzy functions in ℜ2 is defined by
yˆ i = hi ( τ i , wˆ i ) = wˆ 0 i + wˆ 1i μ iimp + wˆ 2 i log(
1 − μ imp i μ imp i
2
) = wˆ 0 i + ∑ wˆ jiτ ji
(3.27)
j =1
ŷi is the estimated output value at tth iteration in ith cluster, τi is the input matrix created by the improved membership values, μiimp, of the ith cluster at tth iteration, along with their log-odd transformations (original scalar input variables, x, are not included) and ŵi’s are estimated parameters of the functions using a linear regression method, e.g., least squares regression method. Log-odds are a user defined transformation of the membership values, which is usually used in fuzzy functions because mostly the distribution of the membership values is Gaussian (bell shaped) in fuzzy clustering algorithms. The analysis of the distribution of the membership values are discussed in the next section. It should be emphasized that, in (3.27) the only inputs are the membership values and their user defined transformations. We want to improve the predictive power of membership values to be used in system modeling as additional inputs. Hence, each tth iteration of IFC optimization tries to minimize the error between actual and estimated output values. Excluding original input variables, we are able to measure and improve the individual effect of the membership values on model performance. The distance function of the IFC algorithm is denoted by
d ikIFC = zk − υi ( z ) + ( yk − h(τ ik , wˆ i ) ) 2
2
(3.28)
where zk∈Z={xk,yk}∈XY ⊆ℜ(nv+1) indicate the input-output vector, vi(z) is the ith cluster center. For each cluster one interim fuzzy function, hi(τi,ŵi), i=1,…,c, is approximated using membership values from the previous iteration as input variables. The second term vanishes when the estimated interim fuzzy functions of each cluster, hi(τi,ŵi), can perfectly explain the observed output variable. The trade off between the precision and squared error enables to define the optimal IFC model.
3.3 Improved Fuzzy Clustering Algorithm (IFC)
71
The solution to minimize the new objective function, JmIFC, can be found by taking the dual of the model by using the Lagrange transformation of the objective function and converting into a maximization problem. This is done by introducing Lagrange Multiplier, λ, with respect to the constraint ∑ci=1 μikimp = 1. We re-write the objective function as follows:
L = ∑∑ ( μikimp ) d ik2 + ∑∑ ( μikimp ) c
m
n
m
i =1 k =1
c
n
i =1 k =1
m
⎛
⎞
⎛ ⎞ 2 ( yk − hi (τ ik , wˆ i ) ) -λ ⎜ ⎜ ∑ ( μikimp ) ⎟ − 1⎟ c
⎝ ⎝ i =1
⎠
⎠
(3.29)
The solution to the maximization of the convex model is found by taking the derivative of the Lagrange function with respect to the unknown parameters, i.e., μikimp, improved membership values and the cluster centers, υi. The input variables of regression models, hi(τi), to be used to approximate output variable are the membership values from the (t-1)th iteration. Therefore the second term, SEik(t-1)= (yk-hi(τik(t-1)))2, is a known variable at the tth iteration. Hence, by taking the derivative of the objective function in (3.29) with respect to the cluster center and the membership values, the optimum membership values of the proposed clustering algorithm are formulated as:
(μ )
imp ( t ) ik 1< i , j ≤ c 1≤ k ≤ n
1/( m −1) ⎛ ⎞ ⎛ ( d (t −1) )2 + ( y − h (τ (t −1) , wˆ ) )2 ⎞ c ik k i ik i ⎜ ⎟ ⎜ ⎟ = ⎜ ∑ j =1 ⎟ ⎜ d (t −1) 2 + y − h (τ (t −1) , wˆ ) 2 ⎟ ( ) ( ⎜ ⎟ jk k j jk j ) ⎠ ⎝ ⎝ ⎠
−1
(3.30)
Since the second term of the objective function JmIFC in (3.26) does not include the cluster center term, when taking the derivative of the objective function with respect to the cluster center, vi(xy), the second term will vanish. Then the objective function will be as same as the objective function of standard FCM algorithm. Therefore, the cluster center function of standard FCM remains unchanged for IFC and is given by
(
(t ) ⎛ n ∀ vi( t ) = ⎜ ∑ ( μikimp ) 1≤ i ≤ c ⎝ k =1
)
m
⎞ xk ⎟ ⎠
∑ (( μ ) n
k =1
)
m imp ( t ) ik
(3.31)
Extraction of membership value calculation equation in (3.30) using Lagrange Transformation is given in Appendix B.2. The membership value calculation equation in (3.30) uses cluster centers from the previous iteration, (t-1), and the cluster center equation in (3.31) uses the membership values from the current iteration. Hence, similar to the FCM algorithm, an iterative algorithm for the IFC algorithm is used to find the optimum membership values. The algorithm terminates according to some termination criteria, e.g., when total number of iterations exceeds a user defined maximum number of iterations or magnitude of the separation between two consecutive objective function values is below a threshold value, etc. Generalized framework of the IFC algorithm is shown in ALGORITHM 3.4.
72
3 Improved Fuzzy Clustering
Let fuzzy partition of each iteration t, t=1,…,max-iteration, be denoted by:
U (t )
(
)
(
)
⎛ μ imp ⎜ 1,1 =⎜ # ⎜ ⎜⎜ μ imp ⎝ 1,n
(t )
…
(μ )
% (t )
"
⎞ ⎟ ⎟ # ⎟ imp ( t ) ⎟ ⎟ μ c ,n ⎠ imp ( t ) c ,n
(
)
ALGORITHM 3.4 Optimization with the Improved Fuzzy Clustering Algorithm (IFC) for Regression Models Given the training dataset, Z={(x1,y1),…, (xn,yn))}. Set m>1.1, c>1 and a termination constant, ε>0, maximum number of iterations (max-iter), specify the structure of regression models such as in (3.27) for each cluster i, i=1,…,c, k=1,…,n, to create the interim input matrix, τi. Using FCM clustering algorithm, initialize the partition matrix, U0. Then, for each iteration, t=1,…max-iter; (1) Populate c number of input matrices, τi (t-1), one for each cluster, using membership values (U(t-1)) from the (t-1)th iteration, and their selected user defined transformations. (2) Approximate c number of interim fuzzy functions such as in (3.27), h(τ ik
( t -1)
).
(3) Update membership values for iteration t using (3.30). (4) Calculate cluster centers for iteration t using (3.31). (5) If (obj(t) - obj(t-1))<ε, then terminate, else return to step 1.
IFC optimization starts with an initial partition matrix, U0, and initial cluster centers, υ0. One may use a crisp clustering such as k-means or a fuzzy clustering such as fuzzy k-means, or FCM clustering, etc., to find the initial partition matrix, U0, and cluster centers, υ0. One should take into account the trade off between choosing a fuzzy clustering such as FCM clustering and discrete clustering methods like K-means. U0 and υ0 are required inputs at the start of the IFC since new membership value calculation equation in (3.30) requires error terms of the regression models, (yk- hi(τi,ŵi))2. The model output ŷik= hi(τi,ŵi) for each cluster i is estimated by using only membership values and their mathematical transformations as input variables as in (3.27). The IFC optimization method searches for the optimum membership values, which are to be used later as additional predictors to estimate the parameters of the Fuzzy Functions of the given system model. In steps 2 of ALGORITHM 3.4, membership values U t-1 that are calculated at (t-1)th iteration are used as input variables at tth iteration to find parameters of regression functions, ŵi, for each cluster. Any function approximator can be used to identify the parameters. In the experiments we use least squares estimation (LSE) to identify linear functions and support vector regression (SVR) [Gunn, 1998] to approximate non-linear functions [Celikyilmaz, 2005; Celikyilmaz and Turksen, 2007b]. Since the aim of IFC is to find membership values which are good predictors of the output, the only
3.3 Improved Fuzzy Clustering Algorithm (IFC)
73
input variables are the membership values and their user defined transformations when finding interim fuzzy functions of each cluster. The optimization algorithm tries to minimize the error of the regression models, hi(τik(t-1),ŵi). One should choose appropriate membership value transformations to approximate the output. The effects of improved membership values, as predictors of the Fuzzy Functions, is investigated in the following sub-section, and entitled justification of the membership values of IFC algorithm. The output of the IFC algorithm for a given m and c value is: ¾
parameters of the interim fuzzy functions of each cluster, ŵi, i=1…c, that are captured from the last iteration step, ¾ improved membership value matrix, U(z), and the cluster centers υ(z), ¾ the interim input matrix structure, τi, composed of membership values and their transformations for each cluster i. The fuzzy function systems utilize membership values from IFC as additional predictors. Hence, the new IFC algorithm is methodically designed to improve the performance of the membership values to be used as additional predictors of Fuzzy Functions systems.
3.3.3 Improved Fuzzy Clustering Algorithm for Classification Models (IFC-C) In this section, an extension of the improved fuzzy clustering (IFC) algorithm to be implemented into classification models will be presented. In the previous section novel IFC algorithm is introduced to be implemented into fuzzy system models with Fuzzy Functions to solve regression problems. To adapt the IFC algorithm for classification problems (IFC-C), one need to change the way the decision function, namely Fuzzy Function parameters are calculated. It should be noted that, for regression problems, Interim Fuzzy Functions, h(τ i( t ) , wˆ i ) , of each cluster (at each step of the IFC iteration), tries to model a continuous output variable. However, for classification problems, the output variable is dichotomous, e.g., y∈{0,1}, y∈{-1,+1}, or ordinal/discrete variable, y∈{0,1,2,..}. One needs to implement a classifier function, e.g., logistic regression [Allison, 2001], support vector machines for classification [Gunn, 1998], or neural networks for classification [Kosko, 1992], etc., in order to assign class labels to each data point. In this work we only dealt with the classification problems, where the output variable is dichotomous, e.g., y∈{0,1}, y∈{-1,+1}. The type of the classification method to choose is usually a tread-off between complexity (non-linearity) and generalization capacity of the classification. The second term of IFC objective function in (3.26) is the squared error calculated between an output of a form of interim fuzzy function and actual output of each data point. Hence, one first estimates output values, ŷik, of each data point k
74
3 Improved Fuzzy Clustering
in cluster i using interim fuzzy functions of ith cluster. Then the error between ŷik and observed output values, i.e., SEik=(yk-ŷik)2, is measured. For classification functions, the output is a binary variable which only takes values 0 or 1. One uses a classification method, i.e., logistic regression (LR), support vector machines for classification (SVC) or neural networks for classification (NNC), to calculate a decision function, h(τ i ) , so that the sign of the decision function, sign(hk(τik)), is the predicted output of object k in cluster i. Instead of predicting the label, sign(hk(τik)), which is a binary outcome, one can calculate posterior probability of the output, pˆ ik (yk=1|)τik) ∈[0,1], of each data point in each cluster based on the probability of the output to be (yk=1). Hence, estimated probability indicates how likely the output of the given observation would be 1. It is a scalar variable and one could form a performance measure between the actual normalized output values of each object and estimated posterior probability, as such (yk - pˆ ik )2. Then, we append the error of these classification functions onto the objective function of the standard FCM clustering to formulize the new hybrid structure of IFC-C as follows:
J mIFC −C = ∑∑ ( μikimp ) dik2 + ∑∑ ( μikimp ) ( yk − pˆ ik ( yk = 1| τ ik , wˆ i ) ) (3.32) i =1 k =1 i =1 k =1
SE of Fuzzy c
n
m
FCM
c
n
m
2
Classifier Function
The objective function to be minimized in (3.32), JmIFC-C , also has a dual structure. The first term is the same as the objective function of the standard FCM clustering. The second term is the total squared error of the “Interim Fuzzy Classification Function” h(τik,ŵi) of each cluster, i, where τi is the corresponding cluster’s input dataset, i=1,..,c, k=1,..,n, and ,ŵi are the interim fuzzy classification function parameters. This term measures the squared deviation of actual class labels, e.g., yi∈{0,1}, from estimated posterior probabilities, pˆ ik (yk=1⎢h(τi,ŵi)). For any cluster i, the second term takes the form:
⎛ ⎡ y1 ⎤ ⎡ pˆ i1 ( y1 = 1) ⎤ ⎞ ⎜⎢ ⎥ ⎢ ⎥⎟ SEi = ⎜ # − ⎢ # ⎥⎟ ⎢ ⎥ ⎜ ⎢ y ⎥ ⎢ pˆ ( y = 1) ⎥ ⎟ ⎦⎠ ⎝ ⎣ n ⎦ ⎣ i ,n n
2
IFC-C algorithm tries to minimize the error of classifier functions using membership values to estimate pˆ ik (yk=1)=e, which is close to e≈1. Thus, during each iteration step, the membership values are reshaped to predict the correct class labels with Interim Fuzzy Classifier Functions, as well as they represent the optimum fuzzy partition of a given dataset. Excluding original input variables from function estimation, we are able to measure and improve the individual effect of the membership values on the performance of each classifier model. If the estimated function can separate the two classes with the given inputs, then the error will be less and the algorithm will converge faster. Since we want to find membership values which can improve classification accuracy, during IFC-C we
3.3 Improved Fuzzy Clustering Algorithm (IFC)
75
only use corresponding cluster’s membership values and their transformations as the only input variables to estimate classifier functions, h(τi,ŵi). The effect of finding better classifiers by shaping membership values can be explained with an example. During tth iteration of IFC-C optimization, let τik and τik’ represent two vectors, k and k′, randomly selected from the dataset τi, the actual output values of which are y(τik)=1 and y(τik’)=0 respectively. An Interim Fuzzy Classifier Function, h(τi), estimates posterior probability equal to pˆ ik =0.90 for τik. Squared error for this kth vector would be, SEik=(1-0.90)2=0.01. This indicates that interim fuzzy classifier function, h(τik), predicts that output value of kth input vector is 1 with 90% probability, which is quite accurate and its effect on objective function will be quite low. Since we are trying to find the global minimum of the objective function, this would mean fast convergence. Similarly, for the second vector k', let the fuzzy classifier predict a probability of pˆ ik ′ =0.15, which indicates that it is very unlikely that the label of vector k′ would be 1. The SE would be (0-0.15)2=0.025. In conclusion, the better the classifier functions separate the two classes, the smaller the objective function, JIFC-C would get and as a result the faster IFC-C would converge. Depending on non-linearity of given dataset, one could use user-defined transformations of the membership values, such as exponential transformation
e( μik
imp
)
or power transformation, (μikimp)p, p∈Z. One could use suitable statistical learning algorithms, e.g., logistic regression (LR), or much effective soft computing approach such as support vector classification (SVC), to approximate fuzzy classifiers. In LR case, for instance, posterior probabilities of each data point in cluster i is calculated by −1
⎡ ⎤ T ⎥ ⎢ ⎡ wˆ i ,0 ⎤ ⎡ 1 ⎤ ⎥ ⎢ ⎢ wˆ ⎥ ⎢ μ imp ⎥ ⎥ ⎢ ⎥ ⎥ , ik 1 i ,1 ⎥ ⎢ (3.33) = ⎢1 + exp(−( ⎢ pˆ ik ( yk = 1| τ ik ) = ⎢ ⎥ )) ⎢ ⎥ # ⎢ (1 + exp(− wi ⋅ τ ik )) # ⎥ ⎥ ⎢ ⎢ ⎥ ⎢ ⎥ imp ⎣⎢ wˆ i , nm ⎦⎥ ⎢⎣ e( μik ) ⎥⎦ ⎥ ⎢
⎥ ⎢ τ ik ∈ℜnm +1 ⎣ ⎦
(
)
where τik denotes input matrix, the constituents of which are the membership values, and their user defined transformations, wi are the LR parameters to be approximated. If support vector classification (SVC) is used to estimate fuzzy classifier function parameters, the output values of data vectors are calculated by hi(τi)=∑k βik yk K(τik,τis)+bi , i=1...c, k=1,…n
(3.34)
In (3.34), K(.) is the kernel function that is used to map original interim data vectors, τi,, onto a higher dimensional space, either linearly or non-linearly. In this work, we analyzed two different kernel functions: linear; K(τik,τil)=τikTτil, where τik and τil are interim vectors of nm dimensions which hold the membership values
76
3 Improved Fuzzy Clustering
and their user defined transformations, and the probabilistic kernel, i.e., radial
{
basis function (RBF); K(τik,τil)= exp −δ τ ik − τ ij
2 2
} (δ>0). β
ik
in (3.34) represents
Lagrange multipliers, one for each interim vector, which are introduced to solve the following SVC optimization algorithm for each local model i (cluster):
Max Qi = ∑ k =1 βik − 12 ∑ k ,l =1 βik βil yk yl K (τ ik ,τ il ) n
s.t. ∑ k =1 βik yk = 0 , n
n
0 ≤ βik ≤ Creg , k ,l = 1,...,n,i = 1...c
(3.35)
Creg is the regularization constant, which balances the complexity of the machine, and the number of the separable data vectors. Interim vectors with non-zero coefficients, βik>0, (Lagrange multipliers) are called “support vectors” obtained for each cluster i, τiks. The fewer the support vectors, the better is the generalization capacity of such models. A brief summary of support vector machines for classification (SVC) methods are given in Appendix C.2. The output values obtained from each SVC fuzzy function in (3.34) is a scalar value that one needs to transform into posterior probabilities. We measure the posterior probabilities using improved Platt’s probability method [Platt 2000; Lin et al., 2003], which is based on the approximation of model output labels with a sigmoid function as follows: pˆ ik (yk =1| hi(τi))) = (1+exp(a1 hi(τi)) + a2))-1
(3.36)
a1 and a2 are found by minimizing the log-likelihood of training data:
min a1 ,a2
y +1⎞ ⎞ n ⎛ y +1 ⎛ − ∑ k =1 ⎜ k log ( ˆpik ) + ⎜1 − k log (1 − ˆpik ) ⎟ ⎟ 2 ⎠ ⎝ ⎝ 2 ⎠
(3.37)
The output values obtained from SVC interim fuzzy functions are not represented as class labels, i.e., yk={-1 or 1} , but rather represented with estimated posterior probabilities. It should be noted that, fuzzy classifier functions estimated at each iteration, h(τi), is only used for estimating and structuring the membership values during IFC-C optimization algorithm which excludes the original input variables, x∈ℜnv. Membership value calculation equation to update the membership values in each step is re-formulated for IFC-C according to the latter objectives as follows: 1/( m −1) ⎛ c ⎡ 2 ⎞ 2 d ik ( xk ,υi ) + ( yk − pˆ ik ) ⎤ ⎜ ⎟ imp ⎥ μik = ∑ ⎢ 2 ⎜ ⎟ 2 1< i ≤ c ˆ jk ) ⎥⎦ j =1 ⎢ d ( x ,υ ) + ( y − p ⎜ ⎟ k j jk k ⎣ 1< k < n ⎝ ⎠
−1
(3.38)
Generalized steps of the Improved Fuzzy Clustering algorithm for classification models (IFC-C) are re-displayed as follows:
3.3 Improved Fuzzy Clustering Algorithm (IFC)
77
Step 1: Initialize clustering parameters: m (fuzziness parameter), c (parameter of the number of clusters), choose termination threshold, ε>0 Step 2: Initialize prototypes, υi∈V, and membership values, μikimp∈U⊂ℜn×c. Repeat
Update membership values using (3.38), Update prototypes using (3.31), Until the change in objective function in (3.32) is less than ε.
In Step 2 of IFC-C algorithm, one may use FCM [Bezdek, 1981a] clustering or a crisp clustering such as k-means to find the initial membership values and cluster prototypes. The IFC-C optimization method searches for the optimum membership values, which are to be used as additional predictors to estimate the parameters of Fuzzy Classifier Functions for the given system. Thus, IFC-C is implemented in structure identification phase and inference phases of the new Improved Fuzzy Functions method (to be presented in Chapter 4). The output of the IFC-C algorithm for any given set of {m,c} values are as follows: (i)
optimal parameters, ŵi, i=1,..,c, of the fuzzy classifier function of each cluster h(τi,ŵi) which is used to calculate the posterior probabilities, pˆ i (y=1⎢h(τi,ŵi)) captured from the last iteration step of IFC-C.
(ii)
structure of τi, in other words list of different types of membership value transformations that were used to approximate each h(τi,ŵi) in IFC-C,
Depending on the type of function approximation method used, the parameter, ŵi, will represent different values. For instance, when support vector classification is used, the list of parameters of each cluster i, ŵi, would represent Lagrange multipliers of each support vector of each cluster, βik, and original support vectors, τiS from each cluster, as shown in (3.34). On the other hand, when LR is used, the parameter list is just the coefficients of each membership value (as column vector) to approximate the fuzzy classifier functions as shown in (3.33). Next, we explain why IFC can be a better method than standard FCM clustering for fuzzy function systems.
3.3.4 Justification of Membership Values of the IFC Algorithm “Are membership values obtained from IFC algorithm better predictors than membership values from FCM clustering algorithm?”
The Improved Fuzzy Clustering (IFC) algorithm to be used for regression and classification problems is a modification of two well-known fuzzy clustering algorithms, i.e., FCM and FCRM. It searches for the optimum membership values, which are later to be used as additional predictors of the “Fuzzy Functions” systems [Turksen, 2007; Celikyilmaz, 2005, Celikyilmaz, and Turksen, 2007b,c].
78
3 Improved Fuzzy Clustering
During the structure identification of the novel fuzzy functions approach, FCM clustering or IFC algorithm is used to partition the data into clusters and assign one membership value to every datum for each of these clusters. For each cluster a separate dataset is formed and using these datasets one Fuzzy Function, i.e., a regression function or a classification function, for each cluster is identified by either one of these clustering algorithms. These membership values are then used as additional predictors in these Fuzzy Functions. The assumption is that, the membership values obtained from the novel IFC, which are used as additional predictors of the Fuzzy Functions, can increase the performance of the models more than the membership values obtained from the FCM clustering algorithm. In this section, the strengths of membership values from the FCM clustering and the IFC algorithm is compared using a simple statistical test. It should be noted that the best comparison can be done when two algorithms are applied to real life or benchmarking dataset and test case performances of each model are compared. Such comparisons are presented in the analysis of experiments chapter (Chapter 6). Here we will demonstrate the performance of the models of the FCM clustering and the IFC method using a small artificial dataset. Generally speaking, membership value calculation equations are used to obtain membership values of inputs vectors or output variables of singletons, indicating to what degree an object belongs to a cluster. Membership value calculation equations can be a function of inputs, outputs, or input and outputs. Membership value calculation equations can be manifested by very different shapes and some of the common shapes are triangular, Gaussian, trapezoidal or singleton membership values as shown in Figure 3.2.
Fig. 3.2 Types of Membership value calculation equations
In our applications, we use fuzzy clustering methods, which are based on distance measures between each data point to each cluster center. Since the algorithm is based on minimization of an objective function based on distance
3.3 Improved Fuzzy Clustering Algorithm (IFC)
79
measure, the membership values obtained from scatter diagram which after appropriate curve fitting can be demonstrated as bell-shapes or s-shapes functions of input variables or the output variable for each cluster. It should be emphasized that, for picture representations, idealized (possibly curve-fitted) membership functions are shown to demonstrate the membership values. In fact such membership values ought to be represented with a scatter diagram. On the other hand, during IFC algorithm, Interim Fuzzy Functions of each cluster, h(τi,ŵi), tries to explain the relationship between membership values and their transformations of a particular cluster and the output value, where the membership values become the input variables to explain the output variable in a linear or a non linear function. Therefore, the relationship between the membership values and their transformations and the output variable to be approximated should be the inverse of the known membership shapes as explained in Table 3.2. In this table, possible examples of relationships between the membership values as input variables and an output variable is shown. To demonstrate the relationship with a “bell-shape” and an “s-shaped” distributions are inverted to draw the membership value versus output variable graphs. When one wants to estimate a user defined linear function for each cluster, one way to obtain best approximations of functions is to transform membership values and then use them as input variables. We assume that the relationship between the membership values and their transformations and the output variable to be estimated would approximately be similar to the inverse of the known membership shapes as shown in Table 3.2. Table 3.2 Membership values as input variables in Fuzzy Function parameter estimations Membership value transformation
S-function-1 :
Inverse s-function-1 : 1− u yˆ ≅ fˆ −1 (u ) ≅ − ln u
f ( y) =
1 1 + e − ( a +by )
Membership value as an input variable versus output variable. 2
output-y
Possible function
0 -2
1 f ( y) = 1 + e( a +by )
Inverse s-function-2 : 1− u yˆ ≅ fˆ −1 (u ) ≅ ln u
0 -2
f ( y ) = ay − b
Inverse power function: yˆ = fˆ −1 (u ) = (u / a)( −1/ b ) , a > 0, b > 0
0
0.5 membership-u
1
0
0.5 membership-u
1
2
output -y
Power function:
0.5 1 membership (u)
2
output-y
S-function-2 :
0
0 -2
80
3 Improved Fuzzy Clustering
Here, we want to discuss and demonstrate the significance of the membership values obtained from IFC algorithm, viz., the confidence of the membership values that covers a portion of the unknown output variables, when used as input variables to approximate an output variable with a function fi, μi ( x) → y . In the experiment presented next, we used statistical significance tests for each function of each cluster to justify the performance of a membership value calculation equation. An artificial dataset of 50 samples with a single input x, and a single output y, as shown in Figure 3.3, is used in this experiment. We wanted to determine if there are local linear relationships between the response variable y and the independent variables, viz. membership values of each data point in each cluster and their possible transformations. 3
Fig. 3.3 Scatter plot of the artificial dataset y-output
2
1
0
-1
-2 -3
-2
-1 0 x-input
1
2
When the standard deviations are not known, the best way is to use F-statistic to test if there is a significant relationship between the dependent and independent variables of a system model. The critical Fα,nm,n-2 value for 1-α confidence level, (α ∈{1%,5%,10%}) is captured from statistical graphs. Null hypowork,
H 0 : F clusteri ≤ F critical , i = 1,..., c , states that there is no significant relationship between the membership values and their transformations and the output variable of the specified cluster. Rejection of null hypowork, H0 , due to
H1 : F clusteri > F critical , states that at least one of the independent variables contributes significantly to the model (nm: number of variables). An alternative way to test the significance is to measure the probability, p, obtained from the regression results. p can be compared to the significance level, α. If p<α is correct, then the null hypowork is rejected, and we conclude that the model has significant explanatory power. Otherwise we fail to reject the model. In this small experiment, we tested these F and p values. In this experiment, standard FCM clustering and IFC algorithm are applied on the artificial dataset using five clusters, c=5, and m=2.0. The membership values obtained from FCM and IFC models are used as independent variables to identify one “Fuzzy Function” for each cluster. We used membership values
3.3 Improved Fuzzy Clustering Algorithm (IFC)
81
and their logistic transformations as independent variables, i.e., f ( μi ) = wˆ 0,i + wˆ 1,i μi + wˆ 2,i ⎡⎣ ± ln((1 − μi ) / μi ) ⎤⎦ , where ŵij, j=1,…,nm are the coefficients, to identify linear functions for the FCM clustering and IFC models, five functions one for each cluster. The critical Fα,nm,n-2 value for 1-α=%99 confidence level (n=50-2=48 and nm=2) , which is
critical F0.01,2,48 = 3.20 , is used to
test the significance of each model. Figure 3.4 shows five fitted regression functions (black surfaces) as linear hyper-surfaces, one for each cluster. The independent variables, x-axis and yaxis, are membership values from FCM clustering method and their logistic transformations, respectively. The dependent variable, y-output, is the z-axis. Figure 3.5 is a similar graph, but this time the membership values are obtained from the IFC method. The white surfaces indicate the actual observed decision surfaces, and the black surfaces are the modeled hyper-surfaces using linear functions, fi. The two surfaces, white and black, in each figure can be explained as follows: Actual Decision Surfaces (White Surfaces) in Figure 3.4 and Figure 3.5 are plotted using the membership values obtained from FCM and IFC, respectively and the actual output variable. They represent the actual decision surfaces (the actual relationship between the membership values and the output variable). Since FCM and IFC have different membership value calculation equations, we expect that the actual decision surfaces of FCM in Figure 3.4 will be different than the actual decision surfaces in Figure 3.5. It should be reminded that, this difference between FCM clustering and IFC emerges from the fact that, during IFC optimization, the shapes of membership values are forced to explain the output variable, e.g., in this experiment using interim linear functions of type, f ( μi ) = wˆ 0,i + wˆ 1,i μi + wˆ 2,i ⎡⎣ ± ln((1 − μi ) / μi ) ⎤⎦ , ŵij, j=1,…,nm, i=1,…,c. Since we chose a linear function to represent them, we expect that the white decision surfaces of IFC would be more flat (linear) than the decision surfaces of FCM clustering models. One could observe from Figure 3.4 and Figure 3.5 that, actual decision surfaces (white surfaces) of the IFC models are more flat than those of the FCM clustering models in most clusters, i.e., cluster 1, 2, 5. We can also measure the linearity of the inputs and the outputs (actual decision surfaces) using correlation analysis. In Table 3.3, x1 represents the first input variable that is the membership values, μi , and x2 represents the second input variable that is the logistic transformation of Table 3.3 Correlation Analysis of FCM clustering and IFC membership values with the output variable
FCM IFC
x1 vs. Output 0.18 0.41
x2 vs. Output 0.21 0.32
82
3 Improved Fuzzy Clustering
membership values, ±ln((1-μi)/μi). Each cell in the table represents the root-meansquares-correlation of each model;
c
FCM
⎛1 c ⎞ ( x1 , y ) = ⎜ ∑ corr FCM ( x1,i , y ) 2 ⎟ ⎝ c i =1 ⎠
1
2
⎛1 c ⎞ c FCM ( x2 , y ) = ⎜ ∑ corr FCM ( x2,i , y ) 2 ⎟ c ⎝ i =1 ⎠ c
IFC
⎛1 c ⎞ ( x1,i , y ) = ⎜ ∑ corr IFC ( x1,i , y ) 2 ⎟ ⎝ c i =1 ⎠
1
⎛1 c ⎞ c IFC ( x2,i , y ) = ⎜ ∑ corr IFC ( x2,i , y ) 2 ⎟ c ⎝ i =1 ⎠
1
2
1
= 0.18, x1,i : μi , y: output variable. 2
= 0.21, x2,i : ln((1 − μi ) / μi )
= 0.41, x1,i : μiimp 2
= 0.32, x2,i : ln((1 − μiimp ) / μiimp ) ,
where μiimp represent the improved membership values obtained from the IFC, and μi represent the membership values obtained from the FCM clustering. For instance, c FCM(x 1,y) represents the overall correlation between the membership values, x 1,i= μi , and the output, y, over the five clusters obtained from the FCM clustering algorithm, and c FCM (x 2,y ) is the overall correlation between the transformed membership values and the output, y, over the five clusters obtained from FCM clustering algorithm. Similarly, cIFC(x1,y) represents the overall correlation between the membership values and the output, y, over the five clusters obtained from the IFC algorithm, and cIFC(x2,y), x2,i : ln((1 − μiimp ) / μiimp ) , is the overall correlation between the transformed membership values and the output, y, over the five clusters obtained from the IFC algorithm. It can be observed from Table 3.3 that the correlation between the output variable and the membership values and their transformations obtained from the IFC are higher than those of FCM clustering. So we expect that even with simple methods such as LSE, we can model the output with the IFC better. Model (Predicted) Decision Surfaces (Black Surfaces) as shown in Figure 3.4 and Figure 3.5 are plotted using the membership values obtained from the FCM and IFC, respectively and the predicted output variable from each functions. They represent the predicted (model) decision surfaces. The F-test and the probability test on the predicted functions are discussed below. In these five experimental trials, when the FCM clustering algorithm is used to find the membership values, one out of five fuzzy functions (estimated decision surfaces) reveal F-test values that are greater than the critical value, see Table 3.4. So, we fail to reject the null hypowork and we conclude that for this data, the membership values from standard FCM clustering cannot explain the output variable.
3.3 Improved Fuzzy Clustering Algorithm (IFC)
83
FCM-cluster 2 F=0.59, p=0.56
FCM-cluster 1 F=2.24, p=0.12
1
3
output-y
0.5
output-y
2
0
-0.5
1
-1 0
-1.5 -2
-1
1
1 0.5
u
10
20
30 0
-10 0 log((1-u)/u
-40
-20
0.5
-20 u
FCM-cluster 3 F=3.55, p=0.037
0 20
0
log((1-u)/u
FCM-cluster 4 F=1.41, p=0.25
2
2
1
output-y
output-y
1
0
0
-1
-1 -20
-2 1 0.5
1 0
0.5
20 0 40 log((1-u)/u
0
u
u
10 log((1-u)/u
0
-10
20
30
FCM-cluster 5 F=1.69, p=0.19 2
ou tput-y
1 0 -1 -2 1 30 20
0.5 u
10 0
0 -10
log((1-u)/u
Fig. 3.4 (White Surface) Actual Decision Surface using Membership values (from the FCM clustering), (Black Linear Surface) the estimated linear decision surface. ‘u’ indicates the membership values.
In the same experimental trial, on the other hand, when we analyze the membership values obtained from the IFC algorithm (Figure 3.5), in 3 out of 5 clusters, membership values can explain the output variable much better, since the F-tests are greater than the critical value, see Table 3.4. The majority of the IFC models (3 out of 5) have passed F-significance test. Therefore, we do not fail to reject the null hypowork and we can conclude that the IFC models can model the output variables better than the FCM clustering algorithms for this dataset.
84
3 Improved Fuzzy Clustering IFC-cluster 2 F=4.61, p=0.01
IFC-cluster 1 F=16.68, p<0.0001 4
2 1 output-y
output-y
2
0
-2
0 -1 -2 1
-4 1
-1 0.5 0.5 u 0
5
10
15
0 log((1-u)/u
0
-10
-5
10 0
u
20
log((1-u)/u
IFC-cluster 4 F=2.39, p=0.1
IFC-cluster 3 F=0.37, p=0.69
4
ou tput-y
output-y
2
0
-2
-4 1
u
5 0
0
-2 1
15 0.5
2
10
0.5
0 -5
log((1-u)/u
0
u
-10
0
10
20
log((1-u)/u
IFC-cluster 5 F=13.94, p<0.00001
o u tp u t-y
4
2
0
-2 1 20 0.5
10 0
u
0
-10
log((1-u)/u
Fig. 3.5 (White Surface) Actual Decision Surface using Membership values (from the IFC), (Black Linear Surface) the estimated linear decision surface. ‘u’ indicates the improved membership values.
The results from this artificial data show that the IFC models in most cases, three out of five, can find the optimum fuzzy partitions of the input space. Using membership values from the IFC as predictors, one can summarize that in general one can approximate the output variable better than models constructed with
3.4 Two New Cluster Validity Indices for IFC and IFC-C
85
membership values obtained from standard FCM clustering models. One should run different experiments by using non-linear models, or trying different membership value transformations, or use different parameters for m and c, before coming to a more definitive conclusion. Even though this would be an exhaustive algorithm which may take time, one may be able to obtain the optimum solution. Nonetheless, here, we wanted to display that even with simple regression models the IFC models can appear to be more powerful estimators than the FCM clustering results in function estimation problems. In this example, it is shown that the membership values cannot display a significant relationship with the output when FCM clustering is used to obtain them. Table 3.4 Significance Test Results of fuzzy functions using membership values obtained from FCM clustering and IFC fuzzy functions
FCM
F-value* 2.24 0.59 3.55 1.41 1.69 = 3.20 is the critical value.
Cluster-1 Cluster-2 Cluster-3 Cluster-4 Cluster-5 critical * F0.01,2,48
IFC
p-value 0.12 0.56 0.037 0.25 0.19
F-value 16.68 4.61 0.37 2.39 13.94
p-value 0.0001 0.01 0.69 0.1 0.0001
3.4 Two New Cluster Validity Indices for IFC and IFC-C Fuzzy clustering methods, including the Improved Fuzzy Clustering (IFC) [Celikyilmaz and Turksen, 2007b] algorithm, assumes that some initialization parameters are known prior to the model execution. This is usually a problematic issue since different parameters may reveal different results which could eventually affect system performance. Literature indicates that, many cluster validity functions have been proposed to validate underlying assumptions of number of clusters, especially for the FCM [Bezdek, 1981a] clustering approach. Among well-known validity functions the ones proposed by [Fukuyama and Sugeno, 1989; Xie and Beni, 1991; Pal and Bezdek, 1995; Bezdek, 1976] are the most commonly used FCM clustering validation measures. In later years, many variations of these functions, e.g., [Boguessa et al., 2006; Dave, 1996; Kim et al., 2003; Kim and Ramakrishna, 2005], are presented by modifying or extending these earlier validity functions. These validity functions measure characteristics of a point-wise fuzzy clustering method, i.e., FCM clustering algorithm. They are limited in determining the best clustering structure, but they can provide some information about underlying information of membership values. The main characteristics of these validity functions are that, they all use either within cluster or between-cluster distances or both as a way of assessing clustering schema. The value of within-cluster distances is interpreted as the compactness and the value of between cluster distances is interpreted as the separability of the clustering structure [Kim et al., 2003]. These validity functions, to be summarized in the
86
3 Improved Fuzzy Clustering
following sections, can be categorized into two groups: ratio type and summation type [Kim et al., 2003]. The type of a validity function is determined by the way they combine the within-cluster and between cluster distances. As stated earlier, most validity indices are designed to validate FCM [Bezdek, 1981a] clustering algorithm. They use characteristics of the FCM to indicate the optimum number of clusters. In this sense, earlier validity indices designed for the FCM clustering method may not be suitable for other variations of fuzzy clustering algorithms, which are designed for different purposes, e.g., Fuzzy CRegression (switching regression) algorithm (FCRM) [Hathaway and Bezdek, 1993]. For these types of the FCM clustering algorithm variations, different validity measures have been introduced. For instance, in [Kung and Lin, 2004] a new validity index is proposed to measure optimum number of clusters of FCRM applications. Their validity function is a modification of Xie-Beni [1991] ratio type validity function. It accounts for the similarity between regression models using standard inner product of unit normal vectors, i.e., estimated regression equations for each cluster, instead of the distance between cluster centers. The validity functions should be designed based on the objectives and structure of the corresponding fuzzy clustering methods. In this work, two new fuzzy clustering methods are proposed, i.e., IFC and IFC-C. Therefore in this section we investigate two new cluster validity functions to measure the optimum number of clusters of the models of these new clustering algorithms. The validity functions, to be discussed below, are ratio type indices, which measure the ratio between the compactness and the separability. Since IFC and IFC-C algorithms are new types of clustering methods, which combine two different fuzzy clustering approaches, i.e., FCM clustering [Bezdek, 1981a] and FCRM [Hathaway and Bezdek, 1993] methods in a novel way and utilize “Fuzzy Functions”, the new validity indices are designed to validate two different concepts in the following way. The compactness is used to combine within-cluster distances and the errors between actual and estimated output obtained from c number of regression functions. The separability, on the other hand, is used to determine the structure of clusters by measuring the ratio of cluster center distances to the angle between their regression functions. If two functions of different clusters happen to be parallel to each other, only cluster center distances are to be used as a separability measure. In the next section, well-known cluster validity indexes that are closely related to the proposed validity measures are reviewed and new cluster validity measures are introduced. Then, using four different artificial datasets, and two real life benchmarking datasets, the performance of new validity function for IFC is compared to three other well-known cluster validity measures, which are closely related to the new validity measure.
3.4.1 Overview of Well-Known Cluster Validity Indices The literature indicates that there is numerous cluster validity formulas (indexes) designed for different clustering methods. This section presents validity formula designed for two different types of fuzzy clustering algorithms: point-wise clustering, e.g., FCM clustering [Bezdek, 1981a] and regression type clustering, e.g., fuzzy c-regression model (FCRM) type algorithms [Hathaway and Bezdek,
3.4 Two New Cluster Validity Indices for IFC and IFC-C
87
1993]. The research indicates that most of the prominent cluster validity indices are designed for FCM algorithms to find the optimum number of clusters. In a more recent study, Kung & Lin [2004] proposed a new validity index for identification of the optimum number of clusters of fuzzy c-regression model (FCRM) clustering algorithm. The fact that the new IFC [Celikyilmaz and Turksen, 2007b] algorithm combines regression and clustering concepts, we hypothesize that the new validity index should measure the concepts from two different types of cluster validity indexes while trying to indicate the optimum number of clusters. Next, we will investigate both types of validity measures from literature before we present the new cluster validity indices. Well-Known CVIs for Point-wise and Regression Type Clustering Algorithms
Most cluster validity functions are structured by combining two different clustering concepts [Kim and Ramakrishna, 2005]:
• •
Compactness: to measure the similarity between cluster elements within each cluster. Most validity indexes use within cluster distances to measure compactness. Separability: to measure dissimilarity between each individual cluster. Most validity measures use between cluster center distances to measure separability.
It has been shown that a clustering algorithm is effective when compactness is small and separability is large [Kim and Ramakrishna, 2005]. Based on the way these two concepts are conjunct, cluster validity indices can be categorized into two different types: ratio-type and summation-type. Ratio-type validity indices are formed by measuring compactness to separability ratio, e.g., Xie-Beni [1991] index. The summation-type validity indices combine separability concepts by adding them in many different ways, e.g., Fukuyama-Sugeno [1989] index. Since new validity index proposed in this paper is a ratio-type validity measure, we will give details of prominent ratio-type validity indexes from literature. A well-known ratio-type cluster validity index (compactness/separability) is XB cluster validity index [Xie and Beni, 1991] which is formulated by:
⎛ c n 2 2⎞ ⎜ ∑∑ μik d ( xk , vi ) ⎟ / n ⎠ , d (x ,v ) = x − v XB (c) = ⎝ i =1 k =1 k i k i 2 min i , j ≠ i d (vi , v j )
2
(3.39)
where xk∈ℜnv represents the kth input vector, k=1,…,n andυi∈ℜnv , i,j=1,…,c, represents cluster center as a vector of nv dimensions. XB decreases monotonically when c is close to total number of data samples, n. Kim and Ramakrishna [2005] discuss the relationship and behavior of compactness and separability between clusters obtained from the FCM clustering method for changing values of number of clusters. In [Kim et al., 2003; Kim and Ramakrishna, 2005], the relationship between compactness and separability is demonstrated using graphs. Figure 3.6 is adopted from [Kim and Ramakrishna,
88
3 Improved Fuzzy Clustering
compactness, seperability
2005]. According to their generalization, compactness increases sharply as c decreases from coptimal to coptimal-1. This means that for c coptimal compactness will be small. Compactness will be zero when each object is a cluster of itself, i.e., c=n. A sudden drop in compactness would be the indicator of the coptimal.
separab ility
compactness c-optimal num be r of cluste rs
Fig. 3.6 Compactness and Separation concepts of ratio-type CVI index
From the analysis of results of FCM clustering, one observes that each cluster may have different compactness values because some clusters are more/less dense than the others. As the number of clusters is increased/decreased, the change in compactness of those clusters would be different (bigger or smaller) than the rest of the clusters. In XB validity index, compactness of overall clustering structure is determined (the numerator in (3.39)) by averaging compactness of every cluster. However, averaging might suppress the effect of large changes in compactness values of some clusters. These changes are usually caused by FCM clustering models with undersized (oversized) number of clusters. Therefore, to exploit these huge compactness value shifts during determination of the optimum number of clusters, the best way is to interpret the compactness of a model by measuring the maximum compactness of clusters. On the other hand, from Figure 3.6 it can be observed that the relative changes of compactness and separability are somewhat similar when c≠ coptimal. Therefore, their effect on ratio-type validity indices should be somewhat similar, i.e., they should both show an increasing/decreasing behavior at the same c values. Concerning the latter discussion, in [Kim and Ramakrishna, 2005] an improved version of XB validity index is proposed as follows:
⎧ n ⎪ max i =1,..,c ⎨∑ μik2 xk − υi ⎪ k =1 n ⎩ * XB (c) = 2 min i , j ≠ i υi − υ j
2
⎫ ⎪ ⎬ ⎪ ⎭
(3.40)
3.4 Two New Cluster Validity Indices for IFC and IFC-C
89
It was proved in [Kim and Ramakrishna, 2005] that XB* index is more effective than XB index, because with XB* one can detect clusters if they have large compactness values. With this information one can determine the optimum number of clusters by observing these ambiguities in clustering structure. Hence, XB* will be the starting point of our new cluster validity formula for IFC algorithm. On the other hand, Kung and Lin [2004] formulated a validity index to validate fuzzy c-regression (FCRM) [Hathaway and Bezdek, 1993] type clustering approaches, i.e., fuzzy adaptation of switching regression problems. FCRM clustering algorithms models identify c regression model parameters and a partition matrix (membership values matrix), which is interpreted as importance or weight attached to error measure between actual output and each regression model output. The validity measure in [Kung and Lin, 2004] is based on XB index; however, compactness is measured by the error between the observed output and the output obtained from a linear or polynomial regression function of each cluster. The separability is measured by inverse dissimilarity between clusters defined by absolute value of the standard inner-product of unit normal vectors, which represent c hyper-planes. Kung-Lin’s cluster validity index is formulized as follows: 2⎞ ⎛ c n 2 T ⎜ ∑∑ μik x β i − yk ⎟ / n ⎠ , where Kung − Lin(c) = ⎝ i =1 k =1 ⎡ ⎤ 1 ⎢ ⎥ ⎢ max u , u + κ ⎥ i j ⎢⎣ i ≠ j ⎥⎦
(3.41)
-1
βi = ⎡⎣ xT μi x ⎤⎦ ⎡⎣ xT μi x ⎤⎦ y Numerator in (3.41) is a compactness measure and denominator represents separability. ui’s represent unit normal vector of each c-regression function. FCRM models [Hathaway and Bezdek, 1993] are represented with regression equations and therefore their corresponding unit vectors are defined by
ui =
[ni ] , ni = [ β i1 " β i , nv −1] ∈ℜ nv +1 ni
(3.42)
ni indicates regression function parameters, βi,nv, in a vector form and ||.|| is Euclidean norm, nv indicate number of variables of input dataset, x =[x1,…,xnv]. The inner product of two unit vectors of two clusters equals to the value of cosine of the angle between them. This value is used to measure the separability of c regression functions of Kung and Lin’s [2004] validity formula. When functions are orthogonal, separability is maximized. They have shown that, their validity function is a good indication for FCRM models.
90
3 Improved Fuzzy Clustering
3.4.2 The New Cluster Validity Indices In this work, two new ratio-type cluster validity indices, cviIFC, and cviIFC-C are presented to validate novel Improved Fuzzy Clustering (IFC) [Celikyilmaz and Turksen, 2007b] and IFC for classification problems, IFC-C [Celikyilmaz and Turksen, 2007i] Firstly, we will focus on IFC for regression problems. In IFC algorithm, clusters are identified by cluster prototypes (centers) and their corresponding regression functions. Membership values calculated from IFC algorithm represent the degree to which each object belongs to each prototype. They are also used as candidate input variables, which can help to identify relationships between input and output variables with regression functions. It should be reminded that, each input vector is mapped onto a new feature space using membership values and/or their user-defined transformations as new additional inputs. A new dataset is formed for each cluster in feature space and then one “Fuzzy Function” is estimated using these datasets. It is therefore expected that, membership values obtained from IFC algorithm can explain output variable, in other words be “good” predictors of the regression functions, as well as represent better fuzzy partitions in the feature space. When validating IFC algorithm, one needs to find a relation between the compactness and separability of clusters by measuring the clustering structure as well as the relationships between their representative regression functions. The new validity measure should include both of these concepts when validating the number of clusters of IFC models. The compactness of the new validity measure will combine two terms. (1) XB* compactness (numerator in (3.40)) will be used as the first term of compactness and (2) a modified version of compactness of Kung-Lin index (numerator in (3.41)) as a second term. The second term of compactness of cviIFC will represent the error between regression model and actual output. Regression models are the “Fuzzy Functions”, f(Φi,Ŵi)Æy, where Φi is a matrix of input variables and their membership values in the corresponding cluster and their transformations, and Ŵi are the regression coefficients, which are the mappings of the input space onto a new space using membership values. On the other hand, the separability of cviIFC will couple the angle between regression functions and the distance between each cluster center prototypes. In the new validity function, original scalar inputs also enter as predictors of functions when measuring the compactness. This can be explained as follows: IFC algorithm finds membership values, μiimp, i=1,…,c, that can predict the local models of a given system. (imp) indicates that membership values are from improved clustering (IFC). These membership values and/or their transformations are also used together with the original input variables to determine “Fuzzy Functions” for each cluster using suitable function approximation method, e.g., least squares, support vector machines, ridge regression, etc., in system modeling with fuzzy functions (to be discussed in Chapter 4). The novel IFC algorithm introduces membership values and their transformations as additional predictors along with the original input variable in order to minimize the error of the local models of each cluster. For this reason, here optimum number of clusters of the
3.4 Two New Cluster Validity Indices for IFC and IFC-C
91
new IFC algorithm is validated by analyzing the behavior of new membership values in addition to the original input variables in regression functions. The following steps describe the configuration of the new validity analysis: (i) A different dataset is structured for any cluster i by using membership values (μiimp) and/or their transformations as additional predictors. This is the same as mapping original input space, x∈ℜnv of nv different input variables, onto a higher dimensional feature space ℜnv+nm, i.e., xÆΦi(x,μiimp)∈ℜnv+nm, for each cluster i, i=1,..,c. Hence, each data vector is represented in (nv+nm) feature space. nm is the number of augmented membership values and their potential transformations as new predictors to the original input space. Below, a special feature space is shown which is formed by mapping the original input matrix of one-dimensional inputs onto a new space of (nv+1+1) dimensions and a single output using only the membership values by
⎡μiimp x1 ⎤ ×1 ⎢ ⎥ imp Φi(x, μi ) = ⎢ # #⎥ ⎢μiimp ⎥ ⎣ ×n xn ⎦
⎡ y1 ⎤ ⎢ # ⎥ ∈ℜnv+nm ⎢ ⎥ ⎢⎣ yn ⎥⎦
(nv=1, nm=1). If it is a need, we use mathematical transformations of membership values such as (μikimp)2, (μikimp) m, exp(μikimp), ln((1- μikimp)/μikimp), etc., where m represents the degree of fuzziness of IFC. (ii) We then fit a regression function for each cluster using their corresponding input datasets, Φi(x,μiimp). Therefore, new cluster validity index designed for the IFC clustering is formulated as; vc * = max i =1,.., c
1 n ∑ μ ikm n k =1
(
⎧ min v − v 2 i j ⎪ i, j≠i vs * = ⎨ ⎪ min i , j ≠ i ⎩
(xy
)
⎫ + ( y k − f i ( Φ i , Wˆi )) 2 ⎪ ⎪ , vc * ⎪ α i , α j , if α i , α j ≠ 0 ⎬ cviIFF = (3.43) * ( c ⋅ vs ) + 1 ⎪ i,j=1,..,c,j ≠ i ⎪ 2 vi − v j , otherwise ⎪⎭ k
k
− vi
)
2
where vc* represents compactness and vs* represents separability of the new validity measure.
Φ
Let ni ,
niΦ = Wˆi1 ,Wˆi 2 ,..,Wˆinm ,Wˆi ( nm +1) ,...,Wˆi ( nm + nv ) ∈ℜnv + nm ,
represent the normal vector, i.e., the vector orthogonal to the plane, of fuzzy functions, obtained from given dataset in the feature space, Φi(x,μiimp)∈ℜnv+nm . α i in the |〈αi,αj〉|∈[0,1] represents unit normal vector of each “Fuzzy Function”, i,
α i = [niΦ ] / niΦ
, where
niΦ is the length of the vector and αi represents the unit
vector. Absolute value of inner product of two unit vectors of fuzzy functions of two clusters, i,j=1,…,c, i≠j, equals to the value of cosine of the angle between them:
92 cosθi , j = α i , α j =
3 Improved Fuzzy Clustering niΦ , nΦj niΦ n Φj
=
(Wˆ ) i1
⎡Wˆi1Wˆ j1 + ... + Wˆi ( nm + nv )Wˆ j ( nm + nv ) ⎤ ⎣ ⎦ 2
(
) ( ) 2
+ ... + Wˆi ( nm + nv ) . Wˆ j1
2
(
+ ... + Wˆ j ( nm + nv )
)
2
(3.44)
When two cluster centers are too close to each other due to oversized number of clusters, the distance between them becomes almost (≅0) invisible, then the validity measure goes to infinity. To prevent this, the denominator of cviIFC in (3.43) is increased by 1. In cviIFC-C for IFC-C models, “Fuzzy Functions” in vc*, fi(Φi(x,μiimp),Ŵi), are replaced with posterior probabilities obtained from classifier models as follows: ⎧1 n vc * = max i =1,..,c ⎨ ∑ μ ikimp ⎩ n k =1
(
) (x m
k
y k − vi
2
)
⎫ + ( yk − Pˆi ( y = 1 | f ( Φ i ( x , μ iimp ), Wˆi ))) 2 ⎬ ⎭
(3.45)
Compactness measure, vc* (numerator of cviIFF) of the new validity function, which is different from the XB* compactness criteria, includes an additional term; squared error of Fuzzy Functions. Nevertheless, the compactness, vc* has still the similar affect on the outcome as the compactness of XB* index as shown in Figure 3.6 for the following reasons. When c>c-optimum, compactness of clusters will be small. This is due to the fact that, as the number of clusters is increased, within cluster distances will decrease since clusters will include more similar objects. In addition there is one regression function estimated for each cluster. Similarly, as the number of functions is increased, the error of the “Fuzzy Functions” will decrease, because regression model output will approach to actual output. Compactness will be zero when c=n, which is when every object becomes its own cluster center and one function, passes through every object. On the other hand, when c=c-optimum, the compactness of each individual cluster will all be very small, so that the difference between maximum and minimum compactness of clusters will be negligible. Hence, we can identify sudden changes by analyzing the maximum compactness values instead of average compactness values. These are assumptions under the optimum conditions and new cluster validity index combines the measure of the separability of clusters within the new validity index to increase the precision of approximation of optimum number of clusters.
3.4 Two New Cluster Validity Indices for IFC and IFC-C
93
Separability of the new validity index also combines the separability measure obtained from two different structures, i.e., regression and clustering. Between cluster distances are represented as Euclidean distances between cluster centers in (3.43). Additionally, absolute value of cosine of the angle between each “Fuzzy Function” |〈αi,αj〉|∈[0,1] is used as an additional separability criteria. If functions are orthogonal, then they are the most dissimilar functions, which is an attribute of an optimum model. The separability, i.e., denominator of cviIFC in (3.43), conditionally combines between-cluster-distances and angles by taking the ratio between them. If an angle between any two function is zero, then they are parallel to each other. So we used the minimum distance between their cluster centers to represent the separability. Better separability would then yield better clustering results. Figure 3.7 is used to explain the changes in separability based on different patterns in the dataset. When two clusters are apart from each other, i.e. the distance between their cluster centers are farther than the rest of the clusters, e.g., cluster1 (Δ’s) and cluster 2 (•’s) in Figure 3.7, no matter what the angle between their regression functions are, their separability will be large. Separability will get larger, if the two vector lines are orthogonal, i.e., cosine of the angle approaches to zero as the value of the angle approaches to 900. On the other hand, when clusters are so close to each other, e.g., cluster 2(•’s), cluster 3 (*’s) and cluster 4 (o’s) in Figure 3.7, the angle between their functions would be dominant separability identifier. If vectors are close to being orthogonal vectors, i.e., cosine of the angle between them is very close to 0 as in cluster3 and cluster 4, then the minimum separability would be very large even if their distance is close to zero. When two 50
40 1 30
20 2
10
0 4 -10 3 -20 -5
-3
-1
1
3
5
Fig. 3.7 Four Different Functions with Varying Separability Relations
94
3 Improved Fuzzy Clustering
cluster centers are close to each other and the functions are almost parallel to each other, i.e., the value of the cosine of the angle would be close to 1, e.g., cluster 2(•’s) and cluster 3 (*’s), then the separability will be small. Using the artificially created dataset, to be presented next, we will justify that the presented cviIFC and cviIFC-C validity measures are good indicators of the new clustering schema, i.e., IFC and IFC-C respectively. The performance of the new validity measures will be investigated in comparison to three other wellknown validity criteria that were mentioned above.
3.4.3 Simulation Experiments [Celikyilmaz and Turksen, 2007i;2008c] In this section we present simulation experiments to measure the performance of cviIFC and cviIFC-C methods by applying IFC and IFC-C methods on artificially created datasets of different structures as well as real datasets. We also apply three different well-known cluster-validity indices to justify the strength of the new cluster validity measures. Later, we will discuss the results obtained form validity experiments. Experiment 1
In order to demonstrate the performance of the new validity indices, we constructed tests on datasets with known structures (number of clusters, or patterns of functions, etc.,). We introduce a dataset structure similar to [Kung and Lin, 2004], but we formed 4 different datasets containing different number of linear models. Each dataset is created using different number of functions with varying Gaussian noise, εl,m(l=1..4, m=1,..,nf) having zero means and variance of 0.9 in each function. m represents each linear function such as in Table 3.5 to be used to generate each dataset l, l=1..4. Firstly, 400 training input vectors, x, which are uniformly distributed in the range of [-5, +5], are generated randomly to form the first dataset. Then the dataset is grouped into 4 separate parts, of 100 observations. Each function from Table 3.5(A) is applied to one group to obtain output values, e.g., 4-clustered datasets are formed using four separate local models. Following the same convention, we generated 500, 490 and 450 more training vectors, x, separately, which are also uniformly distributed in the range of [-5, +5] to generate the datasets with 5, 7 and 9 patterns using the corresponding functions in Table 3.5 (B), (C), and (D), respectively. Using 500, 490 and 450 training vector sets; we applied each of the 100, 70 and 50 observations to the 5, 7, and 9 functions correspondingly to form 3 more single input- single output datasets, e.g., dataset2, dataset3, and dataset4. We applied IFC algorithm [Celikyilmaz and Turksen, 2007b] onto these 4 datasets separately using 2 different degree of fuzziness values, m=1.3, and m=2.0 and 14 different number of clusters, c=2,…,15. We measured the new cluster validity index, cviIFC as well as XB, XB*, and Kung-Lin validity measures using the membership values obtained from these IFC models.
3.4 Two New Cluster Validity Indices for IFC and IFC-C
95
Table 3.5 Functions used to generate Artificial Datasets (A) 4-cluster (dataset1) y1 = x1T β1 = 2 x + 5 + ε1,1
(C) 7-cluster (dataset3) y1 = x1T β1 = 2 x + ε 3,1
(D) 9-cluster (dataset4) y1 = x1T β1 = x + ε 4,1
y2 = x2T β 2 = 0.2 x − 5 + ε1,2
y2 = x2T β 2 = 7 x − 5 + ε 3,2
y2 = x2T β 2 = 3x + 2 + ε 4,2
y3 = x β3 = − x + 1 + ε1,3
y3 = x3T β 3 = − x + 1 + ε 3,3
y3 = x3T β 3 = 0.5 x + 1 + ε 4,3
y4 = x β 4 = x − 8 + ε1,4
y4 = x β 4 = x − 14 + ε 3,4
y4 = x4T β 4 = 3x − 3 + ε 4,4
(B) 5-cluster (dataset2) y1 = x1T β1 = 2 x + ε 2,1
y5 = x β5 = −5 x − 6 + ε 3,5
y5 = x5T β5 = − x + 2 + ε 4,5 y6 = x6T β 6 = −2 x − 9 + ε 4,6
y2 = x β 2 = 7 x − 5 + ε 2,2
y6 = x β 6 = 6ε 3,6 y7 = x7T β 7 = 3x − 25 + ε 3,7
y7 = x7T β 7 = −2 x + ε 4,7
T 3
T 4
T 2
y3 = x3T β 3 = − x + 1 + ε 2,3
T 4
T 5
T 6
y8 = x8T β8 = x − 0.5 + ε 4,8
y4 = x4T β 4 = x + 14 + ε 2,4
y9 = x9T β9 = − x + 0.2 + ε 4,9
y5 = x β5 = −5 x − 6 + ε 1,5 T 5
Figure 3.8 illustrates the values obtained from the four cluster validity functions using membership values of the IFC models on dataset1 for changing values of number of clusters, c, and two fuzziness values, m=1.3 and 2.0. m=1.3 is used to represent a more crisp model where overlapping of clusters are negligible, while m=2 is a model where the local fuzzy clusters (models) overlap to a degree of 2.0. In an analogical manner, Figure 3.9, Figure 3.10, Figure 3.11 compare the values obtained from these four cluster validity measures using the membership values of the IFC models on dataset2, dataset3 and dataset4, respectively. Hence, it is expected that c* would be 4, 5, 7, and 9 for dataset1, dataset 2, dataset3 and dataset4, respectively. 15
8
m=1.3 m=2.0
m=1.3 m=2.0
6 XB*
XB
10 5
4 2
0
0
4 8 12 number of clusters
0
15
0.15 m=1.3 m=2.0
0.6 0.4 0.2
4 8 12 number of clusters
15
m=2.0 m=1.3
0.125 cviFF
Kung-Lin
0.8
0
0
0.1 0.075 0.05 0.025 0
0
4 8 12 number of clusters
15
0
4 8 12 number of clusters
15
Fig. 3.8 Dataset 1 - Cluster validity measures, XB, XB*, Kung-Lin and cviIFC, versus c for two m values;m=1.3 (.-) and m=2.0(*-) using 4-patterned dataset
96
3 Improved Fuzzy Clustering 60
8
m=1.3 m=2.0
XB
4
20 0
m=2.0 m=1.3
6 XB*
40
2 0 0
5 10 number of clusters
15
0
5 10 number of clusters
15
0.8 m=1.3 m=2.0 cviFF
Kung-Lin
0.6 0.4 0.2 0
0
5 10 number of clusters
m=1.3 m=2.0
0.2 0.1 0
15
0
5 10 number of clusters
15
Fig. 3.9 Dataset 2- Cluster validity measures, XB, XB*, Kung-Lin and cviIFC, versus c for two m values; m=1.3 (.-) and m=2.0(*-) using 5-patterned dataset 3
6 m=1.3 m=2.0
m=1.3 m=2.0
2
XB
XB*
4
1
2 0
0 1
4 7 10 13 number of clusters
15
13
15
m=1.3 m=2.0
m=1.3 m=2.0
cviFF
0.75 Kung-Lin
4 7 10 number of clusters
1.5
1
0.5
1 0.5
0.25 0
1
1
4 7 10 13 number of clusters
15
0
1
4 7 10 number of clusters
13
15
Fig. 3.10 Dataset 3- Cluster validity measures, XB, XB*, Kung-Lin and cviIFC, versus c for two m values; m=1.3 (.-) and m=2.0(*-) using 7-patterned dataset
Experiment 2
In order to explain the strength of the proposed validity measure on real datasets, we applied the proposed IFC on a dataset of historical stock prices of a major Canadian Financial Institution. The aim in this experiment is to predict the last two months’ stock prices using estimated models obtained from the previous 10 month stock prices. In real datasets, just as in artificial datasets, there are hidden
3.4 Two New Cluster Validity Indices for IFC and IFC-C 3
6
m=1.3 m=2.0
m=2.0 m=1.3
2
XB
XB*
4
1
2 0
1
4
7 10 numer of clusters
13
0
3 6 9 12 number of clusters
15
0.15 m=2.0 m=1.3 cviFF
m=2.0 m=1.3
0.8 0.6
0.1
0.05
0.4 0.2
0
15
1 Kung-Lin
97
0 0
3
6 9 12 number of clusters
15
0
3 6 9 12 number of clusters
15
Fig. 3.11 Dataset 4-Cluster validity measures, XB, XB*, Kung-Lin and cviIFC, versus c for two m values; m=1.3 (.-) and m=2.0(*-) using 9-patterned dataset -3
x 10
Potential Cluster #1
3.5
Potential Cluster #2
3
density
2.5 2 1.5 1 0.5 0
70 60 50 45 40 Bollinger-Band 20 Day
50
55
60
65
EMA 50 day
Fig. 3.12 Density graph of stock price dataset using two financial indicators. Two components that are well separated are indicated.
components viz., clusters, within which input-output relations can be locally identified. Since we do not have a prior conception of the actual number of clusters of real datasets, we can conduct an exhaustive search by changing the number of clusters and keeping other parameters constant and investigate the
98
3 Improved Fuzzy Clustering
performance of each model. The optimum model would be the one with the best performance, e.g., the least model error, etc. We identify the optimum number of clusters based on the number of cluster of the optimum model. Here, we used the proposed fuzzy system modeling tools based on type-1 Improved Fuzzy Functions, (T1IFF) to be discussed in chapter 4. T1IFF system modeling tools implement novel IFC method to find hidden structures in a given dataset. Improved membership values are obtained from IFC to be used as additional predictors to identify the local fuzzy functions. We iterated the T1IFF for different values of the number of clusters, keeping the rest of the parameters constant and captured the optimum number of clusters, c*, based on the minimum error criteria. At this point we should pause and explain the multi-modal structure of real systems. One can observe hidden structures in a given system by just analysing the probability distributions of input variables. For instance, in Figure 3.12 the Gaussian density graphs of the given stock price dataset is illustrated using two financial indicators. Two separate components can easily be noticed from the graph. The degree of overlap between these components might affect the number of clusters perceived by the human operator. We expect to distinguish between any hidden overleaping clusters in real datasets using the cviIFC based on the structure of the system. In this experiment, stock prices collected over 12 months are divided into two parts. Indicator values from the first ten months, i.e., from 27 July, 2005 to 11 May, 2006, are used to train models and to optimize model parameters. The last two months, i.e., from 11 May-21 July 2006, are holdout for testing the model’s performances. Experiments were repeated with 20 random subsets of above sizes. Model performances using holdout datasets are measured using root-mean-errorsquare (RMSE) and averaged over 20 repetitions. Financial indicators such as moving average, exponential moving average are used as predictors. Detailed explanations on the financial indicators of such experiments are given in the chapter of experiments. We applied an exhaustive search method to identify the optimum T1IFF model by changing values of number of clusters, bounded with c≤ n*(1/10), where n is the number of training samples. Improved membership values, μimp, their logit transformations, i.e., log((1-μimp)/μimp), and exponential transformations, i.e., exp(μimp), are used as additional dimensions (input variables) to approximate fuzzy functions in a new feature space. Based on the minimum RMSE values, the average optimum c, c*, of the best T1IFF models over 20 repetitions is identified as c*=4.9±1.77. In short, we found that this stock price dataset may consist of c*∈[3, 6] structures. We now want to test if the new cluster validity index could have found the optimum c*, by just applying IFC clustering without executing the exhaustive search method using T1IFF strategy. To validate the latter statement, we calculated the values of the proposed validity index, cviIFC, as well as XB, XB* and King-Lin indices using membership values obtained from the proposed IFC models. We plotted validity index values measured for different c values and two fuzziness values, m=1.3 and
3.4 Two New Cluster Validity Indices for IFC and IFC-C 8
3 m=1.3 m=2.0
4
1
2 2
4 6 8 number of clusters
0
10
2
4 6 8 number of clusters
10
1.5 m=1.3 m=2.0
0.01
cviFF
Kung-Lin
0.015
0.005 0
m=1.3 m=2.0
2 XB*
XB
6
0
99
2
4 6 8 number of clusters
10
m=1.3 m=2.0
1 0.5 0
2
4 6 8 number of clusters
10
Fig. 3.13 Stock Price Dataset - Cluster validity measures, XB, XB*, Kung-Lin and cviIFC, versus c for two m values; m=1.3 (.-) and m=2.0(*-)
2.0. The CVI graphs in Figure 3.13 indicates the values of different CVI measures using membership values from IFC algorithm applied on stock price dataset. Experiment 3
In addition to the first two experiments above, we wanted to validate the performance of the novel Improved Fuzzy Clustering for classification (IFC-C) clustering algorithm using the new cviIFC-C method. Since proposed IFC-C is specifically designed for binary classification problems; we have chosen the Ionosphere classification dataset from UCI repository [Newman et al., 1998] to demonstrate the performance of the new cviIFC-C. The targets of ionosphere dataset take on the values of “good” or “bad” indicating the free electrons in the ionosphere. "Good" radar returns are those showing evidence of some type of structure in the ionosphere. "Bad" returns are those that do not; their signals pass through the ionosphere. The details of this datasets are given in chapter of experiments (Chapter 6). In this experiment, again the actual number of clusters that the Ionosphere dataset might hold, viz., overlapping or discrete structures that represent multiple classification surfaces of the ionosphere dataset, is an unknown parameter. Therefore we applied our proposed Improved Fuzzy Functions approach (T1IFFC) for classification problems (to be discussed in Chapter 4) on the ionosphere dataset. We only changed the number of clusters keeping the rest of parameters constant. The list of parameters of previous T1IFF models of experiment 2 that
100
3 Improved Fuzzy Clustering
are used to model stock prices is also used to build different models T1IFF-C of this experiment. The experiment is repeated 10 times. The results have yielded that the average c* is around c*=3.5±0.92, viz, c*∈[3-5] based on the average highest classification accuracy over 10 iterations. To validate the optimum number of clusters, c*, we applied the proposed cviIFC-C as well as XB, XB* and KingLin indices on the outcome of 10 different IFC-C models and took the average of the values of each individual validity measure. Figure 3.14 demonstrates the results for two fuzziness values, m=1.3 and 2.0. 60
400
200
0
2
4 6 8 number of clusters
2
4 6 8 number of clusters
10
0.8 m=1.3 m=2.0
0.06 0.04 0.02 2
4 6 8 number of clusters
m=1.3 m=2.0
0.6 cviFF-C
Kung-Lin
20 0
10
0.08
0
m=1.3 m=2.0
40 XB*
XB
m=1.3 m=2.0
10
0.4 0.2 0
2
4 6 8 number of clusters
10
Fig. 3.14 Ionosphere Dataset- Cluster validity measures, XB, XB*, Kung-Lin and cviIFCC, versus c for two m values; m=1.3 (.-) and m=2.0(*-)
3.4.4 Discussions on Performances of New Cluster Validity Indices Using Simulation Experiments We have reached the following conclusions based on the analysis of the results of the experiments on 4 different artificial datasets and a real life dataset of prediction nature (stock price dataset). We analyzed the outcome of IFC to measure the values of cviIFC along with the values of the known cluster validity measures; XB, XB*, and Kung-Lin. The discussions also include the analysis of models of IFC-C on a binary classification dataset, i.e., Ionosphere dataset. For these experiments we measured cviIFC-C as well as XB, XB*, and Kung-Lin to validate the IFC-C model results.
Analysis of Experiment 1. In Table 3.6 the predicted optimum number of clusters obtained from four different validity measures using 4 different artificial datasets for two different fuzziness levels, i.e., m=1.3 and m=2.0, are shown.
3.4 Two New Cluster Validity Indices for IFC and IFC-C
101
Table 3.6 Optimum number of clusters of artificial datasets for m∈{1.3,2.0}
Actual # of Clusters Î m=1.3 XB XB* Kung-Lin Proposed cviIFC m=2.0 XB XB* Kung-Lin Proposed cviIFC
Dataset 1 c*=4
Dataset 2 c*=5
Dataset 3 c*=7
Dataset 4 c*=9
2-6 2-9 9 4,5
2-6, 11 2-7, 11 10 4-5
5 10 8 7
2, 8 4 9,10 9
2-6 2-8 9, 13 4
2-6, 11 2-9 10 5,6
3, 5, 6 8 6 6,7
4, 7,9 9-10 9,10 9
The results as shown in Table 3.6, which are obtained from the application of the proposed cviIFC on 4 different datasets (Figure 3.8 - Figure 3.11), indicate that cviIFC can successfully suggest the optimum number of clusters. It is also not effected by changing values of the levels of fuzziness, i.e., m=1.3 and m=2.0. It converges at the optimum number of clusters, and then it asymptotes to zero as the number of clusters goes to infinity, e.g., c=15 in this case. In all of the four datasets, the new validity criteria captures c* at the actual value, and then converges to zero as c>c*. The difference between cviIFC and XB* is that, proposed cviIFC is expected to show an asymptotical behaviour towards larger number of clusters, whereas XB* validity index can increase or decrease but c* should be where index is at its minimum value. When actual number of clusters are small, as in dataset1 and dataset2, XB* cannot directly identify c* for either level of fuzziness values. For larger c values, e.g., dataset3 in Figure 3.10, XB* can identify the actual number of clusters more precisely. We can conclude that when the number of different components (clusters, different patterns) in a given dataset are large, XB* can validate the IFC clustering applications to a certain degree. When the system has fewer models, XB* is not the optimum validity measure of number of clusters for the IFC method. As Kung-Lin (2004) cluster validity index asymptotes to zero, the knee-point is the indicator for the optimum number of clusters, c*. From the cluster validity graphs of four artificial datasets demonstrated in Figure 3.8-Figure 3.11, Kung-Lin index is unable to identify the optimum number of clusters for small-clustered datasets and is capable of identifying c* for datasets with larger number of clusters to a certain extent. In [Kim and Ramakrishna, 2005], it was proven that the XB index is inefficient to identify the optimum number of clusters, c*, of FCM clustering models. We wanted to test if it can identify the c* of the IFC models. The minimum XB values
102
3 Improved Fuzzy Clustering
indicate the predicted c*. XB index was unable to identify an exact c* in most of the datasets. It can however give a wider range which includes the optimum c*, however, this information is not enough to determine a precise value for the optimum number of clusters.
Analysis of Experiment 2. The optimum number of clusters was estimated to be c*∈[3,6] based on the models of T1IFF fuzzy system where proposed IFC is used for structure identification on real stock price dataset [Celikyilmaz and Turksen, 2007b]. Hence, we applied the proposed cviIFC function and averaged over 20 repetitions. In Figure 3.13 the values obtained from the cluster validity indexes of four different validity measures are shown using two levels of fuzziness values. The results are summarized in Table 3.7. The values of the proposed cviIFC indicate that c* should around 4 or 6 for two different m values. Kung-Lin index can also correctly validate c*. In addition, XB index can somewhat identify c* within the interval of c*∈[4-8]. Among 4 different cluster validity measures, the cviIFC has still the closest estimations to the actual c*. Table 3.7 Optimum Number of Clusters of IFC models of stock price dataset identified by different validity indices
Actual # of Clusters m=1.3 XB XB* Kung-Lin Proposed cviIFC m=2.0 XB XB* Kung-Lin Proposed cviIFC
Î
[3,6] 6-8 4, 6-8 4 4,6 5,7 5,7 4 4,6
Analysis of Experiment 3. In experiment 3, we wanted to demonstrate that the new cviIFC-C, specifically designed for the classification type datasets, could actually identify the optimum number of clusters of a real dataset, viz., ionosphere dataset from UCI repository. From exhaustive analysis based on models of IFF-C for changing c values and two different levels of fuzziness, we obtained that c* should be c*∈[3-5]. From the validity graphs shown in Figure 3.14, we obtained the optimum number of clusters from each cluster for two different fuzziness values and listed in Table 3.8. It is to be noted that cviIFC-C is able to identify overlapping clusters and the elbow indicates that optimum c* should be 4 or 6. Kung-Lin index could also identify the approximate c*. None of the other cvi measures were able to confidently identify c* for changing values of fuzziness.
3.5 Summary
103
Table 3.8 Optimum Number of Clusters of IFC models of Ionosphere dataset indicated by different validity indices
Actual # of Clusters m=1.3 XB XB* Kung-Lin Proposed cviIFC m=2.0 XB XB* Kung-Lin Proposed cviIFC
Î
[3,6] 6-9 8-9 6 6 4,7,8 7, 8 6 4
3.5 Summary In data mining practices, one of the challenges one encounters is that the datasets under study may have different properties, e.g., number of variables, number of data vectors, noise level, number of different model structures, the dependent variable characteristics, etc., that different approaches should be followed to get better results from the representative models of a system. Based on these characteristics of datasets, different fuzzy clustering algorithms have been introduced in the literature to find the hidden structures in them. If one designs a fuzzy clustering algorithm for an image processing dataset, which composes of the pixel intensities, the neighborhood pixel intensities might be one of the important measures that the clustering algorithm should utilize. Similarly, for a web log dataset, similarity based on recursive occurrences of corresponding words will be the most discriminative property. The fuzzy clustering should include corresponding similarity measures based on the type of the dataset under study. In this chapter different fuzzy clustering methods are reviewed and the structural differences between them are discussed. Most importantly, in this chapter a new improved clustering algorithm is proposed to be used for a new fuzzy system modeling approach, namely “Improved Fuzzy Functions” approaches. The logic behind proposed improved fuzzy clustering algorithm is that, feature spaces are not only related to input space but also to input-output mapping space, so the given data objects could be grouped by considering local models of the dataset as well as linear relationship between input and output variables. Therefore, new improved clustering algorithm combines regression type clustering algorithms to find membership values and their transformations that may help to explain local model relationships and basic clustering structure, which helps to a separate datasets into meaningful clusters at the same time. We also present an extension of the new fuzzy clustering method for classification problem domains.
104
3 Improved Fuzzy Clustering
This chapter also deals with one of the problems common to almost all of the clustering algorithms, that is to say, determination of optimum number of clusters. In current literature, since there are numerous fuzzy clustering methods, many different cluster validity measures are proposed. Especially there are numerous validity measures just to identify the optimum number of clusters of the wellknown Fuzzy C-Means Clustering algorithm. In this chapter two new cluster validity indices are introduced corresponding to the two new fuzzy clustering methods in order to validate their structures. Experimental analysis on artificial and real life datasets showed that the new cluster validity indices can help to identify optimum number of clusters of the models of the new improved fuzzy clustering methods. We preferred to present the experimental analysis on the new validity indices in this chapter instead of displaying them in the chapter of experiments. The main reason is that in chapter 4 and chapter 5 a new evolutionary system modeling structure based on proposed fuzzy functions approaches will be introduced to dynamically optimize the number of clusters based on stochastic search. Hence, the proposed approaches in Chapter 4 and Chapter 5 will be used to optimize the number of clusters with genetic algorithms.
Chapter 4
Fuzzy Functions Approach 4 Fuzzy Functions Approach
This chapter presents mathematical theory of the new “Fuzzy Functions” and “Improved Fuzzy Functions” approaches to build fuzzy system models for regression and classification problems using type-1 fuzzy sets.1 Additionally, Evolutionary Type-1 Fuzzy Functions and “Improved Fuzzy Functions” approach2 are presented. Mathematics seems to endow one with something like a new sense. Charles Robert Darwin (1809-1882)
4.1 Introduction This chapter introduces the theory of novel Type-1 Fuzzy System Modeling based on the “Fuzzy Functions” approach [Turksen, 2008; Celikyilmaz, 2005; Celikyilmaz and Turksen, 2007a]. Type-1 Fuzzy Systems utilize “first order” fuzzy sets. System modeling with higher order fuzzy sets, such as type-2 fuzzy sets will be investigated in the next chapter. Fuzzy systems based on Fuzzy Rule Bases (FRB) have been successfully used to model human problem-solving activities. A classical way to represent the human knowledge is with ‘IF…THEN’ fuzzy rules. The ‘IF’ part represents the antecedents (input fuzzy sets), and the ‘THEN’ part represents the consequents (output fuzzy sets). There are various FRB systems, based on the way the antecedent and consequent parameters are formed. These FRB system modeling strategies has been reviewed in the previous chapter (Chapter 2); hence we will not go into detail about the underlying methods here. One should note that in these systems, membership values of fuzzy sets represent different concepts such as the “degree of belongingness,” the “degree of fire”, the “degree of compatibility”, the “weight or strength of local functions or individual objects”. For instance in Fuzzy c-Regression methods [Hathaway and Bezdek, 1993], membership values are used as weights (strengths) assigned to each local function identified by the system 1
The first part of this chapter is an extension of [Turksen, and Celikyilmaz, 2006], [Celikyilmaz and Turksen, 2007a;b;e;i;k;2008e;f;g]. 2 The last section of this chapter is an extension of [Celikyilmaz and Turksen, 2007g]. A. Celikyilmaz and I.B. Türksen: Model. Uncertain. Fuzzy Logic, STUDFUZZ 240, pp. 105–147. springerlink.com © Springer-Verlag Berlin Heidelberg 2009
106
4 Fuzzy Functions Approach
model. Nonetheless, in this work a new representation of membership values are introduced to formulate the structure of “Fuzzy Functions” approaches. The novel fuzzy functions approach is initially introduced by Turksen in 2005 [Turksen, 2008]. Recently the algorithm was extended and combined with other soft computing approaches for performance improvement and automatic parameter identification [Celikyilmaz, 2005; Celikyilmaz and Turksen, 2007b,d,g,k]. The standard fuzzy functions approaches are multi-variable crisp valued functions which implement type-1 fuzzy sets. This chapter will explore these methods. In the next chapter, “Interval Type-2 Fuzzy Functions” will be presented with the new improvements on “Fuzzy Functions” theory. “Fuzzy Functions” approaches have emerged from the idea of representing each unique rule of an FRB system in terms of the ‘Fuzzy Functions’. One of their prominent features is that; the degree of belongingness of each sample vector in a fuzzy set has a direct affect on how the local fuzzy functions of the particular set are defined. Domain experts, who are not familiar with the theory of fuzzy logic, can build various fuzzy system models with “Fuzzy Functions” approaches instead of FRB systems, since “Fuzzy Functions” strategies offer reduced steps for identification of each rule and reasoning with them. Domain experts do not need to know fuzzy set operations, e.g., fuzzification or defuzzification, and fuzzy set operators such as t-norms and co-norms, modus ponens, etc. With the new “Fuzzy Functions” approach, one can build models for various system structures as with the other fuzzy system modeling tools. The goal of general system modeling depends on the type of the system under study. If the aim is to assign class labels to objects, such as in classification problems, the goal of system modeling is to reduce the number of misclassified cases. On the other hand if the problem involves estimation of a relationship between given independent variables and the dependent variable using functions, then the goal of a system modeling is to find a representation function that can minimize the prediction error. In this chapter, both of these system modeling techniques will be investigated using the “Fuzzy Functions” approaches. The constituents of the general system modeling frameworks [Uncu, 2003] are summarized as follows: • • •
•
Data Pre-processing: This step ensures that raw data extracted from data pool, e.g., data warehouse, is free from noise and missing values. Structure Identification: The training dataset is used to approximate the learning parameters of a mathematical model of the system to predict an output value or a label for a given object. Structure Tuning and Model Selection: In this section, parameters of the mathematical model are tuned by applying it on a validation dataset to measure its performance. This process is usually called the validation mechanism. Training and validation mechanisms are repeated several times for different sets of parameters, which, in turn, tune such parameters. The final model is selected when the desired error rate or computation time is reached. Structure Verification: This process verifies the performance of a final model using a testing dataset which has not been used during previous stages of system modeling.
4.2 Motivation
107
In every system modeling technique, one or more of these steps are implemented; however underlying theories of each step may vary from one strategy to another. Hence, in this chapter we introduce novel structure identification, tuning and reasoning algorithms for the “Type-1 Fuzzy Functions” approach. In this work, data pre-processing steps, e.g., data cleaning, selection of input variables, etc., are assumed to be completed in the first step of the fuzzy system modeling. Structure identification and parameter tuning are called the “training algorithm” and the structure validation is called the “inference algorithm (for testing data)”. Therefore we divided the system modeling with fuzzy functions method into two parts based on the two concepts; ‘training’ and ‘inference’ or ‘testing’. The fuzzy clustering methods form the basis of the fuzzy functions strategies. Fuzzy c-Means (FCM) Clustering algorithm [Bezdek, 1981a] is generally used in fuzzy functions approaches to obtain membership values. In chapter 3, the new Improved Fuzzy Clustering (IFC) [Celikyilmaz and Turksen, 2007b] is introduced and hypothesized that it would yield better performances than FCM clustering algorithm when implemented into the fuzzy functions system. Since FCM clustering and IFC require different initial parameters and processing steps, in this chapter we investigated Fuzzy Functions using FCM clustering and Improved Fuzzy Clustering (IFC) separately. Next, we will present our motivation of the fuzzy functions systems in comparison to earlier FRB systems.
4.2 Motivation In the earlier traditional fuzzy systems models, expert knowledge was encoded to define linguistic properties using fuzzy sets. However, these methods have major drawbacks of being subjective and suffer a lack of generalization. To reduce expert knowledge intervention and rather build self learning systems, more objective fuzzy system models are developed, e.g., [Babuška and Verbruggen, 1997; Emami et al., 1998; Kasabov and Song, 2002; Kilic et al. 2001; Zarandi et al. 2004]. Fuzzy sets in these methods are learned from data under study through optimization algorithms, such as fuzzy clustering. In recent studies more sophisticated optimization algorithms to build hybrid fuzzy systems, e.g., neuro-fuzzy algorithms, [Jang, 1993; Kasabov and Quin, 2002; Pedrycz, 2004], genetic fuzzy systems [Pedrycz and Reformat, 2003; Bodur et al. 2006, Cococcioni et al. 2007] are utilized for performance enhancement. Although the aforementioned approaches improve the efficiency of fuzzy systems, they have various challenges that should not be neglected [Turksen, and Celikyilmaz, 2006]. Let us investigate next the standard fuzzy systems based on fuzzy rule bases. Among some of their challenges are:
¾ identification of the types of antecedent and consequent membership functions, and their parameters,
¾ identification of most suitable combination operators (t-norm, t-conorm, etc.), conjunction operators during aggregation of antecedents, and consequents,
108
4 Fuzzy Functions Approach
¾ identification of implication operator types to capture uncertainty associ¾
ated with the linguistic “AND”, “OR”, “IMP” for the representation of the rules, and reasoning with them, identification of the type of defuzzification method.
Over the course of many years these challenges have been investigated and many different methods are used to optimize the parameters of these system models to reduce expert intervention, etc., [Vergara and Moraga, 1997; Wang and Mendel, 1992, Cococcioni et al. 2007]. Thus in this chapter we introduce a unique approach, entitled “Fuzzy Functions” approaches originally proposed by Turksen [2008], instead of FRB approaches. The aim is to reduce the number of fuzzy operators, e.g., fuzzification, aggregation of antecedents, implication, aggregation of consequents, optimization of antecedent and consequent membership function, etc. The “Fuzzy Function” strategy is designed to be less complicated than FRB systems since they will eliminate most of the aforementioned operations. In a somewhat simplified view, type-1 fuzzy functions approach works as follows: ¾ The domain x∈X ⊆ ℜ , nv dimensional input space, where X is the domain of x, is partitioned into c overlapping clusters, represented with cluster centers, vi, i=1,..,c. ¾ To each of these regions, a local fuzzy model fi: viÆℜ is assigned. Given some input data x∈X, the system then identifies one fuzzy output from each fuzzy model and then weighs these outputs based on the degree of belongingness (membership values) of the given input vector in each cluster. nv
Already, at this point, we ought to outline two requirements that need to be fulfilled in order to obtain a fuzzy system that requires a smaller number of fuzzy operators, and yet yield good prediction performance: (i)
Concerning the partition: The partition must identify the local models accurately so that during reasoning, the local functions could be as close to the true local relationships as possible. As a first step, each variable may be partitioned separately and the partition of the multivariate space can be obtained as the cross-product of the uni-variable partitions. However, in real life datasets, separating variables is usually impossible since variables are very much related and/or interact. Thus, it is more useful to consider them in combination. A simple example is using variables like “location” instead of using {“longitude” and “latitude”}, and similarly “speed” instead of {“distance” and “time to travel”}. Hence, in “Fuzzy Function” systems, we assume that each variable interact with each other and should be analyzed together by representing the antecedent part with the interactive membership values. This is the interactivity assumption of antecedent fuzzy sets, to be discussed in the following section.
(ii)
Concerning the complexity of the local models: Given the input variables are within a limited range in each cluster, the local models identify the behavior of the output variable. In order to improve the prediction performance, the local models should be close enough to the actual local relationships, as well as should have good generalization capability. Additionally,
4.2 Motivation
109
the local functions should not be complex such that they cause over-fitting. One should choose the optimum functions to identify the local models. In many cases, a small set of simple models should be sufficient. In our models a fuzzy clustering algorithm is executed to form the “Fuzzy Functions” system models and to identify hidden local structures. The “Fuzzy Functions” systems are unique because membership values obtained from fuzzy clustering and their transformations are used to explain the local fuzzy relationships. This is a new and novel implementation of membership values in fuzzy system modeling studies. In this respect, clustering algorithms should find improved membership values to enhance the prediction performance of the local “Fuzzy Functions”. Hence, the performance of “Fuzzy Function” system models relies on the fact that, in principle, the input/output behavior can be explained with each local “Fuzzy Function” structure, which poses a third requirement: (iii) Concerning the behavior of membership values: Given fuzzy partitions, Vi, (either univariate or multivariate), the fuzzy membership value calculation equations in “Fuzzy Functions” system models capture the fuzzy membership of belongingness of individual objects in fuzzy clusters together with their original independent variables to identify the behavior of an output variable in each cluster. Therefore, improved membership values should be extracted with the improved fuzzy clustering algorithm in (3.26) and (3.32). This is achieved by using intelligent fuzzy clustering methods discussed in Chapter 3 to improve the predictability of the output behavior of each “Fuzzy Functions” model. A common property of conventional Fuzzy Rule Base (FRB) approaches is the non-interactivity of antecedent fuzzy sets. In the earlier FRB systems, a separate membership function for each explanatory variable in each rule is identified by domain experts, usually using triangular membership functions. However, each input variable affects the output variable in relation to the other input variables. One should analyze the input variables together while trying to estimate a function to explain the behavior between inputs and outputs. One way to approach this issue is by assuming that the input variables interact with one another; so their antecedent fuzzy sets should be analyzed as a whole. Several structure identification methods are proposed that address this concept, e.g., [Delgado et.al., 1997; Babuska and Verbruggen, 1997]. In these approaches, it is assumed that there is interactivity between antecedent fuzzy sets. Hence, using clustering methods, a single membership function is identified for the entire antecedent part of fuzzy rules instead of separate membership functions for each variable. Input membership functions are defined by projecting the membership function obtained from different domains, e.g., input-output, input or output domains, onto input and output domains, separately. In this work, the “Fuzzy Functions” approach also assumes interactivity between each input variable. The new “Fuzzy Functions” system models and traditional fuzzy system models of FRB structures and their extensions [Bodur et al. 2006; Jang, 1993; Kasabov and Song, 2002; Kilic et al. 2001; Kim et al. 1998] share similar system design steps, as shown in Figure 4.1, but they differ in structure identification
110
4 Fuzzy Functions Approach
Reasoning with Fuzzy Functions Fuzzy System Modeling with Fuzzy Functions
System IdentificationTraining Engine
Input variables and Membership value Structure Identification
Parameter Identification
Rule Generation with Fuzzy Functions Inference Engine
Fig. 4.1 Framework of fuzzy system models with “Fuzzy Functions” approach
technologies, namely in finding the fuzzy models (rules) for each pattern identified [Turksen, and Celikyilmaz, 2006] and inference methods. The new “Fuzzy Functions” approach first clusters the given data into several overlapping fuzzy clusters, each of which is used to define a separate decision rule. Initially Fuzzy cmeans clustering (FCM) has been the clustering algorithm utilized in these methods to find fuzzy partitions. The novelty of the “Fuzzy Functions” approaches is that, similarity of objects are enhanced with additional fuzzy identifiers viz. membership values and their transformation, by utilizing them as additional predictors to estimate local relations of input-output data along with original input variables. Thus, membership values and their list of possible (user-defined) transformations are augmented to original dataset as new predictors to identify the structure of the different datasets for each cluster. Using the datasets of each cluster, local functions are identified to explain the local input-output relationships. These functions are named the “Fuzzy Functions” as originally proposed by [Turksen, 2008]. In system modeling approaches, where similarities are based on distances between vector objects, membership values play a crucial role in shaping these “Fuzzy Functions” of each cluster [Celikyilmaz and Turksen, 2007a]. Earlier research [Celikyilmaz. 2005; Turksen and Celikyilmaz, 2006], has pointed out the importance of these membership values on fuzzy models. Results from these studies indicate that Fuzzy Functions strategies can minimize the error between the system output and the model output better than traditional FRB approaches. These systems implemented the Fuzzy C-Means (FCM) clustering algorithm [Bezdek, 1981a] into structure identification. In the experiments section, the results of the investigations of performance improvements are presented. In this regard, one might argue that the FCM clustering algorithm may not be the optimum clustering method. This is because the FCM clustering method is not designed to find optimal membership values, which can, at the same time, explain the behavior of input and output variables in local models. In response, in chapter 3, we introduce a new fuzzy clustering algorithm, namely Improved Fuzzy Clustering (IFC), which achieves two objectives: ¾ ¾
To find a good representation of the partition matrix which captures the multiple model structures of the given system by identifying hidden patterns, To identify the membership values obtained from these “Fuzzy Functions” those improve the representation of the local input-output relationships with each local model.
4.2 Motivation
111
The IFC method ensures that the membership values can help to predict the relationship between independent and dependent variables in local structures. As a result of this improvement, the new IFC identifies better membership values for regression and classification (pattern analysis) models as in (3.30) and (3.38). Recall that the improved fuzzy clustering models can be applied on regression system domains (IFC) or for classification system domains (IFC-C). We hypothesize that membership values obtained from improved clustering algorithms and possibly their transformations can increase the prediction power of “Fuzzy Functions” of each cluster, when used together with original input variables to explain the behavior of the input and output variables in local models. In this sense, the resulting Fuzzy Functions are referred as “Improved Fuzzy Functions” and this approach is denoted as Type-1 Improved Fuzzy Functions, in short T1IFF. Type-1 Improved Fuzzy Functions for classification problem domains is represented by T1IFF-C. The novel T1IFF approach introduces a structure identification (training) and inference (reasoning) method, different from the training and inference methods of Type-1 Fuzzy Functions using FCM clustering models, which will be denoted with T1FF. The membership value calculation equation of the new IFC or IFC-C requires additional prior information, so that the training method of T1IFF is different than that of T1FF. Consequently, the new inference algorithm of T1IFF and T1IFF-C implement a semi-parametric case-based inference method to estimate the membership values of new data vectors different from the T1FF approach. Previous fuzzy inference system models, in other words fuzzy system models based on fuzzy rule bases (FRB), require various initial parameters to be set prior to execution. Recent publications [Ishibuchi et al., 1999; Tettamanzi and Tomassini, 2001; Pedrycz and Reformat, 2003; Roland, 2003] have shown that evolutionary algorithms to optimize the FRB parameters such as the shape and number of fuzzy rules and the type of the rule base operators mentioned above have major impact on the performance of fuzzy systems based on fuzzy rule bases. Similarly, an important characteristic of “Fuzzy Functions” system models is that they require a few key parameters that may affect the performance of a system model. A procedure that learns the Fuzzy Functions system model automatically from data must respect the properties of such parameters. In this work an evolutionary algorithm method is applied to encode these parameters as hybrid genetic code. Then “Fuzzy Function” models are evolved to capture optimum system parameters based on a stochastic search3 method. Hence, a new Evolutionary Type-1 fuzzy structure identification and inference method are proposed by combining evolutionary algorithms with improved fuzzy functions. The stochastic optimization model within the “Fuzzy Functions” approach would reduce the number of 3
The evolutionary algorithms are stochastic search methods because of the fact that certain steps are based on random choice. Initial population is randomly assigned. The individuals to go through cross-over or mutation operations are randomly selected. Some cross-over or mutation operations are based on probabilistic approach. Due to the fact that the probability of the evolutionary process depends on the stochastic process but not solely on the data, there will be different results of the same evolutionary method in two consecutive runs. A brief definition of genetic algorithms, a type of evolutionary computation, is presented in Appendix C.3.
112
4 Fuzzy Functions Approach
iterations to find the optimum model compared to “Type-1 Improved Fuzzy Functions” based on an exhaustive search method to optimize these parameters. The aforementioned Type-1 Fuzzy Functions and its variations are demonstrated in one graph as shown in Figure 4.2.
Evolutionary Type 1 Fuzzy Functions
Evolutionary Type 1 Improved Fuzzy Functions
Type 1 Fuzzy Functions
Type 1 Improved Fuzzy Functions
FCM
Implementation of Improved Membership Values
IFC
Fig. 4.2 Evolution of Type-1 Fuzzy Functions Strategies
Firstly, we introduce Type-1 Fuzzy Function system models based on the FCM clustering algorithm – T1FF (middle part of inverted triangle on the left side of Figure 4.2). Later, improved Fuzzy Functions System modeling using the novel Improved Fuzzy Clustering (IFC) method – T1IFF– is introduced (middle part of the inverted triangle on the right side of Figure 4.2). To optimize the systems parameters of both methods, hybrid fuzzy function methods are applied, entitled Evolutionary Type-1 Fuzzy Functions –ET1FF– and Evolutionary Type-1 Improved Fuzzy Functions –ET1IFF–, using an evolutionary algorithm based on a stochastic search method to solve fuzzy data mining problems. In this chapter these four methods will be briefly discussed. In the following, we start with the definition of the design principles of the proposed Fuzzy Functions algorithm using FCM clustering for regression –T1FF– and classification models –T1FF-C–, separately. Then, the architecture of the training and inference steps of the new T1IFF using IFC algorithm for regression and T1IFF-C using IFC-C for classification problem domains will be discussed. In the last part, a new approach to optimize the parameters of T1FF and T1IFF approaches will be presented using Evolutionary Algorithms. These methods will be denoted with ET1FF and ET1IFF for the regression domains and ET1FF-C and ET1IFF-C for the classification models.
4.3 Proposed Type-1 Fuzzy Functions Approach Using FCM – T1FF 4.3.1 Structure Identification of FF for Regression Models (T1FF ) Fuzzy function systems have two separate components: training and inference algorithms. During the training algorithm, the system model is identified using a training
4.3 Proposed Type-1 Fuzzy Functions Approach Using FCM – T1FF
113
dataset, randomly selected from the entire dataset, and the parameters are optimized using another randomly selected sample dataset called validation dataset. During the inference algorithm, the testing dataset, which is also randomly selected from the entire dataset, is used to measure the model performance. A datum can only be present in one of three partitions. This process is sketched in Figure 4.3. As observed from several articles listed in the references, a common way to optimize the parameters of fuzzy systems is to separate the datasets into several randomly selected training and testing subsets. The training dataset is used to learn the system structure and optimize the parameters. Similarly, the testing dataset is used to measure the modeling performance. This process is usually repeated several times with different random subsets to form training and testing datasets. The overall performance is calculated by averaging the performances of the models applied on the testing samples in these repeated experiments. In this work, a subsampling cross validation method, which is slightly different from the standard cross validation methods, is implemented to optimize system model parameters using a separate dataset from the training set, as shown in Figure 4.3. We use an additional subset called validation dataset, which is different from the training and testing datasets. The validation dataset is used to optimize the model parameters obtained from the training dataset. Also, the testing dataset is neither used for training nor for validation purposes: it is only used to measure the performance of the optimum model. This way, we can ensure an unbiased capacity (performance) for our models. After the optimum fuzzy function model parameters are obtained using training and validation datasets, to be discussed next, overall performance of models is measured on a separate testing dataset. In order to measure the robustness of the models, we repeat the above learning and optimization processes several times, using different subsets randomly selected from the entire dataset. c Training Dataset
m,c
Structure Idendtification with FCM
.
.
1 Fuzzy Function per cluster
No Validation Dataset
Inference
max performance optimum model parameters YES
Testing data semi-non-parametric inference
Fig. 4.3 Modeling with Fuzzy Functions Systems
Scoring the model with optimal parameters
114
4 Fuzzy Functions Approach
Multi-input, single-output (MISO) problems are the main interest of this work. We begin with structure identification of fuzzy function strategies for regression type problems. Later the same strategy will be extended for classification type problems. Let Z(x,y)={(x1,y1), (x2,y2),…, (xn,yn)} represent the input-output dataset, where nv+1 z(xk,yk)⊂ℜ denotes any data point (vector) from the training set, and every data nv point is composed of (nv+1) dimensions of inputs vectors, xk=(x1,k,…,xnv,k)∈ℜ , k=1,…,n, a total of n vectors, and an output, yk∈ℜ. Z is the (n×(nv+1)) input-output matrix, n is the total number of data vectors, i, i=1…c, is the cluster identifier, c represents the total number of clusters identified, and m is the degree of fuzziness (i.e., overlapping degree) parameters of the Fuzzy C-Means (FCM) clustering th method. Let μik ∈[0,1] represent the membership value of the k datum in cluster i. Therefore the list of parameters of the training algorithm is given by the; • • • •
number of clusters of the system model, c, which are discrete values such as c∈{2,3,… n(1/℘)}, 0<℘
The programming steps for T1FF systems for regression models of a MISO domain is displayed in ALGORITHM 4.1. ALGORITHM 4.1 Training Algorithm for the Type-1 Fuzzy Functions Approach ( T1FF ) using Standard FCM clustering algorithm Step 1: Choose FCM clustering parameters, m≥1.1 (degree of fuzziness) and c>1(the number of clusters) and ε (a termination threshold). Step 2: Execute FCM using Z(x, y) to find the cluster centers, υi(xy), and interactive (I/O) membership values for m and c by;
⎛ c ⎛ d ( xy ) ⎞ ∀ μik ( xy ) = ⎜ ∑ ⎜ ik ⎟ 1≤ i ≤ c d ( xy ) ⎜ j =1 jk ⎠ 1≤ k ≤ n ⎝ ⎝
2 /( m −1)
⎞ ⎟ ⎟ ⎠
−1
, where d ikxy = ( xk , yk ) − vi ( x, y ) .
Step 3: Find membership values of the input space using −1
2 /( m −1) ⎛ c ⎛ d ( x) ⎞ ⎞ ⎟ , where dik ( x) = xk − vi ( x) . ∀ μik ( x) = ⎜ ∑ ⎜ ik ⎟ 1≤ i ≤ c d jk ( x) ⎠ ⎜ j =1 ⎝ ⎟ 1≤ k ≤ n ⎝ ⎠
Step 4: For each cluster i, 4.1. Membership values of each input data sample, μik and their user defined transformations in ℜ nm space is augmented to the original input space to map the original input matrix onto a new feature space ℜnv+nm , i.e., xÆΦi(x,μi), for each cluster i. 4.2. Estimate the parameters, Ŵi={Ŵ0 ,…,Ŵnv+nm}, of each fuzzy function fi(Φi,Ŵi) of each cluster.
4.3 Proposed Type-1 Fuzzy Functions Approach Using FCM – T1FF
115
The novel training algorithm in ALGORITHM 4.1 applies standard FCM [Bezdek, 1981a] clustering method on the input-output data, z(xk,yk), k=1…n, to generate membership values μ(x,y) and cluster centers, vi(x,y), i=1…c. In step 3, we obtain membership values corresponding to input space, μi(x), and cluster centers of the given input space, denoted by vi(x). It should be noted that membership values are not projections of the clusters onto each input space of the input variables, but rather induced on the whole input space by fuzzy clusters [Babuska and Verbruggen, 1997]. Due to interactivity assumption between input variables, we do not map the membership values onto every input variable. Therefore there is one membership value assigned to each input vector to represent the whole input space. In step 4, a different dataset is formed for each cluster i by using membership values, μi(x) and/or their transformations as additional dimensions. This is the n nv same as mapping the original input space, x∈ℜ × , of nv input variables, x={x1 n (nv+nm) …xnv}, onto a higher dimensional feature space ℜ × , i.e., xÆΦi(x,μi), for nv+nm each cluster i, i=1,..,c. Each data vector is represented by Φik∈ℜ , which is a matrix composed of the original input variables of nv dimensions and membership values and their (nm) possible different transformations. The feature space dimension (nv+nm) is determined by the user and the optimum dimensions are sought based on exhaustive search method. And, nm is the number of augmented membership values appended to the original input space as new dimensions. Optimum regression function parameters are sought in this new space. In the following sections, with the implementation of the evolutionary algorithms, the feature space dimension will be determined automatically based on the system model performance. These approaches will be explained in the following sections of this chapter. A special case of this approach is shown in Figure 4.4 using a Toy dataset of n data points, of a single input and single output, where only membership values are nv nv+1 used as an additional dimension, i.e., ℜ Æℜ , nm=1, to form a new dataset in each cluster as follows: ⎡ μi ,1
μik > α - cut , i=1.. Φ i ( x , μi ) ∈ℜnv +1 = ⎢⎢ M
⎢ μi , p ⎣
x1×1 M x1× p
xnv×1 ⎤ ⎥ M ⎥ L xnv× p ⎥⎦ K M
μik > α - cut 0< p≤n
(4.1)
c, k=1...p
For this example, a nonlinear regression function (i.e., support vector regression) is fit for each cluster in (nv+1+1) space, i.e., one additional dimension for the intercept. A prominent feature of the fuzzy functions approaches is that, if the relations between input variables and the output can not be defined in the original nv+nm space using additional space, we can explain their relationship in the ℜ information induced by the membership values and their transformations. Note from (4.1) that, for each cluster a different subset is obtained by the following constraint:
116
4 Fuzzy Functions Approach
μik >α-cut, i=1..c, k=1..p
(4.2)
Using an alpha-cut value, α-cut>0, we eliminate vectors from each cluster that do not affect the decision surfaces, viz, far away from the cluster centers. The α-cut in general eliminates anomalities generated by the clustering algorithms. In the experiment we usually use α-cut=0, when the number of data points in one cluster is below n/(#clusters). The training algorithm would focus on vectors that can support the decision surfaces, in other words, the data points that can explain the local input-output relationships with less error will be retained to find the fuzzy functions. Hence, for each local model i, functions are shaped using different datasets, Φi(x,μ i), the dimensions of which are determined by the user. Again, the optimum values of the parameters that define the dimensions of each dataset of each cluster, viz., nm and α-cut, can be automatically determined by the proposed evolutionary fuzzy functions, to be discussed in the following sections of this chapter. 4
y
2
0
-2 -2
0 Input (x)
Output (y)
0 -5 1
2
0.5 membership
0 -2
1
2
Cluster2
5
Cluster1
5 Output (y)
-1
0 Input (x)
0 -5 1
2
0.5 membership
0 -2
0 Input (x)
Fig. 4.4 Mapping the Input-Output space onto individual clusters using a one dimensional Toy Example. Support Vector Regression is used to find the decision surfaces for two clusters in (1+1) space, i.e., only membership values are used as additional dimensions.
In step 4.2 of ALGORITHM 4.1, one regression function f(Φi,Ŵi) is identified for each cluster, i, i=1…c. These functions are called the “Fuzzy Functions” of type-1[Turksen, 2008, Celikyilmaz, 2005, Celikyilmaz and Turksen, 2007a]. One can map the original input matrix onto a higher dimension by using various forms of membership functions as additional dimensions, depending on the non-linearity
4.3 Proposed Type-1 Fuzzy Functions Approach Using FCM – T1FF
117
of the dataset. Where appropriate, we usually use mathematical transformations of 2 m membership values such as μik , μik , exp(μik), ln((1-μik)/μik), etc., where m represents the degree of fuzziness. Our extensive research indicates that the exponential and various logarithmic transformations of membership values can improve the performance of the system models better than several others. However, one should apply as many forms of fuzzy membership values as possible to improve the performance. In addition, one could use any regression function type. In this work, the simple least squares estimation (LSE) and the more sophisticated support vector regression (SVR) [Smola, 1996; Gunn, 1998] methods are implemented using different kernel functions to estimate fuzzy function parameters for each cluster, f(Φi,Ŵi), where Ŵi={Ŵi,0,Ŵi,1…Ŵi,(nv+nm)} are the parameters of these functions for each cluster i. When LSE is used to find the parameters of the fuzzy functions, Ŵi represents the linear regression parameters as follows:
) { (
(
)
(
)
f Φi ,Wˆi = Φi x , μi | Wˆ0,i + Wˆ1,i ( μi ) + " + Wˆnm,i μip >0 + Wˆnm+1,i x1 + " + Wˆnm+nv,i xnv
}
(4.3)
represents input space, which composes of a column vector Each Φi(x,μi)∈ℜ of 1’s for the intercept, membership values of the ith cluster and their possible transformations as separate nm number of columns, and input matrix x (n×nv) of the nv variables with n data vectors. On the other hand, if support vector regression (SVR) is used to approximate nv+nm
regression functions of the local models instead of the LSE approach, Ŵi represents a more complex structure including the Lagrange multipliers of the support vectors obtained from each fuzzy partition and selected support training vectors for each cluster. Hence, the SVR optimization model for each fuzzy function of each cluster is represented by:
(
)(
)
max L = − 12 ∑ k ,l =1 α ik* − α ik α il* − α il K Φ ( xk , μ ik ) ,Φ ( xl , μ il ) n
i
(
)
(
)
− ε ∑ k =1 α ik* + α ik + ∑ k =1 α ik* − α ik yk n
(
n
)
⎧ ∑ n α * − α = 0, ik ik ⎪ subject to ⎨ k =1 * ⎪⎩ α ik ,α ik ∈ [0,Creg ]
(4.4)
k ,l = 1,...,n ,Creg > 0,i = 1,...,c
In (4.4), the SVR [Smola, 1996; Gunn, 1998] optimization algorithm is applied separately to each cluster i to find the support vectors and their corresponding La* grange multipliers, (αik,αik ), k=1,…,n. Some data vectors are eliminated from the training datasets of each cluster based on the criteria in (4.2) viz., vectors with μik>α-cut values, before the SVR model execution. In that case the total number of vectors of each cluster would be less than the total number of training vectors. When SVR is used to estimate function parameters, each fuzzy function of the system model is represented by:
118
4 Fuzzy Functions Approach n
(
)
fi ( Φi ) = ∑ α ik − α ik* K Φ ( xk , μik ) ,Φ ( x , μi ) + bi k =1
(4.5)
The number of initial parameters that one should determine prior to execution of the T 1FF method changes based on the function approximation type used. If one uses LSE to approximate the fuzzy function parameters, then there are only three initial parameters; m (degree of fuzziness), and c (the number of clusters) from the FCM clustering and the structure of fuzzy functions. Whereas if SVR is used to model the fuzzy functions, on top of those three initial parameters, four additional initial parameters, i.e., C-regularization constant (C-reg), ε-margin for epsilon insensitive regression, kernel type, and parameters for non-linear SVR, are also required to be determined by the expert. Therefore, optimizing the T1FF approach with SVR methods would take a longer time than T1FF methods when LSE is used. One could use any other regression method such as kernel regression, or weighted least squares regression to estimate fuzzy function parameters. This work only makes use of two methods to demonstrate the implementation, a simple and a more complicated learning method into the novel T1FF methodologies. Depending on the structure of the dataset in hand, one should choose an appropriate function estimation method. Recall that the structure identification of T1FF system models requires fewer operators when compared to FRB systems. One reason being is that the list of operations and operators starting from fuzzification, aggregation of input fuzzy sets, implication, aggregation of output fuzzy sets, and finally defuzzification steps are replaced with easily interpretable steps, summarized as follows: ¾ ¾ ¾
clustering the given dataset and finding the type-1 membership functions for each cluster, forming sub-datasets for each cluster by using data points that have membership values μik>α-cut and additional predictors using membership values and their transformations as input vectors, estimating one function, linear or non-linear.
The challenge with T1FF systems, as in FRB systems, is that one should apply a search method such as an exhaustive search to optimize each parameter. Even though an optimum model would be certain when an exhaustive search is applied with higher precision, the searching process takes time, which may not be a desirable solution for many system domains. In this work, we applied an evolutionary algorithm to optimize the parameters of T1FF systems. The parameters of T1FF system that are optimized by the evolutionary method –ET1FF– are the degree of fuzziness and number of clusters to shape membership values, the α-cut, the structure of fuzzy functions and their parameters to shape local system model structures. The aim for applying evolutionary algorithms for optimization of the parameters of fuzzy functions strategies is to reduce the number of iterations it takes to optimize them.
4.3 Proposed Type-1 Fuzzy Functions Approach Using FCM – T1FF
119
4.3.2 Structure Identification of the Fuzzy Functions for Classification Models (T1FF-C ) The structure identification of T1FF systems for classification problems, T1FF-C, is similar to the structure of T1FF systems for the regression problems. The only difference between them is that the output variables in classification problems have a discrete domain, y={0,1,2…}, whereas in regression/prediction domains the output variable has a continuous domain, y∈ℜ. So the fuzzy function types that are used to approximate decision surfaces would be different in T1FF-C than T1FF systems. In this work we only deal with binary classification problems. Hence, output variables only take on dichotomous values e.g., y={-1,+1}. During the reasoning (inference) method, we obtain posterior probabilities of data vectors as a result of classification models and make decisions with these model posterior probabilities based on an optimum threshold value. The optimum threshold is captured whilst sub-sampling cross validation and is used to measure recognition performance of the testing data vectors. Among many different classification functions, we use a linear classifier and a sophisticated non-linear classifier in order to approximate fuzzy classifier functions and obtain posterior probabilities of each input vector in each local structure. Hence, in this section, we will shortly review these well-known classifiers and calculations of posterior probabilities of the dataset instances. Structure Identification of T1FF-C Using Posterior Probabilities from Logistic Regression (LR) Given a training dataset of n samples, Z(x,y)={(x1,y1),…(xn,yn)}, where the binary output is represented with, yk∈{0,1}. Firstly, the FCM clustering algorithm is applied on the training dataset to obtain membership values and cluster centers, as in Step 1-3 of ALGORITHM 4.1. Then, the training data is mapped onto a feature space as such; xÆΦi(x,μi), for each cluster i, i=1,..,c. Each new input matrix Φi(x,μi) of a cluster i such as in (4.1) comprises of the original input variables and membership values of the corresponding cluster i and their transformations. We also eliminate those data vectors from each cluster based on the following criteria, μik>α-cut≥0. At this point, instead of building a linear regression model for each dataset of each cluster, Φi(x,μi), a logistic regression (LR) function is applied. LR tries to estimate a posterior probability, Pˆi (yk=1|Φik), for each vector k, xk, indicating the probability of a datum k belonging to the class of positives, yk=1. The posterior probabilities of any xk in each cluster i is measured by: T Pˆ (y =|Φ (x ,μ )) = 1/ (1 + exp(-(Ŵ ,+Ŵ Φ ))) (4.6) i
k
i
k
ik
0,i
i
i
Ŵi={Ŵi,0…Ŵi,(nv+nm)} is the list of coefficients of the logistic regression function of cluster i, where Ŵ0,i refers to the intercept, and Ŵj,i, j=1…nv+nm, are the coefficients of each input variable. These are composed of the original input variables and the membership values of cluster i and their transformation, Pˆi (yk=0|Φik)=1Pˆ (y =1|Φ ). Parameters Ŵ and Ŵ are estimated using the maximum likelihood i
k
formula.
ik
j,i
0,i
120
4 Fuzzy Functions Approach
Structure Identification of the T1FF-C Using Posterior Probabilities from the Support Vector Classification (SVC)
If support vector classification (SVC) is used to estimate fuzzy classifier function parameters, the output values of data vectors are calculated by fi(Φi)=∑k βik yk K(Φik,Φi)+bi , i=1...c, k=1,…n
(4.7)
In (4.7), K(.) is the kernel function that is used to map original data vector onto a higher dimensional space, either linearly or non-linearly. In this work, we anaT lyzed two different kernel functions: linear, K(xk,xl)=xk xl, and probabilistic kernel,
{
i.e., the radial basis function (RBF), K ( xk , x j ) = exp −δ xk − x j
2 2
} (δ>0). And, β
ik
in (4.7) represents Lagrange multipliers, one for each training data vector, which are introduced to solve the following SVC optimization algorithm for each local model i (cluster):
Max Qi = ∑ k =1 βik − 12 ∑ k ,l =1 βik βil yk yl K (Φ i ( xk , μik ) ,Φ i ( xl , μil ) ) n
s.t. ∑ k =1 βik yk = 0 , n
n
(4.8)
0 ≤ βik ≤ Creg , k ,l = 1,...,n,i = 1...c
where Creg is the regularization constant, that balances the complexity of the machine, and the number of the separable data vectors. Training vectors with non-zero coefficients, βik>0, (Lagrange multipliers) are called “support vectors” obtained for each cluster i. The fewer the support vectors, the better is the generalization capacity of such models. A brief summary of support vector machines for classification (SVC) methods are given in Appendix C.2. During the learning phase of the proposed T1FF-C method, one fuzzy classifier function is identified for each cluster obtained from FCM clustering algorithm. The output values obtained from each SVC fuzzy function is a scalar value that one needs to transform into posterior probabilities. We measure the posterior probabilities using the improved Platt’s probability method [Platt 2000; Lin et al., 2003], which is based on the approximation of model output labels with a sigmoid function as follows: -1 Pˆik (yk =1| fi(Φi)) = (1+exp(a1 f(Φi) + a2)) ,
(4.9)
where a1 and a2 are found by minimizing the log-likelihood of training data:
min a1 ,a2
y +1⎞ n ⎛ y +1 ⎛ ˆ − ∑ k =1 ⎜ k log( Pˆik ) + ⎜1 − k ⎟ log( 1 − Pik 2 2 ⎝ ⎠ ⎝
⎞ )⎟ ⎠
(4.10)
The output values obtained from SVC fuzzy functions are not represented as class labels, e.g., yk={-1 or 1} , but rather represented with estimated posterior probabilities. The optimum threshold value of a specific T1FF-C model will be
4.3 Proposed Type-1 Fuzzy Functions Approach Using FCM – T1FF
121
determined based on the estimated posterior probabilities obtained from the validation dataset during structure identification. The optimum threshold values obtained during the structure identification are used to score testing datasets during performance evaluation of each T1FF-C model. The algorithm to obtain the optimum threshold value will be introduced in the next section in Algorithm 4.3.
4.3.3 Inference Mechanism of T1FF for Regression Models In supervised learning strategies a common method for inference is to implement a case-based (instance-based) reasoning methodology. In case-based reasoning, knowledge is represented in terms of specific cases or experiences. The reasoning relies on flexible matching methods to retrieve these cases and applies them to new situations. In a very simple case-based reasoning, some descriptive nearest neighbors- according to some distance measure- are captured during the training algorithm to be later used for reasoning. A recent addition to cased-based reasoning algorithms is Support Vector Machines (SVM)4 [Vapnik, 1995]. In SVM, only cases that define interclass boundaries in a newly transformed feature space are retained after learning; these become the support vectors upon which inference occurs. The proposed inference method uses selected training cases to infer the output values for the new cases. In standard exhaustive search methods, parameters of an optimum model are sought by iteratively executing the same methodology for the different values of its parameters. In this work, the structure identification to learn the system models is applied on the training dataset. The inference method is used to calculate the performance of those models captured during structure identification by using the provided datasets, i.e., validation or testing datasets. The validation dataset is used to optimize the model parameters during sub-sampling cross validation that it is a part of the structure identification method. The optimum model is obtained based on exhaustive search. The selection of the optimum model is based on the best performance obtained from inference (reasoning) methodology. Then, using the parameters of the optimum model, we test the generalization capacity of the model with a testing dataset. v v v v th Let the validation dataset be represented as X ={x1 ,x2 …,xndv }, and every k data v v v nv vector is composed of (nv) dimensional input vectors, xk =(x1,k ,…,xnv,k )∈ℜ , and an v v output which we do not know beforehand, yk ∈ℜ where X is the (ndv×nv) input matrix, ndv is the total number of validation data vectors, i, i=1…c, is the cluster identifier, c is the total number of clusters identified, and m is the degree of fuzziness parameters of Fuzzy C-Means (FCM) clustering method. On the other hand, test test test test the testing dataset is represented with X ={x1 ,x2 ,…,xnte }, and every testing data test test test nv vector is composed of (nv) dimensions of vectors, xk =(x1,k ,…,xnv,k )∈ℜ , and an test test output variable, yk ∈ℜ. X is the (nte×nv) input matrix, nte is the total number of testing data vectors. In the Fuzzy Functions reasoning mechanism one wants to obtain a crisp output value from the approximated model and the estimated output values along with the actual output values of corresponding vectors are used to
4
A background of Support Vector Machines for regression and classification models are presented in Appendix C.2.
122
4 Fuzzy Functions Approach
calculate the performance of the corresponding model, as shown in ALGORITHM 4.2. Here, we used the validation dataset to explain the new inference mechanism. ALGORITHM 4.2 Inference Algorithm of the Type-1 Fuzzy Functions Approach ( T1FF ) using Standard FCM clustering algorithm v
Step 1: Find input membership values of each validation sample, xk ,k=1,..,ndv using
⎛⎛ c v d ∀ μikv = ⎜ ⎜ ∑ ikv 1≤i ≤c ⎜ ⎜ j =1 d jk 1≤ k ≤ ndv ⎝⎝
⎞ ⎟ ⎟ ⎠
2/( m −1) −1
⎞ ⎟ , i=1,…,c, where dikv = xkv − υi ( x ) . ⎟ ⎠
Step 2: Membership values of validation data, μik and their user defined transformations optimized during training algorithm, are used to map the original validation data onto v v nv+nm v space. each cluster x ÆΦi(x ,μi ), in ℜ v
Step 3: Infer output value of new data vectors using Fuzzy Function parameters, ŷik=f(Φik,Ŵi) using equation (4.3) or (4.5) depending on the type of the model.
Step 4: Find single output value for validation data samples by weighting inferred fuzzy output values from each cluster with their corresponding membership values:
∑ ˆyik μik ,i = 1...c,k = 1,...,ndv ˆyk = i c ∑ i μik c
The first step of the inference algorithm of the T1FF system, as shown in ALGORITHM 4.2, is fuzzification. Here, membership values of each of the validation data v c nv samples, x , is calculated using cluster centers, v(x)={v1(x),…, vc(x)}∈ℜ × obtained from the standard FCM clustering algorithm for a given m and c values. This is interpreted as finding the membership values of testing cases in each cluster during the learning stage. These membership values are used as additional dimensions of original validation input matrix. Hence, original validation matrix is mapped onto a user v v nv+nm defined feature space as determined in the training algorithm i.e., x ÆΦi(x ,μi)∈ℜ , to form the datasets of each cluster. The fuzzy functions of each cluster i, f(Φi,Ŵi) obtained from the training algorithm, are used to infer output values of the validation samples. Hence, for each cluster, one fuzzy model output, ŷi = f(Φi) is obtained for each data point of each dataset of each cluster in the feature space. In order to obtain a crisp output, each fuzzy output is weighted with the membership values by using the fuzzy weighted average formula as follows:
∑ i ˆyik μikv c ∑ i μikv c
ˆyk =
(4.11)
4.3.4 Inference Mechanism of T1FF for Classification Models The proposed inference mechanism of the T1FF-C approach for classification problems is similar to the inference mechanism of the T1FF systems for the regression problems. The only difference is in Step 4 of ALGORITHM 4.2, where decision
4.3 Proposed Type-1 Fuzzy Functions Approach Using FCM – T1FF
123
functions are used to obtain estimated output values. Here, posterior probabilities of ˆ (y =1|f (Φi)), are obtained from the fuzzy each data sample k, xk, in each cluster i, P i ik k functions using (4.6) or {(4.7) and (4.9)} depending on the model type used in the training algorithm. An estimated crisp posterior probability is obtained for each verification vector by the fuzzy weighted average method as follows:
ˆ = ∑i P k
c
ˆ μ P ik ik
∑ i μik c
(4.12)
The objective of building classification models is to predict a class label, ŷk=(-1) or ŷk=(+1), for each data object which represent either one of the two dichotomous outputs. Based on the estimated class labels, we can measure the performance of a model in terms of accuracy. However, the output values we obtain from the T1FFC inference method are posterior probabilities, as in (4.12). Some performance measures such as accuracy (to be discussed in the chapter on experiments) require estimated class labels instead of probabilities. In order to obtain a crisp output label such as ŷk=(-1) or ŷk=(+1), one needs to apply a discretization method. In order to apply discretization, one needs to determine a threshold value, so that data vectors with estimated probabilities above some threshold would belong to the class of ŷk∈(+1), or otherwise would belong to the class of ŷk∈(-1). The threshold is especially required to measure the performance of the testing dataset using area under the receiver operating curve analysis in chapter of experiments, Chapter 6. After estimated posterior probabilities are obtained from structure identification of the T1FF-C method, model class labels are determined based on the optimum threshold value (OTV). OTV is captured using a threshold determination method during the structure identification stage as shown in ALGORITHM 4.3. The threshold value obtained during the training method is used to measure the accuracy of the testing method. The recognition performance (RP), i.e., the accuracy, measure in ALGORITHM 4.3. is a probabilistic measure that captures the correctly classified samples as a ratio of the total samples in the dataset. The threshold at the point where the RP is optimum is selected as the optimum threshold value (OTV). Optimum threshold value is an additional parameter optimized during the structure identification of the proposed the T1FF-C method and it is retained after the structure identification to discretisize the testing posterior probabilities to measure the model’s performance where necessary. In the conclusion of this section, we hypothesize that the optimum membership values estimated from the standard FCM clustering [Bezdek, 1981a] method and their possible transformations will be one of the optimum input variables of the T1FF or T1FF-C strategies. Our research on empirical datasets [Turksen, 2008; Celikyilmaz, 2005; Turksen and Celikyilmaz, 2006; Celikyilmaz and Turksen, 2007a] indicated that the T1FF model performances using these membership values can be improved if one can find better membership functions. Therefore, as a first step, a new approach is proposed to enhance the accuracy and generalization capability of the T1FF models using the proposed Improved Fuzzy Clustering (IFC) approach. Later, we will present an hybrid fuzzy functions method, that can further reduce some of the challenges of the T1FF methods, using other optimization methods that can search for global optimum solutions.
124
4 Fuzzy Functions Approach
ALGORITHM 4.3 Threshold determination for classification problem domain
Step 1: Create a 2 column dataset of actual class labels of the training data samples, yk∈{0,1}, and predicted posterior probabilities from the T1FF-C model, Pˆk (yk=1)∈[0,1], using the training dataset. An example is shown below: actual − yk
predicted probability − Pˆk
1 0
0.95 0.28
0 1 0 1
0.36 0.58 0.57 0.87
Step 2: Sort the above dataset in descending order according to the second column, predicted probabilities, Pˆk . Assign the probability of the first data vector to optimum-threshold value (OTV) and ŷk=1t=1=1 and ŷk>1t=1=0 Measure the recognition performance (RPt=1), e.g., total number of correctly classified points, using thresholdt=1 Step 3: Start iterating t=2…n-1; - Assign threshold t = Pˆk = t . - For k=1…n; If Pˆk > threshold t, then assign labels ŷkt =1, otherwise ŷkt =0. - Measure the recognition performance (RPt) using thresholdt - If RPt > RPt-1, OTV = threshold t, otherwise continue. predicted probability − Pk
ˆykt =1
ˆykt =2
ˆykt =3
1
0.95
1
1
1
1
0.87
0
1
1
1 0
0.58 0.57
0 0
0 0
1 0
0
0.36
0
0
0
0
0.28
0
0
actual − yk
accuracy ⇒
67% 84%
0 100%
In Chapter 3, the novel Improved Fuzzy Clustering (IFC) algorithm is presented and its advantages on Fuzzy Functions approaches are investigated using artificial datasets. The following section introduces the new training and inference algorithms of the Fuzzy Function systems by implementing the IFC algorithm, namely the Type-1 Improved Fuzzy Functions (T1IFF) system. The hypotheses is that membership values obtained from the IFC, can identify the parts of the behavior of the input-output variables in local models better than when membership values are obtained from the standard FCM clustering algorithm. The extension of the T1IFF approach to classification problems denoted with T1IFF-C, is also presented in the next section.
4.4 Proposed Type-1 Improved Fuzzy Functions with IFC – T1IFF
125
4.4 Proposed Type-1 Improved Fuzzy Functions with IFC – T1IFF This section presents new fuzzy system architecture with improved fuzzy functions - T1IFF - for regression and classification type problems (T1IFF-C). Novel T1IFF applies the proposed IFC algorithm during the learning stage to approximate improved membership values. These membership values and their transformations are then used as additional predictors of fuzzy regression functions of the system model along with candidate input variables. The reasoning mechanism is based on a new semi-parametric case-based reasoning approach. The structure of T1IFF and previous T1FF systems are similar to each other; however, the new training and inference algorithms are changed due to the new IFC method. The general framework of the proposed T1IFF is depicted in Figure 4.5 and Figure 4.6. The next sub-sections present learning and reasoning mechanisms of T1IFF and T1IFF-C strategies.
Fig. 4.5 Type-1 Improved Fuzzy Functions Framework based on sub-sampling cross validation method
4.4.1 Structure Identification of T1IFF for Regression Models Proposed Improved Fuzzy Clustering (IFC) optimization method searches for the optimum membership values, which are to be used later as additional predictors to estimate the parameters of the local fuzzy functions of a given system model. For regression problems, these fuzzy functions can be built using any type of function estimation method, e.g., LSE, SVR, etc. The structures of functions to be approximated depend on the structure of the membership values. One should choose appropriate transformations of membership values to identify improved fuzzy functions. The first step of T1IFF is to capture hidden patterns in the given dataset and to obtain improved membership values using the IFC method. For the readers’ convenience, we re-list the output values obtained from the IFC algorithm for given m and c values as follows:
126
4 Fuzzy Functions Approach
¾ ŵi : optimum parameters of interim fuzzy functions hi(τi|ŵi) of each cluster, i=1…c, that are captured from the last iteration step of IFC, ¾ the structure of the input matrix, τi, viz. list of different types of optimum improved membership value transformations that are used to approximate each Interim Fuzzy Function, hi(τi|ŵi), during IFC,
⎡ imp ⎢ μ1,i ⎢ ⎢ μ imp τ i = ⎢ i ,2 ⎢ # ⎢ ⎢ μ imp ⎣ i ,n
(μ ) (μ )
imp p ≠ 0 i ,1 imp i ,2
p≠0
(μ ) ⎤ " e ⎥ ⎥ μ ) ( ⎥ " e imp i ,1
imp i ,2
imp p ≠ 0 i ,n
p ≠0
⎥ ⎥ ⎥ p ≠0 μiimp ( ,n ) ⎥ " e ⎦ "
(μ )
p ≠0
(4.13)
¾ optimized membership values matrix, Uimp(x,y), and the cluster centers imp imp v (x,y), ( ) indicates optimum results from the new Improved Fuzzy Clustering algorithm. It should be recalled that, the interim input matrix τi is composed of improved membership values and their potential transformations as shown in (4.13). τi is obtained from the clustering algorithm IFC, which contains both the classical FCM clustering objective function term and the added “Interim Fuzzy Function”, hi(τi,ŵi), error estimation. It should be noted that Interim Fuzzy Functions contain only the membership values and their transformations in structures as shown in (4.13), i.e., the original input variables are not a part of the “Interim Fuzzy Functions”, hi(τi,ŵi), where ŵi is the coefficients of the function. τi matrix considers all possible transformations of membership values obtained from the IFC that may be considered in identifying the optimum Fuzzy Function parameters, which we usually refer to “System Fuzzy Functions”, or “Primary Fuzzy Functions” or simply “Fuzzy Functions” to differentiate them from “Interim Fuzzy Functions”. The system “Fuzzy Functions” are identified by using the improved membership values obtained from IFC and their transformations, τi, as additional inputs along with the original input variables, xk. We denote the new matrix Φi(x,τi) and the local system “Fuzzy Functions” using this matrix, fi(Φi,Ŵi), whose parameters are denoted with capital Ŵi. We also retain the cluster centers vectors of the optimum model to be later used for inference. Proposed Learning Schema of Type-1 Improved Fuzzy Functions Approach
Structure identification steps of the proposed approach as shown in Figure 4.6 is briefly explained in ALGORITHM 4.4.
4.4 Proposed Type-1 Improved Fuzzy Functions with IFC – T1IFF
127
Fig. 4.6 Structure of the T1IFF approach. (Top) Structure Identification with Improved Fuzzy Functions, (Bottom) Improved fuzzy function i.
Clustering Using Proposed IFC
The steps for applying the IFC method into the T1IFF system are shown in Figure 4.7. A user defined fuzzy or non-fuzzy clustering algorithm, usually Fuzzy C-means Clustering (FCM) [Bezdek, 1981a] algorithm is applied to capture initial cluster prototypes and the partition matrix of the membership values. Then, input-output dataset is re-clustered using Improved Fuzzy Clustering (IFC) to capture the interactive imimp proved membership values, μi (xy), i=1…c, and cluster centers, υi(x1,…,xnv,y) for c number of clusters and user defined fuzzy function structures, τi, such as in (3.29). imp Next, membership values corresponding to the input space, μi (x), is calculated using imp input matrix, x, and the cluster centers, υi (x1,…,xnv), of the input vectors.
Fig. 4.7 Implementation of Improved Fuzzy Clustering in T1IFF
128
4 Fuzzy Functions Approach
ALGORITHM 4.4 Proposed Training Algorithm of Type-1 Improved Fuzzy Functions Approach using the Improved Fuzzy clustering (IFC) algorithm Step 1: Compute an initial fuzzy partition, Uinit, of the input-output space Z(x,y) using a clustering method (of the users’ choice), e.g. typically FCM is used. Step 2: Compute Improved Fuzzy Clustering – IFC- algorithm in ALGORITHM 3.4 as follows: Use membership values Uinit as initial parameters to find the improved partition matrix, Uimp(xy) of the training samples, the cluster centers v(xy) and the optimum parameters, ǒi, of the regression functions hi(IJi|ǒi) for a given m and c value, while minimizing the IFC objective function:
J mIFC = ¦ ¦ ( μikimp ) d ik2 + ¦ ¦ ( μikimp ) c
n
i =1 k =1
m
c
n
m
i =1 k =1
( yk − hi (IJ ik , wˆ i ) )
2
(Equation (3.26) in chapter 3)
Step 3: Find improved membership values of the input space using the membership function, (Equation (3.30) from chapter 3) 1/( m −1) 2 2 § · § · ¨ c ¨ ( dik ) + ( yk − hi (τ ik , wˆ i ) ) ¸ ¸ imp μik ( x,τ i ) = ¨ ¦ j =1 2 2 ¸ ¨ ¸ 1< i , j ≤ c ˆ ¨ ¸ 1≤ k ≤ n © ( d jk ) + ( yk − h j (τ jk , w j ) ) ¹ © ¹
−1
where dik = xk − υiimp ( x ) and h(τi,ǒi) using improved membership values, μiimp , is obtained from the results of the IFC method in Step 2. Step 4: Repeat for each cluster i for τ i ( μiimp )∈Uimp 4.1. Using improved membership values of each of the training input data sample, μikimp and their user defined transformations (a total of nm new variables ℜnm space), IJi map the original input space onto each cluster feature space i, ℜnv+nm, i.e., xÆΦi(x,τi), 4.2. Estimate the regression parameters Ǒi of the fuzzy functions, fi(ĭi,Ǒi) for each cluster i. 4.3. Infer the output value of the data vectors using Fuzzy Functions:
ǔi =fi(Φi(x,τi),Ǒi). Step 5: Find a single output value by weighting inferred output values from each cluster with their corresponding membership values: ˆy =
(¦ ( μ c
i =1
imp ˆyi i
)) ( ¦ ( μ )) c
i =1
imp i
At this stage, we apply the cluster validity index, cviIFF, to measure the optimum number of clusters. In chapter 3, the implementation of the cviIFF into regression models is explained with experiments. Approximation of Improved Fuzzy Functions
The interim matrix, τi(μi )∈U , composed of the improved membership values and their transformations, are used as additional predictors of fuzzy regression functions in this step. The novelty of the “Fuzzy Functions” approach is that a different dataset is structured for each cluster i by using improved membership imp values (μi ) with their possible other forms as additional predictors of the original input matrix, see Figure 4.6. This is similar to mapping the original input space, imp
imp
4.4 Proposed Type-1 Improved Fuzzy Functions with IFC – T1IFF
129
ℜ onto a higher dimensional feature space ℜ , viz., xÆΦi(x,τi), for each cluster i, i=1…c. Hence, each datum is represented in (nv+nm) feature space. It should be noted that improved fuzzy functions are represented the same way as the fuzzy functions in equation (4.1) and Figure 4.4, but this time improved membership values obtained from the IFC method are used. For instance, a sample dataset of multi input and single output that was used to demonstrate the fuzzy functions in the previous section can also be used to demonstrate the improved fuzzy function structure. A new dataset is formed for each cluster by mapping the original input matrix onto a higher dimension using improved membership values as additional nv nv+1 predictors as new dimensions, i.e., ℜ Æℜ , as follows; nv
nv+nm
⎡ μ imp ⎢ i ,1 nv +1 Φi ( x ,τ i ) ∈ℜ =⎢ # ⎢ imp ⎢⎣ μi ,niα
xnv×1 ⎤ ⎥ # ⎥ ⎥ " xnv×nα ⎥ i ⎦ … #
x1×1 # x1×nα i
μikimp > α - cut 0 < niα ≤ n
(4.14)
In (4.14), i=1…c, represents the number of clusters, nm=1 denotes that there is only one additional column for this special example; nv indicates the original input dimension. Here, one fuzzy regression function of each cluster is identified in (nv+1+1) space (Additional dimension for the intercept.) It should be noted that these functions are nothing but crisp multi-variable-single output functions. Using μikimp>α-cut>0, we eliminate clustering anomalities which may be created by outliers, and only use the rest of the vectors to identify local fuzzy functions. A prominent feature of fuzzy function approximation is that, if the relations between input and output variables can not be defined in the original space, we can use the nv+nm+1 proposed fuzzy functions approach to explain their relationship in the ℜ space using additional information induced by the membership values and their transformations. Consequently, one fuzzy function, f(Φi), is identified for each cluster i, as shown in Figure 4.6, to obtain fuzzy output values for each vector. For instance, a 2 fuzzy function of a cluster can be represented with a set of linear planes in ℜ , such as in (4.14), by ŷ=f(Φi)=Ŵ0i+Ŵ1i(μi )+Ŵ2ix= Φi Ŵi imp
T
(4.15)
Ŵi=[Ŵ0i Ŵ1i Ŵ2i] are regression coefficients of fuzzy functions. One can use various forms of the improved membership values to identify local fuzzy decision surfaces. We use mathematical transformations of the membership values such as imp p imp p imp imp imp (μi ) , exp((μi ) ), ln((1-(μi ))/(μi )), 1/(1+exp(μi )), p∈ℜ,etc. The optimum membership function transformations depend on merely the structure of the dataset, similar to the simple regression problems where the input variable relationships with the output depends on the linearity issues (polynomials should be used if the relations cannot be explained with the linear functions). Hence, we identify the optimum structure based on the given dataset. Mainly search methods such as exhaustive search or genetic algorithms are used to optimize these parameters. We refer these parameters of fuzzy function systems as fuzzy function structures. In
130
4 Fuzzy Functions Approach
addition, the membership value transformations also depend on the clustering structure, the clustering parameters, etc. It is therefore difficult to construct a general rule about the suitability of a type of fuzzy function for a specific type of the problem set. If SVR is used to identify fuzzy functions, the general function to find fuzzy outputs for each cluster is given by:
(
)
ˆyi = fi (Φi ,α i ,α i* ) = ∑ k =1 α ik − α ik* K Φik , Φi + bi n
(4.16)
In (4.16), (αi,αi ) are the non-zero Lagrange multipliers, which denote multipliers for support vectors in each cluster i, K(.) is the kernel function that generates new nv+nm feature space for each cluster using Φi∈ℜ , i.e., ΦiÆK(Φi). A linear or a nonlinear kernel can be used depending on non-linearity of the input variables and the output variable. Just as in LSE structure, in SVR models, Ŵi represents a more complex structure indicating the model parameters of each cluster as follows *
Struct Ǒi { [Įi,Įi*]: Lagrange multipliers of support vectors of cluster i, [Φis]: training vectors of cluster i that are support vectors, (s) [K]: type of kernel mapping (K), i.e., Gaussian, linear, etc. }
When SVR is used to identify the fuzzy functions, the system fuzzy function pa* s rameters are represented with four-tuple form Ŵi=〈αi,αi ,Φi ,K〉. Finally, as the last step of the learning mechanism, one could infer single output value of each training data sample through fuzzy weighted average functions as follows;
μ imp ˆy i ∑ i =1 i ˆy = c ∑ i =1 μ iimp c
(4.17)
In conclusion, after the structure identification of T1IFF is completed, the outputs we obtain from each model will be: ¾
parameters of the interim fuzzy functions h(τi) of each cluster ŵi, i=1…c, which are used during IFC, and the interim input matrix, τi, viz., the list of different types of improved membership value transformations that are used to approximate each h(τi), ¾ degree of fuzziness, m, and number of cluster, c, parameter of IFC, ¾ parameters of system fuzzy functions f(Φi) of each cluster Ŵi, i=1…c, Ŵi=〈αi,αi,Φi,K〉, and the optimum α-cut value. ¾ If SVR is used to find fuzzy functions, C-regularization constant, ε-margin for epsilon insensitive regression and kernel type and parameters with the addition to the initial parameters for a non-linear SVR. The optimum model is captured with an exhaustive search method during structure identification using the training and validation datasets, i.e., training a particular model with a set of initial parameters on the training dataset and validating the particular model using a validation dataset for every set of parameters. The overall
4.4 Proposed Type-1 Improved Fuzzy Functions with IFC – T1IFF
131
performance of the model is measured by applying the optimum model on the testing dataset. Hence, the inference mechanism, to be discussed next, is used both for validating the model (optimization of the model parameters) and testing the generalization capacity of the optimum model.
4.4.2 Structure Identification of T1IFF-C for Classification Models Structure identification (training) of the proposed T1IFF-C systems for the classification type problems is very similar to structure identification of T1IFF systems as shown in Figure 4.8. Instead of implementing the IFC clustering, IFC-C clustering [Celikyilmaz and Turksen, 2007i] is implemented for the classification domains. It should be remained that, the matrix, τi is obtained from the clustering algorithm IFC-C. It contains only the membership values and their transformations, a total of nm different input variables such as in (4.13) to approximate interim fuzzy classifier functions, hi(τi|ŵi), with a classification function method of user choice, e.g., logistic regression (LR) or support vector classification (SVC). A sample interim classifier fuzzy function using a linear LR method is represented by: h(τi) = 1/ (1 + exp(-(ŵ0,i + ∑j ŵj,i τj,i))), i=1,..c, j=1...nm. T
(4.18)
After the IFC is executed and improved membership values are obtained, in order to find the local decision surfaces of each cluster i, a fuzzy classifier function, nm+nv f(Φi) is identified in feature space, ℜ . Here, nm is the number of additional membership value transformations augmented to the original input matrix and nv is the number of original input variables, i.e., xÆΦi(x,τi). Then posterior probabilities, Pˆi (ŷi=1|fi(Φi)) are calculated from f(Φi). For instance, if the support vector classification (SVC) is used to estimate fuzzy classification functions, f(Φi), the output values of data vectors are calculated by: Improved Fuzzy Classifier Functions gi, i=1,...,c
Inputs
Fuzzy Classification Function 1
μ
imp 1
Improved Fuzzy Clustering (IFC-C) hi(τi, ŵi)
x1
μiimp
e.g., ⎡
τ i = ⎢ μiimp ⎣
(
μiimp
)
2
imp ⎤ " e μi ⎥ ⎦
μ
. . . xk . . . xnv
imp c
Φ1 = [ x1 x2 ...xnvτ 1 ]
Outputs
μ
imp 1
g1
. . .
Fuzzy Classification Function i
Φ i = [ x1 x2 ...xnvτ i ]
μ1imp Pˆ1
μ
imp i
gi
. . .
Fuzzy Classification Function c
Φ c = [ x1 x2 ...xnvτ c ]
μcimp
μiimp Pˆi
∑μ
μcimp Pˆc
threshold
Pˆ
imp i i
Pˆ
Set Signal
gc
Fig. 4.8 Topology of Type-1 Improved Fuzzy Functions Systems for Classification Models
132
4 Fuzzy Functions Approach
f i (Φ i ) = ∑ nk βik* yk K (Φ ik ,Φ i ) + bi
(4.19)
In (4.19), βi are non-zero LaGrange multipliers, which denote the multipliers of nv+nm to the support vectors of cluster i, K(.) is the kernel function which uses Φi∈ℜ generate a new feature space, i.e., ΦiÆK(Φi), where K could be linear or a nonlinear function. The parameters of fuzzy functions such as in (4.19) are represented by Ŵi , which is a rather complex structure as in the T1IFF method. The parameters of the models for each cluster i, Ŵi will include: *
Struct Ǒi { [βi*]: Lagrange multipliers of support vectors of cluster i, [Φis]: training vectors of cluster i, that are support vectors, (s). [K]: type of kernel mapping (KT), i.e., Gaussian, linear, etc. }
We denote Struct Ŵi with three-tuple notation as Ŵi=〈βi ,Φi ,K〉. Fuzzy th posterior probabilities for the i cluster are calculated by; *
( (
Pˆi ( y = 1| f i (Φ i ) ) = 1 + exp − ( a1 fi (Φ i ) + a2 )
))
s
−1
(4.20)
Then, each fuzzy posterior probability of each cluster obtained from the classification function of the users’ choice, e.g., LR, SVC, etc., are weighted with their membership values,
gi = μiimp Pˆi , i = 1,..., c
(4.21)
Overall crisp posterior probabilities are calculated by the membership weighted method as follows: imp ∑ c gi ∑ c μ Pˆ Pˆ = c i =1 imp = i =c1 i imp i , i = 1,..., c ∑ i =1 μi ∑ i =1 μi
(4.22)
Actual class labels are calculated based on the optimum classification threshold value (OTV) obtained during the structure identification of T1IFF-C. The OTV is calculated from the application of ALGORITHM 4.3.
4.4.3 Inference Mechanism of T1IFF for Regression Problems Inference (reasoning) mechanism is used for validating a model (optimization of the model parameters) or testing the generalization capacity of an optimum model. This work applies a sub-sampling cross validation method to optimize model parameters based on the performance of the models applied on the validation dataset. Hence, the learning mechanism uses the training dataset and the inference mechanism uses * validation dataset to search for the optimum parameters of a given system. Let ( )
4.4 Proposed Type-1 Improved Fuzzy Functions with IFC – T1IFF
133
indicate the optimum parameters obtained from exhaustive search m ethod. After the optimum T1IFF system parameters obtained, which are; ¾ ¾ ¾ ¾
optimum number of clusters, c*, optimum degree of fuzziness, m* , * optimum parameters of the fuzzy classifier functions of IFC, ŵi , optimum system fuzzy function parameters, four-tuple form, * * Ŵi =(αi,αi ,Φi,K), i=1…c*, and ¾ some additional parameters specific to the selected function estimation type, e.g., kernel parameters (if SVR is used), are obtained from structure identification of the T1IFF system, we measure the performance of each T1IFF model with the reasoning (inference) method. In the following, the new reasoning (inference) mechanism is described using the testing dataset. In chapter 3, we introduced a new membership value calculation equation for IFC, i.e., 1/( m −1) ⎛ ⎞ ⎛ ( d ) 2 + ( y − h (τ * , wˆ * ) )2 ⎞ k i ik i ⎜ c ⎜ ik ⎟ imp ⎟ μik = ⎜ ∑ j =1 2 2 ⎟ * * ⎜ ⎟ 1< i , j ≤ c ˆ j )) ( , + − d y h τ w ⎜ ⎟ ( ) ( jk k j jk 1≤ k ≤ n ⎝ ⎠ ⎝ ⎠
−1
(4.23)
which is used by the new inference method to find the membership values of the testing data samples in each cluster. In order to calculate membership values of the testing data vectors whose output values are unknown, we need to review the new membership value calculation equation in (4.23). Note that this has an additional * * 2 squared error term, SEik=(yk-h(τ ik,ŵi )) . One should measure the value of this error term to calculate the improved membership values of the testing samples. Here, yk, indicates the actual output value of each testing sample, which is an unknown term * for the inference method. And, τi is the input matrix of the improved membership values such as in (4.13) and their user defined transformations, e.g., logistic, power functions, etc., and it is an unknown parameter, since we have not yet calculated improved membership values of the testing samples. The only known parameter that would be used during fuzzification of testing samples is the estimated * optimum function parameters, ŵi , i=1…c, of the membership values of IFC model from training algorithm. Hence, we need to approximate the unknown value of SEik, to be able to calculate new improved membership values of testing data samples. For this, we propose a case-based inference approach to estimate the error term using training data samples and their membership values. Since, the algorithm partially relies on the training samples, it is called a semi-parametric inference method. The following parameters are given at the beginning of the inference method of the T1IFF strategy; ¾ ¾
te
X ={xk, k=1…nte} represents the testing dataset, c*, represents the optimum number of clusters,
134
4 Fuzzy Functions Approach
¾ ¾ ¾ ¾ ¾
m* be the optimum fuzziness value, X={(xk, yk), k=1…n}, represent the training data samples, n be the total number of training data samples, imp μik ∈U*(x), i=1…c*, represent the improved membership values of the training samples, τi* is the optimal interim fuzzy function matrix composed of improved * membership values and their transformations, and ŵi be the parameters * of the interim fuzzy functions identified using τi , * Ŵi are the structures of system local fuzzy regression functions of the T1IFF system model.
ALGORITHM 4.5 shows the inference mechanism of the proposed T1IFF method, which is summarized in Figure 4.9.
Fig. 4.9 Proposed Inference Mechanism of T1IFF
ALGORITHM 4.5 can be summarized as follows: The first step of the new inference mechanism is to find κ training data samples that are the nearest to each testing data sample, r, r=1…nte, using the Euclidean distance measure, where A is the norm. Improved membership values of κ-nearest training data samples are * * * T used to build the interim matrix, τi =[τi1 …τiκ ] , a total of κ vectors for each cluster to be used to estimate the membership values of the corresponding testing sample, * r. Thus, using regression function parameters, ŵi , of interim fuzzy functions cal* culated during the IFC algorithm, hi, and the input matrix structure, τi , i=1…c, formed with the improved membership values, the output values of each κ-nearest
ˆ i ) , q=1…κ. training sample is calculated using hi (τ iq , w *
*
The next step is to measure the squared error values of these nearest κ data
ˆ i ) ) , q=1…κ, i=1…c, to be samples for each cluster, i.e., SEiq=(y q- hi (τ iq , w *
th
*
*
2
used to approximate the mSEir of the r test data sample in each cluster i. Next, error values, SEiq, are weighted with weight constants,ηrq, which represent the normalized distances of the κ-training samples to the testing sample r. The averth th age approximate squared error of the r testing sample in i cluster is calculated with weighted square error, m SEir , which is used in the new membership function to calculate improved membership values of testing samples. Then type-1
4.4 Proposed Type-1 Improved Fuzzy Functions with IFC – T1IFF
135
fuzzy outputs are weighted by their membership values to get a crisp output value of type-0. ALGORITHM 4.5 Inference Mechanism of the Improved Fuzzy Functions (T1IFF) for regression models Select k∈Ζ+. Iterate for each testing data sample r:1,…,nte; Step 1: Find the distance of the rth testing vector to each training sample, x and store them in D:
{
Dr = drq ∈ℜ drq = xte r - xq
2 A
}
, q=1,…,n, te: testing vectors
Step 2: Sort distances in Dr in ascending order and choose the first κ training vectors as the κnearest data sample to the testing sample r. Step 3: Save the distances of κ-nearest training samples in Dr=[dr1…drκ]T, their corresponding improved membership values in cluster i in Ui imp=[μi1imp,…,μiκimp]T, i=1,…,c*, and their actual output values in yr=[y1,…,yκ]T. Step 4: Calculate the average error term (SEir) of the κ-nearest data samples in each cluster: m SEir
= ∑ κq =1 SEiq ⋅η rq
(
SEiq = yq - hi (τ iq* , wˆ i* )
η rq = 1 - (drq
κ
∑ drs ) s =1
) =(y 2
q
- yˆ i (τ iq* )
)
2
Step 5: Use the membership function in (4.23) to calculate improved membership values of testing sample r in each cluster, μirimp, using dir = xr − υiimp ( x test ) and mSEir Step 6: Map the rth testing vector onto every cluster i using the structure Φi to form mapped testing vector in each cluster space, Φir, similar to input matrixes such as in (4.14) and infer their output values, ŷir, in each cluster using functions such as (4.16). Step 7: Find a single output value for each data object in the testing dataset using (4.17).
4.4.4 Inference with T1IFF-C for Classification Problems The inference (reasoning) mechanism of the T1IFF-C approach differs from the inference mechanism of the T1IFF in the way the fuzzy function parameters are obtained. Hence, the new membership function for IFC-C is re-displayed as follows:
μ ikimp
1< i ≤ c 1< k < n
( (
)) ))
⎛ ⎡ 2 * * ⎜ c ⎢ d ik ( xk ,υi ) + yk − pˆ ik yk =1 h (τ ik , wˆ i ) = ⎜∑⎢ ⎜⎜ j =1 ⎢ d jk2 ( xk ,υ j ) + yk − pˆ jk yk =1 h (τ *jk , wˆ *j ) ⎝ ⎣
( (
2
⎤ ⎥ 2 ⎥ ⎥⎦
1 /( m −1)
⎞ ⎟ ⎟ ⎟⎟ ⎠
−1
(4.24)
The membership function in (4.24) is used in the new inference method to find the improved membership values of the testing data samples in each cluster. In order to calculate the membership values of the testing data vectors, whose output
136
4 Fuzzy Functions Approach
values are unknown, we need to review the new membership value calculation equation in (4.24). The new membership function of IFC-C has an additional term, * * 2 SEik=(yk - pˆ k (yk=1|h(τiq ,ŵi ))) , where yk’s are class labels and pˆ i (.) are posterior probabilities. One should measure the value of the error term before calculating improved membership values of the testing samples. The inference algorithm of T1IFFC is similar to the T1IFF systems for regression models; so the ALGORITHM 4.5 is slightly updated for T1IFF-C models. In Step 4 of ALGORITHM 4.6, the IFC-C * * model output values ŷiq= h(τiq |ŵi ) of each training sample are replaced with model posterior probabilities, pˆ i (.) . ALGORITHM 4.6 Inference Mechanism of Type-1 Improved Fuzzy Functions (T1IFF-C) for classification problem Select k∈Ζ+. Iterate for each testing data sample r:1,…,nte; Step 1: Find the distance of the rth testing vector to each training sample, x and store them in D:
{
Dr = drq ∈ℜ drq = xte r - xq
2 A
}
, q=1,…,n, te: testing vectors
Step 2: Sort distances in Dr in ascending order and choose the first k training vectors as the κnearest data sample to the testing sample r. Step 3: Save the distances of κ-nearest training sample in Dr=[dr1…drκ]T, their corresponding improved membership values in cluster i in U*i=[μi1imp,…,μiκimp]T, i=1,…,c*, and their actual class labels in yr=[y1,…,yκ]T. Step 4: Calculate the average error term (SEir) of the κ-nearest data samples in each cluster: m SEir
= ∑ κq =1 SEiq ⋅η rq
η rq = 1 - (drq
κ
∑ drs ) s =1
SEiq = ( yq - pˆ i (τ iq* , wˆ i* ) )
2
Step 5: Using the membership value calculation equation in (4.24) to calculate the improved membership values of testing sample r in each cluster, μirimp. Step 6: Map the rth testing vector onto every cluster i using the structure Φi to form the mapped testing vector in each cluster space, Φir , input matrixes such as in (4.14) and infer their probability values, Pˆir , for each cluster using functions such as in (4.19) and (4.20). Step 7: Find the single posterior probability for each data point in testing dataset using (4.22).
4.5 Proposed Evolutionary Type-1 Improved Fuzzy Function Systems Genetic Algorithms (GAs) [Holland, 1973; Goldberg, 1989] are adaptive heuristic search algorithms premised on the evolutionary ideas of natural selection and genetics. GA’s were introduced as a computational analogy of adaptive systems and they are the most commonly implemented evolutionary algorithm methods. They are modeled broadly on the principles of the evolution via natural selection,
4.5 Proposed Evolutionary Type-1 Improved Fuzzy Function Systems
137
employing a population of individuals that undergo selection in the presence of variation-inducing operators such as mutation and recombination (crossover). Each individual represent one chromosome and a genetic algorithm model hold a number of individuals, determined by the domain expert, to apply the genetic operations on. A fitness function is used to evaluate individuals, and their reproductive success. Nearly everyone can gain benefits from GA, once they can encode solutions of a given problem to chromosomes in GA, and compare the relative performance (fitness) of such solutions. An effective GA representation and meaningful fitness evaluation are the keys to success in GA applications. The appeal of GAs comes from their simplicity and elegance as robust search algorithms as well as from their power to discover good solutions rapidly for difficult high-dimensional problems. GA’s are useful and efficient when the; ¾ ¾
Search space is large, Domain knowledge is scarce or expert knowledge is difficult to encode to narrow the search space, ¾ Traditional search methods fail. The advantage of the GA approach is the ease with which it can handle arbitrary kinds of constraints and objectives; all such things can be handled as weighted components of the fitness function, making it easy to adapt the GA scheduler to the particular requirements of a very wide range of possible overall objectives. GAs has been used for problem-solving and for modeling practices. Researchers have been investigating automatic learning methods for designing fuzzy inference systems by deriving an appropriate Knowledge Base automatically, without the necessity of its human operator. Genetic Algorithms [Goldberg, 1989] are stochastic search methods, based on natural genetics, which are valid approaches for fuzzy systems requiring efficient and effective search processes. One of the areas to improve the performance of the proposed T1IFF systems as in Figure 4.5, is that the optimum system parameters of both fuzzy clustering and local fuzzy functions could be determined automatically by an optimization method instead of an exhaustive search algorithm. Among many search algorithms, e.g., hill climbing, simulated annealing, gradient descent, etc., genetic algorithms have been the optimum solutions for fuzzy systems, as mentioned above. Fuzzy systems have a complex decision space which has a highly nonlinear structure due to the non-linear membership functions. This makes it hard to implement gradient descent algorithms while searching for the global minima; therefore, genetic algorithms, which are rather generalized optimization methods, are preferable for fuzzy system parameter optimization. One of the strengths of genetic algorithms is that different types of parameters can be easily encoded and with a diverse population, the algorithm can choose its own path based on fitness evaluation towards where the optimum parameters are in the global search space. Hence, the genetic algorithms utilize mutation operators to generate diversities in the population which present a global search space. The algorithm tends to keep the best solutions and generate similar ones until it finds the optimum solution.
138
4 Fuzzy Functions Approach
In this work, we introduce a new hybrid fuzzy system by encoding different types of parameters, e.g., binary or scalar values, of Type-1 Fuzzy Functions (T1FF) and Type-1 Improved Fuzzy Functions (T1IFF) systems into genomes to optimize them with GAs. Since, genetic algorithms are the most commonly known type of evolutionary computing methods, the new fuzzy functions methods are called “Evolutionary Type-1 Fuzzy Functions", in short denoted with ET1FF and “Evolutionary Type-1 Improved Fuzzy Functions", in short denoted with ET1IFF. Structurally speaking ET1IFF and ET1FF systems are similar to each other as do the T1FF and T1IFF systems are. The difference between them is that ET1FF systems implement the standard FCM clustering algorithm, whereas ET1IFF methods implement the proposed IFC method. Therefore, their structure identification and inference methods differ slightly. Details of these two methods are presented in the previous section. Since ET1IFF and ET1FF systems optimize the parameters of T1FF and T1IFF systems, we will only present ET1IFF systems in the next section. In the chapter of experiments (Chapter 6), both of these methods will be applied on benchmarking datasets to compare their performances based on generalization and robustness perspectives. The structure of each chromosome (genome) encodes the proposed ET1IFF system, which are parameters of the IFC algorithm including parameters of the membership function and the structure of the fuzzy functions, i.e., interim and local fuzzy functions. The proposed evolutionary fuzzy system with improved fuzzy functions –ET1IFF- optimizes system parameters with genetic algorithms which automatically determines the optimum parameters and at the same time reduces the number of iterations to optimize the learning parameters as opposed to T1IFF strategies which are based on exhaustive search methods. ET1IFF is introduced to solve regression type problems. An extension of ET1IFF for classification problems, ET1IFF-C, is also briefly presented in the next section.
Fig. 4.10 Evolutionary Improved Fuzzy Functions (ET1IFF) System Modeling Framework –Extension of the Improved Fuzzy Functions (T1IFF) Approach to Genetic Fuzzy Systems. i=1…c.
4.5 Proposed Evolutionary Type-1 Improved Fuzzy Function Systems
139
ET1IFF is an iterative hybrid system, in which, the structure is build and the parameters are constructed and tuned by the genetic learning algorithm. The learning algorithm determines the size and structure of the information structures in a novel way, which is the two fundamental phases of system identification [Pedrycz and Reformat, 2003]. The proposed fuzzy model as depicted in Figure 4.10, is comprised of two fundamental phases based on cross validation: ¾
Phase 1: Determination of optimum parameters using genetic learning process: learning from training and fitness evaluation with validation data, ¾ Phase 2: Inference with testing data using optimum model parameters.
4.5.1 Genetic Learning Process: Genetic Tuning of Improved Membership Functions and Improved Fuzzy Functions This section presents, in turn, the basic mechanism of the genetic learning process, e.g., coding, initial population creation, fitness function, genetic operators and stopping criterion, as implemented into the fuzzy functions learning strategy. Chromosome Encoding
The structure of each chromosome (genome) encodes the proposed ET1IFF model, which are parameters of the Improved Fuzzy Clustering (IFC) algorithm and structures of the fuzzy functions. The parameters of IFC identify the shapes of the improved membership functions. The membership function parameters are encoded as a separate gene (token) in each chromosome of the proposed ET1IFF systems. Since we implement the new IFC method into ET1IFF systems, the parameters to tune the membership functions will represent the parameters of the IFC method. Hierarchical heterogeneous chromosome formulation of genetic algorithms [Wang et al. 2005] is implemented to build ET1IFF model chromosomes. The genes of chromosomes are classified into two different types and structures: parameter genes that are real numbers, and control genes that are binary codes. Hierarchical arrangement of genes is shown in Figure 4.11. The structure of the parameter genes, as shown in Figure 4.11, vary based on the selected fuzzy regression type. The parameter genes take on real or integer values. The first four parameter genes are common in all ET1IFF systems, and they represent type of fuzzy function approximators, e.g., 0 indicates linear regression, 1 indicates support vector regression, and so on, clustering parameters, + m∈[1.1,∞], c∈Z ⊂[2,n(1/℘)], ℘>0 where n is the number of training data samples and α-cut≥0. The rest of the parameter genes are determined by the structure of the selected fuzzy regression type. For instance, if a simple linear regression is used to identify the fuzzy functions, then parameter genes consist of only the first four IFC parameters as seen in Figure 4.11 (Top). For other modeling types such as the support vector regression (SVR), additional model parameters are augmented to the genome. In the SVR case, two additional parameter genes that are Creg∈ℜ>0, and epsilon are appended to the chromosome structure. Creg is the
140
4 Fuzzy Functions Approach
type
Membership Function Structure
Fuzzy Function Structure
m
μ
c
alpha
Parameter Genes
type
Control Genes
Membership function structure
Fuzzy Function Type
Fuzzy Function Structure
m
Creg
μ
c
alpha
K(.)
Parameter Genes 1
... 1/(1+e−μ)
eμ
μp
1.75
3
0.15
μp
eμ
1/(1+e−μ)
Control Genes 54.4
0.115
1
0
0
1
0
Fig. 4.11 Genome Encoding for ET1IFF systems. (Top) Hierarchical structure genome encoding when linear regression is used (Middle) Encoding when SVM is used (Bottom) Example chromosome structure.
regularization parameter to balance the SVR objective function, viz., the weight vector and the error margin, and epsilon (ε) limits the error margin. These parameters determine the shape of the improved membership values as well as the improved fuzzy function structures. Similarly, the list of control genes, which take on only binary values, {0,1}, also varies based on the modeling type that is chosen by the expert. Only the first part of the control gene differs for the ET1IFF systems based on the type of the regression models used. For simple regression models such as LSE, no additional control gene is defined (Figure 4.11- Top). However, if SVR is implemented into the ET1IFF modeling structure, then the first control genes start with a kernel type, K(⋅), see Figure 4.11 (Middle). Two separate kernel types are used for SVM T formulation: linear K(xk,xj)=xk xj, and, non-linear Gaussian radial basis kernel (RBF), K(xk,xj)=exp(-δ||xk-xj||), δ>0. The rest of the control genes in Figure 4.11, are common to all types of ET1IFF systems. They represent the structure of the fuzzy functions, in other words, the list of membership values to be used as additional predictors of the input matrix to be mapped onto higher dimensions. Thus, the ET1IFF approach dynamically determines the feature mapping structure of the fuzzy functions. They are identified with different forms of type-1 fuzzy sets. The length of the fuzzy function structures, viz. the number of membership value types used in regression functions, is determined prior to chromosome formation. These form the control genes. The control genes represent fuzzy function structures, τi, implemented into IFC. These structures will then be used to identify local fuzzy functions structures, i.e., Φi,(x,τi) ,i=1,..,c. Activation of the control genes is set accordingly: an integer 1 is assigned to represent ignition, whereas 0 is for turning it off. Hence 0 in control gene represents that the corresponding membership value transformation will not be used in
4.5 Proposed Evolutionary Type-1 Improved Fuzzy Function Systems
141
the model if that chromosome is used to set the parameters. In other words, fuzzy function structure tokens, i.e., alleles (gene location), are assigned 1 if genetic algorithm decides that the particular dimension will appear in shaping fuzzy functions and 0 vice versa. In ET1IFF models, we used the same fuzzy function structures and parameters for each cluster. Therefore, static length chromosomes represent every cluster structure. An example of a chromosome structure in Figure 4.11 (Bottom) indicates that, for the given model, only eμ will be used as an additional dimension to map the original input space onto a feature space. In the next chapter, with the implementation of evolutionary methods using type-2 improved fuzzy functions, different structures of the fuzzy function for every cluster can be identified. Initial Population Creation and Fitness Function
The initial population is randomly generated. Parameter genes are formed by random numbers that can assume values between type∈{0,1,..}, alpha∈[0,1], + m∈[1.1,∞] and c∈Z ⊂[2,n(1/℘)], Creg∈ℜ>0. The epsilon variable should be between zero and the range of the target values. The control genes are randomly assigned 0 or 1 at the start of the genetic learning process. Fitness function could be any function determined by the user. Some of the fitness functions used in this work is discussed in the chapter of experiments in Chapter 6. The sub-sampling (three-way) cross validation method that is described above is also implemented in ET1IFF strategies. Even though the learning strategy of the ET1IFF approach is applied on the training dataset, the performance of each individual in the population is measured using the randomly selected validation dataset. Genetic Operators, Selection and Stopping Criteria
Different genetic operators are utilized for the parameter and control genes since they are real and binary numbers, respectively. For parameter genes, we used arithmetic and simple crossover and non-uniform and uniform mutation operators. For control genes, simple crossover and shift mutation operators are utilized. Selection is the process by which individuals in the population are chosen to reproduce. We implemented a tournament selection that extracts k individuals from the population with uniform probability (without re-insertion) and makes them play a “tournament”, where the probability for an individual to win is generally proportional to its fitness. Elitism is a variation on selection in which a user determined percentage of the highest ranked individuals is allowed to “walk-over” into the new population. This ensures that there will always be a minimum amount of highly fit genetic material to apply genetic operators to. In our models we employed tournament selection based on elitist strategy. The purpose of ET1IFF is to optimize the learning parameters, i.e., the number (c), the level of fuzziness, m, the membership value structure and parameters as well as the structure and parameters that would hold the optimum interim and local fuzzy functions structure and parameters, i.e., τ, Φ, ŵ, Ŵ, Creg, ε, K(⋅), etc., so
142
4 Fuzzy Functions Approach
that the optimum model to represent the actual system has a good performance. The learning process of proposed ET1IFF, as shown in Figure 4.10, Phase 1, is given in ALGORITHM 4.7. chrpop=1,…,max-population-size, indicates each chromosome, th i.e., individual, in the population of chromosomes at t iteration. The genetic learning process uses training and validation datasets to learn the model structure and select the best individuals to form different generations (the population). The T1IFF training dataset is used to learn a model using the parameters of each chromosome where GLP identifies optimum values of such parameters. At the start of the GLP process, each chromosome is randomly assigned linear or non-linear fuzzy function approximator types. One individual may identify a linear model, which uses membership values as additional predictors to shape membership values and fuzzy functions. Another individual may be assigned a non-linear fuzzy function type to build a T1IFF model. Based on the evolution strategies, the algorithm determines which fuzzy function approximator type would yield better performance based on the fitness function. During the genetic learning process, the performance of each individual is measured by applying T1IFF’s inference mechanism as in ALGORITHM 4.5 on the validation dataset by decoding their chromosomes. The individuals with the best performance are selected based on the tournament selection method. ALGORITHM 4.7 Genetic Learning Process of ET1IFF systems GA initializes chromosomes to form the initial populationt=0 as in Figure 4.11 depending on the system model type, i.e., linear or non-linear functions. Start iterating, t=1… max-number-iterations, { chrpop : popth chromosome in the population at tth iteration including parameters of typepop, alphapop, mpop, cpop, Cregpop, εpop, Kernel-typepop {K(⋅)}, and list of membership value transformations to construct {τpop,Φpop}. if chrpop has not been used in the past iterations { LEARNING - ALGORITHM 4.4 - Compute IFC with parameters from the chrpop using training data to obtain improved membership values, (μipimp) and calculate their possible transformations to form the input matrix τi,pop∈Uimppop - Map each original data onto the individual cluster to form feature vectors for each cluster cpop, xÆΦi,pop(x,τi,pop) using control genes to determine which membership value form to chose for chrpop - Approximate fuzzy functions, fi,pop(Φipop,Ŵipop) for each cluster i=1…cpop such as (4.15), (4.16) using chrpop parameters such as in Figure 4.13. -
REASONING- ALGORITHM 4.5 Find improved membership values of the validation data and infer their output values using ALGORITHM 4.5 based on chrpop, Measure fitness value for validation data.
} Genetic Algorithm generates next populationt+1 by means of selection, crossover and mutation operations Next iteration (t=t+1) } End
4.5 Proposed Evolutionary Type-1 Improved Fuzzy Function Systems
143
The difference between the proposed T1IFF and ET1IFF methods is that, in T1IFF systems only one model is built for a set of parameters. However, in ET1IFF systems, we deal with a population of set of parameters to build different ET1IFF models. These models are structurally different from each other. The best parameter is sought among potential individuals based on their fitness value on a validation dataset. In ET1IFF systems, each individual in the population represent different fuzzy decision surfaces based on their parameter and control genes of chromosome structure. They also represent different functions, such as a least squares regression, support vector regression, etc., which is identified by the first token of each chromosome. On the other hand, in T1IFF approaches, the fuzzy function of each cluster would have the same structure and type. At the start of the T1IFF, the domain expert determines that the model would use linear or non-linear fuzzy function approximators during the exhaustive search process. Hence, different sets
2
output-y
1 0 -1 1
1 0.5
output-y
2
2 output-y
Cluster 3
Cluster 2
Cluster 1
1
-2 1
0 1
0
0
1
1 0.5
0.5
uc1
0
input-x
0.5
uc2
0 0
0.5
0.5
uc3
input-x
0
0
input-x
(a) Fuzzy Functions – f(x,eμ), SVM-Linear Kernel, Kernel token=0. Cluster 1
Cluster 2
4
output-y
2 0
2
2 output-y
4
output-y
Cluster 3
0 -2
-2 1
1 1 0.5
u
c1
0.5
0.5 0
0
1
u
input-x
c2
0.5 0
0
input-x
1 0 -1 1 1 0.5
u
c3
0.5 0
0
input-x
μ
(b) Fuzzy Functions – f(x,e ), SVM-Gaussian Kernel, Kernel token=1. Fig. 4.12 Decision surfaces of one chromosomes of ET1IFF model using two different fuzzy function structures. (m,c,Creg, )={1.75,3,54.5,0.115}. uci represents membership values of corresponding cluster.
144
4 Fuzzy Functions Approach
of models are to be constructed for T1IFF models, whereas ET1IFF handles the fuzzy function approximators type during the optimization process and chooses the optimum type based on the stochastic search method. The crossover and mutation operators are applied on the randomly chosen chromosomes of the same type. Figure 4.12 depicts two sample decision surfaces of two separate models from a corresponding gene pool constructed for a non-linear sinusoidal artificial dataset. The top and bottom graphs represent two separate decision surfaces of T1IFF fuzzy functions of each cluster identified by two separate chromosomes of the population of chromosomes of a ET1IFF model. The difference between the upper and lower graphs is that all of their tokens of their corresponding chromosomes are kept the same but the kernel type (token #7 in Figure 4.13), which determines the non-linearity of the fuzzy functions. Hence, the upper three graphs represent the decision surfaces of a model from the gene pool with a three clustered structure where linear support vector is used to estimate each hyper-plane. Whereas in the lower graphs, a radial basis kernel function is implemented to build non-linear hyper-planes. The presented ET1IFF modeling strategy searches for the best local fuzzy decision surfaces by optimizing the parameters that are represented with each chromosome.
Fig. 4.13 Genome Two different chromosomes from the GLP algorithm of ET1IFF modeling approach applied on the Artificial Dataset. The dark colored token is the only difference between the two chromosomes. ‘1’:Linear Kernel Model, ‘2’:Non-Linear Kernel Model.
We can extend the proposed ET1IFF systems to build models of evolutionary improved fuzzy functions for classification problems, ET1IFF-C. The only difference between the ET1IFF and ET1IFF-C models is that in ET1IFF-C models, we approximate decision surfaces that can discern between two dichotomous classes. Posterior probabilities for each data vector in each cluster are estimated instead of continuous output values. Thus, each chromosome indicates the parameters of a T1IFF-C model. In this work, we only deal with binary classification problems. The proposed ET1IFF-C applies IFC-C during structure identification. Therefore, the structure identification (learning) phase – Phase 1 – of ET1IFF-C system is slightly different than the structure identification stage of the ET1IFF approach in ALGORITHM 4.7. The learning process of the ET1IFF-C utilizes classification type fuzzy functions, e.g., logistic regression, support vector classification (SVC), etc., and the chromosome structures will be slightly different than the ET1IFF chromosome structures, as shown in Figure 4.14. When the SVC is selected to approximate fuzzy classifier functions, the chromosomes will
4.5 Proposed Evolutionary Type-1 Improved Fuzzy Function Systems
145
not include the epsilon parameter, since it is a parameter for support vector regression models. The rest of the parameters will be as same as the ET1IFF parameters.
type
Membership Function Structure
Fuzzy Function Structure
m
μ
c
alpha
Parameter Genes
Control Genes
Membership function Fuzzy Function Type structure type
m
c
alpha
Creg
Parameter Genes
1/(1+e−μ)
eμ
μp
K(.)
Fuzzy Function Structure
μ
μp
eμ
1/(1+e−μ)
Control Genes
Fig. 4.14 Genome Encoding for ET1IFF-C systems for Classification Domains. (Top) Hierarchical structure genome encoding for linear regression (Middle) Encoding when SVM is used (Bottom) Example chromosome structure.
The same genetic operators used in ET1IFF approach is also used in ET1IFF-C approach. The fitness function of ET1IFF-C is different than ET1IFF methods’ fitness function due to the structural differences between the problem domains. For ET1IFF-C fitness evaluation, we implemented measures such as accuracy, area under the curve, etc., to be discussed in the chapter of experiments (Chapter 6). The best models of the gene pool are evaluated based on their fitness function. For each model, one T1IFF-C model is identified and then evaluated using the inference method in ALGORITHM 4.6. As a result of the genetic learning process, in both ET1IFF and ET1IFF-C, the parameters of the individual, i.e., chromosome, which identifies a model that has the optimum performance measure is retained to do reasoning for new data vectors. The chromosomes represent the optimum model parameters of the T1IFF model. Similarly, the optimum chromosome of ET1IFF-C represents the parameters of the T1IFF-C model. The next step is to measure the general performance of the optimum model by the new inference method.
4.5.2 Inference Method for ET1IFF and ET1IFF-C The main objective of the proposed ET1IFF and ET1IFF-C strategies, which are based on cross validation techniques, is to find a crisp output for a given input vector. The crisp output value would either be a real number for the ET1IFF models, or dichotomous class label for the ET1IFF-C models. Therefore, as a result of the algorithm, the type of the output is reduced from type-1 to type 0 by using the ET1IFF or ET1IFF-C model parameters. For the regression problem domains, during the first phase of the ET1IFF method, the genetic learning phase, the best
146
4 Fuzzy Functions Approach
individual is identified based on a performance measure. It should be noted that, each individual in the population represents one T1IFF model. During the genetic learning process, the inference engine of T1IFF system in ALGORITHM 4.5 is used to estimate the model output of the validation data in order to measure the fitness values of each population. After the genetic learning process, the optimum parameter of the best model from the gene pool is used to score the testing dataset to evaluate the optimum model performance using the same T1IFF inference mechanism. At this point ET1IFF and T1IFF implement the same inference mechanism as shown in ALGORITHM 4.5. Herein, the second phase of the ET1IFF as shown in Figure 4.10 is the inference method using testing data. The testing data, which has not been used for learning or validation purposes, is used to evaluate the overall model performance. Similarly, the proposed inference method for classification models in ALGORITHM 4.6 applies for ET1IFF-C model performance evaluation on testing datasets as well.
4.5.3 Reduction of Structure Identification Steps of T1IFF Using the Proposed ET1IFF Method In type-1 improved fuzzy functions - T1IFF systems, initial parameters are optimized with a exhaustive search based on the supervised learning method by iterating the list of parameters; degree of fuzziness (m), number of clusters (c), types of membership value transformations (τ, Φ) the of fuzzy functions, both of which use the same membership value transformations to identify interim and local fuzzy functions, alpha-cut constant and support vector regression (SVR) parameters such as regularization constant (C-reg), and error margin ( ). The values of these parameters should all be set prior to the model execution. Assuming each parameter has N different discrete values and two different types of kernel types are iterated, 6 the number of iterations of an exhaustive search would be 2N as shown in Table 4.1. Roughly speaking, N can correspond to ~10. For instance, we use 10 different values of number of clusters, e.g., c={2,3,4,…,11}, during exhaustive search optimization process. Table 4.1 Number of parameters of a Type-1 Improved Fuzzy Functions – T1IFF experiment
Parameter m: degree of fuzziness c: number of clusters α-cut {τ, Φ} matrix structures to form fuzzy functions Creg: regularization constant of SVR epsilon constant for SVR Kernel Type Total
Number of discrete values N N N N N N 2 6 2N
4.6 Summary
147
On the other hand, for the ET1IFF, let the total number of iterations of GA be 2 N and each chromosome represent a randomly initialized T1IFF model. Roughly speaking, we set the number of iterations to 100, so the correspondence of N dif2 ferent values of a parameter of T1IFF listed in Table 4.1 will be ~N . The popula2 tion size will be set to N , since we use 50-100 different populations in our experiments. At each iteration, two child chromosomes are selected for crossover and their fitness function is evaluated, and one child is mutated from two selected parents and its fitness is also evaluated. Hence, three T1IFF models are built at each iteration of GA. Therefore the number of iterations of the ET1IFF would be 2 2 2 N +2N =3N as shown in Table 4.2. Table 4.2 The number of parameters of an Evolutionary Type-1 Imssproved Fuzzy Function experiments
Parameter 2 Initial Run: population size = N Secondary run: evaluation of 1 crossover and 1 mutation operations Total
Number of T1IFF iterations 2 N 2 2N 2
3N
It is evident that the number of iterations is dramatically reduced when ET1IFF is used for modeling instead of the T1IFF method. On the other hand, one of the steps of T1IFF structure identification method is to analyze optimum number of clusters with a cluster validity index. Cluster validity functions are not used in the proposed ET1IFF due to the fact that number of clusters is optimized during ET1IFF structure identification. Therefore, ET1IFF systems eliminate the need for an additional function to evaluate the number of clusters. With ET1IFF systems, we evaluate the optimum number of clusters based on the system model performance. The optimum model is chosen from among the populations with the best performance measure on the validation dataset at the end of the GA iteration.
4.6 Summary In knowledge discovery applications, one tries to discover models with potentially higher prediction accuracy. Fuzzy Functions can be useful in such cases, if one succeeds in learning them automatically while at the same time preserving their predictability. In this chapter, we outline some fundamental properties of new fuzzy functions systems that are introduced instead of fuzzy rule based systems. The new fuzzy functions systems are designed on the bases of fuzzy clustering and regression techniques. We discussed structure identification and reasoning mechanism of the fuzzy function methods for two different system domain structures; regression and classification systems. We developed a hybrid fuzzy function system using evolutionary algorithms to optimize the membership functions and fuzzy functions dynamically with less user interventions. In the final part, we briefly presented the potential opportunity to reduce the number of iterations to find the optimum model with the hybrid fuzzy function systems using genetic algorithms as opposed to standard fuzzy functions systems based on the exhaustive search method.
Chapter 5
Modeling Uncertainty with Improved Fuzzy Functions 5 Modeling Uncertainty with Improved Fuzzy Functions This chapter introduces a new uncertainty modeling architecture for the new improved fuzzy functions systems1. The theory is based on a new interval type-2 fuzzy system. The uncertainties are captured by automatic identification of the structure of fuzzy functions, and upper and lower boundaries of key parameters that define fuzzy sets2. A new type reduction method is introduced.
“Inferences of Science and Common Sense differ from those of deductive logic and mathematics in a very important respect, namely, when the premises are true and the reasoning correct, the conclusion is only probable.”
—Bertrand Russell
5.1 Motivation Clearer characterization of mathematical models are required, when uncertainty in distinguishing various facets of fuzzy systems increase. For instance, we use fuzzy sets of type-1 when there are uncertainties in determining the membership value of an element in a set. Similarly, when there are additional uncertainties to determine type-1 fuzzy sets, or the circumstances are so fuzzy that we cannot determine the membership values in a type-1 fuzzy set, we may use type-2 fuzzy sets, which were initially introduced by Prof. Zadeh in 19753. We can continue on this line of 1
The first part of this chapter is an extension of [Celikyilmaz and Turksen, 2007f;2008d]. The last section of this chapter is an extension of [Celikyilmaz and Turksen, 2007h]. 3 Type-1 fuzzy sets are described by membership functions that are certain, whereas type-2 fuzzy sets are described by membership functions that are themselves type-1 fuzzy sets. 2
A. Celikyilmaz and I.B. Türksen: Model. Uncertain. Fuzzy Logic, STUDFUZZ 240, pp. 149–215. © Springer-Verlag Berlin Heidelberg 2009 springerlink.com
150
5 Modeling Uncertainty with Improved Fuzzy Functions
thinking and use type 3, and higher fuzzy sets and end up with type ∞ fuzzy sets, which are not feasible and as a result the complexity of the fuzzy logic system increases rapidly. Most of the latter uncertain circumstances arise from uncertain measurements in system models. There are situations where it is not possible to determine the exact value of a measurement. For instance, different people would obtain different results in decimals while measuring the weight of 1 pack of a substance, or the relevancy of a document would differ from one person to another while determining the class of relevant user requests to retrieve documents. In such cases, we use type-1 fuzzy sets instead of crisp fuzzy sets; however, as Mendel [2001] points out, “…even in type-1 fuzzy sets we specify the membership function exactly, which is ‘counter-intuitive. If we cannot determine the exact value of an uncertain quantity, how can we determine its exact membership value in a type-2 fuzzy set? We need to use a type-∞ fuzzy set to ‘completely’ represent uncertainty”. So, type-1 fuzzy sets can be thought of as a first-order approximation to the uncertainty in real life. With the type-2 fuzzy sets, one tries to get at a second-order approximation. One may investigate higher types; but as one goes on to higher types, the complexity of systems increases rapidly. Thus, in this part, we just deal with type-2 sets. In the previous chapter, Type-1 Improved Fuzzy Function (T1IFF) systems are proposed as opposed to conventional Fuzzy Rule Base systems by implementing the improved fuzzy functions approach to reduce the number of fuzzy operators and operations, as well as to improve system model performance. Additionally, T1IFF optimized with evolutionary algorithms (ET1IFF), are introduced to optimize the learning parameters by a stochastic search method. Although these systems are hypothesized to improve the performance, they have limited capacity in identifying uncertainties, because type-1 fuzzy sets are used in these systems. Let us explain the uncertainty concept in T1IFF approaches. In an analogical structure to Type-1 Fuzzy Rule Base (FRB) as sketched in [Mendel, 2001], we identify four components in T1IFF where we use (1) “Fuzzifier” in place of Improved Fuzzy Clustering (IFC), (2) “Rules” in place of “Fuzzy Functions”, (3) “Inference” and (4) “Output Processor” is the function to obtain the membership weighted average output value, in other words “Fuzzy Output Weighing”. We make this distinction to emphasize that in our work we introduce processes that are distinctly different from the processes applied by Mendel’s approach. Furthermore, as it was pointed out, our structures are different from Mendel’s, e.g., the structure of fuzzy rules and structure of “Fuzzy Functions” are different. In addition, in Mendel’s approach, generally, one obtains membership functions from experts, whereas in Fuzzy Functions strategies the membership values are obtained through the use of fuzzy clustering methods which are to be designed to structure these fuzzy functions. We have introduced the details of each T1IFF system component separately in the previous chapter. The question we are concerned with is that; why is it necessary to identify uncertainties in fuzzy function systems? Quite often, the knowledge that we use to construct the fuzzy functions is uncertain. T1IFF systems are not able to deal with fuzzy function uncertainties directly. Such uncertainties lead to fuzzy functions, whose membership values are
5.1 Motivation
151
uncertain. Type-2 fuzzy sets, on the other hand, can be very useful to identify an inexact membership function of a fuzzy set. In this respect, they can be used to handle fuzzy function uncertainties, and measurement uncertainties while determining membership values. In a T1IFF system as shown in Figure 5.1, crisp input numbers are mapped onto fuzzy sets using fuzzy clustering methods, e.g., Improved Fuzzy Clustering (IFC). The fuzzy sets are represented by membership values in each cluster. We can easily list some possible uncertainties of T1IFF systems here; the inputs to T1IFF systems prior to fuzzification may be uncertain, the parameters of fuzzy clustering method may be uncertain. In particular, among the most important uncertain parameters of the improved fuzzy clustering method are: (i) the number of clusters representing each local fuzzy functions, (ii) the degree of overlap between clusters, (iii) the type of similarity measure, and most importantly (iv) the type of membership value calculation equation, (v) the structure of local models, viz., the interim fuzzy function parameters. “RULES”* Fuzzy Functions Crisp Inputs
Crisp Output Type-0
“FUZZIFIER”* Improved Fuzzy Clustering
Fuzzy Input Sets of Type-1
“OUTPUT PROCESSOR”* Fuzzy Output Weighing
“INFERENCE”*
Fuzzy Output Set of Type-1
Fig. 5.1 Type-1 Improved Fuzzy Functions Systems. ‘*’ are Mendel’s terminology which correspond to terminology used for fuzzy function strategies in this book.
The inference (reasoning) process in Figure 5.1 determines the computation of improved fuzzy functions. It handles the way the fuzzy functions are activated to what degree and combines them. That is; the types of functions depend on the problem domain. For classification models, the classifier functions such as logistic regression is used, whereas for prediction (regression) domains regression type functions such as ordinary least squares are implemented. The inference mechanism of T1IFF systems is simpler than in Type-1 FRB systems, since aggregation, and implication operators are not required during inference. Other features of T1IFF are as follows: One fuzzy output from each fuzzy function is obtained. The fuzzy outputs can be continuous valued numbers for prediction problem domains or posterior probabilities, e.g., between 0 and 1, for classification problem domains. T1IFF systems can identify different fuzzy function structures for each local structure (cluster). Even within one cluster, the choice of an optimum function structure may be uncertain based on parameter uncertainties. Here, the degree of membership affects the activation and structure of a fuzzy function.
152
5 Modeling Uncertainty with Improved Fuzzy Functions
In many applications, a crisp output value must be obtained at the end of the modeling process. The output processor in Figure 5.1 converts the fuzzy output set into a crisp output. Several different defuzzification methods can be used to accomplish this. For Mamdani type fuzzy systems [Mamdani, Assilian, 1974], where the output is a fuzzy set, a defuzzification method such as center of gravity is used in order to obtain a crisp value. In T1IFF systems, since one single output from each fuzzy function is obtained, a defuzzification method is not required. Another point one should consider in T1IFF systems is that, some of its essential system parameters (such as the number of clusters, degree of fuzziness, parameter and structure of fuzzy functions of each cluster) are provided by the designer before the method of execution, which potentially introduces various sorts of uncertainties. We used the exhaustive search method to optimize these parameters for given list of their values. For continuous valued parameters such as fuzzifier, m>1.001, we use crisp (discretisized) list of values such as m={1.1, 1.3, .. 2.6..}. Unless one uses an infinite number of values for continuous valued parameters, capturing the global optimal solution is not certain. Infact this may increase the processing time which is not something we want to increase in the knowledge extraction processes. Since type-1 fuzzy sets are used in these models, most of the uncertainties cannot be captured and this may hamper the possibility of developing an optimal model. To capture some of the uncertainties of system models, as well as to improve performance of earlier T1IFF approaches, in this section, a new Type-2 Improved Fuzzy Function System – T2IFF – is introduced. The topology of the new approach is shown in Figure 5.2. It is very similar to T1IFF systems; however, in the new type-2 fuzzy system, the fuzzifier component i.e., degree of fuzziness of structures, identifies type-2 fuzzy sets and a new case-based type reduction method reduces the type of fuzzy sets down one level, from two to one, right at the beginning of the inference method. The rest of the components in the framework are similar to the T1IFF systems. The fuzzifier of T2IFF systems maps the crisp input onto fuzzy sets of type-2. The distinction between type-1 and type-2 systems can be identified based on the nature of fuzzy membership values. In this book, the fuzzy membership values of T2IFF systems are expressed with interval type-2 fuzzy sets. Crisp Inputs
“RULES”* Fuzzy Functions
“FUZZIFIER”* Improved Fuzzy Clustering
“OUTPUT PROCESSOR”* Fuzzy Output Weighing
Crisp Output Type-0
Input Fuzzy Sets of Type-2
Case-Based Type-Reducer*
Input Fuzzy Sets of Type-1
“INFERENCE”
Fuzzy Output Set of Type-1
Fig. 5.2 Type-2 Improved Fuzzy Functions System. * indicate difference between Mendel’s terminology and Fuzzy Function terminology.
5.1 Motivation
153
In analogical structure to Type-2 Fuzzy Logic System [Mendel, 2001], we identify five components in Type-2 Improved Fuzzy Functions (T2IFF) strategy as follows: (1) “Fuzzifier” in place of “Improved Fuzzy Clustering (IFC)”, (2) “Rules” in place of “Fuzzy Functions”, (3) “Inference”, (4) Case Based TypeReducer and (5) “Output Processor” in place of “Fuzzy Output Weighing”. After the fuzzification step, the type of the interval type-2 membership values are reduced down to type-1 based on the new Case-Based Type-Reduction method. This method is based on a search algorithm, which does not require complex fuzzy operators as do the other type reduction methods of standard Type-2 Fuzzy Logic Systems based on Fuzzy Rule Bases [Karnik et al. 1999; Mendel 2001]. After the type of the fuzzy sets is reduced down to type-1, the rest of the steps of the new T2IFF system are similar to T1IFF system. In this work, we introduce two new architectures of structure identification for the Type-2 Improved Fuzzy Functions (T2IFF) systems, which identify interval type-2 fuzzy functions. Therefore, we present them as two separate system modeling tools (strategies) as; ¾
Descrete Interval Type-2 Improved Fuzzy Functions (DIT2IFF),
¾
Evolutionary Design of Discrete Interval Type-2 Improved Fuzzy Functions (EDIT2IFF),
The two new T2IFF strategies are structurally different from each other based on the search method used to identify the global optimum solutions. Structure identification of the new DIT2IFF is based on a supervised Improved Fuzzy Clustering (IFC) method. Uncertainties are identified based on the learning parameters of the IFC and fuzzy function structures and the parameters are optimized based on an exhaustive search. On the other hand, EDIT2IFF has a three-phase structure identification method, which initially identifies the optimum uncertainty interval of the membership values, interval fuzzy functions and the optimum values of the rest of the inference parameters by implementing genetic algorithms. Then using these optimum parameters, the new system implements DIT2IFF system structure based on the identified uncertainty interval. The EDIT2IFF is proposed to reduce the number of iterations required for structure identification of the DIT2IFF by optimizing the system parameters automatically based on a stochastic search algorithm. The fact is that, there is infinite number of discrete type-1 membership values within the interval type-2 fuzzy sets. Even if one chooses the optimum ones from within this interval, it would either take too long time to find the optimum one(s) or there is a chance to get stuck at the local minima and not achieve the best model at all. With the implementation of evolutionary search algorithms, we are trying to identify the uncertainty interval, which encapsulates the optimum DIT2IFF model. There is still a chance of getting stuck at the local minima with the evolutionary methods; however, with more iterations and larger populations, this can be avoided to a certain degree. EDIT2IFF will have still less number of iterations compared to DIT2IFF, to be proved at the very last section of this chapter. An evolutionary model would be capable to shift, locate and possibly narrow the interval to find the optimum interval based on the given data and interval valued parameters.
154
5 Modeling Uncertainty with Improved Fuzzy Functions
It should be pointed out that, the type-2 fuzzy system modeling tools in this chapter applies the Improved Fuzzy Clustering (IFC) methodology. In addition to IFC, Fuzzy C-Means Clustering (FCM) algorithm is also used in these type-2 fuzzy system modeling tools in structure identification method to build two new strategies: ¾
Discrete Interval Type-2 Fuzzy Functions (DIT2FF),
¾
Evolutionary Design of Discrete Interval Type-2 Fuzzy Functions (EDIT2FF).
The common characteristic of these strategies is that they implement FCM clustering algorithm [Bezdek, 1981a] instead of IFC clustering methods. In Chapter 4, we presented the differences between the application of FCM and IFC methods into type-1 fuzzy functions approaches. These differences also apply to type-2 fuzzy functions methods of this chapter. Thus, we will only present details of the type-2 fuzzy functions using Improved Fuzzy Clustering (IFC) method. In this work, we not only deal with the regression models, where we estimate continuous fuzzy functions, we also extend the two novel T2IFF methods for classification problems as well and denote them with Discrete Interval Type-2 Improved Fuzzy Functions for Classification (DIT2IFF-C) and Evolutionary Design of Discrete Interval Type-2 Improved Fuzzy Functions for classification (EDIT2IFF-C). The only difference between DIT2IFF/EIT2IFF models of regression and DIT2IFF-C/EDIT2IFF-C models of classification is the way the output variables are represented. DIT2IFF and EDIT2IFF models try to model a continuous output variable by approximating regression surfaces of each local domain (cluster). On the other hand, DIT2IFF-C and EDIT2IFF-C models estimate decision surfaces for each cluster to discriminate two separate classes of input data points denoted by their dichotomous output variable. The extensions of type-2 fuzzy functions methods for classification problem domains are also investigated. Hence, Discrete Interval Type-2 Fuzzy Functions for Classification (DIT2FF-C) and Evolutionary Design of Discrete Interval Type-2 Fuzzy Functions for classification (EDIT2FF-C) are two methodologies that implement FCM clustering method instead of IFC method. Thus, in this chapter we will only give the details of the type-2 improved fuzzy function strategies for classification using IFC method, since the differences between them for regression problem domains also apply in classification problem domains. In what follows, we present the uncertainty concept using fuzzy theory and preview different types of uncertainty modeling methods. We then present novel uncertainty modeling approach, namely Discrete Interval Type-2 Improved Fuzzy Functions system, DIT2IFF. We introduce the case-based type reduction method. Finally, we present evolutionary design of discrete interval type-2 improved fuzzy function system, EDIT2IFF, which is capable of automatically identifying uncertainty bounds of DIT2IFF system parameters.
5.2 Uncertainty Uncertainty is one of the key characteristics of any mathematical system model. It has a pivotal role in any efforts to maximize the usefulness of system models [Klir and Yuan, 1995]. Uncertainty becomes very valuable when analyzed in
5.2 Uncertainty
155
combination with different characteristics of models: in general, identifying uncertainty to reduce the complexity and increase the performance. The term ‘modeling uncertainty’ defines the effort for identifying uncertainties when building a system model. The challenging part of modeling uncertainty is to determine the optimal level of allowable uncertainty. In this work, this allowable uncertainty will be identified automatically by evolving stochastic models of T1IFF approach. In many research papers on uncertainty modeling and books on mathematics of uncertainty, it is generally agreed that Zadeh’s seminal paper in 1965 has the pioneering role as the starting point of the modern concept of uncertainty. In his paper, Zadeh [1965] introduced a theory whose objects are fuzzy sets of which the boundaries are not precise and the membership to a fuzzy set is not a matter of confirmation, but rather a matter of degree. Let x be an object in fuzzy set A. The proposition “x is a member of A” is expressed with a degree to which x is actually a member of fuzzy set A. The capabilities of fuzzy sets provide us the meaningful and powerful representations of measurement uncertainties, but also meaningful representation of vague concepts expressed in natural language.
Randomness Fuzziness
Possibility
Ambiguity
Uncertainty
Probability
Imprecision
Vagueness Similarity
Fig. 5.3 Different Types of Uncertainty
Among many types of uncertainties, as shown in Figure 5.3, two categories of uncertainty emerge quite naturally: vagueness and ambiguity. Vagueness is usually defined as the difficulty of making sharp and precise distinctions in the world. On the other hand, ambiguity is defined as one-to-many relations in which the choice between many alternatives is left unspecified. Each of these concepts relates to fuzzy set concepts differently. Vagueness is connected with fuzziness, in which fuzzy set theory provides a basic mathematical framework. On the other hand, the concept of the fuzzy measure provides our underlying framework for dealing with ambiguity. Hence, fuzzy measure and fuzzy sets reflect two fundamentally different types of uncertainty⎯vagueness and ambiguity concepts, and they provide frameworks for different types of uncertainty.
156
5 Modeling Uncertainty with Improved Fuzzy Functions
Mathematical models are simplified representations of the phenomena being studied and a key aspect of the modeling process is the careful choices of model assumptions. The structure of mathematical models employed to represent systems is often a key source of uncertainty based on these assumptions. In this context, different uncertainty modeling methods are summarized and they are categorized into four different groups: 1. Uncertainty in Natural Language Processing. The meaning of words in natural language that corresponds to fuzzy sets can be different, since same words may be used for different meanings. This phenomenon brings about the notion of computing with words [Zadeh, 1996]. In addition, the difficulty arises when one wants to model with words such as “most”, “popular”, which have meanings that are vague. In addition, uncertainty in decision-making arises due to the non-uniform data characteristics such as noisy data. 2. Uncertainty defined in the break down of classical equations. More explicitly, as proposed by Turksen [1986, 1989], representation of combined linguistic expression of linguistic values with linguistic connectives should not just be mapped arbitrarily to one of the two fuzzy canonical forms, but to the “interval” generated by both fuzzy canonical forms. This is required because Turksen proved that the equivalence between DNF(.)≡CNF(.) breaks down in fuzzy theory, i.e., FDCF(.)≠FCCF(.)4 for certain families of t-norms and tconorms, and more specifically he explains that the containment between two fuzzy canonical forms, should be represented as FDCF(.)⊆FCCF(.) for archimedian t-norms and t-conorms. 3. Uncertainty defined with Type-2 Fuzzy Sets, which handles uncertainties about the meanings of words. All the uncertainty about the words can be explained by blurring the boundaries of type-1 membership function into a footprint of uncertainty, which is then called type-2 membership function [Mendel and Karnik, 1998; Mendel et al., 2006; Mendel and Liang, 2000; Klir and Folger, 1988; Klir and Wierman, 1998; etc.]. 4. Uncertainty defined with the changing values of parameters (attributes) of Fuzzy Systems, e.g., degree of freedom, number of clusters, alpha-cut, fuzzy function structures etc. Turksen [2001; 2002], Uncu, and Turksen [2007], Uncu et al. [2004a,b], Uncu [2003], Ozkan-Turksen [2004, 2007] identified other sources of uncertainties and introduced type-2 fuzziness analysis, which can explain uncertainties about the parameters of the fuzzy systems, especially unknown interval values of degrees of fuzziness of Fuzzy c-Means Clustering (FCM) algorithm. In [Ozkan and Turksen, 2007], lower and upper boundaries of the level of fuzziness parameter of the FCM clustering method is proven to be [1.4, 2.6] based on numerical analysis. In addition to investigating the boundaries of level of fuzziness of systems models, in this work, an additional source of parameter uncertainty is used, namely, the structure uncertainty of the fuzzy functions [Turksen, 2006; Turksen and Celikyilmaz, 2006; 4
FCCF: Fuzzy Conjunctive Canonical Form, FDCF: Fuzzy Disjunctive Canonical Form.
5.3 Conventional Type-2 Fuzzy Systems
157
Celikyilmaz and Turksen, 2007g, 2007h]. This uncertainty emerges from identification of different membership value transformations in approximating local fuzzy functions. In this chapter, the last two categories will be explained in more detail. For the first two categories, references on computing with words [Zadeh 1996;2001;Turksen, 2002; 2007] are suggested.
5.3 Conventional Type-2 Fuzzy Systems Today, the research on Type-2 Fuzzy Systems is a growing area and more publications are presented in recent conferences than ever. This section briefly introduces well-known Type-2 Fuzzy Systems and their recent advancements, which were mentioned in the last two categories of uncertainty modeling approaches in the previous subsection. These systems are based on Fuzzy Rule Base approaches and they implement type-2 fuzzy sets. (type-2 fuzzy sets and operations are reviewed in Chapter 2.) This sub-section presents a brief description of most commonly implemented and applied type-2 fuzzy systems presented in series of papers by Mendel, Karnik, and Liang [Mendel and Karnik, 1998; Liang and Mendel, 2000; Mendel et al, 1999; Mendel and Liang, 2000; Mendel, 2001;2003]. Additionally, some recent developments on type 2 fuzzy systems using discrete interval type-2 fuzzy sets e.g., [Uncu and Turksen, 2007] are reviewed and comparative analyses between proposed and earlier discrete type-2 fuzzy systems are discussed. It should be recalled that there are two types of type-2 fuzzy sets, generalized type-2 and interval type-2 fuzzy sets, as presented in Chapter 2. Type-2 fuzzy systems are categorized based on these two different types of fuzzy sets. Hence, ‘Generalized Type-2 Fuzzy Systems’ is the general name given to type-2 fuzzy systems whose antecedent and/or consequent part of the rule base implements a type-2 fuzzy set. Interval type-2 Fuzzy Systems use interval type-2 fuzzy sets.
5.3.1 Generalized Type-2 Fuzzy Rule Bases Systems (GT2FRB) A generalized Type-2 Fuzzy Rule Base (GT2FRB) [Mendel and Karnik, 1999] is an extended version of Type-1 Fuzzy Rule Base structure, which was presented in chapter 2. The GT2FRB is represented by; nv
% i : IF AND R j =1
(x
j
∈ X j isr %Aij
)
%i THEN y ∈ Y isr B
(5.1)
Let u and w represent primary memberships values and f(u) and f(w) represent the secondary membership values. In (5.1) Ãij is the linguistic label associated with jth input variable in the ith rule represented by a type-2 membership function, % is μ% i ( x j ) ∈ X j → f x j (u ) / u, u ∈ J x j , J x j ⊆ [0,1] , which is depicted in Figure 5.4. B i the linguistic label associated with output variable in the ith rule with type-2 membership function μ% i ( y ) ∈ Y → f y ( w) / w, w ∈ J y , J y ⊆ [0,1] .
158
5 Modeling Uncertainty with Improved Fuzzy Functions
Fig. 5.4 Example of a full type 2 membership function. The shaded area is the ‘Footprint of Uncertainty’ (FOU). The amplitude of the sticks is secondary membership values.
Each combination of (x,u, f(u)) denotes an embedded set and there may be infinite number of embedded sets for the universe of discourse X and Jx. The operations of GT2FRB are very much the same as the Type-1 FRB operations; however, in GT2FRB systems type-2 fuzzy operations are used. This introduces an additional step, e.g., type reduction, that is shown in Figure 5.5. Identification of Degree of Fire Fuzzification + Aggregation of Antecendets
Implication
Fuzzy Input Sets
Rules + Aggregation of Deduced output Fuzzy sets
Fuzzy Output Sets
Type Reduction
Defuzzifier
Reduction of Type 2 to Type 1
Defuzzification Reduction of Type1 to Type 0
Crisp Output
Fig. 5.5 The structure of Generalized Type-2 Fuzzy Rule Base Systems (GT2FRB)
The following type-2 fuzzy rule base operations are a summary of definitions extracted from [Mendel, 2001] by using PRODUCT, MIN or MAX connectives via Mamdani approach. In general however, other t-norms and t-conorms can also be used. From (5.1), the antecedent type-2 fuzzy sets Ãi is determined follows: nv
%A = ∩ i
j =1
( %A ) ij
∀i = 1,...,c
(5.2)
In (5.2) the membership values of an object x is calculated by
% i= μ
nv j =1
% i (xj ) μ
(5.3)
5.3 Conventional Type-2 Fuzzy Systems
159
It should be noted that, there is an independency assumption in type-2 fuzzy system models, in that, each type-2 fuzzy set associated with each antecedent variable is defined separately in each rule. is the meet operator used to combine type-2 fuzzy sets, which was defined in Chapter 2. Consequently, the relation between % is determined by the ith type-2 fuzzy relation as inputs and output in each rule R i follows:
μ% R% ( x, y ) = μ% A% ( x) → μ% B% ( y ) = ⎛⎜ μ% A% ( x) ⎞⎟ μ% B% ( y ) j nv
i
⎝
i
i
ij
⎠
i
(5.4)
Equation (5.4) is an extension of equation (2.80) for type-2 fuzzy sets. From (5.4), the output fuzzy set of each rule, %Bi* , is calculated using Generalized Modus Ponens as follows:
( )(
B%i* = A%i′ ° A%i → B%i
)
(5.5)
For a given observation x′, (5.5) is represented with membership functions by:
μ% i* ( y ) =
x
⎡ ⎣
( μ% ′( x′), μ%
R%i
)
( x, y ) ⎤ = ⎦
x
⎡ μ% R% ( x, y ) ⎤ , ∀i = 1,..., c ⎣ i ⎦
(5.6)
Since the given observation x′ is a crisp number, (5.6) is reduced to ⎡% ′ ⎤ x ⎣ μ R%i ( x , y ) ⎦ . Finally, each output fuzzy set i, i=1…c, is aggregated to obtain one fuzzy set, and the aggregation formula is computed by:
μ% * ( y ) = μ% i* ( y ), ∀i = 1,..., c i
(5.7)
Mendel and Karnik’s GT2FRB strategy is an extension of Type-1 FRB using type-2 fuzzy sets and extension principle proposed by Zadeh [1975a]. GT2FRB strategies [Mendel, 2001] can be computationally challenging when the number of variables and number of data vectors are large. The output set corresponding to each rule of the type-2 fuzzy system, as shown in (5.1), is a type-2 fuzzy set. After the implication step in (5.6), defuzzification takes place. Defuzzification of a type-2 fuzzy set consists of two stages: ¾
Type-reduction, creating a type-1 fuzzy set, and
¾
Defuzzification of the resultant type-1 fuzzy set.
The second step, defuzzification, can be much easily implemented compared to the first stage, which is more problematic. The type-reduction is reliant on the concept of embedded type-2 sets. A type-reducer combines all these type-2 output fuzzy sets, just as type-1 defuzzifier method, and then performers a centroid calculation on this type-2 fuzzy set (similar to center of gravity of type-1 fuzzy sets, as presented in Chapter 2). This leads to a type-1 fuzzy set; therefore, this operation was named “type-reduction”. In the next, we will present the most common type reduction method. The challenge with the type-reduction method is that the
160
5 Modeling Uncertainty with Improved Fuzzy Functions
centroids and the membership value computations has to be repeated for every (x, u, f(u)) pair, and there can be infinite number of membership functions. In the experiments, these intervals can be discretisized into embedded sets, however, the degree of discretization is an unknown value. Therefore, computational complexity can be high in generalized type-2 fuzzy sets if the discretization degree is very small, i.e., precision is high, which results too many embedded sets. This would increase the complexity of system modeling strategy. Most recently, to reduce some of the complexities of type-2 fuzzy logic systems, a geometric approach to implement GT2FRB is proposed in [Coupland and John, 2007]. Although it is hypothesized that it presents promising results, it is still in development stage and the secondary membership values are still determined by the experts. A challenging part of generalized type-2 fuzzy systems is the uncertainty in the determination of secondary membership values. For instance, the primary membership values can be determined from the given dataset by fuzzy clustering methods. Even the footprint of uncertainty can be defined by varying the fuzziness parameter of the FCM clustering algorithm [Hwang, Rhee, 2007; Ozkan and Turksen, 2007], however, to date, there is no algorithm that can automatically extract the secondary membership functions from the data. Initial studies of automatic determination of full type-2 membership values, i.e., the secondary membership values, are presented in [Celikyilmaz and Turksen, 2008a] and they use fuzzy regression clustering methods, which is rather a simple method compared to earlier type-2 fuzzy systems, to identify secondary membership functions. To cope with these computational complexities of generalized type-2 fuzzy systems, many researchers today use interval type-2 fuzzy structure identification and inference methods instead of GT2FRB.
5.3.2
Interval Valued Type-2 Fuzzy Rule Bases Systems (IT2FRB)
In the previous section, the generalized Type-2 Fuzzy Rule Base (GT2FRB) proposed by Karnik and Mendel [Mendel and Karnik, 1998; Karnik et al, 1999] are presented. Karnik and Mendel have pointed out the complexity of the GT2FRB systems. Thus, Liang and Mendel have proposed a special case of GT2FRB systems in [Liang and Mendel, 2000] in which only interval type-2 fuzzy sets are used. In an analogical manner to the GT2FRB systems, the Interval Valued Type2 Fuzzy Rule Bases (IT2FRB) systems considered only crisp MIN, PRODUCT and MAX connectives. In general however, other t-norms and t-conorms can also be used. Here we will give a brief definition of interval type-2 fuzzy sets as explained in Chapter 2. Let à be a linguistic label with type-2 membership function on the universe of discourse of base variable x, μ% i ( x j ) ∈ X j → f x (u ) / u, u ∈ J x , J x ⊆ [0,1] . In order to j
j
j
consider μ% A ( x) as interval valued type-2 membership function, the following condition has to be satisfied:
f x (u ) = 1, ∀x ∈ X , u ∈ J x , J x ⊆ [0,1]
(5.8)
5.3 Conventional Type-2 Fuzzy Systems
161
Let upper, μ% UA ( x) , and lower membership values, μ% AL ( x) , of a base variable are defined as follows:
μ UA% ( x) ∈ J xU , μ AL% ( x) ∈ J xL
(5.9)
The upper and lower bounds are denoted with JxU and JxL [Liang and Mendel, 2000]. Then, each data vector is assigned a secondary membership value of 1 between the upper and lower boundaries:
μ A% ( x) : X → 1/ u , u ∈ ⎡⎣ μ AL% ( x ), μ UA% ( x ) ⎤⎦
(5.10)
As in GT2FRB approach, we define the fuzzy rule base structure of the interval type-2 fuzzy rule base structure (IT2FRB) as follows: nv
% i : IF AND R j =1
(x
j
∈ X j isr %Aij
)
%i , THEN y ∈ Y isr B
(5.11)
where Ãij is the linguistic label associated with jth antecedent in the ith rule with type-2 membership function of μ% i ( x j ) : X → 1/ u, u ∈ ⎡⎣ μ% iL ( x j ), μ% iU ( x j ) ⎤⎦ . In general case each ith rule is represented by:
% i : IF x ∈ X isr %Ai THEN y ∈ Y isr B %i , R
(5.12)
where Ãi is the linguistic label, associated with aggregated antecedents in the ith rule with type-2 membership function of μ% i ( x) : X → 1/ u, u ∈ ⎡⎣ μ%iL ( x), μ%iU ( x) ⎤⎦ . Aggregated antecedent membership values for each rule are calculated by
μ% iL ( x) = T jnv=1 ( μ% iL ( x j ) ) , μ% iU ( x) = T jnv=1 ( μ% iU ( x j ) )
(5.13)
In (5.13), T denotes the t-norm connective. Interval valued type-2 membership functions further reduce (5.6) as follows:
μ% i* ( y ) : Y → 1/ w, w ∈ ⎡⎣ μ% i*L ( y ), μ% i*U ( y ) ⎤⎦
(5.14)
In (5.14), ⎡⎣ μ% i*L ( y ), μ% i*U ( y ) ⎤⎦ are lower and upper membership functions that define the boundaries of the interval valued type-2 model output fuzzy set μ% i* ( y ) . Let T denote the t-norm connective, then, for each given data object, the interval model output-membership function in the ith rule is calculated as follows:
μ% i* L ( y ) = T ( μ% iL ( x), μ% iL ( y ) )
μ% i*U ( y ) = T ( μ% iU ( x), μ% iU ( y ) )
(5.15)
In order to obtain one single type-2 membership function of the final model output fuzzy set, μ% * ( y ) , output type-2 membership function of each ith rule are aggregated as follows:
162
5 Modeling Uncertainty with Improved Fuzzy Functions
μ% *L ( y ) = Sic=*1 ( μ% iL ( y ) )
μ% *U ( y ) = Sic=*1 ( μ% iU ( y ) )
(5.16)
In (5.16) S denotes t-conorm, which was taken as MAX operator in [Liang, and Mendel, 2000]. In order to obtain a crisp value, type-reduction can be applied on [ μ% i*L ( y ), μ% i*U ( y )] as explained in the previous section with only one distinction that is the secondary membership functions only take on values of 1. Thus the output of type-reduction applied on [ μ% i*L ( y ), μ% i*U ( y )] , y% * , is an interval where the lower and upper bounds are denoted as y*L and y*U respectively, i.e., y% * ∈[y*L,y*U]. If the goal is to obtain a crisp output, Liang and Mendel proposed to calculate the model output, y*, as follows: y* = [y*L+y*U]/2
(5.17)
5.3.3 Most Common Type-Reduction Methods The output fuzzy sets corresponding to each rule of Generalized or Interval Type-2 Fuzzy Rule Base system is a type-2 fuzzy set. In general case, type-reducer combines type-2 output fuzzy sets to perform centroid calculation. As a result, the type of the fuzzy set is reduced from type-2 to type-1 and resulting type-1 fuzzy set is called the “type-reduced” set. Here we will introduce type reduction method for generalized and interval type-2 fuzzy systems. Generalized type-2 fuzzy set, Ã, has primary membership values (type-2 membership functions), μÃ(x), i.e., Ã={(x,u),μÃ(x,u)|∀x∈X,∀u∈Jx⊆Jx[0,1]}, in which 0≤μÃ(x,u)≤1, which are mapped to a secondary membership function f(x), which maps values in [0,1] to values in [0,1]. Jx is the primary membership function of x, and the amplitude of a secondary membership function is called a secondary grade, fx(u), where u’s are the primary membership values. The type-2 fuzzy set is represented with: à =∫x∈X [∫u∈Jx fx(u)/u]/x
(5.18)
In (5.18) Jx⊆[0,1], x∈X, u∈[0,1], fx(u)∈[0,1] [Mendel et al., 2006]. Hence, each datum X has many primary membership values, u, which also has secondary membership functions. Each u of x has a secondary degree of membership; as shown in Fig. 2.9, that assigns the degree to which u belongs to x. On the other
% , has secondary membership values, μÃ(x), hand the interval type-2 fuzzy set, A which are mapped to a secondary membership function f(u), which maps values in [0,1] to values in {0,1}. Let the domain of the secondary membership function denoted by Jx; (5.19) à =∫ [ ∫ 1/u]/x x∈X
u∈Jx
Type-2 membership functions are a limited version of generalized type-2 fuzzy sets, where the secondary membership grade is always 1. Only boundaries of membership values are defined for each datum, which form the footprint of
5.3 Conventional Type-2 Fuzzy Systems
163
uncertainty, as shown in Fig. 2.11. This limitation allows type-2 interval fuzzy sets to be processed a lot more quicker than generalized type-2 fuzzy sets. The interval type-2 fuzzy sets are bounded with upper and lower type-1 fuzzy sets. Type reduction takes a type-2 fuzzy set and reduces its type down to a type-1 fuzzy set. Type reducing an interval type-2 fuzzy set results in crisp interval or discretisized values. Next, type reduction for both type-2 fuzzy sets will be briefly reviewed. The most commonly used type-reduction method of type-2 fuzzy rule base systems is the ‘centroid type reducer’. In this section, we will only review one type reducer in order to present how a well-known type reduction can be applied. More detailed information about other types of type-reducers can be found in [Mendel, 2001]. A general type reduction works as follows: type-2 fuzzy sets can be represented as a collection of embedded type-2 fuzzy sets [Mendel et al. 2006]. An embedded set is contained in a type-2 set in such a way that for every value of x, there is one value of u (primary membership function), and the secondary membership grade being determined by f(u), Each of these embedded type-2 fuzzy sets has a centroid that can be calculated in different ways one of the common method is the centroid method. Each of these centroid values provides a point in the domain of the type-reduced set. The centroid of a type-1 fuzzy set A, whose domain x∈X is discretized into N points, x1,…xN is given as:
∑ k =1 xk μ A ( xk ) N ∑ k =1 μ A ( xk ) N
cA =
(5.20)
Similarly, the centroid of a type-2 set Ã, Ã={(x,u),μÃ(x,u)}, whose x domain is discretized into N points, so that Ã= ∑k=1 [∫u∈Jxi fxi(u)/u]/xi, can be defined using the Extension Principle as follows:
★ ★f
C A% = ∫θ1∈J x L ∫θ N ∈J xN ⎡⎣ f x1 (θ1 ) L 1
In (5.21),
C A% is a type-1 fuzzy set,
xN
N ∑ k =1 xkθ k (θ N ) ⎤⎦ ∑ kN=1θ k
(5.21)
★ represent the general t-norm (minimum or
product) and ∫ is the union operator. Every combination θ1,…, θN, and its associated secondary grade f x1 (θ1 ) L f xN (θ N ) forms an embedded type-2 set Ã, and
★ ★
we can express them as follows: a(θ ) =
∑ kN=1 xkθ k ; ∑ kN=1θ k
★ ★f
b(θ ) = f x1 (θ1 ) L
xN
(θ N )
(5.22)
The computation of CÃ involves computing the tuple (a(θ),b(θ)) as many times as possible, possibly infinite times. A special sequence of computations to obtain CÃ for discretized intervals is as follows: ¾ ¾
Discretize the x domain into N points, x1,…, xN Discretize each J xk (the primary membership of xj) into a suitable number of points, say Mk.
164
¾ ¾
5 Modeling Uncertainty with Improved Fuzzy Functions
Enumerate all the embedded type-1 sets: there will be ∏k Mk, k=1…N, embedded type-1 fuzzy sets. Compute the centroid using (5.21), i.e., compute the tuples (ak,bk), say p times, l=1,2,…,∏kMk, k=1…N, where ak and bk are given in (5.22), respectively. In this case, p=∏kMk, k=1…N.
A simpler choice would be to implement interval type-2 fuzzy sets and use a type reducer for interval type-2 fuzzy sets. The generalized centroid of an interval type-2 fuzzy set à over the domain X is given by: GC A% = ∫θ1∈J x L ∫θ N ∈J x 1 1
N
∑ kN=1 xkθ k = [Cl , Cr ] ∑ kL=1θ k
(5.23)
In (5.23) JxN is the secondary membership grade at N in the secondary membership function Jx and x∈X. Since it is a crisp interval, the type-reduced set only needs two end-points to define it, Cleft and Cright, i.e., Cl, Cr. Briefly, the goal in type reduction is to identify the embedded sets that represent the centroid of a given set. With generalized type-2 fuzzy sets, the abundance of embedded sets make type-reduction by far the most computationally expensive stage of the inference process. In this work, we try to solve this problem with a different type-reducer method based on a semi-parametric “cased-based” method. The IT2FRB methods presented above are most commonly used uncertainty modeling tools based on fuzzy rule base methodologies. One of the challenges of these methodologies are that they require quite a lot of parameter initializations. Generally, these system models are subjective, since expert knowledge is required to decide on the shapes and parameters of fuzzy sets, e.g., triangular, Gaussian, etc, their parameters, number of rules, etc. These models also assume that antecedent variables are separable and neglect the interactivity assumption, which increases the number of fuzzy operators to conduct fuzzification. Recent investigations mostly lean towards new approaches to find different shapes of fuzzy sets and further reduce complexity and subjectivity of type-2 fuzzy systems. One of the recent investigations [Uncu, 2003; Uncu and Turksen, 2007], to be discussed next, implements a discrete interval type-2 fuzzy rule base algorithm, DIT2FRB, where the interval membership function are obtained from fuzzy c-means (FCM) clustering method and the consequents are represented with linear functions. The discretization concept in DIT2FRB strategy is similar to the the new type-2 strategy of this work.
5.3.4 Discrete Interval Type-2 Fuzzy Rule Bases (DIT2FRB) Mendel identifies type-2 fuzzy set parameters associated with each variable in each rule mostly using supervised learning methods. Turksen [1999] introduced the concept of implementing Fuzzy c-means (FCM) [Bezdek, 1981a] clustering algorithm to identify type-2 fuzziness in system models. In [Kilic, 2002; Uncu et al, 2004a; Uncu and Turksen, 2007] FCM clustering algorithm is used to identify the structure of the system models. In [Kilic, 2002] a new type-2 FCM algorithm is constructed using
5.3 Conventional Type-2 Fuzzy Systems
165
upper and lower values of the overlapping constant of clusters. Generally, in these studies, they assume that the selection of the degree of fuzziness parameter, m, of the FCM clustering algorithm is the only source of uncertainty and identify embedded type-1 fuzzy sets for each m to represent Discrete Interval Valued Type-2 Fuzzy Rule Base System (DIT2FRB). Exhaustive search methods are used to optimize the parameters of fuzzy rule base structures. Here we will briefly explain the most recent version of these types of type-2 studies [Uncu and Turksen, 2007]. Let mr be the rth level of fuzziness, mr∈{m1.. mM}, where M is the number of disjoint m values. Thus, rth embedded type-1 FRB structure is identified by using mr as the learning parameter. μAr represents the membership function obtained from using mr value. Each fuzzy rule represents one embedded type-1 fuzzy set A. r They construct Takagi-Sugeno (TS) rule base structure, represented with R% i , i=1,..,c*, as follows:
% T + b%i R%ir : IF x ∈ X isr A%i THEN y% i = ax
(5.24)
or more specifically, r r r r T r R% i : IF x∈X is Ai THEN yi =ai x +bi
(5.25)
In (5.25) r=1…M, air xT +bir are the regression coefficients associated with the ith rule of the rth embedded type-1 fuzzy system. Thus, the problem of building DIT2FRB systems is reduced to finding type-1 fuzzy rules. They also implement Mizumoto [1989] type fuzzy rule bases system models instead of Takagi-SugenoKang models. Mizumoto fuzzy rule base systems require a defuzzification step to obtain crisp output from output fuzzy sets. They assume interactivity between antecedents, so they assign one joint input-membership function for the complete antecedent part of each rule using Fuzzy c-Means (FCM) clustering algorithm. Implication
Discrete Type 2 Fuzzy Rule Base
New Data Vector
Type Reduction
Fuzzification Identification of Degree of Fire
Aggregation of Deduced Model Output For Each Rule Defuzzification
m-lookup table Deduced Output
Fig. 5.6 DIT2FRB inference strategy by [Uncu, 2003; Uncu and Turksen, 2007]
Their algorithm initially captures one output value for each possible 〈mr,c〉 pair by using the min-max fuzzy inference method. For each datum, xk, k=1,..n, the 〈mr,c〉 that yields the minimum prediction error is retained in an m-lookup table as follows:
166
5 Modeling Uncertainty with Improved Fuzzy Functions
⎛ rkc = ⎜ q yk − yˆ kq ,c ⎝
(
)
2
M
(
= min yk − yˆ kr ,c r =1
)
2
⎞ , q = 1...M , k = 1..n, c = cmin ...cmax ⎟ (5.26) ⎠
Here the cmin and cmax are the boundaries of number of clusters that the algorithm is iterated on. The mr value that reveals the minimum error for the kth data vector, is selected as the best m value for the corresponding data vector and retained in the lookup table. When a new testing vector, x′ is introduced, the first step is the type reduction, as shown in Figure 5.6. They identify the first closest training data vector xk ′ based on Euclidean distance as follows:
( (
))
n ⎛ ⎞ xk ′ = ⎜ xk d xk , xk ′ = min d xk , xk ′ , k = 1...n ⎟ k =1 ⎝ ⎠
(
)
(5.27)
The model output of a given observation x′ using the optimum embedded type-1 fuzzy system model is determined based on the value of the m parameter of xk ′ in the lookup table. After selecting the best parameter, the inference is as the same as the type-1 fuzzy rule base inference as explained in Chapter 2. Based on the rule base structure identified, e.g., Takagi-Sugeno, Mizumoto, etc., their reasoning calculations differ. A detailed mathematical background of the algorithm can be found in [Uncu, 2003; Uncu and Turksen, 2007]. The challenging part of DIT2FRB method is to determine the initial parameters, e.g.,: ¾
M value to discretisize the membership interval into discrete membership values.
¾
Boundaries of number of clusters, cmin and cmax
¾
The type of the fuzzy operators, MIN, MAX, PRODUCT, etc., used in fuzzy rule base structure.
With DIT2FRB, Uncu and Turksen [2007] identify embedded Type-1 Fuzzy Rule Base (FRB) for each selected m to represent Discrete Interval Type-2 FRB. Their fuzzy rule base is essentially a collection of embedded type-1 fuzzy systems. They assume interactivity between antecedent variables, i.e., the antecedent fuzzy sets are not independent from each other and only one membership function is defined for the entire antecedent part of each rule, which eventually combine fuzzification and antecedent aggregation steps to one, reducing the amount of work. Their inference mechanism does not require a defuzzification step when first order Takagi-Sugeno-Kang Fuzzy Systems (TSK) [Takagi and Sugeno; 1995; Sugeno and Kang; 1996] used; only a type reduction is conducted using an additional parameter from the learning stage, the m-lookup table. Inference is conducted by assigning one embedded Type-1 FSM for each vector. In a more recent study, Garibaldi and Ozen [2007] investigated the process of modeling the variation in human decision-making processes by applying a fuzzy expert system. They introduced a non-stationary fuzzy inference approach. Their
5.4 Discrete Interval Type-2 Improved Fuzzy Functions
167
underlying idea is that any conventional Type-1 Fuzzy Expert System always provides the same output(s) when supplied with the same input(s). However, experts do not always make the same decisions; even one expert can make a different decision one day from the next. Considering that the expert decisions are vague, they designed a method, which accounts these variations in the outcome of the system. Type-1 fuzzy systems are deterministic and do not allow this variation because of the obvious fact that given the same inputs the outputs are usually the same. So, in [Garibaldi and Ozen, 2007] such vagueness in measurements is investigated by initially perturbing the membership function parameters slightly and then constructing various type-1 fuzzy expert systems by combining the fuzzy-output and defuzzification to get a crisp output. They do not utilize the type-2 fuzzy sets in their models. Hwang and Rhee [2007] focus on the uncertainty associated with the fuzzifier parameter, i.e., level of fuzziness parameter denoted with m, that controls the amount of fuzziness of the final –partition in the fuzzy -means (FCM) clustering algorithm [Bezdek, 1981a]. To design and manage uncertainty of the fuzzifier parameter, they extend a pattern set to identify interval type-2 fuzzy sets using two fuzzifiers, which creates boundaries of the footprint of uncertainty (FOU) using the interval valued fuzzifier. Then, they incorporate this interval type-2 fuzzy set into FCM clustering to observe the effect of managing uncertainty from the two fuzzifiers. They also provide some solutions to type-reduction and defuzzification (i.e., cluster center updating and hard-partitioning) in FCM clustering algorithm. This study is another approach to finding uncertainty interval of membership functions based on FCM clustering parameters, namely the fuzzifier, m. It should be pointed out that there are other parameters in the fuzzy systems that should be considered to obtain the unknown uncertainty interval. This chapter will present a new approach that focuses on these topics using the new Improved Fuzzy Clustering methodology. The new type-2 fuzzy system modeling tools are inspired by the last type-2 fuzzy rule base systems that identify the uncertainty in fuzzy systems based on fuzzy clustering parameters. Next, we introduce the new Type-2 Improved Fuzzy Functions strategy based on discrete interval valued type-2 fuzzy membership values.
5.4 Discrete Interval Type-2 Improved Fuzzy Functions The type-2 fuzzy systems that are closest to the type-2 improved fuzzy functions, e.g., [Uncu, et al., 2004; Uncu and Turksen, 2007; Hwang and Rhee, 2007], demonstrated better performances than conventional type-1 and type-2 fuzzy systems in terms of modeling error. In these studies it is argued that the computation complexity is reduced to a degree, i.e., less number of fuzzy operators, an easier typereduction method, etc., compared to previous interval type-2 fuzzy systems of [Mendel and Karnik, 1999]. Nevertheless, these approaches are based on fuzzy rule base structures. We are replacing these fuzzy rule base structures with “Fuzzy Functions” to improve performance and require even less fuzzy operations and operators as explained in Chapter 4. Furthermore, Uncu and Turksen [2007] and Hwang and Rhee [2007] only deal with the uncertainty of system models using
168
5 Modeling Uncertainty with Improved Fuzzy Functions
one of the learning parameters of FCM clustering method; that is the fuzziness constant (m) of FCM clustering algorithm, leaving out other important sources of uncertainties of fuzzy systems. Since in these systems Takagi-Sugeno-Kang [Takagi and Sugeno; 1995; Sugeno and Kang; 1996] rule base structures are employed, a general function structure is used for each fuzzy rule. Employing different mathematical models for each rule, viz. fuzzy functions, in structure identification as opposed to building one general structure as in latter fuzzy rule base systems, or as in type-1 improved fuzzy functions (T1IFF) strategies, can also potentially identify some of the uncertainties in structure identification. Hence, in this work we propose a new uncertainty-modeling framework for T1IFF systems to identify uncertainties that may arise from two separate concerns: ¾
selection of the learning parameters of the proposed system,
¾
determination of the structure of fuzzy functions.
5.4.1
Background of Type-2 Improved Fuzzy Functions Approaches
During structure identification of the type-1 improved fuzzy functions (T1IFF) system, each data vector is assigned a membership value in each fuzzy set. Improved membership values are obtained through fuzzy membership functions of IFC method. We assume interactivity between each input variable (antecedent), so for each rule (cluster) there is only one type-1 membership function identified via IFC algorithm5. The challenge of working with T1IFF models is that; since type-1 fuzzy sets express belongingness of an object, xk, to a fuzzy set A by a crisp membership value, μA, they cannot capture the uncertainties in identifying membership functions nor the uncertainties in determining structure of fuzzy functions. In order to remedy this problem, we propose to implement type-2 fuzzy sets into fuzzy functions systems. Next, we will present the underlying representation of the interval membership functions into fuzzy functions systems. Generally; Fuzzy c-means (FCM) clustering algorithm is employed in order to identify fuzzy sets during structure identification of recent fuzzy systems based on fuzzy rule bases [Emami et al., 1998; Uncu and Turksen, 2007; Delgado et al., 1997; Hwang and Rhee, 2007] as well as fuzzy systems based on fuzzy functions [Turksen, 2008; Turksen and Celikyilmaz, 2006; Celikyilmaz, 2005]. In chapter 3, we introduced Improved Fuzzy Clustering (IFC), which can re-shape membership values in order to predict the output variables in local models. One of the common parameters of the FCM clustering and IFC method, is the degree of fuzziness, m, viz., a constant to represent overlapping degree of identified clusters. Fuzziness parameter has been investigated by researchers, because it is believed to have the capability to identify the uncertainty intervals of membership functions of any
5
The membership values obtained from clustering method are not mapped onto each individual variable due to interactivity assumption between variables. Only one input membership function is identified to define fuzzy sets of the antecedents.
5.4 Discrete Interval Type-2 Improved Fuzzy Functions
169
given system [Ozkan and Turksen, 2007]. Nonetheless, there is also conflicting research on fuzziness of clustering methods. In [Yu et al, 2004], a method to choose the optimum weighing exponent is investigated around the mid-points of the clusters. They proved that, m=2 is not a reasonable heuristic rule for FCM clustering algorithm. In [Ozkan and Turksen, 2007], upper and lower bounds of the degree of fuzziness is approximated to be [m-lower=1.4, m-upper=2.6], by investigating the FCM model behavior around mass center for large m values and around cluster centers for smaller m values. They suggest that the fuzziness interval represents the uncertainty interval of the fuzzy systems. This interval is investigated and identified via a Taylor expansion of actual membership functions. Nonetheless, further experimental analysis is required to prove the performance of this interval in system modeling practices. In a similar study, Celikyilmaz and Turksen [2008b] investigated whether similar boundaries could be identified for different fuzzy clustering methods as well. On top of the uncertainties induced by the determination of the level of fuzziness parameters, uncertainties in fuzzy function systems may also arise from determination of the structure of the local fuzzy functions. By the structure of the fuzzy functions, we generally mean different forms of membership value transformations used to identify the local system fuzzy functions. A unique property of fuzzy function modeling strategy is that, membership values, μi(x), i=1,…,c, and their transformations are introduced to local functions as additional predictors. The way these membership values are used to map the given input dataset to a feature domain depends on the user and one may use many different variations of membership values, e.g., power transformations, (μi(x))p, p>0, logarithmic, ln((1μi(x))/μi(x)), etc. These power sets of membership values help to identify the structure of fuzzy functions. Hence, optimum membership forms to structure a fuzzy function of a cluster are an unknown parameter prior to the model execution. In brief, there are uncertainties in identifying membership values, as well as fuzzy functions, which are formed based on these membership value calculation equations. Therefore, in this chapter, in order to identify these uncertainties we introduce a new architecture with interval valued type-2 fuzzy sets and define the upper and lower boundaries of their interval. Hence, one of the parameters we use to identify the interval valued type-2 fuzzy sets is by analyzing different structures of fuzzy functions. It should be summarized again that the fuzzy clustering algorithms are used to generate the interval valued type-2 membership value sets with a clustering algorithm. Then, one defines a membership function via a curvefitting technique and this is used symbolically as the membership function to represent the scatter of the membership value set. In this work we use the fuzzy clustering methods, e.g., Fuzzy C-Means Clustering [Bezdek, 1981a], or Improved Fuzzy Clustering [Celikyilmaz and Turksen, 2007b] algorithm to obtain membership functions. In the following, when we use the membership function term, we mean the membership value calculation equation of a corresponding fuzzy clustering algorithm. As pointed out in the previous section, in modern practices, interval type-2 fuzzy logic systems are most efficient uncertainty modeling methods because it is easier to conduct type-reduction with them. Secondary membership functions are not required in interval valued type-2 fuzzy functions. Membership values of
170
5 Modeling Uncertainty with Improved Fuzzy Functions
interval valued type-2 fuzzy sets have secondary membership values equal to 1 and one tries to identify the upper and lower bounds of these membership functions to form an interval. Thus; interval type-2 fuzzy sets, Ã, map the domain of base variable onto membership values in the interval of [0,1] as follows;
μ A% ( x) : x → 1/ u , u ∈ [ μ AL% ( x), μ UA% ( x )]
(5.28)
In (5.28) μ AL% ( x) and μ UA% ( x) are lower and upper membership functions, u is the primary membership value assigned to each datum x. One of the most common ways to fuzzify a given observation, which is most commonly used in fuzzy system modeling approaches, is by using a membership function. The shape of the membership functions is generally selected a priori and a surface or a curve is fitted depending on whether the antecedent is represented with a single multi-dimensional fuzzy set or aggregation of fuzzy sets of each input dimension. Hence, membership of a crisp observation, x', in a type-1 fuzzy set A, μA(x'), is a function of x' and pM is the set of parameters which mathematically defines the fitted membership curve. Thus μA(x') is formulated as
μA(x')=f(x'; pM)
(5.29)
Let A be linguistic label on the universe of discourse of base variable x with type-1 membership function characterized by a set of membership function parameters pM=(p1M,p2M,…,pnpM), where np is the number of parameters used to characterize the membership function, ptM is the membership function parameter, for which the domain is denoted by PtM, i.e., ptM∈PtM. If we assume that all ptM are independent parameters, the domain of set of membership function parameters, PM, can be defined as the Cartesian product of the domains of its elements, i.e., P M = P1M × P2M × ... × PnpM . For instance, rth specific combination of parameters, pM,r, is defined as pM,r=(p1M,r,p2M,r,…,pnpM,r). Thus, one can define PM as, PM= { pM,r} , r=1,…,nr, where nr is the number of all possible value that pM,r can take. Then the number of all possible values for ptM can take is defined by | PtM |, then NM=∏t | PtM |, t=1,…,np. The challenge is that, if PtM has a continuous domain, then there will be infinitely many pM,r. Based on this information, one can define the primary type-1, interval valued and discrete interval valued type-2 membership functions as follows: Definition 5.1. Primary Membership Function: The type-2 membership function for a given parameter combination pM,r , is defined as the rth primary membership function of interval type-2 fuzzy membership function, μÃ(x) and it is denoted with μrA(x). Definition 5.2. Interval Type-2 Membership Function: If the primary membership function of an interval type-2 membership function can be characterized by the set of parameters pM=(p1M,p2M,…,pnpM) and PM is a “continuous” domain, then interval valued type-2 membership function is the collection of primary membership functions determined for all possible values of pM,r ∈PM.
5.4 Discrete Interval Type-2 Improved Fuzzy Functions
171
Definition 5.3. Discrete Interval Type-2 Membership Function: If the primary membership functions of an interval type-2 membership function can be characterized by the set of parameters pM=(p1M,p2M,…,pnpM) and PM is a “discrete” domain, then interval valued type-2 membership function is the collection of primary membership functions determined for all possible values of pM,r ∈PM. Therefore, the membership function of a discrete interval type-2 fuzzy set A can be reformulated as follows:
μÃ(x)={ μrA(x)}, r=1,…,nr
(5.30)
In (5.30), μ Ar ( x) is the rth primary membership function of a datum x and μÃ(x) is the collection of all the primary membership functions that define an interval and nr is the total number of possible values of pM . We explain the idea above with an example. Let a Gaussian membership function (5.31) be defined with two parameters, the mean, m , and the standard deviation, σ . Let the mean parameter be a constant value, and standard deviation be a variable. The primary membership function of an interval type-2 fuzzy set formed with Gaussian membership function is defined as having a fixed mean, m , i.e., Pm =0, and an uncertain standard deviation parameter, σ , i.e., Pσ = nr. ‘nr’ is the number of possible values that the standard deviation parameter may get. The rth primary membership function of the discrete interval valued type-2 fuzzy set A, μ Ar ( x) for the particular Gaussian membership function is defined as;
⎛ 1 ⎛ x − m ⎞⎞
μ Ar ( x) = exp ⎜ − ⎜ r ⎟ ⎟ , ∀σ r ∈ {σ 1 ,...,σ nr } , r = 1,..., nr ⎝ 2 ⎝ σ ⎠⎠
(5.31)
Since membership values are within [0,1] interval, a type-2 membership function for a given datum x of every set of parameters, will eventually have a maximum and minimum membership value, the parameters of which will define the upper and lower limits of the parameter combination respectively. In this example, let the σ and σ U represent the lower (L) and upper(U) limits of the standard deviation parameter, which results in minimum and maximum values of membership values for a given data vector in a fuzzy set A. Then, the boundaries of the discrete interval valued type-2 membership function can be defined for this example using the upper and lower values of the uncertain standard deviation parameter, σ , as follows: L
μ Ar ( x) = μ ( x, mean,σ U ) ⎫⎪ ⎬ , r = 1,..., nr μ Ar ( x) = μ ( x, mean,σ L ) ⎪⎭ In (5.32)
(5.32)
μ Ar ( x) is the upper primary membership function which uses upper limit
value of the uncertain standard deviation parameter, σ U and μ Ar ( x) is the lower primary membership function which uses the lower limit value of the uncertain standard deviation parameter, σ L . Figure 5.7 displays the graph of the membership
172
5 Modeling Uncertainty with Improved Fuzzy Functions
function of a given variable, x, i.e., x={0,0.01,…,10}, with mean, mean=5, and uncertain standard deviation, σ = [1,1.01,.., 2] .The upper and lower boundaries of the interval membership function in Figure 5.7 are drawn in bold.
Fig. 5.7 Interval Type-2 membership function with fixed mean and uncertain standard deviation. The shaded area includes infinite number of membership values.
Each point within the uncertainty interval denotes a membership value obtained from one embedded membership function, which represents a type-1 fuzzy set. In literature, this interval is denoted as the ‘footprint of uncertainty’ [Mendel, 2001] as shown in Figure 5.7. In this work, one of the aims of the proposed methods is to identify the footprint of uncertainty with the proposed methodologies. The latter representation of discrete interval type-2 fuzzy sets will be the foundation of the Discrete Interval Valued Type-2 Improved Fuzzy Functions (DIT2IFF) approach to be introduced next. We will start with the identification of the uncertainty interval of fuzzy functions based on parameters of the Fuzzy c-Means (FCM) clustering method. We will then move on to the IFC method to represent the uncertainty interval of the improved fuzzy function systems.
Representation of Type-2 Fuzzy Sets Using FCM Clustering Method The fuzzy function systems implement the FCM [Bezdek, 1981a] clustering method and then identify local fuzzy functions using membership values obtained from the FCM clustering method. The parameters of the membership value calculation equation of the FCM clustering algorithm are: the number of clusters in the system, c, the degree of fuzziness parameter (weighing exponent) m, which defines the degree of overlap between the clusters, and the cluster centers of each cluster defined for each m, υX,m ∈X⊂ℜnv, where nv is the number of the input variables in the system. We may represent only a part of the learning parameters of the type-2 fuzzy functions with FCM clustering algorithm to build models for Discrete Interval Type-2 Fuzzy Functions (T2FF) strategies in the following way. Let pL represent each of the learning parameters of FCM clustering algorithm, pL ={m,υX,m,c}, where L implies learning, the membership function of the FCM clustering algorithm for cluster i can be defined as follows:
5.4 Discrete Interval Type-2 Improved Fuzzy Functions
173
μi (x) = f (x, m, υX,m,c )
(5.33) th
Using the parameters defined in (5.33), membership value of any k input vector, xk, k=1,…,n, in ith cluster is calculated with the membership value calculation equation as follows: ⎛ ⎜ c ⎛ d xk ,υix, m μik ( x) = ⎜ ∑ ⎜ m ⎜ j =1 ⎜ d xk ,υ xj , m ⎜ ⎝ ⎝
( (
(
In (5.34) d xk ,υi
x ,m
) )
2 ⎞ ⎞ m −1 ⎟ ⎟ ⎟ ⎟ ⎟ ⎠ ⎟ ⎠
−1
(5.34)
) is the distance defined as: (
)
d xk ,υix,m = ( x − υix,m ) A( x − υix,m )T
(5.35)
The norm matrix A in (5.35) is equal to the covariance inverse of the training matrix. Since each input variable is normalized to mean=0 and standard deviation =1, A is the identity matrix, d(xk,vi) is the Euclidean distance. Therefore the membership function of the FCM clustering algorithm is determined by the combination of parameters, i.e., (m,υX,m,c). It should be noted that, based on the FCM clustering iterative algorithm, the cluster centers are calculated as follows:
⎛
n
( )
υix ,m = ⎜ ∑ μik ⎝ k =1
m
⎞ xk ⎟ ⎠
∑ ( μik ) n
m
, ∀i = 1,..., c
(5.36)
k =1
In cluster center calculation function in (5.36) the 〈m,x,μi〉 are input variables and the cluster centers are the output variables. They are measured values and therefore they are not the parameters of the FCM clustering. Cluster centers are however affected by different values of m since it is one of the input parameters of the cluster center function in (5.36). However, for FCM clustering algorithm, one still needs to determine the optimum number of clusters, c*, and the optimum degree of fuzziness value m* before calculating cluster centers. (*) indicate the optimum value of the corresponding parameter. Here we will assume that the optimum number of cluster is identified by a cluster validity measure or by an exhaustive search based on systems’ performance measure [Turksen and Celikyilmaz, 2006]. Therefore, c* will be assumed as a fixed variable of FCM clustering algorithm. From the latter discussion, one may conclude that the only uncertain parameter that is left to identify the uncertainty interval is the degree of fuzziness, m. Thus, the domain of the learning parameters, PL, of the FCM clustering method is defined for the changing values of m parameter and is defined as, PL={c*,υX,r | mr}, r=1,…, nr
(5.37)
where nr is the number of different values of the degree of fuzziness parameter of the FCM. The parameter list PL in (5.37) indicates that, the parameter, c*, is fixed and the degree of fuzziness parameter, m, is kept uncertain and vX,r is measured during the FCM clustering process based on changing values of m. These
174
5 Modeling Uncertainty with Improved Fuzzy Functions
parameters make up of a part of the learning parameters of the type-2 fuzzy functions approaches. The affect of choosing the right m value on the cluster centers and membership values can be demonstrated with an example. Here, we used an artificial sparse dataset of 100 data samples including one input variable, x, and a single output variable, y. This dataset was introduced in [Celikyilmaz and Turksen, 2007a] which is depicted in Figure 5.8. The dataset is clustered into 2 clusters, i.e., c* is fixed to c=2, using mr ={1.2, 1.5, 1.8, 2.0, 2.5, 3.2}, nr=6 representing each six different m values. Artificial Dataset 4
output - y
2 0 -2 -4 -1.5
-1
-0.5
0
0.5 input - x 1
1
1.5
2
2.5
Fig. 5.8 Artificial dataset with one input variable (x1) and a single output (y)
For each x vector in the dataset, x={x1, x1,.., x100}, a membership value is calculated of each cluster i,i=1,2, using the membership value calculation equation of FCM clustering as follows:
⎛ ⎜ ⎛ d x ,υ x,mr k i c ⎜ μik ( x) = ⎜⎜ ∑ ⎜ x ,mr j =1 mr ⎜ ⎜ d xk ,υ j ⎜ ⎝ ⎝
( (
−1 2 ⎞ r ⎞ m −1 ⎟
) ⎟⎟ ) ⎟⎠
⎟ , ∀i = 1, 2, r = 1,.,6, k = 1,..,100 ⎟ ⎟ ⎟ ⎠
(5.38)
r
In (5.38), υix ,m is the cluster center of cluster i calculated using the degree of fuzziness parameter mr. For each discrete mr, one can identify one discrete membership function, i.e., the membership values of each cluster that form scatter clouds and the membership function is the idealized distribution of these membership values of a particular cluster. Hence, interval type-2 membership values of FCM clustering algorithm can be constructed by:
μ% i ( x) = {μir ( x)}, ∀i = 1, 2, r = 1,., 6
(5.39)
Membership value calculation equation of a particular cluster for each discrete mr is shown in Figure 5.9 and Figure 5.10, which are bounded with the degree of fuzziness, i.e., [mL,mU]=[1.2,3.2]. Natural outcome of this definition is that, one can identify upper and lower membership values, μir ( x), μir ( x) for each cluster i,
5.4 Discrete Interval Type-2 Improved Fuzzy Functions
175
eventually identifying the uncertainty interval of the membership value calculation equation. In Figure 5.10 we observe the changes in cluster center locations for cluster 1 with respect to the changes in m values. It can be seen that, one can define an interval of membership values based on the m values which form an interval, i.e., [m-lower, m-upper]. In this example the m interval is determined by the maximum and minimum values of the mr values.
1 m1=1.2 m2=1.5 m3=2.0 m4=2.3 m5=2.5 m6=3.2
0.8 0.6 0.4 0.2 0 -1.5
-1
-0.5
0
0.5
1
1.5
2
2.5
Fig. 5.9 Membership values for cluster 1 using the artificial dataset determined by changing m values
m 1=1.2
m2=1.5
m3=2.0
m 4=2.3
m5=2.5
m6=3.2
2 1 0 -0.6
-0.59
-0.58
-0.57
-0.56
-0.55 X
-0.54
-0.53
-0.52
-0.51
-0.5
Fig. 5.10 Changing cluster center locations of cluster 1 based on changing m values. The square dots in bold indicate the cluster centers for changing m values.
Representation of Type-2 Improved Fuzzy Sets Using Improved Fuzzy Clustering
In the previous section, the representation of the uncertainty interval of the membership functions of the standard Fuzzy c-Means (FCM) clustering algorithm is presented based on discretization of the level of fuzziness, m, parameter. This section presents the representation of the uncertainty interval when one applies the new improved fuzzy clustering (IFC) method for the determination of fuzzy functions. The IFC method has a unique membership value calculation equation that can capture improved membership values to reduce the error of local fuzzy functions as follows:
176
5 Modeling Uncertainty with Improved Fuzzy Functions
1/( m −1) ⎛ ⎞ ⎛ ( d )2 + ( y − h (τ , wˆ ) )2 ⎞ c ik k i ik i ⎜ ⎟ imp ⎜ ⎟ μik = ⎜ ∑ j =1 2 2 ⎟ ⎜ ( d ) + ( y − h (τ , wˆ ) ) ⎟ 1
−1
(5.40)
In (5.40), hi(τi|ŵi) is the interim fuzzy function of cluster i to approximate an output value, ŷk, using the input matrix, τi, which is composed of the membership values and/or their user defined transformations, e.g., τi′=[( μiimp ) (( μiimp )m) e μ ]. ŵi‘s are the interim fuzzy function parameters of membership value calculation equation of the IFC method. For instance, when the LSE is used to approximate the interim fuzzy functions, the latter sample fuzzy function structure, τi′, where three different membership value transformations are used to form the interim fuzzy functions, (nm=3) takes the form: imp i
(
)
(
hi (τ i , wˆ i )=wˆ 0,i +wˆ1,i μiimp +wˆ 2,i μiimp
τ i ∈ R n×( nm +1)
⎡ imp ⎢1 μi ,1 = ⎢M M ⎢ ⎢ imp ⎢⎣1 μi , n
(μ ) imp i ,1
M
(μ ) imp i ,n
)
m
( )
⎤ ⎥ M ⎥ , wˆ i ∈ R ( nm +1)×1 = [ wˆ 0,i ⎥ μ imp ⎥ e i ,n ⎥ ⎦ e
imp
μ +wˆ 3,i e i
μiimp ,1
(5.41) wˆ1,i
wˆ 2,i
wˆ 3,i ]T
It is to be noted from the new IFC membership value calculation equation in (5.40) that, during optimization, the error between the actual output yk and the estimated output, ŷk, obtained from approximated interim fuzzy functions of each cluster, hi(⋅), is used as additional input variables of the new membership value calculation equation of the IFC algorithm. Hence, input variables of these fuzzy functions, hi(⋅), are membership values and their user defined transformation. One should decide on the type of membership value transformation to be used to identify the behavior of the output variable. These eventually constitute the structure of local fuzzy functions. When the IFC method is used during the structure identification of the local fuzzy functions, an additional parameter is introduced, denoted with ‘τ’ (spelled TAO) which is an interim matrix that represents the structure of interim fuzzy functions constructed during the IFC algorithm, as shown in the sample in (5.41). Thus, this introduces an additional uncertainty during the extraction of the membership values, since the error of the interim fuzzy functions of each cluster is an input variable for the new membership value calculation equation. During IFC algorithm, we assume that the interim fuzzy functions hi(τi|ŵi) all have the same structure in an experimental trial. Let τ represent the structure of the input interim matrix composed of membership values and their possible transformations, i.e., squared membership value, m-powered membership value, logarithmic transformation of the membership value, etc., to form the structure of the interim fuzzy functions during the IFC algorithm. Since, different types of membership values make up the structure of the τ-matrix, in shortτ, and membership values are affected by the degree of fuzziness
5.4 Discrete Interval Type-2 Improved Fuzzy Functions
177
variable m, each different structure of τ for each mr is represented as τs⊂ℜnif . Here, nif, i.e., number of interim fuzzy functions, is the number of different types of interim fuzzy functions composed of different membership value forms, r represents each discrete value of m variable, r=1,…, nr, s represents each different interim fuzzy function, s=1,…, nif, as shown in Table 5.1. For instance, one interim fuzzy function might use only membership values, and another might use membership values and their logarithmic transformations. One should identify the optimum one that would improve the performance of the model; hence, there is an uncertainty in identifying the interim fuzzy function structures. The domain of learning parameters, PL, of the new membership value extraction function of the Improved Fuzzy Clustering algorithm can be re-defined as, Table 5.1 Improved Fuzzy Functions Parameter Representation
Parameter
Name
m∈P
Level of fuzziness
τs∈ Pτ L
Interim Fuzzy Function Matrix composed of only membership values and/or their different transformations. (System) Fuzzy Function Matrix composed of original input variables, membership values and/or their transformations.
r
L m
Φψ∈ PΦL
PL{ PmL , Pτ L }={ c*,υX,r,s| mr, τs}, r=1,…, nr, s=1,…, nif
Range r=1…nr
s=1…nif
ψ=1…nf
(5.42)
In (5.42) PmL indicates the domain of parameters identified by level of fuzziness, m, and Pτ L indicates the domain of parameters identified by different interim matrix structure. Let nr represent the values of the level of fuzziness of the IFC and nif is the number of different structures of interim fuzzy functions. The parameter list PL in (5.42) indicates that the parameters, c*, and υX,r,s are fixed and two different parameters namely the degree of fuzziness parameter mr and the structure of the input matrix τs are kept uncertain. If the Cartesian product of each discrete values of these two parameters, 〈mr,τs〉 form each tth IFC model, then there would be t=1,…,nr*nif different IFC models, where (nr*nif ) is the total number of parameters and at this point only two types of parameters are used to describe the interval valued type-2 improved membership values with IFC method. Up to this point, we presented the representation of discrete parameters of interval valued type-2 membership functions due to uncertainties in forming type-1 improved membership functions. There are other uncertainties that one should take into account when building models using type-1 fuzzy function systems. It is mentioned earlier that based on the structure of type-1 improved fuzzy
178
5 Modeling Uncertainty with Improved Fuzzy Functions
functions (T1IFF) systems, local system fuzzy functions are identified for each cluster after the structures of fuzzy sets are captured. The difference between the “Interim Fuzzy Functions”, hi(τi|ŵi), and the “System Fuzzy Functions”, in short the “Fuzzy Functions”, fi(Φi|Ŵi), presented in Chapter 4, should be stated at this point. The “Interim Fuzzy Functions” are used to approximate the membership values in Improved Fuzzy Clustering (IFC) algorithm. The parameters of interim fuzzy functions of cluster i, ŵi, are identified using the matrix, τi, which is composed of the membership values and/or their transformations, evaluated at each step of the IFC algorithm, excluding the original input variables. These functions are called the interim functions since they are only used for finding the improved membership values. Then, after the improved membership values, μiimp(x), are obtained as a result of the IFC algorithm, the parameters of the “System (Local) Fuzzy Functions” in short, “Fuzzy Functions”, denoted with Ŵi, are approximated for each cluster using the matrix, Φi, (spelled FEE), which composes of the original input variables of nv different candidate input variables, x={x1,x2,…,xnv}, their membership values of the corresponding cluster, and their possible user defined transformations. For instance a fuzzy function of cluster i, fi(Φi|Ŵi), can be identified using a sample input matrix structure, Φi′, e.g., Φi′=[( μiimp ) (( μiimp )m) e μ (x1) (x2) (x3)], where there are nm=3 different membership values transformations, and nv=3 different input variables. For instance, when the LSE is used to approximate the fuzzy functions, fi(Φi|Ŵi), using Φi′ matrix, the fuzzy functions, their coefficients and the input matrix used to identify these parameters are denoted as follows: imp i
(
)
(
)
fi Φ i ,Wˆ i =Wˆ0,i +Wˆ1,i μiimp +Wˆ2,i
Φ′i ∈ℜ n×( nm + nv +1) nm =3, nv =3
⎡ imp ⎢1 μi ,1 = ⎢M M ⎢ ⎢ 1 μi*,n ⎣⎢
(( μ ) )+Wˆ (e ) + Wˆ m
imp i
(μ )
imp m i ,1
3,i
e
M
(μ )
imp m i ,n
e
μiimp ,1
x1,1 M M μiimp , n xn ,1
μiimp
x1,2 M xn ,2
4,i x1
⎤ x1,3 ⎥ M ⎥, ⎥ xn ,3 ⎥ ⎦⎥
+ Wˆ5,i x2 + Wˆ3+ 3,i x3
(5.43)
Wˆi ∈ℜ( nm+ nv +1)×1 = [Wˆ0,i L Wˆnm ,i Wˆnm +1,i Wˆnm + nv +1,i ]T nm =3, nv =3
A unique property of the type-1 improved fuzzy functions (T1IFF) strategies is that in approximating fuzzy functions, the membership values are used to map the input space onto a feature space by using them as additional predictors and different functions can be defined as a result. Based on the type and number of different forms of membership values, the matrix to identify the fuzzy functions, Φψ, ψ=1,…,nf, can take on ψ (spelled SIGH) different function types, which would be an additional parameter to determine in optimizing the model parameters. Note that, the interim fuzzy functions and system fuzzy function of each cluster of a particular fuzzy functions model may have different membership value transformations, which is a unique property to be presented in this chapter. Each set of parameters can form one fuzzy function structure for each cluster i, i=1…c*, c* being the optimum number of clusters. Consequently, the domain of
5.4 Discrete Interval Type-2 Improved Fuzzy Functions
179
the T1IFF can be represented with the learning parameters of membership functions as well as the structure of the fuzzy functions as shown in Table 5.1 by: PL={ Pm , Pτ , PΦ }={ c*,υX,r,s| mr, τs, Φψ}, r=1,…, nr, s=1,…, nif, ψ=1,…, nf L
L
L
(5.44)
L
Let PΦ be the domain of different local improved fuzzy function structures that one can define and q represent a particular improved fuzzy function structure of type-1. Then, there would be q= 1,…,nr×nif×(nf)c* different type-1 improved fuzzy functions (T1IFF) models within an interval valued type-2 improved fuzzy functions system. In short, we identify the uncertainties of T1IFF models using m, τ, Φ parameters to define the uncertainty interval of membership values and fuzzy functions. These parameters denote in sequence, each of the level of fuzziness (mr), r=1,…,nr, each interim matrix, (τs), s=1,…,nif, to identify different interim fuzzy functions and each input matrix (Φψ), ψ=1,…,nf, to define (local) improved fuzzy functions, keeping the rest of the parameters constant. In the next section, an example to these parameters of the type-2 fuzzy system modeling strategy will be explained with the help of Figure 5.12.
5.4.2 Discrete Interval Type-2 Improved Fuzzy Functions System (DIT2IFF) In this section, we present Discrete Interval Type-2 Improved Fuzzy Functions system (DIT2IFF) – that is based on the new Improved Fuzzy Clustering (IFC) method. The Discrete Interval Type-2 Fuzzy Functions –DIT2FF– which implement the standard FCM clustering method, is an extension of DIT2IFF. Here, we will only give the background of the DIT2IFF systems. The difference between two fuzzy functions strategies, which implement FCM clustering and IFC methods, is explained in Chapter 4 in detail. Hence, we will present a detailed representation of interval valued type-2 improved fuzzy functions in this section. The new DIT2IFF system introduces some unique properties that no other type-2 fuzzy system modeling tools implement. Nonetheless, some of the ideas are inspired by earlier type-2 fuzzy system, the closest one being the type-2 fuzzy rule base approach, in short DIT2FRB [Uncu et al., 2004a; Uncu and Turksen, 2007], which implement discretization of membership values to represent embedded fuzzy rules. In Table 5.2 some of the major differences are listed to emphasize the uniqueness of the proposed DIT2IFF compared to DIT2FRB approach. Next, we will present components of the new structure identification and reasoning methods of the proposed type-2 fuzzy systems approach in detail.
I. Structure Identification of Proposed DIT2IFF Let mr be the rth level of fuzziness, mr∈{m1.. mnr}, where nr is the number of discrete m values, e.g., mr∈{1.1, 1.3,…, 3.5}, and τs be the sth interim fuzzy function structure, τs∈{τ1... τnif}, where nif is the number of different interim fuzzy-function structures to build nif different IFC models. Examples of different interim fuzzy functions can be
180
5 Modeling Uncertainty with Improved Fuzzy Functions
identified with different interim matrixes denoted with τs∈{τ1=[(μi)], ..., τnif=[(μiimp) ((μiimp)p>0) (ln(1-μiimp)/μiimp)]}, or as shown in (5.41). Let t represent each IFC model. Then, one can define t=1,…,(nrif=nr×nif) different embedded improved fuzzy clustering – IFC – models, each are represented with tuple6 of 〈c,mr,τs〉, r=1,…,nr, s=1,…,nif. In addition, a list of local fuzzy function structures are represented with Φψ, ψ=1,…,nf. A fuzzy function of a particular cluster is represented with an improved fuzzy function structure, Φiψ from within the list of improved fuzzy functions. Hence that in one DIT2IFF model, two different clusters may or may not have the same structures, i.e., Φi=1ψ≠Φi=2ψ. A sample fuzzy function of a cluster i using the LSE approximators are shown in (5.43). Here ψ represents a single fuzzy function structure, which comprises of membership values and their possible user defined transformations as additional input variables. Table 5.2 Differences between the and Earlier Type-2 Fuzzy System Modeling Approach
Discrete Interval Type-2 Fuzzy Rule Bases approach (DIVT2FRB) ¾ Based on Fuzzy Rule Bases ¾ ¾ ¾ ¾
¾ ¾
6
Discrete Interval Type-2 Improved Fuzzy Functions (DIT2IFF) ¾ Based on Fuzzy Functions approach Implements FCM clustering dur- ¾ Implements FCM clustering or ing structure identification IFC during structure identification For each cluster a linear hyper¾ Linear or non-linear models can plane is approximated be applied to any cluster The structure of the linear func¾ Different fuzzy function structures tions are identical in each cluster can be used across clusters Membership values are used as ¾ Membership values are used as weighing parameter in defuzzifiadditional predictors to approxication method during inference. mate local functions as well as the weighing parameter in finding crisp outputs during inference. Implements parametric inference ¾ Implements semi-parametric inmethod to estimate membership ference method to estimate memvalues. bership values. Identifies the uncertainty interval ¾ Identifies an uncertainty interval of the system model based on user of the system model based on user defined discretisized fuzzifier, m, defined discretized fuzzifier paparameter. rameter, m, different types of interim fuzzy function structures, τ and local fuzzy functions and their parameters
In this work tuple represent finite sequence of different parameters with no particular order. A tuple containing n objects is also represented with "n-tuple".
5.4 Discrete Interval Type-2 Improved Fuzzy Functions
181
Initially, the embedded IFC models will form the interval membership values as described in the following: Step 1: Clustering with IFC
IFC clustering is applied to partition the training data into c clusters for each parameter set 〈c,mr,τs〉. As a result, c number of cluster centers, represented with υic,r,s(x|y)=(υi,1c,r,s,υi,2c,r,s,…,υi,nvc,r,s,υi,nv+1c,r,s) and membership values, μic,r,s(x|y) are captured in input-output (x|y) space. Implementation of IFC into T1IFF strategies for a set of variables, 〈c,mr,τs〉, is presented in Chapter 4, section 4.4. Next, membership values of each training data vector of the input space, μic,r,s(x), is calcux, and the cluster centers, lated using input matrix, υic,r,s(x)=(υi,1c,r,s,υi,2c,r,s,…,υi,nvc,r,s) as input parameters of membership value calculation equation in (5.40). Figure 5.11 depicts the uncertainty interval of a membership functions induced by changing values of mr and τs. It should be recalled that these membership functions are idealized representations obtained after a curvefitting over the scatter diagram of membership values. The bounded interval in Figure 5.11 comprises of discrete membership functions (of a particular cluster i) one for each different parameter set obtained from Cartesian product of its discrete values, nr×nif . Hence, for each datum x, e.g., x=7.5, t=1,…, nr×nif different membership values can be defined within this interval. Each one of these t membership values are represented with μic,r,s(x), associated with tth fuzzy clustering model using tuples of 〈c,mr,τs〉.
7.5 Fig. 5.11 Uncertainty interval of a cluster represented with membership functions of 3-tuples, 〈c,mr,τs〉. c is the number of clusters, mr defines the level of fuzziness, τs represents the interim fuzzy function structures of the Improved Fuzzy Clustering (IFC).
Step 2: Approximation of Improved Fuzzy Functions
Once membership values are obtained from the IFC method, nf different improved fuzzy functions, each of them represented with Φiψ, ψ=1,…,nf=nif, i=1,..c, is approximated for each cluster using a list of fuzzy function structures represented
182
5 Modeling Uncertainty with Improved Fuzzy Functions
with Φψ∈{Φ1…Φnf}, ψ=1,…,nf, i=1,…c. It should be noted that, since the same list of membership value transformations are used to identify interim and local fuzzy functions, the total number of different IFC models and the number of fuzzy functions of each cluster would be equal to the Cartesian product parameters, i.e., nr×nif=nr×nf. The local fuzzy functions are formed by using different membership value transformations in each local fuzzy function. Here ψ represent a fuzzy function structure. Thus, for each tth embedded IFC model viz., each set of 〈c,mr,τs〉 parameters using improved membership values, μ r,s(x), nf different system fuzzy functions for each cluster, denoted with fir,s,ψ, i=1,…c, are identified. It should be recalled that τs denotes a different membership value transformation matrix than Φψ to identify the interim fuzzy functions of the IFC clustering algorithm, whereas, Φiψ identifies one fuzzy function structure that may or may not pickup the same membership value transformations used in τs matrix as additional input variables along with the original input variables. Therefore different indices are listed for each of the matrix structures, i.e., interim, τs, and local input matrices, Φψ. Thus, from the pool of different membership value transformations defined in the beginning of the structure identification method, the IFC model uses Cartesian product of the these membership values to identify different interim fuzzy functions in order to build t different IFC models, and then the same pool of membership value transformations are used to approximate t different fuzzy functions for each cluster The novelty of fuzzy functions systems is that a different dataset is structured for each cluster i by using improved membership values, μr,s(x), and/or their transformations as additional predictors to the original input matrix, as shown in Figure 4.6. It should emphasized again that the membership value transformations used to construct the Φiψ matrix should not be the same as the membership value transformation used to construct τs matrix. Hence, each cluster may have a different structure of fuzzy functions. For each tth IFC model denoted with 〈c,mr,τs〉 tuples, one can define different models using different fuzzy functions structures in each cluster from a list of fuzzy function structures. Figure 5.12 depicts an example topology for structure identification of the DIT2IFF system using the following parameter list;
mr∈{m1,m2,m3}, nr=3, τs∈{τ1,τ2}, nif=2 and Φψ∈{Φ1,Φ2}, nf=2. The parameter mr can be assigned any real (scalar) value, i.e., 1<mr<∞. The matrix structures τs, such as in (5.41), and Φψ such as in (5.43), can also be formed with any possible transformation of membership values to convert them into a higher dimensional feature space. For this example there are only two different types of fuzzy functions identified and the interim and local fuzzy functions are approximated using these structures. Each τs represents one interim fuzzy function structure to be used to build a tth IFC model, a total of 2 different interim fuzzy fu