Stochastic Global Optimization and Its Applications with Fuzzy Adaptive Simulated Annealing

Hime Aguiar e Oliveira Junior, Lester Ingber, Antonio Petraglia, Mariane Rembold Petraglia, and Maria Augusta Soares Machado Stochastic Global Optimization and Its Applications with Fuzzy Adaptive Simulated Annealing

Intelligent Systems Reference Library, Volume 35 Editors-in-Chief Prof. Janusz Kacprzyk Systems Research Institute Polish Academy of Sciences ul. Newelska 6 01-447 Warsaw Poland E-mail: [email protected]

Prof. Lakhmi C. Jain University of South Australia Adelaide Mawson Lakes Campus South Australia 5095 Australia E-mail: [email protected]

Further volumes of this series can be found on our homepage: springer.com Vol. 11. Samuli Niiranen and Andre Ribeiro (Eds.) Information Processing and Biological Systems, 2011 ISBN 978-3-642-19620-1

Vol. 24. Dawn E. Holmes and Lakhmi C. Jain (Eds.) Data Mining: Foundations and Intelligent Paradigms, 2011 ISBN 978-3-642-23240-4

Vol. 12. Florin Gorunescu Data Mining, 2011 ISBN 978-3-642-19720-8

Vol. 25. Dawn E. Holmes and Lakhmi C. Jain (Eds.) Data Mining: Foundations and Intelligent Paradigms, 2011 ISBN 978-3-642-23150-6

Vol. 13. Witold Pedrycz and Shyi-Ming Chen (Eds.) Granular Computing and Intelligent Systems, 2011 ISBN 978-3-642-19819-9

Vol. 26. Tauseef Gulrez and Aboul Ella Hassanien (Eds.) Advances in Robotics and Virtual Reality, 2011 ISBN 978-3-642-23362-3

Vol. 14. George A. Anastassiou and Oktay Duman Towards Intelligent Modeling: Statistical Approximation Theory, 2011 ISBN 978-3-642-19825-0

Vol. 27. Cristina Urdiales Collaborative Assistive Robot for Mobility Enhancement (CARMEN), 2011 ISBN 978-3-642-24901-3

Vol. 15. Antonino Freno and Edmondo Trentin Hybrid Random Fields, 2011 ISBN 978-3-642-20307-7

Vol. 28. Tatiana Valentine Guy, Miroslav K´arn´y and David H. Wolpert (Eds.) Decision Making with Imperfect Decision Makers, 2012 ISBN 978-3-642-24646-3

Vol. 16. Alexiei Dingli Knowledge Annotation: Making Implicit Knowledge Explicit, 2011 ISBN 978-3-642-20322-0 Vol. 17. Crina Grosan and Ajith Abraham Intelligent Systems, 2011 ISBN 978-3-642-21003-7 Vol. 18. Achim Zielesny From Curve Fitting to Machine Learning, 2011 ISBN 978-3-642-21279-6 Vol. 19. George A. Anastassiou Intelligent Systems: Approximation by Artificial Neural Networks, 2011 ISBN 978-3-642-21430-1 Vol. 20. Lech Polkowski Approximate Reasoning by Parts, 2011 ISBN 978-3-642-22278-8 Vol. 21. Igor Chikalov Average Time Complexity of Decision Trees, 2011 ISBN 978-3-642-22660-1 Vol. 22. Przemyslaw Rz˙ ewski, Emma Kusztina, Ryszard Tadeusiewicz, and Oleg Zaikin Intelligent Open Learning Systems, 2011 ISBN 978-3-642-22666-3 Vol. 23. Dawn E. Holmes and Lakhmi C. Jain (Eds.) Data Mining: Foundations and Intelligent Paradigms, 2011 ISBN 978-3-642-23165-0

Vol. 29. Roumen Kountchev and Kazumi Nakamatsu (Eds.) Advances in Reasoning-Based Image Processing Intelligent Systems, 2012 ISBN 978-3-642-24692-0 Vol. 30. Marina V. Sokolova and Antonio Fern´andez-Caballero Decision Making in Complex Systems, 2012 ISBN 978-3-642-25543-4 Vol. 31. Ludomir M. Lauda´nski Between Certainty and Uncertainty, 2012 ISBN 978-3-642-25696-7 Vol. 32. Jos´e J. Pazos Arias, Ana Fern´andez Vilas, and Rebeca P. D´ıaz Redondo Recommender Systems for the Social Web, 2012 ISBN 978-3-642-25693-6 Vol. 33. Jie Lu, Lakhmi C. Jain, and Guangquan Zhang Handbook on Decision Making, 2012 ISBN 978-3-642-25754-4 Vol. 34. Ernesto Sanchez, Giovanni Squillero, and Alberto Tonda Industrial Applications of Evolutionary Algorithms, 2012 ISBN 978-3-642-27466-4 Vol. 35. Hime Aguiar e Oliveira Junior, Lester Ingber, Antonio Petraglia, Mariane Rembold Petraglia, and Maria Augusta Soares Machado Stochastic Global Optimization and Its Applications with Fuzzy Adaptive Simulated Annealing, 2012 ISBN 978-3-642-27478-7

Hime Aguiar e Oliveira Junior, Lester Ingber, Antonio Petraglia, Mariane Rembold Petraglia, and Maria Augusta Soares Machado

Stochastic Global Optimization and Its Applications with Fuzzy Adaptive Simulated Annealing

123

Authors Dr. Hime Aguiar e Oliveira Junior Rio de Janeiro Brazil Prof. Lester Ingber Lester Ingber Research Ashland USA

Prof. Mariane Rembold Petraglia Universidade Federal do Rio de Janeiro Rio de Janeiro Brazil Prof. Maria Augusta Soares Machado IBMEC-RJ Rio de Janeiro Brazil

Prof. Antonio Petraglia Universidade Federal do Rio de Janeiro Rio de Janeiro Brazil

ISSN 1868-4394 e-ISSN 1868-4408 ISBN 978-3-642-27478-7 e-ISBN 978-3-642-27479-4 DOI 10.1007/978-3-642-27479-4 Springer Heidelberg New York Dordrecht London Library of Congress Control Number: 2011945156 c Springer-Verlag Berlin Heidelberg 2012  This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. Exempted from this legal reservation are brief excerpts in connection with reviews or scholarly analysis or material supplied specifically for the purpose of being entered and executed on a computer system, for exclusive use by the purchaser of the work. Duplication of this publication or parts thereof is permitted only under the provisions of the Copyright Law of the Publisher’s location, in its current version, and permission for use must always be obtained from Springer. Permissions for use may be obtained through RightsLink at the Copyright Clearance Center. Violations are liable to prosecution under the respective Copyright Law. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. While the advice and information in this book are believed to be true and accurate at the date of publication, neither the authors nor the editors nor the publisher can accept any legal responsibility for any errors or omissions that may be made. The publisher makes no warranty, express or implied, with respect to the material contained herein. Printed on acid-free paper Springer is part of Springer Science+Business Media (www.springer.com)

Preface

Global optimization is a very important subject. It finds applications in Biology, Statistics, Engineering, Mathematics itself, Medicine, Management Science, Economics and in virtually everything you can imagine. Although many excellent methods have been developed so far, several of them assume certain conditions about functions to be processed, as convexity or differentiability, and, in practice, we often get into situations in which objective functions to be optimized (minimized or maximized) are not differentiable, convex or even continuous. In such settings, gradientbased methods are not of much help and it is necessary to find more general ways to get good results. During the last decades an expressive number of new global optimization methods were idealized, aiming to reach that more general objective and substantial part of them belong to the category of metaheuristic methods, being also known generically as metaheuristics. Their majority are of a probabilistic nature, that is to say, use probability theory results to get to their target, consequently being classified as stochastic methods as well. Knowledge of the capabilities and limitations of these algorithms leads to a better understanding of their reach over various applications and points us the way to future research on improving and extending algorithms’ theoretical foundations and respective implementations. Our goal in this book is to present a description of some techniques for solving stochastic global optimization problems and to detail one, in particular - Fuzzy Adaptive Simulated Annealing (Fuzzy ASA). By presenting several, detailed examples of its application we will try to stimulate the reader’s intuition and make the use of Fuzzy ASA (or ASA) easier to all wishing to solve their problems with this tool. It is important to note that, in this book, all example program codes are presented in the hope that they will be useful in the learning process, but without any warranty - without even the implied warranty of fitness for a particular purpose. Besides, it is advisable to highlight that the architecture of the example routines is not necessarily the most efficient in computational terms, taking into account that they were constructed for pedagogical reasons only. Formal mathematical requirements are kept to a minimum and our focus will be on continuous problems, although ASA is able to handle discrete optimization tasks as well. This book could be used in courses related to optimization as well

VI

Preface

as by researchers and practitioners in Engineering and industry, and is adequate for self-study, also. Prerequisites for reading this book include some knowledge of Linear Algebra, introductory Numeric Analysis and basic Probability Theory. The work is divided in three parts: • An introductory set of chapters, exposing basic facts about some important global optimization methods and their overall structure; • A second part containing a detailed description of Adaptive Simulated Annealing (ASA) and its fuzzy controlled version, how to make constrained and unconstrained optimization with them and several illustrative examples that, we hope, will be helpful to the reader; • A final part containing chapters that describe applications of Fuzzy ASA to several areas of knowledge like signal processing, statistical estimation, fuzzy modeling and nonlinear equation solving. To complement the material contained in the text and make the learning time shorter, we invite the reader to try doing some global optimization tasks by him or herself. It suffices to download the publicly available code at www.ingber.com and start coding. In case of doubts, suggestions or anything else, please, feel free to contact us. We hope you enjoy the book and that it will be useful in your work. Rio de Janeiro, BRAZIL Ashland, USA October 2011

Hime Aguiar e Oliveira Junior Lester Ingber Antonio Petraglia Mariane Rembold Petraglia Maria Augusta Soares Machado

Acknowledgements

We would like to thank Dr. Leontina Di Cecco (Publishing Editor), Mr. Holger Sch¨ape (Engineering Editorial) and all staff of Springer-Verlag for their kind support.

Contents

Part I Fundamentals 1

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1 Why to Optimize? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 Kinds of Optimization Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3 How to Optimize? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3 3 5 6 10

2

Global Optimization and Its Applications . . . . . . . . . . . . . . . . . . . . . . . . 2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Stochastic or Deterministic ? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3 Considerations about General Global Optimization Tasks . . . . . . . . 2.4 Some Popular Approaches and Final Comments . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

11 11 12 13 18 20

3

Metaheuristic Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Genetic Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Particle Swarm Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 Differential Evolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5 Cross-Entropy Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.6 Simulated Annealing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

21 21 23 24 25 26 27 30

Part II ASA, Fuzzy ASA and Their Characteristics 4

Adaptive Simulated Annealing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1.1 LICENSE and Contributions . . . . . . . . . . . . . . . . . . . . . . . . . 4.1.2 Organization of Chapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Theoretical Foundations of Adaptive Simulated Annealing (ASA) .

33 33 34 34 35

X

5

Contents

4.2.1 Shades of Simulated Annealing . . . . . . . . . . . . . . . . . . . . . . 4.2.2 Critics of SA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.3 “Standard” Simulated Annealing (SA) . . . . . . . . . . . . . . . . 4.2.4 Boltzmann Annealing (BA) . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.5 Simulated Quenching (SQ) . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.6 Fast Annealing (FA) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.7 Adaptive Simulated Annealing (ASA) . . . . . . . . . . . . . . . . . 4.2.8 VFSR and ASA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 Practical Implementation of ASA . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3.1 Generating Probability Density Function . . . . . . . . . . . . . . . . 4.3.2 Acceptance Probability Density Function . . . . . . . . . . . . . . 4.3.3 Reannealing Temperature Schedule . . . . . . . . . . . . . . . . . . . 4.3.4 QUENCH PARAMETERS=FALSE . . . . . . . . . . . . . . . . . . 4.3.5 QUENCH COST=FALSE . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3.6 QUENCH COST SCALE=TRUE . . . . . . . . . . . . . . . . . . . . . 4.4 Tuning Guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4.1 The Necessity for Tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4.2 Construction of the Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4.3 Motivations for Tuning Methodology . . . . . . . . . . . . . . . . . 4.4.4 Some Rough But Useful Guidelines . . . . . . . . . . . . . . . . . . . . 4.4.5 Quenching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4.6 Options for Large Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4.7 Shunting to Local Codes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4.8 Judging Importance-Sampling . . . . . . . . . . . . . . . . . . . . . . . . 4.4.9 User References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.5 Adaptive OPTIONS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.5.1 VFSR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.5.2 ASA FUZZY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.6 Multiple Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.6.1 SELF OPTIMIZE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.6.2 ASA PARALLEL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.6.3 TRD Example of Multiple Systems . . . . . . . . . . . . . . . . . . . . 4.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

35 36 36 36 39 40 40 44 44 44 45 45 46 47 47 47 47 48 50 50 52 53 54 55 55 56 56 56 56 56 57 57 58 59

Unconstrained Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1 Fuzzy ASA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 Unconstrained (or Rectangular Constrained) Optimization Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2.1 Rastrigin Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2.2 Schwefel Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2.3 Ackley Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2.4 Krishnakumar Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2.5 Rosenbrock Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2.6 Griewangk Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

63 63 67 70 73 76 78 80 83

Contents

6

XI

5.2.7 Special Function 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2.8 Special Function 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

85 88 92 93

Constrained Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2 Constrained Global Optimization Using ASA and Fuzzy ASA . . . . 6.2.1 Function G01 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2.2 Function G02 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2.3 Function G03 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2.4 Function G04 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2.5 Function G05 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2.6 Function G06 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2.7 Function G07 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2.8 Function G08 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2.9 Function G09 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2.10 Function G10 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2.11 Function G11 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2.12 Function G12 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2.13 Function G13 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.3 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

95 95 97 98 102 105 106 107 107 108 109 110 111 112 113 113 114 115

Part III Applications 7

Applications to Signal Processing - Blind Source Separation . . . . . . . 7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3.1 Example 1 - Separation by TSK MIMO System . . . . . . . . . . 7.3.2 Example 2 - Separation by TSK MIMO System . . . . . . . . . . 7.3.3 Example 3 - Separation by TSK MIMO System . . . . . . . . . . 7.3.4 Example 4 - Separation by TSK MIMO System . . . . . . . . . . 7.3.5 Example 5 - Mixture by PNL Model . . . . . . . . . . . . . . . . . . . 7.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

119 119 124 124 124 127 128 129 132 137 138

8

Fuzzy Modeling with Fuzzy Adaptive Simulated Annealing . . . . . . . . 8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2 Affine Takagi-Sugeno Fuzzy Systems . . . . . . . . . . . . . . . . . . . . . . . . . 8.3 The Fuzzy Modeling Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.1 Approximation in Lower Dimensions . . . . . . . . . . . . . . . . . . 8.3.2 Approximation in Higher Dimensions . . . . . . . . . . . . . . . . . .

139 139 140 141 141 144

XII

Contents

8.4 Ideas for Fuzzy Clustering Using ASA . . . . . . . . . . . . . . . . . . . . . . . . 145 8.5 Conclusions about the Presented Methods . . . . . . . . . . . . . . . . . . . . . 147 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148 9

Statistical Estimation and Global Optimization . . . . . . . . . . . . . . . . . . . 9.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.2 Maximum Likelihood Estimation with ASA . . . . . . . . . . . . . . . . . . . 9.3 Implementation and Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.3.1 Exponential Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.3.2 Normal Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.3.3 Lognormal Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.3.4 Cauchy Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.3.5 Triangular Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.3.6 Mixture (Laplace and Uniform) Distribution . . . . . . . . . . . . 9.3.7 Gamma Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.4 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

149 149 150 151 152 156 157 158 160 164 164 166 167

10 Nonlinear Equation Solving . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.2 Statement of the Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.3 The Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.4 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.4.1 Example 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.4.2 Example 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.4.3 Example 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.4.4 Example 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.4.5 Example 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.4.6 Example 6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.4.7 Example 7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

169 169 170 171 172 172 176 178 181 181 182 184 184 187

11 Space-Filling Curves and Fuzzy ASA . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Key Results from General Topology, Ergodic and Measure Theories 11.3 Composing Space-Filling Curves and ASA . . . . . . . . . . . . . . . . . . . . 11.3.1 Algorithm Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.4 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

189 189 190 196 196 197 199 201

12 Epilogue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203 12.1 Final Thoughts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203 Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205

Part I

Fundamentals

Chapter 1

Introduction

Abstract. This chapter aims to introduce the readers to the fundamental ideas of global optimization, presenting, in a friendly way, its main techniques. Although we are also going to browse some traditional global optimization techniques, our emphasis along this book will be on evolutionary and nature-inspired algorithms, focusing on the adaptive simulated annealing paradigm and its more representative applications until now. The book aims to show readers how to use global optimization techniques to get their problems solved in practice, using simple and inexpensive tools.

1.1 Why to Optimize? Optimization applications are common nowadays and certainly will be much more frequent in the future. For example, when driving back home from work we usually try to take the best route, usually the shortest one. Sometimes we have to add some constraints as, for instance, a restaurant should be located along the way, in case you don’t have time for preparing your dinner and needs, let us say, to go to the theater later. One additional constraint could be, usually in a big city, the preference for less congested streets or highways. In more technical areas, like Engineering, Statistics etc., many applications of global optimization methods are well-known as well [1, 7, 10]. Below, you can find some examples: • Electrical Engineering – FIR filter design – Filter bank design – Blind source separation • Transport engineering – Traveling salesman problems H.A. e Oliveira Jr. et al.: Stochastic Global Optimization & Its Applications, ISRL 35, pp. 3–10. c Springer-Verlag Berlin Heidelberg 2012 springerlink.com 

4

1 Introduction

• Statistical inference – Maximum likelihood estimation • Soft computing – Neural network training – Neuro-fuzzy system training – Fuzzy systems modeling • Biology – Protein folding problems • Economics – Nash equilibrium approximation – Optimum portfolio design • Numerical analysis – Nonlinear equation solving – Fixed point approximation

Even in introductory courses global optimization techniques could be helpful: whenever we make curve or surface fitting, for example, an optimization process is in progress, taking into account that we are trying to minimize a specific measure of distance between a parametric model (curve or surface) and a set of points, normally obtained through a practical experiment - parameters defining the model are chosen so as to get exactly that. Another very important point is that, whenever we are faced with a hard problem for which there is no really definitive or satisfactory theoretical model upon which to base our reasoning, optimization techniques could be the only way to go in order to complete the task, at least while a better modeling is not discovered. We can cite several examples of this situation, like numerical solution of certain classes of nonlinear systems of equations, nonlinear multivariable regression, complex engineering design, and so on. If we are able to express the problem under consideration in a parametric form and to put our design objectives ”inside” a given objective function, there is an opportunity to solve the problem by means of (global) optimization techniques. In that case, it is like if the original ”theory of operation” could be totally replaced by the ”minimization power” of effective algorithms - by the way, orthodox designers get very upset whenever this happens. Jokes apart, the truth is that there is no free lunch and, when we don’t have good models for solving certain difficult problems, there is something we can take advantage of - global optimization algorithms. Intuitively, it seems like if the ”intelligence” of taking cost functions to lower values could be replaced for the ”intelligence” of a good model for the original problem, embedded in its original setting.

1.2 Kinds of Optimization Problems

5

In this book, we opted for considering only minimization problems. Maximization problem with objective function H will be converted into a minimization one, by considering -H - maximization of H is equivalent to minimization of -H.

1.2 Kinds of Optimization Problems In many scenarios, optimization problems involve variables that assume integer values only. For example, suppose that a certain factory produces cars. In this case, its production will be measured by means of integers (that is, the number of cars shipped) rather than real numbers. It would not make much sense to talk about producing 1,200.45 cars and sending 456.76 to a reseller, for instance. The somewhat naive strategy of ignoring the integrality requirement, solving the problem with real variables and then rounding all the components to the nearest integer is by no means guaranteed to give solutions that are close to optimal. This kind of problem should be handled using discrete optimization tools. The mathematical formulation is changed by adding the constraint that independent variables should assume integer values only. This problem becomes an integer programming one. When the domain of the function to minimize is countable, the problem is called combinatorial or discrete. Optimization problems for real valued variables are named continuous whenever cost function’s domain is not enumerable. Finally, we can have a multivariable problem in which there are continuous and discrete independent variables, here we say that the problem is mixed. In constrained optimization, additional conditions are usually imposed by means of the so-called constraints, expressed by linear or nonlinear equalities and/or inequalities. Points satisfying all constraints are called feasible. The term ”discrete optimization” refers, in general, to problems in which the desired solution belongs to a finite set. On the opposite side, in continuous optimization problems we search for a solution from an uncountable set - typically a set of vectors with real components. Continuous optimization problems involving smooth objective functions are typically easier to handle because their ”good behavior” makes it possible to use objective and constraint information at a particular point to obtain information about the function’s behavior at points close to this basic point. The same is not valid in discrete problems, where points that are close in some sense may have significantly different function values. Besides, the set of possible solutions can be too large to make an exhaustive search for the best value in this finite domain. Constrained problems [2, 3] can be classified according to the nature of the objective function and constraints (linear, nonlinear etc.), whether their objective functions are smooth or not (differentiable or nondifferentiable), and so on. Possibly the most important distinction is between problems that have constraints on the independent variables and those that do not. Unconstrained optimization problems can arise directly in many applications and also as reformatted constrained optimization problems, in which the constraints are replaced by penalty terms in the cost

6

1 Introduction

functions that have the effect of repelling constraint violations [4, 5]. Constrained optimization problems arise from models that demand constraining values that variables may assume. Constraints may be expressed by simple imposition of belonging to a certain set (a hyper-rectangle, for instance) and/or satisfying linear or nonlinear equalities or inequalities, representing arbitrarily complex relationships among the variables. When both the objective function and all the constraints are linear functions of x, the problem is a linear programming one. By the way, financial management and economics make extensive use of linear models. On the other hand, nonlinear programming problems, in which at least one of the constraints or the objective functions are nonlinear, arise naturally in the physical sciences and Engineering, and are becoming more widely used in social sciences. Gradient-based optimization algorithms [1, 6, 8] tend to be the fastest ones when we seek only a local optimizer, a point at which the cost function is smaller than all other feasible points in its neighborhood, or attraction basin. Whenever the cost function has more than one local minimum, they don’t always find the best of all such minima, that is, the global solution. Global solutions are highly desirable in some applications, but they are usually difficult to locate. So, general nonlinear problems may have local solutions that are not global ones. As we said before, in this book global optimization is our focus. One more classification dimension can be introduced among optimization algorithms - they can be classified as stochastic or deterministic. Stochastic optimization algorithms usually work by simulating (probabilistically, of course) certain stochastic processes, aiming to find a local or global minimizer for the problem at hand - their output can vary from one execution to another, depending on the particular sample path resulting from a specific simulation. Deterministic algorithms, on the other hand, give always the same result provided we start the whole process with exactly the same initial conditions.

1.3 How to Optimize? There are several ways to tackle a global optimization problem. Firstly it will be useful to classify the problem at hand and, depending on its type, we will have more or less methods and tools available. For example, if our objective function is smooth and the problem is an unconstrained one, there are many alternative methods, ranging from gradient-based to nature-inspired ones. Also, many excellent implementations exist and several of them are open-source and free. On the other hand, if the cost function is not differentiable, our alternatives get scarcer because we just can’t recur to algorithms using gradients, for instance: they (gradients) are simply not defined for this kind of function. In general, the worst situation is when we are faced with discontinuous, non-Lipschitz and multimodal cost functions - here is where heuristic paradigms bright more intensely. Nevertheless, multimodal functions and those presenting graphs with large flat regions offer a big challenge to any algorithm, being it

1.3 How to Optimize?

7

deterministic, stochastic, gradient-based or nature-inspired. To understand why this is so, please, take a look at figures 1.1 to 1.5 below. Graphs corresponding to Schwefel, Griewangk, Rastrigin and Ackley functions illustrate the multiple local minima problem, and that one relative to Easom function shows a scenario where there exists a large region without expressive variation, making it difficult, for typical algorithms, to find the central global minimizer, unless it starts from its attraction basin. The fact is that common optimization algorithms take their ”decisions” based upon local information, collected from neighborhoods of their current (population of) points [4, 5, 6, 8]. This information is definitely related to values assumed by the functions under optimization and their trends - after all, our aim is to get to regions of smallest value, so it is natural to ground the decision making on that elements. In this fashion, whenever the state of a given algorithm finds itself in large flat or local minimum attraction regions it tends to get ”confused” (flat region) or stagnated (local minimum). Population-based methods [9] attenuate but not eliminate this kind of problem, taking into account that known methods normally use finite populations and continuous optimization deals with noncountable domains. As an example, imagine a gradient-based method started from a region where the function under study is constant - the state couldn’t move, considering that the gradient would be null in a given vicinity of the current point.

Fig. 1.1 Schwefel function

In the next chapter we will continue our reasoning about these and other related issues.

8

Fig. 1.2 Griewangk function

Fig. 1.3 Easom function

1 Introduction

1.3 How to Optimize?

Fig. 1.4 Rastrigin function

Fig. 1.5 Ackley function

9

10

1 Introduction

References 1. Bazaraa, M., Sherali, H., Shetty, C.: Nonlinear Programming, Theory and Applications. John Wiley & Sons, New York (1993) 2. Bertsekas, D.P.: Constrained Optimization and Lagrange Multiplier Methods. Academic Press, New York (1982) 3. Bertsekas, D.P.: Nonlinear Programming. Athena Scientific, Belmont (1995) 4. Brent, R.P.: Algorithms for minimization without derivatives. Prentice Hall, Englewood Cliffs (1973) 5. Fletcher, R.: Practical Methods of Optimization. John Wiley & Sons, New York (1987) 6. Gill, P., Murray, W., Wright, M.H.: Practical Optimization. Academic Press (1981) 7. Hartmann, A.K., Riger, H.: Optimization Algorithms in Physics. Wiley, Berlin (2002) 8. Lawson, C.L., Hanson, R.J.: Solving Least Squares Problems. Prentice-Hall, Englewood Cliffs (1974) 9. Michalewicz, Z.: Genetic Algorithms + Data Structures = Evolution Programs. Springer, Heidelberg (1994) 10. Nocedal, J., Wright, S.J.: Numerical Optimization. Springer, New York (1999)

Chapter 2

Global Optimization and Its Applications

Abstract. In this chapter we will introduce ideas about global optimization methods, their types and limitations, exposing, in a succint way, several of their main characteristics. As our focus in this book will be on adaptive simulated annealing and its applications, we’ll start to pave the way that will take us to pragmatic utilization of that method and stochastic optimization methods, in general.

2.1 Introduction As human kind has always strived for reaching optimal states of things, be them maximum profits and industrial production levels or minimum costs, we can infer that optimization problems always existed, even though human beings could not be conscious of them. In Nature, for instance, cell formation obeys laws driven by optimization processes and atoms form bonds aiming minimization of certain energy levels. Molecules forming solid bodies during the process of freezing try to assume energy-minimum crystal structures. Biological principle of survival of the fittest [17] which, conjugated to species evolution theory [8], tries to model adaptation of the species to their environment is highly related to optimization ideas. Also, humans want to reach maximum utility with the least amount of effort. In consequence, optimization is not only present in our ordinary lives but Universe itself features optimization processes happening all the time, at the microscopic or macroscopic levels. As we can face problems of global or local types, there are techniques aimed at finding local or global optima - whenever we need a minimum, for example, within a certain neighborhood, the problem is local. Global optimization treats the case in which we search for optimum points over all the domain of the objective function. Accordingly, the goal of global optimization is to find the best possible elements from a set, subject to a group of criteria. These criteria are expressed as mathematical functions and/or relationships. The objective function is the index we want to optimize (in this book, minimize) and the constraints are the relationships we want to be satisfied [10, 11, 13]. H.A. e Oliveira Jr. et al.: Stochastic Global Optimization & Its Applications, ISRL 35, pp. 11–20. c Springer-Verlag Berlin Heidelberg 2012 springerlink.com 

12

2 Global Optimization and Its Applications

Typically, the image of an objective function as well as its range are subsets of the real numbers set R. The domain of the function can contain many kinds of elements like numbers, vectors of real or integer numbers etc.. Depending on the problem to be solved we have different kinds of domain. Objective function values could be obtained not only by mathematical expressions, but as results of simulations that can, for example, involve long and computationally expensive calculations - this could justify why a large number of publications takes the number of cost function evaluations as a figure of merit for comparison to other methods. Roughly speaking, global optimization could be seen as the area of applied mathematics that deals with methods that can be used to find the best elements in a whole domain satisfying a given set of constraints.

2.2 Stochastic or Deterministic ? It is not trivial to build a taxonomy of global optimization methods. As we said before, in general, optimization algorithms can be divided in two basic classes: deterministic and stochastic (or probabilistic). Deterministic ones are most often used whenever the cost function presents certain analytical characteristics that allow us to explore known theoretical results so as to find optimal points [5, 10, 16, 18]. For example, in gradient-based methods we frequently simulate a dynamical system that evolves towards local optima, according to decreasing directions of the cost function. This way the search space can be efficiently explored using, for instance, a divide and conquer scheme. If the relation between a solution candidate and its fitness are too complicated, presenting nonsmooth behavior, it could be difficult to solve a global optimization problem deterministically. Trying it by exhaustive enumeration of the search space is not feasible even for small dimension problems. At this point stochastic algorithms come into play. A relevant family of probabilistic algorithms are the Monte Carlo-based approaches . They present solutions back in reduced time but don’t guarantee convergence to global solutions. Their results might be non-global optima - in case of multimodal functions, even deterministic algorithms could not, in general, ensure that they have reached global optima. Here we have an example of a heuristic paradigm - in particular, the decision is taken with basis in previous experience and experimental data. Heuristics used in global optimization help deciding which one of a set of possible solutions is to be examined next. A heuristic could be described as the component of an optimization algorithm that uses information currently gathered by the algorithm to help deciding which candidate solution should be tested next or how the next individual can be produced. By the other side, a metaheuristic can be understood as a method for solving general classes of problems. It combines objective function values and reasoning rules in a somewhat abstract and hopefully effective way (in many cases, inspired by natural phenomena), usually without exploring the structure of the particular problem at hand [12, 14, 20]. This combination is often performed stochastically by

2.3 Considerations about General Global Optimization Tasks

13

using samples from the search space. Standard simulated annealing, for example, decides which candidate will be evaluated next according to the Boltzmann probability distribution. Evolutionary algorithms follow the behavior of natural evolution and consider solution candidates as individuals that compete in an artificial setting. An important class of stochastic metaheuristics is evolutionary computing. It contains algorithms that are population-based and iteratively refined. Some of its most important members are genetic and swarm intelligence algorithms. Besides these nature-inspired and evolutionary approaches, there exist also methods that try to behave similarly to physical processes: simulated annealing, parallel tempering, and grenade explosion method [1, 12, 20], to cite only a few.

2.3 Considerations about General Global Optimization Tasks Although until now we have cited some global optimization algorithms and we will talk about a reasonable variety of them in the chapters ahead, the methods introduced in this book will deal with only a small part of the actual number of available methods and we intend to focus into just one of them. Anyway, it is a natural question to ask why there are so many different algorithms and why is this variety needed. One reason could be simply because there are many different kinds of global optimization tasks, each of them putting different obstacles to optimizers and representing specific and particular difficulties. In what follows we would like to discuss concisely the most important of these complications and the main problems usually encountered by existing programs during global optimization, namely, multimodality, stagnation, premature convergence, poor exploratory ability and related phenomena. Whenever a given method faces even a single such feature in a given function under study, it can get caught in a trap and simply not be able to reach its final goal, that is the global optimum. This could occur even if highly efficient optimization techniques are applied. In this fashion, we want to alert researchers and practitioners about those ”threats” and hope the information will be helpful to them in the use of global optimization algorithms. Figures 2.1 – 2.6 show a set of different types of landscapes which we are going to analyze briefly. As we remarked before, all problems will be minimization ones and the graphs aim to illustrate the difficulties faced when trying to get to global minima starting from arbitrary points inside function’s domain. Here, the illustrations show unidimensional cases but in higher dimensions there are analogous scenarios, complicated by the additional dimensional complexity. Before investigating about what makes these landscapes hard to deal with, we should understand the situation in the context of optimization. The degree of difficulty of solving a certain problem with a specific algorithm is closely related to its computational complexity, that is, the amount of resources such as time and memory required to get the job done. One approach to obtain near-optimal solutions for complex problems in reasonable time is to apply metaheuristic, probabilistic

14

2 Global Optimization and Its Applications

Fig. 2.1 Best possible scenario

Fig. 2.2 Multimodal function

optimization procedures. After all, optimization algorithms are guided primarily by values of cost functions. A function is hard from a mathematical perspective if it is not continuous, not differentiable and/or if it has multiple maxima and minima. This understanding of difficulty is well represented by the graphs shown in the figures. In many real world applications of metaheuristic optimization, the characteristics of the objective functions are not known in advance and result from real-time simulations dependent upon physical processes. So, it is seldom possible to estimate precise boundaries for the performance of optimizers in advance, not even mentioning accurate numerical estimates for their results. Generally, experimental data,

2.3 Considerations about General Global Optimization Tasks

15

Fig. 2.3 Orientation is very difficult

Fig. 2.4 Potential stagnation configuration

rules of thumb and models inspired from results obtained in other areas such as physics, biology etc., are the only available tools. An optimization algorithm is considered to have converged if it cannot produce new candidates anymore or if it continues producing solution candidates from a very restricted region of the problem domain. One of the main problems in global optimization is that it is often not possible to determine whether the best solution currently known is a local or global optimum and thus if definitive convergence was reached. In this fashion, it is usually not clear when the optimization process could be interrupted, should concentrate on refining the current optimum, or should

16

2 Global Optimization and Its Applications

Fig. 2.5 Needle configuration

Fig. 2.6 Many needles

examine different parts of the search space. This situation is particularly significant if we are dealing with multimodal and nondifferentiable functions. As expected, an optimization process is said to have prematurely converged to a local optimum if it is no longer able to explore other parts of the search space than the area currently being visited (it was caught inside a suboptimal attraction basin), existing another region that contains a better solution. By another side, the existence of multiple global optima is not problematic by itself and the discovery of only a subset of them could be considered a successful result in most practical situations. The occurrence of numerous local (and nonglobal) optima, however, could cause

2.3 Considerations about General Global Optimization Tasks

17

problems, taking into account the possibility of premature convergence to suboptimal states. Related to this setting is the exploration x exploitation problem, that can be seen in virtually every nontrivial case of global optimization. In the optimization realm, exploration means visiting new (and perhaps promising) areas of the search space which have not been investigated before. Exploration is a procedure by which we try to find novel and better solution states. Certain operators (like mutation, in genetic algorithms) have the chance of creating inferior solutions by destroying good individuals, but also have a small chance of finding fitter candidates, although that is not guaranteed at all. By the other side, exploitation processes operate by incorporating small changes into already existing individuals, leading to new but nearby solution candidates. There is always the disadvantage that other, possibly better, solutions located in different areas of the problem space will not be discovered. General optimization strategies usually possess mechanisms that allow us to balance exploitation and exploration levels. Methods that favor exploitation over exploration have higher convergence speed but offer the risk of not finding the optimal solution and may get stuck at a local optimum. On the other hand, algorithms that perform excessive exploration may never improve their candidates well enough to find the global optimum or it may spend too much time trying to discover it by chance. An excellent example is given by the standard simulated annealing paradigm, that can do quenching, priorizing exploitation over exploration and losing the guaranteed (theoretical) convergence in distribution to the global miniimum. Unfortunately, there is no known approach that could prevent premature convergence, in general. The probability that an optimization process gets caught in a local optimum depends on several factors, including the configuration of the function to be minimized, the features of the algorithm at hand etc.. In general, global optimization methods adopt certain heuristics that try to attenuate the risk of being caught in suboptimal regions. Increasing the degree of exploration may reduce the chance of premature convergence, taking into account that, by doing so, we have the opportunity to improve the mapping of the whole cost function landscape. Actually, many methods have been devised to drive the search away from areas which have already been sampled. In genetic algorithms, for example, using low selective pressure decreases the chance of premature convergence but also slows the exploitation of good candidates. Another approach against premature convergence is to use selfadaptation, allowing the optimization algorithm to alter strategies and/or to change its parameters, depending on current state and performance figures - Fuzzy ASA does exactly this. Such mechanisms are often implemented not in order to prevent premature convergence, but to speed up the minimization process (rising the risk of converging prematurely to a suboptimal minimum). Now, let’s make a few comments about the situations represented by means of the previous figures. Figure 2.1 represents the best possible and easiest configuration, considering that there is only one minimum, the cost function is smooth and there is only one attraction basin. In consequence, for every starting point in the domain, a simple gradientbased algorithm will find the global optimum.

18

2 Global Optimization and Its Applications

Figure 2.2 portrays a graph of a smooth cost function presenting several local minima and attraction basins, but only two global minima in the given domain. In this fashion, depending on the starting point certain algorithms will converge to a nonglobal minimum. So, a certain amount of preliminary exploration could help to find the best region to start the exploitation process. Figure 2.3 shows a very oscillatory graph with a central (almost) flat region. A gradient-based algorithm would have difficulties in finding global minima when applied to this cost function because, starting from the center, it would converge to a nearby local minimum, and starting from more peripheral regions those kind of methods would tend also to get trapped into suboptimal valleys. Once more, methods featuring stronger exploratory ability could perform better. Figure 2.4 illustrates another scenario in which there exists a central flat region presenting null gradients. In such a case, starting points located there will not evolve, taking into account that their evolution is usually based on increments proportional to gradient values. In addition, there are two peripheral regions at which typical gradient methods will perform well, finding global optima. Figure 2.5 displays yet another situation with flat peripheral regions inside which gradient algorithms would get stagnated, considering the null gradients (derivatives) assumed at their elements. There is also a thin central part that contains the function’s global minimum - if a given state gets there, it almost surely will converge to the global minimum. Figure 2.6 represents a true defiance for global optimizations methods, considering that the cost function have three deep minima separated by almost linear segments whose derivatives (gradients) could give not much useful information about good directions to follow. Besides, the valleys are very ”thin” and it is not a trivial task, for a computer program, to find the global minimum.

2.4 Some Popular Approaches and Final Comments There are many methods aimed at globally optimizing functions, but only a few have gained high popularity. In the deterministic and differentiable realm we have the gradient-based ones and its variations [2, 3, 4, 18], that guide their search based on the negative of cost functions’ gradient (please, remember that we are talking about minimization problems). In other words, given a base point, these methods usually evolve by adding to this current state a term or terms that tend to drive the search toward regions inside which the cost function assumes lower values. This works well inside attraction basins because the gradient in a given point conveys information about the variation trend of the function in a neighborhood of that point but, as said before, whenever the seeds are located in ”wrong” attraction regions the search is driven to a local minimum. Also in the deterministic area we can highlight partition algorithms [19] for continuous global optimization problems. These methods have a very broad scope and are adequate for minimizing continuous or

2.4 Some Popular Approaches and Final Comments

19

Lipschitz-continuous objective functions over compact domains contained in Rn under very general conditions. On the other extreme, we have several stochastic and metaheuristic approaches that are able to face successfully difficult nonlinear, discontinuous and ”misbehaved” objective functions. Among them we can cite, for example: genetic algorithms, simulated annealing, cross-entropy, artificial bee colony, particle swarm optimization, grenade explosion method, cuckoo search, differential evolution, free search and nature-inspired methods [6, 7, 9, 14, 15, 17, 20], in general. We will have the opportunity to say more about their structure in the next chapter, but, the fact is that the diversity of their applications is growing every day and they are showing themselves as excellent tools for solving many complex, real world problems, not easily solvable by conventional methods. This could be partially understood by recognizing their capability to escape from strict local minima, and that is possible thanks to the flexibility in accepting temporary degradations at a given point in time in order to attain better states later. In other words, based on a particular neighborhood, it is a question of allowing, once in a while, a temporary degradation of the present state, by means of a change in the current configuration. A mechanism for controlling the degradations makes it possible to avoid the divergence of the overall process. In this fashion, it becomes possible to go away from the trap which represents a local minimum and explore another more promising region. Population-based metaheuristics also feature mechanisms allowing the departure from states near local ”traps” of the objective function. These devices, as the mutation operation in evolutionary algorithms for instance, affect individuals to assist the collective apparatus in getting out of the local minima.

20

2 Global Optimization and Its Applications

References 1. Ahrari, A., Atai, A.A.: Grenade Explosion Method - A novel tool for optimization of multimodal functions. Applied Soft Computing 10, 1132–1140 (2010) 2. Bazaraa, M., Sherali, H., Shetty, C.: Nonlinear Programming, Theory and Applications. John Wiley & Sons, New York (1993) 3. Bertsekas, D.P.: Constrained Optimization and Lagrange Multiplier Methods. Academic Press, New York (1982) 4. Bertsekas, D.P.: Nonlinear Programming. Athena Scientific, Belmont (1995) 5. Brent, R.P.: Algorithms for minimization without derivatives. Prentice Hall, Englewood Cliffs (1973) 6. Cherruault, Y., Mora, G.: Optimisation Globale - Theorie des courbes alpha-denses. Economica, Paris (2005) 7. Clerc, M.: Particle Swarm Optimization. ISTE Publishing Company, London (2006) 8. Deb, K.: Multi-Objective Optimization Using Evolutionary Algorithms. John Wiley & Sons, New York (2001) 9. Dorigo, M., St¨utzle, T.: Ant Colony Optimization. Bradford Books (2004) 10. Fletcher, R.: Practical Methods of Optimization. John Wiley & Sons, New York (1987) 11. Gill, P., Murray, W., Wright, M.H.: Practical Optimization. Academic Press (1981) 12. Hartmann, A.K., Riger, H.: Optimization Algorithms in Physics. Wiley, Berlin (2002) 13. Himmelblau, D.M.: Applied Nonlinear Programming. McGraw-Hill (1972) 14. Karaboga, D., Basturk, B.: A powerful and efficient algorithm for numerical function optimization: artificial bee colony (ABC) algorithm. Journal of Global Optimization 39, 459–471 (2007) 15. Karaboga, D., Basturk, B.: On the performance of artificial bee colony (ABC) algorithm. Applied Soft Computing 8, 687–697 (2008) 16. Lawson, C.L., Hanson, R.J.: Solving Least Squares Problems. Prentice-Hall, Englewood Cliffs (1974) 17. Michalewicz, Z.: Genetic Algorithms + Data Structures = Evolution Programs. Springer, Heidelberg (1994) 18. Nocedal, J., Wright, S.J.: Numerical Optimization. Springer, New York (1999) 19. Pint´er, J.D.: Global Optimization in Action. Kluwer Academic Publishers, Dordrecht (1996) 20. Weise, T.: Global Optimization Algorithms - Theory and Application, http://www.it-weise.de/ (accessed July 11, 2011)

Chapter 3

Metaheuristic Methods

Abstract. In this chapter we start to focus our attention only on heuristic methods, describing several important, well-established methods and trying to point out how and why they are useful whenever we face certain difficult optimization problems. Although (meta)heuristic algorithms are numerous, we opted for presenting here just a few of them, that, we believe, can give the reader a good view of the whole class. The emphasis will be on their qualitative aspects.

3.1 Introduction Nowadays, engineers, researchers and designers, in general, are confronted with problems of growing complexity, from many diverse technical and scientific sectors. Those problems can be many times expressed as optimization tasks, frequently global optimization ones. That being the case, it is possible to define an objective function, that is sought to be minimized or maximized with respect to the concerned parameters. Usually, the optimization problem is supplemented by the imposition of constraints [16, 20]. Of course, all the parameters of the adopted solutions must satisfy these constraints. In this chapter, our interest is focused on a group of methods, called metaheuristics, that include, for example, the simulated annealing paradigm, the genetic algorithms, the ant colony optimization method etc.. All those models have a common target: to solve optimization problems, usually known by their severity, as well as possible and inspired by analogies related to other, artificial or natural, systems, be them biological, chemical, electrical or thermal [1, 2, 3, 5, 6, 7]. The metaheuristics label is sometimes considered an inadequate term, often used to describe a very important subfield of stochastic optimization, that is the general class of algorithms and techniques which employ randomness to find optimal (or as optimal as possible) solutions to difficult problems. Metaheuristics could be considered the most general of these types of algorithms, being applied to a very wide range of problems [8]. H.A. e Oliveira Jr. et al.: Stochastic Global Optimization & Its Applications, ISRL 35, pp. 21–30. c Springer-Verlag Berlin Heidelberg 2012 springerlink.com 

22

3 Metaheuristic Methods

It will be possible for the reader to infer that metaheuristics methods are, to a large extent, based on a common set of principles that make it possible to design powerful algorithms. The large variety of existing metaheuristics could be attributed to different ways to combine those fundamental ideas. By the way, it is not uncommon to hear from some (thanks God, just a very tiny number) people the following phrase: ”I don’t know what is the difference between genetic algorithms and simulated annealing - they do the same thing, after all.”. Indeed, at a very simplistic viewpoint all stochastic global optimization algorithms do the same thing: exploration and exploitation in a probabilistic way, many of them employing nature-inspired models to guide their search towards better regions as fast as possible. If we follow this line of reasoning, it is unavoidable to conclude that all existent metaheuristics are only one, what is not true, definitely. Proceeding with the weird analogy, we could also say that, considering that all cars have four wheels, need fuel to move and have seats, all makes and models perform equivalently, and that is not true either. Digressions apart, it is worth to consider some ways to classify metaheuristics. We can find many categories into which metaheuristic algorithms can be classified. Depending on the subset of chosen characteristics, several classifications are possible, each of them being the result of a specific viewpoint. Here, we cite the most common ways of classifying metaheuristics. A very natural way of metaheuristics classification might be based on the roots of the algorithm, and it could be classified as a nature-inspired or not nature-inspired one, for instance. So, examples of nature-inspired algorithms are genetic and ant colony algorithms [9, 10, 11, 12], and not nature-inspired ones could be grenade explosion and cross-entropy methods. Of course, it is sometimes difficult to attribute a paradigm to one of these two classes, but references to this classification dimension are frequent in the literature. Another possibility is related to the number of states kept by algorithms during their search for optimal elements: population-based x single point search. It is based on the classification of metaheuristics by the number of candidate solutions maintained simultaneously: does the algorithm work on a population or on a single candidate solution at any time? Single solution methods have the property of describing a trajectory in the search space during the whole optimization process. Differently, population-based metaheuristics simulate multistate processes, evolving a set of points in the search space. We can also consider metaheuristics as being classified according to the way they use cost functions. While some algorithms keep the given objective function untouched, others modify it during the search, trying to escape from local minima by modifying the original landscape. Consequently, during the search process the objective function is modified according to information collected so far. So, we have fixed or variable cost functions. Although we could introduce many more classes to segregate metaheuristic algorithms, let’s go ahead and know a little more about some of them. As promised, in what follows we will describe the main characteristics of a few methods.

3.2 Genetic Algorithms

23

3.2 Genetic Algorithms Genetic algorithms were invented in the 1970s and even today new variations are being created [10, 15]. They work by iterating through fitness assessment (cost function evaluation), selection, breeding and population reassembly (they are population-based). The primary difference among the several ”flavors” is in how selection and breeding, in particular, occurs. In another dimension, implementations can differ by how coding of chromossomes is effected. Usually, we find two basic types, namely, binary and floating-point. Each representation allows different kinds of operations of mutation, reproduction and recombination. To proceed to a new population, we begin with an empty set and then select two parents from the original population, copy them, cross them over with one another, and mutate the results. In the more conventional methods, this procedure gives rise to two children, which we add to the new population. This process is repeated until we get the number of individuals nedeed to form a new population. As a family of computational models inspired by biological evolution, this family of algorithms usually encode potential solutions to particular problems in chromosome-like data structures, submitting them to recombination operations so as to preserve desirable properties. Implementations begin with a population of individuals (chromossomes) that are usually chosen at random. From this point on, evolution proceeds by evaluating elements in each generation and conducting reproduction so that better solutions have greater probability of generating offspring. The degree of adequacy (fitness) of a given element is typically relative to the current generation. Thus, better individuals are given more opportunities to produce offspring and the genetic operators (usually mutation and crossover) are applied to the individuals in the mating buffer, producing offspring that will compose the next generation. The rates at which mutation and crossover are applied are a design decision. If the rates are low enough, it is likely that some of the offspring produced will be identical to their parents. Another point is related to how many offspring are produced by crossover, and how many individuals are selected and paired in the mating buffer. After the new offspring have been created via the genetic operators the two populations of parents and children must be combined to create a new population. Considering that most genetic algorithms maintain a fixed-sized population with, say, P elements, this means that a total of P individuals need to be selected from previous populations to create a new population. One alternative is to use all the generated children (not more than P) and randomly select individuals from the old population to bring the new population up to size P. If only one or two new offspring are produced, this in effect means randomly replacing one or two individuals in the old population with the new offspring. If the number of offspring is equal to P, then the old parent population vanishes, being fully replaced by the new population. There are some choices for biasing selection: selection for reproduction and selection from the parent and child populations to produce the new one. If reproduction with emphasis on fitness degree is used, then the probability of an individual being chosen is mainly based upon its fitness function value. A direct way

24

3 Metaheuristic Methods

of doing this would be to scale fitness values of all individuals in the parent population and calculate the probability of any individual being selected based upon the obtained values. Although scaling can eliminate the problem of reduced selective pressure, often GAs using fitness proportional selection have the opposite problem - excessive selective pressure. If an individual is much fitter than the rest of population, the probability of selecting this element may become quite high. There is also the risk that many copies of this individual will be placed in the mating buffer, and the new population will be crowded with its clones, leading to premature convergence. To fix this problem, we can replace fitness proportional selection by ranked selection, that ranks individuals in the parent population, making the probability of selection a function of rank instead of fitness. Another frequent method of performing selection is tournament selection, in which a small subset of individuals is randomly chosen and the best individuals in this set are selected for mating. There is another common type of selective criterion that preserves a copy of the best individual(s) found so far, and is referred to as elitism. It is an attempt to preserve the present quality, in the hope that it could help to find better and better elements in future populations. As a matter of fact, to the term genetic algorithm are attributed several meanings, but we could say that a genetic algorithm may be defined as any population-based method using selection, reproduction and mutation operations to evolve a set of candidates in a given search space. Along the past decades, hundreds of types of genetic operators and many chromossome representations were proposed for solving virtually all kinds of optimization problems.

3.3 Particle Swarm Optimization The particle swarm optimization (PSO) method can be classified as a member of the wide category of swarm intelligence methods, normally used for solving global optimization problems [3, 17]. It was originally proposed as a simulation of social behavior and initially introduced as an optimization method in 1995. PSO is related to swarm-based theories and evolutionary computing. It can be (and it was) easily implemented in many programming languages and requires very few computational resources. Besides, it does not require gradient information of the cost function under study, only its values, and showed itself as an efficient and effective method for many difficult tasks. PSO is similar to evolutionary computation techniques in that it uses a population of candidate solutions to the problem under analysis to explore the search space. On the other hand, each individual in the population has an adaptive velocity, according to which it moves in the configuration space. Moreover, each element has a kind of particular memory, remembering the best position of the search space it has ever visited until the present. Thus, its movement shows a collective, composite trend towards its best previously visited position and the best individual of a specific neighborhood.

3.4 Differential Evolution

25

The PSO method is compatible with the basic principles of swarm intelligence, as it is able to perform simple space and time computations and to respond to quality factors in the environment, doesn’t not change its behavior every time the environment alters and is able to change its behavior when the computational cost is not prohibitive. Indeed, the swarm in typical PSO implementations does space calculations for several time steps, responding to the quality factors resulting from particle’s best position. In addition, the group changes its state only when the fittest element in the swarm changes. In this fashion, the method is adaptive and stable. There are some differences between PSO and evolutionary computing techniques. In the latter, there are (usually) three main operators involved, namely, recombination, mutation and selection. PSO does not have a direct recombination operator, but the stochastic acceleration of a particle towards its previous best position, as well as towards the best particle of the swarm, looks like the recombination procedure in evolutionary techniques. Also, the information exchange occurs only among the particle’s own experience and that of the best particle in the population, instead of being carried from fitness dependent selected ”parents” to ”descendants”, as in GAs. Moreover, the directional position updating operation of PSO could be compared to the mutation of GA. So, PSO shows itself as an useful technique for solving complex global optimization tasks and a good alternative in cases where other techniques are not sucessful, although further research is required to fully understand the dynamics and the potential limits of this technique.

3.4 Differential Evolution Differential evolution is an efficient method for global optimization of real-valued, multimodal objective functions [19]. It is conceptually simple, easy to use and to implement. In its standard implementations, it uses few control variables which remain fixed during all the optimization procedure, although we can find variants of DE endowed with adaptive behavior. In the differential evolution model, the population consists of individuals that are represented by vectors. It synthesizes new parameter vectors by adding the weighted difference between two population vectors to a third vector. If the resulting vector is fitter than a certain member of the population, the former substitutes the latter. There are many variants of this paradigm, depending on what perturbations an existing vector suffers: we can add it more than one weighted difference vector or mix the parameters of the old vector with those of the perturbed one before comparisons. The existing schemes usually differ in how a new individual is generated. An operation analogous to GA’s crossover is done after generation, by combining features of both generated individual and the current one. In this way, a random position within the vector and a crossover size are chosen. The control parameter for the crossover size is a number ∈ [0, 1], governing its probability distribution. This ”crossover” is done by taking a strip of a certain size from the generated individual

26

3 Metaheuristic Methods

to replace the same positions of the current one. After that, the resulting candidate is evaluated and substitutes the current one if its fitness is better. As stated before, the number of parameters necessary to use this method is small, namely, population size, crossover size and F ∈ [−1, 0)∪(0, 1], the latter controlling the degree of influence that differences have on the generation of mutant vectors. Thus the algorithm is very simple to understand but there is no precise indication of which scheme is the best, if any, some of them presenting better results in some problems while performing worse on others, nor is there a definitive indication of which values to use for the parameters. Anyway, differential evolution is continuously improving since its original version, being applied successfully to several scientific and technical problems and showing itself as a very promising paradigm.

3.5 Cross-Entropy Method An adaptive importance sampling algorithm was initiated by R. Rubinstein in 1997, for estimating probabilities of rare events by minimizing sample variance of the IS estimator. It was later modified to minimize the Kullback-Leibler (cross-entropy) divergence, and this evolved into the present CE method [18]. Besides simulation, the CE method can be easily applied for solving continuous multi-extremal optimization problems [18]. Regardless of the problem that the cross-entropy method is applied to, it can be described as an iterative algorithm which consists of two main steps in each iteration: • Generation of a sample of random data with a set of dynamic parameters; • Updating the previously used set of parameters controlling the generation of random data using the sample itself, aiming to improve data in the next iteration. The advantages of the cross-entropy method lie in its versatile structure, which is easily adjustable to a wide range of applications, and its simple updating procedure, which is fast and efficient. Let’s describe the measurements (individuals) obtained from an experiment as a random vector X = (X1 , ..., Xn ) with probability density function f. As in the case of genetic algorithms, information about the problem is contained in the population of potential solutions. Similarly, the information about the experiment is found in the pdf f. Nonetheless, it is best if we determine our information about the experiment with just the main components of the pdf f. This way we narrow down the search for information by compressing it. Particularly, the pdf f in the theory of point estimation relies upon a parameter vector v. The accuracy with which v can be estimated from a result of X refers to the amount of information about v which is contained in the data X. For example, the expectation and the covariance matrix provide information about the mean measurements and the variability of the measurements, respectively. In addition, a method of measuring the amount of information is to use the concept of distance which, in this case, is the Kullback-Leibler divergence. An alternative way of measuring information is to use the Shannon entropy, which defines the average number of bits required to send a

3.6 Simulated Annealing

27

message X through a (binary) communication channel. This slightly resembles the GAs that work with fixed-length binary strings as representation for individuals to obtain information. As we said, the algorithm consists of two phases: at first a solution is produced at random according to a specified probabilistic mechanism, then the parameters of the mechanism are modified on the basis of the solution obtained, in order to obtain a better solution in the next iteration. The CE method uses a graphbased formulation of the problems and views deterministic optimization problems as stochastic optimization ones. In this fashion, the random component lies either on the edges, or on the nodes of the graph. In summary, we can view CE method as the search for an importance sampling distribution that approximates as precisely as possible the true distribution corresponding to the landscape under minimization and, to pursue that target, KullbackLeibler divergence is taken as a kind of distance (although it is not), guiding the approximation process. So, it can be seen as a minimization problem whose search space is a (usually parametric) class of probability distribution functions and the cost function involves the ”distance” between a candidate p.d.f. and the true one.

3.6 Simulated Annealing Simulated Annealing was originally proposed as a combinatorial optimization method [13], that is, when the cost function is defined in a discrete domain. The method is reported to perform well in high-dimensional domains and is based on random evaluations of the cost function, allowing transitions out of local minima. There is no deterministic guarantee of finding global minima but, in a certain probabilistic sense, the algorithm tends to approach them. This method is also able to discriminate between huge variations of the landscape and small ”ripples”. First, it exploits function’s domain and reaches an area where at least one global minimum should be present. Then, it develops finer details, trying to find a good, near-optimal local minimum, if not a global minimum itself. In [4] some modifications to this algorithm are proposed, in order to apply it to the optimization of functions defined in a continuous domain - these functions do not need to be smooth or even continuous in their domain - the algorithm works based only upon cost function evaluations. In [4], the method assumes that f : C → Rn is the (bounded) function to minimize, where C is a hyper-rectangle and f might be discontinuous. It starts from a given point x0 and generates a succession of points x1 , x2 , ... approaching the global minimum of the cost function f . New candidate points are generated around the current point xi by perturbating it in a random way along each coordinate direction. The new coordinate values are uniformly distributed in intervals centered around the corresponding coordinate of xi . Half the size of these intervals along each coordinate is recorded in a so-called step vector. If the point falls outside the definition domain, a new point is randomly generated until a point belonging to the definition domain is found. The acceptance of a new candidate point xC is decided according to the well-known Metropolis criterion [4]:

28

3 Metaheuristic Methods

If Δ f ≤ 0, then accept the new point xi+1 = xC else accept the new point with probability p(Δ f ) = exp(−Δ f /T ), where Δ f = f (xC ) − f (xi ) and T is called the temperature. The SA algorithm starts at a user defined temperature T0 and generates a sequence of points until statistical mixing (or equilibrium) occurs. After that, the temperature T is reduced and a new sequence of moves is made, starting from the best point resulting from the previous phase, until a new equilibrium happens again, and so on. The process is interrupted at a temperature low enough such that no more significant improvement could be reached, according to a established stopping criterion. The SA optimization paradigm can be compared to a physical process by which a material changes state while minimizing its energy. A sufficiently slow cooling brings the material to a highly ordered state of lowest energy and, by the other side, a rapid cooling provokes defects inside the material. Indeed, a search process accepting only new points with lowest function values tends to drive states to local minima. Thanks to its decision criterion, SA permits uphill moves, depending on the values of temperature parameter. At higher temperatures, only the global behavior of the cost function is relevant to the search dynamics. As temperature values decrease, reduced neighborhoods can be explored, allowing us to reach more refined results. Although the final point is not deterministically guaranteed to be a global optimum, the method is able to proceed toward better minima even in the presence of many local minima. This method needs many function evaluations, but it is effective in finding the global minimum of difficult functions with huge number of local minima. Simulated annealing method can provide a very high reliability in the minimization of multimodal functions at relatively high computational costs, increasing with the number of dimensions of the problem. However, some design issues have to be addressed. The first one is the choice of the starting temperature T0 . If we choose a very high T0 , there will be a waste of computational resources. In the contrary case, we could get caught in suboptimal regions. Besides, each cost function behaves in a particular way, so that it is very hard to establish a general rule for calculating the ideal starting temperature and, in practice, some experimental monitoring is of great help in finding good values for it, for the number of function evaluations at each temperature and about conditions for stopping the whole process, that’s to say, to decide whether the algorithm converged or not. Observe that, at high temperatures, almost any change is accepted - the algorithm sweeps a relatively large neighborhood of the current state. At lower temperatures, the transitions to higher energy states becomes less frequent and the search tends to be located in smaller regions. The cooling schedule is of paramount importance to the performance of SA. It consists of initial temperature, decrement function for the temperature, final temperature specified by a stop criterion, and the number of transitions in the homogeneous Markov chain at each temperature step. Of course, there is a tradeoff between the quality of the final solution and the execution time

3.6 Simulated Annealing

29

in SA, the latter being sensitive to the decrement speed of the temperature. As we said before, if the temperature drops too rapidly, the current state is likely to be trapped in poor attraction basins and, in addition, it is not a trivial task to identify an equilibrium of the system at each temperature, so that the corresponding Markov chain can be considered as having mixed. In this fashion, designing an adequate cooling schedule that could generally lead to global optima in a fast and effective way is not an easy task [14].

30

3 Metaheuristic Methods

References 1. Ahrari, A., Atai, A.A.: Grenade Explosion Method - A novel tool for optimization of multimodal functions. Applied Soft Computing 10, 1132–1140 (2010) 2. Birbil, S.I., Fang, S.: An Electromagnetism-like Mechanism for Global Optimization. Journal of Global Optimization 25, 263–282 (2003) 3. Clerc, M.: Particle Swarm Optimization. ISTE Publishing Company, London (2006) 4. Corana, A., Marchesi, M., Martini, C., Ridella, S.: Minimizing multimodal functions of continuous variables with the simulated annealing algorithm. ACM Trans. Mathematical Software 13, 262–280 (1987) 5. Deb, K.: Multi-Objective Optimization Using Evolutionary Algorithms. John Wiley & Sons, New York (2001) 6. Dorigo, M., St¨utzle, T.: Ant Colony Optimization. Bradford Books (2004) 7. Dr´eo, J., P´etrowski, A., Siarry, P., Taillard, E.: Metaheuristics for Hard Optimization Methods and Case Studies - Simulated Annealing, Tabu Search, Evolutionary and Genetic Algorithms, Ant Colonies. Springer, Berlin (2006) 8. Glover, F., Kochenberger, G.A.: Handbook of metaheuristics. Springer, Heidelberg (2003) 9. Hartmann, A.K., Riger, H.: Optimization Algorithms in Physics. Wiley, Berlin (2002) 10. Holland, J.H.: Adaptation in Natural and Artificial Systems. University of Michigan Press, Ann Arbor (1975) 11. Karaboga, D., Basturk, B.: A powerful and efficient algorithm for numerical function optimization: artificial bee colony (ABC) algorithm. Journal of Global Optimization 39, 459–471 (2007) 12. Karaboga, D., Basturk, B.: On the performance of artificial bee colony (ABC) algorithm. Applied Soft Computing 8, 687–697 (2008) 13. Kirkpatrick, S., Gelatt Jr., C.D., Vecchi, M.P.: Optimization by simulated annealing. Science 220, 671–680 (1983) 14. van Laarhoven, P.J.M., Aarts, E.H.L.: Simulated Annealing: Theory and Applications. D. Reidel, Dordrecht (1987) 15. Michalewicz, Z.: Genetic Algorithms + Data Structures = Evolution Programs. Springer, Heidelberg (1994) 16. Nocedal, J., Wright, S.J.: Numerical Optimization. Springer, New York (1999) 17. Parsopoulos, K.E., Vrahatis, M.N.: Recent approaches to global optimization problems through Particle Swarm Optimization. Natural Computing 1, 235–306 (2002) 18. Rubinstein, R.Y., Kroese, D.P.: The cross-entropy method: A unified approach to combinatorial optimization, Monte-Carlo simulation, and machine learning. Springer, New York (2004) 19. Storn, R., Price, K.: Differential evolution - a simple and efficient heuristic for global optimization over continuous spaces. Journal of Global Optimization 11(4), 341–359 (1997) 20. Weise, T.: Global Optimization Algorithms - Theory and Application, http://www.it-weise.de/ (accessed July 11, 2011)

Part II

ASA, Fuzzy ASA and Their Characteristics

Chapter 4

Adaptive Simulated Annealing Lester Ingber

Abstract. Adaptive Simulated Annealing (ASA) is a C-language code that finds the best global fit of a nonlinear cost-function over a D-dimensional space. ASA has over 100 OPTIONS to provide robust tuning over many classes of nonlinear stochastic systems. These many OPTIONS help ensure that ASA can be used robustly across many classes of systems.

4.1 Introduction Simulated annealing (SA) presents an optimization technique that can: (a) process cost functions possessing quite arbitrary degrees of nonlinearities, discontinuities, and stochasticity; (b) process quite arbitrary boundary conditions and constraints imposed on these cost functions; (c) be implemented quite easily with the degree of coding quite minimal relative to other nonlinear optimization algorithms; (d) statistically guarantee finding an optimal solution. Adaptive Simulated Annealing (ASA) is a C-language code that finds the best global fit of a nonlinear cost-function over a D-dimensional space. The basic algorithm was originally published as Very Fast Simulated Reannealing (VFSR) in 1989 [9], after two years of application on combat simulations. The code [13] can be used at no charge and downloaded from http://www.ingber.com/#ASA with mirrors at: http://alumni.caltech.edu/˜ingber http://asa-caltech.sourceforge.net https://code.google.com/p/adaptive-simulated-annealing . ASA has over 100 OPTIONS to provide robust tuning over many classes of nonlinear stochastic systems. The current number as of this chapter is 152. These H.A. e Oliveira Jr. et al.: Stochastic Global Optimization & Its Applications, ISRL 35, pp. 33–62. c Springer-Verlag Berlin Heidelberg 2012 springerlink.com 

34

4 Adaptive Simulated Annealing

many OPTIONS help ensure that ASA can be used robustly across many classes of systems. In the context of this book, it will be seen in the discussions that the “QUENCHing” OPTIONS are among the most important for controlling Adaptive Simulated Annealing. Fuzzy ASA algorithms in particular offer new ways of controlling how these QUENCHing OPTIONS may be applied across many classes of problems.

4.1.1 LICENSE and Contributions The code originally was issued under a BSD-type License. This was changed to a form consistent with the less restrictive New BSD License http://en.wikipedia.org/wiki/BSD License beginning with Version 28.1 in February 2011. I have had several queries as to why I did not follow a GPL license. I felt and still feel, similar to many other people who make code available at no charge to others, that the GPL license is just too cumbersome and onerous. I have made my code available at no charge to anyone or any company, subject to very simple terms. If some user contributions do not quite fit into the code per se, I have put or referenced their contributions into the asa contrib.txt or ASA-NOTES files. I do not think this has stymied people from contributing to the code. For example, in http://www.ingber.com/asa contrib.txt there are references to several major contributions made by other people, e.g., Matlab interface, RLAB interface, AMPL interface, and Haskell Interface, The ASA PARALLEL OPTIONS were contributed as a team effort I led, as Principal Investigator of a 1994 National Science Foundation Parallelizing ASA and PATHINT Project (PAPP). One of the authors of this book has contributed FUZZY ASA OPTIONS [44, 45, 46]. Another user referenced in http://www.ingber.com/asa contrib.txt contributed explicit code used in ASA to help parallelize optimization of chip design. The current list of CONTRIBUTORS in the ASA-CHANGES file that comes with code numbers 56. All these contributions have resulted in many versions of the code. The current list of VERSION DATES in the ASA-CHANGES file that comes with code numbers 586 since 1987. A few ASA papers showed how the code could be useful for many projects [1, 14, 16].

4.1.2 Organization of Chapter The next two sections give a short introduction to simulated annealing and to ASA. The first section discusses the theoretical foundations of ASA, and the second section discusses the practical implementation of ASA. The following section gives an overview and several approaches that consider why tuning is necessary in any sampling algorithm like SA, GA, etc. These issues have been addressed according to

4.2 Theoretical Foundations of Adaptive Simulated Annealing (ASA)

35

user feedback, i.e., what helps many users in many disciplines with a broad range of experience to no experience. This work follows theoretical development of the algorithm that can be found in other ASA papers [9, 14, 16]. Other sections that follow illustrate the use of OPTIONS and are devoted to Adaptive OPTIONS and Multiple Systems. The last section is the conclusion. Most of this chapter has organized information that has been collected on the use of the code since 1987, and is contained in some form in multiple files, e.g., ASA-README, ASA-NOTES, asa contrib.txt, asa examples.txt, etc.

4.2 Theoretical Foundations of Adaptive Simulated Annealing (ASA) The unique aspect of simulated annealing (SA) is its property of (weak) ergodicity, permitting such code to statistically and reasonably sample a parameter space. Note that for very large systems, ergodicity is not an entirely rigorous concept when faced with the real task of its computation [41]. In this chapter “ergodic” is used in a very weak sense, as it is not proposed, theoretically or practically, that all states of the system are actually to be visited.

4.2.1 Shades of Simulated Annealing Even “standard” SA is not without its critics. Some negative features of SA are that it can: (A) be quite time-consuming to find an optimal fit, especially when using the “standard” Boltzmann technique; (B) be difficult to fine tune to specific problems, relative to some other fitting techniques; (C) suffer from “over-hype” and faddish misuse, leading to misinterpretation of results; (D) lose its ergodic property by misuse, e.g., by transforming SA into a method of “simulated quenching” (SQ) for which there is no statistical guarantee of finding an optimal solution. There also is a large and growing domain of SA-like techniques, which do not theoretically predict general statistical optimality, but which are extremely powerful for certain classes of problems. There are many examples given in published papers addressing robust problems across many disciplines. There are many reviews of simulated annealing, comparisons among simulated annealing algorithms, and between simulated annealing and other algorithms [6, 7, 14, 16, 39, 49]. It is important to compare the basic theoretic constraints of true SA with actual practice on a range of problems spanning many disciplines. This may help to address what may yet be expected in terms of better necessary conditions on SA to make it a more efficient algorithm, as many believe that the present sufficiency conditions are overly restrictive.

36

4 Adaptive Simulated Annealing

4.2.2 Critics of SA The primary criticism is that it is too slow. This is partially addressed here by summarizing some work in appropriately adapting SQ to many problems. Another criticism is that it is “overkill” for many of the problems on which it is used. This is partially addressed here by pointing to much work demonstrating that it is not insignificant that many researchers are using SA/SQ because of the ease in which constraints and complex cost functions can easily be approached and coded. There is another class of criticisms that the algorithm is too broadly based on physical intuition and is too short on mathematical rigor [5]. In some particular bitter and scathing critiques authors take offense at the lack of reference to other prior work [47], the use of “metaphysical non-mathematical ideas of melting, cooling, and freezing” reference to the physical process of annealing as used to popularize SA [40], and they give their own calculations to demonstrate that SA can be a very poor algorithm to search for global optima in some instances. That there are undoubtedly other references that should be more regularly referenced is an objective issue that has much merit, with respect to SA as well as to other research projects. The other criticisms may be considered by some to be more subjective, but they are likely no more extreme than the use of SQ to solve for global optima under the protective umbrella of SA.

4.2.3 “Standard” Simulated Annealing (SA) The Metropolis Monte Carlo integration algorithm [43] was generalized by the Kirkpatrick algorithm to include a temperature schedule for efficient searching [40]. A sufficiency proof was then shown to put an lower bound on that schedule as 1/ log(t), where t is an artificial time measure of the annealing schedule [8]. However, independent credit usually goes to several other authors for independently developing the algorithm that is now recognized as simulated annealing [4, 47].

4.2.4 Boltzmann Annealing (BA) Credit for the first simulated annealing is generally recognized as a Monte Carlo importance-sampling technique for doing large-dimensional path integrals arising in statistical physics problems [43]. This method was generalized to fitting non-convex cost-functions arising in a variety of problems, e.g., finding the optimal wiring for a densely wired computer chip [40]. The choices of probability distributions described in this section are generally specified as Boltzmann annealing (BA) [48]. The method of simulated annealing consists of three functional relationships. 1. g(x): Probability density of state-space of D parameters x = {xi ; i = 1, D}. 2. h(Δ E): Probability for acceptance of new cost-function given the just previous value.

4.2 Theoretical Foundations of Adaptive Simulated Annealing (ASA)

37

3. T (k): schedule of “annealing” the “temperature” T in annealing-time steps k, i.e., of changing the volatility or fluctuations of one or both of the two previous probability densities.

The acceptance probability is based on the chances of obtaining a new state with “energy” Ek+1 relative to a previous state with “energy” Ek , h(Δ E) =

exp(−Ek+1 /T ) exp(−Ek+1 /T ) + exp(−Ek /T )

(4.1)

1 1 + exp(Δ E/T )

(4.2)

≈ exp(−Δ E/T ),

(4.3)

=

where Δ E represents the “energy” difference between the present and previous values of the energies (considered here as cost functions) appropriate to the physical problem, i.e., Δ E = Ek+1 − Ek . This essentially is the Boltzmann distribution contributing to the statistical mechanical partition function of the system [2]. This can be described by considering: a set of states labeled by x, each with energye(x);a set of probability distributions p(x); and the energy distribution per state d e(x) , giving an aggregate energy E,

∑ p(x)d

  e(x) = E.

(4.4)

x

The principle of maximizing the entropy, S, S = − ∑ p(x) ln[p(x)/p(x)], ¯

(4.5)

x

where x¯ represents a reference state, using Lagrange multipliers [42] to constrain the energy to average value T , leads to the most likely Gibbs distribution G(x),   1 exp − H(x)/T , (4.6) Z in terms of the normalizing partition function Z, and the Hamiltonian H operator as the “energy” function, G(x) =

  Z = ∑ exp − H(x)/T .

(4.7)

x

For such distributions of states and acceptance probabilities defined by functions such as h(Δ E), the equilibrium principle of detailed balance holds. I.e., the distributions of states before, G(xk ), and after, G(xk+1 ), applying the acceptance criteria, h(Δ E) = h(Ek+1 − Ek ) are the same:   G(xk )h Δ E(x) = G(xk+1 ).

(4.8)

38

4 Adaptive Simulated Annealing

This is sufficient to establish that all states of the system can be sampled, in theory. However, the annealing schedule interrupts equilibrium every time the temperature is changed, and so, at best, this must be done carefully and gradually. An important aspect of the SA algorithm is to pick the ranges of the parameters to be searched. In practice, computation of continuous systems requires some discretization, so without loss of much generality for applications described here, the space will be assumed to be discretized. There are additional constraints that are required when dealing with generating and cost functions with integral values. Many practitioners use novel techniques to narrow the range as the search progresses. For example, based on functional forms derived for many physical systems belonging to the class of Gaussian-Markovian systems, one could choose an algorithm for g, g(Δ x) = (2π T )−D/2 exp[−Δ x2 /(2T )],

(4.9)

where Δ x = x − x0 is the deviation of x from x0 (usually taken to be the justpreviously chosen point), proportional to a “momentum” variable, and where T is a measure of the fluctuations of the Boltzmann distribution g in the D-dimensional x-space. Given g(Δ x), it has been proven [8] that it suffices to obtain a global minimum of E(x) if T is selected to be not faster than T (k) =

T0 , ln k

(4.10)

with T0 “large enough.” A heuristic demonstration shows that this equation for T will suffice to give a global minimum of E(x) [48]. In order to statistically assure, i.e., requiring many trials, that any point in x-space can be sampled infinitely often in annealing-time (IOT), it suffices to prove that the products of probabilities of not generating a state x IOT for all annealing-times successive to k0 yield zero, ∞

∏ (1 − gk) = 0.

(4.11)

k=k0

This is equivalent to ∞

∑ gk = ∞.

(4.12)

k=k0

The problem then reduces to finding T (k) to satisfy this equation. For BA, if T (k) is selected to be the Boltzmann criteria above, then the generating distribution g above gives ∞

∑

k=k0

gk ≥

∞

∑

k=k0

exp(− ln k) =

∞

∑

1/k = ∞.

(4.13)

k=k0

Although there are sound physical principles underlying the choices of the Boltzmann criteria above [43], it was noted that this method of finding the global minimum in x-space was not limited to physics examples requiring bona fide

4.2 Theoretical Foundations of Adaptive Simulated Annealing (ASA)

39

“temperatures” and “energies.” Rather, this methodology can be readily extended to any problem for which a reasonable probability density h(Δ x) can be formulated [40].

4.2.5 Simulated Quenching (SQ) Many researchers have found it very attractive to take advantage of the ease of coding and implementing SA, utilizing its ability to handle quite complex cost functions and constraints. However, the long time of execution of standard Boltzmann-type SA has many times driven these projects to utilize a temperature schedule too fast to satisfy the sufficiency conditions required to establish a true (weak) ergodic search. A logarithmic temperature schedule is consistent with the Boltzmann algorithm, e.g., the temperature schedule is taken to be ln k0 , (4.14) lnk where T is the “temperature,” k is the “time” index of annealing, and k0 is some starting index. This can be written for large k as Tk = T0

Δ T = −T0

ln k0 Δ k ,k  1 k(ln k)2

Tk+1 = Tk − T0

ln k0 . k(ln k)2

(4.15) (4.16)

However, some researchers using the Boltzmann algorithm use an exponential schedule, e.g., Tk+1 = cTk , 0 < c < 1

ΔT = (c − 1)Δ k, k  1 Tk   Tk = T0 exp (c − 1)k ,

(4.17) (4.18) (4.19)

with expediency the only reason given. While perhaps someday some less stringent necessary conditions may be developed for the Boltzmann algorithm, this is not now the state of affairs. The question arises, what is the value of this clear misuse of the claim to use SA to help solve these problems/systems? Adaptive simulated annealing (ASA) [9, 13], in fact does justify an exponential annealing schedule, but only if a particular distribution is used for the generating function. In many cases it is clear that the researchers already know quite a bit about their system, and the convenience of the SA algorithm, together with the need for some global search over local optima, makes a strong practical case for the use of SQ. In

40

4 Adaptive Simulated Annealing

some of these cases, the researchers have been more diligent with regard to their numerical SQ work, and have compared the efficiency of SQ to some other methods they have tried. Of course, the point must be made that while SA’s true strength lies in its ability to statistically deliver a true global optimum, there are no theoretical reasons for assuming it will be more efficient than any other algorithm that also can find this global optimum.

4.2.6 Fast Annealing (FA) Although there are many variants and improvements made on the “standard” Boltzmann algorithm described above, many textbooks finish just about at this point without going into more detail about other algorithms that depart from this explicit algorithm [49]. Specifically, it was noted that the Cauchy distribution has some definite advantages over the Boltzmann form [48]. The Cauchy distribution, g(Δ x) =

T , (Δ x2 + T 2 )(D+1)/2

(4.20)

has a “fatter” tail than the Gaussian form of the Boltzmann distribution, permitting easier access to test local minima in the search for the desired global minimum. It is instructive to note the similar corresponding heuristic demonstration, that the Cauchy g(Δ x) statistically finds a global minimum. If the Boltzmann T is replaced by T (k) =

T0 , k

(4.21)

then here ∞

T0

∞

1

∑ gk ≈ Δ xD+1 ∑ k = ∞. k0

(4.22)

k0

Note that the “normalization” of g has introduced the annealing-time index k, giving some insights into how to construct other annealing distributions. The method of FA is thus seen to have an annealing schedule exponentially faster than the method of BA. This method has been tested in a variety of problems [48].

4.2.7 Adaptive Simulated Annealing (ASA) In a variety of physical problems we have a D-dimensional parameter-space. Different parameters have different finite ranges, fixed by physical considerations, and different annealing-time-dependent sensitivities, measured by the derivatives of the cost-function at local minima. BA and FA have distributions that sample infinite ranges, and there is no provision for considering differences in each parameter-dimension; e.g., different sensitivities might require different annealing

4.2 Theoretical Foundations of Adaptive Simulated Annealing (ASA)

41

schedules. This prompted the development of a new probability distribution to accommodate these desired features [9], leading to a variant of SA that in fact justifies an exponential temperature annealing schedule. These are among several considerations that gave rise to Adaptive Simulated Annealing (ASA). Full details are available by obtaining the publicly available source code [13]. ASA considers a parameter αki in dimension i generated at annealing-time k with the range

αki ∈ [Ai , Bi ], calculated with the random variable

(4.23)

yi ,

i αk+1 = αki + yi (Bi − Ai ),

(4.24)

yi ∈ [−1, 1].

(4.25)

Define the generating function gT (y) =

D

D

1

∏ 2(|yi | + Ti) ln(1 + 1/Ti) ≡ ∏ giT (yi ). i=1

(4.26)

i=1

Its cumulative probability distribution is 1

GT (y) =

y

−1

D

...

y

dy 1 . . . dy D gT (y ) ≡

(4.27)

i=1

−1

GiT (yi ) =

D

∏ GiT (yi ),

1 sgn (yi ) ln(1 + |yi |/Ti ) + . 2 2 ln(1 + 1/Ti)

(4.28)

yi is generated from a ui from the uniform distribution ui ∈ U [0, 1],

(4.29)

i 1 yi = sgn (ui − )Ti [(1 + 1/Ti)|2u −1| − 1]. 2 It is straightforward to calculate that for an annealing schedule for Ti

Ti (k) = T0i exp(−ci k1/D ),

(4.30)

(4.31)

a global minima statistically can be obtained. I.e., ∞

∞

D

k0

k0

i=1

1

1

∑ gk ≈ ∑ [ ∏ 2|yi|ci ] k = ∞. It seems sensible to choose control over ci , such that

(4.32)

42

4 Adaptive Simulated Annealing

T f i = T0i exp(−mi ) when k f = expni ,

(4.33)

ci = mi exp(−ni /D),

(4.34)

where mi and ni can be considered “free” parameters to help tune ASA for specific problems. It has proven fruitful to use the same type of annealing schedule for the acceptance function h as used for the generating function g, but with the number of acceptance points, instead of the number of generated points, used to determine the k for the acceptance temperature. i are generated from old parameters αki from New parameters αk+1 i αk+1 = αki + yi (Bi − Ai ),

(4.35)

i αk+1 ∈ [Ai , Bi ].

(4.36)

constrained by

I.e., yi ’s are generated until a set of D are obtained satisfying these constraints.

4.2.7.1 Reannealing Whenever doing a multi-dimensional search in the course of a real-world nonlinear physical problem, inevitably one must deal with different changing sensitivities of the α i in the search. At any given annealing-time, it seems sensible to attempt to “stretch out” the range over which the relatively insensitive parameters are being searched, relative to the ranges of the more sensitive parameters. This can be by periodically rescaling the annealing-time k, essentially reannealing, e.g., every hundred or so acceptance-events, in terms of the sensitivities si calculated at the most current minimum value of the cost function, L, si = ∂ L/∂ α i .

(4.37)

In terms of the largest si = smax , ASA can reanneal by using a rescaling for each ki of each parameter dimension, ki → k i ,

(4.38)

T ik = Tik (smax /si ),

(4.39)

 D k i = ln(Ti0 /Tik )/ci .

(4.40)

Ti0 is set to unity to begin the search, which is ample to span each parameter dimension.

4.2 Theoretical Foundations of Adaptive Simulated Annealing (ASA)

43

The acceptance temperature is similarly rescaled. In addition, since the initial acceptance temperature is set equal to a trial value of L, this is typically very large relative to the global minimum. Therefore, when this rescaling is performed, the initial acceptance temperature is reset to the most current minimum of L, and the annealing-time associated with this temperature is set to give a new temperature equal to the lowest value of the cost-function encountered to annealing-date. Also generated are the “standard deviations” of the theoretical forms, calculated as [∂ 2 L/(∂ α i )2 ]−1/2 , for each parameter αi . This gives an estimate of the “noise” that accompanies fits to stochastic data or functions. At the end of the run, the offdiagonal elements of the “covariance matrix” are calculated for all parameters. This inverse curvature of the theoretical cost function can provide a quantitative assessment of the relative sensitivity of parameters to statistical errors in fits to stochastic systems. A few other twists can be added, and such searches undoubtedly will never be strictly by rote. Physical systems are so different, some experience with each one is required to develop a truly efficient algorithm.

4.2.7.2 Self Optimization Another feature of ASA is its ability to recursively self optimize its own Program Options, e.g., the ci parameters described above, for a given system. An application is described below.

4.2.7.3 Quenching Another adaptive feature of ASA is its ability to perform quenching. This is applied by noting that the temperature schedule above can be redefined as Qi /D

Ti (ki ) = T0i exp(−ci ki

),

ci = mi exp(−ni Qi /D),

(4.41) (4.42)

in terms of the “quenching factor” Qi . The above proof fails if Qi > 1 as D

∑ ∏ 1/kQi/D = ∑ 1/kQi < ∞. k

(4.43)

k

This simple calculation shows how the “curse of dimensionality” arises, and also gives a possible way of living with this disease. In ASA, the influence of large dimensions becomes clearly focused on the exponential of the power of k being 1/D, as the annealing required to properly sample the space becomes prohibitively slow. So, if we cannot commit resources to properly sample the space ergodically, then

44

4 Adaptive Simulated Annealing

for some systems perhaps the next best procedure would be to turn on quenching, whereby Qi can become on the order of the size of number of dimensions. The scale of the power of 1/D temperature schedule used for the acceptance function can be altered in a similar fashion. However, this does not affect the annealing proof of ASA, and so this may be used without damaging the (weak) ergodicity property.

4.2.8 VFSR and ASA The above defines this method of adaptive simulated annealing (ASA), previously called very fast simulated reannealing (VFSR) [9] only named such to contrast it the previous method of fast annealing (FA) [48]. The annealing schedules for the temperatures Ti decrease exponentially in annealing-time k, i.e., Ti = Ti0 exp(−ci k1/D ). Of course, the fatter the tail of the generating function, the smaller the ratio of acceptance to generated points in the fit. However, in practice, when properly tuned, it is found that for a given generating function, this ratio is approximately constant as the fit finds a global minimum. Therefore, for a large parameter space, the efficiency of the fit is determined by the annealing schedule of the generating function. A major difference between ASA and BA algorithms is that the ergodic sampling takes place in an n + 1 dimensional space, i.e., in terms of n parameters and the cost function. In ASA the exponential annealing schedules permit resources to be spent adaptively on reannealing and on pacing the convergence in all dimensions, ensuring ample global searching in the first phases of search and ample quick convergence in the final phases. The acceptance function h(Δ x) chosen is the usual Boltzmann form satisfying detailed balance, and the acceptance-temperature reannealing paces the convergence of the cost function to permit ergodic searching in the n-parameter space considered as the independent variables of the dependent cost function.

4.3 Practical Implementation of ASA Details of the ASA algorithm are best obtained from the code itself and from published papers. There are three parts to its basic structure.

4.3.1 Generating Probability Density Function In a D-dimensional parameter space with parameters pi having ranges [Ai , Bi ], about the k’th last saved point (e.g., a local optima), pik , a new point is generated using a distribution defined by the product of distributions for each parameter, gi (yi ; Ti ) in

4.3 Practical Implementation of ASA

45

terms of random variables yi ∈ [−1, 1], where pik+1 = pik + yi (Bi − Ai ), and “temperatures” Ti , gi (yi ; Ti ) =

1 . 2(|yi | + Ti) ln(1 + 1/Ti)

(4.44)

The OPTIONS USER GENERATING FUNCTION permits using an alternative to this ASA distribution function.

4.3.2 Acceptance Probability Density Function The cost functions, C(pk+1 ) −C(pk ), are compared using a uniform random generator, U ∈ [0, 1), in a “Boltzmann” test: If   exp[− C(pk+1 ) − C(pk ) /Tcost ] > U,

(4.45)

where Tcost is the “temperature” used for this test, then the new point is accepted as the new saved point for the next iteration. Otherwise, the last saved point is retained. The OPTIONS USER ACCEPT ASYMP EXP or USER ACCEPT THRESHOLD permit using alternatives to this Boltzmann distribution function.

4.3.3 Reannealing Temperature Schedule The annealing schedule for each parameter temperature, Ti from a starting temperature Ti0 , is 1/D

Ti (ki ) = T0i exp(−ci ki

).

(4.46)

The annealing schedule for the cost temperature is developed similarly to the parameter temperatures. However, the index for reannealing the cost function, kcost , is determined by the number of accepted points, instead of the number of generated points as used for the parameters. This choice was made because the Boltzmann acceptance criteria uses an exponential distribution that is not as fat-tailed as the ASA distribution used for the parameters. This schedule can be modified using several OPTIONS. In particular, the Pre-Compile OPTIONS USER COST SCHEDULE permits quite arbitrary functional modifications for this annealing schedule. As determined by the Program Options selected, the parameter “temperatures” may be periodically adaptively reannealed, or increased relative to their previous values, using their relative first derivatives with respect to the cost function, to guide the search “fairly” among the parameters. As determined by the Program Options selected, the reannealing of the cost temperature resets the scale of the annealing of the cost acceptance criteria as

46

4 Adaptive Simulated Annealing 1/D

Tcost (kcost ) = T0cost exp(−ccost kcost ).

(4.47)

The new T0cost is taken to be the minimum of the current initial cost temperature and the maximum of the absolute values of the best and last cost functions and their difference. The new kcost is calculated taking Tcost as the maximum of the current value and the absolute value of the difference between the last and best saved minima of the cost function, constrained not to exceed the current initial cost temperature. This procedure essentially resets the scale of the annealing of the cost temperature within the scale of the current best or last saved minimum. This default algorithm for reannealing the cost temperature, taking advantage of the ASA importance sampling that relates most specifically to the parameter temperatures, while often is quite efficient for some systems, may lead to problems in dwelling too long in local minima for other systems. In such case, the user may also experiment with alternative algorithms effected using the Reanneal Cost OPTIONS. For example, ASA provides an alternative calculation for the cost temperature, when Reanneal Cost < -1 or > 1, that periodically calculates the initial and current cost temperatures or just the initial cost temperature, resp., as a deviation over a sample of cost functions. These reannealing algorithms can be changed adaptively by the user, e.g., by using USER REANNEAL COST and USER REANNEAL PARAMETERS.

4.3.4 QUENCH PARAMETERS=FALSE This OPTIONS permits you to alter the basic algorithm to perform selective “quenching,” i.e., faster temperature cooling than permitted by the ASA algorithm. This can be very useful, e.g., to quench the system down to some region of interest, and then to perform proper annealing for the rest of the run. However, note that once you decide to quench rather than to truly anneal, there no longer is any statistical guarantee of finding a global optimum. Once you decide you can quench, there are many more alternative algorithms you might wish to choose for your system, e.g., creating a hybrid global-local adaptive quenching search algorithm, e.g., using USER REANNEAL PARAMETERS. Note that just using the quenching OPTIONS provided with ASA can be quite powerful, as demonstrated in the http://www.ingber.com/asa examples.txt file. Setting QUENCH PARAMETERS to TRUE can be extremely useful in very large parameter dimensions; see the ASA-NOTES file under the section on Quenching. Many parameters can be conveniently read in from the asa opt file. E.g., User Quench Cost Scale and User Quench Param Scale all are read in if OPTIONS FILE DATA, QUENCH COST, and QUENCH PARAMETERS are TRUE.

4.4 Tuning Guidelines

47

4.3.5 QUENCH COST=FALSE If QUENCH COST is set to TRUE, the scale of the power of 1/D temperature schedule used for the acceptance function can be altered in a similar fashion to that described above when QUENCH PARAMETERS is set to TRUE. However, note that this OPTIONS does not affect the annealing proof of ASA, and so this may be used without damaging the statistical ergodicity of the algorithm. Even greater functional changes can be made using the following Pre-Compile OPTIONS: USER USER USER USER

COST SCHEDULE ACCEPT ASYMP EXP ACCEPT THRESHOLD ACCEPTANCE TEST

If QUENCH COST=TRUE User Quench Cost Scale must be defined.

4.3.6 QUENCH COST SCALE=TRUE When QUENCH COST is TRUE, if QUENCH COST SCALE is TRUE, then the temperature scale and the temperature index are affected by User Quench Cost Scale. This can have the effect of User Quench Cost Scale appear contrary, as the effects on the temperature from the temperature scale and the temperature index can have opposing effects. However, these defaults are perhaps most intuitive when User Quench Cost Scale is on the order of the parameter dimension. When QUENCH COST is TRUE, if QUENCH COST SCALE is FALSE, only the temperature index is affected by User Quench Cost Scale. The same effect could be managed by raising Temperature Anneal Scale to the appropriate power, but this may not be as convenient.

4.4 Tuning Guidelines 4.4.1 The Necessity for Tuning I am often asked how I can help someone tune their system, and they send me their cost function or a list of the ASA OPTIONS they are using. Most often, the best help I can provide is based on my own experience that nonlinear systems typically are non-typical. In practice, that means that trying to figure out the nature of the cost function under sampling in order to tune ASA (or likely to similarly tune a hard problem under any sampling algorithm), by examining just the cost function, likely will not be as productive as generating more intermediate printout, e.g., setting

48

4 Adaptive Simulated Annealing

ASA PRINT MORE to TRUE, and looking at this output as a “grey box” of insight into your optimization problem. Larger files with more information is provided by setting ASA PIPE FILE to TRUE. Treat the output of ASA as a simulation in the ASA parameter space, which usually is quite a different space than the variable space of your system. For example, you should be able to see where and how your solution might be getting stuck in a local minima for a very long time, or where the last saved state is still fluctuating across a wide portion of your state space. These observations should suggest how you might try speeding up or slowing down annealing/quenching of the parameter space and/or tightening or loosening the acceptance criteria at different stages by modifying the OPTIONS, e.g., starting with the OPTIONS that can be easily adjusted using the asa opt file. The ASA-NOTES file that comes with the ASA code provides some guidelines for tuning that may provide some insights, especially the section Some Tuning Guidelines. An especially important guide is to examine the output of ASA at several stages of sampling, to see if changes in parameter and temperatures are reasonably correlated to changes in the cost function. Examples of useful OPTIONS and code that often give quick changes in tuning in some problems are in the file http://www.ingber.com/asa examples.txt under WWW. Some of the reprint files of published papers in the ingber.com site provide other examples in harder systems, and perhaps you might find some examples of harder systems using ASA similar to your own in http://www.ingber.com/asa papers.html under WWW. This is the best way to add some Art to the Science of annealing. While the upside of using ASA is that it has many OPTIONS available for tuning, derived in large part from feedback from many users over many years, making it extremely robust across many systems, the downside is that the learning curve can be steep especially if the default settings or simple tweaking in asa opt do not work very well for your particular system, and you then must turn to using more ASA OPTIONS. Most of these OPTIONS have useful guides in the ASA TEMPLATEs in asa usr.c, as well as being documented here. If you really get stuck, you may consider working with someone else who already has climbed this learning curve and whose experience might offer quick help. Tuning is an essential aspect of any sampling algorithm if it is to be applied to many classes of systems. It just doesn’t make sense to compare sampling algorithms unless you are prepared to properly tune each algorithm to each system being optimized or sampled.

4.4.2 Construction of the Code I sometimes get a query like: “I used your ASA code some years ago with good results and want to thank you for providing it. However even back then i noticed that it was in urgent need of a good refactoration, as described in http://en.wikipedia.org/wiki/Refactor .

4.4 Tuning Guidelines

49

I encourage you to go over your code and split it up in more readable chunks. today’s compilers are pretty good at optimizing the result so it will not impact your programs performance. Again, thank you very much for your excellent program.”

My reply is typically along these lines: “When I first wrote the code it was in broken into multiple files which were easy to take care of. I made the decision, which feedback has shown to be a good one, to make the code look less formidable to many users by aggregating the code into just a few files. The code is used widely across many disciplines, but often by expert people or groups without computer science skills, and often tuning can be accomplished by tweaking the parameter file and not having to deal with the .c files very much. Even if I choose to keep just a few files, I just do not have the time to rewrite the code into better code similar to how I write code now, 20 years later (I first wrote the VFSR code in 1987). However, for me at least, the structure of the code makes it very easy to maintain, and I been able to be responsive to any major changes that might come up. The ASA-CHANGES files reflects this. I have led teams of extremely bright and competent math-physics and computerscience people in several disciplines over the years, and I have also seen how code that may be written in exemplary languages, whether C, Java, C++, python, etc., nonetheless can be rotten to maintain if it is not written in a “functional” manner that better reflects the underlying algebra or physical process, e.g., as most people would program in an algebraic language like Macsyma/Maxima, Maple, etc. In many of these projects, we had no problem using ASA. This does not excuse a lot of the clumsy writing in ASA, but it does reflect on the difference between code that is just well written but not flexible and robust to maintain. By now, ASA represents a lot of feedback from thousands of users. A major strength of the code is that it has well over 100 tuning OPTIONS, albeit in many case only a few are usually required. This is the nature of sampling algorithms, and I have broken out all such code-specific parameters into a top-level meta-language that is easy for an end-user to handle. Other very good sampling algorithms do not give such robust tuning, and too often do not work on some complex systems for some users just for this reason. This also has added a lot of weight to the code, but since most of these ASA OPTIONS are chosen at pre-compile time, this does not affect the executables in typical use. I have had at least half a dozen exceptional coders start to rewrite the code into another language, e.g., C++, Java, Matlab, etc., but they gave up when faced with integrating all the ASA OPTIONS. (There is no way I could influence them to start or stop such projects.) I think all these OPTIONS are indeed necessary for such a generic code. I very much appreciate your writing to me.”

The OPTIONS are not just a way of compiling in only code that may be needed for systems so it can run efficiently. The OPTIONS provide a clear meta-language for users to understand how to adjust and tune the code for their own needs. Indeed, there are several OPTIONS that provide hooks for users to insert their own generating and acceptance distribution functions. This leads to a transparency of the code to end-users, at the expense of muddling the code for object-oriented coders.

50

4 Adaptive Simulated Annealing

4.4.3 Motivations for Tuning Methodology Nonlinear systems are typically not typical, and so it is difficult if not impossible to give guidelines for ASA defaults similar to what you might expect for “canned” quasi-linear systems. I have tried to prepare the ASA-README to give some guidelines, and if all else fails you could experiment a bit using a logical approach with the SELF OPTIMIZE OPTIONS. I still advise some experimentation that might yield a bit of insight about a particular system. In many case, the best approach is probably a “blend”: Make a guess or two, then fine-tune the guesses with SELF OPTIMIZE in some rather finer range of the parameter(s). The reason this is slow is because ASA does what you expect it to do: It truly samples the space. When SELF OPTIMIZE is turned on, for each call of the top-level ASA parameters selected, the “inner” shell of your system’s parameters are optimized, and this is performed for an optimization of the “outer” top-level shell of ASA parameters. If you find that indeed this is a necessary and valuable approach to your problem, then one possible short cut might be to turn on Quenching for the outer shell. The ASA proof of statistical convergence to a global optimal point gives sufficient, not necessary, conditions. This still is a pretty strong statement since one can only importance-sample a large space in a finite time. Note that some spaces would easily require CPU times much greater than the lifetime of the universe to sample all points. If you “tucked away” a “pathological” singular optimal point in an otherwise “smooth” space, indeed ASA might have to run “forever.” If the problem isn’t quite so pathological, you might have to slow down the annealing, to permit ASA to spend more time at each scale to investigate the finer scales; then, you would have to explore some other OPTIONS. This could be required if your problem looks different at different scales, for then you can often get trapped in local optima, and thus ASA could fail just as any other “greedy” quasi-Newton algorithm. Because of its exponential annealing schedule, ASA does converge at the end stages of runs quite well, so if you start with your setup akin to this stage, you will search for a very long time (possibly beyond your machine’s precision to generate temperatures) to get out. Or, if you start with too broad a search, you will spin your wheels at first before settling down to explore multiple local optima. ASA has demonstrated many times that it is more efficient and gets the global point better than other importance-sampling techniques, but this still can require “tuning” some ASA OPTIONS. E.g., as mentioned in the ASA-README, a quasiNewton algorithm should be much more efficient than ASA for a parabolic system.

4.4.4 Some Rough But Useful Guidelines Here are some crude guidelines that typically have been useful to tune many systems. At least ASA has a formal proof of convergence to the global minimum of your system. However, no sampling proof is general enough for all systems to guarantee this will take place within your lifetime. This is where the true power of ASA

4.4 Tuning Guidelines

51

comes into play as the code provides many tuning OPTIONS, most which can be applied adaptively at any time in the run, to give you tools to tune your system to provide reasonably efficient optimizations. Depending on your system, this may be easy or hard, possibly taxing anyone’s intuitive and analytic capabilities. In general, respect the optimization process as a simulation in parameter space. The behavior of a system in this space typically is quite different from the system defined by other variables in the system. • Three Stages of Optimization - It is useful to think of the optimization process as having three main stages: initial, middle and end. In the initial stage you want to be sure that ASA is jumping around a lot, visiting all regions of the parameter space within the bounds you have set. In the end stage you want to be sure that the cost function is in the region of the global minimum, and that the cost function as well as the parameter values are being honed to as many significant figures as required. The middle stage typically can require the most tuning, to be sure it smoothly takes the optimization from the initial to the end stage, permitting plenty of excursions to regularly sample alternative regions/scales of the parameter space. • Tuning Information - Keep ASA PRINT MORE set to TRUE during the tuning process to gather information in asa out whenever a new accepted state is encountered. If you have ASA PIPE and/or ASA PIPE FILE set to TRUE, additional information (in relatively larger files) is gathered especially for purposes of graphing key information during the run. Graphical aids can be indispensable for gaining some intuition about your system. If ASA SAVE OPT is set to TRUE then you have the ability to restart runs from intermediate accepted states, without having to reproduce a lot of the original run each time you wish to adaptively change some OPTIONS after a given number of accepted or generated states. • Parameter Temperatures - As discussed above in the section ParameterTemperature Scales, the temperature schedule is determined by T0i , ci , ki , Qi , and D. The default is to have all these the same for each parameter temperature. Note that the sensitivity of the default parameter distributions to the parameter temperatures is logarithmic. Therefore, middle stage temperatures of 10E-6 or 10E-7 still permit very large excursions from the last local minima to visit new generated states. Typically (of course depending on your system), values of 10E10 are appropriate for the end stage of optimization. It is advisable to start by changing the ci to get a reasonable temperature schedule throughout the run. If it becomes difficult to do this across the 3 stages, work with the Qi QUENCH PARAMETERS as these provide different sensitivities at different stages. Generally, it is convenient to use the ci to tune the middle stage, then add in Qi modifications for the end stage. As long as the sum Qi ≤ 1, then the sampling proof is intact. However, once you are sure of the region of the global minima, it can be convenient to turn on actual quenching wherein sum Qi > 1. Turning on Reanneal Parameters can be very useful for some systems to adaptively adjust the temperatures to different scales of the system.

52

4 Adaptive Simulated Annealing

• Cost Temperature - Note that the sensitivity of the default cost distribution to the cost temperatures is exponential. In general, you would like to see the cost temperatures throughout the run be on the scale of the difference between the best and last generated states, where the last generated state in the run is at the last local minima from which new states are explored. Therefore, pay careful attention to these values. Note that the last generated state is set to the most recently accepted state, and if the recently accepted state also is the current best state then the last generated state will be so reported. Therefore, this sensitivity to the last generated state works best during parts of the run where the code is sampling alternate multiple minima. The default is to baseline the cost temperature scale to the default parameter temperature scale, using Cost Parameter Scale Ratio (default = 1). It is advisable to first tune your parameter temperature schedule using Temperature Ratio Scale, then to tune your cost temperature schedule using Cost Parameter Scale Ratio. If it becomes difficult to do this across the 3 stages, work with the Q QUENCH COST as this provides a different sensitivity at a different stage. Generally, it is convenient to use the c scale via Cost Parameter Scale Ratio to tune the middle stage, then add in Q modifications for the end stage. Turning on Reanneal Cost can be very useful for some systems to adaptively adjust the temperature to different scales of the system. • Large Parameter Dimensions - As the number of parameter dimensions D increases, you may see that your temperatures are changing more than you would like with respect to D. The default is to keep the parameter exponents of the ki summed to 1 with each exponent set to 1/D. The effective scale of the default exponential decay of the temperatures is proportional to ck−Q/D , so smaller D gives smaller decay rates for the same values of c, k and Q. Modifications to this behavior of the parameter and cost temperatures are easily made by altering the Qi and Q, resp., as Qi , Q and D enter the code as Qi /D and Q/D, resp. The scales c are set as c = - log (Temperature Ratio Scale) exp (-log (Temperature Anneal Scale) (Q/D). Therefore, the sensitivity of c to D can be controlled by modifying Temperature Anneal Scale or Q.

4.4.5 Quenching If you have a large parameter space, and if a “smart” quasi-local optimization code won’t work for you, then any true global optimization code will be faced with the “curse of dimensionality”. I.e., global optimization algorithms must sample the entire space, and even an efficient code like ASA must do this. As mentioned in the ASA-README, there are some features to explore that might work for your system.

4.4 Tuning Guidelines

53

SQ techniques like genetic algorithms (GA) obviously are important and are crucial to solving many systems in time periods much shorter than might be obtained by standard SA. In ASA, if annealing is forsaken, and Quenching turned on, voiding the proof of sampling, remarkable increases of speed can be obtained, apparently sometimes even greater than other “greedy” algorithms. In large D space, this can be especially useful if the parameters are relatively independent of each other, by noting that the arguments of the exponential temperature schedules are proportional to kQ/D . Then, you might do better thinking of changing Q/D in fractional moves, instead of only small deviations of Q from 1. For example, in http://www.ingber.com/asa92 saga.pdf, along with 5 GA test problems from the UCSD GA archive, another harder problem (the ASA TEST problem that comes with the ASA code) was used. As reported in http://www.ingber.com/asa93 sapvt.pdf, Quenching was applied to this harder problem. The resulting SQ code was shown to speed up the search by as much as a factor of 86 (without even attempting to see if this could be increased further with more extreme quenching). In the asa examples.txt file, even more dramatic efficiencies were obtained. This is a simple change of one number in the code, turning it into a variant of SQ, and is not equivalent to tuning any of the other many ASA options, e.g., like SELF OPTIMIZE, USER COST SCHEDULE, etc. Note that SQ will not suffice for all systems; several users of ASA reported that Quenching did not find the global optimal point that was otherwise be found using the correct SA algorithm. As mentioned in the ASA-README, note that you also can use the Quenching OPTIONS quite differently, to slow down the annealing process by setting User Quench Param Scale to values less than 1. This can be useful in problems where the global optimal point is at a quite different scale from other local optima, masking its presence. This likely might be most useful for low dimensional problems where the CPU time incurred by slower annealing might not be a major consideration. Once you decide you can quench, there are many more alternative algorithms you might wish to choose for your system, e.g., creating a hybrid global-local adaptive quenching search algorithm, e.g., using USER REANNEAL PARAMETERS. Note that just using the quenching OPTIONS provided with ASA can be quite powerful, as demonstrated in the asa examples.txt file.

4.4.6 Options for Large Spaces For very large parameter-space dimensions, the following guide is useful if you desire to speed up the search: Pre-Compile Options: add USER REANNEAL PARAMETERS=TRUE add USER COST SCHEDULE=TRUE

54

4 Adaptive Simulated Annealing add ASA PRINT INTERMED=FALSE SMALL FLOAT may have to be decreased set QUENCH PARAMETERS to TRUE [negates SA sampling if Q > 1] set QUENCH COST to TRUE Perhaps set QUENCH PARAMETERS SCALE and QUENCH COST SCALE to FALSE

Program Options: set Curvature 0 to TRUE decrease Temperature Ratio Scale increase Cost Parameter Scale Ratio increase Maximum Cost Repeat decrease Acceptance Frequency Modulus decrease Generated Frequency Modulus

If the parameter space dimension, D, is huge, e.g., 256x256=65536, then the exponential of the generating or acceptance index to the 1/D power hardly changes over even a few million cycles. True annealing in such huge spaces can become prohibitively slow as the temperatures will hardly be diminished over these cycles. This “curse of dimensionality” will face any algorithm seeking to explore an unknown space. Then, the QUENCH PARAMETERS and QUENCH COST OPTIONS should be tried. However, note that slowing down annealing sometimes can speed up the search by avoiding spending too much time in some local optimal regions.

4.4.7 Shunting to Local Codes I have always maintained in e-mails and in VFSR/ASA publications since 1987, that SA techniques are best suited for approaching complex systems for which little or no information is available. When the range of a global optima is discovered, indeed it may be best to then turn to another algorithm. I have done this myself in several papers, shunting over to a quasi-local search, the Broyden-Fletcher-GoldfarbShanno (BFGS) algorithm, to “polish” off the last 2 or 3 decimals of precision, after I had determined just what final level of precision was acceptable. In the problems where I shunted to BFGS, I simply used something the value of Cost Precision or Limit Acceptances (which were pretty well correlated in some problems) to decide when to shunt over. (I got terrible results if I shunted over too quickly.) However, that was before the days I added OPTIONS like USER COST SCHEDULE and USER ACCEPTANCE TEST, and if and when I redo some of those calculations I will first experiment adaptively using these to account for different behaviors of my systems at different scales. When FITLOC is set to TRUE, three modified simplex subroutines, not requiring derivatives of cost functions, become active to perform a local fit after leaving asa ().

4.4 Tuning Guidelines

55

4.4.8 Judging Importance-Sampling If the cost function is plotted simply as a function of decreasing temperature(s), often the parameter space does appear to be continually sampled in such a plot, but the plot is misleading. That is, there really is importance sampling taking place, and the proof of this is to do a log-log plot of the cost function versus the number of generated states. Then you can see that if the temperature schedule is not enforced you will have a poor search, if quenching is turned on you will get a faster search (though you may miss the global minimum), etc. You can test these effects using quenching and “reverse quenching” (slowing down the annealing); it likely would be helpful to set: QUENCH COST and QUENCH PARAMETERS to TRUE QUENCH PARAMETERS SCALE and QUENCH COST SCALE to FALSE perhaps NO PARAM TEMP TEST and NO COST TEMP TEST to TRUE The point is that the ASA distribution is very fat-tailed, and the effective widths of the parameters being searched change very slowly with decreasing parameter temperatures; the trade-off is that the parameter temperatures may decrease exponentially and still obey the sampling proof. Thus, the experience is that ASA finds global minimum when other sampling techniques fail, and it typically finds the global minimum faster than other sampling techniques as well. Furthermore, the independence of cost and parameter temperatures permits additional tuning of ASA in many difficult problems. While the decreasing parameter temperatures change the way the parameter states are generated, the decreasing cost temperature changes the way the generated states are accepted. The sensitivity to the acceptance criteria to the cost temperature schedule can be very important in many systems. An examination of a few runs using ASA PRINT MORE set to TRUE can reveal premature holding onto local minimum or not enough holding time, etc., requiring tuning of some ASA OPTIONS.

4.4.9 User References Collaborators and I have published some papers in several disciplines that have used or expanded the use of ASA [1, 3, 10, 11, 12, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38]. The file http://www.ingber.com/asa papers.html contains a short list of users who have sent me their papers using ASA. Many other users also have had to list ASA as a tool since it was used in the patents. That file also gives URLs to search patent filings for the use of ASA. The results reveal its use in many disciplines and companies.

56

4 Adaptive Simulated Annealing

4.5 Adaptive OPTIONS 4.5.1 VFSR The first VFSR code [9] added adaptive options by reannealing, i.e., increasing rather than decreasing, the temperature schedules for parameters and the cost function, to enable easier passage through multi-dimensional spaces en route to finding global optima. Of several such OPTIONS, most effective on many systems are Temperature Ratio Scale, Cost Parameter Scale Ratio, and Temperature Anneal Scale.

4.5.2 ASA FUZZY One of the authors of this book contributed ASA FUZZY code to ASA, to help guide QUENCHing OPTIONS to make ASA more efficient for several kinds of problems [44, 45, 46]. Often, ASA FUZZY turns on QUENCHing > 1, violating the proof of ASA. For many systems, this speeding up of the sampling process can be a welcome efficiency, but in some systems global minima may be missed. An active research program is to make ASA FUZZY more adaptive to decreasing as well as increasing QUENCHing.

4.6 Multiple Systems Many times hard problems present themselves as multiple systems to be optimized or sampled. Experience shows that all criteria are not always best considered by lumping them all into one cost function, even with some typical methods as Pareto sampling, but rather good judgment should be applied to multiple stages of preprocessing and post-processing when performing such optimization or sampling.

4.6.1 SELF OPTIMIZE The SELF OPTIMIZE OPTIONS was an early OPTIONS to use ASA itself to optimize parameters used for a particular problem using ASA. A few ASA TEMPLATEs that come with the code give examples of using SELF OPTIMIZE. SELF OPTIMIZE is not particularly useful as the CPU time is the cross product of the outer-shell using SELF OPTIMIZE and the inner-shell optimizing the selected problem for each generated state from SELF OPTIMIZE.

4.6 Multiple Systems

57

SELF OPTIMIZE is a recursive algorithm, which may be useful as a guide to sample or optimize other recursive systems. At least, it is demonstrated that ASA is ready for such systems.

4.6.2 ASA PARALLEL For many hard problems, most CPU resources are spent on the cost function calculations, not the overhead of running ASA per se. This knowledge plus the nature of the fat-tailed ASA distribution, which typically gives rise to a high generated state to acceptance state ratio, gave rise to the opportunity to insert hooks for parallel code within ASA, essentially running many generated states in parallel, and then checking for the best acceptance state. The concept was originally tested on a Connection Machine circa 1990, then in the 1994 National Science Foundation Parallelizing ASA and PATHINT Project (PAPP) mentioned above. It is known to have been used in several industrial settings, including chip design.

4.6.3 TRD Example of Multiple Systems The file http://www.ingber.com/asa examples.txt gives several kinds of use for ASA. An interesting example is in a trading code, Trading in Risk Dimensions (TRD) [34]. TRD provides examples of both recursive and sequential use of ASA. There are three levels of optimization/sampling: The section @@OPTIONAL DATA PTR and MULTI MIN in http://www.ingber.com/asa examples.txt gives details and explicit code used in some past versions to demonstrate how this is set up in ASA. A parameterized trading-rule outer-shell uses the global optimization code Adaptive Simulated Annealing (ASA) to fit parameters of the trading system, e.g., trading rules and trading indicators, to historical data. This is necessary during a Training phase with in-sample data. A simple fitting algorithm, sometimes requiring ASA, is used for an inner-shell fit of incoming market data to real-world probability distributions. The cost function is typically a simple parameterized exponential distribution representing observed fat-tailed distribution. A risk-management middle-shell develops portfolio level distributions of copula transformed multivariate distributions (with constituent markets possessing typically different marginal distributions in returns space), generated by Monte Carlo samplings. This The copula code essentially transforms different real-world market distributions into a common multivariate Gaussian space where it makes sense to calculate correlations. There are inverse transformations to come back to individual

58

4 Adaptive Simulated Annealing

distributions as needed for some trading indicators. ASA is used to importancesample weightings (contract sizes) of these markets. Together with the outer-shell optimization, both the middle-shell portfolio sampling and the inner-shell market distribution fits are processed in Training of insample data, Testing of out-of-sample data, e.g., using walk-forward scripts, and during Real-Time trading of incoming market data. This means that during Training, there are recursive uses of ASA: For example, for each generated state of tradingrule and trading-indicator parameters in the outer-shell cost function, ASA is used for both middle-shell and inner-shell optimizations and sampling. During Testing and Real-Time, after the Training stage has determined a set of best (or sets of good parameters to be post-processed using different technical or fundamental criteria by a different ASA cost function, e.g., during walk forwards), the outer-shell parameters the middle-shell and inner-shell cost functions are run sequentially using their cost functions. ASA can process these multiple cost functions, using a top-level function to set the OPTIONAL DATA PTR OPTIONS to information required to set up each level of optimization. ASA gives ASA TEMPLATEs in asa usr.c to process all these OPTIONS: If the Pre-Compile Option OPTIONAL DATA PTR is set to TRUE, an additional Program Option pointer, Asa Data Ptr, becomes available to define an array, of type OPTIONAL PTR TYPE defined by the user, which can be used to pass arbitrary arrays or structures to the user module from the asa module. This information communicates with the asa module, and memory must be allocated for it in the user module. For example, struct DATA might contain an array data[10] to be used in the cost function. Asa Data Dim Ptr might have a value 2. Set OPTIONAL PTR TYPE to DATA. Then, data[3] in struct Asa Data Ptr[1] could be set and accessed as Asa Data Ptr[1].data[3] in the cost function. For example, your main program that calls asa main() would have developed a struct SelectedType *SelectedPointer, and you can call asa main (SelectedPointer, ...). In asa usr asa.h, you would have OPTIONAL PTR TYPE set to SelectedType. In asa usr.c (and asa usr.h) you would develop asa main (OPTIONAL PTR TYPE *OptionalPointer, ...) and, close to the appropriate ASA TEMPLATE, you would set Asa Data Ptr to OptionalPointer. See the ASA TEMPLATE in asa usr.c. I realize this may sound complex, but with the example provided in http://www.ingber.com/asa examples.txt all this work is fairly easy to implement.

4.7 Conclusion A sampling of theory, practical considerations, and experience gained from many users over many years, has produced the current ASA code. If you are “lucky” then a simple entry into the code, e.g., just using the asa opt file to control some OPTIONS, may do very well for you. However, to keep the ASA code robust for many classes of hard problems, there are many OPTIONS available to properly tune your system to provide a valuable optimization or sampling algorithm.

References

59

References 1. Atiya, A., Parlos, A., Ingber, L.: A reinforcement learning method based on adaptive simulated annealing. In: Proceedings International Midwest Symposium on Circuits and Systems (MWCAS). IEEE CAS, Egypt (2003), http://www.ingber.com/asa03_reinforce.pdf 2. Binder, K., Stauffer, D.: A simple introduction to monte carlo simulations and some specialized topics. In: Binder, K. (ed.) Applications of the Monte Carlo Method in Statistical Physics, pp. 1–36. Springer, Berlin (1985) 3. Bowman, M., Ingber, L.: Canonical momenta of nonlinear combat. In: Proceedings of the 1997 Simulation Multi-Conference, Atlanta, GA, April 6-10, Society for Computer Simulation, San Diego, CA (1997), http://www.ingber.com/combat97_cmi.pdf 4. Cerny, V.: A thermodynamical approach to the travelling salesman problem: An efficient simulation algorithm. Tech. Rep. Report, Comenius University, Bratislava, Czechoslovakia (1982) 5. Charnes, A., Wolfe, M.: Extended pincus theorems and convergence of simulated annealing. International Journal Systems Science 20(8), 1521–1533 (1989) 6. Collins, N., Egelese, R., Golden, B.: Simulated annealing — an annotated bibliography. American Journal Mathematical Management Science 8(3,4), 209–307 (1988) 7. Gelfand, S.: Analysis of simulated annealing type algorithms. Tech. Rep. Ph.D. Thesis, MIT, Cambridge, MA (1987) 8. Geman, S., Geman, D.: Stochastic relaxation, gibbs distribution and the bayesian restoration in images. IEEE Transactions Pattern Analysis Machine Intelligence 6(6), 721–741 (1984) 9. Ingber, L.: Very fast simulated re-annealing. Mathematical Computer Modelling 12(8), 967–973 (1989), http://www.ingber.com/asa89_vfsr.pdf 10. Ingber, L.: Statistical mechanical aids to calculating term structure models. Physical Review A 42(12), 7057–7064 (1990), http://www.ingber.com/markets90_interest.pdf 11. Ingber, L.: Statistical mechanics of neocortical interactions: A scaling paradigm applied to electroencephalography. Physical Review A 44(6), 4017–4060 (1991), http://www.ingber.com/smni91_eeg.pdf 12. Ingber, L.: Generic mesoscopic neural networks based on statistical mechanics of neocortical interactions. Physical Review A 45(4), R2183–R2186 (1992), http://www.ingber.com/smni92_mnn.pdf 13. Ingber, L.: Adaptive simulated annealing (ASA). Tech. Rep. Global optimization C-code, Caltech Alumni Association, Pasadena, CA (1993a), http://www.ingber.com/#ASA-CODE 14. Ingber, L.: Simulated annealing: Practice versus theory. Mathematical Computer Modelling 18(11), 29–57 (1993b), http://www.ingber.com/asa93_sapvt.pdf 15. Ingber, L.: Statistical mechanics of combat and extensions. In: Jones, C. (ed.) Toward a Science of Command, Control, and Communications, pp. 117–149. American Institute of Aeronautics and Astronautics, Washington, DC (1993c); ISBN 1-56347-068-3, http://www.ingber.com/combat93_c3sci.pdf 16. Ingber, L.: Adaptive simulated annealing (ASA): Lessons learned. Control and Cybernetics 25(1), 33–54 (1996a), Invited paper to Control and Cybernetics on Simulated Annealing Applied to Combinatorial Optimization, http://www.ingber.com/asa96_lessons.pdf

60

4 Adaptive Simulated Annealing

17. Ingber, L.: Statistical mechanics of nonlinear nonequilibrium financial markets: Applications to optimized trading. Mathematical Computer Modelling 23(7), 101–121 (1996b), http://www.ingber.com/markets96_trading.pdf 18. Ingber, L.: Canonical momenta indicators of financial markets and neocortical EEG. In: Amari, S.I., Xu, L., King, I., Leung, K.S. (eds.) Progress in Neural Information Processing, pp. 777–784. Springer, New York (1996c); Invited paper to the 1996 International Conference on Neural Information Processing (ICONIP 1996), Hong Kong, September 24-27 (1996); ISBN 981 3083-05-0, http://www.ingber.com/markets96_momenta.pdf 19. Ingber, L.: Statistical mechanics of neocortical interactions: Applications of canonical momenta indicators to electroencephalography. Physical Review E 55(4), 4578–4593 (1997), http://www.ingber.com/smni97_cmi.pdf 20. Ingber, L.: Data mining and knowledge discovery via statistical mechanics in nonlinear stochastic systems. Mathematical Computer Modelling 27(3), 9–31 (1998a), http://www.ingber.com/path98_datamining.pdf 21. Ingber, L.: Statistical mechanics of neocortical interactions: Training and testing canonical momenta indicators of EEG. Mathematical Computer Modelling 27(3), 33–64 (1998b), http://www.ingber.com/smni98_cmi_test.pdf 22. Ingber, L.: Adaptive simulated annealing (ASA) and path-integral (PATHINT) algorithms: Generic tools for complex systems. Tech. Rep. ASA-PATHINT Lecture Plates, Lester Ingber Research, Chicago, IL (2001a); Invited talk U Calgary, Canada (April 2001), http://www.ingber.com/asa01 lecture.pdfandasa01 lecture.html 23. Ingber, L.: Statistical mechanics of combat (SMC): Mathematical comparison of computer models to exercise data. Tech. Rep. SMC Lecture Plates, Lester Ingber Research, Chicago, IL (2001b), http://www.ingber.com/combat01 lecture.pdfandcombat01 lecture.html 24. Ingber, L.: Statistical mechanics of financial markets (SMFM): Applications to trading indicators and options. In: Tech. Rep. SMFM Lecture Plates, Lester Ingber Research, Chicago, IL (2001c); Invited talk U Calgary, Canada (April 2001); Invited talk U Florida, Gainesville (April 2002); Invited talk Tulane U, New Orleans (January 2003), http://www.ingber.com/markets01_lecture.pdf, http://www.ingber.com/markets01_lecture.html 25. Ingber, L.: Statistical mechanics of neocortical interactions (SMNI): Multiple scales of short-term memory and EEG phenomena. Tech. Rep. SMNI Lecture Plates, Lester Ingber Research, Chicago, IL (2001d); Invited talk U Calgary, Canada (April 2001), http://www.ingber.com/smni01 lecture.pdfandsmni01 lecture.html 26. Ingber, L.: Trading in risk dimensions (TRD). Tech. Rep. Report 2005: TRD, Lester Ingber Research, Ashland, OR (2005), http://www.ingber.com/markets05_trd.pdf 27. Ingber, L.: Ideas by statistical mechanics (ISM). Tech. Rep. Report 2006: ISM, Lester Ingber Research, Ashland, OR (2006), http://www.ingber.com/smni06_ism.pdf 28. Ingber, L.: Ideas by statistical mechanics (ISM). Journal Integrated Systems Design and Process Science 11(3), 31–54 (2007a); Special Issue: Biologically Inspired Computing

References

61

29. Ingber, L.: Real options for project schedules (ROPS). Tech. Rep. Report 2007:ROPS, Lester Ingber Research, Ashland, OR (2007b), http://www.ingber.com/markets07_rops.pdf 30. Ingber, L.: AI and ideas by statistical mechanics (ISM). In: Rabu˜nal, J., Dorado, J., Pazos, A. (eds.) Encyclopedia of Artificial Intelligence, pp. 58–64. Information Science Reference, New York (2008a); ISBN 978-1-59904-849-9 31. Ingber, L.: Statistical mechanics of neocortical interactions (SMNI): Testing theories with multiple imaging data. NeuroQuantology Journal 6(2), 97–104 (2008b), http://www.ingber.com/smni08_tt.pdf 32. Ingber, L.: Statistical mechanics of neocortical interactions: Nonlinear columnar electroencephalography. NeuroQuantology Journal 7(4), 500–529 (2009), http://www.ingber.com/smni09_nonlin_column_eeg.pdf 33. Ingber, L.: Real options for project schedules (ROPS). International Journal of Science, Technology & Management 2(2), 15–20 (2010a); Invited paper 34. Ingber, L.: Trading in risk dimensions. In: Gregoriou, G. (ed.) The Handbook of Trading: Strategies for Navigating and Profiting from Currency, Bond, and Stock Markets, pp. 287–300. McGraw-Hill, New York (2010b) 35. Ingber, L., Fujio, H., Wehner, M.: Mathematical comparison of combat computer models to exercise data. Mathematical Computer Modelling 15(1), 65–90 (1991), http://www.ingber.com/combat91_data.pdf 36. Ingber, L., Mondescu, R.: Optimization of trading physics models of markets. IEEE Trans. Neural Networks 12(4), 776–790 (2001); Invited paper for special issue on Neural Networks in Financial Engineering, http://www.ingber.com/markets01_optim_trading.pdf 37. Ingber, L., Mondescu, R.: Automated internet trading based on optimized physics models of markets. In: Howlett, R., Ichalkaranje, N., Jain, L., Tonfoni, G. (eds.) Intelligent Internet-Based Information Processing Systems, pp. 305–356. World Scientific, Singapore (2003), http://www.ingber.com/markets03_automated.pdf 38. Ingber, L., Sworder, D.: Statistical mechanics of combat with human factors. Mathematical Computer Modelling 15(11), 99–127 (1991), http://www.ingber.com/combat91_human.pdf 39. Johnson, D., Aragon, C., McGeoch, L., Schevon, C.: Optimization by simulated annealing: An experimental evaluation (parts 1 and 2). Tech. Rep. Report, AT&T Bell Laboratories, Murray Hill, NJ (1987) 40. Kirkpatrick, S., Gelatt Jr., C., Vecchi, M.: Optimization by simulated annealing. Science 220(4598), 671–680 (1983) 41. Ma, S.K.: Statistical Mechanics. World Scientific, Singapore (1985) 42. Mathews, J., Walker, R.: Mathematical Methods of Physics, 2nd ed. Benjamin, New York (1970) 43. Metropolis, N., Rosenbluth, A., Rosenbluth, M., Teller, A., Teller, E.: Equation of state calculations by fast computing machines. Journal of Chemical Physics 21(6), 1087–1092 (1953) 44. Oliveira Jr., H.: Fuzzy control of stochastic global optimization algorithms and very fast simulated reannealing. Tech. Rep. Report, [email protected], Rio de Janeiro, Brazil (2001), http://www.optimization-online.org/DB_FILE/2003/11/779.pdf 45. Oliveira Jr., H., Petraglia, A., Petraglia, M.: Frequency domain fir filter design using fuzzy adaptive simulated annealing. Circuits, Systems, and Signal Processiing 28(6), 899–911(2009); DOI: 10.1007/s00034-009-9128-1

62

4 Adaptive Simulated Annealing

46. Oliveira Jr.,H., Petraglia, H., Petraglia, A.: Frequency domain fir filter design using fuzzy adaptive simulated annealing. In: 7th International Symposium on Signal Processing and Information Technology, Proceedings of ISSPIT, Cairo, vol. 1, pp. 899–903 (2007) 47. Pincus, M.: A monte carlo method for the approximate solution of certain types of constrained optimization problems. Operations Research 18, 1225–1228 (1970) 48. Szu, H., Hartley, R.: Fast simulated annealing. Physics Letters A 122(3-4), 157–162 (1987) 49. van Laarhoven Jr., P., Aarts, E.: Simulated Annealing: Theory and Applications. D. Reidel, Dordrecht (1987)

Chapter 5

Unconstrained Optimization

Abstract. In this chapter we will start to focus on operational aspects of (Fuzzy) ASA and how to use it for practical purposes. Several detailed examples of unconstrained problems will be presented, aiming to show the readers how to benefit from using the method in their own tasks. Initially, an overview of Fuzzy ASA will be shown and, after that, many cost functions will be defined and minimized with ASA and Fuzzy ASA, being the results compared in each case.

5.1 Fuzzy ASA Conventional simulated annealing typically presents low speed of convergence, being the performance presented by most implementations not very encouraging [2, 3, 5, 6, 9]. Despite this, there are ways to overcome the limitations of original annealing schemes, such as ASA, introduced in the previous chapter, that is a sophisticated and really effective global optimization method [4]. ASA method is particularly well-suited to applications involving hard cost functions, like those related to neuro-fuzzy systems design and neural network training, for example, taking into account its superior performance and simplicity. Adaptive simulated annealing brings us the benefits of being publicly available, adjustable and wellmaintained. Besides, ASA shows itself as an alternative to other global optimization tools, according to the published benchmarks, that demonstrate its good quality. Unfortunately, stochastic global optimization algorithms share a few inconvenient characteristics like, for example, large periods of poor improvement in their way to a global extremum. In SA implementations, that is mainly due to the cooling schedule, whose speed is limited by the characteristics of probability density functions (PDFs) used to generate new candidate points. In this fashion, if we choose to employ the so-called Boltzmann annealing, the temperature has to be lowered at a maximum rate of T (k) = T (0)/ln(k). In the case of fast annealing, the schedule becomes T (k) = T (0)/k, if assurance of convergence with probability 1 is to be H.A. e Oliveira Jr. et al.: Stochastic Global Optimization & Its Applications, ISRL 35, pp. 63–93. c Springer-Verlag Berlin Heidelberg 2012 springerlink.com 

64

5 Unconstrained Optimization

maintained, resulting in a faster schedule. ASA has an even better default scheme, given by Ti (k) = Ti (0)exp(−Ci k1/D )

(5.1)

where Ci is an user-defined constant, thanks to its improved generating distribution. The subscripts indicate independent evolution of temperatures for each parameter dimension. In addition, it is possible for the ASA user to take advantage of Simulated Quenching, resulting in Ti (k) = Ti (0)exp(−Ci kQi /D )

(5.2)

where Qi is the quenching parameter corresponding to dimension i. Setting quenching parameters to values greater than 1 results in greater speed, but the convergence to a global optimum is no longer assured (please, see [4]). Such a procedure could be used for very high-dimensional parameter spaces, whenever computational resources are scarce. Despite (or because) all those features, there is much tuning to be done, from the user’s point of view. In the sequel we will describe Fuzzy ASA, a well-succeeded approach to accelerate ASA algorithm using a simple Mamdani fuzzy controller that dynamically adjusts certain user parameters related to quenching. It will be shown that, by increasing the algorithm’s perception of slow convergence, it is possible to speed it up significantly and to reduce the task of parameter tuning. That is done without changes in the original ASA code. Simulated annealing algorithms are based on the ideas introduced by N. Metropolis and others [7], usually known as Monte Carlo importance-sampling techniques. The method uses three fundamental components, that have large impact on the final implementation: • A probability density function g(.), used in the generation of new candidate points; • A PDF a(.), used in the acceptance/rejection of new generated points; • A schedule T(.), that determines how the temperatures will vary during the execution of the algorithm, that is, their dynamic profile. The basic approach is to generate a starting point, chosen according to some criteria, and to set the initial temperature so that the space state could be sufficiently explored. After that, new points are iteratively generated according to the generating PDF g (.) and probabilistically accepted or rejected, as dictated by PDF a(.). If acceptance occurs, the candidate point becomes the current base point. During the run, temperatures are lowered and this reduces the probability of acceptance of new generated points with higher cost values than that of the current point. However, there is a non-zero probability of going uphill, giving the opportunity to escape from local minima.

5.1 Fuzzy ASA

65

As said before, ASA is based upon the concept of simulated annealing, possessing in addition a lot of convenient features. Among them we cite: • Re-annealing – it is the dynamical re-scaling of parameter temperatures, adapting generating distributions for each dimension according to the sensitivities shown in a given search direction. In a few words, if the cost function doesn’t show significant variations when we vary one given parameter, it may be worth to extend the search interval for that particular dimension and vice-versa; • Quenching facilities – as we cited before, ASA code has several user-settable parameters related to quenching that allow us to improve the convergence speed. So, it is possible to tailor parameter and cost temperatures evolution by changing selected quenching factors in an easy and clean manner; • High level of parameterization – ASA is coded in such a way that we can alter virtually any building block without significant effort. This way, it is possible to change generation/acceptance processes behavior, stopping criteria, starting point generation, log file detail level, and so on. ASA was designed to find global minima belonging to a given compact subset of n-dimensional Euclidean space. It generates points component-wise, according to xi+1 = xi + Δ xi , with Δ xi = yi (Bi − Ai ), [Ai , Bi ] = interval corresponding to i-th dimension, yi ∈ [−1, 1] is given by yi = sign(ui − 1/2)Ti[(1 + 1/Ti)|2ui −1| − 1], where ui ∈ [0, 1] is generated by means of the uniform distribution, and Ti = present temperature relative to dimension i. The compactness of the search space is not a severe limitation in practice and, in the absence of prior information about the possible location of global minima, it suffices to choose a sufficiently large hyper-rectangular domain. As we said before, by using the so-called simulated quenching it is possible to improve the efficiency of the annealing schedule, taking the risk of reaching a nonglobal minimum. In certain cases, however, we have no choice, as is the case for domains with very large number of dimensions, for instance. To solve this problem, a fuzzy controller was designed. The approach is as folllows: we consider ASA as a MISO (Multiple Input Single Output) dynamical system and close the loop, by sampling ASA’s output (best cost function value found so far) and acting on its inputs (a subset of settable parameters related to quenching) according to a fuzzy law (quenching controller) that does nothing more than emulate human reasoning about the underlying process. So, by the use of an intelligent controller we can control the speed of the temperature schedule, in addition to being able to take evasive actions in case of premature convergence.

66

5 Unconstrained Optimization

There are two main obstacles to get there: • How the sampled outputs (cost function values) could tell us the present status of the progressing run ? • How do we change ASA inputs in order to leave undesirable situations (permanence near non-global minima / slow progress) ? The first question was handled thanks to the concept of sub-energy function, used in the TRUST method [1]. The sub-energy function is given by S(x, x0 ) = log(1/[1 + exp(−( f (x) − f (x0)) − a)])

(5.3)

where a ∈ R and x0 is the current base point. The base point is defined as the best minimum point found so far. In this way, the function S behaves qualitatively like the original f when the search visits better points than the current minimum, and tends to be flat in worse points. Thus, it is possible to assess when the search is located above, near or under the current minimum point by inspecting the values assumed by the sub-energy function. Such a detection process results in approximate conclusions like The search is NEAR the current minimum or The search is VERY FAR from the current minimum leading naturally to a fuzzy modeling opportunity. The second question above is related to the consequent parts of the fuzzy rule base, in which we have to place corrective actions to keep the search progressing toward the global minimum. That was done by varying quenching degrees for generating and acceptance PDFs. The implementation used individual quenching factors for each dimension and one cost quenching factor. The fuzzy controller’s rule-base could contain rules like • IF AveSub IS NEAR ZERO THEN increase Quenching • IF AveSub IS NEAR current minimum THEN increase Quenching • IF StdDevSub IS ZERO THEN decrease Quenching where AveSub is a linguistic variable corresponding to the crisp average of last 100 sub-energy values and StdDevSub is a linguistic variable corresponding to the crisp standard deviation of last 100 sub-energy values. Having outlined the structure of the whole scheme, it is time to start showing you how to get some practical results from ASA and Fuzzy ASA.

5.2 Unconstrained (or Rectangular Constrained) Optimization Examples

67

5.2 Unconstrained (or Rectangular Constrained) Optimization Examples In this section we’ll present several examples of unconstrained optimization using ASA and Fuzzy ASA, that is ASA with the fuzzy controller turned on. We have chosen well-known, difficult functions that will allow us to assess ASA’s ability as a true global optimization tool. It is our intention to make a step by step presentation so that readers could figure out how to use it fluently. It is worth to mention that although we are referring to the problems as unconstrained ones, in practical situations it is usually necessary to define numerical bounds for each dimension in order to allow programs to generate points inside a well-defined region, namely, a hyper-rectangle. On the other hand, that bounded region can be as large as we wish and, in practice, it doesn’t represent a true limitation. To make our task easier, we’ll use, among other things, C++ language source code implementing cost functions according to a certain protocol designed specifically for working with standard ASA code. Although it has been tailored to work with a particular GUI application, it is possible to use it inside any program, provided the user follows the corresponding protocol. As we said, a graphical user interface was constructed as a wrapper for (Fuzzy) ASA and in the hope of saving development efforts when using it, for example, in optimization-based design tasks. The ’global minimization engine’ is encapsulated in the main program itself and, in normal conditions, does not need to be recompiled frequently. The cost functions themselves reside inside DLLs (Dynamic Linking Libraries or Dynamically Linked Libraries) and must obey a very simple protocol for talking to the central program, as the code examples will demonstrate. There are 5 functions that need to be inside the DLL, as follows: • BOOL DllMain(HINSTANCE hInst,DWORD dwReason,LPVOID Reserved) • extern ’C’ double declspec(dllexport) (NAME OF THE COST FUNCTION) ( int NoOfDimensions, // Number of elements of the following vector double *Vector, // Argument void *Custom // User defined information ) • extern ’C’ char * declspec(dllexport) CostFunctionName(void) • extern ’C’ unsigned int declspec(dllexport) NumberOfParametersCF(void) • extern ’C’ int declspec(dllexport) LimitsOfDomain(double *Left,double *Right) The first one is standard and necessary for interfacing to the host operating system when using DLLs. The second function is called by a given application whenever it needs to have the value of the cost function calculated at a given point - in our case, this function is directly called by ASA code during the minimization process. The third function returns the name of the routine implementing the cost function, so that the caller could know ’who’ to call when loading, in run-time, the corresponding code. The fourth function returns the dimension of cost function’s domain

68

5 Unconstrained Optimization

and the fifth one returns arrays containing lower and upper bounds for the values of each coordinate of the vector argument, so defining the frontiers of the bounding (hyper)rectangle. From the caller side, we’ll show a few code snippets that will illustrate how the above code is used. Firstly, the desired DLL has to be loaded and information about the cost function and its domain obtained, as shows the code below

void *dllhandle ; #define DEFAULTDLLNAME DefaultDLL #define ROUTINENAME CostFunctionName #define ROUTINENUMBER NumberOfParametersCF #define ROUTINEDOMAIN LimitsOfDomain if (DLLName->getText().length() > 0) strcpy(DLLTOLOAD, DLLName->getText().text()) ; else strcpy(DLLTOLOAD, DEFAULTDLLNAME) ; GlobalHandleToDLL = dllhandle = fxdllOpen( (const FXchar *) DLLTOLOAD ); CFNAME = (CostFunctionName) fxdllSymbol(dllhandle,(const FXchar *) ROUTINENAME); CFPARNO = (NoOfParametersCF) fxdllSymbol(dllhandle,(const FXchar *) ROUTINENUMBER); DOMAINLIMITS = (LimitsOfHyperRectangle) fxdllSymbol(dllhandle,(const FXchar *) ROUTINEDOMAIN); NoParame = CFPARNO(); strcpy(SIMBOL,CFNAME()); LeftLimits = (double *) calloc (NoParame, sizeof (double)); RightLimits = (double *) calloc (NoParame, sizeof (double)); TemplateCF = (CostFunction) fxdllSymbol(dllhandle ,(const FXchar *) SIMBOL);

After that, it is possible to call indirectly the cost function as shown below, that is, ASA’s algorithm will call MyCostFunction, and the latter will call the actual objective function, resident in the DLL - variable TemplateCF contains a pointer to it.

5.2 Unconstrained (or Rectangular Constrained) Optimization Examples

69

double MyCostFunction (double *x, double *parameterlowerbound, double *parameterupperbound, double *costtangents, double *costcurvature, ALLOC INT * parameterdimension, int *parameterintreal, int *costflag, int *exitcode, USER DEFINES * USEROPTIONS) { double presentvalue=0; int i; char ACCEPTSPOINT[10]; ACCEPTSPOINT[0] = (char) 1 ; presentvalue = TemplateCF(*parameterdimension,x, (void *) ACCEPTSPOINT); if (ACCEPTSPOINT[0]) *costflag = TRUE; else *costflag = FALSE; .............................. ........................ .................. return presentvalue; }

Finally, ASA’s routine could be activated by code like shown below

costvalue = asa (::MyCostFunction, ::randflt, randseed, costparameters, parameterlowerbound, parameterupperbound, costtangents, costcurvature, parameterdimension, parameterintreal, costflag, exitcode, USEROPTIONS);

As can be seen in the ASA documentation, function asa receives pointers to a cost function and a (pseudo) random number generator, a pointer to a random seed, an array of doubles holding the set of starting parameters which should satisfy any constraints or boundary conditions (upon return from the asa procedure, the array will contain the best set of parameters found by asa to minimize the users cost function), the bounds of search space, two arrays of doubles for returning first and second derivatives of the cost function with respect to its parameters, the Euclidean dimension of the domain, an integer array indicating which parameters are real and which

70

5 Unconstrained Optimization

ones are integers, a flag to handle constraints, a pointer for returning terminating codes and a pointer to a structure containing ASA parameters. Now, we will proceed by showing applications of ASA and Fuzzy ASA to several cost functions. To better evaluate the results, comparative graphs showing cost function values against number of function evaluations are included in the examples. All ASA parameters were kept fixed during the experiments, the more relevant being listed below

USER USER USER USER USER USER USER USER USER USER USER USER USER USER USER USER USER USER USER

OPTIONS-> Limit Invalid Generated States = 100000000000 OPTIONS->Accepted To Generated Ratio = 1E-4 OPTIONS->Cost Precision = 1.0E-18 OPTIONS->Maximum Cost Repeat = 0 OPTIONS->Number Cost Samples = 25 OPTIONS->Temperature Ratio Scale = 1E-10 OPTIONS->Cost Parameter Scale Ratio = 1.0 OPTIONS->Temperature Anneal Scale = 10000 OPTIONS->Include Integer Parameters = FALSE OPTIONS->User Initial Parameters = TRUE OPTIONS->Sequential Parameters = -1 OPTIONS->Initial Parameter Temperature = 100 OPTIONS->Acceptance Frequency Modulus = 100 OPTIONS->Generated Frequency Modulus = 10000 OPTIONS->Reanneal Cost = 1 OPTIONS->Reanneal Parameters = TRUE OPTIONS->Delta X = 1E-15 OPTIONS->User Tangents = FALSE OPTIONS->Curvature 0 = TRUE

5.2.1 Rastrigin Function Δ

n

f (x) = 10n + ∑ (x2i − 10cos(2π xi)) i=1

where n is the domain dimension, x = (x1 , x2 , ..., xn ) ∈ Rn and −5.12 ≤ xi ≤ 5.12, i ∈ {1, ..., n}. This function has a global minimizer at x∗ = (0, 0, ..., 0) with value 0. In this example, we have chosen n = 30.

(5.4)

5.2 Unconstrained (or Rectangular Constrained) Optimization Examples

71

Fig. 5.1 Rastrigin function minimization

As can be seen from Fig. 5.1, Fuzzy ASA converged in less than 100,000 cost function evaluations and standard ASA is still in its way to the global minimum after 500,000 function evaluations. In this case, Fuzzy ASA has shown itself more efficient. Below, we present the real DLL code implementing the computation of the cost function and the auxiliary routines that interface to the global minimization apparatus.

#include <stdio.h> #include <windows.h> #include <math.h> char NameWithUnderscore[] = ” Rastrigin”; #define DIMENSION 30

72

5 Unconstrained Optimization

BOOL DllMain(HINSTANCE hInst,DWORD dwReason, LPVOID Reserved) { switch (dwReason) { case DLL PROCESS ATTACH: break; case DLL PROCESS DETACH: break; case DLL THREAD ATTACH: break; case DLL THREAD DETACH: break; } return TRUE; } extern ”C” double declspec(dllexport) Rastrigin( int NoOfDimensions, // No of elements of the following vector double *Vector, void *Custom // User defined information ) { int i,k; double PresentValue = 0 , xd ; k = 10; for (i=0 ; i < NoOfDimensions ; i++) { xd = Vector[i]; PresentValue += xd * xd - k * cos (2 * M PI * xd); } PresentValue = PresentValue + NoOfDimensions * k; return PresentValue ; } extern ”C” char * declspec(dllexport) CostFunctionName(void) { return (char *) NameWithUnderscore; } extern ”C” unsigned int declspec(dllexport) NumberOfParametersCF(void) {

5.2 Unconstrained (or Rectangular Constrained) Optimization Examples

73

return DIMENSION; } extern ”C” int declspec(dllexport) LimitsOfDomain(double *Left,double *Right) { for (int i=0; i ¡ DIMENSION ; i++) { Left[i] = -5.12 ; Right[i] = 5.12 ; } return DIMENSION; }

5.2.2 Schwefel Function n  Δ f (x) = 418.9829n + ∑ −xi sin( |xi |)

(5.5)

n  Δ f (x) = 418.9829n + ∑ xi sin( |xi |)

(5.6)

i=1

or

i=1

where n is the domain dimension, x = (x1 , x2 , ..., xn ) ∈ Rn and −500 ≤ xi ≤ 500, i ∈ {1, ..., n}. Functions (5.5) and (5.6) have global minimizers at x∗ = (420.968746, 420.968746, ..., 420.968746) and x∗ = (−420.968746, −420.968746, ..., −420.968746), respectively, with value 0. In this example, we have chosen n = 10. As can be seen from Fig. 5.2, when minimizing (5.6), standard ASA converged in less than 20,000 cost function evaluations and Fuzzy ASA is very far from convergence (if any) after 500,000 function evaluations. In this case, standard ASA has shown itself very effective. Below, we present the DLL code implementing the computation of only one of the two cost functions ( (5.6) ) and just a few auxiliary routines that interface to

74

5 Unconstrained Optimization

Fig. 5.2 Schwefel function minimization

the global minimization apparatus, taking into account that CostFunctionName and NumberOfParametersCF have a fixed format.

#include <stdio.h> #include <windows.h> #include <math.h> char NameWithUnderscore[] = ” Schwefel”; #define DIMENSION 10 BOOL DllMain(HINSTANCE hInst,DWORD dwReason, LPVOID Reserved) { switch (dwReason) {

5.2 Unconstrained (or Rectangular Constrained) Optimization Examples

case DLL case DLL case DLL case DLL }

PROCESS ATTACH: break; PROCESS DETACH: break; THREAD ATTACH: break; THREAD DETACH: break;

return TRUE; } extern ”C” double declspec(dllexport) Schwefel( int NoOfDim, double *Vector, void *Custom ) { int i; double valoratu = 0 , xd ; for (i=0 ; i < NoOfDim ; i++) { xd = Vector[i]; valoratu += xd*sin( sqrt( fabs( xd ) ) ) ; } valoratu = valoratu + 418.9829*NoOfDim ; return valoratu ; } extern ”C” int declspec(dllexport) LimitsOfDomain(double *Left, double *Right) { for (int i=0; i ¡ DIMENSION ; i++) { Left[i] = -500 ; Right[i] = 500 ; } return DIMENSION; }

75

76

5 Unconstrained Optimization

5.2.3 Ackley Function  Δ

f (x) = −20exp(−0.2

∑ni=1 x2i ∑n cos(2π xi ) ) − exp( i=1 ) + 20 + e n n

(5.7)

where n is the domain dimension, x = (x1 , x2 , ..., xn ) ∈ Rn and −32.768 ≤ xi ≤ 32.768, i ∈ {1, ..., n}. This function has a global minimizer at x∗ = (0, 0, ..., 0) with value 0. In this example, we have chosen n = 10.

Fig. 5.3 Ackley function minimization

As Fig. 5.3 displays, standard ASA and Fuzzy ASA converged in less than 25,000 cost function evaluations and showed almost identical performance in this case both were effective in finding the global minimum. Below, we present the DLL code implementing the computation of the cost function and only a few auxiliary routines that interface to the global minimization ”engine”.

5.2 Unconstrained (or Rectangular Constrained) Optimization Examples

#include <windows.h> #include <math.h> char NameWithUnderscore[] = ” Ackley”; #define DIMENSION 10 BOOL DllMain(HINSTANCE hInst,DWORD dwReason, LPVOID Reserved) { switch (dwReason) { case DLL PROCESS ATTACH: break; case DLL PROCESS DETACH: break; case DLL THREAD ATTACH: break; case DLL THREAD DETACH: break; } return TRUE; } extern ”C” double declspec(dllexport) Ackley( int NoOfDimensions, double *Vector, void *Custom ) { int i; double valoratu = 0 , xd , E NEPER=exp(1) ; double sum1 , sum2 ; sum1 = sum2 = 0; for (i=0 ; i < NoOfDimensions ; i++) { xd = Vector[i]; sum1=sum1+xd*xd; sum2=sum2+cos(2*M PI*xd); }

77

78

5 Unconstrained Optimization

valoratu = -20*exp(-0.2*sqrt(sum1/NoOfDimensions))exp(sum2/NoOfDimensions)+20+E NEPER ; return valoratu ; } extern ”C” int declspec(dllexport) LimitsOfDomain(double *Left, double *Right) { for (int i=0; i < DIMENSION ; i++) { Left[i] = -32.768 ; Right[i] = 32.768 ; } return DIMENSION; }

5.2.4 Krishnakumar Function Δ

f (x) =

n−1

∑ (sin(xi + xi+1) + sin(2xixi+1/3))

(5.8)

i=1

where n is the domain dimension, x = (x1 , x2 , ..., xn ) ∈ Rn and −5 ≤ xi ≤ 5, i ∈ {1, ..., n} This function has a global minimum of −2(n − 1). In this example, we have chosen n = 15. As Fig. 5.4 shows, standard ASA spent approximately three times more cost function evaluations than Fuzzy ASA to get to the global minimum. Fuzzy ASA was much more efficient in this example. Below, we present the DLL code that calculates the cost function.

#include <windows.h>

5.2 Unconstrained (or Rectangular Constrained) Optimization Examples

Fig. 5.4 Krishnakumar function minimization

#include <math.h> char NameWithUnderscore[] = ” KrishnaKumar”; #define DIMENSION 15 BOOL DllMain(HINSTANCE hInst,DWORD dwReason, LPVOID Reserved) { switch (dwReason) { case DLL PROCESS ATTACH: break; case DLL PROCESS DETACH: break; case DLL THREAD ATTACH: break; case DLL THREAD DETACH: break;

79

80

5 Unconstrained Optimization

} return TRUE; } extern ”C” double declspec(dllexport) KrishnaKumar( int NoOfDimensions, double *Vector, void *Custom // User defined data ) { int i; double valoratu = 0 ; for (i=0 ; i < NoOfDimensions-1 ; i++) valoratu += sin( Vector[i]+Vector[i+1] ) + sin( 2 * Vector[i] * Vector[i + 1] / 3 ); return valoratu ; } extern ”C” int declspec(dllexport) LimitsOfDomain(double *Left, double *Right) { for (int i=0; i < DIMENSION ; i++) { Left[i] = -5 ; Right[i] = 5 ; } return DIMENSION; }

5.2.5 Rosenbrock Function Δ

f (x) =

n−1

∑ 100(xi+1 − x2i )2 + (1 − xi)2

i=1

(5.9)

5.2 Unconstrained (or Rectangular Constrained) Optimization Examples

81

where n is the domain dimension, x = (x1 , x2 , ..., xn ) ∈ Rn and −2.048 ≤ xi ≤ 2.048, i ∈ {1, ..., n} This function has a global minimizer at x∗ = (1, 1, ..., 1) with value 0. In this example, we have chosen n = 20.

Fig. 5.5 Rosenbrock function minimization

As Fig. 5.5 shows, standard ASA spent approximately five times more cost function evaluations than Fuzzy ASA to get to the global minimum. Fuzzy ASA was much more efficient in this example as well. Below, we present the DLL code that calculates the cost function.

#include <windows.h> #include <math.h>

82

5 Unconstrained Optimization

char NameWithUnderscore[] = ” Rosenbrock”; #define DIMENSION 20 BOOL DllMain(HINSTANCE hInst,DWORD dwReason, LPVOID Reserved) { switch (dwReason) { case DLL PROCESS ATTACH: break; case DLL PROCESS DETACH: break; case DLL THREAD ATTACH: break; case DLL THREAD DETACH: break; } return TRUE; } extern ”C” double declspec(dllexport) Rosenbrock( int NoOfDimensions, double *Vetor, void *Custom ) { int i; double valoratu = 0 , aux , aux1 ; for (i=0 ; i < NoOfDimensions-1 ; i++) { aux = Vetor[i]*Vetor[i]-Vetor[i+1]; aux1 = 1-Vetor[i] ; valoratu += 100*aux*aux + aux1*aux1 ; } return valoratu ; } extern ”C” int declspec(dllexport) LimitsOfDomain(double *Left, double *Right) {

5.2 Unconstrained (or Rectangular Constrained) Optimization Examples

83

for (int i=0; i < DIMENSION ; i++) { Left[i] = -2.048 ; Right[i] = 2.048 ; } return DIMENSION; }

5.2.6 Griewangk Function Δ

f (x) =

n 1 n 2 xi x − cos( √ ) + 1 ∑ ∏ i 4000 i=1 i i=1

(5.10)

where n is the domain dimension, x = (x1 , x2 , ..., xn ) ∈ Rn and −600 ≤ xi ≤ 600, i ∈ {1, ..., n} This function has a global minimizer at x∗ = (0, 0, ..., 0) with value 0. In this example, we have chosen n = 100. As Fig. 5.6 shows, standard ASA stagnated and Fuzzy ASA converged before 150,000 cost function evaluations. Below, we present the DLL code that calculates the cost function. #include <windows.h> #include <math.h> char NameWithUnderscore[] = ” Griewangk”; #define DIMENSION 100 BOOL DllMain(HINSTANCE hInst,DWORD dwReason, LPVOID Reserved) { switch (dwReason)

84

5 Unconstrained Optimization

Fig. 5.6 Griewangk function minimization

{ case DLL case DLL case DLL case DLL }

PROCESS ATTACH: break; PROCESS DETACH: break; THREAD ATTACH: break; THREAD DETACH: break;

return TRUE; } extern ”C” double declspec(dllexport) Griewangk( int NoOfDimensions, double *Vector, void *Custom ) {

5.2 Unconstrained (or Rectangular Constrained) Optimization Examples

int i; double valoratu = 0 , xd ; double sum1 = 0 , prod2 = 1.0 ; for (i=0 ; i < NoOfDimensions ; i++) { xd = Vector[i]; sum1=sum1+xd*xd; prod2 = prod2*cos(xd/sqrt(i+1)); } valoratu = (1.0/4000.0)*sum1 - prod2 + 1 ; return valoratu ; } extern ”C” int declspec(dllexport) LimitsOfDomain(double *Left, double *Right) { for (int i=0; i < DIMENSION ; i++) { Left[i] = -600 ; Right[i] = 600 ; } return DIMENSION; }

5.2.7 Special Function 1 This function appears in example 1, ref. [8], p. 183.

85

86

5 Unconstrained Optimization Δ

f (x) = f1 (x) + f2 (x) + f3 (x) Δ

(5.11)

n

f1 (x) = ∑ ix2i

(5.12)

f2 (x) = ∑ (xi−1 + 5sin(xi) + x2i+1 )2 )

(5.13)

i=1

Δ

n

i=1

Δ

n

f3 (x) = ∑ ln2 (1 + |isin2 xi−1 + 2xi + 3xi+1|) i=1

Δ

x 0 = xn Δ

xn+1 = x1 where n is the domain dimension, x = (x1 , x2 , ..., xn ) ∈ Rn and −10 ≤ xi ≤ 10, i ∈ {1, ..., n} This function has a global minimizer at x∗ = (0, 0, ..., 0) with value 0. In this example, we have chosen n = 100.

Fig. 5.7 Special function 1 minimization

(5.14)

5.2 Unconstrained (or Rectangular Constrained) Optimization Examples

87

As Fig. 5.7 shows, standard ASA is progressing towards the global minimum after spending 500,000 function evaluations and Fuzzy ASA converged before 100,000 cost function evaluations. So, Fuzzy ASA was more efficient in this case. Below, we present the DLL code that calculates the cost function.

#include <windows.h> #include <math.h> char NameWithUnderscore[] = ” Pinter1”; #define DIMENSION 100 BOOL DllMain(HINSTANCE hInst,DWORD dwReason, LPVOID Reserved) { switch (dwReason) { case DLL PROCESS ATTACH: break; case DLL PROCESS DETACH: break; case DLL THREAD ATTACH: break; case DLL THREAD DETACH: break; } return TRUE; } extern ”C” double declspec(dllexport) Pinter1( int NoOfDimensions, double *x, void *Custom ) { int i; double f1,f2,f3,aux; f1=0; for (i=0;i < NoOfDimensions;i++) f1 += (i+1)*x[i]*x[i]; f2 = pow(x[NoOfDimensions-1]+5*sin(x[0])+x[1]*x[1],2);

88

5 Unconstrained Optimization

aux = 1])+x[0]*x[0],2) ; f2 = f2 + aux;

pow(x[NoOfDimensions-2]+5*sin(x[NoOfDimensions-

for (i=1 ; i < NoOfDimensions-1 ; i++) f2 += pow(x[i-1]+5*sin(x[i])+x[i+1]*x[i+1],2); f3 = log(1+fabs(pow(x[NoOfDimensions-1],2)+2*x[0]+3*x[1])); f3 = f3*f3 ; aux = log(1+fabs(NoOfDimensions*pow(x[NoOfDimensions2],2)+2*x[NoOfDimensions-1]+3*x[0])); aux = aux*aux; f3 = f3+aux; for (i=1 ; i < NoOfDimensions-1 ; i++) { aux = log(1+fabs((1+i)*pow(x[i-1],2)+2*x[i]+3*x[i+1])); f3 += aux*aux ; } aux=f1+f2+f3; return aux ; } extern ”C” int declspec(dllexport) LimitsOfDomain(double *Left, double *Right) { for (int i=0; i < DIMENSION ; i++) { Left[i] = -10 ; Right[i] = 10 ; } return DIMENSION; }

5.2.8 Special Function 2 This function appears in example 2, ref. [8], p. 187.

5.2 Unconstrained (or Rectangular Constrained) Optimization Examples Δ

f (x) = f1 (x) + f2 (x) + f3 (x)

89

(5.15)

n

Δ

f 1 (x) = ∑ i.x2i

(5.16)

f2 (x) = ∑ i.sin2 (xi−1 sin(xi ) − xi + sin(xi+1 ))

(5.17)

f3 (x) = ∑ i.ln(1 + i(x2i−1 − 2xi + 3xi+1 − cos(xi ) + 1)2)

(5.18)

i=1

Δ

n

i=1

Δ

n

i=1

Δ

x 0 = xn Δ

xn+1 = x1 where n is the domain dimension, x = (x1 , x2 , ..., xn ) ∈ Rn and −10 ≤ xi ≤ 10, i ∈ {1, ..., n} This function has a global minimizer at x∗ = (0, 0, ..., 0) with value 0. In this example, we have chosen n = 50.

Fig. 5.8 Special function 2 minimization

90

5 Unconstrained Optimization

As Fig. 5.8 shows, standard ASA is progressing towards the global minimum after spending 5,000,000 function evaluations and Fuzzy ASA converged before 600,000 cost function evaluations. So, Fuzzy ASA has been more efficient in this case too. Below, we present the DLL code that calculates the cost function.

#include <windows.h> #include <math.h> char NameWithUnderscore[] = ” Pinter2”; #define DIMENSION 50 BOOL DllMain(HINSTANCE hInst,DWORD dwReason, LPVOID Reserved) { switch (dwReason) { case DLL PROCESS ATTACH: break; case DLL PROCESS DETACH: break; case DLL THREAD ATTACH: break; case DLL THREAD DETACH: break; } return TRUE; } extern ”C” double declspec(dllexport) Pinter2( int NoOfDimensions, double *x, void *Custom ) { int i; double f1,f2,f3,aux; f1=0; for (i=0;i < NoOfDimensions;i++) f1 += (i+1)*x[i]*x[i];

5.2 Unconstrained (or Rectangular Constrained) Optimization Examples

f2 = pow(sin(x[NoOfDimensions-1]*sin(x[0])-x[0]+sin(x[1])),2); aux = NoOfDimensions*pow(sin(x[NoOfDimensions2]*sin(x[NoOfDimensions-1])-x[NoOfDimensions-1]+sin(x[0])),2) ; f2 = f2 + aux; for (i=1 ; i < NoOfDimensions-1 ; i++) f2 += (i+1)*pow(sin( x[i-1]*sin(x[i])-x[i]+sin(x[i+1]) ),2); f3 = log(1+ pow(pow(x[NoOfDimensions-1],2)-2*x[0]+3*x[1]cos(x[0])+1,2)); aux = NoOfDimensions*log(1+ NoOfDimensions*pow(pow(x [NoOfDimensions-2],2)-2*x[NoOfDimensions-1]+3*x[0]cos(x[NoOfDimensions-1])+1,2)); f3 = f3+aux; for (i=1 ; i < NoOfDimensions-1 ; i++) { f3 += (i+1)*log(1+ (i+1)*pow(pow(x[i-1],2)-2*x[i]+3*x[i+1]cos(x[i])+1,2)); ; } aux=f1+f2+f3; return aux ; } extern ”C” int declspec(dllexport) LimitsOfDomain(double *Left, double *Right) { for (int i=0; i < DIMENSION ; i++) { Left[i] = -10 ; Right[i] = 10 ; } return DIMENSION; }

91

92

5 Unconstrained Optimization

5.3 Conclusion In this chapter we presented several examples of (unconstrained) minimization performed by the ASA method and its fuzzy controlled version (Fuzzy ASA). In all cases ASA and/or Fuzzy ASA found the correct solutions for the functions to be optimized and it is possible to observe that the algorithm reaches the solution within a satisfactory number of function evaluations. We can also note that in many cases Fuzzy ASA performed better than standard ASA because it is able to activate the quenching mechanism, making convergence to the global minimum easier. However, in a few cases, ”pure” ASA reached the global minimum faster, suggesting that the conjunction of the two modes of operation could be more effective than one or another in isolation. The suggested approach is suitable to a very large class of functions and shows itself adequate to handle multimodal landscapes, what makes it ideal to application in real world problems. As intended, (Fuzzy) ASA was shown to be effective in difficult global minimization tasks and a good alternative to global optimization problems.

References

93

References 1. Barhen, J., Protopopescu, V., Reister, D.: TRUST: A Deterministic Algorithm for Global Optimization. Science 276, 1094–1097 (1997) 2. Corana, A., Marchesi, M., Martini, C., Ridella, S.: Minimizing multimodal functions of continuous variables with the simulated annealing algorithm. ACM Trans. Mathematical Software 13, 262–280 (1987) 3. Dr´eo, J., P´etrowski, A., Siarry, P., Taillard, E.: Metaheuristics for Hard Optimization Methods and Case Studies. Simulated Annealing, Tabu Search, Evolutionary and Genetic Algorithms, Ant Colonies,.. Springer, Berlin (2006) 4. Ingber, L.: Adaptive simulated annealing (ASA): Lessons learned. Control and Cybernetics 25(1), 33–54 (1996) 5. Kirkpatrick, S., Gelatt Jr., C.D., Vecchi, M.P.: Optimization by simulated annealing. Science 220, 671–680 (1983) 6. van Laarhoven, P.J.M., Aarts, E.H.L.: Simulated Annealing: Theory and Applications. D. Reidel, Dordrecht (1987) 7. Metropolis, N., Rosenbluth, A., Rosenbluth, M., Teller, A., Teller, E.: Equation of state calculations by fast computing machines. Journal of Chemical Physics 21(6), 1087–1092 (1953) 8. Pint´er, J.D.: Global Optimization in Action. Kluwer Academic Publishers, Dordrecht (1996) 9. Weise, T.: Global Optimization Algorithms - Theory and Application, http://www.it-weise.de/ (accessed July 11, 2011)

Chapter 6

Constrained Optimization

Abstract. The Adaptive Simulated Annealing method (ASA) has been successful in numerous areas of knowledge, ranging from optimization-based engineering design to statistical estimation, and can be very useful in constrained global optimization tasks as well. That is what we will see in this chapter by means of a series of examples containing difficult problems. In this fashion, ASA and Fuzzy ASA can be considered as good alternatives to well-established paradigms, like ABC, DE, PSO and GA, for instance, in CGOPs.

6.1 Introduction Constrained global optimization of numerical functions is very important in many practical applications [3], especially in real world, technical and scientific problems, considering that they always involve limitations, mainly physical and financial ones. So, all efforts spent at discovering effective methods for doing such tasks are welcome. In particular, well-succeeded methods in unconstrained global optimization tasks could be excellent candidates for testing in constrained ones. Adaptive Simulated Annealing is well suited to many applications in several areas of knowledge thanks to its good performance and robustness, showing itself as an alternative to other stochastic paradigms [1, 2, 15, 17], according to the published benchmarks that demonstrate its effectiveness in the unconstrained global optimization realm [6, 10, 12, 13, 14]. Although it has been used recently in complex constrained engineering design problems [11], it would be interesting to measure its performance when applied to a generic test suite. In this chapter, our aim is to demonstrate that ASA and its fuzzy-controlled version (Fuzzy ASA) are effective in approximating the solution of difficult constrained global optimization problems and, besides, can overcome in some cases well-established and accepted methods. To do that, we will show several examples offering different degrees of difficulty and for which there are several published results, using many other methods. Another important issue is related to the way we have used ASA to do constrained optimization, considering H.A. e Oliveira Jr. et al.: Stochastic Global Optimization & Its Applications, ISRL 35, pp. 95–115. c Springer-Verlag Berlin Heidelberg 2012 springerlink.com 

96

6 Constrained Optimization

that it is (or has been) mainly directed to unconstrained optimization, although its standard, public implementation has built-in mechanisms to deal with constraints. One such feature is the mandatory specification of the hyper-rectangle inside which the candidate point generation is to take place, and the other one is the possibility of rejecting synthesized points from inside the cost function, defined by the user. The tests have used penalty functions in different ways, dynamic and static, not taking advantage of ”native” ASA devices, aiming to assess its constraint handling ability without those tools. To carry out the intended evaluation we have chosen some problems in the test suite proposed in [8], taking into account the substantial number of previous results, using diverse, good quality methods. A constrained optimization problem can be stated as follows [4, 5, 7, 9, 16, 18]: Find a parameter vector x∗ ∈ Rn that minimizes a given objective function f : S → R, S ⊂ Rn , subject to the following restrictions: x∗ = (x1 ∗ , x2 ∗ , x3 ∗ , ..., xn ∗ ) ∈ Rn ai ≤ xi ∗ ≤ bi , i = 1, ..., n

(6.1)

g j (x∗ ) ≤ 0, j = 1, ..., q hk (x∗ ) = 0, k = q + 1, ..., m The search space, S, defined as a n-dimensional rectangle in Rn , is the domain of the objective function f. The ranges of the variables are defined by their lower and upper bounds, ai and bi , and a feasible region F ⊂ S is defined by means of a set of m additional constraints. At a given point x ∈ F, constraints g j satisfying g j (x) = 0 are termed active constraints at x. To find approximate solutions for the system (6.1) we have used penalty functions to transform each constrained problem in an unconstrained one, as described by the following algorithm: Step 1. Transform the original problem into an unconstrained one, by defining a new objective function, extended by 2 penalty terms, one corresponding to inequality constraints (G(x)) and one to equality ones (H(x)). This is done as usual: if a given point violates a certain restriction, we penalize it by adding to the original cost function a term proportional to the violation (we assume a minimization problem here). This new function is given by: F(x) = f (x) + α (G(x) + H(x)), where

(6.2)

q

G(x) = ∑ max(0, gi (x)) i=1

(6.3)

6.2 Constrained Global Optimization Using ASA and Fuzzy ASA

and H(x) =

m

∑

h2i (x)

97

(6.4)

i=q+1

Step 2. Design an adequate α factor by deciding whether it will be a simple constant or a dynamic factor; Step 3. Submit the extended function F(·) in (6.2) to the (Fuzzy) ASA algorithm. As is well-known, the reasoning underlying penalty methods is that unfeasible points should be repelled, and a sensible way to do this is to alter the original cost function in such a way that whenever candidate points are located in undesired regions, the values of the penalized function should drive the search toward feasible parts of the domain. As a result, the cost function remains unaltered inside feasible regions, otherwise the returned values tend to force algorithms to leave unfeasible ones. In any case, the main task is to effectively minimize the function f within a given precision, assumed adequate for the problem at hand. Of course, the quality of the algorithm is strongly based on the minimization power of the used global optimization algorithm (Fuzzy ASA, in our method), particularly in its ability to handle arbitrarily complex landscapes of highly nonlinear cost functions. In the tests, several penalty approaches were employed and the ASA algorithm was used with and without its fuzzy controller. Also, the α factor was dynamic in some cases and static in others, as pointed out below. As usual, all equality constraints have been, in practice, converted into inequality constraints, |h j | ≤ δ , j = q + 1, ..., m, with δ = 0.0001. In other words, the tolerance for equality constraints is δ .

6.2 Constrained Global Optimization Using ASA and Fuzzy ASA As aforementioned, some functions described in [8] were chosen as our examples and, in the following, we present the results of applying ASA to them. To assess the ability of the ASA method to generalize in constrained global optimization problems, we kept its working parameters fixed during all tests (there is one exception in function G07), just allowing activation or not of the fuzzy controller. The most important ones are shown in Table 6.1. For each tested function, 25 independent global optimization sessions were done using ASA and Fuzzy ASA, and the results are summarized in tables having 5 columns, whose head labels are in the following order: method, minimum value obtained over all tests for that particular method, mean value calculated over all results, respective standard deviation, and mean number of function evaluations along all executions.

98

6 Constrained Optimization

Table 6.1 Parameters of the ASA algorithm. Parameter

Value

Accepted To Generated Ratio

1E-4

Cost Precision

1.0E-18

Temperature Ratio Scale

1E-10

Cost Parameter Scale Ratio

1.0

Temperature Anneal Scale

10000

Initial Parameter Temperature

100

Delta X

1E-15

Detailed definitions of the above parameters can be found in ASA’s manual (www.ingber.com).

6.2.1 Function G01 Minimize 4

4

13

i=1

i=1

i=1

f (x) = 5 ∑ xi − 5 ∑ x2i − ∑ xi

(6.5)

subject to g1 (x) = 2x1 + 2x2 + x10 + x11 − 10 ≤ 0 g2 (x) = 2x1 + 2x3 + x10 + x12 − 10 ≤ 0 g3 (x) = 2x2 + 2x3 + x11 + x12 − 10 ≤ 0 g4 (x) = −8x1 + x10 ≤ 0 g5 (x) = −8x2 + x11 ≤ 0 g6 (x) = −8x3 + x12 ≤ 0

(6.6)

g7 (x) = −2x4 − x5 + x10 ≤ 0 g8 (x) = −2x6 − x7 + x11 ≤ 0 g9 (x) = −2x8 − x9 + x12 ≤ 0 The domain of this function is contained in R13 , and the problem presents 9 linear inequality constraints. Its hyper-rectangle is defined by 0 ≤ xi ≤ 100, for i = 10, 11, 12, and 0 ≤ xi ≤ 1, for the remaining coordinates. The global minimum is at x∗ = (1, 1, 1, 1, 1, 1, 1, 1, 1, 3, 3, 3, 1), where f (x∗ ) = −15. The results are presented in Table 6.2, where we observe that standard ASA performed better than Fuzzy ASA in reaching the global minimum. So, best results

6.2 Constrained Global Optimization Using ASA and Fuzzy ASA

99

were obtained without activating the fuzzy controller of the ASA algorithm. In addition, a dynamic penalty was employed by initializing the α factor value to 1300, and raising it by 1000 each 1000 function evaluations. This approach worked very well in this instance, tending to keep the search inside the feasible region most of the time. Table 6.2 Results for function G01. Method

Best found

Mean

Std. dev.

Mean f.e.

ASA

-15

-15

0

201,092

-7.6396

1.13442

250,000

Fuzzy ASA -8.6462347

Below we show selected parts of the DLL code used to implement the example. Note that the code is illustrative, not optimized and for educational purposes only.

#include <windows.h> #include <math.h> #include <stdio.h> #define NG 9 #define NH 1 #define FREQUENCY 1000 #define MAXIMUM WEIGHT 7000000.0 #define MINIMUM WEIGHT 1300.0 typedef void (WINAPI * PPROC) (double *, double *, double *, double *, int, int, int, int); int Activations ; double PresR = MINIMUM WEIGHT ; char NameWithUnderscore[] = ” FunctionG01”; static PPROC pfcn; static HINSTANCE hLibrary; BOOL DllMain(HINSTANCE hInst,DWORD dwReason, LPVOID Reserved) { switch (dwReason) { case DLL PROCESS ATTACH: hLibrary = LoadLibrary (”fcnsuite BCC.dll”);

100

6 Constrained Optimization

if (hLibrary == NULL) return FALSE; pfcn = (PPROC) GetProcAddress ((HMODULE) hLibrary, ” g01”); if (pfcn == NULL) return FALSE; break; case DLL PROCESS DETACH: FreeLibrary ((HMODULE) hLibrary); break; case DLL THREAD ATTACH: break; case DLL THREAD DETACH: break; } return TRUE; } extern ”C” double declspec(dllexport) FunctionG01( int NoOfDimensions, double *Vector, void *Custom ) { double f , gfunction[NG] , h[NH] , aux ; /* Function pfcn returns values of cost functions and inequality/equality constraints calculated at input vector (Vector) */ /* This function and several other good benchmark routines can be found (by the time of this writing) at http://www3.ntu.edu.sg/home/EPNSugan/ */ pfcn( Vector , &f ,( double * ) gfunction , ( double * ) h , NoOfDimensions , 1 , NG , 0 ); aux = 0; for ( int i=0 ; i < NG ; i++ ) aux += ( gfunction[i] > 0 ? gfunction[i] : 0 ) ; aux = PresR*aux ; Activations++ ;

6.2 Constrained Global Optimization Using ASA and Fuzzy ASA

if ( ! ( Activations % FREQUENCY ) ) { Activations = 0 ; if (PresR < MAXIMUM WEIGHT) PresR += 1000 ; } return ( f + aux ) ; } extern ”C” char * declspec(dllexport) CostFunctionName(void) { return (char *) NameWithUnderscore; } extern ”C” unsigned int declspec(dllexport) NumberOfParametersCF(void) { return 13 ; } extern ”C” int declspec(dllexport) LimitsOfDomain(double *Left, double *Right) { for ( int i=0 ; i < 9 ; i++ ) { Left[i] = 0 ; Right[i] = 1 ; } for ( int i=9 ; i < 12 ; i++ ) { Left[i] = 0 ; Right[i] = 100 ; } Left[12] = 0 ; Right[12] = 1 ; return 13 ; }

101

102

6 Constrained Optimization

6.2.2 Function G02 Minimize f (x) = −

| ∑ni=1 cos4 (xi ) − 2 ∏ni=1 cos2 (xi )|  ∑ni=1 ix2i

(6.7)

subject to n

g1 (x) = 0.75 − ∏ xi ≤ 0

(6.8)

i=1

n

g2 (x) = ∑ xi − 7.5n ≤ 0 i=1

n = 20 The domain of this function is contained in R20 , and the problem presents 1 linear inequality and 1 nonlinear inequality constraints. Its hyper-rectangle is defined by 0 ≤ xi ≤ 10, for i = 1, 2, ..., 20. The best known value for this function is −0.80361910412559 at (3.16246061572185, 3.12833142812967, 3.09479212988791, 3.06145059523469, 3.02792915885555, 2.99382606701730, 2.95866871765285, 2.92184227312450, 0.49482511456933, 0.48835711005490, 0.48231642711865, 0.47664475092742, 0.47129550835493, 0.46623099264167, 0.46142004984199, 0.45683664767217, 0.45245876903267, 0.44826762241853, 0.44424700958760, 0.44038285956317). The results in this case are given in Table 6.3. As in the previous case, best results were obtained for the ASA algorithm without activating its fuzzy controller. Also, a dynamic penalty was employed by initializing the α factor value to 1,300, and raising it by 1,000 each 1,000 function evaluations. This approach worked well in this instance, tending to keep the search inside the feasible region most of the time.

Table 6.3 Results for function G02. Method

Best found

Mean

Std. dev.

Mean f.e.

ASA

-0.803238

-0.793719

0.005

350,000

Fuzzy ASA -0.620802

-0.614665

0.023629

350,000

Below we show selected parts of the DLL code used to implement the example. Please, note that the code is illustrative, not optimized and for educational purposes only.

6.2 Constrained Global Optimization Using ASA and Fuzzy ASA

103

#include <windows.h> #include <math.h> #include <stdio.h> #define NG 2 #define NH 1 #define FREQUENCY 1000 #define MAXIMUM WEIGHT 7000000.0 #define MINIMUM WEIGHT 1300.0 typedef void (WINAPI * PPROC) (double *, double *, double *, double *, int, int, int, int); double PresentR = MINIMUM WEIGHT ; int Activations = 0; char NameWithUnderscore[] = ” FunctionG02”; static PPROC pfcn; static HINSTANCE hLibrary;

BOOL DllMain(HINSTANCE hInst,DWORD dwReason, LPVOID Reserved) { switch (dwReason) { case DLL PROCESS ATTACH: hLibrary = LoadLibrary (”fcnsuite BCC.dll”); if (hLibrary == NULL) return FALSE; pfcn = (PPROC) GetProcAddress ((HMODULE) hLibrary, ” g02”); if (pfcn == NULL) return FALSE; break; case DLL PROCESS DETACH: FreeLibrary ((HMODULE) hLibrary); break; case DLL THREAD ATTACH: break; case DLL THREAD DETACH: break;

104

6 Constrained Optimization

} return TRUE; } extern ”C” double declspec(dllexport) FunctionG02( int NoOfDimensions, double *Vector, void *Custom ) { double f , functiong[NG] , h[NH] , aux ; pfcn( Vector ,&f ,( double * ) functiong , ( double * ) h , NoOfDimensions , 2 , NG , 0 ); aux = 0; for ( int i=0 ; i < NG ; i++ ) aux += ( functiong[i] > 0 ? functiong[i] : 0 ) ; aux = PresentR*aux ; Activations++ ; if ( ! ( Activations % FREQUENCY ) ) { Activations = 0 ; if (PresentR < MAXIMUM WEIGHT) PresentR += 1000 ; } return ( f + aux ) ; } extern ”C” char * declspec(dllexport) CostFunctionName(void) { return (char *) NameWithUnderscore; } extern ”C” unsigned int declspec(dllexport) NumberOfParametersCF(void) { return 20 ; } extern ”C” int declspec(dllexport) LimitsOfDomain(double *Left, double *Right) {

6.2 Constrained Global Optimization Using ASA and Fuzzy ASA

105

for ( int i=0 ; i < 20 ; i++ ) { Left[i] = 0 ; Right[i] = 10 ; } return 20 ; }

6.2.3 Function G03 Minimize n √ f (x) = −( n)n ∏ xi

(6.9)

i=1

subject to n

h1 (x) = ∑ x2i − 1 = 0

(6.10)

i=1

n = 10 The domain of this function is contained in R10 , and the problem presents 1 nonlinear equality constraint. Its hyper-rectangle is defined by 0 ≤ xi ≤ 1, for i = 1, 2, ..., 10. The best value for this function is −1.00050010001 at (10−1/2 , 10−1/2, 10−1/2, 10−1/2 , 10−1/2 , 10−1/2, 10−1/2 , 10−1/2 , 10−1/2, 10−1/2). The results are displayed in Table 6.4, where it can be seen that the performance of the ASA algorithm was satisfactory. In this example the results were almost indifferent with respect to ASA’s fuzzy controller activation, that is, results were very similar in both configurations. A dynamic penalty was employed by initializing the α factor value to 1000, and raising it by 1000 each 1,000 function evaluations. This approach worked well in this instance, tending to keep the search inside the feasible region most of the time.

Table 6.4 Results for function G03. Method

Best found

Mean

Std. dev.

Mean f.e.

ASA

-1

-0.99999

0.000012

350,000

0.00131

350,000

Fuzzy ASA -1.00002147458 -0.9999737

106

6 Constrained Optimization

6.2.4 Function G04 Minimize f (x) = 5.35787547x23 + 0.8356891x1x5 + 37.293239x1 − 40792.141

(6.11)

subject to

g1 (x) = 85.334407 + 0.0056858x2x5 + 0.0006262x1x4 − 0.0022053x3x5 − 92 ≤ 0 g2 (x) = −85.334407 − 0.0056858x2x5 − 0.0006262x1x4 + 0.0022053x3x5 ≤ 0 g3 (x) = 80.51249 + 0.0071317x2x5 + 0.0029955x1x2 + 0.0021813x23 − 110 ≤ 0 g4 (x) = −80.51249 − 0.0071317x2x5 − 0.0029955x1x2 − 0.0021813x23 + 90 ≤ 0 g5 (x) = 9.300961 + 0.0047026x3x5 + 0.0012547x1x3 + 0.0019085x3x4 − 25 ≤ 0 g6 (x) = −9.300961−0.0047026x3x5 −0.0012547x1x3 −0.0019085x3x4 +20 ≤ 0 (6.12) The domain of this function is contained in R5 , and the problem presents 6 nonlinear inequality constraints. Its hyper-rectangle is defined by 78 ≤ x1 ≤ 102, 33 ≤ x2 ≤ 45, 27 ≤ xi ≤ 45, for i = 3, 4, 5. The best known value for this function is −30665.539 at (78, 33, 29.995256025682, 45, 36.775812905788). In Table 6.5 we present the results obtained for this case. It can be observed that that ASA algorithm performed well, approaching optimum value with reasonable precision although not reaching the best known value. The best results achieved by the ASA algorithm occurred without activating its fuzzy controller. A dynamic penalty was employed by initializing the α factor value to 1,000, and raising it by 1,000 each 1,000 iterations. This approach worked well in this instance, tending to keep the search inside the feasible region most of the time. The best result we found was −30665.5385 at (78 , 33.0000001, 29.99525694036198, 44.99999999995895, 36.77581059282999), which is a good approximation to the best known value.

Table 6.5 Results for function G04. Method

Best found

Mean

Std. dev.

Mean f.e.

ASA

-30665.5385

-30664.2342

98.7878

350,000

-30274.77

157.6392

350,000

Fuzzy ASA -30480.8

6.2 Constrained Global Optimization Using ASA and Fuzzy ASA

107

6.2.5 Function G05 Minimize f (x) = 3x1 + 0.000001x31 + 2x2 + (0.000002/3)x32

(6.13)

subject to g1 (x) = −x4 + x3 − 0.55 ≤ 0 g2 (x) = −x3 + x4 − 0.55 ≤ 0 h3 (x) = 1000sin(−x3 − 0.25) + 1000sin(−x4 − 0.25) + 894.8 − x1 = 0

(6.14)

h4 (x) = 1000sin(x3 − 0.25) + 1000sin(x3 − x4 − 0.25) + 894.8 − x2 = 0 h5 (x) = 1000sin(x4 − 0.25) + 1000sin(x4 − x3 − 0.25) + 1294.8 = 0 The domain of this function contained in R4 , and the problem presents 2 linear inequality and 3 nonlinear equality constraints. Its hyper-rectangle is defined by 0 ≤ x1 ≤ 1200, 0 ≤ x2 ≤ 1200, −0.55 ≤ x3 ≤ 0.55, −0.55 ≤ x4 ≤ 0.55.The best known value for this function is 5126.4967140071 at (679.945148297028709, 1026.06697600004691, 0.118876369094410433,−0.39623348521517826). The results are shown in Table 6.6, where we present results for this case and from which it can be noticed that the ASA algorithm performed well, approaching optimum value with reasonable precision, although not reaching the best known value. The best results were obtained without activating its fuzzy controller. Also, an almost static penalty was employed by initializing the α factor value to 1,000, raising it to 2,000 at the beginning, and keeping it fixed untill the end of the whole process. This approach worked well in this instance, tending to keep the search inside the feasible region most of the time. The best result we found was 5126.50452 at (681.6106281329841 , 1024.284088913455 , 0.1176875873806129 , −0.3967969948378958). Table 6.6 Results for function G05. Method

Best found

Mean

Std. dev.

Mean f.e.

ASA

5126.50452

5367.37155

401.66

350,000

5230.89866

70.318

350,000

Fuzzy ASA 5142.91

6.2.6 Function G06 Minimize f (x) = (x1 − 10)3 + (x2 − 20)3

(6.15)

108

6 Constrained Optimization

subject to g1 (x) = −(x1 − 5)2 − (x2 − 5)2 + 100 ≤ 0 g2 (x) = (x1 − 6)2 + (x2 − 5)2 − 82.81 ≤ 0

(6.16)

The domain of this function is contained in R2 , and the problem presents 2 nonlinear inequality constraints. Its hyper-rectangle is defined by 13 ≤ x1 ≤ 100, 0 ≤ x2 ≤ 100. The best known value for this function is −6961.81388 at (14.095, 0.84296). The results are listed in Table 6.7, where it can be noticed that the ASA algorithm performed well, approaching the best known global minimum with reasonable precision and showing small variance through the tests. The best results were obtained independently of ASA’s fuzzy controller activation. A dynamic penalty function was employed by initializing the α factor value to 1,300, and raising it by 1,000 each 1,000 function evaluations. This approach worked well in this instance, tending to keep the search inside the feasible region most of the time. Our best result was −6961.81387 at (14.095 , 0.8429607892). Table 6.7 Results for function G06. Method

Best found

Mean

Std. dev.

Mean f.e.

ASA

-6961.81387

-6961.81276

0.0117

439,427

0.12

363,662

Fuzzy ASA -6961.8116109 -6961.81

6.2.7 Function G07 Minimize f (x) = x21 + x22 + x1 x2 − 14x1 − 16x2 + (x3 − 10)2 + 4(x4 − 5)2 + (x5 − 3)2 +2(x6 − 1)2 + 5x27 + 7(x8 − 11)2 + 2(x9 − 10)2 + (x10 − 7)2 + 45 subject to

(6.17)

6.2 Constrained Global Optimization Using ASA and Fuzzy ASA

109

g1 (x) = −105 + 4x1 + 5x2 − 3x7 + 9x8 ≤ 0 g2 (x) = 10x1 − 8x2 − 17x7 + 2x8 ≤ 0 g3 (x) = −8x1 + 2x2 + 5x9 − 2x10 − 12 ≤ 0 g4 (x) = 3(x1 − 2)2 + 4(x2 − 3)2 + 2x23 − 7x4 − 120 ≤ 0 g5 (x) = 5x21 + 8x2 + (x3 − 6)2 − 2x4 − 40 ≤ 0

(6.18)

g6 (x) = x21 + 2(x2 − 2)2 − 2x1 x2 + 14x5 − 6x6 ≤ 0 g7 (x) = 0.5(x1 − 8)2 + 2(x2 − 4)2 + 3x25 − x6 − 30 ≤ 0 g8 (x) = −3x1 + 6x2 + 12(x9 − 8)2 − 7x10 ≤ 0 The domain of this function is contained in R10 , and the problem presents 3 linear inequality and 5 nonlinear inequality constraints. Its hyper-rectangle is defined by −10 ≤ xi ≤ 10, for i = 1, 2, ..., 10. The best known value for this function is 24.30620906818 at (2.17199634142692, 2.3636830416034, 8.77392573913157, 5.09598443745173, 0.990654756560493, 1.43057392853463, 1.32164415364306, 9.82872576524495, 8.2800915887356, 8.3759266477347). The results are presented in Table 6.8 and ASA algorithm has shown its ability to approximate the best known global minimum with small error. Good results were obtained independently of ASA’s fuzzy controller activation. A static penalty function was employed by keeping the α factor value at 2,000 and setting the ASA parameter “Cost Parameter Scale Ratio” (see Table 6.1) to 10. This approach worked well in this instance, maintaining the search inside the feasible region most of the time. Our best result was 24.406799 at (2.165477268566057, 2.399189018094574, 8.752590658972402, 5.108363718791631, 1.040529752116427, 1.530772264809844, 1.303912972679436, 9.805987101906659, 8.241093653825358, 8.340014082473985). Table 6.8 Results for function G07. Method

Best found

Mean

Std. dev.

Mean f.e.

ASA

24.406799

24.406799

0

350,000

24.85181201

0.14377

350,000

Fuzzy ASA 24.48653709

6.2.8 Function G08 Minimize f (x) = −

sin3 (2π x1)sin(2π x2 ) x31 (x1 + x2 )

(6.19)

110

6 Constrained Optimization

subject to g1 (x) = x21 − x2 + 1 ≤ 0

(6.20)

g2 (x) = 1 − x1 + (x2 − 4)2 ≤ 0

The domain of this function is contained in R2 , and the problem presents 2 nonlinear inequality constraints. Its hyper-rectangle is defined by 0 ≤ xi ≤ 10, for i = 1, 2. The best known value for this function is -0.0958250414180359 at (1.22797135260752599, 4.24537336612274885). In Table 6.9 we present results for this case and it is possible to see that they coincided with the best known one. The results were obtained independently of fuzzy controller activation of the ASA algorithm, that is, ASA and Fuzzy ASA got to the global minimum. A dynamic penalty function was employed by initializing the α factor value to 1,300, and raising it by 1,000 each 1,000 function evaluations. This approach worked well in this instance, tending to keep the search inside the feasible region most of the time.

Table 6.9 Results for function G08. Method

Best found

Mean

Std. dev.

Mean f.e.

ASA

-0.095825

-0.095825

0

350,000

Fuzzy ASA -0.095825

-0.095825

0

350,000

6.2.9 Function G09 Minimize f (x) = (x1 − 10)2 + 5(x2 − 12)2 + x43 + 3(x4 − 11)2 +10x65 + 7x26 + x47 − 4x6 x7 − 10x6 − 8x7

(6.21)

subject to g1 (x) = −127 + 2x21 + 3x42 + x3 + 4x24 + 5x5 ≤ 0 g2 (x) = −282 + 7x1 + 3x2 + 10x23 + x4 − x5 ≤ 0 g3 (x) = −196 + 23x1 + x22 + 6x26 − 8x7 ≤ 0

(6.22)

g4 (x) = 4x21 + x22 − 3x1x2 + 2x23 + 5x6 − 11x7 ≤ 0 The domain of this function is contained in R7 , and the problem presents 4 nonlinear inequality constraints. Its hyper-rectangle is defined by −10 ≤ xi ≤ 10, for i = 1, 2, ..., 7. The best known value for this function is 680.630057374402 at (2.33049935147405174, 1.95137236847114592, -0.477541399510615805,

6.2 Constrained Global Optimization Using ASA and Fuzzy ASA

111

4.36572624923625874, -0.624486959100388983, 1.03813099410962173, 1.5942266780671519). The results are given in Table 6.10, where it can be seen that the ASA algorithm approached the best known global minimum very well, showing small variance over the 30 runs. The best results were obtained without activating ASA’s fuzzy controller. A dynamic penalty function was employed by initializing the α factor value to 1,300, and raising it by 1,000 each 1,000 function evaluations. This approach worked well in this instance, tending to keep the search inside the feasible region most of the time. Our best result was 680.6386 at (2.325847631527188, 1.944020111070723, -0.4928144211316623, 4.386380745855409, -0.6270087900969463, 1.03486140558863, 1.592094482097159).

Table 6.10 Results for function G09. Method

Best found

Mean

Std. dev.

Mean f.e.

ASA

680.6386

680.8967

0.0839

350,000

710.5354

9.31946

350,000

Fuzzy ASA 695.53093

6.2.10 Function G10 Minimize f (x) = x1 + x2 + x3

(6.23)

subject to g1 (x) = −1 + 0.0025(x4 + x6 ) ≤ 0 g2 (x) = −1 + 0.0025(x5 + x7 − x4 ) ≤ 0 g3 (x) = −1 + 0.01(x8 − x5 ) ≤ 0 g4 (x) = −x1 x6 + 833.33252x4 + 100x1 − 83333.333 ≤ 0

(6.24)

g5 (x) = −x2 x7 + 1250x5 + x2 x4 − 1250x4 ≤ 0 g6 (x) = −x3 x8 + 1250000 + x3x5 − 2500x5 ≤ 0 The domain of this function is contained in R8 , and the problem presents 3 linear inequality and 3 nonlinear inequality constraints. Its hyper-rectangle is defined by 100 ≤ x1 ≤ 10000, 1000 ≤ xi ≤ 10000, for i = 2, 3, 10 ≤ xi ≤ 1000, for i = 4, 5, ..., 8. The best known value for this function is 7049.24802052867 at (579.306685017979589, 1359.97067807935605, 5109.97065743133317, 182.01769963061534, 295.601173702746792, 217.982300369384632, 286.41652592786852, 395.601173702746735).

112

6 Constrained Optimization

The results are displayed in Table 6.11, where it can be observed that the best results were obtained by activating ASA’s fuzzy controller. A dynamic penalty function was employed by initializing the α factor value to 1,300, and raising it by 1,000 each 1,000 function evaluations. This approach worked well in this instance, tending to keep the search inside the feasible region most of the time. Our best result was 7049.6881 at (5.765597157646735e+02, 1.327841387106032e+03, 5.145287021394519e+03, 1.817878283352716e+02, 2.941885191442533e+02, 2.182121716602349e+02, 2.875993091910136e+02, 3.941885191442367e+02).

Table 6.11 Results for function G10 Method

Best found

Mean

Std. dev.

Mean f.e.

ASA

7078.6

7236.937

39.3016

350,000

7052.4576

1.673

128,765

Fuzzy ASA 7049.6881

6.2.11 Function G11 Minimize f (x) = x21 + (x2 − 1)2

(6.25)

h1 (x) = x2 − x21 = 0

(6.26)

subject to

The domain of this function is contained in R2 , and the problem presents 1 nonlinear equality constraint. Its hyper-rectangle is defined by −1 ≤ xi ≤ 1, for i = 1, 2. The best known value for this function is 0.7499 at (-0.707036070037170616, 0.500000004333606807) . The results are shown in Table 6.12, where it can be seen that the performance of the ASA algorithm was very good, reaching the best known result. The best results were obtained without activating ASA’s fuzzy controller. A dynamic penalty function was employed by initializing the α factor value to 1,300, and raising it by 1,000 each 1,000 function evaluations. This approach worked well in this instance, tending to keep the search inside the feasible region most of the time. The best result was 0.74988 at (-0.7048443193003002, 0.4970723651368366), with a small constraint violation (2.67e-4).

6.2 Constrained Global Optimization Using ASA and Fuzzy ASA

113

Table 6.12 Results for function G11. Method

Best found

Mean

Std. dev.

Mean f.e.

ASA

0.74988

0.758275

0.0126

350,000

0.75461

0.01

350,000

Fuzzy ASA 0.74998

6.2.12 Function G12 Minimize f (x) = −(100 − (x1 − 5)2 − (x2 − 5)2 − (x3 − 5)2 )/100

(6.27)

g1 (x) = (x1 − p)2 + (x2 − q)2 + (x3 − r)2 − 0.0625 ≤ 0

(6.28)

subject to

where p, q, r = 1, 2, ..., 9 The domain of this function is contained in R3 , and the problem presents 93 nonlinear inequality constraints (the feasible region is composed of balls in R3 ). Its hyper-rectangle is defined by 0 ≤ xi ≤ 10, for i = 1, 2, 3. The best value for this function is −1 at (5,5,5). The results are shown in Table 6.13. Notice that ASA’s best result coincided with the best known and was obtained without the activation of ASA’s fuzzy controller. A dynamic penalty function was employed by initializing the α factor value to 1,300, and raising it by 1,000 each 1,000 function evaluations. This approach worked well in this instance, tending to keep the search inside the feasible region.

Table 6.13 Results for function G12. Method

Best found

Mean

Std. dev.

Mean f.e.

ASA

-1.000

-1.000

0

92,459

-0.99766

0.0012007

350,000

Fuzzy ASA -0.999911

6.2.13 Function G13 Minimize f (x) = ex1 x2 x3 x4 x5

(6.29)

114

6 Constrained Optimization

subject to h1 (x) = x21 + x22 + x23 + x24 + x25 − 10 = 0 h2 (x) = x2 x3 − 5x4 x5 = 0 h3 (x)

= x31 + x32 + 1 =

(6.30)

0

The domain of this function is contained in R5 , and the problem presents 3 nonlinear equality constraints. Its hyper-rectangle is defined by −2.3 ≤ xi ≤ 2.3, for i = 1, 2, and −3.2 ≤ xi ≤ 3.2, for i = 3, 4, 5. The best known value for this function is 0.053941514041898 at (-1.71714224003, 1.59572124049468, 1.8272502406271, -0.763659881912867, -0.76365986736498). The results are shown in Table 6.14, from which we conclude that the ASA algorithm approximated well the best known global minimum. The best performance was obtained with the activation of ASA’s fuzzy controller, although the difference between the results is very small and the two could be considered equivalent in this test. A dynamic penalty function was employed by initializing the α factor value to 100, and raising it by 1,000 each 1,000 function evaluations. This approach worked well in this instance, tending to keep the search inside the feasible region. Our best result was 0.05607 at (-1.718816029195466, 1.597645922887044, 1.812042933996276, -0.6556804868422581, -0.8830530128448459).

Table 6.14 Results for function G13. Method

Best found

Mean

Std. dev.

Mean f.e.

ASA

0.0563788

0.06473

0.09591

350,000

0.0689905

0.0033908

350,000

Fuzzy ASA 0.0550396

6.3 Conclusion The Fuzzy ASA and standard ASA approaches have been used to approximate the solutions to well-known constrained global optimization problems. Comparisons between the two showed that they are complementary and effective in constrained optimization tasks. The tests were also useful in assessing the ability of the ASA algorithm to tackle difficult constrained global optimization problems and in comparing its performance relatively to Fuzzy ASA. Simple dynamic and static penalty function schemes were employed in different situations and relevant ASA parameters were kept fixed along all tests.

References

115

References 1. Ahrari, A., Atai, A.A.: Grenade Explosion Method - A novel tool for optimization of multimodal functions. Applied Soft Computing 10, 1132–1140 (2010) 2. Ahrari, A., Shariat-Panahi, M., Atai, A.A.: GEM: A novel evolutionary optimization method with improved neighborhood search. Applied Mathematics and Computation 210, 376–386 (2009) 3. Barbosa, H.J.C., Lemonge, A.C.C.: An adaptive penalty method for genetic algorithms in constrained optimization problems. In: Iba, H. (ed.) Frontiers in Evolutionary Robotics, pp. 9–34. I-Tech Education Publ., Austria (2008) 4. Deb, K.: An efficient constraint handling method for genetic algorithms. Computer Methods in Applied Mechanics and Engineering 186(2-4), 311–338 (2000) 5. Farmani, R., Wright, J.A.: Self-Adaptive Fitness Formulation for Constrained Optimization. IEEE Transactions on Evolutionary Computation 7(5), 445–455 (2003) 6. Ingber, L.: Adaptive simulated annealing (ASA): Lessons learned. Control and Cybernetics 25(1), 33–54 (1996) 7. Karaboga, D., Akay, B.: A modified Artificial Bee Colony (ABC) algorithm for constrained optimization problems. Applied Soft Computing 11, 3021–3031 (2011) 8. Liang, J.J., Runarsson, T.P., Mezura-Montes, E., Clerc, M., Suganthan, P.N., Coello, C.A.C., Deb, K.: Problem Definitions and Evaluation Criteria for the CEC 2006 Special Session on Constrained Real-Parameter Optimization. Technical Report, Nanyang Technological University, Singapore (2005) 9. Lu, H., Chen, W.: Self-adaptive velocity particle swarm optimization for solving constrained optimization problems. J. Glob. Optim. 41, 427–445 (2008) 10. Oliveira Jr., H.: Fuzzy control of stochastic global optimization algorithms and VFSR. Naval Research Magazine 16, 103–113 (2003) 11. Oliveira Jr., H.A., Petraglia, A., Petraglia, M.R.: Frequency Domain FIR Filter Design Using Fuzzy Adaptive Simulated Annealing. Circuits, Systems and Signal Processing 28 (6), 899–911 (2009) 12. Oliveira Jr., H.A., Petraglia, A.: Global Optimization Using Space-Filling Curves and Measure-Preserving Transformations. In: Gaspar-Cunha, A., Takahashi, R., Schaefer, G., Costa, L., et al. (eds.) Soft Computing in Industrial Applications. AISC, vol. 96, pp. 121– 130. Springer, Heidelberg (2011) 13. Oliveira Jr., H.A., Petraglia, A.: Global optimization using dimensional jumping and fuzzy adaptive simulated annealing. Applied Soft Computing 11, 4175–4182 (2011) 14. Pachter, R., Wang, Z.: Adaptive Simulated Annealing and its Application to Protein Folding. In: Floudas, C.A., Pardalos, P.M. (eds.) Encyclopedia of Optimization, pp. 21–26. Springer, Heidelberg (2009) 15. Price, K., Storn, R., Lampinen, J.: Differential Evolution - A Practical Approach to Global Optimization. Springer, Heidelberg (2005) 16. Rocha, A.M.A.C., Fernandes, E.M.G.P.: Electromagnetism-Like Augmented Lagrangian Algorithm for Global Optimization. In: Gaspar-Cunha, A., Takahashi, R., Schaefer, G., Costa, L., et al. (eds.) Soft Computing in Industrial Applications. AISC, vol. 96, pp. 415–425. Springer, Heidelberg (2011) 17. Rosen, B.: Function optimization based on advanced simulated annealing. In: IEEE Workshop on Physics and Computation - Phys. Comp. 1992, pp. 289–293 (1992) 18. Runarsson, T.P., Yao, X.: Stochastic ranking for constrained evolutionary optimization. IEEE Transactions on Evolutionary Computation 4, 284–294 (2000)

Part III

Applications

Chapter 7

Applications to Signal Processing - Blind Source Separation

Abstract. In this chapter an alternative method to make independent component analysis and source separation is introduced. It is based upon Fuzzy Adaptive Simulated Annealing and uses mainly mutual information measures to achieve its final goals. After presenting the central arguments of the method, some experimental results are shown and comparison to previous work is done.

7.1 Introduction In many applications of Medicine (EEG, ECG), Acoustics (noise filtering), Econometrics, Defense (radar and sonar systems) etc. [7], several signals are received by sensors that process and transmit them to posterior stages in order to be further conditioned. By their own physical structure, such sensors distort original signals and, commonly, such a distortion should be reversed. Otherwise the whole acquisition apparatus may be compromised and not achieve its aim, that is to identify the proper signals under investigation. In some cases, it is possible to model such a phenomenon as being the result of a linear mixture, that is, original signals would have been processed by a linear and time invariant system, represented by an invertible matrix. Such a premise resumes the problem to the search for another matrix that, composed to the first one, takes back the actual readings. Assuming that original signals are statistically independent, such a problem was solved by many researchers, using diverse methods [2]. It occurs that not always the assumption of linearity is compliant with physical device structure and has thus to be discarded if we want to implement certain functional characteristics - from that comes the need of creation of nonlinear mixture separation methods, that keep straight relationship to independent component analysis. Although there are several and conclusive results concerning independent component analysis of linear mixtures (obtained through linear combinations of two or more signals), the case of nonlinear mixtures is yet in its infancy, taking into account the relatively few conclusive results in the literature concerning separation of H.A. e Oliveira Jr. et al.: Stochastic Global Optimization & Its Applications, ISRL 35, pp. 119–138. c Springer-Verlag Berlin Heidelberg 2012 springerlink.com 

120

7 Applications to Signal Processing - Blind Source Separation

mixtures resulting from nonlinear operations. Among the most relevant methods we find MISEP [2], that extends INFOMAX [3], the latter having its scope restricted to linear mixtures of statistically independent signals. The MISEP approach enlarges the reach of the INFOMAX model in two directions, that is, it is able to separate nonlinear mixtures and uses adaptive nonlinearities at their outputs. Taking into account that such features are related to the probability distributions of vectors resulting from the final analysis, the extra flexibility allows us to handle a larger number of cases. In what follows, it will be presented a brief description of the problem under investigation and the proposed contribution to tackle the problem of blind separation of nonlinear mixtures and related independent component analysis. At last, the advantages of the proposed algorithm will be shown through specific experiments. The problem under investigation consists of, given observed vectors (o = [o1 , o2 ]), obtained from nonlinear sensor readings (mixers) and whose original signals (s = [s1 , s2 ]) were generated according to unknown distributions, obtaining the transformation y = F(o) = [y1 , y2 ] that estimates original vectors (s) without previous knowledge of specific characteristics, admitting only the hypothesis that original signals are mutually statistically independent and the nonlinearity inserted in the acquisition device is an invertible function, at least when restricted to the operational domain of the physical apparatus. Fig. 7.1 shows schematically the described setting

Fig. 7.1 General diagram of the separation problem

Both INFOMAX and MISEP approaches aim to minimize the mutual information of components of random variable y, defined by the formula I(y) = ∑ H(yi ) − H(y)

(7.1)

i

where H(y), the Shannon differential entropy [2], is defined by H(y) = −



p(y)logp(y)dy

(7.2)

7.1 Introduction

121

where p is the probability density function of the corresponding random variable and the logarithm is the Neperian one, although it is possible to adopt other bases, like the decimal one, for instance. The mutual information I(y) is nonnegative, being null only when components of vector y are statistically independent, and considered a good measure of independence among the yi for utilization in independent component analysis [2]. So, the present work will use the same index to solve the described problem, although its estimation will be realized in a completely different fashion, whenever compared to previous research efforts. Both MISEP and INFOMAX methods use neural approaches with gradient based training, the second one restricted only to linear mixtures. In consequence, they are susceptible to local minima convergence whenever trying to minimize the mutual information or any cost functions aimed at signal separation. From that scenario, it is worth trying to synthesize a training method that may offer at least theoretical guarantee of convergence to the global minimum of the chosen objective function. It is proposed here to construct F by means of Takagi-Sugeno-Kang MIMO (Multiple Input Multiple Output) fuzzy inference systems, that possess approximation properties under very general conditions [1] and are equivalent to typical feedforward neural networks, ψ1 and ψ2 by means of sigmoid and nondecreasing polynomials composition and reinforcement learning using Fuzzy ASA [5, 6], that offers theoretical global optimum convergence warranty under conditions satisfied by the problem at hand, that is, to minimize mutual information among components of y. This way, and considering equal number of signals and sensors, F will be realized using a TSK MIMO system, that can be trained both in antecedent and consequent parts by varying their characteristics parameters, the examples having been built so as to adapt only the consequent coefficients and in dimension 2, for the sake of better understanding and comparison to previous results. As cited previously, the objective function to minimize is the mutual information I(y) and we search for a computationally adequate expression to reach such a goal. Supposing that each function at the output stage is a CDF (Cumulative Distribution Function) of the yi , it is easy to see that the zi will be uniformly distributed in the [0,1] interval, resulting in H(zi ) = 0, and giving the following expression for I(z) I(z) = ∑ H(zi ) − H(z) = −H(z)

(7.3)

i

Besides, each zi is related to yi by means of an invertible (bijective) function, resulting in I(y) = I(z). Consequently, we get I(y) = −H(z)

(7.4)

We conclude that, in maximizing the output entropy H(z), we will be minimizing the mutual information I(y), that is the final desired effect. In this way, the present method will try to maximize H(z), tending simultaneously to the configuration in

122

7 Applications to Signal Processing - Blind Source Separation

which functions ψi coincide with the CDFs of outputs yi . The parameterized expression for each ψi is

ψi (x) = sig(a + bx + cx3 + dx5 + ex7 )

(7.5)

1 and a, b, c, d, e ≥ 0 1 + e−x

(7.6)

where sig(x) =

Because they are nondecreasing and assume values restricted to the [0,1] interval, they are adequate to represent cumulative distributions. The seventh degree for the polynomial showed itself to be sufficient for all experimental cases. Contrarily to INFOMAX method, such functions will be adjusted during global training, and, differently from MISEP, the training will not be done by neural means. As the ψi have an auxiliary role, at the end of the process the output stage is discarded, being used only the trained system F. To do the actual optimization, it is necessary to have at hand expressions relating the measurements (o) and the entropy H(z), considering that the adaptation should be realized with basis in available experimental data. That is done by observing the following expression H(z) = H(o)+ < log(|detJ|) >

(7.7)

where J is the Jacobian matrix of the transformation composed by F and the output stage, and the operator <> denotes expected value. As the entropy H(o) is independent of the parameters under adaptation, it can be dropped from the optimization process, remaining just the second term, that is approximated by the sample mean of the training set elements (observed values) and whose expression is 1 K < log(|detJ|) >∼ = ∑ log(|detJk |) = E , K k=1

(7.8)

Δ

J = [∂ zi /∂ o j ] where detJk denotes the value of the Jacobian at each generic point belonging to the training set and K is its cardinality. This function is used in the fitting process, and depends on the values Jk and parameters under adaptation. This way, while the minimization process evolves, the system F and the functions ψi tend to assume favorable configurations relatively to the recovery of the original random vector (s), presented firstly to the input sensors. Looking specifically at the examples, we will have 64 parameters to estimate, being 54 relative to the TSK system and 10 to the auxiliary functions, because the model includes a MIMO system with 2 inputs and 2 outputs, fixed grid subdivision of the input domain (normalized in [0,1]x[0,1]), presenting 9 rules with 3 consequent parameters each, used in the form y = pi0 + pi1 x1 + pi2 x2 , and resulting in 9x3x2=54 elements to determine. The ψi are composed by standard sigmoids

7.1 Introduction

123

composed with seventh degree polynomials that have nonnegative coefficients at odd degree terms, but the constant term, and suppress the even degree terms, what results in 5 parameters for each of them - as the output space dimension is 2, we have a total of 2x5=10 coefficients. In this fashion, the vector presented to Fuzzy ASA algorithm will consist of the concatenation of 27 real numbers corresponding to the first TSK system output, 27 to the second one, 5 to the first function (ψ1 ) and 5 to the second function (ψ2 ). The decomposition of the normalized input domain was done by using 3 membership functions [1] corresponding to the concepts of high, median and low, what has been effective in the approximation of inverse functions. Considering the fact that independent component analysis using nonlinear methods may not present just one solution to a given problem without additional restrictions to the minimization of mutual information, we also considered the so-called post nonlinear model (PNL), that consists in the assumption of the hypothesis that the observed mixture is the result of a linear operation followed by nonlinear ones, one for each dimension. So, the diagram shown in Fig. 7.1 would assume the particular configuration in Fig. 7.2.

Fig. 7.2 PNL model

In this setting, it was assumed that the generic nonlinearity of original mixture consisted of multiplication by a matrix (M) followed by unidimensional application of nonlinear operators (f1 and f2), what is compatible with situations found in practice [4]. To achieve the separation, an inverse configuration is employed, that is, application of nonlinear operators (g1 and g2) followed by matrix multiplication (by matrix P), whose parameters shall be determined. Here, g1 and g2 are SISO (Single Input Single Output) Takagi-Sugeno-Kang fuzzy systems with normalized domain in the [-1,1] interval and granularity given by fuzzy terms named LOW, MEDIAN and HIGH, all fixed. In the consequent part we find the usual expressions, parameterized by the coefficients of affine expressions. Similarly to the first model, the adaptation will be done in such coefficients, keeping antecedent expressions unaltered, with 3 fuzzy rules for each inference system. The functions ψ1 and ψ2 , as before, correspond to the CDFs of outputs yi , and their treatment will be exactly the same.

124

7 Applications to Signal Processing - Blind Source Separation

7.2 Implementation In the specific case of the PNL model, we will have 26 parameters to estimate, with 12 of them corresponding to TSK systems, 10 to auxiliary functions ψi and 4 to matrix P, taking into account that the realization uses 2 SISO systems, fixed grid subdivision of the input domain (normalized in [-1,1]), presenting each one, 3 rules with 2 consequent parameters and used in the form y = pi0 + pi1 x, resulting in 3x2x2=12 elements to determine. The functions ψi are the result of composing standard sigmoids with seventh degree polynomials, exactly like those used in the precedent SISO model - as the number of outputs is 2, we have a total of 2x5=10 coefficients. So, the vector presented to the Fuzzy ASA algorithm will be composed by the concatenation of 12 real numbers corresponding to the TSK systems, 5 to the first function (ψ1 ), 5 to the second function (ψ2 ) and, at last, 4 relative to the elements of matrix P. Similarly to the previous problems, mutual information is to be minimized by means of approximations obtained through evaluations of Jacobian determinant at training set points, as described before. All specific operations are encapsulated in a single module, that is activated repeatedly by the main nucleus in its trajectory toward the final desired configuration.

7.3 Results From this point on some results used in the validation of proposed approach will be presented. The examples illustrate the effectiveness of our approach and, in each one, a comparison to the solution resulting from the MISEP method will be done.

7.3.1 Example 1 - Separation by TSK MIMO System In this example the original samples were generated through unidimensional Laplace and uniform distributions, so as to synthesize (nearly) statistically independent signals, in compliance with problem premises, as shown in Fig. 7.3. To attain that end the coordinates of each dimension were generated and conditioned separately, tending to a joint distribution in a cross-like shape, indicating satisfaction of the above condition. After that, a nonlinearity expressed by (o1 = s1 + 4s22 , o2 = s2 + 4s21 ) was applied, producing the training set to be used in the analysis process, as can be seen in Fig. 7.4. Note that, as in the linear case, order permutations and scale changes are allowed in the separated signals. The results of the two methods are shown in Figs. 7.5 and 7.6 by means of joint distributions of resulting signals, and demonstrate pictorially the effectiveness of both in this particular case, discounted some peculiarities as nonlinear distortion at peripheral regions.

7.3 Results

Fig. 7.3 Joint distribution of original signals

Fig. 7.4 Joint distribution of transformed signals

125

126

7 Applications to Signal Processing - Blind Source Separation

Fig. 7.5 Joint distribution of signals recovered by the proposed method

Fig. 7.6 Joint distribution of signals recovered by the MISEP method

7.3 Results

127

7.3.2 Example 2 - Separation by TSK MIMO System In this section, the original samples were generated using unidimensional Laplace and uniform distributions, so as to get (nearly) statistically independent signals, as shown in Fig. 7.7(a). To that end, coordinates of each dimension were separately generated and conditioned, tending to a joint distribution similar to a vertical bar, indicating compliance with the hypothesis of the problem. After that, a nonlinearity expressed by (o1 = s1 + 4s22 , o2 = s2 + 4s21 ) was applied, producing the training set to be used in the analysis process itself, as represented in Fig. 7.7(b).

(a) Original

(b) Measured

Fig. 7.7 Joint distribution of signals

We can see in Fig. 7.8(b) an example of failure of the MISEP method in the recovery of the original configurations, taking into account mainly the lower horizontal segment. In Fig. 7.8(a), it is shown the successful recovery by the proposed method.

128

7 Applications to Signal Processing - Blind Source Separation

(a) Proposed method

(b) MISEP

Fig. 7.8 Recovered distributions

7.3.3 Example 3 - Separation by TSK MIMO System Here the original points were generated through unidimensional uniform distributions, customized so as to get signals compliant with problem premises. For that, the coordinates were separately conditioned and tended to a joint distribution resembling a segmented vertical column, indicating that the desired conditions were satisfied, as shows Fig. 7.9(a). After that, a nonlinearity expressed by (o1 = s1 + 4s22 , o2 = s2 + 4s21 ) was applied, producing the training set to be used in the rest of the process, as seen in Fig. 7.9(b). It is evident that there was a permutation in the order of the recovered signals, shown in Figs. 7.10(a) and 7.10(b), besides the usual change in scale.

7.3 Results

129

(a) Original signals

(b) Measured signals

Fig. 7.9 Joint distribution

7.3.4 Example 4 - Separation by TSK MIMO System Here the original points were generated by means of unidimensional Laplace distributions, customized so as to furnish statistically independent signals, according to problem hypothesis (Fig. 7.11(a)). For that, coordinates in each dimension were separately conditioned and the resulting points tended to a joint distribution similar to a segmented vertical bar. After that, a linear transformation expressed by (o1 = s1 + 4s2 , o2 = s2 + s1 ) was applied, succeeded by a second coordinate distortion through the function tanh(0.4x), producing the training set used in the rest of the recovering process and represented in Fig. 7.11(b).

130

7 Applications to Signal Processing - Blind Source Separation

(a) Proposed method

(b) MISEP

Fig. 7.10 Recovered signals

We can see here an example of failure of the MISEP method in trying to recover the original configuration of signals (Fig. 7.12(b)), when compared to results obtained by the proposed method (Fig. 7.12(a)), examining particularly the lower and upper parts of the resulting configuration.

7.3 Results

131

(a) Original signals

(b) Measured signals

Fig. 7.11 Joint distribution

(a) Proposed method

Fig. 7.12 Recovered signals

(b) MISEP

132

7 Applications to Signal Processing - Blind Source Separation

7.3.5 Example 5 - Mixture by PNL Model In this example, the original signals (Fig. 7.13) were obtained by discretization of real world sounds and mixed through a linear transformation followed by distortion functions tanh(x) and tanh(0.5x) in dimensions 1 and 2, respectively, producing a training set containing 1000 elements to be used in the adaptation process properly said and represented in Figs. 7.15 and 7.16. By observing the sample joint distribution of nondistorted signals, as shown in Fig. 7.14, it is possible to see the high degree of statistical independence between the two signals, satisfying, at least approximately, the premises of the method.

Fig. 7.13 Original signals

To obtain greater precision in the evaluation of results, it was used an index known as residual crosstalk and defined by the expression E[(y − s)2 ], where y is the recovered signal and s the original one, both conditioned so as to present unitary variance. Taking into account that the lower the value of the chosen index, the better the quality of recovery, we need to compute it using the signals resultant of the two approaches and compare the numerical results. The values obtained by the proposed method are -7.7314 dB (or 0.41061) for the first signal and -20.8643 dB (or 0.090528) for the second signal, the corresponding amounts for the MISEP method being 23.3090 dB (or 14.6369) and -4.1943 dB (or 0.6169), respectively, showing that the proposed algorithm performed better than MISEP in this example.

7.3 Results

Fig. 7.14 Joint distribution of original signals

Fig. 7.15 Mixtures used in the training process

133

134

7 Applications to Signal Processing - Blind Source Separation

Fig. 7.16 Joint distribution of mixed signals

Fig. 7.17 Signals recovered by proposed method

7.3 Results

Fig. 7.18 Joint distribution of signals recovered by the proposed method

Fig. 7.19 Signals recovered by MISEP method

135

136

7 Applications to Signal Processing - Blind Source Separation

Fig. 7.20 Joint distribution of signals recovered by MISEP method

7.4 Conclusion

137

7.4 Conclusion In this chapter, we presented an alternative method to blind source separation and independent component analysis in nonlinear mixtures of statistically independent signals, particularly the so-called PNL (Post NonLinear) model, that presupposes certain realistic conditions upon the structure of the input stage of sensors or data acquisition apparatus. The new approach uses an approximation of mutual information between output signals to guide the automatic design of the separation devices, that use parametric fuzzy inference systems to accomplish the nonlinear part of the inversion task and reverse the nonlinearity inserted at the input - the linear inversion is effected by a matrix, whose parameters are found simultaneously by the training mechanism. Finally, some examples aimed at exemplifying the shown ideas are presented and performance indexes evaluated.

138

7 Applications to Signal Processing - Blind Source Separation

References 1. Abonyi, J.: Fuzzy Model Identification for Control. Birkh¨auser, Boston (2003) 2. Almeida, L.B.: Nonlinear Source Separation. Morgan & Claypool Publishers (2006) 3. Bell, A., Sejnowski, T.: An information-maximization approach to blind separation and blind deconvolution. Neural Computation 7, 1129–1159 (1995) 4. G´orriz, J.M., Puntonet, C.G., Rojas, F., Martin, R., Hornillo, S., Lang, E.W.: Optimizing Blind Source Separation with Guided Genetic Algorithms. Neurocomputing 69, 1442– 1457 (2006) 5. Ingber, L.: Adaptive simulated annealing (ASA): Lessons learned. Control and Cybernetics 25(1), 33–54 (1996) 6. Oliveira Jr, H.: Fuzzy control of stochastic global optimization algorithms and VFSR. Naval Research Magazine 16, 103–113 (2003) 7. Tan, Y., Wang, J.: Nonlinear Blind Source Separation Using Higher Order Statistics and a Genetic Algorithm. IEEE Transactions on Evolutionary Computation 5(6), 600–612 (2001)

Chapter 8

Fuzzy Modeling with Fuzzy Adaptive Simulated Annealing

Abstract. Data-based fuzzy system modeling usually depends on effective optimization methods to fit experimental data to parametric fuzzy models. Here, an approach that uses Takagi-Sugeno models and Adaptive Simulated Annealing (ASA) is presented and discussed, showing that (Fuzzy) ASA could also be helpful in such a kind of task. The problem to solve is well-defined - given a training set containing a finite number of input-output pairs, construct a fuzzy system approximating the behavior of the actual system that originated that set, within a pre-established precision. Such an approximation must have generalization ability to be useful in the real world, considering the finiteness of the training set and other constraints. Besides, other suggestions for application of (Fuzzy) ASA to fuzzy logic related problems are offered.

8.1 Introduction Black box, realistic fuzzy system modeling typically gives rise to nonlinear global optimization problems [1]. By black box we mean modeling only from observed data, not taking into account any prior knowledge about the system under study. So, supposing that a given system have n inputs and one output - the so-called MISO (Multiple Input Single Output) system - all we have is a training set, formed by points in (n+1)-dimensional Euclidean space. The first n components of each (n+1)tuple are the inputs to the given system and the last coordinate is the corresponding response. In general, the goal is to develop a mathematical model that approximates the behavior of the actual system within a certain error. In the approach to be presented here, that error is the function to minimize - it depends on the training set elements and model parameters in a strongly nonlinear way. Besides, the resulting functions present high-dimensional domains and the COD (Curse Of Dimensionality) comes into play. Restricting our attention to fuzzy models, the usual choice is to employ Mamdani or Takagi-Sugeno systems, since there are theoretical results assuring their approximating properties under very general conditions. In this H.A. e Oliveira Jr. et al.: Stochastic Global Optimization & Its Applications, ISRL 35, pp. 139–148. c Springer-Verlag Berlin Heidelberg 2012 springerlink.com 

140

8 Fuzzy Modeling with Fuzzy Adaptive Simulated Annealing

chapter, we focus on TS models, but the same arguments apply to the Mamdani model as well. After all, the global optimization algorithm used in this chapter (ASA) [3, 4, 5, 6, 7] just receives a certain number of parameters, process them and gives them back in an interactive way, until the final convergence.

8.2 Affine Takagi-Sugeno Fuzzy Systems Their structure consists of rules in the following form: n

Ri : IF x IS Ai T HEN yi = ai0 + ∑ aik xk

(8.1)

k=1

where • • • • • •

x = (x1 , x2 , x3 , ..., xn ) ∈ D ⊂ Rn represents the input Ai is a fuzzy set with membership function defined on D yi ∈ R is the scalar output corresponding to rule i aik ∈ R are the output parameters associated with rule i i ∈ {1, ..., NR} NR = total number of fuzzy rules The global output of such a MISO system is given by y(x) =

∑NR k=1 μAk (x)yk ∑NR k=1 μAk (x)

(8.2)

where μAk is the membership function associated with the k-th rule. Sometimes, it can be more convenient to represent the antecedent part as IF x1 IS Ai1 AND IF x2 IS Ai2 AND...AND IF xn IS Ain where Aik has a scalar membership function. In this case, the multidimensional function can be computed as n μAi (x) = μAi (x1 , x2 , ..., xn ) = Λk=1 μAik (xk )

(8.3)

Obviously we can use other t-norms than the minimum operator, such as the product one.

8.3 The Fuzzy Modeling Problem

141

8.3 The Fuzzy Modeling Problem Given p (n+1)-tuples representing the behavior of a MISO system, build a TakagiSugeno fuzzy system that, when submitted to the same inputs (first n coordinates), produces an approximation of the true output (last coordinate). The set of tuples form the training set and can be put in a matrix-like configuration: ⎛ ⎞ i11 i12 · · · i1n o1 ⎜i21 i22 · · · i2n o2 ⎟ ⎜ ⎟ (8.4) ⎜ .. .. . . .. ⎟ ⎝ . . ⎠ . . i p1 i p2 · · · i pn o p where the n first columns represent the several inputs and the last one, the respective outputs, that is, we have p input n-dimensional row vectors and p scalar outputs. Exactly as in crisp modeling, it is important to use test and validation sets to evaluate the quality of the final model (generalization ability etc.). Those sets look exactly like training sets, but are not used in the fitting process itself.

8.3.1 Approximation in Lower Dimensions Initially, we solve the approximation problem by means of the following steps: Step 1 - Normalize the original training set (8.4) by scaling the input and output variables and transforming the original intervals into [0,1]. Suppose that it has p elements and is given by (8.4). We get a new (normalized) training set by computing the linear transformations i∗kl =

ikl − min(i1l , i2l , i3l , ..., i pl ) max(i1l , i2l , i3l , ..., i pl ) − min(i1l , i2l , i3l , ..., i pl )

(8.5)

o∗l =

ol − min(o1, o2 , o3 , ..., o p ) max(o1 , o2 , o3 , ..., o p ) − min(o1 , o2 , o3 , ..., o p )

(8.6)

and

From this point on we work with the new training set, that represents a mapping from [0,1]x[0,1]x...x[0,1] into [0,1]. It’s important to save some parameters from the original TS so that we can de-normalize the final results, using the inverse transformation.

142

8 Fuzzy Modeling with Fuzzy Adaptive Simulated Annealing

Step 2 - At this point we have a new function to fit by means of a TakagiSugeno fuzzy system - the domain is the n-dimensional unitary cube and the image is contained in [0,1]. Our choice, in this first method, was to define five input fuzzy terms over the universe of discourse [0,1] and evaluate each dimension separately. After that, the activation degree is found by composing the several membership values. The functions were named VERY LOW (VL), LOW (L), ZERO (Z), HIGH(H) and VERY HIGH (VH). Their definitions are shown below

⎧ ⎪ if x < 0 ⎨1 V L(x) = 1 − 3x if x ∈ [0, 1/3] ⎪ ⎩ 0 if x > 1/3 ⎧ ⎪ if x ∈ [0, 1/3) ⎨3x L(x) = 3(1 − 2x) if x ∈ [1/3, 1/2) ⎪ ⎩ 0 , otherwise

⎧ ⎪ ⎨2(3x − 1) if x ∈ [1/3, 1/2) Z(x) = 2(2 − 3x) if x ∈ [1/2, 2/3) ⎪ ⎩ 0 , otherwise

⎧ ⎪ ⎨3(2x − 1) if x ∈ [1/2, 2/3) H(x) = 3(1 − x) if x ∈ [2/3, 1] ⎪ ⎩ 0 , otherwise

⎧ ⎪ if x < 2/3 ⎨0 V H(x) = 3x − 2 if x ∈ [2/3, 1] ⎪ ⎩ 1 if x > 1 Each fuzzy rule will have the following general form

(8.7)

(8.8)

(8.9)

(8.10)

(8.11)

8.3 The Fuzzy Modeling Problem

143

Δ

Ri = IF x1 IS Ti1 AND...AND xn IS Tin n

T HEN yi = ai0 + ∑ ai j x j

(8.12)

j=1

where yi ∈ [0, 1]. At first, and to investigate the efficiency of the ASA method in this kind of task, our choice was to allow linguistic variables to assume each of the 5 terms above. By doing so, we arrived at the following table relating the dimension of the input space and the number of parameters and rules to find in each case

Table 8.1 Number of parameters for parametric TS fuzzy system Dimension

Number of rules

Number of parameters

1

5

10

2

25

75

3

125

500

n

5n

(n + 1)5n

For input dimensions greater than 3 the COD shows up and even a powerful algorithm needs an alternative approach to finish the minimization task in an acceptable period of time, taking into account the computational power available at present. We will describe such algorithm in the next section. Step 3 - Now, it is time to synthesize the error function that is going to guide the fitting process and will be used by the ASA algorithm as its cost function. We’ll adopt the so-called batch training method, in which all deviations (relative to the training set) are taken into account simultaneously, opposite to the incremental learning approach. That said, we define the expression of the error function as Δ

p Global training error = ∑k=1 Ek2

where p is the cardinality of the training set.

144

8 Fuzzy Modeling with Fuzzy Adaptive Simulated Annealing Δ

Ek = yapp k − ok , ∑Ri=1 AD(i)(x)yi (x) ∑Ri=1 AD(i)(x) where x ∈ Rn , R = number of rules ok = output part of the k-tuple of the training set Δ

yapp = k

AD(i)(x) = Activation degree of rule i at point x ∈ R, defined as Δ

n

AD(i)(x) = ∏ μTi (xl ) l

l=1

yi (x) = output of rule i corresponding to input x, defined as Δ

n

yi (x) = ai0 + ∑ aik xk k=1

with x = (x1 , x2 , x3 , ..., xn ) ∈ Rn ai j are the free parameters that the global minimization algorithm will use to fit the fuzzy model to the original function. Step 4 - Having defined the cost function, all we have to do is to start the training phase by activating the ASA algorithm, that will guide the fitting process.

8.3.2 Approximation in Higher Dimensions The previous method works well for lower dimensional input spaces, but for higher number of dimensions it was necessary to build a less sensitive algorithm. We now describe a different and more adequate method for application in higher dimensional systems: Step 1 - Identical to the previous one (the training set has to be normalized). Step 2 - The new training set is submitted to a clustering process that produces a set of cluster centers. Each center (xc ) will define a multidimensional membership function given by Δ

μxc (x) =

2 1 + e10NCd(x,xc )

(8.13)

where NC is the total number of cluster centers and d(., .) is the Euclidean distance. Each μxc (x) will be used to synthesize fuzzy rules like

8.4 Ideas for Fuzzy Clustering Using ASA

145 n

Δ

Ri = IF x IS Ai T HEN yi = ai0 + ∑ ai j x j

(8.14)

j=1

where Ai is the fuzzy set whose membership function is defined in (8.13). We observe that each cluster center will originate one fuzzy rule and the number of clusters is defined in advance. In the implementation, we suggest the use of the Kohonen SOM (Self Organizing Map) to realize the clustering (or vector quantization, if you prefer), but any similar mechanism will do the intended task, such as FCM (Fuzzy C-Means) or Gustafson-Kessel fuzzy clustering algorithm. Our aim is to get a finite (and as small as possible) set of points that can represent the given training set. It’s also possible to improve the approximation accuracy by tuning the location of cluster centers. Step 3 - The expression of the error function is the same as before, with a different activation degree Δ

AD(i)(x) = μAi (x) =

2 1 + e10NCd(x,xci )

(8.15)

An advantage of this type of membership function is that it is defined over the whole input domain and its evaluation is very simple. Step 4 - Same as in lower dimensions - the ASA algorithm is started with the error function as its cost function. This time the number of free parameters is R(2n+1), where R is the number of rules (the same of cluster centers) and n is the dimension of the input space - to each rule corresponds one cluster center (n parameters) and n+1 consequent coefficients. For example, if we choose R=16 and the input space has dimension 3, the number of parameters to find is 16(2x3+1)=112. In the first method, we would have 500 parameters to fit, that is certainly a harder task.

8.4 Ideas for Fuzzy Clustering Using ASA Clustering can be viewed as the classification of data into groups based on similarities between the elements of a given sample, and we could speak about hard and fuzzy clustering. In hard clustering elements might belong to only one cluster, that’s to say, its membership degree relatively to a certain cluster is 1 and to the others is 0. In fuzzy clustering we can have the same point belonging to more than one cluster with different degrees - in this sense, fuzzy clustering is an extension of hard clustering. Although there are many methods capable of doing fuzzy clustering, fuzzy c-means (FCM, for short) and Gustafson-Kessel algorithm are well-known and widely tested methods. Here, we will sketch a way of doing fuzzy clustering more directly by facing the problem as a global optimization one and applying ASA to search for a global minimum.

146

8 Fuzzy Modeling with Fuzzy Adaptive Simulated Annealing

So, suppose that we are given N vectors, denoted by zk ∈ Rn , k = 1, ..., N, that should be attributed to NC clusters, represented by their centers or prototypes vi ∈ Rn , i = 1, ..., NC . To classify the data in a fuzzy logic way (fuzzy clustering) we have to represent their membership degrees to each one of the clusters, and a good way to do that is to find the corresponding fuzzy partition matrix M, with NC rows and N columns, whose element mi j represents the membership degree of vector z j to cluster i. In this fashion, we have to find not only the elements of matrix M but also the cluster centers in such a way that nearby vectors zi get classified whether in the same cluster or in close clusters, so as to reflect their original proximity. There is a certain class of clustering algorithms [1] that try to find the best M and vi by solving the following constrained optimization problem Minimize NC N

P(v1 , v2 , ..., vN , M) = ∑ ∑ (μi j )m d 2 (z j , vi )

(8.16)

i=1 j=1

subject to NC

∑ μi j = 1,

i=1 N

0<

j = 1, ..., N

∑ μ ji < N,

(8.17) j = 1, ..., NC

i=1

where m ∈ (1, ∞) dictates the fuzziness of clusters, that’s to say, whenever m gets higher, clusters become more overlapped, and as m → 1, membership degrees of vectors z j tend to values in {0,1}, what would mean hard (or crisp) clustering. Expression d(z j , vi ) denotes a generic distance between z j and vi , and when we use different definitions for it, cluster configurations vary (of course), allowing us to shape final results according to our needs by fine tuning them. Whenever d is the usual Euclidean distance, we have the objective function used by the fuzzy c-means algorithm [2], and if the distance is defined by d 2 (x, vi ) = (x − vi )T Mi (x − vi ), i = 1, ..., NC

(8.18)

we get the function used by the Gustafson-Kessel algorithm, where Mi is a positive definite matrix associated to each cluster, making this method more adaptable. In this fashion, it’s not difficult to see that there are infinite ways to cluster data, but one problem remains: how to globally optimize (8.16) with constraints (8.17) ? ASA could be used for that in an uniform way, taking into account that it is able to do constrained optimization, as we saw in a previous chapter.

8.5 Conclusions about the Presented Methods

147

8.5 Conclusions about the Presented Methods Methods for data based fuzzy modeling using (Fuzzy) ASA were presented and can be useful as an alternative to other existing paradigms, like those using genetic algorithms or gradient methods [1] to fit (or train) experimental data to parametric models. That approach can be used for function approximation, in general, and specific tasks such as fuzzy controller design and synthesis can be realized in a relatively uniform manner. Although the focus was placed over Takagi-Sugeno fuzzy systems, the reasoning can easily be applied to Mamdani systems as well, taking into account that the underlying problem can be faced as the global minimization of a numeric error function. In particular, the present method is an alternative to the neuro-fuzzy modeling approach that, in the end, faces a global optimization problem as well. With respect to fuzzy clustering, the exposition highlighted the fact that we can, in general, face the problem as a constrained global optimization one and, in that the case, it’s possible to take advantage of ASA’s good performance in such tasks. Besides, such an approach could allow us to find solutions in a very uniform way, as ASA only ”sees” functions to optimize, working independently of the meanings of specific problems.

148

8 Fuzzy Modeling with Fuzzy Adaptive Simulated Annealing

References 1. Hellendoorn, H., Driankov, D. (eds.): Fuzzy Model Identification - Selected approaches. Springer, Heidelberg (1997) 2. H¨oppner, F., Klawonn, F., Kruse, R., Runkler, T.: Fuzzy Cluster Analysis. Wiley (1999) 3. Ingber, L.: Adaptive simulated annealing (ASA): Lessons learned. Control and Cybernetics 25(1), 33–54 (1996) 4. Oliveira Jr, H.: Fuzzy control of stochastic global optimization algorithms and VFSR. Naval Research Magazine 16, 103–113 (2003) 5. e Oliveira Jr., H.A., Petraglia, A.: Global Optimization Using Space-Filling Curves and Measure-Preserving Transformations. In: Gaspar-Cunha, A., Takahashi, R., Schaefer, G., Costa, L., et al. (eds.) Soft Computing in Industrial Applications. AISC, vol. 96, pp. 121– 130. Springer, Heidelberg (2011) 6. Oliveira Jr., H.A., Petraglia, A.: Global optimization using dimensional jumping and fuzzy adaptive simulated annealing. Applied Soft Computing 11, 4175–4182 (2011) 7. Rosen, B.: Function optimization based on advanced simulated annealing. In: IEEE Workshop on Physics and Computation - Phys. Comp.1992, pp. 289–293 (1992)

Chapter 9

Statistical Estimation and Global Optimization

Abstract. A very popular method for the estimation of probability density functions (PDFs) associated to a set of random samples of a given population or process is the so-called Maximum Likelihood Estimation Method, based upon the premise that the best estimative for the set of parameters corresponding to the actual PDF of the population under study is that one maximizing the likelihood function, relative to that set of sample points. To find global maximizers for that function it is common practice to submit it, or its logarithm, to gradient based deterministic algorithms, aiming at better numerical conditioning. The present work presents an alternative approach to obtain the global maximum, based on Fuzzy ASA. After a brief theoretical introduction, several experimental results will be presented to illustrate the effectiveness of the proposed method.

9.1 Introduction The maximum likelihood estimation method is a well-known technique for obtaining estimatives of parameters corresponding to probability density functions [5, 6, 10]. The general procedure could be described as follows: Given an independent and identically distributed sample (x1 , x2 , ..., xn ) of a specific population with supposed pdf f (x|θ1 ; θ2 ; . . . ; θk ), the corresponding likelihood function is defined by [10] n

Δ

L(Θ|x) = L(θ1 , θ2 , . . . , θk |x1 , x2 , . . . , xn ) = ∏ f (xi |θ1 , θ2 , . . . , θk )

(9.1)

i=1

where x = (x1 , x2 , . . . , xn ) and Θ = (θ1 , θ2 , . . . , θk ). H.A. e Oliveira Jr. et al.: Stochastic Global Optimization & Its Applications, ISRL 35, pp. 149–167. c Springer-Verlag Berlin Heidelberg 2012 springerlink.com 

150

9 Statistical Estimation and Global Optimization

Whenever independency and/or distribution equality assumptions are not satisfied, the likelihood function is defined as the joint density f (x1 , x2 , . . . , xn |θ ), considered as a function of vector θ , taking into account that the sample is fixed. The vector θ ∗ maximizing L(θ |x) is called the maximum likelihood estimate, and is an approximation of the parametric vector corresponding to the actual distribution of the population or process under study. Despite of being defined by means of the above expression, its maximization is usually done by means of its logarithm (log-likelihood function), considering that such a transformation avoids the ocurrence of numerical underflows, besides simplifying the expression of the original likelihood function. Because of the monotonicity of the logarithm function, the maximizers of the transformed function will coincide with those of the original function. In the literature [1, 3] it is common to find several examples of procedures that aim to find the MLE by equating the gradient of the (log) likelihood function to zero and solving the resulting system of equations, possibly nonlinear ones. However, such an approach might not be effective in cases presenting nonsmooth likelihood functions, existing also the possibility of convergence to non-global extrema, in case of multimodal objective functions. Another weakness of those approaches is evidenced when the equations to be solved are very complex and don’t have known closed form solution. In summary, it would be sensible to say that an adequate method for use in the MLE problem should: • Be able to optimize discontinuous or, at least, nondifferentiable objective functions; • Be able to optimize globally in the great majority of cases, not being easily caught in suboptimal extrema; • Present theoretical guarantee of convergence to global extrema, at least in distribution. The Fuzzy Adaptive Simulated Annealing method presents all the above characteristics and showed itself successful in MLE problems, as will be shown in the next sections.

9.2 Maximum Likelihood Estimation with ASA Whenever sample data have been collected and the likelihood function of a model (given the data) is chosen, it is possible to make statistical inferences about the population, that is, to find approximations for the probability distribution that underlies the data. Of course, different parameter values give rise to different probability distributions, and we are interested in finding the parameter values that correspond to the desired PDF in a given (parametric) family. The principle of maximum

9.3 Implementation and Experiments

151

likelihood estimation, originally developed by R. A. Fisher in the 1920s, states that the desired probability distribution should make the observed data most likely, which is obtained by seeking the value of the parameter vector that maximizes the likelihood function (9.1). According to the MLE principle, the population who ”owns” the found PDF is the most likely to have generated the observed data. So, maximum likelihood estimation is a method by which the probability distribution that makes the observed data most likely is sought. It is worth to remark that MLE estimates need not exist or even be unique. As we said in the previous section, it is usually not possible to obtain an analytic form solution for the MLE estimate, especially when the model family involves many parameters and its respective PDF is highly nonlinear. In such settings, the MLE estimate must be sought numerically using nonlinear optimization algorithms. The basic idea of nonlinear optimization is to quickly find optimal parameters that maximize the log-likelihood. This is done by searching much smaller subsets of the multidimensional parameter domain rather than exhaustively searching the whole configuration space, which becomes intractable as the number of parameters increases. The fundamental idea is that the search should proceed iteratively, by taking into account the results from the previous iterations. It is very important to be aware that, in general, optimization algorithms does not necessarily guarantee that a set of parameter values that uniquely maximizes the loglikelihood will be found. Finding optimum parameters numerically is essentially and typically a heuristic process in which algorithms try to improve upon an initial set of parameters that is supplied by some means. Depending upon the choice of the initial parameter values, the algorithm could prematurely return a sub-optimal estimative - this is called the problem of premature convergence. As that is a very serious and difficult problem with no general known solution, a variety of heuristics have been developed, trying to avoid the problem, although there is no full guarantee of their effectiveness. This is particularly true, of course, for multimodal and multivariate objective functions. Fuzzy ASA can be used in difficult MLE problems by facing the negative of (9.1) as its objective function and restrictions over parameters as constraints [2, 4, 7, 8, 9]. From this point on it suffices to proceed exactly like in a regular constrained global optimization problem, as we will see in the next section.

9.3 Implementation and Experiments To estimate parameters by means of the previous approach, we adopted the usual practice and coded each objective function in a separate dynamic link library that can be called by the generic optimization code (ASA / Fuzzy ASA). This way, each specific problem can be encapsulated in one self-contained module. As we want to maximize the likelihood function, it suffices to change its sign and submit it to the global minimization process. As the final result, we get approximations of the parameters corresponding to maximum likelihood. The modules corresponding to objective functions are coded in the C++ language, as usual.

152

9 Statistical Estimation and Global Optimization

The original Fuzzy ASA code and the same ASA parameters were kept intact through all tests, to assess ASA’s minimization ability without exploiting specific features of objective functions. As said before, the method needs to maximize a given function and it was necessary to take the negative of original log-likelihood functions. To find the optimal parameters for a certain distribution class, given the corresponding samples, it’s enough to synthesize the log-likelihood function, change its sign and submit it to (Fuzzy) ASA, for minimization. Such architecture allows us to decouple the optimization mechanism itself from the code relative to particular objective functions, as before. Although the ASA implementation provides many adjustable parameters, we present here the most important ones and that were decisive to get satisfactory results, that’s to say, to approximate log-likelihood functions’ global extrema. Temperature Ratio Scale = 1E-5 Cost Parameter Scale Ratio = 1.0 Temperature Anneal Scale = .01 User Initial Parameters = TRUE Initial Parameter Temperature = 1000.0 Reanneal Cost = 1 Reanneal Parameters = TRUE In what follows, we will present numerical examples of successful parameter estimation relative to well-known probability density distributions. They illustrate the ability of Fuzzy ASA in obtaining good approximations at diverse scenarios and using training samples of very different sizes. To sample from some distributions we used the excellent and very useful C++ library PROB, authored by John Burkardt and freely downloadable (by the time of this writing) from his site (http://people.sc.fsu.edu/∼jburkardt/).

9.3.1 Exponential Distribution   1 A−y φ (y) = exp B B

(9.2)

This test was aimed at approximating parameters A (location) and B (scale) of an exponential distribution, given a training set of 200 iid samples, as shown in the histogram of figure 9.1(a). In figure 9.1(b), we can see a histogram obtained from 20000 samples of the exponential distribution using the actual parameters (A=3, B=2), and in figure 9.1(c) we find the one corresponding to the estimated parameters (A=3.010291305178826, B=1.950514385349573). To achieve such results, the

9.3 Implementation and Experiments

153

algorithm used 216422 evaluations of the objective function. The search intervals are shown in Table 9.1: Table 9.1 Search intervals used by Fuzzy ASA - exponential distribution Parameter

Search interval

A B

[-10,10] [0,10]

(a) Histogram of training samples

(b) Histogram of 20000 samples (true parameters)

(c) Histogram of 20000 samples (estimated parameters)

Fig. 9.1 Histograms corresponding to exponential distribution parameter estimation

Below, we show the source code corresponding to the DLL submitted to ASA in order to solve the related constrained global optimization problem. Of course, this

154

9 Statistical Estimation and Global Optimization

particular task could be (and usually is) solved by simpler methods but, exactly for its simplicity, is a good example for the sake of faster learning.

# include <windows.h> # include <stdio.h> # include <stdlib.h> # include <math.h> # include <exceptio.h> # include # include # include # include # include #include ”prob.h” // This file contains the iid samples (training set) #define SAMPLEFILE ”SAMPLES.DAT” #define MAXNOOFENTRIES 10000 char NameWithUnderscore[] = ” MLEExponential”; static int NoOfSamples ; double SampleArray[MAXNOOFENTRIES]; BOOL DllMain(HINSTANCE hInst,DWORD dwReason, LPVOID Reserved) { FILE *fp; char sample[100] ; switch (dwReason) { case DLL PROCESS ATTACH: NoOfSamples = 0; fp = fopen(SAMPLEFILE,”r”); while ( fscanf(fp,”%s”,sample) != EOF && NoOfSamples < MAXNOOFENTRIES ) { SampleArray[NoOfSamples] = strtod(sample,NULL); NoOfSamples++ ;

9.3 Implementation and Experiments

} fclose(fp); if (NoOfSamples == 0) return FALSE; break; case DLL PROCESS DETACH: break; case DLL THREAD ATTACH: break; case DLL THREAD DETACH: break; } return TRUE; }

extern ”C” double declspec(dllexport) MLEExponential ( int noofdimensions, double *x, void *Custom ) { double aux = 0 ; double resu ; for ( int i=0 ; i < NoOfSamples ; i++ ) { resu = exponential pdf(SampleArray[i],x[0],x[1]) ; if (resu < 1e-10) aux += -10; else aux += log(resu) ; } return -aux ; }

extern ”C” char * declspec(dllexport) CostFunctionName(void) { return (char *) NameWithUnderscore; }

extern ”C” unsigned int declspec(dllexport)

155

156

9 Statistical Estimation and Global Optimization

NumberOfParametersCF(void) { return 2; }

extern ”C” int declspec(dllexport) LimitsOfDomain(double *Left,double *Right) { Left[0] = -10 ; Right[0] = 10 ; Left[1] = 0 ; Right[1] = 10 ; return 2 ; }

9.3.2 Normal Distribution   1 1 2 φ (y) = √ exp − ((y − A)/B) 2 B 2π

(9.3)

This test was aimed at approximating the true mean (parameter A) and standard deviation (parameter B>0) of a normal distribution, given a training set of 100 iid samples, as shown in the histogram of figure 9.2(a). In figure 9.2(b), we can see a histogram obtained from 20000 samples of the normal distribution using the actual parameters (A=7, B=2), and in figure 9.2(c) we find the one corresponding to the estimated parameters (A=6.964106651898014, B=2.046661885014805). To obtain such results, the algorithm used 292605 evaluations of the objective function. The search intervals are shown in Table 9.2: Table 9.2 Search intervals used by Fuzzy ASA - normal distribution Parameter

Search interval

A B

[-10,10] [0,10]

9.3 Implementation and Experiments

(a) Histogram of training samples

157

(b) Histogram of 20000 samples (true parameters)

(c) Histogram of 20000 samples (estimated parameters)

Fig. 9.2 Histograms corresponding to normal distribution parameter estimation

9.3.3 Lognormal Distribution   1 1 2 φ (y) = √ exp − ((log(y) − A)/B) 2 By 2π

(9.4)

This test was aimed at approximating parameters A and B>0 of a lognormal distribution, given a training set of 200 iid samples, as shown in the histogram of figure 9.3(a). In figure 9.3(b), we can see a histogram obtained from 20000 samples of the referred type of distribution using the actual parameters (A=0.2, B=2), and in figure 9.3(c) we find the one corresponding to the estimated parameters (A=0.1621867681197146, B=2.020362941500447). Here, it was necessary to spend 541100 evaluations of the objective function. The search intervals are shown in Table 9.3:

158

9 Statistical Estimation and Global Optimization

Table 9.3 Search intervals used by Fuzzy ASA - lognormal distribution Parameter

Search interval

A B

[0,10] [0,10]

(a) Histogram of training samples

(b) Histogram of 20000 samples (true parameters)

(c) Histogram of 20000 samples (estimated parameters)

Fig. 9.3 Histograms corresponding to lognormal distribution parameter estimation

9.3.4 Cauchy Distribution φ (y) =

1 π B[1 + ((y − A)/B)2]

(9.5)

This test was aimed at approximating parameters A (location) and B>0 (scale) of a Cauchy distribution, given a training set of 200 iid samples, as shown in the histogram of figure 9.4(a). In figure 9.4(b), we can see a histogram obtained from

9.3 Implementation and Experiments

159

20000 samples using the actual parameters (A=4, B=0.5), and in figure 9.4(c) we find the one corresponding to the estimated parameters (A=4.001930721620335, B=0.5395334272165981 ). In this example, the method spent 867000 evaluations of the objective function. The search intervals are shown in Table 9.4.

Table 9.4 Search intervals used by Fuzzy ASA - Cauchy distribution Parameter

Search interval

A B

[-10,10] [0,10]

(a) Histogram of training samples

(b) Histogram of 20000 samples (true parameters)

(c) Histogram of 20000 samples (estimated parameters)

Fig. 9.4 Histograms corresponding to Cauchy distribution parameter estimation

160

9 Statistical Estimation and Global Optimization

9.3.5 Triangular Distribution 

φ (y) =

2(y−A) (B−A)(C−A) , y 2(B−y) (B−A)(B−C) , y




≥C

(9.6)

This test was aimed at approximating the parameters A, B and C of a triangular distribution (A<=B<=C and A
Search interval

A B C

[-1,10] [0,25] [0,10]

In this example the constrained optimization capabilities of ASA were tested by imposing penalties to candidate points not satisfying the conditions A≤B≤C and A
# include <windows.h> # include <stdio.h> # include <stdlib.h> # include <math.h> # include <exceptio.h> # include # include # include # include

9.3 Implementation and Experiments

(a) Histogram of training samples

161

(b) Histogram of 20000 samples (true parameters)

(c) Histogram of 20000 samples (estimated parameters)

Fig. 9.5 Histograms corresponding to triangular distribution parameter estimation

# include #include ”prob.h” // This file contains the iid samples (training set) #define SAMPLEFILE ”SAMPLES.DAT” #define MAXNOOFENTRIES 10000 char NameWithUnderscore[] = ” MLETriangle”; static int NoOfSamples ; double SampleArray[MAXNOOFENTRIES]; BOOL DllMain(HINSTANCE hInst,DWORD dwReason, LPVOID Reserved) { FILE *fp;

162

9 Statistical Estimation and Global Optimization

char sample[100] ; switch (dwReason) { case DLL PROCESS ATTACH: NoOfSamples = 0; fp = fopen(SAMPLEFILE,”r”); while ( fscanf(fp,”%s”,sample) != EOF && NoOfSamples < MAXNOOFENTRIES ) { SampleArray[NoOfSamples] = strtod(sample,NULL); NoOfSamples++ ; } fclose(fp); if (NoOfSamples == 0) return FALSE; break; case DLL PROCESS DETACH: break; case DLL THREAD ATTACH: break; case DLL THREAD DETACH: break; } return TRUE; }

extern ”C” double declspec(dllexport) MLETriangle ( int noofdimensions, double *x, void *Custom ) { double aux = 0 ; double resu ; for ( int i=0 ; i < NoOfSamples ; i++ ) { // Please, note that input vector x[] is generated by ASA in // such a way that x[0] corresponds to A, x[1] to B, and x[2] to C

9.3 Implementation and Experiments

resu = triangle pdf(SampleArray[i],x[0],x[1],x[2]) ; if (resu < 1e-10) aux += -10; else aux += log(resu) ; } //////////////////// Constraints ////////////////////////////// if ( x[0] > x[1]) aux -= 1000*(x[0]-x[1]) ; //// A has to be < B if (x[1] > x[2]) aux -= 1000*(x[1]-x[2]) ; //// B has to be < C if ( x[0] > x[2] ) aux -= 1000*(x[0]-x[2]) ; //// A has to be < C ////////////////////////////////////////////////////////////////

return -aux ; }

extern ”C” char * declspec(dllexport) CostFunctionName(void) { return (char *) NameWithUnderscore; }

extern ”C” unsigned int declspec(dllexport) NumberOfParametersCF(void) { return 3; }

extern ”C” int declspec(dllexport) LimitsOfDomain(double *Left,double *Right) { Left[0] = -1 ; Right[0] = 10 ; Left[1] = 0 ; Right[1] = 25 ; Left[2] = 0 ; Right[2] = 10; return 3 ; }

163

164

9 Statistical Estimation and Global Optimization

9.3.6 Mixture (Laplace and Uniform) Distribution   p |y − A| χ (C, D) exp − φ (y) = + (1 − p) 2B B D −C

(9.7)

where χ (C, D) is the characteristic function of the interval [C,D]. This test was aimed at approximating the five parameters (A,B,C,D,p) of a mixture distribution (Laplace(A,B) & Uniform(C,D) with blending proportion p), given a training set of 1000 iid samples, as shown in the histogram of figure 9.6(a). Here, the restrictions are B>0 , C
Search interval

A B C D p

[1,90] [0,5] [0,10] [0,10] [0,1]

In this example, once more, the constrained optimization capabilities of ASA were tested by imposing penalties to candidate points not satisfying the condition C
9.3.7 Gamma Distribution

φ (y) =

1 BΓ (C)



y−A B



C−1 exp

A−y B

 (9.8)

This test was aimed at approximating the parameters A, B and C of a gamma distribution (B > 0 and 0 < C ≤ 100), given a training set of 1000 iid samples, as shown in

9.3 Implementation and Experiments

(a) Histogram of training samples

165

(b) Histogram of 20000 samples (true parameters)

(c) Histogram of 20000 samples (estimated parameters)

Fig. 9.6 Histograms corresponding to a mixture distribution parameter estimation

the histogram of figure 9.7(a). In figure 9.7(b), we can see a histogram obtained from 20000 samples using the actual parameters (A=1, B=1, C=2), and in figure 9.7(c) we find the one corresponding to the estimated parameters (A=0.9973459425880535, B=0.8731245632506847, C=2.247411717671696). In this example, the method spent 110121 evaluations of the objective function. The search intervals are shown in Table 9.7. Table 9.7 Search intervals used by Fuzzy ASA - gamma distribution Parameter

Search interval

A B C

[0.1, 20] [0.1, 5] [0.1, 10]

166

9 Statistical Estimation and Global Optimization

(a) Histogram of training samples

(b) Histogram of 20000 samples (true parameters)

(c) Histogram of 20000 samples (estimated parameters)

Fig. 9.7 Histograms corresponding to gamma distribution parameter estimation

9.4 Conclusions Considering the difficulties related to the application of conventional optimization techniques to maximum likelihood estimation problems when the distributions under study give rise to highly complex likelihood functions, it was shown that, by using a global optimization technique, it is possible to get to good ML estimates in a systematic and uniform fashion, independently of specific or particular characteristics presented by functional expressions of probability density functions under investigation. Besides, histograms corresponding to several probability distribution functions were given so as to illustrate the quality of estimated parameters. Also, source code extracts were presented to illustrate practical aspects of using generic ASA code to obtain maximum likelihood estimates.

References

167

References 1. Blatt, D., Hero, A.O.: On Tests for Global Maximum of the Log-Likelihood Function. IEEE Transactions on Information Theory 53(7), 2510–2526 (2007) 2. Brooks, S.P., Friel, N., King, R.: Classical model selection via simulated annealing. J.R. Statist. Soc. B 65(2), 503–520 (2003) 3. Fang, K.T., Hickernell, F.J., Winker, P.: Some global optimization algorithms in Statistics. In: Du, D.Z., Zhang, X.S., Cheng, K. (eds.) Lecture Notes in Operations Research. K. World Publishing Corporation (1996) 4. Ingber, L.: Adaptive simulated annealing (ASA): Lessons learned. Control and Cybernetics 25(1), 33–54 (1996) 5. Jerrel, M.E., Campione, W.A.: Global Optimization of Econometric Functions. Journal of Global Optimization 20, 273–295 (2001) 6. Myung, I.J.: Tutorial on maximum likelihood estimation. Journal of Mathematical Psychology 47, 90–100 (2003) 7. Oliveira Jr, H.: Fuzzy control of stochastic global optimization algorithms and VFSR. Naval Research Magazine 16, 103–113 (2003) 8. Pachter, R., Wang, Z.: Adaptive Simulated Annealing and its Application to Protein Folding. In: Floudas, C.A., Pardalos, P.M. (eds.) Encyclopedia of Optimization. Springer, Heidelberg (2009) 9. Rosen, B.: Function optimization based on advanced simulated annealing. In: IEEE Workshop on Physics and Computation - Phys. Comp.1992, pp. 289–293 (1992) 10. Severini, T.A.: Likelihood Methods in Statistics. Oxford University Press (2000)

Chapter 10

Nonlinear Equation Solving

Abstract. This chapter introduces a global optimization approach for finding solutions of nonlinear systems of functional equations using Fuzzy ASA. The original problem is transformed into a global optimization one by synthesizing objective functions whose global minima, if any, are also solutions to the original system. The global minimization process is triggered from different starting points so as to find as many solutions as possible. To demonstrate its utility, the method is applied to several types of equations, presenting very good results. The equation systems are composed of n equations on n-dimensional Euclidean spaces.

10.1 Introduction Finding solutions of nonlinear systems of equations has been a serious concern of modern Applied Mathematics [2, 7, 8, 9, 12]. One good reason for this is the importance of the subject in Engineering, Biology , Chemistry, Physics and Mathematics itself [3, 4]. Unfortunately, although several methods for solving systems of linear equations exist, results about finding closed form solutions for nonlinear systems of equations are rare, and, sometimes, too specific to be useful in practice. On the other side, we have very elegant and useful results such as the fixed point theorem of S. Banach [1], that allows us to devise methods for finding approximate solutions for a significant class of nonlinear systems, although it could be hard to assure that certain systems are compatible with its hypothesis. Equations could be described as expressions of correspondences between elements of given spaces by means of operators defined on those spaces, be them, for example, finite dimensional Euclidean or infinite dimensional function spaces. Using properties of the involved transformations (defining the particular correspondences and properties of the underlying state spaces), one try to find conditions under which solutions exist and, in that case, how to get them in closed or approximate form. Solving simultaneous systems of equations involves finding elements of the state space satisfying all component equations simultaneously - in our case, vectors of the n-dimensional Euclidean space Rn . In this fashion, considering that H.A. e Oliveira Jr. et al.: Stochastic Global Optimization & Its Applications, ISRL 35, pp. 169–187. c Springer-Verlag Berlin Heidelberg 2012 springerlink.com 

170

10 Nonlinear Equation Solving

closed form solutions are hard to find in the case of nonlinear equations systems, algorithms that can discover solutions efficiently and precisely are welcome, especially in situations presenting non-differentiable or even discontinuous operators, when traditional methods are even scarcer. In the literature [3, 4] researchers describe a significant number of methods for solving numerically the problems and propose new methods. At present, there is a very interesting line of research that is based upon transforming a system of nonlinear equations into a multi-objective optimization problem. The transformed problem is then solved by means of Pareto dominance relationship between candidate solutions and an iterative strategy, evolving the respective populations towards better and (hopefully) optimal solutions. Such techniques use principles of evolutionary computing and are able to handle very general systems, not demanding additional conditions on the component functions, such as differentiability or continuity, for instance, differently from most common paradigms. Such works represent a modern trend of investigation and there is a significative number of applications of multiobjective programming to specific problems, although only a few methods for solving general systems of nonlinear equations using computational intelligence techniques seem to be publicly known until now. Following another line of action, in this chapter we are going to show how to approach the same kind of problems by using a stochastic global optimization algorithm - Fuzzy ASA, of course [6, 10, 11].

10.2 Statement of the Problem The general problem we are concerned with in this chapter is that of finding solutions to systems of simultaneous nonlinear equations of the form ⎛ ⎞ f1 (x) ⎜ f2 (x)⎟ ⎜ ⎟ f (x) = ⎜ . ⎟ = 0 (10.1) ⎝ .. ⎠ fn (x) where 0, x ∈ R and n

f : Ω → Rn , fi : Ω → R Δ

n

Ω = ∏[ai , bi ] ⊂ Rn , ai < bi i=1

x = (x1 , x2 , ..., xn ) 0 = (0, 0, 0, ..., 0) f1 , f2 , ..., fn are nonlinear functions in the class of all real valued (not necessarily continuous) functions on Ω . At least one of the equations should be nonlinear. A

10.3 The Algorithm

171

solution for a nonlinear system of equations is, by definition, a simultaneous solution for every equation appearing in the system, what means f1 (s1 , s2 , ..., sn ) = f2 (s1 , s2 , ..., sn ) = ... = fn (s1 , s2 , ..., sn ) = 0

(10.2)

for some vector (s1 , s2 , ..., sn ) ∈ Rn . Unfortunately, it is not always possible to find closed form, exact solutions for nonlinear equations, in general. From this fact follows the need to synthesize numerical methods capable of finding accurate approximations for their roots. This setting gets worse whenever functional relationships present in the component equations lack crucial attributes, like continuity or differentiability, taking into account that Newton-like/gradient based methods become unfeasible, given the impossibility to evaluate derivatives, for instance. Here, we present an approach that handles the problem by means of a global optimization method, facing simultaneous equations satisfaction as a multi-objective minimization task (each equation should be satisfied) and solving it by the minimization of a single objective function that encapsulates all the constraints and has a lower bound of 0. Such a function is the sum of the absolute values of left sides of component equations, as we will see below.

10.3 The Algorithm To find approximate solutions for system (10.1) we will use the following algorithm: • Step 1 – Transform the original problem into a global optimization one, by taking as the objective function F(x), the sum of the absolute values of component functions fi , and finding the global minima of Δ

n

F(x) = ∑ | fi (x)| x ∈ Ω

(10.3)

i=1

Δ

n

Ω = ∏[ai , bi ] ⊂ Rn , ai < bi

(10.4)

i=1

• Step 2 – Submit F(x) to the Fuzzy ASA algorithm, aiming at finding different solutions to the new global optimization problem defined by (10.3). Here, it is possible to simply run the algorithm several times or use its multistart version, taking into account that, as a stochastic method, Fuzzy ASA is able to explore different regions during different activations. • Step 3 – If sufficient solutions (presenting global error F(x) less than a predefined value) were found, stop. Otherwise, repeat step 2 until satisfaction or reaching a certain stopping criterion (for instance, reaching a maximum number of trials).

172

10 Nonlinear Equation Solving

The reasoning underlying the algorithm is extremely simple: if it is possible to minimize the sum of absolute errors of all component equations F(x) below a certain desired level, then each individual error is also below that given threshold. So, once chosen the acceptable precision beyond which a given vector is considered a solution for system (10.1), solving problem (10.3) is equivalent to solving the original system of equations (10.1). Obviously, it is also possible to place additional restrictions over individual equations by modifying the cost function F(x) and forcing some (or all) equations to satisfy stricter conditions. In any case, the main task is to effectively minimize the function defined by (10.3) within a given precision, considered adequate for the problem at hand. So, the quality of the algorithm is strongly based on the minimization power of the employed global optimization algorithm (Fuzzy ASA, in the present case), particularly in its ability to face arbitrarily complex landscapes of highly nonlinear cost functions.

10.4 Examples In this section we present solutions for several examples of nonlinear systems, demonstrating the efficiency and effectiveness of the suggested method. To get a better assessment, we are going to use problems of several sources ([3, 4], for example), that (we hope) represent good references for evaluation. It is worth alerting the reader that, due to differences between implementations of numerical runtime libraries corresponding to the several programming environments available at present, it is possible for a particular numerical solution to produce slightly different functional results, when submitted to distinct computer programs, coded in different languages.

10.4.1 Example 1 The problem is to solve the following system [4] ⎧ ⎪ ⎪ ⎪2x1 + x2 + x3 + x4 + x5 − 6 = 0 ⎪ ⎪ ⎪ ⎨x1 + 2x2 + x3 + x4 + x5 − 6 = 0 x1 + x2 + 2x3 + x4 + x5 − 6 = 0 ⎪ ⎪ ⎪x1 + x2 + x3 + 2x4 + x5 − 6 = 0 ⎪ ⎪ ⎪ ⎩x x x x x − 1 = 0 1 2 3 4 5

(10.5)

By applying the proposed method we were able to find two solutions: V1 = (−0.5790430884941372, −0.579043088494136, −0.5790430884940641, − 0.5790430884941251, 8.895215442470592)

10.4 Examples

173

with error = 8.406003757649438412e − 14 (it spent 3,134,277 cost function evaluations to get the final solution) and V2 = (1.000000091483145, 1.000000091464961, 1.000000091683518 , 1.000000091688527, 0.9999995421100667) with error = 9.19938387201110069400e − 08 (it spent 31,396,047 cost function evaluations to converge).

The search intervals were [-100,200] for all dimensions and in all experiments, and the evolution of the minimization of the merit function for the two solutions can be seen in Figures 10.1 and 10.2.

Fig. 10.1 Merit function minimization evolution corresponding to V1 - example 1

To complement the example, we will list the code implementing the merit function

#include <windows.h> #include <math.h>

174

10 Nonlinear Equation Solving

Fig. 10.2 Merit function minimization evolution corresponding to V2 - example 1

#include <malloc.h> #include <stdio.h> char NameWithUnderscore[] = ” Ex1”; #define DIMENSION 5 BOOL DllMain(HINSTANCE hInst, DWORD dwReason, LPVOID Reserved) { switch (dwReason) { case DLL PROCESS ATTACH: case DLL PROCESS DETACH: case DLL THREAD ATTACH: case DLL THREAD DETACH: break; } return TRUE;

10.4 Examples

175

} extern ”C” double declspec(dllexport) Ex1( int NoOfDimensions, double *Vector, void *Custom ) { int i; double sum1 , sum2 , sum3 , sum4 , sum5 ; sum1 = fabs( 2*Vector[0]+Vector[1]+Vector[2]+Vector[3]+Vector[4]-6 ) ; sum2 = fabs( Vector[0]+2*Vector[1]+Vector[2]+Vector[3]+Vector[4]-6 ) ; sum3 = fabs( Vector[0]+Vector[1]+2*Vector[2]+Vector[3]+Vector[4]-6 ) ; sum4 = fabs( Vector[0]+Vector[1]+Vector[2]+2*Vector[3]+Vector[4]-6 ) ; sum5 = fabs( Vector[0]*Vector[1]*Vector[2]*Vector[3]*Vector[4]-1 ) ; return sum1+ sum2 + sum3 + sum4 + sum5 ; } extern ”C” char * declspec(dllexport) CostFunctionName(void) { return (char *) NameWithUnderscore; } extern ”C” unsigned int declspec(dllexport) NumberOfParametersCF(void) { return DIMENSION ; } extern ”C” int declspec(dllexport) LimitsOfDomain(double *Left,double *Right) { for (int i=0; i < DIMENSION ; i++) { Left[i] = -100 ; Right[i] = 200 ; } return DIMENSION ; }

176

10 Nonlinear Equation Solving

10.4.2 Example 2 The problem is to solve the following system [4]  10000x1x2 − 1 = 0 e−x1 + e−x2 − 1.001 = 0

(10.6)

By applying the proposed method we have found the solutions: V1 = (21.57315449146697, 4.635390713933557e − 06) with error = 1.00463495250795176170e − 03 and search region [0,25]x[0,2] (it spent 115,036,256 cost function evaluations to get the final solution) and V2 = (7.679406480976657, 1.302183967338437e − 05) with error = 5.50772763563226798e − 04 and search region = [0,50]x[0,2] (it spent 162,561,656 cost function evaluations to converge).

The evolution of the minimization of the merit function for the solution V2 can be seen in Figure 10.3. Complementing the example, we give the source code implementing the merit function, as follows

#include <windows.h> #include <math.h> #include <malloc.h> #include <stdio.h> char NameWithUnderscore[] = ” Ex2”; #define DIMENSION 2 BOOL DllMain(HINSTANCE hInst, DWORD dwReason, LPVOID Reserved) { switch (dwReason) {

10.4 Examples

Fig. 10.3 Merit function minimization evolution corresponding to V2 - example 2

case DLL PROCESS ATTACH: case DLL PROCESS DETACH: case DLL THREAD ATTACH: case DLL THREAD DETACH: break; } return TRUE; } extern ”C” double declspec(dllexport) Ex2( int NoOfDimensions, double *Vector, void *Custom ) { int i; double sum1 , sum2 ; sum1 = fabs( 10000*Vector[0] * Vector[1] -1 ) ; sum2 = fabs( exp(-Vector[0]) + exp(-Vector[1]) -1.001 ) ;

177

178

10 Nonlinear Equation Solving

return sum1+ sum2 ; } extern ”C” char * declspec(dllexport) CostFunctionName(void) { return (char *) NameWithUnderscore; } extern ”C” unsigned int declspec(dllexport) NumberOfParametersCF(void) { return DIMENSION ; } extern ”C” int declspec(dllexport) LimitsOfDomain(double *Left,double *Right) { Left[0] = Left[1] = 0 ; Right[0] = 50 ; Right[1] = 2 ;

return DIMENSION ; }

10.4.3 Example 3 The problem is to solve the following system [4]  0.25 π x2 + 0.5x1 − 0.5sin(x1x2 ) = 0 e 0.25 2x1 − e) = 0 π x2 − 2ex1 + (1 − π )(e

(10.7)

By applying the proposed method we have found the solutions V1 = (0.5, 3.141592653589793) with error = 1.09162207588998601800e − 16 and search region [-25,25]x[10,20] (it took 30,000 cost function evaluations to get to the final solution)

10.4 Examples

179

and V2 = (2.994489808124304e − 01, 2.836928465564936) with error = 1.393144978885402e − 07 and search region [0.25,1]x[1.5,2 π] (it took 32,834,082 cost function evaluations to get to the final solution). The evolution of the minimization process for the solution V1 can be seen in Fig. 10.4.

Fig. 10.4 Merit function minimization evolution corresponding to the solution V1 of example 3

It is easy to see that the solution V1 is an approximation for (0.5, π ). Here goes the source code for example 3

#include <windows.h> #include <math.h>

180

10 Nonlinear Equation Solving

#include <malloc.h> #include <stdio.h> char NameWithUnderscore[] = ” Ex3”; #define DIMENSION 2 BOOL DllMain(HINSTANCE hInst, DWORD dwReason, LPVOID Reserved) { switch (dwReason) { case DLL PROCESS ATTACH: case DLL PROCESS DETACH: case DLL THREAD ATTACH: case DLL THREAD DETACH: break; } return TRUE; } extern ”C” double declspec(dllexport) Ex3( int NoOfDimensions, double *Vector, void *Custom ) { int i; double sum1 , sum2 ; sum1 = fabs( (0.25/M PI)*Vector[1] + 0.5*Vector[0] - 0.5*sin( Vector[0]*Vector[1] ) ) ; sum2 = fabs( (M E / M PI)* Vector[1] - 2*M E*Vector[0] + (1-0.25/M PI)*(exp(2*Vector[0])-M E) ) ;

return sum1+ sum2 ; } extern ”C” char * declspec(dllexport) CostFunctionName(void) { return (char *) NameWithUnderscore;

10.4 Examples

181

} extern ”C” unsigned int declspec(dllexport) NumberOfParametersCF(void) { return DIMENSION ; } extern ”C” int declspec(dllexport) LimitsOfDomain(double *Left,double *Right) { Left[0] = 0.25 ; Right[0] = 1 ; Left[1] = 1.5 ; Right[1] = 2*M PI ;

return DIMENSION ; }

10.4.4 Example 4 The problem is to solve the following system [4]  4x31 + 4x1 x2 + 2x22 − 42x1 − 14 = 0 4x32 + 2x21 + 4x1 x2 − 26x2 − 22 = 0

(10.8)

The search region is [-25,250]x[-100,20] and the found solutions are shown in Table 10.1. The evolution of the minimization of the merit function for a typical solution can be seen in Fig. 10.5.

10.4.5 Example 5 The problem is to solve the following system ⎧ x1 −10 10 ⎪ 1+ γ x1 ⎨x − (1 − R)( D − x )e =0 1 1 10(1+β1) x2 −10 10 ⎪ ⎩ 1+ γ x2 D x1 − (1 + β2)x2 + (1 − R)( 10 − β1 x1 − (1 − β2)x2 )e =0

(10.9)

182

10 Nonlinear Equation Solving

Table 10.1 Solutions for example 4 Point

Global error

No of function evaluations

(-1.279613467306801e-01, -1.953714980244576)

1.74339709335669113e-15

774,000

(-2.805118086952745, 3.131312518250573)

7.133182933216631e-15

1,479,000

(3,2)

0

1,834,308

(-0.2708445906673476, -0.9230385564799815)

3.070460552478949e-16

1,689,000

(3.584428340330492, -1.848126526964404)

2.869232629265639e-15

2,626,303

(0.08667750455539627, 2.884254701174776)

1.755193212993333e-14

5,192,525

(-3.779310253377747, -3.28318599128617)

1.747213485003840e-14

1,830,000

(-3.07302575076439, -0.0813530442879675)

4.007211229506424e-15

7,725,739

(3.385154183607749, 0.07385187983270787)

7.255659267846326e-11

2,448,341

where D = 22, β1 = 2, β2 = 2, R = 0.935, γ = 1000. By applying the proposed method we have found the solution V1 = (0.03280365031638163, 0.04194069461727134) with error = 3.984442983884229e − 18 and search region [-25,25]x[-10,20] (it took 42,000 cost function evaluations to get to the final solution). The evolution of the minimization of the merit function for the solution can be seen in Fig. 10.6.

10.4.6 Example 6 The problem is to solve the following system [5] ⎧ x1 10 10 ⎪ 1+ γ x1 ⎨x − (1 − R)( D − x )e =0 1 1 10(1+β1) x2 ⎪ ⎩x − (1 + β )x + (1 − R)( D − β x − (1 + β )x )e10 1+ 10γ x2 = 0 1 2 2 1 1 2 2 10

(10.10)

10.4 Examples

183

Fig. 10.5 Merit function minimization evolution corresponding to a typical solution - example 4

where D = 22, β1 = 2, β2 = 2, R = 0.935, γ = 1000. Although this system of equations is very similar to that of previous example, its solutions are considered hard to find and it concerns a model of two continuous non-adiabatic stirred tank reactors. These reactors are in series, at steady state with a recycle component, and have an exothermic first-order irreversible reaction [5]. After simplified, the model becomes a system of two nonlinear equations, as described above. x1 and x2 represent the dimensionless temperatures in the two reactors and are bounded to the unit interval [0,1]. By applying the proposed method we have found the solution V1 = (0.7249868948022162, 0.2452408205983374) with error = 3.51161648723266953100e − 15 and search region [0,1]x[0,1] (it took 1,026,000 cost function evaluations to get to the final solution). The evolution of the minimization of the merit function for the solution can be seen in Fig. 10.7.

184

10 Nonlinear Equation Solving

Fig. 10.6 Merit function evolution corresponding to the solution of example 5

10.4.7 Example 7 The problem is to solve the following system [5]  −sin(x1 )cos(x2 ) − 2cos(x1 )sin(x2 ) = 0 −cos(x1 )sin(x2 ) − 2sin(x1 )cos(x2 ) = 0

(10.11)

We have found the solutions shown in Table 10.2 The evolution of the minimization of a typical merit function can be seen in Fig. 10.8 ( the search region is [0, 2π ]x[0, 2π ] ⊂ R2 ).

10.5 Conclusions Fuzzy ASA method has been effective in solving several nonlinear systems of functional equations, finding accurately several solutions for them. Taking into account that, in practice, we find numerous cases of nonlinear systems not solvable analytically (at least, not with known closed form solution), such a kind of tool can be valuable for researchers of virtually all areas of science and technology.

10.5 Conclusions

185

Fig. 10.7 Merit function minimization evolution corresponding to the solution of example 6 Table 10.2 Solutions for example 7 Point

Global error

No of function evaluations

( 1.228336034002354e-16, 6.283185307179586 )

3.677548741647109e-16

51392

( 4.712388980384658, 1.570796326794961 )

9.604572996299576e-14

102793

( 6.283185307179586, 1.224630856454763e-16 )

3.673843564099518e-16

154191

( 3.141592653589777, 1.230243531215396e-14 )

2.841210750213845e-14

205580

( 6.283185307179586, 4.406816274589072e-16 )

6.856028982233827e-16

256978

( 4.712388980384689, 4.712388980384691 )

1.776356839400250e-15

411160

( 3.141592653589792, 3.141592653589794 )

1.776356839400250e-15

668138

( 6.283185307179587, 6.283185307179586 )

1.195007634513523e-15

770949

186

10 Nonlinear Equation Solving

Fig. 10.8 Merit function minimization evolution corresponding to a typical solution of example 7

References

187

References 1. Edgar, G.: Measure, Topology, and Fractal Geometry. Springer, Heidelberg (2008) 2. Effati, S., Nazemi, A.R.: A new method for solving a system of the nonlinear equations. Appl. Math. Comput. 168(2), 877–894 (2005) 3. Grosan, C., Abraham, A.: A new approach for solving nonlinear equations systems. IEEE Transactions on Systems, Man and Cybernetics - Part A: Systems and Humans 38(3), 698–714 (2008) 4. Grosan, C., Abraham, A.: Multiple solutions for a system of nonlinear equations. International Journal of Innovative Computing, Information and Control 4(9), 2161–2170 (2008) 5. Hirsch, M.J., Pardalos, P.M., Resende, M.G.C.: Solving systems of nonlinear equations with continuous GRASP. Nonlinear Analysis: Real World Applications 10, 2000–2006 (2009) 6. Ingber, L.: Adaptive simulated annealing (ASA): Lessons learned. Control and Cybernetics 25(1), 33–54 (1996) 7. Moore, R.E.: Methods and Applications of Interval Analysis. SIAM, Philadelphia (1979) 8. Morgan, A.P.: Computing all solutions to polynomial systems using homotopy continuation. Appl. Math. Comput. 24(2), 115–138 (1987) 9. Morgan, A.P.: Solving Polynomial Systems Using Continuation for Scientific and Engineering Problems. Prentice-Hall, Englewood Cliffs (1987) 10. Oliveira Jr., H.: Fuzzy control of stochastic global optimization algorithms and VFSR. Naval Research Magazine 16, 103–113 (2003) 11. Pachter, R., Wang, Z.: Adaptive Simulated Annealing and its Application to Protein Folding. In: Floudas, C.A., Pardalos, P.M. (eds.) Encyclopedia of Optimization. Springer, Heidelberg (2009) 12. Verschelde, J., Verlinden, P., Cools, R.: Homotopies exploiting Newton polytopes for solving sparse polynomial systems. SIAM J. Numer. Anal. 31(3), 915–930 (1994)

Chapter 11

Space-Filling Curves and Fuzzy ASA

Abstract. This chapter introduces a multi-start global optimization algorithm that uses dimensional reduction techniques based upon approximations of space-filling curves and fuzzy adaptive simulated annealing, aiming at finding global minima of real-valued (possibly multimodal) functions that are not necessarily well behaved, that is, are not required to be differentiable, continuous, or even satisfying Lipschitz conditions. The overall idea is as follows: given a real-valued function with a multidimensional and compact domain, the method builds an equivalent, onedimensional problem by composing it with a space-filling curve, searches for a small group of candidates and returns to the original higher-dimensional domain, this time with a small set of promising starting points. In this fashion, it is possible to overcome difficulties related to capture in inconvenient attraction basins and, simultaneously, to bypass the complexity associated to finding the global minimum of the auxiliary one dimensional problem, whose graph is typically fractal-like, as we shall see along the chapter. New space-filling curves are built with basis on the well-known Sierpi´nski SFC, a subtle modification of a theorem by Hugo Steinhaus and several results of Ergodic Theory.

11.1 Introduction A number of different techniques for global optimization of numerical functions based upon space-filling curves or its approximations have been proposed in the literature in recent years [2, 3, 8, 10]. One common characteristic shown in those contributions is that the functions under study have certain regularity properties, such as being of Lipschitz type or differentiable. If those properties are not satisfied, or we cannot prove whenever they are, the problem at hand cannot benefit from the corresponding method. It is thus of interest to devise a way of handling such limitations as well. Another situation that occurs frequently is related to the poor precision attained by some methods whenever the domain dimension gets higher such a fact is related to the difficulty in minimizing the resulting one-dimensional H.A. e Oliveira Jr. et al.: Stochastic Global Optimization & Its Applications, ISRL 35, pp. 189–201. c Springer-Verlag Berlin Heidelberg 2012 springerlink.com 

190

11 Space-Filling Curves and Fuzzy ASA

auxiliary functions that exhibit a large number of local minima and very ”noisy” graphs. Considering that the great majority, if not all, of global minimization methods fail in such extreme situations, such an approach is, in some cases, considered to be ineffective. The reasons for this are related to the way the multidimensional domain is filled and to the quality of the one-dimensional minimization process applied to the resulting auxiliary function. To be successful in practice, the SFC based global optimization approach should satisfy, at least, the following two conditions: (i) to offer a good approximation of a space-filling curve whose image is, or contains, the compact domain of the function under study; (ii) to use an onedimensional global minimization algorithm that can find precise approximations to optimizing points of the composed map, whose domain is a closed interval, say [0,1], and that assumes real values, possessing the same extremes as the original function, among which the desired global optimum is included. Unfortunately, such conditions are not easily satisfied and past efforts were only partially successful in finding good results. In [2, 3], for example, an interesting and promising paradigm is presented, but the results shown therein focus on low dimensional spaces. Besides, the objective function must satisfy Lipschitz conditions, property that can be difficult to prove or disprove in many situations. Another issue is related to the adequacy of the chosen filling of the original multidimensional domain - by projecting the image of the presented approximation of the space-filling curve in the space generated by vectors corresponding to directions that have larger variance by means of principal component analysis (PCA), we can show that there are poorly visited regions, inside which extreme points could be located. In fact, although there are several SFCs filling up multidimensional domains, at least in theory, the respective numerical approximations can exhibit ”holes” because of truncation and/or limitations of present day digital computers. This behavior becomes more important as the number of dimensions increases. An example of this phenomenon occurs with the Schoenberg or Iseki SFCs (see [13] for the corresponding definitions). To find ways of obtaining other space-filling curves capable of overcoming such inconvenience, measure-preserving transformations and important theorems of General Topology and Ergodic Theory were used as inestimable tools [1, 4, 5]. From this point on it is assumed, without loss of generality, that all optimization problems are related to unconstrained global minimization of real functions and all space-filling curves have the unitary interval [0,1] as their domain.

11.2 Key Results from General Topology, Ergodic and Measure Theories We can define a space-filling curve as a surjective and continuous function from a real interval, say [0,1], to a compact subset of a finite dimensional vector space, which can be identified to Rn , the n-dimensional Euclidean space. Spacefilling curves were well-studied in the past and there are many theoretical results stating necessary conditions for their existence [13]. Besides, long before the

11.2 Key Results from General Topology, Ergodic and Measure Theories

191

invention of digital computers, a number of great mathematicians, for example, Hilbert, Cantor, Lebesgue, Peano and Sierpi´nski, proposed constructive examples and established several interesting properties [13] of SFCs. However, the application of all that mathematical apparatus to real world problems seemed improbable at the occasion. More recently, researchers have found ways to apply previous knowledge about space-filling curves to several important areas, including global optimization of numerical functions, indexing in database management systems and antenna design [8]. Here, the basic idea is to compose a given objective function with a SFC corresponding to a compact superset of the respective domain. This way it is possible to reduce a multivariate problem to a univariate one. Hence, at least in theory, it would be possible, by solving the auxiliary one-dimensional problem, to go back to the n-dimensional domain and retrieve the desired optimum point. However, as such ideas are implemented by means of digital computers, some complications arise, particularly whenever the number of dimensions increases. The main drawback concerning practical issues is that virtually all those curves idealized in the past did not take into account the finite word length of present day digital computers. Typically, the proposed space-filling curves were based on infinite expansions and used, for instance, the property that elements in [0,1] can be represented as 0.t1t2 t3 t4 ... in a given basis B, where each ti is an integer between 0 and B-1 (extremes included). In this fashion, it is necessary to find adequate approximations of SFCs if we want to pursue this kind of approach. Initially, a sensible alternative seems to be the use of Sierpi´nski SFCs, taking into account the availability of their precise defining formulas, as follows [13]: x(t) = f(t), y(t) = f(t − 1/4), t ∈ [0, 1]

(11.1)

where f is a bounded, even and continuous real function whose expression is given by Θ(t) Θ(t)Θ(τ1 (t)) Θ(t)Θ(τ1 (t))Θ(τ2 (t)) f (t) = − + − ... (11.2) 2 4 8 The 1-periodic functions Θ(t) and τk (t) are defined in [13], so that the resulting curve (x(t), y(t)) is a 2-dimensional SFC and shows to be well-suited to numerical calculations. To build higher-dimensional SFCs starting from this one, some results were crucial, as stated by the following theorems, whose proofs can be found in [13]. At this point, it is necessary to present some definitions. Definition 11.1. A function ϕ : [0, 1] → R is uniformly distributed with respect to the Lebesgue measure if, for any (Lebesgue) measurable set A ⊂ R, we have

Λ1 (ϕ −1 (A)) = Λ1 (A)

(11.3)

where Λ1 is the Lebesgue measure in the real line. Definition 11.2. n measurable functions ϕ1 , ϕ2 , ..., ϕn : [0, 1] → R are stochastically independent with respect to Lebesgue measure if, for any n measurable sets A1 , A2 , ..., An ⊂ R,

192

11 Space-Filling Curves and Fuzzy ASA

Λ1 (

n  j=1

n

−1 ϕ −1 j (A j )) = ∏ Λ1 (ϕ j (A j ))

(11.4)

j=1

Theorem 11.1. (H. Steinhaus) If ϕ1 , ϕ2 , ..., ϕn : [0, 1] → R are continuous, nonconstant and stochastically independent with respect to the Lebesgue measure, then f = (ϕ1 , ϕ2 , ..., ϕn ) : [0, 1] → ϕ1 ([0, 1]) × ϕ2 ([0, 1]) × ... × ϕn([0, 1])

(11.5)

is a SFC. Theorem 11.2. If f = (ϕ , ψ ) : [0, 1] → [0, 1] × [0, 1] is (Lebesgue) measurepreserving and onto, then its coordinate functions ϕ , ψ are uniformly distributed and stochastically independent. Taking into account that the Sierpi´nski SFC is measure-preserving ([13], page 111), we conclude that its coordinate functions are uniformly distributed and stochastically independent, and can be used to synthesize higher-dimensional SFCs with coordinates ⎧ ⎪ x1 (t) = ϕ (t) ⎪ ⎪ ⎪ ⎪ ⎪ ⎨x2 (t) = ϕ ◦ ψ (t) (11.6) ...................................... ⎪ ⎪ ⎪ xn (t) = ϕ ◦ ψ ◦ ψ ◦ · · · ◦ ψ (t) ⎪ ⎪ ⎪ ⎩where t ∈ [0, 1] that are non-constant, continuous and stochastically independent. Unfortunately, after a few experiments it was clear that, for higher-dimensional domains (around 8 dimensions), approximations of such “pure” Sierpi´nski based SFCs failed to fill up adequately the compact domains of interest, as will be shown in the sequel. This fact is due to distortions caused by numerical approximations, despite the theoretical curve being really a space-filling one. In order to find a more robust curve, we can compose the original Sierpi´nski function with an invertible (Lebesgue) measure-preserving transformation that is a natural extension of a particular generalized L¨uroth series transformation [5], mapping [0,1)x[0,1) onto itself. To generate a new SFC mapping [0,1] onto [-1,1]x[-1,1], it was necessary to use linear homeomorphisms from [-1,1]x[-1,1] to [0,1]x[0,1] and vice-versa. Let us denote such a transformation by T and define it as 1 y Δ T (x, y) = ((k − 1)(kx − 1), (1 + )), x ∈ Ik , y ∈ [0, 1) k k−1 1 Δ 1 Ik = [ , ), k ∈ {2, 3, ...} k k−1 To build the proposed SFC we applied the following sequence of operations:

(11.7)

11.2 Key Results from General Topology, Ergodic and Measure Theories

193

[0, 1] → [−1, 1]× [−1, 1] → [0, 1]× [0, 1] → [0, 1]× [0, 1] → [−1, 1]× [−1, 1] (11.8) Sierpi´nski

Homeomorphism

τ

Homeomorphism

It should also be observed that although T was initially defined only in [0,1)x[0,1), it was extended to [0,1]x[0,1] in an obvious way, so that the composite path can reach all regions of the desired domain. For the sake of assessing the filling degree of the curves relatively to the compact set [−1, 1]n, we present in Figs. 11.1 and 11.2 the plots produced by PCA projections of the corresponding multidimensional curves onto the two maximum variance directions. Qualitatively, it can be stated that the more filled the graph area is, the more adequate the filling curve will be. The generated paths consisted of 20,000 8-dimensional points of each kind of curve, and conventional PCA was carried out through a customized computer program. The parameterizing domain was chosen as the closed interval [0,1] in all cases.

Fig. 11.1 Projection of original Sierpi´nski curve points (20,000 samples)

As can be seen from the plot in Fig. 11.1, the projected points of pure Sierpi´nski curves did not fill adequately the maximum variance region. The composed transformation, in Fig. 11.2, presented substantially better performance, showing denser and more uniform covering. Going ahead, let us now see what happens to the graph of the function f (x1 , x2 ) = (x1 − 0.3)2 + (x2 − 0.3)2 , restricted to the square [-1,1]x[-1,1], when it is composed with the proposed space-filling curve, yielding a

194

11 Space-Filling Curves and Fuzzy ASA

Fig. 11.2 Projection of modified space-filling curve (20,000 sample points)

univariate, real-valued function with compact domain [0,1]. The result of plotting the function values at 1000 uniformly spaced points in the interval [0,1] is displayed in Fig. 11.3, from which we conclude that a simple dimensional reduction from 2 to 1 transforms a smooth surface into the very rough line g(t) = f (ϕ1 (t), ϕ2 (t)). This phenomenon tends to worsen as the original number of dimensions increases. Then, at a first glance, we could interpret experimental data by saying that the order in which the SFC visits points belonging to the compact domain causes a kind of disorder in the resulting graphs, transforming simple higher-dimensional optimization problems into very difficult one-dimensional optimization tasks, and hence that such a method would be ineffective. A deeper analysis, however, shows that such is not the case, particularly in the global minimization of multimodal and high-dimensional functions, considering the absence of general effective deterministic methods for finding even the approximate location of the attraction basins corresponding to globally minimizing points. By carefully observing Fig. 11.3, we notice that there are regions where the maximum (3.38) and minimum (0) values of f in [-1,1]x[-1,1] are visibly approximated, and could serve as hints to start more effectively preexisting and efficient global

11.2 Key Results from General Topology, Ergodic and Measure Theories

195

Fig. 11.3 Graph of the reduced function

optimization algorithms, be them stochastic or deterministic. Returning to the above example function we find that for t = 0.589 the 1-dimensional function reaches the value 0.0001201, which is the minimum of the 1000 points used in the illustrative discretization and an approximation for the actual 0 value, corresponding to (0.3, 0.3). Going back to the two-dimensional domain, we find (0.2907, 0.2942) as the corresponding bi-dimensional point, which is reasonably close to the actual minimizer, specially in view of such a coarse discretization. On the other side, for t=0.425, we find the function value 3.38, which is exactly the maximum of the function in [-1,1]x[-1,1]. The corresponding bi-dimensional point is (-1,-1). Another subtle but important issue is that although Steinhaus’ theorem is stated only for continuous functions, it holds true for surjective (over [0,1]), piecewise continuous coordinate functions as well, as is the case for the components of the proposed space-filling curve. In fact, we could state the following theorem, whose proof follows from the corresponding one of Steinhaus’ theorem in [13]. Theorem 11.3. (extended Steinhaus) If ϕ1 , ϕ2 , . . . , ϕn : [0, 1] → [0, 1] are piecewise continuous, surjective, non-constant and stochastically independent with respect to the Lebesgue measure, then

196

11 Space-Filling Curves and Fuzzy ASA Δ

f = (ϕ1 , ϕ2 , ..., ϕn ) : [0, 1] → ϕ1 ([0, 1]) × ϕ2([0, 1]) × · · · × ϕn ([0, 1])

(11.9)

is a space-filling curve.

11.3 Composing Space-Filling Curves and ASA We propose the following algorithm to find a global minimum of a given function f : C → R, where C is a compact subset of some n-dimensional Euclidean space Rn . No condition of regularity, such as differentiability or even continuity, is imposed on f , and C is usually a hyper-rectangle. If it is not, we can always find one hyperrectangle containing it, taking into account its compactness [6]. In this fashion, from this point on we assume that C = [a1 , b1 ] × [a2, b2 ] × ... × [an, bn ].

11.3.1 Algorithm Description • (i) Using (11.6) and the sequence of operations shown in (11.8), find an SFC Δ ϕ = (ϕ1 , ϕ2 , . . . , ϕn ) : [0, 1] → [−1, 1] × [−1, 1] × · · ·× [−1, 1] ; • (ii) Transform the original, multidimensional minimization problem into a unidimensional one by composing ϕ and a linear isomorphism Ψ : [−1, 1] × [−1, 1] × · · ·× [−1, 1] → [a1 , b1 ] × [a2 , b2 ] × ... × [an, bn ], defining g as the composition of ϕ , Ψ and f , from [0,1] onto [a1 , b1 ]×[a2 , b2 ]×· · ·×[an , bn ], sharing with f the same extreme values; • (iii) Submit g to a one-dimensional global minimization process and find a finite subset of best candidates to global minimizers of f , say {t1 ,t2 ,t3 , ...,tN }, contained in [0,1]. Here, this set has N=7 elements, but such a number could be easily reconfigured, if necessary; • (iv) Compute the set φ = {Ψ(ϕ (t1 )), Ψ (ϕ (t2 )), . . . , Ψ (ϕ (tN ))}, jumping back to the original domain; • (v) Use the elements of φ as starting points for the Fuzzy ASA algorithm [9, 11]; • (vi) Choose the best point (corresponding to the minimum value of f ) as the final output of the algorithm. The unidimensional minimization process in (iii) is to be chosen by the implementer. In this chapter we used intentionally a simple scheme in our experiments (uniform sequential search), aiming to highlight the filling ability of the proposed curve.

11.4 Experiments

197

11.4 Experiments As noticed in [10], it is common in the literature to employ certain sets of test functions for evaluation of global optimization methods [12]. However, the chosen problems might not be the best ones for that task, as the functions belonging to them are relatively simple and regions of attraction of the global minimizers could be easily caught, despite their complicated appearances. Consequently, it is argued that they do not present enough difficulty to effectively stress the minimization ability of new optimization approaches. Hence, there is a need for synthesizing more sophisticated and systematic tests to verify their performances. To assess the effectiveness of the proposed method, a scheme similar to the one used in [10] was adopted, by employing classes of test functions, as the ones listed in Table 11.1, produced by the GKLS generator [7], which allows us to evaluate algorithms in a more complete way. It generates classes of 100 test functions having the same number of local minima plus one global minimum, supplying complete parametric information about each one, such as their domains’ dimensions, the values of local minimizers and respective coordinates, placement and sizes of attraction regions of the global minimizer, which are described by the radius of the approximate attraction region of the global minimizer, rg , and the distance from the global minimizer to the paraboloid vertex, d. We refer the reader to [7], that describes in detail the cited software. In what follows, we consider a√global minimum found when candidate points reach a ball Bi of radius ρ = 0.01 N, where N is the Euclidean dimension of the function domain, that is,    ≤ ρ} Bi = {y ∈ RN : y − yG i

(11.10)

where yG i is the global minimizer of the i-th function in a given test class and i ∈ {1, ..., 100}. As in the original tests, the three function classes shown in Table 11.1 were used in the experiments. It should also be noted that all functions are non-differentiable, for (expected) greater difficulty. Initially a set of 100 functions belonging to the Class 6, listed in Table 11.1, was generated and the performance of the proposed algorithm (described in subsection 11.3.1) compared to that of the AG algorithm, as reported in [10]. To make all tests more severe, we generated continuous, non-differentiable functions, referred to as ND-type in [7], and submitted them to our algorithm. It was successful in reaching the global minimum in all cases, computing a smaller number of function evaluations than what is described in [10] for the basic AG algorithm, which failed to find the global minimum for 1 test function out of 100, within 90000 function evaluations. By contrast, the proposed approach was able to find all 100 global minima in less than the maximum of 87307 evaluations per session. In the second phase of experiments, the present method was tested against the best one presented in [10], therein termed ALI. The authors proposed 4 new global optimization

198

11 Space-Filling Curves and Fuzzy ASA

methods denoted as AG, AGI, AL and ALI. Figure 11.4 displays the number of function evaluations and respective global minima found by the proposed technique, in a similar manner as those reported in [10] to compare AG and ALI for functions of Class 5. Again, as indicated in Table 11.2, the proposed algorithm produced better performance.

Table 11.1 Parameters corresponding to the function classes used in the experiments FunctionDomain class dimension

No. of lo- Global cal minima minimum value

d

rg

Function type

3

3

10

-1

0.66

0.33

ND

5

4

10

-1

0.66

0.33

ND

6

4

10

-1

0.90

0.20

ND

Table 11.2 Comparison of ALI and our method - minimization of 100 ND-type class 5 functions Method

Average number of Maximum number of function evaluafunction evaluations tions in individual minimization operations

ALI

14910

48210

Presented method

13600

19810

Finally, the algorithm described in subsection 11.3.1 was tested against ALI using 100 functions belonging to Class 3. The number of function evaluations performed by the proposed algorithm and respective number of found global minima are plotted in Fig. 11.5. In this case, the algorithm was able to minimize all 100 functions in less than 2700 function evaluations per session, outperforming ALI - and hence AG, as well - which required a maximum of 2952 function evaluations in at least one minimization operation. The average number of function evaluations computed by the proposed approach in the 100 minimization operations was 2091, against 1349 attained by the ALI method. Considering the main performance index as the upper bound on the number of function evaluations until finding global minima, we conclude that the proposed method was successful in this case too. The results are summarized in Table 11.3.

11.5 Conclusions

199

Fig. 11.4 Results for 100 class 5 functions - average 13600 and maximum 19810 evaluations

Table 11.3 Comparison of ALI and our method - minimization of 100 ND-type class 3 functions Method

Average number of Maximum number of function evaluafunction evaluations tions in individual minimization operations

ALI

1349

2952

Presented method

2091

2700

11.5 Conclusions This chapter introduced a multi-start approach for global minimization of multidimensional functions using dimensional reduction and a nonlinear stochastic method. When compared to other published techniques, the method shows good performance and adequacy for use in difficult cost functions. This is also due to the ability of space-filling curves in serializing the space, thereby allowing us to reduce the search domain to real line intervals. As an important and desirable byproduct, the method is capable of finding points located in the attraction basins of global minimum points that will be used as seeds when jumping back to the original, multidimensional domain. That ability is of great help in functions presenting large planar regions, where

200

11 Space-Filling Curves and Fuzzy ASA

Fig. 11.5 Results for 100 class 3 functions - average 2091 and maximum 2700 evaluations

the great majority of methods get trapped, due mainly to the lack of differential indications. Although the reduced function typically presents a fractal-like graph even when the original one is smooth or well-behaved, existing one-dimensional optimization techniques can obtain good estimates of near optimal points. To synthesize computationally adequate space-filling curves, several results were essential, including an extension of a theorem due to Hugo Steinhaus, the existence of the Sierpi´nski’s space-filling curve and several other facts from Ergodic Theory and General Topology. Although experiments were realized by means of continuous, low-dimensional functions, the proposed algorithm can be applied to real functions with arbitrarily large compact domains, not necessarily obeying Lipschitz, H¨older, continuity or differentiability conditions.

References

201

References 1. Boyarsky, A., G´ora, P.: Laws of Chaos: Invariant Measures and Dynamical Systems in One Dimension. Birkh¨auser, Boston (1997) 2. Cherruault, Y.: A new reducing transformation for global optimization (with Alienor method). Kybernetes 34(7/8), 1084–1089 (2005) 3. Cherruault, Y., Mora, G.: Optimisation Globale - Theorie des courbes α -denses. Economica, Paris (2005) 4. Choe, G.H.: Computational Ergodic Theory. Springer, Berlin (2005) 5. Dajani, K., Kraaikamp, C.: Ergodic Theory of Numbers. The Mathematical Association of America, Washington, DC (2002) 6. Edgar, G.: Measure, Topology, and Fractal Geometry. Springer, Heidelberg (2008) 7. Gaviano, M., Kvasov, D.E., Lera, D., Sergeyev, Y.D.: Software for generation of classes of test functions with known local and global minima for global optimization. ACM Transactions on Mathematical Software 29(4), 469–480 (2003) 8. Goertzel, B.: Global Optimization with Space-Filling Curves. Applied Mathematics Letters 12, 133–135 (1999) 9. Ingber, L.: Adaptive simulated annealing (ASA): Lessons learned. Control and Cybernetics 25(1), 33–54 (1996) 10. Lera, D., Sergeyev, Y.D.: Lipschitz and H¨older global optimization using space-filling curves. Applied Numerical Mathematics 60, 115–129 (2010) 11. Oliveira Jr, H.: Fuzzy control of stochastic global optimization algorithms and VFSR. Naval Research Magazine 16, 103–113 (2003) 12. Pint´er, J.: Global Optimization in Action. Kluwer Academic Publishers, Dordrecht (1996) 13. Sagan, H.: Space-Filling Curves. Springer, New York (1994)

Chapter 12

Epilogue

Abstract. In this very brief chapter we will synthesize the ideas we tried to offer along this book.

12.1 Final Thoughts There is no novelty in the assertion that Global Optimization has a unique role in the realm of Applied Mathematics, as it allows us to attain practical results even in the absence of theoretical/closed form results about a given problem, be it related to systems of strongly nonlinear equations, digital filter design or nonlinear fuzzy modeling. Of course, if we have a simpler and more direct means to get to the solution, it is not necessary to use global optimization techniques at all. In this book, we hope to have accomplished the task of explaining the readers some applications of Fuzzy ASA and how to use it to solve their own problems. Naturally, all mathematical methods need formal proof to be considered valid and this is a necessary tenet that must be rigorously respected in order to not get trapped inside dangerous intellectual blind alleys. Many well-known metaheuristic techniques already have firm theoretical ground justifying their results and, in consequence, could be applied to many problems as well. It is very important to highlight that although a given problem might have been solved by means of global optimization techniques, there is no reason to abandon the research in order to find other ways to solve it, perhaps in a more efficient manner. Intuitively, real world problem solving through global optimization methods could be viewed (in some cases) as a kind of ”temporary intelligence loan”, in the sense that while a given knowledge area has not yet enough theory to solve a specific problem, it is possible to come to a solution by indirect paths, transporting the cited problem to a different context, solving it without using results of the original field and coming back to the original setting. At first sight it seems unlikely that things could happen this way, but they do, as you can see by applying effective global optimization algorithms to your particular problems. H.A. e Oliveira Jr. et al.: Stochastic Global Optimization & Its Applications, ISRL 35, pp. 203–204. c Springer-Verlag Berlin Heidelberg 2012 springerlink.com 

204

12 Epilogue

Another point to highlight is that this book has shown that Adaptive Simulated Annealing has a lot of unexploited potential and is able to solve many different kinds of difficult problems, provided they are modeled under the right viewpoint. Although there is ongoing research to investigate the benefits of population-based ASA, the present implementation performs very well and rarely gets caught in nonglobal minimum points. In this fashion, we hope that Fuzzy ASA be useful in the solution of your own problems, always finding new and significant applications.

Index

1-periodic functions, 191 Acceptance criteria, 37, 48 Acceptance function, 44 Acceptance temperature, 42, 43 Ackley function, 76 Adaptive quenching, 46 Adaptive Simulated Annealing, 34, 44, 95, 204 Affine Takagi-Sugeno fuzzy systems, 140 Annealing proof, 47 Annealing schedule, 36, 41 Artificial bee colony, 19 ASA, 39, 41, 46, 55, 92, 143, 145, 147, 150, 164 ASA cost function, 58 ASA distribution function, 45 ASA OPTIONS, 47 ASA parameters, 70 ASA FUZZY, 56 Attraction basins, 194 Banach, S., 169 Biological evolution, 23 Blind source separation, 3 Boltzmann, 35 Boltzmann acceptance criteria, 45 Boltzmann annealing, 36 Candidate point generation, 96 Cantor, 191 Cauchy distribution, 40, 158 Closed form solutions, 169 Clustering, 145

Constrained global optimization, 95, 114 Continuous systems, 38 Convergence in distribution, 17 Cooling schedule, 29 Cost acceptance criteria, 45 Cost temperature, 45 Cost Parameter Scale Ratio, 52 Covariance matrix, 26 Crisp modeling, 141 Cross-entropy, 19, 22 Curse of dimensionality, 43, 139 Detailed balance, 37, 44 Deterministic, 12 Differential evolution, 19, 25 DLLs, 67 Dynamic penalty, 105 Entropy, 37, 122 Ergodic property, 35 Ergodic Theory, 190, 200 Euclidean space, 65 Evolutionary techniques, 25 Exponential annealing, 44 Exponential annealing schedule, 39 Exponential distribution, 152 Fast annealing, 44 Feasible region, 112 Filter bank design, 3 FIR filter design, 3 Fisher, 151 Fitness, 23 Fixed point approximation, 4

206 Fixed point theorem, 169 Fuzzy ASA, 17, 70, 73, 90, 114, 121, 123, 151, 152, 170, 171, 184, 203 Fuzzy c-means, 145 Fuzzy clustering, 145 Fuzzy controller, 65, 67, 97 Fuzzy inference system, 121 Fuzzy modeling, 66 Fuzzy systems modeling, 4, 139 Gamma distribution, 164 Gaussian-Markovian systems, 38 General Topology, 190, 200 Generalized L¨uroth series, 192 Generating function, 41 Genetic algorithms, 19, 21 Gibbs distribution, 37 Global minimum, 43 Global optimization, 12, 166, 189 Global optimization algorithm, 172 Gradient based training, 121 Grenade explosion, 22 Grenade explosion method, 19 Griewangk function, 83 Gustafson-Kessel algorithm, 145 Hamiltonian, 37 Hard clustering, 145 Hilbert, 191 Histogram, 152, 156 Hyper-rectangle, 196 Importance sampling, 26, 46 Independent component analysis, 119, 123 Inference system, 123 INFOMAX, 121, 122 Initial cost temperature, 46 Intelligent controller, 65 Inverse curvature, 43 Iseki, 190 Jacobian, 122 Jacobian determinant, 124 Kirkpatrick, 36 Krishnakumar function, 78 Kullback-Leibler divergence, 26 Lagrange multipliers, 37 Laplace, 124, 129

Index Lebesgue, 191 Likelihood function, 150 Linear homeomorphisms, 192 Linear mixtures, 120, 121 Linear transformation, 132 Linguistic variable, 66 Lipschitz, 189 Lipschitz conditions, 190 Local minima, 48, 64 Log-likelihood function, 150 Lognormal distribution, 157 Mamdani, 139 Mamdani fuzzy controller, 64 Markov chain, 28 Maximum likelihood estimate, 150 Maximum likelihood estimation, 4, 149, 166 Measure-preserving transformation, 192 Membership degree, 145 Merit function, 173, 176, 184 Metaheuristic optimization, 14 Metaheuristic techniques, 203 Metaheuristics, 21 Metropolis, 27 MIMO system, 122 MISEP, 121, 122, 127, 130, 132 Mixture distribution, 164 MLE, 150 Monte Carlo, 12, 36 Multimodal, 12 Multimodal functions, 28 Multimodal landscapes, 92 Multiobjective programming, 170 Mutation, 23 Mutual information, 121, 124, 137 n-dimensional Euclidean space, 190 Nash equilibrium, 4 Neighborhood, 28 Neural network training, 4 Neuro-fuzzy system training, 4 Nonlinear equation, 4 Nonlinear mixtures, 120 Nonlinear systems, 184 Nonlinear systems of equations, 169 Normal distribution, 156 Objective function, 19, 22 Offspring, 23 Optimization, 3

Index Optimum portfolio design, 4 Pareto dominance, 170 Pareto sampling, 56 Particle swarm optimization, 19 Path integrals, 36 PCA, 193 Peano, 191 Penalty functions, 96 Point estimation, 26 Population-based metaheuristics, 22 Post nonlinear model, 123 Premature convergence, 17, 65 Principal component analysis, 190 PROB, 152 Probability density functions, 149 Protein folding, 4 PSO, 25 QUENCH COST, 47 QUENCH COST SCALE, 47 QUENCH PARAMETERS, 47 Quenching, 17, 34, 43 Quenching factors, 66 Quenching OPTIONS, 46 Rastrigin function, 70 Ratio of acceptance, 44 Real world sounds, 132 Reanneal Cost, 52 Reannealing, 42, 44 Recombination, 23 Reproduction, 23 Rescaling, 43 Rosenbrock function, 80 Sampling, 56 Schoenberg, 190 Schwefel function, 73 Search space, 12 SELF OPTIMIZE, 50, 53 SFC, 191 Shannon entropy, 26 Sierpi´nski, 191, 192 Sierpi´nski SFC, 192 Sierpi´nski’s space-filling curve, 200

207 Simulated annealing, 17, 19, 21, 64 Simulated Quenching, 35, 64 Simultaneous nonlinear equations, 170 Space-filling curves, 189, 190 Static penalty, 107 Statistical independence, 132 Statistical mixing, 28 Statistical physics, 36 Statistically independent, 121 Steinhaus, 192 Steinhaus’ theorem, 195 Stirred tank reactors, 183 Stochastic optimization, 21 Stochastically independent, 192 Sub-energy, 66 Sub-energy function, 66 Swarm, 25 Swarm intelligence, 25 Takagi-Sugeno fuzzy system, 141 Takagi-Sugeno-Kang, 121 Tanh, 132 Temperature, 28 Temperature scale, 47 Temperature schedule, 65 Temperature Ratio Scale, 52 Training set, 122, 124, 127, 132, 139, 157 Traveling salesman problems, 3 TRUST, 66 TSK system, 122 Tuning, 48 Unconstrained global minimization, 190 Unconstrained optimization, 67 Uniform random generator, 45 USER ACCEPT ASYMP EXP, 45 USER ACCEPT THRESHOLD, 45 USER ACCEPTANCE TEST, 54 USER COST SCHEDULE, 45 USER GENERATING FUNCTION, 45 User Quench Cost Scale, 47 USER REANNEAL PARAMETERS, 53 Very fast simulated reannealing, 44 Volatility, 37