Evolution and Optimum Seeking

Evolution and Optimum Seeking Hans-Paul Schwefel

2

Preface In 1963 two students at the Technical University of Berlin met and were soon to collaborate on experiments which used the wind tunnel of the Institute of Flow Engineering. During the search for the optimal shapes of bodies in a ow, which was then a matter of laborious intuitive experimentation, the idea was conceived of proceeding strategically. However, attempts with the coordinate and simple gradient strategies were unsuccessful. Then one of the students, Ingo Rechenberg, now Professor of Bionics and Evolutionary Engineering, hit upon the idea of trying random changes in the parameters de ning the shape, following the example of natural mutations. The evolution strategy was born. A third student, Peter Bienert, joined them and started the construction of an automatic experimenter, which would work according to the simple rules of mutation and selection. The second student, I myself, set about testing the eciency of the new methods with the help of a Zuse Z23 computer for there were plenty of objections to these random strategies. In spite of an occasional lack of nancial support, the Evolutionary Engineering Group which had been formed held rmly together. Ingo Rechenberg received his doctorate in 1970 for the seminal thesis: Optimierung technischer Systeme nach Prinzipien der biologischen Evolution. It contains the theory of the two membered evolution strategy and a rst proposal for a multimembered strategy, which in the nomenclature introduced here, is of the (
+1) type. In the same year nancial support from the Deutsche Forschungsgemeinschaft (German Research Association) enabled the initiation of the work which comprises most of the present book. This work was concluded, at least temporarily, in 1974 with the thesis Evolutionsstrategie und numerische Optimierung and published by Birkhauser, Basle, Switzerland, in 1977 under the title Numerische Optimierung von Computer-Modellen mittels der Evolutionsstrategie as well as by Wiley, Chichester, in 1981 as monograph Numerical optimization of computer models. Between 1976 and 1985 the author was not able to continue his work in the eld of Evolution Strategies (nowadays abbreviated: ESs). The general interest in this type of optimum seeking algorithms was not broad enough for there to be nancial support. On the other hand, the number of articles, journals, and books devoted to (mathematical) optimization has increased tremendously. Looking back upon the development from 1964 on, when the rst ES version was devoted to experimental optimization, i.e., upon 30 years, or roughly one human generation, reveals three interesting facts:


First, ESs are not at all outdated. On the contrary, three consecutive conferences on Parallel Problem Solving from Nature (PPSN) in 1990 (see Schwefel and Manner, 1991), 1992 (Manner and Manderick, 1992), and 1994 (Davidor, Schwefel, and Manner, 1994) have demonstrated a revived and increasing interest.




Secondly, the computational environment has changed over time, not only with respect to the number of (also personal) computers and their data processing power, but even more with respect to new architectures. MIMD (Multiple Instructions v

vi Multiple Data) machines with many processors working in parallel for one task seem to wait for inherently parallel problem solving concepts like ESs. Biological metaphors prevail within the new branch of Arti cial Intelligence, called Articial Life (AL).

Third, updating this dissertation from 1974/1975 once more (after adding only a few pages to Chapter 7 in 1981) can be done without rewriting the bulk of the chapters on traditional approaches. Since the emphasis always has been centered on derivativefree direct optimum-seeking methods, it should be sucient to add material on three concepts now, i.e., Genetic Algorithms (GAs), Simulated Annealing (SA), and Tabu Search (TS). This was done with the new Sections 5.3 to 5.5 in Chapter 5. Another innovation is a oppy disk with all those procedures which had been used for the test series in the 1970s, along with a users' manual. Hopefully, some incorrectnesses have been deleted now, too. A rst thank goes again to my friend Dr. Mike Finnis whose translation of my German original into English still forms the core of this book. Thanks go also to those who helped me in completing this update, especially Ms. Heike Bracklo, who brought the scanned ASCII text into LaTeX formats, Mr. Ulrich Hermes, Mr. Jorn Mehnen, and Mr. Joachim Sprave for the many graphs and ready for use computer programs, as well as all those who helped in the process of proofreading the complete work. Finally, I would like to thank the Wiley team for the fruitful collaboration during the process of editing the camera-ready script.


Dortmund, Autumn 1994

Hans-Paul Schwefel

Contents Preface 1 Introduction 2 Problems and Methods of Optimization

2.1 General Statement of the Problems : : : : : : : : : : : : 2.2 Particular Problems and Methods of Solution : : : : : : 2.2.1 Experimental Versus Mathematical Optimization 2.2.2 Static Versus Dynamic Optimization : : : : : : : 2.2.3 Parameter Versus Functional Optimization : : : : 2.2.4 Direct (Numerical) Versus Indirect (Analytic) Optimization : : : : : : : : : 2.2.5 Constrained Versus Unconstrained Optimization : 2.3 Other Special Cases : : : : : : : : : : : : : : : : : : : : :

3 Hill climbing Strategies

v 1 5

: : : : :

5 6 6 9 10

: : : : : : : : : : : : : : : : : : : : : : : : : : : : : :

13 16 18

: : : : :

: : : : :

: : : : :

: : : : :

3.1 One Dimensional Strategies : : : : : : : : : : : : : : : : : : : : 3.1.1 Simultaneous Methods : : : : : : : : : : : : : : : : : : : 3.1.2 Sequential Methods : : : : : : : : : : : : : : : : : : : : : 3.1.2.1 Boxing in the Minimum : : : : : : : : : : : : : 3.1.2.2 Interval Division Methods : : : : : : : : : : : : 3.1.2.2.1 Fibonacci Division. : : : : : : : : : : : 3.1.2.2.2 The Golden Section. : : : : : : : : : : 3.1.2.3 Interpolation Methods : : : : : : : : : : : : : : 3.1.2.3.1 Regula Falsi Iteration. : : : : : : : : : 3.1.2.3.2 Newton-Raphson Iteration. : : : : : : 3.1.2.3.3 Lagrangian Interpolation. : : : : : : : 3.1.2.3.4 Hermitian Interpolation. : : : : : : : : 3.2 Multidimensional Strategies : : : : : : : : : : : : : : : : : : : : 3.2.1 Direct Search Strategies : : : : : : : : : : : : : : : : : : 3.2.1.1 Coordinate Strategy : : : : : : : : : : : : : : : 3.2.1.2 Strategy of Hooke and Jeeves: Pattern Search : 3.2.1.3 Strategy of Rosenbrock: Rotating Coordinates : 3.2.1.4 Strategy of Davies, Swann, and Campey (DSC) 3.2.1.5 Simplex Strategy of Nelder and Mead : : : : : vii

: : : : :

: : : : : : : : : : : : : : : : : : :

: : : : :

: : : : : : : : : : : : : : : : : : :

: : : : :

: : : : : : : : : : : : : : : : : : :

: : : : :

: : : : : : : : : : : : : : : : : : :

: : : : :

: : : : : : : : : : : : : : : : : : :

: : : : : : : : : : : : : : : : : : :

23 25 26 27 28 29 29 32 33 34 35 35 37 38 40 41 44 48 54 57

viii 3.2.1.6 Complex Strategy of Box : : : : : : : : : : : : : : 3.2.2 Gradient Strategies : : : : : : : : : : : : : : : : : : : : : : : 3.2.2.1 Strategy of Powell: Conjugate Directions : : : : : : 3.2.3 Newton Strategies : : : : : : : : : : : : : : : : : : : : : : : : 3.2.3.1 DFP: Davidon-Fletcher-Powell Method (Quasi-Newton Strategy, Variable Metric Strategy) 3.2.3.2 Strategy of Stewart: Derivative-free Variable Metric Method : : : : : : : 3.2.3.3 Further Extensions : : : : : : : : : : : : : : : : : :

: : : :

61 65 69 74

: : : :

77

: : : : : : : :

78 81

: : : :

4 Random Strategies 5 Evolution Strategies for Numerical Optimization

5.1 The Two Membered Evolution Strategy : : : : : : : : : : : : : : : : : 5.1.1 The Basic Algorithm : : : : : : : : : : : : : : : : : : : : : : : 5.1.2 The Step Length Control : : : : : : : : : : : : : : : : : : : : : 5.1.3 The Convergence Criterion : : : : : : : : : : : : : : : : : : : : 5.1.4 The Treatment of Constraints : : : : : : : : : : : : : : : : : : 5.1.5 Further Details of the Subroutine EVOL : : : : : : : : : : : : 5.2 A Multimembered Evolution Strategy : : : : : : : : : : : : : : : : : : 5.2.1 The Basic Algorithm : : : : : : : : : : : : : : : : : : : : : : : 5.2.2 The Rate of Progress of the (1 , ) Evolution Strategy : : : : : 5.2.2.1 The Linear Model (Inclined Plane) : : : : : : : : : : 5.2.2.2 The Sphere Model : : : : : : : : : : : : : : : : : : : 5.2.2.3 The Corridor Model : : : : : : : : : : : : : : : : : : 5.2.3 The Step Length Control : : : : : : : : : : : : : : : : : : : : : 5.2.4 The Convergence Criterion for
> 1 Parents : : : : : : : : : : 5.2.5 Scaling of the Variables by Recombination : : : : : : : : : : : 5.2.6 Global Convergence : : : : : : : : : : : : : : : : : : : : : : : : 5.2.7 Program Details of the (
+ ) ES Subroutines : : : : : : : : : 5.3 Genetic Algorithms : : : : : : : : : : : : : : : : : : : : : : : : : : : : 5.3.1 The Canonical Genetic Algorithm for Parameter Optimization 5.3.2 Representation of Individuals : : : : : : : : : : : : : : : : : : 5.3.3 Recombination and Mutation : : : : : : : : : : : : : : : : : : 5.3.4 Reproduction and Selection : : : : : : : : : : : : : : : : : : : 5.3.5 Further Remarks : : : : : : : : : : : : : : : : : : : : : : : : : 5.4 Simulated Annealing : : : : : : : : : : : : : : : : : : : : : : : : : : : 5.5 Tabu Search and Other Hybrid Concepts : : : : : : : : : : : : : : : :

: : : :

: : : : : : : : : : : : : : : : : : : : : : : : :

: : : :

: : : : : : : : : : : : : : : : : : : : : : : : :

: : : : : : : : : : : : : : : : : : : : : : : : :

87 105 105 106 110 113 115 115 118 118 120 124 127 134 142 145 146 149 149 151 152 153 155 157 158 160 162

6 Comparison of Direct Search Strategies for Parameter Optimization 165 6.1 Diculties : : : : : : : : : : : 6.2 Theoretical Results : : : : : : 6.2.1 Proofs of Convergence 6.2.2 Rates of Convergence : 6.2.3 Q-Properties : : : : : :

: : : : :

: : : : :

: : : : :

: : : : :

: : : : :

: : : : :

: : : : :

: : : : :

: : : : :

: : : : :

: : : : :

: : : : :

: : : : :

: : : : :

: : : : :

: : : : :

: : : : :

: : : : :

: : : : :

: : : : :

: : : : :

: : : : :

: : : : :

: : : : :

: : : : :

165 166 167 168 169

ix 6.2.4 Computing Demands : : : : : : : : : : : : : : : 6.3 Numerical Comparison of Strategies : : : : : : : : : : : 6.3.1 Computer Used : : : : : : : : : : : : : : : : : : 6.3.2 Optimization Methods Tested : : : : : : : : : : 6.3.3 Results of the Tests : : : : : : : : : : : : : : : : 6.3.3.1 First Test: Convergence Rates for a Quadratic Objective Function : : 6.3.3.2 Second Test: Reliability : : : : : : : : 6.3.3.3 Third Test: Non-Quadratic Problems with Many Variables : : : : : : : : : : 6.4 Core storage required : : : : : : : : : : : : : : : : : : :

: : : : :

170 173 174 175 179

: : : : : : : : : : : : : : : : : : : : : :

179 204

: : : : : : : : : : : : : : : : : : : : : :

217 232

: : : : :

: : : : :

: : : : :

: : : : :

: : : : :

: : : : :

: : : : :

: : : : :

: : : : :

: : : : :

7 Summary and Outlook 8 References Appendices A Catalogue of Problems

235 249 325 325

B Program Codes

367

C Programs

415

A.1 Test Problems for the First Part of the Strategy Comparison : : : : : : : : 325 A.2 Test Problems for the Second Part of the Strategy Comparison : : : : : : : 327 A.3 Test Problems for the Third Part of the Strategy Comparison : : : : : : : 361

B.1 (1+1) Evolution Strategy EVOL : : : : : : : : : : : : : : : : : : : : : : : : 367 B.2 (
, ) Evolution Strategies GRUP and REKO : : : : : : : : : : : : : : : : 375 B.3 (
+ ) Evolution Strategy KORR : : : : : : : : : : : : : : : : : : : : : : : 386 C.1 Contents of the Floppy Disk : : : : : : : : : : : : : : : : : : : : : C.2 About the Program Disk : : : : : : : : : : : : : : : : : : : : : : : C.3 Running the C Programs : : : : : : : : : : : : : : : : : : : : : : : C.3.1 How to Install OptimA on a PC Using LINUX or on a UNIX Workstation : : : : : : : : : : : : : : : : : : C.3.2 How to Install OptimA on a PC Under DOS : : : : : : : C.3.3 Running OptimA : : : : : : : : : : : : : : : : : : : : : : : C.4 Description of the Programs : : : : : : : : : : : : : : : : : : : : : C.4.1 How to Incorporate New Functions : : : : : : : : : : : : : C.5 Examples : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : C.5.1 An Application of the Multimembered Evolution Strategy to the Corridor Model : : : : : : : : : : : : : : : : : : : : C.5.2 OptimA Working in Batch Mode : : : : : : : : : : : : : :

Index

: : : : : : : : : : : : : : :

415 416 417

: : : : : :

417 418 418 418 419 421

: : : : : : : : : :

421 422

: : : : : :

: : : : : :

: : : : : :

: : : : : :

425

x

Chapter 1 Introduction There is scarcely a modern journal, whether of engineering, economics, management, mathematics, physics or the social sciences, in which the concept optimization is missing from the subject index. If one abstracts from all specialist points of view, the recurring problem is to select a better or best (Leibniz, 1710 eventually, he introduced the term optimal ) alternative from among a number of possible states of a airs. However, if one were to follow the hypothesis of Leibniz, as presented in his Theodicee, that our world is the best of all possible worlds, one could justi ably sink into passive fatalism. There would be nothing to improve or to optimize. Biology, especially since Darwin, has replaced the static world picture of Leibniz' time by a dynamic one, that of the more or less gradual development of the species culminating in the appearance of man. Paleontology is providing an increasingly complete picture of organic evolution. So-called missing links repeatedly turn out to be not missing, but rather hitherto undiscovered stages of this process. Very much older than the recognition that man is the result (or better, intermediate state) of a meliorization process is the seldom-questioned assumption that he is a perfect end product, the \pinnacle of creation." Furthermore, long before man conceived of himself as an active participant in the development of things, he had unconsciously inuenced this evolution. There can be no doubt that his ability and e orts to make the environment meet his needs raised him above other forms of life and have enabled him, despite physical inferiority, to nd, to hold, and to extend his place in the world{so far at least. As long as mankind has existed on our planet, spaceship earth, we, together with other species have mutually inuenced and changed our environment. Has this always been done in the sense of meliorization? In 1759, the French philosopher Voltaire (1759), dissatis ed with the conditions of his age, was already taking up arms against Leibniz' philosophical optimism and calling for conscious e ort to change the state of a airs. In the same way today, when we optimize we nd that we are both the subject and object of the history of development. In the desire to improve an object, a process, or a system, Wilde and Beightler (1967) see an expression of the human striving for perfection. Whether such a lofty goal can be attained depends on many conditions. It is not possible to optimize when there is only one way to carry out a task{then one has no alternative. If it is not even known whether the problem at hand is soluble, the 1

2

Introduction

situation calls for an invention or discovery and not, at that stage, for any optimization. But wherever two or more solutions exist and one must decide upon one of them, one should choose the best, that is to say optimize. Those independent features that distinguish the results from one another are called (independent) variables or parameters of the object or system under consideration they may be represented as binary, integer, otherwise discrete, or real values. A rational decision between the real or imagined variants presupposes a value judgement, which requires a scale of values, a quantitative criterion of merit, according to which one solution can be classi ed as better, another as worse. This dependent variable is usually called an objective (function) because it depends on the objective of the system{the goal to be attained with it{and is functionally related to the parameters. There may even exist several objectives at the same time{the normal case in living systems where the mix of objectives also changes over time and may, in fact, be induced by the actual course of the evolutionary paths themselves. Sometimes the hardest part of optimization is to de ne clearly an objective function. For instance, if several subgoals are aimed at, a relative weight must be attached to each of the individual criteria. If these are contradictory one only can hope to nd a compromise on a trade-o subset of non-dominated solutions. Variability and distinct order of merit are the unavoidable conditions of any optimization. One may sometimes also think one has found the right objective for a subsystem, only to realize later that, in doing so, one has provoked unwanted side e ects, the rami cations of which have worsened the disregarded total objective function. We are just now experiencing how narrow-minded scales of value can steer us into dangerous plights, and how it is sometimes necessary to consider the whole Earth as a system, even if this is where di erences of opinion about value criteria are the greatest. The second diculty in optimization, particularly of multiparameter objectives or processes, lies in the choice or design of a suitable strategy for proceeding. Even when the objective has been suciently clearly de ned, indeed even when the functional dependence on the independent variables has been mathematically (or computationally) formulated, it often remains dicult enough, if not completely impossible, to nd the optimum, especially in the time available. The uninitiated often think that it must be an easy matter to solve a problem expressed in the language of mathematics, that most exact of all sciences. Far from it: The problem of how to solve problems is unsolved{and mathematicians have been working on it for centuries. For giving exact answers to questions of extremal values and corresponding positions (or conditions) we are indebted, for example, to the dierential and variational calculus, of which the development in the 18th century is associated with such illustrious names as Newton, Euler, Lagrange, and Bernoulli. These constitute the foundations of the present methods referred to as classical, and of the further developments in the theory of optimization. Still, there is often a long way from the theory, which is concerned with establishing necessary (and sucient) conditions for the existence of minima and maxima, to the practice, the determination of these most desirable conditions. Practically signi cant solutions of optimization problems rst became possible with the arrival of (large and) fast programmable computers in the mid-20th century. Since then the ood of publications on the subject of optimization has been steadily rising in volume it is a

Introduction

3

simple matter to collect several thousand published articles about optimization methods. Even an interested party nds it dicult to keep pace nowadays with the development that is going on. It seems far from being over, for there still exists no all-embracing theory of optimization, nor is there any universal method of solution. Thus it is appropriate, in Chapter 2, to give a general survey of optimization problems and methods. The special r^ole of direct, static, non-discrete, and non-stochastic parameter optimization emerges here, for many of these methods can be transferred to other elds the converse is less often possible. In Chapter 3, some of these strategies are presented in more depth, principally those that extract the information they require only from values of the objective function, that is to say without recourse to analytic partial derivatives (derivative-free methods). Methods of a probabilistic nature are omitted here. Methods which use chance as an aid to decision making, are treated separately in Chapter 4. In numerical optimization, chance is simulated deterministically by means of a pseudorandom number generator able to produce some kind of deterministic chaos only. One of the random strategies proves to be extremely promising. It imitates, in a highly simpli ed manner, the mutation-selection game of nature. This concept, a two membered evolution strategy, is formulated in a manner suitable for numerical optimization in Chapter 5, Section 5.1. Following the hypothesis put forward by Rechenberg, that biological evolution is, or possesses, an especially advantageous optimization process and is therefore worthy of imitation, an extended multimembered scheme that imitates the population principle of evolution is introduced in Chapter 5, Section 5.2. It permits a more natural as well as more e ective speci cation of the step lengths than the two membered scheme and actually invites the addition of further evolutionary principles, such as, for example, sexual propagation and recombination. An approximate theory of the rate of convergence can also be set up for the (1 ,
) evolution strategy, in which only the best of
descendants of a generation become parents of the following one. A short excursion, new to this edition, introduces nearly concurrent developments that the author was unaware of when compiling his dissertation in the early 1970s, i.e., genetic algorithms, simulated annealing, and tabu search. Chapter 6 then makes a comparison of the evolution strategies with the direct search methods of zero, rst, and second order, which were treated in detail in Chapter 3. Since the predictive power of theoretical proofs of convergence and statements of rates of convergence is limited to simple problem structures, the comparison includes mainly numerical tests employing various model objective functions. The results are evaluated from two points of view:


Eciency, or speed of approach to the objective




Eectivity, or reliability under varying conditions

The evolution strategies are highly successful in the test of e ectivity or robustness. Contrary to the widely held opinion that biological evolution is a very wasteful method of optimization, the convergence rate test shows that, in this respect too, the evolution methods can hold their own and are sometimes even more ecient than many purely deterministic methods. The circle is closed in Chapter 7, where the analogy between

4

Introduction

iterative optimization and evolution is raised once again for discussion, with a look at some natural improvements and extensions of the concept of the evolution strategy. The list of test problems that were used can be found in Appendix A, and FORTRAN codes of the evolution strategies, with detailed guidance for users, are in Appendix B. Finally, Appendix C explains how to use the C and FORTRAN programs on the oppy disk.

Chapter 2 Problems and Methods of Optimization 2.1 General Statement of the Problems According to whether one emphasizes the theoretical aspect (existence conditions of optimal solutions) or the practical (procedures for reaching optima), optimization nowadays is classi ed as a branch of applied or numerical mathematics, operations research, or of computer-assisted systems (engineering) design. In fact many optimization methods are based on principles which were developed in linear and non-linear algebra. Whereas for equations, or systems of equations, the problem is to determine a quantity or set of quantities such that functions which depend on them have speci ed values, in the case of an optimization problem, an initially unknown extremal value is sought. Many of the current methods of solution of systems of linear equations start with an approximation and successively improve it by minimizing the deviation from the required value. For non-linear equations and for incomplete or overdetermined systems this way of proceeding is actually essential (Ortega and Rheinboldt, 1970). Thus many seemingly quite di erent and apparently unrelated problems turn out, after a suitable reformulation, to be optimization problems. Into this class come, for example, the solution of di erential equations (boundary and initial value problems) and eigenvalue problems, as well as problems of observational calculus, adaptation, and approximation (Stiefel, 1965 Schwarz, Rutishauser, and Stiefel, 1968 Collatz and Wetterling, 1971). In the rst case, the basic problem again is to solve equations in the second, the problem is often reduced to minimize deviations in the Gaussian sense (sum of squares of residues) or the Tchebyche sense (maximum of the absolute residues). Even game theory (Vogelsang, 1963) and pattern or shape recognition as a branch of information theory (Andrews, 1972 Niemann, 1974) have features in common with the theory of optimization. In one case, from among a stored set of idealized types, a pattern will be sought that has the maximum similarity to the one presented in another case, the search will be for optimal courses of action in conict situations. Here, two or more interests are competing. Each player tries to maximize his chance of winning with regard to the way in which his opponent supposedly plays. Most optimization 5

6

Problems and Methods of Optimization

problems, however, are characterized by a single interest, to reach an objective that is not inuenced by others. The engineering aspect of optimization has manifested itself especially clearly with the design of learning robots, which have to adapt their operation to the prevailing conditions (see for example Feldbaum, 1962 Zypkin, 1970). The feedback between the environment and the behavior of the robot is e ected here by a program, a strategy, which can perhaps even alter itself. Wiener (1963) goes even further and considers self-reproducing machines, thus arriving at a consideration of robots that are similar to living beings. Computers are often regarded as the most highly developed robots, and it is therefore tempting to make comparisons with the human brain and its neurons and synapses (von Neumann, 1960, 1966 Marfeld, 1970 Steinbuch, 1971). They are nowadays the most important aid to optimization, and many problems are intractable without them.

2.2 Particular Problems and Methods of Solution The lack of a universal method of optimization has led to the present availability of numerous procedures that each have only limited application to special cases. No attempt will be made here to list them all. A short survey should help to distinguish the parameter optimization strategies, treated in detail later, from the other procedures, but while at the same time exhibiting some features they have in common. The chosen scheme of presentation is to discuss two opposing concepts together.

2.2.1 Experimental Versus Mathematical Optimization

If the functional relation between the variables and the objective function is unknown, one is forced to experiment either on the real object or on a scale model. To do so one must be as free as possible to vary the independent variables and have access to measuring instruments with which the dependent variable, the quality, can be measured. Systematic investigation of all possible states of the system will be too costly if there are many variables, and random sampling of various combinations is too unreliable for achieving the desired result. A procedure must be signi cantly more e ective if it systematically exploits whatever information is retained about preceding attempts. Such a plan is also called a strategy. The concept originated in game theory and was formulated by von Neumann and Morgenstern (1961). Many of the search strategies of mathematical optimization to be discussed later were also applied under experimental conditions{not always successfully. An important characteristic of the experiment is the unavoidable e ect of (stochastic) disturbances on the measured results. A good experimental optimization strategy has to take account of this fact and approach the desired extremum with the least possible cost in attempts. Two methods in particular are most frequently mentioned in this connection:
The EVOP (evolutionary operation) method proposed by G. E. P. Box (1957), a development of the experimental gradient method of Box and Wilson (1951)
The strategy of arti cial evolution designed by Rechenberg (1964)

Particular Problems and Methods of Solution

7

The algorithm of Rechenberg's evolution strategy will be treated in detail in Chapter 5. In the experimental eld it has often been applied successfully: for example, to the solution of multiparameter shaping problems (Rechenberg, 1964 Schwefel, 1968 Klockgether and Schwefel, 1970). All variables are simultaneously changed by a small random amount. The changes are (binomially or) normally distributed. The expected value of the random vector is zero (for all components). Failures leave the starting condition unchanged, only successes are adopted. Stochastic disturbances or perturbations, brought about by errors of measurement, do not a ect the reliability but inuence the speed of convergence according to their magnitude. Rechenberg (1973) gives rules for the optimal choice of a common variance of the probability density distribution of the random changes for both the unperturbed and the perturbed cases. The EVOP method of G. E. P. Box changes only two or three parameters at a time{if possible those which have the strongest inuence. A square or cube is constructed with an initial condition at its midpoint its 22 = 4 or 23 = 8 corners represent the points in a cycle of trials. These deterministically established states are tested sequentially, several times if perturbations are acting. The state with the best result becomes the midpoint of the next pattern of points. Under some conditions, one also changes the scaling of the variables or exchanges one or more parameters for others. Details of this altogether heuristic way of proceeding are described by Box and Draper (1969, 1987). The method has mainly been applied to the dynamic optimization of chemical processes. Experiments are performed on the real system, sometimes over a period of several years. The counterpart to experimental optimization is mathematical optimization. The functional relation between the criterion of merit or quality and the variables is known, at least approximately to put it another way, a more or less simpli ed mathematical model of the object, process or system is available. In place of experiments there appears the manipulation of variables and the objective function. It is sometimes easy to set up a mathematical model, for example if the laws governing the behavior of the physical processes involved are known. If, however, these are largely unresearched, as is often the case for ecological or economic processes, the work of model building can far exceed that of the subsequent optimization. Depending on what deliberate inuence one can have on the process, one is either restricted to the collection of available data or one can uncover the relationships between independent and dependent variables by judiciously planning and interpreting tests. Such methods (Cochran and Cox, 1950 Kempthorne, 1952 Davies, 1954 Cox, 1958 Fisher, 1966 Vajda, 1967 Yates, 1967 John, 1971) were rst applied only to agricultural problems, but later spread into industry. Since the analyst is intent on building the best possible model with the fewest possible tests, such an analysis itself constitutes an optimization problem, just as does the synthesis that follows it. Wald (1966) therefore recommends proceeding sequentially, that is to construct a model as a hypothesis from initial experiments or given a priori information, and then to improve it in a stepwise fashion by a further series of tests, or, alternatively, to sometimes reject the model completely. The tting of model parameters to the measured data can be considered as an optimization problem insofar as the expected error or maximum risk is to be minimized. This is a special case of optimization, called calculus of observations , which involves sta-

8

Problems and Methods of Optimization

tistical tests like regression and variance analyses on data subject to errors, for which the principle of maximum likelihood or minimum 2 is used (see Heinhold and Gaede, 1972). The cost of constructing a model of large systems with many variables, or of very complicated objects, can become so enormous that it is preferable to get to the desired optimal condition by direct variation of the parameters of the process, in other words to optimize experimentally. The fact that one tries to analyze the behavior of a model or system at all is founded on the hope of understanding the processes more fully and of being able to solve the synthesis problem in a more general way than is possible in the case of experimental optimization, which is tied to a particular situation. If one has succeeded in setting up a mathematical model of the system under consideration, then the optimization problem can be expressed mathematically as follows:

F (x) = F (x1 x2 : : : xn) ! extr The round brackets symbolize the functional relationship between the n independent variables fxi  i = 1(1)ng1 and the dependent variable F , the quality or objective function. In the following it is always a scalar quantity. The variables can be scalars or functions of one or more parameters. Whether a maximum or a minimum is sought for is of no consequence for the method of optimization because of the relation maxfF (x)g = ; minf;F (x)g Without loss of generality one can concentrate on one of the types of problem usually the minimum problem is considered. Restrictions do arise, insofar as in many practical problems the variables cannot be chosen arbitrarily. They are called constraints. The simplest of these are the non-negative conditions:

xi  0 

for all i = 1(1)n

They are formulated more generally like the objective function: 8 9 > <> = Gj (x) = Gj (x1 x2 : : : xn ) > = > 0  for all j = 1(1)m : The notation chosen here follows the convention of parameter optimization. One distinguishes between equalities and inequalities. Each equality constraint reduces the number of true variables of the problem by one. Inequalities, on the other hand, simply reduce the size of the allowed space of solutions without altering its dimensionality. The sense of the inequality is not critical. Like the interchanging of minimum and maximum problems, one can transform one type into the other by reversing the signs of the terms. It is sucient to limit consideration to one formulation. For minimum problems this is 1

The term 1(1)n stands for 1,2,3,...,n.

Particular Problems and Methods of Solution

9

normally the type Gj (x)  0: Points on the edge of the (closed) allowed space are thereby permitted. A di erent situation arises if the constraint is given as a strict inequality of the form Gj (x) > 0: Then the allowed space can be open if Gj (x) is continuous. If for Gj (x)  0, with other conditions the same, the minimum lies on the border Gj (x) = 0, then for Gj (x) > 0, there is no true minimum. One refers here to an inmum, the greatest lower bound, at which actually Gj (x) = 0. In analogous fashion one distinguishes between maxima and suprema (smallest upper bounds). Optimization in the following means always to nd a maximum or a minimum, perhaps under inequality constraints.

2.2.2 Static Versus Dynamic Optimization

The term static optimization means that the optimum is time invariant or stationary. It is sucient to determine its position and size once and for all. Once the location of the extremum has been found, the search is over. In many cases one cannot control all the variables that inuence the objective function. Then it can happen that these uncontrollable variables change with time and displace the optimum (non-stationary case). The goal of dynamic optimization2 is therefore to maintain an optimal condition in the face of varying conditions of the environment. The search for the extremum becomes a more or less continuous process. According to the speed of movement of the optimum, it may be necessary, instead of making the slow adjustment of the independent variables by hand{as for example in the EVOP method (see Chap. 2, Sect. 2.2.1), to give the task to a robot or automaton. The automaton and the process together form a control loop. However, unlike conventional control loops this one is not required to maintain a desired value of a quantity but to discover the most favorable value of an unknown and time-dependent quantity. Feldbaum (1962), Frankovic et al. (1970), and Zach (1974) investigate in detail such automatic optimization systems, known as extreme value controllers or optimizers. In each case they are built around a search process. For only one variable (adjustable setting) a variety of schemes can be designed. It is signi cantly more complicated for an optimal value loop when several parameters have to be adjusted. Many of the search methods are so very costly because there is no a priori information about the process to be controlled. Hence nowadays one tries to build adaptive control systems that use information gathered over a period of time to set up an internal model of the system, or that, in a sense, learn. Oldenburger (1966) and, in more detail, Zypkin (1970) tackle the problems of learning and self-learning robots. Adaptation is said to take place if the change in the control characteristics is made on the basis of measurements of those input quantities to the process that cannot be altered{also known as disturbing variables. If the output quantities themselves are used (here the objective function) to adjust the control system, the process is called self-learning or self-adaptation. The latter possibility is more reliable but, because of the time lag, slower. Cybernetic engineering is concerned with learning processes in a more general form and always sees or even seeks links with natural analogues. An example of a robot that adapts itself to the environment is the homeostat of Ashby 2

Some authors use the term dynamic optimization in a dierent way than is done here.

10

Problems and Methods of Optimization

(1960). Nowadays, however, one does not build one's own optimizer every time there is a given problem to be solved. Rather one makes use of so-called process computers, which for a new task only need another special program. They can handle large and complicated problems and are coupled to the process by sensors and transducers in a closed loop (online) (Levine and Vilis, 1973 McGrew and Haimes, 1974). The actual computer usually works digitally, so that analogue-digital and digital-analogue converters are required for input and output. Process computers are employed for keeping process quantities constant and maintaining required pro les as well as for optimization. In the latter case an internal model (a computer program) usually serves to determine the optimal process parameters, taking account of the latest measured data values in the calculation. If the position of the optimum in a dynamic process is shifting very rapidly, the manner in which the search process follows the extremum takes on a greater signi cance for the overall quality. In this case one has to go about setting up a dynamic model and specifying all variables, including the controllable ones, as functions of time. The original parameter optimization goes over to functional optimization.

2.2.3 Parameter Versus Functional Optimization

The case when not only the objective function but also the independent variables are scalar quantities is called parameter optimization. Numerical values fx
i  i

= 1(1)ng

of the variables or parameters are sought for which the value of the objective function

F
= F (x
) = extrfF (x)g is an optimum. The number of parameters describing a state of the object or system is

nite. In the simplest case of only one variable (n = 1), the behavior of the objective function is readily visualized on a diagram with two orthogonal axes. The value of the parameter is plotted on the abscissa and that of the objective function on the ordinate. The functional dependence appears as a curve. For n = 2 a three dimensional Cartesian coordinate system is required. The state of the system is represented as a point in the horizontal plane and the value of the objective function as the vertical height above it. A mountain range is obtained, the surface of which expresses the relation of dependent to independent variables. To further simplify the representation, the curves of intersection between the mountain range and parallel horizontal planes are projected onto the base plane, which provides a contour diagram of the objective function. From this three dimensional picture and its two dimensional projection, concepts like peak, plateau, valley, ridge, and contour line are readily transferred to the n-dimensional case, which is otherwise beyond our powers of description and visualization. In functional optimization, instead of optimal points in three dimensional Euclidean space, optimal trajectories in function spaces (such as Banach or Hilbert space) are to be determined. Thus one refers also to in nite dimensional optimization as opposed to the nite dimensional parameter optimization. Since the variables to be determined are

Particular Problems and Methods of Solution

11

themselves functions of one or more parameters, the objective function is a function of a function, or a functional. A classical problem is to determine the smooth curve down which a point mass will slide between two points in the shortest time, acted upon by the force of gravity and without friction. Known as the brachistochrone problem, it can be solved by means of the ordinary variational calculus (Courant and Hilbert, 1968a,b Denn, 1969 Clegg, 1970). If the functions to be determined depend on several variables it is a multidimensional variational problem (Klotzler, 1970). In many cases the time t appears as the only parameter. The objective function is commonly an integral, in the integrand of which will appear not only the independent variables x(t) = fx1(t) x2(t) : : : xn(t)g but also their derivatives x_ i(t) = @xi=@t and sometimes also the parameter t itself: Z t2 F (x(t)) = t f (x(t) x_ (t) t) dt ! extr 1

Such problems are typical in control theory, where one has to nd optimal controlling functions for control processes (e.g., Chang, 1961 Lee, 1964 Leitmann, 1964 Hestenes, 1966 Balakrishnan and Neustadt, 1967 Karreman, 1968 Demyanov and Rubinov, 1970). Whereas the variational calculus and its extensions provide the mathematical basis of functional optimization (in the language of control engineering: optimization with distributed parameters), parameter optimization (with localized parameters) is based on the theory of maxima and minima from the elementary di erential calculus. Consequently both branches have followed independent paths of development and become almost separate disciplines. The functional analysis theory of Dubovitskii and Milyutin (see Girsanov, 1972) has bridged the gap between the problems by allowing them to be treated as special cases of one fundamental problem, and it could thus lead to a general theory of optimization. However di erent their theoretical bases, in cases of practical signi cance the problems must be solved on a computer, and the iterative methods employed are then broadly the same. One of these iterative methods is the dynamic programming or stepwise optimization of Bellman (1967). It was originally conceived for the solution of economic problems, in which time-dependent variables are changed in a stepwise way at xed points in time. The method is a discrete form of functional optimization in which the trajectory sought appears as a steplike function. At each step a decision is taken, the sequence of which is called a policy. Assuming that the state at a given step depends only on the decision at that step and on the preceding state{i.e., there is no feedback{, then dynamic programming can be applied. The Bellman optimum principle implies that each piece of the optimal trajectory that includes the end point is also optimal. Thus one begins by optimizing the

nal decision at the transition from the last-but-one to the last step. Nowadays dynamic programming is frequently applied to solving discrete problems of optimal control and regulation (Gessner and Spremann, 1972 Lerner and Rosenman, 1973). Its advantage compared to other, analytic methods is that its algorithm can be formulated as a program suitable for digital computers, allowing fairly large problems to be tackled (Gessner and

12

Problems and Methods of Optimization

Wacker, 1972). Bellman's optimum principle can, however, also be expressed in di erential form and applied to an area of continuous functional optimization (Jacobson and Mayne, 1970). The principle of stepwise optimization can be applied to problems of parameter optimization, if the objective function is separable (Hadley, 1969): that is, it must be expressible as a sum of partial objective functions in which just one or a very few variables appear at a time. The number of steps (k) corresponds to the number of the partial functions at each step a decision is made only on the (`) variables in the partial objective. They are also called control or decision variables. Subsidiary conditions (number m) in the form of inequalities can be taken into account. The constraint functions, like the variables, are allowed to take a nite number (b) of discrete values and are called state variables. The recursion formula for the stepwise optimization will not be discussed here. Only the number of required operations (N ) in the calculation will be mentioned, which is of the order

N  k bm+` For this reason the usefulness of dynamic programming is mainly restricted to the case ` = 1 k = n and m = 1. Then at each of the n steps, just one control variable is speci ed with respect to one subsidiary condition. In the other limiting case where all variables have to be determined at one step, the normal case of parameter optimization, the process goes over to a grid method (complete enumeration) with a computational requirement of order O(b(n+m) ). Herein lies its capability for locating global optima, even of complicated multimodal objective functions. However, it is only especially advantageous if the structure of the objective function permits the enumeration to be limited to a small part of the allowed region. Digital computers are poorly suited to solving continuous problems because they cannot operate directly with functions. Numerical integration procedures are possible, but costly. Analogue computers are more suitable because they can directly imitate dynamic processes. Compared to digital computers, however, they have a small numerical range and low accuracy and are not so easily programmable. Thus sometimes digital and analogue computers are coupled for certain tasks as so-called hybrid computers. With such systems a set of di erential equations can be tackled to the same extent as a problem in functional optimization (Volz, 1965, 1973). The digital computer takes care of the iteration control, while on the analogue computer the di erentiation and integration operations are carried out according to the parameters supplied by the digital computer. Korn and Korn (1964), and Bekey and Karplus (1971), describe the operations involved in trajectory optimization and the solution of di erential equations by means of hybrid computers. The fact that random methods are often used for such problems has to do with the computational imprecision of the analogue part, with which deterministic processes usually fail to cope. If requirements for accuracy are very high, however, purely digital computation has to take over, with the consequent greater cost in computation time.

Particular Problems and Methods of Solution

13

2.2.4 Direct (Numerical) Versus Indirect (Analytic) Optimization

The classi cation of mathematical methods of optimization into direct and indirect procedures is attributed to Edelbaum (1962). Especially if one has a computer model of a system, with which one can perform simulation experiments, the search for a certain set of exogenous parameters to generate excellent results asks for robust direct optimization methods. Direct or numerical methods are those that approach the solution in a stepwise manner (iteratively), at each step (hopefully) improving the value of the objective function. If this cannot be guaranteed, a trial and error process results. An indirect or analytic procedure attempts to reach the optimum in a single (calculation) step, without tests or trials. It is based on the analysis of the special properties of the objective function at the position of the extremum. In the simplest case, parameter optimization without constraints, one proceeds on the assumption that the tangent plane at the optimum is horizontal, i.e., the rst partial derivatives of the objective function exist and vanish in x
:  @F  = 0  for all i = 1(1)n (2.1) @xi x=x
This system of equations can be expressed with the so-called Nabla operator (r) as a single vector equation for the stationary point x
: rF (x
) = 0

(2.2)

Equation (2.1) or (2.2) transforms the original optimization problem into a problem of solving a set of, perhaps non-linear, simultaneous equations. If F (x) or one or more of its derivatives are not continuous, there may be extrema that do not satisfy the otherwise necessary conditions. On the other hand not every point in IRn{the n-dimensional space of real variables{ that satis es conditions (2.1) need be a minimum it could also be a maximum or a saddle point. Equation (2.2) is referred to as a necessary condition for the existence of a local minimum. To give sucient conditions requires further processes of di erentiation. In fact, di erentiations must be carried out until the determinant of the matrix of the second or higher partial derivatives at the point x
is non-zero. Things remain simple in the case of only one variable, when it is required that the lowest order non-vanishing derivative is positive and of even order. Then and only then is there a minimum. If the derivative is negative, x
represents a maximum. A saddle point exists if the order is odd. For n variables, at least the n2 (n + 1) second partial derivatives

@ 2F (x)  @xi @xj

for all i j = 1(1)n

must exist at the point x
. The determinant of the Hessian matrix r2F (x
) must be positive, as well as the further n ; 1 principle subdeterminants of this matrix. While MacLaurin had already completely formulated the sucient conditions for the existence of minima and maxima of one parameter functions in 1742, the corresponding theory

14

Problems and Methods of Optimization

for functions of several variables was only completed nearly 150 years later by Schee er (1886) and Stolz (1893) (see also Hancock, 1960). Sucient conditions can only be applied to check a solution that was obtained from the necessary conditions. The analytic path thus always leads the original optimization problem back to the problem of solving a system of simultaneous equations (Equation (2.2)). If the objective function is of second order, one is dealing with a linear system, which can be solved with the aid of one of the usual methods of linear algebra. Even if noniterative procedures are used, such as the Gaussian elimination algorithm or the matrix decomposition method of Cholesky, this cannot be done with a single-step calculation. Rather the number of operations grows as O(n3 ): With fast digital computers it is certainly a routine matter to solve systems of equations with even thousands of variables however, the inevitable rounding errors mean that complete accuracy is never achieved (Broyden, 1973). One can normally be satis ed with a suciently good approximation. Here relaxation methods, which are iterative, show themselves to be comparable or superior. It depends in detail on the structure of the coecient matrix. Starting from an initial approximation, the error as measured by the residues of the equations is minimized. Relaxation procedures are therefore basically optimization methods but of a special kind, since the value of the objective function at the optimum is known beforehand. This a priori information can be exploited to make savings in the computations, as can the fact that each component of the residue vector must individually go to zero (e.g., Traub, 1964 Wilkinson and Reinsch, 1971 Hestenes, 1973 Hestenes and Stein, 1973). Objective functions having terms or members of higher than second order lead to non-linear equations as the necessary conditions for the existence of extrema. In this case, the stepwise approach to the null position is essential, e.g., with the interpolation method, which was conceived in its original form by Newton (Chap. 3, Sect. 3.1.2.3.2). The equations are linearized about the current approximation point. Linear relations for the correcting terms are then obtained. In this way a complete system of n linear equations has to be solved at each step of the iteration. Occasionally a more convenient approach is to search for the minimum of the function !2 n ~F (x) = X @F @x i=1

i

with the help of a direct optimization method. Besides the fact that F~ (x) goes to zero, not only at the sought for minimum of F (x) but also at its maxima and saddle points, it can sometimes yield non-zero minima of no interest for the solution of the original problem. Thus it is often preferable not to proceed via the conditions of Equation (2.2) but to minimize F (x) directly. Only in special cases do indirect methods lead to faster, more elegant solutions than direct methods. Such is, for example, the case if the necessary existence condition for minima with one variable leads to an algebraic equation, and sectioning algorithms like the computational scheme of Horner can be used or if objective functions are in the form of so-called posynomes, for which Dun, Peterson, and Zener (1967) devised geometric programming, an entirely indirect method.

Particular Problems and Methods of Solution

15

Subsidiary conditions, or constraints, complicate matters. In rare cases equality constraints can be expressed as equations in one variable, that can be eliminated from the objective function, or constraints in the form of inequalities can be made superuous by a transformation of the variables. Otherwise there are the methods of bounded variation and Lagrange multipliers, in addition to penalty functions and the procedures of mathematical programming. The situation is very similar for functional optimization, except that here the indirect methods are still dominant even today. The variational calculus provides as conditions for optima di erential instead of ordinary equations{actually ordinary di erential equations (Euler-Lagrange) or partial di erential equations (Hamilton-Jacobi). In only a few cases can such a system be solved in a straightforward way for the unknown functions. One must usually resort again to the help of a computer. Whether it is advantageous to use a digital or an analogue computer depends on the problem. It is a matter of speed versus accuracy. A hybrid system often turns out to be especially useful. If, however, the solution cannot be found by a purely analytic route, why not choose from the start the direct procedure also for functional optimization? In fact with the increasing complexity of practical problems in numerical optimization, this eld is becoming more important, as illustrated by the work of Daniel (1969), who takes over methods without derivatives from parameter optimization and applies them to the optimization of functionals. An important point in this is the discretization or parameterization of the originally continuous problem, which can be achieved in at least two ways:


By approximation of the desired functions using a sum of suitable known functions or polynomials, so that only the coecients of these remain to be determined (Sirisena, 1973)




By approximation of the desired functions using step functions or sides of polygons, so that only heights and positions of the connecting points remain to be determined

Recasting a functional into a parameter optimization problem has the great advantage that a digital computer can be used straightaway to nd the solution numerically. The disadvantage that the result only represents a suboptimum is often not serious in practice, because the assumed values of parameters of the process are themselves not exactly known (Dixon, 1972a). The experimentally determined numbers are prone to errors or to statistical uncertainties. In any case, large and complicated functional optimization problems cannot be completely solved by the indirect route. The direct procedure can either start directly with the functional to be minimized, if the integration over the substituted function can be carried out (Rayleigh-Ritz method) or with the necessary conditions, the di erential equations, which specify the optimum. In the latter case the integral is replaced by a nite sum of terms (Beveridge and Schechter, 1970). In this situation gradient methods are readily applied (Kelley, 1962 Klessig and Polak, 1973). The detailed way to proceed depends very much on the subsidiary conditions or constraints of the problem.

16

Problems and Methods of Optimization

2.2.5 Constrained Versus Unconstrained Optimization

Special techniques have been developed for handling problems of optimization with constraints. In parameter optimization these are the methods of penalty functions and mathematical programming. In the rst case a modi ed objective function is set up, which
For the minimum problem takes the value F (x) = + 1 in the forbidden region, but which remains unchanged in the allowed (feasible) region (barrier method e.g., used within the evolution strategies, see Chap. 5)
Only near the boundary inside the allowed region, yields values di erent from F (x) and thus keeps the search at a distance from the edge (partial penalty function e.g., used within Rosenbrock's strategy, see Chap. 3, Sect. 3.2.1.3)
Di ers from F (x) over the whole space spanned by the variables (global penalty function) This last is the most common way of treating constraints in the form of inequalities. The main ideas here are due to Carroll (1961 created response surface technique) and to Fiacco and McCormick (1964, 1990 SUMT, sequential unconstrained minimization technique). For the problem

F (x) ! min Gj (x)  0  for all j = 1(1)m Hk (x) = 0  for all k = 1(1)` the penalty function is of the form (with r vj  wk > 0  and Gj > 0) ` m X X F~ (x) = F (x) + r G v(jx) + 1r wk "Hk (x)]2 j =1 j

k=1

The coecients vj and wk are weighting factors for the individual constraints and r is a free parameter. The optimum of F~ (x) will depend on the choice of r, so it is necessary to alter r in a stepwise way. The original extreme value problem is thereby solved by a sequence of optimizations in which r is gradually reduced to zero. One can hope in this way at least to nd good approximations for the required minimum problem within a

nite sequence of optimizations. The choice of suitable values for r is not, however, easy. Fiacco (1974) and Fiacco and McCormick (1968, 1990) give some indications, and also suggest further possibilities for penalty functions. These procedures are usually applied in conjunction with gradient methods. The hemstitching method and the riding the constraints method of Roberts and Lyvers (1961) work by changing the chosen direction whenever a constraint is violated, without using a modi ed objective function. They orient themselves with respect to the gradient of the objective and the derivatives of the constraint functions (Jacobian matrix). In hemstitching, there is always a return into the feasible region, while in riding the constraints the search runs along the active constraint boundaries. The variables are

Particular Problems and Methods of Solution

17

reset into the allowed region by the complex method of M. J. Box (1965) (a direct search strategy) whenever explicit bounds are crossed. Implicit constraints on the other hand are treated as barriers (see Chap. 3, Sect. 3.2.1.6). The methods of mathematical programming, both linear and non-linear, treat the constraints as the main aspect of the problem. They were specially evolved for operations research (Muller-Merbach, 1971) and assume that all variables must always be positive. Such non-negativity conditions allow special solution procedures to be developed. The simplest models of economic processes are linear. There are often no better ones available. For this purpose Dantzig (1966) developed the simplex method of linear programming (see also Krelle and Kunzi, 1958 Hadley, 1962 Weber, 1972). The linear constraints, together with the condition on the signs of the variables, span the feasible region in the form of a polygon (for n = 2) or a polyhedron, sometimes called simplex. Since the objective function is also linear, except in special cases, the desired extremum must lie in a corner of the polyhedron. It is therefore sucient just to examine the corners. The simplex method of Dantzig does this in a particularly economic way, since only those corners are considered in which the objective function has progressively better values. It can even be thought of as a gradient method along the edges of the polyhedron. It can be applied in a straightforward way to many hundreds, even thousands, of variables and constraints. For very large problems, which may have a particular structure, special methods have also been developed (Kunzi and Tan, 1966 Kunzi, 1967). Into this category come the revised and the dual simplex methods, the multiphase and duplex methods, and decomposition algorithms. An unpleasant property of linear programs is that sometimes just small changes of the coecients in the objective function or the constraints can cause a big alteration in the solution. To reveal such dependencies, methods of parametric linear programming and sensitivity analysis have been developed (Dinkelbach, 1969). Most strategies of non-linear programming resemble the simplex method or use it as a subprogram (Abadie, 1972). This is the case in particular for the techniques of quadratic programming, which are conceived for quadratic objective functions and linear constraints. The theory of non-linear programming is based on the optimality conditions developed by Kuhn and Tucker (1951), an extension of the theory of maxima and minima to problems with constraints in the form of inequalities. These can be expressed geometrically as follows: at the optimum (in a corner of the allowed region) the gradient of the objective function lies within the cone formed by the gradients of the active constraints. To start with, this is only a necessary condition. It becomes sucient under certain assumptions concerning the structure of the objective and constraint functions. For minimum problems, the objective function and the feasible region must be convex, that is the constraints must be concave. Such a problem is also called a convex program. Finally the Kuhn-Tucker theorem transforms a convex program into an equivalent saddle point problem (Arrow and Hurwicz, 1956), just as the Lagrange multiplier method does for constraints in the form of equalities. A complete theory of equality constraints is due to Apostol (1957). Non-linear programming is therefore only applicable to convex optimization, in which, to be precise, one must distinguish at least seven types of convexity (Ponstein, 1967). In addition, all the functions are usually required to be continuously di erentiable, with an

18

Problems and Methods of Optimization

analytic speci cation of their partial derivatives. There is an extensive literature on this subject, of which the books by Arrow, Hurwicz, and Uzawa (1958), Zoutendijk (1960), Vajda (1961), Kunzi, Krelle, and Oettli (1962), Kunzi, Tzschach, and Zehnder (1966, 1970), Kunzi and Krelle (1969), Zangwill (1969), Suchowitzki and Awdejewa (1969), Mangasarian (1969), Stoer and Witzgall (1970), Whittle (1971), Luenberger (1973), and Varga (1974) are but a small sample. Kappler (1967) considers some of the procedures from the point of view of gradient methods. Kunzi and Oettli (1969) give a survey of the more extended procedures together with an extensive bibliography. FORTRAN programs are to be found in McMillan (1970), Kuester and Mize (1973), and Land and Powell (1973). Of special importance in control theory are optimization problems in which the constraints are partly speci ed as di erential equations. They are also called non-holonomous constraints. Pontrjagin et al. (1967) have given necessary conditions for the existence of optima in these problems. Their trick was to distinguish between the free control functions to be determined and the local or state functions which are bound by constraints. Although the theory has given a strong foothold to the analytic treatment of optimal control processes, it must be regarded as a case of good luck if a practical problem can be made to yield an exact solution in this way. One must usually resort in the end to numerical approximation methods in order to obtain the desired optimum (e.g., Balakrishnan and Neustadt, 1964, 1967 Rosen, 1966 Leitmann, 1967 Kopp, 1967 Mufti, 1970 Tabak, 1970 Canon, Cullum, and Polak, 1970 Tolle, 1971 Unbehauen, 1971 Boltjanski, 1972 Luenberger, 1972 Polak, 1973).

2.3 Other Special Cases According to the type of variables there are still other special areas of mathematical optimization. In parameter optimization for example the variables can sometimes be restricted to discrete or integer values. The extreme case is if a parameter may only take two distinct values, zero and unity. Mixed variable types can also appear in the same problem hence the terms discrete, integer, binary (or zero-one), and mixed-integer programming. Most of the solution procedures that have been worked out deal with linear integer problems (e.g., those proposed by Gomory, Balas, and Beale). An important class of methods, the branch and bound methods, is described for example by Weinberg (1968). They are classed together with dynamic programming as decision tree strategies. For the general non-linear case, a last resort can be to try out all possibilities. This kind of optimization is referred to as complete enumeration. Since the cost of such a procedure is usually prohibitive, heuristic approaches are also tried, with which usable, not necessarily optimal, solutions can be found (Weinberg and Zehnder, 1969). More clever ways of proceeding in special cases, for example by applying non-integer techniques of linear and non-linear programming, can be found in Korbut and Finkelstein (1971), Greenberg (1971), Plane and McMillan (1971), Burkard (1972), Hu (1972), and Gar nkel and Nemhauser (1972, 1973). By stochastic programming is meant the solution of problems with objective functions, and sometimes also constraints, that are subject to statistical perturbations (Faber, 1970). It is simplest if such problems can be reduced to deterministic ones, for example by working

Other Special Cases

19

with expectation values. However, there are some problems in which the probability distributions signi cantly inuence the optimal solution. Operational methods at rst only existed for special cases such as, for example, warehouse problems (Beckmann, 1971). Their numbers as well as the elds of application are growing steadily (Hammer, 1984 Ermoliev and Wets, 1988 Ermakov, 1992). In general, one has to make a clear distinction between deterministic solution methods for more or less noisy or stochastic situations and stochastic methods for deterministic but dicult situations like multimodal or fractal topologies. Here we refer to the former in Chapter 4 we will do so for the latter, especially under the aspect of global optimization. In a rather new branch within the mathematical programming eld, called non-smooth or non-dierentiable optimization, more or less classical gradient-type methods for nding solutions still persist (e.g., Balinski and Wolfe, 1975 Lemarechal and Mi$in, 1978 Nurminski, 1982 Kiwiel, 1985). For successively approaching the zero or extremum of a function if the measured values are subject to uncertainties, a familiar strategy is that of stochastic approximation (Wasan, 1969). The original concept is due to Robbins and Monro (1951). Kiefer and Wolfowitz (1952) have adapted it for problems in which the maximum of a unimodal regression function is sought. Blum (1954a) has proved that the method is certain to converge. It distinguishes between test or trial steps and work steps. With one variable, starting at the point x(k), the value of the objective function is obtained at the two positions x(k)  c(k). The slope is then calculated as (k) (k) (k ) (k) y(k) = F (x + c )2c;(kF) (x ; c ) A work step follows from the recursion formula (for minimum searches) x(k+1) = x(k) ; 2a(k)y(k) The choice of the positive sequences c(k) and a(k) is important for convergence of the process. These should satisfy the relations lim c(k) ! 0 k!1 1 X a(k) = 1 k=1 1 X a(k)c(k) < 1 k=1 1 X a(k) !2 < 1 c(k) k=1

One chooses for example the sequences (0) a(k) = ak  a(0) > 0 c(0)  c(0) > 0  k > 0 c(k) = p 4 k

20

Problems and Methods of Optimization

This means that the work step length goes to zero very much faster than the test step length, in order to compensate for the growing inuence of the perturbations. Blum (1954b) and Dvoretzky (1956) describe how to apply this process to multidimensional problems. The increment in the objective function, hence an approximation to the gradient vector, is obtained from n + 1 observations. Sacks (1958) uses 2 n trial steps. The stochastic approximation can thus be regarded, in a sense, as a particular gradient method. Yet other basic strategies have been proposed these adopt only the choice of step lengths from the stochastic approximation, while the directions are governed by other criteria. Thomas and Wilde (1964) for example, combine the stochastic approximation with the relaxation method of Southwell (1940, 1946). Kushner (1963) and Schmitt (1969) even take random directions into consideration. All the proofs of convergence of the stochastic approximation assume unimodal objective functions. A further disadvantage is that stability against perturbations is bought at a very high cost, especially if the number of variables is large. How many steps are required to achieve a given accuracy can only be stated if the probability density distribution of the stochastic perturbations is known. Many authors have tried to devise methods in which the basic procedure can be accelerated: e.g., Kesten (1958), who only reduces the step lengths after a change in direction of the search, or Odell (1961), who makes the lengths of the work steps dependent on measured values of the objective function. Other attempts are directed towards reducing the e ect of the perturbations (Venter, 1967 Fabian, 1967), for example by making only the direction and not the size of the gradients determine the step lengths. Bertram (1960) describes various examples of applications. More of such work is that of Krasulina (1972) and Engelhardt (1973). In this introduction many classes of possible or practically occurring optimization problems and methods have been sketched briey, but the coverage is far from complete. No mention has been made, for example, of broken rational programming, nor of graphical methods of solution. In operations research especially (Henn and Kunzi, 1968) there are many special techniques for solving transport, allocation, routing, queuing, and warehouse problems, such as network planning and other graph theoretical methods. This excursion into the vast realm of optimization problems was undertaken because some of the algorithms to be studied in more depth in what follows, especially the random methods of Chapter 4, owe their origin and nomenclature to other elds. It should also be seen to what extent methods of direct parameter optimization permeate the other branches of the subject, and how they are related to each other. An overall scheme of how the various branches are interrelated can be found in Saaty (1970). If there are two or more objectives at the same time and occasion, and especially if these are not conict-free, single solution points in the decision variable space can no longer give the full answer to an optimization question, not even in the otherwise simplest situation. How to look for the whole subset of ecient, non-dominated, or Paretooptimal solutions can be found under keywords like vector optimization, polyoptimization or multiple criteria decision making (MCDM) (e.g., Bell, Keeney, and Rai a, 1977 Hwang and Masud, 1979 Peschel, 1980 Grauer, Lewandowski, and Wierzbicki, 1982 Steuer, 1986). Game theory comes into play when several decision makers have access to di erent

Other Special Cases

21

parts of the decision variable set only (e.g., Luce and Rai a, 1957 Maynard Smith, 1982 Axelrod, 1984 Sigmund, 1993). No consideration is given here to these special elds.

22

Problems and Methods of Optimization

Chapter 3 Hill climbing Strategies In this chapter some of the direct, mathematical parameter optimization methods will be treated in more detail for static, non-discrete, non-stochastic, mostly unconstrained functions. They come under the general heading of hill climbing strategies because their manner of searching for a maximum corresponds closely to the intuitive way a sightless climber might feel his way from a valley up to the highest peak of a mountain. For minimum problems the sense of the displacements is simply reversed, otherwise uphill or ascent and downhill or descent methods (Bach, 1969) are identical. Whereas methods of mathematical programming are dominant in operations research and the special methods of functional optimization in control theory, the hill climbing strategies are most frequently applied in engineering design. Analytic methods often prove unsuitable in this eld


Because the assumptions are not satis ed under which necessary conditions for extrema can be stated (e.g., continuity of the objective function and its derivatives)




Because there are diculties in carrying out the necessary di erentiations




Because a solution of the equations describing the conditions does not always lead to the desired optimum (it can be a local minimum, maximum, or saddle point)




Because the equations describing the conditions, in general a system of simultaneous non-linear equations, are not immediately soluble

To what extent hill climbing strategies take care of these particular characteristics depends on the individual method. Very thorough presentations covering some topics can be found in Wilde (1964), Rosenbrock and Storey (1966), Wilde and Beightler (1967), Kowalik and Osborne (1968), Box, Davies, and Swann (1969), Pierre (1969), Pun (1969), Converse (1970), Cooper and Steinberg (1970), Ho mann and Hofmann (1970), Beveridge and Schechter (1970), Aoki (1971), Zahradnik (1971), Fox (1971), C%ea (1971), Daniel (1971), Himmelblau (1972b), Dixon (1972a), Jacoby, Kowalik and Pizzo (1972), Stark and Nicholls (1972), Brent (1973), Gottfried and Weisman (1973), Vanderplaats (1984), and Papageorgiou (1991). More variations or theoretical and numerical studies of older methods can be found as individual publications in a wide variety of journals, or in the volumes of collected articles such as Graves and Wolfe (1963), Blakemore and Davis (1964), Lavi 23

24

Hill climbing Strategies

and Vogl (1966), Klerer and Korn (1967), Abadie (1967, 1970), Fletcher (1969a), Rosen, Mangasarian, and Ritter (1970), Geo rion (1972), Murray (1972a), Lootsma (1972a), Szego (1972), and Sebastian and Tammer (1990). Formulated as a minimum problem without constraints, the task can be stated as follows: n min (3.1) x fF (x) j x 2 IR g The column vector x
(at the extreme position) is required 2 x
3 66 x1
2 77 x
= 66 .. 77 = (x
1 x
2 : : : x
n)T 4 . 5 x
n and the associated extreme value F
= F (x
) of the objective function F (x), in this case the minimum. The expression x 2 IRn means that the variables are allowed to take all real values x can thus be represented by any point in an n-dimensional Euclidean space IRn. Di erent types of minima are distinguished: strong and weak, local and global. For a local minimum the following relationship holds:

F (x
)  F (x) for 0

v u n uX
kx ; x k = t (xi ; x
i )2 i=1

and

(3.2) "

x 2 IRn This means that in the neighborhood of x
de ned by the size of " there is no vector x for which F (x) is smaller than F (x
). If the equality sign in Equation (3.2) only applies when x = x
, the minimum is called strong, otherwise it is weak. An objective function that only displays one minimum (or maximum) is referred to as unimodal. In many cases, however, F (x) has several local minima (and maxima), which may be of di erent heights. The smallest, absolute or global minimum (minimum minimorum) of a multimodal objective function satis es the stronger condition F (x
)  F (x) 

for all x 2 IRn

(3.3)

This is always the preferred object of the search. If there are also constraints, in the form of inequalities or equalities

Gj (x)  0 

for all j = 1(1)m

(3.4)

Hk (x) = 0 

for all k = 1(1)`

(3.5)

One Dimensional Strategies

25

then IRn in Equations (3.1) to (3.3) must either be replaced by the hopefully non-empty subset M 2 IRn to represent the feasible region in IRn de ned by Equation (3.4), or by IRn;` , the subspace of lower dimensionality spanned by the variables that now depend on each other according to Equation (3.5). If solutions at in nity are excluded, then the theorem of Weierstrass holds (see for example Rothe, 1959): \In a closed compact region a  x  b every function which is continuous there has at least one (i.e., an absolute) minimum and maximum." This can lie inside or on the boundary. In the case of discontinuous functions, every point of discontinuity is also a potential candidate for the position of an extremum.

3.1 One Dimensional Strategies The search for a minimum is especially easy if the objective function only depends on one variable. F(x)

x

a

b c

d

e

f

g

Figure 3.1: Special points of a function of one variable

a: b: c: d-e: f: g: h:

local maximum at the boundary local minimum at a point of discontinuity of Fx(x) saddle point, or point of inection weak local maximum local minimum maximum (may be global) at a point of discontinuity of F (x) global minimum at the boundary

h

26

Hill climbing Strategies

This problem would be of little practical interest, however, were it not for the fact that many of the multidimensional strategies make use of one dimensional minimizations in selected directions, referred to as line searches. Figure 3.1 shows some possible ways minima and other special points can arise in the one dimensional case.

3.1.1 Simultaneous Methods

One possible way of discovering the minimum of a function with one parameter is to determine the value of the objective function at a number of points and then to declare the point with the smallest value the minimum. Since in principle all trials can be carried out at the same time, this procedure is referred to as simultaneous optimization. How closely the true minimum is approached depends on the choice of the number and location of the trial points. The more trials are made, the more accurate the solution can be. One will be concerned, however, to obtain a result at the lowest cost in time and computation (or material). The two requirements of high accuracy and lowest cost are contradictory thus an optimum compromise must be sought. The e ectiveness of a search method is judged by the size of the largest remaining interval of uncertainty (in the least favorable case) relative to the position of the minimum for a given number of trials (the so-called minimax concept, see Wilde, 1964 Beamer and Wilde, 1973). Assuming that the points in the series of trials are so densely distributed that several at a time are in the neighborhood of a local minimum, then the length of the interval of uncertainty is the same as the distance between the two points in the neighborhood of the smallest value of F (x). The number of necessary trials can thus get very large unless one has at least some idea of whereabouts the desired minimum is situated. In practice one must limit investigation of the objective function to a nite interval "a b]. It is obvious, and it can be proved theoretically, that the optimal choice of all simultaneous search methods is the one in which the trial points are evenly distributed over the interval "a b] (Boas, 1962, 1963a{d). If N equidistant points are used, the interval of uncertainty is of length `N = N 2+ 1 (b ; a) and the e ectiveness takes the value  = N 2+ 1 Put another way: To be sure of achieving an accuracy of " > 0, the equidistant search (also called lattice, grid, or tabulation method ) requires N trials, where 2(b ; a) ; 1 < N  2(b ; a)  N integer (3.6) " " Apart from the requirements that the chosen interval "a b] should contain the absolute minimum being sought and that N should be big enough in relation to the \wavyness" of the objective function, no further conditions need to be ful lled in order for the grid method to succeed.

One Dimensional Strategies

27

Even more e ective search schemes can be devised if the objective function is unimodal in the interval "a b]. Wilde and Beightler (1967) describe a procedure, using evenly distributed pairs of points, which is also referred to as a simultaneous dichotomous search. The distance between two points of a pair must be chosen to be suciently large that their objective function values are di erent. As ! 0 the dichotomous search with an even number of trials (even block search) is the best. The number of trials required is 2(b ; a) ; 2 < N  2(b ; a) ; 1  N integer (3.7) " " This is one less than for equidistant searches with the same accuracy requirement. Such a scheme is referred to as optimal in the sense of the -minimax concept. Investigations of arrangements of trials also for uneven numbers (odd block search), with non-vanishing , can be found in Wilde (1966) and Wilde and Beightler (1967). The one dimensional search procedures dealt with so far, which strive to reduce the interval of uncertainty, are called by Wilde (1964) direct elimination procedures. As a second category one can consider the various interpolation methods. These can be much more e ective, but only when the objective function F (x) satis es certain \smoothness" conditions, or can be suciently well approximated in the interval under consideration by a polynomial P (x). In this case, instead of the minimum of F (x), a zero of Px (x) = dP (x)=dx is determined. The number of points at which the objective function must be investigated depends on the order of the chosen polynomial and on the type of information at the points that determine it. Besides the values of the objective function itself, consideration is given to its rst, second, and, less frequently, higher derivatives. In general, no exact statements can be made regarding the quality of the approximation to the desired minimum for a given number of trials. Details of the various methods of locating real zeros of rational functions of one variable, such as regula falsi, Lagrangian and Newtonian interpolation, can be found in books on practical or numerical mathematics under the heading of non-linear equations: e.g., Booth (1967), Faddejew and Faddejewa (1973), Saaty and Bram (1964), Traub (1964), Zurmuhl (1965), Walsh (1966), Ralston and Wilf (1967, 1969), Householder (1970), Ortega and Rheinboldt (1970), and Brent (1973). Although from a fundamental point of view interpolation methods represent indirect optimization procedures, they are of interest here as line search strategies, especially when they are applied iteratively and obtain information about derivatives from function values.

3.1.2 Sequential Methods

If the trials for determining a minimum can be made sequentially, the intermediate results retained at a given time can be used to locate the next trial of the sequence more favorably than would be possible without this information. With the digital computers usually available nowadays, which work in a serial way, one is actually obliged to execute all steps one after the other. Sequential methods, in which the solution is approached in a stepwise, or iterative, manner, are advantageous here. The evaluation of the intermediate results and prediction

28

Hill climbing Strategies

of favorable conditions for the next trial presupposes a more or less precise internal model of the objective function the better the model corresponds to reality, the better will be the results of the interpolation and extrapolation processes. The simplest assumption is that the objective function is unimodal, which means that local minima also always represent global minima. On this basis a number of sequential interval-dividing procedures have been constructed (Sect. 3.1.2.2). Iterative interpolation methods demand more \smoothness" of the objective function (Sect. 3.1.2.3). In the former case it is necessary, in the latter useful, to determine at the outset a suitable interval, "a(0) b(0)], in which the desired extremum lies (Sect. 3.1.2.1).

3.1.2.1 Boxing in the Minimum

If there are no clues as to whereabouts the desired minimum might be situated, one can start with two points x(0) and x(1) = x(0) + s and determine the objective function there. If F (x(1)) < F (x(0)) one proceeds in the chosen direction keeping the same step length: x(k+1) = x(k) + s until F (x(k+1)) > F (x(k)) If, however, F (x(1)) > F (x(0)), one chooses the opposite direction: x(2) = x(0) ; s and x(k+1) = x(k) ; s  for k  2 similarly, until a step past the minimum is taken, one has thus determined the minimum of the unimodal function to within an uncertainty interval of length 2 s (Beveridge and Schechter, 1970). In numerical optimization problems the values of the variables often run through several powers of 10, or alternatively they must be precisely determined at many points. In this case the boxing-in method with a very small xed step length is too costly. Box, Davies, and Swann (1969) therefore suggest starting with an initial step length s(0) and doubling it at each successful step. Their recursion formula is as follows: x(k+1) = x(0) + 2k s(0) It is applied as long as F (x(k+1))  F (x(k)) holds. As soon as F (x(k+1)) > F (x(k)) is registered, however, b(0) = x(k+1) is set as the upper bound to the interval and the starting point x(0) is returned to. The lower bound a(0) is found by a corresponding process with negative step lengths going in the opposite direction. In this way a starting interval "a(0) b(0)] is obtained for the one dimensional search procedure to be described below. It can happen, because of the convention for equality of two function values, that the search for a bound to the interval does not end if the objective function reaches a constant horizontal level. It is therefore useful to specify a maximum step length that may not be exceeded.

One Dimensional Strategies

29

The boxing-in method has also been proposed occasionally as a one dimensional optimization strategy (Rosenbrock, 1960 Berman, 1966) in its own right. In order not to waste too many trials far from the target when the accuracy requirement is very high, it is useful to start with relatively large steps. Each time a loop ends with a failure the step length is reduced by a factor less than 0:5, e.g., 0:25. If the above rules for increasing and reducing the step lengths are combined, a very exible procedure is obtained. Dixon (1972a) calls it the success/failure routine. If a starting interval "a(0) b(0)] is already at hand, however, there are signi cantly better strategies for successively reducing the size of the interval.

3.1.2.2 Interval Division Methods

If an equidistant division method is applied repeatedly, the interval of uncertainty is reduced at each step by the same factor , and thus for k steps by k . This exponential progression is considerably stronger than the linear dependence of the value of on the number of trials per step. Thus as few simultaneous trials as possible would be used. A comparison of two schemes, with two and three simultaneous trials, shows that except in the rst loop, only two new objective function values must be obtained at a time in both cases, since of three trial points in one step, one coincides with a point of the previous step. The total number of trials required with sequential application of the equidistant three point scheme is     2 log b;" a 2 log b;" a N odd (3.8) 1 + log 2 < N  3 + log 2  Even better results are provided by the sequential dichotomous search with one pair per step. For the limiting case ! 0 one obtains     2 log b;" a 2 log b;" a N even (3.9) log 2 < N  2 + log 2  Detailed investigations of the inuence of on various equidistant and dichotomous search schemes can be found in Avriel and Wilde (1966b), and Beamer and Wilde (1970). Of greater interest are the two purely sequential elimination methods described in the following chapters, which only require a single objective function value per step. They require that the objective function be unimodal, otherwise they only guarantee that a local minimum or maximum will be found. Shubert (1972) describes an interval dividing procedure that is able to locate all local extrema, including the global optimum. To use it, however, an upper bound to the slope of the function must be known. The method is rather costly, especially with regard to the storage space required.

3.1.2.2.1 Fibonacci Division. This interval division strategy was introduced by

Kiefer (1953). It operates with a series due to Leonardo of Pisa, which bears his pseudonym Fibonacci: f0 = f1 = 1

30

Hill climbing Strategies

fk = fk;1 + fk;2  for k  2 An initial interval "a(0) b(0)] is required, containing the extremum together with a number N , which represents the total number of intended interval divisions. If the general interval is called "a(k) b(k)], the lengths s(k) = t(k) (b(k) ; a(k)) = (b(k+1) ; a(k+1)) are subtracted from its ends, with the reduction factor t(k) = ffN ;k;1 (3.10) N ;k giving c(k) = a(k) + s(k) d(k) = b(k) ; s(k) The values of the objective function at c(k) and d(k) are compared and whichever subinterval contains the better (in a minimum search, lower) value is taken as de ning the interval for the next step. If F (d(k)) < F (c(k)) then a(k+1) = a(k) and b(k+1) = c(k) If F (d(k)) > F (c(k)) then a(k+1) = d(k) and b(k+1) = b(k) A consequence of the Fibonacci series is that, except for the rst interval division, at all of the following steps one of the two new points c(k+1) and d(k+1) is always already known. If F (d(k)) < F (c(k)) then c(k+1) = d(k) and if F (d(k)) > F (c(k)) then d(k+1) = c(k)

One Dimensional Strategies

31

F(x)

x a

d

a

d

c

a

d c

b

c

b

b

Step k Step k+1 Step k+2

Figure 3.2: Interval division in the Fibonacci search

so that each time only one new value of the objective function needs to be obtained. Figure 3.2 illustrates two steps of the procedure. The process is continued until k = N ; 2. At the next division, because f2 = 2 f1 d(k) and c(k) coincide. A further interval reduction can only be achieved by slightly displacing one of the test points. The displacement must be at least big enough for the two objective function values to still be distinguishable. Then the remaining interval after N trials is of length `N = f1 (b(0) ; a(0)) + N As ! 0 the e ectiveness tends to fN;1 . Johnson (1956) and Kiefer (1957) show that this value is optimal in the sense of the -minimax concept, according to which the Fibonacci search is the best of all sequential interval division procedures. However, by taking account of the displacement, not only at the last but at all the steps, Oliver and Wilde (1964) give a recursion formula that for the same number of trials yields a slightly smaller residual interval. Avriel and Wilde (1966a) provide a proof of optimality. If one has a priori information about the structure of the objective function it can be exploited to advantage (Gal, 1971) in order to reduce further the number of trials. Overholt (1967a, 1973) suggests that in general there is no a priori information available to x suitably, and it is therefore better to omit the nal division using a displacement rule and to choose N one bigger from the start. In order to obtain the minimum with accuracy " > 0 one

32

Hill climbing Strategies

should choose N such that

(0) (0) fN > b ;" a  fN ;1 Then the e ectiveness of the procedure becomes =f2 N +1 and since (Lucas, 1876) 2 p !N +1 p !N +13 p !N +1 1 1 + 1 ; 1 5 5 5' p 1+ 5 fN = p 4 2 ; 2 2 5 5

the number of trials is approximately

p

b(0) ;a(0) + log 5 log " p N' log 1+2 5



(0) b ; a(0) log "

(3.11)

Overholt (1965) shows by means of numerical tests that the procedure must often be terminated prematurely as F (d(k+1)) becomes equal to F (c(k+1)), for example because of computing with a nite number of signi cant gures. Further divisions of the interval of uncertainty are then pointless. For the boxing-in method of determining the initial interval one would x an initial step length of about 10 " and a maximum step length of about 5  109 ", so that for a 36-bit computer the number range of integers is not exceeded by the largest required Fibonacci number. Finally, two further applications of the Fibonacci procedure may be mentioned. By reversing the scheme, Wilde and Beightler (1967) obtain a method of boxing in the minimum. Kiefer (1957) shows how to proceed if values of the objective function can only be obtained at discrete, not necessarily equidistant, points. More about such lattice search problems can be found in Wilde (1964), and Beveridge and Schechter (1970).

3.1.2.2.2 The Golden Section. It can sometimes be inconvenient to have to specify in advance the number of interval divisions. In this case Kiefer (1953) and Johnson (1956) propose, instead of the reduction factor t(k), which varies with the iteration number in the Fibonacci search, a constant factor (3.12) t = 2p ' 0:618  (positive root of: t2 + t = 1) 1+ 5 For large N ; k t(k) reduces to t. In addition, t is identical to the ratio of lengths a to b, which is obtained by dividing a total length of a + b into two pieces such that the smaller, a, has the same ratio to the larger, b, as the larger to the total. This harmonic division (after Euclid) is also known as the golden section, which gave the procedure its name (Wilde, 1964). After N function calls the uncertainty interval is of length

`N = tN ;1 (b(0) ; a(0))

One Dimensional Strategies For the limiting case N

! 1,

33 since lim (tN ;1 fN ) = 1:17

N !1

the number of trials compared to the Fibonacci procedure is about 17% higher. Compared to the Fibonacci search without displacement, since 1  N ;1 f lim t N +1 ' 0:95 N !1 2 the number of trials is about 5% lower. It should further be noted that, when using the Fibonacci method on digital computers, the Fibonacci numbers must rst be generated, or a sucient number of them must be provided and stored. The number of trials needed for a sequential golden section is 2 b(0);a(0) 3 (0) (0) N = 66 loglog "t 77 ; 1  log b ;" a (3.13) 6 7 Other properties of the iteration sequence, including the criterion for termination at equal function values, are the same as those of the method of interval division according to Fibonacci numbers. Further details can be found, for example, in Avriel and Wilde (1968). Complete programs for the interval division procedures have been published by Pike and Pixner (1965), and Overholt (1967b,c) (see also Boothroyd, 1965 Pike, Hill, and James, 1967 Overholt, 1967a).

3.1.2.3 Interpolation Methods

In many cases one is dealing with a continuous function, the minimum of which is to be determined. If, in addition to the value of the objective function, its slope can be speci ed everywhere, many methods can be derived that may converge faster than the optimal elimination methods. One of the oldest schemes is based on the procedure named after Bolzano for determining the zeros of a function. Assuming that one has two points at which the slopes of the objective function have opposite signs, one bisects the interval between them and determines the slope at the midpoint. This replaces the interval end point, which has a slope of the same sign. The procedure can then be repeated iteratively. At each trial the interval is halved. If the slope has to be calculated from the di erence of two objective function values, the bisection or midpoint strategy becomes the sequential dichotomous search. Avriel and Wilde (1966b) propose, as a variant of the Bolzano search, evaluating the slope at two points in the interval so as to increase the reduction factor. They show that their diblock strategy is slightly superior to the dichotomous search. If derivatives of the objective function are available, or at least if it can be assumed that these exist, i.e., the function F (x) is continuous and di erentiable, far better strategies for the minimum search can be devised. They determine analytically the minimum of a trial function that coincides with the objective function, and possibly also its derivatives, at selected argument values. One distinguishes linear, quadratic, and cubic models according to the order of the trial polynomial. Polynomials of higher order are virtually never used.

34

Hill climbing Strategies

They require too much information about the function F (x). Furthermore, it turns out that in contrast to all the methods referred to so far such strategies do not always converge, for reasons other than rounding error.

3.1.2.3.1 Regula Falsi Iteration. Given two points a(k) and b(k), with their function values F (a(k)) and F (b(k)), the simplest approximation formula for a zero c(k) of F (x) is

(k) b ; a(k) c = a ; F (a ) F (b(k)) ; F (a(k)) This technique, known as regula falsi or regula falsorum, predicts the position of the zero correctly if F (x) depends linearly on x. For one dimensional minimization it can be applied to nd a zero of Fx(x) = dF (x)=dx: (k)

(k)

(k)

(k) a(k) c(k) = a(k) ; Fx(a(k)) F (bb(k)) ; ; F (a(k))

x

x

(3.14)

The underlying model here is a second order polynomial with linear slope. If Fx(a(k)) and Fx(b(k)) have opposite sign, c(k) lies between a(k) and b(k). If Fx(c(k)) 6= 0, the procedure can be continued iteratively by using the reduced interval "a(k+1) b(k+1)] = "a(k) c(k)] if Fx(c(k)) and Fx(b(k)) have the same sign, or using "a(k+1) b(k+1)] = "c(k) b(k)] if Fx(c(k)) and Fx(a(k)) have the same sign. If Fx(a(k)) and Fx(b(k)) have the same sign, c(k) must lie outside "a(k) b(k)]. If Fx(c(k)) has the same sign again, c(k) replaces the argument value at which jFxj is greatest. This extrapolation is also called the secant method. If Fx(c(k)) has the opposite sign, one can continue using regula falsi to interpolate iteratively. As a termination criterion one can apply Fx(c(k)) = 0 or jFx(c(k))j  ", " > 0. A minimum can only be found reliably in this way if the starting point of the search lies in its neighborhood. Otherwise the iteration sequence can also converge to a maximum, at which, of course, the slope also goes to zero if Fx(x) is continuous. Whereas in the Bolzano interval bisection method only the sign of the function whose zero is sought needs to be known at the argument values, the regula falsi method also makes use of the magnitude of the function. This extra information should enable it to converge more rapidly. As Ostrowski (1966) and Jarratt (1967, 1968) show, for example, this is only the case if the function corresponds closely enough to the assumed model. The simpler bisection method is better, even optimal (as a zero method in the minimax sense), if the function has opposite signs at the two starting points, is not linear and not convex. In this case the linear interpolation sometimes converges very slowly. According to Stanton (1969), a cubic interpolation as a line search in the eccentric quadratic case often yields even worse results. Dixon (1972a) names two variants of the regula falsi recursion formula, but it is not known whether they lead to better convergence. Fox (1971) proposes a combination of the Bolzano method with the linear interpolation. Dekker (1969) (see also Forsythe, 1969) accredits this procedure with better than linear convergence. Even greater reliability and speed is attributed to the algorithm of Brent (1971), which follows Dekker's method by a quadratic interpolation process as soon as the latter promises to be successful.

One Dimensional Strategies

35

It is inconvenient when dealing with minimization problems that the derivatives of the function are required. If the slopes are obtained from function values by a di erence method, diculties can arise from the nite accuracy of such a process. For this reason Brent (1973) combines regula falsi iteration with division according to the golden section. Further variations can be found in Schmidt and Trinkaus (1966), Dowell and Jarratt (1972), King (1973), and Anderson and Bjorck (1973).

3.1.2.3.2 Newton-Raphson Iteration. Newton's interpolation formula for improv-

ing an approximate solution x(k) to the equation F (x) = 0 (see for example Madsen, 1973) (k) x(k+1) = x(k) ; FF ((xx(k))) x uses only one argument value, but requires the value of the derivative of the function as well as the function itself. If F (x) is linear in x, the zero is correctly predicted here, otherwise an improved approximation is obtained at best, and the process must be repeated. Like regula falsi, Newton's recursion formula can also be applied to determining Fx(x) = 0, with of course the reservations already stated. The so-called Newton-Raphson rule is then (k) (3.15) x(k+1) = x(k) ; FFx((xx(k))) xx

If F (x) is not quadratic, the necessary number of iterations must be made until a termination criterion is satis ed. Dixon (1972a) for example uses the condition jx(k+1) ; x(k)j < ". To set against the advantages that only one argument value is required, and, for quadratic objective functions, one iteration is sucient to nd a point where Fx(x) = 0, there are several disadvantages:






If the derivatives Fx and Fxx are obtained approximately by numerical di erentiation, the eciency of the procedure is worsened not only by rounding errors but also by inaccuracies in the approximation. This is especially true in the neighborhood of the minimum being sought, since Fx becomes vanishingly small there. Minima, maxima, and saddle points are not distinguished. The starting point x(0) must already be located as close as possible to the minimum being sought. If the objective function is of higher than second order, the Newton-Raphson iteration can diverge. The condition for convergence towards a minimum is that Fxx(x) > 0 for all x.

3.1.2.3.3 Lagrangian Interpolation. Whereas regula falsi and Newton-Raphson it-

eration attempt to approximate the minimum using information about the derivatives of the objective function at the argument values and can therefore be classi ed as indirect optimization methods, in Lagrangian interpolation only values of the objective function itself are required. In its general form the procedure consists of tting a pth order polynomial through p + 1 points (Zurmuhl, 1965). One usually uses three argument values and a

36

Hill climbing Strategies

parabola as the model function (quadratic interpolation). Assuming that the three points are a(k) < b(k) < c(k), with the objective function values F (a(k)) F (b(k)) and F (c(k)), the trial parabola P (x) has a vanishing rst derivative at the point (k) 2 (k) 2 (k) (k) 2 ; (a(k))2 ] F (b(k)) + "(a(k))2 ; (b(k) )2 ] F (c(k)) d(k) = 12 "(b ) ;"b((ck) ;) c](Fk)](Fa (a)(k+))"(+c "c()k) ; a(k)] F (b(k)) + "a(k) ; b(k)] F (c(k)) (3.16) This point is a minimum only if the denominator is positive. Otherwise d(k) represents a maximum or a saddle point. In the case of a minimum, d(k) is introduced as a new argument value and one of the old ones is deleted: 9 ( (k) (k) (k) a(k+1) = a(k) > = if a < d < b b(k+1) = d(k) > and F (d(k)) < F (b(k)) c(k+1) = b(k) 

9 ( (k) (k) (k) > a(k+1) = d(k) = if a < d < b ( k +1) ( k ) b =b > and F (d(k)) > F (b(k)) c(k+1) = c(k)  9 ( (k) (k) (k) a(k+1) = b(k) > = if b < d < c (k+1) (k) b =d > and F (d(k)) < F (b(k)) c(k+1) = c(k)  9 ( (k) (k) (k) a(k+1) = a(k) > = if b < d < c b(k+1) = b(k) > (3.17) and F (d(k)) > F (b(k)) c(k+1) = d(k)  Figure 3.3 shows one of the possible cases. It is useful to determine the ends of the interval a(0) and c(0) at the start of the one dimensional search by a procedure such as one of those described in Section 3.1.2.1. The third argument value is best positioned at the center of the interval. It can be seen from the recursion formula (Equation (3.17)) that only one value of the objective function needs to be obtained at each iteration, at the current position of d(k). The three points clearly do not in general remain equidistant. When the interpolation formula (Equation (3.16)) predicts a minimum that coincides to the desired accuracy " with one of the argument values, the search can be terminated. As the result, one will select the smallest objective function value from among the argument values at hand and the computed minimum. There is little prospect of success if the minimum is predicted to lie outside the interval "a(k) c(k)], at any rate when a bounding procedure of the type described in Section 3.1.2.1 has been applied initially. In this case too the procedure is best terminated. The same holds if a negative denominator in Equation (3.16) indicates a maximum, or if a point of inection is expected because the denominator vanishes. If the measured objective function values are subject to error, for example in experimental optimization, special precautions must be taken. Hotelling (1941) treats this problem in detail. How often the interpolation must be repeated until the minimum is suciently well approximated cannot be predicted in general it depends on the level of agreement between

One Dimensional Strategies F P

F(x)

37

Minimum P(x)

P(x)

x a

(k)

(k+1)

a

d

(k)

b

(k)

c

(k)

(k+1) (k+1)

b

c

Figure 3.3: Lagrangian quadratic interpolation

the objective function and the trial function. In the most favorable case the objective function is also quadratic. Then one iteration is sucient. This is why it can be advantageous to use an interpolation method rather than an interval division method such as the optimal Fibonacci search. Dijkhuis (1971) describes a variant of the basic procedure in which four argument values are taken. The two inner ones and each of the outer ones in turn are used for two separate quadratic interpolations. The weighted mean of the two results yields a new iteration point. This procedure is claimed to increase the reliability of the minimum search for non-quadratic objective functions.

3.1.2.3.4 Hermitian Interpolation. If one chooses, instead of a parabola, a third

order polynomial as a test function, more information is needed to make it agree with the objective function. Beveridge and Schechter (1970) describe such a cubic interpolation procedure. In place of four argument values and associated objective function values, two points a(k) and b(k) are enough, if, in addition to the values of the objective function, values of its slope, i.e., the rst order di erentials, are available. This Hermitian interpolation is mainly used in conjunction with gradient or quasi-Newton methods, because in any case they require the partial derivatives of the objective function, or they approximate them using nite di erence methods. The interpolation formula is: (k) c(k) = a(k) + (b(k) ; a(k)) 2 w +w F; F(bx((ka)) ;) F; (za(k)) x x

38 where and

Hill climbing Strategies (k) (k ) z = 3 "F ((aa(k));;bF(k)()b )] ; Fx(a(k)) ; Fx(b(k))

(3.18)

q w = + z2 ; Fx(a(k)) Fx(b(k)) Recursive exchange of the argument values takes place according to the sign of Fx(c(k)) in a similar way to the Bolzano method. It should also be veri ed here that a(k) and b(k) always bound the minimum. Fletcher and Reeves (1964) use Hermitian interpolation in their conjugate gradient method as a subroutine to approximate a relative minimum in speci ed directions. They terminate the iteration as soon as ja(k) ; b(k)j < ". As in all interpolation procedures, the speed and reliability of convergence depend on the degree of agreement between the model function and the objective function. Pearson (1969), for example, reports that the Fibonacci search is superior to Hermitian interpolation if the objective function is logarithmic or a polynomial of high order. Guilfoyle, Johnson, and Wheatley (1967) propose a combination of cubic interpolation and either the Fibonacci search or the golden section.

3.2 Multidimensional Strategies There have been frequent attempts to extend the basic ideas of one dimensional optimization procedures to several dimensions. The equidistant grid strategy, also known in the experimental eld as the method of factorial design, places an evenly meshed grid over the space under consideration and evaluates the objective function at all the nodes. If n is the dimension of the space under consideration and Ni (i = 1 2 : : :  n) is the number of discrete values that the variable xi can take, then the number of combinations to be tested is given by the product n Y N = Ni (3.19) i=1

If Ni = N1 for all i = 1(1)n, one obtains N = N1n. This exponential increase in the number of trials and the computational requirements is what provoked Bellman's now famous curse of dimensions (see Wilde and Beightler, 1967). On a traditional computer, which works sequentially, the trials must all be carried out one after another. The computation time therefore increases as O(cn ), in which the constant c depends on the required accuracy and the size of the interval to be investigated. Proceeding sequentially brought considerable advantages in the one dimensional case if it could only be assumed that the objective function was unimodal. Krolak and Cooper (1963) (see also Krolak, 1968) and Sugie (1964) have given an extension to several dimensions of the Fibonacci search scheme. For n = 2, two points are chosen on one of the two coordinate axes within the given interval in the same way as for the usual one dimensional Fibonacci search. The values of this variable are rst held constant while two complete Fibonacci searches are made to nd the relative optima with respect to

Multidimensional Strategies

39

the second variable. Both end results are then used to reject one of the values of the

rst variable that were held constant, and to reduce the size of the interval with respect to this parameter. By analogy, a three dimensional minimization consists of a recursive sequence of two dimensional Fibonacci searches. If the number of function calls to reduce the uncertainty interval "ai bi] suciently with respect to the variable xi is Ni, then the total number N also obeys Equation (3.19). The advantage compared to the grid method is simply that Ni depends logarithmically on the ratio of initial interval size to accuracy (see Equation (3.11)). Aside from the fact that each variable must be suitably xed in advance, and that the unimodality requirement of the objective function only guarantees that local minima are approached, there is furthermore no guarantee that a desired accuracy will be reached within a nite number of objective function calls (Kaupe, 1964). Other elimination procedures have been extended in a similar way to the multivariable case, such as, for example, the dichotomous search (Wilde, 1965) and a sequential boxingin method (Berman, 1969). In each case the e ort rises exponentially with the number of variables. Another elimination concept for the multidimensional case, the method of contour tangents, is due to Wilde (1963) (see also Beamer and Wilde, 1969). It requires, however, the determination of gradient vectors. Newman (1965) indicates how to proceed in the two dimensional case, and also for discrete values of the variables (lattice search). He requires that F (x) be convex and unimodal. Then the cost should only increase linearly with the number of variables. For n  3, however, no applications of the contour tangent method are as yet known. Transferring interpolation methods to the n-dimensional case means transforming the original minimum problem into a series of problems, in the form of a set of equations to be solved. As non-linear equations can only be solved iteratively, this procedure is limited to the special case of linear interpolation with quadratic objective functions. Practical algorithms based on the regula falsi iteration can be found in Schmidt and Schwetlick (1968) and Schwetlick (1970). The procedure is not widely used as a minimization method (Schmidt and Vetters, 1970). The slopes of the objective function that it requires are implicitly calculated from function values. The secant method described by Wolfe (1959b) for solving a system of non-linear equations also works without derivatives of the functions. From n + 1 current argument values, it extracts the required information about the structure of the n equations. Just as the transition from simultaneous to sequential one dimensional search methods reduces the e ort required at the expense of global convergence, so each further acceleration in the multidimensional case is bought by a reduction in reliability. High convergence rates are achieved by gathering more information and interpreting it in the form of a model of the objective function. If assumptions and reality agree, then this procedure is successful if they do not agree, then extrapolations lead to worse predictions and possibly even to abandoning an optimization strategy. Figure 3.4 shows the contour diagram of a smooth two parameter objective function. All the strategies to be described assume a degree of smoothness in the objective function. They do not converge with certainty to the global minimum but at best to one of the local minima, or sometimes only to a saddle point.

40

Hill climbing Strategies x2

b

c d

e a

a:

Global minimum

b:

Local minimum

c:

Local maxima

d,e:

Saddle points

c

x1

Figure 3.4: Contour lines of a two parameter function (

F x1
x2

)

Various methods are distinguished according to the kind of information they need, namely:
Direct search methods, which only need objective function values F (x)
Gradient methods, which also use the rst partial derivatives rF (x) ( rst order strategies)
Newton methods, which in addition make use of the second partial derivatives r2F (x) (second order strategies) The emphasis here will be placed on derivative-free strategies, that is on direct search methods, and on such higher order procedures as glean their required information about derivatives from a sequence of function values. The recursion scheme of most multidimensional strategies is based on the formula: x(k+1) = x(k) + s(k) v(k) (3.20) They di er from each other with regard to the choice of step length s(k) and search direction v(k), the former being a scalar and the latter a vector of unit length.

3.2.1 Direct Search Strategies

Direct search strategies do without constructing a model of the objective function. Instead, the directions, and to some extent also step lengths, are xed heuristically, or by

Multidimensional Strategies

41

a scheme of some sort, not always in an optimal way under the assumption of a speci ed internal model. Thus the risk is run of not being able to improve the objective function value at each step. Failures must accordingly be planned for, if something can also be \learned" from them. This trial character of search strategies has earned them the name of trial-and-error methods. The most important of them that are still in current use will be presented in the following chapters. Their attraction lies not in theoretical proofs of convergence and rates of convergence, but in their simplicity and the fact that they have proved themselves in practice. In the case of convex or quadratic unimodal objective functions, however, they are generally inferior to the rst and second order strategies to be described later.

3.2.1.1 Coordinate Strategy

The oldest of multidimensional search procedures trades under a variety of names (e.g., successive variation of the variables, relaxation, parallel axis search, univariate or univariant search, one-variable-at-a-time method, axial iteration technique, cyclic coordinate ascent method, alternating variable search, sectioning method, Gauss-Seidel strategy) and manifests itself in a large number of variations. The basic idea of the coordinate strategy, as it will be called here, comes from linear algebra and was rst put into practice by Gauss and Seidel in the single step relaxation method of solving systems of linear equations (see Ortega and Rocko , 1966 Ortega and Rheinboldt, 1967 VanNorton, 1967 Schwarz, Rutishauser, and Stiefel, 1968). As an optimization strategy it is attributed to Southwell (1940, 1946) or Friedmann and Savage (1947) (see also D'Esopo, 1959 Zangwill, 1969 Zadeh, 1970 Schechter, 1970). The parameters in the iteration formula (3.20) are varied in turn individually, i.e., the search directions are xed by the rule: (  if k = p n  p integer v(k) = e`  with ` = nk (mod n)  otherwise where e` is the unit vector whose components have the value zero for all i 6= `, and unity for i = `. In its simplest form the coordinate strategy uses a constant step length s(k). Since, however, the direction to the minimum is unknown, both positive and negative values of s(k) must be tried. In a rst and easy improvement on the basic procedure, a successful step is followed by further steps in the same direction, until a worsening of the objective function is noted. It is clear that the choice of step length strongly inuences the number of trials required on the one hand and the accuracy that can be achieved in the approximation on the other. One can avoid the problem of the choice of step length most e ectively by using a line search method each time to locate the relative optimum in the chosen direction. Besides the interval division methods, the Fibonacci search and the golden section, Lagrangian interpolation can also be used, since all these procedures work without knowledge of the partial derivatives of the objective function. A further strategy for boxing in the minimum must be added, in order to establish a suitable starting interval for each one dimensional minimization.

42

Hill climbing Strategies

The algorithm can be described as follows: Step 0: (Initialization) Establish a starting point x(00) and choose an accuracy bound " > 0 for the one dimensional search. Set k = 0 and i = 1. Step 1: (Boxing in the minimum) Starting from x(ki;1) with an initial step length s = smin (e.g., smin = 10 "), box in the minimum in the direction ei. Double the step length at each successful trial, as long as s < smax (e.g., smax = 5  109 "). De ne the interval limits a(ik) and b(ik). Step 2: (Line search) By varying xi within the interval "a(ik) b(ik)] search for the relative minimum x0 with the required accuracy " (line search with an interval division procedure or an iterative interpolation method): (ki;1) + s e )g F (x0) = min i s fF (x Step 3: (Check for improvement) If F (x0)  F (x(ki;1)), then set x(ki) = x0 otherwise set x(ki) = x(ki;1). Step 4: (Inner loop over all coordinates) If i < n, increase i i + 1 and go to step 1. Step 5: (Termination criterion, outer iteration loop) 0) = x(kn) . Set x(k+1(k+1 If all jxi 0) ; x(ik0)j  5 " for all i = 1(1)n, then end the search otherwise increase k k + 1, set i = 1, and go to step 1. Figure 3.5 shows a typical sequence of iteration points for n = 2 variables. In theory the coordinate strategy always converges if F (x) has continuous partial derivatives and the line searches are carried out exactly (Kowalik and Osborne, 1968 Schechter, 1968 Ortega and Rheinboldt, 1970). Rapid convergence is only achieved, however, if the contour surfaces F (x) = const: are approximately concentric surfaces of a hypersphere, or, in the case of elliptic contours, if the principle axes almost coincide with the coordinate axes. If the signi cant system parameters inuence each other (non-separable objective function), then the distances covered by each line search are small without the minimum being within striking distance. This is especially true when the number of variables is large. At discontinuities in the objective function it may happen that improvements in the objective function can only be made in directions that are not along a coordinate axis. In this case the coordinate strategy fails. There is a similar outcome at steep \valleys" of a continuously di erentiable function, i.e., if the step lengths that would enable successful

Variable values

k

i

x x

(0)

0

0

0

9

(1)

0

1

3

9

(2)

0

(2)

1

0

3

5

(3)

1

1

7

5

(4)

1

(4)

2

0

7

3

(5)

2

1

9

3

(6)

2

2

9

2

Numbering

Direction index

43 Iteration index

Multidimensional Strategies

e2

Start x x

(0,1)

(0,0)

x

(1,1)

(0,2) (1,0)

x

=

x

x

(2,1)

(1,2) (2,0)

x

=

x

x

(2,2) =

(3,0)

x

End

1

2

2 3 5 carried over

2 7 3 carried over

carried over

e1

(6)

3

0

9

2

Figure 3.5: Coordinate strategy

convergence are so small that the number of signi cant gures to which data are handled by the computer is insucient for the variables to be signi cantly altered. Numerical tests with the coordinate strategy show that an exact determination of the relative minima is unnecessary, at least at distances far from the objective. It can even happen that one inaccurate line search can make the next one particularly e ective. This phenomenon is exploited in the procedures known as under- or overrelaxation (Engeli, Ginsburg, Rutishauser, and Stiefel, 1959 Varga, 1962 Schechter, 1962, 1968 Cryer, 1971). Although the relative optimum is determined as before, either an increment is added on in the same direction or an iteration point is de ned on the route between the start and nish of the one dimensional search. The choice of the under- or overrelaxation factor requires assumptions about the structure of the problem. The necessary information is available for the problem of solving systems of linear equations with a positive de nite matrix of coecients, but not for general optimization problems. Further possible variations of the coordinate strategy are obtained if the sequence of searches parallel to the axes is not made to follow the cyclic scheme. Southwell (1946), for example, always selects either the direction in which the slope of the objective function Fxi (x) = @F@x(x) i is maximum, or the direction in which the largest step can be taken. To evaluate the choice

44

Hill climbing Strategies

of direction, Synge (1944) uses the ratio Fxi =Fxixi of rst to second partial derivatives at the point x(k). Whether or not the additional e ort for this scheme is worthwhile depends on the particular topology of the contour surface. Adding directions other than parallel to the axes is also often found to accelerate the convergence (Pinsker and Tseitlin, 1962 Elkin, 1968). Its great simplicity has always made the coordinate strategy attractive, despite its sometimes slow convergence. Rules for handling constraints{not counting here penalty function methods{have been devised, for example, by Singer (1962), Murata (1963), and Mugele (1961, 1962, 1966). Singer's maze method departs from the coordinate directions as soon as a constraint is violated and progresses into the feasible region or along the boundary. For this, however, the gradient of the active constraints must be known. Mugele's poor man's optimizer, a discrete coordinate strategy without line searches, not only handles active constraints, but can also cope with narrow valleys that do not run parallel to the coordinate axes. In this case diagonal steps are permitted. Similar to this strategy is the direct search method of Hooke and Jeeves, which because it has become very widely used will be treated in detail in the following chapter.

3.2.1.2 Strategy of Hooke and Jeeves: Pattern Search

The direct pattern search of Hooke and Jeeves (1961) was originally devised as an automatic experimental strategy (see Hooke, 1957 Hooke and VanNice, 1959). It is nowadays much more widely used as a numerical parameter optimization procedure. The method by which the direct pattern search works is characterized by two types of move. At each iteration there is an exploratory move, which represents a simpli ed Gauss-Seidel variation with one discrete step per coordinate direction. No line searches are made. On the assumption that the line joining the rst and last points of the exploratory move represents an especially favorable direction, an extrapolation is made along it (pattern move) before the variables are varied again individually. The extrapolations do not necessarily lead to an improvement in the objective function value. The success of the iteration is only checked after the following exploratory move. The length of the pattern step is thereby increased each time, while the optimal search direction only changes gradually. This pays o to most advantage where there are narrow valleys. An ALGOL implementation of the strategy is due to Kaupe (1963). It was improved by Bell and Pike (1966), as well as by Smith (1969) (see also DeVogelaere, 1968 Tomlin and Smith, 1969). In the rst case, the sequence of plus and minus exploratory steps in the coordinate directions is modi ed to suit the conditions at any instant. The second improvement aims at permitting a retrospective scaling of the variables as the step lengths can be chosen individually to be di erent from each other. The algorithm runs as follows: Step 0: (Initialization) Choose a starting point x(00) = x(;1n), an accuracy bound " > 0, and initial (0) step lengths si 6= 0 for all i = 1(1)n (e.g., s(0) = 1 if no more plausible 1 values are at hand). Set k = 0 and i = 1.

Multidimensional Strategies

45

Step 1: (Exploratory move) Construct x0 = x(ki;1) + s(ik) ei (discrete step in positive direction). If F (x0) < F (x(ki;1)), go to step 2 (successful rst trial) otherwise replace x0 x0 ; 2 s(ik) ei (discrete step in negative direction). If F (x0) < F (x(ki;1)), go to step 2 (success) otherwise replace x0 x0 + s(ik) ei (back to original situation). Step 2: (Retention and switch to next coordinate) Set x(ki) = x0. If i < n, increase i i + 1 and go to step 1. Step 3: (Test for total failure in all directions) If F (x(kn))  F (x(k0)), set x(k+10) = x(k0) and go to step 9. Step 4: (Pattern move) Set x(k+10) = 2 x(kn) ; x(k;1n) (extrapolation), and s(ik+1) = s(ik) sign(x(ikn) ; x(ik;1n)) for all i = 1(1)n. (This may change the sequence of positive and negative directions in the next exploratory move.) Increase k k + 1 and set i = 1. (Observe: There is no success control of the pattern move so far.) Step 5: (Exploration after extrapolation) Construct x0 = x(ki;1) + s(ik) ei. If F (x0) < F (x(ki;1)), go to step 6 otherwise replace x0 x0 ; 2 s(ik) ei. If F (x0) < F (x(ki;1)), go to step 6 otherwise replace x0 x0 + s(ik) ei. Step 6: (Inner loop over coordinates) Set x(ki) = x0. If i < n, increase i i + 1 and go to step 5. Step 7: (Test for failure of pattern move) If F (x(kn))  F (x(k;1n)) (back to position before pattern move), set x(k+10) = x(k;1n) s(ik+1) = s(ik) for all i = 1(1)n, and go to step 10. Step 8: (After successful pattern move, retention and rst termination test) If jx(ikn) ; x(ik;1n)j  12 js(ik)j for all i = 1(1)n, set x(k+10) = x(k;1n) and go to step 9 otherwise go to step 4 (for another pattern move). Step 9: (Step size reduction and termination test) If js(ik)j  " for all i = 1(1)n, end the search with result x(k0) otherwise set s(ik+1) = 12 s(ik) for all i = 1(1)n.

46

Hill climbing Strategies

Step 10: (Iteration loop) Increase k k + 1, set i = 1, and go to step 1. Figure 3.6, together with the following table, presents a possible sequence of iteration points. From the starting point (0), a successful step (1) and (3) is taken in each coordinate direction. Since the end point of this exploratory move is better than the starting point, it serves as a basis for the rst extrapolation. This leads to (4). It is not checked here whether or not any improvement over (3) has occurred. At the next exploratory move, from (4) to (5), the objective function value can only be improved in one coordinate direction. It is now checked whether the condition (5) is better than that of point (3). This is the case. The next extrapolation step, to (8), has a changed direction because of the partial failure of the exploration, but maintains its increased length. Now it will be assumed that, starting from (8) with the hitherto constant exploratory step length, no success will be scored in any coordinate direction compared to (8). The comparison with (5) shows that a reduction in the value of the objective function has nevertheless occurred. Thus the next extrapolation to (13) remains the same as the previous one with respect to direction and step length. The next exploratory move leads to a point (15), which although better than (13) is worse than (8). Now there is a return to (8). Only after the exploration again has no success here, are the step lengths halved in order to make further progress possible. The fact that at some points in this case the objective function was tested several times is not typical for n > 2. Starting point

(2)

Success Failure Extrapolation

(1) (0)

Final point (7) (3) (21) (12)

(5) (4)

(18) (17) (22) (9) (6)

(10) (19)

(23) (8) (24)

(25)

(11) (20)

(15)

(13)

(16)

Figure 3.6: Strategy of Hooke and Jeeves

(14)

Multidimensional Strategies

47

Numbering Iteration Direction Variable Comparison Step index index values point lengths k i x1 x2 s1 s2 (0) (1) (2) (3) (4) (5) (6) (7) (8)

0 0 0 0 1 1 1 1 2

0 1 2 2 0 1 2 2 0

0 9 2 9 2 11 2 7 4 5 6 5 6 3 6 7 10 3

(0) (1) (1) (4),(3) (5) (5) -,(5)

(9) (10) (11) (12) (13) (14) (15) (16) (17) (8) (18) (19) (20) (21) (8)

2 2 2 2 3 3 3 3 3 4 4 4 4 4 5

1 1 2 2 0 1 1 2 2 0 1 1 2 2 0

12 3 8 3 10 1 10 5 14 1 16 1 12 1 12 ;1 12 3 10 3 12 3 8 3 10 1 10 5 10 3

(8) (8) (8) (8) (13) (13),(8) (15) (15) (8) (8) (8) (8) -

(22) (23) (24) (25)

5 5 5 6

1 1 2 0

11 9 9 8

(8) (8) (23),(8) -

3 3 2 1

2

2

2

;2

2

;2

2

;2

2

;2

1

;1

Remarks starting point success failure success extrapolation success, success failure failure extrapolation, success failure failure failure failure extrapolation failure success, failure failure failure return failure failure failure failure step lengths halved failure success success, success extrapolation

;1 ;1 A proof of convergence of the direct search of Hooke and Jeeves has been derived by C%ea (1971) it is valid under the condition that the objective function F (x) is strictly convex and continuously di erentiable. The computational operations are very simple and even in unforeseen circumstances cannot lead to invalid arithmetical manipulations such as, for example, division by zero. A further advantage of the strategy is its small storage requirement. It is of order O(n). The selected pattern accelerates the search in valleys, provided they are not sharply bent. The extrapolation steps follow, in an approximate way, the gradient trajectory. However, the limitation of the trial steps to coordinate directions can also lead to a premature termination of the search here, as in the coordinate strategy. Further variations on the method, which have not achieved such popularity, are due to, among others, Wood (1960, 1962, 1965 see also Weisman and Wood, 1966 Weisman,

48

Hill climbing Strategies

Wood, and Rivlin, 1965), Emery and O'Hagan (1966 spider method), Fend and Chandler (1961 moment rosetta search), Bandler and MacDonald (1969 razor search see also Bandler, 1969a,b), Pierre (1969 bunny-hop search), Erlicki and Appelbaum (1970), and Houston and Hu man (1971). A more detailed enumeration of older methods can be found in Lavi and Vogl (1966). Some of these modi cations allow constraints in the form of inequalities to be taken into account directly. Similar to them is a program designed by M. Schneider (see Drenick, 1967). Aside from the fact that in order to use it one must specify which of the variables enter the individual constraints, it does not appear to work very e ectively. Excessively long computation times and inaccurate results, especially with many variables, made it seem reasonable to omit M. Schneider's procedure from the strategy comparison (see Chap. 6). The problem of how to take into account constraints in a direct search has also been investigated by Klingman and Himmelblau (1964) and Glass and Cooper (1965). The resulting methods, to a greater or lesser extent, transform the original problem. They have nowadays been superseded by the general penalty function methods. Automatic \optimizators" for on-line optimization of chemical processes, which once were well known under the names Opcon ( Bernard and Sonderquist, 1959) and Optimat (Weiss, Archer, and Burt, 1961), also apply modi ed versions of the direct search method. Another application is described by Sawaragi at al. (1971).

3.2.1.3 Strategy of Rosenbrock: Rotating Coordinates

Rosenbrock's idea (1960) was to remove the limitation on the number of search directions in the coordinate strategy so that the search steps can move parallel to the axes of a coordinate system that can rotate in the space IRn . One of the axes is set to point in the direction that appears most favorable. For this purpose the experience of successes and failures gathered in the course of the iterations is used in the manner of Hooke and Jeeves' direct search. The remaining directions are xed normal to the rst and mutually orthogonal. To start with, the search directions comprise the unit vectors vi(0) = ei  for all i = 1(1)n Starting from (0the point x(00), a trial is made in each direction with the discrete initial 0) step lengths si for all i = 1(1)n. When a success is scored (including equality of the objective function values), the changed variable vector is retained and the step length is multiplied by a positive factor > 1 for a failure, the vector of variables is left unchanged and the step length is multiplied by a negative factor ;1 <  < 0. Rosenbrock proposes the choice = 3 and  = ;0:5. This process is repeated until at least one success followed (not necessarily immediately) by a failure is registered in each direction. As a rule several cycles must be run through, because if there is a failure in any particular direction the opposite direction is not tested in the same cycle. Following this rst part of the search, the coordinate axes are rotated. Rosenbrock uses for this the orthogonalization procedure of Gram and Schmidt (see, for example, Birkho and MacLane, 1953 Rutishauser, 1966 Nake, 1966). The recursion formulae are as follows: vi(k+1) = kwwik  for all i = 1(1)n i

Multidimensional Strategies where and

49

8 > < ai  i;1 wi = > ai ; P (aT v(k+1)) v(k+1)  : j j j =1 i ai =

n X j =i

for i = 1 for i = 2(1)n

(3.21)

d(jk) vj(k)  for all i = 1(1)n

A scalar d(ik) represents the distance covered in direction vi(k) in the kth iteration. Thus v1(k+1) points in the overall successful direction of the step k. It is expected that a particularly large search step can be taken in this direction at the next iteration. The requirement of waiting for (at least one success in each direction has the e ect that no k) direction is lost, and the vi always span the full n-dimensional Euclidean space. The termination rule or convergence criterion is determined by the lengths of the vectors a(1k) and a(2k). Before each orthonormalization there is a test whether ka(1k)k < " and ka(2k) k > 0:3 ka(1k) k. When this condition is satis ed in six consecutive iterations, the search is ended. The second condition is designed to ensure that a premature termination of the search does not occur just because the distances covered have become small. More signi cantly, the requirement is also that the main success direction changes suciently rapidly something that Rosenbrock regards as a sure sign of the proximity of a minimum. As the strategy comparison will show (see Chap. 6), this requirement is often too strong. It even hinders the ending of the procedure in many cases. In his original publication Rosenbrock has already given detailed rules as to how inequality constraints can be treated. His procedure for doing this can be viewed as a partial penalty function method, since the objective function is only altered in the neighborhood of the boundaries. Immediately after each variation of the variables, the objective function value is tested. If the comparison is unfavorable, a failure is registered as in the unconstrained case. For equality or an improvement, however, if the iteration point lies near a boundary of the region, the success criterion changes. For example, for constraints of the form Gj (x)  0 for all j = 1(1)m, the extended objective function F~ (x) takes the form (this is one of several suggestions of Rosenbrock): m X ~ F (x) = F (x) + 'j (x) (fj ; F (x)) j =1

in which and

8 > < 0 'j (x) = > 3  ; 4 2 + 2 3  : 1

if Gj (x)  if 0 < Gj (x) < if Gj (x)  0

(3.22)

 = 1 ; 1 Gj (x) The auxiliary item fj is the value of the objective function belonging to the last success of the search that did not fall in the region of the j th boundary. As a reasonable value for the boundary zone one can take = 10;4 . (Rosenbrock sets = 10;4 "bj (x(k)) ; aj (x(k))]

50

Hill climbing Strategies

for constraints of the form aj (x)  Gj (x)  bj (x) this kind of double sided bounding is not always given however). The basis of the procedure is fully described in Rosenbrock and Storey (1966). Using the notations xi object variables si step sizes vi direction components di distances travelled i success/failure indications the extended algorithm of the strategy runs as follows: Step 0: (Initialization) Choose a starting point x(00) such that Gj (x(00)) Choose an accuracy parameter " > 0 (Rosenbrock takes " = 10;4  = 10;4 ). Set vi(0) = ei for all i = 1(1)n. Set k = 0 (outer loop counter),  = 0 (inner loop counter). If there are constraints (m > 0), set fj = F (x(00)) for all j = 1(1)m.



> 0 for all j = 1(1)m.

Step 1: (Initialization of step sizes, distances travelled, and indicators) Set s(ik0) = 0:1, d(ik) = 0, and i(k) = ;1 for all i = 1(1)n. Set ` = 0 and i = 1. Step 2: (Trial step) Construct x0 = x(kn` + i ; 1) + s(ik`) vi(k). If F (x0) > F (x(kn` + i ; 1)), go to step 6 otherwise ( 0 go to step 5 if m = 6= 0 set F~ = F (x0) and j = 1: Step 3: (Test of feasibility) 8 >  0 > <  If Gj (x0) >  otherwise, > : Step 4: (Constraints loop) If j < m increase j

go to step 7 set fj = F (x0) and go to step 4 replace F~ F~ + 'j (x0) (fj ; F~ ) as Equation (3.22). If F~ > F (x(kn` + i ; 1)) go to step 6.

j + 1 and go to step 3.

Multidimensional Strategies

51

Step 5: (Store the success and update the internal memory) Set x(kn` + i) = x0 s(ik` + 1) = 3 s(ik`) , and replace d(ik) d(ik) + s(ik`). If i(k) = ; 1 set i(k) = 0. Go to step 7. Step 6: (Internal memory update in case of failure) kn` + i) = x(kn` + i ; 1), Set x((k` si + 1) = ; 21 s(ik`). If i(k) = 0 set i(k) = 1. Step 7: (Main loop) If j(k) = 1 for all j = 1(1)n, go to step 8 otherwise ( n increase i i + 1 if i < = n increase ` ` + 1 and set i = 1: Go to step 2. Step 8: (Preparation for the orthogonalization and check for termination) n Set x(k + 10) = x(kn` + i) = x(k0) + P d(jk) vj(k). j=1 n (k) (k) P (k ) Construct the vectors a = d v for all i = 1(1)n i

j=i

j

j

(a1 is the total progress during the loop just nished). If ka(1k)k < " and (if n > 1)ka(2k)k > 0:3 ka(1k)k, increase   + 1 otherwise set  = 0. If  = 6, end the search. Step 9: (Orthogonalization) If n > 1, construct new direction vectors vi(k + 1) for i = 1(1)n according to the recursion formula (Equation (3.21)) of the Gram-Schmidt orthogonalization. Increase k k + 1 and go to step 1.

52

Hill climbing Strategies

(2) (0)

v2

(0)

v

1

(3)

(1)

(0) (4)

v

(5)

(1)

Starting point

1

Success

(1)

v2

(7)

Failure Overall success (8) (6)

(9) (2)

v2

(2)

v

1

(10)

Figure 3.7: Strategy of Rosenbrock

(11)

Multidimensional Strategies

53

Numbering Iteration Test index/ index iteration k nl + i (0) (1) (2) (3) (4) (5) (6) (4)

0 0 0 0 0 0 0 1

0 1 2 3 4 5 6 0

Variable values x1 x2 0 2 2 8 2 ;1 2 2

9 9 11 9 8 8 5 8

Step lengths s1 s2 2 2 6 ;3 2

2 2 ;1 ;3 2

Remarks

starting point success failure failure success failure failure transformation and orthogonalization (7) 1 1 3.8 7.1 2 success (8) 1 2 2.9 5.3 2 success (9) 1 3 8.3 2.6 6 success (10) 1 4 5.6 ;2.7 6 failure (11) 1 5 24.4 ;5.4 18 failure (9) 2 0 8.3 2.6 2 2 transformation and orthogonalization In Figure 3.7, including the following table, a few iterations of the Rosenbrock strategy for n = 2 are represented geometrically. At the starting point x(00) the search directions are the same as the unit vectors. After three runs through (6 trials), the trial steps in each direction have led to a success followed by a (1) failure. At the best condition thus attained, (4) at x(04) = x(10), new direction vectors v1 and v2(1) are generated. Five further trials lead to the best point, (9) at x(13) = x(20), of the second iteration, at which a new choice of directions is again made. The complete sequence of steps can be followed, if desired, with the help of the accompanying table. Numerical experiments show that within a few iterations the rotating coordinates become oriented such that one of the axes points along the gradient direction. The strategy thus allows sharp valleys in the topology of the objective function to be followed. Like the method of Hooke and Jeeves, Rosenbrock's procedure needs no information about partial derivatives and uses no line search method for exact location of relative minima. This makes it very robust. It has, however, one disadvantage compared to the direct pattern search: The orthogonalization procedure of Gram and Schmidt is very costly. It requires storage space of order O(n2) for the matrices A = faij g and V = fvij g, and the number of computational operations even increases with O(n3 ). At least in cases where the objective function call costs relatively little, the computation time for the orthogonalization with many variables becomes highly signi cant. Besides this, the number of parameters is in any case limited by the high storage space requirement. If there are constraints, care must be taken to ensure that the starting point is inside the allowed region and suciently far from the boundaries. Examples of the application of

54

Hill climbing Strategies

Rosenbrock's strategy can be found in Storey (1962), and Storey and Rosenbrock (1964). Among them is also a discretized functional optimization problem. For unconstrained problems there exists the code of Machura and Mulawa (1973). The Gram-Schmidt orthogonalization has been programmed, for example, by Clayton (1971). Lange-Nielsen and Lance (1972) have proposed, on the basis of numerical experiments, two improvements in the Rosenbrock strategy. The rst involves not setting constant step lengths at the beginning of a cycle or after each orthogonalization, but rather modifying them and simultaneously scaling them according to the successes and failures during the preceding cycle. The second improvement concerns the termination criterion. Rosenbrock's original version is replaced by the simpler condition that, according to the achievable computational accuracy, several consecutive trials yield the same value of the objective function.

3.2.1.4 Strategy of Davies, Swann, and Campey (DSC)

A combination of the Rosenbrock idea of rotating coordinates with one dimensional search methods is due to Swann (1964). It has become known under the name Davies-SwannCampey (abbreviated DSC) strategy. The description of the procedure given by Box, Davies, and Swann (1969) di ers somewhat from that in Swann, and so several versions of the strategy have arisen in the subsequent literature. Preference is given here to the original concept of Swann, which exhibits some features in common with the method of conjugate directions of Smith (1962) (see also Sect. 3.2.2). Starting from x(00), a line search is made in each of the unit directions vi(0) = ei for all i = 1(1)n. This process is followed by a one dimensional minimization in the direction of the overall success so far achieved (0n) x(00) vn(0)+1 = kxx(0n) ; ; x(00)k

with the result x(0n + 1). The orthogonalization follows this, e.g., by the Gram-Schmidt method. If one of the line searches was unsuccessful the new set of directions would no longer span the complete parameter space. Therefore only those old direction vectors along which a prescribed minimum distance has been moved are included in the orthogonalization process. The other directions remain unchanged. The DSC method, however, places a further hurdle before the coordinate rotation. If the distance covered in one iteration is smaller than the step length used in the line search, the latter is reduced by a factor 10, and the next iteration is carried out with the old set of directions. After an orthogonalization, one of the new directions (the rst) coincides with that of the (n +1)-th line search of the previous step. This can therefore also be interpreted as the

rst minimization in the new coordinate system. Only n more one dimensional searches need be made to nish the iteration. As a termination criterion the DSC strategy uses the length of the total vector between the starting point and end point of an iteration. The search is ended when it is less than a prescribed accuracy bound.

Multidimensional Strategies The algorithm runs as follows: Step 0: (Initialization) Specify a starting point x(00) and an initial step length s(0) (the same for all directions). De ne an accuracy requirement " >(0)0. Choose as a rst set of directions vi = ei for all i = 1(1)n. Set k = 0 and i = 1. Step 1: (Line search) (ki) Starting from x(ki(k;1) , seek the relative minimum x in the direction vi ) such that (k ) F (x(ki)) = F (x(ki;1) + d(ik) vi(k)) = min fF (x(ki;1) + d vi )g: d Step 2: (Main 8 loop) > < < n If i > = n : = n + 1 Step 3:

Step 4: Step 5:

Step 6:

Step 7:

increase i i + 1, and go to step 1 go to step 3 go to step 4. (Eventually one more line search) Construct z = x((kkn) ) ; x(k0). If kzk > 0, set vn+1 = z = kzk  i = n + 1 , and go to step 1 otherwise set x(kn+1) = x(kn) d(nk+1) = 0 , and go to step 5. (Check appropriateness of step length) If kx(kn+1) ; x(k0)k  s(k), go to step 6. (Termination criterion) Set s(k+1) = 0:1 s(k) . If s(k+1)  " end the search otherwise set x(k+10) = x(kn+1), increase k k + 1, set i = 1, and go to step 1. (Check appropriateness of orthogonalization) (k) (k ) Reorder ( the directions vi and associated distances di such that > " for all i = 1(1)p (k) jdi j  " for all i = p + 1 (1)n: If p < 2, thus always for n = 1, go to step 5. (Orthogonalization) Construct new direction vectors vi(k+1) for i = 1(1)p by means of the orthogonalization process of Gram and Schmidt (Equation (3.21)). Set s(k+1) = s(k) d(1k+1) = d(nk+1) , and x(k+10) = x(kn) x(k+11) = x(kn+1). Increase k k + 1 set i = 2, and go to step 1.

55

56

Hill climbing Strategies

No geometric representation has been attempted here, since the ne deviations from the Rosenbrock method would hardly be apparent on a simple diagram. The line search procedure of the DSC method has been described in detail by Box, Davies, and Swann (1969). It boxes in the minimum in the chosen direction using three equidistant points and then applies a single Lagrangian quadratic interpolation. The authors state that, in their experience, this is more economical with regard to the number of objective function calls than an exact line search with a sequence of interpolations. The algorithm of the line search is: Step 0: (Initialization) Specify a starting point x0, a step length s, and a direction v (all given from the main program). Step 1: (Step forward) Construct x = x0 + s v. If F (x)  F (x0), go to step 3. Step 2: (Step backward) Replace x x ; 2 s v and s ;s. If F (x)  F (x0), go to step 3 otherwise (both rst trials without success) go to step 5. Step 3: (Further steps) Replace s 2 s and set x0 = x. Construct x = x0 + s v. If F (x)  F (x0), repeat step 3. Step 4: (Prepare interpolation) Replace s 0:5 s. Construct x = x0 + s v. Of the four points just generated, x0 ; s x0 x0 + s, and x0 + 2 s, reject the one which is furthest from the point that has the smallest value of the objective function. Step 5: (Interpolation) De ne the three available equidistant points x1 < x2 < x3 and the associated function values F1 F2, and F3 (this can be done in the course of the trial steps to box in the minimum). Fit a trial parabola through the three points and solve the necessary condition for its minimum. Because the argument values are equidistant, Equation (3.16) for the Lagrangian quadratic interpolation simpli es to x = x2 + 2 (Fs (;F12;F F+3) F ) 1

2

3

If the denominator vanishes or if it turns out that F (x) > F2, then the line search ends with the result x00 = x2 F~0 = F2 otherwise with the result x00 = x and F~0 = F (x).

Multidimensional Strategies

57

A numerical strategy comparison by M. J. Box (1966) shows the method to be a very e ective optimization procedure, in general superior both to the Hooke and Jeeves and the Rosenbrock methods. However, the tests only refer to smooth objective functions with few variables. If the number of parameters is large, the costly orthogonalization process makes its inconvenient presence felt also in the DSC strategy. Several suggestions have been made to date as to how to simplify the Gram-Schmidt procedure and to reduce its susceptibility to numerical rounding error (Rice, 1966 Powell, 1968a Palmer, 1969 Golub and Saunders, 1970, Householder method). Palmer replaces the conditions of Equation (3.21) by: 8 (k) Pn d2 = 0 otherwise > v  if > i > j =1 j > > > < Pn (k) s Pn 2 (k+1) for i = 1 vi = > j=1 dj vj = j=1 dj  > > > ! sn > n n 2 P Pn P > (k) (k) P > : di;1 j=i dj vj ; vi;1 j=i dj = j=i d2j j=i;1 d2j  for i = 2(1)n He shows that even if no success was obtained in one of the directions vi(k), that is di = 0, the new vectors vi(k+1) for all i = 1(1)n still span the complete parameter space, because k+1) vi(+1 is set equal to ;vi(k). Thus the algorithm does not need to be restricted to directions for which di > ", as happens in the algorithm with Gram-Schmidt orthogonalization. The signi cant advantage of the revised procedure lies in the fact that the number of computational operations remains only of the order O(n2). The storage requirement is also somewhat less since one n  n matrix as an intermediate storage area is omitted. For problems with linear constraints (equalities and inequalities) Box, Davies, and Swann (1969) recommend a modi cation of the orthogonalization procedure that works in a similar way to the method of projected gradients of Rosen (1960, 1961) (see also Davies, 1968). Non-linear constraints (inequalities) can be handled with the created response surface technique devised by Carroll (1961), which is one of the penalty function methods. Further publications on the DSC strategy, also with comparison tests, are those of Swann (1969), Davies and Swann (1969), Davies (1970), and Swann (1972). Hoshino (1971) observes that in a narrow valley the search causes zigzag movements. His remedy for this is to add a further search, again in direction v1(k), after each set of n line searches. With the help of two examples, for n = 2 and n = 3, he shows the accelerating e ect of this measure.

3.2.1.5 Simplex Strategy of Nelder and Mead

There are a group of methods called simplex strategies that work quite di erently to the direct search methods described so far. In spite of their common name they have nothing to do with the simplex method of linear programming of Dantzig (1966). The idea (Spendley, Hext, and Himsworth, 1962) originates in an attempt to reduce, as much as possible, the number of simultaneous trials in the experimental identi cation procedure

58

Hill climbing Strategies

of factorial design (see for example Davies, 1954). The minimum number according to Brooks and Mickey (1961) is n + 1. Thus instead of a single starting point, n + 1 vertices are used. They are arranged so as to be equidistant from each other: for n = 2 in an equilateral triangle for n = 3 a tetrahedron and in general a polyhedron, also referred to as a simplex. The objective function is evaluated at all the vertices. The iteration rule is: Replace the vertex with the largest objective function value by a new one situated at its re ection in the midpoint of the other n vertices. This rule aims to locate the new point at an especially promising place. If one lands near a minimum, the newest vertex can also be the worst. In this case the second worst vertex should be reected. If the edge length of the polyhedron is not changed, the search eventually stagnates. The polyhedra rotate about the vertex with the best objective function value. A closer approximation to the optimum can only be achieved by halving the edge lengths of the simplex. Spendley, Hext, and Himsworth suggest doing this whenever a vertex is common to more than 1:65 n + 0:05 n2 consecutive polyhedra. Himsworth (1962) holds that this strategy is especially advantageous when the number of variables is large and the determination of the objective function prone to error. To this basic procedure, various modi cations have been proposed by, among others, Nelder and Mead (1965), Box (1965), Ward, Nag, and Dixon (1969), and Dambrauskas (1970, 1972). Richardson and Kuester (1973) have provided a complete program. The most common version is that of Nelder and Mead, in which the main di erence from the basic procedure is that the size and shape of the simplex is modi ed during the run to suit the conditions at each stage. The algorithm, with an extension by O'Neill (1971), runs as follows: Step 0: (Initialization) Choose a starting point x(00), initial step lengths s(0) i for all i = 1(1)n (0) (if no better scaling is known, si = 1), and an accuracy parameter " > 0 (e.g., " = 10;8 ). Set c = 1 and k = 0. Step 1: (Establish the initial simplex) x(k) = x(k0) + c s(0)  e for all  = 1(1)n. Step 2: (Determine worst and best points for the normal reection) Determine the indices w (worst point) and b (best point) such that (k ) F (x(kw)) = max  fF (x )   = 0(1)ng (k ) F (x(kb)) = min  fF (x )   = 0(1)ng n Construct x( = n1 P x(k) and x0 = 2 x( ; x(kw) (normal reection).  =0 6=w F (x(kb)), go to

If F (x0) < step 4. Step 3: (Compare trial with other vertices) Determine the number  for which F (x0)  F (x(k)) holds for all  = 0(1)n. 8 > < > 1 set x(k+1w) = x0 and go to step 8 If  > = 1 go to step 5 : = 0 go to step 6:

Multidimensional Strategies

59

Step 4: (Expansion) Construct x00 = 2 x0 ; x(. If F (x00) < F (x(kb)), set x(k+1w) = x00 otherwise set x(k+1w) = x0. Go to step 8. Step 5: (Partial outside contraction) Construct x00 = 0:5 ((x + x0). If F (x00)  F (x0), set x(k+1w) = x00 and go to step 8 otherwise go to step 7. Step 6: (Partial inside contraction) Construct x00 = 0:5 ((x + x(kw)). If F (x00)  F (x(kw)), set x(k+1w) = x00 and go to step 8. Step 7: (Total contraction) Construct x(k+1) = 0:5 (x(kb) + x(k)) for all  = 0(1)n. Go to step 9. Step 8: (Normal iteration loop) Assign x(k+1) = x(k) for all  = 0(1)n except  = w. Step 9: (Termination criterion) Increase n k k + 1.  Pn 2 ! P 1 F 2(x(k)) ; n+1 F (x(k)) < "2, go to step 10 If n1  =0

 =0

otherwise go to step 2. Step 10: (Restart test note that index w points to the new vertex) Test whether any vector (kw)). x = x(kw)  0:001 s(0) i ei for all i = 1(1)n exists, such that F (x) < F (x (k0) If so, set x = x c = 0:001, and go to step 1 (restart) otherwise end the search with result x(kw). The criterion for ending the minimum search is based on testing whether the variance of the objective function values at the vertices of the simplex is less than a prescribed limit. A few hypothetical iterations of the procedure for two variables are shown in Figure 3.8 including the following table. The sequence of reections, expansions, and contractions is taken from the accompanying table, in which the simplex vertices are numbered sequentially and sorted at each stage in order of their objective function values.

60

Hill climbing Strategies

(1)

(2)

Starting point Vertex point First and last simplex

(3)

(4) (5)

(9) (6) (7) (15) (17) (14) (12)

(11)

(16) (10) (8)

(13)

Figure 3.8: Simplex strategy of Nelder and Mead

Multidimensional Strategies Iteration index

Simplex vertices worst best

0

1

1

2

2

3

3

6

4

5

5 6 7

61

9 14 15

11 17

2 2 2 3 3 6 6 6 5 5 9 10 9 9 11 11 11 7 7 7 17 16

6 8 9 11 13 16

3 3 3 5 5 5 5 5 7 7 7 7 7 7 7 7 12 12 12 12 12

Remarks 4 5

start simplex reection expansion (successful) reection

7

reection expansion (unsuccessful) reection reection partial outside contraction

12

reection expansion (unsuccessful) reection partial inside contraction total contraction

The main di erence between this program and the original strategy of Nelder and Mead is that after a normal ending of the minimization there is an attempt to construct a new starting simplex. To this end, small trial steps are taken in each coordinate direction. If just one of these tests is successful, the search is started again but with a simplex of considerably reduced edge lengths. This restart procedure recommends itself because, especially for a large number of variables, the simplex tends to no longer span the complete parameter space, i.e., to collapse, without reaching the minimum. For few variables the simplex method is known to be robust and reliable, but also to be relatively costly. There are n + 1 parameter vectors to be stored and the reection requires a number of computational operations of order O(n2). According to Nelder and Mead, the number of function calls increases approximately as O(n2:11) however, this empirical value is based only on test results with up to 10 variables. Parkinson and Hutchinson (1972a,b) describe a variant of the strategy in which the real storage requirement can be reduced by about half (see also Spendley, 1969). Masters and Drucker (1971) recommend altering the expansion or contraction factor after consecutive successes or failures respectively.

3.2.1.6 Complex Strategy of Box

M. J. Box (1965) calls his modi cation of the polyhedron strategy the complex method, an abbreviation for constrained simplex, since he conceived it also for problems with

62

Hill climbing Strategies

inequality constraints. The starting point of the search does not need to lie in the feasible region. For this case Box suggests locating an allowed point by minimizing the function m X F~ (x) = ; Gj (x) j (x) j =1

with until

(

j (x) = 01 

if Gj (x)  0 otherwise

(3.23)

F~ (x) = 0 The two most important di erences from the Nelder-Mead strategy are the use of more vertices and the expansion of the polyhedron at each normal reection. Both measures are intended to prevent the complex from eventually spanning only a subspace of reduced dimensionality, especially at active constraints. If an allowed starting point is given or has been found, it de nes one of the n + 1  N  2 n vertices of the polyhedron. The remaining vertex points are xed by a random process in which each vector inside the closed region de ned by the explicit constraints has an equal probability of selection. If an implicit constraint is violated, the new point is displaced stepwise towards the midpoint of the allowed vertices that have already been de ned until it satis es all the constraints. Implicit constraints Gj (x)  0 are dealt with similarly during the course of the minimum search. If an explicit boundary is crossed, xi  ai, the o ending variable is simply set back in the allowed region to a value near the boundary. The details of the algorithm are as follows: Step 0: (Initialization) Choose a starting point x(0) and a number of vertices N  n + 1 (e.g., N = 2 n). Number the constraints such that the rst j  m1 each depend only on one variable, x`j (Gj (x`j ), explicit form). Test whether x(0) satis es all the constraints. If not, then construct a substitute objective function according to Equation (3.23). Set up the initial complex as follows: x(01) = x(0) n and x(0) = x(0) + P z e for  = 2(1)N , i=1

i i

where the zi are uniformly distributed random numbers from the range ( "a  b ], if constraints are given in the form a  x  b  i i i i i h (0) (0) otherwise xi ; 0:5 s xi + 0:5 s ] where, e.g., s = 1: If Gj (x(0)) < 0 for any j  m1  > 1 replace x(0`j ) 2 x(0`j 1) ; x(0`j ): If Gj (x(0)) < 0 for any j  m  > 1  ;1 1 P (0) replace x(0) 0:5 "x(0) + ;1 x ]. =1

Multidimensional Strategies

63

(If necessary repeat this process until Gj (x(0))  0 for all j = 1(1)m.) Set k = 0. Step 1: (Reection) Determine the index w (worst vertex) such that (k ) F (x(kw)) = max  fF (x )  = 1(1)N g: N Construct x( = 1 P x(k) N ;1

Step 2: Step 3:

Step 4:

Step 5:

Step 6: Step 7:

Step 8:


=1

 6=w ; x(kw))

and x0 = x( + ((x (over-reection factor = 1:3): (Check for constraints) If m = 0, go to step 7 otherwise set j = 1. If m1 = 0 , go to step 5. (Set vertex back into bounds for explicit constraints) Obtain g = Gj (x0) = Gj (x0`j ). If g  0, go to step 4 otherwise replace x0`j x0`j + g + " (backwards length " = 10;6 ). If Gj (x0) < 0, replace x0`j x0`j ; 2 (g + "). (Explicit constraints loop) Increase j j + 1. 8 > go to step 3 < j  m1 If > m1 < j  m go to step 5 : j > m go to step 7: (Check implicit constraints) If Gj (x0)  0, go to step 6 otherwise go to step 8, unless the same constraint caused a failure ve times in a row without its function value Gj (x0) being changed. In this case go to step 9. (Implicit constraints loop) If j < m, increase j j + 1 and go to step 5. (Check for improvement) If F (x0) < F (x((k)) for at least one  = 1(1)N except  = w, (k ) for all  = 1(1)N except  = w set x(k+1) = xx0  for  = w increase k k + 1 and go to step 1 otherwise go to step 8, unless a failure occurred ve times in a row with no change in the objective function value F (x0). In this case go to step 9. (Contraction) Replace x0 0:5 ((x + x0). Go to step 2.

64

Hill climbing Strategies

Step 9: (Termination) Determine the index b (best vertex) such that (k ) F (x(kb)) = min  fF (x )  = 1(1)N g: End the search with the result x(kb) and F (x(kb)): Box himself reports that in numerical tests his complex strategy gives similar results to the simplex method of Nelder and Mead, but both are inferior to the method of Rosenbrock with regard to the number of objective function calls. He actually uses his own modi cation of the Rosenbrock method. Investigation of the e ect of the number of vertices of the complex and the expansion factor (in this case 2 n and 1.3 respectively) lead him to the conclusion that neither value has a signi cant e ect on the eciency of the strategy. For n > 5 he considers that a number of vertices N = 2 n is unnecessarily high, especially when there are no constraints. The convergence criterion appears very reliable. While Nelder and Mead require that the standard deviation of all objective function values at the polyhedron vertices, referred to its midpoint, must be less than a prescribed size, the complex search is only ended when several consecutive values of the objective function are the same to computational accuracy. Because of the larger number of polyhedron vertices the complex method needs even more storage space than the simplex strategy. The order of magnitude, O(n2 ), remains the same. No investigations are known of the computational e ort in the case of many variables. Modi cations of the strategy are due to Guin (1968), Mitchell and Kaplan (1968), and Dambrauskas (1970, 1972). Guin de nes a contraction rule with which an allowed point can be generated even if the allowed region is not convex. This is not always the case in the original method because the midpoint to which the worst vertex is reected is not tested for feasibility. Mitchell nds that the initial con guration of the complex inuences the results obtained. It is therefore better to place the vertices in a deterministic way rather than to make a random choice. Dambrauskas combines the complex method with the step length rule of the stochastic approximation. He requires that the step lengths or edge lengths of the polyhedron go to zero in the limit of an in nite number of iterations, while their sum tends to in nity. This measure may well increase the reliability of convergence however, it also increases the cost. Beveridge and Schechter (1970) describe how the iteration rules must be changed if the variables can take only discrete values. A practical application, in which a process has to be optimized dynamically, is described by Tazaki, Shindo, and Umeda (1970) this is the original problem for which Spendley, Hext, and Himsworth (1962) conceived their simplex EVOP (evolutionary operation) procedure. Compared to other numerical optimization procedures the polyhedra strategies have the disadvantage that in the closing phase, near the optimum, they converge rather slowly and sometimes even stagnate. The direction of progress selected by the reection then no longer coincides at all with the gradient direction. To remove this diculty it has been suggested that information about the topology of the objective function, as given by function values at the vertices of the polyhedron, be exploited to carry out a quadratic interpolation. Such surface tting is familiar from the related methods of test planning and

Multidimensional Strategies

65

evaluation (lattice search, factorial design), in which the task is to set up mathematical models of physical or other processes. This territory is entered for example by G. E. P. Box (1957), Box and Wilson (1951), Box and Hunter (1957), Box and Behnken (1960), Box and Draper (1969, 1987), Box et al. (1973), and Beveridge and Schechter (1970). It will not be covered in any more detail here.

3.2.2 Gradient Strategies

The Gauss-Seidel strategy very straightforwardly uses only directions parallel to the coordinate axes to successively improve the objective function value. All other direct search methods strive to advance more rapidly by taking steps in other directions. To do so they exploit the knowledge about the topology of the objective function gleaned from the successes and failures of previous iterations. Directions are viewed as most promising in which the objective function decreases rapidly (for minimization) or increases rapidly (for maximization). Southwell (1946), for example, improves the relaxation by choosing the coordinate directions, not cyclically, but in order of the size of the local gradient in them. If the restriction of parallel axes is removed, the local best direction is given by the (negative) gradient vector rF (x) = (Fx1 (x) Fx2 (x) : : : Fxn (x))T with @F (x)  for all i = 1(1)n Fxi (x) = @x i

at the point x . All hill climbing procedures that orient their choice of search directions v(0) according to the rst partial derivatives of the objective function are called gradient strategies. They can be thought of as analogues of the total step procedure of Jacobi for solving systems of linear equations (see Schwarz, Rutishauser, and Stiefel, 1968). So great is the number of methods of this type which have been suggested or applied up to the present, that merely to list them all would be dicult. The reason lies in the fact that the gradient represents a local property of a function. To follow the path of the gradient exactly would mean determining in general a curved trajectory in the ndimensional space. This problem is only approximately soluble numerically and is more dicult than the original optimization problem. With the help of analogue computers continuous gradient methods have actually been implemented (Bekey and McGhee, 1964 Levine, 1964). They consider the trajectory x(t) as a function of time and obtain it as the solution of a system of rst order di erential equations. All the numerical variants of the gradient method di er in the lengths of the discrete steps and thereby also with regard to how exactly they follow the gradient trajectory. The iteration rule is generally rF (x(k)) x(k+1) = x(k) ; s(k) kr F (x(k))k It assumes that the partial derivatives everywhere exist and are unique. If F (x) is continuously di erentiable then the partial derivatives exist and F (x) is continuous. (0)

66

Hill climbing Strategies

A distinction is sometimes drawn between short step methods, which evaluate the gradients again after a small step in the direction rF (x(k)) (for maximization) or ;rF (x(k)) (for minimization), and their equivalent long step methods. Since for nite step lengths s(k) it is not certain whether the new variable vector is really better than the old, after the step the value of the objective function must be tested again. Working with small steps increases the number of objective function calls and gradient evaluations. Besides F (x) n partial derivatives must be evaluated. Even if the slopes can be obtained analytically and can be speci ed as functions, there is no reason to suppose that the number of computational operations per function call is much less than for the objective function itself. Except in special cases, the total cost increases roughly as the weighting factor (n + 1) and the number of objective function calls. This also holds if the partial derivatives are approximated by di erential quotients obtained by means of trial steps

Fxi (x) = @F@x(x) = F (x + e i) ; F (x) + O( 2)  i

for all i = 1(1)n

Additional diculties arise here since for values of that are too small the subtraction is subject to rounding error, while for trial steps that are too large the neglect of terms O( 2) leads to incorrect values. The choice of suitable deviations requires special care in all cases (Hildebrand, 1956 Curtis and Reid, 1974). Cauchy (1847), Kantorovich (1940, 1945), Levenberg (1944), and Curry (1944) are the originators of the gradient strategy, which started life as a method of solving equations and systems of equations. It is rst referred to as an aid to solving variational problems by Hadamard (1908) and Courant (1943). Whereas Cauchy works with xed step lengths s(k), Curry tries to determine the distance covered in the (not normalized) direction v(k) = ;rF (x(k)) so as to reach a relative minimum (see also Brown, 1959). In principle, any one of the one dimensional search methods of Section 3.1 can be called upon to nd the optimal value for s(k): (k) (k) F (x(k) + s(k) v(k)) = min s fF (x + s v )g

This variant of the basic strategy could thus be called a longest step procedure. It is better known however under the name optimum gradient method, or method of steepest descent (for maximization, ascent). Theoretical investigations of convergence and rate of convergence of the method can be found, e.g., in Akaike (1960), Goldstein (1962), Ostrowski (1967), Forsythe (1968), Elkin (1968), Zangwill (1969), and Wolfe (1969, 1970, 1971). Zangwill proves convergence based on the assumptions that the line searches are exact and the objective function is continuously twice di erentiable. Exactness of the one dimensional minimization is not, however, a necessary assumption (Wolfe, 1969). It is signi cant that one can only establish theoretically that a stationary point will be reached (rF (x) = 0) or approached (krF (x)k < "  " > 0). The stationary point is a minimum, only if F (x) is convex and three times di erentiable (Akaike, 1960). Zellnik, Sondak, and Davis (1962), however, show that saddle points are in practice an obstacle, only if the search is started at one, or on a straight gradient trajectory passing through one. In other cases numerical rounding errors ensure that the path to a saddle point is unstable.

Multidimensional Strategies

67

The gradient strategy, however, cannot distinguish global from local minima. The optimum at which it aims depends only on the choice of the starting point for the search. The only chance of nding absolute extrema is to start suciently often from various initial values of the variables and to iterate each time until the convergence criterion is satis ed (Jacoby, Kowalik, and Pizzo, 1972). The termination rules usually recommended for gradient methods are that the absolute value of the vector krF (x(k))k < "1 "1  0 or the di erence F (x(k;1)) ; F (x(k)) < "2 "2  0 vanishes or falls below a given limit. The rate of convergence of the strategy of steepest descent depends on the structure of the objective function, but is no better than rst order apart from exceptional cases like contours that are concentric, spherically symmetric hypersurfaces (Forsythe, 1968). In the general quadratic case (3.24) F (x) = 21 xT A x + xT b + c with elliptic contours, i.e., positive de nite matrix A, the convergence rate depends on the ratio of smallest to greatest eigenvalue of A, or geometrically expressed, on the oblateness of the ellipsoid. It can be extremely small (Curry, 1944 Akaike, 1960 Goldstein, 1962) and is in principle no better than the coordinate strategy with line searches (Elkin, 1968). In narrow valleys both procedures execute zigzag movements with very small changes in the variable values in relation to the distance from the objective. The individual steps can even become too small to e ect any change in the objective function value if this is de ned with a nite number of decimal places. Then the search ends before reaching the desired optimum. To obviate this diculty, Booth (1949, 1955) has suggested only going 90% of the way to the relative minimum in each line search (see also Stein, 1952 Kantorovich, 1952 Faddejew and Faddejewa, 1973). In fact, one often obtains much better results with this kind of underrelaxation. Even more advantageous is a modi cation due to Forsythe and Motzkin (1951). It is based on the observation that the search movements in the minimization of quadratic objective functions oscillate between two asymptotic directions (Stiefel, 1952 Forsythe, 1968). Forsythe and Motzkin therefore from time to time include a line search in the direction v(k) = x(k) ; x(k;2)  for k  2 in order to accelerate the convergence. For n = 2 the gradient method is thereby greatly improved with many variables the eciency advantage is lost again. Similar proposals for increasing the rate of convergence have been made by Baer (1962), Humphrey and Cottrell (1962, 1966), Witte and Holst (1964), Schinzinger (1966), and McCormick (1969). The Partan (acronym for parallel tangents) methods of Buehler, Shah, and Kempthorne (1961, 1964) have attracted particular attention. One of these, continued gradient Partan, alternates between gradient directions v(k) = ;rF (x(k))  for k = 0 as well as k  1 odd

68

Hill climbing Strategies

and those derived from previous iteration points

v(k) = x(k) ; x(k;3)  for k  2 even (with x(;1) = x(0)) For quadratic functions the minimum is reached after at most 2 n ; 1 line searches (Shah, Buehler, and Kempthorne, 1964). This desirable property of converging after a

nite number of iterations, that is also called quadratic convergence, is only shared by strategies that apply conjugate gradients, of which the Partan methods can be regarded as forerunners (Pierre, 1969 Sorenson, 1969). In the fties, simple gradient strategies were very popular, especially the method of steepest descent. Today they are usually only to be found as components of program packages together with other hill climbing methods, e.g., in GROPE of Flood and Leon (1966), in AID of Casey and Rustay (1966), in AESOP of Hague and Glatt (1968), and in GOSPEL of Huelsman (1968). McGhee (1967) presents a detailed ow diagram. Wasscher (1963a,b) has published two ALGOL codings (see also Haubrich, 1963 Wallack, 1964 Varah, 1965 Wasscher, 1965). The partial derivatives are obtained numerically. A comprehensive bibliography by Leon (1966b) names most of the older versions of strategies and gives many examples of their application. Numerical comparison tests have been carried out by Fletcher (1965), Box (1966), Leon (1966a), Colville (1968, 1970), and Kowalik and Osborne (1968). They show the superiority of rst (and second) order methods over direct search strategies for objective functions with smooth topology. Gradient methods for solving systems of di erential equations are described for example by Talkin (1964). For such problems, as well as for functional optimization problems, analogue and hybrid computers have often been applied (Rybashov, 1965a,b, 1969 Sydow, 1968 Fogarty and Howe, 1968, 1970). A literature survey on this subject has been compiled by Gilbert (1967). For the treatment of variational problems see Kelley (1962), Altman (1966), Miele (1969), Bryson and Ho (1969), C%ea (1971), Daniel (1971), and Tolle (1971). In the experimental eld, there are considerable diculties in determining the partial derivatives. Errors in the values of the objective function can cause the predicted direction of steepest descent to lie almost perpendicular to the true gradient vector (Kowalik and Osborne, 1968). Box and Wilson (1951) attempt to compensate for the perturbations by repeating the trial steps or increasing their number above the necessary minimum of (n + 1). With 2n trials, for example, a complete factorial design can be constructed (e.g., Davies, 1954). The slope in one direction is obtained by averaging the function value di erences over 2n;1 pairs of points (Lapidus et al., 1961). Another possibility is to determine the coecients of a linear polynomial such that the sum of the squares of the errors between measured and model function values at N  n + 1 points is a minimum. The linear function then represents the tangent plane of the objective function at the point under consideration. The cost of obtaining the gradients when there are many variables is too great for practical application, and only justi ed if the aim is rather to set up a mathematical model of the system than simply to perform the optimization. In the EVOP (acronym for evolutionary operation) scheme, G. E. P. Box (1957) has presented a practical simpli cation of this gradient method. It actually counts as a direct search strategy because it does not obtain the direction of the gradient but only one of a

nite number of especially good directions. Spendley, Hext, and Himsworth (1962) have

Multidimensional Strategies

69

devised a variant of the procedure (see also Sections 3.2.1.5 and 3.2.1.6). Lowe (1964) has gathered together the various schemes of trial steps for the EVOP strategy. The philosophy of the EVOP strategy is treated in detail by Box and Draper (1969). Some examples of applications are given by Kenworthy (1967). The eciency of methods of determining the gradient in the case of stochastic perturbations is dealt with by Mlynski (1964a,b, 1966a,b), Sergiyevskiy and Ter-Saakov (1970), and others.

3.2.2.1 Strategy of Powell: Conjugate Directions

The most important idea for overcoming the convergence diculties of the gradient strategy is due to Hestenes and Stiefel (1952), and again comes from the eld of linear algebra (see also Ginsburg, 1963 Beckman, 1967). It trades under the names conjugate directions or conjugate gradients. The directions fvi  i = 1(1)ng are said to be conjugate with respect to a positive de nite n  n matrix A if (Hestenes, 1956) viT A vj = 0  for all i j = 1(1)n  i 6= j A further property of conjugate directions is their linear independence, i.e., n X

i vi = 0 i=1

only holds if all the constants f i  i = 1(1)ng are zero. If A is replaced by the unit matrix, A = I , then the vi are mutually orthogonal. With A = r2F (x) (Hessian matrix) the minimum of a quadratic function is obtained exactly in n line searches in the directions vi. This is a factor two better than the gradient Partan method. For general non-linear problems the convergence rate cannot be speci ed. As it is frequently assumed, however, that many problems behave roughly quadratically near the optimum, it seems worthwhile to use conjugate directions. The quadratic convergence of the search with conjugate directions comes about because second order properties of the objective function are taken into account. In this respect it is not, in fact, a rst order gradient method, but a second order procedure. If all the n rst and n2 (n + 1) second partial derivatives are available, the conjugate directions can be generated in one process corresponding to the Gram-Schmidt orthogonalization (Kowalik and Osborne, 1968). It calls for expensive matrix operations. Conjugate directions can, however, be constructed without knowledge of the second derivatives: for example, from the changes in the gradient vector in the course of the iterations (Fletcher and Reeves, 1964). Because of this implicit exploitation of second order properties, conjugate directions has been classi ed as a gradient method. The conjugate gradients method of Fletcher and Reeves consists of a sequence of line searches with Hermitian interpolation (see Sect. 3.1.2.3.4). As a rst search direction v(0) at the starting point x(0), the simple gradient direction v(0) = ;rF (x(0)) is used. The recursion formula for the subsequent iterations is v(k) = (k) v(k;1) ; rF (x(k))  for all k = 1(1)n (3.25)

70

Hill climbing Strategies

with the correction factor (k) T F (x(k))

(k) = rFr(Fx((kx;1)))T r rF (x(k;1))

For a quadratic objective function with a positive de nite Hessian matrix, conjugate directions are generated in this way and the minimum is found with n line searches. Since at any time only the last direction needs to be stored, the storage requirement increases linearly with the number of variables. This often signi es a great advantage over other strategies. In the general, non-linear, non-quadratic case more than n iterations must be carried out, for which the method of Fletcher and Reeves must be modi ed. Continued application of the recursion formula (Equation (3.25)) can lead to linear dependence of the search directions. For this reason it seems necessary to forget from time to time the accumulated information and to start afresh with the simple gradient direction (Crowder and Wolfe, 1972). Various suggestions have been made for the frequency of this restart rule (Fletcher, 1972a). Absolute reliability of convergence in the general case is still not guaranteed by this approach. If the Hessian matrix of second partial derivatives has points of singularity, then the conjugate gradient strategy can fail. The exactness of the line searches also has an important e ect on the convergence rate (Kawamura and Volz, 1973). Polak (1971) de nes conditions under which the method of Fletcher and Reeves achieves greater than linear convergence. Fletcher (1972c) himself has written a FORTRAN program. Other conjugate gradient methods have been proposed by Powell (1962), Polak and Ribi)ere (1969) (see also Klessig and Polak, 1972), Hestenes (1969), and Zoutendijk (1970). Schley (1968) has published a complete FORTRAN program. Conjugate directions are also produced by the projected gradient methods (Myers, 1968 Pearson, 1969 Sorenson, 1969 Cornick and Michel, 1972) and the memory gradient methods (Miele and Cantrell, 1969, 1970 see also Cantrell, 1969 Cragg and Levy, 1969 Miele, 1969 Miele, Huang, and Heidemann, 1969 Miele, Levy, and Cragg, 1971 Miele, Tietze, and Levy, 1972 Miele et al., 1974). Relevant theoretical investigations have been made by, among others, Greenstadt (1967a), Daniel (1967a, 1970, 1973), Huang (1970), Beale (1972), and Cohen (1972). Conjugate gradient methods are encountered especially frequently in the elds of functional optimization and optimal control problems (Daniel, 1967b, 1971 Pagurek and Woodside, 1968 Nenonen and Pagurek, 1969 Roberts and Davis, 1969 Polyak, 1969 Lasdon, 1970 Kelley and Speyer, 1970 Kelley and Myers, 1971 Speyer et al., 1971 Kammerer and Nashed, 1972 Szego and Treccani, 1972 Polak, 1972 McCormick and Ritter, 1974). Variable metric strategies are also sometimes classi ed as conjugate gradient procedures, but more usually as quasi-Newton methods. For quadratic objective functions they generate the same sequence of points as the Fletcher-Reeves strategy and its modi cations (Myers, 1968 Huang, 1970). In the non-quadratic case, however, the search directions are di erent. With the variable metric, but not with conjugate directions, Newton directions are approximated. For many practical problems it is very dicult if not impossible to specify the partial derivatives as functions. The sensitivity of most conjugate gradient methods to imprecise

Multidimensional Strategies

71

speci cation of the gradient directions makes it seem inadvisable to apply nite di erence methods to approximate the slopes of the objective function. This is taken into account by some procedures that attempt to construct conjugate directions without knowledge of the derivatives. The oldest of these was devised by Smith (1962). On the basis of numerical tests by Fletcher (1965), however, the version of Powell (1964) has proved to be better. It will be briey presented here. It is arguable whether it should be counted as a gradient strategy. Its intermediate position between direct search methods that only use function values, and Newton methods that make use of second order properties of the objective function (if only implicitly), nevertheless makes it come close to this category. The strategy of conjugate directions is based on the observation that a line through the minimum of a quadratic objective function cuts all contours at the same angle. Powell's idea is then to construct such special directions by a sequence of line searches. The unit vectors are taken as initial directions for the rst n line searches. After these, a minimization is carried out in the direction of the overall result. Then the rst of the old direction vectors is eliminated, the indices of the remainder are reduced by one and the direction that was generated and used last is put in the place freed by the nth vector. As shown by Powell, after n cycles, each of n + 1 line searches, a set of conjugate directions is obtained provided the objective function is quadratic and the line searches are carried out exactly. Zangwill (1967) indicates how this scheme might fail. If no success is obtained in one of the search directions, i.e., the distance covered becomes zero, then the direction vectors are linearly dependent and no longer span the complete parameter space. The same phenomenon can be provoked by computational inaccuracy. To prevent this, Powell has modi ed the basic algorithm. First of all, he designs the scheme of exchanging directions to be more exible, actually by maximizing the determinant of the normalized direction vectors. It can be shown that, assuming a quadratic objective function, it is most favorable to eliminate the direction in which the largest distance was covered (see Dixon, 1972a). Powell would also sometimes leave the set of directions unchanged. This depends on how the value of the determinant would change under exchange of the search directions. The objective function is here tested at the position given by doubling the distance covered in the cycle just completed. Powell makes the termination of the search depend on all variables having changed by less than 0:1 " within an iteration, where " represents the required accuracy. Besides this rst convergence criterion, he o ers a second, stricter one, according to which the state reached at the end of the normal procedure is slightly displaced and the minimization repeated until the termination conditions are again ful lled. This is followed by a line search in the direction of the di erence vector between the last two endpoints. The optimization is only nally ended when the result agrees with those previously obtained to within the allowed deviation of 0:1 " for each component. The algorithm of Powell runs as follows: Step 0: (Initialization) Specify a starting point x(0) and accuracy requirements "i > 0 for all i = 1(1)n.

72

Hill climbing Strategies

Step 1: (Specify rst set of directions) Set vi(0) = ei for all i = 1(1)n and set k = 0. Step 2: (Start outer loop) Set x(k0) = x(k) and i = 1. Step 3: (Line search) Determine x(ki) such that (ki;1) F (x(ki)) = min + s vi(k))g: s fF (x Step 4: (Inner loop) If i < n increase i i + 1 and go to step 3. Step 5: (First termination criterion) Increase k k + 1. (k) Set x(k) = x(k;1n) and vi = vi(k;1) for all i = 1(1)n. If jx(ik) ; x(ik;1)j < 0:1 "i for all i = 1(1)n, go to step 9. Step 6: (First test for direction exchange) Determine F~ = F (2 x(k) ; x(k;1)). If F~  F (x(k;1)), go to step 2. Step 7: (Second test for direction exchange) Determine the index ` 1  `  n such that '` = max f'i i = 1(1)ng where 'i = F (x(k;1i;1)) ; F (x(k;1i)). i If "F (x(k;1)) ; 2 F (x(k)) + F~ ] "F (x(k;1)) ; F (x(k)) ; '` ]2   21 '` "F (x(k;1)) ; F~ ]2, go to step 2. Step 8: (New direction set and additional line search) Eliminate v`(k) from the list of directions so that vn(k) becomes free. Set vn(k) = x(k) ; x(k;1) = x(k;1n) ; x(k;10). Determine a new x(k) such that (k;1n) F (x(k)) = min + s vn(k))g: s fF (x Go to step 2. Step 9: (Second termination criterion) n Set y(1) = x(k) and replace x(0) y(1) + P 10 " ei . i=1 Repeat steps 1 to 8 until the convergence criterion (step 5) is ful lled again and call the result y(2). Determine y(3) such that (2) (2) (1) F (y(3)) = min s fF (y + s (y ; y ))g: If jyi(3) ; yi(2)j < 0:1 "i for all i = 1(1)n and jyi(3) ; yi(1)j < 0:1 "i for all i = 1(1)n, then end the search with the result y(3) and F (y(3))

Multidimensional Strategies

73

otherwise set x(0) = y(3) v1(0) = y(3) ; y(1) vi(0) = vi(k) for i = 2(1)n, k = 0, and go to step 2. Figure 3.9 illustrates a few iterations for a hypothetical two parameter function. Each of the rst loops consists of n + 1 = 3 line searches and leads to the adoption of a new search direction. If the objective function had been of second order, the minimum would certainly have been found by the last line search of the second loop. In the third and fourth loops it has been assumed that the trial steps have led to a decision not to exchange directions, thus the old direction vectors, numbered v3 and v4 are retained. Further loops, e.g., according to step 9, are omitted. The quality of the line searches has a strong inuence on the construction of the conjugate directions. Powell uses a sequence of Lagrangian quadratic interpolations. It is terminated as soon as the required accuracy is reached. For the rst minimization within an iteration three points and Equation (3.16) are used. The argument values taken in direction vi are: x (the starting point), x + si vi, and either x + 2 si vi or x ; si vi, according to whether F (x + si vi) < F (x) or not. The step length si is given initially by the associated accuracy "i multiplied by a maximum factor and is later adjusted in the course of the iterations. In the direction constructed from the successful results of n one dimensional searches, the argument values are called x(k0) x(kn), and 2 x(kn) ; x(k0). With three points (a < b < c) and associated objective function values (Fa Fb Fc), not only the minimum but also the second derivative of the quadratic trial function P (x) can be speci ed. In the notation of Section 3.1.2.3.3, the formula for the curvature i in the direction vi is Starting point End points of line searches Tests for direction exchange v 1

End points of iterations

(0,1)

(0,0)

v 2 (0,2)

v 3

(0,3)=(1,0)

(1,1) (3,1)

(2,1)

(1,2) v 4

(1,3)=(2,0)

(3,3)=(4,0) v (3,2) 5

(2,2)=(3,0)

Figure 3.9: Strategy of Powell, conjugate directions

74

Hill climbing Strategies

@ 2 (P (x + s v )) = ;2 (b ; c) Fa + (c ; a) Fb + (a ; b) Fc i = @s i 2 (b ; c) (c ; a) (a ; b) Powell uses this quantity i for all subsequent interpolations in the direction vi as a scale for the second partial derivative of the objective function. He scales the directions vi, p which in his case are not normalized, by 1= i. This allows the possibility of subsequently carrying out a simpli ed interpolation with only two argument values, x and x + si vi. It is a worthwhile procedure, since each direction is used several times. The predicted minimum, assuming that the second partial derivatives have value unity, is then   x0 = x + 12 si ; s1 "F (x + si vi) ; F (x)] vi i For the trial step lengths si, Powell uses the empirical recursion formula q s(ik) = 0:4 F (x(k)) ; F (x(k;1)) Because of the scaling, all the step lengths actually become the same. A more detailed justi cation can be found in Ho mann and Hofmann (1970). Contrary to most other optimization procedures, Powell's strategy is available as a precise algorithm in a tested code (Powell, 1970f). As Fletcher (1965) reports, this method of conjugate directions is superior for the case of a few variables both to the DSC method and to a strategy of Smith, especially in the neighborhood of minima. For many variables, however, the strict criterion for adopting a new direction more frequently causes the old set of directions to be retained and the procedure then converges slowly. A problem which had a singular Hessian matrix at the minimum made the DSC strategy look better. In a later article, Fletcher (1972a) de nes a limit of n = 10 to 20, above which the Powell strategy should no longer be applied. This is con rmed by the test results presented in Chapter 6. Zangwill (1967) combines the basic idea of Powell with relaxation steps in order to avoid linear dependence of the search directions. Some results of Rhead (1971) lead to the conclusion that Powell's improved concept is superior to Zangwill's. Brent (1973) also presents a variant of the strategy without derivatives, derived from Powell's basic idea, which is designed to prevent the occurrence of linear dependence of the search directions without endangering the quadratic convergence. After every n + 1 iterations the set of directions is replaced by an orthogonal set of vectors. So as not to lose all the information, however, the unit vectors are not chosen. For quadratic objective functions the new directions remain conjugate to each other. This procedure requires O(n3 ) computational operations to determine orthogonal eigenvectors. As, however, they are only performed every O(n2 ) line searches, the extra cost is O(n) per function call and is thus of the same order as the cost of evaluating the objective function itself. Results of tests by Brent con rm the usefulness of the strategy.

3.2.3 Newton Strategies

Newton strategies exploit the fact that, if a function can be di erentiated any number of times, its value at the point x(k+1) can be represented by a series of terms constructed at

Multidimensional Strategies

75

another point x(k): where

F (x(k+1)) = F (x(k)) + hT rF (x(k)) + 12 hT r2F (x(k)) h + : : :

(3.26)

h = x(k+1) ; x(k) In this Taylor series, as it is called, all the terms of higher than second order are zero if F (x) is quadratic. Di erentiating Equation (3.26) with respect to h and setting the derivative equal to zero, one obtains a condition for the stationary points of a second order function: rF (x(k+1)) = rF (x(k)) + r2 F (x(k)) (x(k+1) ; x(k) ) = 0

or

x(k+1) = x(k) ; "r2F (x(k))];1 rF (x(k)) (3.27) If F (x) is quadratic and r2F (x(0)) is positive-de nite, Equation (3.27) yields the solution x(1) in a single step from any starting point x(0) without needing a line search. If Equation (3.27) is taken as the iteration rule in the general case it represents the extension of the Newton-Raphson method to functions of several variables (Householder, 1953). It is also sometimes called a second order gradient method with the choice of direction and step length (Crockett and Cherno , 1955) v(k) = ;"r2F (x(k))];1 rF (x(k)) s(k) = 1

(3.28)

The real length of the iteration step is hidden in the non-normalized Newton direction v . Since no explicit value of the objective function is required, but only its derivatives, the Newton-Raphson strategy is classi ed as an indirect or analytic optimization method. Its ability to predict the minimum of a quadratic function in a single calculation at rst sight looks very attractive. This single step, however, requires a considerable e ort. Apart from the necessity of evaluating n rst and n2 (n + 1) second partial derivatives, the Hessian matrix r2F (x(k)) must be inverted. This corresponds to the problem of solving a system of linear equations r2 F (x(k)) 4 x(k) = ;rF (x(k)) (3.29) for the unknown quantities 4x(k). All the standard methods of linear algebra, e.g., Gaussian elimination (Brown and Dennis, 1968 Brown, 1969) and the matrix decomposition method of Cholesky (Wilkinson, 1965), need O(n3) computational operations for this (see Schwarz, Rutishauser, and Stiefel, 1968). For the same cost, the strategies of conjugate directions and conjugate gradients can execute O(n) steps. Thus, in principle, the Newton-Raphson iteration o ers no advantage in the quadratic case. If the objective function is not quadratic, then
v (0) does not in general point towards a minimum. The iteration rule (Equation (3.27)) must be applied repeatedly. (k )

76

Hill climbing Strategies


s(k) = 1 may lead to a point with a worse value of the objective function. The search diverges, e.g., when r2F (x(k)) is not positive-de nite.




It can happen that r2F (x(k)) is singular or almost singular. The Hessian matrix cannot be inverted.

Furthermore, it depends on the starting point x(0) whether a minimum, a maximum, or a saddle point is approached, or the whole iteration diverges. The strategy itself does not distinguish the stationary points with regard to type. If the method does converge, then the convergence is better than of linear order (Goldstein, 1965). Under certain, very strict conditions on the structure of the objective function and its derivatives even second order convergence can be achieved (e.g., Polak, 1971) that is, the number of exact signi cant gures in the approximation to the minimum solution doubles from iteration to iteration. This phenomenon is exhibited in the solution of some test problems, particularly in the neighborhood of the desired extremum. All the variations of the basic procedure to be described are aimed at increasing the reliability of the Newton iteration, without sacri cing the high convergence rate. A distinction is made here between quasi-Newton strategies, which do not evaluate the Hessian matrix explicitly, and modied Newton methods, for which rst and second derivatives must be provided at each point. The only strategy presently known which makes use of higher than second order properties of the objective function is due to Biggs (1971, 1973). The simplest modi cation of the Newton-Raphson scheme consists of determining the step length s(k) by a line search in the Newton direction v(k) (Equation (3.28)) until the relative optimum is reached (e.g., Dixon, 1972a): (k) (k) F (x(k) + s(k) v(k)) = min s fF (x + s v )g

(3.30)

To save computational operations, the second partial derivatives can be redetermined less frequently and used for several iterations. Care must always be taken, however, that v(k) always points \downhill," i.e., the angle between v(k) and ;rF (x(k)) is less than 900 . The Hessian matrix must also be positive-de nite. If the eigenvalues of the matrix are calculated when it is inverted, their signs show whether this condition is ful lled. If a negative eigenvalue appears, Pearson (1969) suggests proceeding in the direction of the associated eigenvector until a point is reached with positive-de nite r2F (x). Greenstadt (1967a) simply replaces negative eigenvalues by their absolute value and vanishing eigenvalues by unity. Other proposals have been made to keep the Hessian matrix positivede nite by addition of a correction matrix (Goldfeld, Quandt, and Trotter, 1966, 1968 Shanno, 1970a) or to include simple gradient steps in the iteration scheme (Dixon and Biggs, 1972). Further modi cations, which operate on the matrix inversion procedure itself, have been suggested by Goldstein and Price (1967), Fiacco and McCormick (1968), and Matthews and Davies (1971). A good survey has been given by Murray (1972b). Very few algorithms exist that determine the rst and second partial derivatives numerically from trial step operations (Whitley, 1962 see also Wasscher, 1963c Wegge, 1966). The inevitable approximation errors too easily cancel out the advantages of the Newton directions.

Multidimensional Strategies

77

3.2.3.1 DFP: Davidon-Fletcher-Powell Method (Quasi-Newton Strategy, Variable Metric Strategy)

Much greater interest has been shown for a group of second order gradient methods that attempt to approximate the Hessian matrix and its inverse during the iterations only from

rst order data. This now extensive class of quasi-Newton strategies has grown out of the work of Davidon (1959). Fletcher and Powell (1963) improved and translated it into a practical procedure. The Davidon-Fletcher-Powell or DFP method and some variants of it are also known as variable metric strategies. They are sometimes also regarded as conjugate gradient methods, because in the quadratic case they generate conjugate directions. For higher order objective functions this is no longer so. Whereas the variable metric concept is to approximate Newton directions, this is not the case for conjugate gradient methods. The basic recursion formula for the DFP method is

x(k+1) = x(k) + s(k) v(k) with and

v(k) = ;H (k)T rF (x(k)) H (0) = I

H (k+1) = H (k) + A(k) The correction A(k) to the approximation for the inverse Hessian matrix, H (k), is derived from information collected during the last iteration thus from the change in the variable vector y(k) = x(k+1) ; x(k) = s(k) v(k) and the change in the gradient vector z(k) = rF (x(k+1)) ; rF (x(k)) it is given by

(k) (k) (k) (k) T (k) (k)T A(k) = yy(k)Tyz(k) ; H z(zk)T H(H(k) z(zk) )

(3.31)

The step length s(k) is obtained by a line search along v(k) (Equation (3.30)). Since the rst partial derivatives are needed in any case they can be made use of in the one dimensional minimization. Fletcher and Powell do so in the context of a cubic Hermitian interpolation (see Sect. 3.1.2.3.4). A corresponding ALGOL program has been published by Wells (1965) (for corrections see Fletcher, 1966 Hamilton and Boothroyd, 1969 House, 1971). The rst derivatives must be speci ed as functions, which is usually inconvenient and often impossible. The convergence properties of the DFP method have been thoroughly investigated, e.g., by Broyden (1970b,c), Adachi (1971), Polak (1971), and Powell (1971, 1972a,b,c). Numerous suggestions have thereby been made for improvements. Convergence is achieved if F (x) is convex. Under stricter conditions it can be proved that the convergence rate is greater than linear and the sequence of iterations

78

Hill climbing Strategies

converges quadratically, i.e., after a nite number (maximum n) of steps the minimum of a quadratic function is located. Myers (1968) and Huang (1970) show that, if the same starting point is chosen and the objective function is of second order, the DFP algorithm generates the same iteration points as the conjugate gradient method of Fletcher and Reeves. All these observations are based on the assumption that the computational operations, including the line searches, are carried out exactly. Then H (k) always remains positivede nite if H (0) was positive-de nite and the minimum search is stable, i.e., the objective function is improved at each iteration. Numerical tests (e.g., Pearson, 1969 Tabak, 1969 Huang and Levy, 1970 Murtagh and Sargent, 1970 Himmelblau, 1972a,b), and theoretical considerations (Bard, 1968 Dixon, 1972b) show that rounding errors and especially inaccuracies in the one dimensional minimization frequently cause stability problems the matrix H (k) can easily lose its positive-de niteness without this being due to a singularity in the inverse Hessian matrix. The simplest remedy for a singular matrix H (k), or one of reduced rank, is to forget from time to time all the experience stored within H (k) and to begin again with the unit matrix and simple gradient directions (Bard, 1968 McCormick and Pearson, 1969). To do so certainly increases the number of necessary iterations, but in optimization as in other activities it is wise to put safety before speed. Stewart (1967) makes use of this procedure. His algorithm is of very great practical interest since he obtains the required information about the rst partial derivatives from function values alone by means of a cleverly constructed di erence scheme.

3.2.3.2 Strategy of Stewart: Derivative-free Variable Metric Method

Stewart (1967) focuses his attention on choosing the length of the trial step d(ik) for the approximation  (k) @F (x)  (k) gi ' Fxi x = @x  (k) i x to the rst partial derivatives in such a way as to minimize the inuence of rounding errors on the actual iteration process. Two di erence schemes are available: h i gi(k) = (1k) F (x(k) + d(ik) ei) ; F (x(k)) (forward di erence) di and h i gi(k) = 1(k) F (x(k) + d(ik) ei) ; F (x(k) ; d(ik) ei) (central di erence) (3.32) 2 di Application of the one sided (forward) di erence (Equation (3.32)) is preferred, since it only involves one extra function evaluation. To simplify the computation, Stewart introduces the vector h(k), which contains the diagonal elements of the matrix (H (k));1 representing information about the curvature of the objective function in the coordinate directions. The algorithm for determining the gi(k)  i = 1(1)n runs as follows:

Multidimensional Strategies Step 0:

Step 1:

Step 2:

(

79

) (k) j jxi j F (x(k))

(k;1)

jgi

Set  = max "b  "c ("b represents an estimate of the error in the calculation of F (x). Stewart sets "b = 10;10 and "c = 5  10;13 :)

   2 If gi(k;1)   h(ik;1) F (x(k))  v    (k)  0 1 u 0  (k;1)  u  h   F ( x )  i i  A  t  (k;1) and i = i0 @1 ; 0  (k;1) de ne i0 = 2 u hi  3 i hi  + 4 gi(k;1) otherwise de ne v  (k) (k;1)  (k;1) 0 1 u u  F ( x ) g  2  g  i i0 = 2 t3  and i = i0 @1 ; 0  (k;1)i  (k;1) A : (k;1) 2 (hi ) 3 i hi  + 4 gi 

Set di(k) = i sign(h(ik;1)) sign(gi(k;1)) ( 0 (k) 0 (k ) d  if d (k) i i0 6= 0 and di = (k;1) di  if di(k) = 0:  (k;1) (k)  di   10; , use Equation (3.32) otherwise If  hi (k;1)  2 gi    r   replace d(ik)  (k1;1) ; gi(k;1) + (gi(k;1))2 + 2  10  F (x(k)) h(ik;1) hi  and use Equation (3.33). (Stewart chooses  = 2.) Stewart's main algorithm takes the following form: Step 0: (Initialization) Choose an initial value x(0), accuracy requirements "ai > 0 i = 1(1)n, and initial step lengths d(0) i for the gradient determination, e.g., 8     if x(0) 6= 0 < 0:05 x(0) (0) i i di = : 0:05 if x(0) i = 0: Calculate the vector g(0) from Equation (3.32) using the step lengths d(0) i : (0) (0) Set H = I hi = 1 for all i = 1(1)n and k = 0: Step 1: (Prepare for line search) Determine v(k) = ;H (k) g(k): If k = 0, go to step 3. If g((kk))T v(k) < 0, go to step 3. If hi > 0 for all i = 1(1)n, go to step 3. 0

80

Hill climbing Strategies

Step 2: (Forget second order information) Replace H(0)(k) H (0) = I (k) hi hi = 1 for all i = 1(1)n and v(k) ;g(k): Step 3: (Line search and eventual break-o ) Determine x(k+1) such that (k) (k) F (x(k+1)) = min s fF (x + s v )g: If F (x(k+1))  F (x(k)), end the search with result x(k) and F (x(k)). Step 4: (Prepare for inverse Hessian update) Determine g(k+1) by the above di erence scheme. Construct y(k)  =(k)x (k+1) ; x(k) and  z(k) = g(k+1) ; g(k): (k) If k > n and vi  < "ai and yi  < "ai for all i = 1(1)n, end the search with result x(k+1) F (x(k+1)). Step 5: (Update inverse Hessian) Construct H (k+1) = H (k) + A(k) using Equation (3.31) and ! (k) (k) (k)T (k) z (k) (k) (k+1) (k) s g v ( k ) i hi = hi + (k)T (k) zi "1 ; (k)T (k) ] + 2 s gi v z v z for all i = 1(1)n: Step 6: (Main loop / termination criterion) If the denominators are non-zero, increase k k + 1, and go to step 1 otherwise terminate with result x(k+1) F (x(k+1)). In place of the cubic Hermitian interpolation, Stewart includes a quadratic Lagrangian interpolation as used by Powell in his conjugate directions strategy. Gradient calculations at the argument values are thereby avoided. One point, x(k), is given each time by the initial vector of the line search. The second, x(k) + s v(k), is situated in the direction v(k) at a distance ( ) (k) 2 ( F ( x ) ; Fm) s = min 1  ; g(k)T v(k)

Fm is an estimate of the value of the objective function at the minimum being sought. It must be speci ed beforehand. s = 1 is an upper bound corresponding to the length of a Newton-Raphson step. The third argument value is to be calculated from knowledge of the points x(k) and x(k) + s v(k), their associated objective function values, and g(k)T v(k), the derivative of the objective function at point x(k) in direction v(k). The sequence of Lagrangian interpolations is broken o if, at any time, the predicted minimum worsens the situation or lies at such a distance outside the interval that it is more than twice as far from the next point as the latter is from the midpoint. Lill (1970, 1971) (see also Kov%acs and Lill, 1971) has published a complete ALGOL program for the derivative-free DFP strategy of Stewart it di ers slightly from the original only in the line search. Fletcher (1969b) reports tests that demonstrate the superiority of Stewart's algorithm to Powell's as the number of variables increases.

Multidimensional Strategies

81

Brown and Dennis (1972) and Gill and Murray (1972) have suggested other schemes for obtaining the partial derivatives numerically from values of the objective function. Stewart himself reports tests that show the usefulness of his rules insofar as the results are completely comparable to others obtained with the help of analytically speci ed derivatives. This may be simply because rounding errors are in any case more signi cant here, due to the matrix operations, than for example in conjugate gradient methods. Kelley and Myers (1971), therefore, recommend carrying out the matrix operations with double precision.

3.2.3.3 Further Extensions

The ability of the quasi-Newton strategy of Davidon, Fletcher, and Powell (DFP) to construct Newton directions without needing explicit second partial derivatives makes it very attractive from a computational point of view. All e orts in the further rapid and intensive development of the concept have been directed to modifying the correction Equation (3.31) so as to reduce the tendency to instability because of rounding errors and inexact line searches while retaining as far as possible the quadratic convergence. There has been a spate of corresponding proposals and both theoretical and experimental investigations on the subject up to about 1973, for example: Adachi (1973a,b) Bass (1972) Broyden (1967, 1970a,b,c, 1972) Broyden, Dennis, and Mor%e (1973) Broyden and Johnson (1972) Davidon (1968, 1969) Dennis (1970) Dixon (1972a,b,c, 1973) Fiacco and McCormick (1968) Fletcher (1969a,b, 1970b, 1972b,d) Gill and Murray (1972) Goldfarb (1969, 1970) Goldstein and Price (1967) Greenstadt (1970) Hestenes (1969) Himmelblau (1972a,b) Hoshino (1971) Huang (1970, 1974) Huang and Chambliss (1973, 1974) Huang and Levy (1970) Jones (1973) Lootsma (1972a,b) Mamen and Mayne (1972) Matthews and Davies (1971) McCormick and Pearson (1969)

82

Hill climbing Strategies

McCormick and Ritter (1972, 1974) Murray (1972a,b) Murtagh (1970) Murtagh and Sargent (1970) Oi, Sayama, and Takamatsu (1973) Oren (1973) Ortega and Rheinboldt (1972) Pierson and Rajtora (1970) Powell (1969, 1970a,b,c,g, 1971, 1972a,b,c,d) Rauch (1973) Ribi)ere (1970) Sargent and Sebastian (1972, 1973) Shanno and Kettler (1970a,b) Spedicato (1973) Tabak (1969) Tokumaru, Adachi, and Goto (1970) Werner (1974) Wolfe (1967, 1969, 1971) Many of the di erently sophisticated strategies, e.g., the classes or families of similar methods de ned by Broyden (1970b,c) and Huang (1970), are theoretically equivalent. They generate the same conjugate directions v(k) and, with an exact line search, the same sequence x(k) of iteration points if F (x) is quadratic. Dixon (1972c) even proves this identity for more general objective functions under the condition that no term of the sequence H (k) is singular. The important nding that under certain assumptions convergence can also be achieved without line searches is attributed to Wolfe (1967). A recursion formula satisfying these conditions is as follows: H (k+1) = H (k) + B (k) where (k ) (k) z (k)) (y (k) ; H (k) z (k) )T (3.33) B (k) = (y ;(H y(k) ; H (k) z(k)) z(k)T The formula was proposed independently by Broyden (1967), Davidon (1968, 1969), Pearson (1969), and Murtagh and Sargent (1970) (see Powell, 1970a). The correction matrix B (k) has rank one, while A(k) in Equation (3.31) is of rank two. Rank one methods, also called variance methods by Davidon, cannot guarantee that H (k) remains positive-de nite. It can also happen, even in the quadratic case, that H (k) becomes singular or B (k) increases without bound. Hence in order to make methods of this type useful in practice a number of additional precautions must be taken (Powell, 1970a Murray, 1972c). The following compromise proposal H (k+1) = H (k) + A(k) + (k) B (k) (3.34) where the scalar parameter (k) > 0 can be freely chosen, is intended to exploit the advantages of both concepts while avoiding their disadvantages (e.g., Fletcher, 1970b). Broyden

Multidimensional Strategies

83

(1970b,c), Shanno (1970a,b), and Shanno and Kettler (1970) give criteria for choosing suitable (k). However, the mixed correction, also known as BFS or Broyden-Fletcher-Shanno formula, cannot guarantee quadratic convergence either unless line searches are carried out. It can be proved that there will merely be a monotonic decrease in the eigenvalues of the matrix H (k). From numerical tests, however, it turns out that the increased number of iterations is usually more than compensated for by the saving in function calls made by dropping the one dimensional optimizations (Fletcher, 1970a). Fielding (1970) has designed an ALGOL program following Broyden's work with line searches (Broyden, 1965). With regard to the number of function calls it is usually inferior to the DFP method but it sometimes also converges where the variable metric method fails. Dixon (1973) de nes a correction to the chosen directions,

v(k) = ;H (k) rF (x(k)) + w(k) where and

w(0) = 0

(k+1) (k) T (k+1) w(k+1) = w(k) + (x (x(k;+1)x ; )x(kr))FT (zx(k) ) (x(k+1) ; x(k)) by which, together with a matrix correction as given by Equation (3.35), quadratic convergence can be achieved without line searches. He shows that at most n + 2 function calls and gradient calculations are required each time if, after arriving at v(k) = 0, an iteration x(k+1) = x(k) ; H (k) rF (x(k)) is included. Nearly all the procedures de ned assume that at least the rst partial derivatives are speci ed as functions of the variables and are therefore exact to the signi cant

gure accuracy of the computer used. The more costly matrix computations should wherever possible be executed with double precision in order to keep down the e ect of rounding errors. Just two more suggestions for derivative-free quasi-Newton methods will be mentioned here: those of Greenstadt (1972) and of Cullum (1972). While Cullum's algorithm, like Stewart's, approximates the gradient vector by function value di erences, Greenstadt attempts to get away from this. Analogously to Davidon's idea of approximating the Hessian matrix during the course of the iterations from knowledge of the gradients, Greenstadt proposes approximating the gradients by using information from objective function values over several subiterations. Only at the starting point must a di erence scheme for the rst partial derivatives be applied. Another interesting variable metric technique described by Elliott and Sworder (1969a,b, 1970) combines the concept of the stochastic approximation for the sequence of step lengths with the direction algorithms of the quasi-Newton strategy. Quasi-Newton strategies of degree one are especially suitable if the objective function is a sum of squares (Bard, 1970). Problems of minimizing a sum of squares arise for example from the problem of solving systems of simultaneous, non-linear equations,

84

Hill climbing Strategies

or determining the parameters of a mathematical model from experimental data (nonlinear regression and curve tting). Such objective functions are easier to handle because Newton directions can be constructed straight away without second partial derivatives, as long as the Jacobian matrix of rst derivatives of each term of the objective function is given. The oldest iteration procedure constructed on this basis is variously known as the Gauss-Newton (Gauss, 1809) method, generalized least squares method, or Taylor series method. It has all the advantages and disadvantages of the Newton-Raphson strategy. Improvements on the basic procedure are given by Levenberg (1944) and Marquardt (1963). Wolfe's secant method (Wolfe, 1959b see also Jeeves, 1958) is the forerunner of many variants which do not require the Jacobian matrix to be speci ed at the start but construct it in the course of the iterations. Further details will not be described here the reader is referred to the specialist literature, again up to 1973: Barnes, J.G.P. (1965) Bauer, F.L. (1965) Beale (1970) Brown and Dennis (1972) Broyden (1965, 1969, 1971) Davies and Whitting (1972) Dennis (1971, 1972) Fletcher (1968, 1971) Golub (1965) Jarratt (1970) Jones (1970) Kowalik and Osborne (1968) Morrison (1968) Ortega and Rheinboldt (1970) Osborne (1972) Peckham (1970) Powell (1965, 1966, 1968b, 1970d,e, 1972a) Powell and MacDonald (1972) Rabinowitz (1970) Ross (1971) Smith and Shanno (1971) Spath (1967) (see also Silverman, 1969) Stewart (1973) Vitale and Taylor (1968) Zeleznik (1968) Brent (1973) gives further references. Peckham's strategy is perhaps of particular interest. It represents a modi cation of the simplex method of Nelder and Mead (1965) and Spendley (1969) and in tests it proves to be superior to Powell's strategy (1965) with regard to the number of function calls. It should be mentioned here at least that non-linear regression, where parameters of a model that enter the model in a non-linear way (e.g., as exponents) have to be estimated, in general requires a global optimization

Multidimensional Strategies

85

method because the squared sum of residuals de nes a multimodal function. Reference has been made to a number of publications in this and the preceding chapter in which strategies are described that can hardly be called genuine hill climbing methods they would fall more naturally under the headings of mathematical programming or functional optimization. It was not, however, the intention to give an introduction to the basic principles of these two very wide subjects. The interested reader will easily nd out that although a nearly exponentially increasing number of new books and journals have become available during the last three decades, she or he will look in vain for new direct search strategies in that realm. Such methods form the core of this book.

86

Hill climbing Strategies

Chapter 4 Random Strategies One group of optimization methods has been completely ignored in Chapter 3: methods in which the parameters are varied according to probabilistic instead of deterministic rules even the methods of stochastic approximation are deterministic. As indicated by the title there is not one random strategy but many, some of which dier considerably from each other. It is common to resort to random decisions in optimization whenever deterministic rules do not have the desired success, or lead to a dead end on the other hand random strategies are often supposed to be essentially more costly. The opinion is widely held that with careful thought leading to cleverly constructed deterministic rules, better results can always be achieved than with decisions that are in some way made randomly. The strategies that follow should show that randomness is not, however, the same as arbitrariness, but can also be made to obey very rened rules. Sometimes only this kind of method solves a problem eectively. Profound considerations do not underlie all the procedures used in hill climbing strategies. The cyclic choice of coordinate directions in the Gauss-Seidel strategy could just as well be replaced by a random sequence. One can also consider increasing the number of directions used. Since there is no good reason for preferring to search for the optimum along directions parallel to the axes, one could also use, instead of only n dierent unit vectors, any number of randomly chosen direction vectors. In fact, suggestions along these lines have been made (Brooks, 1958) in order to avoid a premature termination of the minimum search in narrow oblique valleys (compare Chap. 3, Sect. 3.2.1.1). Similar concepts have been developed for example by O'Hagan and Moler (after Wilde and Beightler, 1967), Emery and O'Hagan (1966), Lawrence and Steiglitz (1972), and Beltrami and Indusi (1972), to improve the pattern search of Hooke and Jeeves (1961, see Chap. 3, Sect. 3.2.1.2). The limitation to a nite number of search directions is not only a disadvantage in narrow oblique valleys but also at the border of the feasible region as determined by inequality constraints. All the deterministic remedies against prematurely ending the iteration sequence assume that more information can be gathered, for example in the form of partial derivatives of the constraint functions (see Klingman and Himmelblau, 1964 Glass and Cooper, 1965 Paviani and Himmelblau, 1969). Providing this information usually means a high extra cost and is sometimes not possible at all. 87

88

Random Strategies

Random directions that are not oriented with respect to the structure of the objective function and the allowed region also imply a higher cost because they do not take optimal single steps. They can, however, be applied in every case. Many deterministic optimization methods, especially those which are guided by the gradient of the objective function, have convergence diculties at points where the partial derivatives are discontinuous. On the contour diagram of a two parameter objective function, of which the maximum is sought, such positions correspond to sharp ridges leading to the summit (e.g., Zwart, 1970). A narrow valley{the geometric picture in the case of minimization{leads to the same problem if the nite step lengths are greater than its width. Then all attempts fail to make improvements in the coordinate directions or, from trial steps in these directions, fail to predict a locally best direction in which to continue (gradient direction). The same phenomenon can also occur when the partial derivatives are specied analytically, because of the rounding errors involved in computing with a nite number of signicant gures. To avoid premature termination of a search in such cases, Norkin (1961) has suggested the following procedure. When the optimization according to the conventional scheme has ended, a step is taken away from the supposed optimum in an arbitrary coordinate direction. The extremum is sought again, excluding this one variable, and the search is only nally ended when deviations in all directions have led back to the same point. This rule should also prevent stagnation at saddle points. Even the simplex method of linear programming makes random decisions if the search for the extremum threatens to be endless because the problem is degenerate. Then following Dantzig's suggestion (1966) the iteration scheme should be interrupted in favor of a random exchange step. A problem is only degenerate, however, because the general rules do not cover the special case (see also Chap. 6, Sect. 6.2). A further example of resorting to chance when a dead end has been reached is Brent's modication of the strategy with conjugate directions (Brent, 1973). Powell's algorithm (Powell, 1964) when applied to problems in many dimensions tends to generate linearly dependent directions and then to proceed within a subspace of IRn. For this reason Brent now and then interrupts the line searches with steps in randomly chosen directions (see also Chap. 3, Sect. 3.2.2.1). One very frequently comes across proposals to let chance take control when the problem is to nd global minima of multimodal objective functions. Such problems frequently crop up in process design (Motskus, 1967 Mockus, 1971) but can also be the result of recasting discrete problems into continuous form (Katkovnik and Shimelevich, 1972). Practically all sequential search procedures can only lead to a local optimum{as a rule, the one nearest to the starting point. There are a few proposals for ensuring global convergence of sequential optimization methods (e.g., Motskus and Feldbaum, 1963 Chichinadze, 1967, 1969 Goldstein and Price, 1971 Ueing, 1971, 1972 Branin and Hoo, 1972 McCormick, 1972 Sutti, Trabattoni, and Brughiera, 1972 Treccani, Trabattoni, and Szego, 1972 Brent, 1973 Hesse, 1973 Opacic, 1973 Ritter and Tui as mentioned by Zwart, 1973). They are often in the form of additional, heuristic rules. Gran (1973), for example, considers gradient methods that are supposed to achieve global convergence by the addition of a random process to the deterministic changes. Hill (1964 see also Hill and Gibson, 1965) suggests subdividing the interval to be explored and gathering sucient information in each section to carry out a cubic interpolation. The best of the results for the

Random Strategies

89

parts is taken as an approximation to the global optimum. However, for n-dimensional interpolations the cost increases rapidly with n this scheme thus looks impractical for more than two variables. To work with several, randomly chosen starting points and to compare each of the local minima (or maxima) obtained is usually regarded as the only course of action for determining the global optimum with at least a certain probability (so-called multistart techniques). Proposals along these lines have been made by, among others, Gelfand and Tsetlin (1961), Bromberg (1962), Bocharov and Feldbaum (1962), Zellnik, Sondak, and Davis (1962), Krasovskii (1962), Gurin and Lobac (1963), Flood and Leon (1964, 1966), Kwakernaak (1965), Casey and Rustay (1966), Weisman and Wood (1966), Pugh (1966), McGhee (1967), Crippen and Scheraga (1971), and Brent (1973). A further problem faces deterministic strategies if the calculated or measured values of the objective function are subject to stochastic perturbations. In the experimental eld, for example in the on-line optimum search, or for control of the optimal conditions in processes, perturbations must be taken into account from the start (e.g., Tovstucha, 1960 Feldbaum, 1960, 1962 Krasovskii, 1963 Medvedev, 1963, 1968 Kwakernaak, 1966 Zypkin, 1967). However, in computational optimization too, where the objective function is analytically specied, a similar eect arises because of rounding errors (Brent, 1973), especially if one uses hybrid analogue computers for solving functional optimization problems (e.g., Gilbert, 1967 Korn and Korn, 1964 Bekey and Karplus, 1971). A simple, if expensive (in the sense of cost in computations or trials) method of dealing with this is the repetition of measurements until a denite conclusion is possible. This is the procedure adopted by Box and Wilson (1951) in the experimental gradient method, and by Box (1957) in his EVOP strategy. Instead of a xed number of repetitions, which while on the safe side may be unnecessarily high, one can follow the concept of sequential analysis of statistical data (Wald, 1966 see also Zigangirov, 1965 Schumer, 1969 Kivelidi and Khurgin, 1970 Langguth, 1972), which is to make only as many trials as the trial results seem to make absolutely necessary. More detailed investigations on this subject have been made, for example, by Mlynski (1964a,b, 1966a,b). As opposed to attempting to improve the decisive data, Brooks and Mickey (1961) have found that one should work with the minimum number of n + 1 comparison points in order to determine a gradient direction, even if this is a perturbed one. One must however depart from the requirement that each step should yield a success, or even the locally greatest success. The motto that following locally the best possible route seldom leads to the best overall result is true not only for rst order gradient strategies but also for Newton and quasi-Newton methods . Harkins (1964), for example, maintains that inexact line searches not only do not worsen the convergence of a minimization procedure but in some cases actually improve it. Similar experiences led Davies, Swann, and Campey in their strategy (see Chap. 3, Sect. 3.2.1.4) to make only one quadratic interpolation in each direction. Also Spendley, Hext, and Himsworth (1962), in the formulation of their simplex method, which generates only near-optimal directions, work on the assumption that random decisions are not necessarily a total disadvantage (see also Himsworth, 1962). Based on similar arguments, the modication of this strategy by M. J. Box (1965) sets up the initial simplex or complex by means of random numbers. Imamura et al. (1970) even go so far as to superimpose articial stochastic variations on an objective function

90

Random Strategies

in order to prevent convergence to inferior local optima. The rigidity of an algorithm based on a xed internal model of the objective function, with which the information gathered during the iterations is interpreted, is advantageous if the objective function corresponds closely enough to the model. If this is not the case, the advantage disappears and may even turn into a disadvantage. Second order methods with quadratic models seem more sensitive in this respect than rst order methods with only linear models. Even more robust are the direct search strategies that work without an explicit model, such as the strategy of Hooke and Jeeves (1961). It makes no use of the sizes of the changes in the objective function values, but only of their signs. A method that uses a kind of minimal model of the objective function is the stochastic approximation (Schmetterer, 1961 see also Chap. 2, Sect. 2.3). This purely deterministic method assumes that the measured or calculated function values are samples of a normally distributed random quantity, of which the expectation value is to be minimized or maximized. The method feels its way to the optimum with alternating exploratory and work steps, whose lengths form convergent series with prescribed bounds and sums. In the multidimensional case this standard concept can be the basis of various strategies for choosing the directions of the work steps (Fabian, 1968). Usually gradient methods show themselves to best advantage here. The stochastic approximation itself is very versatile. Constraints can be taken into account (Kaplinskii and Propoi, 1970), and problems of functional optimization can be treated (Gersht and Kaplinskii, 1971) as well as dynamic problems of maintaining or seeking optima (Chang, 1968). Tsypkin (1968a,b,c, 1970a,b see also Zypkin, 1966, 1967, 1970) discusses these topics very thoroughly. There are also, however, arguments against the reliability of convergence for certain types of objective function (Aizerman, Braverman and Rozonoer, 1965). The usefulness of the strategy in the multidimensional case is limited by its high cost. Hence there has been no shortage of attempts to accelerate the convergence (Fabian, 1967 Berlin, 1969 Saridis, 1968, 1970 Saridis and Gilbert, 1970 Janac, 1971 Kwatny, 1972 see also Chap. 2, Sect. 2.3). Ideas for using random directions look especially promising some of the many investigations of this topic which have been published are Loginov (1966), Stratonovich (1968, 1970), Schmitt (1969), Ermoliev (1970), Svechinskii (1971), Tsypkin (1971), Antonov and Katkovnik (1972), Berlin (1972), Katkovnik and Kulchitskii (1972), Kulchitskii (1972), Poznyak (1972), and Tsypkin and Poznyak (1972). The original method is not able to determine global extrema reliably. Extensions of the strategy in this direction are due to Kushner (1963, 1972) and Vaysbord and Yudin (1968). The sequence of work steps is so designed that the probability of the following state being the global optimum is maximized. In contrast to the gradient concept, the information gathered is not interpreted in terms of local but of global properties of the objective function. In the case of two local minima, the eort of the search is gradually concentrated in their neighborhood and only when one of them is signicantly better is the other abandoned in favor of the one that is also a global minimum. In terms of the cost of the strategy, the acceleration of the local search and the reliability of the global search are diametrically opposed. Hill and Gibson (1965) show that their global strategy is superior to Kushner's, as well as to one of Bocharov and Feldbaum. However, they only treat cases with n
2 parameters. More recent research results have been presented by

Random Strategies

91

Pardalos and Rosen (1987), Torn and Z ilinskas (1989), Floudas and Pardalos (1990), Zhigljavsky (1991), and Rudolph (1991, 1992b). Now there are even specialized journals established in the eld, see Horst (1991). All the strategies mentioned so far are fundamentally deterministic. They only resort to chance in dead-end situations, or they operate on the assumption that the objective function is stochastically perturbed. Jarvis (1968), who compares deterministic and probabilistic optimization methods, nds that random methods that do not stick to any particular model are most suitable when an optimum must be located under particularly dicult conditions, such as a perturbed objective function or a \pathological" problem structure with several extrema, discontinuities, plateaus, forbidden regions, etc. The homeostat of Ashby (1960) is probably the oldest example of the application of a random strategy. Its objective is to maintain a condition of equilibrium against stochastic disturbances. It may happen that no optimum is sought, but only a point in an allowed region (today one calls such task a constraints satisfaction problem or CSP). Nevertheless, corresponding solution methods are closely tied to optimization, and there are a series of various heuristic planning methods available (e.g., Weinberg and Zehnder, 1969). Ashby's strategy, which he calls a blind homeostatic process, becomes active whenever the apparatus strays from equilibrium. Then the controllable parameters are randomly varied until the desired condition is restored. The nite number (in this case) of discrete settings of the variables all enter the search process with equal probability. Chichinadze (1960) later constructed an electronic model on the same principle and used it for synthesizing simple optimal control systems. Brooks (1958), probably stimulated by R. L. Anderson (1953), is generally regarded as the initiator of the use of random strategies for optimization problems. He describes the simple, later also called blind or pure random search for nding a minimum or maximum in the experimental eld. In a closed interval a
x
b several points are chosen at random. The probability density w(x) is constant everywhere within the region and zero outside. ( for all a
x
b w(x) = 10=V  otherwise V , the volume of the cube with corners ai and bi for i = 1(1)n, is given by n Y V = (bi ; ai) i=1

The value of the objective function must be determined at all selected points. The point that has the lowest or highest function value is taken as optimum. How well the true extremum is approximated depends on the number of trials as well as on the actual random results. Thus one can only give a probability p that the optimum will be found within a given number N of trials with a prescribed accuracy.

p = 1 ; (1 ; v=V )N

(4.1)

The volume v < V < 1 contains all points that satisfy the accuracy requirement. By

92

Random Strategies

rearranging Equation (4.1), the number of trials is obtained N = ln (1 ; vp) (4.2) ln (1 ; V ) that is required in order to place with probability p at least one trial in the volume v. Brooks concludes from this that the cost is independent of the number of variables. In their criticism Hooke and Jeeves (1958) point out that it is not feasible to consider the accuracy in terms of the volumev ratio for problems with many variables. For n = 100 parameters, a volume ratio of V = 0:1 corresponds to a length ratio of the side length D of V and d of v of s d = n  v  ' 0:98 D V This means that the uncertainty in the variables xi is 98% of the original interval ai bi], although the volume containing the optimum has been reduced to one tenth of the original. Shimizu (1969) makes the same mistake as Brooks and attempts to implement the strategy for problems with more general constraints. A comparison of the pure random search and deterministic search methods known at the time for experimental optimization problems (Brooks, 1959) also shows no advantage of the stochastic strategy. The test only covers four dierent objective functions, each with two variables. Brooks then recommends applying his random method if the number of parameters is large or if the determination of objective function values is subject to large perturbations. McArthur (1961) concludes on the basis of numerical experiments that the random strategy is also preferable for complicated problem structures. Just this circumstance has led to the use, even today, of the pure random search, often called the Monte-Carlo method, for example in computer optimization of building construction (Golinski and Lesniak, 1966 Lesniak, 1970 Hupfer, 1970). In principle, all the trials of the simple random strategy can be made simultaneously. It is thus numbered among the simultaneous optimization methods. The decision to choose a particular state vector of variables does not depend on the results of preceding trials, since the probability of scoring according to the uniform distribution is the same at all times. However, in applications on the traditional, serially operating computers, the trials must be made sequentially. This can be used to advantage by storing the current best value of the objective function and its associated variable value. In Chapter 3, Section 3.1.1 and 3.2 the grid or tabulation method was referred to as optimal in the minimax sense. The blind random strategy should thus not be any better. Dening the interval length Di = bi ; ai for the variable xi, with required accuracy di , and assuming that all the Di = D and di = d for i = 1(1)n, then for the volume ratio in Equations (4.1) and (4.2)  ! v= d n V D If Vv is small, which when there are many variables must be the case, one can use the approximation ln (1 + y) ' y for y  1

Random Strategies

93

to write the number of required trials as

 n N ' ; ln(1 ; p) Dd

Assuming that Dd is an integer, the grid method requires  D n N= d trials (compare Chap. 3, Sect. 3.2, Equation (3.19)). The value is the same for both procedures if p ' 0:63. Supposing that the probability of at least one score of the required accuracy is p = 0:90, then the random strategy results in  D n N ' 2:3 d which is clearly worse than the grid strategy (Spang, 1962). The reason for the extra cost, however, should not be attributed to the randomness of decisions itself, but to the fact that for an equiprobable, continuous selection of variables, the trials can be very close together or, in the discrete case, they can repeat themselves. If one can avoid that, the disadvantage would no longer exist. A randomized sequence of trials even might hit upon the optimal result earlier than an ordered one. Nevertheless Spang's proof has for some time brought all random methods, not only the simple Monte-Carlo strategy, into disrepute. Nowadays the term Monte-Carlo methods is understood to cover, in general, simulation methods that have to do with stochastic events. They are applied eectively to solving dicult dierential equations (Little, 1966) or for evaluating integrals (Cowdrey and Reeves, 1963 McGhee and Walford, 1968). Besides the simple hit-or-miss scheme, however, greatly improved variants have been developed (e.g., W. F. Bauer, 1958 Hammersley and Handscomb, 1964 Korn, 1966, 1968 Hull, 1967 Brandl, 1969). Amann (1968a,b) reports a Monte-Carlo method with information storage and a sequential extension for the solution of a linear boundary value problem, and Curtiss (1956) describes a Monte-Carlo procedure for solving systems of linear equations. Both are supposed to be less costly than comparable deterministic strategies. Pinkham (1964) and Pincus (1970) describe modications for the problems of nding zeros of a non-linear function and of constrained optimization. Since only relatively few publications treat random optimization methods in any depth (Karnopp, 1961, 1963 Idelsohn, 1964 Dickinson, 1964 Rastrigin, 1963, 1965a,b, 1966, 1967, 1968, 1969, 1972 Lavi and Vogl, 1966 Schumer, 1967 Jarvis, 1968 Heydt, 1970 Cockrell, 1970 White, 1970, 1971 Aoki, 1971 Kregting and White, 1971), the improved strategies will be brie!y presented here. They all operate with sequential and sometimes both simultaneous and sequential random trials and in one way or another exploit the information from preceding trials to accelerate the convergence. Brooks himself already suggests several improvements. Thus to exclude repetitions or closely situated trials, the volume to be investigated can be subdivided into, for example, cubic subspaces, into each of which only one random trial is placed. According to one's

94

Random Strategies

knowledge of the approximate position of the optimum, the subspaces will be assigned dierent sizes (Idelsohn, 1964). The original uniform distribution is thereby replaced by one with a greater density in the neighborhood of the expected optimum. Karnopp (1961, 1963, 1966) has treated this problem in detail without, however, giving any practical procedure. Mathematically based investigations of the same topic are due to Motskus (1965), Hupfer (1970), Pluznikov, Andreyev, and Klimenko (1971), Yudin (1965, 1966, 1972), Vaysbord (1967, 1968, 1969), Taran (1968a,b), Karumidze (1969), and Meerkov (1972). If after several (simultaneous) samples the search is continued in an especially promising looking subregion, the procedure becomes sequential in character. Suggestions of this kind have been made for example by McArthur (1961), Motskus (1965), and Hupfer (1970) (shrinkage random search). Zakharov (1969, 1970) applies the stochastic approximation for the successive shrinkage of the region in which Monte-Carlo samples are placed. The most thoroughly worked out strategy is that of McMurtry and Fu (1966, probabilistic automaton see also McMurtry, 1965). The problem considered is to adjust the variable parameters of a control system for a dynamic process in such a way that the optimum of the system is found and maintained despite perturbations and (slow) drift (Hill, McMurtry, and Fu, 1964 Hill and Fu, 1965). Initially the probabilities are equal for all subregions, at the center of which the function values are measured (assumed to be stochastically perturbed). In the course of the iterations the probability matrix is altered so that regions with better objective function values are tested more often than others. The search ends when only one subregion remains: the one with the highest probability of containing the global optimum. McMurtry and Fu use a so-called linear intensication to adjust the probability matrix. Suggestions for further improving the convergence rate have been made by Nikolic and Fu (1966), Fu and Nikolic (1966), Shapiro and Narendra (1969), Asai and Kitajima (1972), Viswanathan and Narendra (1972), and Witten (1972). Strongin (1970, 1971) treats the same problem from the point of view of decision theory. All these methods lay great emphasis on the reliability of global convergence. The quality of the approximation depends to a large extent on the number of subdivisions of the n-dimensional region under investigation. High accuracy requirements cannot be met for many variables since, at least initially, the number of subregions to investigate rises exponentially with the number of parameters. To improve the local convergence properties, there are suggestions for replacing the midpoint tests in a subvolume by the result of an extreme value search. This could be done with one of the familiar search strategies such as a gradient method (Hill, 1969) or any other purely sequential random search method (Jarvis 1968, 1970) with a high convergence rate, even if it were only guaranteed to converge locally. Application, however, is limited to problems with at most seven or eight variables, as reported. Another possibility for giving a sequential character to random methods consists of gradually shifting the expectation value of a random variable with a restricted probability density distribution. Brooks (1958) calls his proposal of this type the creeping random search. Suitable random numbers are provided for example by a Gaussian distribution with expectation value  and standard deviation . Starting from a chosen initial condition x(0), several simultaneous trials are made, which most likely fall in the neighborhood of the starting point ( = x(0)). The coordinates of the point with the best function value form

Random Strategies

95

the expectation value for the next set of random trials. In contrast to other procedures, the data from the other trials are not exploited to construct a linear or even quadratic model from which to calculate a best possible step (e.g., Brooks and Mickey, 1961 Aleksandrov, Sysoyev, and Shemeneva, 1968 Pugachev, 1970). For small  and a large number of samples, the best value will in any case fall in the locally most favorable direction. In order to approach a solution with high accuracy, the variance 2 must be successively reduced. Brooks, however, gives no practical rule for this adjustment. Many algorithms have since been published that are extensions of Brooks' basic concept of the creeping random search. Most of them no longer choose the best of several trials they accept each improvement and reject each worsening (Favreau and Franks, 1958 Munson and Rubin, 1959 Wheeling, 1960). The iteration rule of a creeping random search is, for the minimum search: ( (k) (k) F (x(k) + z(k))
F (x(k)) (success) ( k +1) x = xx(k) + z  ifotherwise (failure) The random vector z(k), which in this notation eects the change in the state vector x, belongs to an n-dimensional (0 2) normal distribution with the expectation value  = 0 and the variance 2, whichp in the simplest case is the same for all components. One can thus regard , or better  n, as a kind of average step length. The direction of z(k) is uniformly distributed in IRn , i.e., purely random. Gaussian distributions for the increments are also used by Bekey et al. (1966), Stewart, Kavanaugh, and Brocker (1967), and De Graag (1970). Gonzalez (1970) and White (1970) use instead of a normal distribution a uniform distribution that covers a small region in the form of an n-dimensional cube centered on the starting point. This clearly favors the diagonal directions, in which the total p step lengths are on average a factor n greater than in the coordinate directions. Pierre (1969) therefore restricts the uniformly distributed random probe to an n-dimensional hypersphere of xed radius. Rastrigin (1960{1972) gives the total step length v u n uX s = t zi2 i=1

a xed value. Instead of the normal distribution he thus obtains a circumferential or hypersphere-surface distribution. In addition, he repeats the evaluation of the objective function when there is a failure in order to reduce the eect of stochastic perturbations. Taking two model functions n X F1(x) = F1(x1 : : : xn) = xi (inclined plane) iv =1 u n uX F2(x) = F2(x1 : : : xn) = t x2i (hypercone) i=1

he investigates the average convergence rate of his strategy and compares it with that of an experimental gradient method, in which the partial derivatives are approximated by quotients of dierences obtained from exploratory steps . He shows that for a linear

96

Random Strategies

p

problem structure like F1 the random strategy needs only O( n) trials, whereas the gradient strategy needs O(n) trials to cover a prescribed distance. For n > 3, the random strategy is always superior to the deterministic method. Whereas Rastrigin shows that the random search always does better than the gradient search in the spherically symmetric eld F2, Movshovich (1966) maintains the opposite. The discrepancy can be traced to diering assumptions about the choice of step length (see also Yvon, 1972 Gaviano and Fagiuoli, 1972). To choose suitable step lengths or variances poses the same problems for sequential random searches as are familiar from deterministic strategies. Here too, a closely related problem is to achieve global convergence with reference to a suitable termination rule, the convergence criterion, and with a degree of reliability. Khovanov (1967) has conceived an individual manner of controlling the random step lengths. He accepts every random change, irrespective of success or failure, increases the variance at each failure and reduces it otherwise. The objective is to increase the probability of lingering in the more promising regions and to abandon states that are irrelevant to the optimum search. No applications of the strategy are known to the author. Favreau and Franks (1958), Bekey et al. (1966), and Adams and Lew (1966) use a constant ratio between i and xi for i = 1(1)n. This measure does have the eect of continuously altering the \step lengths," but its merit is not obvious. Just because a variable value xi is small in no way indicates that it is near to the extreme position being sought. Karnopp (1961) was the rst to propose a step length rule based on the number of successes or failures, according to which the i or s are all uniformly reduced or enlarged such that a success always occurs after two or three trials. Schumer (1967), and Schumer and Steiglitz (1968), submit Rastrigin's circumferential random direction method to a thorough examination by probability theory. For the model n X F3(x) = x2i = r2 i=1

with the condition n  1 and the continuously optimal step length s ' 1:225 prn they obtain a rate of progress ', which is the average distance covered in the direction of the objective (minimum) per random step: ' ' 0:203 nr and a success rate ws which is the average number of successes per trial: ws ' 0:270 They are only able to treat the general quadratic case theoretically for n = 2. Their result can be interpreted in the sense that ' is dependent on the smallest radius of curvature of the elliptic contour passing through r. Since neither r nor s can be assumed to be known in advance, it is not clear how to keep to the optimal step length. Schumer and Steiglitz (1968) give an adaptive method with which the correct size of s can be

Random Strategies

97

maintained at least approximately during the course of the iterations. At the starting point x(0) two random changes are made with step lengths s(0) and s(0) (1 + a), where 0 < a
1. If both samples are successful, for the next iteration s(1) = s(0) (1 + a) is taken, i.e., the greater value. If only one sample yields an improvement in the objective function, its step length is taken nally if no success is scored, s(1) remains equal to s(0). A reduction in s is only made if several consecutive trials are unsuccessful. This is also the procedure of Maybach (1966). This adjustment to the local conditions assists the strategy in achieving high convergence rates but reduces the chances of locating global optima among several local ones. For this reason a sample with a signicantly larger step length (a > 1) should be included from time to time. Numerical tests show that the computation cost, or number of trials, actually only increases linearly with the number of variables. Schumer and Steiglitz have tested this using the model functions F3 and n X F4(x) = x4i i=1

A comparison with a Newton-Raphson strategy, in which the partial rst and second derivatives are determined numerically and the cost increases as O(n2 ), favors the random method when n > 78 for F3 and when n > 2 for F4. For the second, biquadratic model function, Nelder and Mead (1965) state that the number of trials or function evaluations in their simplex strategy grows as O(n2:11), so that the sequential random method is superior from n > 10. White and Day (1971) report numerical tests in which the cost in iterations with Schumer's strategy increases more sharply than linearly with n, whereas a modication by White (1970) shows exact linear dependence. A comparison with the strategy of Fletcher and Powell (1963) favors the latter, especially for truly quadratic functions. Rechenberg (1973), with an n-dimensional normal distribution (see Chap. 5, Sect. 5.1), reaches almost the same theoretical results as Schumer for the circumferential distribution, if one notes that the overall step length v u n p uX tot = t i2 =  n i=1

for equal variances i2 = 2 in each random component zi is proportional to the square root of the number of variables. The reason for this lies in the property of Euclidean space that, as the number of dimensions increases, the volume of a hypersphere becomes concentrated more and more in the boundary region near the surface. Rechenberg's adaptation rule is founded on the relation between optimal variance and probability of success derived from two essentially dierent models of the objective function. The adaptation rule which is thereby formulated makes the frequency and size of the  increments respectively dependent on the number of variables and independent of the structure of the objective function. This will be discussed in more detail in Chapter 5, Section 5.1. Convergence proofs for the sequential random strategy have been given by Matyas (1965, 1967) and Rechenberg (1973) only for the case of constant variance 2. Gurin (1966) has proved convergence also for stochastically perturbed objective functions. The

98

Random Strategies

convergence rate is still reduced by perturbations (Gurin and Rastrigin, 1965), but not as much as in gradient methods. Global convergence can be achieved if the reference value of the objective function is measured more than once at the comparison point (Saridis and Gilbert, 1970). As soon as any attempt is made to achieve higher rates of convergence by adjusting the variances or step lengths, the chance of nding a global optimum diminishes. Then the random strategy itself becomes a path-oriented instead of a volume-oriented strategy. The probability of global convergence still always remains nite it may simply become very small, especially in the case of many dimensions. Apart from adjusting the step lengths, one can consider modifying the directions. Several proposals of this kind have been published: Satterthwaite (1959a following McArthur, 1961), Wheeling (1960), Smith and Rudd (1964 following Dickinson, 1964), Matyas (1965, 1967), Bekey et al. (1966), Stewart, Kavanaugh, and Brocker (1967), De Graag (1970), and Lawrence and Emad (1973). They are all heuristic in nature. In the simplest case of a directed random search, a successful random direction is maintained until a failure occurs (Satterthwaite). Bekey, Lawrence, and Rastrigin actually make use of each random direction. If the rst step leads to a failure, they use the opposite direction (positive and negative absolute biasing). Smith and Rudd store the two currently best points from a larger series of samples and obtain from their separation a step length for continuing the optimization. Wheeling's history vector method adds to each random increment a deterministic portion, derived from experience. This additional vector is initially zero. It is increased at each success by a fraction of the increment vector, and correspondingly decreased at each failure. Such a learning and forgetting process also forms the basis of the algorithms of De Graag and Matyas. The latter has received the most attention, in spite of the fact that it gives no precise guidance on how to choose the variances. Schrack and Borowski (1972), who apply their own step length rule in Matyas' strategy, were able to show by numerical tests that the simple algorithm of Schumer and Steiglitz, without direction orientation, is at least as good as Matyas' for unperturbed as well as perturbed measurements of the objective function. A quite dierent kind of method, due to Kjellstrom (1965), in which the random search takes place in varying three dimensional subspaces of the space IRn, shows itself here to be very much worse. Another method that sets out to accept only especially favorable directions is the threshold strategy of Stewart, Kavanaugh and Brocker (1967), in which only those random changes are accepted that result in a specied minimum improvement in the objective function value. A more recent version of the same idea has been given by Dueck and Scheuer (1990). The simultaneous adjustment of step lengths and directions has seldom been attempted. The suggestions of Favreau and Franks (1958) and Matyas (1965, 1967) remain too imprecise to be practicable. Gaidukov (1966 see also Hupfer, 1970) and Furst, Muller, and Nollau (1968) provide more exact information for this purpose, based on either the concepts of Rastrigin or Matyas. Modication of the expectation values and variances of the random vectors is made according to the success or failure of iterations. No applications of the strategy are known, however, so that for the time being the observation of Schrack and Borowski (1972) still stands, namely that a careful choice of the step lengths is the most important prerequisite for the rapid convergence of a random method. A method devised by Rastrigin (1965a,b, 1968) and developed further by Heydt (1970)

Random Strategies

99

works entirely with a restricted choice of directions. With a xed step length, a direction can be randomly selected only from within an n-dimensional hypercone. The angle subtended by the cone and its height (and thus the overall step length) are controlled in an adaptive way. For a spherical objective function, e.g., the model functions F2 (hypercone), F3 (hypersphere), or F4 (something intermediate between hypersphere and hypercube), there is no improvement in the convergence behavior. Advantages can only be gained if the search has to follow a particular direction for a long time along a narrow valley. Sudden changes in direction present a problem, however, which leads Heydt to consider substituting for the cone conguration a hyper-parabolic or hyper-hyperbolic distribution, with which at least small step lengths would retain sucient freedom of direction. In every case the striving for rapid convergence is directly opposed to the reliability of global convergence. This has led Jarvis (1968, 1970) to investigate a combination of the method of Matyas (1965, 1967) with that of McMurtry and Fu (1966). Numerical tests by Cockrell (1969, 1970 see also Fu and Cockrell, 1970) show that even here the basic strategy of Matyas (1965) or Schumer and Steiglitz (1967) is clearly the better alternative. It oers high convergence rates besides a fair chance of locating global optima, at least for a small number of variables. In the case of many dimensions, every attempt to reach global reliability is thwarted by the excessive cost. This leaves the globally convergent stochastic approximation method of Vaysbord and Yudin (1968) far behind the rest of the eld. Furthermore, the sequential or creeping random search is the least susceptible if perturbations act on the objective function. Users of random strategies always draw attention to their simplicity, !exibility and resistance to perturbations. These properties are especially important if one wishes to construct automatic optimalizers (e.g., Feldbaum, 1958 Herschel, 1961 Medvedev and Ruban, 1967 Krasnushkin, 1970). Rastrigin actually built the rst optimalizer with a random search strategy, which was designed for automatic frequency control of an electric motor. Mitchell (1964) describes an extreme value controller that consists of an analogue computer with a permanently wired-in digital part. The digital part serves for storage and !ow control, while the analogue part evaluates the objective function. The development of hybrid analogue computers, in which the computational inaccuracy is determined by the system, has helped to bring random methods, especially of the sequential type, into more general use. For examples of applications besides those of the authors mentioned above, the following publications can be referred to: Meissinger (1964), Meissinger and Bekey (1966), Kavanaugh, Stewart, and Brocker (1968), Korn and Kosako (1970), Johannsen (1970, 1973), and Chatterji and Chatterjee (1971). Hybrid computers can be applied to best advantage for problems of optimal control and parameter identication, because they are able to carry out integrations and dierentiations more rapidly than digital computers. Mutseniyeks and Rastrigin (1964) have devised a special algorithm for the dynamic control problem of keeping an optimum. Instead of the variable position vector x, a velocity vector with components @xi=@t is varied. A randomly chosen combination is retained as long as the objective function is decreasing in value (for minimization @F=@t < 0). As soon as it begins to increase again, a new velocity vector is chosen at random. It is always striking, if one observes living beings, how well adapted they are in shape, function, and lifestyle . In many cases, biological structures, processes, and systems even

100

Random Strategies

surpass the capabilities of highly developed technical systems. Recognition of this has for years led many authors to suspect that nature is in possession of optimal solutions to her problems. In some cases the optimality of biological subsystems can even be demonstrated mathematically, for example for the ratios of diameters in branching arteries (Cohn, 1954), for the hematocrit value (the volume fraction of solid particles in the blood Lew, 1972), and the position of branch points in a level system of blood vessels (Kamiya and Togawa, 1972 see also Grassmann, 1967, 1968 Rosen, 1967 Rein and Schneider, 1971). According to the theory of the descent of the species, all organisms that exist today are the (intermediate) result of a long process of development: evolution. Based on the multitude of nds of transitional species that have since become extinct, paleontology is providing a gradually more complete picture of this development. Leaving aside supernatural explanations, one must assume that the development of optimal or at least very good structures is a property of evolution, i.e., evolution is, or possesses, an optimization (or better, meliorization) strategy. In evolution, the mechanism of variation is the occurrence of random exchanges, even \errors," in the transfer of genetic information from one generation to the next. The selection criterion favors the better suited individuals in the so-called struggle for existence. The similarity of variation and selection to the iteration rules of direct optimization methods is, in fact, striking. This analogy is most often drawn for random strategies, since mutations can best be interpreted as random changes. Thus Ashby (1960) regards as mutations the stochastic parameter variations in his blind homeostatic process. For many variables, however, the pure or blind random search requires so many trials that it offers no acceptable explanation of the capabilities of natural structures, processes, and systems. With the highest possible physical rate of transfer of information, as given by Bremermann (1962 see also Ashby, 1965, 1968) of 1047 bits per second and gram of computer mass, the mass of the earth and the extent of its lifetime up to now would not be sucient to solve even simple combinatorial problems by complete enumeration or a blind random search, never mind to determine the optimal conguration of the 104 to 105 genes with their information content of around 1010 bits (Bremermann, 1963). Evolution must rather be considered as a sequential process that exploits the information from preceding successes and failures in order to follow a trajectory, although not a completely deterministic one, in the n-dimensional parameter space. Brooks (1958) and Favreau and Franks (1958) are therefore right to compare their creeping random search with biological evolution. Yet it is also certainly a very much simplied imitation of the natural process of development. In the 1960s, two proposals that consciously think of higher evolution principles as optimization rules to be simulated are due to Rechenberg (1964, 1973) and Bremermann (1962, 1963, 1967, 1968a,b,c, 1970, 1971, 1973a,b see also Bremermann, Rogson, and Sala, 1965, 1966 Bremermann and Lam, 1970). Bremermann reasons from the (nowadays!) low mutation rates observed in nature that only one component of the variable vector should be varied at a time. He then encounters with this scheme the same diculties as arise in the coordinate method. On the basis of his failure with the mutation-selection scheme, for example on linear programming problems, he comes to the conclusion that ecological niches are actually only stagnation points in development, and they do not represent optimal states of adaptation. None of his many attempts to invoke

Random Strategies

101

the principles of population, sexual inheritance, recombination, dominance, and recessiveness to improve the convergence behavior yield the hoped for breakthrough. He thus eventually resigns himself to a largely deterministic strategy. In the linear programming problem, he chooses from the starting point several random directions and follows these in turn up to the boundary of the feasible region. The best states on the individual bounding hyperplanes are used to determine a new starting point by taking the arithmetic mean of the component parameters. Because of the convexity of the allowed region, the new starting point is always within it. The simultaneous choice of several search directions is supposed to be the analogue of the population principle and the construction of the average the analogue of recombination in sexual propagation. To tackle the problem of nding the minimum or maximum of an unconstrained, non-linear function, Bremermann even applies a ve point Lagrangian interpolation to determine relative extrema in the random directions. Rechenberg's evolution strategy changes all the components of the variable vector at each mutation. In his case, the low mutation rate for many dimensions is expressed by choosing small values for the step lengths, or the spread in the random changes. On the basis of theoretical work with two model functions he nds that the standard deviations of the random components are set optimally when they are inversely proportional to the number of parameters. His two membered evolution strategy resembles the scheme of Schumer and Steiglitz (1968), which is acknowledged to be particularly good, except that a (0 2) normally distributed random quantity replaces the xed step length s. He has also added to it a step length modication rule, again derived from theory, which makes this look a very promising search method. It is rened in Chapter 5, Section 5.1 to meet the requirements of numerical optimization with digital computers. A multimembered strategy is treated in Section 5.2, which follows the same basic concept however, by imitating the principles of population and recombination, it can operate without external control of the step lengths. Incorporating more than one descendant at a time and forgetting \parental wisdom" at the end of each iteration loop has provoked erce objections against a more natural evolution strategy. Box (1957) also considers that his EVOP (evolutionary operation) strategy resembles the biological mutation-selection process. He regards the vertices of his pattern of trial points, of which the best becomes the center of the next pattern, as individuals of a population, of which only the best \survives." The \ospring" are, however, generated by purely deterministic rules. Random decisions, as used by Satterthwaite (1959a after Lowe, 1964) in his REVOP (random evolutionary operation) variant, are actually explicitly rejected by Box (see Youden et al., 1959 Satterthwaite, 1959b Budne, 1959 Anscombe, 1959). From a biological or cybernetic point of view, Pask (1962, 1971), Schmalhausen (1964), Berg and Timofejew-Ressowski (1964), Dobzhansky (1965), Moran (1967), and Kussul and Luk (1971) among others have examined the analogy between optimization and evolution. The fact that no practical algorithms have come out of this is no doubt because the evolutionary processes are described only verbally. Although they sometimes even include their more subtle eects, they have so far not produced a really quantitative, predictive theory. Exceptions, such as the work of Eigen (1971 see also Schuster, 1972), Merzenich

102

Random Strategies

(1972), and Papentin (1972) are so dierent in emphasis that they are not applicable to the kind of problems considered here. The ways in which a process of mathematization can be implemented in theoretical biology are documented in for example the books by Waddington (1968) and Locker (1973), which contain a number of contributions of interest from the optimization point of view, as well as many articles in the journal Mathematical Biosciences, which has been published by R. W. Bellman since 1967, and some papers from two Berkeley symposia (LeCam and Neyman, 1967 LeCam, Neyman, and Scott, 1972). Whereas many modern books on biology, such as Riedl (1976) and Roughgarden (1979), still give mainly verbal explanations of organic evolution, in general, this is no longer the case. Physicists like Ebeling and Feistel (see Feistel and Ebeling, 1989) and biologists like Maynard Smith (1982, 1989) meanwhile have contributed mathematical models. The following two paragraphs thus no longer represent the actual situation, but before we add some new aspects they will be presented, nevertheless, to characterize the situation as perceived by the author in the early 1970s (Schwefel, 1975a): Relationships have been seen between random strategies and biological evolution on the one hand and the psychology of recognition processes on the other, for example, by Campbell (1960) and Khovanov (1967). The imitation of such processes{the catch phrase is articial intelligence{always leads to the problem of choosing or designing a suitable search algorithm, which should rather be heuristic than strictly deterministic. Their simplicity, reliability (even in extreme, unfamiliar situations), and !exibility give the random strategies a special r^ole in this eld. The topic will not be discussed more fully here, except to mention some publications that explicitly deal with the relationship to optimization strategies: Friedberg (1958), Friedberg, Dunham, and North (1959), Minsky (1961), Samuel (1963), J. L. Barnes (1965), Vagin and Rudelson (1968), Thom (1969), Minot (1969), Ivakhnenko (1970), Michie (1971), and Slagle (1972). A particularly impressive example is given by the work of Fogel, Owens, and Walsh (1965, 1966a,b), in which imitation of the biological evolutionary principles of mutation and selection gives a (mathematical) automaton the ability to recognize prescribed sequences of numbers. It may be that in order to match the capabilities of the human brain{and to understand them{there must be a move away from the digital methods of present serial computers to quite dierent kinds of switching elements and coupling principles. Such concepts, as pursued in neurocybernetics and neurobionics, are described, for example, by Brajnes and Svecinskij (1971). The development of the perceptron by Rosenblatt (1958) can be seen as a rst step in this direction. Two research teams that have emphasized the adaptive capacity of evolutionary procedures and who have shown interesting computer simulations are Allen and McGlade (1986), and Galar, Kwasnicka, and Kwasnicki (see Galar, Kwasnicka, and Kwasnicki, 1980 Galar, 1994). In terms of the optimization tasks looked at throughout this book, one might call their point of view dynamic or on-line optimization, including optimum holding against environmental changes. As Schwefel and Kursawe (1992) have pointed

Random Strategies

103

out, a limited life span of all individuals is an important ingredient in such cases (principle of forgetting). Two others who have tried to explain brain processes on an evolutionary, at least selectionist, basis are Edelman (1987) and Conrad (1988). Though their approach has not yet been embraced by the main stream of neural network research, this might happen in the near future (e.g., Banzhaf and Schmutz, 1992). An even more general paradigm shift in the eld of articial intelligence (AI) has emerged under the label articial life (AL see Langton, 1989, 1994a,b Langton et al., 1992 Varela and Bourgine, 1992). Whereas Lindenmayer (see Prusinkiewicz and Lindenmayer, 1990) demonstrates the possibility of (re-)creating plant forms by means of rather simple computer algorithms, the AL community tries to imitate animal behavior computationally. In most cases the goal is to design \intelligent" robots, sometimes called knowbots or animats (Meyer and Wilson, 1991 Meyer, 1992 Meyer, Roitblat, and Wilson, 1993). The attraction of even simple evolutionary models (re-)producing fairly complex behavior of multi-individual systems simulated on computers is already spreading across the narrow bounds of computer science as such. New ideas are emerging from evolutionary computation, not only towards the organization of software development (Huberman, 1988), but also into the eld of economics (e.g., Witt, 1992 Nissen, 1993, 1994) and even beyond (Schwefel, 1988 Haefner, 1992). It may be questionable whether worthwhile conclusions from the new ndings can reach as far as that, but ecology at least should be a eld that could benet from a consequent use of evolutionary thinking (see Wol, Soeder, and Drepper, 1988). Computers have opened a third way of systems analysis aside from the classical mathematical/analytical and experimental/empirical main roads: i.e., numerical and/or symbolical simulation experiments. There is some hope that we may learn this way quickly enough so that we can maintain life on earth before we more or less unconsciously destroy it. Real evolution always had to deal with unpredictable environmental changes, not only those resulting from exogenous in!uences, but also self-induced endogenous ones. The landscape is some kind of n-dimensional trampoline, and every good imitation of organic evolution, whether it be called adaptive or meliorizing, must be able to work properly under such hard conditions. The multimembered evolution strategy (see Chap. 5, Sect. 5.2) with limited life span of the individuals fullls that requirement to some extent.

104

Random Strategies

Chapter 5 Evolution Strategies for Numerical Optimization The task of mimicking biological structures and processes with the object of solving technical problems is as old as engineering itself. Mimicry itself, as a natural \strategy", is even older than mankind. The legend of Daedalus and Icarus bears early witness to such human endeavor. A sign of its scientic coming of age is the formation of the distinct branch of science known as bionics (e.g., Hertel, 1963 Gerardin, 1968 Beier and Gla%, 1968 Nachtigall, 1971 Heynert, 1972 Zerbst, 1987), which is concerned with the recognition of existing biological solutions to problems that also happen to arise in engineering, and with the adequate emulation of these examples. It is always thereby supposed that evolution has found particularly good, perhaps even optimal solutions. This assumption has often proved to be correct, or at any rate useful. Only a few attempts to imitate the actual methods of natural development are known (Ashby, 1960 Bremermann, 1962{1973 Rechenberg, 1964, 1973 Fogel, Owens, and Walsh, 1965, 1966a,b Holland, 1975 see also Chap. 4) since they are curiously regarded a priori as being especially bad, meaning costly. Rechenberg proposed the hypothesis \that the method of organic evolution represents an optimal strategy for the adaptation of living things to their environment," and he concludes \it should therefore be worthwhile to take over the principles of biological evolution for the optimization of technical systems."

5.1 The Two Membered Evolution Strategy Rechenberg's two membered evolution scheme, suggested in similar form by other authors as a random strategy (see Chap. 4) will be expressed in this chapter as an algorithm for solving non-discrete, non-stochastic, parameter optimization problems. As in Chapter 3, the problem is F (x) ! min where x 2 IRn. In the constrained case x has to be in an allowed region G IRn, where

G = fx 2 IRn j Gj (x) 0  j = 1(1)n  Gj restriction functionsg 105

106

Evolution Strategies for Numerical Optimization

In this, as in all direct search methods, it is not possible to deal with constraints in the form of equalities.

5.1.1 The Basic Algorithm

The two membered scheme is the minimal concept for an imitation of organic evolution. The two principles of mutation and selection, which Darwin (1859) recognized to be most important, are taken as rules for variation of the parameters and for ltering during the iteration sequence respectively. In the language of biology, the rules are as follows: Step 0: (Initialization) A given population consists of two individuals, one parent and one descendant. They are each identied by their genotype according to a set of n genes. Only the parental genotype has to be specied as starting point. Step 1: (Mutation) The parent E (g) of the generation g produces a descendant N (g), whose genotype is slightly dierent from that of the parent. The deviations refer to the individual genes and are random and independent of each other. Step 2: (Selection) Because of their dierent genotypes, the two individuals have a dierent capacity for survival (in the same environment). Only one of them can produce further descendants in the next generation, namely the one which represents the higher survival value. It becomes the parent E (g+1) of the generation g + 1. Thus the simplest possible assumptions are made:  The population size remains constant  An individual has in principle an innitely long life span and capacity for producing descendants (asexually)  No dierence exists between genotype (encoding) and phenotype (appearance), or that one is unambiguously and reproducibly associated with the other  Only point mutations occur, independently of each other at all single parameter locations  The environment and thus the criterion of survival is constant over time This minimal concept takes no account of the evolutionary factors familiar to the modern synthetic evolution theory (e.g., Stebbins, 1968 Cizek and Hodanova, 1971 Osche, 1972), such as chromosome mutations, bisexuality, recombination, diploidy, dominance and recessiveness, population size, niching, isolation, migration, etc. Even the concepts of mutation and selection are not applied here with their full biological meaning. Natural selection does not simply mean the struggle between just two individuals in which the

The Two Membered Evolution Strategy

107

better survives, but far more accurately that an individual with more favorable properties produces on average more descendants than one less well adapted to the environment. Neither does the present work go more deeply into the connections between cause and eect in the transmission of inherited information, of which so much has been revealed by molecular biology. Mutation is used in the widest biological sense as a synonym for all types of alteration of the substance of inheritance. In his book Evolutionsstrategie, Rechenberg (1973) examines in more detail the analogy between natural evolution and technical optimization. He compares in particular the biological with the technical parameter space, and interprets mutations as steps in the nucleotide space. Expressed in mathematical language, the rules are as follows: Step 0: (Initialization) There should be storage allocated in a (digital) computer for two points of an n-dimensional Euclidean space. Each point is characterized by a position vector consisting of a set of n components. Step 1: (Variation) Starting from point E (g), with position vector x(Eg), in iteration g, a second g) of point N (g), with position vector x(Ng), is generated, the components x(Ni g) which dier only slightly from the x(Ei . The dierences come about by the addition of (pseudo) random numbers zi(g), which are mutually independent. Step 2: (Filtering) The two points or vectors are associated with dierent values of an objective function F (x). Only one of them serves as a starting point for the new variation in the next iteration g + 1: namely the one with the better (for minimization, smaller) value of the objective function. Taking account of constraints in the form of a barrier penalty function, this algorithm can be formalized as follows: Step 0: (Initialization) (0) (0) T Dene x(0) E = fxEi  i = 1(1)ng , such that Gj (xE ) 0 for all j = 1(1)m. Set g = 0. Step 1: (Mutation) Construct x(Ng) = x(Eg) + z(g) with components g) = x(g) + z (g) for all i = 1(1)n. x(Ni Ei i Step 2: (Selection) Decide ( (g) (g) (g ) (g ) x(Eg+1) = xN(g) if F (xN )
F (xE ) and Gj (xN ) 0 for all j = 1(1)m xE  otherwise: Increase g g + 1 and go to step 1 as long as the termination criterion does not hold.

108

Evolution Strategies for Numerical Optimization

The question remains of how to choose the random vectors z(g). This choice has the r^ole of mutation. Mutations are understood nowadays to be random, purposeless events, which furthermore only occur very rarely. If one interprets them, as is done here, as a sum of many individual events, it is natural to choose a probability distribution according to which small changes occur frequently, but large ones only rarely (the central limit theorem of statistics). For discrete variations one can use a binomial distribution, for example, for continuous variations a Gaussian or normal distribution. Two requirements then arise together by analogy with natural evolution:  That the expectation value i for a component zi has the value zero  That the variance i2, the average squared deviation from the mean, is small The probability density function for normally distributed random events is (e.g., Heinhold and Gaede, 1972):  2! ( z ;  ) 1 i i exp ; 2 2 w(zi) = p (5.1) 2 i i If i = 0, one obtains a so-called (0 i2) normal distribution. There are still however a total of n free parameters fi i = 1(1)ng with which to specify the standard deviations of the individual random components. By analogy with other, deterministic search strategies, the i can be called step lengths, in the sense that they represent average values of the lengths of the random steps. For the occurrence of a particular random vector z = fzi i = 1(1)ng, with the independent (0 i2) distributed components zi, the probability density function is  X n n  z 2 ! Y 1 1 i exp ; 2 w(z1 z2 : : : zn) = w(zi) = (5.2) n n Q  i 2 i=1 i =1 (2) i i=1

or more compactly, if i =  for all i = 1(1)n, !n  T !  1 (5.3) exp ;2zz2 w(z) = p 2  q p For the length of the overall random vector S = Pni=1 zi2 a  2 distribution is obtained. q The1  22 distribution with n degrees of freedom approximates, for large n, to value for the total length a ( n ; 2  2 ) normal distribution. Thus the expectation p of the random vector2 for many variables is E (S ) =  n, the variance is D2 (S ) = E ((S ; E (S ))2) = 2 , and the coecient of variation is

D(S ) = p1 E (S ) 2n This means that the most probable value for the length of the random vector at constant  increases as the square root of the number of variables and the relative width of variation decreases with the reciprocal square root of parameters.

The Two Membered Evolution Strategy x

2

109

Line of equal probability density

Contours F (x) = const.

(g) N (g) z (g) x N (g) xE

Opt.

(g) (g+1) E =E

(g+1) z

E : Parent

(g+1) (g+2) N =E

N : Descendant (g) : Generation index

(g+2) xE x

1

Figure 5.1: Two membered evolution strategy

The geometric locus of equally likely changes in variation of the variables can be derived immediately from the probability density function, Equation (5.2). It is an ndimensional hyperellipsoid (n-fold variance ellipse) with the equation n  z 2 X i = const:  i=1 i referred to its center, which is the starting point x(Eg). In the multidimensional case, the random changes can be regarded as a vector ending on the surface of a hyperellipsoid with the semi-axes i or if i =  for all i = 1(1)n, in the language of two dimensions they are distributed circumferentially. Figure 5.1 serves to illustrate two iteration steps of the evolution strategy on a two dimensional contour diagram. Whereas in other, fully deterministic search strategies both the direction and length of the search step are determined in the procedure in a xed manner, or on the basis of previously gathered information and plausible assumptions about the topology of the objective function, in the evolution strategy the direction is purely random and the step length{except for a small number of variables{is practically xed. This should be emphasized again to distinguish this random method from Monte-Carlo procedures, in which the selected trial point is always fully independent of the previous choice and its outcome. Darwin (1874) himself emphasized that the evolution of living things is not a purely random process. Yet against his theory of descendancy, a polemic is still waged in which the impossibility is demonstrated that life could arise by a purely random process (e.g., Jordan, 1970). Even at the level of the simplest imitation of organic evolution, a suitable choice of the step lengths or variances turns out to be of fundamental signicance.

110

Evolution Strategies for Numerical Optimization

5.1.2 The Step Length Control

In experimental optimization, the appropriate step lengths can frequently be predicted. The values of the variables usually have to be determined exactly at only a few points. Thus constant values of the variances are often all that is required to complete an extreme value search. It is a matter of fact that in most experimental applications of the simple evolution strategy xed (and discrete) distributions of mutations have been used. By contrast, in mathematically formulated problems that are to be solved on a digital computer, the variables often run over much of the number range of the computer, which corresponds to many powers of 10. In a numerical optimum search the step lengths must be continuously modied if the algorithm is to be ecient{a problem reminiscent of steering safely between Scylla and Charybdis for if the step length is too small the search takes an unnecessarily large number of iterations if it is too large, on the other hand, the optimum can only be crudely approached and the search can even get stuck far from the optimum, for example, if the route to the minimum passes along a narrow valley. Thus in all optimization strategies the step length control is the most important part of the algorithm after the recursion formula, and it is furthermore closely linked to the convergence behavior. The corresponding remarks hold for the evolution strategy, with the following dierence: In place of a predetermined step length for a parameter of the objective function there is the variance of the random changes in this parameter, and instead of the statement that an improvement will or will not be made in a given direction with a specied step length, there can only be a statement of probability of the success or failure for a chosen variance. In his theoretical investigations of the two membered evolution strategy, Rechenberg discovered using two basically dierent model objective functions (sphere model = Problem 1.1, corridor model = Problem 3.8 of the problem catalogue see Appendix A) that the maximal rate of convergence corresponds to a particular value for the probability of a success, i.e., an improvement in the objective function value. He was thus led to formulate the following rule for controlling the size of the random changes: The 1=5 success rule: From time to time during the optimum search obtain the frequency of successes, i.e., the ratio of the number of successes to the total number of trials (mutations). If the ratio is greater than 1=5, increase the variance, if it is less than 1=5, decrease the variance.

In many problems this rule proves to be extremely eective in maintaining approximately the highest possible rate of progress towards the optimum. While in the rightangled corridor model the variances are adjusted once and for all in accordance with this rule and subsequently remain constant, in the sphere model they must steadily become smaller. The question then arises as to how often the success criterion should be tested and by what factor the variances are most eectively reduced or increased. This question will be answered with reference to the sphere model introduced by Rechenberg, since this is the simplest non-linear model objective function and requires

The Two Membered Evolution Strategy

111

the greatest and most frequent changes in the step lengths. The following results of Rechenberg's theory can be used here: For the maximum rate of progress 'max = k1 nr  k1 ' 0:2025 (5.4) with a common variance 2, which is always optimal given by opt = k2 nr  k2 ' 1:224 (5.5) for all components zi of the random vector z. In these expressions r is the current distance from the goal (optimum) and n is the number of variables. The rate of progress is dened as the expectation value of the radial dierence covered per trial (mutation), as illustrated in Figure 5.2. '(g) = r(g) ; r(g+1) (5.6) From Equations (5.4) to (5.6) one obtains a relation for the changes in the variance after a generation (iteration, or mutation) under the condition of maximum convergence rate x

2

Line of constant probability density (g)

(g)

E

ϕ

r

(g)

(g+1) E

r

(g+1) x

1 Contours 2 2 F(x) = x + x = const. 1

Figure 5.2: The rate of progress for the sphere model

2

112

or after n generations

Evolution Strategies for Numerical Optimization (g+1) opt r(g+1) = 1 ; k1 = (g) r(g) n opt

!n (g+n)  opt k 1 (g) = 1 ; n opt If n is large compared to one, and the formulae derived by Rechenberg are only valid under this assumption, the step length factor tends to a constant:  !n k 1 ;k1 ' 0:817 ' 1 lim 1 ; = e n!1 n 1:224 The same result is obtained by considering the rate of progress as a dierential quotient ' = dr=dg, in which g represents the iteration number. This matches the limiting case of very many variables because, according to Equation (5.4) the rate of progress is inversely proportional to the number of variables. The fact that the rate of progress ' near its maximum is quite insensitive to small changes in the variances, together with the fact that the probability of success can only be determined from an average over several mutations, leads to the following more precise formulation of the 1=5 success rule for numerical optimization: After every n mutations, check how many successes have occurred over the preceding 10 n mutations. If this number is less than 2 n, multiply the step lengths by the factor 0:85 divide them by 0:85 if more than 2 n successes occurred. The 1=5 success rule enables the step lengths or variances of the random variations to be controlled. One might do even better by looking for a control mechanism with additional dierential and integral coecients to avoid the oscillatory behavior of a mere proportional feedback. However, the probability of success unfortunately gives no indication of how appropriate are the ratios of the variances i2 to each other. The step lengths can only be all reduced together, or all increased. One would sometimes rather like to build in a scaling of the variables, i.e., to determine ratios of the step lengths to each other. This can be achieved by a suitable formulation of the objective function, in which new parameters are introduced in place of the original variables. The functional dependence can be freely chosen and in the simplest case it is given by multiplicative factors. In the formulation of the numerical procedure for the two membered evolution strategy (Appendix B, Sect. B.1) the possibility is therefore included of specifying an initial step length for each individual variable. The ratios of the variances to each other remain constant during the optimum search, unless specied lower bounds to the step lengths are not operating at the same time. All digital computers handle data only in the form of a nite number of units of information (bits). The number of signicant gures and the range of numbers is thereby limited. If a quantity is repeatedly divided by a factor greater than one, the stored value of

The Two Membered Evolution Strategy

113

the quantity eventually becomes zero after a nite number of divisions. Every subsequent multiplication leaves the value as zero. If it happens to one of the standard deviations i, the aected variable xi remains constant thereafter. The optimization continues only in a subspace of IRn . To guard against this it must be required that i > 0 for all i = 1(1)n. The random changes should furthermore be suciently large that at least the last stored place of a variable is altered. There are therefore two requirements: Lower limits for the \step lengths": i(g) "a for all i = 1(1)n and   (g) "b x(g)  for all i = 1(1)n where

i

i

) "a > 0 according to the computational accuracy 1 + "b > 1 It is thereby ensured that the random variations are always active and the region of the search stays spanned in all dimensions.

5.1.3 The Convergence Criterion

In experimental optimization it is usually decided heuristically when to terminate the series of trials: for example, when the trial results indicate that no further signicant improvement can be gained. One always has an overall view of how the experiment is running. In numerical optimization, if the calculations are made by computer, one must build into the program a rule saying when the iteration sequence is to be terminated. For this purpose objective, quantitative criteria are needed that refer to the data available at any time. Sometimes, although not always, one will be concerned to obtain a solution as exactly as possible, i.e., accurate to the last stored digit. This requirement can relate to the variables or to the objective function. Remember that the optimum may be a weak one. Towards the minimum, the step lengths and distances covered normally become smaller and smaller. A frequently used convergence criterion consists of ending the search when the changes in the variables become zero (in which case no further improvement in the objective function is made), or when the step lengths have become zero. As a rule one sets the lower bound not to zero but to a suciently small, nite value. This procedure has however one disadvantage that can be serious. Small step lengths occur not only if the minimum is nearby, but also if the search is moving through a narrow valley. The optimization may then be practically halted long before the extreme value being sought is reached. In Equations (5.4) and (5.5), r can equally well be thought of as the local radius of curvature. Neither ', the distance covered, nor , the step length, are a measure of the closeness to the optimum. Rather they convey information about the complexity of the minimum problem: the number of variables and the narrowness of valleys encountered. The requirement  > " or kx(g) ; x(g;1)k > " for the continuation of the search is thus no guarantee of sucient convergence.

114

Evolution Strategies for Numerical Optimization

Gradient methods, which seek a point with vanishing rst derivatives, frequently also apply this necessary condition for the existence of an extremum as a termination criterion. Alternatively the search can be continued until 4F = F (x(k;1)) ; F (x(k)), the change in the objective function value in one iteration, goes to zero or to below a prescribed limit. But this requirement can also be fullled far from the minimum if the valley in which the deepest point is sought happens to be very !at in shape. In this case the step length control of the two membered evolution strategy ensures that the variances become larger, and thus the function value dierences between two successful trials also on average become larger. This is guaranteed even if the function values are equal (within computational accuracy), since a change in the variables is always then registered as a success. One thus has only to take care that 4F is summed over a number of results in order to derive a termination criterion. Just as lower bounds are dened for the step lengths, an absolute and a relative bound can be specied here: Termination rule: End the search if

or

F (x(Eg;4g)) ; F (x(Eg))
"c 1 hF (x(g;4g)) ; F (x(g))i
jF (x(g))j E E E "d

where and

4g 20 n "c > 0 1 + "d > 1

)

according to the computational accuracy

The condition 4g 20 n is designed to ensure that in the extreme case the standard deviations are reduced or increased within the test period by at least the factor (0:85)20 ' (25)1 , in accordance with the 1=5 success rule. This will prevent the search being terminated only because the variances are forced to change suddenly. It is clear from Equation (5.4) that the more variables are involved in the problem, the slower is the rate of progress. Hence it does not make sense to test the convergence criterion very frequently. A recommended procedure is to make a test every 20 n mutations. Only one additional function value then needs to be stored. Another reason can be adduced for linking the termination of the search to the function value changes. While every success in an optimum search means, in the end, an economic prot, every iteration costs computer time and thus money. If the costs exceed the prot, the optimization may well provide useful information, but it is certainly not on the whole of any economic value. Thus someone who only wishes to optimize from an economic point of view can, by a suitable choice of values for the accuracy parameters, restrain the search process as soon as it starts running into a loss.

The Two Membered Evolution Strategy

115

5.1.4 The Treatment of Constraints

Inequality constraints Gj (x) 0 for all j = 1(1)m are quite acceptable. Sign conditions may be formulated in the same manner and do not receive any special treatment. In contrast to linear and non-linear programming, no sign conditions need to be set in order to keep within a bounded region. If a mutation falls in the forbidden region it is assessed as a worsening (in the sense of a lethal mutation) and the variation of the variables is not accepted. No particular penalty function, such as Rosenbrock chooses for his method of rotating coordinates, has been developed for the evolution strategy. The user is free to use the techniques for example of Carroll (1961), Fiacco and McCormick (1968), or Bandler and Charalambous (1974), to construct a suitable sequence of substitute objective functions and to solve the original constrained problem as a sequence of unconstrained problems. This, however, can be done outside the procedure. It is sometimes dicult to specify an allowed initial vector of the variables. If one were to wait until by chance a mutation satised all the constraints, it could take a very long time. Besides, during this search period the success checks could not be carried out as described above. It would nevertheless be desirable to apply the normal search algorithm eectively to nd an allowed state. Box (1965) has given in the description of his complex method a simple way of proceeding from a forbidden starting point. He constructs an auxiliary objective function from the sum of the constraint function values of the violated constraints: m X F~ (x) = Gj (x) j (x) where

j =1

(

1 j (x) = ; 0

if Gj (x) < 0 otherwise

(5.7)

Each decrease in the value of F~ (x) represents an approach to the feasible region. When eventually F~ (x) = 0, then x satises all the constraints and can serve as a starting vector for the optimization proper. This procedure can be taken over without modication for the evolution strategy.

5.1.5 Further Details of the Subroutine EVOL

In Appendix B, Section B.1 a complete FORTRAN listing is given of a subroutine corresponding to the two membered evolution scheme that has been described. Thus no detailed algorithm will be formulated here, but a few further programming details will be mentioned. In nearly all digital computers there are library subroutines for generating uniformly distributed pseudorandom numbers. They work, as a rule, according to the multiplicative or additive congruence method (see Johnk, 1969 Niederreiter, 1992 Press et al., 1992). From any two numbers taken at random from a uniform distribution in the range 0 1], by using the transformation rules of Box and Muller (1958) one can generate two independent,

116

Evolution Strategies for Numerical Optimization

normally distributed random numbers with the expectation values zero and the variances unity. The formulae are q Z10 = ;2 ln Y1 sin (2 Y2) and q (5.8) Z20 = ;2 ln Y1 cos (2 Y2)

where the Yi are the uniformly distributed and the Zi0 (0 1)-normally distributed random numbers respectively. To obtain a distribution with a variance dierent from unity, the Zi0 must simply be multiplied by the desired standard deviation i (the \step length"):

Zi = i Zi0

The transformation rules are contained in a function procedure separate from the actual subroutine. To make use of both Equations (5.8) a switch with two settings is dened, the condition of which must be preset in the subroutine once and for all. In spite of Neave's (1973) objection to the use of these rules with uniformly distributed random numbers that have been generated by a multiplicative congruence method, no signicant dierences could be observed in the behavior of the evolution strategy when other random generators were used. On the other hand the trapezium method of Ahrens and Dieter (1972) is considerably faster. Most algorithms for parameter optimization include a second termination rule, independent of the actual convergence criterion. They end the search after no more than a specied number of iterations, in order to avoid an innite series of iterations in case the convergence criterion should fail. Such a rule is eectively a bound on the computation time. The program libraries of computers usually contain a procedure with which the CPU time used by the program can be determined. Thus instead of giving a maximum number of iterations one could specify a maximum computation time as a termination criterion. In the present program the latter option is adopted. After every n iterations the elapsed CPU time is checked. As soon as the limit is reached the search ends and output of the results can be initiated from the main program. The 1=5 success rule assumes that there is always some combination of variances i > 0 with which, on average, at least one improvement can be expected within ve mutations. In Figure 5.3 two contour diagrams are shown for which the above condition cannot always be met. At some points the probability of a success cannot exceed 1=5 : for example, at points where the objective function has discontinuous rst partial derivatives or at the edge of the allowed region. Especially in the latter case, the selection principle progressively forces the sequence of iteration points closer up to the boundary and the step lengths are continuously reduced in size, without the optimum being approached with comparable accuracy. Even in the corridor model (Problem 3.8 of Appendix A, Sect. A.3) diculties can arise. In this case the rate of progress and probability of success depend on the current position relative to the edges of the corridor. Whereas the maximum probability of success in the middle of the corridor is 1=2, at the corners it is only 2;n . If one happens to be in the neighborhood of the edge of the corridor for several mutations, the probability of success

The Two Membered Evolution Strategy x 2

To the optimum

117 x2 Forbidden region

x 1 Circle : line of equal probability density Bold segment : fraction where success can be scored

Optimum x1

Figure 5.3: Failure of the 1/5 success rule

calculated by the above rule will be very dierent from that associated with the same step length if an average over the corridor cross section were taken. If now, on the basis of this low estimate of the success probability, the step length is further reduced, there is a corresponding decrease in the probability of escaping from the edge of the corridor. It would therefore be desirable in this special case to average the probability of success over a longer time period. Opposed to this, however, is the requirement from the sphere model that the step lengths should be adjusted to the topology as directly as possible. The present subroutine oers several means of dealing with the problem. For example, the lower bounds on the variances (variables EA, EB in the subprogram EVOL) can be chosen to be relatively large, or the number of mutations (the variable LS) after which the convergence criterion is tested can be altered by the user. The user has besides a free choice with regard to the required probability of success (variable LR) and the multiplier of the variance (variable SN). The rate of change of the step lengths, given by the factor 0:85 per n mutations, was xed on the basis of the sphere model. It is not ideal for all types of problems but rather in the nature of a lower bound. If it seems reasonable to operate with constant variances, the parameter in question should be set equal to one. An indication of a suitable choice for the initial step lengths (variable array SM) can be obtained from Equation (5.4). Since r increases as the root of the number of parameters, one is led to set pxni i(0) = 4 in which 4xi is a rough measure of the expected distance from the optimum. This does not actually give the optimal step length because r is a kind of local scale of curvature of the contours of the objective function. However, it does no harm to start with variances that are too large they will quickly be reduced to a suitable size by the 1=5 success rule. During this transition phase there is still a chance of escaping from the neighborhood of a merely local optimum but very little chance afterwards. The global convergence

118

Evolution Strategies for Numerical Optimization

property (see Rechenberg, 1973) of the evolution strategy can only be proved under the condition of constant step lengths. With the introduction of the success rule, it is lost, or to be more precise: the probability of nding the global minimum among several local minima decreases continuously as a local minimum is approached with continuous reduction in the step lengths. Rapid convergence and reliable global convergence behavior are two contradictory requirements. They cannot be reconciled if one has absolutely no knowledge of the topology of the objective function. The 1=5 success rule is aimed at high convergence rates. If several local optima are expected, it is thus advisable to keep the variances large and constant, or at least to start with large i(0) and perhaps to require a lower success probability than 1/5. This measure naturally costs extra computation time. Once one is sure of having located a point near the global extremum, the accuracy can be improved subsequently in a follow-up computation. For more sophisticated investigations of the global convergence see Born (1978), Rappl (1984), Scheel (1985), Back, Rudolph, and Schwefel (1993), and Beyer (1993).

5.2 A Multimembered Evolution Strategy While the simple, two membered evolution strategy is successful in application to many optimization problems, it is not a satisfactory method of solving certain types of problem. As we have seen, by following the 1=5 success rule, the step lengths can be permanently reduced in size without thereby improving the rate of progress. This phenomenon occurs frequently if constraints become active during the search, and greatly reduce the size of the success scoring region. A possible remedy would be to alter the probability distribution of the random steps in such a way as to keep the success probability suciently large. To do so the standard deviations i would have to be individually adjustable. The contour surfaces of equal probability could then be stretched or contracted along the coordinate axes into ellipsoids. Further possibilities for adjustment would arise if the random components were allowed to depend on each other. For an arbitrary quadratic problem the rate of convergence of the sphere model could even be achieved if the random changes of the individual variables were correlated so as to make the regression line of the random vector run parallel to the concentric ellipsoids F (x) = const:, which now lie at some angle in the space. To put this into practice, information about the topology of the objective function would have to be gathered and analyzed during the optimum search. This would start to turn the evolution strategy into something resembling one of the familiar deterministic optimization methods, as Marti (1980) and recently again Ostermeier (1992) have done this is contrary to the line pursued here, which is to apply biological evolution principles to the numerical solution of optimization problems. Following Rechenberg's hypothesis, construction of an improved strategy should therefore be attempted by taking into account further evolution principles.

5.2.1 The Basic Algorithm

When the ground rules of the two membered evolution strategy were formulated in the language of biology, reference was to one parent and one ospring the basic population

A Multimembered Evolution Strategy

119

thus consisted of two individuals. In order to reach a higher level of imitation of the evolutionary process, the number of individuals must be increased. This is precisely the concept behind the evolution strategy referred to in the following as multimembered. In his basic work (Rechenberg, 1973), Rechenberg already presented a scheme for a multimembered evolution. The one considered here is somewhat dierent. It turns out to be particularly useful with respect to the individual control of several step lengths to be described later. As yet, however, no detailed comparison of the two variants has been undertaken. It is useful to introduce at this point a nomenclature for the dierent evolution strategies. We shall call the number of parents of a generation , and the number of descendants , so that the selection takes place between  +  = 1+1 = 2 individuals in the two membered strategy. We thus characterize the simplest imitation of evolution in abbreviated notation as the (1+1) strategy. Since the multimembered evolution scheme described by Rechenberg allows a selection between  > 1 parents and  = 1 ospring it should be called the (+1) strategy. Accordingly a more general form, a (+) evolution strategy, should be formulated in such a way that a basic population of  parents of generation g produces  ospring. The process of selection only allows the  best of all  +  individuals to proceed as parents of the following generation, be they ospring of generation g or their parents. In this model it could happen that a parent, because of its vitality, is far superior to the other parents in the same generation, \lives" for a very long time, and continues to produce further ospring. This is at variance to the biological fact of a limited life span, or more precisely a limited capacity for reproduction. Aging phenomena do not, as far as is known, aect biological selection (see Savage, 1966 Osche, 1972). As a further conceptual model, therefore, let us introduce a population in which  parents produce  >  ospring but the  parents are not included in the selection. Rather the parents of the following generation should be selected only from the  ospring. To preserve a constant population size, we require that each time the  best of the  ospring become parents of the following generation. We will refer to this scheme in what follows as the ( , ) strategy. As for the (1+1) strategy in Section 5.1.1, the algorithm of the multimembered ( , ) strategy will rst be formulated in the language of biology. Step 0: (Initialization) A given population consists of  individuals. Each is characterized by its genotype consisting of n genes, which unambiguously determine the vitality, or tness for survival. Step 1: (Variation) Each individual parent produces = ospring on average, so that a total of  new individuals are available. The genotype of a descendant diers only slightly from that of its parents. The number of genes, however, remains to be n in the following, i.e., neither gene duplication nor gene deletion occurs. Step 2: (Filtering) Only the  best of the  ospring become parents of the following generation. In mathematical notation, taking constraints into account, the rules are as follows:

120

Evolution Strategies for Numerical Optimization

Step 0: (Initialization) (0) (0) (0) T Dene x(0) k = xEk = (xk1  : : : xkn ) for all k = 1(1): (0) x(0) k = xEk is the vector of the k th parent Ek , such that Gj (x(0) k ) 0 for all k = 1(1) and all j = 1(1)m: Set the generation counter g = 0: Step 1: (Mutation) Generate x(`g+1) = x(kg+1) + z(g + `) such that Gj (x(`g+1)) 0 j = 1(1)m ` = 1(1) where k 2 1 ] ( if ` = p   p integer e.g., k =  `(mod ) otherwise. x`(g+1) = x(Ng`+1) = (x(`g1+1) : : :  x(`ng+1))T is the vector of the `th ospring N` and z(g+`) is a normally distributed random vector with n components: Step 2: (Selection) Sort the x(`g+1) for all ` = 1(1) so that F (x(`g1 +1))
F (x(`g2 +1)) for all `1 = 1(1) `2 =  + 1(1) ( g +2) ( g +1) Assign xk = x`1  for all k `1 = 1(1): Increase the generation counter g g + 1: Go to step 1, unless some termination criterion is fullled. What happens in one generation for a (2 , 4) evolution strategy is shown schematically on the two dimensional contour diagram of a non-linear optimization problem in Figure 5.4.

5.2.2 The Rate of Progress of the (1 ,
) Evolution Strategy

In this section we attempt to obtain approximately the rate of progress of the multimembered, or ( , ) strategy{at least for  = 1. For this purpose the n-dimensional sphere and corridor models, as used by Rechenberg (1973), are employed for calculating the progress for the (1+1) strategy. In the two membered evolution strategy ' was the expectation value of the useful distance covered in each mutation. It is convenient here to dene the rate in terms of the number of generations.  ' = expectation value kx^ ; x'(g)k ; kx^ ; x'(g;1)k where x^ is the vector of the optimum and x'(g) is the average vector of the parents of generation g. From the chosen n-dimensional normal distribution of the random vector, which has expectation value zero and variance 2 for all independent vector components, the probability density for going from a point E with vector xE = (xE1 : : : xEn)T to another

A Multimembered Evolution Strategy x

2

121

Circles : lines of constant probability density

(g) (g+1) N=E 2 2

Opt.

: Parents k N : Offspring

E

(g) (g+1) N=E 3 1

(g) E 1 (g) N1

(g) : Generation index

(g) E 2

(g) N 4

x

1

Figure 5.4: Multimembered (2 , 4) evolution strategy

point N with vector xN = (xN1 : : : xNn)T is !n   ! n X 1 1 2 exp ; 2 2 (xEi ; xNi) w (E ! N ) = p 2  i=1

(5.9)

The distance kxE ; xN k between xE and xN is v u n tX(xEi ; xNi)2 kxE ; xN k = u i=1

But of this, only a part, s = f (xE  xN ), is useful in the sense of approaching the objective. To discover the total probability density for covering a useful distance s, an integration must be performed over the locus of points for which the useful distance is s, measured from the starting point xE . This locus is the surface of a nite region in n-dimensional space: Z Z  p(s) = w(E ! N ) dxN1 dxN2 : : : dxNn (5.10) f (xE  xN ) = s The result of the integration depends on the weighting function f (xE  xN ) and thus on the topology of the objective function F (x). So far only one random change has been considered. In the multimembered evolution strategy, however, the average over the  best of the  ospring must be taken, in which each of the ospring is to be associated with its own distance s`. We rst have to nd the probability density w (s0) for the  th best descendant of a generation to cover the useful distance s0. It is a combinatorial product of

122

Evolution Strategies for Numerical Optimization

 The probability density w(s` = s0) that a particular descendant N` gets exactly

s0 closer to the objective  The probability p(s`2 > s0) that a descendant N`2 advances further than s0  The probability p(s`3 < s0) that a descendant N`3 advances less than s0 Better results must be given by  ; 1 descendants and worse by  ;  . This results in a large number of combinations, since it is of no signicance which descendant is in which place.  ( ;X  +2 ( ;X  +3 ( X 0 0 0 w (s ) = w(s`1 = s )  p(s`2 > s )  p(s`3 > s0)  `2 =1 `3 =`2 +1 `1 =1 `2 = 6 `1 `3 = 6 `1 ( ( ;X  +4  X 0  p(s`4 > s )    p(s` > s0)  `4 =`3 +1 ` =` ;1 +1 `4 62f`1`2 g ` 62 f`1 `2 ::: ` ;2 g ) ))))  Y 0  p(s`+1 < s )    (5.11) 1

1

` +1 =1

` +1 62 f`1  `2 ::: ` g

As an average of the  best descendants one obtains  X 1 0 w(s ) =  w (s0)  =1 and hence the rate of progress Z1 '= s0 w(s0) ds0 s0 =su

(5.12)

(5.13)

The meaning of su will be described later. To evaluate ', besides  and , all components of the position vectors of all parents of the generation must be given, together with the values of  for producing each descendant. If ' is to become independent of a particular initial conguration, it is necessary to dene representative or average values of the relative positions of the parents, which are established during the optimization as a function of the topology. To do so would require setting up and solving an integral equation. This has not yet been achieved. To be able to say something nevertheless about the rate of convergence some simplifying assumptions will be made. All parents will be represented by a single position vector xk , and the standard deviations `i will be assumed equal for all components i = 1(1)n and for the descendants ` = 1(1). Equation (5.11) thereby simplies to  !  ; 1 0 w (s ) =   ; 1 w(s` = s0) p(s` < s0)]; p(s` > s0)];1 Since p(s` > s0) + p(s` < s0) = 1

A Multimembered Evolution Strategy and



we have

123

! ( ; 1) ! ;1 = ;1 ( ; 1) ! ( ;  ) !

w (s0) = ( ; 1) ! (! ;  ) ! w(s` = s0) p(s` < s0)]; 1 ; p(s` < s0)];1

(5.14)

Henceforth the number of parents is reduced to  = 1. One parent produces all  descendants. Of these, because of the assumption of constant population size, only the best survives. All the others are rejected. Accordingly we are now dealing with a (1 , ) strategy, for which Equation (5.12) reduces to w(s0) = w1(s0) =  w(s` = s0) p(s` < s0)];1 (5.15) where

w(s` = s0) = and

Z

Z



 p1 f (xE  xN ) = s 2 

!n



! n X 1 2 exp ; 2 2 (xEi ; xNi) dxN1 : : : dxNn i=1

p(s` < s0) =

Zs0 s` =;1

w(s` = s0) ds`

If we now make use of the corridor and sphere model objective functions, as chosen by Rechenberg in his work, we can directly take over some of his results in particular the integrations for the calculation of w(s` = s0) and p(s` < s0). The nal integration (Equation (5.13)) for determining ' turns out to be impossible to evaluate in closed form. To nd a suitable way around this let us take a closer look at Equation (5.13). It has the form of an equation for the mean value (expectation value) of the probability density w(s0) in the interval su
s0
1. The lower limit su of the range depends on whether, in cases when none of the ospring represent an improvement over the parent, the selection allows either the parent to survive (so-called \plus" version), or only the best of all ospring (socalled \comma" version), in which case the chance of deterioration is greater than zero. It will turn out later that the optimization can actually benet if setbacks are permitted. We therefore distinguish the two cases:  The parent is included in the selection process and can in theory survive an innite number of generations: su = 0 (1+) strategy  The parent is no longer considered in the selection: su = ;1 (1 , ) strategy In the second case the integral for p extends over the total interval in which the variable of integration s0 can lie. Now if the function w(s0) happens to be symmetrical and unimodal, the expectation value can be found in a dierent way. Namely, it would be equal to the value of s0 at which the probability density w(s0) reaches its maximum. For

124

Evolution Strategies for Numerical Optimization

a skew distribution this is not the case. Perhaps, however, the skewness is only slight, so that one can determine at least approximately the expectation value from the position of the maximum. Before treating the sphere and corridor models in this way, we will check the usefulness of the scheme with an even simpler objective function. 5.2.2.1 The Linear Model (Inclined Plane)

The simplest way the objective function can depend on the variables is linearly. Imagining the function to be a terrain in the (n + 1)-dimensional space, it appears as an inclined plane. In the two dimensional projection the contours are straight, parallel lines in this case. Without loss of generality one can orient the coordinate system so that the plane only slopes in the direction of one axis x1 and the starting point or parent under consideration lies at the origin (Fig. 5.5). The useful distance s` towards the objective that is covered by descendant N` of the parent E is just the part of the random vector z lying along the x1 axis. Since the components zi of z are independent, we have  02 ! 1 0 w(s` = s ) = p exp ; 2s2 2  and

"  0 !#  2! 1 1 s p exp ; 2 ` 2 ds` = 2 1 + erf ps p(s` < s0) = 2 s` =;1 2  Substituting these two results in Equation (5.15) we obtain the probability density for the best of  ospring of a parent covering the useful distance s0:  0 !#!;1  02 !  " 1 1 ; s 0 w(s ) =  p exp 2 2 2 1 + erf ps (5.16) 2  2 To obtain the position of the maximum we dierentiate with respect to s0 and set the result equal to zero. The associated value of s0 is then the sought for approximation '~ to the rate of progress '. From  @w(s0)  =! 0 @s0 s0 ='~ it follows that  !# p '~  '~2 ! " ' ~  = 1 + p exp 2 2 1 + erf p (5.17) 2 2 Figure 5.6 shows how the function '= ~ , which is just '~ for  = 1, depends on . For  = 1 the rate of progress is equal to zero, independent of the step length. This must be so because for only one descendant the probability of improvement is the same as that of worsening. As the number of descendants increases so does the rate of progress, Zs0

A Multimembered Evolution Strategy

125

x 2 N z E

Contours F (x) = const. To the minimum x E : Parent1 N : th offspring

s = z ,1

Figure 5.5: The inclined plane model function

sublinearly however, probably proportional to the logarithm of . To compare the above approximation '~ with the exact value ' the following integral must be evaluated:  02 !  "  0 !#!;1 Z1 0 s s 1 exp ; 22 2 1 + erf ps ds0 '= p 2   2  s0 =su For small values of  the integration can be performed by elementary methods, but not for general values of . The value of ' was therefore obtained by simulation on the computer rst for the case in which the parent survives if the best of the descendants is worse than the parent ('sur with su = 0) and secondly for the case in which the parent is no longer considered in the selection ('ext with su = ;1). The two results are shown in Figure 5.6 for comparison with the approximate solution '~. It is immediately striking that for only ve ospring the extinction of the parent has hardly any eect on the rate of progress, i.e., for  5 it is as good as certain that at least one of the descendants will be better than the parent. The greatest dierences between 'sur and 'ext naturally appear when  = 1. Whereas 'ext goes to zero, 'sur keeps a nite value. This can be determined exactly. Omitting here the details of the derivation, which is straightforward, the result is simply 'sur ( = 1) = p 2 The relationship to the (1+1) evolution scheme is thereby established. The dierences between the approximate theory ('~) and the simulation ('ext) indicate that the assumption of the symmetry of w(s0) is not correct. The discrepancy with regard to '= seems to tend to a constant value as  increases. While the approximate theory is shown by this

126

Evolution Strategies for Numerical Optimization

Rate of progress for σ = 1

ϕ

2

ϕ ext (λ) Simulation with “extinction” ϕ sur (λ) Simulation with “survival” ϕ (λ) (1, λ ) approximate theory (1+1) Theory

1

Number of offspring

0 1

5

10

15

20

25

λ

Figure 5.6: Rate of progress for the inclined plane model

comparison to be poor for making exact quantitative predictions, it nevertheless correctly reproduces the qualitative relation between the rate of progress and the number of descendants in a generation. The probability distributions w(s0) are illustrated in Figure 5.7 for ve dierent values of  2 f1 3 10 30 100g, according to Equation (5.16). For the inclined plane model the question of an optimal step length does not arise. The rate of progress increases linearly with the step length. Another question that does arise, however, is how to choose the optimal number of ospring per parent in a generation. The immediate answer is: the bigger  is, the faster the evolution advances. But in nature, since resources are limited (territory, food, etc.) it is not possible to increase the number of descendants arbitrarily. Likewise in applications of the strategy to solving problems on the digital computer, the requirements for computation time impose limits. The computers in common use today can only work in a serial rather than parallel way. Thus all the mutations must be produced one after the other, and the more descendants the longer the computation time. We should therefore turn our attention instead to nding the optimum value of '=. In the case where the parent survives if it is not bettered by any descendant, we have the trivial solution

opt = 1 The corresponding value for the (1 , ) strategy is, however, larger. With Equation (5.17)

A Multimembered Evolution Strategy

127

1.0

100

Probability density w(s’) for = 1 0.9 30 0.8

0.7

10

Parameter of offspring

0.6

= number

3 0.5

0.4 = 1

0.3

0.2

0.1

0 −4

−2

0

2

4 Useful distance s’

Figure 5.7: Probability distribution w(s ) 0

one obtains from the requirement

@  '~  ! =  @  =opt 0

the relation

 2 @ = '~2 opt = '~ @ '~  =opt and, by substituting it back in Equation (5.17), the result 0 13  !2 s  1 1 A5 opt = 1 + 2  exp 2  41 + erf @ q opt opt 2 opt

The value obtained iteratively is

opt ' 2:5 (as an integer: opt = 2 or 3) 5.2.2.2 The Sphere Model

We will now try to calculate the rate of progress for the simple spherically symmetrical model, which is of importance for considering the convergence rate properties of the strategy. The contours of the objective function F (x) are concentric hypersphere surfaces, given for example by n X F (x) = x2i = const: i=1

128

Evolution Strategies for Numerical Optimization x2 Contours 2

2

F(x) = x 1 + x 2 = const

r

2

N

a 2 a r

2

E x1

1

1

N

1

rE s >0 s <0 2

1

Figure 5.8: Hypersphere model function

Figure 5.8 illustrates the case of two variables. The solution is obtained in much the same way as for the inclined plane. We shall take over some of the steps and subsidiary results from the derivation of Rechenberg (1973) in the two membered evolution strategy. The normalized probability density for production of a descendant N` with position vector x` = (x`1 : : : x`n)T from parent E with position vector xE = (xE1 : : : xEn)T again corresponds to an n-dimensional normal distribution with expectation value  = 0 and variance 2 (the same for all vector components). Without aecting the generality of the result, the special position xE = (rE  0 : : : 0)T can be selected for the starting point E in relation to the coordinate system. With the notation v u n uX r` = t x2`i i=1

Equation (5.9) yields



w(E ! N` ) = p 1 2 

!n



 1 2 2 exp ; 2 2 (r` + rE ; 2 rE x`1)

A Multimembered Evolution Strategy

129

For the distance covered towards the objective, s` , the portion is now calculated that contributes to an improvement of the objective function, i.e., in this case the radial difference s` = rE ; r` (see Fig. 5.8). The locus of all points N` for which s` is the same is the surface of the n-dimensional hypersphere about the origin with radius r` = rE ; s` . Accordingly the total probability density that a mutation (index `) starting from point E will cover the distance s` is the n-fold line integral: Z Z  1 !n  1   2 2 p  w(s` ) = exp ; 2 2 r` + rE ; 2 rE x`1 dx`1 : : :dx`n 2   rE ; r` = s` By transforming to spherical coordinates one obtains a simple integral   2 2! !n n;2 1 Z2  rE r` cos   r + r 1  E ` n ; 1  exp ; 2 r` w(s` ) = p exp sinn;2  d 2 2  2  ; n;2 1

=0 The remaining integral can be expressed as a modied Bessel function: n 1; n  2 2!   2 2 r r E ` w(s` ) = 2 exp ; rE2+2r` I n2 ;1 rE2r` To simplify the notation we now introduce the following denitions: 2  = n2  a = rE2  v = rr` E We thereby obtain 2 w(s` ) = ra e; a2 v e; a 2v I;1(a v) with s` = rE (1 ; v) E In order to use Equation (5.15) to calculate the total probability that the best of  descendants will cover the distance s0 = max fs` j ` = 1(1)g = rE ; r0 `

the following quantities are still required: 2 w(s` = s0) = ra e; a2 u e; a 2u I;1(a u) E with r0 = u and s0 = r (1 ; u) E rE and p(s` < s0) = 1 ; p(s` > s0) s0 R = 1 ; s =r w(s` ) ds` ` E 2 Ru = 1 ; a e; a2 v e; a 2v I;1(a v) dv v=0

130

Evolution Strategies for Numerical Optimization

This nally gives the probability function for the useful distance s0 covered in one generation, expressed in units of u: 0 1;1 Zu a a  ; a u2 a a v2 0 ; ;  ; w(s ) = r e 2 u e 2 I;1(a u) @1 ; a e 2 v e 2 I;1(a v) dvA E v=0 Since the expectation value of this distribution is not readily obtainable, we shall determine its maximum to give an approximation '~. From the necessary condition  @w(s0)  =! 0 @s0 s0 ='~ with the more concise notation a y2 D(y) = a e; a2 y e; 2 I;1(a y) we obtain the relation 0 1;'=r 1  ~ E Z @D ( u )  = 1 + @u  D(1 ; '=r ~ E )];2 B D(v) dvC (5.18) @1 ; A u=1;'=r ~ E v=0 Except for the upper limit of integration, this is the same integral that made it so dicult to obtain the exact solution for the rate of progress in the (1+1) evolution strategy (see Rechenberg, 1973). Under the condition   1 and =a  1, which means for many variables and at a large enough distance from the optimum, Rechenberg obtained an estimate by expanding Debye's asymptotic series representation of the Bessel function (e.g., Jahnke-Emde-Losch, 1966) in powers of =a. Without giving here the individual steps in the derivation, the result is !# "   2 !# pa "  ( ; 1)2 ! Z1   1 + p exp ; ; exp ; (5.19) D(v) dv ' 2 1 ; erf p 2 a 2 a 2 a 2  2  v=0 It is clear from Equation (5.4) that the rate of progress of the (1+1) strategy for the two membered evolution varies inversely as the number of variables. Even if a higher convergence rate is expected from the multimembered scheme, with many descendants per parent, there will be no change in the relation to n, the number of parameters. In addition to the assumptions already made regarding  and =a, without further risk to the validity of the approximate theory we can assume that 1 ; '=r ~ E ' 1. Equation (5.19) can now also be applied here. For the partial dierential  @D(u)  @u u=1;'=r ~ E we obtain with the use of the Debye series again:  "  ! #  @D(u)  1 = D(1 ; '=r ~ E ) a exp a (1 ; '=r ~ E) @u u=1;'=r ~ E ) + 1 ; '=r ~ E ; a (1 ; '=r ~ E

A Multimembered Evolution Strategy

Figure 5.9: Rate of progress for the sphere model

131

132

Evolution Strategies for Numerical Optimization

If the result is substituted into Equation (5.18) a longer expression is obtained, of the form:  = (' ~  rE  n) In the expectation of an end result similar to Equation (5.4) and since a particular starting point rE is of no interest, we will introduce new variables: ' = '~r n and  = r n E E If '~ and  are now replaced by ' and  , taking the limit nlim !1 ('    rE  n) we nd that the quantities n and rE disappear from the parameter list of . ' and  can therefore be regarded as \universal" variables. We obtain  ! 2 !23 "  !# p '   '   4 5  = ('   ) = 1 +  p + p exp p + p 1 + erf p (5.20) 2 8 2 8 8 As in the case of the inclined plane considered previously, this equation cannot be simply solved for ' . Figure 5.9 shows the family of curves ' = ' (  ). For  ! 0, as expected, ' ! 0. For  = 1, the rate of progress is always negative. Since the parent in the (1 , ) strategy is not included in the selection after it has served to produce a descendant,  = 1 means that every mutation is adopted, whether better or worse. For the sphere model, except for  = 0, the region of success is always smaller than half of the variable space. With increasing  , the ratio becomes even worse ' is thus always
0, and more strongly negative the greater is  . For  2 the rate of progress increases at rst as a function of the variance, reaches a maximum, and then decreases continuously until it becomes negative. From this behavior one can see even more clearly than in the (1+1) strategy how important the correct choice of variance is for the optimization. In the (1 , ) strategy, the progress can turn retrograde if all the ospring are worse than the parent that produced them. Only with an immortal parent having an innite capacity for reproduction would progress be guaranteed or, at least, would retrogression be ruled out. We shall see later why the model with \extinction" is nevertheless advantageous. Except for small values of , the maximum rate of progress is almost the same in the \survival" and \extinction" cases. So if the optimal variance can be maintained throughout, leaving the parents out of the selection is not a disadvantage. The position of the maxima of ' with respect to  at a constant  is obtained by simple dierentiation and equating the partial derivative to zero. Dening popt = + and p'max = '+ 8 2 opt the equation is   p +('+ + +) exp(;+2) + (+ ; '+) 12 + ('+ + +)2 1 + erf(+) =! 0 (5.21)

A Multimembered Evolution Strategy 2.0

133

Maximal universal rate of progress ϕ (σ ) opt max

1.0 (1+1) - theory

Numer of offspring 1

5

10

15

20

25

30

λ

Figure 5.10: Maximal rate of progress for the sphere model

) can only be obtained iteratively. To Points on the curve ' max = ' ( = opt express  = ('max ), the non-linear system of equations consisting of Equations (5.20) and (5.21) must be solved. The results as obtained with the multimembered evolution strategy are shown in Figure 5.10. A convenient formula can only be obtained by assuming 2 '+ ' + i.e., 2 ' max ' opt This estimate is actually not far wrong, since the second term in Equation (5.21) goes to zero. We thus nd   1 q  q  ' 1 +  'max exp('max) 1 + erf 2 ' max (5.22) a relation with comparable structure to the result for the inclined plane. Finally we ask whether ' max= has a maximum, as in the inclined plane case. If the parent can survive the ospring, opt = 1 here too if not the condition  p  opt = 2  21 + ('+ + +)2 exp('+ + +)2] 1 + erf(+)] '+ (5.23) must be added to Equations (5.20) and (5.21). The solution, obtained iteratively, is: opt ' 4:7 (as an integer: opt = 5)

134

Evolution Strategies for Numerical Optimization

Both the (1 , ) and (1+) schemes were run on the computer for the sphere model, with n = 100 rE = 100, and variable . In each case ' was evaluated over 10 000 generations. The resulting data are shown in terms of ' and  in Figure 5.9. In . comparison with the approximate theory, deviations are apparent mainly for  > opt The skewness of the probability distribution w(s0) and the error in the estimate of the R integral D(y) dy have only a weak eect in the region of greatest interest, where the rate of progress is maximum. Furthermore, the results of the simulation fall closer to the approximate theory if n is taken to be greater than 100 however, the computation time then becomes excessive. For large values of  the possible survival of the parent only becomes noticeable when the variance is too large to allow rapid convergence. The greatest dierences, as expected, appear for  = 1. On the whole we see that the theory worked out here gives at least a qualitative account of the behavior of the (1 , ) strategy. A much more elegant method yielding an even better approximation may be found in Back, Rudolph, and Schwefel (1993), or Beyer (1993, 1994a,b). 5.2.2.3 The Corridor Model

As a third and last model objective function, we will now consider the right-angled corridor. The contours of F (x) in the two dimensional picture (Fig. 5.11) are straight and parallel, but not necessarily equidistant. n X F (x) = c0 + ci xi i=1

For the sake of simplifying the calculation we will again give the coordinate system a particular position and orientation with c1 = ;1 ci = 0 for all i = 2 3 : : :  n: The right-angled corridor (Problem 2.37, see Appendix A, Sect. A.2){we are using here three dimensional concepts for the essentially n-dimensional case{is dened by constraints of the form Gj (x) = jxj j
b for j = 2(1)n It has the width 2 b for all coordinate directions xi i = 2(1)n hence the cross section (2 b)n;1 . As a starting point, the position xE of the parent E , we choose the origin with respect to x1 = 0. The useful part of a random step is just its component z1 in the x1 direction, which is the negative gradient direction. The formulae for w(s` = s0) and p(s` < s0) derived previously for the inclined plane also apply here. We cannot, however, insert them immediately into Equation (5.15) rst we must pay attention to the rule that mutants that violate one or more of the constraints are not accepted. For a given mutation, the probability of not jumping through the corridor wall associated with the variable xi i = 2(1)n, is " Zb 2# 1 ( x ; x ) Ei `i p exp ; 2 2 p(jx`ij
b) = dx`i 2   x`i =;b

A Multimembered Evolution Strategy

135

x2 N

3

Line of equal probability density

Contours F(x) = const. Downwards

E

2b

N

2

0

N

x1

1

Allowed region Forbidden region s <0 2

s >0 1

Figure 5.11: Corridor model function

"  !  !# b ; x b + x 1 Ei Ei + erf p = 2 erf p 2 2 That is, the probability depends on the current position xEi of the starting point E . We can only construct an average value for all possible situations if we know the probability pa of certain situations occurring. It could well be that, during the minimum search, positions near the border are occupied less often than others. The same problem of nding the occupation probability pa has arisen already in the theoretical treatment of the (1+1) strategy. Rechenberg (1973) discovered that pa = 21b (with respect to one of the variables xi i = 2(1)n) which is a constant independent of the current values of the variables. We will assume that this also holds here. Thus the average probability that one of the n ; 1 constrained variables will remain within the corridor can be given as: Zb p~(jx`ij
b) = pa p(jx`ij
b) dxEi xEi =;b  !# Zb "  b ; xEi ! b + x 1 Ei = 4b + erf p dxEi erf p 2  2  xEi =;b

136

Evolution Strategies for Numerical Optimization

Making use of the relation (see Ryshik and Gradstein, 1963) Zp 2 2 erf( y) dy = p erf( p) + exp(;p p ) ; 1 y=0 one nally obtains

p ! "  2! # 2 b 1  p~(jx`ij
b) = erf  +  p exp ; 2b2 ; 1 (5.24) 2b In the following we refer to this expression as item v. v = p~(jx`ij
b) With the above denition of v, the total probability that a descendant N` is feasible, i.e., that it satises all the constraints, is n Y pfeas = p~(jx`ij
b) i=2 = vn;1 and the probability that N` is lethal is

pleth = 1 ; pfeas = 1 ; vn;1 Only non-lethal mutants come into consideration as parents of the next generation. Hence, instead of w(s` = s0) we must insert into Equation (5.15) the expression  02 ! 1 0 w(s` = s ) pfeas = p exp ; 2s2 vn;1 2  and instead of p(s` < s0) we should take "  0 !# 1 0 p(s` < s ) pfeas + pleth = 2 1 + erf ps vn;1 + 1 ; vn;1 2 The rst term expresses the probability that the descendant N` both falls within the allowed region and progresses by s0 the second term represents the probability that a descendant N` is either non-lethal and advanced by s` < s0, or lethal. If we now insert both these quantities into Equation (5.15) we obtain  02 ! "  0 !# !;1 n;1 s s  v n ; 1 n ; 1 0 exp ; 2 1 + erf p v + 2 (1 ; v ) (5.25) w~(s ) = p 2 2  2;1 2 where v is given by Equation (5.24). So far we have not considered the special case of all the descendants being lethal mutants. If we were to abide by the rules of the (1 , ) strategy as followed up to now, the

A Multimembered Evolution Strategy

137

outcome would be extinction of the population and the rate of progress would no longer be dened. The probability of extinction of the population is given by the product of the lethal probabilities: pstop = (1 ; vn;1) To be able to optimize further in such situations let us adopt the following procedure: If all the mutations lead to forbidden points, the parent will survive and produce another generation of descendants. Thus for this generation the rate of progress takes the value zero. Equation (5.25) then only holds for s0 6= 0 and we must reformulate the probability of advancing by s0 in one generation as follows: w(s0) = w~(s0) +  pstop where ( 0 0  = 01  ifif ss0 6= =0 The distribution w(s0) is no longer continuous, and even if w0(s0) is symmetric we cannot assume that the maximum of the distribution is a useful approximation to the average rate of progress (Fig. 5.12). The following condition must be satised: Z1 Z1 0 0 w(s ) ds = w~(s0) ds0 + wstop = 1 (5.26) 0 0 s =;1 s =;1 We can think of w(s0) as a superposition of two density distributions, with conditional mathematical expectation values Z1 '1 = s0 w~(s0) ds0 s0 =;1 and '2 = 0 and with associated frequencies Z1 w~(s0) ds0 = 1 ; pstop p1 = s0 =;1 and p2 = pstop The events belonging to the two density distributions are mutually exclusive and by virtue of Equation (5.26) they make up together a complete set of events. The expectation value is then given by (e.g., Gnedenko, 1970 Sweschnikow , 1970). Z1 '= s0 w(s0) ds0 = '1 p1 + '2 p2 = '1 (1 ; pstop ) s0 =;1

138

Evolution Strategies for Numerical Optimization

w(s’)

Probability density

w(s’) p

stop

s’ = 0

~ w(s’= 0)

s’ Useful distance covered

Figure 5.12: Estimation of the rate of progress from the probability density for the corridor model

Since we are unable to calculate '1 directly, we make an approximation: '~ = '^ (1 ; pstop) = '^1 ; (1 ; vn;1)]

(5.27)

taking for '^ the position of the maximum of w~(s0). We require  @ w~ (s0)  =! 0 @s0  s0 ='^

By dierentiating Equation (5.25) and setting the rst derivative to zero:  ! #  2 !" p ' ^ ' ^ ' ^ 1 ; n  = 1 +  p exp 2 2 1 + erf p + 2 (v ; 1) (5.28) 2 2 Apart from an extra term, this formula is similar to the relation  = (' ~ ) found for the inclined plane (Equation (5.17)). The main dierence here, however, is that in place of ' ~ '^ appears, as dened by Equation (5.27). As in the case of the sphere model, we will introduce here \universal parameters" ' = '~bn and  = bn and take the limit n ! 1 in order to arrive at a practically useful relation  = ('   ).

A Multimembered Evolution Strategy

139

With the new quantities ' and  , Equation (5.24) for v becomes  2 !# p ! "  2 n 1 ; exp ; 2n 2 v = erf  ; p 2 n Since the argument of the error function increases as n, the number of variables, the approximation erf(y) ' 1 ; p1 y exp (;y2) can be used to give and with



 v =1; p n 2

for n  1

 1 n lim n!1 1 + n = e nally  ! 1 ; n v = exp p 2 The desired relation  = ('   ) is thus  ! # p '~ 2 '~ !23 "  '~ !  = 1 + p exp 4 p 5 erf p + 2 exp p ; 1 2 2 2 2 in which, from Equation (5.27),

(5.29)

'  i 1 ; 1 ; exp ; p2 Pairs of values obtained iteratively are shown in Figure 5.13 together with simulation results for the cases of \survival" and \extinction" of the parent (n = 100 b = 100, average over 10 000 successful generations). As in the case of the sphere model, the deviations can be attributed to the simplifying assumptions made in deriving the approximate theory. For  = 1 ' is always zero if the parent is not included in the selection. The transition to the inclined plane model is correctly reproduced in this respect. Negative rates of progress cannot occur. ) at constant  are obtained in the The position of the maxima ' max = ' ( = opt same way as for the sphere model. The condition to be added to Equation (5.29) is !   exp (;+) 1 ; exp (;+)];1 ; 1  c + h ! ih 2 2i 2 + + + + +  erf(' ) + 2 exp ( ) ; 1 1 + 2 ' + p ' exp (;' ) + 2 exp(+) =! 0 (5.30) in which the following new quantities are introduced again for compactness: '~ =

h

140

Evolution Strategies for Numerical Optimization j

j

Figure 5.13: Rate of progress for the corridor model

A Multimembered Evolution Strategy

7

141

Maximal universal rate of progress ϕ*

max

( σ*

opt

)

6

5

4

(1+1) - theory

3

2

1

Number of descendants 0 1

10

20

30

40

50

λ

60

Figure 5.14: Maximal rate of progress for the corridor model

 + = popt 2 '+ = p'max c 2 opt "  !#  c = 1 ; 1 ; exp ; popt 2 Pairs of values found by iteration are shown in Figure 5.13. Figure 5.14 shows ' max versus . To determine opt for the (1 , ) strategy, i.e., the value of  for which ' max= is a maximum, it is necessary to solve the system of three non-linear equations, comprising Equation (5.29), Equation (5.30), and np o opt = '+  exp ('+2) erf ('+) + 2 exp (+) ; 1] 1 + 2'+ 2] + 2'+  ( )  opt +  +  c 1 ; exp (; )] ln1 ; exp(; )] + 1 (5.31) The result is opt ' 6:0 (as an integer: opt = 6)

142

Evolution Strategies for Numerical Optimization

5.2.3 The Step Length Control

How should one proceed in order to still achieve the maximum rate of progress, i.e., to maintain the optimum variances i2 i = 1(1)n, for the case of the multimembered evolution scheme? For the (1+1) strategy this aim was met by the 1=5 success rule, which was based on the probability of success at maximum convergence rate of the sphere and corridor model functions. Such control from outside the actual mutation-selection game does not correspond to the biological paradigm. It should rather be assumed that the step lengths, or more precisely the variances, have adapted and are still adapting to circumstances arising in the course of natural evolution. Although the environmentally induced rate of mutation cannot be interfered with directly, the existence of mutator genes and repair enzymes strongly suggests that the consequences of such environmental in!uences are always reduced to the appropriate level. In the multimembered evolution strategy the fact that the observed rates of mutation are also small, indeed that they must be small to be optimal, comes out of the universal rate of progress and standard deviation introduced above, which require  to be inversely proportional to the number of variables, as in the (1+1) strategy. If we wish to imitate organic evolution, we can proceed as follows. Besides the variables xEi i = 1(1)n, a set of parameters Ei i = 1(1)n, is assigned to a parent E . These describe the variances of the random changes. Each descendant N` of the parent E should dier from it both in x`i and `i. The changes in the variances should also be random and small, and the most probable case should be that there is no change at all. Whether a descendant can become a parent of the next generation depends on its vitality, thus only on its x`i. Which values of the variables it represents depends, however, not only on the xEi of the parent, but also on the standard deviations `i, which aect the size of the changes zi = x`i ; xEi. In this way the \step lengths" also play an indirect r^ole in the selection mechanism. The highest possible probability that a descendant is better than the parent is normally

wemax = 0:5 It is attained in the inclined plane case, for example, and for other model functions in the limit of innitely small step lengths. In order to prevent that a reduction of the i always gives rise to a selection advantage,  must be at least 2. But the optimal step lengths can only take eect if  > w1 eopt

This means that on average at least one descendant represents an improvement of the value of the objective function. The number of descendants per parent thus plays a decisive r^ole in the multimembered scheme, just as does the check on the success ratio in the two membered evolution scheme. For comparison let us tabulate here the opt of the (1 , ) strategy and weopt of the (1+1) strategy for the three model functions considered. The values of weopt are taken from the work of Rechenberg (1973).

A Multimembered Evolution Strategy

143

opt 1 2 2.5 2 0:27 3.7 4.7 1 5.4 6.0 2e How should the step lengths now be altered? We shall rst consider only a single variance 2 for changes in all the variables. In the production of the random changes, the standard deviation  is always a positive factor. It is therefore reasonable to generate new step lengths from the old by a multiplicative rather than additive process, according to the scheme N(g) = E(g) Z' (g) (5.32) The median ' of the random distribution for the quantity Z' must equal one to satisfy the condition that there is no deterministic drift without selection. Furthermore an increase of the step length should occur with the same frequency as a decrease more precisely, the probability of occurrence of a particular random value must be the same as that of its reciprocal. The third requirement is that small changes should occur more often than large ones. All three requirements are satised by the log-normal distribution. Random quantities obeying this distribution are obtained from (0  2) normally distributed numbers Y by the process Z' = eY (5.33) The probability distribution for Z' is then  2! (ln z ' ) 1 1 w('z ) = p z' exp ; 2  2 2  The next question concerns the choice of  , and we shall answer it, in the same way as for the (1+1) strategy, with reference to the rate of change of step lengths that maintains the maximum rate of progress in the sphere model. Regarding ' as a dierential quotient ;dr=dg leads to the relation (see Sect. 5.1.2) (g+1)  '  opt = exp ; max (5.34) ( g ) n opt for the optimal step lengths of two consecutive generations, where ' max now has a different, larger value that depends on  and . The actual size of the average changes in the variances, using the proposed mutation scheme based on Equations (5.32) and (5.33), depends on the topology of the objective function and the number of parents and descendants. If n, the number of variables, is large, the optimal variance will only change slightly from generation to generation. We will therefore assume that the selection in any generation is more or less indierent to reductions and increases in the step length. We thereby obtain the multiplicative change in the random quantity X , averaged over n generations: 0 n 1 0n 1 n1 X Y 1 X = @ Z' (g)A = exp @ n Y (g)A

Model function Inclined plane Sphere Corridor

weopt

1 weopt

g=1

g=1

144

Evolution Strategies for Numerical Optimization

Since the Y (g) are all (0  2) normally distributed, it follows from the addition theorem of the normal distribution (Heinhold and Gaede, 1972) that n 1X (g ) Y n g=1 2=n) normally distributed random quantity. Accordingly, the two quantities is a (0  p exp(= n) are characteristic of the average changes (minus sign for reduction) in the step lengths per generation. The median of w('z) is of course just e0 = 1. Together with Equation (5.34), our observation leads us to the requirement  !  '   max p exp ' exp n n or  ' 'pmax (5.35) n The variance  2 of the normally distributed random numbers Y , from which the lognormally distributed random multipliers for the standard deviations (\step sizes") of the changes in the object variables are produced, thus must vary inversely as the number of variables. Its actual value should depend on the expected rate of convergence ' and hence on the choice of the number of descendants . Instead of only one common strategy parameter , each individual can now have a complete set of n dierent i i = 1(1)n, for every alteration in the corresponding n object variables xi i = 1(1)n. The two following schemes can be envisioned: (g) =  (g) Z' (g) Ni (5.36) Ei i or (g) =  (g) Z' (g) Z' (g) Ni (5.37) 0 Ei i But only the second one should be taken into further consideration, because otherwise in the case of n  1 the average overall step size of the ospring v u n uX 2 sN = t Ni i=1

could not be substantially dierent from that of its parent v u n X u 2 t Ei sE = i=1

due to the levelling eect of the many random multiplication events (law of large number of events). In order to split the mutation eects to the overall step size and the individual step sizes one could choose 0 ' p'2n  for Z'0 (5.38)   ' p'2pn  for all Z'i  i = 1(1)n (5.39)

A Multimembered Evolution Strategy

145

We shall not go into further details since another kind of individual step length control will oer itself later, i.e., recombination. At this point a further word should be said about the alternative (1+) or (1 , ) strategies. Let us assume that by virtue of a jump landing far from the expectation value, a descendant has made a very large and useful step towards the optimum, thus becoming a parent of the next generation. While the variance allocated to it was eminently suitable for the preceding situation, it is not suited to the new one, being in general much too big. The probability that one of the new descendants will be successful is thereby low. Because the (1+) strategy permits no worsening of the objective function value, the parent survives{and may do so for many generations. This increases the probability of a successful mutation still having a poorly adapted step length. In the (1 , ) strategy such a stray member will indeed also turn up in a generation, but it will be in eect revoked in the following generation. The descendant that regresses the least survives and is therefore probably the one that most reduces the variance. The scheme thus has better adaptation properties with regard to the step length. In fact this phenomenon can be observed in the simulation. Since we have seen that for  5 the maximum rate of progress is practically independent of whether or not the parent survives, we should favor a ( , ) strategy, at least when = is not chosen to be very small, e.g., less than 5 or 6.

5.2.4 The Convergence Criterion for  > 1 Parents

In Section 5.2.2 we were really looking for the rate of progress of a ( , ) evolution method. Because of the analytical diculties, however, we had to fall back on the  = 1 case, with only one parent. We shall now proceed again on the assumption that  > 1. In each generation  state vectors xE and associated step lengths are stored, which should always be the  best of the  mutants of the previous generation. We naturally require more storage space for doing this on the computer, but on the other hand we have more suitable values at our disposal for each variable. Supposing that the topology of the objective function is complicated or even \pathological," and an individual reaches a point that is unfavorable to further progress, we still have sucient alternative starting points, which may even be much more favorable. According to the usefulness of their parameter sets, some parents place more mutants in the prime group of descendants than others. In general the best individuals of a generation will dier with respect to their variable vectors and objective function values as long as the optimum has not been reached. This provides us with a simple convergence criterion. From the population of  parents Ek  k = 1(1), we let Fb be the best objective function value: Fb = min fF (x(kg)) k = 1(1)g k and Fw the worst Fw = max fF (x(kg)) k = 1(1)g k Then for ending the search we require that either

Fw ; Fb
"c

146

Evolution Strategies for Numerical Optimization

or

   (F ; F )
X (g)  "d w b k=1 F (xk ) where "c and "d are to be dened such that ) "c > 0 1 + "d > 1 according to the computational accuracy Either absolutely or relatively, the objective function values of the parents in a generation must fall closely together before convergence is accepted. The reason for basing the criterion on function values, rather than variable values or step lengths, has already been discussed in connection with the (1+1) strategy (see Sect. 5.1.3).

5.2.5 Scaling of the Variables by Recombination

The ( , ) method opens up the possibility of imitating a further principle of organic evolution, which is of particular interest from the point of view of numerical optimization problems, namely sexual propagation. By combining the genes of two parents a new source of variation is added to point mutation. The fact that only a few primitive organisms do without this mechanism of recombination leads us to expect that it is very favorable for ( g ) ( g ) evolution. Instead of one vector xE now there are  distinct vectors xk for k = 1(1) in a population. In biology, the totality of all genes in a generation is known as a gene pool. Among the concerns of population genetics (e.g., Wilson and Bossert, 1973) is the frequency distribution of certain alleles in a population, the so-called gene frequencies. Until now, we did not argue on that level of detail, nor did we go down to the !oor of only four nucleic acids in order to model, for example, the mutation process within evolution strategies. This might be worthwhile for quaternary optimization, but not in our case of continuous parameters. It would be a tedious task to model all the intermediate processes from nucleic acids to proteins, cell, organs, etc., taking into account the genetic code and the whole epigenetic apparatus. We shall now apply the principle of recombination to numerical optimization with continuous parameters, once again in a simplied fashion. In our population of  parents we have stored  dierent values of each component xi i = 1(1)n. From this gene pool we now draw one of the  values of xi for each i = 1(1)n. The draw should be random so that the probability that an xi comes from any particular parent (k) of the  is just 1= for all k = 1(1). The variable vector constructed in this way forms the starting point for the subsequent variation of the components. The Figure 5.15 should help to clarify that kind of global recombination. By imitating recombination in this way we have, so as to speak, replaced bisexuality by multisexuality. This was less for reasons of principle than as a result of practical considerations of programming. A crude test yielded only a slight further increase in the rate of progress in changing from the bisexual to the multisexual scheme, whereas appreciable acceleration was achieved by introducing the bisexual in place of the asexual scheme, which allowed no recombination. A more detailed and exact comparison has yet to be carried out. Without some guidance from theory it is hard to choose the correct initial step lengths and rates of change of step lengths for each of the dierent algorithms.

A Multimembered Evolution Strategy

147

x x x ...

x

1 2 3

n

Parents of generation g

Recombination by choosing components columnwise and at random Discrete global recombination Descendants

Figure 5.15: Scheme of global uniform recombination

This is, however, the only way to arrive at quantitative statements, free from confusing side eects. It is thus hard to explain the origin of the accelerating eect of recombination. It may, for example, lie in the fact that instead of  dierent starting points, the bisexual scheme oers nX ;2 i 2  +  ( ; 1) 2 i=1

possible combinations in the case of n variables. With multirecombination, as chosen here, there are as many as n , which is far more than could be put into eect. A more detailed investigation may be found in Back (1994a). So far we have only considered recombination of the object variables, but the strategy variables, the step lengths, can be recombined in just the same way. Even if all the parents start with equal i =  for all i = 1(1)n, and if all the step length components are varied by a common random factor in the production of descendants, the variances i of all the individuals for each i = 1(1)n dier from each other in the subsequent generations. Thus by recombination is it possible for the step lengths to adapt individually in this way to circumstances. A better combination aords a higher chance of survival to its bearer. It can therefore be expected that in the course of the optimum search, the currently best combination of the fi i = 1(1)ng prevails{the one that is associated with the fastest rate of progress. In attempting to verify this in a practical test, an unpleasant phenomenon occurs. It can happen that one of the standard deviations i is suddenly

148

Evolution Strategies for Numerical Optimization

(e.g., by a random value very far from the expectation value) so much reduced in size that the associated variable xi can now hardly be changed. The total change in the vector x is then roughly speaking within an (n ; 1)-dimensional subspace of IRn. Contrary to what one might hope, that such a descendant would have less chance of surviving than others, it turns out that the survival of such a descendant is actually favored. The reason is that the rate of progress with an optimal step length is proportional to 1=n. If the number of variables n decreases, the rate of convergence, together with the optimal step length, increases. The optimum search therefore only proceeds in a subspace of IRn . Not until the only improvement in the objective function entails changing the variable that has hitherto been omitted from the variation will the mutation-selection mechanism operate to increase its associated variance and so restore it to the range for which noticeable changes are possible. The minimum search proceeds by jumps in the value of the objective function and with rates of progress that vary alternately above and below what would otherwise be smooth convergence. Such unstable behavior is most pronounced when , the number of parents, is small. With suciently large  the reserve of step length combinations in the gene pool is always big enough to avoid overadaptation, or to compensate for it quickly. From an experimental study (Schwefel, 1987) the conclusion could be drawn that punctuated equilibrium evolution (Gould and Eldredge, 1977, 1993) can be avoided by using a suciently large population ( > 1) and a suciently low selection pressure (= ' 7). A further improvement can be made by using as the starting point in the variation of the step lengths the current average of two parents' variances, rather than the value from only one or the other parent. This measure too has its biological justication it represents an imitation of what is called intermediary recombination (instead of discrete recombination). In this context chromosome mutations should be very eective, those in which for example, the positions of two individual step lengths are exchanged. As well as the haploid scheme of inheritance on which the present work is based, some forms of life also exhibit the diploid scheme. In this case each individual stores two sets of variable values. Whilst the formation of the phenotype only makes use of one allele, the production of ospring brings both alleles into the gene pool. If both alleles are the same one speaks of homozygosity, otherwise of heterozygosity. Heterozygote alleles enlarge the set of variants in the gene pool and thus the range of possible combinations. With regard to the stability of the evolutionary process this also appears to be advantageous. The true gain made by diploidy only becomes apparent, however, when the additional evolutionary factors of recessiveness and dominance are included. For multiple criteria optimization, the usefulness of this concept has been demonstrated by Kursawe (1991, 1992). Many possible extensions of the multimembered scheme have yet to be put into practice. To nd their theoretical eect on the rate of progress, one would rst have to construct a theory of the ( , ) strategy for  > 1. If one goes beyond the  = 1 scheme followed here, signicant dierences between approximate theory and simulation results arise for  > 1 because of the greater asymmetry of the probability distribution w(s0).

A Multimembered Evolution Strategy

149

5.2.6 Global Convergence

In our discussion of deterministic optimization methods (Chap. 3) we have established that only simultaneous strategies are capable of locating with certainty global minima of arbitrary objective functions. The computational cost of their application increases with the volume of the space under consideration and thus with the power of n. The dynamic programming technique of Bellman allows the reliability of global convergence to be maintained at less cost, but only if the objective function has a rather special structure, such that only a part of the space IRn needs to be investigated. Of the stochastic search procedures, the Monte-Carlo method has the best chance of global convergence it oers a high probability rather than certainty of nding the global optimum. If one requires a 90% probability, its cost is greater than that of the equidistant grid search. However, the (1+1) evolution strategy can also be credited with a nite probability of global convergence if the step lengths (variances) of the random changes are held constant (see Rechenberg, 1973 Born, 1978 Beyer, 1989, 1990). How great the chance is of nding an absolute minimum among several local minima depends on the topology, in particular on the disposition and \width" of the minima. If the user wishes to realize the possibility of a jump from a local to a global extremum, it requires a trial of patience. The requirement of approaching an optimum as quickly and as accurately as possible is always diametrically opposed to maintaining the reliability of global convergence. In the formulation of the algorithms of the evolution strategies we have mainly strived to satisfy the rst requirement of rapid convergence, by adaptation of the step lengths. Thus for both strategies no claims can be made for good global convergence properties. With  > 1 in the multimembered evolution scheme, several state vectors x(kg) 2 IRn  k = 1(1) are stored in each generation g. If the x(kg) are very dierent, the probability is greater that at least one point is situated near the global optimum and that the others will approach it in the process of generation. The likelihood of this is less if the x(kg) fall close together, with the associated reduction in the step lengths. It always remains nite, however, and increases with , the number of parents. This advantage over the (1+1) strategy is best exploited if one starts the search with  initial vectors x(0) k roughly evenly distributed over the whole region of interest, and chooses fairly large initial values of the standard deviations k(0) 2 IRn k = 1(1). Here too the ( ) scheme is preferable to the ( + ) because concentration at a locally very favorable position is at least delayed.

5.2.7 Program Details of the ( +
) ES Subroutines

Appendix A, Section A.2 contains FORTRAN listings of the multimembered ( + ) evolution strategy developed here, with the alternatives GRUP without recombination REKO with recombination (intermediary recombination for the step lengths) KORR the so far most general form with correlated mutations as well as ve dierent recombination types (see Chap. 7)

150

Evolution Strategies for Numerical Optimization

In the choice of  (number of parents) and  (number of descendants) there is no need to ensure that  is exactly divisible by . The association of descendants to parents is made by a random selection of uniformly distributed random integers from the range 1 ]. It is only necessary that  exceeds  by a sucient margin that on average at least one descendant can be better than its parent. From the results of Section 5.2.3 a suitable choice would be for example  6 . The transformation from 0 1] evenly distributed random numbers to (0 2) normally distributed pseudorandom numbers is carried out in the same way as in subroutine EVOL of the (1+1) strategy (see Sect. 5.1.5). The log-normally distributed variance multipliers are produced by the exponential function. The step lengths (standard deviations of the individual random components) can initially be specied individually. During the subsequent process of generation they satisfy the constraints ) i(g) "a for all i = 1(1)n and i(g) "b jx(ig)j where ) "a > 0 and 1 + "b > 1 according to the computational accuracy can be specied in advance. The parameter  which in!uences the averageprate of change of the step lengths should be given a value roughly proportional to 1= n in case of two factors (the case to be preferred), a global and an individual one, the values given in Section 5.2.3 are recommended. The constant of proportionality depends mainly on another adjustable feature, =, which may be called the selection pressure. For a (10 , 100) strategy it should be set at about unity to allow the fastest convergence of simple optimization problems like the hypersphere. With increasing  this value ' can be changed sublinearly according to p   ' e' (compare Equation (5.22)). If the initial step lengths i(0) are chosen to be too large, what may have been an especially well situated starting point x(0) can be thrown away. Nevertheless, this step backwards in the rst generation works in favor of reaching a global minimum among several local minima. In principle, for  > 1 each of the  dierent starting vectors (0) n n x(0) k 2 IR and k 2 IR  k = 1(1) can be specied. In the present program this dierentiation of the parent generation is carried out automatically the x(0) k are produced (0) (0) 2 from x by addition of (0 ( ) ) normally distributed random vectors. The k(0) = (0) are initially equal for all parents. The convergence criterion is described in Section 5.2.4. It is based on the dierence in objective function values between the current best and worst parents of a generation. As accuracy parameters, an absolute and a relative quantity ("c and "d) must be specied (compare Sect. 5.1.3). Furthermore, an upper bound on the computation time for the search can be given so that whatever the outcome results can be output from the main program (see also Sect. 5.1.5). Inequality constraints are treated as described for subroutine EVOL (Sect. 5.1.4) so

Genetic Algorithms

151

too is the case of the starting point x(0) lying outside the feasible region. Whereas the subroutine GRUP with option REKO has been taken into account in the test series of Chapter 6, this is not so for the third version KORR, which was created later (Schwefel, 1974). Still, more often than any multimembered version, the (1+1) strategy has been used in practice. Nonetheless it has proved its usefulness in several applications: for example, in conjunction with a linearization method for minimizing quadratic functions in surface tting problems (Plaschko and Wagner, 1973). In this case the evolution process provides useful approximate values that enable the deterministic method to converge. It should also serve to locate the global minimum of the multimodal objective function. Another practically oriented multiparameter case was to nd the optimum weight disposition of lightweight rigidly jointed frameworks (Ho!er, Ley%ner, and Wiedemann, 1973 Ley%ner, 1974). Here again the evolution strategy is combined with another method, this time the simplex method of linear programming. Each strategy is applied in turn until the possible improvements remaining at a step are very small. The usefulness of this procedure is demonstrated by checking against known solutions. A third example is provided by Hartmann (1974), who seeks the optimal geometry of a statically loaded shell support. He parameterizes the functional optimization problem by assuming that the shape of the cross section of the cylindrical shell is described by a suitable polynomial. Its coecients are to be determined such that the largest absolute value of the transverse moment is as small as possible. For various cases of loading, Hartmann nds optimal shell geometries diering considerably from the shape of circular cylinders, with sometimes almost vanishingly small transverse moments. More examples are mentioned in Chapter 7.

5.3 Genetic Algorithms At almost the same time that evolution strategies (ESs) were developed and used at the Technical University of Berlin, two other lines of evolutionary algorithms (EAs) emerged in the U.S.A., all independently of each other. One of them, evolutionary programming (EP), was mentioned at the end of Chapter 4 and goes back to the work of L. J. Fogel (1962 see also Fogel, Owens, and Walsh, 1965, 1966a,b). For a long time, activity on this front seemed to have become quiet. However, in 1992 a series of yearly conferences was started by D. B. Fogel and others (Fogel and Atmar, 1992, 1993 Sebald and Fogel, 1994) to disseminate recent results on the theory and applications of EP. Since EP uses concepts that are rather similar to either ESs or genetic algorithms (GAs) (Fogel, 1991, 1992), it will not be described in detail here, nor will it be compared to ESs on the basis of test results. This was done in a paper presented at the second EP conference (Back, Rudolph, and Schwefel, 1993). Similarly, contributions to comparing ESs and GAs in detail may be found in Homeister and Back (1990, 1991, 1992 see also Back, Homeister, and Schwefel, 1991 Back and Schwefel, 1993). The third line of EAs mentioned above, genetic algorithms, has become rather popular today and diers from the others in several aspects. This approach will be explained in the following according to its classical (also called canonical) form. Even to attentive scientists, GAs did not become apparent before 1975 when the rst

152

Evolution Strategies for Numerical Optimization

book of Holland (1975) and the dissertation of De Jong (1975) were published. Thus this work was unknown in Europe at the time when Rechenberg's and the author's dissertations were completed and, later on, published as books. Only 10 years later, however, in 1985, a series of biennial conferences (ICGA, International Conferences on Genetic Algorithms) has been started (Grefenstette, 1985, 1987 Schaer, 1989 Belew and Booker, 1991 Forrest, 1993) to bring together those who are interested in the theory or application of GAs. On the Eastern side of the Atlantic, a similar revival of the eld began in 1990 with the rst conference on parallel problem solving from nature (PPSN) (Schwefel and Manner, 1991 Manner and Manderick, 1992 Davidor, Schwefel, and Manner, 1994). During the PPSN 90 and the ICGA 91 events, proponents of GAs and ESs agreed upon the common denominators evolutionary algorithms (EAs) for both approaches as well as evolutionary computation (EC) for a new international journal (see De Jong, 1993). The latter term has been adopted among others by the Institute of Electrical and Electronics Engineers (IEEE) for an international conference during the 1994 World congress on computational intelligence (WCCI). Surveys of the history have been attempted by De Jong and Spears (1993) and Spears et al. (1993). As forerunners of the genetic simulation, Fraser (1957), Friedberg (1958), and Hollstien (1971) should at least be mentioned here.

5.3.1 The Canonical Genetic Algorithm for Parameter Optimization

Even if the originators of the GA approach emphasized that GAs were designed for general adaptation processes, most applications reported up to now concern numerical optimization by means of digital computers, including discrete as well as combinatorial optimization. Books by Ackley (1987), Goldberg (1989), Davis (1987, 1991), Davidor (1990), Rawlins (1991), Michalewicz (1992, 1994), Stender (1993), and Whitley (1993) may serve as sources for more details in this eld. As for so-called classier systems (CS see Holland et al., 1986) and genetic programming (GP see Koza, 1992), two very interesting special areas of evolutionary computation{in which GAs play an important r^ole in searching for production rules in so-called knowledge-based systems and for correct expressions in computer programs, respectively{the reader must be referred to the relevant and vast literature (Alander, 1994 he compiled more than 3,000 references). The GA for parameter optimization usually has been presented in the following general form: Step 0: (Initialization) A given population consists of  individuals. Each is characterized by its genotype consisting of n genes, which determine the vitality, or tness for survival. Each individual's genotype is represented by a (binary) bit string, representing the object parameter values either directly or by means of an encoding scheme. Step 1: (Selection) Two parents are chosen with probabilities proportional to their relative position in the current population, either measured by their contribution to the

Genetic Algorithms

153

mean objective function value of the generation (proportional selection) or by their rank (e.g., linear ranking selection). Step 2: (Recombination) Two dierent preliminary ospring are produced by recombination of two parental genotypes by means of crossover at a given recombination probability pc  only one of those ospring (at random) is actually taken into further consideration. Steps 1 and 2 are repeated until  individuals represent the (next) generation. Step 3: (Mutation) The ospring eventually (with a given xed and small probability pm ) underly further modication by means of point mutations working on individual bits, either by reversing a one to a zero, or vice versa or by throwing a dice for choosing a zero or a one, independent of the original value. At rst glance, this scheme looks very similar to that of a multimembered ES with discrete recombination. To reveal the dierences one has to take a closer look at the so-called operators, \selection (S)", \mutation (M)", and \recombination (R)." The GA sequence of events, i.e., S { R { M, as opposed to M { R { S within ESs, should not matter signicantly since the whole process is a circular one, and whether one likes to reverse the order of mutation and recombination is a matter of avoiding unnecessary operations or not. In applications, the evaluation of the individuals with respect to their corresponding objective function values normally dominates all other operations. Canonical values for the recombination probability are pc = 0:6, for the number of crossover points nc = 2, and for the mutation probability pm = 0:001.

5.3.2 Representation of Individuals

One of the most apparent dierences between GAs and ESs is the fact, that completely dierent representations of the object variables are used. Organic evolution uses four dierent nucleotides to encode the genotype in pairs of triplets. By means of the genetic code these are translated to 20 dierent amino acids. Since there are 43 = 64 dierent triplets, the genetic code is largely redundant. A closer look reveals its property of maintaining similarity on the amino acid level despite most of the small variations on the level of single nucleotides. Similar transmission laws between chains of amino acids and proteins, proteins and higher aggregates like cells and organs, up to the overall phenotype are called the epigenetic apparatus (Riedl, 1976). As a matter of fact, biologists as well as behaviorists report that dierences among several children of the same parents as well as dierences between two consecutive generations can well be described by normal distributions with zero mean and characteristic, probably genetically coded, variances. That is why ESs, when used for seeking optimal values for continuous variables use the more aggregate model of normal distributions for mutations and discrete or intermediary recombination as described in Sections 5.1 and 5.2. GAs, however, rely on binary representations of the object variables. One might call this genotypic modelling of the variation process, instead of phenotypic modelling as is

154

Evolution Strategies for Numerical Optimization

practiced in ESs and EP. An important link between both levels, i.e., the genetic code as well as the so-called epigenetic apparatus, is neglected at least in the canonical GA. For dealing with integer or real values on the level of the object variables GAs make use of a normal Boolean representation or they use the so-called Gray code. Both, however, present the diculty of so-called Hamming clis. Depending on its position, a single bit reversal thus can lead to small or very large changes on the phenotypic level. This important fact has advantages and disadvantages. The advantage lies in the broad range of dierent phenotypes available in a GA population at the same time, a matter aecting its global convergence reliability (for a thorough convergence analysis of the canonical GA see Rudolph, 1994a). The corresponding disadvantage stems from the other side of the same coin, i.e., the inability to focus the search eort in a close enough vicinity of the current positions of individuals in one generation. There is a second reason to cling to binary representations of object variables within GAs, i.e., Holland's schema theorem (Holland, 1975, 1992). This theorem tries to assure exponential penetration of the population by individuals with above average tness under proportional selection, with suciently higher reproduction rates for better individuals, one point crossover with xed crossover probability, and small, xed mutation rates. If, at some time, especially when starting the search, the population contains the globally optimal solution, this will persist in the case where there are zero probabilities for mutation and recombination. Mutation, according to the theorem, is an always destructive force and thus called a subordinate operator. It only serves to introduce missing or reintroduce lost correct bits into nite populations. Recombination (here, one point crossover) may or may not be destructive, depending on whether the crossover point happens to lie within a so-called building block, i.e., a short substring of the bit string that contributes to above-average tness of one of the mating individuals, or not. Building blocks are especially important in case of decomposable objective functions (for a more detailed description see Goldberg, 1989). GAs in their original form do not permit the handling of implicit inequality or equality constraints. On the other hand, explicit upper and lower bounds have to be provided for the range of the object variables:

ui
xi
vi for all i = 1(1)n in order to have a basis for the binary decoding and encoding process, e.g., l v i ; ui X xi = ui + 2l ; 1 aij 2j;1 j =1 where aij for j = 1(1)l represents the bit string segment of length l for encoding the ith element of the object variable vector x. Instead of this Boolean mapping one also may choose the Gray code, which has the property that neighboring values for the xi dier in one bit position only. Looking for the probability distribution p(*xi) of phenotypic changes *xi from one generation to the next at a given position x(0) i and a given mutation probability pm shows that changing the code from Boolean to Gray only shifts, but never avoids, the so-called Hamming

Genetic Algorithms

155

p(∆x)

p(∆x)

1

1

1e-5

1e-5

1e-10

1e-10

1e-15

1e-15 -6

-4

-2

0

2

4

6

8

10 ∆x

-6

-4

-2

0

2

4

6

8

10 ∆x

Figure 5.16: Probability distributions for GA mutations / left: normal binary code right: Gray code

clis. As Figure 5.16 clearly shows for a one dimensional case with x(0) = 5, l = 4, and pm = 0:001, the expectation values for changes *x are dierent from zero in both cases, and the distribution is in no case unimodal.

5.3.3 Recombination and Mutation

Innovation during evolutionary processes occurs in two dierent ways, for so-called higher organisms at least. Only the most early and primitive species operate asexually. People have often said that GAs can do their work without mutations, which, according to the schema theorem, always hamper the adaptation or optimization process, and that, on the other hand, ESs can do their work without recombination. The latter is not true if self-adaptation of the individual mutation variances and covariances is to work properly (see Schwefel, 1987), whereas the former conjecture has been disproved by Back (1993, 1994a,b). For a GA the probability of containing the correct bits for the global solution, dispersed over its random start population, is 1 ; L 2; , which may be close enough to 1 for  = 50 as population size and L = 1000 as length of the bit string (actually it is 0:999999999999) however, it cannot be guaranteed that those bits will not get lost in the course of generations. Whether this happens or not, largely depends on the problem structure, the phenomenon being called deception (e.g., Whitley, 1991 Page and Richardson, 1992). If one looks for recombination eects within GAs on the level of phenotypes, one stumbles over the fact that a recombined ospring of two parents that are close together in the phenotype space may largely deviate from both parental positions there. This

156

Evolution Strategies for Numerical Optimization Table 5.1: Two point crossover within a GA and its eect on the phenotypes Bit strings

Parent 1 Parent 2

0111 1100

1000 1011

Two point crossover Ospring 1 0000 1000 Ospring 2 1111 1111

Phenotype 7 12 8 11 0 8 15 15

completely contradicts the proverbial saying that the apple never falls far from the tree. Table 5.1 shows a simple situation with two parents producing two ospring by means of two point crossover, on a bit string of length 8, and encoding two phenotypic variables in the range 0 15] in the standard Boolean form. Neither discrete nor intermediary recombination within ESs can be that disruptive intermediary recombination always delivers phenotypic values for the ospring between those of their parents. The assumption that mutations are not necessary for the GA process may even stem from that disruptive character of recombination that permits crossover points not only at the boundaries of meaningful parental information but also within the genes themselves. ESs obey the general rule, that mutations are undirected, by means of using normally distributed changes with zero mean{even in the case of correlated mutations. That this is not so for GAs can easily be seen from Figure 5.16. Without selection, the GA process thus provides biased genetic drift, depending on the actual situation. Table 5.2 presents the probability transition matrix for one phenotypic integer variable xi in the range 0 3] encoded by means of two bits only. Let p = pm  12 single bit inversion probability and q = 1 ; pm probability of not inverting the bit From Table 5.2 it is obvious that among all possible transitions (except for those withTable 5.2: Transition probabilities for mutations within a GA Genotype

xi old

00 01 10 11

00 Phenotype 0 0 q2 1 pq 2 pq 3 p2

xi new 01 10 1 2 pq pq q2 p2 p2 q 2 pq pq

11 3 p2 pq pq q2

Genetic Algorithms

157

out any change) between the four dierent genetic states 00 01 10 11 (e.g., phenotypes 0 1 2 3), those from 01 to 10 and from 10 to 01 are the most improbable ones despite their phenotypic vicinity. Let pm = 10;3  then q2 = 0:998001 p q = 0:000999 and p2 = 0:000001:

5.3.4 Reproduction and Selection

Whether selection is the rst or last operator in the generation loop of EAs should not matter except for the rst iteration. The dierence in this respect between ESs and GAs, however, is that both mingle several aspects of the generation transition. Let us look rst, therefore, at the biological facts to be modelled by a selection operator. An ospring may or may not be able to survive the time span between birth and reproduction. If it is vital up to its reproductive age it may have varying numbers of ospring with one or more partners of its own generation. Thus, the term \selection" in EAs comprises at least three dierent aspects:

 Survival to adult state (ontogeny)  Mating behavior (perhaps including promiscuity)  Reproductive activity Both ESs and GAs select parents for each ospring anew, thus modelling maximal promiscuity. GAs assign higher mating and reproductive activities to individuals with better objective function values (both for proportional as well as linear or other ranking selection). But even the worst ospring of generation g may become parents for generation g + 1. The probability, however, may be very low. If this is the case, most ospring are descendants of a few best parents only. The corresponding loss of diversity in the population may lead to premature stagnation (not convergence!) of the evolutionary seeking process. Reducing the proportionality factor in the selection function, on the other hand, ultimately leads to random walk behavior. This enhances the reliability in multimodal situations, but reduces the convergence velocity and the precision of locating the optimum. For proportional selection, after Holland derived from an analogy to the game-theoretic multiarmed bandit problem, the average number of ospring for an individual with genotype ak , phenotype xk , and vitality f (xk ) is

(ak ) =  ps (ak ) =

+(f (xk )) = +k  +' 1X +( f ( x i ))  i=1

The transformation +(f ) is necessary for introducing the proportionality factor mentioned above as well as for dealing with negative values of the objective function. ps often is called the survival probability, which is misleading. No parent really survives its generation except in an elitist GA version. Then the best parent is put into the next generation

158

Evolution Strategies for Numerical Optimization

without applying the selection operator. Otherwise it may happen simply by chance that one or the other descendant is not dierent from one of its parents. In contrast to ESs, the number of ospring always is equal to the number of parents ( = ). There is no surplus of descendants to cope with lethal mutations and recombinations. ESs need that kind of surplus for handling constraints, at least. In the non-preserving case of its comma-version, a multimembered ES also needs a surplus ( > ) for the selection process. The  ;  worst ospring are handled as if they do not survive to the adult reproductive state the  best, however, have the same reproduction probability ps = 1=, which does not depend on their individual phenotypes or corresponding objective function values. Thus, on average, every parent has = descendants. This is depicted on the left-hand side of Figure 5.17, where the average number of descendants of the two best of  = 10 descendants (evenly distributed on the tness scale just for simplication purposes) is just = = 5 for a (2,10) ES, and zero for all others. Within a GA it largely depends on the scaling function +(f ), how many ospring are produced on average by their ancestors. The right-hand part of Figure 5.17 presents two possible situations. Crosses (+) belong to a steep, triangles (4) to a !at reproduction probability curve (average number of ospring) over the tness of the individuals. In the former case it typically happens that, just like in ESs, only the best individuals produce ospring (here the best parent has 6, the second best 3, the third best only 1, and all others zero ospring). One would call this strong selection. Weak selection, on the contrary, characterizes the other case (only the worst parent has no ospring, the best one just 2, and all others 1). It will strongly depend on the actual topology how one should choose the proportionality factor and it may even be necessary to change it during one optimum seeking process. Self-adaptation of internal strategy parameters is possible within the framework of GAs, too. Back (1992a,b, 1993, 1994a,b) has demonstrated this with respect to the mutation rate. For that purpose he adopts the selection mechanism of the multimembered ES. Last but not least, the question remains whether a stochastic or a deterministic approach to modelling selection is more appropriate. The argument that a stochastic model is closer to reality, is not sucient for the purpose at hand: optimization and adaptation.

5.3.5 Further Remarks

Of course, one would like to incorporate at least one close-to-canonical GA version into the comparative test series with all the other optimization procedures. But there are problems with that kind of endeavor. First, GAs do not permit general inequality constraints. This does not matter too much, since there are other algorithms that are not applicable directly in such cases, too. Next, GAs must be provided with lower and upper bounds for all parameters, which of course have to be chosen to contain the solution, probably in or near the middle of the hypercube dened by the explicit bounds. The GA thus would be provided with information that is not available for the other algorithms. For all other methods the starting point is of great importance, not only because it

Genetic Algorithms

159

Average Number of Offspring

Average Number of Offspring

5

5

4

4

3

3

2

2

1

1

0

0

1

5

10 Fitness

1

5

10 Fitness

Figure 5.17: Comparison of selection consequences in EAs left: ES right: GA

denes the initial distance from the optimum and thus determines largely the number of iterations needed to approximate the solution at the predened accuracy, but also because it may provide more or less topological diculties in its vicinity. GAs, however, should be started at random in the whole hypercube dened by the lower and upper bounds of the variables, in order to give them a chance of approaching the global or, at least, a very good local optimum. Reliability tests (see Appendix A, Sect. A.2), especially in cases of multimodal functions would thus be biased against all other methods, if one allows the GA to start from many points at the same time and if one gives the GA the needed extra information about the relevant search region that is not available for the other methods. One might provide special test conditions to compare dierent EAs with each other without giving one of them an advantage from the very beginning, but no large eort of this kind has been made so far. Even in cases of special constraints or side conditions one may formulate appropriate instantiations of suitable GA versions. This has been done, for example, for the combinatorial optimization task of solving the travelling salesperson problem (TSP) by Gorges-Schleuter (1991a,b) repair mechanisms were used in cases where unfeasible tours were caused by recombination. Beyer (1992) has investigated ESs for solving TSP-like optimization problems. It is much better to look for data structures tted to the special task and to redene the genetic operators to keep to the feasible solution set (see Michalewicz, 1992, 1994). The time for developing such special EAs must be added to the run time on the computer, and one argument in favor of EAs is lost, i.e., their simplicity of use or generality of application. As the short analysis of GA mutation and recombination operators above has clearly

160

Evolution Strategies for Numerical Optimization

shown, GAs other than ESs favor in-breadth search and thus are especially prepared to solve global and discrete optimization problems, where a volume-oriented approach is more appropriate than a path-oriented one. They have so far done their best in all kinds of combinatorial optimization (e.g., Lawler et al., 1985), a eld that has not been pursued in depth throughout this book. One example in the domain of computational intelligence has been the combined topology and parameter optimization of articial neural networks (e.g., Mandischer, 1993) another is the optimization of membership function parameters within fuzzy controllers (e.g., Meredith, Karr, and Kumar, 1992).

5.4 Simulated Annealing The simulated annealing approach to solve optimization problems does not really belong to the biologically motivated evolutionary algorithms. However, it belongs to the realm of problem solving methods that make use of other natural paradigms. This is the reason why this section has not been placed elsewhere among the traditional hill climbing strategies. In order to harden steel one rst heats it up to a high temperature not far away from the transition to its liquid phase. Subsequently one cools down the steel more or less rapidly. This process is known as annealing. According to the cooling schedule the atoms or molecules have more or less time to nd positions in an ordered pattern (e.g., a crystal structure). The highest order, which corresponds to a global minimum of the free energy, can be achieved only when the cooling proceeds slowly enough. Otherwise the frozen status will be characterized by one or the other local energy minimum only. Similar phenomena arise in all kinds of phase transitions from gaseous to liquid and from liquid to solid states. A descriptive mathematical model abstracts from local particle-to-particle interactions. It describes statistically the correspondences between macro variables like density, temperature, and entropy. It was Boltzmann who rst formulated a probability law to link the temperature with the relative frequencies of the very many possible micro states. Metropolis et al. (1953) simulated on that basis the evolution of a solid in a heat bath towards thermal equilibrium. By means of a Monte-Carlo method new particle congurations were generated. Their free energy Enew was compared with that of the former state (Eold). If Enew
Eold then the new conguration \survives" and forms the basis for the next perturbation. The new state may survive also if Enew > Eold , but only with a certain probability w   w = 1c exp EoldK;TEnew where K denotes the famous Boltzmann constant and T the current temperature. The constant c serves to normalize the probability distribution. This Metropolis algorithm thus is in line with the probability law of Boltzmann. Kirkpatrick, Gelatt, and Vecchi (1983) and C erny (1985) published optimization methods based on Metropolis' simulation algorithm. These methods are used quite frequently nowadays as simulated annealing (SA) procedures. Due to the fact that good intermediate positions may be \forgotten" during the search for a minimum or maximum, the algorithm is able to escape from local extrema and nally might reach the global optimum.

Simulated Annealing

161

There are two loops within the SA process:  Lowering the temperature (outer loop) Tnew = f (Told) < Told, e.g., Tnew =  Told 0 <  < 1 until the ground state T = 0 is reached  Waiting until the equilibrium state is found (inner loop) Metropolis simulations are performed at T = const: until no further improvements occur Two questions arise immediately. First, how long should the equilibration phase last, or which constructive criterion should be used for stopping the search for an optimum at a given temperature? Secondly, how large should the cooling steps be? Another question concerns the step size for the perturbations of the variables during the equilibration stage. There are many empirical suggestions for partial answers to the questions a lot of successful applications of the method, e.g., for the combinatorial optimization of the travelling salesperson problem (TSP) as well as some rigorous theoretical results concerning the global convergence but very few investigations about the convergence rates that can be obtained. A good summary may be found in the books of van Laarhoven and Aarts (1987), Aarts and Korst (1989), and Azencott (1992). The relation between SA and evolutionary algorithms (EAs) has been stressed by Rudolph (1993), especially under the parallel computing point of view. In the following a more detailed pseudocode is given: Step 0: (Initialization) Choose a start position x(00), a start temperature T (0), a start width d(0) for the variations of x. Set x = x~ = x(00) k = 0, and ` = 1. Step 1: (Metropolis simulation) Construct x(kl) = x~ + d(k) z, where z is uniformly distributed for all components zi for all i = 1(1)n in the range zi 2 ; 21  + 12 ]  or normally distributed according to w(zi) = p12 exp ; 12 zi2 . If F (x(kl)) < F (x ) , set x = x(kl). If F (x(kl)) < F (~x) , go to step 3 otherwise draw  F (x(kla )uniform random number, , from the interval 0 1]: ) ; F (~ x ) , go to step 3. If
exp T (k) Step 2: (Check for equilibrium) If F (x ) has not been improved within the last N trials, go to step 4. Step 3: (Inner loop) Set x~ = x(kl), increase ` ` + 1, and go to step 1.

162

Evolution Strategies for Numerical Optimization

Step 4: (Termination criterion) If T (k)
", end the search with result x . Step 5: (Cooling, outer loop) Set x(k+10) = x  x~ = x , and T (k+1) =  T (k) 0 <  < 1. Eventually, decrease d(k+1) =  d(k) 0 <  < 1. Set ` = 1, increase k k + 1, and go to step 1. The most important feature of the SA algorithm is its ability to escape from inferior local optima by allowing deteriorations with a certain probability. This kind of forgetting principle cannot be found in most numerical optimization routines. In EAs, however, it is more or less built-in as well. Though the overall structure of the algorithm is rather simple, it turns out to be quite dicult to decide upon the free parameters T (0) the temperature to start with d(0) the start width for the step sizes  the cooling factor  the step size reduction factor N the criterion upon which to state \equilibrium" " the lower bound on the temperature All rules that have been devised rely upon assumptions concerning the special type of objective function. The reader is referred to the literature in this special eld, which is closely related to the eld of global and stochastic optimization. Lau%ermair (1992a,b) recently devised a special set of rules called hyperplane annealing, and Rudolph (1993) points to similarities with ESs in case of parallel function evaluations.

5.5 Tabu Search and Other Hybrid Concepts Many heuristic optimum seeking methods, especially those that are called more or less greedy, are in danger of getting trapped in inferior local optima in case of multimodal objective functions. This is especially enhanced by measures to achieve ultimate eciency, e.g., by controlling the step size or search domain. Tabu search (TS) is a metastrategy aimed at avoiding the local optimality trap and can be superimposed onto many traditional direct optimization methods. Glover (1986, 1989 see also Glover and Greenberg, 1989) tries to overcome the problem by setting up short-, medium-, and long-term memories of successful as well as unsuccessful trials. According to that history of events, some rules are set up to alternate between three modes of operation:  Aggressive exploration  Intensication  Diversication

Tabu Search and Other Hybrid Concepts

163

Aggressive exploration using a short-term memory forms the core of the TS. From a candidate list of (non-exhaustive) moves the best admissible one is chosen. The decision is based on tabu restrictions on the one hand and on aspiration criteria on the other. Whereas aspiration criteria aim at perpetuating former successful operations, tabu restrictions help to avoid stepping back to inferior solutions and repeating already investigated trial moves. Although the best admissible step does not necessarily lead to an improvement, only better solutions are stored as real moves. Successes and failures are used to update the tabu list and the aspiration memory. If no further improvements can be found, or after a specied number of iterations, one transfers the results to the longer-term memories and switches to either an intensication or a diversication mode. Intensication combined with the medium-term memory refers to procedures for reinforcing move combinations historically found good, whereas diversication combined with the long-term memory refers to exploring new regions of the search space. The rst articles of Glover (1986, 1989) present many ideas to decide upon switching back and forth between the three modes. Many more have been conceived and published together with application results. In some cases complete procedures from other optimization paradigms have been used within the dierent phases of the TS, e.g., line search or gradient-like techniques during intensication, and GAs during diversication. Instead of going into further details here, it seems appropriate to give some hints that point to rather similar hybrid methods, more or less centered around either GAs, ESs, or SA as the main strategy. One could start again with Powell's rule to look for further restart points in the vicinity of the nal solutions of his conjugate direction method (Chap. 3, Sect. 3.2.2.1) or with the restart rule of the simplex method according to Nelder and Mead (Chap. 3, Sect. 3.2.1.5), in order to interpret them in terms of some kind of diversication phase. But in general, both approaches cannot be classied as better ideas than starting a specic optimum seeking method from dierent initial solutions and simply comparing all the (maybe dierent) outcomes, and choosing the best one as the nal solution. It might even be more promising to use dierent strategies from the same starting point and to select the overall best outcome again as a new start condition. On MIMD (multiple instructions, multiple data) parallel computers or nets of workstations the competition of dierent search methods could even be used to set up a knowledge base that adapts to a specic situation (e.g., Peters, 1989, 1991). Only individual conclusions for one or the other special application can be drawn from this kind of metastrategic approach, however. At the close of this general survey, only a few further hints will be given regarding the vast number of recent proposals. Ablay (1987), for example, uses a basic search routine similar to Rechenberg's (1+1) ES and interrupts it more or less frequently by a pure random search in order to avoid premature stagnation as well as convergence to a non-global local optimum. The replicator algorithm of Voigt (1989) also refers to organic evolution as a metaphor (see also Voigt, Muhlenbein, and Schwefel, 1990). Its modelling technique may be called descriptive, according to earlier work of Feistel and Ebeling (1989). Ebeling (1992) even proposes to incorporate ontogenetic learning features (so-called Haeckel strategy). Muhlenbein and Schlierkamp-Voosen (1993a,b) proposed a so-called breeder GA, which

164

Evolution Strategies for Numerical Optimization

combines a greedy algorithm to locate nearest local optima very quickly, with a genetic algorithm to allocate recombined start positions for further local optimum seeking cycles. This has proven to be very successful in special situations where the local optima are situated in a regular pattern in the search space. Dueck and Scheuer (1990) have devised a so-called threshold accepting strategy, which is rather similar to the simulated annealing approach but pretends to deliver superior results. Later on Dueck (1993) elaborated his great deluge algorithm, which adds to the threshold accepting method some kind of diversication mode like the tabu search in order to avoid premature stagnation at a non-global local optimum. Lohmann (1992) and Herdy (1992) propose a hierarchical ES according to Rechenberg's extended notation (Rechenberg, 1978, 1989, 1994) of the multimembered scheme to solve so-called structural optimization problems. Whereas this term normally points to situations in which a solid structure subject to stresses and deformations has to be designed in order to have least weight or production cost, Lohmann and Herdy do not mean anything else than a mixed-integer optimization problem. The solution is sought for in an outer ES-loop that varies the integer object variables only and an inner ESloop that varies the real-valued variables. Thus the outer loop compares relative optima found in the inner loops. This kind of cyclical subspace search, somehow similar to the Gauss-Seidel approach, must not represent the ultimate solution to mixed-integer problems, however. It is more or less prone to nding non-global local optima only. A more general evolutionary algorithm should be able to change{at the same time, by appropriate mutation and recombination operators{both the discrete and the real-valued object variables. But this speculation must be proved in forthcoming further steps towards a more general evolutionary algorithm, perhaps a hybrid of ES and GA ingredients.

Chapter 6 Comparison of Direct Search Strategies for Parameter Optimization 6.1 Diculties The vast and steadily increasing number of optimization methods necessarily raises the question of which is the best strategy. There seems to be no unique answer. If indeed there were an optimal optimization method all the others would be superuous and would have been long ago forgotten. Because of the strong competition between already existing strategies it is necessary nowadays that whenever any proposal for a new method or variant is made, its advantages and improvements compared to older strategies be displayed. The usual way is to refer to a minimum problem for which the known methods fail to nd a solution whereas the new proposal is successful. Or it is shown with reference to chosen examples that computation time or iterations can be saved by using the new version. The series of publications along these lines can in principle be continued indenitely. With sucient insight into the working of any strategy a special optimization problem can always be constructed for which the strategy fails. Likewise for any problem a special method of solution can be devised that is superior to the other procedures. One simply needs to exploit to the full what one knows of the problem structure as contained in its mathematical formulation. Progress in the eld of optimization methods does not, however, consist in developing an individual method of solution for each problem or type of problem. A practitioner would much rather manage with just one strategy, which can solve all the practically occurring problems for as small a total cost as possible. But as yet there is no such universal optimization method, and some authors doubt if there ever will be (Arrow and Hurwicz, 1957). All the methods presently known can only be used without restriction in particular areas of application. According to the nature of the particular problem, one or another strategy o ers a more successful solution. The question of which is the best strategy is itself a kind of optimization problem. To be able to answer it objectively an objective function would have to be formulated for deciding which of two methods 165

166

Comparison of Direct Search Strategies for Parameter Optimization

was best from the point of view of its results. So long as no generally recognized quality function of this kind exists, the question of which optimization method is optimal remains unanswered.

6.2 Theoretical Results Classical optimization theory is concerned with establishing necessary and sucient existence criteria for maxima and minima. It provides systems of equations but no iterative methods of nding their solutions. Not even Dantzig's simplex method (1966) for solving linear programming problems can be regarded as a direct result of theory{theoretical considerations of the linear problem only show that the extremum sought, except in special cases, must always lie in a corner of the polyhedron dened by the constraints. With n variables and m constraints (together with n non-negativity conditions) the number of corners or points of intersection
 of the hypersurfaces formed by the constraints is also m + n limited to a maximum of n . Even the systematic inspection of all the points of intersection would be a nite optimization method. But not all the points of intersection are also within the allowed region (Saaty, 1955, 1963). Muller-Merbach (1971) gives m n ; m + 2 as an upper bound to the number of feasible corner points. The simplex method, which is a method of steepest ascent along the edges of the polyhedron only traverses a tiny fraction of all the corners. Dantzig (1966) refers to empirical evidence that the number of necessary iterations increases as n, the number of variables, if the number of constraints m is constant, or as m if (n ; m) is not too small. Since, in the least favorable case, between m and 2 m exchange operations must be performed on the tableau of (m + 1)(n + 1) coecients, the average computation time increases as O(m2 n). In so-called degenerate cases, however, the simplex method can also become innite. The repeated cycling through the same corners must then be broken by a rule for randomly choosing the iteration step (Dantzig). From a theoretical point of view the ellipsoid method of Khachiyan (1979) and the interior point method of Karmarkar (1984) do have the advantage of polynomial time consumption even in the worst case. The question of niteness of iterative methods is also a central theme of non-linear programming. In this case the solution can lie at any point on the border or interior of the enclosed region. For the special case that the objective function and all the constraint functions are convex and multiply di erentiable Kuhn and Tucker (1951) and John (1948) have derived necessary and sucient conditions for extremal solutions. Most of the iteration methods that have been developed on this basis are designed for problems with a quadratic objective function and linear constraints. Representative of quadratic programming are, for example, the methods of Beale (1956) and Wolfe (1959a). They make extensive use of the algorithm of the simplex method and thus belong, according to Hadley (1969), to the class of neighboring extremal point methods. Other strategies can move into the allowed region in the course of the iterations. As far as the constraints permit they take the direction of the gradient of the objective function. They are therefore known as gradient methods of non-linear programming (Kappler, 1967). As their name may suggest, however, they are not suitable for all non-linear problems. Their convergence can be proved at best for di erentiable quasi-convex programs (Kunzi, Krelle, and

Theoretical Results

167

Oettli, 1962). For these conditions the number of required iterations and rate of convergence cannot be stated in general. The same is true for the methods of Khachiyan (1979) and Karmarkar (1984). In the following chapters a short summary is attempted of the convergence properties of non-linear optimization methods in the unconstrained case (hill climbing methods).

6.2.1 Proofs of Convergence

A proof of convergence of an iterative method will aim to show that a sequence of iteration points x(k) tends monotonically with the index k towards the point x which is sought: 0

lim kx(k) ; x k ! 0

or

k

0

!1

kx(k) ; x k  " "  0 for K (")  k < 1 0

If a nite accuracy of approximation is required, e.g., in terms of a distance from the solution measured by the Euclidean norm, the number of necessary iterations is usually nite. In the case of optimization strategies it is shown that the rst partial derivatives vanish at the point x : rF (x ) = 0 This necessary condition for an extremum of a continuously di erentiable function F (x) is at the same time the termination criterion of the procedure. There are numerous convergence proofs of this kind covering a very wide range of minimization methods. A good survey is given by Polak (1971). It contains convergence proofs for, among others  The Newton-Raphson method Assumption: F (x) is twice continuously di erentiable, r2F (x) has an inverse  A generalized gradient method based on the method of steepest descent Assumption: F (x) is once continuously di erentiable  A derivative-free method with local variation of the variables, similar to the GaussSeidel iteration method Assumption: F (x) is continuously di erentiable In many optimization methods that deal with a function of several variables, each iteration consists of a number of one dimensional minimizations. For such a procedure to be nite it is not enough to show that the sequence of iteration points tends monotonically to the desired solution. The number of arithmetic operations in each iteration must also be nite. However, a line search may only become exact in the limit of innitely many steps, while for the overall procedure to be nite, each one dimensional minimization must be terminated. This can result in the loss of convergence. Polak therefore distinguishes between conceptual algorithms, with an arbitrary number of calculation steps in one iteration, and practical algorithms in which this number is nite. To ensure the convergence 0

0

168

Comparison of Direct Search Strategies for Parameter Optimization

of a practical method, one must usually introduce adaptive rules for the termination of subroutines that would in principle run forever (Nickel, 1967 Nickel and Ritter, 1972). A further limitation to the predictive power of proofs of convergence arises from the properties of the point x referred to above. Even if confusion of maxima and minima is eliminated, the approximate solution x can still be a saddle point. To exclude this possibility, the second and sometimes even higher partial derivatives must be constructed and tested. It still always remains uncertain whether the solution that is nally found represents the global minimum or only a local minimum of the objective function. The only possibility of proving the global convergence of a sequential optimization method seems to be to require unimodality of the objective function. Then only one local optimum exists that is also a global optimum. Some global convergence properties are only possessed by a few simultaneous methods, such as for example the systematic grid method or the Monte-Carlo method. They place no continuity requirements on the objective function but the separation of the trial points must be signicantly smaller than the distance between neighboring minima and the required accuracy. The fact that its cost rises exponentially with the number of variables usually precludes the practical application of such a method. How does the convergence of the evolution strategy compare? For xed step lengths, or more precisely for xed variances i2 > 0 of the normally distributed mutation steps, there is always a positive probability of going from any starting point (e.g., a local minimum) to any other point with a better objective function value, provided that the separation of the points is nite. For the two membered method, Rechenberg (1973) gives necessary and sucient conditions that the probability of success should exceed a specied value. Estimates of the computation cost can only be made for special objective functions. In this respect there are problems in determining the rules for controlling the mutation step lengths and deciding when the search is to be terminated. It is hard to reconcile the requirements for rapid convergence in one case and for a certain minimum probability of global convergence in another. 0

0

6.2.2 Rates of Convergence While it may be of importance from a mathematical point of view to show that under certain assumptions a particular method leads with certainty to the objective, it is even more important to know how much computational e ort is required, or what is the rate of convergence. The question of how fast an optimal solution is approached, or how many iterations are needed to reach a prescribed small distance from the objective, can only be answered for a few abstract methods and under even more restrictive assumptions. One distinguishes between rst and second order convergence. Although some authors reserve the term quadratic convergence for the case when the solution of a quadratic problem is found within a nite number of iterations, it will be used here as a synonym for second order convergence. A sequence of iteration points x(k) converges linearly to x if it satises the condition kx(k) ; x k  c k 



Theoretical Results

169

where 0   < 1 and c < 1, constant. All methods which progress to the objective as a geometric progression in this way are said to display rst order convergence. For a suitable choice of step lengths, e.g., following Polak (1971), the strategy of steepest descents satises this condition if the objective function is at least twice continuously di erentiable and strictly convex in the neighborhood of the local minimum x . A sequence x(k) is said to be quadratically convergent if it satises the condition 

kx(k+1) ; x k  c kx(k) ; x k2 < 1 

0



where c < 1, constant. Strategies providing iteration points such that the error after a step is proportional to the square of the preceding error exhibit second order convergence. The number of exact signicant gures approximately doubles at each iteration. If a Newton method converges, then it converges quadratically either if the objective function is four times continuously di erentiable or if it is three times di erentiable and the Hessian matrix of second partial derivatives is denite. Under the second condition it can be shown that the method of conjugate gradients with cyclic restart converges quadratically. If furthermore the objective function can be treated as convex, second order convergence can also be proved for the variable metric method. Polak (1971) shows that under the weaker assumption of a bounded Hessian matrix and an only twice continuously di erentiable objective function, quadratic convergence can no longer be proved for the Newton-Raphson method. Its rate of convergence is still however greater than linear, i.e., 0

kx(k) ; x k ! 0 for  2 (0 1] lim k k Quadratic convergence makes an optimization method attractive from a mathematical point of view. Unfortunately this desirable property is coupled with a tendency to diverge if the objective function is of higher than second order and the search is not started near the solution. For this reason combinations have often been proposed of a rst order strategy at the start of an optimum search followed by a second order strategy in the neighborhood of the minimum. 

!1

6.2.3 Q-Properties

While it is to be expected that a quadratically convergent strategy will take fewer iterations to locate a minimum than one that only converges linearly, it is still of interest to know the explicit number of calculation steps required. This can only be given in a general form for the simplest case of a non-linear minimization problem, namely for quadratic objective functions

F (x) = xT A x + b x + c with a positive denite matrix of coecients A. Since all second order methods also employ a quadratic function as an internal model of the objective function for the purpose of predicting suitable directions and sometimes also step lengths, they can at least in principle nd the exact solution within a nite number of steps. They are referred to by

170

Comparison of Direct Search Strategies for Parameter Optimization

their so-called Q-properties. Thus if a strategy takes p iteration steps for locating exactly the quadratic optimum it is said to have the property Q p. The Newton-Raphson method, for example, takes only a single step because the second partial derivatives are constant over the whole IRn and all higher order derivatives vanish. If the iteration rule is followed exactly it gives the position of the minimum right at the rst step without the necessity of a line search. As no objective function values need to be evaluated explicitly one also refers to it as an indirect optimization method. It has the property Q 1. A conjugate gradients method, e.g., that of Fletcher and Reeves (1964), requires up to n cycles before a complete set of conjugate directions is assembled and a line search leads to the minimum. It therefore has the property Q n. Powell's (1964) derivative-free search method of conjugate directions requires n + 1 line searches for determining each of the n direction vectors and thus has the property Q n(n + 1) or Q O(n2 ) in terms of the number of one dimensional minimizations. The variable metric strategy of Davidon (1959) in the formulation of Fletcher and Powell (1963) can be interpreted both as a quasi-Newton method and as a method with conjugate directions. If the objective function is quadratic, then the iteratively improved approximate matrix agrees with the exact inverse of the Hessian matrix after n iterations. This method has the property Q n. Apart from the fact that any practical algorithm can require more than the theoretically predicted number of iterations due to the e ect of rounding errors, for peculiar types of coecient matrix in the quadratic problem the algorithm can fail completely. For example Zangwill (1967) demonstrates such a source of error in the Powell method if no improvement is achieved in one direction.

6.2.4 Computing Demands

The specication of the Q-properties of individual strategies is only the rst step towards estimating the computing demands. In di erent procedures an iteration or a cycle comprises various di erent operations. It is useful to distinguish ordinary calculation operations like additions and multiplications from the evaluation of functions such as the objective function and its derivatives. The number of variables is the basic quantity that determines the computation cost. A crude but adequate measure is therefore given by the power p of n, the number of parameters, with which the expected computation times increase. For the case of many variables, since the highest powers are dominant, lower order terms can be neglected. In the Newton-Raphson method, at each iteration the gradient vector rF and the Hessian matrix r2F must be evaluated, which means n rst and n (n + 1) second partial derivatives. Objective function values are not required. In fact 2 the most costly step is the matrix inversion. It requires in the order of O(n3 ) operations. A cycle of the conjugate gradient method consists of a line search and a gradient determination. The one dimensional minimization requires several calls of the objective function. Their number depends on the choice of method but it can be regarded as constant, or at least as independent of the number of variables. The remaining steps in the calculation, including vector multiplications, are composed of O(n) elementary arithmetical opera-

Theoretical Results

171

tions. Similar results apply in the case of the variable metric strategy, except that there are an additional O(n2) basic operations for matrix additions and multiplications. The direct search method due to Powell evaluates neither rst nor second partial derivatives. After every n + 1 line searches the direction vectors are redened, which requires O(n2) values to be assigned. But since each one dimensional optimization counts as an iteration step, only O(n) direct operations are attributed to each iteration. A convenient summary of the relationships is given in Table 6.1. For simplicity only the terms of highest order in the number of parameters n are accounted for, without their coecients of proportionality. So far we have no scale for comparison of the di erent function evaluations with each other. Fletcher (1972a) and others consider an evaluation of the Hessian matrix to be equivalent to O(n) gradient determinations or O(n2) objective function calls. This type of scaling is valid whenever the partial derivatives cannot be obtained in analytic form and provided as functions, but are calculated approximately as quotients of di erences obtained by trial steps in the coordinate directions. In any case it ought to be about right if the objective function is of higher than second order. Accordingly the following weighting of the function evaluations can be introduced on the table:

F : rF : r2F = n0 : n1 : n2 ^

Before anything can be said about the overall computation cost, or time, one must know how many operations are required for calculating a value of the objective function. In general a function of n variables will entail a cost that rises at least linearly with n. Table 6.1: Number of operations required by the most important basic strategies to minimize a quadratic objective function in terms of the number of variables n (only orders of magnitude) Number of Number of operations per iteration evaluations Elementary Strategy iterations Function F rF r2F operations Newton n0 | n0 n0 n3 e.g., Newton-Raphson Variable metric n1 n0 n0 | n2 e.g., Davidon Conjugate gradients n1 n0 n0 | n1 e.g., Fletcher-Reeves Conjugate directions n2 n0 | | n1 e.g., Powell n0 n1 n2 Weighting factors

172

Comparison of Direct Search Strategies for Parameter Optimization

For a quadratic function with a full matrix of coecients, just to evaluate the expression xT A x requires O(n2 ) basic arithmetical operations. If the order of magnitude is denoted by O(nf ) then, assuming f  1, for all the optimization methods considered so far the computation time is given by: T  n2+f  n3 The advantage of having fewer function-independent operations in the Fletcher-Reeves method, therefore, only makes itself felt if the number of variables is small and the time for one function evaluation is short. All the variants of the basic second order strategies mentioned here can be tted, with similar assumptions, into the above scheme. Among these are (Broyden, 1972)  Modied and quasi-Newton methods

 Methods of conjugate gradients and conjugate directions  Variable metric strategies, with their variations using correction matrices of rank one

There is no optimization method that has a cost rising with less than the third power of the number of variables. Even the indirect procedure, in which the equations for the necessary conditions for an extremum are set up and solved by conventional methods, does not a ord any basic reduction in the computational e ort. If the objective function is quadratic, a system of n simultaneous linear equations is obtained. To solve for the n unknowns the Gaussian elimination method requires 13 n3 basic operations (multiplications and divisions). According to Zurmuhl (1965) all the other direct methods, meaning here non-iterative methods, are more costly, except in special cases. Methods involving a stepwise approach to the solution of systems of linear equations (relaxation methods) require an innite number of iterations to reach an absolutely exact result. They converge linearly and correspond to rst order optimization strategies (single step or Gauss-Seidel methods and total step or gradient methods see Schwarz, Rutishauser, and Stiefel, 1968). Only the method of Hestenes and Stiefel (1952) converges after a nite number of calculation steps, assuming that the calculations are exact. It is a conjugate gradient method for solving systems of linear equations with a symmetrical, positive-denite matrix of coecients. The main concern here is with direct, i.e., derivative-free, search strategies for optimization. Finiteness of the search in the quadratic case and greater than linear convergence can only be proved for the Powell method of conjugate directions and for the Davidon-Fletcher-Powell variable metric method, which Stewart reformulated as a derivative-free quasi-Newton method. Of the coordinates strategy, at best it can be said that it converges linearly. The same holds for the simple gradient methods. There are also versions of them in which the partial derivatives are obtained numerically. Since various comparison tests have shown them to be rather ine ective in highly non-linear situations, none is considered here. No theoretically founded statements about convergence rates and Q-properties are available for the other direct strategies. The rate of progress dened by Rechenberg (1973) for the evolution strategy with adaptive step length control

Numerical Comparison of Strategies

173

represents an average measure of convergence. It could, however, only be determined theoretically for two selected model objective functions. The one with concentric contour lines, or contour hypersurfaces, can be regarded as a special case of a quadratic objective function. The formula for the local rate of progress in both the two membered and the multimembered strategies has the form '(r) = c nr  c = const: r is the current distance from the objective: r = kx(k) ; x k and ' is the change in r at one iteration or mutation '(r) = 4r = kx(k) ; x k ; kx(k+1) ; x k Rearrangement of the above formulae gives   c ( k +1) ( k ) kx ; x k = kx ; x k 1 ; n or k  kx(k) ; x k = kx(0) ; x k 1 ; nc which because 0 < 1 ; nc < 1  for 1  n < 1 proves the linear convergence property of the evolution strategy. 













6.3 Numerical Comparison of Strategies While the statements about convergence and rates of convergence derived from theory are not without value, they can say little about the capability of optimization methods in the general non-linear case because of the frequently rather limiting assumptions or restrictions. The computational e ort for example could only be specied for quadratic objective functions. The need therefore arises for numerical tests even for mathematically based methods in the case of non-linear optimization. Many of the direct strategies are only heuristic in nature anyway. They owe their success simply to the experimental evidence of their usefulness in practical situations. Iteration methods usually require a considerable number of calculation steps. Without mechanical assistance they frequently cannot be applied at all. There is thus an evident parallel between the development of rapid digital computers and optimization methods. The use of such systems entails, however, one diculty. The possibly unpleasant consequences of nite accuracy in line searches have already been pointed out. The nite number of decimal places to which data are stored implies that all calculation operations are subject to rounding errors, unless they are dealing with integers. Proofs of convergence, however, assume that the calculations are performed exactly. They therefore only

174

Comparison of Direct Search Strategies for Parameter Optimization

hold for the idealized concept of an algorithm, not for a particular computer program. The susceptibility of a strategy to rounding errors depends on how it is coded. Thus, for this reason too there is a need to check the convergence properties of numerical methods experimentally. Because of the nite word length of a digital computer the number range is also limited. If it is exceeded, the program that is running normally terminates. Such fatal execution errors (oating overow, oating divide check), are usually the consequence of rounding errors in previous steps if the error is in going below the absolutely smallest number value (oating underow) it is not regarded as fatal. Only few algorithms, e.g., Brent (1973), take special account of nite machine accuracy. In spite of the frequent mention of the importance of numerical comparisons of strategies, few publications to date have reported results on several di erent test problems using a large number of minimization methods. By virtue of its scope, the work of Colville (1968, 1970) stands out among the older studies by Brooks (1959), Spang (1962), Dickinson (1964), Leon (1966a), Box (1966), and Kowalik and Osborne (1968). It included 30 strategies and 8 di erent problems, but not many direct search methods compared to gradient methods. In some other tests by Jacoby, Kowalik, and Pizzo (1972), Himmelblau (1972a), Smith (1973), and others in the collection of Lootsma (1972a), derivative-free strategies receive much more attention. The comparisons of Gorvits and Larichev (1971) and Larichev and Gorvits (1974) treat only gradient methods, and that of Tapley and Lewallen (1967) deals with some schemes for the numerical treatment of functional optimization problems. The huge collection of test problems of Hock and Schittkowski (1981) is biased towards standard methods of mathematical programming and their capabilities (Schittkowski, 1980).

6.3.1 Computer Used

The machine on which the numerical experiments were carried out was a PDP 10 from the rm Digital Equipment Corporation, Maynard, Massachusetts. It had the following specications: Core storage area: 64K (1K = 1024 words) Word length: 36 bits Cycle time: 1.65 or 1.8 s The time-sharing operating system accounted for about 34K of core, so that only 30K remained available to the user. To tackle some problems with as many variables as possible, the computations were generally worked only to single
precision. The main 2 n K words, program, which was the same for all strategies, occupied about 5 + 1024 and the FORTRAN library a further 5K. The consequent maximum number nmax of parameters is given for each search method under test in Table 6.2. The nite word length of a digital computer means that its number range is limited. The absolute bounds for oating point arithmetic were given by: Largest absolute number: 2127 ' 1:7  1038 Smallest absolute number: 2 128 ' 2:9  10 39 ;

;

Numerical Comparison of Strategies

175

Only a part of the word is available for the mantissa of a number. This imposed the di erential accuracy limit, which is much lower and usually more important: Smallest di erence relative to unity: 2 27 ' 7:5  10 9 Accordingly the following equalities hold for this computer: ;

" = 0 1 + " = 1

;

for j"j < 2 for j"j < 2

128 27

; ;

These computer-specic data play a r^ole when testing for zero or for the equality of two quantities. The same programs can therefore lead to di erent results on di erent computers. Strategies are often judged by the computation time they require to achieve a result, for example, with a specied accuracy. The basic quantity for this purpose is the occupation time of the central processor unit (CPU). It also depends on the machine. Word lengths and cycle times are not enough to allow comparison between runs that were made on di erent computers. So-called MIX-times, which are average values of the duration of certain operations, also prove to be unsuitable, since the speed of calculation is so strongly dependent on the frequency of its individual steps. A method proposed by Colville (1968) has received wide recognition. Its design was particularly suited to optimization methods. According to this scheme, measured computation times are expressed relative to the time required for 10 consecutive inversions of a 40 40 matrix, using the FORTRAN program written by Colville. In our case this unit was around 110 seconds. Because of the timesharing operation, with its rather variable load on the PDP 10, there were deviations of 10% and above on the reported CPU times. This was especially marked for short programs.

6.3.2 Optimization Methods Tested

One goal of this work is to compare evolution strategies with other derivative-free methods of continuous parameter optimization. To this end we consider not only direct search methods in the narrower sense, but also those methods that glean their required knowledge of partial derivatives by means of trial steps and nite di erence methods. Altogether 14 strategies or versions of basic strategies are considered. Their names and abbreviations used for them are listed in Table 6.2. All tests were run on the PDP 10 mentioned in the previous section. Finite computer accuracy implies that in the case of quadratic objective functions the iteration process could or should not be continued until the exact solution has been obtained. The decision when to terminate the optimum search is a necessary and often crucial component of any iterative method. Just as the procedures of the individual strategies di er, so too do their termination or convergence criteria. As a rule, the user is given the chance to exert an inuence on the termination criterion by means of an input parameter dened as the required accuracy. It refers either to the values of the variables (change in xi within one iteration or size of the step lengths si) or to values of the objective function. Both criteria harbor the danger that the search will be terminated

176

Comparison of Direct Search Strategies for Parameter Optimization

prematurely, that is before arriving as close to the objective as is required. This is made clear by Figure 6.1. Neither 4x < "x nor 4 F < "F alone are sucient conditions for being close to the solution x . The condition krF k < "g , which is often applied for gradient methods, can lead to termination of the search near a saddle point and is in any case not always appropriate in the presence of constraints or discontinuities. Thus the e ectiveness of a convergence criterion is always closely linked to the procedure of a particular strategy and not automatically transferable to other strategies. Since each method converges to the optimum along a di erent path, in spite of having the same required accuracy, di erent methods do not nish the search with the same result. The termination criteria are also tested at di erent points in time and not always with the same frequency. These factors make it more dicult to compare the test results of di erent methods. For this reason Himmelblau replaces the strategy specic termination criteria by the tests kx(k) ; x k < "1 and F (x(k)) ; F (x ) < "2 after each iteration. He thereby obtains results that can be compared quite easily, but they are valid only for strategies deprived of one of their major components. We have retained here the original termination criteria of all the search methods. The required accuracies were set as high as the computer permitted. The actual values used are given in Table 6.2. Their meaning can be found in the description of the strategies in Chapter 3. 





F (x)

F

x x

x*

Figure 6.1: The adequacy of termination criteria

Numerical Comparison of Strategies

177

Table 6.2: Strategies applied: their abbreviations, maximum number of variables and accuracy parameters Strategy Coordinate strategy with Fibonacci search Coordinate strategy with golden section Coordinate strategy with Lagrangian interpolation Direct search of Hooke and Jeeves Davies-Swann-Campey method with Gram-Schmidt orthogonalization Davies-Swann-Campey method with Palmer orthogonalization Powell's method of conjugate directions Stewart's modication of the Davidon-Fletcher-Powell method Simplex Method of Nelder and Mead Method of Rosenbrock with Gram-Schmidt orthogonalization Complex method of Box (1 + 1) Evolution strategy (10 100) Evolution strategy (10 100) Evolution strategy with recombination

Abbreviation Maximum number of variables FIBO 2900

" = 7:5  10

9

GOLD

2910

" = 7:5  10

9

LAGR

2540

" = 7:5  10

9

HOJE

4090

" = 7:5  10

9

DSCG

75

" = 7:5  10

9

DSCP

95

" = 7:5  10

9

POWE

135

" = 7:5  10

9

DFPS

180

"a = "b = "c = 7:5  10 9

SIMP

135

" = 10

;

ROSE

75

" = 10

;

COMP EVOL

95 4000 ) ) 435 ) 435 ) )

GRUP REKO

zValues xed by the author. yIn place of the values set in Lill's program:

Accuracy parameter ;

;

;

;

;

;

;

;

y

8 4

z

z

" = 10 6 "a = "c = 3:0  10 39 ;

z

;

"b = "d = 7:5  10 9 ;

= 10;6 "b = 10;10 "c = 5 10;13: The maximum number of variables refers to an available core storage area of 30K words, which includes the main program and the FORTRAN library. "a




178

Comparison of Direct Search Strategies for Parameter Optimization

Besides their considerable cost in programming and computation time, numerical strategy comparisons entail further diculties. The e ectiveness of a method can be strongly inuenced by small programming details. A number of methods were not fully worked out by their originators and require heuristic rules to be introduced before they can be applied. The way in which this degree of freedom is exercised to dene the procedure depends on the skill and experience of the programmer, which leads to large discrepancies between the results of investigations and the judgements of di erent authors on one and the same strategy. We have therefore, as far as possible, used already published programs (FORTRAN or ALGOL) for the algorithms or parts of them under study:

 One dimensional search with the Fibonacci method of Kiefer: M. C. Pike, J. Pixner (1965) J. Boothroyd (1965) M. C. Pike, I. D. Hill, F. D. James (1967)

Algorithm 2, Fibonacci search Certication of Algorithm 2 Note on Algorithm 2

 One dimensional search with the golden section method of Kiefer: K. J. Overholt (1967)

Algorithm 16, Gold

 Direct search (pattern search) of Hooke and Jeeves:

A. F. Kaupe, Jr. (1963) Algorithm 178, direct search M. Bell, M. C. Pike (1966) Remark on Algorithm 178 R. DeVogelaere (1968) Remark on Algorithm 178 F. K. Tomlin, L. B. Smith (1969) Remark on Algorithm 178 L. B. Smith (1969) Remark on Algorithm 178

 Orthogonalization method for the strategies of Rosenbrock and of Davies, Swann, and Campey: J. R. Palmer (1969)

An improved procedure for orthogonalizing the search vectors in Rosenbrock's and Swann's direct search optimization methods  Derivative-free method of conjugate directions of M. J. D. Powell: M. J. Hopper (1971) Harwell subroutine library. A catalogue of subroutines, from which subroutine VA04A, updated May 20, 1970 (received as a card deck).  Variable metric method of Davidon, Fletcher, and Powell as formulated by Stewart: S. A. Lill (1970) Algorithm 46. A modied Davidon method for nding the minimum of a function, using di erence approximation for the derivatives.

Numerical Comparison of Strategies

179

S. A. Lill (1971) Note on Algorithm 46 Z. Kovacs (1971) Note on Algorithm 46 Some of the parameters a ecting the accuracy were altered, either because the small values dened by the author could not be realized on the available computer or because the closest possible approach to the objective could not have been achieved with them.  Simplex method of Nelder and Mead: R. O'Neill (1971) Algorithm AS 47, function minimization using a simplex procedure  A complete program for the Rosenbrock strategy: M. Machura, A. Mulawa (1973) Algorithm 450, Rosenbrock function minimization This was not applied because it could only treat the unconstrained case.  The same applies to the code for the complex method of M. J. Box: J. A. Richardson, J. L. Kuester Algorithm 454, the complex method for con(1973) strained optimization The part of the strategy that, when the starting point is not feasible seeks a basis in the feasible region, is not considered here. Whenever the procedures named were published in ALGOL they have been translated into FORTRAN. All the other optimization strategies not mentioned here have also been programmed in FORTRAN, with close reference to the original publications. If one wanted to repeat the test series today, a much larger number of codes could be made use of from the book of More and Wright (1993).

6.3.3 Results of the Tests

6.3.3.1 First Test: Convergence Rates for a Quadratic Objective Function

In the rst part of the numerical strategy comparison the theoretical predictions of convergence rates and Q-properties will be tested, or, where these are not available, experimental data will be supplied instead. For this purpose two quadratic objective functions are used (Appendix A, Sect. A.1). In the rst (Problem 1.1) the matrix of coecients is diagonal with unit diagonal elements, i.e., a scalar matrix. This simplest of all quadratic problems is characterized by concentric contour lines or surfaces that can be represented or imagined as circles in the two parameter case, spheres in the three parameter case, and surfaces of hyperspheres in the general case. The same pattern of contours but with arbitrary monotonic variation in the objective function occurs in the sphere model for which the average rates of progress of the evolution strategies could be determined theoretically (Rechenberg, 1973 and Chap. 5 of this book). The second objective function (Problem 1.2) has a matrix of coecients with all nonzero elements. It represents a full quadratic problem (except for the missing linear term)

180

Comparison of Direct Search Strategies for Parameter Optimization

with concentric, oblique ellipses, or ellipsoids as the contour lines or surfaces. The condition number of the matrix of coecients increases quadratically with the number of parameters (see Appendix A, Sect. A.1). In general, the time required to calculate one value of the objective function increases as O(n2 ) for a quadratic problem, because, for a full form matrix, n2 (n + 1) distinct second order terms aij xi xj must be evaluated. The objective function of Problem 1.2 has been formulated with the intention of reducing the computation time per function call to O(n), without it being such a particular quadratic problem that one of the strategies could nd it especially advantageous. The strategy comparison for this problem could thereby be made for much larger numbers of variables for the prescribed maximum computation time (Tmax = 8 hours). The storage requirement for the full matrix A would also have been an obstacle to numerical tests with many parameters. To enable comparison of the experimental and theoretical results, the required number of iterations, line searches, orthogonalizations, objective function calls, and the computation time were measured in going from the initial values (; 1)i  p x(0) = x + for i = 1(1)n i i n to an approximation  (k)  1 x(0) ; x   for i = 1(1)n xi ; xi   i 10 i The interval of uncertainty of the variables thus had to be reduced by at least 90%. The distance covered is e ectively independent of the number of variables. The above conditions were tested after each iteration, and as soon as they were satised the search was terminated. The convergence criteria of the strategies themselves were not suppressed, but they could not generally take e ect as they were much stricter. If they did actually operate it could be regarded as a failure of the method being applied. The results of the rst test are given in Tables 6.3 and 6.4. The number of function calls and the number of iterations or other characteristic processes involved are displayed in Figures 6.2 to 6.13 as a function of the number of parameters n on a log-log scale. As the data show, the computation time and e ort of a strategy increase sharply with n. The large range in the number of variables compared to other investigations allows the trends to be seen clearly. To facilitate an overall view, the computation times of all the strategies are plotted as a function of the number of variables in Figures 6.14 and 6.15. 





Numerical Comparison of Strategies

181

Table 6.3: Results of all strategies for test Problem 1.1 FIBO{Coordinate strategy with Fibonacci search Number of variables Number of cycles Number of objective Computation time function calls in seconds 3 6 10 20 30 60 100 200 300 600 1000 2000 (max) 2900

1 1 1 1 1 1 1 1 1 1 1 1 1

158 278 456 866 1242 2426 3870 7800 10562 21921 38701 67451 103846

0.13 0.28 0.53 1.66 3.07 10.7 26.5 106 210 826 2500 8270 19300

1 cycle = n line searches

GOLD{Coordinate strategy with golden section Number of variables Number of cycles Number of objective Computation time function calls in seconds 3 6 10 20 30 60 100 200 300 600 1000 2000 2900 1 cycle = n line searches

1 1 1 1 1 1 1 1 1 1 1 1 1

158 279 458 866 1242 2426 3870 7802 10562 21921 38703 67431 103834

0.10 0.22 0.51 1.48 3.14 11.3 27.6 114 221 808 2670 8410 18300

182

Comparison of Direct Search Strategies for Parameter Optimization

Table 6.3 (continued) LAGR{Coordinate strategy with Lagrangian interpolation Number of variables Number of cycles Number of objective Computation time function calls in seconds 3 6 10 20 30 60 100 200 300 600 1000 2000 (max) 2540

1 1 1 1 1 1 1 1 1 1 1 1 1

85 163 271 521 781 1561 2501 5001 7201 14401 24001 46803 64545

0.04 0.12 0.30 0.88 1.80 6.68 17.3 68.6 153 546 1620 6020 10300

1 cycle = n line searches HOJE-Direct search of Hooke and Jeeves Number of variables Number of cycles Number of objective Computation time function calls in seconds 3 6 10 20 30 60 100 200 300 600 1000 2000 3000 (max) 4090 1 cycle = n to 2 n individual steps

4 4 3 7 3 8 2 8 9 7 12 9 10 13

20 43 48 274 168 874 352 3104 4954 7503 23505 35003 58504 104300

0.02 0.04 0.06 0.50 0.43 3.70 2.37 40.1 100 286 1460 4270 11200 25600

Numerical Comparison of Strategies

183

Table 6.3 (continued)

DSCG{Davies-Swann-Campey method with Gram-Schmidt orthogonalization Number of Number of Number of line Number of objec- Computation time variables orthog. searches tive function calls in seconds 3 6 10 20 30 50 (max) 75

0 0 0 0 0 0 0

3 6 10 20 30 50 75

20 34 56 111 136 226 338

0.04 0.10 0.20 0.68 1.18 2.80 6.10

DSCP{Davies-Swann-Campey method with Palmer orthogonalization Number of Number of Number of line Number of objec- Computation time variables orthog. searches tive function calls in seconds (max) 95

0

95

428

9.49

Results for n
75 identical to those of DSCG in addition.

POWE{Powell's method of conjugate directions Number of Number of Number of line Number of objec- Computation time variables iterations searches tive function calls in seconds 3 6 10 20 30 60 100 (max) 135 1 complete iteration =

1 1 1 1 1 1 1 1 n

3 6 10 20 30 60 100 135

11 20 32 62 92 182 202 407

0.02 0.06 0.12 0.32 0.60 1.96 3.72 8.60

+ 1 line searches included are all the iterations begun

184

Comparison of Direct Search Strategies for Parameter Optimization

Table 6.3 (continued) DFPS{Stewart's modication of the Davidon-Fletcher-Powell method Number of variables Number of iterations Number of objective Computation time function calls in seconds 3 6 10 20 30 60 100 135 (max) 180

1 1 1 1 1 1 1 1 1

10 16 24 44 64 124 204 274 364

0.02 0.04 0.06 0.16 0.32 1.14 3.19 5.42 9.56

1 iteration = 1 gradient evaluation and 1 line search SIMP{Simplex method of Nelder and Mead (with restart) Number of variables Number of restarts Number of objective Computation time function calls in seconds 3 6 10 20 30 60 100 (max) 135

0 0 0 0 0 0 0 1

28 104 138 301 664 1482 1789 5142

0.09 0.64 1.49 8.24 37.4 277 862 5270

ROSE{Rosenbrock's method with Gram-Schmidt orthogonalization Number of variables Number of orthog. Number of objective Computation time function calls in seconds 3 6 10 20 30 40 50 60 (max) 75

1 2 2 1 0 1 2 2 2

27 60 120 181 121 281 550 600 899

0.08 0.32 0.91 2.56 1.18 13.7 48.4 78.3 145

Numerical Comparison of Strategies

185

Table 6.3 (continued)

COMP{Complex method of Box (2 n vertices) Number of variables Number of objective Computation time function calls in seconds 3 6 10 20 30 60 (max) 95

69 259 535 1447 2621 7263 14902

0.22 1.62 6.72 72.0 211 2240 11000

All numbers are averages over several attempts.

EVOL{(1+1) evolution strategy (average values) Number of variables Number of mutations Computation time in seconds 3 6 10 20 30 60 100 150 200 300 600 1000

49 154 224 411 630 1335 2192 3322 4232 6666 13819 23607

0.17 0.79 1.74 6.47 14.0 60.0 149 340 565 1310 5440 15600

Maximum number of variables (4,000) not achieved because too much computation time required. Number of objective function calls = 1 + number of mutations

186

Comparison of Direct Search Strategies for Parameter Optimization

Table 6.3 (continued) GRUP{(10 , 100) evolution strategy (average values) Number of variables Number of generations Computation time in seconds 3 6 10 20 30 60 100 200 300 (max) 435

4 10 17 37 55 115 194 377 551 854

1.81 6.75 16.8 64.5 145 519 1600 5720 13600 28300

Number of objective function calls: 10 + 100 times number of generations. REKO{(10 , 100) evolution strategy with recombination (average values) Number of variables Number of generations Computation time in seconds 3 6 10 20 30 60 100 200 300 (max) 435 Number of objective function calls: 10 + 100 times number of generations.

4 6 13 23 34 53 84 136 180 289

2.67 7.42 23.3 82.5 177 514 1420 4380 9340 21100

Numerical Comparison of Strategies

187

Table 6.4: Results of all strategies for test Problem 1.2 FIBO{Coordinate strategy with Fibonacci search Number of variables Number of cycles Number of objec- Computation time tive function calls in seconds 3 6 10 20 30 50 60 100

8 928 22 4478 40 12644 87 50265 132 110423 227 297609 282 422911 Search terminates prematurely

0.68 4.44 15.6 102 298 1290 2120

GOLD{Coordinate strategy with golden section Number of variables Number of cycles Number of objec- Computation time tive function calls in seconds 3 6 10 20 30 50 60 100

8 946 22 4418 40 12622 86 50131 133 111219 226 296570 279 423471 Search terminates prematurely

0.61 3.96 14.5 102 287 1330 2040

LAGR{Coordinate strategy with Lagrangian interpolation Number of variables Number of cycles Number of objec- Computation time tive function calls in seconds 3 6 10 20 30 60 100 150

8 586 22 2826 40 8023 87 32452 134 70889 272 263067 519 703130 Search terminates prematurely

0.39 2.48 9.55 62.8 192 1320 5770

188

Comparison of Direct Search Strategies for Parameter Optimization

Table 6.4 (continued) HOJE{Direct search of Hooke and Jeeves Number of variables Number of cycles Number of objec- Computation time tive function calls in seconds 3 6 10 20 30 60 100 200 300

11 30 26 78 111 212 367 727 1117

65 353 502 3035 6443 24801 71345 284060 656113

0.04 0.34 0.62 5.70 16.3 119 547 4270 14800

DSCG{Davies-Swann-Campey method with Gram-Schmidt orthogonalization Number of Number of Number of line Number of objec- Computation time variables orthog. searches tive function calls in seconds 3 6 10 20 30 40 50 60 (max) 75

3 7 8 16 21 42 27 44 87

16 55 101 361 691 1802 1451 2822 6676

87 195 323 1209 2181 5883 4453 9308 20365

0.22 0.87 2.70 29.2 110 484 582 1540 5790

DSCP{Davies-Swann-Campey method with Palmer orthogonalization Number of Number of Number of line Number of objec- Computation time variables orthog. searches tive function calls in seconds 3 6 10 20 30 50 75 (max) 95

3 7 8 16 28 28 79 100

16 55 101 361 901 1501 6076 9691

84 194 324 1208 2809 4610 18591 29415

0.22 0.78 1.54 10.3 33.8 89.7 547 1090

Numerical Comparison of Strategies

189

Table 6.4 (continued)

POWE{Powell's method of conjugate directions Number of Number of Number of line Number of objec- Computation time variables iterations searches tive function calls in seconds 3 6 10 20 30 40 50 60 9 70 > > > > 80 > = 90 > > 100 > > > (max) 135

3 5 9 17 53

11 27 35 77 99 215 354 744 1621 3401 search becomes innite { no convergence 175 8864 21532 138 8367 19677

0.08 0.30 0.97 4.82 24.1 235 222

search becomes innite { no convergence

DFPS{Stewart's modication of the Davidon-Fletcher-Powell method Number of Number of Number of objec- Computation time Fatal errors variables iterations tive function calls in seconds 3 6 10 20 30 60 100 135 (max) 180

3 4 5 7 9 13 17 20 22

20 41 74 178 333 926 2003 3190 4757

0.04 0.14 0.34 1.36 3.63 19.7 67.9 145 270

2 oating divide checks

190

Comparison of Direct Search Strategies for Parameter Optimization

Table 6.4 (continued) SIMP{Simplex method of Nelder and Mead (with restart) Number of Number of Number of objec- Computation time variables restarts tive function calls in seconds 3 6 10 20 30 40 50 60 70

0 1 0 0 0 2 1 1 1

29 173 304 2415 8972 28202 53577 62871 86043

0.09 1.06 3.17 77.6 579 3030 8870 13700 25800

ROSE{Rosenbrock's method with Gram-Schmidt orthogonalization Number of Number of Number of objec- Computation time variables orthog. tive function calls in seconds 3 6 10 20 30 40 50 60 (max) 75

3 4 8 12 14 19 21 23 34

38 182 678 2763 5499 10891 15396 20911 43670

0.12 0.82 4.51 35.6 114 329 645 1130 3020

COMP{Complex method of Box (2 n vertices) Number of Number of objective Computation time variables function calls in seconds 3 6 10 20 30 40

60 0.21 302 2.06 827 12.0 5503 235 24492 2330 Search sometimes terminates prematurely Search always terminates prematurely

All numbers are averages over several attempts

Numerical Comparison of Strategies

191

Table 6.4 (continued) EVOL{(1+1) evolution strategy (average values) Number of variables Number of mutations Computation time in seconds 3 6 10 20 30 60 100 150

85 213 728 2874 5866 24089 69852 152348

0.33 1.18 6.15 44.4 136 963 4690 15200

GRUP{(10 , 100) evolution strategy (average values) Number of variables Number of generations Computation time in seconds 3 6 10 20 30 50 80 100

5 14 53 183 381 1083 2977 4464

2.02 9.36 49.4 326 955 4400 18600 35100

REKO{(10 , 100) evolution strategy with recombination (average values) Number of variables Number of generations Computation time in seconds 3 6 10 20 30 40

6 15 42 162 1322 9206

2.44 18.9 76.2 546 6920 61900

Figures 6.2 to 6.13 translate the numerical data into vivid graphics. The abbreviations used here are: OFC stands for objective function calls ORT stands for orthogonalizations The parameters 1.1 and 1.2 refer to Problems 1.1 and 1.2 as mentioned above.

192

Comparison of Direct Search Strategies for Parameter Optimization

Figure 6.2: Coordinate strategy with Fibonacci search

Figure 6.3: Coordinate strategy with golden section

Numerical Comparison of Strategies

Figure 6.4: Coordinate strategy with Lagrangian interpolation

Figure 6.5: Strategy of Hooke and Jeeves

193

194

Comparison of Direct Search Strategies for Parameter Optimization

.

Figure 6.6: Strategy of Davies, Swann, and Campey with Gram-Schmidt orthogonalization

Figure 6.7: Strategy of Davies, Swann, and Campey with Palmer orthogonalization

Numerical Comparison of Strategies

195

no convergence

no convergence

Figure 6.8: Strategy of Powell with conjugate directions

Figure 6.9: Strategy of Davidon, Fletcher, Powell, and Stewart as formulated by Lill (variable metric)

196

Comparison of Direct Search Strategies for Parameter Optimization

Figure 6.10: Strategy of Rosenbrock with Gram-Schmidt orthogonalization

Figure 6.11: Left: Simplex strategy of Nelder and Mead, Right: Complex strategy of Box

Numerical Comparison of Strategies

197

Mutations 1.2

Mutations 1.1

Figure 6.12: (1+1) evolution strategy

Figure 6.13: Left: (10 , 100) evolution strategy without recombination, Right: (10 , 100) evolution strategy with recombination

198

Comparison of Direct Search Strategies for Parameter Optimization Computation time (sec)

10

1

(Number of variables)

2

COMP

SIMP

ROSE

10

0

GRUP

−1 10

EVOL

−2 10

−3 10

...................

FIBO HOJE LAGR DSCP/ DSCG POWE DFPS

Number of variables

−4 10 10

0

10

1

10

2

(10,100) evolution strategy without recombination (1+1) evolution strategy Complex strategy of Box Strategy of Rosenbrock Simplex strategy of Nelder and Mead Strategy of Davidon−Fletcher−Powell−Stewart (10,100) evolution strategy, parallel

10

3

Strategy of Powell DSC−Strategy (Palmer−Orthog.) DSC−Strategy (Gram−Orthog.) Strategy of Hooke and Jeeves Coordinate / Lagrange Coordinate / Fibonacci

.......

Figure 6.14: Result of the rst comparison test: computation times for Problem 1.1

−4 10

Numerical Comparison of Strategies

199

Computation time (sec) (Number of variables) 10

3

COMP

0

SIMP

DSCG 10

−1

ROSE

GRUP

−2

FIBO/ GOLD

10

LAGR EVOL

DSCP

−3 10

POWE HOJE .................................

10

−4

DFPS Number of Variables

−5 10 10

0

10

1

10

Figure 6.15: Result of the rst comparison test: computation times for Problem 1.2 Meanings of the symbols as in Figure 6.14

2

10

3

200

Comparison of Direct Search Strategies for Parameter Optimization

Points that deviated greatly from the trends have been omitted. To emphasize the di erences between the methods, instead of the computation time T the quantities T=n2 for Problem 1.1 and T=n3 for Problem 1.2 have been plotted on a logarithmic scale. For solving Problem 1.1 nearly all strategies require computation times of the order of O(n2 ). This corresponds to O(n) objective function calls, each requiring O(n) computation time. As expected, the most successful methods are the two that theoretically show quadratic convergence, namely the method of conjugate directions (Powell) and the variable metric method (DFPS). They obtain the solution within one iteration and n line searches respectively. For this simple problem, however, the same can be said for strategies with cyclic variation of the variables, since the search directions are the same. Of the three coordinate methods, the one with quadratic interpolation is a bit faster than the two which use sequential interval division. The latter two are of equal merit. The strategy of Davies, Swann, and Campey (DSC) also performs very well. Since the objective is reached within the rst n line searches, no orthogonalizations need to be carried out. For this reason too both versions yield identical results for n  75. The evolution strategies live up to expectations in so far as the number of mutations or generations increases linearly with n. The number of objective function calls and the computation times are, however, considerably higher than those of the previously mentioned methods. For r(0)=r(M ) = 10 the approximate theory of the two membered evolution strategy with optimal step length control predicts the number of mutations to be M ' (5 ln 10) n ' 11:5 n In fact nearly twice as many objective function calls (about 22 n) are required. This is partly because of the discrete way in which the variances are adjusted and partly because the chosen reduction factor of 0.85 corresponds to a success rate below the optimal value of 0.27. The ASSRS (adaptive step size random search) method of Schumer and Steiglitz (1968), which resembles the simple evolution strategy, is presently the most e ective random method as far as we know. According to the experimental results of Schumer (1967) for Problem 1.2, taking into account the di erent initial and nal conditions, it requires about the same number of steps as the (1+1) evolution strategy. It is noteworthy that the (10 , 100) strategy without recombination only takes about 10 times as much time as the (1+1) method, in spite of having to execute 100 mutations per generation. This factor of acceleration is signicantly higher than the theory for a (1 , 10) version would indicate and is closer to the calculated value for a (1 , 30) strategy. In the case of many variables, recombination further reduces the required number of generations by two thirds. This is less apparent in the computation time that is increased by the extra arithmetic operations, compared to the relatively inexpensive calculation of one objective function value. Thus, in the gures showing computation times only the (10 , 100) evolution without recombination has been included. The strategy of Hooke and Jeeves appears to require computation times rather more than O(n2) on average for many variables, nearer O(n2:2). This arises from the slight increase with n of the number of exploratory moves. The likely cause is the xed initial step length, which for problems with many variables is signicantly too big and must rst be reduced to the appropriate size. Three search strategies exhibit strikingly di erent behavior.

Numerical Comparison of Strategies

201

The method of Rosenbrock requires computation times on the order of O(n3 ). This can be readily understood. Up to the single exception of n = 30, in each case one or two orthogonalizations are performed. The Gram-Schmidt method employed performs O(n3) operations. If the number of variables is large the orthogonalization time is of major signicance whenever the time for one function call increases less than quadratically with the number of variables. One can see here that the number of objective function calls is not always sucient to characterize the cost of a strategy. In this case the DSC method succeeds with no orthogonalizations. The introduction of quadratic interpolation proves to give better results than the single step method of Rosenbrock. Computation times for the simplex and complex strategies also increase as n3, or even somewhat more steeply with n for many variables. The determining factor for the cost in this case is calculating the centroid of the simplex (or complex), about which the worst of the (n + 1) or 2 n vertices is reected. This process takes O(n2) additions. Since the number of reections and objective function calls increases as n, the cost increases, simply on this basis, as O(n3). Even in this simplest of all quadratic problems the simplex of the Nelder-Mead method collapses if the number of variables is large. To avoid premature termination of the optimum search, in the presently used algorithm for this strategy the simplex is initialized again. The search can thereby be prevented from stagnating in a subspace of IRn , but the required computation time increases even more rapidly than O(n3 ). The situation is even worse for the complex method of Box. The author suggests using 2 n vertices for problems with few variables and considers that this number could be reduced for many variables. However, the attempt to solve Problem 1.1 for n = 30 with a complex of 40 vertices fails in one of three cases with di ering sequences of random numbers: i.e., the search process ends before achieving the required approximation to the objective. For n = 40 and 50 vertices the complex collapsed prematurely in all three attempts. With 2 n vertices the complex strategy is successful up to the maximum possible number of variables, n = 95. Here again, however, for n > 30 the computation time increases faster than O(n3) with the number of parameters. It is therefore dubious whether the search would have been pursued to the point of reaching the maximum internally specied accuracy. The second order methods only distinguish themselves from other strategies for solving Problem 1.1 in that their required computation time

T = c n2 c = const: is characterized by a small constant of proportionality c. Their full capabilities should become apparent in solving the true quadratic problem (Problem 1.2). The variable metric method lives up to this expectation. According to theory it has the property Q n, which means that after n iterations, n2 line searches, and O(n3 ) computation time the problem should be solved. It comes as something of a surprise to nd that the numerical tests indicate a requirement for only about O(n0:5) iterations and O(n2:5) computation time. This apparent discrepancy between theory and experiment is explained if we note that the property Q n signies absolute accuracy within at most n iterations, while in this example only a nite reduction of the uncertainty interval is required. More surprising than the good results of the DFPS method is the behavior of the

202

Comparison of Direct Search Strategies for Parameter Optimization

strategy of Powell, which in theory is also quadratically convergent. Not only does it require signicantly more computation time, it even fails completely when the number of parameters is large. And in the case of n = 40 variables the step length goes to zero along a chosen direction. The convergence criterion is subsequently not satised and the search process becomes innite it must be interrupted externally. For n = 50 and n = 60 the Powell method does converge, but for n = 70 80 90 100 and 130 it fails again. The origin of this behavior was not investigated further, but it may well have to do with the objection raised by Zangwill (1967) against the completeness of Powell's (1964) proof of convergence. It appears that rounding errors combined with small step lengths in the one dimensional search can cause linearly dependent directions to be generated. However, independence of the n directions is the precondition for them to be conjugate to each other. The coordinate strategies also fail to converge when the number of variables in Problem 1.2 becomes very large. With the Fibonacci search and golden section as interval division methods they fail for n  100, and with quadratic interpolation for n  150. For successful line searching the step lengths would have to be smaller than allowed by the nite word length of the computer used. This phenomenon only occurs for many variables because the condition of the matrix of coecients in Problem 1.2 varies as O(n2). In this proportion the elliptic contour surfaces F (x) = const: become gradually more extended and the relative minimizations along the coordinate directions become less and less e ective. This failure is typical of methods with variation of individual parameters and demonstrates how important it can be to choose other search directions. This is where random directions can prove advantageous (see Chap. 4). Computation times for the method of Hooke and Jeeves and the method of DaviesSwann-Campey (DSC) clearly increase as O(n3) if Palmer orthogonalization is employed for the latter. For the method of Hooke and Jeeves this corresponds to O(n) exploratory moves and O(n2) function calls for the DSC method it corresponds to O(n) orthogonalizations and O(n2) line searches and objective function evaluations. The original Gram-Schmidt procedure for constructing mutually orthogonal directions requires O(n3) rather than O(n2) arithmetic operations. Since the type of orthogonalization seems to hardly alter the sequence of iterations, with the Gram-Schmidt subroutine the DSC strategy takes O(n4) instead of O(n3) basic operations to solve Problem 1.2. For the same reason the Rosenbrock method requires computation times that increase as O(n4). It is, however, striking that the single step method (Rosenbrock) in conjunction with the suppression of orthogonalization until at least one successful step has been made in each direction requires less time than line searching, even if only one quadratic interpolation is performed. In both these methods the number of objective function calls, which is of order O(n2), plays only a secondary r^ole. Once again the simplex and complex strategies are the most expensive. From n = 30, the method of Nelder and Mead does not come within the required distance of the objective without restarts. Even for just six variables the search simplex has to be re-initialized once. The number of objective function calls increases approximately as O(n3), hence the computation time increases as O(n5). The strategy of Box with 2 n vertices shows a correspondingly steep increase in the time with the number of variables. For n = 30

Numerical Comparison of Strategies

203

Problem 1.2 was actually only solved in one out of three attempts, and for n = 40 not at all. If the number of vertices of the complex is reduced to n + 10 the method fails from n = 20. As in Problem 1.1, the cost of the evolution strategies increases rather smoothly with the number of parameters{more so than for several of the deterministic search methods. To solve Problem 1.2, O(n2 ) objective function calls are required, corresponding to O(n3) computation time. Since the distance to be covered is no greater than it was in Problem 1.1, the greater cost must have been caused by the locally smaller curvatures. These are related to the lengths of the semi-axes of the contour ellipsoids. Because of the regular structure of the matrix of coecients A of the quadratic objective function in Problem 1.2, the condition number K , the ratio of greatest to least semi-axes (cf. test Problem 1.2 in Appendix A, Sect. A.1) 2  K = aamax min

can be considered as the only quantity of signicance in determining the geometry of the contour pattern. The remaining semi-axes will distribute themselves uniformly between amin and amax. The fact that K increases as O(n2) suggests that the rate of progress ', the average change in the distance from the objective per mutation or generation, only decreases as the square root of the condition number. There is so far no theory for the general quadratic case. Such a theory will also look more complicated, since apart from the ratio of greatest to smallest semi-axis a further n ; 2 parameters that determine the shape of the hyperellipsoid will play a r^ole. The position of the starting point will also have an e ect, although in the case of many variables only at the beginning of the search. After a transition phase the starting point of mutations will always lie in the vicinity of a point where the objective function contour surfaces are most curved. In the sphere model theory of Rechenberg, if r is regarded as the average local radius of curvature, the rate of progress at worst should become inversely proportional to the square root of the condition number. The convergence rate of the evolution strategy would then be comparable to that of the strategy of steepest descents, for which function values of two consecutive iterations in the quadratic case are in the ratio (Akaike, 1960) amax ; amin amax + amin Compared to other methods having costs in computation time that increase as O(n3 ), the evolution strategies fare better than they did in Problem 1.1. Besides the fact that the coordinate strategies do not converge at all when the number of variables becomes large, they are surpassed in speed by the two membered evolution strategy. The relative performance of the two membered and multimembered evolution strategies without recombination remains about the same. The behavior of the (10 , 100) evolution strategy with recombination deviates from that of the other versions. It requires considerably more computation time to solve Problem 1.2. This can be attributed to the fact that, although the probability distribution for mutation steps alters, it cannot adapt continuously to the local conditions. Whilst the mutation ellipsoid, the locus of all equiprobable mutation steps, can extend and contract

204

Comparison of Direct Search Strategies for Parameter Optimization

along the coordinate directions, it cannot rotate in the space. To do so, not only the variances but also the orientation or covariances would need to be variable (for such an extension see Chap. 7 and subroutine KORR). As the results show, starting from a spherical shape the mutation ellipsoid adopts a conguration that initially accelerates the search process. As it progresses towards the objective the ellipsoid must become smaller but it should also gradually rotate to follow the orientation of the contour lines. That is not possible because the mechanism adopted here allows no mutual dependence of the components of the random vector. The ellipsoid rst has to form itself into a sphere again, or to become generally small, before it extends again with the longer axes in new directions. This awkward process actually occurs, but it causes an appreciable delay in the search. There is a further undesirable phenomenon. Supposing that a single variance suddenly becomes very much smaller. The associated variation in the variables then takes place in an (n ; 1)-dimensional subspace of IRn (for a more detailed analysis see Schwefel, 1987). Other things being equal, the probability of a success is thereby greater than if all the parameters had varied. Step length alterations of this kind are therefore favored and, together with the resistance to rotation of the mutation ellipsoid, they enhance the unstable behavior of the strategy with recombination. This can be prevented by having a large population, in which there is always a sucient supply of di erent kinds of parameter combinations for the variances as well. Another possibility is to allow one individual to execute several consecutive mutations with one setting of the step length parameters. Then the overall success depends rather less on the instantaneous probability of success and more on the size of the partial successes. The quality of the strategy parameters is thereby assessed more objectively. It should be noticed that Problem 1.2 is actually the only one in which recombination appears troublesome. In many other cases it led to a reduction in the computation cost, even in the simple form applied here (see second and third test).

6.3.3.2 Second Test: Reliability Convergence in the quadratic case is a minimum requirement of non-linear optimization methods. The unsatisfactory results of the coordinate strategies and of Powell's method for a large number of variables conrm the necessity of numerical tests even when convergence is assured by theory. Even more important, in fact unavoidable, are experimental tests of the reliability of convergence of optimization methods on non-quadratic, non-linear problems. Some methods with an internal quadratic model of the objective function have to be modied in order to deal with more general problems. Such, for example, is the method of conjugate gradients. The method of Fletcher and Reeves (1964) actually terminates after the relative minimum has been obtained in each of n conjugate directions. However, for higher order objective functions the optimum will not have been reached after n iterations. Even in quadratic problems, if they are ill-conditioned, more iterations may be required. There are two possible ways to proceed. Either the iteration process can be formally continued beyond n line searches or it can be repeated in a cyclic way. Fletcher and Reeves recommend destroying all the accumulated information after each set of n + 1

Numerical Comparison of Strategies

205

iterations and beginning again, i.e., with uncorrected gradient directions. This procedure is said to be more e ective for non-quadratic objective functions. On the other hand, Fox (1971) suggests that a periodic restart of the search can prevent convergence in the quadratic case, whereas a simple continuation of the sequence of iterations is successful. Further suggestions for the way to restart are made by Fletcher (1972a). The situation is similar for the quasi-Newton methods in which the Hessian matrix or its inverse is approximated in discrete steps. Some of the proposed formulae for improving the approximation matrix can lead to division by zero sometimes due to rounding errors (Broyden, 1972), but in other cases even on theoretical grounds. If the Hessian matrix has singular points, the optimization process stagnates before reaching the optimum. Bard (1968) and others recommend as a remedy replacing the approximation matrix from time to time by the unit matrix. The information gathered over the course of the iterations is destroyed again in this process. Pearson (1969) proposes a restart period of 2 n cycles, while Powell (1970b) suggests regularly adding steps di erent from the predicted ones. It is thus still true to say of the property of quadratic termination that its \relevance for general functions has always been questionable" (Fletcher, 1970b). No guarantee is given that Newtonian directions are better than the (anti-) gradient. As there is no single objective function that can be taken as representative for determining experimentally the properties of a strategy in the non-quadratic case, as large and as varied a range of problem types as possible must be included in the numerical tests. To a certain extent, it is true to say that the greater their number and the more skillfully they are chosen, the greater the value of strategy comparisons. Some problems have become established as standard examples, others are added to each experimenter's own taste. Thus in the catalogue of problems for the second series of tests in the present strategy comparison, both familiar and new problems can be found the latter were mainly constructed in order to demonstrate the limits of usefulness of the evolution strategies. It appears that all the previously published tests use as a basis for judging performance the number of function calls (with objective function, gradient, and Hessian matrix weighted in the ratio 1 : n : n2 (n +1)) and the computation time for achieving a prescribed accuracy. Usually the objective functions considered are several times continuously differentiable and depend on relatively few variables, and the results lack compatibility from problem to problem and from strategy to strategy. With one method, a rst minimum may be found very quickly, and a second much more slowly another method may work just the opposite way round. The abundance of individual results actually makes a comprehensive judgement more dicult. Hence average values are frequently calculated for the required computation time and the number of function calls. Such tests then result in establishing that second order methods are faster than rst order and these in turn are faster than direct search methods. These conclusions, which are compatible with the test results for quadratic problems, lead one to suspect that the selected objective functions behave quadratically, at least in the neighborhood of the objective. Thus it is also frequently noted that, at the beginning of a search, gradient methods converge faster, whereas towards the end Newton methods are faster. The average values that are measured therefore depend on the chosen starting point and the required closeness of approach to the objective.

206

Comparison of Direct Search Strategies for Parameter Optimization

The assessment is tricky if a method does not converge for a particular problem but terminates the search following its own criteria without getting anywhere near the solution. Any strategy that fails frequently in this way cannot be recommended for use in practice even if it is especially fast in other cases. In a practical problem, unlike a test problem, the correct solution is not, of course, known in advance. One therefore has to be able to rely on the results given by a strategy if they cannot be checked by another method. Hence, reliability is just as important a criterion for assessing optimization methods as speed. The second part of the strategy comparison is therefore designed to test the robustness of the optimization methods. The scale for assessing this is the number of problems that are solved by a given method. Since in this respect it is the complexity rather than size of the problem that is signicant, the number of variables ranges only from one to six. All numerical iteration methods in practice can only approximate a solution with a nite accuracy. In order to be able either to accept the end result of an optimum search as adequate, or to reject it as inadequate, a border must be dened explicitly, on one side of which the solution is exact enough and on the other side of which it is unsatisfactory. It is the structure of the objective function that is the decisive factor determining the accuracy that can be achieved (Hyslop, 1972). With this in mind the border values for the purpose of ranking the test results were obtained by the following scheme. Starting from the known exact or best solution

x = ( x1 x2 : : :  xn )T 







the variables were individually altered by the amounts (

 for xi = 0 4xi =

xi  for xi 6= 0 





in all combinations. For example for n = 2 one obtains eight di erent test values of the objective function (see Fig. 6.16). In the general case there are 3n ; 1 di erent values. The greatest deviation 4F () from the optimal value F (x ) denes the border between results that approach the objective suciently closely and results that do not. To obtain a number of grades of merit, four di erent test increments j  j = 1(1)4 were selected: 1 = 10 38 2 = 10 8 3 = 10 4 4 = 10 2 A problem is deemed to have been solved \exactly" at x~ if 

; ; ; ;

F (~x)  F (x ) + 4F (1) 

is attained. On the other hand, if at the end of the search

F (~x) > F (x ) + 4F (4) 

Numerical Comparison of Strategies

207

x2 Optimum

x1

Test position

Figure 6.16: Eight dierent test values of the objective function in case of n = 2

the strategy employed has failed. Three intermediate classes of approximation are dened in the obvious way. The maximum possible accuracy was required of all strategies. The corresponding free parameters of the strategies that enter the termination criteria have already been dened in Table 6.2. In contrast to the rst test, no additional common termination rule was employed. A total of 50 problems were to be solved. The mathematical formulations of the problems are given in Appendix A, Section A.2. Some of them are only distinguished by the chosen initial conditions, others by the applied constraints. Nine out of 14 strategies or versions of basic strategies are not suited to solving constrained problems, at least not directly. Methods involving transformations of the variables and penalty function methods were not employed. An exception is the method of Rosenbrock, which only alters the objective function near the boundaries and can be applied in one pass otherwise penalty functions require a sequence of partial optimizations to be executed. The second series of tests therefore comprises one set of 28 unconstrained problems for all 14 strategies and a second set of 22 constrained problems for 5 of the strategies. The results are displayed together in Tables 6.5 to 6.8. The approximation to the objective that has been achieved in each case is indicated by a corresponding symbol, using the classes of accuracy dened above. Any interesting features in the solution of individual problems are documented in the Appendix A, Section A.2, in some cases together with a brief analysis. Thus at this point it is only necessary to make some general observations about the reliability of the search methods for the totality of problems.

Unconstrained Problems

The results of the three versions of the coordinate strategies are very similar and generally unsatisfactory. A third of all the problems cannot be solved with them at all, or only very inaccurately. Exact solutions ( = 10 38 ) are the exception and only in less than a ;

208

Comparison of Direct Search Strategies for Parameter Optimization Table 6.5: Results of all strategies in the second comparison test, unconstrained problems

Problem No. 2.1 2.2 2.3 2.4 2.5 2.6 2.7 2.8 2.9 2.10 2.11 2.12 2.13 2.14 2.15 2.16 2.17 2.18 2.19 2.20 2.21 2.22 2.23 2.24 2.25 2.26 2.27 2.28 Sum

F I B O

G O L D

L A G R

3 3 3 3 2 2 3 3 2 2 5 5 5 5 5 5 3 3 5 5 5 5 5 5 3 3 3 3 3 3 2 1 2 2 5 5 5 5 2 2 5 5 2 2 1 1 3 3 3 3 1 1 1 1 4 4 91 90

3 3 3 3 2 5 4 4 3 4 5 5 3 3 3 2 1 5 5 2 5 2 1 3 3 2 5 4 93

H O J E

D S C G

D S C P

P O W E

D F P S

S I M P

R O S E

C O M P

1 1 1 2 1e 2n 1 2 1 1 1 1 2 2a 1a 2 1 1 1 1 1 1n 1a 4 1 1 1 2 2e 3 1 3 2 1 1 2 1e 2a 1 2 5 2 2 5 3 2a 1 3 2 5ea 5ea 5e 5 3 5 3 3 5ea 5ea 5e 5 3 5 2 2 2 2 2e 2 1a 3 1 3 2 2 5a 4e 4n 2 2 3 2 2 2 4 4n 2 2 3 2 3 2 4e 2 2 2 2 2 1 3 2 3 3 3 2 2 2 2a 5e 2n 2 2 2 2 2 2ea 3 2 2 2 2 1 1 2 2 2n 2 3 2 2 2 2e 2 1a 2 1 2 2 2 2e 2 1an 1 1 5 2 2 5 2e 2 2 3 2 3 2 2 1e 3n 2 2 2 4 2 5 2e 5a 2 5 2 2 2 2 5 5a 2 5 1 1 1 5a 5e 1a 1a 1 2 1 1 2 2 2 2 2 2 1 1 2e 2e 3 1 3 1 1 1 1e 1 1n 1 4 2 1 1 1 5 1 1 2 3 4 3 2 4e 2 3 1 61 56 52 74 79 65 54 68

Meaning of the number and letter symbols used above: 1 Accuracy achieved better than 10 38 2 Accuracy achieved better than 10 84 3 Accuracy achieved better than 10 2 4 Accuracy achieved better than 10 2 5 Accuracy achieved worse than 10 e Fatal execution error (oating overow, oating divide check) a Termination rule ineective search innite with no further convergence r Computation time too long or convergence too slow search terminated n Concerns the simplex method of Nelder and Mead: restart(s) required ;

;

;

;

;

E V O L

G R U P

R E K O

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 2 1 4 3 1 4 3 1a 4 3 1a 2 1 1 3r 3r 3r 3r 3r 3r 3 2 2 1 1 1 1 1 1 3 2 3 1 1 1 1 1 1 1 1 1 1 1 1 2 2 1 1 1 1 1 1 1 1 1 1 4 4 3 51 51 37

Numerical Comparison of Strategies

209

third of all cases are the end results good (  10 8 ). As already shown by the quadratic objective function models, it appears again that progress along the directions of the unit vectors becomes possible only in very small step lengths. The limit of smallest possible changes in the variables, as dened by the nite word and mantissa lengths of the digital computer, is often reached before the search has come suciently close to the objective. The three methods with rotating axes also behave similarly to one another, namely the strategies of Rosenbrock and of Davies, Swann, and Campey. Although the choice of orthogonalization method (Gram-Schmidt or Palmer) has a considerable e ect on the computation times it makes little di erence to the accuracies achieved. If \exact" solutions are required, all three methods prove useful in about 4 out of 10 cases. This proportion is doubled if the accuracy requirement is lowered by a grade. Two problems (Problems 2.7 and 2.8) are not solved by any of the three variants. In the Rosenbrock method, the search is ended a very long way from the objective, while in the DSC method a line search becomes innite. To prepare for the single quadratic interpolation it uses a subroutine for bounding the relative minimum in the chosen direction. In this case, however, the relative minimum is situated at innity thus, after some time, the range of numbers that can be handled by the computer is exceeded. It eventually makes a fatal execution error with the message: \oating overow." In most computers, a program would terminate at this point, but the PDP 10 continues the calculation using its largest number 2127 in place of the value that exceeded the number range. Nevertheless the bounding procedure does not end because in the DSC method any steps that do not change the value of the objective function are also regarded as successful. The convergence criterion is not tested within this subroutine, so the whole procedure becomes innite without any further change in value of the objective function. It must be terminated externally. The convergence criterion of the Rosenbrock method fails in three cases, in spite of the fact that the exact solutions have already been found. It is noted on the tables wherever fatal execution errors occur or the optimization does not terminate normally. With 11 or 12 exact results, and altogether 23 good results, these three rotating axes methods rank highly. Fatal errors occur especially frequently in applying the more \thoroughbred" methods, the method of Powell and the DFPS strategy. They are not always accompanied by termination diculties or bad nal results. The accuracies achieved have therefore been evaluated independently of the execution errors. Good approximations, of which there are 20 (Powell) and 16 (DFPS) out of 28, are also less frequent than in the orthogonalization strategies. In many cases both of these methods that are so advantageous in theory completely fail to approach the desired solution usually in the same problems that present diculties with the much simpler coordinate methods. Apart from failure of a line search because of a relative minimum at innity, the causes are:  The confusion of minima and saddle points because of ambiguity in quadratic interpolation (Problem 2.19 for the Powell strategy, Problem 2.27 for the variable metric method)  Discontinuities in the objective function or its derivatives (Problems 2.6, 2.21, 2.22)  A singular Hessian matrix (Problem 2.14 in the DFPS method) ;

210

Comparison of Direct Search Strategies for Parameter Optimization

However, even a completely regular, several times di erentiable objective function of 10th order (Problem 2.23) is not managed by either of the quadratically convergent strategies. Their concept of using all the data that can be accumulated during the iterations to adjust their internal quadratic model apparently leads to completely wrong predictions of favorable directions and step lengths if the function is of appreciably higher than second order. Not one of the other direct search methods fails on this problem in fact they all nd the exact solution. With Powell's method one can choose between two di erent convergence criteria. The di erence between the stricter one and the simple one is that the former displaces slightly the best position obtained after the sequence of iterations has ended normally and searches again for the minimum. The search is only nally terminated if both results are the same within the specied accuracy. Otherwise the search is continued after a line search in the direction of the di erence vector between the two solutions. Because of the extreme accuracy requirements in the present cases the search usually ends with the message that rounding errors in the objective function prevent any closer approach to the objective. In such cases no additional variation in the nal result is made. Even in other cases, the stricter convergence criterion only makes a very slight improvement of the results the grades of merit of the results are not changed at all. In four problems the search becomes innite because the step lengths vanish and the termination criterion is no longer tested. The search has to be terminated externally. Fatal execution errors occur very frequently. In three cases there is a \oating overow" and in seven cases a \oating divide check." This concerns a total of eight problems. The DFPS strategy is even more susceptible. There are ve occurrences of \oating overow" and eleven of \oating divide check." Twelve problems are involved. In contrast, the direct search of Hooke and Jeeves works without errors, but even this method fails on two problems one because of sharp corners in the pattern of contour lines (Problem 2.6) and another in the neighborhood of a stationary point with a very narrow valley leading to the objective (Problem 2.19). Nevertheless it yields 6 exact solutions and 21 good approximations. The overall behavior of the simplex and complex strategies is similar, but there are di erences in detail. There are 17 good solutions together with 6 exact ones to set against two failures (Problems 2.21 and 2.22). These are provoked by edges on the contour surfaces in the multidimensional space. The restart rule in the Nelder-Mead method is invoked during 9 of the solutions. The termination criterion based only on function values at the simplex corners does not operate in 9 cases. The optimum search becomes innite with no apparent improvement in the objective function values. The results of the complex strategy depend strongly on the initial conguration, which is determined by random numbers. In this case the evaluation was made for the best of three attempts each with di erent sequences of pseudorandom numbers. It is especially worth noting the performance of the complex method in solving Problem 2.28, for which it is better than all the other methods. All three versions of the evolution strategy are distinguished by the fact that in no case do they completely fail, and they are able to solve far more than half of all the problems exactly (in the sense dened above). Since their behavior, like that of the complex method,

Numerical Comparison of Strategies

211

is inuenced by random numbers, the same rule was followed: namely, out of three tests the one with the best end result was accepted. In contrast to the strategy of Box, however, the evolution methods prove to be less dependent on the actual sequence of random numbers. This is especially true of the multimembered versions. Recombination almost always improves the chance of getting very close to the desired solutions. Fatal errors due to exceeding the maximum number range or dividing by zero do not occur by virtue of the simple computational operations in these strategies. Discontinuities in the partial derivatives, saddle points, and the like have no obvious adverse e ects. The search does, however, become rather time consuming when the minimum is reached via a long, narrow valley. The step lengths or variances that are set in this case are very small and impose slow convergence in comparison to methods that can perform a line search along the valley. The average rate of progress of an evolution strategy is not, however, a ected by bends in the valley, which would retard a one dimensional minimization procedure. Line searches only a ord a signicant advantage to the rate of progress if there are directions in the space along which successful steps can be made of a size that is large compared to the local radius of curvature of the objective function contour surface. Examples are provided by Problems 2.14, 2.15, and 2.28. In these cases, long before reaching the minimum the optimal variances of the evolution methods have reached the lower limit as determined by the machine accuracy. The desired solution cannot therefore be approximated to the required accuracy. In Problems 2.14 and 2.15 the computation time limit did not allow the convergence criterion to be satised although it was actually progressing slowly but surely, the search was terminated. Diculties with the termination rule based on function values only occurred in the solution of one type of problem (Problems 2.11, 2.12) using the (10 , 100) evolution strategy with recombination. The multimembered method selects the 10 best individuals of a generation only from the current 100 descendants. Their 10 parents are not included in the selection process, for reasons associated with the step length adaptation. In general, the objective function value of the best descendant is closer to the solution than that of the best parent. In the case of the two problems referred to above, this is initially the case. As the solution is approached, however, it happens more and more frequently that the best value occurring in a generation is lost again. This is related to the fact that because of rounding errors in evaluating values near the minimum, the objective function behaves practically stochastically. Thus the population wanders around in the neighborhood of the (quasi-singular) optimal solution without being able to satisfy the convergence criterion. These diculties do not beset the other search methods, including the multimembered evolution without recombination, because they do not come nearly so close to the optimum. The fact that the third problem of the same type (Problem 2.10) is solved without diculties in a nite time, even with recombination, can be considered a uke. Here too the minimum was reached long before the termination criterion was satised. On the whole, the multimembered evolution strategy with recombination is the surest and safest of all the search methods tested. In only 5 out of 28 cases is the solution not located exactly, and the greatest deviations of the variables were in the accuracy class  = 10 4 . ;

212

Comparison of Direct Search Strategies for Parameter Optimization Table 6.6: Summary of the results from Table 6.5 Strategy

Total number of problems No solution Fatal No normal solved in the accuracy class

or
> 10 2 computation termination 10 38 10 8 10 4 10 2 errors ;

;

FIBO GOLD LAGR HOJE DSCG DSCP POWE DFPS SIMP ROSE COMP EVOL GRUP REKO

y y y

3 4 2 6 11 12 4 5 7 11 5 17 18 23

;

;

;

9 9 7 21 23 24 20 16 18 23 17 20 22 24

18 18 17 26 24 26 21 18 24 26 24 24 27 28

19 19 21 26 26 26 21 22 26 26 26 28 28 28

9 9 7 2 2 2 7 6 2 2 2 0 0 0

0 0 0 0 2 2 8 12 0 0 0 0 0 0

0 0 0 0 2 2 4 0 9 3 0 0 0 2

Table 6.6 presents again a summary of the number of unconstrained problems that were solved with given accuracy by the search methods under test, together with the number of unsolved problems, the number of cases of fatal execution errors, and the number of cases in which the termination criteria failed.

Constrained Problems

Tables 6.7 and 6.8 show the results of 5 strategies in the 22 constrained problems. Execution errors such as exceeding the number range or dividing by zero did not occur in any case. Neither were there any diculties in the termination of the searches. The method of Rosenbrock can only be applied if the starting point of the search lies within the allowed or feasible region. For this reason the initial values of the variables in seven problems had to be altered. All other methods very quickly found a feasible solution to start with. As in the unconstrained problems, the strategies that depend on random numbers were each run three times with di erent sequences of random numbers. The best of the three results was accepted for evaluation. The results of the complex method and the two membered evolution turned out to be very variable in quality, whereas the multimembered versions of the strategy, especially with recombination, proved to be less inuenced by the particular random numbers. Two problems (Problems 2.40 and 2.41) caused great diculty to all the search methods. These are simple linear programs that can be solved rapidly and exactly by, for example, the simplex method of Dantzig. In ySearch terminated twice in each case due to too slow convergence

Numerical Comparison of Strategies

213

Table 6.7: Results of all strategies in the second comparison test, constrained problems Problem No. ROSE COMP EVOL GRUP REKO 2.29 2.30 2.31 2.32 2.33 2.34 2.35 2.36 2.37 2.38 2.39 2.40 2.41 2.42 2.43 2.44 2.45 2.46 2.47 2.48 2.49 2.50 Sum

3 1 3v 3v 3 1 3v 1 3 3v 3 5 5 3 3v 1 3 3 3v 3v 3 3 62

1 5 3 3 2 2 1 1 1 3 3 5 5 3 3 5 2 2 1 3 2 1 57

4 1 1 1 5 3 4 1 1 1 4 5 5 2 2 1 4 3 1 1 4 1 55

3 1 1 1 4 3 4 1 1 1 4 5 5 2 2 1 2 3 1 1 3 1 50

3 1 1 1 1 2 4 1 1 1 3 5 5 1 1 1 1 1 1 1 1 1 38

The meaning of the symbols is as in Table 6.5  \v" is used in connection with the Rosenbrock method for constrained cases: The starting point had to be displaced since it was not feasible for this method.

each case the closest to the objective was again the (10 , 100) evolution strategy with recombination, but even that result had to be classied as \no solution." On the whole the evolution methods cope with constrained problems no worse than the Rosenbrock or complex strategies, but they do reveal inadequacies that are not apparent in unconstrained problems. In particular the 1=5 success rule for adapting the variances of the mutation step lengths in the (1+1) evolution strategy appears to be unsuitable for attaining an optimal rate of convergence when several constraints become active. In problems with active constraints, the tendency of the evolution methods to follow the average gradient trajectory causes the search to come quickly up against one or more boundaries of the feasible region. The subsequent migration towards the objective along

214

Comparison of Direct Search Strategies for Parameter Optimization Table 6.8: Summary of the results from Table 6.7 Total number of problems solved with accuracy class

No solution Strategy 10 38 10 8 10 4 10 2 or
> 10 2 ;

ROSE COMP EVOL GRUP REKO

4 6 10 10 16

;

;

;

4 11 12 13 17

20 18 14 17 19

20 18 19 20 20

;

2 4 3 2 2

such edges takes considerable e ort and time. In Figure 6.17 the situation is illustrated for the case of two variables and one constraint. The contours of the objective function run at a narrow angle to the boundary of the region. For a mutation to count as successful it must fall within the feasible region as well as improve the objective function value. For simplicity let us assume that all the mutations fall on the circumference of a circle about the current starting point. In the case of many variables this point of view is very reasonable (see Chap. 5, Sect. 5.1). To start with the center of the circle (P1) will still lie some way from the boundary. If the angle between the contours of the objective function and the edge of the feasible region is small and the step size, or variance of the mutation step size, is large then only a small fraction of the mutations will be successful (thickly drawn part of the circle 1). The 1=5 success rule ensures that this fraction is raised to 20%, which if the angle is small enough can only be achieved by reducing the variance to 2. The search point P is driven closer and closer to the boundary and eventually lies on it (P2). Since there is no longer any nite step size that can provide a suciently large success rate, the variance is permanently reduced to the minimum value specied in the program. Depending on the particular problem structure and the chosen values of the parameters in the convergence criteria the search is either slowly continued or it is terminated before reaching the optimum. The more constraints become active during the search, the smaller is the probability that the objective will be closely approached. In fact, even in problems with only two variables and one constraint (Problem 2.46) the angle between the contours and the edge of the feasible region can become vanishingly small in the neighborhood of the minimum. Similar situations to the one depicted in Figure 6.17 can even arise in unconstrained problems if the objective function displays discontinuities in its rst partial derivatives. Examples of this kind of behavior are provided by Problems 2.6 and 2.21. If only a few variables are involved there is still a good chance of reaching the objective. Other search methods, especially those which execute line searches, are generally defeated by such points of discontinuity. The multimembered evolution strategy, although it works without a rigid step length adaptation, also loses its otherwise reliable convergence characteristics when the region of

Numerical Comparison of Strategies

Forbidden region

215

σ1 σ2

α

To the minimum

σ2 P2

Lines of constant F(x)

P1 Negative gradient direction

Circles : lines of equal probability of a step

Figure 6.17: The situation at active constraints

success is very much narrowed down by constraints. While the individuals are not yet at the edge of the feasible region, those descendants whose step lengths have become smaller have a higher probability of survival. Thus here too the entire population eventually concentrates itself in a smaller and smaller area at the edge of the feasible region. The theory of the rate of progress in the corridor model did not foresee this kind of diculty, indeed it gives an optimal success rate, almost the same as in the sphere model, simply because the gradient vector of the objective function always runs parallel to the boundaries. In this case the search weaves backwards and forwards between the center and side of the corridor. The reduced probability of success at multidimensional edges is compensated by the fact that with a uniform probability of occupation over the cross section of the corridor, the space that counts as near to the edges represents a very small fraction of the total. Provided that the success rate is obtained over long enough periods the 1=5 success rule does not lead to permanent reduction of the variances but to a constant near optimal step size (it really uctuates) that depends only on the width of the corridor and the number of variables. The situation is happier than in Figure 6.17 if the constraints are given explicitly as

xi  ai or xi  bi For any one variable, the region of success at a boundary is reduced by one half. If at some position m variables are each bounded on one side, then on average it costs 2m mutations before one lands within the feasible region. Here again, the 1=5 success rule for m > 2 will continuously reduce the variances until they reach their minimum value. Depending on the route chosen by the search process the limiting values of the variances, which are individually adjustable for each variable, will be reached at di erent times. Their relative values thereby alter, and with the new combination of step lengths the convergence can be faster. The extra exibility of the multimembered evolution strategy with recombination, in which the variances of the changes in the variables are individually adaptable during

216

Comparison of Direct Search Strategies for Parameter Optimization

the whole of the optimization process, is a very clear advantage in solving constrained problems. Suitable combinations of variances are set up in this case before the smallest possible step lengths are reached. Thus the total computation time is reduced and the nal accuracy is better. The recombination option also appears to have a benecial e ect at boundaries that are not explicit it clearly improves the chance that descendants, even with a larger step size, will be successful near the boundary. In any case the population clusters more slowly together than when there is no recombination.

Global Convergence Properties

Among the 50 test problems there are 8 having at least a second local minimum besides the global one. In the reliability test, the accuracy achieved was only assessed with respect to the particular optimum that was being approximated. What now is the capability of each strategy for locating global minima? Several problems were specically designed to investigate this question by having very many local optima, namely Problems 2.3, 2.26, 2.30, and 2.44. In Table 6.9 this aspect of the test results is evaluated. Except for one problem (Problem 2.32), whose global minimum was found by all the strategies under test, the method of Rosenbrock only converged to local optima. The complex method and the (1+1) evolution strategy were only better in one case: namely, in Problem 2.45 they both approached the global minimum. Table 6.9: Results of all strategies in the second comparison test: global convergence properties Problem F I B No. O 2.3 2.36 2.30 2.32 2.44 2.45 2.47 2.48

G O L D

L A G R

H O J E

D S C G

D P D S R S O F I O C W P M S P E S P E

C O M P

L1 L1 L3 L1 L7 L7 L1 L3 L1 L6 L1 L1 L1 L1 L1 L1 L1 L1 L1 L1 L1 L1 L4 L1 G G L1 L1 L G L3 L1 L2 Lm

E V O L

G R U P

R E K O

Lm G G L1 G G Lm G G G G G L1 G G G G G L2 G G Lm GL GL

Meaning of symbols: L Search converges to local minimum. L3 Search converges to the 3rd local minimum (in order of decreasing objective function values). Lm Search converges to various local minima depending on the random numbers. G Search converges to global minimum. GL Search converges to local or global minimum depending on the random numbers.

Numerical Comparison of Strategies

217

The multimembered evolution strategy displays much better global convergence properties, with or without recombination. Although its actual path through the space was determined by chance, it always found its way to the absolute minimum. Only in Problem 2.48 was the global optimum not always approached. In this case the feasible region is not simply connected: Between the starting point and the global minimum there is no connecting line that does not pass through the region excluded by constraints. The path of the simple evolution strategy and the initial condition of the complex method are also both dependent on the particular sequence of pseudorandom numbers however, the main di erence between the results of three trials in each case was simply that di erent local minima were approached. In one case the (1+1) evolution rejected 33 local optima only to converge at the 34th (Problem 2.3). In spite of the good convergence properties of the multimembered evolution manifested in the tests, a certain measure of scepticism is called for. If the search is started with only small step lengths in the neighborhood of a local minimum, while the global minimum is far removed and is surrounded by only a relatively small region with small objective function values, then the probability of getting there can be very small. If in addition there are very many variables, so that the step sizes of the mutations are small compared to the Euclidean distance between two points in IRn , the search for a global optimum among many local optima is like the proverbial search for a needle in a haystack. Locating singular minima, even with only a few variables, is a practically hopeless task. Although the multimembered evolution increases the probability of nding global minima compared to other methods, it cannot guarantee to do so because of its basically sequential character.

6.3.3.3 Third Test: Non-Quadratic Problems with Many Variables In the rst series of tests we investigated the rates of convergence for a quadratic objective function, and in the second the reliability of convergence for the general non-linear case. The aim of the third test is now to study the computational e ort required for nonquadratic problems. Because of their small number of variables, the problems of the second test series appear unsuitable for this purpose, as rates of convergence and computation times are only of interest in relation to the number of variables. The construction of non-quadratic objective functions of a type that can also be extended to an arbitrary number of variables is not a trivial problem. Another reason, however, for this third strategy comparison being restricted to only 10 di erent problems is that it required a large amount of computation time. In some cases CPU times of several hours were needed to test just one strategy on one problem with a particular number of variables. Seven of the problems are unconstrained and three have constraints. Appendix A, Section A.3 contains the mathematical formulation of the problems together with their solutions. The procedure followed was the same as in the rst test. Besides the termination criterion specic to each strategy, which demanded maximum accuracy, a further convergence criterion was applied in common to all strategies. According to the latter the search was to be ended when a specied distance had been covered from the starting point towards the minimum. The number of variables was varied up to the maximum allowed

218

Comparison of Direct Search Strategies for Parameter Optimization

by the storage capacity, taking the values 3, 10, 30, 100, 300, and 1000. Of course, if a problem with, for example, 30 variables could not be solved by a strategy, or if no result was forthcoming at the end of the maximum computation time of 8 hours, the number of variables was not increased any further. As in the rst test, the initial conditions were specied by (;p1)i  i = 1(1)n x(0) = x + i i n Two exceptions are Problem 3.3 with

(;p1)i x(0) = x + i i 10 n to ensure that the search always converged to the desired minimum and not to one of the many others of equal value and Problem 3.10 with 1 x(0) i = xi + pn to start the search within the feasible region. Problems 3.8 and 3.9, whose minima are at innity, required special treatment of the starting point and termination conditions (see Appendix A, Sect. A.3). The results are presented in Table 6.10. For comparison, some of the results of the rst test (Problem 1.1) are also displayed. The numbers enable one to assess critically on the one hand the reliability of a strategy and on the other the computation times it requires. 





Numerical Comparison of Strategies

219

Table 6.10: Results of all strategies in the third comparison test The following notation is used in the tables: n: Number of variables Case: A label for the convergence behavior, taking the values: 1 Normal end of search required approximation to the objective was achieved. 2 The search was ended before reaching the desired accuracy. 3 The search became unending without converging it had to be terminated externally. 4 The maximum computation time of 8 hours was insucient to end the search successfully (occasionally more computation time was invested in trials with the multimembered evolution strategy that promised to be successful). - No trial was undertaken. 1(2) Depending on the sequence of random numbers various cases occurred the entries in the table refer to the rst case dened. OFC: Number of objective function calls. CFC: Number of constraint function calls. Time: Computation time in seconds (CPU time). Iterations, cycles, exploratory cycles, line searches, orthogonalizations, restarts, etc., were counted as in the rst comparison test. Fatal execution errors were only registered in the Powell and DFPS methods and it is not further specied here in which problems they occurred. As a rule the same types of problem were involved as in the second test. In unconstrained problems no numbers are tabulated for the number of objective function calls made by the evolution strategies. This can be calculated from the number of mutations or generations as follows: EVOL: 1 + number of mutations GRUP, REKO: 10 + 100 times number of generations

(continued)

1

1 10 1250

1

1

2

1

1

1

1

1 10 1250

1

1

2

1

1

1

1

1 10

1

1

2

1

1

Probl.

3.1

3.2

3.3

3.4

3.5

3.6

3.7

1.1

3.1

3.2

3.3

3.4

3.5

3.6

3.7

1.1

3.1

3.2

3.3

3.4

3.5

3.6

3.7

1

1

1

1

4

2

1

2

1

1

1

4

3

1

2

1

1

1

4

3

1

case

1

cycles

1.1

OFC

87

138

82

140

321

745

198

85

183

192

85

192

630

415

158

183

192

85

192

630

415

158

n=3

cycles

1 2 18 4381 286.00 1 2 1 1 1 1 1 2 1 2 1 1 1 1 1 2 24 3178 199.00 1 2 1 1

0.14 0.08 0.18 0.18 0.10 0.34 0.84 3.96 0.12 0.08 0.14 0.18 0.04 0.16 0.56 2.08 0.10 0.06 0.12 0.06

458

296

567

567

5.54

3.38

0.51

0.48

1.10

0.72

5.90

271

296

589

589

1

1

1

264

437

436

9 2142

5 1285

1

1

1

1

0.42

0.64

0.56

3.52

2.22

0.30

0.48

1.10

0.68

9 2143 136.00

9 3543

5 1993

1

1

1

1

9 3470

4.12

4.26

5 1997

1

0.53

1.04

time 0.38

case 1 456

time

0.13 1

OFC n = 10

case 1

1

2

1

2

1

1

1

1

1

2

1

2

1

1

1

1

1

2

1

2

1

1

1

cycles 1

1

1

6

4

1

1

1

1

6

4

1

1

1

1

6

4

1

OFC 3.07

time 3.14

2.92

6.92

4.68

1.80

3.00

6.98

4.68

785

1274

1274

2.80

4.62

3.20

4012 17.60

2816 12.00

781

816

1715

1715

6347 28.60

4441 18.70

1242

816

1709

1709

6472 27.50

4270 18.90

1242

n = 30

case 1

1

2

1

-

1

1

1

1

1

2

1

-

1

1

1

1

1

2

1

-

1

1

1

1

1

1

2

2

1

2

1

1

2

2

1

2

1

1

2

2

1

cycles OFC 27.6

52.3

53.9

40.0

94.5

93.5

26.5

time

2509

3695

3695

4790

4808

2501

4849

5207

5207

7013

25.5

35.3

30.1

64.4

66.3

17.3

52.5

53.1

42.4

91.2

7017 101.0

3870

4818

5193

5193

7139

7041

3870

n = 100

Table 6.10 continued : Coordinate strategies FIBO, GOLD, LAGR (from top to bottom)

case 1

1

2

1

-

1

1

1

1

1

2

1

-

1

1

1

1

1

2

1

-

1

1

1

cycles 1

1

1

1

1

1

2

1

1

1

1

1

2

1

1

1

1

1

OFC 7204

11270

11270

8046

8073

7201

14633

15111

15111

11218

11244

10562

14634

15127

15127

11218

11244

10562

n = 300

time 240

319

291

345

345

153

476

440

332

439

430

221

478

428

334

435

434

210

case 1

1

2

1

-

1

1

1

1

1

2

1

-

1

1

1

1

1

2

1

-

1

1

1

cycles 2

1

1

1

1

1

2

1

1

1

1

1

2

1

1

1

1

1

OFC 42993

35247

35247

26332

26415

24001

47767

49132

49132

35303

35300

38703

47799

49255

49255

35303

35300

38701

n = 1000

time 7890

3240

2600

3500

3600

1620

6720

4750

3680

4580

4740

2670

6770

4590

3670

4730

4590

2500

220 Comparison of Direct Search Strategies for Parameter Optimization

Probl.

case

cycles

4

3

4

4

4

OFC

20

19

20

20

20

1

1

4

4

20

20

2 25 151

1

1

1

1

1

1

0.02 48

48

1

0.02

3 3

2

0.18

0.02

48

1

48

3 3

1 12 237 1 3 48

1

time 0.02

48

3

cycles

0.12

1

OFC

0.02

case

0.02

time 0.08

0.08

0.06

16.70

0.10

0.08

0.06

case

44

29

44

28

28

268

case

1 1

1 3 1 5

1 3

1 2

1 2

1 2

1 2

1 1

1 1

Probl.

1.1

3.1 3.2

3.3

3.4

3.5

3.6

3.7

3.8

3.9

3.10 1 8

OFC

29

45

41 98

27

n=3

time

367

1.16

0.16 0.10

28

CFC

101

0.10

0.14

0.12

0.14

0.40

0.14 0.30

0.08

case 4 3 4 2

1

1 1 1

2

1

1

1

2

427

309

91 1128

152

279

295

279

73

161 282

120

CFC

orth.

0.43 0.74 0.74

168 168 168

3 3

time

3

cycles

3 3 0.70 0.62

168 168

3.46

1.60

1.10

2.06

2.02

1.98

5.02

1.36 2.10

0.91

12 2953 9766 30.50

6

1

1

1

1

3 4

2

orth.

1 1

1

n = 10

OFC

1

1

1

case

3.72 3.68

352 352

2 2

121 121 121

0 0 0

1 1 1

2

1

1

512

241

121

0 1

1

3077

8 1

1

575 575

121

215

7861

1

2.86

352

2

9.76

17.30

1.32

1.32

1.36

1.20

1690.00

18.70 19.10

1.18

1

4.78

352

2

2

1

-

4.82

352

2

3 3

0

1

1 1

758 119 153 137

17714 4954 4954 4954

2

9

9

0 2

1 -

3

47

9

47

3 4

2

1

1

1

1

1

-

1 1

1

1

784

case

2833

226

1871

83059

7337

83059

2352 3879

899

OFC

23505 1700 23505 2160 23505 2410

12 12

969

42004 5710

12

cycles

42004 5440

23505 1460

n = 1000

time

12 21 21

33448

n = 75 (max)

2

1

-

1

100

orth.

n = 300

18612

cycles

time OFC

4954

OFC

case CFC

9 1 33 1 32 1

2.37

352

n = 100

OFC

2

cycles

1 1

1

OFC n = 30

1

1

2

1 130 7493 4210.00 1 3 168 0.48 1

1

1

1

time

ROSE - Rosenbrock method with Gram-Schmidt orthogonalization

3.7

3.6

3.5

3.4

3.3

3.2

3.1

1.1

OFC n = 30

case

time

CFC

n = 10

orth.

case

time

n=3

194

85

236

4830

728

4660

242 342

145

time

Table 6.10 continued : HOJE - Direct search of Hooke and Jeeves

Numerical Comparison of Strategies 221

1

3.7

orth.

1

0

1

0

2

3

1

0 0.04 0.08 0.12 0.28 0.06 0.08 0.06 0.10

OFC

20 32 48 35 20 30 20 30

6

12

9

3

6

3

6

lin. search

time

3

case

1

1

1

1

1

1

1

1

orth.

0

0 10

10

10

10

0 0

112

40

30

10

8

3

2

0

1.12

147 0.20 0.20 0.22 0.22

56 47 56 56

377 26.00

0.86

119

OFC

0.20

time

56

case

1

1

1

1

1

1

1

1

Probl.

1.1

3.1

3.2

3.3

3.4 3.5

3.6

3.7

orth.

1

0

0 1

2

6

3

3 6

9

6

12

1

3

3

0

lin. search

n=3

OFC

30

20

20 31

35

48

32

20

time

0.08

0.06

0.06 0.08

0.28

0.12

0.08

0.04

case

1

1

1

1

1

1

1

1

orth.

0

0

0 0

8

3

2

0

lin. search

10

10

10 10

112

40

30

10

n = 10

OFC

0.72

0.58

0.22

time

56

56

56 47

0.22

0.22

0.20 0.20

383 25.50

147

119

56

1

1

1

1

2

1

1

1

1

1

1

1

1

0

0

0 0

1 28

3

1 1

0

1

case

DSCP - Davies-Swann-Campey method with Palmer orthonormalization

1

3.6

1

3.3

1

1

3.2

1

1

3.1

3.5

1

Probl.

3.4

case

1.1

lin. search

n = 10

orth.

535

321

136

527

321

136

30

30

30

30

6.56

3.58

1.16

136

136

165

136

30

30

30 30

136

136

136 165

1.26

1.36

1.28 1.60

932 2924 1670.00

151

91

30

151

91

30

OFC

18.70

7.58

1.18

1.26

1.48

1.58

1.18

1087 3366 2030.00

n = 30

0

0

0

0

28

3

1

0

n = 30

lin. search

case orth.

lin. search OFC

n=3

time

1

1

1

1

-

1

1

1

orth.

1

0

0 1

1

1

0

case

75 150 75 150

0 1

150

1

1

150

1

0

75

0

150

75

75 150

150

150

75

6.68 7.08 564 76.90

338

490 78.20

338

563 79.00

6.10

564 13.70

7.18

338 6.36 490 12.90

563 14.80

338

6.10

time

563 78.40

338

563 14.90

338

n = 75 (max)

1

1

1

1

-

1

1

1

n = 75 (max)

orth. lin. search

time case

lin. search OFC

OFC time

Table 6.10 continued : DSCG - Davies-Swann-Campey method with Gram-Schmidt orthonormalization

1

1

1

1

-

1

1

1

case

OFC

lin. search orth.

95

95

0

0

0 0

95

95

95 95

1 190

0

0

9.49

time

428

428

428 334

11.2

11.1

9.96 9.22

713 23.20

428 11.90

448

n = 95 (max)

222 Comparison of Direct Search Strategies for Parameter Optimization

Probl.

iterations

case

7

3 3

3

13

2

1 1

1

4

1

1

1

1

1

2

7

11

2

3

1

OFC

166

20

20 55

23

38

32

11

1

case

time

0.46

0.06

0.06 0.14

0.18

0.12

0.08

10

iterations

1

lin. search

0.02 32 96 20 20

9 2 2

1 1 1

3

1

2

21

2

3

1 0.46 0.44 0.48

128 128

257 16.90

0.36

84

0.12

time

112

32

1

1

1

15 101

1

1

1

1

1

2

1

1

Probl.

3.2

3.3

3.4

3.5

3.6

3.7

10

11

10

15

25

4

2

20

10

3

1

case

1.1

iterations

3.1

OFC

n=3

time

0.26

0.02

0.02

0.02

0.12

0.06

0.06

0.02

case

1

1

2

1

1

1

1

1

iterations

24

24

61

65

48

24

16 307

1

1

4

4

3

1

OFC

n = 10

1.42

0.08

0.06

3.96

0.30

0.20

0.06

time

3

1

2

64

160

160

1.78

1.82

0.32

780

2 15

64 64

1

7.84

0.40

0.34

204

3.19 612 22.00

2

1

1

204

3.92

3.34

1

1

1

1

2

1

2 1

iterations

8.74 502

100

364

9.56

364 11.80

364 10.20

3

1

2 1

1

1 7.82 502

100

135 811 19.00

135 811 16.20

135 541 15.50

1

701 15.00

-

135 541 15.50 1

1

1

701 15.10

200

8.60

200

135 407 1

1

time

3.72

6 1274 84.20 1

lin. search

202

OFC

6 1274 84.40

1

OFC

n = 135 (max)

time

100

n = 100

lin. search

n = 180 (max)

1 204

3

1

2

1

1

2

1

1 1

1

3

2

2

1

-

5 612 23.10

5

1

n = 100

1.14

0.94

1

case

-

1

1

1

152

30

45 1640 932.00

4

4

1

1

1

2

1

1

1

1

1

n = 30

2.32

2.50

0.60

700 1963 1100.00

1

1

1

288 152

23

1

304

91

92

91

30

30

3 3

1 1

1

case

1

iterations

DFPS - Stewart’s modification of the Davidon-Fletcher-Powell method

3.7

3.6

3.4 3.5

3.3

3.2

3.1

lin. search

3

OFC case

lin. search

time

1

iterations

OFC

case

iterations

1

OFC

1.1

OFC

time time

iterations case

n = 30

OFC

n = 10

time

n=3

iterations

case

Table 6.10 continued : POWE - Powell’s method of conjugate directions

Numerical Comparison of Strategies 223

Probl.

0

0

0 0

40

35

1

1

1 1

1

1

1

1

1

1

1.1

3.1

3.2

3.3 3.4

3.5

3.6

3.7

3.8

3.9

3.10

33

76

85

57

58 74

83

76

69

1

Probl.

1 1 1 1 1 1 1 1

0.09 0.10 0.12 0.28 0.08 0.08 0.08 0.08

28 34 40 32 25 32 25 28 48

0

0 185

0 0

0

0

3968

163

163 22362

300 163

152

138

39.10

208

44

186

n=3

0.25

0.19

0.33

0.30

0.32

0.23

0.55 0.24

0.28

0.28

0.22

966 529 1266 556 587

1 1 1(2) 1 1 1 1 1(4)

486

384

207

527

1

1

535 691

1

8.46

8.32

7.68

7.34

17.20

74.30 6.71

6.99

9.30

6.72

8919 12.10

600

4680

n = 10

1.68

1.62 206.00

1100.0 4030.0

7 17357 1 2 547 101142

4

2

1

1

1(2)

2(4)

2(4) 1(2)

1

1

1

2622

259

247

211

4508

357

240

840

781

1450

12100 402

1041 75867

9537

9011

16936

19407 4722

3092

2770

2621

n = 30

-

-

4

1

4

4

4

1

1

1

4

1140.0 1020.0

17357 21387

7 4

4 -

1 -

1

1

1 181

1

79.7 8130.0

95.2

37.4

1277 12936

664

0 1

1 1

3.54 12.10

0 1514

1

1.56

0

1

1.49

COMP - Complex method of Box ( no. of vertices = 2n )

1

1

1 1

0 0

0

0

case

3.7

3.6

3.4 3.5

1 1

1

restarts

OFC

3.2 3.3

case

1

CFC

3.1

time time

1.1

OFC

case case

n = 30

case

n = 10

time

OFC

restarts OFC

restarts

CFC

OFC CFC

time time

n=3

case

time

OFC

Table 6.10 continued : SIMP - Simplex method of Nelder and Mead (with restart rule)

restarts

4 -

6190

0

32196

17431

17397

14902

25300

13300

13100

11000

n = 95 (max)

-

-

-

4

10082

1

862

1789 42099 25800

case

1

0

OFC n = 100

OFC

case case

time time

OFC

restarts 1

5142

time 5270

n = 135 (max)

224 Comparison of Direct Search Strategies for Parameter Optimization

case

1 1 1 1 1 1 1 1 1 1 1

case

Probl.

3.5 3.6 3.7 3.8 3.9 3.10

Probl.

49 67 179 61 66 74 89 63 99 78 925

1 47720 2 1 37985 1 5145 1 1696 1 4633 -

3.3 3.4 3.5 3.6 3.7 3.8 3.9 3.10

2192 2185 2208

1 1 1

1.1 3.1 3.2

1.1 3.1 3.2 3.3 3.4

45 31 592

998 1870

n = 100

CFC CFC 3320

149 164 188

2710 389 255392 803 4634 336

301 79 3913

0.17 0.22 0.52 0.50 0.20 0.36 0.32 0.20 0.54 0.29 4.06

6666 7638 6916

4 2 4 1 14818 1 3161 1 13035 -

1 1 1

1310 1700 1480

3060 3129 1883309 14100 5374 13036 2830

n = 300

2 4 -

1 1 1

23607 23374 23819

OFC

15600 17100 17200

24629 1436

14.0 16.6 26.5 4060.0 244.0 365.0 26.7 46.2 34.5

n = 1000

300 579

n = 30 1 630 730 1 1 1041 1 7103 1 10939 2 1 15769 1 1154 1 668 1 1435 4

n = 10

CFC

1.74 1 224 1.88 1 221 4.46 1 537 1 2244 156.00 1 257 2.18 1.82 200 2 2.87 325 1 1 206 1.74 3980 1 319 136 5.80 389 1 388 148 3.33 1 59757 39076 802511 824.00

case case

mutations

mutations

mutations mutations

OFC OFC

OFC OFC

time time

time time

CFC

n=3

OFC

case case

CFC CFC

mutations mutations

time time

Table 6.10 continued : EVOL - (1+1) evolution strategy

Numerical Comparison of Strategies 225

4 7 4 5 3 4 5 5

case

1 1 1 1 1 1 1 1

1 9 1 6 1 103

1.1 3.1 3.2 3.3 3.4 3.5 3.6 3.7

3.8 3.9 3.10

194 213 199

1 1 1

2 1653 2 1650 1 473 1 109 1 329 -

Probl.

1.1 3.1 3.2

3.3 3.4 3.5 3.6 3.7 3.8 3.9 3.10

Probl.

case

540 414 6937

n=3

13200

1600 1730 1660

5.19 2.68 58.0

13200 3790 5925 1614891 5090 15941 32915 2620

n = 100

2940 610 45174

1.81 3.18 1.56 5.02 1.44 1.90 2.26 2.08 33663 1 25 1355 3710 1 37 2067 1 3707 230227 4745318

17 20 25 23 18 21 19 20

1 1 1 1 1 1 1 1

n = 10

Table 6.10 continued : GRUP - (10,100) evolution strategy

generations

generations

case 1 4 1 -

1 1 1

case

OFC OFC

generations 986

1655

551 677 604

47168 98673

n = 300

generations

time time

OFC OFC

CFC CFC

CFC CFC

48.0 35.4 5360

16.8 20.3 24.1 164.0 20.9 21.8 19.6 20.8

time 23200

38100

13600 16400 14600

time

55 77 67 1462 848 481 446 80 51 96

1 1 1 1 1(2) 2 1 1 1 1 4

case -

-

1 1 1

case

854 865 940

2983 226805 9610 4801

n = 30

OFC

n = 435 (max)

OFC

generations generations

CFC CFC

390 239

145 201 194 82100 2140 1300 1130 200

time 28300 30400 33100

time

226 Comparison of Direct Search Strategies for Parameter Optimization

generations

4 5 6 1 3 5 3 5

case

1 1 1 1 1 1 1 1

1.1 3.1 3.2 3.3 3.4 3.5 3.6 3.7

OFC

1250 28200 1420 2370 12354 2529790 9080 13547 18610 3110

1 1 1 4 1 -

1 1 1

494

205 399

210

180 206 212

1 77 2 1653 1 82 1 137 1 128 1 186 -

1420 1420 1350

3.3 3.4 3.5 3.6 3.7 3.8 3.9 3.10

84 80 77

1 1 1

36708

n = 300

1.1 3.1 3.2

n = 100

1410 1978 27875

20 28 456

1 1 1

8.25 7.09 6.88

3549 1010 3321

584 638 462

10 10 8

1 1 1

3.8 3.9 3.10

13 15 11 14 15 19 12 17

Probl.

1 1 1 1 1 1 1 1

case

n = 10

generations

Probl.

OFC

CFC CFC

2.67 3.56 4.32 1.40 2.06 3.43 2.22 3.60

time time

case case

generations generations

23.3 31.3 19.8 122.0 26.3 35.7 21.5 30.4 60.5 30006 2810 53.9 586357 1050.0

CFC 10600 20800

10800

9340 10800 10800

49410 25200

CFC

OFC OFC

time time

n=3

case 1 1 -

1 1 1

1 1 4

45 92

34 1 32 1 35 1 1 1365 28 1 1(2) 213 1 29 1 41

case

307

305

289 305 257

OFC

n = 435 (max)

23000

21900

21100 22700 19400

526 487

177 170 198 79700 152 1120 148 209 3920 242482 6321 9210

n = 30

CFC CFC

OFC

generations generations

time time

Table 6.10 continued : REKO - (10,100) evolution strategy with recombination

Numerical Comparison of Strategies 227

228

Comparison of Direct Search Strategies for Parameter Optimization

With only three variables, nearly all the problems were solved perfectly by all strategies i.e., the required approximation to the objective was achieved. The only exception is Problem 3.5, which ended in failure for the coordinate strategies, the method of Hooke and Jeeves, and the methods of Powell and of Davidon-Fletcher-Powell-Stewart. In apparent contradiction to this, the corresponding Problem 2.21 for n = 5 was satisfactorily solved by the Hooke-Jeeves strategy and the DFPS method. The causes are to be found in the di erent initial values of the variables. With the variable metric method, fatal execution errors occurred in both cases. If there are 10 or more variables, even the two membered evolution strategy does not nd the minimum in Problem 3.5, due to the extremely unfavorable starting point. The probability of making from there a rst step with a lower objective function value is 2 n . Thus with many variables, the termination condition is usually met before a single success has been scored. The simplex method of Nelder and Mead with n = 10 took 185 restarts to reach the desired approximation to the objective. For more than 10 parameters the solution can no longer be suciently well approximated in spite of an increasing number of restarts. With stricter accuracy requirements the simplex method fails much sooner (Problem 2.21 with n = 5). The complex strategy likewise was no longer able to solve the same problem for n  30. Depending on the sequence of random numbers it either ended the search before achieving the required accuracy, or it was still far from the minimum when the allowed computation time (8 hours) expired. The multimembered evolution strategy also proved to be dependent, although less strongly, on the particular sequence of random numbers. The version without recombination failed on Problem 3.5 for n  30 with recombination it failed for n  100. Without recombination and for n  100 it ended the minimum search prematurely also in Problems 3.4 and 3.6. The simplex and complex methods had convergence diculties with both types of objective function, usually even for only a few variables. Several times they had to be interrupted because of exceeding the time limit. Further details can be found in the tables and Appendix A, Section A.3. The search for the minima in Problems 3.4 and 3.6 presents no diculties to the coordinate strategies, and the methods of Hooke and Jeeves, Rosenbrock, Davies-SwannCampey, Powell, and Davidon-Fletcher-Powell-Stewart. The three rotating coordinate strategies are the only ones that manage to solve Problem 3.5 satisfactorily for any number of variables. Nevertheless it would be hasty to conclude that these methods are therefore clearly better than the others an attempt to analyze the reasons for their success reveals that only slight changes in the objective functions are enough to undermine their apparently advantageous way of working. The signicant di erence in this respect between the above group of strategies and the others (complex, simplex, and evolution strategies) is that the former operate with a much more limited set of search directions than the latter. There are usually only n directions, e.g., the n coordinate directions of the axes-parallel search methods, compared to an innite number (in principle) in the evolution methods. In the case of Problems 3.4 to 3.6 the most favorable search directions are the n directions of the unit vectors. All methods with one dimensional minimizations use precisely these directions in their rst iteration cycle, so they do not usually require any further iterations to achieve the required ;

Numerical Comparison of Strategies

229

accuracy. By keeping the starting conditions the same but rotating the coordinates with respect to the contours of the objective function (Problem 3.6), or slightly tilting the contours with respect to the coordinate axes (Problem 3.5), or both together (Problem 3.4), one could easily cause all the line searches to fail. On the other hand the strategies without line searches would not be impaired by these changes. Thus the advantage of selected directions can turn into a disadvantage. These coordinated strategies can never solve the problem referred to, whereas, as we have seen, the strategies that have a large set of search directions at their disposal only fail when a particular number of variables is exceeded. Problems 3.4 and 3.6 are therefore suitable for assessing the reliability of simplex, complex, and evolution strategies, but not for the other methods. Together they belong to the type of problems which Himmelblau designates as \pathological." Leaning more to the conservative side are the several times continuously di erentiable objective functions of Problems 3.1, 3.2, 3.3, and 3.7. The rst two problems were tackled successfully by all the strategies for any number of variables. The simplex method did, however, need at least one restart for Problem 3.1 with n  100. For 135 variables it exceeded the time limit before Problems 3.1 and 3.2 were solved to sucient accuracy. Problem 3.3 gave trouble to several search procedures when there were 10 or more variables. The coordinate strategies were the rst to fail. For only n = 10, the step lengths of the line searches would have had to be smaller than allowed by the number precision of the computer used. At n = 30, the DSC strategy with Gram-Schmidt orthogonalization also ends without having located the minimum accurately enough. The simplex method with one restart still found the solution for n = 30, but the complex strategy failed here, either by premature termination of the search or by reaching the maximum permitted computation time. Problem 3.3, because the cost per objective function evaluation increases as O(n2), requires the longest computation times for its solution. Since the objective function also took O(n2) units of storage, this problem could not be used for more than 30 variables. Problem 3.7, like the analogous Problem 2.31, gave trouble to the two quadratically convergent strategies. The method of Powell was only successful for n = 3. For more variables it became stuck in the search process without the termination rule taking e ect. The variable metric strategy behaved in just the same way. For n  30, it no longer came as near as required to the optimum. Under the stricter conditions of the second set of tests it failed already at n = 5. With both methods fatal execution errors occurred during the search. No other direct search strategies had any diculty with Problem 3.7, which is a simple 10th order polynomial. Only the simplex method would not have found the solution suciently accurately without the restart rule. For n = 100, it reached the time limit before the search simplex had collapsed for the rst time. The advantage shown by the complex strategy was due to the complex's having 2 n vertices, which is almost twice as many as the n + 1 of the simplex. An attempt to solve Problems 3.1 to 3.10 for n = 30 with a complex constructed of 40 points failed completely. The search ended, in every case, without having reached the required accuracy. How do the computation times compare when the problems are no longer only quadratically non-linear? For solving the \pathological" Problems 3.4 to 3.6 all the methods with a line search take about the same times, with the same number of variables, as they do

230

Comparison of Direct Search Strategies for Parameter Optimization

for solving the simple quadratic Problem 1.1, if indeed they actually can nd a solution. With any of the remaining methods the computation times increase somewhat more steeply with the number of variables, up to the limiting number beyond which convergence cannot be guaranteed in every case. The solution times for Problems 3.1 and 3.2 usually turn out to be several times greater than those for Problem 1.1. The cost of the coordinate strategies is up to 1000% more for a few variables, which reduces to 100% as the number of variables increases. As in the case of Problem 1.1, the solution times for Problems 3.1 and 3.2 using the method of Hooke and Jeeves increase somewhat faster than the square of the number of variables. For very many variables 250% more computation time is required. For n  30, the Rosenbrock method requires 70% to 250% more time (depending on the number of orthogonalizations) for the rst two problems of the third set of tests than for the simple quadratic problem. The computation time still increases as O(n3) in all cases because of the costly procedure of rotating the coordinates. For example, for n = 75, up to 90% of the total time is taken up by the orthogonalizations. The DSC strategies reached the desired accuracy in Problem 1.1 without orthogonalizations. Since solving Problems 3.1 and 3.2 requires more than n line searches in each case, the computation times di er signicantly, depending on the chosen method of orthogonalization. Palmer's program holds down the increase in the computation times to O(n2) whereas the GramSchmidt method leads to an O(n3) increase. It therefore is not meaningful to quote the extra cost as a percentage with respect to Problem 1.1. In the extreme case instead of 6 seconds at n = 75 the procedure took nearly 80 seconds. The method of Powell requires two to four times as much time, depending on whether one or two extra iterations are needed. However, even for the same number of iterations, i.e., also with the same number of line searches (n = 135), the number of function calls in Problems 3.1 and 3.2 is greater than in Problem 1.1. The reason for this is that in the quadratic reference problem a simplied form of the parabolic interpolation can be used. The variable metric strategy, in order to solve the two non-quadratic problems (Problems 3.1 and 3.2) with n = 180, requires about nine times as much computation time as for Problem 1.1. This factor increases with n since the number of gradient determinations increases gradually with n. The pattern of behavior of the simplex method of Nelder and Mead is very irregular. If the number of variables is small, the computation times for all three problems are about equal. However, for n = 100, Problem 3.2 requires about seven times as much time to be solved as Problem 1.1 and, because of a restart, Problem 3.1 requires even thirty times as much. With n = 135, neither of the two non-quadratic problems can be solved within 8 hours, whereas 1:5 hours are sucient for Problem 1.1. On the other hand the complex strategy requires only slightly more time, about 20%, than in the simple quadratic case, provided 2 n vertices are taken. The time taken by this method on the whole for all problems, however, exhibits the strongest rate and range of variation with the number of parameters. The evolution strategies prove to be completely una ected by the altered topology of the objective function as compared with the case of spherically symmetrical contour surfaces. Within the expected deviations, due to di erent sequences of random numbers,

Numerical Comparison of Strategies

231

the measured computation times for all three problems are equal. The results show that Rechenberg's (1973) theory of the rate of progress, which does not assume a quadratic objective function but merely concentric hypersphere contour surfaces, is valid over a wide range of conditions. Even more surprising, however, is the behavior of the (10 , 100) evolution method with recombination in the solution of Problems 3.4 and 3.6, whose objective functions have discontinuous rst derivatives, i.e., their contour surfaces display sharp edges and corners. The mixing of the components of variables representing individuals on di erent sides of a discontinuity appears sometimes to have a kind of smoothing e ect. In any case it can be seen that the strategy with recombination needs no more computation time or objective function calls for Problems 3.4 and 3.6 than for Problems 1.1, 3.1, and 3.2. With all the methods under test, the computation times for solving Problem 3.7 are about twice as high as those measured in the simple quadratic case. Only the simplex method is signicantly more demanding of time. Since the search simplex frequently collapses in on itself it must repeatedly be reinitialized. Since Problem 3.3 could only be tackled with 3, 10, and 30 variables it is not easy to analyze the resulting data. In addition, the dependence of the increase in diculty on the number of parameters is not so clear-cut in this problem. Nevertheless the results seem to indicate that at least the number of objective function calls, in many strategies, increases with n in a way similar to that in the pure quadratic Problem 1.2. Because an objective function evaluation takes about O(n2) operations in Problem 3.3, the total cost generally increases as one higher power of n than in Problem 1.2. The cost of the variable metric strategy and both versions of the (10 , 100) evolution strategy seems to increase even more rapidly. In the latter case there is a suspicion that the chosen initial step lengths are too large for this problem when there are very many variables. Their reduction to a suitable size then takes a few additional generations. The two membered evolution strategy, which is able to adjust unsuitable initial step lengths relatively quickly, needed about the same number of mutations for both Problems 1.2 and 3.3. Since only one experiment per strategy and number of variables was performed, the e ect of the particular sequence of random numbers on the recorded computation times is not known. The particularly advantageous behavior of the DFPS method on exactly quadratic objective functions is clearly wasted once the problem deviates from this model structure in fact it seems that the search process is appreciably held back by an interpretation of the measured data in terms of an inappropriate internal model. So far we have only discussed the results for the seven unconstrained problems, since they were amenable to solution by all the search strategies. Problem 3.8, with constraints, corresponds to the second model function (corridor model) for which Rechenberg (1973) has obtained theoretically the rate of progress of the two membered evolution strategy with optimal adaptation of variances. According to his analysis, one expects a linear rate of convergence increasing with the width of the corridor and inversely proportional to the number of variables. The results of the third set of tests conrm that the number of mutations or generations increases linearly with n if the width of the corridor and the reference distance to be covered are held constant. The picture for the Rosenbrock strategy is as usual: the time consumption increases as O(n3 ) again. The point at n = 75

232

Comparison of Direct Search Strategies for Parameter Optimization

departs from the general trend of the others simply because no orthogonalizations were performed in this case. But the di erence is not dramatic, because the cost of testing the constraints is of the same order of magnitude as that of rotating the coordinates. The complex method takes computation times that initially increase somewhat more rapidly than O(n3 ). This corresponds to a greater than linearly increasing number of objective function evaluations. As we have already seen in other problems, the increase becomes even steeper as the number of parameters increases. With n = 95 variables, the required distance was only partially covered within the maximum computation time. Problem 3.9 represents a modication of Problem 3.8 with respect to the constraints. In place of the (2 n ; 2) linear constraints, the corridor is bounded by a single non-linear boundary condition. The cost of testing the feasibility of an iteration point is thereby greatly reduced. The number of mutations or generations of the evolution strategies is higher than in Problem 3.8 but still increases as O(n) the computation times in contrast to Problem 3.8 only increase as O(n2). The Rosenbrock method also has no diculty with this problem, although the necessary rotations of the coordinate system make the times of order O(n3 ). The complex method could only solve Problem 3.9 for n = 3 upwards of n = 10 it no longer converged. The last problem, Problem 3.10, which also has inequality constraints, turned out to be extremely dicult for all the search methods in the test. The main problem is one of scaling. Convergence in the neighborhood of the minimum can be achieved if, and practically only if, the step lengths in the coordinate directions are individually adjustable. They have to di er from each other by several powers of 10. For n = 30, no strategy managed to solve the problem within the maximum allowed computation time. The complex method sometimes failed to end the search within this time for n = 10. The intermediate results achieved after 8 hours are presented in Appendix A, Section A.3. All of the evolution strategies do better than the methods of Rosenbrock and Box. The result that the two membered evolution strategy came closer to the objective than the multimembered evolution without recombination was not completely unexpected, because considerably fewer generations than mutations can occur within the allowed time. What is more surprising is that the (10 , 100) strategy with recombination does almost as well as the two membered version. Here once again, the degree of freedom gained by the possibilities of recombination shows itself to advantage. The variances of the mutation step lengths do adjust themselves individually quite di erently according to the situation and thus permit much faster convergence than with equal variances for all variables. The other evolution strategies only come as close as they do to the solution because the variances reach their relative lower bounds at di erent times, whereby di erences in their sizes are introduced. This scaling process is, however, very much slower than the continuous process of adaptation brought about by the recombination mechanism.

6.4 Core storage required Up to now, only the time has been considered as a measure of the computational cost. There is, however, another important characteristic that a ects the applicability of optimization strategies, namely the core storage required. (Today nobody would use this

Core storage required

233

term \core" here, but at the time these tests were performed, it was so called.) All indirect methods of quadratic optimization, which solve the linear equations for the extremal, require storage of order O(n2 ) for the matrix of coecients. The same holds for quasiNewton methods, except that here the signicant r^ole is played by the approximation to the inverse Hessian matrices. Most strategies that perform line searches in other than coordinate directions also require O(n2 ) words for the storage of n vectors, each with n coecients. An exception to this rule is the conjugate gradient method of Fletcher and Reeves, which at each stage only needs to retain the latest generated direction vector for the subsequent iteration. Of the direct search methods included in the tests, the coordinate methods, the method of Hooke and Jeeves, and the evolution strategies work with only O(n) words of core storage. How important the formal storage requirement of an optimization method can be is shown by the maximum number of variables for the tested strategies in Table 6.2. The limiting values range from 75 to 4,000 under the given conditions. There exist, of course, tricks such as segmentation for enabling larger programs to be run on smaller machines the cost of the strategy should then take into account, however, the extra cost in preparation time for an optimization. (Here again, modern virtual storage techniques and the relative cheapness of memory chips make the considerations above look rather old-fashioned.) In the following Table 6.11, all the strategies compared are listed again, together with the order of magnitude of their required computation time as obtained from the rst set of tests (columns 1 and 2). The third column shows how the computation time would vary if each function call performed O(n2) rather than O(n) operations, as would occur for the worst case of a general quadratic objective function. The fourth column gives the storage requirement, again only as an order of magnitude, and the fth displays the product of the time and storage requirements from the two previous columns. Judging by the computation computation time alone, the variable metric strategy seems the best suited for true quadratic problems. In the least favorable case, however, it is more expensive than an indirect method and only faster in special cases. Problems having a very simple structure (e.g., Problem 1.1) can be solved just as well by direct search methods the time they take is at worst only a constant factor more than that of a second order method. If the total cost is measured by the product of time and storage requirements, all those strategies that store a two dimensional array of data, show up badly at least for problems with many variables. Since the coordinate methods have shown unreliable convergence, the method of Hooke and Jeeves and the evolution strategies remain as the least costly optimization methods. Their cost does not exceed that of indirect methods. The product of time and storage is not such a bad measure of the total cost in many computing centers jobs have been, in fact, charged with the product of storage requested in K words and the time in seconds of occupation of the central processing unit (K-core-sec). A comparison of the two membered and multimembered evolution strategies seems clearly to favor the simpler method. This is not surprising as several individuals in the multimembered procedure have to nd their way towards the optimum. In nature, this process runs in parallel. Already in the early 1970s, rst e orts towards constructing multi-processor computers were undertaken (see Barnes et al., 1968 Miranker, 1971).

234

Comparison of Direct Search Strategies for Parameter Optimization Table 6.11: The dependence of the total costs of the search methods on the number of variables (n) Strategy Computation Computation Computation Core K-core-sec time for time for time for gen. storage Problem 1.1 Problem 1.2 quadr. probl.

FIBO,GOLD,LAGR HOJE DSCG DSCP POWE DFPS SIMP ROSE COMP EVOL,GRUP,REKO

n2 > n2 n2 n2 n2 n2 > n3 n3 > n3 n2

n3 y n3 n4 n3 n3 y n2:5 n5 n4 n5 y n3

n4 n4 n4 n4 n4 n3:5 n5 n4 n5 n4

n n n2 n2 n2 n2 n2 n2 n2 n

n5 n5 n6 n6 n6 n5:5 n7 n6 n7 n5

On such a parallel computer, supposing it had 100 sub-units, one could simultaneously perform all the mutations and objective function evaluations of one generation in the (10 , 100) evolution strategy. The time required for the optimization would be about two orders of magnitude less than it is with a serially operating machine. In Figures 6.14 and 6.15 the dotted lines show the results that would be obtained by the (10 , 100) strategy without recombination in the hypothetical case of parallel operation. No other methods can make use of parallel operations to such an extent. On SIMD (single instructions, multiple data) architectures, the possible speedup is sharply limited by the percentages of a program's scalar and vector operations. Using array arithmetic for all matrix and vector operations, the execution time of a program may be accelerated at most by a factor of ve, given that these operations would serially take 80% of the computation time. On MIMD (multiple instructions, multiple data) machines, the speedup is limited by the number of processing units a program can make use of and by the amount of communication needed between the processors and the data store(s). Most classical optimization algorithms cannot economically employ large MIMD computers{even the less, the more sophisticated the procedures are. Multimembered evolution strategies, however, are easily scalable to any number of processors and communication links between them. For a taxonomy of parallel versions of evolution strategies, see Ho meister and Schwefel (1990).

yNot sure to converge

Chapter 7 Summary and Outlook So, is the evolution strategy the long-sought-after universal method of optimization? Unfortunately, things are not so simple and this question cannot be answered with a clear \yes." In two situations, in particular, the evolution strategies proved to be inferior to other methods: for linear and quadratic programming problems. These cases demonstrate the full eectiveness of methods that are specially designed for them, and that cannot be surpassed by strategies that operate without an adequate internal model. Thus if one knows the topology of the problem to be solved and it falls into one of these categories, one should always make use of such special methods. For this reason there will always rightly exist a number of dierent optimization methods. In other cases one would naturally not search for a minimum or maximum iteratively if an analytic approach presented itself, i.e., if the necessary existence conditions lead to an easily and uniquely soluble system of equations. Nearest to this kind of indirect optimization come the hill climbing strategies, which operate with a global internal model. They approximate the relation between independent and dependent variables by a function (e.g., a polynomial of high order) and then follow the analytic route, but within the model rather than reality. Since the approximation will inevitably not be exact, the process of analysis and synthesis must be repeated iteratively in order to locate an extremum exactly. The rst part, identication of the parameters or construction of the model, costs a lot in trial steps. The cost increases with n, the number of variables, and p, the order of the tting polynomial, as O(n ). For this reason hill climbing methods usually keep to a linear model (rst order strategies, gradient methods) or a quadratic model (second order strategies, Newton methods). All the more highly developed methods also try as infrequently as possible to adjust the model to the local topology (e.g., the method of steepest descents) or to advance towards the optimum during the model construction stage (e.g., the quasi-Newton and conjugate gradient strategies). Whether this succeeds, and the information gathered is su cient, depends entirely on the optimization problem in question. A quadratic model seems obviously more suited to a non-linear problem than a linear model, but both have only a limited, local character. Thus in order to prove that the sequence of iterations converges and to make general statements about the speed of convergence and the Q-properties, very strict conditions must be satised by the objecp

235

236

Summary and Outlook

tive function and, if they exist, also by the constraints, such as unimodality, convexity, continuity, and dierentiability. Linear or quadratic convergence properties require not only conditions on the structure of the problem, which frequently cannot be satised, but also presuppose that the mathematical operations are in principle carried out with innite accuracy . Many an attractive strategy thus fails not only because a problem is \pathological," having non-optimal stationary points, an indenite Hessian matrix, or discontinuous partial derivatives, but simply because of inevitable rounding errors in the calculation, which works with a nite number of signicant gures. Theoretical predictions are often irrelevant to practical problems and the strength of a strategy certainly lies in its capability of dealing with situations that it recognizes as precarious: for example, by cyclically erasing the information that has been gathered or by introducing random steps. As the test results conrm, the second order methods are particularly susceptible. A questionable feature of their algorithms is, for example, the line search for relative optima in prescribed directions. Contributions to all conferences in the late 1970s clearly showed a leaning towards strategies that do not employ line searches, thereby requiring more iterations but oering greater stability. The simpler its internal model, the less complete the required information, the more robust an optimization strategy can be. The more rigid the representation of the model is, the more eect perturbations of the objective function have, even those that merely result from the implementation on digital, analogue, or hybrid computers. Strategies that accept no worsening of the objective function are very easily led astray. Every attempt to accelerate the convergence is paid for by loss in reliability. The ideal of guaranteed absolute reliability, from which springs the stochastic approximation (in which the measured objective function values are assumed to be samples of a stochastic, e.g., Gaussian distribution), leads directly to a large reduction in the rates of convergence. The starkest contradiction, however, between the requirements for speed and reliability can be seen in the problem of discovering a global optimum among several local optima. Imagine the situation of a blind person who arrives at New York and wishes, without assistance or local knowledge, to reach the summit of Mt. Whitney. For how long might he seek? The task becomes far more formidable if there are more than two variables (here longitude and latitude) to determine. The most reliable global search method is the volume-oriented grid method, which at the same time is the costliest. In the multidimensional case its information requirement is too huge to be satised. There is, therefore, often no alternative but to strike a compromise between reliability and speed. Here we might adopt the sequential random search with normally distributed steps and xed variances. It has the property of always maintaining a chance of global convergence, and is just as reliable (although slower) in the presence of stochastic perturbations. It also has a path-oriented character: According to the sizes of the selected standard deviations of the random components, it follows more or less exactly the gradient path and thus avoids testing systematically the whole parameter space. A further advantage is that its storage requirement increases only linearly with the number of variables. This can sometimes be a decisive factor in favor of its implementation. Most of the deterministic hill climbing methods require storage space of order O(n2 ). The simple operations of the algorithm guarantee the least eect of rounding errors and are safe from forbidden

Summary and Outlook

237

numerical operations (division by zero, square root of a negative number, etc.). No conditions of continuity or dierentiability are imposed on the objective function. These advantages accrue from doing without an internal model, not insisting on an improvement at each step, and having an almost unlimited set of search directions and step lengths. It is surely not by chance that this method of zero order corresponds to the simplest rules of organic evolution, which can also cope, and has coped, with di cult situations. Two objections are nevertheless sometimes raised against the analogy of mutations to random steps. The rst is directed against randomness as such. A common point of view, which need not be explicitly countered, is to equate randomness with arbitraryness, even going so far as to suppose that \random" events are the result of a superhuman hand sharing out luck and misfortune but it is then further asserted that mutations do after all have causes, and it is concluded that they should not be regarded as random. Against this it can be pointed out that randomness and causality are not contradictory concepts. The statistical point of view that is expressed here simply represents an avoidance of statements about individual events and their causes. This is especially useful if the causal relation is very complicated and one is really only interested in the global behavioral laws of a stochastic set of events, as they are expressed by probability density distributions. The treatment of mutations as stochastic events rather than otherwise is purely and simply a reection of the fact that they represent undirected and on average small deviations from the initial condition. Since one has had to accept that non-linear dynamic systems rather frequently produce behaviors called deterministic chaos (which in turn is used to create pseudorandom numbers on computers), arguments against speaking of random events in nature have diminished considerably. The second objection concerns the unbelievably small probability, as proved by calculation, that a living thing, or even a mere wristwatch, could arise from a chance step of nature. In this case, biological evolution is implicitly being equated to the simultaneous, pure random methods that resemble the grid search. In fact the achievements of nature are not explicable with this model concept. If mutations were random events evenly distributed in the whole parameter space it would follow that later events would be completely independent of the previous results that is to say that descendants of a particular parent would bear no resemblance to it. This overlooks the sequential character of evolution, which is inherent in the consecutive generations. Only the sequential random search can be regarded as an analogue of organic evolution. The changes from one generation to the next, expressed as rates of mutation, are furthermore extremely small. The fact that this must be so for a problem with so many variables is shown by Rechenberg's theory of the two membered evolution strategy: optimal (i.e., fastest possible) progress to the optimum is achieved if, and only if, the standard deviations of the random components of the vector of changes are inversely proportional to the number of variables. The 1=5 success rule for adaptation of the step length parameters does not, incidentally, have a biological basis rather it is suited to the requirements of numerical optimization. It allows rates of convergence to be achieved that are comparable to those of most other direct search strategies. As the comparison tests show, because of its low computational cost per iteration the evolution strategy is actually far superior to some

238

Summary and Outlook

methods for many variables, for example those that employ costly orthogonalization processes. The external control of step lengths sometimes, however, worsens the reliability of the strategy. In \pathological" cases it leads to premature termination of the search and reduces besides the chance of global convergence. Now instead of concluding like Bremermann that organic evolution has only reached stagnation points and not optima, for example, in ecological niches, one should rather ask whether the imitation of the natural process is su ciently perfect. One can scarcely doubt the capability of evolution to create optimal adaptations and ever higher levels of development the already familiar examples of the achievements of biological systems are too numerous. Failures with simulated evolution should not be imputed to nature but to the simulation model. The two membered scheme incorporates only the principles of mutation and selection and can only be regarded as a very simple basis for a true evolution strategy. On the other hand one must proceed with care in copying nature, as demonstrated by Lilienthal's abortive attempt, which is ridiculed nowadays, to build a ying machine by imitating the birds. The objective, to produce high lift with low drag, is certainly the same in both cases, but the boundary conditions (the ow regime, as expressed by the Reynolds number) are not. Bionics, the science of evaluating nature's patents, teaches us nowadays to beware of imitating in the sense of slavishly copying all the details but rather to pay attention to the principle. Thus Bremermann's concept of varying the variables individually instead of all together must also be regarded as an inappropriate way to go about an optimization with continuously variable quantities. In spite of the many, often very detailed investigations made into the phenomenon of evolution, biology has oered no clues as to how an improved imitation should look, perhaps because it has hitherto been a largely descriptive rather than analytic science. The di culties of the two membered evolution with step length adaptation teach us to look here to the biological example. It also alters the standard deviations through the generations, as proved by the existence of mutator genes and repair enzymes. Whilst nature cannot inuence the mutation-provoking conditions of the environment, it can reduce their eects to whatever level is suitable. The step lengths are genetically determined they can be thought of as strategy parameters of nature that are subject to the mutation-selection process just like the object parameters. To carry through this principle as the algorithm of an improved evolution strategy one has to go over from the two membered to a multimembered scheme. The ( , ) strategy does so by employing the population principle and allowing  parents in each generation to produce  descendants, of which the  best are selected as parents of the following generation. In this way the sequential as well as the simultaneous character of organic evolution is imitated the two membered concept only achieves this insofar as a single parent produces descendents until it is surpassed by one of them in vitality, the biological criterion of goodness. According to Rechenberg's hypothesis that the forms and intricacies of the evolutionary process that we observe today are themselves the result of development towards an optimal optimization strategy, our measures should lead to improved results. The test results show that the reliability of the (10 , 100) strategy, taken as an example, is indeed better than that of the (1+1) evolution strategy. In particular, the chances of locating global optima in multimodal problems have become

Summary and Outlook

239

considerably greater. Global convergence can even be achieved in the case of a non-convex and disconnected feasible region. In the rate of convergence test the (10 , 100) strategy does a lot worse, but not by the factor 100 that might be expected. In terms of the number of required generations, rather than the computation time, the multimembered strategy is actually considerably faster. The increase in speed compared to the two membered method comes about because not only the sign of 4F , the change in the function value, but also its magnitude plays a r^ole in the selection process. Nature possesses a way of exploiting this advantage that is denied to conventional, serially operating computers: It operates in parallel. All descendants of a generation are produced at the same time, and their vitality is tested simultaneously. If nature could be imitated in this way, the ( , ) strategy would make both a very reliable and a fast optimization method. The following two paragraphs, though completely out of date, have been left in place mainly to demonstrate the considerable shift in the development of computers during the last 20 years (compare with Schwefel, 1975a). Meanwhile parallel computers are beginning to conquer desk tops. Long and complicated iterative processes, such as occur in many other branches of numerical mathematics, led engineers and scientists of the University of Illinois, U.S.A., to consider new ways of reducing the computation times of programs. They built their own computer, Illiac IV, which has especially short data retrieval and transfer times (Barnes et al., 1968). They were unable to approach the 1020 bits/sec given by Bledsoe (1961) as an upper limit for serial computers, but there will inevitably always be technological barriers to achieving this physical maximum. A novel organizational principle of Illiac IV is much more signicant in this connection. A bank of satellite computers are attached to a central unit, each with its own processor and access to a common memory. The idea is for the sub-units to execute simultaneously various parts of the same program and by this true parallel operation to yield higher eective computation speeds. In fact not every algorithm can take advantage of this capability, for it is impossible to execute two iterations simultaneously if the result of one inuences the next. It may sometimes be necessary to reconsider and make appropriate modications to conventional methods, e.g., of linear algebra, before the advantages of the next generation of computers can be exploited. The potential and the problems of implementing parallel computers are already receiving close attention: Shedler (1967), Karp and Miranker (1968), Miranker (1969, 1971), Chazan and Miranker (1970), Abe and Kimura (1970), Sameh (1971), Patrick (1972), Gilbert and Chandler (1972), Hansen (1972), Eisenberg and McGuire (1972), Casti, Richardson, and Larson (1973), Larson and Tse (1973), Miller (1973), Stone (1973a,b). A version of FORTRAN for parallel computers has already been devised (Millstein, 1973). Another signicant advantage of the multimembered as against the two membered scheme that also holds for serial calculations is that the self-adjustment of step lengths can be made individually for each component. An automatic scaling of the variables results from

240

Summary and Outlook

this, which in certain cases yields a considerable improvement in the rate of progress. It can be achieved either by separate variation of the standard deviations  for i = 1(1)n, by recombination alone, or, even better, (0) by both measures together. Whereas in the two membered scheme, in which (unless the  are initially given dierent values) the contour lines of equiprobable steps are circles, or hyperspherical surfaces, they are now ellipses or hyperellipsoids that can extend or contract along the coordinate directions following the n-dimensional normal distribution of the set of n random components z for i = 1(1)n :
X  2 ! 1 1 z w(z) = exp ; 2 =1  (2) n2 Q  i

i

i

n

i

n

i

=1

i

i

i

This is not yet, however, the most general form of a normal distribution, which is rather: pDet A  1  w(z) = (2) n2 exp ; 2 (z ; ) A (z ; ) The expectation value vector  can be regarded as a deterministic part of the random step z. However, the comparison made by Schrack and Borowski (1972) between the random strategies of Schumer-Steiglitz and Matyas shows that even an ingenious learning scheme for adapting  to the local conditions only improves the convergence in special cases. A much more important feature seems to be the step length adaptation. It is now possible for the elements of the matrix A to be chosen so as to give the ellipsoid of variation any desired orientation in the space. Its axes, the regression directions of the random vector, only coincide with the coordinate axes if A is a diagonal matrix. In that case the old scheme is recovered whereby the variances  or the 2 reappear as diagonal elements of the inverse matrix A;1. If, however, the other elements, the covariances  =  are nonzero, the ellipsoids are rotated in the space. The random components z become mutually dependent, or correlated. The simplest kind of correlation is linear, which is the only case to yield hyperellipsoids as surfaces of constant step probability. Instead of just n strategy parameters  one would now have to vary 2 (n + 1) dierent quantities  . Although in principle the multimembered evolution strategy allows an arbitrary number of strategy variables to be included in the mutation-selection process, in practice the adaptation of so many parameters could take too long and cancel out the advantage of more degrees of freedom. Furthermore, the  must satisfy certain compatibility conditions (Sylvester's criteria, see Faddejew and Faddejewa, 1973) to ensure an orthogonal coordinate system or a positive denite matrix A. In the simplest case, n = 2, with T

ii

i

ij

ji

i

n

ij

i

ij

A;1 = there is only one condition: and the quantity dened by

"

11 12 21 22

#

122 = 212 < 11 22 = 12 22 12 = q 12 (1 2)

;1 <  < 1 12

Summary and Outlook

241

is called the correlation coe cient. If the covariances were generated independently by a mutation process in the multimembered evolution scheme, with subsequent application of the rules of Scheuer and Stoller (1962) or Barr and Slezak (1972), there would be no guarantee that the surfaces of equal probability density would actually be hyperellipsoids. It follows that such a linear correlation of the random changes can be constructed more easily by rst generating as before (0 2) normally distributed, independent random components and then making a coordinate rotation through prescribed angles. These angles, rather than the covariances  , represent the additional strategy variables. In the most general case there are a total of n = 2 (n ; 1) such angles, which can take all values between 00 and 3600 (or ; and ). Including the n = n \step lengths"  , the total number of strategy parameters to be specied in the population by mutation and selection is 2 (n + 1). It is convenient to generate the angles by an additive mutation process (cf. Equations (5.36) and (5.37)) i

ij

p

n

s

i

n

j

( ) = ( ) + Z^ ( ) for j = 1(1)n g

g

N
j

E
j

g

p

j

where the Z^ ( ) can again be normally distributed, for example, with a standard deviation 4 which is the same for all angles. Let 4x0 represent the mutations as produced by the old scheme and 4x the correlated changes in the object variables produced by the rotation for the two dimensional case (n = n = 2 n = 1) the coordinate transformation for the rotation can simply be read o from Figure 7.1 g

j

i

i

s

p

4x = 4x0 cos ; 4x0 sin

4x = 4x0 sin + 4x0 cos

1

1

2

2

1

2

For n = n = 3 three consecutive rotations would need to be made: s

 In the (4x 4x ) plane through an angle

 In the (4x0 4x0 ) plane through an angle

 In the (4x00 4x00) plane through an angle

Starting from the uncorrelated random changes 4x000 4x000 4x000 these rotations would have to be made in the reverse order. Thus also, in the general case with (n ; 1) 1

2

1

1

2

2

2

3

3

1

2

3

n

2

rotations, each one only involves two coordinates so that the computational cost increases as O(n ). The validity of this algorithm has been proved by Rudolph (1992a). An immediate simplication can be made if not all the n step lengths are dierent, i.e., if the hyperellipsoid of equal probability of a mutation has rotational symmetry about one or more axes. In the extreme case n = 2 there are n ; n such axes and only n = n ; 1 relevant angles of rotation. Except for one distinct principle axis, the ellipsoid resembles a sphere. If in the course of the optimization the minimum search leads through a narrow valley (e.g., in Problem 2.37 or 3.8 of the catalogue of test problems), it will often be quite adequate to work with such a greatly reduced variability of the mutation ellipsoid. p

s

s

s

p

242

Summary and Outlook

x

2

x’

x’

2

1

∆ x1

α

x

∆ x2’

∆ x’

1

1

∆x

2

Mutation

Figure 7.1: Generation of correlated mutations

Between the two extreme cases n = n and n = 2 (n = 1 would be the uncorrelated case with hyperspheres as mutation ellipsoids) any choice of variability is possible. In general we have 2n n  n n = n ; 2 (n ; 1) For a given problem the most suitable choice of n , the number of dierent step lengths, would have to be obtained by numerical experiment. For this purpose the subroutine KORR and its associated subroutines listed in Appendix B, Section B.3 is exibly coded to give the user considerable freedom in the choice of quantities that determine the strategy parameters. This variant of the evolution strategy (Schwefel, 1974) could not be fully included in the strategy test (performed in 1973) however, initial results conrmed that, as expected, it is able to construct a kind of variable metric for the changes in the object variables by adapting the angles to the local topology of the objective function. The slow convergence of the two membered evolution strategy can often be traced to the fact that the problem has long elliptical (or nearly elliptical) contours of constant objective function value. If the function is quadratic, their extension (or eccentricity) can be expressed by the condition number of the matrix of second order coe cients. In the worst case, in which the search is started at the point of greatest curvature of the contour surface F (x) = const:, the rate of progress seems to be inversely proportional s

s

s

s

s

p

s

s

Summary and Outlook

243

to the product of the number of variables and the square root of the condition number. This dependence on the metric would be eliminated if the directions of the axes of the variance ellipsoid corresponded to those of the contour ellipsoid, which is exactly what the introduction of correlated random numbers should achieve. Extended valleys in other than coordinate directions then no longer hinder the search because, after a transition phase, an initially elliptical problem is reduced to a spherical one. In this way the evolution strategy acquires properties similar to those of the variable metric method of DavidonFletcher-Powell (DFP). In the test, for just the reason discussed above, the latter proved to be superior to all other methods for quadratic objective functions. For such problems one should not expect it to be surpassed by the evolution strategy, since compared to the Q n property of the DFP method the evolution strategy has only a Q O(n) property i.e., it does not nd the optimum after exactly n iterations but rather it reaches a given approximation to the objective after O(n) generations. This disadvantage, only slight in practice, is outweighed by the following advantages:  Greater exibility, hence reliability, in other than quadratic cases  Simpler computational operations  Storage required increases only as O(n) (unless one chooses n = n) While one has great hopes for this extension of the multimembered evolution strategy, one should not be blinded by enthusiasm to limitations in its capability. It would yield computation times no better than O(n3) if it turns out that a population of O(n) parents is needed for adjusting the strategy parameters and if pure serial rather than parallel computation is necessary. Does the new scheme still correspond to the biological paradigm? It has been discovered that one gene often inuences several phenotypic characteristics of an individual (pleiotropy) and conversely that many characteristics depend on the cooperative eect of several genes (polygeny). These interactions just mean that the characteristics are correlated. A linear correlation as in Figure 7.1 represents only one of the many conceivable types in which (x01 x02) is the plane of the primary, independent genetic changes and (x1 x2) that of the secondary, mutually correlated changes in the characteristics. Particular kinds of such dependence, for example, allometric growth, have been intensively studied (e.g., Grasse, 1973). There is little doubt that the relationships have also adapted, during the history of development, to the topological requirements of the objective function. The observable dierences between life forms are at least suggestive of this. Even non-linear correlations may occur. Evolution has indeed to cope with far greater di culties, for it has no ordered number system at its disposal. In the rst place it had to create a scale of measure{with the genetic code, for example, which has been learned during the early stages of life on earth. Whether it is ultimately worth proceeding so far or further to mimic evolution is still an open question, but it is surely a path worth exploring perhaps not for continuous, but for discrete or mixed parameter optimization. Here, in place of the normal distribution of random changes, a discrete distribution must be applied, e.g., a binomial or better still a distribution with maximum entropy (see Rudolph, 1994b), so that for small \total s

244

Summary and Outlook

step lengths" the probability really is small that two or more variables are altered simultaneously. Occasional stagnation of the search will only be avoided, in this case, if the population allows worsening within a generation. Worsening is not allowed by the two membered strategy, but it is by the multimembered ( , ) strategy , in which the parents, after producing descendants, no longer enter the selection process. Perhaps this shows that the limited life span of individuals is no imperfection of nature, no consequence of an inevitable weakness of the system, but rather an intelligent, indeed essential means of survival of the species. This conjecture is again supported by the genetically determined, in eect preprogrammed, ending of the process of cell division during the life of an individual. Sequential improvement and consequent rapid optimization is only made possible by the following of one generation after another. However, one should be extremely wary of applying such concepts directly to mankind. Human evolution long ago left the purely biological domain and is more active nowadays in the social one. One properly refers now to a cultural evolution. There is far too little genetic information to specify human behavior completely. Little is known of which factors are genetically inherited and which socially acquired, as shown by the continuing discussions over the results of behavioral research and the diametrically opposite points of view of individual scientists in the eld. The two most important evolutionary principles, mutation and selection, also belong to social development (Alland, 1970). Actually, even more complicated mechanisms are at work here. Oversimplications can have quite terrifying consequences, as shown by the example of social Darwinism, to which Koch (1973) attributes responsibility for racist and imperialist thinking and hence for the two World Wars. No such further speculation with the evolution strategy will therefore be embarked upon here. The fact remains that the recognition of evolution as representing a sequential optimization process is too valuable to be dismissed to oblivion as evolutionism (Goll, 1972). Rather one should consider what further factors are known in organic evolution that might be worth imitating, in order to make of the evolution strategy an even more general optimization method for up to now several developments have conrmed Rechenberg's hypothesis that the strategy can be improved by taking into account further factors, at least when this is done adequately and the biological and mathematical boundary conditions are compatible with each other. Furthermore, by no means all evolutionary principles have yet been adopted for optimizing technical systems. The search for global optima remains a particularly di cult problem. In such cases nature seems to hunt for all, or at least a large number of maxima or minima at the same time by the splitting of a population (the isolation principle). After a transition phase the individuals of both or all the subpopulations can no longer intermix. Thereafter each group only seeks its own specic local optimum, which might perhaps be the global one. This principle could easily be incorporated into the multimembered scheme if a criterion could be dened for performing the splitting process. Many evolution principles that appeared later on the scene can be explained as affording the greater chance of survival to a population having the better mechanism of inheritance (for these are also variable) compared to an other forming a worse \strategy of life." In this way the evolution method could itself be optimized by organizing a compe-

Summary and Outlook

245

tition between several populations that alter the concept of the optimum seeking strategy itself. The simplest possibility, for example, would be to vary the number of parents  and of descendants  two or more groups would be set up each with its own values of these parameters, each group would be given a xed time to seek the optimum then the group that has advanced the most would be allowed to \survive." In this way these strategy variables would be determined to best suit the particular problem and computer, with the objective of minimizing the required computation time. One might call such an approach meta- or hierarchical evolution strategy (see Back, 1994a,b). The solution of problems with multiple objectives could also be approached with the multimembered evolution strategy. This is really the most common type of problem in nature. The selection step, the reduction to the best of the descendants, could be subdivided into several partial steps, in each of which only one of the criteria for selection is applied. In this way no weighting of the partial objectives would be required. First attempts with only two variables and two partial objectives showed that a point on the Pareto line is always approached as the optimum. By unequal distribution of the partial selections the solution point could be displaced towards one of the partial objectives. At this stage subjective information would have to be applied because all the Pareto-optimal solutions are initially equally good (see Kursawe, 1991, 1992). Contrary to many statements or conjectures that organic evolution is a particularly wasteful optimization process, it proves again and again to be precisely suited to advancing with maximum speed without losing reliability of convergence, even to better and better local optima. This is just what is required in numerical optimization. In both cases the available resources limit what can be achieved. In one case these are the limitations of food and the nite area of the earth for accommodating life, in the other they are the nite number of satellite processors of a parallel-organized mainframe computer and its limited (core) storage space. If the evolution strategy can be considered as the sought-after universal optimization method, then this is not in the sense that it solves a particular problem (e.g., a linear or quadratic function) exactly, with the least iterations or generations, but rather refers to its being the most readily extended concept, able to solve very di cult problems, problems with particularly many variables, under unfavorable conditions such as stochastic perturbations, discrete variables, time-varying optima, and multimodal objective functions (see Hammel and Back, 1994). Accordingly, the results and assessments introduced in the present work can at best be considered as a rst step in the development towards a universal evolution strategy. Finally, some early applications of the evolution strategy will be cited. Experimental tasks were the starting point for the realization of the rst ideas for an optimization strategy based on the example of biological evolution. It was also rst applied here to the solution of practical problems (see Schwefel, 1968 Klockgether and Schwefel, 1970 Rechenberg, 1973). Meanwhile it is being applied just as widely to optimization problems that can be expressed in computational or algorithmic form, e.g., in the form of simulation models. The following is a list of some of the successful applications, with references to the relevant publications. 1. Optimal dimensioning of the core of a fast sodium-type breeder reactor (Heusener, 1970)

246

Summary and Outlook

2. Optimal allocation of investments to various health-service programs in Columbia (Schwefel, 1972) 3. Solving curve-tting problems by combining a least-squares method with the evolution strategy (Plaschko and Wagner, 1973) 4. Minimum-weight designing of truss constructions partly in combination with linear programming (Ley!ner, 1974 and Hoer, 1976) 5. Optimal shaping of vaulted reinforced concrete shells (Hartmann, 1974) 6. Optimal dimensioning of quadruple-joint drives (Anders, 1977) 7. Approximating the solution of a set of non-linear dierential equations (Rodlo, 1976) 8. Optimal design of arm prostheses (Brudermann, 1977) 9. Optimization of urban and regional water supply systems (Cembrowicz and Krauter, 1977) 10. Combining the evolution strategy with factorial design techniques (Kobelt and Schneider, 1977) 11. Optimization within a dynamic simulation model of a socioeconomic system (Krallmann, 1978) 12. Optimization of a thermal water jet propulsion system (Markwich, 1978) 13. Optimization of a regional system for the removal of refuse (von Falkenhausen, 1980) 14. Estimation of parameters within a model of oods (North, 1980) 15. Interactive superimposing of dierent direct search techniques onto dynamic simulation models, especially models of the energy system of the Federal Republic of Germany (Heckler, 1979 Drepper, Heckler, and Schwefel, 1979). Much longer lists of references concerning applications as well as theoretical work in the eld of evolutionary computation have been compiled meanwhile by Alander (1992, 1994) and Back, Homeister, and Schwefel (1993). Among the many dierent elds of applications only one will be addressed here, i.e., non-linear regression and correlation analysis. In general this leads to a multimodal optimization problem when the parameters searched for enter the hypotheses non-linearly, e.g., as exponents. Very helpful under such circumstances is a tool with which one can switch from one to the other minimization method. Beginning with a multimembered evolution strategy and rening the intermediate results by means of a variable metric method has often led to practically useful results (e.g., Frankhauser and Schwefel, 1992). In some cases of practical applications of evolution strategies it turns out that the number of variables describing the objective function has to vary itself. An example was

Summary and Outlook

247

the experimental optimization of the shape of a supersonic one-component two-phase ow nozzle (Schwefel, 1968). Conically bored rings with xed lengths could be put side by side, thus forming potentially millions of dierent inner nozzle contours. But the total length of the nozzle had to be varied itself. So the number of rings and thus the number of variables (inner diameters of the rings) had to be mutated during the search for an optimum shape as well. By imitating gene duplication and gene deletion at randomly chosen positions, a rather simple technique was found to solve the variable number of variables problem. Such a procedure might be helpful for many structural optimization problems (e.g., Rozvany, 1994) as well. If the decision variables are to be taken from a discrete set only (the distinct values may be equidistant or not integer and binary values just form special subclasses), ESs may be used sometimes without any change. Within the objective function the real values must simply undergo a suitable rounding-o process as shown at the end of Appendix B, Section B.3. Since all ESs handle unchanged objective function values as improvements, the selfadaptation of the standard deviations on a plateau will always lead to their enlargement, until the plateaus F (x) = const: built by rounding o can be left. On a plateau, the ES performs a random walk with ever increasing step sizes. Towards the end of the search, however, more and more of the individual step sizes have to become very small, whereas others{singly or in some combination{should be increased to allow hopping from one to the next n-cube in the decision-variable space. The chances for that kind of adaptation are good enough as long as sequential improvements are possible the last few of them will not happen that way, however. A method of escaping from that awkward situation has been shown (Schwefel, 1975b), imitating multicellular individuals and introducing so-called somatic mutations. Even in the case of binary variables an ES thus can reach the optimum. Since no real application has been done this way until now, no further details will be given here. An interesting question is whether there are intermediate cases between a plus and a comma version of the multimembered ES. The answer must be, \Yes, there are." Instead of neglecting the parents during the selection step (within comma-ESs), or allowing them to live forever in principle (within plus-ESs only until ospring surpass them, of course), one might implant a generation counter into each individual. As soon as a prexed limit is reached, they leave the scene automatically. Such a more general ES version could be termed a ( ) strategy, where denotes the maximal number of generations (iterations), an individual is allowed to \survive" in the population. For = 1 we then get the old comma-version, whereas the old plus-version is reached if goes to innity. There are some preliminary results now, but as yet they are too unsystematic to be presented here. Is the strict synchronization of the evolutionary process within ESs as well as GAs the best way to do the job? The answer to this even more interesting question is, \No," especially if one makes use of MIMD parallel machines or clusters of workstations. Then one should switch to imitating life more closely: Birth and death events may happen at the same time. Instead of modelling a central decision maker for the selection process (which is an oversimplication) one could use a predator-prey model like that of Lotka and Volterra. Adding a neighborhood model (see Gorges-Schleuter, 1991a,b Sprave, 1993, 1994) for the

248

Summary and Outlook

recombination process would free the whole strategy from all kinds of synchronization needs. Initial tests have shown that this is possible. Niching and migration as used by Rudolph (1991) will be the next features to be added to the APES (asynchronous parallel evolution strategy). A couple of earlier attempts towards parallelizing ESs will be mentioned at the end of this chapter. Since all of them are somehow intermediate solutions, however, none of them will be explained in detail. The reader is referred to the literature. A taxonomy, more or less complete with respect to possible ways of parallelizing EAs, may be found in Homeister and Schwefel (1990) or Homeister (1991). Rudolph (1991) has realized a coarse-grained parallel ES with subpopulations on each processor and more or less frequent migration events, whereas Sprave (1994) gave preference to a ne-grained diusion model. Both of these more volume-oriented approaches delivered great advances in solving multimodal optimization problems as compared with the more greedy and pathoriented \canonical" ( , ) ES. The comma version, by the way, is necessary to follow a nonstationary optimum (see Schwefel and Kursawe, 1992), and only such an ES is able to solve on-line optimization problems. Nevertheless, one should never forget that there are many other specialized optimum seeking methods. For a practitioner, a tool box with many dierent algorithms might always be the \optimum optimorum." Whether he or she chooses a special tool by hand, so to speak (see Heckler and Schwefel, 1978 Heckler, 1979 Schwefel, 1980, 1981 Hammel, 1991 Bendin, 1992 Back and Hammel, 1993), or relies upon some knowledgebased selection scheme (see Campos, 1989 Campos and Schwefel, 1989 Campos, Peters, and Schwefel, 1989 Peters, 1989, 1991 Lehner, 1991) will largely depend on his or her experience.

Chapter 8 References Glossary of abbreviations at the end of this list Aarts, E., J. Korst (1989), Simulated annealing and Boltzmann machines, Wiley, Chichester Abadie, J. (Ed.) (1967), Nonlinear programming, North-Holland, Amsterdam Abadie, J. (Ed.) (1970), Integer and nonlinear programming, North-Holland, Amsterdam Abadie, J. (1972), Simplex-like methods for non-linear programming, in: Szego (1972), pp. 41-60 Abe, K., M. Kimura (1970), Parallel algorithm for solving discrete optimization problems, IFAC Kyoto Symposium on Systems Engineering Approach to Computer Control, Kyoto, Japan, Aug. 1970, paper 35.1 Ablay, P. (1987), Optimieren mit Evolutionsstrategien, Spektrum der Wissenschaft (1987, July), 104-115 (see also discussion in (1988, March), 3-4 and (1988, June), 3-4) Ackley, D.H. (1987), A connectionist machine for genetic hill-climbing, Kluwer Academic, Boston Adachi, N. (1971), On variable-metric algorithms, JOTA 7, 391-410 Adachi, N. (1973a), On the convergence of variable-metric methods, Computing 11, 111-123 Adachi, N. (1973b), On the uniqueness of search directions in variable-metric algorithms, JOTA 11, 590-604 Adams, R.J., A.Y. Lew (1966), Modied sequential random search using a hybrid computer, University of Southern California, Electrical Engineering Department, report, May 1966 249

250

References

Ahrens, J.H., U. Dieter (1972), Computer methods for sampling from the exponential and normal distributions, CACM 15, 873-882, 1047 Aizerman, M.A., E.M. Braverman, L.I. Rozonoer (1965), The Robbins-Monro process and the method of potential functions, ARC 26, 1882-1885 Akaike, H. (1960), On a successive transformation of probability distribution and its application to the analysis of the optimum gradient method, Ann. Inst. Stat. Math. Tokyo 11, 1-16 Alander, J.T. (Ed.) (1992), Proceedings of the 1st Finnish Workshop on Genetic Algorithms and their Applications, Helsinki, Nov. 4-5, 1992, Bibliography pp. 203-281, Helsinki University of Technology, Department of Computer Science, Helsinki, Finland Alander, J.T. (1994), An indexed bibliography of genetic algorithms, preliminary edition, Jarmo T. Alander, Espoo, Finland Albrecht, R.F., C.R. Reeves, N.C. Steele (Eds.) (1993), Articial neural nets and genetic algorithms, Proceedings of an International Conference, Innsbruck, Austria, Springer, Vienna Aleksandrov, V.M., V.I. Sysoyev, V.V. Shemeneva (1968), Stochastic optimization, Engng. Cybern. 6(5), 11-16 Alland, A., Jr. (1970), Evolution und menschliches Verhalten, S. Fischer, Frankfort/Main Allen, P., J.M. McGlade (1986), Dynamics of discovery and exploitations|the case of the Scotian shelf groundsh sheries, Can. J. Fish. Aquat. Sci. 43, 1187-1200 Altman, M. (1966), Generalized gradient methods of minimizing a functional, Bull. Acad. Polon. Sci. 14, 313-318 Amann, H. (1968a), Monte-Carlo Methoden und lineare Randwertprobleme, ZAMM 48, 109-116 Amann, H. (1968b), Der Rechenaufwand bei der Monte-Carlo Methode mit Informationsspeicherung, ZAMM 48, 128-131 Anders, U. (1977), Losung getriebesynthetischer Probleme mit der Evolutionsstrategie, Feinwerktechnik und Me!technik 85(2), 53-57 Anderson, N., # A. Bjorck (1973), A new high order method of Regula Falsi type for computing a root of an equation, BIT 13, 253-264 Anderson, R.L. (1953), Recent advances in nding best operating conditions, J. Amer. Stat. Assoc. 48, 789-798 Andrews, H.C. (1972), Introduction to mathematical techniques in pattern recognition, Wiley-Interscience, New York

References

251

Anscombe, F.J. (1959), Quick analysis methods for random balance screening experiments, Technometrics 1, 195-209 Antonov, G.E., V.Ya. Katkovnik (1972), Method of synthesis of a class of random search algorithms, ARC 32, 990-993 Aoki, M. (1971), Introduction to optimization techniques|fundamentals and applications of nonlinear programming, Macmillan, New York Apostol, T.M. (1957), Mathematical analysis|a modern approach to advanced calculus, Addison-Wesley, Reading MA Arrow, K.J., L. Hurwicz (1956), Reduction of constrained maxima to saddle-point problems, in: Neyman (1956), vol. 5, pp. 1-20 Arrow, K.J., L. Hurwicz (1957), Gradient methods for constrained maxima, Oper. Res. 5, 258-265 Arrow, K.J., L. Hurwicz, H. Uzawa (Eds.) (1958), Studies in linear and non-linear programming, Stanford University Press, Stanford CA Asai, K., S. Kitajima (1972), Optimizing control using fuzzy automata, Automatica 8, 101-104 Ashby, W.R. (1960), Design for a brain, 2nd ed., Wiley, New York Ashby, W.R. (1965), Constraint analysis of many-dimensional relations, in: Wiener and Schade (1965), pp. 10-18 Ashby, W.R. (1968), Some consequences of Bremermann's limit for information-processing systems, in: Oestreicher and Moore (1968), pp. 69-76 Avriel, M., D.J. Wilde (1966a), Optimality proof for the symmetric Fibonacci search technique, Fibonacci Quart. 4, 265-269 Avriel, M., D.J. Wilde (1966b), Optimal search for a maximum with sequences of simultaneous function evaluations, Mgmt. Sci. 12, 722-731 Avriel, M., D.J. Wilde (1968), Golden block search for the maximum of unimodal functions, Mgmt. Sci. 14, 307-319 Axelrod, R. (1984), The evolution of cooperation, Basic Books, New York Azencott, R. (Ed.) (1992), Simulated annealing|parallelization techniques, Wiley, New York Bach, H. (1969), On the downhill method, CACM 12, 675-677 Back, T. (1992a), Self-adaptation in genetic algorithms, in: Varela and Bourgine (1992), pp. 263-271

252

References

Back, T. (1992b), The interaction of mutation rate, selection, and self-adaptation within a genetic algorithm, in: Manner and Manderick (1992), pp. 85-94 Back, T. (1993), Optimal mutation rates in genetic search, in: Forrest (1993), pp. 2-9 Back, T. (1994a), Evolutionary algorithms in theory and practice, Dr. rer. nat. Diss., University of Dortmund, Department of Computer Science, Feb. 1994 Back, T. (1994b), Parallel optimization of evolutionary algorithms, in: Davidor, Schwefel, and Manner (1994), pp. 418-427 Back, T., U. Hammel (1993), Einsatz evolutionarer Algorithmen zur Optimierung von Simulationsmodellen, in: Szczerbicka and Ziegler (1993), pp. 1-22 Back, T., U. Hammel, H.-P. Schwefel (1993), Modelloptimierung mit evolutionaren Algorithmen, in: Sydow (1993), pp. 49-57 Back, T., F. Homeister, H.-P. Schwefel (1991), A survey of evolution strategies, in: Belew and Booker (1991), pp. 2-9 Back, T., F. Homeister, H.-P. Schwefel (1993), Applications of evolutionary algorithms, technical report SYS-2/92, 4th ext. ed., Systems Analysis Research Group, University of Dortmund, Department of Computer Science, July 1993 Back, T., G. Rudolph, H.-P. Schwefel (1993), Evolutionary programming and evolution strategies|similarities and dierences, in: Fogel and Atmar (1993), pp. 11-22 Back, T., H.-P. Schwefel (1993), An overview of evolutionary algorithms for parameter optimization, Evolutionary Computation 1, 1-23 Baer, R.M. (1962), Note on an extremum locating algorithm, Comp. J. 5, 193 Balakrishnan, A.V. (Ed.) (1972), Techniques of optimization, Academic Press, New York Balakrishnan, A.V., M. Contensou, B.F. DeVeubeke, P. Kree, J.L. Lions, N.N. Moiseev (Eds.) (1970), Symposium on optimization, Springer, Berlin Balakrishnan, A.V., L.W. Neustadt (Eds.) (1964), Computing methods in optimization problems, Academic Press, New York Balakrishnan, A.V., L.W. Neustadt (Eds.) (1967), Mathematical theory of control, Academic Press, New York Balinski, M.L., P. Wolfe (Eds.) (1975), Nondierentiable optimization, vol. 3 of Mathematical Programming Studies, North-Holland, Amsterdam Bandler, J.W. (1969a), Optimization methods for computer-aided design, IEEE Trans. MTT-17, 533-552

References

253

Bandler, J.W. (1969b), Computer optimization of inhomogeneous waveguide transformers, IEEE Trans. MTT-17, 563-571 Bandler, J.W., C. Charalambous (1974), Nonlinear programming using minimax techniques, JOTA 13, 607-619 Bandler, J.W., P.A. MacDonald (1969), Optimization of microwave networks by razor search, IEEE Trans. MTT-17, 552-562 Banzhaf, W., M. Schmutz (1992), Some notes on competition among cell assemblies, Int'l J. Neural Syst. 2, 303-313 Bard, Y. (1968), On a numerical instability of Davidon-like methods, Math. Comp. 22, 665-666 Bard, Y. (1970), Comparison of gradient methods for the solution of nonlinear parameter estimation problems, SIAM J. Numer. Anal. 7, 157-186 Barnes, G.H., R.M. Brown, M. Kato, D.J. Kuck, D.L. Slotnick, R.A. Stokes (1968), The Illiac IV computer, IEEE Trans. C-17, 746-770 Barnes, J.G.P. (1965), An algorithm for solving non-linear equations based on the secant method, Comp. J. 8, 66-72 Barnes, J.L. (1965), Adaptive control as the basis of life and learning systems, Proceedings of the IFAC Tokyo Symposium on Systems Engineering Control and Systems Design, Tokyo, Japan, Aug. 1965, pp. 187-191 Barr, D.R., N.L. Slezak (1972), A comparison of multivariate normal generators, CACM 15, 1048-1049 Bass, R. (1972), A rank two algorithm for unconstrained minimization, Math. Comp. 26, 129-143 Bauer, F.L. (1965), Elimination with weighted row combinations for solving linear equations and least squares problems, Numer. Math. 7, 338-352 Bauer, W.F. (1958), The Monte Carlo method, SIAM J. 6, 438-451 Beale, E.M.L. (1956), On quadratic programming, Nav. Res. Log. Quart. 6, 227-243 Beale, E.M.L. (1958), On an iterative method for nding a local minimum of a function of more than one variable, Princeton University, Statistical Techniques Research Group, technical report 25, Princeton NJ, Nov. 1958 Beale, E.M.L. (1967), Numerical methods, in: Abadie (1967), pp. 133-205 Beale, E.M.L. (1970), Computational methods for least squares, in: Abadie (1970), pp. 213-228

254

References

Beale, E.M.L. (1972), A derivation of conjugate gradients, in: Lootsma (1972a), pp. 39-43 Beamer, J.H., D.J. Wilde (1969), An upper bound on the number of measurements required by the contour tangent optimization technique, IEEE Trans. SSC-5, 2730 Beamer, J.H., D.J. Wilde (1970), Minimax optimization of unimodal functions by variable block search, Mgmt. Sci. 16, 529-541 Beamer, J.H., D.J. Wilde (1973), A minimax search plan for constrained optimization problems, JOTA 12, 439-446 Beckman, F.S. (1967), Die Losung linearer Gleichungssysteme nach der Methode der konjugierten Gradienten, in: Ralston and Wilf (1967), pp. 106-126 Beckmann, M. (Ed.) (1971), Unternehmensforschung heute, Springer, Berlin Beier, W., K. Gla! (1968), Bionik|eine Wissenschaft der Zukunft, Urania, Leipzig, Germany Bekey, G.A., M.H. Gran, A.E. Sabro, A. Wong (1966), Parameter optimization by random search using hybrid computer techniques, AFIPS Conf. Proc. 29, 191-200 Bekey, G.A., W.J. Karplus (1971), Hybrid-Systeme, Berliner Union und Kohlhammer, Stuttgart Bekey, G.A., R.B. McGhee (1964), Gradient methods for the optimization of dynamic system parameters by hybrid computation, in: Balakrishnan and Neustadt (1964), pp. 305-327 Belew, R.K., L.B. Booker (Eds.) (1991), Proceedings of the 4th International Conference on Genetic Algorithms, University of California, San Diego CA, July 13-16, 1991, Morgan Kaufmann, San Mateo CA Bell, D.E., R.E. Keeney, H. Raia (Eds.) (1977), Conicting objectives in decisions, vol. 1 of Wiley IIASA International Series on Applied Systems Analysis, Wiley, Chichester Bell, M., M.C. Pike (1966), Remark on algorithm 178 (E4)|direct search, CACM 9, 684-685 Bellman, R.E. (1967), Dynamische Programmierung und selbstanpassende Regelprozesse, Oldenbourg, Munich Beltrami, E.J., J.P. Indusi (1972), An adaptive random search algorithm for constrained minimization, IEEE Trans. C-21, 1004-1008

References

255

Bendin, F. (1992), Ein Praktikum zu Verfahren zur Losung zentraler und dezentraler Optimierungsprobleme und Untersuchungen hierarchisch zerlegter Optimierungsaufgaben mit Hilfe von Parallelrechnern, Dr.-Ing. Diss., Technical University of Ilmenau, Germany, Faculty of Technical Sciences, Sept. 1992 Berg, R.L., N.W. Timofejew-Ressowski (1964), Uber Wege der Evolution des Genotyps, in: Ljapunov, Kammerer, and Thiele (1964b), pp. 201-221 Bergmann, H.W. (1989), Optimization|methods and applications, possibilities and limitations, vol. 47 of Lecture Notes in Engineering, Springer, Berlin Berlin, V.G. (1969), Acceleration of stochastic approximations by a mixed search method, ARC 30, 125-129 Berlin, V.G. (1972), Parallel randomized search strategies, ARC 33, 398-403 Berman, G. (1966), Minimization by successive approximation, SIAM J. Numer. Anal. 3, 123-133 Berman, G. (1969), Lattice approximations to the minima of functions of several variables, JACM 16, 286-294 Bernard, J.W., F.J. Sonderquist (1959), Progress report on OPCON|Dow evaluates optimizing control, Contr. Engng. 6(11), 124-128

Bertram, J.E. (1960), Control by stochastic adjustment, AIEE Trans. II Appl. Ind. 78, 485-491 Beveridge, G.S.G., R.S. Schechter (1970), Optimization|theory and practice, McGrawHill, New York Beyer, H.-G. (1989), Ein Evolutionsverfahren zur mathematischen Modellierung stationarer Zustande in dynamischen Systemen, Dr. rer. nat. Diss., University of Architecture and Civil Engineering, Weimar, Germany, June 1989 Beyer, H.-G. (1990), Simulation of steady states in dissipative systems by Darwin's paradigm of evolution, J. of Non-Equilibrium Thermodynamics 15, 45-58 Beyer, H.-G. (1992), Some aspects of the `evolution strategy' for solving TSP-like optimization problems, in: Manner and Manderick (1992), pp. 361-370 Beyer, H.-G. (1993), Toward a theory of evolution strategies|some asymptotical results from the (1 + ) - theory, Evolutionary Computation 1, 165-188 Beyer, H.-G. (1994a), Towards a theory of `evolution strategies'|results for (1 + )strategies on (nearly) arbitrary tness functions, in: Davidor, Schwefel, and Manner (1994), pp. 58-67

256

References

Beyer, H.-G. (1994b), Towards a theory of `evolution strategies'|results from the N dependent ( ) and the multi-recombinant (= ) theory, technical report SYS5/94, Systems Analysis Research Group, University of Dortmund, Department of Computer Science, Oct. 1994 Biggs, M.C. (1971), Minimization algorithms making use of non-quadratic properties of the objective function, JIMA 8, 315-327 (errata in 9 (1972)) Biggs, M.C. (1973), A note on minimization algorithms which make use of non-quadratic properties of the objective function, JIMA 12, 337-338 Birkho, G., S. MacLane (1965), A survey of modern algebra, 3rd ed., Macmillan, New York Blakemore, J.W., S.H. Davis, Jr. (Eds.) (1964), Optimization techniques, AIChE Chemical Engineering Progress Symposium Series 60, no. 50 Bledsoe, W.W. (1961), A basic limitation on the speed of digital computers, IRE Trans. EC-10, 530 Blum, J.R. (1954a), Approximation methods which converge with probability one, Ann. Math. Stat. 25, 382-386 Blum, J.R. (1954b), Multidimensional stochastic approximation methods, Ann. Math. Stat. 25, 737-744 Boas, A.H. (1962), What optimization is all about, Chem. Engng. 69(25), 147-152 Boas, A.H. (1963a), How to use Lagrange multipliers, Chem. Engng. 70(1), 95-98 Boas, A.H. (1963b), How search methods locate optimum in univariable problems, Chem. Engng. 70(3), 105-108 Boas, A.H. (1963c), Optimizing multivariable functions, Chem. Engng. 70(5), 97-104 Boas, A.H. (1963d), Optimization via linear and dynamic programming, Chem. Engng. 70(7), 85-88 Bocharov, I.N., A.A. Feldbaum (1962), An automatic optimizer for the search for the smallest of several minima|a global optimizer, ARC 23, 260-270 Bohling, K.H., P.P. Spies (Eds.) (1979), 9th GI-Jahrestagung, Bonn, Oct. 1979, Springer, Berlin Boltjanski, W.G. (1972), Mathematische Methoden der optimalen Steuerung, Hanser, Munich Booth, A.D. (1949), An application of the method of steepest descents to the solution of systems of non-linear simultaneous equations, Quart. J. Mech. Appl. Math. 2, 460-468

References

257

Booth, A.D. (1955), Numerical methods, Butterworths, London Booth, R.S. (1967), Location of zeros of derivatives, SIAM J. Appl. Math. 15, 1496-1501 Boothroyd, J. (1965), Certication of algorithm 2|Fibonacci search, Comp. Bull. 9, 105, 108 Born, J. (1978), Evolutionsstrategien zur numerischen Losung von Adaptationsaufgaben, Dr. rer. nat. Diss., Humboldt University at Berlin Box, G.E.P. (1957), Evolutionary operation|a method for increasing industrial productivity, Appl. Stat. 6, 81-101 Box, G.E.P., D.W. Behnken (1960), Simplex-sum designs|a class of second order rotatable designs derivable from those of rst order, Ann. Math. Stat. 31, 838-864 Box, G.E.P., N.R. Draper (1969), Evolutionary operation|a statistical method for process improvement, Wiley, New York Box, G.E.P., N.R. Draper (1987), Empirical model-building and response surfaces, Wiley, New York Box, G.E.P., J.S. Hunter (1957), Multi-factor experimental designs for exploring response surfaces, Ann. Math. Stat. 28, 195-241 Box, G.E.P., M.E. Muller (1958), A note on the generation of random normal deviates, Ann. Math. Stat. 29, 610-611 Box, G.E.P., K.B. Wilson (1951), On the experimental attainment of optimum conditions, J. of the Royal Statistical Society B, Methodological 8, 1-45 Box, M.J. (1965), A new method of constrained optimization and a comparison with other methods, Comp. J. 8, 42-52 Box, M.J. (1966), A comparison of several current optimization methods and the use of transformations in constrained problems, Comp. J. 9, 67-77 Box, M.J., D. Davies, W.H. Swann (1969), Nonlinear optimization techniques, ICI Monograph 5, Oliver Boyd, Edinburgh Bracken, J., G.P. McCormick (1970), Ausgewahlte Anwendungen nicht-linearer Programmierung, Berliner Union und Kohlhammer, Stuttgart Brajnes, S.N., V.B. Sve%cinskij (1971), Probleme der Neurokybernetik und Neurobionik, 2nd ed., G. Fischer, Stuttgart Brandl, V. (1969), Ein wirksames Monte-Carlo-Schatzverfahren zur simultanen Behandlung hinreichend eng verwandter Probleme angewandt auf Fragen der Neutronenphysik, Tagungsbericht der Reaktortagung des Deutschen Atomforums, Frankfort/ Main, April 1969, Sektion 1, pp. 6-7

258

References

Branin, F.H., Jr., S.K. Hoo (1972), A method for nding multiple extrema of a function of n variables, in: Lootsma (1972a), pp. 231-237 Brazdil, P.B. (Ed.) (1993), Machine learning|ECML '93, vol. 667 of Lecture Notes in Articial Intelligence, Springer, Berlin Brebbia, C.A., S. Hernandez (Eds.) (1989), Computer aided optimum design of structures|applications, Proceedings of the 1st International Conference, Southampton UK, June 1989, Springer, Berlin Bremermann, H.J. (1962), Optimization through evolution and recombination, in: Yovits, Jacobi, and Goldstein (1962), pp. 93-106 Bremermann, H.J. (1963), Limits of genetic control, IEEE Trans. MIL-7, 200-205 Bremermann, H.J. (1967), Quantitative aspects of goal-seeking self-organizing systems, in: Snell (1967), pp. 59-77 Bremermann, H.J. (1968a), Numerical optimization procedures derived from biological evolution processes, in: Oestreicher and Moore (1968), pp. 597-616 Bremermann, H.J. (1968b), Principles of natural and articial intelligence, AGARD report AD-684-952, Sept. 1968, pp. 6c1-6c2 Bremermann, H.J. (1968c), Pattern recognition, functionals, and entropy, IEEE Trans. BME-15, 201-207 Bremermann, H.J. (1970), A method of unconstrained global optimization, Math. Biosci. 9, 1-15 Bremermann, H.J. (1971), What mathematics can and cannot do for pattern recognition, in: Grusser and Klinke (1971), pp. 31-45 Bremermann, H.J. (1973a), On the dynamics and trajectories of evolution processes, in: Locker (1973), pp. 29-37 Bremermann, H.J. (1973b), Algorithms and complexity of evolution and self-organization, Kybernetik-Kongre! der Deutschen Gesellschaft fur Kybernetik und der Nachrichtentechnischen Gesellschaft im VDE, Nuremberg, Germany, March 1973 Bremermann, H.J., L.S.-B. Lam (1970), Analysis of spectra with non-linear superposition, Math. Biosci. 8, 449-460 Bremermann, H.J., M. Rogson, S. Sala (1965), Search by evolution, in: Maxeld, Callahan, and Fogel (1965), pp. 157-167 Bremermann, H.J., M. Rogson, S. Sala (1966), Global properties of evolution processes, in: Pattee et al. (1966), pp. 3-41

References

259

Brent, R.P. (1971), An algorithm with guaranteed convergence for nding a zero of a function, Comp. J. 14, 422-425 Brent, R.P. (1973), Algorithms for minimization without derivatives, Prentice-Hall, Englewood Clis NJ Bromberg, N.S. (1962), Maximization and minimization of complicated multivariable functions, AIEE Trans. I Comm. Electron. 80, 725-730 Brooks, S.H. (1958), A discussion of random methods for seeking maxima, Oper. Res. 6, 244-251 Brooks, S.H. (1959), A comparison of maximum-seeking methods, Oper. Res. 7, 430-457 Brooks, S.H., M.R. Mickey (1961), Optimum estimation of gradient direction in steepest ascent experiments, Biometrics 17, 48-56 Brown, K.M. (1969), A quadratically convergent Newton-like method based upon Gaussian elimination, SIAM J. Numer. Anal. 6, 560-569 Brown, K.M., J.E. Dennis, Jr. (1968), On Newton-like iteration functions|general convergence theorems and a specic algorithm, Numer. Math. 12, 186-191 Brown, K.M., J.E. Dennis, Jr. (1972), Derivative free analogues of the LevenbergMarquardt and Gauss algorithms for non-linear least squares approximation, Numer. Math. 18, 289-297 Brown, R.R. (1959), A generalized computer procedure for the design of optimum systems, AIEE Trans. I Comm. Electron. 78, 285-293 Broyden, C.G. (1965), A class of methods for solving nonlinear simultaneous equations, Math. Comp. 19, 577-593 Broyden, C.G. (1967), Quasi-Newton methods and their application to function minimisation, Math. Comp. 21, 368-381 Broyden, C.G. (1969), A new method of solving nonlinear simultaneous equations, Comp. J. 12, 94-99 Broyden, C.G. (1970a), The convergence of single-rank quasi-Newton methods, Math. Comp. 24, 365-382 Broyden, C.G. (1970b), The convergence of a class of double-rank minimization algorithms, part 1|general considerations, JIMA 6, 76-90 Broyden, C.G. (1970c), The convergence of a class of double-rank minimization algorithms, part 2|the new algorithm, JIMA 6, 222-231 Broyden, C.G. (1971), The convergence of an algorithm for solving sparse non-linear systems, Math. Comp. 25, 285-294

260

References

Broyden, C.G. (1972), Quasi-Newton methods, in: Murray (1972a), pp. 87-106 Broyden, C.G. (1973), Some condition-number bounds for the Gaussian elimination process, JIMA 12, 273-286 Broyden, C.G., J.E. Dennis, Jr., J.J. More (1973), On the local and superlinear convergence of quasi-Newton methods, JIMA 12, 223-245 Broyden, C.G., M.P. Johnson (1972), A class of rank-l optimization algorithms, in: Lootsma (1972a), pp. 35-38 Brudermann, U. (1977), Entwicklung und Anpassung eines vollstandigen Ansteuersystems fur fremdenergetisch angetriebene Ganzarmprothesen, Fortschrittberichte der VDI-Zeitschriften, vol. 17 (Biotechnik), no. 6, Dec. 1977 Bryson, A.E., Jr., Y.C. Ho (1969), Applied optimal control, Blaisdell, Waltham MA Budne, T.A. (1959), The application of random balance designs, Technometrics 1, 139155 Buehler, R.J., B.V. Shah, O. Kempthorne (1961), Some properties of steepest ascent and related procedures for nding optimum conditions, Iowa State University, Statistical Laboratory, technical report 1, Ames IA, April 1961 Buehler, R.J., B.V. Shah, O. Kempthorne (1964), Methods of parallel tangents, in: Blakemore and Davis (1964), pp. 1-7 Burkard, R.E. (1972), Methoden der Ganzzahligen Optimierung, Springer, Vienna Campbell, D.T. (1960), Blind variation and selective survival as a general strategy in knowledge-processes, in: Yovits and Cameron (1960), pp. 205-231 Campos Pinto, I. (1989), Wissensbasierte Unterstutzung bei der Losung von Optimierungsaufgaben, Dr. rer. nat. Diss., University of Dortmund, Department of Computer Science, June 1989 Campos, I., E. Peters, H.-P. Schwefel (1989), Zwei Beitrage zum wissensbasierten Einsatz von Optimumsuchverfahren, technical report 311 (green series), University of Dortmund, Department of Computer Science Campos, I., H.-P. Schwefel (1989), KBOPT|a knowledge based optimisation system, in: Brebbia and Hernandez (1989), pp. 211-221 Canon, M.D., C.D. Cullum, Jr., E. Polak (1970), Theory of optimal control and mathematical programming, McGraw-Hill, New York Cantrell, J.W. (1969), Relation between the memory gradient method and the FletcherReeves method, JOTA 4, 67-71

References

261

Carroll, C.W. (1961), The created response surface technique for optimizing nonlinear, restrained systems, Oper. Res. 9, 169-185 Casey, J.K., R.C. Rustay (1966), AID|a general purpose computer program for optimization, in: Lavi and Vogl (1966), pp. 81-100 Casti, J., M. Richardson, R. Larson (1973), Dynamic programming and parallel computers, JOTA 12, 423-438 Cauchy, A. (1847), Methode generale pour la resolution des syst&emes d'equations simultanees, Compt. Rend. Acad. Sci. URSS (USSR), New Ser. 25, 536-538 Cea, J. (1971), Optimisation|theorie et algorithmes, Dunod, Paris C% erny, V. (1985), Thermodynamical approach to the traveling salesman problem|an e cient simulation algorithm, JOTA 45, 41-51 Cembrowicz, R.G., G.E. Krauter (1977), Optimization of urban and regional water supply systems, Proceedings of the Conference on Systems Approach for Development, Cairo, Egypt, Nov. 1977 Chang, S.S.L. (1961), Synthesis of optimum control systems, McGraw-Hill, New York Chang, S.S.L. (1968), Stochastic peak tracking and the Kalman lter, IEEE Trans. AC13, 750 Chatterji, B.N., B. Chatterjee (1971), Performance optimization of a self-organizing feedback control system in presence of inherent coupling signals, Automatica 7, 599-605 Chazan, D., W.L. Miranker (1970), A nongradient and parallel algorithm for unconstrained minimization, SIAM J. Contr. 8, 207-217 Checkland, P., I. Kiss (Eds.) (1987), Problems of constancy and change|the complementarity of systems approaches to complexity, papers presented at the 31st Annual Meeting of the International Society for General System Research, Budapest, Hungary, June 1-5, International Society for General System Research Cheng, W.-M. (Ed.) (1988), Proceedings of the International Conference on Systems Science and Engineering (ICSSE '88), Beijing, July 25-28, 1988, International Academic Publishers/Pergamon Press, Oxford UK Chichinadze, V.K. (1960), Logical design problems of self-optimizing and learning-optimizing control systems based on random searching, Proceedings of the 1st IFAC Congress, Moscow, June-July 1960, vol. 2, pp. 653-657 Chichinadze, V.K. (1967), Random search to determine the extremum of the functions of several variables, Engng. Cybern. 5(1), 115-123

262

References

Chichinadze, V.K. (1969), The Psi-transform for solving linear and non-linear programming problems, Automatica 5, 347-356 C%i%zek, F., D. Hodan%ova (1971), Evolution als Selbstregulation, G. Fischer, Jena, Germany Clayton, D.G. (1971), Algorithm AS-46|Gram-Schmidt orthogonalization, Appl. Stat. 20, 335-338 Clegg, J.C. (1970), Variationsrechnung, Teubner, Stuttgart Cochran, W.G., G.M. Cox (1950), Experimental designs, Wiley, New York Cockrell, L.D. (1969), A comparison of several random search techniques for multimodal surfaces, Proceedings of the National Electronics Conference, Chicago IL, Dec. 1969, pp. 18-23 Cockrell, L.D. (1970), On search techniques in adaptive systems, Ph.D. thesis, Purdue University, Lafayette IN, June 1970 Cohen, A.I. (1972), Rate of convergence of several conjugate gradient algorithms, SIAM J. Numer. Anal. 9, 248-259 Cohn, D.L. (1954), Optimal systems I|the vascular system, Bull. Math. Biophys. 16, 59-74 Collatz, L., W. Wetterling (1971), Optimierungsaufgaben, 2nd ed., Springer, Berlin Colville, A.R. (1968), A comparative study on nonlinear programming codes, IBM New York Science Center, report 320-2949, June 1968 Colville, A.R. (1970), A comparative study of nonlinear programming codes, in: Kuhn (1970), pp. 487-501 Conrad, M. (1988), Prolegomena to evolutionary programming, in: Kochen and Hastings (1988), pp. 150-168 Converse, A.O. (1970), Optimization, Holt, Rinehart, Winston, New York Cooper, L. (Ed.) (1962), Applied mathematics in chemical engineering, AIChE Engineering Progress Symposium Series 58, no. 37 Cooper, L., D. Steinberg (1970), Introduction to methods of optimization, W.B. Saunders, Philadelphia Cornick, D.E., A.N. Michel (1972), Numerical optimization of distributed parameter systems by the conjugate gradient method, IEEE Trans. AC-17, 358-362 Courant, R. (1943), Variational methods for the solution of problems of equilibrium and vibrations, Bull. Amer. Math. Soc. 49, 1-23

References

263

Courant, R., D. Hilbert (1968a), Methoden der mathematischen Physik, 3rd ed., vol. 1, Springer, Berlin Courant, R., D. Hilbert (1968b), Methoden der mathematischen Physik, 2nd ed., vol. 2, Springer, Berlin Cowdrey, D.R., C.M. Reeves (1963), An application of the Monte Carlo method to the evaluation of some molecular integrals, Comp. J. 6, 277-286 Cox, D.R. (1958), Planning of experiments, Wiley, New York Cragg, E.E., A.V. Levy (1969), Study on a supermemory gradient method for the minimization of functions, JOTA 4, 191-205 Crippen, G.M., H.A. Scheraga (1971), Minimization of polypeptide energy, X|a global search algorithm, Arch. Biochem. Biophys. 144, 453-461 Crockett, J.B., H. Cherno (1955), Gradient methods of maximization, Pacif. J. Math. 5, 33-50 Crowder, H., P. Wolfe (1972), Linear convergence to the conjugate gradient method, IBM T.J. Watson Research Center, report RC-3330, Yorktown Heights NY, May 1972 Cryer, C.W. (1971), The solution of a quadratic programming problem using systematic overrelaxation, SIAM J. Contr. 9, 385-392 Cullum, J. (1972), An algorithm for minimizing a dierentiable function that uses only function values, in: Balakrishnan (1972), pp. 117-127 Curry, H.B. (1944), The method of steepest descent for non-linear minimization problems, Quart. Appl. Math. 2, 258-261 Curtis, A.R., J.K. Reid (1974), The choice of step lengths when using dierences to approximate Jacobian matrices, JIMA 13, 121-126 Curtiss, J.H. (1956), A theoretical comparison of the e ciencies of two classical methods and a Monte Carlo method for computing one component of the solution of a set of linear algebraic equations, in: Meyer (1956), pp. 191-233 Dambrauskas, A.P. (1970), The simplex optimization method with variable step, Engng. Cybern. 8, 28-36 Dambrauskas, A.P. (1972), Investigation of the e ciency of the simplex method of optimization with variable step in a noise situation, Engng. Cybern. 10, 590-599 Daniel, J.W. (1967a), The conjugate gradient method for linear and nonlinear operator equations, SIAM J. Numer. Anal. 4, 10-26

264

References

Daniel, J.W. (1967b), Convergence of the conjugate gradient method with computationally convenient modications, Numer. Math. 10, 125-131 Daniel, J.W. (1969), On the approximate minimization of functionals, Math. Comp. 23, 573-581 Daniel, J.W. (1970), A correction concerning the convergence rate for the conjugate gradient method, SIAM J. Numer. Anal. 7, 277-280 Daniel, J.W. (1971), The approximate minimization of functionals, Prentice-Hall, Englewood Clis NJ Daniel, J.W. (1973), Global convergence for Newton methods in mathematical programming, JOTA 12, 233-241 Dantzig, G.B. (1966), Lineare Programmierung und Erweiterungen, Springer, Berlin Darwin, C. (1859), Die Entstehung der Arten durch naturliche Zuchtwahl, translation from \The origin of species by means of natural selection", Reclam, Stuttgart, 1974 Darwin, C. (1874), Die Abstammung des Menschen, translation of the 2nd rev. ed. of \The descent of man", Kroner, Stuttgart, 1966 Davidon, W.C. (1959), Variable metric method for minimization, Argonne National Laboratory, report ANL-5990 rev., Lemont IL, Nov. 1959 Davidon, W.C. (1968), Variance algorithm for minimization, Comp. J. 10, 406-410 Davidon, W.C. (1969), Variance algorithm for minimization, in: Fletcher (1969a), pp. 13-20 Davidor, Y. (1990), Genetic algorithms and robotics, a heuristic strategy for optimization, World Scientic, Singapore Davidor, Y., H.-P. Schwefel (1992), An introduction to adaptive optimization algorithms based on principles of natural evolution, in: Sou%cek (1992), pp. 183-202 Davidor, Y., H.-P. Schwefel, R. Manner (Eds.) (1994), Parallel problem solving from nature 3, Proceedings of the 3rd PPSN Conference, Jerusalem, Oct. 9-14, 1994, vol. 866 of Lecture Notes in Computer Science, Springer, Berlin Davies, D. (1968), The use of Davidon's method in nonlinear programming, ICI Management Service report MSDH-68-110, Middlesborough, Yorks, Aug. 1968 Davies, D. (1970), Some practical methods of optimization, in: Abadie (1970), pp. 87118 Davies, D., W.H. Swann (1969), Review of constrained optimization, in: Fletcher (1969a), pp. 187-202

References

265

Davies, M., I.J. Whitting (1972), A modied form of Levenberg's correction, in: Lootsma (1972a), pp. 191-201 Davies, O.L. (Ed.) (1954), The design and analysis of industrial experiments, Oliver Boyd, London Davis, L. (Ed.) (1987), Genetic algorithms and simulated annealing, Pitman, London, 1987 Davis, L. (Ed.) (1991), Handbook of genetic algorithms, Van Nostrand Reinhold, New York Davis, R.H., P.D. Roberts (1968), Method of conjugate gradients applied to self-adaptive digital control systems, IEE Proceedings 115, 562-571 DeGraag, D.P. (1970), Parameter optimization techniques for hybrid computers, Proceedings of the VIth International Analogue Computation Meeting, Munich, Aug.Sept. 1970, pp. 136-139 Dejon, B., P. Henrici (Eds.) (1969), Constructive aspects of the fundamental theorem of algebra, Wiley-Interscience, London De Jong, K. (1975), An analysis of the behavior of a class of genetic adaptive systems, Ph.D. thesis, University of Michigan, Ann Arbor MI De Jong, K. (Ed.) (1993), Evolutionary computation (journal), MIT Press, Cambridge MA De Jong, K., W. Spears (1993), On the state of evolutionary computation, in: Forrest (1993), pp. 618-623 Dekker, L., G. Savastano, G.C. Vansteenkiste (Eds.) (1980), Simulation of systems '79, Proceedings of the 9th IMACS Congress, Sorento, Italy, North-Holland, Amsterdam Dekker, T.J. (1969), Finding a zero by means of successive linear interpolation, in: Dejon and Henrici (1969), pp. 37-48 Demyanov, V.F., A.M. Rubinov (1970), Approximate methods in optimization problems, Elsevier, New York Denn, M.M. (1969), Optimization by variational methods, McGraw-Hill, New York Dennis, J.E., Jr. (1970), On the convergence of Newton-like methods, in: Rabinowitz (1970), pp. 163-181 Dennis, J.E., Jr. (1971), On the convergence of Broyden's method for nonlinear systems of equations, Math. Comp. 25, 559-567 Dennis, J.E., Jr. (1972), On some methods based on Broyden's secant approximation to the Hessian, in: Lootsma (1972a), pp. 19-34

266

References

D'Esopo, D.A. (1956), A convex programming procedure, Nav. Res. Log. Quart. 6, 33-42 DeVogelaere, R. (1968), Remark on algorithm 178 (E4)|direct search, CACM 11, 498 Dickinson, A.W. (1964), Nonlinear optimization|some procedures and examples, Proceedings of the XIXth ACM National Conference, Philadelphia, Aug. 1964, paper E1.2 Dijkhuis, B. (1971), An adaptive algorithm for minimizing a unimodal function of one variable, ZAMM 51(Sonderheft), T45-T46 Dinkelbach, W. (1969), Sensitivitatsanalysen und parametrische Programmierung, Springer, Berlin Dixon, L.C.W. (1972a), Nonlinear optimization, English University Press, London Dixon, L.C.W. (1972b), The choice of step length, a crucial factor in the performance of variable metric algorithms, in: Lootsma (1972a), pp. 149-170 Dixon, L.C.W. (1972c), Variable metric algorithms|necessary and su cient conditions for identical behavior of nonquadratic functions, JOTA 10, 34-40 Dixon, L.C.W. (1973), Conjugate directions without linear searches, JOTA 11, 317-328 Dixon, L.C.W., M.C. Biggs (1972), The advantages of adjoint-control transformations when determining optimal trajectories by Pontryagin's Maximum Principle, Aeronautical J. 76, 169-174 Dobzhansky, T. (1965), Dynamik der menschlichen Evolution|Gene und Umwelt, S. Fischer, Frankfort/Main Dowell, M., P. Jarratt (1972), The Pegasus method for computing the root of an equation, BIT 12, 503-508 Drenick, R.F. (1967), Die Optimierung linearer Regelsysteme, Oldenbourg, Munich Drepper, F.R., R. Heckler, H.-P. Schwefel (1979), Ein integriertes System von Schatzverfahren, Simulations- und Optimierungstechnik zur rechnergestutzten Langfristplanung, in: Bohling and Spies (1979), pp. 115-129 Dueck, G. (1993), New optimization heuristics, the great deluge algorithm and the record-to-record-travel, J. Computational Physics 104, 86-92 Dueck, G., T. Scheuer (1990), Threshold accepting|a general purpose optimization algorithm appearing superior to simulated annealing, J. Computational Physics 90, 161-175 Du n, R.J., E.L. Peterson, C. Zener (1967), Geometric programming|theory and application, Wiley, New York

References

267

Dvoretzky, A. (1956), On stochastic approximation, in: Neyman (1956), pp. 39-56 Ebeling, W. (1992), The optimization of a class of functionals based on developmental strategies, in: Manner and Manderick (1992), pp. 463-468 Edelbaum, T.N. (1962), Theory of maxima and minima, in: Leitmann (1962), pp. 1-32 Edelman, G.B. (1987), Neural Darwinism|the theory of group selection, Basic Books, New York Eigen, M. (1971), Self-organization of matter and the evolution of biological macromolecules, Naturwissenschaften 58, 465-523 Eisenberg, M.A., M.R. McGuire (1972), Further comments on Dijkstra's concurrent programming control problem, CACM 15, 999 Eisenhart, C., M.W. Hastay, W.A. Wallis (Eds.) (1947), Selected techniques of statistical analysis for scientic and industrial research and production and management engineering, McGraw-Hill, New York Elkin, R.M. (1968), Convergence theorems for Gauss-Seidel and other minimization algorithms, University of Maryland, Computer Science Center, technical report 68-59, College Park MD, Jan. 1968 Elliott, D.F., D.D. Sworder (1969a), A variable metric technique for parameter optimization, Automatica 5, 811-816 Elliott, D.F., D.D. Sworder (1969b), Design of suboptimal adaptive regulator systems via stochastic approximation, Proceedings of the National Electronics Conference, Chicago IL, Dec. 1969, pp. 29-33 Elliott, D.F., D.D. Sworder (1970), Applications of a simplied multidimensional stochastic approximation algorithm, IEEE Trans. AC-15, 101-104 Elliott, D.G. (Ed.) (1970), Proceedings of the 11th Symposium on Engineering Aspects of Magnetohydrodynamics, Caltech, March 24-26, 1970, California Institute of Technology, Pasadena CA Emery, F.E., M. O'Hagan (1966), Optimal design of matching networks for microwave transistor ampliers, IEEE Trans. MTT-14, 696-698 Engelhardt, M. (1973), On upper bounds for variances in stochastic approximation, SIAM J. Appl. Math. 24, 145-151 Engeli, M., T. Ginsburg, H. Rutishauser, E. Stiefel (1959), Rened iterative methods for computation of the solution and the eigen-values of self-adjoint boundary value problems, Mitteilungen des Instituts fur Angewandte Mathematik, Technical University (ETH) of Zurich, Switzerland, Birkhauser, Basle, Switzerland

268

References

Erlicki, M.S., J. Appelbaum (1970), Solution of practical optimization problems, IEEE Trans. SSC-6, 49-52 Ermakov, S. (Ed.) (1992), Int'l J. on Stochastic Optimization and Design, Nova Science, New York Ermoliev, Yu. (1970), Random optimization and stochastic programming, in: Moiseev (1970), pp. 104-115 Ermoliev, Yu., R.J.-B. Wets (1988), Numerical techniques for stochastic optimization, Springer, Berlin Faber, M.M. (1970), Stochastisches Programmieren, Physica-Verlag, Wurzburg, Germany Fabian, V. (1967), Stochastic approximation of minima with improved asymptotic speed, Ann. Math. Stat. 38, 191-200 Fabian, V. (1968), On the choice of design in stochastic approximation methods, Ann. Math. Stat. 39, 457-465 Faddejew, D.K., W.N. Faddejewa (1973), Numerische Methoden der linearen Algebra, 3rd ed., Oldenbourg, Munich Falkenhausen, K. von (1980), Optimierung regionaler Entsorgungssysteme mit der Evolutionsstrategie, Proceedings in Operations Research 9, Physica-Verlag, Wurzburg, Germany, pp. 46-51 Favreau, R.F., R. Franks (1958), Random optimization by analogue techniques, Proceedings of the IInd Analogue Computation Meeting, Strasbourg, Sept. 1958, pp. 437-443 Feigenbaum, E.A., J. Feldman (Eds.) (1963), Computers and thought, McGraw-Hill, New York Feistel, R., W. Ebeling (1989), Evolution of complex systems, Kluwer, Dordrecht, The Netherlands Feldbaum, A.A. (1958), Automatic optimalizer, ARC 19, 718-728 Feldbaum, A.A. (1960), Statistical theory of gradient systems of automatic optimization for objects with quadratic characteristics, ARC 21, 111-118 Feldbaum, A.A. (1962), Rechengerate in automatischen Systemen, Oldenbourg, Munich Fend, F.A., C.B. Chandler (1961), Numerical optimization for multi-dimensional problems, General Electric, General Engineering Laboratory, report 61-GL-78, March 1961

References

269

Fiacco, A.V. (1974), Convergence properties of local solutions of sequences of mathematical programming problems in general spaces, JOTA 13, 1-12 Fiacco, A.V., G.P. McCormick (1964), The sequential unconstrained minimization technique for nonlinear programming|a primal-dual method, Mgmt. Sci. 10, 360-366 Fiacco, A.V., G.P. McCormick (1968), Nonlinear programming|sequential unconstrained minimization techniques, Wiley, New York Fiacco, A.V., G.P. McCormick (1990), Nonlinear programming|sequential unconstrained minimization techniques, vol. 63 of CBMS-NSF Regional Conference Series on Applied Mathematics and vol. 4 of Classics in Applied Mathematics, SIAM, Philadelphia Fielding, K. (1970), Algorithm 387 (E4)|function minimization and linear search, CACM 13, 509-510 Fisher, R.A. (1966), The design of experiments, 8th ed., Oliver Boyd, Edinburgh Fletcher, R. (1965), Function minimization without evaluating derivatives|a review, Comp. J. 8, 33-41 Fletcher, R. (1966), Certication of algorithm 251 (E4)|function minimization, CACM 9, 686-687 Fletcher, R. (1968), Generalized inverse methods for the best least squares solution of systems of non-linear equations, Comp. J. 10, 392-399 Fletcher, R. (Ed.) (1969a), Optimization, Academic Press, London Fletcher, R. (1969b), A review of methods for unconstrained optimization, in: Fletcher (1969a), pp. 1-12 Fletcher, R. (1970a), A class of methods for nonlinear programming with termination and convergence properties, in: Abadie (1970), pp. 157-176 Fletcher, R. (1970b), A new approach to variable metric algorithms, Comp. J. 13, 317-322 Fletcher, R. (1971), A modied Marquardt subroutine for non-linear least squares, UKAEA Research Group, report AERE-R-6799, Harwell, Oxon Fletcher, R. (1972a), Conjugate direction methods, in: Murray (1972a), pp. 73-86 Fletcher, R. (1972b), A survey of algorithms for unconstrained optimization, in: Murray (1972a), pp. 123-129 Fletcher, R. (1972c), A Fortran subroutine for minimization by the method of conjugate gradients, UKAEA Research Group, report AERE-R-7073, Harwell, Oxon

270

References

Fletcher, R. (1972d), Fortran subroutines for minimization by quasi-Newton methods, UKAEA Research Group, report AERE-R-7125, Harwell, Oxon Fletcher, R., M.J.D. Powell (1963), A rapidly convergent descent method for minimization, Comp. J. 6, 163-168 Fletcher, R., C.M. Reeves (1964), Function minimization by conjugate gradients, Comp. J. 7, 149-154 Flood, M.M., A. Leon (1964), A generalized direct search code for optimization, University of Michigan, Mental Health Research Institute, preprint 129, Ann Arbor MI, June 1964 Flood, M.M., A. Leon (1966), A universal adaptive code for optimization|GROPE, in: Lavi and Vogl (1966), pp. 101-130 Floudas, C.A., P.M. Pardalos (1990), A collection of test problems for constrained global optimization algorithms, vol. 455 of Lecture Notes in Computer Science, Springer, Berlin Fogarty, L.E., R.M. Howe (1968), Trajectory optimization by a direct descent process, Simulation 11, 127-135 Fogarty, L.E., R.M. Howe (1970), Hybrid computer solution of some optimization problems, Proceedings of the VIth International Analogue Computation Meeting, Munich, Aug.-Sept. 1970, pp. 127-135 Fogel, D.B. (1991), System identication through simulated evolution, Ginn Press, Needham Heights MA Fogel, D.B. (1992), Evolving articial intelligence, Ph.D. thesis, University of California at San Diego Fogel, D.B., J.W. Atmar (Eds.) (1992), Proceedings of the 1st Annual Conference on Evolutionary Programming, San Diego, Feb. 21-22, 1992, Evolutionary Programming Society, La Jolla CA Fogel, D.B., J.W. Atmar (Eds.) (1993), Proceedings of the 2nd Annual Conference on Evolutionary Programming, San Diego, Feb. 25-26, 1993, Evolutionary Programming Society, La Jolla CA Fogel, L.J. (1962), Autonomous automata, Ind. Research 4, 14-19 Fogel, L.J., A.J. Owens, M.J. Walsh (1965), Articial intelligence through a simulation of evolution, in: Maxeld, Callahan, and Fogel (1965), pp. 131-155 Fogel, L.J., A.J. Owens, M.J. Walsh (1966a), Adaption of evolutionary programming to the prediction of solar ares, General Dynamics-Convair, report NASA-CR-417, San Diego CA

References

271

Fogel, L.J., A.J. Owens, M.J. Walsh (1966b), Articial intelligence through simulated evolution, Wiley, New York Forrest, S. (Ed.) (1993), Proceedings of the 5th International Conference on Genetic Algorithms, University of Illinois, Urbana-Champaign IL, July 17-21, 1993, Morgan Kaufmann, San Mateo CA Forsythe, G.E. (1968), On the asymptotic directions of the s-dimensional optimum gradient method, Numer. Math. 11, 57-76 Forsythe, G.E. (1969), Remarks on the paper by Dekker, in: Dejon and Henrici (1969), pp. 49-51 Forsythe, G.E., T.S. Motzkin (1951), Acceleration of the optimum gradient method, Bull. Amer. Math. Soc. 57, 304-305 Fox, R.L. (1971), Optimization methods for engineering design, Addison-Wesley, Reading MA Frankhauser, P., H.-P. Schwefel (1992), Making use of the Weidlich-Haag model in the case of reduced data sets, in: Gritzmann et al. (1992), pp. 320-323 Frankovi%c, B., S%. Petra%s, J. Skakala, B. Vykouk (1970), Automatisierung und selbsttatige Steuerung, Verlag Technik, Berlin Fraser, A.S. (1957), Simulation of genetic systems by automatic digital computers, Australian J. Biol. Sci. 10, 484-499 Friedberg, R.M. (1958), A learning machine I, IBM J. Res. Dev. 2, 2-13 Friedberg, R.M., B. Dunham, J.H. North (1959), A learning machine II, IBM J. Res. Dev. 3, 282-287 Friedmann, M., L.J. Savage (1947), Planning experiments seeking maxima, in: Eisenhart, Hastay, and Wallis (1947), pp. 365-372 Friedrichs, K.O., O.E. Neugebauer, J.J. Stoker (Eds.) (1948), Studies and essays, Courant anniversary volume, Interscience, New York Fu, K.S., L.D. Cockrell (1970), On search techniques for multimodal surfaces, IFAC Kyoto Symposium on Systems Engineering Approach to Computer Control, Kyoto, Japan, Aug. 1970, paper 17.3 Fu, K.S., Z% .J. Nikolic (1966), On some reinforcement techniques and their relation to the stochastic approximation, IEEE Trans. AC-11, 756-758 Furst, H., P.H. Muller, V. Nollau (1968), Eine stochastische Methode zur Ermittlung der Maximalstelle einer Funktion von mehreren Veranderlichen mit experimentell ermittelbaren Funktionswerten und ihre Anwendung bei chemischen Prozessen, ChemieTechnik 20, 400-405

272

References

Gaidukov, A.I. (1966), Primeneniye sluchainovo poiska pri optimalnom projektirovanii, Prikladnye zadichi tekhnicheskoi kibernetiki (1966), 420-436 Gal, S. (1971), Sequential minimax search for a maximum when prior information is available, SIAM J. Appl. Math. 21, 590-595 Gal, S. (1972), Multidimensional minimax search for a maximum, SIAM J. Appl. Math. 23, 513-526 Galar, R. (1994), Evolutionary simulations and insights into progress, in: Sebald and Fogel (1994), pp. 344-352 Galar, H., H. Kwasnicka, W. Kwasnicki (1980), Simulation of some processes of development, in: Dekker, Savastano, and Vansteenkiste (1980), pp. 133-142 Garnkel, R.S., G.L. Nemhauser (1972), Integer programming, Wiley, New York Garnkel, R.S., G.L. Nemhauser (1973), A survey of integer programming emphasizing computation and relations among models, in: Hu and Robinson (1973), pp. 77-155 Gauss, C.F. (1809), Determinatio orbitae observationibus quotcumque quam proxime satisfacientis, Werke, Band 7 (Theoria motus corporum coelestium in sectionibus conicis solem ambientium), Liber secundus, Sectio III, pp. 236-257, Hamburgi sumtibus Frid. Perthes et I.H. Besser, 1809 reprint: Teubner, Leipzig, Germany, 1906 Gaviano, M., E. Fagiuoli (1972), Remarks on the comparison between random search methods and the gradient method, in: Szego (1972), pp. 337-349 Gelfand, I.M. M.L. Tsetlin (1961), The principle of nonlocal search in automatic optimization systems, Soviet Physics Doklady 6(3), 192-194 Georion, A.M. (Ed.) (1972), Perspectives on optimization, Addison-Wesley, Reading MA Gerardin, L. (1968), Natur als Vorbild|die Entdeckung der Bionik, Kindler, Munich Gersht, A.M., A.I. Kaplinskii (1971), Convergence of the continuous variant of the Robbins-Monro procedure, ARC 32, 71-75 Gessner, P., K. Spremann (1972), Optimierung in Funktionenraumen, Springer, Berlin Gessner, P., H. Wacker (1972), Dynamische Optimierung|Einfuhrung, Modelle, Computerprogramme, Hanser, Munich Gilbert, E.G. (1967), A selected bibliography on parameter optimization methods suitable for hybrid computation, Simulation 8, 350-352 Gilbert, P., W.J. Chandler (1972), Interference between communicating parallel processes, CACM 15, 427-437

References

273

Gill, P.E., W. Murray (1972), Quasi-Newton methods for unconstrained optimization, JIMA 9, 91-108 Ginsburg, T. (1963), The conjugate gradient method, Numer. Math. 5, 191-200 Girsanov, I.V. (1972), Lectures on mathematical theory of extremum problems, Springer, Berlin Glass, H., L. Cooper (1965), Sequential search|a method for solving constrained optimization problems, JACM 12, 71-82 Glover, F. (1986), Future paths for integer programming and links to articial intelligence, Comp. Oper. Res. 13, 533-549 Glover, F. (1989), Tabu search|part I, ORSA-J. on Computing 1, 190-206 Glover, F., H.-J. Greenberg (1989), New approaches for heuristic search|a bilateral linkage with articial intelligence, Europ. J. Oper. Res. 39, 119-130 Gnedenko, B.W. (1970), Lehrbuch der Wahrscheinlichkeitsrechnung, 6th ed., AkademieVerlag, Berlin Goldberg, D.E. (1989), Genetic algorithms in search, optimization, and machine learning, Addison-Wesley, Reading MA Goldfarb, D. (1969), Su cient conditions for the convergence of a variable metric algorithm, in: Fletcher (1969a), pp. 273-282 Goldfarb, D. (1970), A family of variable-metric methods derived by variational means, Math. Comp. 24, 23-26 Goldfeld, S.M., R.E. Quandt, H.F. Trotter (1966), Maximization by quadratic hillclimbing, Econometrica 34, 541-551 Goldfeld, S.M., R.E. Quandt, H.F. Trotter (1968), Maximization by improved quadratic hill-climbing and other methods, Princeton University, Econometric Research Program, research memo. RM-95, Princeton NJ, April 1968 Goldstein, A.A. (1962), Cauchy's method of minimization, Numer. Math. 4, 146-150 Goldstein, A.A. (1965), On Newton's method, Numer. Math. 7, 391-393 Goldstein, A.A., J.F. Price (1967), An eective algorithm for minimization, Numer. Math. 10, 184-189 Goldstein, A.A., J.F. Price (1971), On descent from local minima, Math. Comp. 25, 569-574 Golinski, J., Z.K. Lesniak (1966), Optimales Entwerfen von Konstruktionen mit Hilfe der Monte-Carlo-Methode, Bautechnik 43, 307-311

274

References

Goll, R. (1972), Der Evolutionismus|Analyse eines Grundbegris neuzeitlichen Denkens, Beck, Munich Golub, G.H. (1965), Numerical methods for solving linear least squares problems, Numer. Math. 7, 206-216 Golub, G.H., M.A. Saunders (1970), Linear least squares and quadratic programming, in: Abadie (1970), pp. 229-256 Gonzalez, R.S. (1970), An optimization study on a hybrid computer, Ann. Assoc. Int'l Calcul Analog. 12, 138-148 Gorges-Schleuter, M. (1991a), Explicit parallelism of genetic algorithms through population structures, in: Schwefel and Manner (1991), pp. 150-159 Gorges-Schleuter, M. (1991b), Genetic algorithms and population structures|a massively parallel algorithm, Dr. rer. nat. Diss., University of Dortmund, Department of Computer Science, Jan. 1991 Gorvits, G.G., O.I. Larichev (1971), Comparison of search methods for the solution of nonlinear identication problems, ARC 32, 272-280 Gottfried, B.S., J. Weisman (1973), Introduction to optimization theory, Prentice-Hall, Englewood Clis NJ Gould, S.J., N. Eldredge (1977), Punctuated equilibria|the tempo and mode of evolution reconsidered, Paleobiology 3, 115-151 Gould, S.J., N. Eldredge (1993), Punctuated equilibrium comes of age, Nature 366, 223-227 Gran, R. (1973), On the convergence of random search algorithms in continuous time with applications to adaptive control, IEEE Trans. SMC-3, 62-66 Grasse, P.P. (1973), Allgemeine Biologie, vol. 5|Evolution, G. Fischer, Stuttgart Grassmann, P. (1967), Verfahrenstechnik und Biologie, Chemie Ingenieur Technik 39, 1217-1226 Grassmann, P. (1968), Verfahrenstechnik und Medizin, Chemie Ingenieur Technik 40, 1094-1100 Grauer, M., A. Lewandowski, A.P. Wierzbicki (Eds.) (1982), Multiobjective and stochastic optimization, Proceedings of the IIASA Task Force Meeting, Nov. 30 - Dec. 4, 1981, IIASA Proceedings Series CP-82-S12, Laxenburg, Austria Grauer, M., D.B. Pressmar (Eds.) (1991), Parallel computing and mathematical optimization, vol. 367 of Lecture Notes in Economics and Mathematical Systems, Springer, Berlin

References

275

Graves, R.L., P. Wolfe (Eds.) (1963), Recent advances in mathematical programming, McGraw-Hill, New York Greenberg, H. (1971), Integer programming, Academic Press, New York Greenstadt, J. (1967a), On the relative e ciencies of gradient methods, Math. Comp. 21, 360-367 Greenstadt, J. (1967b), Bestimmung der Eigenwerte einer Matrix nach der JacobiMethode, in: Ralston and Wilf (1967), pp. 152-168 Greenstadt, J. (1970), Variations on variable-metric methods, Math. Comp. 24, 1-22 Greenstadt, J. (1972), A quasi-Newton method with no derivatives, Math. Comp. 26, 145-166 Grefenstette, J.J. (Ed.) (1985), Proceedings of the 1st International Conference on Genetic Algorithms, Carnegie-Mellon University, Pittsburgh PA, July 24-26, 1985, Lawrence Erlbaum, Hillsdale NJ Grefenstette, J.J. (Ed.) (1987), Proceedings of the 2nd International Conference on Genetic Algorithms, MIT, Cambridge MA, July 28-31, 1987, Lawrence Erlbaum, Hillsdale NJ Gritzmann, P., R. Hettich, R. Horst, E. Sachs (Eds.) (1992), Operations Research '91, Extended Abstracts of the 16th Symposium on Operations Research, Trier, Sept. 9-11, 1991, Physica-Verlag, Heidelberg Grusser, O.J., R. Klinke (Eds.) (1971), Zeichenerkennung durch biologische und technische Systeme, Springer, Berlin Guilfoyle, G., I. Johnson, P. Wheatley (1967), One-dimensional search combining golden section and cubic t techniques, Analytical Mechanics Associates Inc., quarterly report 67-1, Westbury, Long Island NY, Jan. 1967 Guin, J.A. (1968), Modication of the complex method of constrained optimization, Comp. J. 10, 416-417 Gurin, L.S. (1966), Random search in the presence of noise, Engng. Cybern. 4(3), 252-260 Gurin, L.S., V.P. Lobac (1963), Combination of the Monte Carlo method with the method of steepest descents for the solution of certain extremal problems, AIAA J. 1, 2708-2710 Gurin, L.S., L.A. Rastrigin (1965), Convergence of the random search method in the presence of noise, ARC 26, 1505-1511

276

References

Hadamard, J. (1908), Memoire sur le probl&eme d'analyse relatif a& l'equilibre des plaques elastiques encastrees, Memoires presentes par divers savants &a l' Academie des sciences de l'Institut national de France, 2nd Ser., vol. 33 (savants etrangers), no. 4, pp. 1-128 Hadley, G. (1962), Linear programming, Addison-Wesley, Reading MA Hadley, G. (1969), Nichtlineare und dynamische Programmierung, Physica-Verlag, Wurzburg, Germany Haefner, K. (Ed.) (1992), Evolution of information processing systems|an interdisciplinary approach for a new understanding of nature and society, Springer, Berlin Hague, D.S., C.R. Glatt (1968), An introduction to multivariable search techniques for parameter optimization and program AESOP, Boeing Space Division, report NASACR-73200, Seattle WA, March 1968 Hamilton, P.A., J. Boothroyd (1969), Remark on algorithm 251 (E4)|function minimization, CACM 12, 512-513 Hammel, U. (1991), Cartoon|combining modular simulation, regression, and optimization in an object-oriented environment, in: Kohler (1991), pp. 854-855 Hammel, U., T. Back (1994), Evolution strategies on noisy functions|how to improve convergence properties, in: Davidor, Schwefel, and Manner (1994), pp. 159-168 Hammer, P.L. (Ed.) (1984), Stochastics and optimization, Annals of Operations Research, vol. 1, Baltzer, Basle, Switzerland Hammersley, J.M., D.C. Handscomb (1964), Monte Carlo methods, Methuen, London Hancock, H. (1960), Theory of maxima and minima, Dover, New York Hansen, P.B. (1972), Structured multiprogramming, CACM 15, 574-578 Harkins, A. (1964), The use of parallel tangents in optimization, in: Blakemore and Davis (1964), pp. 35-40 Hartmann, D. (1974), Optimierung balkenartiger Zylinderschalen aus Stahlbeton mit elastischem und plastischem Werkstoverhalten, Dr.-Ing. Diss., University of Dortmund, July 1974 Haubrich, J.G.A. (1963), Algorithm 205 (E4)|ative, CACM 6, 519 Heckler, R. (1979), OASIS|optimization and simulation integrating system|status report, technical report KFA-STE-IB-2/79, Nuclear Research Center (KFA) Julich, Germany, Dec. 1979

References

277

Heckler, R., H.-P. Schwefel (1978), Superimposing direct search methods for parameter optimization onto dynamic simulation models, in: Highland, Nielsen, and Hull (1978), pp. 173-181 Heinhold, J., K.W. Gaede (1972), Ingenieur-Statistik, 3rd ed., Oldenbourg, Munich Henn, R., H.P. Kunzi (1968), Einfuhrung in die Unternehmensforschung I und II, Springer, Berlin Herdy, M. (1992), Reproductive isolation as strategy parameter in hierarchical organized evolution strategies, in: Manner and Manderick (1992), pp. 207-217 Herschel, R. (1961), Automatische Optimisatoren, Elektronische Rechenanlagen 3, 30-36 Hertel, H. (1963), Biologie und Technik, Band 1: Struktur - Form - Bewegung, Krausskopf Verlag, Mainz Hesse, R. (1973), A heuristic search procedure for estimating a global solution of nonconvex programming problems, Oper. Res. 21, 1267-1280 Hestenes, M.R. (1956), The conjugate-gradient method for solving linear systems, Proc. Symp. Appl. Math. 6, 83-102 Hestenes, M.R. (1966), Calculus of variations and optimal control theory, Wiley, New York Hestenes, M.R. (1969), Multiplier and gradient methods, in: Zadeh, Neustadt, and Balakrishnan (1969a), pp. 143-163 Hestenes, M.R. (1973), Iterative methods for solving linear equations, JOTA 11, 323-334 (reprint of the original from 1951) Hestenes, M.R., M.L. Stein (1973), The solution of linear equations by minimization, JOTA 11, 335-359 (reprint of the original from 1951) Hestenes, M.R., E. Stiefel (1952), Methods of conjugate gradients for solving linear systems, NBS J. Research 49, 409-436 Heusener, G. (1970), Optimierung natriumgekuhlter schneller Brutreaktoren mit Methoden der nichtlinearen Programmierung, report KFK-1238, Nuclear Research Center (KfK) Karlsruhe, Germany, July 1970 Heydt, G.T. (1970), Directed random search, Ph.D. thesis, Purdue University, Lafayette IN, Aug. 1970 Heynert, H. (1972), Einfuhrung in die allgemeine Bionik, Deutscher Verlag der Wissenschaften, Berlin Highland, H.J., N.R. Nielsen, L.G. Hull (Eds.) (1978), Proceedings of the Winter Simulation Conference, Miami Beach FL, Dec. 4-6, 1978

278

References

Hildebrand, F.B. (1956), Introduction to numerical analysis, McGraw-Hill, New York Hill, J.C. (1964), A hill-climbing technique using piecewise cubic approximation, Ph.D. thesis, Purdue University, Lafayette IN, June 1964 Hill, J.C., J.E. Gibson (1965), Hill-climbing on hills with many minima, Proceedings of the IInd IFAC Symposium on the Theory of Self Adaptive Control Systems, Teddington UK, Sept. 1965, pp. 322-334 Hill, J.D. (1969), A search technique for multimodal surfaces, IEEE Trans. SSC-5, 2-8 Hill, J.D., K.S. Fu (1965), A learning control system using stochastic approximation for hill-climbing, VIth Joint Automatic Control Conference, Troy NY, June 1965, session 14, paper 2 Hill, J.D., G.J. McMurtry, K.S. Fu (1964), A computer-simulated on-line experiment in learning control systems, AFIPS Conf. Proc. 25, 315-325 Himmelblau, D.M. (1972a), A uniform evaluation of unconstrained optimization techniques, in: Lootsma (1972b), pp. 69-97 Himmelblau, D.M. (1972b), Applied nonlinear programming, McGraw-Hill, New York Himsworth, F.R. (1962), Empirical methods of optimisation, Trans. Inst. Chem. Engrs. 40, 345-349 Hock, W., K. Schittkowski (1981), Test examples for nonlinear programming codes, vol. 187 of Lecture Notes in Economics and Mathematical Systems, Springer, Berlin Hofestadt, R., F. Kruckeberg, T. Lengauer (Eds.) (1993), Informatik in der Biowissenschaft, Springer, Berlin Homann, U., H. Hofmann (1970), Einfuhrung in die Optimierung mit Anwendungsbeispielen aus dem Chemie-Ingenieur-Wesen, Verlag Chemie, Weinheim Homeister, F. (1991), Scalable parallelism by evolutionary algorithms, in: Grauer and Pressmar (1991), pp. 177-198 Homeister, F., T. Back (1990), Genetic algorithms and evolution strategies|similarities and dierences, technical report 365 (green series), University of Dortmund, Department of Computer Science, Nov. 1990 Homeister, F., T. Back (1991), Genetic algorithms and evolution strategies|similarities and dierences, in: Schwefel and Manner (1991), pp. 445-469 Homeister, F., T. Back (1992), Genetic algorithms and evolution strategies|similarities and dierences, technical report SYS-1/92, Systems Analysis Research Group, University of Dortmund, Department of Computer Science, Feb. 1992

References

279

Homeister, F., H.-P. Schwefel (1990), A taxonomy of parallel evolutionary algorithms, in: Wolf, Legendi, and Schendel (1990), pp. 97-107 Hoer, A. (1976), Formoptimierung von Leichtbaufachwerken durch Einsatz einer Evolutionsstrategie, Dr.-Ing. Diss., Technical University of Berlin, Department of Transportation Technologies, June 1976 Hoer, A., U. Ley!ner, J. Wiedemann (1973), Optimization of the layout of trusses combining strategies based on Michell's theorem and on the biological principles of evolution, IInd Symposium on Structural Optimization, Milan, April 1973, AGARD Conf. Proc. 123, appendix A Holland, J.H. (1975), Adaptation in natural and articial systems, University of Michigan Press, Ann Arbor MI Holland, J.H. (1992), Adaptation in natural and articial systems, 2nd ed., MIT Press, Cambridge MA Holland, J.H., K.J. Holyoak, R.E. Nisbett, P.R. Thagard (1986), Induction|process of interference, learning, and discovery, MIT Press, Cambridge MA Hollstien, R.B. (1971), Articial genetic adaptation in computer control systems, Ph.D. thesis, University of Michigan, Ann Arbor MI Hooke, R. (1957), Control by automatic experimentation, Chem. Engng. 64(6), 284-286 Hooke, R., T.A. Jeeves (1958), Comments on Brooks' discussion of random methods, Oper. Res. 6, 881-882 Hooke, R., T.A. Jeeves (1961), Direct search solution of numerical and statistical problems, JACM 8, 212-229 Hooke, R., R.I. VanNice (1959), Optimizing control by automatic experimentation, ISA J. 6(7), 74-79 Hopper, M.J. (Ed.) (1971), Harwell subroutine library|a catalogue of subroutines, UKAEA Research Group, report AERE-R-6912, Harwell, Oxon Horst, R. (Ed.) (1991), J. of Global Optimization, Kluwer, Dordrecht, The Netherlands Hoshino, S. (1971), On Davies, Swann, and Campey minimisation process, Comp. J. 14, 426-427 Hoshino, S. (1972), A formulation of variable metric methods, JIMA 10, 394-403 Hotelling, H. (1941), Experimental determination of the maximum of a function, Ann. Math. Stat. 12, 20-45 House, F.R. (1971), Remark on algorithm 251 (E4)|function minimisation, CACM 14, 358

280

References

Householder, A.S. (1953), Principles of numerical analysis, McGraw-Hill, New York Householder, A.S. (1970), The numerical treatment of a single nonlinear equation, McGraw-Hill, New York Houston, B.F., R.A. Human (1971), A technique which combines modied pattern search methods with composite designs and polynomial constraints to solve constrained optimization problems, Nav. Res. Log. Quart. 18, 91-98 Hu, T.C. (1972), Ganzzahlige Programmierung und Netzwerkusse, Oldenbourg, Munich Hu, T.C., S.M. Robinson (Eds.) (1973), Mathematical programming, Academic Press, New York Huang, H.Y. (1970), Unied approach to quadratically convergent algorithms for function minimization, JOTA 5, 405-423 Huang, H.Y. (1974), Method of dual matrices for function minimization, JOTA 13, 519-537 Huang, H.Y., J.P. Chambliss (1973), Quadratically convergent algorithms and onedimensional search schemes, JOTA 11, 175-188 Huang, H.Y., J.P. Chambliss (1974), Numerical experiments on dual matrix algorithms for function minimization, JOTA 13, 620-634 Huang, H.Y., A.V. Levy (1970), Numerical experiments on quadratically convergent algorithms for function minimization, JOTA 6, 269-282 Huberman, B.A. (Ed.) (1988), The ecology of computation, North Holland, Amsterdam Huelsman, L.P. (1968), GOSPEL|a general optimization software package for electrical network design, University of Arizona, Department of Electrical Engineering, report, Tucson AZ, Sept. 1968 Hull, T.E. (1967), Random-number generation and Monte-Carlo methods, in: Klerer and Korn (1967), pp. 63-78 Humphrey, W.E., B.J. Cottrell (1962/66), A general minimizing routine, University of California, Lawrence Radiation Laboratory, internal memo. P-6, Livermore CA, July 1962, rev. March 1966 Hupfer, P. (1970), Optimierung von Baukonstruktionen, Teubner, Stuttgart Hwang, C.L., A.S.M. Masud (1979), Multiple objective decision making|methods and applications, vol. 164 of Lecture Notes in Economics and Mathematical Systems, Springer, Berlin Hyslop, J. (1972), A note on the accuracy of optimisation techniques, Comp. J. 15, 140

References

281

Idelsohn, J.M. (1964), Ten ways to nd the optimum, Contr. Engng. 11(6), 97-102 Imamura, H., K. Uosaki, M. Tasaka, T. Suzuki (1970), Optimization methods in the multimodal case and their application to automatic lens design, IFAC Kyoto Symposium on Systems Engineering Approach to Computer Control, Kyoto, Japan, Aug. 1970, paper 7.4 Ivakhnenko, A.G. (1970), Heuristic self-organization in problems of engineering cybernetics, Automatica 6, 207-219 Jacobson, D.H., D.Q. Mayne (1970), Dierential dynamic programming, Elsevier, New York Jacoby, S.L.S., J.S. Kowalik, J.T. Pizzo (1972), Iterative methods for nonlinear optimization problems, Prentice-Hall, Englewood Clis NJ Jahnke-Emde-Losch (1966), Tafeln hoherer Funktionen, 7th ed., Teubner, Stuttgart Jana%c, K. (1971), Adaptive stochastic approximations, Simulation 16, 51-58 Jarratt, P. (1967), An iterative method for locating turning points, Comp. J. 10, 82-84 Jarratt, P. (1968), A numerical method for determining points of inection, BIT 8, 31-35 Jarratt, P. (1970), A review of methods for solving nonlinear algebraic equations in one variable, in: Rabinowitz (1970), pp. 1-26 Jarvis, R.A. (1968), Hybrid computer simulation of adaptive strategies, Ph.D. thesis, University of Western Australia, Nedlands WA, March 1968 Jarvis, R.A. (1970), Adaptive global search in a time-variant environment using a probabilistic automaton with pattern recognition supervision, IEEE Trans. SSC-6, 209-217 Jeeves, T.A. (1958), Secant modication of Newton's method, CACM 1, 9-10 Johnk, M.D. (1969), Erzeugen und Testen von Zufallszahlen, Physica-Verlag, Wurzburg, Germany Johannsen, G. (1970), Entwicklung und Optimierung eines vielparametrigen nichtlinearen Modells fur den Menschen als Regler in der Fahrzeugfuhrung, Dr.-Ing. Diss., Technical University of Berlin, Department of Transportation Technologies, Oct. 1970 Johannsen, G. (1973), Optimierung vielparametriger Bezugsmodelle mit Hilfe von Zufallssuchverfahren, Regelungstechnische Proze!-Datenverarbeitung 21, 234-239 John, F. (1948), Extremum problems with inequalities as subsidiary conditions, in: Friedrichs, Neugebauer, and Stoker (1948), pp. 187-204

282

References

John, P.W.M. (1971), Statistical design and analysis of experiments, Macmillan, New York Johnson, S.M. (1956), Best exploration for maximum is Fibonaccian, RAND Corporation, report P-856, Santa Monica CA Jones, A. (1970), Spiral|a new algorithm for non-linear parameter estimation using least squares, Comp. J. 13, 301-308 Jones, D.S. (1973), The variable metric algorithm for non-denite quadratic functions, JIMA 12, 63-71 Joosen, W., E. Milgrom (Eds.) (1992), Parallel computing|from theory to sound practice, Proceedings of the European Workshop on Parallel Computing (EWPC '92), Barcelona, Spain, March 1992, IOS Press, Amsterdam Jordan, P. (1970), Schopfung und Geheimnis, Stalling, Oldenburg, Germany Kamiya, A., T. Togawa (1972), Optimal branching structure of the vascular tree, Bull. Math. Biophys. 34, 431-438 Kammerer, W.J., M.Z. Nashed (1972), On the convergence of the conjugate gradient method for singular linear operator equations, SIAM J. Numer. Anal. 9, 165-181 Kantorovich, L.V. (1940), A new method of solving of some classes of extremal problems, Compt. Rend. Acad. Sci. URSS (USSR), New Ser. 28, 211-214 Kantorovich, L.V. (1945), On an eective method of solving extremal problems for quadratic functionals, Compt. Rend. Acad. Sci. URSS (USSR), New Ser. 48, 455-460 Kantorovich, L.V. (1952), Functional analysis and applied mathematics, NBS report 1509, March 1952 Kaplinskii, A.I., A.I. Propoi (1970), Stochastic approach to non-linear programming problems, ARC 31, 448-459 Kappler, H. (1967), Gradientenverfahren der nichtlinearen Programmierung, O. Schwartz, Gottingen, Germany Karmarkar, N. (1984), A new polynomial-time algorithm for linear programming, Combinatorica 4, 373-395 Karnopp, D.C. (1961), Search theory applied to parameter scan optimization problems, Ph.D. thesis, MIT, Cambridge MA, June 1961 Karnopp, D.C. (1963), Random search techniques for optimization problems, Automatica 1, 111-121

References

283

Karnopp, D.C. (1966), Ein direktes Rechenverfahren fur implizite Variationsprobleme bei optimalen Prozessen, Regelungstechnik 14, 366-368 Karp, R.M., W.L. Miranker (1968), Parallel minimax search for a maximum, J. Comb. Theory 4, 19-35 Karreman, H.F. (Ed.) (1968), Stochastic optimization and control, Wiley, New York Karumidze, G.V. (1969), A method of random search for the solution of global extremum problems, Engng. Cybern. 7(6), 27-31 Katkovnik, V.Ya., O.Yu. Kulchitskii (1972), Convergence of a class of random search algorithms, ARC 33, 1321-1326 Katkovnik, V.Ya., L.I. Shimelevich (1972), A class of heuristic methods for solution of partially-integer programming problems, Engng. Cybern. 10, 390-394 Kaupe, A.F. (1963), Algorithm 178 (E4)|direct search, CACM 6, 313-314 Kaupe, A.F. (1964), On optimal search techniques, CACM 7, 38

Kavanaugh, W.P., E.C. Stewart, D.H. Brocker (1968), Optimal control of satellite attitude acquisition by a random search algorithm on a hybrid computer, AFIPS Conf. Proc. 32, 443-452 Kawamura, K., R.A. Volz (1973), On the rate of convergence of the conjugate gradient reset method with inaccurate linear minimizations, IEEE Trans. AC-18, 360-366 Kelley, H.J. (1962), Methods of gradients, in: Leitmann (1962), pp. 205-254 Kelley, H.J., G.E. Myers (1971), Conjugate direction methods for parameter optimization, Astron. Acta 16, 45-51 Kelley, H.J., J.L. Speyer (1970), Accelerated gradient projection, in: Balakrishnan et al. (1970), pp. 151-158 Kempthorne, O. (1952), The design and analysis of experiments, Wiley, New York Kenworthy, I.C. (1967), Some examples of simplex evolutionary operation in the paper industry, Appl. Stat. 16, 211-224 Kesten, H. (1958), Accelerated stochastic approximation, Ann. Math. Stat. 29, 41-59

Khachiyan, L.G. (1979), (abstract on the ellipsoid method), Doklady Akademii Nauk SSSR (USSR) 244, 1093-1096 Khovanov, N.V. (1967), Stochastic optimization of parameters by the method of variation of the search region, Engng. Cybern. 5(4), 34-39

284

References

Kiefer, J. (1953), Sequential minimax search for a maximum, Proc. Amer. Math. Soc. 4, 502-506 Kiefer, J. (1957), Optimum sequential search and approximation methods under minimum regularity assumptions, SIAM J. 5, 105-136 Kiefer, J., J. Wolfowitz (1952), Stochastic estimation of the maximum of a regression function, Ann. Math. Stat. 23, 462-466 King, R.F. (1973), An improved Pegasus method for root nding, BIT 13, 423-427

Kirkpatrick, S., C.D. Gelatt, M.P. Vecchi (1983), Optimization by simulated annealing, Science 220, 671-680 Kivelidi, V.Kh., Ya.I. Khurgin (1970), Construction of probabilistic search, ARC 31, 1892-1894

Kiwiel, K.C. (1985), Methods of descent for nondierentiable optimization, vol. 1133 of Lecture Notes in Mathematics, Springer, Berlin Kjellstrom, G. (1965), Network optimization by random variation of component values, Ericsson Technical 25, 133-151 Klerer, M., G.A. Korn (Eds.) (1967), Digital computer user's hand-book, McGraw-Hill, New York Klessig, R., E. Polak (1972), E cient implementations of the Polak-Ribi&ere conjugate gradient algorithm, SIAM J. Contr. 10, 524-549 Klessig, R., E. Polak (1973), An adaptive precision gradient method for optimal control, SIAM J. Contr. 11, 80-93 Klingman, W.R., D.M. Himmelblau (1964), Nonlinear programming with the aid of a multiple-gradient summation technique, JACM 11, 400-415 Klir, G.J. (Ed.) (1978), Applied general systems research, Plenum Press, New York Klockgether, J., H.-P. Schwefel (1970), Two-phase nozzle and hollow core jet experiments, in: Elliott (1970), pp. 141-148 Klotzler, R. (1970), Mehrdimensionale Variationsrechnung, Birkhauser, Basle, Switzerland Kobelt, D., G. Schneider (1977), Optimierung im Dialog unter Verwendung von Evolutionsstrategie und Einu!gro!enrechnung, Chemie-Technik 6, 369-372 Koch, H.W. (1973), Der Sozialdarwinismus|seine Genese und sein Einu! auf das imperialistische Denken, Beck, Munich

References

285

Kochen, M., H.M. Hastings (Eds.) (1988), Advances in cognitive science|steps toward convergence, AAAS Selected Symposium 104 Kohler, E. (Ed.) (1991), 36th International Scientic Colloquium, Ilmenau, Oct. 21-24, 1991, Technical University of Ilmenau, Germany Kopp, R.E. (1967), Computational algorithms in optimal control, IEEE Int'l Conv. Record 15, part 3 (Automatic Control), 5-14 Korbut, A.A., J.J. Finkelstein (1971), Diskrete Optimierung, Akademie-Verlag, Berlin Korn, G.A. (1966), Random process simulation and measurement, McGraw-Hill, New York Korn, G.A. (1968), Hybrid computer Monte Carlo techniques, in: McLeod (1968), pp. 223-234 Korn, G.A., T.M. Korn (1961), Mathematical handbook for scientists and engineers, McGraw-Hill, New York Korn, G.A., T.M. Korn (1964), Electronic analog and hybrid computers, McGraw-Hill, New York Korn, G.A., H. Kosako (1970), A proposed hybrid-computer method for functional optimization, IEEE Trans. C-19, 149-153 Kovacs, Z., S.A. Lill (1971), Note on algorithm 46|a modied Davidon method for nding the minimum of a function, using dierence approximation for derivatives, Comp. J. 14, 214-215 Kowalik, J. (1967), A note on nonlinear regression analysis, Austral. Comp. J. 1, 51-53 Kowalik, J., J.F. Morrison (1968), Analysis of kinetic data for allosteric enzyme reactions as a nonlinear regression problem, Math. Biosci. 2, 57-66 Kowalik, J., M.R. Osborne (1968), Methods for unconstrained optimization problems, Elsevier, New York Koza, J. (1992), Genetic programming, MIT Press, Cambridge MA Krallmann, H. (1978), Evolution strategy and social sciences, in: Klir (1978), pp. 891903 Krasnushkin, E.V. (1970), Multichannel automatic optimizer having a variable sign for the feedback, ARC 31, 2057-2061 Krasovskii, A.A. (1962), Optimal methods of search in continuous and pulsed extremum control systems, Proceedings of the 1st IFAC Symposium on Optimization and Adaptive Control, Rome, April 1962, pp. 19-33

286

References

Krasovskii, A.A. (1963), Problems of continuous systems theory of extremal control of industrial processes, Proceedings of the IInd IFAC Congress, Basle, Switzerland, Aug.-Sept. 1963, vol. 1, pp. 519-526 Krasulina, T.P. (1972), Robbins-Monro process in the case of several roots, ARC 33, 580-585 Kregting, J., R.C. White, Jr. (1971), Adaptive random search, Eindhoven University of Technology, Department of Electrical Engineering, Group Measurement and Control, report TH-71-E-24, Eindhoven, The Netherlands, Oct. 1971 Krelle, W., H.P. Kunzi (1958), Lineare Programmierung, Verlag Industrielle Organisation, Zurich, Switzerland Krolak, P.D. (1968), Further extensions of Fibonaccian search to linear programming problems, SIAM J. Contr. 6, 258-265 Krolak, P.D., L. Cooper (1963), An extension of Fibonaccian search to several variables, CACM 6, 639-641 Kuester, J.L., J.H. Mize (1973), Optimization techniques with Fortran, McGraw-Hill, New York Kuhn, H.W. (Ed.) (1970), Proceedings of the Princeton Symposium on Mathematical Programming, Aug. 1967, Princeton University Press, Princeton NJ Kuhn, H.W., A.W. Tucker (1951), Nonlinear programming, in: Neyman (1951), pp. 481-492 Kulchitskii, O.Yu. (1972), A non-gradient random search method for an extremum in a Hilbert space, Engng. Cybern. 10, 773-780 Kunzi, H.P. (1967), Mathematische Optimierung gro!er Systeme, Ablauf- und Planungsforschung 8, 395-407 Kunzi, H.P., W. Krelle (1969), Einfuhrung in die mathematische Optimierung, Verlag Industrielle Organisation, Zurich, Switzerland Kunzi, H.P., W. Krelle, W. Oettli (1962), Nichtlineare Programmierung, Springer, Berlin Kunzi, H.P., W. Oettli (1969), Nichtlineare Optimierung|neuere Verfahren|Bibliographie, Springer, Berlin Kunzi, H.P., S.T. Tan (1966), Lineare Optimierung gro!er Systeme, Springer, Berlin Kunzi, H.P., H.G. Tzschach, C.A. Zehnder (1966), Numerische Methoden der mathematischen Optimierung, Teubner, Stuttgart

References

287

Kunzi, H.P., H.G. Tzschach, C.A. Zehnder (1970), Numerische Methoden der mathematischen Optimierung mit Algol- und Fortran-Programmen|Gebrauchsversion der Computerprogramme, Teubner, Stuttgart Kuo, F.F., J.F. Kaiser (Eds.) (1966), System analysis by digital computer, Wiley, New York Kursawe, F. (1991), A variant of evolution strategies for vector optimization in: Schwefel and Manner (1991), pp. 193-197 Kursawe, F. (1992), Evolution strategies for vector optimization, in: Tzeng and Yu (1992), vol. 3, pp. 187-193 Kushner, H.J. (1963), Hill climbing methods for the optimization of multiparameter noise disturbed systems, Trans. ASME D, J. Basic Engng. (1963), 157-164 Kushner, H.J. (1972), Stochastic approximation algorithms for the local optimization of functions with nonunique stationary points, IEEE Trans. AC-17, 646-654 Kussul, E., A. Luk (1971), Evolution als Optimierungsproze!, Ideen des exakten Wissens (1971), 821-826 Kwakernaak, H. (1965), On-line iterative optimization of stochastic control systems, Automatica 2, 195-208 Kwakernaak, H. (1966), On-line dynamic optimization of stochastic control systems, Proceedings of the IIIrd IFAC Congress, London, June 1966, paper 29-D Kwatny, H.G. (1972), A note on stochastic approximation algorithms in system identication, IEEE Trans. AC-17, 571-572 Laarhoven, P.J.M. van, E.H.L. Aarts (1987), Simulated annealing, theory and applications, Reidel, Dordrecht, The Netherlands Land, A.H., S. Powell (1973), Fortran codes for mathematical programming|linear, quadratic and discrete, Wiley, London Lange-Nielsen, T., G.M. Lance (1972), A pattern search algorithm for feedback-control system parameter optimization, IEEE Trans. C-21, 1222-1227 Langguth, V. (1972), Ein Identikationsverfahren fur lineare Systeme mit Hilfe von stochastischen Suchverfahren und unter Anwendung der Sequentialanalyse fur stochastische Fehlersignale, messen-steuern-regeln 15, 293-296 Langton, C.G. (Ed.) (1989), Articial life, Proceedings of an Interdisciplinary Workshop on the Synthesis and Simulation of Living Systems, Los Alamos NM, Sept. 1987, Proceedings vol. VI of Santa Fe Institute Studies in the Science of Complexity, Addison-Wesley, Redwoood City CA

288

References

Langton, C.G. (Ed.) (1994a), Articial life III, Proceedings of the Workshop on Articial Life, Santa Fe NM, June 1992, Proceedings vol. XVII of Santa Fe Institute Studies in the Science of Complexity, Addison-Wesley, Reading MA Langton, C.G. (Ed.) (1994b), Articial life (journal), MIT Press, Cambridge MA Langton, C.G., C. Taylor, J.D. Former, S. Rasmussen (Eds.) (1992), Articial life II, Proceedings of the Second Interdisciplinary Workshop on the Synthesis and Simulation of Living Systems, Santa Fe NM, Feb. 1990, Proceedings vol. X of Santa Fe Institute Studies in the Science of Complexity, Addison-Wesley, Reading MA Lapidus, L., E. Shapiro, S. Shapiro, R.E. Stillman (1961), Optimization of process performance, AIChE J. 7(2), 288-294 Larichev, O.I., G.G. Gorvits (1974), New approach to comparison of search methods used in nonlinear programming problems, JOTA 13, 635-659 Larson, R.E., E. Tse (1973), Parallel processing algorithms for the optimal control of nonlinear dynamic systems, IEEE Trans. C-22, 777-786 Lasdon, L.S. (1970), Conjugate direction methods for optimal control, IEEE Trans. AC-15, 267-268 Lau!ermair, T. (1992a), Hyperachen-Annealing|ein paralleles Optimierungsverfahren basierend auf selbstorganisierter Musterbildung durch Relaxation auf gekrummten Hyperachen, Dr. rer. nat. Diss., Technical University of Munich, Department of Mathematics and Computer Science, April 1992 Lau!ermair, T. (1992b), Hyperplane annealing and activator-inhibitor-systems, in: Manner and Manderick (1992), pp. 521-530 Lavi, A., T.P. Vogl (Eds.) (1966), Recent advances in optimization techniques, Wiley, New York Lawler, E.L., J.K. Lenstra, A.H.G. Rinooy Kan, D.B. Shmoys (Eds.) (1985), The travelling salesman problem, a guided tour of combinatorial optimization, WileyInterscience, New York Lawrence, J.P., III, F.P. Emad (1973), An analytic comparison of random searching for the extremum and gradient searching of a known objective function, IEEE Trans. AC-18, 669-671 Lawrence, J.P., III, K. Steiglitz (1972), Randomized pattern search, IEEE Trans. C-21, 382-385 LeCam, L.M., J. Neyman (Eds.) (1967), Proceedings of the Vth Berkeley Symposium on Mathematical Statistics and Probability, 1965/66, vol. 4: Biology and Problems of Health, University of California Press, Berkeley CA

References

289

LeCam, L.M., J. Neyman, E.L. Scott (Eds.) (1972), Proceedings of the VIth Berkeley Symposium on Mathematical Statistics and Probability, 1970/71, vol. 5: Darwinian, Neo-Darwinian and Non-Darwinian Evolution, University of California Press, Berkeley CA Lee, R.C.K. (1964), Optimal estimation, identication, and control, MIT Press, Cambridge MA Lehner, K. (1991), Einsatz wissensbasierter Systeme in der Strukturoptimierung dargestellt am Beispiel Fachwerkoptimierung, Dr.-Ing. Diss., University of Bochum, Faculty of Civil Engineering, May 1991 Leibniz, G.W. (1710), Theodicee, 4th rev. ed., Forster, Hannover, 1744 Leitmann, G. (Ed.) (1962), Optimization techniques with applications to aerospace systems, Academic Press, New York Leitmann, G. (1964), Einfuhrung in die Theorie optimaler Steuerung und der Dierentialspiele|eine geometrische Darstellung, Oldenbourg, Munich Leitmann, G. (Ed.) (1967), Topics in optimization, Academic Press, New York Lemarechal, C., R. Mi'in (Eds.) (1978), Nonsmooth optimization, vol. 3 of IIASA Proceedings Series, Pergamon Press, Oxford UK Leon, A. (1966a), A comparison among eight known optimizing procedures, in: Lavi and Vogl (1966), pp. 23-46 Leon, A. (1966b), A classied bibliography on optimization, in: Lavi and Vogl (1966), pp. 599-649 Lerner, A.Ja., E.A. Rosenman (1973), Optimale Steuerungen, Verlag Technik, Berlin Lesniak, Z.K. (1970), Methoden der Optimierung von Konstruktionen unter Benutzung von Rechenautomaten, W. Ernst, Berlin Levenberg, K. (1944), A method for the solution of certain non-linear problems in least squares, Quart. Appl. Math. 2, 164-168 Levine, L. (1964), Methods for solving engineering problems using analog computers, McGraw-Hill, New York Levine, M.D., T. Vilis (1973), On-line learning optimal control using successive approximation techniques, IEEE Trans. AC-18, 279-284 Lew, H.S. (1972), An arithmetical approach to the mechanics of blood ow in small caliber blood vessels, J. Biomech. 5, 49-69

290

References

Ley!ner, U. (1974), Uber den Einsatz Linearer Programmierung beim Entwurf optimaler Leichtbaustabwerke, Dr.-Ing. Diss., Technical University of Berlin, Department of Transportation Technologies, June 1974 Lill, S.A. (1970), Algorithm 46|a modied Davidon method for nding the minimum of a function, using dierence approximation for derivatives, Comp. J. 13, 111-113 Lill, S.A. (1971), Note on algorithm 46|a modied Davidon method, Comp. J. 14, 106 Little, W.D. (1966), Hybrid computer solutions of partial dierential equations by Monte Carlo methods, AFIPS Conf. Proc. 29, 181-190 Ljapunov, A.A. (Ed.), W. Kammerer, H. Thiele (Eds.) (1964a), Probleme der Kybernetik, vol. 4, Akademie-Verlag, Berlin Ljapunov, A.A. (Ed.), W. Kammerer, H. Thiele (Eds.) (1964b), Probleme der Kybernetik, vol. 5, Akademie-Verlag, Berlin Locker, A. (Ed.) (1973), Biogenesis - evolution - homeostasis, Springer, Berlin Loginov, N.V. (1966), Methods of stochastic approximation, ARC 27, 706-728 Lohmann, R. (1992), Structure evolution and incomplete induction, in: Manner and Manderick (1992), pp. 175-185 Lootsma, F.A. (Ed.) (1972a), Numerical methods for non-linear optimization, Academic Press, London Lootsma, F.A. (1972b), A survey of methods for solving constrained minimization problems via unconstrained minimization, in: Lootsma (1972a), pp. 313-347 Lowe, C.W. (1964), Some techniques of evolutionary operation, Trans. Inst. Chem. Engrs. 42, T334-T344 Lucas, E. (1876), Note sur l'application des series recurrentes &a la recherche de la loi de distribution de nombres premiers, Compt. Rend. Hebdomad. Seances Acad. Sci. Paris 82, 165-167 Luce, A.D., H. Raia (1957), Games and decisions, Wiley, New York Luenberger, D.G. (1972), Mathematical programming and control theory|trends of interplay, in: Georion (1972), pp. 102-133 Luenberger, D.G. (1973), Introduction to linear and nonlinear programming, AddisonWesley, Reading MA Machura, M., A. Mulawa (1973), Algorithm 450 (E4)|Rosenbrock function minimization, CACM 16, 482-483 Madsen, K. (1973), A root-nding algorithm based on Newton's method, BIT 13, 71-75

References

291

Mamen, R., D.Q. Mayne (1972), A pseudo Newton-Raphson method for function minimization, JOTA 10, 263-277 Mandischer, M. (1993), Representation and evolution of neural networks, in: Albrecht, Reeves, and Steele (1993), pp. 643-649 Mangasarian, O.L. (1969), Nonlinear programming, McGraw-Hill, New York Manner, R., B. Manderick (Eds.) (1992), Parallel problem solving from nature 2, Proceedings of the 2nd PPSN Conference, Brussels, Sept. 28-30, 1992, North-Holland, Amsterdam Marfeld, A.F. (1970), Kybernetik des Gehirns|ein Kompendium der Grundlagenforschung, Safari Verlag, Berlin Markwich, P. (1978), Der thermische Wasserstrahlantrieb auf der Grundlage des oenen Clausius-Rankine-Prozesses|Konzeption und hydrothermodynamische Analyse, Dr.-Ing. Diss., Technical University of Berlin, Department of Transportation Technologies Marquardt, D.W. (1963), An algorithm for least-squares estimation of non linear parameters, SIAM J. 11, 431-441 Marti, K. (1980), On accelerations of the convergence in random search methods, Methods of Oper. Res. 37, 391-406 Masters, C.O., H. Drucker (1971), Observations on direct search procedures, IEEE Trans. SMC-1, 182-184 Matthews, A., D. Davies (1971), A comparison of modied Newton methods for unconstrained optimisation, Comp. J. 14, 293-294 Matyas, J. (1965), Random optimization ARC 26, 244-251

Matyas, J. (1967), Das zufallige Optimierungsverfahren und seine Konvergenz, Proceedings of the Vth International Analogue Computation Meeting, Lausanne, Aug.-Sept. 1967, vol. 1, pp. 540-544 Maxeld, M., A. Callahan, L.J. Fogel (Eds.) (1965), Biophysics and cybernetic systems, Spartan, Washington, DC Maybach, R.L. (1966), Solution of optimal control problems on a high-speed hybrid computer, Simulation 9, 238-245 McArthur, D.S. (1961), Strategy in research|alternative methods for design of experiments, IRE Trans. EM-8, 34-40 McCormick, G P. (1969), Anti-zig-zagging by bending, Mgmt. Sci. 15, 315-320

References

293

Meissinger, H.F., G.A. Bekey (1966), An analysis of continuous parameter identication methods, Simulation 6, 94-102 Meredith, D.L., C.L. Karr, K.K. Kumar (1992), The use of genetic algorithms in the design of fuzzy-logic controllers, 3rd Workshop on Neural Networks|Academic / Industrial / Defence (WNN '92), vol. SPIE-1721, pp. 545-555, International Society of Optical Engineering Merzenich, W. (1972), Ein einfaches mathematisches Evolutionsmodell, GMD Mitteilungen 21, Bonn Metropolis, N., A.W. Rosenbluth, M.W. Rosenbluth, A.H. Teller, E. Teller (1953), Equations of state calculations by fast computing machines, J. Chem. Phys. 21, 10871092 Meyer, H.A. (Ed.) (1956), Symposium on Monte Carlo methods, Wiley, New York Meyer, J.-A. (Ed.) (1992), Adaptive behavior (journal), MIT Press, Cambridge MA Meyer, J.-A., H.L. Roitblat, S.W. Wilson (Eds.) (1993), From animals to animats 2, Proceedings of the 2nd International Conference on Simulation of Adaptive Behavior (SAB '92), Honolulu HI, Dec. 7-11, 1992, MIT Press, Cambridge MA Meyer, J.-A., S.W. Wilson (Eds.) (1991), From animals to animats, Proceedings of the 1st International Conference on Simulation of Adaptive Behavior (SAB), Paris, Sept. 24-28, 1990, MIT Press, Cambridge MA Michalewicz, Z. (1992), Genetic algorithms + data structures = evolution programs, Springer, Berlin Michalewicz, Z. (1994), Genetic algorithms + data structures = evolution programs, 2nd ext. ed., Springer, Berlin Michie, D. (1971), Heuristic search, Comp. J. 14, 96-102 Miele, A. (1969), Variational approach to the gradient method|theory and numerical experiments, in: Zadeh, Neustadt, and Balakrishnan (1969b), pp. 143-157 Miele, A., J.W. Cantrell (1969), Study on a memory gradient method for the minimization of functions, JOTA 3, 459-470 Miele, A., J.W. Cantrell (1970), Memory gradient method for the minimization of functions, in: Balakrishnan et al. (1970), pp. 252-263 Miele, A., J.N. Damoulakis, J.R. Cloutier, J.L. Tietze (1974), Sequential gradientrestoration algorithm for optimal control problems with nondierential constraints, JOTA 13, 218-255

294

References

Miele, A., H.Y. Huang, J.C. Heidemann (1969), Sequential gradient-restoration algorithm for the minimization of constrained functions|ordinary and conjugate gradient versions, JOTA 4, 213-243 Miele, A., A.V. Levy, E.E. Cragg (1971), Modications and extensions of the conjugate gradient-restoration algorithm for mathematical programming problems, JOTA 7, 450-472 Miele, A., J.L. Tietze, A.V. Levy (1972), Summary and comparison of gradient-restoration algorithms for optimal control problems, JOTA 10, 381-403 Miller, R.E. (1973), A comparison of some theoretical models of parallel computation, IEEE Trans. C-22, 710-717 Millstein, R.E. (1973), Control structures in Illiac IV Fortran, CACM 16, 621-627 Minot, O.N. (1969), Articial intelligence and new simulations, Simulation 13, 214-215 Minsky, M. (1961), Steps toward articial intelligence, IRE Proc. 49, 8-30 Miranker, W.L. (1969), Parallel methods for approximating the root of a function, IBM J. Res. Dev. 13, 297-301 Miranker, W.L. (1971), A survey of parallelism in numerical analysis, SIAM Review 13, 524-547 Mitchell, B.A., Jr. (1964), A hybrid analog-digital parameter optimizer for Astrac II, AFIPS Conf. Proc. 25, 271-285 Mitchell, R.A., J.L. Kaplan (1968), Nonlinear constrained optimization by a non-random complex method, NBS J. Res. C, Engng. Instr. 72, 249-258 Mlynski, D. (1964a), Der Wirkungsgrad experimenteller Optimierungsstrategien, Dr.Ing. Diss., Technical University (RWTH) of Aachen, Germany, Dec. 1964 Mlynski, D. (1964b), Maximalisierung durch logische Suchprozesse, in: Steinbuch and Wagner (1964), pp. 82-94 Mlynski, D. (1966a), Ein Beitrag zur statistischen Theorie der Optimierungsstrategien I and II, Regelungstechnik 14, 209-215 and 325-330 Mlynski, D. (1966b), E ciency of experimental strategies for optimising feedback control of disturbed processes, Proceedings of the IIIrd IFAC Congress, London, June 1966, paper 29-G Mockus, J.B. see also under Motskus, I.B. Mockus, J.B. (1971), On the optimization of power distribution systems, in: Schwarz (1971), technical papers, vol. 3, pp. 6.3.2-1 to 6.3.2-14

References

295

Moiseev, N.N. (Ed.) (1970), Colloquium on methods of optimization, Springer, Berlin Moran, P.A.P. (1967), Unsolved problems in evolutionary theory, in: LeCam and Neyman (1967), pp. 457-480 More, J.J., S.J. Wright (1993), Optimization software guide, vol. 14 of Frontiers in Applied Mathematics, SIAM, Philadelphia Morrison, D.D. (1968), Optimization by least squares, SIAM J. Numer. Anal. 5, 83-88 Motskus, I.B. see also under Mockus, J.B. Motskus, I.B. (1965), Some experiments related to the capabilities of man in solving multiextremal problems heuristically, Engng. Cybern. 3(3), 40-44 Motskus, I.B. (1967), Mnogoekstremalnye sadachi v projektirovanii, Nauka, Moscow Motskus, I.B., A.A. Feldbaum (1963), Symposium on multiextremal problems, Trakay, June 1963, Engng. Cybern. 1(5), 154-155 Movshovich, S.M. (1966), Random search and the gradient method in optimization problems, Engng. Cybern. 4(6), 39-48 Mufti, I.H. (1970), Computational methods in optimal control problems, Springer, Berlin Mugele, R.A. (1961), A nonlinear digital optimizing program for process control systems, AFIPS Conf. Proc. 19, 15-32 Mugele, R.A. (1962), A program for optimal control of nonlinear processes, IBM Systems J. 1, 2-17 Mugele, R.A. (1966), The probe and edge theorems for non-linear optimization, in: Lavi and Vogl (1966), pp. 131-144 Muhlenbein, H., D. Schlierkamp-Voosen (1993a), Predictive models for the breeder genetic algorithm I. Continuous Parameter Optimization, Evolutionary Computation 1, 25-49 Muhlenbein, H., D. Schlierkamp-Voosen (1993b), Optimal interaction of mutation and crossover in the breeder genetic algorithm, in: Forrest (1993), pp. 648 Muller-Merbach, H. (1971), Operations Research|Methoden und Modelle der Optimalplanung, 2nd ed., F. Vahlen, Berlin Munson, J.K., A.I. Rubin (1959), Optimization by random search on the analog computer, IRE Trans. EC-8, 200-203 Murata, T. (1963), The use of adaptive constrained descent in systems design, University of Illinois, Coordinated Science Laboratory, report R-189, Urbana IL, Dec. 1963

296

References

Murray, W. (Ed.) (1972a), Numerical methods for unconstrained optimization, Academic Press, London Murray, W. (1972b), Second derivative methods, in: Murray (1972a), pp. 57-71 Murray, W. (1972c), Failure, the causes and cures, in: Murray (1972a), pp. 107-122 Murtagh, B.A. (1970), A short description of the variable-metric method, in: Abadie (1970), pp. 525-528 Murtagh, B.A., R.W.H. Sargent (1970), Computational experience with quadratically convergent minimisation methods, Comp. J. 13, 185-194 Mutseniyeks, V.A., L.A. Rastrigin (1964), Extremal control of continuous multiparameter systems by the method of random search, Engng. Cybern. 2(1), 82-90 Myers, G.E. (1968), Properties of the conjugate-gradient and Davidon methods, JOTA 2, 209-219 Nachtigall, W. (1971), Biotechnik|statische Konstruktionen in der Natur, Quelle und Meyer, Heidelberg, Germany Nachtigall, W. (Ed.) (1992), Technische Biologie und Bionik 1, Proceedings of the 1st Congress on Bionics, Wiesbaden, June 11-13, 1992, BIONA report 8, G. Fischer, Stuttgart Nake, F. (1966), Zertikat zu Algorithmus 2|Orthonormierung von Vektoren nach E. Schmidt, Computing 1, 281 Neave, H.R. (1973), On using the Box-Muller transformation with multiplicative congruential pseudo-random number generators, Appl. Stat. 22, 92-97 Nelder, J.A., R. Mead (1965), A simplex method for function minimization, Comp. J. 7, 308-313 Nenonen, L.K., B. Pagurek (1969), Conjugate gradient optimization applied to a copper converter model, Automatica 5, 801-810 Neumann, J. von (1960), Die Rechenmaschine und das Gehirn, Oldenbourg, Munich Neumann, J. von (1966), Theory of self-reproducing automata, University of Illinois Press, Urbana-Champaign IL Neumann, J. von, O. Morgenstern (1961), Spieltheorie und wirtschaftliches Verhalten, Physica-Verlag, Wurzburg, Germany Newman, D.J. (1965), Location of the maximum on unimodal surfaces, JACM 12, 395398

References

297

Neyman, J. (Ed.) (1951), Proceedings of the IInd Berkeley Symposium on Mathematical Statistics and Probability, 1950, University of California Press, Berkeley CA Neyman, J. (Ed.) (1956), Proceedings of the IIIrd Berkeley Symposium on Mathematical Statistics and Probability, 1954/55, University of California Press, Berkeley CA Neyman, J. (Ed.) (1961), Proceedings of the IVth Berkeley Symposium on Mathematical Statistics and Probability, 1960, University of California Press, Berkeley CA Nickel, K. (1967), Allgemeine Forderungen an einen numerischen Algorithmus, ZAMM 47(Sonderheft), T67-T68 Nickel, K., K. Ritter (1972), Termination criterion and numerical convergence, SIAM J. Numer. Anal. 9, 277-283 Niederreiter, H. (1992), Random number generation and quasi-Monte Carlo methods, vol. 63 of CBMS-NSF Regional Conference Series in Applied Mathematics, SIAM, Philadelphia Niemann, H. (1974), Methoden der Mustererkennung, Akademische Verlagsgesellschaft, Frankfort/Main Nikolic, Z% .J., K.S. Fu (1966), An algorithm for learning without external supervision and its application to learning control systems, IEEE Trans. AC-11, 414-442 Nissen, V. (1993), Evolutionary algorithms in management science, report 9303 of the European Study Group for Evolutionary Economics Nissen, V. (1994), Evolutionare Algorithmen|Darstellung, Beispiele, betriebswirtschaftliche Anwendungsmoglichkeiten, DUV Deutscher Universitatsverlag, Wiesbaden Norkin, K.B. (1961), On one method for automatic search for the extremum of a function of many variables, ARC 22, 534-538 North, M. (1980), Time-dependent stochastic model of oods, Proc. ASCE, J. Hydraulics Div. 106-HY5, 649-665 Nurminski, E.A. (Ed.) (1982), Progress in nondierentiable optimization, IIASA Collaborative Proceedings Series CP-82-58, International Institute for Applied Systems Analysis, Laxenburg, Austria Odell, P.L. (1961), An empirical study of three stochastic approximation techniques applicable to sensitivity testing, report NAVWEPS-7837 Oestreicher, H.L., D.R. Moore (Eds.) (1968), Cybernetic problems in bionics, Gordon Breach, New York Oi, K., H. Sayama, T. Takamatsu (1973), Computational schemes of the DavidonFletcher-Powell method in innite-dimensional space, JOTA 12, 447-458

298

References

Oldenburger, R. (Ed.) (1966), Optimal and self optimizing control, MIT Press, Cambridge MA Oliver, L.T., D.J. Wilde (1964), Symmetric sequential minimax search for a maximum, Fibonacci Quart. 2, 169-175 O'Neill, R. (1971), Algorithm AS 47|function minimization using a simplex procedure, Appl. Stat. 20, 338-345 Opa%cic, J. (1973), A heuristic method for nding most extrema of a nonlinear functional, IEEE Trans. SMC-3, 102-107 Oren, S.S. (1973), Self-scaling variable metric algorithms without line search for unconstrained minimization, Math. Comp. 27, 873-885 Ortega, J.M., W.C. Rheinboldt (1967), Monotone iterations for nonlinear equations with application to Gauss-Seidel methods, SIAM J. Numer. Anal. 4, 171-190 Ortega, J.M., W.C. Rheinboldt (1970), Iterative solution of nonlinear equations in several variables, Academic Press, New York Ortega, J.M., W.C. Rheinboldt (1972), A general convergence result for unconstrained minimization methods, SIAM J. Numer. Anal. 9, 40-43 Ortega, J.M., M.L. Rocko (1966), Nonlinear dierence equations and Gauss-Seidel type iterative methods, SIAM J. Numer. Anal. 3, 497-513 Osborne, M.R. (1972), Some aspects of nonlinear least squares calculations, in: Lootsma (1972a), pp. 171-189 Osche, G. (1972), Evolution|Grundlagen, Erkenntnisse, Entwicklungen der Abstammungslehre, Herder, Freiburg, Germany Ostermeier, A. (1992), An evolution strategy with momentum adaptation of the random number distribution, in: Manner and Manderick (1992), pp. 197-206 Ostrowski, A.M. (1966), Solution of equations and systems of equations, 2nd ed., Academic Press, New York Ostrowski, A.M. (1967), Contributions to the theory of the method of steepest descent, Arch. Ration. Mech. Anal. 26, 257-280 Overholt, K.J. (1965), An instability in the Fibonacci and golden section search methods, BIT 5, 284-286 Overholt, K.J. (1967a), Note on algorithm 2|Fibonacci search, and algorithm 7|Minx, and the golden section search, Comp. J. 9, 414 Overholt, K.J. (1967b), Algorithm 16|Gold, Comp. J. 9, 415

References

299

Overholt, K.J. (1967c), Algorithm 17|Goldsec, Comp J. 9, 415 Overholt, K.J. (1973), E ciency of the Fibonacci search method, BIT 13, 92-96 Page, S.E., D.W. Richardson (1992), Walsh functions, schema variance, and deception, Complex Systems 6, 125-135 Pagurek, B., C.M. Woodside (1968), The conjugate gradient method for optimal control problems with bounded control variables, Automatica 4, 337-349 Palmer, J.R. (1969), An improved procedure for orthogonalising the search vectors in Rosenbrock's and Swann's direct search optimisation methods, Comp. J. 12, 69-71 Papageorgiou, M. (1991), Optimierung|Statische, dynamische, stochastische Verfahren fur die Anwendung, Oldenbourg, Munich Papentin, F. (1972), A Darwinian evolutionary system, Dr. rer. nat. Diss., University of Tubingen, Germany Pardalos, P.M., J.B. Rosen (1987), Constrained global optimization|algorithms and applications, vol. 268 of Lecture Notes in Computer Science, Springer, Berlin Parkinson, J.M., D. Hutchinson (1972a), A consideration of non-gradient algorithms for the unconstrained optimization of functions of high dimensionality, in: Lootsma (1972a), pp. 99-113 Parkinson, J.M., D. Hutchinson (1972b), An investigation into the e ciency of variants on the simplex method, in: Lootsma (1972a), pp. 115-135 Pask, G. (1962), Physical and linguistic evolution in self-organizing systems, Proceedings of the 1st IFAC Symposium on Optimization and Adaptive Control, Rome, April 1962, pp. 199-227 Pask, G. (1971), A cybernetic experimental method and its underlying philosophy, Int'l J. Man-Machine Stud. 3, 279-337 Patrick, M.L. (1972), A highly parallel algorithm for approximating all zeros of a polynomial with only real zeros, CACM 15, 952-955 Pattee, H.H., E.A. Edelsack, L. Fein, A.B. Callahan (Eds.) (1966), Natural automata and useful simulations, Spartan, Washington, DC Paviani, D.A., D.M. Himmelblau (1969), Constrained nonlinear optimization by heuristic programming, Oper. Res. 17, 872-882 Pearson, J.D. (1969), Variable metric methods of minimization, Comp. J. 12, 171-178 Peckham, G. (1970), A new method for minimising a sum of squares without calculating gradients, Comp. J. 13, 418-420

300

References

Peschel, M. (1980), Ingenieurtechnische Entscheidungen|Modellbildung und Steuerung mit Hilfe der Polyoptimierung, Verlag Technik, Berlin Peters, E. (1989), OptimiEst|an optimizing expert system using topologies, in: Brebbia and Hernandez (1989), pp. 222-232 Peters, E. (1991), Ein Beitrag zur wissensbasierten Auswahl und Steuerung von Optimierverfahren, Dr. rer. nat. Diss., University of Dortmund, Department of Computer Science, May 1991 Pierre, D.A. (1969), Optimization theory with applications, Wiley, New York Pierson, B.L., S.G. Rajtora (1970), Computational experience with the Davidon method applied to optimal control problems, IEEE Trans. SSC-6, 240-242 Pike, M.C., I.D. Hill, F.D. James (1967), Note on algorithm 2|Fibonacci search, and on algorithm 7|Minx, and algorithm 2 modied|Fibonacci search, Comp. J. 9, 416-417 Pike, M.C., J. Pixner (1965), Algorithm 2|Fibonacci search, Comp. Bull. 8, 147 Pincus, M. (1970), A Monte Carlo method for the approximate solution of certain types of constrained optimization problems, Oper. Res. 18, 1225-1228 Pinkham, R.S. (1964), Random root location, SIAM J. 12, 855-864 Pinsker, I.Sh., B.M. Tseitlin (1962), A nonlinear optimization problem, ARC 23, 15101518 Plane, D.R., C. McMillan, Jr. (1971), Discrete optimization|integer programming and network analysis for management decisions, Prentice-Hall, Englewood Clis NJ Plaschko, P., K. Wagner (1973), Evolutions-Linearisierungs-Programm zur Darstellung von numerischen Daten durch beliebige Funktionen, report DLR-FB-73-55, DFVLR Porz-Wahn, Germany Pluznikov, L.N., V.O. Andreyev, E.S. Klimenko (1971), Use of random search method in industrial planning, Engng. Cybern. 9, 229-235 Polak, E. (1971), Computational methods in optimization|a unied approach, Academic Press, New York Polak, E. (1972), A survey of methods of feasible directions for the solution of optimal control problems, IEEE Trans. AC-17, 591-596 Polak, E. (1973), An historical survey of computational methods in optimal control, SIAM Review 15, 553-584 Polak, E., G. Ribi&ere (1969), Note sur la convergence de methodes de directions conjuguees, Rev. Franc. Inf. Rech. Oper. 3(16), 35-43

References

301

Polyak, B.T. (1969), The conjugate gradient method in extremal problems, USSR Comp. Math. and Math. Phys. 9(4), 94-112 Ponstein J. (1967), Seven kinds of convexity, SIAM Review 9, 115-119 Pontrjagin, L.S., V.G. Boltjanskij, R.V. Gamkrelidze, E.F. Mis%cenko (1967), Mathematische Theorie optimaler Prozesse, 2nd ed., Oldenbourg, Munich Powell, D.R., J.R. MacDonald (1972), A rapidly convergent iterative method for the solution of the generalized nonlinear least squares problem, Comp. J. 15, 148-155 Powell, M.J.D. (1962), An iterative method for nding stationary values of a function of several variables, Comp. J. 5, 147-151 Powell, M.J.D. (1964), An e cient method for nding the minimum of a function of several variables without calculating derivatives, Comp. J. 7, 155-162 Powell, M.J.D. (1965), A method for minimizing a sum of squares of nonlinear functions without calculating derivatives, Comp. J. 7, 303-307 Powell, M.J.D. (1966), Minimization of functions of several variables, in: Walsh (1966), pp. 143-158 Powell, M.J.D. (1968a), On the calculation of orthogonal vectors, Comp. J. 11, 302-304 Powell, M.J.D. (1968b), A Fortran subroutine for solving systems of non-linear algebraic equations, UKAEA Research Group, report AERE-R-5947, Harwell, Oxon Powell, M.J.D. (1969), A theorem on rank one modications to a matrix and its inverse, Comp. J. 12, 288-290 Powell, M.J.D. (1970a), Rank one methods for unconstrained optimization, in: Abadie (1970), pp. 139-156 Powell, M.J.D. (1970b), A survey of numerical methods for unconstrained optimization, SIAM Review 12, 79-97 Powell, M.J.D. (1970c), A Fortran subroutine for unconstrained minimization, requiring rst derivatives of the objective function, UKAEA Research Group, report AERER-6469, Harwell, Oxon Powell, M.J.D. (1970d), A hybrid method for nonlinear equations, in: Rabinowitz (1970), pp. 87-114 Powell, M.J.D. (1970e), A Fortran subroutine for solving systems of nonlinear algebraic equations, in: Rabinowitz (1970), pp. 115-161 Powell, M.J.D. (1970f), Subroutine VA04A (Fortran), updated May 20th, 1970, in: Hopper (1971), p. 72

302

References

Powell, M.J.D. (1970g), Recent advances in unconstrained optimization, UKAEA Research Group, technical paper AERE-TP-430, Harwell, Oxon, Nov. 1970 Powell, M.J.D. (1971), On the convergence of the variable metric algorithm, JIMA 7, 21-36 Powell, M.J.D. (1972a), Some properties of the variable metric algorithm, in: Lootsma (1972a), pp. 1-17 Powell, M.J.D. (1972b), Quadratic termination properties of minimization algorithms I|Statement and discussion of results, JIMA 10, 333-342 Powell, M.J.D. (1972c), Quadratic termination properties of minimization algorithms II|Proofs and theorems, JIMA 10, 343-357 Powell, M.J.D. (1972d), A survey of numerical methods for unconstrained optimization, in: Georion (1972), pp. 3-21 Powell, M.J.D. (1972e), Problems related to unconstrained optimization, in: Murray (1972a), pp. 29-55 Powell, M.J.D. (1972f), Unconstrained minimization algorithms without computation of derivatives, UKAEA Research Group, technical paper AERE-TP-483, Harwell, Oxon, April 1972 Poznyak, A.S. (1972), Use of learning automata for the control of random search, ARC 33, 1992-2000 Press, W.H., S.A. Teukolsky, W.T. Vetterling, F.B. Flannery (1992), Numerical recipes in Fortran, 2nd ed., Cambridge University Press, Cambridge UK (especially Chap. 7, Random numbers, pp. 266-319) Prusinkiewicz, P., A. Lindenmayer (1990), The algorithmic beauty of plants|the virtual laboratory, Springer, Berlin Pugachev, V.N. (1970), Determination of the characteristics of complex systems using statistical trials and analytical investigation, Engng. Cybern. 8, 1109-1117 Pugh, E.L. (1966), A gradient technique of adaptive Monte Carlo, SIAM Review 8, 346-355 Pun, L. (1969), Introduction to optimization practice, Wiley, New York Rabinowitz, P. (Ed.) (1970), Numerical methods for nonlinear algebraic equations, Gordon Breach, London Ralston, A., H.S. Wilf (Eds.) (1967), Mathematische Methoden fur Digitalrechner, Oldenbourg, Munich

References

303

Ralston, A., H.S. Wilf (Eds.) (1969), Mathematische Methoden fur Digitalrechner II, Oldenbourg, Munich Rappl, G. (1984), Konvergenzraten von Random-Search-Verfahren zur globalen Optimierung, Dr. rer. nat. Diss., Hochschule der Bundeswehr, Munich-Neubiberg, Department of Computer Science, Nov. 1984 Rastrigin, L.A. (1960), Extremal control by the method of random scanning, ARC 21, 891-896 Rastrigin, L.A. (1963), The convergence of the random search method in the extremal control of a many-parameter system, ARC 24, 1337-1342 Rastrigin, L.A. (1965a), Sluchainyi poisk v zadachakh optimisatsii mnogoparametricheskikh sistem, Zinatne, Riga, (for a translation into English see next item) Rastrigin, L.A. (1965b), Random search in optimization problems for multiparameter systems, Air Force System Command, Foreign Technical Division, FTD-HT-67-363 Rastrigin, L.A. (1966), Stochastic methods of complicated multi-parameter system optimization, Proceedings of the IIIrd IFAC Congress, London, June 1966, paper 3-F Rastrigin, L.A. (1967), Raboty po teorii i primeneniyu statisticheskikh metodov optimisatsii v institute elektroniki i vychislitelnoi tekhniki Akademii Nauk Latviiskoi SSR, Avtomatika i Vychislitelnaya Tekhnika (1967, 5), 31-40 Rastrigin, L.A. (1968), Statisticheskiye metody poiska, Nauka, Moscow Rastrigin, L.A. (1969), Teorija i primenenije sluchainovo poiska, Zinatne, Riga Rastrigin, L.A. (1972), Adaptivnye sistemy, vol. 1, Zinatne, Riga Rauch, S.W. (1973), A convergence theory for a class of nonlinear programming problems, SIAM J. Numer. Anal. 10, 207-228 Rawlins, G.J.E. (Ed.) (1991), Foundations of genetic algorithms, Morgan Kaufmann, San Mateo CA Rechenberg, I. (1964), Cybernetic solution path of an experimental problem, Royal Aircraft Establishment, Library Translation 1122, Farnborough, Hants, Aug. 1965, English translation of the unpublished written summary of the lecture \Kybernetische Losungsansteuerung einer experimentellen Forschungsaufgabe", delivered at the joint annual meeting of the WGLR and DGRR, Berlin, 1964 Rechenberg, I. (1973), Evolutionsstrategie|Optimierung technischer Systeme nach Prinzipien der biologischen Evolution, Frommann-Holzboog, Stuttgart Rechenberg, I. (1978), Evolutionsstrategien, in: Schneider and Ranft (1978), pp. 83-114

304

References

Rechenberg, I. (1989), Evolution strategy|nature's way of optimization, in: Bergmann (1989), pp. 106-126 Rechenberg, I. (1994), Evolutionsstrategie '94, Frommann-Holzboog, Stuttgart Rein, H., M. Schneider (1971), Einfuhrung in die Physiologie des Menschen, Springer, Berlin Rhead, D.G. (1971), Some numerical experiments on Zangwill's method for unconstrained minimization, University of London, Institute of Computer Science, working paper ICSI-319 Ribi&ere, G. (1970), Sur la methode de Davidon-Fletcher-Powell pour la minimisation des fonctions, Mgmt. Sci. 16, 572-592 Rice, J.R. (1966), Experiments on Gram-Schmidt orthogonalization, Math. Comp. 20, 325-328 Richardson, J.A., J.L. Kuester (1973), Algorithm 454 (E4)|the complex method for constrained optimization, CACM 16, 487-489 Riedl, R. (1976), Die Strategie der Genesis, Piper, Munich Robbins, H., S. Monro (1951), A stochastic approximation method, Ann. Math. Stat. 22, 400-407 Roberts, P.D., R.H. Davis (1969), Conjugate gradients, Control 13, 206-210 Roberts, S.M., H.I. Lyvers (1961), The gradient method in process control, Ind. Engng. Chem. 53, 877-882 Rodlo, R.K. (1976), Bestimmung der Geschwindigkeit von Versetzungsgruppen in neutronen-bestrahlten Kupfer-Einkristallen, Dr. rer. nat. Diss., Technical University of Braunschweig, Germany, Sept. 1976 Rosen, J.B. (1960), The gradient projection method for nonlinear programming I| Linear constraints, SIAM J. 8, 181-217 Rosen, J.B. (1961), The gradient projection method for nonlinear programming II| Nonlinear constraints, SIAM J. 9, 514-532 Rosen, J.B. (1966), Iterative solution of nonlinear optimal control problems, SIAM J. Contr. 4, 223-244 Rosen, J.B., O.L. Mangasarian, K. Ritter (Eds.) (1970), Nonlinear programming, Academic Press, New York Rosen, J.B., S. Suzuki (1965), Construction of nonlinear programming test problems, CACM 8, 113

References

305

Rosen, R. (1967), Optimality principles in biology, Butterworths, London Rosenblatt, F. (1958), The perceptron|a probabilistic model for information storage and organization in the brain, Psychol. Rev. 65, 386-408 Rosenbrock, H.H. (1960), An automatic method for nding the greatest or least value of a function, Comp. J. 3, 175-184 Rosenbrock, H.H., C. Storey (1966), Computational techniques for chemical engineers, Pergamon Press, Oxford UK Ross, G.J.S. (1971), The e cient use of function minimization in non-linear maximumlikelihood estimation, Appl. Stat. 19, 205-221 Rothe, R. (1959), Hohere Mathematik fur Mathematiker, Physiker und Ingenieure, I| Dierentialrechnung und Grundformeln der Integralrechnung nebst Anwendungen, 18th ed., Teubner, Leipzig, Germany Roughgarden, J.W. (1979), Theory of population genetics and evolutionary ecology, Macmillan, New York Rozvany, G. (Ed.) (1994), J. on Structural Optimization, Springer, Berlin Rudolph, G. (1991), Global optimization by means of distributed evolution strategies, in: Schwefel and Manner (1991), pp. 209-213 Rudolph, G. (1992a), On Correlated mutation in evolution strategies, in: Manner and Manderick (1992), pp. 105-114 Rudolph, G. (1992b), Parallel approaches to stochastic global optimization, in: Joosen and Milgrom (1992), pp. 256-267 Rudolph, G. (1993), Massively parallel simulated annealing and its relation to evolutionary algorithms, Evolutionary Computation 1(4), 361-383 Rudolph, G. (1994a), Convergence analysis of canonical genetic algorithms, IEEE Trans. NN-5, 96-101 Rudolph, G. (1994b), An evolutionary algorithm for integer programming, in: Davidor, Schwefel, and Manner (1994), pp. 139-148 Rutishauser, H. (1966), Algorithmus 2|Orthonormierung von Vektoren nach E. Schmidt, Computing 1, 159-161 Rybashov, M.V. (1965a), The gradient method of solving convex programming problems on electronic analog computers, ARC 26, 1886-1898 Rybashov, M.V. (1965b), Gradient method of solving linear and quadratic programming problems on electronic analog computers, ARC 26, 2079-2089

306

References

Rybashov, M.V. (1969), Insensitivity of gradient systems in the solution of linear problems on analog computers, ARC 30, 1679-1687 Ryshik, I.M., I.S. Gradstein (1963), Summen-, Produkt- und Integraltafeln, 2nd ed., Deutscher Verlag der Wissenschaften, Berlin Saaty, T.L. (1955), The number of vertices of a polyhedron, Amer. Math. Monthly 62, 326-331 Saaty, T.L. (1963), A conjecture concerning the smallest bound on the iterations in linear programming, Oper. Res. 11, 151-153 Saaty, T.L. (1970), Optimization in integers and related extremal problems, McGrawHill, New York Saaty, T.L., J. Bram (1964), Nonlinear mathematics, McGraw-Hill, New York Sacks, J. (1958), Asymptotic distribution of stochastic approximation procedures, Ann. Math. Stat. 29, 373-405 Sameh, A.H. (1971), On Jacobi and Jacobi-like algorithms for a parallel computer, Math. Comp. 25, 579-590 Samuel, A.L. (1963), Some studies in machine learning using the game of checkers, in: Feigenbaum and Feldman (1963), pp. 71-105 Sargent, R.W.H., D.J. Sebastian (1972), Numerical experience with algorithms for unconstrained minimization, in: Lootsma (1972a), pp. 45-68 Sargent, R.W.H., D.J. Sebastian (1973), On the convergence of sequential minimization algorithms, JOTA 12, 567-575 Saridis, G.N. (1968), Learning applied to successive approximation algorithms, Proceedings of the 1968 Joint Automatic Control Conference, Ann Arbor MI, pp. 1007-1013 Saridis, G.N. (1970), Learning applied to successive approximation algorithms, IEEE Trans. SSC-6, 97-103 Saridis, G.N., H.D. Gilbert (1970), Self-organizing approach to the stochastic fuel regulator problem, IEEE Trans. SSC-6, 186-191 Satterthwaite, F.E. (1959a), REVOP or random evolutionary operation, Merrimack College, report 10-10-59, North Andover MA Satterthwaite, F.E. (1959b), Random balance experimentation, Technometrics 1, 111137 Satterthwaite, F.E., D. Shainin (1959), Pinpoint important process variable with polyvariable experimentation, J. Soc. Plast. Engrs. 15, 225-230

References

307

Savage, J.M. (1966), Evolution, Bayerischer Landwirtschafts-Verlag, Munich Sawaragi, Y., T. Takamatsu, K. Fukunaga, E. Nakanishi, H. Tamura (1971), Dynamic version of steady state optimizing control of distillation column by trial method, Automatica 7, 509-516 Schaer, J.D. (Ed.) (1989), Proceedings of the 3rd International Conference on Genetic Algorithms, George Mason University, Fairfax VA, June 4-7, 1989, Morgan Kaufmann, San Mateo CA Schechter, R.S. (1962), Iteration methods for nonlinear problems, Trans. Amer. Math. Soc. 104, 179-189 Schechter, R.S. (1968), Relaxation methods for convex problems, SIAM J. Numer. Anal. 5, 601-612 Schechter, R.S. (1970), Minimization of a convex function by relaxation, in: Abadie (1970), pp. 177-190 Scheeer, L. (1886), Uber die Bedeutung der Begrie \Maximum und Minimum" in der Variationsrechnung, Mathematische Annalen 26, 197-208 Scheel, A. (1985), Beitrag zur Theorie der Evolutionsstrategie, Dr.-Ing. Diss., Technical University of Berlin, Department of Process Engineering Scheuer, E.M., D.S. Stoller (1962), On the generation of normal random vectors, Technometrics 4, 278-281 Schinzinger, R. (1966), Optimization in electromagnetic system design, in: Lavi and Vogl (1966), pp. 163-214 Schittkowski, K. (1980), Nonlinear programming codes, vol. 183 of Lecture Notes in Economics and Mathematical Systems, Springer, Berlin Schley, C.H., Jr. (1968), Conjugate gradient methods for optimization, General Electric Research and Development Center, report 68-C-008, Schenectady NY, Jan. 1968 Schmalhausen, I.I. (1964), Grundlagen des Evolutionsprozesses vom kybernetischen Standpunkt, in: Ljapunov, Kammerer, and Thiele (1964a), pp. 151-188 Schmetterer, L. (1961), Stochastic approximation, in: Neyman (1961), vol. 1, pp. 587609 Schmidt, J.W., H. Schwetlick (1968), Ableitungsfreie Verfahren mit hoherer Konvergenzgeschwindigkeit, Computing 3, 215-226 Schmidt, J.W., H.F. Trinkaus (1966), Extremwertermittlung mit Funktionswerten bei Funktionen von mehreren Veranderlichen, Computing 1, 224-232

308

References

Schmidt, J.W., K. Vetters (1970), Ableitungsfreie Verfahren fur nichtlineare Optimierungsprobleme, Numer. Math. 15, 263-282 Schmitt, E. (1969), Adaptive computer algorithms for optimization and root-nding, NTZ-report 6, VDE Verlag, Berlin Schneider, B., U. Ranft (Eds.) (1978), Simulationsmethoden in der Medizin und Biologie, Springer, Berlin Schrack, G., N. Borowski (1972), An experimental comparison of three random searches, in: Lootsma (1972a), pp. 137-147 Schumer, M.A. (1967), Optimization by adaptive random search, Ph.D. thesis, Princeton University, Princeton NJ, Nov. 1967 Schumer, M.A. (1969), Hill climbing on a sample function of a Gaussian Markov process, JOTA 4, 413-418 Schumer, M.A., K. Steiglitz (1968), Adaptive step size random search, IEEE Trans. AC-13, 270-276 Schuster, P. (1972), Vom Makromolekul zur primitiven Zelle|die Entstehung biologischer Funktion, Chemie in unserer Zeit 6(1), 1-16 Schwarz, H. (Ed.) (1971), Multivariable technical control systems, North-Holland, Amsterdam Schwarz, H.R., H. Rutishauser, E. Stiefel (1968), Numerik symmetrischer Matrizen, Teubner, Stuttgart Schwefel, D. et al. (1972), Gesundheitsplanung im Departamento del Valle del Cauca, report of the German Development Institute, Berlin, July 1972 Schwefel, H.-P. (1968), Experimentelle Optimierung einer Zweiphasenduse Teil I, report 35 for the project MHD-Staustrahlrohr, AEG Research Institute, Berlin, Oct. 1968 Schwefel, H.-P. (1974), Adaptive Mechanismen in der biologischen Evolution und ihr Einu! auf die Evolutionsgeschwindigkeit, Internal report of the Working Group of Bionics and Evolution Techniques at the Institute for Measurement and Control Technology, Technical University of Berlin, Department of Process Engineering, July 1974 Schwefel, H.-P. (1975a), Evolutionsstrategie und numerische Optimierung, Dr.-Ing. Diss., Technical University of Berlin, Department of Process Engineering Schwefel, H.-P. (1975b), Binare Optimierung durch somatische Mutation, Internal report of the Working Group of Bionics and Evolution Techniques at the Institute of for Measurement and Control Technology, Technical University of Berlin (and the Central Animal Laboratory of the Medical High School of Hannover, SFB 146 Versuchstierforschung of the Veterinary High School of Hannover), May 1975

References

309

Schwefel, H.-P. (1980), Subroutines EVOL, GRUP, KORR|Listings and User's Guides, Internal report of the Programme Group of Systems Analysis and Technological Development, KFA-STE-IB-2/80, April 1980, Nuclear Research Center (KFA) Julich, Germany Schwefel, H.-P. (1981), Optimum Seeking Methods|User's Guides, Internal report of the Programme Group of Systems Analysis and Technological Development, KFASTE-IB-7/81, Oct. 1981, Nuclear Research Center (KFA) Julich, Germany Schwefel, H.-P. (1987), Collective phenomena in evolutionary systems, in: Checkland and Kiss (1987), vol. 2, pp. 1025-1033 Schwefel, H.-P. (1988), Towards large-scale long-term systems analysis, in: Cheng (1988), pp. 375-381 Schwefel, H.-P., F. Kursawe (1992), Kunstliche Evolution als Modell fur naturliche Intelligenz, in: Nachtigall (1992), pp. 73-91 Schwefel, H.-P., R. Manner (Eds.) (1991), Parallel problem solving from nature, Proceedings of the 1st PPSN Workshop, Dortmund, Oct. 1-3, 1990, vol. 496 of Lecture Notes in Computer Science, Springer, Berlin Schwetlick, H. (1970), Algorithmus 12|Ein ableitungsfreies Verfahren zur Losung endlich-dimensionaler Gleichungssysteme, Computing 5, 82-88 and 393 Sebald, A.V., L.J. Fogel (Eds.) (1994), Proceedings of the 3rd Annual Conference on Evolutionary Programming, San Diego CA, Feb. 24-26, 1994, World Scientic, Singapore Sebastian, H.-J., K. Tammer (Eds.) (1990), System Modelling and Optimization, vol. 143 of Lecture Notes in Control and Information Sciences, Springer, Berlin Sergiyevskiy, G.M., A.P. Ter-Saakov (1970), Factor experiments in many-dimensional stochastic approximation of an extremum, Engng. Cybern. 8, 949-954 Shah, B.V., R.J. Buehler, O. Kempthorne (1964), Some algorithms for minimizing a function of several variables, SIAM J. 12, 74-92 Shanno, D.F. (1970a), Parameter selection for modied Newton methods for function minimization, SIAM J. Numer. Anal. 7, 366-372 Shanno, D.F. (1970b), Conditioning of quasi-Newton methods for function minimization, Math. Comp. 24, 647-656 Shanno, D.F., P.C. Kettler (1970), Optimal conditioning of quasi-Newton methods, Math. Comp. 24, 657-664 Shapiro, I.J., K.S. Narendra (1969), Use of stochastic automata for parameter selfoptimization with multimodal performance criteria, IEEE Trans. SSC-5, 352-360

310

References

Shedler, G.S. (1967), Parallel numerical methods for the solution of equations, CACM 10, 286-291 Shimizu, T. (1969), A stochastic approximation method for optimization problems, JACM 16, 511-516 Shubert, B.O. (1972), A sequential method seeking the global maximum of a function, SIAM J. Numer. Anal. 9, 379-388 Sigmund, K. (1993), Games of life|explorations in ecology, evolution, and behavior, Oxford University Press, Oxford UK Silverman, G. (1969), Remark on algorithm 315 (E4)|the damped Taylor's series method for minimizing a sum of squares and for solving systems of non-linear equations, CACM 12, 513 Singer, E. (1962), Simulation and optimization of oil renery design, in: Cooper (1962), pp. 62-74 Sirisena, H.R. (1973), Computation of optimal controls using a piecewise polynomial parameterization, IEEE Trans. AC-18, 409-411 Slagle, J.R. (1972), Einfuhrung in die heuristische Programmierung|kunstliche Intelligenz und intelligente Maschinen, Verlag Moderne Industrie, Munich Smith, C.S. (1962), The automatic computation of maximum likelihood estimates, National Coal Board, Scientic Department, report SC-846-MR-40, London, June 1962 Smith, D.E. (1973), An empirical investigation of optimum-seeking in the computer simulation situation, Oper. Res. 21, 475-497 Smith, F.B., Jr., D.F. Shanno (1971), An improved Marquardt procedure for non-linear regressions, Technometrics 13, 63-74 Smith, J. Maynard (1982), Evolution and the theory of games, Cambridge University Press, Cambridge UK Smith, J. Maynard (1989), Evolutionary genetics, Oxford University Press, Oxford UK Smith, L.B. (1969), Remark on algorithm 178 (E4)|direct search, CACM 12, 638 Smith, N.H., D.F. Rudd (1964), The feasibility of directed random search, University of Wisconsin, Department of Chemical Engineering, report Snell, F.M. (Ed.) (1967), Progress in theoretical biology, vol. 1, Academic Press, New York Sorenson, H.W. (1969), Comparison of some conjugate direction procedures for function minimization, J. Franklin Inst. 288, 421-441

References

311

Sou%cek, B. and the IRIS Group (Eds.) (1992), Dynamic, genetic, and chaotic programming, vol. 5 of Sixth-Generation Computer Technology Series, Wiley-Interscience, New York Southwell, R.V. (1940), Relaxation methods in engineering science|a treatise on approximate computation, Oxford University Press, Oxford UK Southwell, R.V. (1946), Relaxation methods in theoretical physics, Clarendon Press, Oxford UK Spath, H. (1967), Algorithm 315 (E4, C5)|the damped Taylor's series method for minimizing a sum of squares and for solving systems of nonlinear equations, CACM 10, 726-728 Spang, H.A., III (1962), A review of minimization techniques for nonlinear functions, SIAM Review 4, 343-365 Spears, W.M., K.A. De Jong, T. Back, D.B. Fogel, H. de Garis (1993), An overview of evolutionary computation, in: Brazdil (1993), pp. 442-459 Spedicato, E. (1973), Stability of Huang's update for the conjugate gradient method, JOTA 11, 469-479 Spendley, W. (1969), Nonlinear least squares tting using a modied simplex minimization method, in: Fletcher (1969a), pp. 259-270 Spendley, W., G.R. Hext, F.R. Himsworth (1962), Sequential application of simplex designs in optimisation and evolutionary operation, Technometrics 4, 441-461 Speyer, J.L., H.J. Kelley, N. Levine, W.F. Denham (1971), Accelerated gradient projection technique with application to rocket trajectory optimization, Automatica 7, 37-43 Sprave, J. (1993), Zellulare Evolutionare Algorithmen zur Parameteroptimierung, in: Hofestadt, Kruckeberg, and Lengauer (1993), pp. 111-120 Sprave, J. (1994), Linear neighborhood evolution strategy, in: Sebald and Fogel (1994), pp. 42-51 Stanton, E.L. (1969), A discrete element analysis of elasto-plastic plates by energy minimization, Ph.D. thesis, Case Western Reserve University, Jan. 1969 Stark, R.M., R.L. Nicholls (1972), Mathematical foundations for design|civil engineering systems, McGraw-Hill, New York Stebbins, G.L. (1968), Evolutionsprozesse, G. Fischer, Stuttgart Stein, M.L. (1952), Gradient methods in the solution of systems of linear equations, NBS J. Research 48, 407-413

312

References

Steinbuch, K. (1971), Automat und Mensch, 4th ed., Springer, Berlin Steinbuch, K., S.W. Wagner (Eds.) (1964), Neuere Ergebnisse der Kybernetik, Oldenbourg, Munich Stender, J. (Ed.) (1993), Parallel genetic algorithms|theory and applications, IOS Press, Amsterdam Steuer, R.E. (1986), Multiple criteria optimization|theory, computation, and application, Wiley, New York Stewart, E.C., W.P. Kavanaugh, D.H. Brocker (1967), Study of a global search algorithm for optimal control, Proceedings of the Vth International Analogue Computation Meeting, Lausanne, Aug.-Sept. 1967, pp. 207-230 Stewart, G.W. (1967), A modication of Davidon's minimization method to accept difference approximations of derivatives, JACM 14, 72-83 Stewart, G.W. (1973), Conjugate direction methods for solving systems of linear equations, Numer. Math. 21, 285-297 Stiefel, E. (1952), Uber einige Methoden der Relaxationsrechnung, ZAMP 3, 1-33 Stiefel, E. (1965), Einfuhrung in die numerische Mathematik, 4th ed., Teubner, Stuttgart Stoer, J., C. Witzgall (1970), Convexity and optimization in nite dimensions I, Springer, Berlin Stolz, O. (1893), Grundzuge der Dierential- und Integralrechnung, erster Teil|reelle Veranderliche und Functionen, Abschnitt V|die gro!ten und kleinsten Werte der Functionen, pp. 199-258, Teubner, Leipzig, Germany Stone, H.S. (1973a), Parallel computation|an introduction, IEEE Trans. C-22, 709-710 Stone, H.S. (1973b), An e cient parallel algorithm for the solution of a tri-diagonal linear system of equations, JACM 20, 27-38 Storey, C. (1962), Applications of a hill climbing method of optimization, Chem. Engng. Sci. 17(1), 45-52 Storey, C., Rosenbrock (1964), On the computation of the optimal temperature prole in a tubular reaction vessel, in: Balakrishnan and Neustadt (1964), pp. 23-64 Stratonovich, R.L. (1968), Does there exist a theory of synthesis of optimal adaptive, self-learning and self-adaptive systems? ARC 29, 83-92 Stratonovich, R.L. (1970), Optimal algorithms of the stochastic approximation type, Engng. Cybern. 8, 20-27 Strongin, R.G. (1970), Multi-extremal minimization, ARC 31, 1085-1088

References

313

Strongin, R.G. (1971), Minimization of many-extremal functions of several variables, Engng. Cybern. 9, 1004-1010 Suchowitzki, S.I., L.I. Awdejewa (1969), Lineare und konvexe Programmierung, Oldenbourg, Munich Sugie, N. (1964), An extension of Fibonaccian searching to multi-dimensional cases, IEEE Trans. AC-9, 105 Sutti, C., L. Trabattoni, P. Brughiera (1972), A method for minimization of a onedimensional nonunimodal function, in: Szego (1972), pp. 181-192 Svechinskii, V.B. (1971), Random search in probabilistic iterative algorithms, ARC 32, 76-80 Swann, W.H. (1964), report on the development of a new direct searching method of optimization, ICI Central Instrument Laboratory, research note 64-3, Middlesborough, Yorks, June 1964 Swann, W.H. (1969), A survey of non-linear optimization techniques, FEBS-Letters 2(Suppl.), S39-S55 Swann, W.H. (1972), Direct search methods, in: Murray (1972a), pp. 13-28 Sweschnikow, A.A. (Ed.) (1970), Wahrscheinlichkeitsrechnung und mathematische Statistik in Aufgaben, Teubner, Leipzig, Germany Sydow, A. (1968), Eine Methode zur exakten Realisierung des Gradientenverfahrens auf dem iterativ-arbeitenden Analogrechner, messen-steuern-regeln 11, 462-465 Sydow, A. (Ed.) (1993), Simulationstechnik, 8th Symposium Simulationstechnik, Berlin, Sept. 1993, Vieweg, Braunschweig, Germany Synge, J.L. (1944), A geometrical interpretation of the relaxation method, Quart. Appl. Math. 2, 87-89 Szczerbicka, H., P. Ziegler (Eds.) (1993), 6th Workshop Simulation und Kunstliche Intelligenz, Karlsruhe, Germany, April 22-23, 1993, Mitteilungen aus den Arbeitskreisen der ASIM, Arbeitsgemeinschaft Simulation in der Gesellschaft fur Informatik (GI), Bonn Szego, G.P. (Ed.) (1972), Minimization algorithms, mathematical theories, and computer results, Academic Press, New York Szego, G.P., G. Treccani (1972), Axiomatization of minimization algorithms and a new conjugate gradient method, in: Szego (1972), pp. 193-216 Tabak, D. (1969), Comparative study of various minimization techniques used in mathematical programming, IEEE Trans. AC-14, 572

314

References

Tabak, D. (1970), Applications of mathematical programming techniques in optimal control|a survey, IEEE Trans. AC-15, 688-690 Talkin, A.I. (1964), The negative gradient method extended to the computer programming of simultaneous systems of dierential and nite equations, AFIPS Conf. Proc. 26, 539-543 Tapley, B.D., J.M. Lewallen (1967), Comparison of several numerical optimization methods, JOTA 1, 1-32 Taran, V.A. (1968a), A discrete adaptive system with random search for the optimum, Engng. Cybern. 6(4), 142-150 Taran, V.A. (1968b), Adaptive systems with random extremum search, ARC 29, 14471455 Tazaki, E., A. Shindo, T. Umeda (1970), Decentralized optimization of a chemical process by a feasible method, IFAC Kyoto Symposium on Systems Engineering Approach to Computer Control, Kyoto, Japan, Aug. 1970, paper 25.1 Thom, R. (1969), Topological models in biology, Topology 8, 313-336 Thomas, M.E., D.J. Wilde (1964), Feed-forward control of over-determined systems by stochastic relaxation, in: Blakemore and Davis (1964), pp. 16-22 Todd, J. (1949), The condition of certain matrices I, Quart. J. Mech. Appl. Math. 2, 469-472 Tokumaru, H., N. Adachi, K. Goto (1970), Davidon's method for minimization problems in Hilbert space with an application to control problems, SIAM J. Contr. 8, 163-178 Tolle, H. (1971), Optimierungsverfahren fur Variationsaufgaben mit gewohnlichen Dierentialgleichungen als Nebenbedingungen, Springer, Berlin Tomlin, F.K., L.B. Smith (1969), Remark on algorithm 178 (E4)|direct search, CACM 12, 637-638 Torn, A., A. Z% ilinskas (1989), Global optimization, vol. 350 of Lecture Notes in Computer Science, Springer, Berlin Tovstucha, T.I. (1960), The eect of random noise on the steady-state operation of a step-type extremal system for an object with a parabolic characteristic, ARC 21, 398-404 Traub, J.F. (1964), Iterative methods for the solution of equations, Prentice-Hall, Englewood Clis NJ Treccani, G., L. Trabattoni, G.P. Szego (1972), A numerical method for the isolation of minima, in: Szego (1972), pp. 239-289

References

315

Tsypkin, Ya.Z. see also under Zypkin, Ja.S. Tsypkin, Ya.Z. (1968a), All the same, does a theory of synthesis of optimal adaptive systems exist? ARC 29, 93-98

Tsypkin, Ya.Z. (1968b), Optimal hybrid adaptation and learning algorithms, ARC 29, 1271-1276 Tsypkin, Ya.Z. (1968c), Self-learning, what is it? IEEE Trans. AC-13, 608-612 Tsypkin, Ya.Z. (1970a), On learning systems, IFAC Kyoto Symposium on Systems Engineering Approach to Computer Control, Kyoto, Japan, Aug. 1970, paper 34.1 Tsypkin, Ya.Z. (1970b), Generalized learning algorithms, ARC 31, 86-92 Tsypkin, Ya.Z. (1971), Smoothed randomized functionals and algorithms in adaptation and learning theory, ARC 32, 1190-1209

Tsypkin, Ya.Z., A.S. Poznyak (1972), Finite learning automata, Engng. Cybern. 10, 478-490 Tzeng, G.-H., P.L. Yu (Eds.) (1992), Proceedings of the 10th International Conference on Multiple Criteria Decision Making, Taipei, July 19-24, 1992, National Chiao Tung University, Taipei, Taiwan Ueing, U. (1971), Zwei Losungsmethoden fur nichtkonvexe Programmierungsprobleme, Springer, Berlin Ueing, U. (1972), A combinatorial method to compute a global solution of certain nonconvex optimization problems, in: Lootsma (1972a), pp. 223-230 Unbehauen, H. (1971), On the parameter optimization of multivariable control systems, in: Schwarz (1971), technical papers, vol. 2, pp. 2.2.10-1 to 2.2.10-11 Vagin, V.N., L.Ye. Rudelson (1968), An example of a self-organizing system, Engng. Cybern. 6(6), 33-40 Vajda, S. (1961), Mathematical programming, Addison-Wesley, Reading MA Vajda, S. (1967), The mathematics of experimental design, Gri n, London Vanderplaats, G.N. (1984), Numerical optimization techniques for engineering design| with applications, McGraw-Hill, New York VanNorton, R. (1967), Losung linearer Gleichungssysteme nach dem Verfahren von Gauss-Seidel, in: Ralston and Wilf (1967), pp. 92-105 Varah, J.M. (1965), Certication of algorithm 203 (E4)|steep 1, CACM 8, 171

316

References

Varela, F.J., P. Bourgine (Eds.) (1992), Toward a practice of autonomous systems, Proceedings of the 1st European Conference on Articial Life (ECAL), Paris, Dec. 11-13, 1991, MIT Press, Cambridge MA Varga, J. (1974), Praktische Optimierung|Verfahren und Anwendungen der linearen und nichtlinearen Optimierung, Oldenbourg, Munich Varga, R.S. (1962), Matrix iterative analysis, Prentice-Hall, Englewood Clis NJ Vaysbord, E.M. (1967), Asymptotic estimates of the rate of convergence of random search, Engng. Cybern. 5(4), 22-32 Vaysbord, E.M. (1968), Convergence of a method of random search, Engng. Cybern. 6(3), 44-48 Vaysbord, E.M. (1969), Convergence of a certain method of random search for a global extremum of a random function, Engng. Cybern. 7(1), 46-50 Vaysbord, E.M., D.B. Yudin (1968), Multiextremal stochastic approximation, Engng. Cybern. 6(5), 1-11 Venter, J.H. (1967), An extension of the Robbins-Monro procedure, Ann. Math. Stat. 38, 181-190 Viswanathan, R., K.S. Narendra (1972), A note on the linear reinforcement scheme for variable-structure stochastic automata, IEEE Trans. SMC-2, 292-294 Vitale, P., G. Taylor (1968), A note on the application of Davidon's method to nonlinear regression problems, Technometrics 10, 843-849 Vogelsang, R. (1963), Die mathematische Theorie der Spiele, Dummler, Bonn Voigt, H.-M. (1989), Evolution and optimization|an introduction to solving complex problems by replicator networks, Akademie-Verlag, Berlin Voigt, H.-M., H. Muhlenbein, H.-P. Schwefel (Eds.) (1990), Evolution and optimization '89|Selected papers on evolution theory, combinatorial optimization and related topics, Wartburg Castle, Eisenach, April 2-4, 1989, Akademie-Verlag, Berlin Voltaire, F.M. Arouet de (1759), Candide oder der Optimismus, Insel Verlag, Frankfort/Main, 1973 Volz, R.A. (1965), The minimization of a function by weighted gradients, IEEE Proc. 53, 646-647 Volz, R.A. (1973), Example of function optimization via hybrid computation, Simulation 21, 43-48 Waddington, C.H. (Ed.) (1968), Towards a theoretical biology I|prolegomena, Edinburgh University Press, Edinburgh

References

317

Wald, A. (1966), Sequential analysis, 8th ed., Wiley, New York Wallack, P. (1964), Certication of algorithm 203 (E4)|steep 1, CACM 7, 585 Walsh, J. (Ed.) (1966), Numerical analysis|an introduction, Academic Press, London Ward, L., A. Nag, L.C.W. Dixon (1969), Hill-climbing techniques as a method of calculating the optical constants and thickness of a thin metallic lm, Brit. J. Appl. Phys. (J. Phys. D), Ser. 2, 2, 301-304 Wasan, M.T. (1969), Stochastic approximation, Cambridge University Press, Cambridge UK Wasscher, E.J. (1963a), Algorithm 203 (E4)|steep 1, CACM 6, 517-519

Wasscher, E.J. (1963b), Algorithm 204 (E4)|steep 2, CACM 6, 519 Wasscher, E.J. (1963c), Remark on algorithm 129 (E4)|minifun, CACM 6, 521

Wasscher, E.J. (1965), Remark on algorithm 205 (E4)|ative, CACM 8, 171 Weber, H.H. (1972), Einfuhrung in Operations Research, Akademische Verlagsgesellschaft, Frankfort/Main Wegge, L. (1966), On a discrete version of the Newton-Raphson method, SIAM J. Numer. Anal. 3, 134-142 Weinberg, F. (Ed.) (1968), Einfuhrung in die Methode Branch and Bound, Springer, Berlin Weinberg, F., C.A. Zehnder (Eds.) (1969), Heuristische Planungsmethoden, Springer, Berlin Weisman, J., C.F. Wood (1966), The use of optimal search for engineering design, in: Lavi and Vogl (1966), pp. 219-228 Weisman, J., C.F. Wood, L. Rivlin (1965), Optimal design of chemical process systems, AIChE Engineering Progress Symposium Series 61, no. 55, pp. 50-63

Weiss, E.A., D.H. Archer, D.A. Burt (1961), Computer sets tower for best run, Petrol Rener 40(10), 169-174

Wells, M. (1965), Algorithm 251 (E4)|function minimization (Flepomin), CACM 8, 169-170 Werner, J. (1974), Uber die Konvergenz des Davidon-Fletcher-Powell-Verfahrens fur streng konvexe Minimierungsaufgaben im Hilbert-Raum, Computing 12, 167-176 Wheeling, R.F. (1960), Optimizers|their structure, CACM 3, 632-638

318

References

White, L.J., R.G. Day (1971), An evaluation of adaptive step-size random search, IEEE Trans. AC-16, 475-478 White, R.C., Jr. (1970), Hybrid-computer optimization of systems with random parameters, Ph.D. thesis, University of Arizona, Tucson AZ, June 1970 White, R.C., Jr. (1971), A survey of random methods for parameter optimization, Simulation 17, 197-205 Whitley, L.D. (1991), Fundamental principles of deception in genetic search, in: Rawlins (1991), pp. 221-241 Whitley, L.D. (Ed.) (1993), Foundations of Genetic Algorithms 2, Morgan Kaufmann, San Mateo CA Whitley, V.W. (1962), Algorithm 129 (E4)|minifun, CACM 5, 550-551 Whittle, P. (1971), Optimization under constraints|theory and applications of nonlinear programming, Wiley-Interscience, London Wiener, N. (1963), Kybernetik|Regelung und Nachrichtenubertragung in Lebewesen und Maschine, Econ-Verlag, Dusseldorf, Germany Wiener, N., J.P. Schade (Eds.) (1965), Progress in biocybernetics, vol. 2, Elsevier, Amsterdam Wilde, D.J. (1963), Optimization by the method of contour tangents, AIChE J. 9(2), 186-190 Wilde, D.J. (1964), Optimum seeking methods, Prentice-Hall, Englewood Clis NJ Wilde, D.J. (1965), A multivariable dichotomous optimum-seeking method, IEEE Trans. AC-10, 85-87 Wilde, D.J. (1966), Objective function indistinguishability in unimodal optimization, in: Lavi and Vogl (1966), pp. 341-350 Wilde, D.J., C.S. Beightler (1967), Foundations of optimization, Prentice-Hall, Englewood Clis NJ Wilkinson, J.H. (1965), The algebraic eigenvalue problem, Oxford University Press, London Wilkinson, J.H., C. Reinsch (1971), Handbook for automatic computation, vol. 2| Linear algebra, Springer, Berlin Wilson, E.O., W.H. Bossert (1973), Einfuhrung in die Populationsbiologie, Springer, Berlin

References

319

Witt, U. (1992), Explaining process and change|approaches to evolutionary economics, University of Michigan Press, Ann Arbor MI Witte, B.F.W., W.R. Holst (1964), Two new direct minimum search procedures for functions of several variables, AFIPS Conf. Proc. 25, 195-209 Witten, I.H. (1972), Comments on \use of stochastic automata for parameter selfoptimization with multimodal performance criteria", IEEE Trans. SMC-2, 289-292 Wolf, G., T. Legendi, U. Schendel (Eds.) (1990), Parcella '90, Proceedings of the 5th International Workshop on Parallel Processing by Cellular Automata and Arrays, Berlin, Sept. 17-21, 1990, vol. 2 of Research in Informatics, Akademie-Verlag, Berlin Wolfe, P. (1959a), The simplex method for quadratic programming, Econometrica 27, 382-398 Wolfe, P. (1959b), The secant method for simultaneous nonlinear equations, CACM 2, 12-13 Wolfe, P. (1966), On the convergence of gradient methods under constraints, IBM Zurich, Switzerland, Research Laboratory report RZ-204, March 1966 Wolfe, P. (1967), Another variable metric method, IBM working paper Wolfe, P. (1969), Convergence conditions for ascent methods, SIAM Review 11, 226-235 Wolfe, P. (1970), Convergence theory in nonlinear programming, in: Abadie (1970), pp. 1-36 Wolfe, P. (1971), Convergence conditions for ascent methods II|some corrections, SIAM Review 13, 185-188 Wol, W., C.-J. Soeder, F.R. Drepper (Eds.) (1988), Ecodynamics|Contributions to theoretical ecology, Springer, Berlin Wood, C.F. (1960), Application of direct search to the solution of engineering problems, Westinghouse Research Laboratory, scientic paper 6-41210-1-P1, Pittsburgh PA, Oct. 1960 Wood, C.F. (1962), Recent developments in direct search techniques, Westinghouse Research Laboratory, research paper 62-159-522-Rl, Pittsburgh PA Wood, C.F. (1965), Review of design optimization techniques, IEEE Trans. SSC-1, 14-20 Yates, F. (1967), A fresh look at the basic principles of the design and analysis of experiments, in: LeCam and Neyman (1967), pp. 777-790

320

References

Youden, W.J., O. Kempthorne, J.W. Tukey, G.E.P. Box, J.S. Hunter, F.E. Satterthwaite, T.A. Budne (1959), Discussion of the papers of Messrs. Satterthwaite and Budne, Technometrics 1, 157-193 Yovits, M.C., S. Cameron (Eds.) (1960), Self-organizing systems, Pergamon Press, Oxford UK Yovits, M.C., G.T. Jacobi, D.G. Goldstein (Eds.) (1962), Self-organizing systems, Spartan, Washington, DC Yudin, D.B. (1965), Quantitative analysis of complex systems I, Engng. Cybern. 3(1), 1-9 Yudin, D.B. (1966), Quantitative analysis of complex systems II, Engng. Cybern. 4(1), 1-13 Yudin, D.B. (1972), New approaches to formalizing the choice of decisions in complex situations, ARC 33, 747-756 Yvon, J.P. (1972), On some random search methods, in: Szego (1972), pp. 313-335 Zach, F. (1974), Technisches Optimieren, Springer, Vienna Zadeh, L.A., L.W. Neustadt, A.V. Balakrishnan (Eds.) (1969a), Computing methods in optimization problems 2, Academic Press, London Zadeh, L.A., L.W. Neustadt, A.V. Balakrishnan (Eds.) (1969b), Computing methods in optimization problems, Springer, Berlin Zadeh, N. (1970), A note on the cyclic coordinate ascent method, Mgmt. Sci. 16, 642-644 Zahradnik, R.L. (1971), Theory and techniques of optimization for practicing engineers, Barnes and Noble, New York Zakharov, V.V. (1969), A random search method, Engng. Cybern. 7(2), 26-30 Zakharov, V.V. (1970), The method of integral smoothing in many-extremal and stochastic problems, Engng. Cybern. 8, 637-642 Zangwill, W.I. (1967), Minimizing a function without calculating derivatives, Comp. J. 10, 293-296 Zangwill, W.I. (1969), Nonlinear programming|a unied approach, Prentice-Hall, Englewood Clis NJ Zeleznik, F.J. (1968), Quasi-Newton methods for nonlinear equation, JACM 15, 265-271 Zellnik, H.E., N.E. Sondak, R.S. Davis (1962), Gradient search optimization, Chem. Engng. Progr. 58(8), 35-41

References

321

Zerbst, E.W. (1987), Bionik, Teubner, Stuttgart Zettl, G. (1970), Ein Verfahren zum Minimieren einer Funktion bei eingeschranktem Variationsbereich der Parameter, Numer. Math. 15, 415-432 Zhigljavsky, A.A. (1991), Theory of global random search, Kluwer, Dordrecht, The Netherlands Zigangirov, K.S. (1965), Optimal search in the presence of noise, Engng. Cybern. 3(4), 112-116 Zoutendijk, G. (1960), Methods of feasible directions|a study in linear and nonlinear programming, Elsevier, Amsterdam Zoutendijk, G. (1970), Nonlinear programming|computational methods, in: Abadie (1970), pp. 37-86 Zurmuhl, R. (1965), Praktische Mathematik fur Ingenieure und Physiker, 5th ed., Springer, Berlin Zwart, P.B. (1970), Nonlinear programming|a quadratic analysis of ridge paralysis, JOTA 6, 331-339 Zwart, P.B. (1973), Nonlinear programming|counterexample to two global optimization algorithms, Oper. Res. 21, 1260-1266 Zypkin, Ja.S. see also under Tsypkin, Ya.Z. Zypkin, Ja.S. (1966), Adaption und Lernen in automatischen Systemen, Oldenbourg, Munich Zypkin, Ja.S. (1967), Probleme der Adaption in automatischen Systemen, messen-steuernregeln 10, 362-365 Zypkin, Ja.S. (1970), Adaption und Lernen in kybernetischen Systemen, Oldenbourg, Munich

322

References Glossary of Abbreviations

AAAS ACM AEG AERE AFIPS AGARD AIAA AIChE AIEE ANL ARC

American Association for the Advancement of Science Association for Computing Machinery Allgemeine Elektricitats-Gesellschaft Atomic Energy Research Establishment American Federation of Information Processing Societies Advisory Group for Aerospace Research and Development American Institute of Aeronautics and Astronautics American Institute of Chemical Engineers American Institute of Electrical Engineers Argonne National Laboratory Automation and Remote Control (cover-to-cover translation of Avtomatika i Telemechanika) ASME American Society of Mechanical Engineers BIT Nordisk Tidskrift for Informationsbehandling CACM Communications of the ACM DFVLR Deutsche Forschungs- und Versuchsanstalt fur Luft{ und Raumfahrt DGRR Deutsche Gesellschaft fur Raketentechnik und Raumfahrt DLR Deutsche Luft- und Raumfahrt FEBS Federation of European Biochemical Societies GI Gesellschaft fur Informatik GMD Gesellschaft fur Mathematik und Datenverarbeitung IBM International Business Machines Corporation ICI Imperial Chemical Industries Limited IEE Institute of Electrical Engineers IEEE Institute of Electrical and Electronics Engineers Transactions AC on Automatic Control BME on Bio-Medical Engineering C on Computers MIL on Military Electronics MTT on Microwave Theory and Techniques NN on Neural Networks SMC on Systems, Man, and Cybernetics SSC on Systems Science and Cybernetics IFAC International Federation of Automatic Control IIASA International Institute for Applied Systems Analysis IMACS International Association for Mathematics and Computers in Simulation IRE Institute of Radio Engineers Transactions EC on Electronic Computers EM on Engineering Management ISA Instrument Society of America JACM Journal of the ACM

323 JIMA JOTA KFA KfK MIT NASA NBS NTZ PPSN SIAM UKAEA VDE VDI WGLR ZAMM ZAMP

Journal of the Institute of Mathematics and Its Applications Journal of Optimization Theory and Applications Kernforschungsanlage (Nuclear Research Center) Julich Kernforschungszentrum (Nuclear Research Center) Karlsruhe Massachusetts Institute of Technology National Aeronautics and Space Administration National Bureau of Standards Nachrichtentechnische Zeitschrift Parallel Problem Solving from Nature Society for Industrial and Applied Mathematics United Kingdom Atomic Energy Authority Verband Deutscher Elektrotechniker Verein Deutscher Ingenieure Wissenschaftliche Gesellschaft fur Luft- und Raumfahrt Zeitschrift fur angewandte Mathematik und Mechanik Zeitschrift fur angewandte Mathematik und Physik

324

References

Appendix A Catalogue of Problems The catalogue is divided into three groups of test problems corresponding to the three divisions of the numerical strategy comparison. The optimization problems are all formulated as minimum problems with a specied objective function F (x) and solution x
. For the second set of problems, the initial conditions x are also given. Occasionally, further local minima and other stationary points of the objective function are also indicated. Inequality constraints are formulated such that the constraint functions Gj (x) are all greater than zero within the allowed or feasible region. If a solution lies on the edge of the feasible region, then the active constraints are mentioned. The values of these constraint functions must be just equal to zero at the optimum. Where possible the structure of the minimum problem is depicted geometrically by means of a two dimensional contour diagram with lines F (x
x ) = const: and as a three dimensional picture in which values of F (x
x ) are plotted as elevation over the (x
x ) plane. Additionally, the values of the objective function on the contour lines are specied. Constraints are shown as bold lines in the contour diagrams. In the 3D plots the objective function is mostly oored to minimal values within non-feasible regions. In some cases there is a brief mention of any especially characteristic behavior shown by individual strategies during their iterative search for the minimum. (0)

1

1

2

2

1

2

A.1 Test Problems for the First Part of the Strategy Comparison Problem 1.1 (sphere model) Objective function: Minimum:

F (x) =

n X i=1

xi

2

x
i = 0
for i = 1(1)n
F (x
) = 0 For n = 2 a contour diagram as well as a 3D plot are sketched under Problem 2.17. For this, the simplest of all quadratic problems, none of the strategies fails. 325

326

Appendix A

Problem 1.2 Objective function:

0i 1 n X X F (x) = @ xj A

2

i=1 j =1

Minimum:

x
i = 0


for i = 1(1)n


F (x
) = 0

A contour diagram as well as a 3D plot for n = 2 are given under Problem 2.9. The objective function of this true quadratic minimum problem can be written in matrix notation as: F (x) = xT A x The n
n matrix of coecients A is symmetric and positive-denite. According to Schwarz, Rutishauser, and Stiefel (1968) its condition number K is a measure of the numerical diculty of the problem. Among other denitions, that of Todd (1949) is useful, namely: K = max = aamax min min 2

2

where

max = max fji j
i = 1(1)ng i

and similarly for min . The i are the eigenvalues of the matrix A, and the ai are the lengths of the semi-axes of an n-dimensional elliptic contour surface F (x) = const. Condition numbers for the present matrix

2n 66 n ; 1 66 n ; 2 66 A = (aij ) = 66 ... 66 n ; i + 1 : : : 66 . 4 .. 1

n;1 n;1 n;2 ... n ; i + 1 ::: ... 1

n;2 n;2 n;2 ... n;i+1 ... 1

::: n ; j + 1 ::: n ; j + 1 ::: n ; j + 1 ... ::: ... ::: 1

::: 1 3 : : : 1 777 : : : 1 77 77 7 : : : 1 777 75 ::: 1

were calculated for various values of n by means of an algorithm of Greenstadt (1967b), which uses the Jacobi method of diagonalization. As can be seen from the following table, K increases with the number of variables as O(n ). 2

Test Problems for the Second Part of the Strategy Comparison

n

K

327

K=n

2

1 1 1 2 6.85 1.71 3 16.4 1.82 6 64.9 1.80 10 175 1.75 20 678 1.69 30 1500 1.67 60 5930 1.65 100 16400 1.64 Not all the search methods achieved the required accuracy. For many variables the coordinate strategies and the complex method of Box terminated the search prematurely. Powell's method of conjugate gradients even got stuck without the termination criterion taking eect.

A.2 Test Problems for the Second Part of the Strategy Comparison Problem 2.1 after Beale (1958) Objective function:

h i h i F (x) = 1:5 ; x (1 ; x )] + 2:25 ; x (1 ; x ) + 2:625 ; x (1 ; x ) 1

2

2

1

2 2

Figure A.1: Graphical representation of Problem 2.1 F (x) = =0:1 1 4 ' 14:20 36 100=

2

1

3 2

2

328

Appendix A

Minimum:

x
= (3
0:5)
F (x
) = 0 Besides the strong minimum x
there is a weak minimum at innity: x0 ! (;1
1)
Saddle point: Start:

x00 = (0
1)


F (x0) ! 0 F (x00) ' 14:20

x = (0
0)
F (x ) ' 14:20 For very large initial step lengths the (1+1) evolution strategy converged once to the weak minimum x0. (0)

(0)

Problem 2.2 As Problem 2.1, but with: Start: x = (0:1
0:1)
(0)

Problem 2.3 Objective function:

F (x ) ' 12:99 (0)

q F (x) = ;jx sin( jxj)j

Figure A.2: Diagram F (x) for Problem 2.3

Test Problems for the Second Part of the Strategy Comparison

329

There are innitely many local minima, the position of which can be specied by a transcendental equation: q q jx
j = 2 tan ( jx
j) For jx
j  1 we have approximately

x
' ( (0:5 + k))


for k = 1
2
3
: : :
integer

2

and

F (x
) ' jx
j Whereas in reality none of the nite local minima is at the same time a global minimum, the nite word length of the digital computer used together with the system-specic method of evaluating the sine function give rise to an apparent global minimum at x
= 4:44487453  10

16

F (x
) = ;4:44487453  10 Counting from the origin it is the 67
108
864th local minimum in each direction. If x is increased above this value, the objective function value is always set to zero. (Note that this behavior is machine dependent.) Start: x = 0
F (x ) = 0 Most strategies located the rst or highest local minimum left or right of the starting point (the origin). Depending on the sequence of random numbers, the two membered evolution method found (for example) the 2nd, 9th, and 34th local minimum. Only the (10, 100) evolution strategy almost always reached the apparent global minimum. 16

(0)

Problem 2.4 Objective function:

F (x) = Minimum: Start:

n  X i=1

(0)



x ; xi + (xi ; 1]
1

x
i = 1


2

2

2

for i = 1(1)n


xi = 10


for i = 1(1)n


(0)

for n = 5

F (x
) = 0 F (x ) = 40905 (0)

Problem 2.5 after Booth (1949) Objective function:

F (x) = (x + 2 x ; 7) + (2 x + x ; 5) 1

2

2

1

2

2

330

Appendix A

Figure A.3: Graphical representation of Problem 2.4 for n = 2 F (x) = =100 101 102 103 104 105 106 107=

This minimum problem is equivalent to solving the following pair of linear equations:

x + 2x = 7 1

2

2x + x = 5 1

2

Figure A.4: Graphical representation of Problem 2.5 F (x) = =1 9 25 49 81 121 169 225=

Test Problems for the Second Part of the Strategy Comparison

331

An approach to the latter problem is to determine those values of x and x that minimize the error in the equations. The error is dened here in the sense of a Gaussian approximation as the sum of the squares of the components of the residual vector. Minimum: x
= (1
3)
F (x
) = 0 Start: x = (0
0)
F (x ) = 74 1

(0)

2

(0)

Problem 2.6 Objective function:

F (x) = maxfjx + 2 x ; 7j
j2 x + x ; 5jg 1

2

1

2

This represents an attempt to solve the previous system of linear equations of Problem 2.5 in the sense of a Tchebyche approximation. Accordingly, the error is dened as the absolute maximum component of the residual vector. Minimum: x
= (1
3)
F (x
) = 0 Start: x = (0
0)
F (x ) = 7 (0)

(0)

Figure A.5: Graphical representation of Problem 2.6 F (x) = =1 2 3 4 5 6 7 8 9 10 11=

332

Appendix A

Several of the search procedures were unable to nd the minimum. They converge to a point on the line x +x = 4, which joins together the sharpest corners of the rhombohedral contours. The partial derivatives of the objective function are discontinuous there in the unit vector directions, parallel to the coordinate axes, no improvement can be made. Besides the coordinate strategies, the methods of Hooke and Jeeves and of Powell are thwarted by this property. 1

2

Problem 2.7 after Box (1966) Objective function: X F (x) = (exp (;0:1 j x ) ; exp (;0:1 j x ) ; x exp (;0:1 j ) ; exp (;j )]) 10

1

j =1

2

2

3

Minima:

x
= (1
10
1)
F (x
) = 0 x
= (10
1
;1)
F (x
) = 0 Besides these two equivalent, strong minima there is a weak minimum along the line x0 = x0
1

2

x0 = 0
3

F (x0) = 0

Because of the nite computational accuracy the weak minimum is actually broadened into a region: x00 ' x00
x00 ' 0
F (x00) = 0
if x  1 1

2

3

1

Figure A.6: Graphical representation of Problem 2.7 on the plane x3 = 1 F (x) = =0:03 0:3 1 ' 3:064 10 30=

Test Problems for the Second Part of the Strategy Comparison

333

Figure A.7: Graphical representation of Problem 2.7 on the planes left: x3 = 0, right: x3 = ;1, F (x) = =0:03 0:3 1 ' 3:064 10 30=

Start:

x = (0
20
20)
F (x ) ' 1022 Many strategies only roughly located the rst of the strong minima dened above. The evolution strategies tended to converge to the weak minimum, since the minima are at equal values of the objective function. The second strong minimum, which is never referred to in the relevant literature, was sometimes found by the multimembered evolution strategy. (0)

Problem 2.8 As Problem 2.7, but with Start: Problem 2.9 Objective function:

Start:

(0)

x
i = 0
xi = 10
(0)

F (x ) ' 1031

x = (0
10
20)
F (x) =

Minimum:

(0)

(0)

n X i X ( xj )
2

i=1 j =1

for i = 1(1)n
for i = 1(1)n


for n = 5

F (x
) = 0 F (x ) = 5500 (0)

334

Appendix A

Figure A.8: Graphical representation of Problem 2.9 for n = 2 F (x) = =4 36 100 196 324 484=

Problem 2.10 after Kowalik (1967 see also Kowalik and Morrison, 1968) Objective function: ! X x ( b + b ix ) i F (x) = ai ; b + b x + x i i i Numerical values of the constants ai and bi for i = 1(1)11 can be taken from the following table: 11

=1

1

2

2

2

2

3

i

ai

b;i

1 2 3 4 5 6 7 8 9 10 11

0.1957 0.1947 0.1735 0.1600 0.0844 0.0627 0.0456 0.0342 0.0323 0.0235 0.0246

0.25 0.5 1 2 4 6 8 10 12 14 16

4

1

In this non-linear tting problem, formulated as a minimum problem, the free parameters

Test Problems for the Second Part of the Strategy Comparison

aj
j = 1(1)4 of a function

335

y(z) = z +(z +z + z) 1

2

2

2

3

4

have to be determined with reference to eleven data points fyi
zig such that the error, as measured by the Euclidean norm, is minimized (Gaussian or least squares approximation). Minimum:

x
' (0:1928
0:1908
0:1231
0:1358)
Start:

F (x
) ' 0:0003075

x = (0
0
0
0)
F (x ) ' 0:1484 Near the optimum, if the variables are changed in the last decimal place (with respect to the machine accuracy), rounding errors cause the objective function to behave almost stochastically. The multimembered evolution strategy with recombination yields the best solution. It deviates signicantly from the optimum solution as dened by Kowalik and Osborne (1968). Since this best value has a quasi-singular nature, it is repeatedly lost by the population of a (10 , 100) evolution strategy, with the result that the termination criterion of the search sometimes only takes eect after a long time if at all. (0)

(0)

Problem 2.11 As Problem 2.10, but with: Start: x = (0:25
0:39
0:415
0:39)


F (x ) ' 0:005316

Problem 2.12 As Problem 2.10, but with: Start: x = (0:25
0:40
0:40
0:40)


F (x ) ' 0:005566

(0)

(0)

(0)

(0)

Problem 2.13 after Fletcher and Powell (1963) Objective function: n X F (x) = (Ai ; Bi(x))
2

where

i=1

for n = 5

9 n Ai = P (aij sin j + bij cos j ) > = j for i = 1(1)n n P Bi(x) = (aij sin xj + bij cos xj ) >  =1

j =1

aij and bij are integer random numbers in the range ;100
100], and i are random numbers in the range ;
]. A minimum of this problem is simultaneously a solution of the equivalent system of n simultaneous non-linear (transcendental) equations:

336

Appendix A

Figure A.9: Graphical representation of Problem 2.13 for n = 2: a11 = ;2 a12 = 27 a21 = ;70 a22 = ;48 b11 = ;76 b12 = ;51 b21 = 63 b22 = ;50 1 = ;3:0882 2 = 2:0559 F (x) = =238:864 581:372 1403:11 3283:14 7153:45 13635:3 21479:6 27961:4 31831:7 33711:8 34533:5= n X j =1

(aij sin xj + bij cos xj ) = Ai


for i = 1(1)n

The solution is again approximated in the least squares sense. Minimum: x
i = i
for i = 1(1)n
F (x
) = 0 Because the trigonometric functions are multivalued there are innitely many equivalent minima (real solutions of the system of equations), of which up to 2n lie in the interval

i ;  xi i + 
Start:

for i = 1(1)n

xi = i + i
for i = 1(1)n where i are random numbers in the range ;=10
=10]. To provide the same conditions for all the search methods the same sequence of random numbers was used in each case, and hence F (x ) ' 1182 Because of the proximity of the starting point to the one solution, x
i = i for i = 1(1)n, all the strategies approached this minimum only. (0)

(0)

Test Problems for the Second Part of the Strategy Comparison

337

Problem 2.14 after Powell (1962) Objective function:

F (x) = (x + 10 x ) + 5 (x ; x ) + (x ; 2 x ) + 10 (x ; x ) 1

Minimum:

2

2

3

4

2

2

x
= (0
0
0
0)


Start:

4

3

1

4

4

F (x
) = 0

x = (3
;1
0
1)
F (x ) = 215 The matrix of second partial derivatives of the objective function goes singular at the minimum. Thus it is not surprising that a quasi-Newton method like the variable metric method of Davidon, Fletcher, and Powell (applied here in Stewart's derivative-free form) got stuck a long way from the minimum. Geometrically speaking, there is a valley which becomes extremely narrow as it approaches the minimum. The evolution strategies therefore ended up by converging very slowly with a minimum step length, and the search had to be terminated for reasons of time. (0)

Problem 2.15 As Problem 2.14, except: Start:

(0)

x = (1
2
3
4)
(0)

F (x ) = 1512 (0)

Problem 2.16 after Leon (1966a) Objective function: F (x) = 100 (x ; x ) + (x ; 1) 2

3 2 1

1

Figure A.10: Graphical representation of Problem 2.16 F (x) = =0:25 4 64 250 1000 5000 10000=

2

338

Appendix A

Minimum:

x
= (1
1)


Start:

x = (;1:2
1)
(0)

Problem 2.17 (sphere model) Objective function: Minimum: Start:

F (x) =

n X i=1

F (x
) = 0 F (x ) ' 749: (0)

xi


for n = 5

2

x
i = 0


for i = 1(1)n


F (x
) = 0

xi = 10


for i = 1(1)n


F (x ) = 500

(0)

(0)

Problem 2.18 after Matyas (1965) Objective function: F (x) = 0:26 (x + x ) ; 0:48 x x Minimum: x
= (0
0)
F (x
) = 0 Start: x = (15
30)
F (x ) = 76:5 2 1

(0)

2 2

1

2

(0)

Figure A.11: Graphical representation of Problem 2.17 for n = 2 F (x) = =4 16 36 64 100 144 196=

Test Problems for the Second Part of the Strategy Comparison

339

Figure A.12: Graphical representation of Problem 2.18 F (x) = =1 3 10 30 100 300=

The coordinate strategies terminated the search prematurely because of the lower bounds on the step lengths (as determined by the machine), which precluded making any more successful line searches in the coordinate directions. Problem 2.19 by Wood (after Colville, 1968) Objective function:   F (x) = 100 x ; x + (x ; 1) + 90 x ; x + (x ; 1) h i + 10:1 (x ; 1) + (x ; 1) + 19:8 (x ; 1)(x ; 1) Minimum: x
= (1
1
1
1)
F (x
) = 0 There is another stationary point near 1

1

2

2 2

2

2

2

3

2

x0 ' (1
;1
1
;1)


2 4

3

1

2

4

2

3

F (x0) ' 8

According to Himmelblau (1972a,b) there are still further local minima. Start: x = (;1
;3
;1
;3)
F (x ) = 19192 A very narrow valley appears to run from the stationary point x0 to the minimum. All the coordinate strategies together with the methods of Hooke and Jeeves and of Powell ended the search in this region. (0)

(0)

340

Appendix A

Problem 2.20 Objective function: Minimum: Start:

F (x) =

n X i=1

jxij


for n = 5

x
i = 0


for i = 1(1)n


F (x
) = 0

xi = 10


for i = 1(1)n


F (x ) = 50

(0)

(0)

Problem 2.21 Objective function:

F (x) = max fjxij
i = 1(1)ng
i Minimum: Start:

for n = 5

x
i = 0


for i = 1(1)n


F (x
) = 0

xi = 10


for i = 1(1)n


F (x ) = 10

(0)

(0)

Since the starting point is at a corner of the cubic contour surface, none of the coordinate strategies could nd a point with a lower value of the objective function. The method of

Figure A.13: Graphical representation of Problem 2.20 for n = 2 F (x) = =2 4 6 8 10 12 14 16 18 20=

Test Problems for the Second Part of the Strategy Comparison

341

Figure A.14: Graphical representation of Problem 2.21 for n = 2 F (x) = =2 4 6 8 10=

Powell also ended the search without making any signicant improvement on the initial condition. Both the simplex method of Nelder and Mead and the complex method of Box also had trouble in the minimum search in their cases the initially constructed simplex or complex collapsed long before reaching the minimum, again near one of the corners. Problem 2.22 Objective function:

F (x) =

Minimum: Start:

x
i = 0


n X i=1

n Y

jxij + jxij


for n = 5

for i = 1(1)n


F (x
) = 0

i=1

xi = 10
for i = 1(1)n
F (x ) = 100050 The simplex and complex methods did not nd the minimum. As in the previous Problem 2.21, this is due to the sharply pointed corners of the contours. The variable metric strategy also nally got stuck at one of these corners and converged no further. In this case the discontinuity in the partial derivatives of the objective function at the corners is to blame for its failure. (0)

(0)

342

Appendix A

Figure A.15: Graphical representation of Problem 2.22 for n = 2 F (x) = =3 8 15 24 35 48 63 80 99=

Problem 2.23 Objective function: Minimum:

F (x) = x
i = 0


n X i=1

xi
10

for n = 5

for i = 1(1)n


F (x
) = 0

Figure A.16: Graphical representation of Problem 2.23 for n = 2 F (x) = =210 410 610 810 1010=

Test Problems for the Second Part of the Strategy Comparison Start:

343

xi = 10
for i = 1(1)n
F (x ) = 5  10 Only the two strategies that have a quadratic internal model of the objective function, namely the variable metric and conjugate directions methods, failed to converge, because the function F (x) is of much higher (10th) order. (0)

(0)

10

Problem 2.24 after Rosenbrock (1960) Objective function: F (x) = 100 (x ; x ) + (x ; 1) Minimum: x
= (1
1)
F (x
) = 0 Start: x = (;1:2
1)
F (x ) = 24:2 2

2 2 1

(0)

Problem 2.25 Objective function:

F (x) = Minimum:

1

(0)

 n  X x ; xi + (xi ; 1)
i=2

x
i = 1


2

1

2

2

for i = 1(1)n


2

for n = 5

F (x
) = 0

Figure A.17: Graphical representation of Problem 2.24 F (x) = =0:5 4 20 100 250 500 1000 2000 5000=

344 Start:

Appendix A

xi = 10
for i = 1(1)n
F (x ) = 32724 For n = 2 this becomes nearly the same as Problem 2.24. (0)

(0)

Problem 2.26 Objective function:

q F (x) = ;x sin( jxj) This problem is the same as Problem 2.3 except for the modulus. The dierence has the eect that the neighboring minima are further apart here. The positions of the local minima and maxima are described under Problem 2.3. Start: x = 0
F (x ) = 0 Again, only the multimembered evolution strategy converged to the apparent global minimum all the other methods only converged to the rst (nearest) local minimum. (0)

(0)

Problem 2.27 after Zettl (1970) Objective function: F (x) = (x + x ; 2 x ) + 0:25 x Minimum: x
' (;0:02990
0)
F (x
) ' ;0:003791 2 1

2 2

1

2

1

Figure A.18: Diagram F (x) of Problem 2.26

Test Problems for the Second Part of the Strategy Comparison

345

Figure A.19: Graphical representation of Problem 2.27 F (x) = =0:03 0:3 1 3 10 30=

Because of rounding errors this same objective function value is reached for various pairs of values of x
x : Local maximum: x0 ' (1:063
0)
F (x0) ' 1:258 Saddle point: x00 ' (1:967
0)
F (x00) ' 0:4962 Start: x = (2
0)
F (x ) = 0:5 1

2

(0)

(0)

Problem 2.28 of Watson (after Kowalik and Osborne, 1968) Objective function: 0 2 3 1   X X j; X F (x) = B @ j ai xj ; 4 aji ; xj 5 ; 1CA + x 30

5

i=1 j =1

where

1

6

+1

j =1

;1 ai = i 29

2

1

2

2 1

The origin of this problem is the approximate solution of the ordinary dierential equation

dz ; z = 1 dy 2

346

Appendix A

on the interval 0 y 1 with the boundary condition z(y = 0) = 0. The function sought, z(y), is to be approximated by a polynomial n X z~(c
y) = cj yj; 1

j =1

In the present case only the rst six terms are considered. Suitable values of the polynomial coecients cj
j = 1(1)6, are to be determined. The deviation from the exact solution of the dierential equation is measured in the Gaussian sense as the sum of the squares of the errors at m = 30 argument values yi, uniformly distributed in the range 0,1] 1 0 m @ z~(c
y )   X F (c) = @ @y  ; z~ (c
y)y ; 1A i y The boundary condition is treated as a second simultaneous equation by means of a similarly constructed term:  F (c) = z~ (c
y) 2

2

1

=1

i

i

2

2

y=0

By inserting the polynomial and redening the parameters ci as variables xi we obtain the objective function F (x) = F (x) + F (x), the minimum of which is an approximate solution of the parameterized functional problem. Minimum: 1

2

x
' (;0:0158
1:012
;0:2329
1:260
;1:513
0:9928)
Start:

F (x
) ' 0:002288

x = (0
0
0
0
0
0)
F (x ) = 30 Judging by the number of objective function evaluations all the search methods found this a dicult problem to solve. The best solution was provided by the complex strategy. (0)

(0)

Problem 2.29 after Beale (1967) Objective function:

F (x) = 2 x + 2 x + x + 2 x x + 2 x x ; 8 x ; 6 x ; 4 x + 9 Constraints: Gj (x) = xj 0
for j = 1(1)3 G (x) = ;x ; x ; 2 x + 3 0 Minimum: 4 7 4
x = 3
9
9
F (x
) = 19
only G active
i.e., G (x
) = 0 Start: x = (0:1
0:1
0:1)
F (x ) = 7:29 2 1

2 2

4

2 3

1

1

2

1

2

3

1

3

3

4

(0)

2

(0)

4

Test Problems for the Second Part of the Strategy Comparison

347

Problem 2.30 As Problem 2.3, but with the constraints

G (x) = ;x + 300 0
1

G (x) = x + 300 0 2

The introduction of constraints gives rise to two equivalent, global minima at the edge of the feasible region: Minima: x
= 300
F (x
) ' ;299:7
G or G active In addition there are ve local minima within the feasible region. Here too, the absolute minima were only located by the multimembered evolution strategy. 1

2

Problem 2.31 As Problem 2.4, but with constraints:

Gj (x) = xj ; 1 0


for j = 1(1)n


n=5

Minimum:

x
i = 1
Start:

for i = 1(1)n


F (x
) = 0


all Gj active

xi = ;10
for i = 1(1)n
F (x ) = 61105 The starting point is located outside of the feasible region. (0)

(0)

Figure A.20: Diagram F (x) for Problem 2.30

348

Appendix A

Problem 2.32 after Bracken and McCormick (1970) Objective function: F (x) = ;x ; x Constraints: Gj (x) = xj 0
for j = 1
2 G (x) = ;x + 1 0
G (x) = ;x ; 4 x + 5 0 Minimum: x
= (1
1)
F (x
) = ;2
G and G active Besides this global minimum there is another local one:  5 0 x = 0
4
F (x0) = ; 25 16
G and G active Start: x = (0
0)
F (x ) = 0 All the search methods converged to the global minimum. 2 1

3

1

2 2

4

1

3

4

1

(0)

2

4

(0)

Problem 2.33 after Zettl (1970) As Problems 2.14 and 2.15, but with the constraints:

Gj (x) = xj ; 2 0
+2

for j = 1
2

Figure A.21: Graphical representation of Problem 2.32 F (x) = =0:04 0:16 0:36 0:64 1:0 1:44 1:96 2:56 3:24 4=

Test Problems for the Second Part of the Strategy Comparison

349

Minimum:

F (x
) ' 189:1


x
= (1:275
0:6348
2:0
2:0)
Start:

all Gj active

x = (1
2
3
4)
F (x ) = 1512 The (1+1) evolution strategy only solved the problem very inaccurately. Due to the 1=5 success rule the mutation variances vanish prematurely. (0)

(0)

Problem 2.34 after Fletcher and Powell (1963) Objective function: h i F (x) = 100 (x ; 10 ) + (R ; 1) + x 2

3

where

2

2 3

x = R cos (2  ) x = R sin (2  ) q R = x +x 1

2

or

2 1

Constraints:

8 arctan x2
>
x1 > > < =>
> > :  + arctan
1 2

if x 6= 0 and x > 0

1 2

if x = 0

1 2

2

x2 x1




if x 6= 0 and x < 0 2

1

G (x) = x + 2:5 0

3

x
= (1
' 0
0)


1

2

G (x) = ;x + 7:5 0
1

Minimum:

2 2

2

3

F (x
) = 0


no constraint is active

The objective function itself has a discontinuity at x = 0, right at the minimum sought. Thus x should only be allowed to approach closely to zero. Because of the multivalued trigonometric functions there are innitely many solutions to the problem, of which only one, however, lies within the feasible region. Start: x = (;1
0
0)
F (x ) = 2500 2

2

(0)

(0)

Problem 2.35 after Rosenbrock (1960) Objective function: F (x) = ;x x x 1

2

3

350

Appendix A

Constraints:

Gj (x) = xj 0
for j = 1(1)3 G (x) = ;x ; 2 x ; 2 x + 72 0 The underlying question here was: What dimension should a parcel of maximum volume have, if the sum of its length and transverse circumference is bounded? Minimum: x
= (24
12
12)
F (x
) = ;3456
G active Start: x = (0
0
0)
F (x ) = 0 All variants of the evolution strategy converged only to within the neighborhood of the minimum sought, because in the end only a fraction of all trials were feasible. 4

1

2

3

4

(0)

(0)

Problem 2.36 This is derived from Problem 2.35 by treating the constraint G , which is active at the minimum, as an equation, and thereby eliminating one of the free variables. With 4

x0 + 2 x0 + 2 x0 = 72 1

2

3

we obtain

F 0(x) = ;(72 ; 2 x0 ; 2 x0 ) x x or by renumbering of the variables a new objective function: 2

3

2

3

F (x) = ;x x (72 ; 2 x ; 2 x ) 1

2

1

2

Figure A.22: Graphical representation of Problem 2.36 F (x) = = ; 3400 ;3000 ;2000 ;1000 ;300 300 1000=

Test Problems for the Second Part of the Strategy Comparison Constraints:

Gj (x) = xj 0


Minimum:

x
= (12
12)
Start:

for j = 1
2

F (x
) = ;3456
x = (1
1)
(0)

no constraints are active

F (x ) = ;68 (0)

Problem 2.37 (corridor model) Objective function: n X F (x) = ; xi


for n = 3

i=1

Constraints: 8 > ;xj + 100 0
> > > > q j;n ;n < xj;n ; jP x + i j ;n i j ;n 0
Gj (x) = > > > > q j; n > : ;xj; n + j; n P xi + jj;; +1

2 +2

for j = 1(1)n for n + 1 j 2 n ; 1

+1

1

=1

1 2 +1

2 +1

i=1

2 2

n+2 n+1

0
for 2 n j 3 n ; 2

Figure A.23: Graphical representation of Problem 2.37 for n = 2 F (x) = = ; 220 ;215 ;210 ;205 ;200 ;195 ;190 ;185 ;180 ;175 ;170 ;165 ;160=

351

352

Appendix A

The constraints form a feasible region, which could be described as a corridor with a square cross section (three dimensionally speaking). The axis of the corridor runs along the diagonal in the space x = x = x = : : : = xn The contours of the linear objective function run perpendicular to this axis. In order to obtain a nite minimum further constraints were added, whereby a kind of pencil point is placed on the end of the corridor. In the absence of these additional constraints the problem corresponds to the corridor model used by Rechenberg (1973), for which he derived theoretically the rate of progress (a measure of the convergence rate) of the two membered evolution strategy. Minimum: x
i = 100
for i = 1(1)n
F (x
) = ;300
G to Gn active Start: xi = 0
for i = 1(1)n
F (x ) = 0 1

2

3

1

(0)

(0)

Problem 2.38 As Problem 2.25, but with the additional constraints:

Gj (x) = xj ; 1 0


Minimum:

x
i = 1
Start:

for j = 1(1)n


for i = 1(1)n


n=5

F (x
) = 0


xi = ;10
for i = 1(1)n
The starting point is not in the feasible region.

all Gj active

F (x ) = 48884

(0)

(0)

Problem 2.39 after Rosen and Suzuki (1965) Objective function: F (x) = x + x + 2 x + x ; 5 x ; 5 x ; 21 x + 7 x Constraints: G (x) = ;2 x ; x ; x ; 2 x + x + x + 5 0 G (x) = ;x ; x ; x ; x ; x + x ; x + x + 8 0 G (x) = ;x ; 2 x ; x ; 2 x + x + x + 10 0 2 1

2 2

2 1

1 2

3

Minimum:

2 3

2 1 2 1

2 4

2 2

2 2

2 3

2 3

2 2

x
= (0
1
2
;1)


1

1

2 4

2 3

2

2

1

2 4

3

4

4

2

1

F (x
) = ;44


3

4

4

G active 1

Test Problems for the Second Part of the Strategy Comparison Start:

353

x = (0
0
0
0)
F (x ) = 0 None of the search methods that operate directly with constraints, i.e., without reformulating the objective functions, managed to solve the problem to satisfactory accuracy. (0)

Problem 2.40 Objective function: Constraints:

(0)

F (x) = ;

X 5

i=1

xi

8 > for j = 1(1)5 > < xj 0
Gj (x) = > P > : ; (9 + i) xi + 50000 0
for j = 6 5

i=1

This is a simple linear programming problem. The solution is in a corner of the allowed region dened by the constraints (simplex). Minimum:

x
= (5000
0
0
0
0)


F (x
) = ;5000


G to G active 2

6

Figure A.24: Graphical representation of Problem 2.40 on the plane x3 = x4 = x5 = 0 F (x) = = ; 10500 ;9500 ;8500 ;7500 ;6500 ;5500 ;4500 ;3500 ;2500 ;1500 ;500 500=

354

Appendix A

Start:

x = (250
250
250
250
250)
F (x ) = ;1250 In terms of the values of the variables, none of the strategies tested achieved accuracies better than 10; . The two variants of the (10 , 100) evolution strategy came closest to the exact solution. (0)

(0)

2

Problem 2.41 Objective function: Constraints: Minimum: Start:

X F (x) = ; (i xi) 5

i=1

as for Problem 2.40

 50000
x = 0
0
0
0
14
F (x
) = ;250000 14 Gj active for j = 1
2
3
4
6

x = (250
250
250
250
250)
F (x ) = ;3750 This problem diers from the previous one only in the numerical values regarding the accuracies achieved, the same remarks apply as for Problem 2.40. (0)

(0)

Figure A.25: Graphical representation of Problem 2.41 on the plane x2 = x3 = x4 = 0 F (x) = = ; 30000 ;25000 ;20000 ;15000 ;10000 ;5000 0=

Test Problems for the Second Part of the Strategy Comparison Problem 2.42 Objective function:

F (x) = Constraints: Minimum: Start:

X 5

i=1

355

(i xi)

as for Problems 2.40 and 2.41

x
= (0
0
0
0
0)


F (x
) = 0


G to G active 1

5

x = (250
250
250
250
250)
F (x ) = 3750 The minimum is at the origin of coordinates. The evolution strategies were thus better able to approach the solution by adjusting the individual step lengths. The multimembered strategy with recombinations yielded an exact solution with variable values less than 10; . (0)

(0)

38

Problem 2.43 As Problem 2.42, except: Start: x = (;250
;250
;250
;250
;250)
The starting point is not in the feasible region. The solutions are the same as in Problem 2.42. (0)

F (x ) = ;3750 (0)

Figure A.26: Graphical representation of Problem 2.42 on the plane x3 = x4 = x5 = 0 F (x) = = ; 1000 1000 3000 5000 7000 9000 11000 13000 15000=

356

Appendix A

Problem 2.44 As Problem 2.26, but with additional constraints:

G (x) = ;x + 300 0


G (x) = x + 300 0

1

Minimum:

2

x
= ;300
F (x
) ' ;299:7
G active Besides this global minimum there are ve more local minima within the feasible region. Start: x = 0
F (x ) = 0 The global minimum could only be located by multimembered evolution. All the other search strategies converged to the local minimum nearest to the starting point. 2

(0)

(0)

Problem 2.45 of Smith and Rudd (after Leon, 1966a) Objective function: n X F (x) = xii e;x
for n = 5 i

Constraints:

i=1

8 > for j = 1(1)n < xj 0
Gj = > : 2 ; xj;n 0
for j = n + 1(1)2 n

Figure A.27: Diagram F (x) for Problem 2.44

Test Problems for the Second Part of the Strategy Comparison

357

Figure A.28: Graphical representation of Problem 2.45 for n = 2 F (x) = = ; 1:0 0:0 0:3 0:4 0:6 0:8 0:9=

Minimum: x
i = 0


for i = 1(1)n


F (x
) = 0
all G to Gn active Besides this global minimum there is another local one: x0i = (2
0
: : :
0)
F (x0) = 2 e;
G to Gn active Start: xi = 1
for i = 1(1)n
F (x ) ' 1:84 In the neighborhood of the minimum sought, the rate of convergence of a search strategy depends strongly on its ability to make widely dierent individual adjustments to the step lengths for the changes in the variables. The multimembered evolution solved this problem best when working with recombination. Rosenbrock's method converged to the local minimum, as did the complex method and the simple evolution strategies. 1

2

2

(0)

(0)

Problem 2.46 Objective function:

F (x) = x + x 2 1

Constraints:

Start:

2 2

G (x) = x + 2 x ; 2 0 1

Minimum:

+1

x
= (0:4
0:8)


1

F (x
) = 0:8


x = (10
10)
(0)

2

G active 1

F (x ) = 200 (0)

358

Appendix A

Figure A.29: Graphical representation of Problem 2.46 F (x) = =0:04 0:36 1:00 1:96 3:24 4:84 6:76=

Problem 2.47 after Ueing (1971) Objective function: Constraints:

Gj (x) G (x) G (x) G (x) G (x) G (x) 3 4 5 6

7

= = = = = =

F (x) = ;x ; x 2 1

2 2

xj 0
for j = 1
2 x + x ; 17 x ; 5 x + 66 0 x + x ; 10 x ; 10 x + 41 0 x + x ; 4 x ; 14 x + 45 0 ;x + 7 0 ;x + 7 0 2 1 2 1 2 1

2 2 2 2 2 2

1

2

1

2

1

2

1

2

Minimum:

x
= (6
0)
F (x
) = ;36
G and G active Besides the global minimum x
there are three other local minima: x0 ' (2:116
4:174)
F (x0) ' ;21:90 x00 = (0
5)
F (x00) = ;25 x000 = (5
2) , F (x000) = ;29 Start: x = (0
0)
F (x ) = 0 2

(0)

(0)

3

Test Problems for the Second Part of the Strategy Comparison

359

Figure A.30: Graphical representation of Problem 2.47 F (x) = = ; 4 ;16 ;36 ;64 ;100 ;144 ;196 ;256=

To the original problem have been added the two constraints G and G . Without them there are two separate feasible regions and the global minimum is at innity, in the external, open region. Depending on the initial step lengths, the evolution strategies were sometimes able to go out from the starting point within the inner, closed region into the external region. After adding G and G , the multimembered strategies converged to the global minimum, all other search methods located other local minima which of these was located by the two membered evolution strategy, depended on the sequence of random numbers. 6

6

Problem 2.48 after Ueing (1971) Objective function: Constraints:

Gj (x) G (x) G (x) G (x) 3 4

5

= = = =

7

7

F (x) = ;x ; x 2 1

2 2

xj 0
for j = 1
2 ;x + x + 4 0 x ;x +4 0 3 x + x ; 10 x ; 10 x + 41 0 1

1

2 1

2

2

2 2

1

2

Minimum:

x
= (12
8)
F (x
) = ;208
G and G active Besides this global minimum there are two more local minima: 3

x0 ' (2:018
4:673)


F (x0) ' ;25:91

4

360

Appendix A

Figure A.31: Graphical representation of Problem 2.48 F (x) = = ; 4 ;16 ;36 ;64 ;100 ;144 ;196 ;256=

Start:

x00 ' (6:293
2:293)


F (x00) ' ;44:86

x = (0
0)
F (x ) = 0 There are two feasible regions which are unconnected and closed. The starting point and the global minimum are separated by a non-feasible region. Only the (10 , 100) evolution strategy converged to the global minimum. It sometimes happened with this strategy that one descendant of a generation would jump from one feasible region to the other however, the group of remaining individuals would converge to one of the other local minima. All other strategies did not converge to the global minimum. (0)

(0)

Problem 2.49 after Wolfe (1966) Objective function: F (x) = 43 (x + x ; x x ) 34 + x Constraints: Gj (x) = xj 0
for j = 1(1)3 Minimum: x
= (0
0
0)
F (x
) = 0
all Gj active Start: x = (10
10
10)
F (x ) ' 52:16 2 1

(0)

2 2

1

2

(0)

3

Test Problems for the Third Part of the Strategy Comparison

361

Problem 2.50 As Problem 2.37, but with some other constraints:

Gj (x) = ;xj + 100 0
for j = 1(1)n 0 n 1 n 1X X Gn (x) = 1 ; @ n (xj ) ; xiA 0 i j 2

+1

=1

Minimum: x
i = 100
Start:

for i = 1(1)n


=1

F (x
) = ;300


for n = 3


G to Gn active 1

xi = 0
for i = 1(1)n
F (x ) = 0 Instead of the 2 n ; 2 linear constraints of Problem 2.37, a non-linear constraint served here to bound the corridor at its sides. From a geometrical point of view, the cross section of the corridor for n = 3 variables is now circular instead of square. For n = 2 variables the two problems become equivalent. (0)

(0)

A.3 Test Problems for the Third Part of the Strategy Comparison These are usually n-dimensional extensions of problems from the second set of tests, whose numbers are given in brackets after the new problem number. Problem 3.1 (analogous to Problem 2.4) Objective function: n h i X F (x) = (x ; xi ) + (1 ; xi) Minimum:

i=1

1

2 2

2

x
i = 1
for i = 1(1)n
F (x
) = 0 No noteworthy diculties arose in the solution of this and the following biquadratic problem with any of the comparison strategies. Away from the minimum, the contour patterns of the objective functions resemble those of the n-dimensional sphere problem (Problem 1.1). Nevertheless, the slight dierences caused most search methods to converge much more slowly (typically by a factor 1=5). The simplex strategy was particularly aected. The computation time it required were about 10 to 30 times as long as for the sphere problem with the same number of variables. With n = 100 and greater, the required accuracy was only achieved in Problem 3.1 after at least one collapse and subsequent reconstruction of the simplex. The evolution strategies on the other hand were all practically unaected by the dierence with respect to Problem 1.1. Also for the complex method the cost was only slightly higher, although with this strategy the computation time increased very rapidly with the number of variables for all problems.

362

Appendix A

Problem 3.2 (analogous to Problem 2.25) Objective function: n h i X F (x) = (x ; xi ) + (1 ; xi) Minimum:

i=2

x
i = 1


1

2 2

2

for i = 1(1)n


F (x
) = 0

Problem 3.3 (analogous to Problem 2.13) Objective function: 0n 1 n X n X X F (x) = @ (aij sin j + bij cos j ) ; (aij sin xj + bij cos xj )A i=1 j =1

2

j =1

where aij
bij for i
j = 1(1)n are integer random numbers from the range ;100
100], and j
j = 1(1)n are random numbers from the range ;
]. Minimum: x
i = i
for i = 1(1)n
F (x
) = 0 Besides this desired minimum there are numerous others that have the same value (see Problem 2.13). The aij and bij require storage space of order O(n ). For this reason the maximum number of variables for which this problem could be set up had to be limited to nmax = 30. The computation time per function call also increases as O(n ). The coordinate strategies ended the search for the minimum before reaching the required accuracy when 10 or more variables were involved. The method of Davies, Swann, and Campey (DSC) with Gram-Schmidt orthogonalization and the complex method failed in the same way for 30 variables. For n = 30 the search simplex of the Nelder-Mead strategy also collapsed prematurely, but after a restart the minimum was suciently well approximated. Depending on the sequence of random numbers, the two membered evolution strategy converged either to the desired minimum or to one of the others. This was not seen to occur with the multimembered strategies however, only one attempt could be made in each case because of the long computation times. 2

2

Problem 3.4 (analogous to Problem 2.20) Objective function: n X F (x) = jxij Minimum:

i=1

x
i = 0
for i = 1(1)n
F (x
) = 0 This problem presented no diculties to those strategies having a line (one dimensional) search subroutine, since the axes-parallel minimizations are always successful. The simplex method on the other hand required several restarts even for just 30 variables, and

Test Problems for the Third Part of the Strategy Comparison

363

for n = 100 variables it had to be interrupted, as it exceeded the maximum permitted computation time (8 hours) without achieving the required accuracy. The success or failure of the (1+1) evolution strategy and the complex method depended upon the actual random numbers. Therefore, in this and the following problems, whenever there was any doubt about convergence, several (at least three) attempts were made with dierent sequences of random numbers. It was seen that the two membered evolution strategy sometimes spent longer near one of the corners formed by the contours of the objective function, where it converged only slowly however, it nally escaped from this situation. Thus, although the computation times were very varied, the search was never terminated prematurely. The success of the multimembered evolution strategy depended on whether or not recombination was implemented. Without recombination the method sometimes failed for just 30 variables, whereas with recombination it converged safely and with no periods of stagnation. In the latter case the computation times taken were actually no longer than for the sphere problem with the same number of variables. Problem 3.5 (analogous to Problem 2.21) Objective function: F (x) = max fxi
i = 1(1)ng i Minimum: x
i = 0
for i = 1(1)n
F (x
) = 0 Most of the methods using a one dimensional search failed here, because the value of the objective function is piecewise constant along the coordinate directions. The methods of Rosenbrock and of Davies, Swann, and Campey (whatever the method of orthogonalization) converged safely, since they consider trial steps that do not change the objective function value as successful. If only true improvements are accepted, as in the conjugate gradient, variable metric, and coordinate strategies, the search never even leaves the chosen starting point at one of the corners of the contour surface. The simplex and complex strategies failed for n > 30 variables. Even for just 10 variables the search simplex of the Nelder-Mead method had to be constructed anew after collapsing 185 times, before the desired accuracy could be achieved. For the evolution strategy with only one parent and one descendant, the probability of nding from the starting point a point with a better value of the objective function is we = 2;n For this reason the (1+1) strategy failed for n 10. The multimembered version without recombination, could solve the problem for up to n = 10 variables. With recombination, convergence was sometimes still achieved for n = 30 variables, but no longer for n = 100 in the three attempts made.

364

Appendix A

Problem 3.6 (analogous to Problem 2.22) Objective function: n n X Y F (x) = jxij + jxij i=1

Minimum:

i=1

x
i = 0
for i = 1(1)n
F (x
) = 0 In spite of the even sharper corners on the contour surfaces of the objective function all the strategies behaved in much the same way as they did in the minimum search of Problem 3.4. The only notable dierence was with the (10 , 100) evolution strategy without recombination. For n = 30 variables the minimum search always converged only for n = 100 and above the search was no longer successful. Problem 3.7 (analogous to Problem 2.23) Objective function: n X F (x) = xi

10

i=1

Minimum:

x
i = 0
for i = 1(1)n
F (x
) = 0 The strategy of Powell failed for n 10 variables. Since all the step lengths were set to zero the search stagnated and the internal termination criterion did not take eect. The optimization had to be interrupted externally. From n = 30, the variable metric method was also ineective. The quadratic model of the objective function on which it is based led to completely false predictions of suitable search directions. For n = 10 the simplex method required 48 restarts, and for n = 30 as many as 181 in order to achieve the desired accuracy. None of the evolution strategies had any convergence diculties in solving the problem. They were not tested further for n > 300 simply for reasons of computation time. Problem 3.8 (similar to Problem 2.37) (corridor model) Objective function: n X F (x) = ; xi i=1

Constraints:

8q Pj x 0
> j +x > ; j i j i > < j Gj (x) = > q j ;n > > : jj;;nn ; xj;n + j;n P xi 0
+1

+1

+2 +1

1

=1

+2

+1

1

+1

i=1

for j = 1(1)n ; 1 for j = n(1)2 n ; 2

The other constraints of Problem 2.37, which bound the corridor in the direction of the minimum being sought, were omitted here. The minimum is thus at innity.

Test Problems for the Third Part of the Strategy Comparison

365

In comparing the results of this and the following circularly bounded corridor problem with the theoretical rates of progress for this model function, the quantity of interest was the cost, not of reaching a given approximation to an objective, but of covering a given distance along the corridor axis. For the half-width of the corridor, b = 1 was taken. The search was started at the origin and terminated as soon as a distance p s 10 b had been covered, or the objective function had reached a value F ;10 n: Start: xi = 0
for i = 1(1)n
F (x ) = 0 All the tested strategies converged satisfactorily. The number of mutations or generations required by the evolution strategies increased linearly with the number of variables, as expected. Since the number of constraints, as well as the computation time per function call, increased as O(n), the total computation time increased as O(n ). Because of the maximum of 8 hours per search adopted as a limit on the computation time, the two membered evolution strategy could only be tested to n = 300, and the multimembered strategies to n = 100. Intermediate results for n = 300, however, conrm that the expected trend is maintained. (0)

(0)

3

Problem 3.9 (similar to Problem 2.50) Objective function: n X F (x) = ; xi i=1

Constraint:

1 0 n n 1X X G(x) = 1 ; @ n (xj ) ; xiA 0 j i Minimum, starting point and convergence criterion as in Problem 3.8. The complex method failed for n 30, but the Rosenbrock strategy simply required more objective function evaluations and orthogonalizations compared to the rectangular corridor. The evolution strategies converged safely. They too required more mutations or generations than in the previous problem. However, since only one constraint instead of 2 n ; 2 was to be tested and respected, the time they took only increased as O(n ). Recombination in the multimembered version was only a very slight advantage for this and the linearly bounded corridor problem. 2

=1

=1

2

Problem 3.10 (analogous to Problem 2.45) Objective function: n X F (x) = xii e;x Constraints:

8 > < xj 0
Gj (x) = > : 2 ; xj;n 0


i

i=1

for j = 1(1)n for j = n + 1(1)2 n

366

Appendix A

Minimum:

x
i = 0


for i = 1(1)n


F (x
) = 0


all G to Gn active 1

Besides this global minimum there is a local one within the feasible region: ( 2
for i = 1 0 xi = F (x0) = 2 e; 0
for i = 2(1)n
2

As in the solution of Problem 2.45 with ve variables, the search methods only converged if they could adjust the step lengths individually. The strategy of Rosenbrock failed for only n = 10. The complex method sometimes converged for the same number of variables after about 1,000 seconds of computation time, but occasionally not even within the allotted 8 hours. For n = 30 variables, none of the strategies reached the objective before the time limit expired. The results obtained after 8 hours showed clearly that better progress was being made by the two membered evolution strategy and the multimembered strategy with recombination. The following table gives the best objective function values obtained by each of the strategies compared. Rosenbrock Complex (1 + 1) evolution (10
100) evolution without recombination (10
100) evolution with recombination

10; 10; 10; 10; 10;

4y 7 30 12

26

The Rosenbrock strategy ended the search prematurely after about 5 hours. All the other values are intermediate results after 8 hours of computation time when the strategy's own termination criteria were not yet satised. The searches could therefore still have come to a successful conclusion. y

Appendix B Program Codes This appendix contains the two FORTRAN programs EVOL and GRUP (with option REKO) used for the test series described in Chapter 6 plus the extension KORR as of 1976, which covers all features of GRUP/REKO as well as correlated mutations (Schwefel, 1974 see also Chap. 7) introduced shortly after the rst German version of this work was nished in 1974 (and reproduced as monograph by Birkh auser, Basle, Switzerland, in 1977). GRUP and REKO thus should no longer be used or imitated. B.1

(1+1) Evolution Strategy EVOL

1. Purpose The EVOL subroutine is a FORTRAN coding of the two membered evolution strategy. It is an iterative direct search strategy for a parameter optimization problem. A search is made for the minimum in a non-linear function of an arbitrary but nite number of continuously variable parameters. Derivatives of the objective function are not required. Constraints in the form of inequalities can be incorporated (right hand side
0). The user must supply initial values for the variables and for the appropriate step sizes. If the initial state is not feasible, a search is made for a feasible point by minimizing the sum of the negative values for the constraints that have been violated. 2. Subroutine parameter list EVOL (N,M,LF,LR,LS,TM,EA,EB,EC,ED,SN,FB,XB,SM,X,F,G,T,Z,R)

All parameters apart from LF, FB, X, and Z must be assigned values or names either before or when the subroutine is called. The variables XB and SM do not retain the values initially assigned to them. N (integer) Number of parameters (>0). M (integer) Number of constraints (
0). 367

368

Appendix B

LF (integer) Return code with the following meaning: LF=;2 Starting point not feasible and search for a feasible state unsuccessful. Feasible region probably empty. LF=;1 Starting point not feasible and search for a feasible state terminated because time limit was reached. LF=0 Starting point not feasible, search for a feasible state successful. The nal values of XB can be used as starting point for a subsequent search for a minimum if EVOL is called again. LF=1 Search for minimum terminated because time limit was reached. LF=2 Search for minimum terminated in an orderly fashion. No further improvement in the value of the objective function could be achieved in the context of the given accuracy parameters. Probably the nal state XB (extreme point) having FB (extreme value) lies near a local minimum, perhaps the global minimum. LR (integer) Auxiliary quantity used in step size management. Normal value 1.0. The step sizes are adjusted so that on average one success (improvement in the value of the objective function) is obtained in 5  LR trials (objective function calls). This is computed on the last 10  N  LR trials. LS (integer) Auxiliary quantity used in convergence testing. Minimum value 2.0. The search is terminated if the value of the objective function has improved by less than EC (absolute) or ED (relative) in the course of 10  N  LR  LS trials. Note: the step sizes are reduced by at most a factor SN10  LS during this period. The factor is 0:2LS if SN = 0.85 is selected. TM (real) Parameter used in controlling the computation time, e.g., the maximum CPU time in seconds, depending on the function designated T (see below). The search is terminated if T > TM. This check is performed after each N  LR mutations (objective function calls). EA (real) Lower bound to step sizes, absolute. EA > 0.0 must be chosen large enough to be treated as dierent from 0.0 within the accuracy of the computer used. EB (real) Lower bound to step sizes relative to values of variables. EB > 0.0 must be chosen large enough for 1:0 + EB to be treated as dierent from 1:0 within the accuracy of the computer used. EC (real) Parameter in convergence test, absolute. See under LS. (EC > 0.0, see EA). ED (real) Parameter in convergence test, relative. See under LS. (1:0 + ED > 1:0, see EB). Convergence is assumed if the data pass one or both tests. If it is desired to suppress a test, it is possible either to set EC = 0.0 or to choose a value for ED such that 1:0 + ED = 1.0 but ED > 0.0 within the accuracy of the machine. SN (real) Auxiliary variable for step size adjustment. Normal value 0.85. The step size can be kept constant during the trials by setting SN = 1:0. The success rate indicated by LR is used to adjust the step size by a factor SN or 1:0/SN after every N  LR trials.

(1 + 1) Evolution Strategy EVOL FB (real) XB (one dimensional real array of length N) SM (one dimensional real array of length N)

369

Best value of objective function obtained during the search. On call: holds initial values of variables. On exit: holds best values of variables corresponding to FB.

On call: holds initial values of step sizes (more precisely, standard deviations of components of the mutation vector). On exit: holds current step sizes of the last (not necessarily successful) mutation. Optimum initialization: somewhere near the local radius of curvature of the objective function hypersurface divided by the number of variables. More practical suggestion: SM(I) = DX(I)/SQRT(N), where DX(I) is a crude estimate of either the distance between start and expected optimum or the maximum uncertainty range for the variable X(I). If the SM(I) are initially set too large, a certain time elapses before they are appropriately adjusted. This is advantageous as regards the probability of locating the global optimum in the presence of several local optima. X (one dimensional Space for holding a variable vector. real array of length N) F (real function) Name of the objective function, which is to be provided by the user. G (real function) Name of the function used in calculating the values of the constraint functions to be provided by the user. T (real function) Name of the function used in controlling the computation time. Z (real function) Name of the function used in transforming a uniform random number distribution to a normal distribution. If the nomenclature Z is retained, the function Z appended to the EVOL subroutine can be used for this purpose. R (real function) Name of the function that generates a uniform random number distribution. 3. Method See I. Rechenberg, Evolution Strategy: Optimization of Technical Systems in Accordance with the Principles of Biological Evolution (in German), vol. 15 of Problemata series, Verlag Frommann-Holzboog, Stuttgart, 1973 also H.-P. Schwefel, Numerical Optimization of Computer Models, Wiley, Chichester, 1981 (translated by M. W. Finnis from Numerische Optimierung von Computer-Modellen mittels der Evolutionsstrategie, vol. 26 of Interdisciplinary Systems Research, Birkh auser, Basle, Switzerland, 1977). The method is based on a very much simpli ed simulation of biological evolution using the principles of mutation (random changes in variables, normal distribution for change vector) and selection (elimination of deteriorations and retention of improvements). The widths of the normal distribution (or step sizes) are controlled by reference to the ratio of the number of improvements to the number of mutations.

370

Appendix B

4. Convergence criterion Based on the change in the value of the objective function (see under LS, EC, and ED). 5. Peripheral I/O: none. 6. Notes If there are several (local) minima, only one is located. Which one actually is found depends on the initial values of variables and step sizes as well as on the random number sequence. In such cases it is recommended to repeat the search several times with dierent sets of initial values and/or random numbers. The approximation to the minimum is usually poor if the search terminates at the boundary of the feasible region de ned by the constraints. Better results can then be obtained by setting LR > 1, LS > 2, and/or SN > 0.85 (maximum value 1.0). In addition, the bounds EA and EB should not be made too small. The same applies if the objective function has discontinuous rst partial derivatives (e.g., in the case of Tchebyche approximation). 7. Subroutines or functions used The function names should be declared as external in the segment that calls EVOL. 7.1 Objective function This is to be written by the user in the form: ----------------------------------------------------FUNCTION F(N,X) DIMENSION X(N) ... F=... RETURN END -----------------------------------------------------

N represents the number of parameters, and X represents the formal parameter vector. The function should be written on the basis that EVOL searches for a minimum if a maximum is to be sought, F must be supplied with a negative sign. 7.2 Constraints function This is to be written by the user in the general style: ----------------------------------------------------FUNCTION G(J,N,X) DIMENSION X(N) GOTO(1,2,3,...,(M)),J 1 G=... RETURN 2 G=...

(1 + 1) Evolution Strategy EVOL

371

RETURN ... (M) G=... RETURN END -----------------------------------------------------

N and X have the meanings described for the objective function, while J (integer) is the serial number of the constraint. The statements should be written on the basis that EVOL will accept vector X as feasible if all the G values are larger than or equal to 0.0. 7.3 Function for controlling the computation time This may be de ned by the user or called from the subroutine library in the particular machine. The following structure is assumed: REAL FUNCTION T(D)

where D is a dummy parameter. T should be assigned the monitored quantity, e.g., the CPU time in seconds limited by TM. Many computers are supplied with ready-made timing software. If this is given as a function, only its name needs to be supplied to EVOL instead of T as a parameter. If it is a subroutine, the user can program the required function. For example, the subroutine might be called SECOND(I), where parameter I is an integer representing the CPU time in microseconds, in which case one could program: ----------------------------------------------------FUNCTION T(D) CALL SECOND(I) T=1.E-6*FLOAT(I) RETURN END -----------------------------------------------------

7.4 Function for transforming a uniformly distributed random number to a normally distributed one See under Section 8. 7.5 Function for generating a uniform random number distribution in the range (0,1] The structure must be REAL FUNCTION R(D)

where D is dummy. R is the value of the random number. Note: The smallest value of R must be large enough for the natural logarithm to be generated without oating-point overow. The standard library usually includes a suitable program, in which case only the appropriate name has to be supplied to EVOL.

372

Appendix B

8. Function Z(S,R) This function converts a uniform random number distribution to a normal distribution pairwise by means of the Box-Muller rules. The standard deviation is supplied as parameter S, while the expectation value for the mean is always 0.0. The quantity LZ is common to EVOL and Z by virtue of a COMMON block and acts as a switch to transmit only one of the two random numbers generated in response to each second call. --------------------------------------------------------SUBROUTINE EVOL(N,M,LF,LR,LS,TM,EA,EB,EC,ED,SN,FB, 1XB,SM,X,F,G,T,Z,R) DIMENSION XB(1),SM(1),X(1),L(10) COMMON/EVZ/LZ EXTERNAL R TN=TM+T(D) LZ=1 IF(M)4,4,1 1 LF=-1 C C FEASIBILITY CHECK C FB=0. DO 3 J=1,M FG=G(J,N,XB) IF(FG)2,3,3 2 FB=FB-FG 3 CONTINUE IF(FB)4,4,5 C C ALL CONSTRAINTS SATISFIED IF FB<=0 C 4 LF=1 FB=F(N,XB) C C INITIALIZATION OF SUCCESS COUNTER C 5 DO 6 K=1,10 6 L(K)=N*K/5 LE=N+N LM=0 LC=0 FC=FB C C MUTATION, START OF MAIN LOOP C

(1 + 1) Evolution Strategy EVOL 7 8 C C C 9

10 11

DO 8 I=1,N X(I)=XB(I)+Z(SM(I),R) IF(LF)9,9,12 AUXILIARY OBJECTIVE FF=0. DO 11 J=1,M FG=G(J,N,X) IF(FG)10,11,11 FF=FF-FG CONTINUE IF(FF)32,32,16

C C ALL CONSTRAINTS SATISFIED IF FF<=0 C 12 IF(M)15,15,13 13 DO 14 J=1,M IF(G(J,N,X))19,14,14 14 CONTINUE 15 FF=F(N,X) 16 IF(FF-FB)17,17,19 C C SUCCESS C 17 LE=LE+1 FB=FF DO 18 I=1,N 18 XB(I)=X(I) 19 LM=LM+1 IF(LM-N*LR)7,20,20 C C SUCCESS RATE CONTROL C 20 K=1 IF(LE-L(1)-N-N)23,22,21 21 K=K-1 22 K=K-1 C C ADAPTATION OF STEP SIZES C 23 DO 24 I=1,N 24 SM(I)=AMAX1(SM(I)*SN**K,ABS(XB(I))*EB,EA) DO 25 K=1,9

373

Appendix B

374 25

L(K)=L(K+1) L(10)=LE LM=0 LC=LC+1 IF(LC-10*LS)31,26,26

C C CONVERGENCE CRITERION C 26 IF(FC-FB-EC)28,28,27 27 IF((FC-FB)/ED-ABS(FC))28,28,30 28 LF=ISIGN(2,LF) 29 RETURN 30 LC=0 FC=FB C C TIME CONTROL C 31 IF(T(D)-TN)7,29,29 32 DO 33 I=1,N 33 XB(I)=X(I) FB=F(N,XB) LF=0 GOTO 29 END --------------------------------------------------------FUNCTION Z(S,R) COMMON/EVZ/LZ DATA ZP/6.28318531/ GOTO(1,2),LZ 1 A=SQRT(-2.*ALOG(R(D))) B=ZP*R(D) Z=S*A*SIN(B) LZ=2 RETURN 2 Z=S*A*COS(B) LZ=1 RETURN END ---------------------------------------------------------

( , ) Evolution Strategies GRUP and REKO B.2


, )

(

375

Evolution Strategies GRUP and REKO

1. Purpose The GRUP subroutine is a FORTRAN program to handle a multimembered (L,LL) evolution strategy with L parents and LL descendants per generation. It is an iterative direct search strategy for handling parameter optimization problems. A search is made for the minimum in a non-linear function of an arbitrary but nite number of continuously variable parameters. Derivatives of the objective function are not required. Constraints in the form of inequalities can be incorporated (right hand side
0). The user must supply initial values for the variables and for the appropriate step sizes. If the initial state is not feasible, a search is made for a feasible point that minimizes the sum of the negative values for the constraints that have been violated. 2. Subroutine parameter list GRUP (REKO,L,LL,N,M,LF,TM,EA,EB,EC,ED,SN,FA,FB,XB,SM,X,FK,XK,SK,F,G,T,Z,R)

All parameters apart from LF, FA, FB, X, FK, XK, SK, and Z must be assigned values or names before or when the subroutine is called. The variables XB and SM do not retain the values initially assigned to them. REKO (logical) Switch for alternative with/without recombination. REKO=.FALSE. No recombination. The step sizes retain the relationship initially assigned to them. REKO=.TRUE. Recombination occurs. The relationships between the step sizes alter during the search. L (integer) Number of parents (
1). This parameter should not be chosen too small if recombination is to occur. LL (integer) Number of descendants (>L). Recommended to choose a value
6  L. N (integer) Number of parameters (>0). M (integer) Number of constraints (
0). LF (integer) Return code with the following meanings: LF=;2 Starting point not feasible and search for a feasible state unsuccessful. Feasible region probably empty. LF=;1 Starting point not feasible and search for a feasible state terminated because time limit was reached. LF=0 Starting point not feasible, search for a feasible state successful. The nal values of XB can be used as starting point for the search for a minimum if GRUP is called again. LF=1 Search for minimum terminated because time limit was reached. LF=2 Search for minimum terminated in an orderly fashion.

376

TM

EA EB EC ED

SN

FA FB

Appendix B LF=2 (continued) No further improvement in the value of the objective function could be achieved in the context of the framework of the given accuracy parameters. Probably the nal state XB (extreme value) lies near a local minimum, perhaps the global minimum. (real) Parameter used in monitoring the computation time, e.g., the maximum CPU time in seconds, depending on the function designated T (see below). The search is terminated if T > TM. This check is performed after every generation = LL objective function calls. (real) Lower bound to step sizes, absolute. EA > 0.0 must be chosen large enough to be treated as dierent from 0.0 within the accuracy of the computer used. (real) Lower bound to step sizes relative to values of variables. EB > 0.0 must be chosen large enough for 1:0 + EB to be treated as dierent from 1.0 within the accuracy of the computer used. (real) Parameter in convergence test, absolute. The search is terminated if the dierence between the best and worst values of the objective function within a generation is less than or equal to EC (EC > 0.0, see EA). (real) Parameter in convergence test, relative. The search is terminated if the dierence between the best and worst values of the objective function within a generation is less than or equal to ED multiplied by the absolute value of the mean of the objective function as taken over all L parents in a generation (1.0 + ED > 1.0, see EB). Convergence is assumed if the data pass one or both tests. If it is desired to delete a test, it is possible either to set EC = 0.0 or to choose a value for ED such that 1.0 + ED = 1.0 but ED > 0.0 within the accuracy of the machine. (real) Auxiliary quantity used in step size adjustment. Normal value C/SQRT(N), with C > 0.0, e.g., C = 1.0 for L = 10 and LL = 100. C can be increased as LL increases, but it must be reduced as L increases. An approximation for L = 1 is LL proportional to SQRT(C)  EXP(C). (real) Current best objective function value for population. (real) Best value of objective function attained during the whole search. The minimum found may not be unique if FB diers from FA because: (1) there is a state with an even smaller value for the objective function (e.g., near a local minimum or even near the global minimum) that has been lost over the generations or (2) the minimum consists of several quasisingular peaks on account of the nite accuracy of the computer used. Usually, the dierence between FA and FB is larger in the rst case than in the second, if EC and ED have been assigned small values.

( , ) Evolution Strategies GRUP and REKO XB (one dimensional real array of length N) SM (one dimensional real array of length N)

377

On call: holds initial values of variables. On exit: holds best values of variables corresponding to FB.

On call: holds initial values of step sizes (more precisely, standard deviations of components of the mutation vector). On exit: holds current step sizes of the last (not necessarily successful) mutation. Optimum initialization: somewhere near the local radius of curvature of the objective function hypersurface divided by the number of variables. More practical suggestion: SM(I) = DX(I)/SQRT(N), where DX(I) is a crude estimate of either the distance between start and expected optimum or the maximum uncertainty range for the variable X(I). If the SM(I) are initially set too large, it may happen that a good starting point is lost in the rst generation. This is advantageous as regards the probability of locating the global optimum in the presence of several local optima. X (one dimensional Space for holding a variable vector. real array of length N) FK (one dimensional Holds objective function values for the L best individuals in real array of each of the last two generations. length 2  L) XK (one dimensional Holds the variable values for N components for each of the L real array of parents in each of the last two generations. XK(1) to XK(N) length 2  L  N) hold the state vector X for the rst individual, the next N locations do the same for the second, and so on. SK (one dimensional Holds the standard deviations, structure as for XK. real array of length 2  L  N) F (real function) Name of the objective function, which is to be programmed by the user. G (real function) Name of the function for calculating the values of the constraints, to be programmed by the user. T (real function) Name of function used in monitoring the computation time. Z (real function) Name of function used in transforming a uniform random number distribution to a normal distribution. If the name Z is retained, the function Z listed after the GRUP subroutine can be used for this purpose. R (real function) Name of the function that generates a uniform random number distribution. 3. Method GRUP has been developed from EVOL. The method is based on a very much simpli ed simulation of biological evolution. See I. Rechenberg, Evolution Strategy: Optimization of Technical Systems in Accordance with the Principles of Biological Evolution (in Ger-

378

Appendix B

man), vol. 15 of Problemata series, Verlag Frommann-Holzboog, Stuttgart, 1973 also H.-P. Schwefel, Numerical Optimization of Computer Models, Wiley, Chichester, 1981 (translated by M. W. Finnis from Numerische Optimierung von Computer-Modellen mittels der Evolutionsstrategie, vol. 26 of Interdisciplinary Systems Research, Birkh auser, Basle, Switzerland, 1977). The current L parameter vectors are used to generate LL new ones by means of small random changes. The best L of these become the initial ones for the next generation (iteration). At the same time, the step sizes (standard deviations) for the changes in the variables (strategy parameters) are altered. The selection leads to adaptation to the local topology if LL/L is assigned a suitably large value, e.g., >6. The random changes in the parameters are produced by the addition of normally distributed random numbers, while those in the step sizes are produced from random numbers with a log-normal distribution by multiplication. 4. Convergence criterion Based on the dierences in value of the objective function (see under EC and ED). 5. Peripheral I/O: none. 6. Notes The multimembered strategy represents an improvement in reliability over the two membered strategy. On the other hand, the run time is greater when an ordinary (serial) digital computer is used. The run time increases less rapidly than in proportion to LL (the number of descendants per generation), because increasing LL increases the convergence rate (over the generations). However, minima at a boundary of the feasible region or at a vertex are attained only slowly or inexactly. In any case, although the certainty of global convergence cannot be guaranteed, numerical tests have shown that the multimembered strategy is far better than other search procedures in this respect. It is capable of handling separated feasible regions provided that the number of parameters is not large and that the initial step sizes are set suitably large. In doubtful cases it is recommended to repeat the search each time with a dierent set of initial values and/or random numbers. If the optimum being sought lies at a boundary of the feasible region, it is probably better to choose a value for SN (the parameter governing the rates of change of the standard deviations) less than the (maximal) value suggested above. 7. Subroutines or functions used The function names are to be declared as external in the segment that calls GRUP. 7.1 Objective function To be written by the user in the form:

( , ) Evolution Strategies GRUP and REKO

379

----------------------------------------------------FUNCTION F(N,X) DIMENSION X(N) ... ... F=... RETURN END -----------------------------------------------------

N represents the number of parameters, and X represents the formal parameter vector. GRUP supplies the actual values. The function should be written on the basis that GRUP searches for a minimum if a maximum is to be sought, F must be supplied with a negative sign. 7.2 Constraints function To be written by the user in the general style: ----------------------------------------------------FUNCTION G(J,N,X) DIMENSION X(N) GOTO(1,2,3,...,(M)),J 1 G=... RETURN 2 G=... RETURN ... ... (M) G=... RETURN END -----------------------------------------------------

N and X have the meanings described for the objective function, while J (integer) is the serial number of the constraint. The statements should be written on the basis that GRUP will accept vector X as feasible if all the G values are larger than or equal to zero. 7.3 Function for monitoring the computation time This may be de ned by the user or called from the subroutine library in the particular machine. The following structure is assumed: REAL FUNCTION T(D)

where D is a dummy parameter. T should be assigned the monitored quantity, e.g., the CPU time in seconds limited by TM. Many computers are supplied with ready-made timing software. If this is given as a function only its name needs to be supplied to GRUP,

380

Appendix B

instead of T, as a parameter. If it is a subroutine, the user can program the required function. For example, the subroutine might be called SECOND(I), where parameter I is an integer representing the CPU time in microseconds, in which case one could program: ----------------------------------------------------FUNCTION T(D) CALL SECOND(I) T=1.E-6*FLOAT(I) RETURN END -----------------------------------------------------

7.4 Function for transforming a uniformly distributed random number to a normally distributed one See under 8. 7.5 Function for generating a uniform random number distribution in the range (0,1] The structure must be REAL FUNCTION R(D)

where D is dummy. R is the value of the random number. Note: The smallest value of R must be large enough for the natural logarithm to be generated without oating-point overow. The standard library usually includes a suitable program, in which case only the appropriate name has to be supplied to GRUP. 8. Function Z(S,R) This function converts a uniform random number distribution to a normal distribution pairwise by means of the Box-Muller rules. The standard deviation is supplied as parameter S, while the expectation value for the mean is always zero. The quantity LZ is common to GRUP and Z by virtue of a COMMON block and acts as a switch to transmit only one of the two random numbers generated in response to each second call. --------------------------------------------------------SUBROUTINE GRUP(REKO,L,LL,N,M,LF,TM,EA,EB,EC,ED,SN, 1FA,FB,XB,SM,X,FK,XK,SK,F,G,T,Z,R) LOGICAL REKO DIMENSION XB(1),SM(1),X(1),FK(1),XK(1),SK(1) COMMON/GRZ/LZ EXTERNAL R KK(RR)=(LA+IFIX(FLOAT(L)*RR))*N C C THE PRECEDING LINE CONTAINS A STATEMENT FUNCTION C TN=TM+T(D)

( , ) Evolution Strategies GRUP and REKO

1 C C C

2 3 C C C 4 5 6

C C C 7 8 9

10 11 12 13 14

LZ=1 IF(M)4,4,1 LF=-1 FEASIBILITY CHECK FB=0. DO 3 J=1,M FG=G(J,N,XB) IF(FG)2,3,3 FB=FB-FG CONTINUE IF(FB)4,4,5 ALL CONSTRAINTS SATISFIED IF FB<=0 LF=1 FB=F(N,XB) DO 6 I=1,N SK(I)=AMAX1(SM(I),ABS(XB(I))*EB,EA) XK(I)=XB(I) FK(1)=FB KA=N KB=0 SETUP OF POPULATION DO 21 K=2,L SA=1. DO 8 I=1,N X(I)=XB(I)+Z(SM(I)*SA,R) IF(LF)9,9,12 FF=0. DO 11 J=1,M FG=G(J,N,X) IF(FG)10,11,11 FF=FF-FG CONTINUE IF(FF)60,60,17 IF(M)16,16,13 DO 15 J=1,M IF(G(J,N,X))14,15,15 SA=SA*.5 GOTO 7

381

382 15 CONTINUE 16 FF=F(N,X) 17 IF(FF-FB)18,19,19 C C STORING OF BEST INTERMEDIATE RESULT C 18 FB=FF KB=K 19 DO 20 I=1,N KA=KA+1 SK(KA)=AMAX1(SM(I)*SA,ABS(X(I))*EB,EA) 20 XK(KA)=X(I) 21 FK(K)=FF IF(KB)24,24,22 22 KB=(KB-1)*N DO 23 I=1,N 23 XB(I)=XK(KB+I) C C START OF MAIN LOOP C 24 LA=L LB=0 C C LA AND LB FORM A ROTATING COUNTER TO AVOID SHUFFLING C GENOTYPES WITHIN THE ARRAYS CONTAINING PARENTS AND C DESCENDANTS C 25 LC=LB LB=LA LA=LC LC=0 LD=0 26 SA=EXP(Z(SN,R)) C C LOG-NORMAL STEP SIZE FACTOR C IF(REKO)GOTO 28 KI=KK(R(D)) DO 27 I=1,N KI=KI+1 SM(I)=SK(KI)*SA 27 X(I)=XK(KI)+Z(SM(I),R) C C MUTATION WITHOUT RECOMBINATION ABOVE

Appendix B

( , ) Evolution Strategies GRUP and REKO C

GOTO 30 SA=SA*.5

28 C C MUTATION WITH RECOMBINATION BELOW C DO 29 I=1,N SM(I)=(SK(KK(R(D))+I)+SK(KK(R(D))+I))*SA 29 X(I)=XK(KK(R(D))+I)+Z(SM(I),R) 30 IF(LF)31,31,34 C C AUXILIARY OBJECTIVE C 31 FF=0. DO 33 J=1,M FG=G(J,N,X) IF(FG)32,33,33 32 FF=FF-FG 33 CONTINUE IF(FF)60,60,38 C C ALL CONSTRAINTS SATISFIED IF FF<=0 C 34 IF(M)37,37,35 35 DO 36 J=1,M IF(G(J,N,X))46,36,36 36 CONTINUE 37 FF=F(N,X) C C LD COUNTS THE NUMBER OF DESCENDANTS PRODUCED C 38 LD=LD+1 IF(LD-L)40,40,39 C C WHEREAS THE FIRST L DESCENDANTS ARE STORED AUTOMATICALLY C FURTHER DESCENDANTS REPLACE THE CURRENT WORST IN CASE OF C BEING BETTER ONLY C 39 IF(FF-FS)41,41,46 40 KS=LB+LD 41 FK(KS)=FF KS=(KS-1)*N DO 42 I=1,N KS=KS+1

383

Appendix B

384 42

SK(KS)=AMAX1(SM(I),ABS(X(I))*EB,EA) XK(KS)=X(I) IF(LD-L)46,43,43

C C DETERMINING THE CURRENT WORST C 43 KS=LB+1 FS=FK(KS) DO 45 K=2,L KA=LB+K FF=FK(KA) IF(FF-FS)45,45,44 44 FS=FF KS=KA 45 CONTINUE 46 LC=LC+1 IF(LC-LL)26,47,47 47 IF(LD-L)26,48,48 C C END OF A GENERATION C 48 KA=LB+1 FA=FK(KA) FC=FA C C DETERMINING THE CURRENT BEST AND SUM C DO 50 K=2,L KB=LB+K FF=FK(KB) FC=FC+FF IF(FF-FA)49,50,50 49 FA=FF KA=KB 50 CONTINUE IF(FA-FB)51,51,53 C C DETERMINING WHETHER THE CURRENT BEST IS BETTER THAN C THE SO FAR OVERALL BEST C 51 FB=FA KB=(KA-1)*N DO 52 I=1,N 52 XB(I)=XK(KB+I)

( , ) Evolution Strategies GRUP and REKO C C CONVERGENCE CRITERION C 53 IF(FS-FA-EC)55,55,54 54 IF((FS-FA)*FLOAT(L)/ED-ABS(FC))55,55,59 55 LF=ISIGN(2,LF) 56 KB=(KA-1)*N DO 57 I=1,N 57 X(I)=XK(KB+I) 58 RETURN C C TIME CONTROL C 59 IF(T(D)-TN)25,56,56 60 DO 61 I=1,N 61 XB(I)=X(I) FB=F(N,XB) FA=FB LF=0 GOTO 58 END --------------------------------------------------------FUNCTION Z(S,R) COMMON/GRZ/LZ DATA ZP/6.28318531/ GOTO(1,2),LZ 1 A=SQRT(-2.*ALOG(R(D))) B=ZP*R(D) Z=S*A*SIN(B) LZ=2 RETURN 2 Z=S*A*COS(B) LZ=1 RETURN END ---------------------------------------------------------

385

Appendix B

386 B.3

(
+ )

Evolution Strategy KORR

Plus additional subroutines: PRUEFG, SPEICH, MUTATI, DREHNG, UMSPEI, MINMAX, GNPOOL, ABSCHA. and functions: ZULASS, GAUSSN, BLETAL. 1. Purpose The KORR subroutine is a FORTRAN coding of a multimembered evolution strategy. It is an iterative direct search strategy for a parameter optimization problem. A search is made for the minimum in a non-linear function of an arbitrary but nite number of continuously variable parameters. Derivatives of the objective function are not required. Constraints in the form of inequalities can be incorporated (right hand side
0). The user must supply initial values for the variables and for the appropriate step sizes. If the initial state is not feasible, a search is made for a feasible point by minimizing the sum of the negative values for the constraints that have been violated. 2. Parameter list for subroutine KORR KORR (IELTER, BKOMMA, NACHKO, IREKOM, BKORRL, KONVKR, IFALLK, TGRENZ, EPSILO, DELTAS, DELTAI, DELTAP, N, M, NS, NP, NY, ZSTERN, XSTERN, ZBEST, X, S, P, Y, ZIELFU, RESTRI, GAUSSN, GLEICH, TKONTR, KANAL)

All parameters apart from IFALLK, ZSTERN, ZBEST, X, and Y must be assigned values or names before or when the subroutine is called. The variables XSTERN, S, and P do not retain the values initially assigned to them. IELTER

(integer)

Number of parents of a generation. IELTER
1 if IREKOM = 111 IELTER > 1 IF IREKOM > 111 BKOMMA (logical) Switch for comma or plus version. BKOMMA=.FALSE. Selection criterion applied to parents and descendants (IELTER + NACHKO) evolution strategy. BKOMMA=.TRUE. Selection criterion applied only to descendants

(IELTER, NACHKO) evolution strategy. NACHKO (integer) Number of descendants in a generation. NACHKO
1 if BKOMMA = .FALSE. NACHKO > IELTER if BKOMMA = .TRUE. IREKOM (integer) Switch for recombination type consisting of three digits each of which has values between 1 and 5. The rst digit applies to the object variables X, the second one to the step sizes S, and the third one to the correlation angles P. Thus 111  IREKOM  555. Each digit controls the recombination in the following way:

( + ) Evolution Strategy KORR 1 2 3 4 5 BKORRL (logical)

387

No recombination Discrete recombination of pairs of parents Intermediary recombination of pairs of parents Discrete recombination of all parents Intermediary recombination of all parents in pairs Switch for variability of the mutation hyperellipsoid (locus of equal probability density). BKORRL=.FALSE. The ellipsoid cannot rotate. BKORRL=.TRUE. The ellipsoid can extend and rotate. KONVKR (integer) Switch for the convergence criterion: KONVKR = 1 The dierence in the objective function values between the best and worst parents at the start of each generation is used to determine whether to terminate the search before the time limit is reached. It is assumed that IELTER > 1. KONVKR > 1 (best
2  N): The change in the mean of all the parental objective function values in KONVKR generations is used as the search termination criterion. In both cases EPSILO(3) serves as the absolute and EPSILO(4) as the relative bound for deciding to terminate the search. IFALLK (integer) Return code with the following meaning: IFALLK = ;2 Starting point not feasible, search terminated on nding a minimal value of the auxiliary objective function without satisfying all the constraints. IFALLK = ;1 Starting point not feasible, search for a feasible parameter vector terminated because time limit was reached. IFALLK = 0 Starting point not feasible, search for a feasible XSTERN vector successful, search for a minimum can be restarted with this. IFALLK = 1 Search for a minimum terminated because time limit was reached. IFALLK = 2 Search for minimum terminated regularly. The convergence criterion was satis ed. IFALLK = 3 As for IFALLK = 1, but time limit reached not at the end of a generation but in an attempt to generate NACHKO viable descendants. TGRENZ (real) Parameter used in monitoring the computation time, e.g., the maximum CPU time in seconds. Search terminated at the latest at the end of the generation for which TKONTR
TGRENZ.

Appendix B

388 EPSILO (one dimensional real array of length 4) EPSILO(1) EPSILO(2) EPSILO(3) EPSILO(4) DELTAS (real)

DELTAI (real)

DELTAP (real)

N M NS

(integer) (integer) (integer)

Holds parameters that aect the attainable accuracy of approximation. The lowest possible values are machinedependent. Lower bound to step sizes, absolute. Lower bound to step sizes relative to values of variables (not implemented in this program). Limit to absolute value of objective function dierences for convergence test. As EPSILO(3) but relative. Factor used in step-size change. All standard deviations (=step sizes) S(I) are multiplied by a common random number EXP(GAUSSN(DELTAS)), where GAUSSN(DELTAS) is a normally distributed random number with zero mean and standard deviation DELTAS. EXP(DELTAS)
1.0. As for DELTAS, but each S(I) is multiplied by its own random factor EXP(GAUSSN(DELTAI)). EXP(DELTAI)
1.0. The S(I) retain their initial values if DELTAS = 0.0 and DELTAI = 0.0. The variables can be scaled only by recombination (IREKOM > 1) if DELTAI = 0.0. The following rules are suggested to provide the most rapid convergence for sphere models: DELTAS = C/SQRT(2.0  N). DELTAI = C/(SQRT(2.0  N/SQRT(NS)). The constant C can increase sublinearly with NACHKO, but it must be reduced as IELTER increases. The empirical value C = 1.0 has been found applicable for IELTER = 10, NACHKO = 100, and BKOMMA = .TRUE., which is a (10 , 100) evolution strategy. Standard deviation in random variation of the position angles P(I) for the mutation ellipsoid. DELTAP > 0.0 if BKORRL = .TRUE. Data in radians. A suitable value has been found to be DELTAP = 5.0  0.01745 (5 degrees) in certain cases. Number of parameters N > 0. Number of constraints M
0. Field length in array S or number of distinct step-size parameters that can be used, 1NSN. The mutation ellipse becomes a hypersphere for NS = 1. All the principal axes of the ellipsoid may be dierent for NS = N, whereas 1
( + ) Evolution Strategy KORR NP NY ZSTERN XSTERN ZBEST X S

P Y ZIELFU RESTRI GAUSSN

(integer)

389

Field length of array P. NP = N  (NS ; 1) ; ((NS ; 1)  NS)/2 if BKORRL = .TRUE. NP = 1 if BKORRL = .FALSE. (integer) Field length of array Y. NY = (N+NS+NP+1)  IELTER  2 if BKORRL = .TRUE. NY = (N+NS+1)  IELTER  2 if BKORRL = .FALSE. (real) Best value of objective function found during search for minimum. (one dimensional On call: initial parameter vector. real array At end of search: best values for parameters correspondof length N) ing to ZSTERN, or feasible vector found for the special case IFALLK = 0. (real) Current best value of objective function for the population, may be dierent from ZSTERN if BKOMMA = .TRUE. (one dimensional Holds the variables for a descendant. real array of length N) (one dimensional Holds the step sizes for a descendant. The user must real array of supply initial values. Universally valid rules for selecting length NS) the best S(I) are not available. If the step sizes are too large, a very good starting point can be wasted (BKOMMA = .TRUE.) or the step size adjustment may be very much delayed (BKOMMA = .FALSE.). If the initial step sizes are too small, there is only a slight chance of locating the global optimum in the presence of several local ones. In general, the optimum overall step sizes vary with the number N of parameters as C/SQRT(N), so the individual standard deviations vary as C/N with C = const. (one dimensional Holds the positional angles of the ellipsoid for a descendreal array ant. The user must supply initial values if BKORRL = of length NP) .TRUE. has been selected. If no better values are known initially, one can set P(I) = ATAN(1.0) for all I = 1(1)NP. (one dimensional Holds the vectors X, S, P, and the objective function real array of values for the parents of the current generation and the length NY) next generation as well. (real function) Name of the objective function, to be programmed by the user. (real function) Name of the function for evaluating all constraints, to be programmed by the user. (real function) Name of the function used in transforming a uniform random number distribution to a Gaussian one.

390 GLEICH

Appendix B

(real function) Name of the function for generating uniformly distributed random numbers. TKONTR (real function) Name of the run-time monitoring function. KANAL (integer) Channel number for output, relates only to messages output by subroutine PRUEFG concerning formal errors detected in the parameter list of subroutine KORR when the latter is called. 3. Method KORR is a development from EVOL, Rechenberg's two membered strategy, and GRUP, the older version of Schwefel's multimembered evolution strategy. The method is based on a very much simpli ed simulation of biological evolution. See I. Rechenberg, Evolution Strategy: Optimization of Technical Systems in Accordance with the Principles of Biological Evolution (in German), vol. 15 of Problemata series, Verlag Frommann-Holzboog, Stuttgart, 1973 also H.-P. Schwefel, Numerical Optimization of Computer Models, Wiley, Chichester, 1981 (translated by M. W. Finnis from Numerische Optimierung von Computer-Modellen mittels der Evolutionsstrategie, vol. 26 of Interdisciplinary Systems Research, Birkh auser, Basle, Switzerland, 1977). The IELTER parameter vectors are used to generate NACHKO new ones by introducing small normally distributed random changes. The IELTER best of these are used as starting points for the next generation (iteration). At the same time the strategy parameters are altered. These are the parameters of the current normal distributions for the lengths of the principal axes (standard deviations = step sizes) and the angular position of the mutation ellipsoid in N-dimensional space. Selection results in adaptation to local topology if the ratio NACHKO/IELTER is set large enough, e.g., at least 6. The random variations in the angles are produced by the addition of normally distributed random numbers, while those in the step sizes are produced from random numbers with a lognormal distribution by multiplication. 4. Convergence criterion The termination criterion is based on value dierences in the objective function, see under KONVKR, EPSILO(3), and EPSILO(4). 5. Peripheral I/O Input: none. Output: via channel KANAL, but only if there are formal errors in the parameter list of KORR. See under KANAL. 6. Notes The two membered strategy (EVOL) usually has the shortest run time of all these evolution strategies (the EVOL, GRUP, and KORR codings so far developed) because ordinary (serial) computers can test the descendants only one after another in a generation, whereas in nature they are tested in parallel. The run times of the multimembered strategies increase less rapidly than in proportion to NACHKO because the convergence rate taken

( + ) Evolution Strategy KORR

391

over the generations tends to increase with NACHKO. However, there are frequent instances where even the simpler multimembered scheme (GRUP) has a run time less than that of EVOL because GRUP and KORR in principle allow one to adapt the step sizes individually to the local topology, which is not possible with EVOL, and this permits one to scale the variables in a exible fashion. For this reason, the reliability and attainable accuracy are appreciably better than those given by EVOL. The new KORR program represents further improvements on GRUP in this respect on account of the increased exibility in the mutation ellipsoid, which improves the variability of the object variables. In addition to the lengths of the principal axes (standard deviations = step sizes) the positions of the principal axes in N-dimensional space are strategy parameters that are adjustable within the population. This together with the scaling provides directional adaptation to any valleys or ridges in the objective function surface. The changes in the object variables are no longer independent but linearly correlated, and this improves the convergence rate (with respect to the number of generations) quite appreciably in many instances. In special cases, however, there may be an increase in the run time arising from the storage and modi cation of the positional angles, and also from coordinate transformation. KORR enables the user to test how many strategy parameters (=degrees of freedom in the mutation ellipsoid) may be adequate to solve his special problem. The correlation can be suppressed completely, in which case KORR becomes equivalent to GRUP. Intermediary stages can be implemented by means of the NS parameter, the number of mutually independent step sizes. For example, for NS = 2 < N we have a hyperellipsoid of rotation with N-NS rotation axes. KORR diers from the older EVOL and GRUP in being divided into numerous small subroutines. This modular structure is disadvantageous as regards core requirement and run time, but it provides insight into the mode of operation of the program as a whole, so that it is easier for the user to modify the algorithm. Although KORR in general allows one to improve the reliability of the optimum search there are still two critical situations. Minima at the boundary of the feasible region or in a vertex are attained only slowly or inaccurately. In any case, certainty of global convergence cannot be guaranteed however, numerical tests have shown that the multimembered strategy is far better than other search procedures in this respect. It is capable of handling separated feasible regions provided that the number of parameters is not large and that the initial step sizes are set suitably large. In doubtful cases it is recommended to repeat the search each time with a dierent set of initial values and/or random numbers. 7. Subroutines or functions used --------------------------------------------------------SUBROUTINES: PRUEFG, SPEICH, MUTATI, DREHNG, UMSPEI, MINMAX, GNPOOL, ABSCHA FUNCTIONS : ZIELFU, RESTRI, GAUSSN, GLEICH, TKONTR, ZULASS, BLETAL ---------------------------------------------------------

The segment that calls KORR should have the name of the functions ZIELFU, RESTRI,

392

Appendix B

GLEICH, and TKONTR declared as external. This applies also to the name of any function used instead of GAUSSN to convert a uniform distribution to a normal one. ZIELFU Objective function, to be programmed by the user in the form: ------------------------------------------------FUNCTION ZIELFU(N,X) DIMENSION X(N) ... ZIELFU=... RETURN END -------------------------------------------------

N represents the number of parameters, and X represents the formal parameter vector. The actual values are supplied by KORR. The function should be written on the basis that KORR searches for a minimum if a maximum is to be sought, F must be supplied with a negative sign. RESTRI Constraints function, to be programmed by the user in the general style: ------------------------------------------------FUNCTION RESTRI(J,N,X) DIMENSION X(N) GOTO(1,2,3,...,(M)),J 1 RESTRI=... RETURN 2 RESTRI=... RETURN ... ... (M) RESTRI=... RETURN END -------------------------------------------------

N and X have the meanings described for the objective function, while J (integer) is the serial number of the constraint. The statements should be written on the basis that KORR will accept vector X as feasible if RESTRI
0.0 for all J = 1(1)M. TKONTR The function for monitoring the computation time may be de ned by the user or called from the subroutine library in the particular machine. The following structure is assumed: REAL FUNCTION TKONTR(D)

where D is a dummy parameter. TKONTR should be assigned the monitored quantity, e.g., the CPU time in seconds limited by TGRENZ. Many computers are supplied with

( + ) Evolution Strategy KORR

393

ready-made timing software. If this is given as a function, only its name needs to be supplied to KORR instead of TKONTR as a parameter. GLEICH Function for generating a uniform random number distribution in the range (0,1]. The structure must be: REAL FUNCTION GLEICH(D)

where D is arbitrary. GLEICH is the value of the random number. The standard library usually includes a suitable program, in which case only the appropriate name has to be supplied to KORR. The other subroutines and functions are explained briey in the program itself. --------------------------------------------------------SUBROUTINE KORR 1(IELTER,BKOMMA,NACHKO,IREKOM,BKORRL,KONVKR,IFALLK, 2TGRENZ,EPSILO,DELTAS,DELTAI,DELTAP,N,M,NS,NP,NY, 3ZSTERN,XSTERN,ZBEST,X,S,P,Y,ZIELFU,RESTRI,GAUSSN, 4GLEICH,TKONTR,KANAL) LOGICAL BKOMMA,BKORRL,BFATAL,BKONVG,BLETAL DIMENSION EPSILO(4),XSTERN(N),X(N),S(NS),P(NP), 1Y(NY) COMMON/PIDATA/PIHALB,PIEINS,PIZWEI EXTERNAL RESTRI,GAUSSN,GLEICH IREKOX = IREKOM / 100 IREKOS = (IREKOM - IREKOX*100) / 10 IREKOP = IREKOM - IREKOX*100 - IREKOS*10 D = 0. CALL PRUEFG 1(IELTER,BKOMMA,NACHKO,IREKOM,BKORRL,KONVKR,TGRENZ, 2EPSILO,DELTAS,DELTAI,DELTAP,N,M,NS,NP,NY,KANAL, 3BFATAL) C C CHECK INPUT PARAMETERS FOR FORMAL ERRORS. C IF(BFATAL) RETURN C C PREPARE AUXILIARY QUANTITIES. TIMING MONITORED IN C ACCORDANCE WITH THE TKONTR FUNCTION FROM HERE C ONWARDS. C TMAXIM=TGRENZ+TKONTR(D) IF(.NOT.BKORRL) GOTO 1 PIHALB=2.*ATAN(1.) PIEINS=PIHALB+PIHALB PIZWEI=PIEINS+PIEINS

Appendix B

394

1 C C C

C C C C

CHECK FEASIBILITY OF INITIAL VECTOR XSTERN. IFALLK=-1 ZSTERN=ZULASS(N,M,XSTERN,RESTRI) IF(ZSTERN.GT.0.) GOTO 3 2 IFALLK=1 ZSTERN=ZIELFU(N,XSTERN) 3 CALL SPEICH 1(0,BKORRL,EPSILO,N,NS,NP,NY,ZSTERN,XSTERN,S,P,Y) THE INITIAL VALUES SUPPLIED BY THE USER ARE STORED IN FIELD Y AS THE DATA OF THE FIRST PARENT. IF(KONVKR.GT.1) Z1=ZSTERN ZBEST=ZSTERN LBEST=0 IF(IELTER.EQ.1) GOTO 16 DSMAXI=DELTAS DPMAXI=AMIN1(DELTAP*10.,PIHALB) DO 14 L=2,IELTER

C C C C C C

IF IELTER > 1, THE OTHER IELTER - 1 INITIAL VECTORS ARE DERIVED FROM THE VECTOR FOR THE FIRST PARENT BY MUTATION (WITHOUT SELECTION). THE STRATEGY PARAMETERS ARE WIDELY SPREAD. 4 5

501 6 7 C C

NL=1+N-NS NM=N-1 NZ=NY/(IELTER+IELTER) IF(M.EQ.0) GOTO 2

DO 4 I=1,NS S(I)=Y(N+I) IF(TKONTR(D).LT.TMAXIM) GOTO 501 IFALLK=-3 GOTO 42 IF(.NOT.BKORRL) GOTO 7 DO 6 I=1,NP P(I)=Y(N+NS+I) CALL MUTATI 1(NL,NM,BKORRL,DSMAXI,DELTAI,DPMAXI,N,NS,NP,X,S,P, 2GAUSSN,GLEICH) MUTATION IN ALL OBJECT AND STRATEGY PARAMETERS.

( + ) Evolution Strategy KORR C 8 C C C C C C

DO 8 I=1,N X(I)=X(I)+Y(I) IF(IFALLK.GT.0) GOTO 9 IF THE STARTING POINT IS NOT FEASIBLE, EACH MUTATION IS CHECKED AT ONCE TO SEE WHETHER A FEASIBLE VECTOR HAS BEEN FOUND. THE SEARCH ENDS WITH IFALLK = 0 IF THIS IS SO.

9 C C C C C C C C

Z=ZULASS(N,M,X,RESTRI) IF(Z)40,40,12 IF(M.EQ.0) GOTO 11 IF(.NOT.BLETAL(N,M,X,RESTRI)) GOTO 11 IF A MUTATION FROM A FEASIBLE STARTING POINT RESULTS IN A NON-FEASIBLE X VECTOR, THEN THE STEP SIZES ARE REDUCED (ON THE ASSUMPTION THAT THEY WERE INITIALLY TOO LARGE) IN ORDER TO AVOID THE THE CONSUMPTION OF EXCESSIVE TIME IN DEFINING THE THE FIRST PARENT GENERATION.

DO 10 I=1,NS S(I)=S(I)*.5 GOTO 5 11 Z=ZIELFU(N,X) 12 IF(Z.GT.ZBEST) GOTO 13 ZBEST=Z LBEST=L-1 DSMAXI=DSMAXI*ALOG(2.) 13 CALL SPEICH 1((L-1)*NZ,BKORRL,EPSILO,N,NS,NP,NY,Z,X,S,P,Y) 10

C C C

STORE PARENT DATA IN ARRAY Y. 14

C C C C C C

IF(KONVKR.GT.1) Z1=Z1+Z CONTINUE THE INITIAL PARENT GENERATION IS NOW COMPLETE. ZSTERN AND XSTERN, WHICH HOLD THE BEST VALUES, ARE OVERWRITTEN WHEN AN IMPROVEMENT OF THE INITIAL SITUATION IS OBTAINED. IF(LBEST.EQ.0) GOTO 16

395

Appendix B

396

15 16

ZSTERN=ZBEST K=LBEST*NZ DO 15 I=1,N XSTERN(I)=Y(K+I) L1=IELTER L2=0 IF(KONVKR.GT.1) KONVZ=0

C C C C

ALL INITIALIZATION STEPS COMPLETED AT THIS POINT. EACH FRESH GENERATION NOW STARTS AT LABEL 17.

C C C C C

LMUTAT IS THE MUTATION COUNTER WITHIN A GENERATION, WHILE L3 IS THE COUNTER FOR LETHAL MUTATIONS WHEN THE PROBLEM INVOLVES CONSTRAINTS.

C C C C C C C C C

IF BKOMMA=.FALSE. HAS BEEN SELECTED, THE PARENTS MUST BE INCORPORATED IN THE SELECTION. THE DATA FOR THESE ARE TRANSFERRED FROM THE FIRST (OR SECOND) PART OF THE ARRAY Y TO THE SECOND (OR FIRST) PART. IN THIS CASE THE WORST INDIVIDUAL MUST ALSO BE KNOWN, THIS IS REPLACED BY THE FIRST BETTER DESCENDANT.

C C C

THE GENERATION OF EACH DESCENDANT STARTS AT LABEL 18

17

18

C C C C C

L3=L2 L2=L1 L1=L3 IF(M.GT.0) L3=0 LMUTAT=0

IF(BKOMMA) GOTO 18

CALL UMSPEI 1(L1*NZ,L2*NZ,IELTER*NZ,NY,Y) CALL MINMAX 1(-1.,L2,NZ,ZSCHL,LSCHL,IELTER,NY,Y)

K1=L1+IELTER*GLEICH(D) RANDOM CHOICE OF A PARENT OR OF A PAIR OF PARENTS IN ACCORDANCE WITH THE VALUE CHOSEN FOR IREKOM. IF IREKOM=3 OR IREKOM=5, THE CHOICE OF PARENTS IS MADE WITHIN GNPOOL.

( + ) Evolution Strategy KORR C

K2=L1+IELTER*GLEICH(D) 19 CALL GNPOOL 1(1,L1,K1,K2,NZ,N,IELTER,IREKOS,NS,NY,S,Y,GLEICH)

C C C C

STEP SIZES SUPPLIED FOR THE DESCENDANT FROM THE POOL OF GENES.

C C C C C

POSITIONAL ANGLES OF ELLIPSOID SUPPLIED FOR THE DESCENDANT FROM THE POOL OF GENES WHEN CORRELATION IS REQUIRED.

C C C C C C C

CALL TO MUTATION SUBROUTINE FOR ALL VARIABLES, INCLUDING POSSIBLY COORDINATE TRANSFORMATION. S (AND P) ARE ALREADY THE NEW ATTRIBUTES OF THE DESCENDANT, WHILE X REPRESENTS THE CHANGES TO BE MADE IN THE OBJECT VARIABLES.

C C C C C C

OBJECT VARIABLES SUPPLIED FOR THE DESCENDANT FROM THE POOL OF GENES AND ADDITION OF THE MODIFICATION VECTOR. X NOW REPRESENTS THE NEW STATE OF THE DESCENDANT.

C C C C

EVALUATION OF THE AUXILIARY OBJECTIVE FUNCTION FOR THE SEARCH FOR A FEASIBLE VECTOR.

IF(BKORRL) CALL GNPOOL 1(2,L1,K1,K2,NZ,N+NS,IELTER,IREKOP,NP,NY,P,Y,GLEICH)

CALL MUTATI 1(NL,NM,BKORRL,DELTAS,DELTAI,DELTAP,N,NS,NP,X,S,P, 2GAUSSN,GLEICH)

CALL GNPOOL 1(3,L1,K1,K2,NZ,0,IELTER,IREKOX,N,NY,X,Y,GLEICH)

LMUTAT=LMUTAT+1 IF(IFALLK.GT.0) GOTO 20

C C

20

Z=ZULASS(N,M,X,RESTRI) IF(Z)40,40,22 IF(M.EQ.0) GOTO 21 CHECK FEASIBILITY OF DESCENDANT. IF THE RESULT IS

397

Appendix B

398 C C C

C C C C C

21

NEGATIVE (LETHAL MUTATION), THE MUTATION IS NOT COUNTED AS REGARDS THE NACHKO PARAMETER. IF(.NOT.BLETAL(N,M,X,RESTRI)) GOTO 21 IF(.NOT.BKOMMA) GOTO 25 LMUTAT=LMUTAT-1 L3=L3+1 IF(L3.LT.NACHKO) GOTO 18 L3=0 TIME CHECK MADE NOT ONLY AFTER EACH GENERATION BUT ALSO AFTER EVERY NACHKO LETHAL MUTATIONS FOR CERTAINTY. IF(TKONTR(D).LT.TMAXIM) GOTO 18 IFALLK=3 GOTO 26 Z=ZIELFU(N,X)

C C C C

EVALUATION OF OBJECTIVE FUNCTION VALUE FOR THE DESCENDANT.

C C C C

TRANSFER OF DATA OF DESCENDANT TO PART OF ARRAY Y HOLDING THE PARENTS FOR THE NEXT GENERATION.

22

IF(BKOMMA.AND.LMUTAT.LE.IELTER) GOTO 23 IF(Z-ZSCHL)24,24,25 23 LSCHL=L2+LMUTAT-1 24 CALL SPEICH 1(LSCHL*NZ,BKORRL,EPSILO,N,NS,NP,NY,Z,X,S,P,Y)

C C C C C 25 C C C 26

IF(.NOT.BKOMMA.OR.LMUTAT.GE.IELTER) CALL MINMAX 1(-1.,L2,NZ,ZSCHL,LSCHL,IELTER,NY,Y) LOOK FOR THE CURRENTLY WORST INDIVIDUAL STORED IN ARRAY Y WITHOUT CONSIDERING THE PARENTS THAT STILL CAN PRODUCE DESCENDANTS IN THIS GENERATION. IF(LMUTAT.LT.NACHKO) GOTO 18 END OF GENERATION. CALL MINMAX 1(1.,L2,NZ,ZBEST,LBEST,IELTER,NY,Y)

( + ) Evolution Strategy KORR C C C C C C

LOOK FOR THE BEST OF THE INDIVIDUALS HELD AS PARENTS FOR THE NEXT GENERATION. IF THIS IS BETTER THAN ANY DESCENDANT PREVIOUSLY GENERATED, THE DATA ARE WRITTEN INTO ZSTERN AND XSTERN.

C C C

TEST CONVERGENCE CRITERION.

C C C

IF(ZBEST.GT.ZSTERN) GOTO 28 ZSTERN=ZBEST K=LBEST*NZ DO 27 I=1,N 27 XSTERN(I)=Y(K+I) 28 IF(IFALLK.EQ.3) GOTO 30 Z2=0. K=L2*NZ DO 29 L=1,IELTER K=K+NZ 29 Z2=Z2+Y(K) CALL ABSCHA 1(IELTER,KONVKR,IFALLK,EPSILO,ZBEST,ZSCHL,Z1,Z2, 2KONVZ,BKONVG)

IF(BKONVG) GOTO 30 CHECK TIME ELAPSED. IF(TKONTR(D).LT.TMAXIM) GOTO 17

C C PREPARE FINAL DATA FOR RETURN FROM KORR IF THE C STARTING POINT WAS FEASIBLE. C 30 K=LBEST*NZ DO 31 I=1,N K=K+1 31 X(I)=Y(K) DO 32 I=1,NS K=K+1 32 S(I)=Y(K) IF(.NOT.BKORRL) RETURN DO 33 I=1,NP K=K+1 33 P(I)=Y(K) RETURN

399

400

Appendix B

C C PREPARE FINAL DATA FOR RETURN FROM KORR IF THE C STARTING POINT WAS NOT FEASIBLE. C 40 DO 41 I=1,N 41 XSTERN(I)=X(I) ZSTERN=ZIELFU(N,XSTERN) ZBEST=ZSTERN IFALLK=0 42 RETURN END ---------------------------------------------------------

Subroutine PRUEFG PRUEFG checks the values given with the parameter list on calling KORR. If discrepancies are found, an attempt is made to eliminate them. If this is not possible, e.g., arrays required are not appropriately dimensioned, the search for the minimum is not initiated. Then PRUEFG outputs a message to the peripheral unit denoted by KANAL on the correction of the error or else a warning message. BFATAL supplies KORR with information on the outcome of the check as a Boolean value. --------------------------------------------------------SUBROUTINE PRUEFG 1(IELTER,BKOMMA,NACHKO,IREKOM,BKORRL,KONVKR,TGRENZ, 2EPSILO,DELTAS,DELTAI,DELTAP,N,M,NS,NP,NY,KANAL, 3BFATAL) LOGICAL BKOMMA,BKORRL,BFATAL DIMENSION EPSILO(4) IREKOX = IREKOM / 100 IREKOS = (IREKOM - IREKOX*100) / 10 IREKOP = IREKOM - IREKOX*100 - IREKOS*10 100 FORMAT(1H ,' CORRECTION. IELTER > 0 . ASSUMED: 2 AND 1 KONVKR = ',I5) 101 FORMAT(1H ,' CORRECTION. NACHKO > 0 . ASSUMED: ',I5) 102 FORMAT(1H ,' WARNING. BETTER VALUE NACHKO >= 6*IELTER') 103 FORMAT(1H ,' CORRECTION. IF BKOMMA = .TRUE., THEN 1 NACHKO > IELTER . ASSUMED: ',I3) 1041 FORMAT(1H ,' CORRECTION. 0 < IREKOX < 6 . ASSUMED: 1') 1042 FORMAT(1H ,' CORRECTION. 0 < IREKOS < 6 . ASSUMED: 1') 1043 FORMAT(1H ,' CORRECTION. 0 < IREKOP < 6 . ASSUMED: 1') 105 FORMAT(1H ,' CORRECTION. IF IELTER = 1, THEN 1 IREKOM = 111 . ASSUMED: 111') 106 FORMAT(1H ,' CORRECTION. IF N = 1 OR NS = 1, THEN 1 BKORRL = .FALSE. . ASSUMED: .FALSE.') 107 FORMAT(1H ,' CORRECTION. KONVKR > 0 . ASSUMED: ',I5)

( + ) Evolution Strategy KORR 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129

1

FORMAT(1H ,' CORRECTION. IF IELTER = 1, THEN 1 KONVKR > 1 . ASSUMED: ',I5) FORMAT(1H ,' CORRECTION. EPSILO(',I1,') > 0. . 1 SIGN REVERSED') FORMAT(1H ,' WARNING. EPSILO(',I1,') TOO SMALL. 1 TREATED AS 0. .') FORMAT(1H ,' CORRECTION. DELTAS >= 0. . 1 SIGN REVERSED') FORMAT(1H ,' WARNING. EXP(DELTAS) = 1. 1 OVER-ALL STEP SIZE CONSTANT') FORMAT(1H ,' CORRECTION. DELTAI >= 0. . 1 SIGN REVERSED') FORMAT(1H ,' WARNING. EXP(DELTAI) = 1. 1 STEP-SIZE RELATIONS CONSTANT') FORMAT(1H ,' CORRECTION. DELTAP >= 0. . 1 SIGN REVERSED') FORMAT(1H ,' WARNING. DELTAP = 0. 1 CORRELATION REMAINS FIXED') FORMAT(1H ,' WARNING. TGRENZ <= 0. 1 ONE GENERATION TESTED') FORMAT(1H ,' CORRECTION. M >= 0 . ASSUMED: 0') FORMAT(1H ,' FATAL ERROR. N <= 0') FORMAT(1H ,' FATAL ERROR. NS <= 0') FORMAT(1H ,' FATAL ERROR. NP <= 0') FORMAT(1H ,' CORRECTION. 1 <= NS <= N . ASSUMED: ',I5) FORMAT(1H ,' CORRECTION. IF BKOORL = .FALSE., THEN 1 NP = 1 . ASSUMED: 1') FORMAT(1H ,' FATAL ERROR. NY < (N+NS+1)*IELTER*2') FORMAT(1H ,' CORRECTION. NY = (N+NS+1)*IELTER*2 . 1 ASSUMED: ',I5) FORMAT(1H ,' FATAL ERROR. NP < N*(NS-1)-((NS-1)*NS)/2') FORMAT(1H ,' CORRECTION. NP = N*(NS-1)-((NS-1)*NS)/2 . 1 ASSUMED: ',I5) FORMAT(1H ,' FATAL ERROR. NY < (N+NS+NP+1)*IELTER*2') FORMAT(1H ,' CORRECTION. NY = (N+NS+NP+1)*IELTER*2 . 1 ASSUMED: ',I5) BFATAL=.TRUE. IF(IELTER.GT.0) GOTO 1 IELTER=2 KONVKR=N+N WRITE(KANAL,100)KONVKR IF(NACHKO.GT.0) GOTO 2 NACHKO=6*IELTER WRITE(KANAL,101)NACHKO

401

Appendix B

402 2

3 301 302 4

5 6

7 8 9 10 11 12 13

IF(.NOT.BKOMMA.OR.NACHKO.GE.6*IELTER) GOTO 3 WRITE(KANAL,102) IF(NACHKO.GT.IELTER) GOTO 3 NACHKO=6*IELTER WRITE(KANAL,103)NACHKO IF(IREKOX.GT.0.AND.IREKOX.LT.6) GOTO 301 IREKOX=1 WRITE(KANAL,1041) IF(IREKOS.GT.0.AND.IREKOS.LT.6) GOTO 302 IREKOS=1 WRITE(KANAL,1042) IF(IREKOP.GT.0.AND.IREKOP.LT.6) GOTO 4 IREKOP=1 WRITE(KANAL,1043) IF(IREKOM.EQ.111.OR.IELTER.NE.1) GOTO 5 IREKOM=111 IREKOX=1 IREKOS=1 IREKOP=1 WRITE(KANAL,105) IF(.NOT.BKORRL.OR.(N.GT.1.AND.NS.GT.1)) GOTO 6 BKORRL=.FALSE. WRITE(KANAL,106) IF(KONVKR.GT.0) GOTO 7 IF(IELTER.EQ.1) KONVKR=N+N IF(IELTER.GT.1) KONVKR=1 WRITE(KANAL,107)KONVKR GOTO 8 IF(KONVKR.GT.1.OR.IELTER.GT.1) GOTO 8 KONVKR=N+N WRITE(KANAL,108)KONVKR DO 12 I=1,4 IF(I.EQ.2.OR.I.EQ.4) GOTO 9 IF(EPSILO(I))10,11,12 IF((1.+EPSILO(I))-1.)10,11,12 EPSILO(I)=-EPSILO(I) WRITE(KANAL,109)I GOTO 12 WRITE(KANAL,110)I CONTINUE IF(EXP(DELTAS)-1.)13,14,15 DELTAS=-DELTAS WRITE(KANAL,111) GOTO 15

( + ) Evolution Strategy KORR 14 15 16 17 18 19 20 21 22 23 24 25 26 27

28 29 30 31

IF(EXP(DELTAI).NE.1.) GOTO 15 WRITE(KANAL,112) IF(EXP(DELTAI)-1.)16,17,18 DELTAI=-DELTAI WRITE(KANAL,113) GOTO 18 IF(IREKOS.GT.1.AND.EXP(DELTAS).GT.1.) GOTO 18 WRITE(KANAL,114) IF(.NOT.BKORRL) GOTO 21 IF(DELTAP)19,20,21 DELTAP=-DELTAP WRITE(KANAL,115) GOTO 21 WRITE(KANAL,116) IF(TGRENZ.GT.0.) GOTO 22 WRITE(KANAL,117) IF(M.GE.0) GOTO 23 M=0 WRITE(KANAL,118) IF(N.GT.0) GOTO 24 WRITE(KANAL,119) RETURN IF(NS.GT.0) GOTO 25 WRITE(KANAL,120) RETURN IF(NP.GT.0) GOTO 26 WRITE(KANAL,121) RETURN IF(NS.LE.N) GOTO 27 NS=N WRITE(KANAL,122)N IF(BKORRL) GOTO 31 IF(NP.EQ.1) GOTO 28 NP=1 WRITE(KANAL,123) NYY=(N+NS+1)*IELTER*2 IF(NY-NYY)29,37,30 WRITE(KANAL,124) RETURN NY=NYY WRITE(KANAL,125)NY GOTO 37 NPP=N*(NS-1)-((NS-1)*NS)/2 IF(NP-NPP)32,34,33

403

404

Appendix B

32

WRITE(KANAL,126) RETURN 33 NP=NPP WRITE(KANAL,127)NP 34 NYY=(N+NS+NP+1)*IELTER*2 IF(NY-NYY)35,37,36 35 WRITE(KANAL,128) RETURN 36 NY=NYY WRITE(KANAL,129)NY 37 BFATAL=.FALSE. RETURN END ---------------------------------------------------------

Function ZULASS This function is required only if there are constraints. If the starting point does not lie in the feasible region, ZULASS generates an auxiliary objective function that is used to search for a feasible initial vector. If ZULASS, the negative sum of the values for the functions representing constraints that have been violated, is zero, then X represents a feasible vector that can be used in restarting the search with KORR. XX represents XSTERN or X. --------------------------------------------------------FUNCTION ZULASS 1(N,M,XX,RESTRI) DIMENSION XX(N) ZULASS=0. DO 1 J=1,M R=RESTRI(J,N,XX) IF(R.LT.0.) ZULASS=ZULASS-R 1 CONTINUE RETURN END ---------------------------------------------------------

Subroutine UMSPEI UMSPEI is required only if BKOMMA = .FALSE., whereupon the parents in the source generation have to be subject to selection. UMSPEI transposes the data on the parents within array Y. K1, K2, and KK are auxiliary quantities transmitted from KORR that de ne the number and addresses of the data to be transposed.

( + ) Evolution Strategy KORR

405

--------------------------------------------------------SUBROUTINE UMSPEI 1(K1,K2,KK,NY,Y) DIMENSION Y(NY) DO 1 K=1,KK 1 Y(K2+K)=Y(K1+K) RETURN END ---------------------------------------------------------

Subroutine GNPOOL GNPOOL supplies a set of variables for a descendant by drawing on the pool of parents taken together in accordance with the type of recombination selected. This subroutine is called once each for the object variables X, the strategy variables S, and possibly also P. To minimize storage demand, the changes in the object variables by mutation are added immediately (J = 3). In intermediary recombination for the positional angle (J = 2), a check must be made on the dierence between the parental angles to establish suitable mean values. J = 1 denotes the case where step sizes are involved. L1 denotes the part of the gene pool from which the parent data are to be drawn if IREKO = 3 or IREKO = 5. K1 denotes the parent selected by KORR whose data are to be used when IREKO = 1 (no recombination). K1 and K2 denote the two parents whose data are to be recombined if IREKO = 2 or IREKO = 4 has been selected. NZ and NN are auxiliary quantities for deriving the addresses in array Y. NX represents N or NS or NP, XX represents X or S or P, IREKO represents one of the digits of IREKOM, i.e., IREKOX or IREKOS or IREKOP. --------------------------------------------------------SUBROUTINE GNPOOL 1(J,L1,K1,K2,NZ,NN,IELTER,IREKO,NX,NY,XX,Y,GLEICH) DIMENSION XX(NX),Y(NY) COMMON/PIDATA/PIHALB,PIEINS,PIZWEI EXTERNAL GLEICH IF(J.EQ.3) GOTO 11 GOTO(1,1,1,7,9),IREKO 1 KI1=K1*NZ+NN IF(IREKO.GT.1) GOTO 3 DO 2 I=1,NX 2 XX(I)=Y(KI1+I) RETURN 3 KI2=K2*NZ+NN IF(IREKO.EQ.3) GOTO 5

Appendix B

406

4 5

6 7 8 9

10 11 12 13 14

15 16 17 18 19

DO 4 I=1,NX KI=KI1 IF(GLEICH(D).GE..5) KI=KI2 XX(I)=Y(KI+I) RETURN DO 6 I=1,NX XX1=Y(KI1+I) XX2=Y(KI2+I) XXI=(XX1+XX2)*.5 IF(J.EQ.1) GOTO 6 DXX=XX1-XX2 IF(ABS(DXX).GT.PIEINS) XXI=XXI+SIGN(PIEINS,DXX) XX(I)=XXI RETURN DO 8 I=1,NX XX(I)=Y((L1+IFIX(IELTER*GLEICH(D)))*NZ+NN+I) RETURN DO 10 I=1,NX XX1=Y((L1+IFIX(IELTER*GLEICH(D)))*NZ+NN+I) XX2=Y((L1+IFIX(IELTER*GLEICH(D)))*NZ+NN+I) XXI=(XX1+XX2)*.5 IF(J.EQ.1) GOTO 10 DXX=XX1-XX2 IF(ABS(DXX).GT.PIEINS) XXI=XXI+SIGN(PIEINS,DXX) XX(I)=XXI RETURN GOTO(12,12,12,18,20),IREKO KI1=K1*NZ+NN IF(IREKO.GT.1) GOTO 14 DO 13 I=1,NX XX(I)=XX(I)+Y(KI1+I) RETURN KI2=K2*NZ+NN IF(IREKO.EQ.3) GOTO 16 DO 15 I=1,NX KI=KI1 IF(GLEICH(D).GE..5) KI=KI2 XX(I)=XX(I)+Y(KI+I) RETURN DO 17 I=1,NX XX(I)=XX(I)+(Y(KI1+I)+Y(KI2+I))*.5 RETURN DO 19 I=1,NX XX(I)=XX(I)+Y((L1+IFIX(IELTER*GLEICH(D)))*NZ+NN+I)

( + ) Evolution Strategy KORR

407

RETURN 20 DO 21 I=1,NX 21 XX(I)=XX(I)+(Y((L1+IFIX(IELTER*GLEICH(D)))*NZ+NN+I) 1+Y((L1+IFIX(IELTER*GLEICH(D)))*NZ+NN+I))*.5 RETURN END ---------------------------------------------------------

Subroutine SPEICH SPEICH transfers to the data pool Y for the parents of the next generation the data of a descendant representing a successful mutation (the object variables X and the strategy parameters S (and P, if used) together with the corresponding value of the objective function). A check is made that S (and P) fall within speci ed bounds. J is the address in array Y from which point onwards the data are to be written and is provided by KORR. ZZ represents ZSTERN or Z, XX represents XSTERN or X. --------------------------------------------------------SUBROUTINE SPEICH 1(J,BKORRL,EPSILO,N,NS,NP,NY,ZZ,XX,S,P,Y) LOGICAL BKORRL DIMENSION EPSILO(4),XX(N),S(NS),P(NP),Y(NY) COMMON/PIDATA/PIHALB,PIEINS,PIZWEI K=J DO 1 I=1,N K=K+1 1 Y(K)=XX(I) DO 2 I=1,NS K=K+1 2 Y(K)=AMAX1(S(I),EPSILO(1)) IF(.NOT.BKORRL) GOTO 4 DO 3 I=1,NP K=K+1 PI=P(I) IF(ABS(PI).GT.PIEINS) PI=PI-SIGN(PIZWEI,PI) 3 Y(K)=PI 4 K=K+1 Y(K)=ZZ RETURN END ---------------------------------------------------------

408

Appendix B

Subroutine MINMAX MINMAX searches for the smallest or largest value in a series of values of the objective function held in an array. KORR calls this subroutine to determine the best or worst parent, in the rst case in order to transfer its data to the location ZBEST (and perhaps also ZSTERN and XSTERN) and in the other case in order to give space for a better descendant. C=1.0 initiates a search for the best (smallest) value of the function, while C=;1.0 does the same for the worst (largest) value. LL and NZ are auxiliary quantities used to transmit information on the position of the required values within array Y. ZM and LM contain the best (or worst) values of the objective function and the number of the corresponding parent minus one. --------------------------------------------------------SUBROUTINE MINMAX 1(C,LL,NZ,ZM,LM,IELTER,NY,Y) DIMENSION Y(NY) LM=LL K1=LL*NZ+NZ ZM=Y(K1) IF(IELTER.EQ.1) RETURN K1=K1+NZ K2=(LL+IELTER)*NZ KM=LL DO 1 K=K1,K2,NZ KM=KM+1 ZZ=Y(K) IF((ZZ-ZM)*C.GT.0.) GOTO 1 ZM=ZZ LM=KM 1 CONTINUE RETURN END ---------------------------------------------------------

Subroutine ABSCHA ABSCHA tests the convergence criterion. If KONVKR = 1 has been selected, the difference between the objective function values representing the best and worst parents (ZBEST and ZSCHL) must be less than the limits set by EPSILO(3) (absolute) or EPSILO(4) (relative). Then the assignment BKONVG = .TRUE. is made. Alternatively, the current dierence ZSCHL-ZBEST is replaced by the change Z1;Z2 in the sum of all the parent objective function values occurring after KONVKR generations divided by IELTER. The Boolean variable BKONVG transmits the result of the convergence test to KORR.

( + ) Evolution Strategy KORR

409

KONVZ is the generation counter if KONVKR > 1. --------------------------------------------------------SUBROUTINE ABSCHA 1(IELTER,KONVKR,IFALLK,EPSILO,ZBEST,ZSCHL,Z1,Z2, 2KONVZ,BKONVG) LOGICAL BKONVG DIMENSION EPSILO(4) IF(KONVKR.EQ.1) GOTO 1 KONVZ=KONVZ+1 IF(KONVZ.LT.KONVKR) GOTO 3 KONVZ=0 DELTAF=Z1-Z2 Z1=Z2 GOTO 2 1 DELTAF=(ZSCHL-ZBEST)*IELTER 2 IF(DELTAF.GT.EPSILO(3)*IELTER) GOTO 3 IF(DELTAF.GT.EPSILO(4)*ABS(Z2)) GOTO 3 IFALLK=ISIGN(2,IFALLK) BKONVG=.TRUE. RETURN 3 BKONVG=.FALSE. RETURN END ---------------------------------------------------------

Function GAUSSN GAUSSN converts a uniform random number distribution to a normal one. The function has been programmed for the trapezium algorithm (J. H. Ahrens and U. Dieter, Computer Methods for Sampling from the Exponential and Normal Distributions, Communications of the Association for Computing Machinery, vol. 15 (1972), pp. 873-882 and 1047). The Box-Muller rules require in many cases (machine-dependent) a longer run time even if both of the pair of numbers can be used. SIGMA is the standard deviation, which is multiplied by the random number derived from a (0.0,1.0) normal distribution. --------------------------------------------------------FUNCTION GAUSSN 1(SIGMA,GLEICH) 1 U=GLEICH(D) U0=GLEICH(D) IF(U.GE..919544406) GOTO 2 X=2.40375766*(U0+U*.825339283)-2.11402808 GOTO 10

410

Appendix B

2 3

IF(U.LT..965487131) GOTO 4 U1=GLEICH(D) Y=SQRT(4.46911474-2.*ALOG(U1)) U2=GLEICH(D) IF(Y*U2.GT.2.11402808) GOTO 3 GOTO 9 4 IF(U.LT..949990709) GOTO 6 5 U1=GLEICH(D) Y=1.84039875+U1*.273629336 U2=GLEICH(D) IF(.398942280*EXP(-.5*Y*Y)-.443299126+Y*.209694057 1.LT.U2*.0427025816) GOTO 5 GOTO 9 6 IF(U.LT..925852334) GOTO 8 7 U1=GLEICH(D) Y=.289729574+U1*1.55066917 U2=GLEICH(D) IF(.398942280*EXP(-.5*Y*Y)-.443299126+Y*.209694057 1.LT.U2*.0159745227) GOTO 7 GOTO 9 8 U1=GLEICH(D) Y=U1*.289729574 U2=GLEICH(D) IF(.398942280*EXP(-.5*Y*Y)-.382544556 1.LT.U2*.0163977244) GOTO 8 9 X=Y IF(U0.GE..5) X=-Y 10 GAUSSN=SIGMA*X RETURN END ---------------------------------------------------------

Subroutine DREHNG DREHNG is called from MUTATI only if BKORRL = .TRUE. and N > 1. DREHNG performs the coordinate transformation of the modi cation vector for the object variables. Although the components of this vector are initially mutually independent, they are linearly related on account of the rotation speci ed by the positional angles P and so are correlated. The transformation involves NP partial rotations, in each of which only two of the components of the modi cation vector are involved.

( + ) Evolution Strategy KORR

411

--------------------------------------------------------SUBROUTINE DREHNG 1(NL,NM,N,NP,X,P) DIMENSION X(N),P(NP) NQ=NP DO 1 II=NL,NM N1=N-II N2=N DO 1 I=1,II X1=X(N1) X2=X(N2) SI=SIN(P(NQ)) CO=COS(P(NQ)) X(N2)=X1*SI+X2*CO X(N1)=X1*CO-X2*SI N2=N2-1 1 NQ=NQ-1 RETURN END ---------------------------------------------------------

Logical function BLETAL BLETAL tests the feasibility of an object variable vector immediately on production if constraints are imposed. The rst constraint to be violated causes BLETAL to signal to KORR via the function name (declared as a Boolean variable) that the mutation was lethal. --------------------------------------------------------LOGICAL FUNCTION BLETAL 1(N,M,X,RESTRI) DIMENSION X(N) DO 1 J=1,M IF(RESTRI(J,N,X).LT.0.) GOTO 2 1 CONTINUE BLETAL=.FALSE. RETURN 2 BLETAL=.TRUE. RETURN END ---------------------------------------------------------

Subroutine MUTATI MUTATI handles the random alteration of the strategy variables and the object variables. First, the step sizes are altered in accordance with the DELTAS and DELTAI

412

Appendix B

parameters by multiplication by two random factors with log-normal distributions. The resulting normal distribution is used in a random vector X that represents the changes in the object variables. If BKORRL = .TRUE. is set when KORR is called, i.e., linear correlation is required, the positional angle P is also mutated, with random numbers from a (0.0,DELTAP) normal distribution added to the original values. Also, DREHNG is called in that case to transform the vector of modi cations to the object variable. NL and NM are auxiliary quantities transmitted from KORR via MUTATI to DREHNG. --------------------------------------------------------SUBROUTINE MUTATI 1(NL,NM,BKORRL,DELTAS,DELTAI,DELTAP,N,NS,NP,X,S,P, 2GAUSSN,GLEICH) LOGICAL BKORRL DIMENSION X(N),S(NS),P(NP) EXTERNAL GLEICH DS=GAUSSN(DELTAS,GLEICH) DO 1 I=1,NS 1 S(I)=S(I)*EXP(DS+GAUSSN(DELTAI,GLEICH)) DO 2 I=1,N 2 X(I)=GAUSSN(S(MIN0(I,NS)),GLEICH) IF(.NOT.BKORRL) RETURN DO 3 I=1,NP 3 P(I)=P(I)+GAUSSN(DELTAP,GLEICH) CALL DREHNG 1(NL,NM,N,NP,X,P) RETURN END ---------------------------------------------------------

Note Without modi cations the subroutines EVOL, GRUP, and KORR may be used to solve optimization problems with integer (or discrete) and mixed-integer variables. The search for an optimum then, however, will only lead into the vicinity of the exact solution. The discreteness may be induced by the user when formulating the objective function, by merely rounding the correspondent variables to integers or by attributing discrete values to them. The following two examples will give hints only to possible formulations. In order to get the results in the form wanted the variables will have to be transformed in the same manner at the end of the optimum search with EVOL, GRUP, or KORR, as is done within the objective function.

( + ) Evolution Strategy KORR Example 1 Minimize

413

X n

F (x) = (x ; i)

with x
0, integer for all i = 1(1)n

i

=1

i

i

--------------------------------------------------------FUNCTION F(N,X) DIMENSION X(N) F=0. DO 1 I=1,N IX=IFIX(ABS(X(I))) XI=FLOAT(IX-I) F=F+XI*XI 1 CONTINUE RETURN END ---------------------------------------------------------

Example 2 Minimize

F (x) = (x1 ; 2)2 + (x1 ; 2x2)2 with x1 from f1.3, 1.5, 2.2, 2.8g only --------------------------------------------------------FUNCTION F(N,X) DIMENSION X(N), Y(4) DATA Y /1.3,1.5,2.2,2.8/ DO 1 I=1,4 X1=Y(I) IF (X(1)-X1) 2,2,1 1 CONTINUE 2 F1=X1-2. F2=X1-X(2)-X(2) F =F1*F1+F2*F2 RETURN END ---------------------------------------------------------

414

Appendix B

Appendix C Programs C.1 Contents of the Floppy Disk The
oppy disk that accompanies this book contains:


Sources of FORTRAN subroutines of the following direct optimization procedures as described in the Chapters 3, 5, and 7 of the book. - FIBO Coordinate strategy with Fibonacci division boh ( boh.f) calls subroutine bo ( bo.f) - GOLD Coordinate strategy with Golden section goldh (goldh.f) calls subroutine gold (gold.f) - LAGR Coordinate strategy with Lagrangian interpolation lagrh (lagrh.f) calls subroutine lagr (lagr.f) - HOJE Strategy of Hooke and Jeeves (pattern search) hoje (hoje.f) calls subroutine hilf (hilf.f) - ROSE Strategy of Rosenbrock (rotating coordinates search) rose (rose.f) calls subroutine grsmr (grsmr.f) - DSCG Strategy of Davies, Swann, and Campey with Gram-Schmidt orthogonalization dscg (dscg.f) calls subroutine lineg (lineg.f) subroutine grsmd (grsmd.f) - DSCP Strategy of Davies, Swann, and Campey with Palmer orthogonalization dscp (dscp.f) calls subroutine linep (linep.f) subroutine palm (palm.f) - POWE Powell's strategy of conjugate directions powe (powe.f) calls - DFPS Davidon, Fletcher, Powell strategy (Variable metric) dfps (dfps.f) calls subroutine seth (seth.f) subroutine grad (grad.f) function updot (updot.f) calls dot (dot.f) function dot (dot.f) 415

416

Appendix C - SIMP simp (simp.f) calls - COMP comp (comp.f) calls - EVOL evol (evol.f) calls - KORR korr2 (korr2.f) calls







Simplex strategy of Nelder and Mead Complex strategy of M. J. Box Two membered evolution strategy function z (included in evol.f) Multimembered evolution strategy function zulass (included in korr2.f) function gaussn (included in korr2.f) function bletal (included in korr2.f) subroutine pruefg (included in korr2.f) subroutine speich (included in korr2.f) subroutine mutati (included in korr2.f) subroutine umspei (included in korr2.f) subroutine minmax (included in korr2.f) subroutine gnpool (included in korr2.f) subroutine abscha (included in korr2.f) subroutine drehng (included in korr2.f) Additionally, FORTRAN function sources of the 50 test problems are included: { ZIELFU(N,X) one objective function with a computed GOTO for 50 entries. { RESTRI(J,N,X) one constraints function with a computed GOTO for 50 entries and J as current number of the single restriction. No runtime package is provided for this set, however. C sources for all strategies mentioned above and C sources for the 50 test problems (GRUP with option REKO is missing since it has become one special case within KORR). A set of simple interfaces to run 13 of the above mentioned optimization routines with the above mentioned 50 test problems on a PC or workstation.

C.2 About the Program Disk The
oppy disk contains both FORTRAN and C sources for each of the strategies described in the book. All test problems presented in the catalogue of problems (see appendix A) exist as C code. A set of simple interfaces, easy to understand and to expand, combines the strategies and functions to OptimA, a ready for use program package. The programs are designed to run on a minimally con gured PC using a math-coprocessor or having an 80486 CPU and running the DOS or LINUX operating system. To accomplish semantic equivalence with the well tested original FORTRAN codes, all strategies have been translated via f2c, a Fortran-to-C converter of AT&T Bell Laboratories. All C codes can be compiled and linked via gcc (Gnu C compiler, version 2.4). Of course,

Running the C Programs

417

any other ANSI C compiler such as Borland C++ that supports 4-byte-integers should produce correct results as well. LINUX and gcc are freely available under the conditions of the GNU General Public License. Information about ordering the Gnu C compiler in the United States is available through the Free Software Foundation by calling 617 876 3296. All C programs should compile and run on any UNIX workstation having gcc or another ANSI C compiler installed.

C.3 Running the C Programs The following instructions are appropriate for installing and running the C programs on your PC or workstation. Installation as well as compilation and linking can be carried out automatically.

C.3.1 How to Install OptimA on a PC Using LINUX or on a UNIX Workstation

First, enter the directory where you want OptimA to be installed. Then copy the installation le via mtools by typing the command: mcopy a:install.sh .

If you don't have mtools, copy wb-1p?.tar from
oppy to workspace and untar it. The instruction sh install.sh

will copy the whole tree of directories from the disk to your local directory. The following directories and subdirectories will be created: fortran funct include lib rstrct strat util

To compile, link, and run OptimA go to the workbench directory and type make

to start a recursive compilation and linking of all C sources.

418

Appendix C

C.3.2 How to Install OptimA on a PC Under DOS

First, enter the directory where you want OptimA to be installed. The instruction a:INSTALL

or b:INSTALLB

will copy the whole tree of directories from the disk to your local directory. The same directories and subdirectories as mentioned above will be created. To compile, link, and run OptimA go to the workbench directory and type mkOptimA

to start a recursive compilation and linking of all C sources. This will take a while, depending on how fast your machine works.

C.3.3 Running OptimA

After the successful execution of make or mkOptimA,respectively, the executable le OptimA is located in the subdirectory bin. Here you can run the program package by issuing the command OptimA

First, the program will list the available strategies. After choosing a strategy by typing its number, a list of test problems is displayed. Type a number or continue the listing by hitting the return key. Depending on the method and the problem, the program will ask for the parameters to con gure the strategy. Please refer to Chapter 6 and Appendix A to choose appropriate values. Of course, you are free to de ne your own parameter values, but please remember that the behavior of each strategy strongly depends on its parameter settings. Warnings during the process will inform the user of inappropriate parameter de nitions or abnormal program behavior. For example, the message timeout reached warns the user that the strategy may nd a better result if the user de ned maximal time were set to a larger value. The strategies COMP, EVOL, and KORR will try at most ve restarts after the rst timeout occurred. If a strategy that can process unrestricted problems only is applied to a restricted problem, a warning will be displayed, too. After the acknowledgement of this message by hitting the return key, the user can choose another function.

C.4 Description of the Programs The following pages brie
y describe the programs on which this package is based. A short description of how to incorporate self-de ned problem functions to OptimA follows.

Description of the Programs

419

The directory FORTRAN lists all the original codes described in the book. The reader may write his own interfaces to these programs. For further information please refer to the C sources or to Schwefel (1980, 1981). All C source codes of the strategies have been translated from FORTRAN to C via f2c. Some modi cations in the C sources were done to gain higher portability and to achieve a homogeneous program behavior. For example, all strategies are minimizing, use standard output functions, and perform operations on the same data types. All modi cations did not change the semantics of any strategy. To each optimization method a dialogue interface has been added. Here the strategy's speci c parameter de nition takes place. In the comments within the program listings the meaning and usage of each parameter is brie
y described. All names of the dialogue interfaces end with the sux \ mod.c." The strategies together with the interfaces are listed in the directory named strat. The whole catalogue of problems (see Appendix A) has been coded as C functions. They are collected in the subdirectory funct. The problems 2.29 to 2.50 (see Appendix A) are restricted. Therefore, constraints functions to these problems were written and listed in directory rstrct. Because in some problems the number of constraints to be applied depends on the dimension of the function to be optimized, this number has to be calculated. This task is performed by the programs with pre x \rsn ." The evaluation of the constraint itself is done in the modules with pre x \rst ." A restriction holds if its value is negative. All strategies perform operations on vectors of varying dimensions. Therefore a set of tools to allocate and to de ne vectors is compiled in the package vec util which is located in the subdirectory util. The procedures from this package are used only in the dialogue interfaces. All other programs perform operations on vectors as if they would use arrays of arbitrary but xed length. The main program \OptimA.c" performs only initialization tasks and runs the dialogue within which the user can choose a strategy and a function number. The strategies and functions are listed in tables, namely \func tab.c" and \strt tab.c." If the user wants to incorporate new problems to OptimA the table \func tab.c" has to be extended. This task is relatively simple for a programmer with little C knowledge if he follows the next instructions carefully.

C.4.1 How to Incorporate New Functions

The following template is typical for every function de nition: #include "f2c.h" #include "math.h" doublereal

probl_2_18(int n,doublereal *x)

420

Appendix C { return(0.26*(x0]*x0] + x1]*x1])-0.48*x0]*x1] ) }

Please add your own function into the directory funct. Here you will nd the le \func tab.c." Include the formal description of your problem into this table. A typical template looks like: { 5, rs_nm_x_x, restr_x_x, "Problem x_x (restricted problem):\n\t x1]+x2]+... ", probl_x_x },

with the data type de nition: struct functions { long int long int doublereal char* name doublereal

dim (*rs_num)() (*restrictions)() (*function)()

/* /* /* /* /* /*

Problem's dimension Calculates the number of constraints Constraints function Mathem. description Objective function

*/ */ */ */ */ */

} typedef struct functions funct_t




The rst item denotes the number of dimensions of the problem. A problem with variable dimension will be denoted by a ;1. In this case the program should inquire the dimension from the user.




The second entry denotes the function that calculates the numbers of constraints to be applied to the problem. If no constraints are needed a NULL pointer has to be inserted.




The next line will be displayed to the user during an OptimA session. This string provides a short description of the problem, typically in mathematical notation.




The last item is a function-pointer to the objective function.

Please do not add a new formal problem description into the func tab behind the last table entry. The latter denotes the end of the table and should not be displaced.

Examples

421

To inform all problems of the new function, its prototype must be included into the header le func names.h. As a last step the Makefile has to be extended. The lists FUNCTSRCS and FUNCTOBJS denote the les that make up the list of problems. These lists have to be extended by the lename of your program code. Now step back to the directory C and issue the command make or mkOptimA, respectively, to compile \OptimA." Restrictions can be incorporated into OptimA like functions. Every C code from the directory rstrct can be taken as template. The name of the constraints function and the name of the function that calculates the number of constraints has to be included in the formal problem description.

C.5 Examples Here two examples of how OptimA works in real life will be presented. The rst one describes an application of the multimembered evolution strategy KORR to the corridor model (problem 2.37, function number 32). The second example demonstrates a batch run. The batch mode enables the user to apply a set of methods to a set of functions in one task.

C.5.1 An Application of the Multimembered Evolution Strategy to the Corridor Model

After calling OptimA and choosing problem 2.37 by typing 32, a typical dialogue will look like: Multimembered evolution strategy applied to function: Problem 2.37 (Corridor model) (restricted problem): Sum-xi],{i,1,n}]

Please enter the parameters for the algorithm: Dimension of the problem Number of restrictions

Number of parents Number of descendants Plus (p) or the comma (c) strategy Should the ellipsoid be able to rotate (y/n) You can choose under several recombination types:

: 3 : 7

: : : :

10 100 c y

422

Appendix C

1 No recombination 2 Discrete recombination of pairs of parents 3 Intermediary recombination of pairs of parents 4 Discrete recombination of all parents 5 Intermediary recombination of all parents in pairs Recombination type for the parameter vector Recombination type for the sigma vector Recombination type for the alpha vector Check for convergence after how many generations (> 2*Dim.) Maximal computation time in sec. Lower bound to step sizes, absolute Lower bound to step sizes, relative Parameter in convergence test, absolute Parameter in convergence test, relative Common factor used in step-size changes (e.g. 1) Standard deviation for the angles of the mutation ellipsoid (degrees) Number of distinct step-sizes Initial values of the variables : 0 0 0 Initial step lengths : 1 1 1

: : : : : : : : : :

2 3 1 10 30 1e-6 1e-7 1e-6 1e-7 1

: 5.0 : 3

Common factor used in step-size changes : 0.408248 Individual factor used in step-size changes : 0.537285 Starting at : F(x) = 0

Time elapsed

: 18.099276

Minimum found : -300.000000 at point : 99.999992 100.000000 99.999992 Current best value of population: -300.000000

C.5.2 OptimA Working in Batch Mode

OptimA also supports a batch mode option. This option was introduced to enable a user to test the behavior of any strategy by varying parameter settings automatically. Of

Examples

423

course, any function or method may be changed during a run, as well. The batch le that will be processed should contain the list of input data you would type in manually during a whole session in non-batch mode. OptimA in batch mode suppresses the listing of the strategies and functions. That reduces the output a lot and makes it better readable. A typical batch run looks like: OptimA -b < bat_file > results

With a \bat file" like: 8 1 100.100 0.98e-6 0.0e+0 5 5 1 1 0.8e-6 0.8e-6 0.111 0.111 y

the le \results" may look like: Method # : 8 Function # : 1 DFPS strategy (Variable metric) applied to function: Problem 2.1 (Beale): (1.5-x*(1-y))^2 + (2.25-x*(1-y^2))^2 + (2.625-x*(1-y^3))^2 Dimension of the problem : 2 Maximal computation time in sec. : 100.100000 Accuracy required : 9.8e-07 Expected value of the objective function at the optimum : 0 Initial values of the variables : 5 5 Initial step lengths : 1 1 Lower bounds of the step lengths :

424

Appendix C

8e-07 8e-07 Initial step lengths for construction of derivatives : 0.111 0.111 Starting at : F(x) = 403069 Time elapsed

: 0.033332

Minimum found : 0.000000 at point : 3.000000 0.500000

Both examples have been run on a SUN SPARC S10/40 workstation. The
oppy disk included into this book may not be copied, sold, or redistributed without the permission of John Wiley & Sons, Inc., New York.

Index

425

Index Aarts, E.H.L., 161 Abadie, J., 17, 24 Abe, K., 239 Ablay, P., 163 Absolute minimum, see global minimum Accuracy of approximation, 26, 27, 29, 32, 38, 41, 70, 76, 78, 81, 91, 92, 94, 116, 146, 167, 168, 173, 175, 206{208, 213, 214, 235 Accuracy of computation, 12, 14, 32, 35, 54, 57, 66, 67, 71, 78, 81, 83, 88, 89, 99, 112{114, 145, 170, 173{ 175, 206, 209, 236, 329 Ackley, D.H., 152 Adachi, N., 77, 81, 82 Adams, R.J., 96 Adaptation, 5, 6, 9, 100, 102, 105, 142, 147, 152 Adaptive step size random search, 96, 97, 200 AESOP program package, 68 Ahrens, J.H., 116 AID program package, 68 Aizerman, M.A., 90 Akaike, H., 66, 67, 203 Alander, J.T., 152, 246 Aleksandrov, V.M., 95 Algebra, 5, 14, 41, 69, 75, 239 Alland, A., Jr., 244 Allen, P., 102 Allometry, 243 Allowed region, see feasible region Altman, M., 68 Amann, H., 93 Analogue computers, 12, 15, 65, 68, 89, 99, 236

Analytic optimization, see indirect optimization Anders, U., 246 Anderson, N., 35 Anderson, R.L., 91 Andrews, H.C., 5 Andreyev, V.O., 94 Animats, 103 Anscombe, F.J., 101 Antonov, G.E., 90 Aoki, M., 23, 93 Apostol, T.M., 17 Appelbaum, J., 48 Applications, 48, 53, 64, 68, 69, 99, 151, 245{246 Approximation problems, 5, 14, see also sum of squares minimization Archer, D.H, 48 Arrow, K.J., 17, 18, 165 Arti cial intelligence, 102, 103 Arti cial life, 103 Asai, K., 94 Ashby, W.R., 9, 91, 100, 105 Atmar, J.W., 151 Automata, 6, 9, 44, 48, 94, 99, 102 Avriel, M., 29, 31, 33 Awdejewa, L.I., 18 Axelrod, R., 21 Azencott, R., 161 Bach, H., 23 Back, T., 118, 134, 147, 151, 155, 159, 245, 246, 248 Baer, R.M., 67 Balakrishnan, A.V., 11, 18 Balas, E., 18 Balinski, M.L., 19

426 Banach, S., 10 Bandler, J.W., 48, 115 Banzhaf, W., 103 Bard, Y., 78, 83, 205 Barnes, G.H., 233, 239 Barnes, J.G.P., 84 Barnes, J.L., 102 Barr, D.R., 241 Barrier penalty functions (barrier methods), 16, 107 Bass, R., 81 Bauer, F.L., 84 Bauer, W.F., 93 Beale, E.M.L., 18, 70, 84, 166, 327, 346 Beamer, J.H., 26, 29, 39 Beckman, F.S., 69 Beckmann, M., 19 Behnken, D.W., 65 Beier, W., 105 Beightler, C.S., 1, 23, 27, 32, 38, 87 Bekey, G.A., 12, 65, 89, 95, 96, 98, 99 Belew, R.K., 152 Bell, D.E., 20 Bell, M., 44, 178 Bellman, R.W., 11, 38, 102 Beltrami, E.J., 87 Bendin, F., 248 Berg, R.L., 101 Berlin, V.G., 90 Berman, G., 29, 39 Bernard, J.W., 48 Bernoulli, Joh., 2 Bertram, J.E., 20 Bessel function, 129, 130 Beveridge, G.S.G., 15, 23, 28, 32, 37, 64, 65 Beyer, H.-G., 118, 134, 149, 159 Biasing, 98, 156, 174 Biggs, M.C., 76 Binary optimization, 18, 247 Binomial distribution, 7, 108, 243 Bionics, 99, 102, 105, 238 Birkho, G., 48 Bisection method, 33, 34

Index Bjorck,  A., 35 Blakemore, J.W., 23 Bledsoe, W.W., 239 Blind random search, see pure random search Blum, J.R., 19, 20 Boas, A.H., 26 Bocharov, I.N., 89, 90 Boltjanski, W.G. (Boltjanskij, V.G.), 18 Boltzmann, L., 160 Bolzano method, 33, 34, 38 Booker, L.B., 152 Booth, A.D., 67, 329 Booth, R.S., 27 Boothroyd, J., 33, 77, 178 Born, J., 118, 149 Borowski, N., 98, 240 Bossert, W.H., 146 Bourgine, P., 103 Box, G.E.P., 6, 7, 65, 68, 69, 89, 101, 115, see also EVOP method Box, M.J., 17, 23, 28, 54, 56{58, 61, 68, 89, 115, 174, 332, see also complex strategy Boxing in the minimum, 28, 29, 32, 36, 41, 56, 209 Brachistochrone problem, 11 Bracken, J., 348 Brajnes, S.N., 102 Bram, J., 27 Branch and bound methods, 18 Brandl, V., 93 Branin, F.H., Jr., 88 Braverman, E.M., 90 Bremermann, H.J., 100, 101, 105, 238 Brent, R.P., 23, 27, 34, 35, 74, 84, 88, 89, 174 Brocker, D.H., 95, 98, 99 Broken rational programming, 20 Bromberg, N.S., 89 Brooks, S.H., 58, 87, 89, 91{95, 100, 174 Brown, K.M., 75, 81, 84 Brown, R.R., 66 Broyden, C.G., 14, 77, 81{84, 172, 205

Index Broyden-Fletcher-Shanno formula, 83 Brudermann, U., 246 Brughiera, P., 88 Bryson, A.E., Jr., 68 Budne, T.A., 101 Buehler, R.J., 67, 68 Bunny-hop search, 48 Burkard, R.E., 18 Burt, D.A., 48 Calculus of observations, see observational calculus Campbell, D.T., 102 Campey, I.G., 54, see also DSC strategy Campos, I., 248 Canon, M.D., 18 Cantrell, J.W., 70 Carroll, C.W., 16, 57, 115 Cartesian coordinates, 10 Casey, J.K., 68, 89 Casti, J., 239 Catalogue of problems, 110, 205, 325{366 Cauchy, A., 66 Causality, 237 Cea, J., 23, 47, 68 Cembrowicz, R.G., 246 C erny, V., 160 Chambliss, J.P., 81 Chandler, C.B., 48 Chandler, W.J., 239 Chang, S.S.L., 11, 90 Charalambous, C., 115 Chatterji, B.N. and Chatterjee, B., 99 Chazan, D., 239 Cherno, H., 75 Chichinadze, V.K., 88, 91
2 distribution, 108 Cholesky, matrix decomposition, 14, 75 Chromosome mutations, 106, 148 Circumferential distribution, 95{97, 109 Cizek, F., 106 Clayton, D.G., 54 Clegg, J.C., 11 Cochran, W.G., 7

427 Cockrell, L.D., 93, 99 Cohen, A.I., 70 Cohn, D.L., 100 Collatz, L., 5 Colville, A.R., 68, 174, 175, 339 Combinatorial optimization, 152 Complex strategy, 17, 61{65, 89, 115, 177, 179, 185, 190, 191, 201, 202, 210, 212, 213, 216, 217, 228{230, 232, 327, 341, 346, 357, 361{363, 365, 366 Computational intelligence, 152 Computer-aided design (CAD), 5, 6, 23 Computers, see analogue, digital, hybrid, parallel, and process computers Concave, see convex Conceptual algorithms, 167 Condition of a matrix, 67, 180, 203, 242, 326 Conjugate directions, 54, 69, 74, 82, 88, 170{172, 202, see also Powell strategy Conjugate gradients, 38, 68, 69, 77, 81, 169{172, 204, 235, see also Fletcher-Reeves strategy Conrad, M., 103 Constraints, 8, 12, 14{18, 24, 44, 48, 49, 57, 62, 87, 90{93, 105, 107, 115, 119, 134, 150, 176, 212{214, 216, 236 Constraints, active, 17, 44, 62, 116, 118, 213, 215 Constraints satisfaction problem (CSP), 91 Contour tangent method, 39 Control theory, 9, 11, 18, 23, 70, 88, 89, 99, 112 Convergence criterion, 113{114, 145{146, see also termination of the search Converse, A.O., 23 Convex, 17, 34, 39, 47, 66, 101, 166, 169, 236, 239 Cooper, L., 23, 38, 48, 87

428 Coordinate strategy, 41{44, 47, 48, 67, 87, 100, 164, 167, 172, 177, 200, 202{204, 207, 209, 228{230, 233, 327, 332, 339, 340, 362, 363, see also Fibonacci division, golden section, and Lagrangian interpolation Coordinate transformation, 241 Cornick, D.E., 70 Correlation, 118, 240, 241, 243, 246 Corridor model objective function, 110, 116, 120, 123, 124, 134{142, 215, 231, 232, 351, 352, 361, 364, 365 Cost of computation, 12, 38, 39, 64, 66, 74, 89, 90, 92, 168, 170, 179, 204, 230, 232, 234, see also rate of convergence Cottrell, B.J., 67 Courant, R., 11, 66 Covariances, 155, 204, 240, 241 Cowdrey, D.R., 93 Cox, D.R., 7 Cox, G.M., 7 Cragg, E.E., 70 Created response surface technique, 16, 57 Creeping random search, 94, 95, 99, 100, 236, 237 Crippen, G.M., 89 Criterion of merit, 2, 7 Crockett, J.B., 75 Crossover, 154 Crowder, H., 70 Cryer, C.W., 43 Cubic interpolation, 34, 37, see also Lagrangian and Hermitian interpolation Cullum, C.D., Jr., 18 Cullum, J., 83 Curry, H.B., 66, 67 Curse of dimensions, Bellman's, 38 Curtis, A.R., 66 Curtiss, J.H., 93 Curve tting, 35, 64, 84, 151, 246

Index Cybernetics, 9, 101, 102, 322 Dambrauskas, A.P., 58, 64 Daniel, J.W., 15, 23, 68, 70 Dantzig, G.B., 17, 57, 88, 166 Darwin, C., 106, 109, 244 Davidon, W.C., 77, 81, 82, 170 Davidon-Fletcher-Powell strategy, see DFP strategy Davidor, Y., 152 Davies, D., 23, 28, 54, 56, 57, 76, 81, see also Davies-Swann-Campey strategy Davies, M., 84 Davies, O.L., 7, 58, 68 Davies-Swann-Campey strategy, see DSC strategy Davis, L., 152 Davis, R.H., 70 Davis, R.S., 66, 89 Davis, S.H., Jr., 23 Day, R.G., 97 Debye series, 130 Decision theory, 94 Decision tree methods, 18 De Graag, D.P., 95, 98 De Jong, K., 152 Dekker, T.J., 34 Demyanov, V.F., 11 Denn, M.M., 11 Dennis, J.E., Jr., 75, 81, 84 Derivative-free methods, 15, 40, 80, 83, 172, 174, see also direct search strategies Derivatives, numerical evaluation of, 19, 23, 35, 66, 68, 71, 76, 78, 81, 83, 95, 97, 170{172 Descendants, number of, 126, 142{144 Descent, theory of, 100, 109 Design and analysis of experiments, 6, 58, 65, 89 D'Esopo, D.A., 41 DeVogelaere, R., 44, 178 DFP strategy, 77{78, 83, 97, 170{172, 243

Index DFP-Stewart strategy, 78{81, 177, 178, 184, 189, 195, 200, 201, 209, 210, 219, 228{231, 337, 341, 343, 363, 364 Diblock search, 33 Dichotomous search, 27, 29, 33, 39 Dickinson, A.W., 93, 98, 174 Dieter, U., 116 Dierential calculus, 2, 11 Digital computers, 6, 10{12, 14, 15, 32, 33, 92, 99, 110, 173, 236 Dijkhuis, B., 37 Dinkelbach, W., 17 Diploidy, 106, 148 Direct optimization, 13{15, 20 Direct search strategies, 40{65, 68, 90 Directed random search, 98 Discontinuity, 13, 23, 25, 42, 88, 91, 116, 176, 211, 214, 231, 236, 341, 349 Discovery, 2 Discrete distribution, 110, 243 Discrete optimization, 11, 18, 32, 39, 44, 64, 88, 91, 108, 152, 160, 243, 247 Discrete recombination, 148, 153, 156 Discretization, see parameterization Divergence, 35, 76, 169 Dixon, L.C.W., 15, 23, 29, 34, 35, 58, 71, 76, 78, 81{83 Dobzhansky, T., 101 Dominance and recessiveness, 101, 106, 148 Dowell, M., 35 Draper, N.R., 7, 65, 69 Drenick, R.F., 48 Drepper, F.R., 103, 246 Drucker, H., 61 DSC strategy, 54{57, 74, 89, 177, 183, 188, 194, 200{202, 209, 228{230, 362, 363 Dubovitskii, A.Ya., 11 Dueck, G., 98, 164 Dun, R.J., 14 Dunham, B., 102 Dvoretzky, A., 20

429 Dynamic optimization, 7, 9, 10, 48, 64, 89{91, 94, 99, 102, 245, 248 Dynamic programming, 11, 12, 18, 149 Ebeling, W., 102, 163 Edelbaum, T.N., 13 Edelman, G.B., 103 Eectivity of a method, see robustness Eciency of a method, see rate of convergence Eigen, M., 101 Eigenvalue problems, 5 Eigenvalues of a matrix, 76, 83, 326 Eisenberg, M.A., 239 Eldredge, N., 148 Elimination methods, see interval division methods Elitist strategy, 157 Elkin, R.M., 44, 66, 67 Elliott, D.F., 83 Ellipsoid method, 166 Emad, F.P., 98 Emery, F.E., 48, 87 Engelhardt, M., 20 Engeli, M., 43 Enumeration methods, see grid method Epigenetic apparatus, 153, 154 Equation, dierential, 15, 65, 68, 93, 246, 345, 346 Equations, system of, 5, 13, 14, 23, 39, 65, 66, 75, 83, 93, 172, 235, 336 Equidistant search, see grid method Erlicki, M.S., 48 Ermakov, S., 19 Ermoliev, Yu., 19, 90 Errors, computational, 47, 174, 205, 209, 210, 212, 219, 228, 229, 236 Euclid of Alexandria, 32 Euclidean norm, 167, 335 Euclidean space, 10, 24, 49, 97 Euler, L., 2, 15 Even block search, 27 Evolution, cultural, 244 Evolution, organic, 1, 3, 100, 102, 105, 106, 109, 142, 153, 237, 238

430 Evolution strategy, 3, 6, 7, 16, 105{151, 168, 173, 175, 177, 179, 200, 203, 210, 213, 219, 228{230, 232{235, 248, 333, 337, 350, 354, 355, 359, 361, 364, 365, 367, 413, see also two membered and multimembered evolution strategies Evolution strategy, asynchronous parallel, 248 Evolution strategy, parallel, 248 Evolution strategy, 1=5 success rule, 110, 112, 114, 116, 118, 142, 200, 213{ 215, 237, 349, 361 Evolution strategy (1+1), 105{119, 125, 163, 177, 185, 191, 200, 203, 212, 213, 216, 217, 228, 231{233, 328, 349, 363 Evolution strategy (1+), 123, 134, 145 Evolution strategy (1 , ), 145 Evolution strategy (10 , 100), 177, 186, 191, 200, 203, 211{215, 217, 228, 231{233 Evolution strategy (+1), 119 Evolution strategy (+), 119 Evolution strategy ( , ), 119, 145, 148, 238, 244, 248 Evolution strategy (    ), 247 Evolution, synthetic theory, 106 Evolutionary algorithms, 151, 152, 161 Evolutionary computation, 152 Evolutionary operation, see EVOP method Evolutionary principles, 3, 100, 106, 118, 146, 244 Evolutionary programming, 151 Evolutionism, 244 EVOP method, 6, 7, 9, 64, 68, 69, 89, 101 Experimental optimization, 6{9, 36, 44, 68, 89, 91, 92, 95, 110, 113, 245, 247, see also design and analysis of experiments Expert system, 248 Extreme value controller, see optimizer Extremum, see minimum

Index Faber, M.M., 18 Fabian, V., 20, 90 Factorial design, 38, 58, 65, 68, 246 Faddejew, D.K. and Faddejewa, W.N., 27, 67, 240 Fagiuoli, E., 96 Falkenhausen, K. von, 246 Favreau, R.F., 95, 96, 98, 100 Feasible region, 8, 9, 12, 16, 17, 25, 101 Feasible region, not connected, 217, 239, 360 Feasible starting point, search for, 62, 91, 115 Feistel, R., 102, 163 Feldbaum, A.A., 6, 9, 88{90, 99 Fend, F.A., 48 Fiacco, A.V., 16, 76, 81, 115, see also SUMT method Fibonacci division, 29{32, 38, 177, 178, 181, 187, 192, 200, 202 Fielding, K., 83 Finiteness of a sequence of iterations, 68, 166, 172 Finkelstein, J.J., 18 Fisher, R.A., 7 Fletcher, R., 24, 38, 68{71, 74, 77, 80{84, 97, 170, 171, 204, 205, 335, 349 Fletcher-Powell strategy, see DFP strategy Fletcher-Reeves strategy, 69, 70, 78, 170{ 172, 204, 233, see also conjugate gradients Flood, M.M., 68, 89 Floudas, C.A., 91 Fogarty, L.E., 68 Fogel, D.B., 151 Fogel, L.J., 102, 105, 151 Forrest, S., 152 Forsythe, G.E., 34, 66, 67 Fox, R.L., 23, 34, 205 Frankhauser, P., 246 Frankovic, B., 9 Franks, R., 95, 96, 98, 100 Fraser, A.S., 152

Index Friedberg, R.M., 102, 152 Friedmann, M., 41 Fu, K.S., 94, 99 Function space, 10 Functional analysis theory, 11 Functional optimization, 10{12, 15, 23, 54, 68, 70, 85, 89, 90, 151, 174 Furst, H., 98 Gaede, K.W., 8, 108, 144 Gaidukov, A.L., 98 Gal, S., 31 Galar, R., 102 Game theory, 5, 6, 20 Gar nkel, R.S., 18 Gauss, C.F., 41, 84 Gauss-Newton method, 84 Gauss-Seidel strategy, see coordinate strategy Gaussian approximation, see sum of squares minimization Gaussian distribution, see normal distribution Gaussian elimination, 14, 75, 172 Gaviano, M., 96 Gelatt, C.D., 160 Gelfand, I.M., 89 Gene duplication and deletion, 247 Gene pool, 146, 148 Generalized least squares, 84 Genetic algorithms, 151{160 Genetic code, 153, 154, 243 Genotype, 106, 152, 153, 157 Georion, A.M., 24 Geometric programming, 14 Gerardin, L., 105 Gersht, A.M., 90 Gessner, P., 11 Gibson, J.E., 88, 90 Gilbert, E.G., 68, 89 Gilbert, H.D., 90, 98 Gilbert, P., 239 Gill, P.E., 81 Ginsburg, T., 43, 69

431 Girsanov, I.V., 11 Glass, H., 48, 87 Gla , K., 105 Glatt, C.R., 68 Global convergence, 39, 88, 94, 96, 98, 117, 118, 149, 216, 217, 238, 239 Global minimum, 24{26, 90, 168, 329, 344, 348, 356, 357, 359, 360 Global optimization, 19, 29, 84, 88{91, 236, 244 Global penalty function, 16 Glover, F., 162, 163 Gnedenko, B.W., 137 Goldberg, D.E., 152, 154 Golden section, 32, 33, 177, 178, 181, 187, 192, 200, 202 Goldfarb, D., 81 Goldfeld, S.M., 76 Goldstein, A.A., 66, 67, 76, 81, 88 Golinski, J., 92 Goll, R., 244 Golub, G.H., 57, 84 Gomory, R.E., 18 Gonzalez, R.S., 95 Gorges-Schleuter, M., 159, 247 Gorvits, G.G., 174 GOSPEL program package, 68 Goto, K., 82 Gottfried, B.S., 23 Gould, S.J., 148 Gradient strategies, 6, 15, 19, 37, 40, 65{ 69, 88{90, 94, 95, 98, 166, 167, 171, 172, 174, 235 Gradient strategies, second order, see Newton strategies Gradstein, I.S., 136 Gram-Schmidt orthogonalization, 48, 53, 54, 57, 69, 177, 178, 183, 188, 194, 201, 202, 209, 229, 230, 362 Gran, R., 88 Graphical methods, 20 Grasse, P.P., 243 Grassmann, P., 100 Grauer, M., 20

432 Graves, R.L., 23 Great deluge algorithm, 164 Greedy algorithm, 162, 248 Greenberg, H., 18 Greenberg, H.-J., 162 Greenstadt, J., 70, 76, 81, 83, 326 Grefenstette, J.J., 152 Grid method, 12, 26, 27, 32, 38, 39, 65, 92, 93, 100, 149, 168, 236 GROPE program package, 68 Guilfoyle, G., 38 Guin, J.A., 64 Gurin, L.S., 89, 97, 98 Hadamard, J., 66 Hadley, G., 12, 17, 166 Haeckel strategy, 163 Haefner, K., 103 Hague, D.S., 68 Haimes, Y.Y., 10 Hamilton, P.A., 77 Hamilton, W.R., 15 Hammel, U., 245, 248 Hammer, P.L., 19 Hammersley, J.M., 93 Hamming clis, 154, 155 Hancock, H., 14 Handscomb, D.C., 93 Hansen, P.B., 239 Haploidy, 148 Harkins, A., 89 Harmonic division, 32 Hartmann, D., 151, 246 Haubrich, J.G.A., 68 Heckler, R., 246, 248 Heidemann, J.C., 70 Heinhold, J., 8, 108, 144 Hemstitching, 16 Henn, R., 20 Herdy, M., 164 Hermitian interpolation, 37, 38, 69, 77, 88 Herschel, R., 99 Hertel, H., 105

Index Hesse, R., 88 Hessian matrix (Hesse, L.O.), 13, 69, 75, 169, 170 Hestenes, M.R., 11, 14, 69, 70, 81, 172 Heuristic methods, 7, 18, 40, 88, 91, 98, 102, 162, 173 Heusener, G., 245 Hext, G.R., 57, 58, 64, 68, 89 Heydt, G.T., 93, 98, 99 Heynert, H., 105 Hilbert, D., 10, 11 Hildebrand, F.B., 66 Hill climbing strategies, 23, 85, 87 Hill, I.D., 33, 178 Hill, J.C., 88, 90 Hill, J.D., 94 Himmelblau, D.M., 23, 48, 81, 87, 174, 176, 229, 339 Himsworth, F.R., 57, 58, 64, 68, 89 History vector method, 98 Hit-or-miss method, 93 Ho, Y.C., 68 Hock, W., 174 Hodanova, D., 106 Homann, U., 23, 74 Homeister, F., 151, 234, 246, 248 Ho
er, A., 151, 246 Hofmann, H., 23, 74 Holland, J.H., 105, 152, 154 Hollstien, R.B., 152 Holst, W.R., 67 Homeostat, 9, 91, 100 Hoo, S.K., 88 Hooke, R., 44, 87, 90, 92 Hooke-Jeeves strategy, 44{48, 87, 90, 177, 178, 182, 188, 193, 200, 202, 210, 228, 230, 233, 332, 339 Hopper, M.J., 178 Horner, computational scheme of, 14 Horst, R., 91 Hoshino, S., 57, 81 Hotelling, H., 36 House, F.R., 77 Householder, A.S., 27, 75

Index Householder method, 57 Houston, B.F., 48 Howe, R.M., 68 Hu, T.C., 18 Huang, H.Y., 70, 78, 81, 82 Huberman, B.A., 103 Huelsman, L.P., 68 Human, R.A., 48 Hull, T.E., 93 Human brain, 6, 102 Humphrey, W.E., 67 Hunter, J.S., 65 Hupfer, P., 92, 94, 98 Hurwicz, L., 17, 18, 165 Hutchinson, D., 61 Hwang, C.L., 20 Hybrid computers, 12, 15, 68, 89, 99, 236 Hybrid methods, 38, 162{164, 169 Hyperplane annealing, 162 Hyslop, J., 206 Idelsohn, J.M., 93, 94 Illiac IV, 239 Imamura, H., 89 Indirect optimization, 13{15, 27, 35, 75, 170, 235 Indusi, J.P., 87 In mum, 9 Information theory, 5 Integer optimization, 18, 247 Interior point method, 166 Intermediary recombination, 148, 153, 156 Interpolation methods, 14, 27, 33{38 Interval division methods, 27, 29{33, 41 Invention, 2 Inverse Hessian matrix, 77, 78 Inversion of a matrix, 76, 170, 175 Isolation, 106, 244 Iterative methods, 11, 13 Ivakhnenko, A.G., 102 Jacobi, C.G.J., 15 Jacobi method, 65, 326 Jacobian matrix, 16, 84

433 Jacobson, D.H., 12 Jacoby, S.L.S., 23, 67, 174 James, F.D., 33, 178 Janac, K., 90 Jarratt, P., 34, 35, 84 Jarvis, R.A., 91, 93, 94, 99 Jeeves, T.A., 44, 84, 87, 90, 92, see also Hooke-Jeeves strategy Johannsen, G., 99 John, F., 166 John, P.W.M., 7 Johnk, M.D., 115 Johnson, I., 38 Johnson, M.P., 81 Johnson, S.M., 31, 32 Jones, A., 84 Jones, D.S., 81 Jordan, P., 109 Kamiya, A., 100 Kammerer, W.J., 70 Kantorovich, L.V., 66, 67 Kaplan, J.L., 64 Kaplinskii, A.I., 90 Kappler, H., 18, 166 Karmarkar, N., 166, 167 Karnopp, D.C., 93, 94, 96 Karp, R.M., 239 Karplus, W.I., 12, 89 Karr, C.L., 160 Karreman, H.F., 11 Karumidze, G.V., 94 Katkovnik, V.Ya., 88, 90 Kaupe, A.F., Jr., 39, 44, 178 Kavanaugh, W.P., 95, 98, 99 Kawamura, K., 70 Keeney, R.E., 20 Kelley, H.J., 15, 68, 70, 81 Kempthorne, O., 7, 67, 68 Kenworthy, I.C., 69 Kesten, H., 20 Kettler, P.C., 82, 83 Khachiyan, L.G., 166, 167 Khovanov, N.V., 96, 102

434 Khurgin, Ya.I., 89 Kiefer, J., 19, 29, 31, 32, 178 Kimura, M., 239 King, R.F., 35 Kirkpatrick, S., 160 Kitajima, S., 94 Kivelidi, V.Kh., 89 Kiwiel, K.C., 19 Kjellstrom, G., 98 Klerer, M., 24 Klessig, R., 15, 70 Klimenko, E.S., 94 Klingman, W.R., 48, 87 Klockgether, J., 7, 245 Klotzler, R., 11 Kobelt, D., 246 Koch, H.W., 244 Kopp, R.E., 18 Korbut, A.A., 18 Korn, G.A., 12, 24, 89, 93, 99 Korn, T.M., 12, 89 Korst, J., 161 Kosako, H., 99 Kovacs, Z., 80, 179 Kowalik, J.S., 23, 42, 67{69, 84, 174, 334, 335, 345 Koza, J., 152 Krallmann, H., 246 Krasnushkin, E.V., 99 Krasovskii, A.A., 89 Krasulina, T.P., 20 Krauter, G.E., 246 Kregting, J., 93 Krelle, W., 17, 18, 166 Krolak, P.D., 38 Kuester, J.L., 18, 58, 179 Kuhn, H.W., 17, 166 Kuhn-Tucker theorem, 17, 166 Kulchitskii, O.Yu., 90 Kumar, K.K., 160 Kunzi, H.P., 17, 18, 20, 166 Kursawe, F., 102, 148, 245, 248 Kushner, H.J., 20, 90 Kussul, E., 101

Index Kwakernaak, H., 89 Kwasnicka, H. and Kwasnicki, W., 102 Kwatny, H.G., 90 Laarhoven, P.J.M. van, 161 Lagrange multipliers, 15, 17 Lagrange, J.L., 2, 15 Lagrangian interpolation, 27, 35{37, 41, 56, 64, 73, 80, 89, 101, 177, 182, 187, 193, 200, 202 Lam, L.S.-B., 100 Lance, G.M., 54 Land, A.H., 18 Lange-Nielsen, T., 54 Langguth, V., 89 Langton, C.G., 103 Lapidus, L., 68 Larichev, O.I., 174 Larson, R.E., 239 Lasdon, L.S., 70 Lattice search, see grid method Lau ermair, T., 162 Lavi, A., 23, 48, 93 Lawler, E.L., 160 Lawrence, J.P., 87, 98 Learning (and forgetting), 9, 54, 70, 78, 98, 101, 103, 162, 236 Least squares method, see sum of squares minimization LeCam, L.M., 102 Lee, R.C.K., 11 Lehner, K., 248 Leibniz, G.W., 1 Leitmann, G., 11, 18 Lemarechal, C., 19 Leon, A., 68, 89, 174, 337, 356 Leonardo of Pisa, 29 Lerner, A.Ja., 11 Lesniak, Z.K., 92 Lethal mutation, 115, 136, 137, 158 Levenberg, K., 66, 84 Levenberg-Marquardt method, 84 Levine, L., 65 Levine, M.D., 10

Index Levy, A.V., 70, 78, 81 Lew, A.Y., 96 Lew, H.S., 100 Lewallen, J.M., 174 Lewandowski, A., 20 Ley ner, U., 151, 246 Lilienthal, O., 238 Lill, S.A., 80, 178, 179 Lindenmayer, A., 103 Line search, 25{38, 42, 54, 66, 70, 71, 77, 89, 101, 167, 170, 171, 173, 180, 214, 228, see also interval division and interpolation methods Linear convergence, 34, 168, 169, 172, 173, 236, 365 Linear model objective function, 96, 124{ 127 Linear programming, 17, 57, 88, 100, 101, 151, 166, 212, 235, 353 Little, W.D., 93, 244 Lobac, V.P., 89 Local minimum, 13, 23{26, 88, 90, 329 Locker, A., 102 Log-normal distribution, 143, 144, 150 Loginov, N.V., 90 Lohmann, R., 164 Long step methods, 66 Longest step procedure, 66 Lootsma, F.A., 24, 81, 174 Lowe, C.W., 69, 101 Lucas, E., 32 Luce, A.D., 21 Luenberger, D.G., 18 Luk, A., 101 Lyvers, H.I., 16 MacDonald, J.R., 84 MacDonald, P.A., 48 Machura, M., 54, 179 MacLane, S., 48 MacLaurin, C., 13 Madsen, K., 35 Mamen, R., 81 Manderick, B., 152 Mandischer, M., 160

435 Mangasarian, O.L., 18, 24 Manner, R., 152 Marfeld, A.F., 6 Markwich, P., 246 Marquardt, D.W., 84 Marti, K., 118 Masters, C.O., 61 Masud, A.S.M., 20 Mathematical biosciences, 102 Mathematical optimization, 6{9 Mathematical programming, 15{17, 23, 85, see also linear, quadratic, and non-linear programming Mathematization, 102 Matthews, A., 76, 81 Matyas, J., 97{99, 240, 338 Maximum likelihood method, 8 Maximum, see minimum Maybach, R.L., 97 Mayne, D.Q., 12, 81 Maze method, 44 McArthur, D.S., 92, 94, 98 McCormick, G.P., 16, 67, 70, 76, 78, 81, 82, 88, 115, 348, see also SUMT method McGhee, R.B., 65, 68, 89, 93 McGlade, J.M., 102 McGrew, D.R., 10 McGuire, M.R., 239 McMillan, C., Jr., 18 McMurtry, G.J., 94, 99 Mead, R., 58, 84, 97, see also simplex strategy Medvedev, G.A., 89, 99 Meerkov, S.M., 94 Meissinger, H.F., 99 Meliorization, 1 Memory gradient method, 70 Meredith, D.L., 160 Merzenich, W., 101 Metropolis, N., 160 Meyer, J.-A., 103 Michalewicz, Z., 152, 159 Michel, A.N., 70

436 Michie, D., 102 Mickey, M.R., 58, 89, 95 Midpoint method, 33 Miele, A., 68, 70 Mi!in, R., 19 Migration, 106, 248 Miller, R.E., 239 Millstein, R.E., 239 Milyutin, A.A., 11 Minima and maxima, theory of, see optimality conditions Minimax concept, 26, 27, 31, 34, 92 Minimum, 8, 13, 16, 24, 36 Minimum
2 method, 8 Minot, O.N., 102 Minsky, M., 102 Miranker, W.L., 233, 239 Missing links, 1 Mitchell, B.A., Jr., 99 Mitchell, R.A., 64 Mixed integer optimization, 18, 164, 243 Mize, J.H., 18 Mlynski, D., 69, 89 Mockus, J.B., see Motskus, I.B. Model, internal (of a strategy), 9, 10, 28, 38, 41, 90, 169, 204, 231, 235{237 Model, mathematical (of a system), 7, 8, 65, 68, 160, 235 Modi ed Newton methods, 76 Moler, C., 87 Moment rosetta search, 48 Monro, S., 19 Monte-Carlo methods, 92{94, 109, 149, 160, 168 Moran, P.A.P., 101 More, J.J., 81, 179 Morgenstern, O., 6 Morrison, D.D., 84 Morrison, J.F., 334 Motskus, I.B., 88, 94 Motzkin, T.S., 67 Movshovich, S.M., 96 Mufti, I.H., 18 Mugele, R.A., 44

Index Muhlenbein, H., 163 Mulawa, A., 54, 179 Muller, M.E., 115 Muller, P.H., 98 Muller-Merbach, H., 17, 166 Multicellular individuals, 247 Multidimensional optimization, 2, 38, 85 Multimembered evolution strategy, 101, 103, 118{151, 153, 158, 235{248, 329, 333, 335, 344, 347, 355{357, 359, 360, 362, 363, 365, 366, 375, 413, see also evolution strategy ( , ) and (+) Multimodality, 12, 24, 85, 88, 157, 159, 239, 245, 248 Multiple criteria decision making (MCDM), 2, 20, 148, 245 Munson, J.K., 95 Murata, T., 44 Murray, W., 24, 76, 81, 82 Murtagh, B.A., 78, 82 Mutation, 3, 100{102, 106{108, 154, 155, 237 Mutation rate, 100, 101, 154, 237 Mutator genes, 142, 238 Mutseniyeks, V.A., 99 Myers, G.E., 70, 78, 81 Nabla operator, 13 Nachtigall, W., 105 Nag, A., 58 Nake, F., 49 Narendra, K.S., 94 Nashed, M.Z., 70 Neave, H.R., 116 Neighborhood model, 247 Nelder, J.A., 58, 84, 97 Nelder-Mead strategy, see simplex strategy Nemhauser, G.L., 18 Nenonen, L.K., 70 Network planning, 20 Neumann, J. von, 6 Neustadt, L.W., 11, 18 Newman, D.J., 39

Index Newton, I., 2, 14 Newton direction, 70, 75{77, 84 Newtonian interpolation, 27, 35 Newton-Raphson method, 35, 75, 76, 97, 167, 169{171 Newton strategies, 40, 71, 74{85, 89, 171, 235 Neyman, J., 102 Niching, 100, 106, 238, 248 Nicholls, R.L., 23 Nickel, K., 168 Niederreiter, H., 115 Niemann, H., 5 Nikolic, Z .J., 94 Nissen, V., 103 Nollau, V., 98 Non-linear programming, 17, 18, 166 Non-smooth or non-dierentiable optimization, 19 Nonstationary optimum, 248 Norkin, K.B., 88 Normal distribution, 7, 90, 94, 95, 97, 101, 108, 116, 120, 128, 153, 236, 240, 243 North, J.H., 102 North, M., 246 Numerical mathematics, 5, 27, 239 Numerical optimization, see direct optimization Nurminski, E.A., 19 Objective function, 2, 8 Observational calculus, 5, 7 Odd block search, 27 Odell, P.L., 20 Oettli, W., 18, 167 O'Hagan, M., 48, 87 Oi, K., 82 Oldenburger, R., 9 Oliver, L.T., 31 One dimensional optimization, 25{38, see also line search One step methods, see relaxation methods O'Neill, R., 58, 179

437 Ontogenetic learning, 163 Opacic, J., 88 Operations research, 5, 17, 20 Optimal control, see control theory Optimality conditions, 2, 13{15, 23, 167, 235 Optimality of organic systems, 99, 100, 105 Optimization, prerequisites for, 1 Optimization problem, 2, 5{8, 14, 20, 24 Optimizer, 9, 10, 48, 99, 248 Optimum, see minimum Optimum, maintaining (and hunting), see dynamic optimization Optimum gradient method, 66 Optimum principle of Bellman, 11, 12 Oren, S.S., 82 Ortega, J.M., 5, 27, 41, 42, 82, 84 Orthogonalization, see Gram-Schmidt and Palmer orthogonalization Osborne, M.R., 23, 42, 68, 69, 84, 174, 335, 345 Osche, G., 106, 119 Ostermeier, A., 118 Ostrowski, A.M., 34, 66 Overadaptation, 148 Overholt, K.J., 31{33, 178 Overrelaxation and underrelaxation, 43, 67 Owens, A.J., 102, 105, 151 Page, S.E., 155 Pagurek, B., 70 Palmer, J.R., 57, 178 Palmer orthogonalization, 57, 177, 178, 183, 188, 194, 202, 209, 230 Papageorgiou, M., 23 Papentin, F., 102 Parallel computers, 161, 163, 234, 239, 243, 245, 247, 248 Parameter optimization, 6, 8, 10{13, 15, 16, 20, 23, 105 Parameterization, 15, 151, 346 Pardalos, P.M., 91 Pareto-optimal, 20, 245

438 Parkinson, J.M., 61 Partan (parallel tangents) method, 67{69 Pask, G., 101 Path-oriented strategies, 98, 160, 236, 248 Patrick, M.L., 239 Pattern recognition, 5 Pattern search, see Hooke-Jeeves strategy Paviani, D.A., 87 Pearson, J.D., 38, 70, 76, 78, 81, 82, 205 Peckham, G., 84 Penalty function, 15, 16, 48, 49, 57, 207 Perceptron, 102 Peschel, M., 20 Peters, E., 163, 248 Peterson, E.L., 14 Phenotype, 106, 153{155, 157, 158 Pierre, D.A., 23, 48, 68, 95 Pierson, B.L., 82 Pike, M.C., 33, 44, 178 Pincus, M., 93 Pinkham, R.S., 93 Pinsker, I.Sh., 44 Pixner, J., 33, 178 Pizzo, J.T., 23, 67, 174 Plane, D.R., 18 Plaschko, P., 151, 246 Pleiotropy, 243 Pluznikov, L.N., 94 Polak, E., 15, 18, 70, 76, 77, 167, 169 Policy, 11 Polyak, B.T., 70 Polygeny, 243 Polyhedron strategies, see simplex and complex strategies Ponstein, J., 17 Pontrjagin, L.S., 18 Poor man's optimizer, 44 Population principle, 101, 119, 238 Posynomes, 14 Powell, D.R., 84 Powell, M.J.D., 57, 70, 71, 74, 77, 82, 84, 88, 97, 170, 202, 205, 335, 337, 349, see also DFP, DFP-Stewart, and Powell strategies

Index Powell, S., 18 Powell strategy, 69{74, 88, 163, 170{172, 177, 178, 183, 189, 195, 200, 202, 204, 209, 210, 219, 228{230, 327, 332, 339, 341, 343, 364 Poznyak, A.S., 90 Practical algorithms, 167 Predator-prey model, 247 Press, W.H., 115 Price, J.F., 76, 81, 88 Probabilistic automaton, 94 Problem catalogue, see catalogue of problems Process computers, 10 Projected gradient method, 57, 70 Proofs of convergence, 42, 47, 66, 77, 97, 167, 168 Propoi, A.I., 90 Prusinkiewicz, P., 103 Pseudo-random numbers, see random number generation Pugachev, V.N., 95 Pugh, E.L., 89 Pun, L., 23 Punctuated equilibrium, 148 Pure random search, 91, 92, 100, 237 Q-properties, 169, 170, 172, 179, 243 Quadratic convergence, 68, 69, 74, 76, 78, 81{83, 168, 169, 200, 202, 236 Quadratic interpolation, see Lagrangian and Hermitian interpolation Quadratic programming, 166, 233, 235 Quandt, R.E., 76 Quasi-Newton method, 37, 70, 76, 83, 89, 170, 172, 205, 233, 235, see also DFP and DFP-Stewart strategies Rabinowitz, P., 84 Raia, H., 20, 21 Rajtora, S.G., 82 Ralston, A., 27 Random direction, 20, 88, 90, 98, 101, 202 Random evolutionary operation, see REVOP method

Index Random exchange step, 88, 166 Random number generation, 115, 150, 210, 212, 217, 237 Random sequence, 87, 93 Random step length, 95, 96, 108 Random strategies, 3, 12, 19, 87{103, 105, 240 Random walk, 247 Randomness, 87, 91, 93, 237 Rank one methods, 82, 83, 172 Raphson, J., see Newton-Raphson method Rappl, G., 118 Raster method, see grid method Rastrigin, L.A., 93, 95, 96, 98, 99 Rate of convergence, 7, 38, 39, 64, 66, 67, 69, 90, 94{98, 101, 110, 118, 120{ 141, 167{169, 179{204, 217{232, 234, 236, 239, 240, 242, see also linear and quadratic convergence Rauch, S.W., 82 Rawlins, G.J.E., 152 Rayleigh-Ritz method, 15 Razor search, 48 Rechenberg, I., 6, 7, 97, 100, 105, 107, 118{120, 130, 142, 149, 164, 168, 172, 179, 231, 238, 245, 352 Recognition processes, 102 Recombination, 3, 101, 106, 146{148, 153{ 159, 186, 191, 200, 203, 204, 211{ 213, 215{217, 228, 231, 232, 240, 335, 355, 357, 363, 365, 366, see also discrete and intermediary recombination Reeves, C.M., 38, 69, 93, 170, 204, see also Fletcher-Reeves strategy References, 249{323 Regression, 8, 19, 84, 235, 246 Regression, non-linear, 84 Regula falsi (falsorum), 27, 34, 35, 39 Reid, J.K., 66 Rein, H., 100 Reinsch, C., 14 Relative minimum, 38, 42, 43, 66, 209

439 Relaxation methods, 14, 20, 41, 172, see also coordinate strategy Reliability, see robustness Repair enzymes, 142, 238 Replicator algorithm, 163 Restart of a search, 61, 67, 70, 71, 88, 89, 169, 201, 202, 205, 210, 219, 228{230, 362, 364 REVOP method, 101 Reynolds, O., 238 Rhead, D.G., 74 Rheinboldt, W.C., 5, 27, 41, 42, 82, 84 Ribi"ere, G., 70, 82 Rice, J.R., 57 Richardson, D.W., 155 Richardson, J.A., 58, 179 Richardson, M., 239 Riding the constraints, 16 Riedl, R., 102, 153 Ritter, K., 24, 70, 82, 88, 168 Rivlin, L., 48 Robbins, H., 19 Roberts, P.D., 70 Roberts, S.M., 16 Robots, 6, 9, 103 Robustness, 3, 13, 34, 37{39, 53, 61, 64, 70, 90, 94, 118, 178, 204{217, 236, 238 Rocko, M.L., 41 Rodlo, R.K., 246 Rogson, M., 100 Roitblat, H., 103 Rosen, J.B., 18, 24, 57, 91, 352 Rosen, R., 100 Rosenblatt, F., 102 Rosenbrock, H.H., 23, 29, 48, 50, 54, 343, 349 Rosenbrock strategy, 16, 48{54, 64, 177, 179, 184, 190, 196, 201, 202, 207, 209, 212, 213, 216, 228, 230{232, 357, 363, 365, 366 Rosenman, E.A., 11 Ross, G.J.S., 84

440 Rotating coordinates method, see Rosenbrock and DSC strategies Rothe, R., 25 Roughgarden, J.W., 102 Rounding error, see accuracy of computation Rozonoer, L.I., 90 Rozvany, G., 247 Ruban, A.I., 99 Rubin, A.I., 95 Rubinov, A.M., 11 Rudd, D.F., 98, 356 Rudelson, L.Ye., 102 Rudolph, G., 91, 118, 134, 151, 154, 161, 162, 241, 243, 248 Rustay, R.C., 68, 89 Rutishauser, H., 5, 41, 43, 48, 65, 75, 172, 326 Rybashov, M.V., 68 Ryshik, I.M., 136 Saaty, T.L., 20, 27, 166 Sacks, J., 20 Saddle point, 13, 14, 17, 23, 25, 35, 36, 39, 66, 76, 88, 168, 176, 209, 211, 345 Sala, S., 100 Sameh, A.H., 239 Samuel, A.L., 102 Sargent, R.W.H., 78, 82 Saridis, G.N., 90, 98 Satterthwaite, F.E., 98, 101 Saunders, M.A., 57 Savage, J.M., 119 Savage, L.J., 41 Sawaragi, Y., 48 Sayama, H., 82 Scaling of the variables, 7, 44, 54, 58, 74, 146{148, 232, 239 Schaer, J.D., 152 Schechter, R.S., 15, 23, 28, 32, 37, 41{43, 64, 65 Scheeer, L., 14 Scheel, A., 118 Schema theorem, 154

Index Scheraga, H.A., 89 Scheuer, E.M., 241 Scheuer, T., 98, 164 Schinzinger, R., 67 Schittkowski, K., 174 Schley, C.H., Jr., 70 Schlierkamp-Voosen, D., 163 Schmalhausen, I.I., 101 Schmetterer, L., 90 Schmidt, E., see Gram-Schmidt orthogonalization Schmidt, J.W., 35, 39 Schmitt, E., 20, 90 Schmutz, M., 103 Schneider, G., 246 Schneider, M., 100 Schrack, G., 98, 240 Schumer, M.A., 89, 93, 96{99, 101, 200, 240 Schuster, P., 101 Schwarz, H.R., 5, 41, 65, 75, 172, 326 Schwefel, D., 246 Schwefel, H.-P., 7, 102, 103, 118, 134, 148, 151, 152, 155, 163, 204, 234, 239, 242, 245{248 Schwetlick, H., 39 Scott, E.L., 102 Sebald, A.V., 151 Sebastian, D.J., 82 Sebastian, H.-J., 24 Secant method, 34, 39, 84 Second order gradient strategies, see Newton strategies Sectioning algorithms, 14 Seidel, P.L., 41, see also coordinate strategy Selection, 3, 100{102, 106, 142, 153, 157 Sensitivity analysis, 17 Separable objective function, 12, 42 Sequential methods, 27, 38, 88, 237 Sequential unconstrained minimization technique, see SUMT method Sergiyevskiy, G.M., 69 Sexual propagation, 3, 101, 106, 146, 147

Index Shah, B.V., 67, 68 Shanno, D.F., 76, 82{84 Shapiro, I.J., 94 Shedler, G.S., 239 Shemeneva, V.V., 95 Shimelevich, L.I., 88 Shimizu, T., 92 Shindo, A., 64 Short step methods, 66 Shrinkage random search, 94 Shubert, B.O., 29 Sigmund, K., 21 Silverman, G., 84 Simplex method, see linear programming Simplex strategy, 57{61, 64, 84, 89, 97, 177, 179, 184, 190, 196, 201, 202, 208, 210, 228{231, 341, 361{364 Simplex, 17, 58, 353 Simulated annealing, 160{162 Simulation, 13, 93, 102, 103, 152, 245, 246 Simultaneous methods, 26{27, 92, 168, 237 Singer, E., 44 Single step methods, see relaxation methods Singularity, 70, 74, 78, 82, 205, 209 Sirisena, H.R., 15 Slagle, J.R., 102 Slezak, N.L., 241 Smith, C.S., 54, 71, 74 Smith, D.E., 174 Smith, F.B., Jr., 84 Smith, J. Maynard, 21, 102 Smith, L.B., 44, 178 Smith, N.H., 98, 356 Soeder, C.-J., 103 Somatic mutations, 247 Sondak, N.E., 66, 89 Sonderquist, F.J., 48 Sorenson, H.W., 68, 70 Southwell, R.V., 20, 41, 43, 65 Spang, H.A., 93, 174 Spath, H., 84 Spears, W., 152

441 Spedicato, E., 82 Spendley, W., 57, 58, 61, 64, 68, 84, 89 Speyer, J.L., 70 Sphere model objective function, 110, 117, 120, 123, 124, 127{134, 142, 173, 179, 203, 215, 325, 338 Spider method, 48 Sprave, J., 247, 248 Spremann, K., 11 Stagnation, 47, 58, 61, 64, 67, 87, 88, 100, 157, 201, 205, 238, 341 Standard deviation, see variance Stanton, E.L., 34 Stark, R.M., 23 Static optimization, 9, 10 Stebbins, G.L., 106 Steepest descent/ascent, 66{68, 166, 169, 235 Steiglitz, K., 87, 96, 98, 99, 101, 200, 240 Stein, M.L., 14, 67 Steinberg, D., 23 Steinbuch, K., 6 Stender, J., 152 Step length control, 110{113, 142{145, 168, 172, 237, see also evolution strategy, 1=5 success rule Steuer, R.E., 20 Stewart, E.C., 95, 98, 99 Stewart, G.W., 78, 84, see also DFPStewart strategy Stiefel, E., 5, 41, 43, 65, 67, 69, 75, 172, 326 Stochastic approximation, 19, 20, 64, 83, 90, 94, 99, 236 Stochastic optimization, 18 Stochastic perturbations, 9, 20, 36, 58, 68, 69, 89, 91, 92, 94, 95, 97, 99, 236, 245 Stoer, J., 18 Stoller, D.S., 241 Stolz, O., 14 Stone, H.S., 239 Storage requirement, 47, 53, 57, 180, 232{ 234, 236

442 Storey, C., 23, 50, 54 Strategy, 2, 6, 100 Strategy comparison, 57, 64, 68, 71, 78, 80, 83, 84, 92, 97, 165{234 Strategy parameter, 144, 204, 238, 240{ 242 Stratonovich, R.L., 90 Strong minimum, 24, 328, 333 Strongin, R.G., 94 Structural optimization, 247 Struggle for existence, 100, 106 Suboptimum, 15 Subpopulations, 248 Success/failure routine, 29 Suchowitzki, S.I., 18 Sugie, N., 38 Sum of squares minimization, 5, 83, 331, 335, 346 SUMT method, 16 Supremum, 9 Sutti, C., 88 Suzuki, S., 352 Svechinskii, V.B. (Svecinskij, V.B.), 90, 102 Swann, W.H., 23, 28, 54, 56, 57, see also DSC strategy Sweschnikow, A.A., 137 Sworder, D.D., 83 Sydow, A., 68 Sylvester, criterion of, 240 Synge, J.L., 44 Sysoyev, V.V., 95 Szego, G.P., 24, 70, 88 Tabak, D., 18, 78, 82 Tabu search, 162{164 Tabulation method, see grid method Takamatsu, T., 82 Talkin, A.I., 68 Tammer, K., 24 Tan, S.T., 17 Tapley, B.D., 174 Taran, V.A., 94 Taylor, G., 84

Index Taylor series (Taylor, B.), 75, 84 Tazaki, E., 64 Tchebyche approximation (Tschebyschow, P.L.), 5, 331, 370 Termination of the search, 35, 38, 49, 54, 59, 64, 67, 71, 96, 113, 114, 117, 145, 146, 150, 167, 168, 175, 176, 180, 212, 238 Ter-Saakov, A.P., 69 Theodicee, 1 Theory of maxima and minima, 11 Thom, R., 102 Thomas, M.E., 20 Three point scheme, 29 Threshold strategy, 98, 164 Tietze, J.L., 70 Timofejew-Ressowski, N.W., 101 Todd, J., 326 Togawa, T., 100 Tokumaru, H., 82 Tolle, H., 18, 68 Tomlin, F.K., 44, 178 Torn, A., 91 Total step procedure, 65 Tovstucha, T.I., 89 Trabattoni, L., 88 Trajectory optimization, see functional optimization Traub, J.F., 14, 27 Travelling salesperson problem (TSP), 159, 161 Treccani, G., 70, 88 Trial and error, 13, 41 Trial polynomial, 27, 33{35, 37, 68, 235 Trinkaus, H.F., 35 Trotter, H.F., 76 Tschebyschow, P.L., see Tchebyche approximation Tse, E., 239 Tseitlin, B.M., 44 Tsetlin, M.L., 89 Tsypkin, Ya.Z., 6, 9, 89, 90 Tucker, A.W., 17, 166 Tui, H., 88

Index Turning point, see saddle point Two membered evolution strategy, 97, 101, 105{118, 172, 238, 329, 352, 357, 359, 363, 366, 367, 374, see also evolution strategy (1+1) Tzschach, H.G., 18 Ueing, U., 88, 358, 359 Umeda, T., 64 Unbehauen, H., 18 Uncertainty, interval of, 26{28, 32, 39, 92, 180 Uniform distribution, 91, 92, 95, 115 Unimodality, 24, 27, 28, 39, 168, 236 Uzawa, H., 18 Vagin, V.N., 102 Vajda, S., 7, 18 Vanderplaats, G.N., 23 VanNice, R.I., 44 VanNorton, R., 41 Varah, J.M., 68 Varela, F.J., 103 Varga, J., 18 Varga, R.S., 43 Variable metric, 70, 77, 83, 169{172, 178, 233, 242, 243, 246, see also DFP and DFP-Stewart strategies Variables, 2, 8, 11 Variance analysis, 8 Variance ellipse, 109 Variance methods, 82 Variational calculus, 2, 11, 15, 66 Vaysbord, E.M., 90, 94, 99 Vecchi, M.P., 160 Venter, J.H., 20 Vetters, K., 39 Vilis, T., 10 Viswanathan, R., 94 Vitale, P., 84 Vogelsang, R., 5 Vogl, T.P., 24, 48, 93 Voigt, H.-M., 163 Voltaire, F.M., 1

443 Volume-oriented strategies, 98, 160, 236, 248 Volz, R.A., 12, 70 Wacker, H., 12 Waddington, C.H., 102 Wagner, K., 151, 246 Wald, A., 7, 89 Walford, R.B., 93 Wallack, P., 68 Walsh, J., 27 Walsh, M.J., 102, 105, 151 Ward, L., 58 Wasan, M.T., 19 Wasscher, E.J., 68, 76 Weak minimum, 24, 25, 113, 328, 332, 333 Weber, H.H., 17 Wegge, L., 76 Weierstrass, K., theorem of, 25 Weinberg, F., 18, 91 Weisman, J., 23, 47, 89 Weiss, E.A., 48 Wells, M., 77 Werner, J., 82 Wets, R.J.-B., 19 Wetterling, W., 5 Wheatley, P., 38 Wheeling, R.F., 95, 98 White, L.J., 97 White, R.C., Jr., 93, 95 Whitley, L.D., 152, 155 Whitley, V.W., 76 Whitting, I.J., 84 Whittle, P., 18 Wiedemann, J., 151 Wiener, N., 6 Wierzbicki, A.P., 20 Wilde, D.J., 1, 20, 23, 26, 27, 29, 31{33, 38, 39, 87 Wilf, H.S., 27 Wilkinson, J.H., 14, 75 Wilson, E.O., 146 Wilson, K.B., 6, 65, 68, 89

444 Wilson, S.W., 103 Witt, U., 103 Witte, B.F.W., 67 Witten, I.H., 94 Witzgall, C., 18 Wolfe, P., 19, 23, 39, 66, 70, 82, 84, 166, 360 Wol, W., 103 Wolfowitz, J., 19 Wood, C.F., 47, 48, 89 Woodside, C.M., 70 Wright, S.J., 179 Yates, F., 7 Youden, W.J., 101 Yudin, D.B., 90, 94, 99 Yvon, J.P., 96 Zach, F., 9 Zadeh, N., 41 Zahradnik, R.L., 23 Zakharov, V.V., 94 Zangwill, W.I., 18, 41, 66, 71, 74, 170, 202 Zehnder, C.A., 18, 91 Zeleznik, F.J., 84 Zellnik, H.E., 66, 89 Zener, C., 14 Zerbst, E.W., 105 Zero-one optimization, see binary optimization Zettl, G., 344, 348 Zhigljavsky, A.A., 91 Zigangirov, K.S., 89 Z ilinskas, A., 91 Zoutendijk, G., 18, 70 Zurmuhl, R., 27, 35, 172 Zwart, P.B., 88 Zypkin, Ja.S., see Tsypkin, Ya.Z.

Index