Design and Control of Intelligent Robotic Systems (Studies in Computational Intelligence, Volume 177)

Dikai Liu, Lingfeng Wang, and Kay Chen Tan (Eds.) Design and Control of Intelligent Robotic Systems

Studies in Computational Intelligence, Volume 177 Editor-in-Chief Prof. Janusz Kacprzyk Systems Research Institute Polish Academy of Sciences ul. Newelska 6 01-447 Warsaw Poland E-mail: [email protected] Further volumes of this series can be found on our homepage: springer.com Vol. 155. Hamid R. Tizhoosh and M. Ventresca (Eds.) Oppositional Concepts in Computational Intelligence, 2008 ISBN 978-3-540-70826-1 Vol. 156. Dawn E. Holmes and Lakhmi C. Jain (Eds.) Innovations in Bayesian Networks, 2008 ISBN 978-3-540-85065-6 Vol. 157. Ying-ping Chen and Meng-Hiot Lim (Eds.) Linkage in Evolutionary Computation, 2008 ISBN 978-3-540-85067-0 Vol. 158. Marina Gavrilova (Ed.) Generalized Voronoi Diagram: A Geometry-Based Approach to Computational Intelligence, 2009 ISBN 978-3-540-85125-7

Vol. 166. Constantinos Koutsojannis and Spiros Sirmakessis (Eds.) Tools and Applications with Artificial Intelligence, 2009 ISBN 978-3-540-88068-4 Vol. 167. Ngoc Thanh Nguyen and Lakhmi C. Jain (Eds.) Intelligent Agents in the Evolution of Web and Applications, 2009 ISBN 978-3-540-88070-7 Vol. 168. Andreas Tolk and Lakhmi C. Jain (Eds.) Complex Systems in Knowledge-based Environments: Theory, Models and Applications, 2009 ISBN 978-3-540-88074-5 Vol. 169. Nadia Nedjah, Luiza de Macedo Mourelle and Janusz Kacprzyk (Eds.) Innovative Applications in Data Mining, 2009 ISBN 978-3-540-88044-8

Vol. 159. Dimitri Plemenos and Georgios Miaoulis (Eds.) Artificial Intelligence Techniques for Computer Graphics, 2009 ISBN 978-3-540-85127-1

Vol. 170. Lakhmi C. Jain and Ngoc Thanh Nguyen (Eds.) Knowledge Processing and Decision Making in Agent-Based Systems, 2009 ISBN 978-3-540-88048-6

Vol. 160. P. Rajasekaran and Vasantha Kalyani David Pattern Recognition using Neural and Functional Networks, 2009 ISBN 978-3-540-85129-5

Vol. 171. Chi-Keong Goh, Yew-Soon Ong and Kay Chen Tan (Eds.) Multi-Objective Memetic Algorithms, 2009 ISBN 978-3-540-88050-9

Vol. 161. Francisco Baptista Pereira and Jorge Tavares (Eds.) Bio-inspired Algorithms for the Vehicle Routing Problem, 2009 ISBN 978-3-540-85151-6 Vol. 162. Costin Badica, Giuseppe Mangioni, Vincenza Carchiolo and Dumitru Dan Burdescu (Eds.) Intelligent Distributed Computing, Systems and Applications, 2008 ISBN 978-3-540-85256-8 Vol. 163. Pawel Delimata, Mikhail Ju. Moshkov, Andrzej Skowron and Zbigniew Suraj Inhibitory Rules in Data Analysis, 2009 ISBN 978-3-540-85637-5 Vol. 164. Nadia Nedjah, Luiza de Macedo Mourelle, Janusz Kacprzyk, Felipe M.G. Fran¸ca and Alberto Ferreira de Souza (Eds.) Intelligent Text Categorization and Clustering, 2009 ISBN 978-3-540-85643-6 Vol. 165. Djamel A. Zighed, Shusaku Tsumoto, Zbigniew W. Ras and Hakim Hacid (Eds.) Mining Complex Data, 2009 ISBN 978-3-540-88066-0

Vol. 172. I-Hsien Ting and Hui-Ju Wu (Eds.) Web Mining Applications in E-Commerce and E-Services, 2009 ISBN 978-3-540-88080-6 Vol. 173. Tobias Grosche Computational Intelligence in Integrated Airline Scheduling, 2009 ISBN 978-3-540-89886-3 Vol. 174. Ajith Abraham, Rafael Falc´on and Rafael Bello (Eds.) Rough Set Theory: A True Landmark in Data Analysis, 2009 ISBN 978-3-540-89886-3 Vol. 175. Godfrey C. Onwubolu and Donald Davendra (Eds.) Differential Evolution: A Handbook for Global Permutation-Based Combinatorial Optimization, 2009 ISBN 978-3-540-92150-9 Vol. 176. Beniamino Murgante, Giuseppe Borruso and Alessandra Lapucci (Eds.) Geocomputation and Urban Planning, 2009 ISBN 978-3-540-89929-7 Vol. 177. Dikai Liu, Lingfeng Wang and Kay Chen Tan (Eds.) Design and Control of Intelligent Robotic Systems, 2009 ISBN 978-3-540-89932-7

Dikai Liu Lingfeng Wang Kay Chen Tan (Eds.)

Design and Control of Intelligent Robotic Systems

123

Dikai Liu

Kay Chen Tan

ARC Center of Excellence for Autonomous Systems (CAS) Faculty of Engineering Level 6, Building 2 University of Technology, Sydney P.O.Box. 123 Broadway, NSW 2007 Australia Email: [email protected]

Department of Electrical and Computer Engineering National University of Singapore 4 Engineering Drive 3 Singapore 117576 Singapore Email: [email protected]

Lingfeng Wang Department of Electrical and Computer Engineering Texas A&M University 214 Zachry Engineering Center College Station, TX 77843-3128 USA Email: [email protected]

ISBN 978-3-540-89932-7

e-ISBN 978-3-540-89933-4

DOI 10.1007/978-3-540-89933-4 Studies in Computational Intelligence

ISSN 1860949X

Library of Congress Control Number: 2008942028 c 2009 Springer-Verlag Berlin Heidelberg  This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilm or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. The use of general descriptive names, registered names, trademarks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. Typeset & Cover Design: Scientific Publishing Services Pvt. Ltd., Chennai, India. Printed in acid-free paper 987654321 springer.com

Preface

With the increasing applications of intelligent robotic systems in various fields, the design and control of these systems have increasingly attracted interest from researchers. This edited book entitled “Design and Control of Intelligent Robotic Systems” in the book series of “Studies in Computational Intelligence” is a collection of some advanced research on design and control of intelligent robots. The works presented range in scope from design methodologies to robot development. Various design approaches and algorithms, such as evolutionary computation, neural networks, fuzzy logic, learning, etc. are included. We also would like to mention that most studies reported in this book have been implemented in physical systems. An overview on the applications of computational intelligence in bio-inspired robotics is given in Chapter 1 by M. Begum and F. Karray, with highlights of the recent progress in bio-inspired robotics research and a focus on the usage of computational intelligence tools to design human-like cognitive abilities in the robotic systems. In Chapter 2, Lisa L. Grant and Ganesh K. Venayagamoorthy present greedy search, particle swarm optimization and fuzzy logic based strategies for navigating a swarm of robots for target search in a hazardous environment, with potential applications in high-risk tasks such as disaster recovery and hazardous material detection. K.N. Krishnanand and D. Ghose, in Chapter 3, address the applications of robotic swarms in localization of multiple signal sources such as light, sound, fire-origins in bush fire, oil spills, etc. A modified glowworm swarm optimization algorithm is proposed and implemented in real-time experiments. Seiji Yamada and Daisuke Katagami, in Chapter 4, describe approaches to simulationbased behavior learning with a genetic algorithm and direct teaching with a learning classifier system (LCS) to enable a robot to learn sufficiently fast in real environments. In Chapter 5, D. Wang, N. M. Kwok, et al. present a ranked Pareto particle swarm optimization approach and its application in the design of force fields for motion planning and collision avoidance in multi-robot systems. In chapter 6, R. Barber, V. Egido, et al. investigate emotional intelligence for robot path planning by taking into account emotional factors, as human beings do, in the decision making process. An example of an emotion inspired planner is demonstrated.

VI

Preface

In Chapter 7, Jim Pugh and Alcherio Martinoli discuss how varying sensor offsets and scaling factors affect parallel swarm-robotic learning of obstacle avoidance behavior using both genetic algorithms and particle swarm optimization, in designing effective behavioral controllers for mobile robots in multi-robot systems. Patrick M. McDowell, Brian S. Bourgeois et al., in Chapter 8, investigate a memorybased learning for robot control in dynamic environments. A neural network which uses sensor inputs and action outputs is used as the basic controller, and genetic algorithms are applied to develop the network weights and transfer functions. In Chapter 9, Dongbing Gu, Jindong Liu, et al. present a behavior based control and learning system for autonomous robots. A Markov decision process is used to model the coordination mechanism, and a Q-learning approach to learn the behavior coordination. The learning system is implemented in two different types of robots. In Chapter 10, Yohannes Kassahun, Jan Hendrik Metzen, et al. present a method, evolutionary acquisition of neural topologies, for evolving the structures and weights of neural networks to find minimal neural structures for solving a given learning task. This method is applied to control a crawling robotic insect to learn and adapt to the environment. Terrence P. Fries, in Chapter 11, presents a fuzzy genetic algorithm approach to robot motion planning in real-time without the limitations of monotone paths. This approach employs a novel chromosome encoding scheme. The imprecision and uncertainty of terrain data are modeled using fuzzy linguistic variables and included in the path planning decision. In Chapter 12, Sukumar Kamalasadan and Adel A. Ghandakly present an intelligent adaptive control approach to precise tracking of autonomous robots. It consists of a neuro-controller that monitors the system’s closed loop dynamics and an adaptive controller for controlling the dynamics. This approach is verified by both simulation and real-time implementation with a mobile wheeled robot. In Chapter 13, Gary B Parker presents a punctuated anytime learning method for robot control. This method uses periodic tests on a robot to improve the accuracy of training done in simulation, and is applied in the learning of a robot control program. Nathan Wiebe and John Anderson, in Chapter 14, address the communication problems in a robot team. An approach is presented to allow a group of robotic agents to develop consistent shared groundings for useful locations in an environment using only local communication and interaction. This approach is demonstrated to be effective in communication in a multi-robot setting. Daniel C. Asmar, Samer M. Abdallah et al., in Chapter 15, develop an inertial-vision Simultaneous Localization and Mapping (SLAM) system that uses real natural objects as landmarks with the objective of creating a purposeful map. In Chapter 16, Florian Hoppe and Gerald Sommer present a programmed robotic system, i.e. a 6 DOF robot arm with two cameras, to learn to solve a shape-sorter puzzle. A new software architecture for artificial cognitive systems is proposed to handle data with different degrees of abstraction and realize nested perception-action cycles. Results from experiments demonstrate the performance of the system and that the system is able to compensate for variations in camera position.

Preface

VII

In Chapter 17, Jacky Baltes and John Anderson describe the work of developing a minimalist humanoid robot, Tao-Pie-Pie, including its walking gait and balancing reflexes. Issues in evaluating humanoid robots are also discussed in this chapter. The design, modeling and control of a four-link redundant parallel ankle rehabilitation robot are presented in Chapter 18 by Yun-Ho Tsoi, Shane Q. Xie et al. Robot kinematic parameters are selected based on singularity and workspace analysis. Controller design is facilitated by the modeling of human ankle (based on the biaxial joint ankle model) and the manipulator. Ankle stiffness and damping parameters are estimated by applying a recursive least squares algorithm. Results from simulations show the performance of the developed controller. In Chapter 19, Hector Hugo Aviles-Arriaga, L. Enrique Sucar, et al. describe a general framework which can be used to develop various applications of service robots. Modules for common tasks are developed and coordinated by a Markov Decision Process (MDP) based planner. With this research, a new application can be created by defining a new MDP whose solution provides an optimal policy that coordinates different behaviors for performing a task. Results from experiments in four different service robot tasks demonstrate the effectiveness of the proposed approach. In Chapter 20, Tetsuyou Watanabe, ZhongWei Jiang, et al. formulate an optimization problem to design desirable grasp operations of a robot hand using defined acceleration and equilibrium-force sets. The fundamental theory and the algorithm for finger configuration and for determining contact positions in grasping and manipulating an object are detailed. The proposed method is validated by simulations. In the last chapter, Gourab Sen Gupta, Chris Messom et al. address the problem of accurate motion control of fast moving wheeled robots by using a triangular targeting algorithm. Results from experiments with soccer robots demonstrate that the algorithm greatly improves the performance of the robot when compared to commonly used methods such as PD controllers. The intended readers of this edited book are engineers, researchers, senior undergraduates, and graduate students who are interested in the field of design and control of intelligent robotic systems. We hope that the readers would find some chapters in this book interesting and useful in their research as well as in practical engineering work in the area of robotics. We would like to thank all the authors who have contributed their work to this book. We are grateful to all the reviewers who have generously devoted their time to review the manuscripts. Special thanks go to Dr. N.M. Kwok for his time and effort in compiling the book. September 2008

D.K. Liu L.F. Wang K.C. Tan

Contents

1

2

Computational Intelligence Techniques in Bio-inspired Robotics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . M. Begum, F. Karray 1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 Bio-inspired Robotics and Tools of Computational Intelligence: The Historic Synergy . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.2 The CI Tools Most Commonly Used in Bio-inspired Robotics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3 Bio-inspired Robotics: An Emerging Research Field . . . . . . . . . . . 1.4 Cognitive Modeling for Autonomous Robots: CI-Based Approaches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4.1 Cognition: Robot and Human . . . . . . . . . . . . . . . . . . . . . . . . 1.4.2 Value System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4.3 Visual Attention . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4.4 Social Cognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Swarm Intelligence for Collective Robotic Search . . . . . . . . . . . Lisa L. Grant, Ganesh K. Venayagamoorthy 2.1 Collective Robotic Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Navigation Strategies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.1 Greedy Search - A Conventional Method . . . . . . . . . . . . . . 2.2.2 Particle Swarm Optimization . . . . . . . . . . . . . . . . . . . . . . . . 2.2.3 Fuzzy Logic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3 Single Target Obstacle-Free Environment . . . . . . . . . . . . . . . . . . . . 2.4 Multiple Target/Obstacle Environment . . . . . . . . . . . . . . . . . . . . . . 2.4.1 Obstacle Avoidance Method . . . . . . . . . . . . . . . . . . . . . . . . . 2.4.2 Case Study 1: Single Swarm . . . . . . . . . . . . . . . . . . . . . . . . . 2.4.3 Case Study 2: Multiple Swarms . . . . . . . . . . . . . . . . . . . . . .

1 1 2 2 3 4 7 7 9 12 14 19 20 29 29 31 31 33 34 38 40 40 41 44

X

Contents

2.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

4

A Glowworm Swarm Optimization Based Multi-robot System for Signal Source Localization . . . . . . . . . . . . . . . . . . . . . . K.N. Krishnanand, D. Ghose 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Prior Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 The GSO Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.1 GSO Algorithm Description . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.2 A Simulation Example to Illustrate Basic Working of GSO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 Algorithmic Modifications for Robotic Implementation . . . . . . . . 3.4.1 Obstacle Avoidance Model . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.2 Implementation Models for Leapfrogging Behavior . . . . . . 3.5 Kinbots: Multi-robot Platform for Real-Robot-Experiments . . . . 3.5.1 Relative Localization of Neighbors . . . . . . . . . . . . . . . . . . . . 3.5.2 Luciferin Glow/Reception . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.6 Real-Robot-Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.6.1 Experiment 1: Basic GSO Behavior . . . . . . . . . . . . . . . . . . . 3.6.2 Experiment 2: Leapfrogging Behavior . . . . . . . . . . . . . . . . . 3.6.3 Experiment 3: Localization of a Single Light Source . . . . . 3.7 Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Evolutionary Robotics: From Simulation-Based Behavior Learning to Direct Teaching in Real Environments . . . . . . . . . Seiji Yamada, Daisuke Katagami 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Simulation-Based Behavior Learning Using Genetic Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.1 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.2 Task: Action-Based Environment Modeling . . . . . . . . . . . . 4.2.3 States, Actions and Environment Vectors . . . . . . . . . . . . . . 4.2.4 Applying GA to Acquire Behaviors . . . . . . . . . . . . . . . . . . . 4.2.5 Simulation Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 Directly Teaching for Robots in Real Environments . . . . . . . . . . . 4.3.1 Supervised Learning and Interactive Evolutionary Computation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3.2 Teacher View and Learner View . . . . . . . . . . . . . . . . . . . . . . 4.3.3 Interactive Classifier System . . . . . . . . . . . . . . . . . . . . . . . . . 4.3.4 Robot Experiments in the Real World . . . . . . . . . . . . . . . . 4.3.5 Teaching Efficiency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

45 46 49 49 51 52 52 54 55 56 56 57 59 60 60 60 63 65 67 67 69 69 70 71 72 73 75 78 83 85 86 86 87 89 93 94 94

Contents

5

6

7

Ranked Pareto Particle Swarm Optimization for Mobile Robot Motion Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D. Wang, N.M. Kwok, D.K. Liu, Q.P. Ha 5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 Ranked Pareto Particle Swarm Optimization . . . . . . . . . . . . . . . . . 5.2.1 Key Concepts in MOP s . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2.2 Particle Swarm Optimization . . . . . . . . . . . . . . . . . . . . . . . . 5.2.3 Ranked Pareto Particle Swarm Optimization . . . . . . . . . . . 5.3 F 2 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.1 Definition of a Force Field . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.2 Attractive Force . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.3 Reaction Force . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.4 V SF 2 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4 Using RPPSO for Multiobjective Optimization of F 2 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4.1 Case 1: Single Robot and One Obstacle . . . . . . . . . . . . . . . 5.4.2 Case 2: Two Robots Navigating in a Corridor . . . . . . . . . . 5.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Path Planning Inspired on Emotional Intelligence . . . . . . . . . . R. Barber, V. Egido, M. Malfaz, M.A. Salichs 6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2 The Path Planning Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.3 Emotions in Robotics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4 Topological Representation of the Environment . . . . . . . . . . . . . . . 6.5 Path Planning Algorithm Inspired on Emotions . . . . . . . . . . . . . . 6.5.1 Environment Representation: The Navigation Chart . . . . 6.5.2 Topological Planning Algorithm: Guided Dijkstra . . . . . . 6.6 How the Guided Dijkstra Algorithm Works . . . . . . . . . . . . . . . . . . 6.6.1 Planning in the AD Control Architecture . . . . . . . . . . . . . . 6.6.2 Deliberative Planner Skill . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.6.3 An Example of the Emotional Inspired Planning . . . . . . . 6.7 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . An Exploration of Online Parallel Learning in Heterogeneous Multi-robot Swarms . . . . . . . . . . . . . . . . . . . . . . . . . Jim Pugh, Alcherio Martinoli 7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3 Parallel Swarm-Robotic Learning Case Study: Obstacle Avoidance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3.1 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.4 Heterogeneity from Sensor Offsets . . . . . . . . . . . . . . . . . . . . . . . . . .

XI

97 97 99 99 100 100 106 107 108 109 109 111 112 115 116 117 119 119 119 120 121 121 122 123 125 125 127 128 131 131

133 133 135 136 138 139 140

XII

Contents

7.4.1 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.4.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.5 Heterogeneity from Sensor Scaling . . . . . . . . . . . . . . . . . . . . . . . . . . 7.5.1 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.5.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.6 Diversity in Evolving Robot Swarms . . . . . . . . . . . . . . . . . . . . . . . . 7.6.1 Euclidean Diversity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.6.2 Entropic Diversity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.6.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.7 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.8 Conclusion and Outlook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

9

Robot Control in Dynamic Environments Using Memory-Based Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Patrick M. McDowell, Brian S. Bourgeois, Frederick E. Petry 8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2 ROV/UUV Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3 Memory Based Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.4 Genetic Algorithm Training of Neural Network Controllers . . . . . 8.5 Reactive Controller . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.6 Memory Based Controller . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.7 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.8 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A Behavior Based Control and Learning Approach to Real Robots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Dongbing Gu, Jindong Liu, Huosheng Hu 9.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.2 Layered Control Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.2.1 Locomotion Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.2.2 Behavior Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.2.3 Cognitive Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.3 Behavior Design Using Fuzzy Logic . . . . . . . . . . . . . . . . . . . . . . . . . 9.4 Q-Learning in the Cognitive Layer . . . . . . . . . . . . . . . . . . . . . . . . . . 9.5 Robotic Fish Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.6 Sony Quadruped Robot Experiments . . . . . . . . . . . . . . . . . . . . . . . . 9.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

141 141 142 143 143 145 145 145 146 148 150 150

153 153 154 156 159 161 162 164 168 169

171 171 173 174 174 175 176 178 180 183 185 186

10 Incremental Acquisition of Neural Structures through Evolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187 Yohannes Kassahun, Jan Hendrik Metzen, Mark Edgington, Frank Kirchner 10.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187

Contents

10.2 Review of Related Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.2.1 Direct Genetic Encoding Approaches . . . . . . . . . . . . . . . . . 10.2.2 Indirect Genetic Encoding Approaches . . . . . . . . . . . . . . . . 10.3 Evolutionary Acquisition of Neural Topologies (EANT) . . . . . . . 10.3.1 Common Genetic Encoding (CGE) . . . . . . . . . . . . . . . . . . . 10.3.2 Genetic Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.3.3 Evaluating a Linear Genome . . . . . . . . . . . . . . . . . . . . . . . . . 10.3.4 Generating the Initial Population . . . . . . . . . . . . . . . . . . . . . 10.3.5 Exploitation and Exploration of Structures . . . . . . . . . . . . 10.4 Practical Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.4.1 Crawling Robotic Insect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.4.2 Pole Balancing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 Evolutionary Terrain-Based Navigation of Autonomous Mobile Robots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Terrence P. Fries 11.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Evolutionary Computation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.3 Motion Planning Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.3.1 Assumptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.3.2 Representation of the Environment as a Grid . . . . . . . . . . 11.3.3 Encoding the Chromosome . . . . . . . . . . . . . . . . . . . . . . . . . . 11.3.4 Initial Population . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.3.5 Genetic Operators and Parameters . . . . . . . . . . . . . . . . . . . 11.3.6 Fuzzy Terrain Conditions in the Fitness Function . . . . . . . 11.3.7 Operation in a Dynamic Environment . . . . . . . . . . . . . . . . . 11.4 Test Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 A Novel Intelligent Control Approach for Precise Tracking of Autonomous Robots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Sukumar Kamalasadan, Adel A. Ghandakly 12.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.2 Formulation of NNPAC Scheme . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.2.1 NNPAC Design Based on Feedback Error Learning . . . . . 12.2.2 Direct Model Reference Adaptive Control . . . . . . . . . . . . . 12.2.3 Generic RBFNN Architecture and Design . . . . . . . . . . . . . 12.3 Theoretical Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.3.1 Development of the RBFNN Controller . . . . . . . . . . . . . . . 12.3.2 Mathematical Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.3.3 Controller Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.3.4 Stability Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

XIII

189 189 190 191 191 193 195 198 198 199 200 202 206 206

209 209 213 214 214 214 215 216 217 217 218 218 223 224

227 227 229 230 231 233 234 234 235 236 236

XIV

Contents

12.3.5 Modifications of Adaptive Laws Based on NNPAC Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.3.6 Online Growing RBFNN Design and Its Properties . . . . . 12.3.7 NNPAC Features and Algorithmic Development . . . . . . . . 12.4 Application to Position Tracking of Robots . . . . . . . . . . . . . . . . . . 12.4.1 Simulation Results and Discussions . . . . . . . . . . . . . . . . . . . 12.5 Real-Time Implementations and Experimental Results . . . . . . . . 12.5.1 Design Details and Algorithmic Flow Chart . . . . . . . . . . . . 12.5.2 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 Punctuated Anytime Learning to Evolve Robot Control for Area Coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Gary B. Parker 13.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.2 Area Coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.2.1 Previous Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.2.2 Research Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.3 The ServoBot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.3.1 Gait Cycles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.3.2 Production of Turns in Gait Cycles . . . . . . . . . . . . . . . . . . . 13.3.3 Simulated Robot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.4 Cyclic Genetic Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.4.1 CGAs Applied to Area Coverage . . . . . . . . . . . . . . . . . . . . . 13.4.2 Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.5 Punctuated Anytime Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.5.1 Anytime Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.5.2 An Extension to Anytime Learning . . . . . . . . . . . . . . . . . . . 13.5.3 Fitness Biasing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.5.4 The Co-evolution of Model Parameters . . . . . . . . . . . . . . . . 13.6 Learning Area Coverage Using the Co-evolution of Model Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.6.1 Partial Recombination Used in the Evolution of Model Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.6.2 Punctuated Anytime Learning in Simulation . . . . . . . . . . . 13.6.3 Punctuated Anytime Learning on the Actual Robot . . . . 13.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

237 237 241 242 244 248 248 253 253 254

255 255 257 257 258 259 259 260 261 262 263 264 265 265 266 267 268 270 271 272 274 275 276

14 Local Methods for Supporting Grounded Communication in Robot Teams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279 Nathan Wiebe, John Anderson 14.1 Introduction: Grounded Communication in Multi-robot Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279 14.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283

Contents

XV

14.3 Developing Grounded Spatial References . . . . . . . . . . . . . . . . . . . . 14.3.1 Grounding Locations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.3.2 Sharing Groundings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.3.3 Consistency among a Group of Agents . . . . . . . . . . . . . . . . 14.4 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.4.1 Location Grounding Behaviours . . . . . . . . . . . . . . . . . . . . . . 14.4.2 Sharing Groundings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.5 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

284 285 286 286 288 289 291 294 299 300

15 Vision SLAM Maps: Towards Richer Content . . . . . . . . . . . . . . . Daniel C. Asmar, Samer M. Abdallah, John S. Zelek 15.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15.2 Inertial Navigation System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15.3 Computer Vision System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15.3.1 Visual Environment Recognition . . . . . . . . . . . . . . . . . . . . . 15.3.2 Tree Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15.3.3 Tree Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15.3.4 Tree Initialization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15.4 Vision SLAM System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15.5 Experiments and Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15.5.1 Run-Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

303

16 Learning of Hierarchical Perception-Action Cycles . . . . . . . . . . Florian Hoppe, Gerald Sommer 16.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16.1.1 Challenges and State of the Art Solutions . . . . . . . . . . . . . 16.1.2 Machine Learning Techniques in Robotics . . . . . . . . . . . . . 16.1.3 Structure of the Chapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16.2 Software Architecture for Artificial Cognitive Systems . . . . . . . . . 16.2.1 Overview of the Software Architecture . . . . . . . . . . . . . . . . 16.2.2 Concrete Visual Robotic System . . . . . . . . . . . . . . . . . . . . . 16.2.3 Dynamic of the Demonstrator . . . . . . . . . . . . . . . . . . . . . . . . 16.3 Example Submodules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16.3.1 Internals of the Perception-Action Module . . . . . . . . . . . . . 16.3.2 Adaptive Visual Servoing Scheme . . . . . . . . . . . . . . . . . . . . 16.3.3 Learning the Long Term Goals of the Puzzle Game . . . . . 16.4 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16.4.1 Validation of the Adaptive Visual Servoing Scheme . . . . . 16.4.2 Validation of the Shape Sorter Puzzle Player . . . . . . . . . . . 16.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

331

303 305 307 308 309 311 311 312 315 321 322 322 328

331 332 333 334 334 335 336 337 337 337 339 342 346 346 348 351 352

XVI

Contents

17 Advancing Artificial Intelligence through Minimalist Humanoid Robotics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jacky Baltes, John Anderson 17.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17.2 Tao-Pie-Pie . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17.3 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17.3.1 Viki . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17.3.2 Pino and Morph . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17.3.3 Robo Erectus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17.3.4 KAIST Robot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17.3.5 Johnny Walker . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17.3.6 Development of Walking Gaits . . . . . . . . . . . . . . . . . . . . . . . 17.4 Design Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17.5 Mechanical Design of Tao-Pie-Pie . . . . . . . . . . . . . . . . . . . . . . . . . . . 17.6 Development of Walking Gaits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17.6.1 Inverse Kinematics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17.6.2 Bezier Curves . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17.6.3 Pattern for Straight Walk . . . . . . . . . . . . . . . . . . . . . . . . . . . 17.6.4 Turning Patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17.6.5 Kicking Pattern . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17.7 Balancing Reflexes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17.7.1 Sensor Feedback in Detecting a Fall . . . . . . . . . . . . . . . . . . . 17.7.2 Balancing Reflexes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17.8 Evaluation of Balancing Reflexes . . . . . . . . . . . . . . . . . . . . . . . . . . . 17.9 Moving Forward: The HuroCup Competition . . . . . . . . . . . . . . . . . 17.10 Discussion: From Tao-Pie-Pie to Blitz . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 Design, Modeling and Control of an Ankle Rehabilitation Robot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yun-Ho Tsoi, Shane Q. Xie, Andrew E. Graham 18.1 Rehabilitation Robots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18.2 Ankle Rehabilitation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18.3 Robot Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18.4 Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18.5 Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18.5.1 Impedance Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18.5.2 Kinematic Self Calibration . . . . . . . . . . . . . . . . . . . . . . . . . . . 18.5.3 Environment Parameter Estimation . . . . . . . . . . . . . . . . . . . 18.6 Controller Simulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18.7 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

355 355 357 358 358 359 359 359 359 359 360 360 362 362 363 364 365 365 366 368 368 370 372 375 376

377 377 378 379 383 386 386 389 392 394 397 397

Contents

19 Markovito: A Flexible and General Service Robot . . . . . . . . . . H´ector Hugo Avil´es-Arriaga, L. Enrique Sucar, Eduardo F. Morales, Blanca A. Vargas, Josu´e S´ anchez, Elva Corona 19.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19.2 Software Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19.3 Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19.3.1 Map Building and Localization . . . . . . . . . . . . . . . . . . . . . . . 19.3.2 Navigation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19.3.3 Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19.3.4 Speech Synthesis and Recognition . . . . . . . . . . . . . . . . . . . . 19.3.5 Visual Perception . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19.4 Coordinator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19.4.1 Markov Decision Processes . . . . . . . . . . . . . . . . . . . . . . . . . . 19.4.2 Task Coordination Based on MDPs . . . . . . . . . . . . . . . . . . . 19.5 Implementation Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19.5.1 Hardware and Software Platforms . . . . . . . . . . . . . . . . . . . . 19.5.2 General Structure and Operation . . . . . . . . . . . . . . . . . . . . . 19.6 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19.6.1 Navigate through Three Places and Return to the Starting Position . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19.6.2 Lost and Found . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19.6.3 Follow a Person under User Request . . . . . . . . . . . . . . . . . . 19.6.4 Deliver Messages and Objects between People . . . . . . . . . . 19.7 Conclusions and Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 Task Based Hybrid Closure Grasping Optimization for Autonomous Robot Hand . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tetsuyou Watanabe, ZhongWei Jiang, Tsuneo Yoshikawa 20.1 Grasping by a Robot Hand . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20.2 Target System and Kinematics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20.2.1 Target System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20.2.2 Nomenclature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20.2.3 Kinematic Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20.2.4 Statics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20.2.5 Frictional Constraint . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20.3 Mechanics of Hybrid Closure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20.4 Dynamics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20.5 Concept of Required External Force Set . . . . . . . . . . . . . . . . . . . . . 20.6 Formulation of Grasping Optimization Problem Using Required External Force Set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20.7 Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20.7.1 Subproblems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20.7.2 Procedure of the Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . 20.8 Numerical Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20.9 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

XVII

401

401 402 403 404 406 408 408 409 413 413 413 414 415 415 416 416 417 418 418 420 421

425 425 428 428 429 430 431 431 431 433 434 436 438 439 440 443 446

XVIII

Contents

20.10 Remark for Gravitational Term of Fingers . . . . . . . . . . . . . . . . . . . 446 20.11 Proof for That Subproblem1 Can Be Solved by Solving Subproblem2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 447 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 449 21 Triangular Targeting Algorithm (TTA) for Accurate Motion Control of Fast Mobile Robots . . . . . . . . . . . . . . . . . . . . . . Gourab Sen Gupta, Chris Messom, Serge Demidenko 21.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.2 Classical PID Control for Robot Positioning and Orientation . . . 21.3 Robot Orientation Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.4 Robot Position Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.5 Advanced Robot Positioning Algorithm . . . . . . . . . . . . . . . . . . . . . 21.5.1 Pre-processing Inputs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.5.2 Re-targeting System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.6 Triangular Targeting Algorithm (TTA) . . . . . . . . . . . . . . . . . . . . . . 21.7 Proximity Positioning Algorithm (PPA) . . . . . . . . . . . . . . . . . . . . . 21.8 Velocity Profiling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.8.1 Angular Velocity Profiling . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.8.2 Linear Velocity Profiling . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.8.3 Cross-Effects Profiling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.9 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.9.1 Angular Motion Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.9.2 Linear Motion Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.9.3 Cross Effects Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . 21.9.4 Testing Triangular Targeting Algorithm . . . . . . . . . . . . . . . 21.10 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

453 453 454 455 456 458 458 459 460 464 464 465 465 466 467 468 468 469 474 474 475

Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 477

List of Contributors

Momotaz Begum Department of Electrical and Computer Engineering University of Waterloo 200 University Avenue West Waterloo, Ontario, Canada N2L 3G1, [email protected]

Department of Electrical and Computer Engineering Missouri University of Science and Technology 301 W 16th Street, Rolla, MO 65409, USA, [email protected]

Fakhri Karray Department of Electrical and Computer Engineering University of Waterloo 200 University Avenue West Waterloo, Ontario, Canada N2L 3G1, [email protected]

K.N. Krishnanand Mobile Robotics Laboratory (GCDSL), Department of Aerospace Engineering, Indian Institute of Science, Bangalore, [email protected]

Lisa Grant Real-Time Power and Intelligent Systems Laboratory Department of Electrical and Computer Engineering Missouri University of Science and Technology 301 W 16th Street, Rolla, MO 65409, USA, [email protected] Ganesh Kumar Venayagamoorthy Real-Time Power and Intelligent Systems Laboratory

D. Ghose Mobile Robotics Laboratory (GCDSL), Department of Aerospace Engineering, Indian Institute of Science, Bangalore, [email protected] S. Yamada National Institute of Informatics, 2-1-2 Hitotsubashi, Chiyoda, Tokyo, Japan, [email protected] D. Katagami Interdisciplinary Graduate School of Science and Engineering, Tokyo Institute of Technology,

XX

List of Contributors

J2-53 4259 Nagatsuta-chou, Midori, Yokohama, Japan, [email protected] D. Wang ARC Centre of Excellence for Autonomous Systems, Faculty of Engineering, University of Technology, Sydney, Broadway, NSW 2007, Australia, [email protected] N.M. Kwok ARC Centre of Excellence for Autonomous Systems, Faculty of Engineering, University of Technology, Sydney, Broadway, NSW 2007, Australia, [email protected] D.K. Liu ARC Centre of Excellence for Autonomous Systems, Faculty of Engineering, University of Technology, Sydney, Broadway, NSW 2007, Australia, [email protected] Q.P. Ha ARC Centre of Excellence for Autonomous Systems, Faculty of Engineering, University of Technology, Sydney, Broadway, NSW 2007, Australia, [email protected]

M. Malfaz Universidad Carlos III de Madrid, Avda de la Universidad 30, 28911-Leganes, Madrid, SPAIN, [email protected] M.A. Salichs Universidad Carlos III de Madrid, Avda de la Universidad 30, 28911-Leganes, Madrid, SPAIN [email protected] Jim Pugh Distributed Intelligent Systems and Algorithms Laboratory ´ Ecole Polytechnique F´ed´erale de Lausanne, Lausanne, Switzerland, [email protected] Alcherio Martinoli Distributed Intelligent Systems and Algorithms Laboratory ´ Ecole Polytechnique F´ed´erale de Lausanne, Lausanne, Switzerland, [email protected] Patrick M. McDowell Southeastern Louisiana University, Computer Science Department, Hammond, LA 70402, USA, [email protected]

R. Barber Universidad Carlos III de Madrid, Avda de la Universidad 30, 28911-Leganes, Madrid, SPAIN, [email protected]

Brian S. Bourgeois Naval Research Laboratory, Code 7440.5, Stennis Space Center, MS, USA 39529, [email protected]

V. Egido Universidad Carlos III de Madrid, Avda de la Universidad 30, 28911-Leganes, Madrid, SPAIN, [email protected]

Frederick E. Petry Naval Research Laboratory, Code 7440.5, Stennis Space Center, MS, USA 39529, [email protected]

List of Contributors

XXI

Dongbing Gu Department of Computing and Electronic Systems, University of Essex, United Kingdom, [email protected]

Faculty 3-Mathematics /Computer Science, University of Bremen, Robert-Hooke-Straße 5, D-28359, Bremen,Germany, [email protected]

Jindong Liu Department of Computing and Electronic Systems, University of Essex, United Kingdom, [email protected]

Terrence P. Fries Department of Computer Science, Coastal Carolina University, Conway, SC 29526, USA, [email protected]

Huosheng Hu Department of Computing and Electronic Systems, University of Essex, United Kingdom, [email protected]

Sukumar Kamalasadan Department of Engineering and Computer Technology University of West Florida, Pensacola, FL-32514, USA, [email protected] Adel A. Ghandakly Department of Electrical and Computer Engineering California State University, Chico, CA- 95929, USA, [email protected]

Yohannes Kassahun Robotics Lab, Faculty 3-Mathematics /Computer Science, University of Bremen, Robert-Hooke-Straße 5, D-28359, Bremen, Germany, [email protected] G.B. Parker Computer Science Jan Hendrik Metzen Connecticut College German Research Center for Artificial 270 Mohegan Ave, New London, CT, Intelligence (DFKI), USA, Robert-Hooke-Strae 5, [email protected] D-28359, Bremen, Germany, jan [email protected] N. Wiebe Department of Computer Science Mark Edgington University of Manitoba Robotics Lab, Faculty Winnipeg, Manitoba, Canada 3-Mathematics/Computer Science, R3T2N2, University of Bremen, [email protected] Robert-Hooke-Straße 5, D-28359, J.E. Anderson Bremen, Germany, [email protected] Department of Computer Science University of Manitoba Frank Kirchner Winnipeg, Manitoba, Canada German Research Center for Artificial R3T2N2, Intelligence (DFKI), Robotics Lab, [email protected]

XXII

List of Contributors

D.C. Asmar Department of Mechanical Engineering, American University of Beirut, Riad El Solh 1107 2020, P.O. Box 11-0236, Beirut, Lebanon, [email protected] S.M. Abdallah, Deloitte, Al Fahad & Co. for Consulting W.L.L. Fahad Al-Salem Street? Salhia Complex, Entrance 2 P.O. Box 23049, Safat 13091? Kuwait City, Kuwait, [email protected] J.S. Zelek Department of Systems Design Engineering, University of Waterloo, 200 University Ave. W., Waterloo, Ontario, Canada, N2L 3G1, [email protected] F. Hoppe Cognitive Systems, Institute of Computer Science, Faculty of Engineering, Christian-Albrechts-University, Kiel, Ohlshausenstr. 40, 24098, Germany, [email protected]

J.E. Anderson Department of Computer Science University of Manitoba Winnipeg, Manitoba, Canada R3T2N2, [email protected] Y.H. Tsoi Medical and Rehabilitation Robotics Research Group, Department of Mechanical Engineering, The University of Auckland, New Zealand, [email protected] S.Q. Xie Medical and Rehabilitation Robotics Research Group, Department of Mechanical Engineering, The University of Auckland, New Zealand, [email protected] A.E. Graham Medical and Rehabilitation Robotics Research Group, Department of Mechanical Engineering, The University of Auckland, New Zealand, [email protected]

G. Sommer Cognitive Systems, Institute of Computer Science, Faculty of Engineering, Christian-Albrechts-University, Kiel, Ohlshausenstr. 40, 24098, Germany, [email protected]

Hector Hugo Aviles-Arriaga Departamento de Ciencias de la Computaci´ on. Instituto de Investigaciones en Matem´aticas Aplicadas y en Sistemas (IIMAS) Universidad Nacional Aut´ onoma de M´exico (UNAM) Mexico, [email protected]

J. Baltes Department of Computer Science University of Manitoba Winnipeg, Manitoba, Canada R3T2N2, [email protected]

L. Enrique Sucar Instituto Nacional de Astrofisica, Optica y Electronica Computer Science Department. Sta. Ma. Tonantzintla, Mexico, 72840, [email protected]

List of Contributors

Eduardo F. Morales Instituto Nacional de Astrofisica, Optica y Electronica Computer Science Department. Sta. Ma. Tonantzintla, Mexico, 72840, [email protected] Blanca A. Vargas Instituto Nacional de Astrofisica, Optica y Electronica Computer Science Department. Sta. Ma. Tonantzintla, Mexico, 72840, [email protected] Josue Sanchez Instituto Nacional de Astrofisica, Optica y Electronica Computer Science Department. Sta. Ma. Tonantzintla, Mexico, 72840, [email protected] Elva Corona Instituto Nacional de Astrofisica, Optica y Electronica Computer Science Department. Sta. Ma. Tonantzintla, Mexico, 72840, [email protected] Tetsuyou Watanabe Graduate School of Natural Science & Technology, Kanazawa University, Kakuma-machi, Kanazawa, 920-1192, Japan, [email protected]

XXIII

ZhongWei Jiang Graduate School of Science and Engineering, Yamaguchi University, Ube, 755-8611, Japan, [email protected] Tsuneo Yoshikawa College of Information Science and Engineering, Ritsumeikan University, Kusatsu, 525-8577, Japan, [email protected] Gourab Sen Gupta School of Engineering and Advanced Technology (SEAT), Massey University, Private Bag 11 222, Palmerston North, New Zealand 4442, [email protected] Chris Messom Institute of Information and Mathematical Sciences (IIMS), Massey University, Private Bag 102-904, Auckland, New Zealand 0745, [email protected] Serge Demidenko School of Engineering and Advanced Technology (SEAT), Massey University, P.O. Box 756, Wellington, New Zealand 6140, [email protected]

1 Computational Intelligence Techniques in Bio-inspired Robotics M. Begum and F. Karray Department of ECE, University of Waterloo, Canada {m2begum,karray}@uwaterloo.ca

Abstract. Biologically inspired robotics is a rapidly emerging research area standing at the cross-cut of biology, artificial intelligence, and robotics. The structural and functional perfection of the biological creatures achieved through thousands of years of evolution in nature have made them a great source of inspiration for designing complex autonomous systems. Accordingly, robotics researchers have initiated designing artificial machines while considering the characteristics of biological systems as the benchmark knowledge. ‘Biomimetics’, an emerging research area, is a consequence of such initiatives. Bio-inspired robotics is actually a subset of biomimectics focusing mostly on the design of autonomous robots while mimicking the intelligence, cognition, and structural properties of the primates and other lower order species. Computational intelligence techniques developed in the past several years are related closely to bio-inspired robotics as both of them have originated from the common inspiration of mimicking human intelligence, although their development have followed different paths. Bio-inspired robotics have used computational intelligence techniques to mathematically model ‘human-like intelligence behavior’ for autonomous robots. This chapter provides an overview on the tools of computational intelligence and their applications in bio-inspired robotics. It highlights the recent progress made in bio-inspired robotics while giving a special emphasis on the usage of computational intelligence tools to design human-like cognitive abilities in the robotic systems.

1.1 Introduction Biologically inspired robotics is a multidisciplinary research area motivated by the inspiration of using nature as the benchmark in designing complex autonomous systems and, at the same time, benefiting nature with the improved system autonomy. Biological creatures, with their adaptability, structural sophistication and varieties in behavior, have always been a great source of inspiration for the researchers in robotics. But substantial research in bio-inspired robotics has started only a few years back. The reasons behind such a delayed start is mainly due to our limited understanding of nature, and to the lack of technological sophistication to mimic biological systems, both at the structural as well as the intellectual level. During the past two decades we have achieved remarkable improvement in closely pertinent areas of bio-robotics such as nano-technology, artificial intelligence (AI), and neuroscience. Achievements in these three, apparently disjoint, disciplines have placed us in a position to perform reverse engineering on biological systems. Increased computing power in gradually diminishing sized computer chips are showing the glimpse of possibility of having computers D. Liu, L. Wang, and K.C. Tan (Eds.): Design and Control of Intel. Robotic Systems, SCI 177, pp. 1–28. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com 

2

M. Begum and F. Karray

comparable to the primates brain (could be lower order primates) in terms of processing power, complexity, and compactness. Our steadily developing understanding of living elements, especially the mechanism of brain functions, theory of interaction among mind, body, and brain, is opening up possibilities of mimicking them in a more robust manner. The AI and related disciplines have already earned remarkable improvement in developing sophisticated computational tools, industry-strength algorithms for machine learning, image, and speech processing which may help to successfully model the functionality of biological systems. At the midst of resonance in development among the related disciplines, bio-inspired robotics has become a topic of significant interest. A substantial amount of research work has been reported during the current decade in various sub-disciplines of bio-inspired robotics and the flow is continuously rising. This chapter provides a brief survey on the trends of bio-inspired robotics while giving a special focus on how its development has been influenced by the approaches rooted in computational intelligence (CI). The rest of the chapter is organized as follows. Section 1.2 sheds light on the history of AI, bio-inspired robotic, and CI techniques, their common origin and underlying goals. Section 1.3 provides a brief discussion on bio-inspired robotics as an emerging research area and highlights existing research directions in bio-inspired robotics. Section 1.4 focuses on one of the most appealing branch of bio-inspired robotics namely, cognitive modeling for autonomous robots. It focuses on different emerging areas of cognitive modeling for autonomous robots, provides a brief survey on the state-of-the art of these areas and sheds light on the role of CI techniques in designing artificial cognition for autonomous robots. Finally, section 1.5 draws some concluding comments.

1.2 Bio-inspired Robotics and Tools of Computational Intelligence: The Historic Synergy 1.2.1

Overview

Computational intelligence techniques, e.g. neural networks, fuzzy logic, genetic algorithms, probabilistic algorithms, are well known to be biomimetic computational tools designed from the inspiration of mimicking the structural and functional elegance of the human to perceive the environment and act accordingly while applying their intelligence. On the other hand, the dream of a human-like machine (in terms of structure and intelligence) was there since the birth of AI. The CI techniques and AI, therefore, should have been complementing each other in the course of their own development as well as in building a complete model of human-like intelligence. In reality, however, a different scenario has taken place. The development of CI techniques followed a different path than that of the AI [1]. The CI techniques have experienced tremendous theoretical growth over the past few decades and have also earned popularity in application areas e.g. control, search and optimization, data mining, knowledge representation, signal processing, and robotics. Specifically in AI robotics, a good number of robotic systems uses CI techniques for robust control, planning, and decision making [2, 3, 4, 5]. Unfortunately, these applications do not reflect the vision with which the CI techniques were originally developed for: modeling human intelligence [6]. The AI, on the other

1 Computational Intelligence Techniques in Bio-inspired Robotics

3

hand, ruled by the norms of GOFAI(R) (Good Old Fashioned AI (and Robotics)) [7, 8] for an extended period of time, failed in many ways to utilize the power of CI techniques in designing human-like intelligence. Thus the long-expected synergy between AI and CI was delayed until the AI revolution of 1990s [9] which replaced the static notion of “ ‘mind-as-computer’ in a symbol-based representation of the world” with the dynamic idea of ‘embodied cognition’. Embodied cognition advocates the notion of bidirectional active interaction among body, mind, and environment as the building block of developing intelligence, both natural and artificial [9, 10, 11, 12, 13]. The current AI robotics, therefore, is focusing on modeling the highly complex, non-linear interaction among mind, body, and environment, the self-motivation to build knowledge base, the automated reasoning, and decision making. The ideas of embodied cognition places CI techniques back in the stage of AI. The CI tools blend naturally with these concepts as they share the common principles of biology. The CI techniques, therefore, are very efficient tools to design bio-inspired intelligence in autonomous robots. 1.2.2

The CI Tools Most Commonly Used in Bio-inspired Robotics

The CI tools developed in the past several years have been widely used in different branches of bio-inspired robotics. New research challenges with higher complexities are arising as we are approaching toward the goal of designing human-like intelligent machines. To meet these challenges new CI tools are being developed as well as modification of the existing tools are being performed. For instance, artificial immune algorithm [14] has been developed based on the inspiration of human immune systems and type 2 fuzzy logic [15] has been proposed as an extension of tradition fuzzy logic to include probabilistic uncertainty in fuzzy reasoning. A brief introduction on the CI tools most commonly used in the existing bio-robotics literature is provided here. 1.2.2.1 Artificial Neural Network Artificial neural network (ANN) is one of the most commonly used CI tools in bioinspired robotics [16]. ANNs mimic the highly parallel structural connectivity and nonlinear operating principle of the brain cells (known as neuron) of biological creatures [17]. The structure of a single artificial neuron (also known as perceptron) is very simple: input terminal(s), one output terminal, and a function defining the mapping from input(s) to the output. The simple perceptrons connect with each other in layered architectures to form different kind of neural networks which are generally capable to handle even very high degree of non-linearity. One elegant property of ANNs is they allow the exact pattern of non-linearity to emerge within themselves through the process of training. There are algorithms to conduct efficient training of ANNs. The theories of neural network have gone through several facets of development in the past decades resulting in varieties of ANNs suitable for different application areas, e.g. data mining, robotics, communication. 1.2.2.2 Fuzzy Reasoning Fuzzy reasoning is another CI tool which mimics the capacity of human mind to process partial truth in observed phenomena [18]. Fuzzy logic, the mathematics of fuzzy

4

M. Begum and F. Karray

reasoning, advocates that the traditional crisp (or binary) logic which deals with the two extreme ends (‘yes’and ‘no’) of the decision paradigm is just a special case of fuzzy logic. Fuzzy reasoning, therefore, is governed by the norms of partial membership and assigns a degree belief in favor of a proposition. Such degree of belief may take any value between zero to one. The elegance of fuzzy logic lies in its simplicity to model even extremely complex, high-dimensional decision problems. This results in its widespread application in areas such as control, pattern recognition, artificial intelligence, and robotics. 1.2.2.3 Evolutionary Algorithms Evolutionary algorithms (EA) are generally considered as search and optimization technique inspired from the principle of natural selection. They provide a structured way to locate the most suitable solution in a highly complex search landscape. EAs generally perform a series of operations (selection, mutation, crossover) to iteratively optimize a fitness function. Starting form the simple form as proposed in [19], the evolutionary algorithms have went through numerous modifications to suit with the increasing complexities of todays search problems. A number of new search and optimization algorithms has been developed in recent years based on the inspiration from nature, e.g. artificial immune algorithm [14], swarm intelligence [20], ant-colony optimization [21]. The application of EAs in bio-inspired robotics is still relatively less appealing due to its high time complexity. 1.2.2.4 Probabilistic Algorithms Probabilistic algorithms have been quite useful in several areas of mobile robotics including bio-inspired robotics. This is because of their capacity to deal with the uncertainty inherent in real-world robotic problems [22]. The probabilistic algorithms used in bio-inspired robotics are generally related to Bayesian inference, Bayes network and different variants of Bayes filter [23, 24, 25]. The mechanism of posterior formation based on prior knowledge and observation likelihood makes Bayes theorem a suitable candidate in modeling intelligence behavior for robotics systems. Besides, the mathematical structures of Bayes network and Markov model are well suited for state based robotic systems.

1.3 Bio-inspired Robotics: An Emerging Research Field Bio-inspired robots are a new kind of research platforms developed to fulfill the twopronged desire of humans: enhancing the quality of human life by incorporating artificial systems in it and, at the same time, improving the system design through integrating biology with it. The major driving forces behind the research on bio-inspired robotics, therefore, can be summarized as follows. • Robots to assist human: During the last decade mobile robotics has experienced a significant forward shift toward real-world applications from the laboratory based research. The Mars rover mission [26], winning of DARPA grand challenge 2005

1 Computational Intelligence Techniques in Bio-inspired Robotics

5

[27], robotics application in surgery [28] and health care [29] are among few examples. The bio-inspired robots are expected to top the current successes of mobile robots with their human-like sophistication in perception, reasoning, decision making and locomotion. These robots will be able to adapt autonomously in changing environment, will apply learned knowledge and reasoning capacity to solve new problems, and above all will be able, to a large extent, to take care of themselves [30]. This sort of machines have potential to drastically improve the quality of human life through providing assistance in several areas such as search and rescue, mining, space-applications, military- surveillance, entertainment, surgery, and health care. • Robots to understand human: The other inspiration of bio-inspired robotics is to have a better understanding of the capacities that the humans are blessed with [31]. Until recently, the flow of knowledge between biology and robotics was unidirectional, from biology to robotics. During the last decade a bi-directional link has been established between biology and robotics based on the realization that lifelike machines provide a wonderful platform to quantitatively test and analyze the computational models and theories from neuroscience [13, 32], psychology, developmental study [33], and human locomotion [34, 35]. As mentioned in [36], the bio-inspired robots act as the ‘mirror to reflect our humanity back at us as we interact with them’. Designing a robot with human-like sophistication in structure and intelligence is a crossdisciplinary endeavor and requires intense collaboration among disciplines such as biology, neuroscience, psychology, linguistics, robotics and AI. The efforts made in this endeavor can be categorized into four distinct categories. • Biomimetic perception: Perception is the key requirement for survival of the living elements [37]. The pre-requisites of primates- like reasoning and decision making is to have primates- like perception. The perceptual system, therefore, is one of the most significant aspects of bio-inspired robot design. There is a steady progress in the development of biomimetic sensors and actuators technology. Such sensors and actuators enable a robot to perceive and interact with the environment in primateslike fashion (more specifically, human like). In other words, biomimetic sensors and actuators help the robots to perceive the same representation of the world as human does and thereby ease the blending of robots in the human environment. The log-polar camera for human like vision [38], haptic devices to mimic touch sensitivity [39], micro electro-mechanical systems (MEMS)[40] and elector-active polymer (EAP) to model the biological muscles [41] are among the many examples of biomimetic sensors and actuators. The possibility of integrating soft tissue to construct flexible and deformable robots are on the way to become full-fledge areas of research and applications [42]. A large group of robotics researchers are currently working on reverse engineering the performance characteristics of biological sensors to formalize control mechanisms of biomimetic sensors as well as their power requirements and efficiency. • Locomotion and control: Locomotion of autonomous robots based on the motion primitives or imitation of biological creatures is a rapidly emerging research area in bio-inspired robotics. The major focus is to mimic lower level sensory-motor

6

M. Begum and F. Karray

skills of biological creatures (ranging from insect to human) in autonomous robots. Bipedal walking by humanoid robots [43, 44], humanoid upper body control [45, 46], design and control of articulated robots e.g. reconfigurable robots [47, 48], crawling and jumping robots [49, 50, 51] are only a few examples of widespread research in bio-inspired locomotion control in robotics. • Control architectures and learning methodologies: These are the sub-discipl-ines of bio-inspired robotics which have been using bio-inspired mechanisms for a long period of time. Majority of the existing control architectures used in autonomous robotics are bio-inspired. The traditional reactive [52], deliberative, and hybrid [53] controllers are inspired from insect locomotion control mechanism whereas the relatively newer behavior-based controller [54] obtains inspiration from human cognition. Similarly, the learning methodologies commonly used in robotic, e.g. reinforcement learning, associative learning, classical conditioning, automated animal like (AA)-learning [55], neural network and statistical learning, are designed based on the learning process of biological creatures ranging from insect to human. • Cognitive modeling: Cognitive modeling deals with implanting human-like cognition, albeit reduced complexity, in autonomous robots with the hope that the cognitive capacities, as they work in human and other biological creatures, will also assist the new generation robots to be coherent, self-motivated, social, persistent in behavior, and aware of themselves and their environments. Cognitive modeling, therefore, requires integration of neuroanatomy, cognitive neuroscience, cognitive psychology, developmental study, linguistics, and AI with robotics. A rapidly increasing number of publications on different aspects of cognitive modeling indicates the growing interest in this research area, although a complete model of robot cognition does not yet exist [56, 57]. The latest findings of neuroscience and psychology are continuously modulating the research on cognitive modeling for autonomous robots. Apart from few general theories on robot cognition (e.g. [9, 46, 58, 59, 60, 61]), the majority of the literature is focused on developing models for discrete cognitive abilities, e.g. attention, learning, value system, reasoning, and decision making. The research on cognitive modeling for autonomous robots, therefore, is still in its infancy and there is a long way to traverse before having a full fledge autonomous robot with human-like cognition. A large number of university research laboratories around the globe is performing research on different areas of bio-inspired robotics: the biologically inspired robotics group at Ecole Polytechnique, the biorobotics laboratories at Harvard University, University of Washington, and MIT, the cognitive robotics labs at Vanderbilt University, Technical University of Munich, University of Genova, Michigan State University, University of Southern California, and Idaho National Laboratory. A growing number of conferences (e.g. EpiRob, ICDL) and journals (e.g. International Journal of Humanoid Robotics, special issues on bio-robotics of the IEEE Transaction on Robotics and Connection Science) are now fully dedicated to the findings from bio-inspired robotic research. The rest of this chapter focuses on the emerging use of CI techniques in cognitive modeling for autonomous robots. It provides a survey on the state-of-the-art of

1 Computational Intelligence Techniques in Bio-inspired Robotics

7

cognitive modeling in pursuit of human-like intelligent machine and investigates on how the CI tools have been invoked in modeling different aspects of cognition. It also advocates the idea that the features of cognition emerges naturally in CI techniques, when properly utilized. The CI tools, therefore, offer a very convenient way to model artificial cognition in autonomous robots.

1.4 Cognitive Modeling for Autonomous Robots: CI-Based Approaches With the tremendous advancement of computer systems, both in speed and storage capacity, AI has achieved significant strides during the past decades. Several AI techniques have emerged as a consequence of the enriched computational power. The advancements in AI robotics, to a large extent, is the consequence of this enriched computational power. However, today’s computers with their enormous speed and storage capacity essentially act as mindless intelligence, executing complex routines with extreme efficiency but showing minimal awareness about the tasks they are performing. AI researchers, therefore, have started working on developing cognitive computers which will be able to learn from its experience and apply the learned knowledge to deal with anomalies [62]. Initiatives taken by AI robotics researchers are one step ahead of this effort. Their idea is to develop embodied physical agents namely, cognitive robots, which are expected to act, behave, and think like humans while interacting with natural human environments. In pursuit of such an ideal goal, the only target device to mimic has always been the human brain. With the advent of new technologies, such as fMRI (functional Magnetic Resonance Imaging) [63], PET (Positron Emission Tomography) [64], neuroscience has been experiencing a significant forward shift toward a better understanding of human brain, yet major part of it is still unknown [37]. The advancements in neuroscience have introduced a remarkable change in the perception of the psychologists about human cognition and have encouraged the thought of explaining cognitive developments in terms of the changes in physical structure of the brain [65, 66, 67, 68]. 1.4.1

Cognition: Robot and Human

Since antiquity cognition has been a major research area in psychology and philosophy. It is probably one of the areas where modern neuroscience and psychology have performed the most significant improvement, although there is still a lot more to know [37, 68]. Elucidating cognition in a precise manner is difficult, yet all of us are generally aware of our cognitive abilities. In simple words, ‘cognition’ of a living element refers to its capacity to process perceptual information and thereby manipulating its behavior. Human cognition encompasses a large collection of processes occurring in human mind. The cognitive capabilities of humans are generally manifested in the following [10, 11, 37, 66, 67, 68]. • • • •

Self awareness, perception, learning, knowledge,

8

M. Begum and F. Karray

• reasoning, • planning and decision making. Generally, action execution is not considered as a part of cognition, although cognitive development is largely influenced by our capability to act upon the environment. Besides, action is the only explicit means onto which we reflect our cognitive skills. Designing an artificial agent with some or all of these powerful capabilities is the new grand challenge of AI [30, 62]. A number of attempts have been made over the past few years to meet this challenge and gave birth to a number of new research areas including ‘developmental engineering’ [46], ‘epigenetic robotics’ [69], ‘developmental robotics’ [58], and ‘developmental cognitive robotics’ [60]. All of them share the common vision of designing human-like cognitive agents, irrespective of their names and the methodologies applied. This chapter addresses these ‘similar vision’ efforts as ‘cognitive modeling for autonomous robots’. Based on the recent neuro-biological and psycho-physical findings about executive cognitive functions of human being (e.g. attention, learning, perception, social

Fig. 1.1. Example robotic systems based on bio- inspired cognitive models (a) Kismet (MIT), (b) ISAC (Vanderbilt University), (c) SAIL robot (Michigan University) (d) Babybot (University of Genova)

1 Computational Intelligence Techniques in Bio-inspired Robotics

9

communication), an impressive amount of research work has been carried out during the current decade in the following sectors of cognitive modeling: 1. Value system 2. Visual attention 3. Social cognition The rest of this chapter will provide a brief survey on the state-of-the-art of each of these sectors of cognitive modeling. Two criteria have been defined for a research work to be included in the survey presented in this chapter. • The research has to apply computational intelligence (CI) techniques to deal with the problem in the associated research area. • The research has to demonstrate the performance of the designed intelligence framework using any physical robotic system. The representative works in different sectors of cognitive modeling (value system, visual attention, and social cognition) are summarized and presented in tabular form for convenience in comparison. 1.4.2

Value System

The value system refers to the capacity of a biological brain to increase the likelihood of neural response to an external phenomena provided that there exists a prior experience of having the similar phenomena [70]. The combined action of internal perception, reasoning and decision making capacity contributes in development of values in the primates. Value is not hard-wired in human since birth, rather it grows developmentally while mediated by the plasticity of human brain [10, 11, 37]. Accordingly, the value with which a biological entity is born with is termed as ‘innate value’ whereas the value that it achieves through experience during the course of development is called ‘acquired value’. The plasticity of human brain, which makes the development of acquired value possible, is a recent discovery in neuroscience. The acquired value of a biological entity is reflected in its behavioral adaptivity. The robotic counterpart of value refers to the capacity of an autonomous robot to plan action upon detection of salient stimuli. A robot with value system performs this action planning after analyzing its internal and external context [13, 71]. Internal context of a robot is the knowledge-base developed through past experiences whereas external context refers to the current environmental situation in favor of a planned action. A value-based robot, therefore, is aware of its own action. Value system is a crucial requirement of developing human-like intelligence in robotic systems. The success of designing truly developmental robot depends largely on the design of a value system. The basic idea of the artificial value system for autonomous robots is to design a mechanism which can reflect the effect of the past experience in the future behaviors (e.g. in planning of specific action, perceptual categorization, conditioning, autonomous exploration) of the robots. Artificial value system essentially tries to modulate the behavior of an autonomous system (e.g Fig. 1.2 shows the modulating effect of value system in the SAIL robot cognitive architecture [72]).

10

M. Begum and F. Karray

Fig. 1.2. Modulating role of value system in the cognitive architecture of SAIL robot in university of Michigan (adapted from [72])

Considering value system in the primates as benchmark [73, 74], the properties expected in an artificial value system for the autonomous robots can be summarized as follows. • Prediction: Predictive power is an important requirement of a value system. The system should be able to infer the consequences of its own action before actually executing it. This sort of predictive action planning requires the capacity to apply reasoning and knowledge-base to analyze perceptual information. • Task non-specific: An artificial value system must be general and should operate in a task non-specific manner. It must be unsupervised and should enable a robot to learn by itself (from the consequence of its action) without any help from human instructor. Instructional scaffolding is a very popular mechanism in the primates (specially, human being) to boost value system development during the very early stage but the supervision obtained through scaffolding is expected to be decayed gradually in order to facilitate the natural cognitive development [80]. In a similar manner, limited support is allowed during the early development of artificial value system in robotic agents but, gradually, the system should be capable of evolving in a an autonomous manner based on the interaction with its environments. • Developmental: Similar to the value system in biological entity, the artificial value system might have innate and acquired components. Innate value, in artificial systems, is modeled through assigning prior determined importance to different actions. The acquired value, on the other hand, should generate developmentally. An artificial value system must have the plasticity to continuously encode the feedback that it receives from its interaction with environment. This encoding of experience should be in such a way that facilitate emergence of adaptive behavior. • Value-based Learning: Learning is the prime mechanism which get modulated through the activities of value system. An artificial values system, therefore, should be capable of generating modulating signals to adapt various learning parameters, e.g bias (positive or negative), learning rate.

1 Computational Intelligence Techniques in Bio-inspired Robotics

11

Table 1.1. Research on artificial value system for autonomous robots

Lead research

Synopsis

CI technique

Robot

Value system [13]

Development of a neuromodulatory system based on conditioning and categorization

NN (continuous) firing rate model

Khepera (Monad)

Value system Development of Q-learning for developmental motivation and robot action planning [71, 72, 75] through value system

HR (SAIL robot)

Intelligent Adaptive Curiosity [76, 77]

Intrinsic motivation NN or generation for any standard action planning learning algorithm

PR (AIBO)

Adaptive behavior [78]

Development of a self-modulating value system

NN

MR (DARWIN V)

Value system Value guided for categorization exploration [79] and perceptual categorization

NN

Khepera

PR: Pet Robot

HR: Humanoid Robot

MR: Mobile Robot

Table 1.1 summarizes the research on value system design in autonomous robots. The initial attempts to design artificial value system were made by a group of researchers using robotic systems as a testbed to simulate the theories of primates nervous system [59, 81, 82, 83]. The efforts on designing artificial value system with the goal of developing human-like intelligence in robotic systems, however, are very limited. With the advancement of developmental robotics, the value system design for autonomous robot is gradually gaining attention of the bio-inspired robotics researchers [13, 71, 72, 75, 76, 77, 78, 79]. Artificial Neural networks generally dominate as a design tool in the development of artificial value system. The reason behind this, partly, is our current understanding about the operation of the primates value system. The findings on developmental plasticity suggest that experience drastically changes the ease with which neurons interact with each other [10, 11, 37]. The architecture of artificial neural networks provides a suitable way to model this synaptic modulation as adaptive weight adjustment among connecting neural processing elements.

12

1.4.3

M. Begum and F. Karray

Visual Attention

Attention, in a biological system, refers to the process of selecting a set of relevant information for further processing. It plays a key role in the survival and normal operation of the biological entities as the gate-keeping activity of attention saves the information processing unit (generally the brain) from being flooded with enormous amount of information. Computational modeling of the primates attention mechanism (more accurately, visual attention mechanism) has become a very popular research area in the recent years due to it potential applications in computational neuroscience, computer vision, automated video surveillance, and AI robotics. The purpose of attention mechanism, when applied on the artificial systems, is essentially the same as that in the biological systems: focusing on behaviorally relevant information for further processing. A properly designed attention system provides a task-executing robot with the capacity to blend with human in natural human environment. Extensive research has been performed on the visual attention mechanism of the primates, both in psychology [65, 68, 84, 85, 86, 87, 88, 89] and neurobiology [90, 91, 92, 93, 94, 95]. Based on these research a good number of bio-inspired computational model of visual attention has been proposed in computer vision [96, 97, 98, 99, 100, 101, 102, 103, 104] and in computational neuroscience [105, 106, 107, 108, 109]. The models in computer vision mostly focus on the technical aspects of the attentional mechanism while the goal of the models in computational neuroscience is to simulate the findings from the primates behavioral data. In the recent years, with the emergence of bio-inspired robotics, the robotic researches have started working on developing artificial attention mechanism for embodied physical robots. Accordingly, a number of attention models has been proposed in robotics literature which are dealing with embedding artificial attention mechanism in robotic systems [103, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121]. This chapter categorizes the existing literature on visual attention into three different groups. • Theoretical model for visual attention: This group includes the theoretical models of visual attention which are based on CI technique and have been implemented in robotic systems for attention modeling. The most notable work in this category is the neuromorphic vision tool-kit (NVT) [99, 122](Fig. 1.3). NVT is an extension of the first computational model of visual attention proposed in [97]. It uses winnertake-all type neural network (WTA-NN) for selective attention modeling. The WTA NN have earned huge popularity in the later years and is a key tool for attention modeling in majority of the existing visual attention models, although it has been considerably modified from its original form as proposed in [97]. A good number of works in robotics uses different variants of WTA NN and NVT to implement visual attention [111, 113, 123, 124, 125]. • Robotic model of visual attention: The visual attention models developed specifically for robotic systems are included in this category [111, 116, 126, 127, 128, 129, 130]. A considerable number of model in this category are based on NVT and WTA-NN. Apart from that, competitive NN, radial basis function (RBF), self organizing map (SOM), arrays of neural processing elements (PE) are popular CI tools used for visual attention modeling.

1 Computational Intelligence Techniques in Bio-inspired Robotics

13

Fig. 1.3. Neuromorphic vision toolkit: an architecture of visual attention mechanism (adapted from [99])

• Task specific visual attention model: Focusing attention on an object of interest is the pre-requisite of learning skill development. A substantial number of robotic system, therefore, use visual attention as the basic mechanism to achieve different cognitive skills. These systems generally develop task-specific models of visual attention where the task in hand might be object manipulation [131, 132, 133, 134], different forms of human robot interaction (HRI) [61, 135, 136, 137, 138, 139], and robot navigation [125]. For quick reference, tables 1.2, 1.3, and 1.4 summarize the leading researches under each of the above mentioned categories, respectively. The existing models of visual attention have achieved significant success addressing the issues of ‘what to attend’, ‘when to attend’, and ‘how to attend’[103]. The majority of the visual attention models for the autonomous robots use different kind of artificial neural networks. The reason behind this, partly, is that the earlier computational models of visual attention were NN-based. The another reason is that the enriched formulation of artificial neurons act as a very strong tool to mimic (albeit reduced complexity) the neural circuity of visual cortex dedicated to attention selection. As the goal of bio-inspired robotics is to mimic the functionality of biological system rather than the exact mechanism, there are possibilities for other computational techniques to be used in developing visual attention models. For instance, fuzzy reasoning could play a substantial role in visual search modeling because of its tolerance

14

M. Begum and F. Karray Table 1.2. Theoretical models of visual attention

Lead research

Synopsis

CI technique

Adaptive Resonance theory [96]

A neural network Neural network based on attention mechanism. Combines top-down and bottom -up attention

Koch’s Model [98], Computational model Winner-take-all Neuromorphic of visual attention. neural network vision toolImplements saliency (WTA NN) kit (NVT) map and inhibition [99, 122] of return (IOR) Selective tuning model [101]

Conncectionist model Neural network of visual attention

Feature-gate model [102]

Conncectionist model Spatially distributed, of visual attention. hierarchical neural Combines feature network integration theory [84] and guided search [85]

to partial truth [56]. Further, statistical learning methodologies offer great versatility in the learning of attention based cognitive skill. Specifically the probabilistic architecture provides the most suitable way to handle the perceptual and decision uncertainty inherent in robotics applications. We have not yet found a general purpose model of visual attention capable to tackle all the existing challenges. Integration of different CI techniques might be a very intriguing way to walk toward this goal since it may capture different functionalities that can not be handled by an isolated CI or AI approach. 1.4.4

Social Cognition

Social cognition development in autonomous systems is a widely explored research area in bio-robotics. The research on socially interactive robots has been experiencing a steady development since the birth of biologically inspired robotics. A number of significant achievements have been made, but researchers are interested about very high-level social interaction conducted by robotic agents. A number of different classifications and taxonomies are available for socially interactive robots ([33, 36, 158, 159] are suitable for extensive survey). This document focuses on the body of research dedicated to design task non-specific social cognition in robotic agents as a part of fulfilling the goal of designing human-like cognition in robotic systems. Such research works generally involve the theories of social development of the human proposed in psychology, ethology, linguistics, neuroscience and developmental cognitive neuroscience. The target robotic systems for these approaches may possess physical embodiment but

1 Computational Intelligence Techniques in Bio-inspired Robotics Table 1.3. Robotic model of visual attention

Lead research

Synopsis

CI technique

Distributed attention [113]

Multi-processor attention model

WTA NN 7 DOF (Koch’s model) HH

Object-based attention [111]

Considers object WTA NN as elemental unit (NVT) for attention. Evaluates objectbased saliency map.

Attention network [140]

Efficient visual search by robotic system

Neural network CH (Feature gate model)

Overt attention [110]

Implements space -based saccade and IOR

Competitive NN HR [141]

Attention and Implements space saccade generation -based saccade [123]

WTA NN CH (Koch’s model)

Autonomous mental development (AMD) [58, 142] [143, 144, 145]

Hierarchically HR connected neural (SAIL robot) processing elements (PE)

A developmental design of visual attention mediated by subsumption [54]

Robot

HR (Babybot)

NeuroBotic Object classifier system based on task[116, 126, 127, 128] relevance

Hierarchical MR radial basis (Peoplebot) function (RBF)

MirrorBot project [129, 130]

A distributed model of spatial attention

Continuum neural field theory

MR (peoplebot)

Multi modal attention [146, 147]

Directing visual attention to respond stimuli arising through different modality

NN

CRA

Brain based devices [32, 59] [148]

Attention-based network of perceptual mean firingcategorization rate neurons and visual binding

MR (Darwin VII)

HH: Humanoid Head

HR: Humanoid Robot

CRA: Camera on Robot Arm

CH: Camera Head

15

16

M. Begum and F. Karray Table 1.4. Task-specific model of visual attention

Lead research

Synopsis

CI technique

Robot

Attention for manipulation and reasoning [131]

visual attention Coupled dynamic MR helps to perform neural field manipulation task and plan task activities

Attention guided visual attention object manipulation mechanism help by robotic hand a robot to focus [132] on target object for grasping task

NN

CH

Visual attention for mobile robots [133, 134]

A connectionist attention model for task executing robots

Spiking and mean firingrate neurons (selective tuning model)

MR (Khepera)

Attention guided robot navigation [125]

attention mechanism is used to calculate the orientation of a robotic system

WTA NN and RBF NN

MR

Intelligent machine architecture for HRI [61, 135] [136, 137]

RBF associated with sensory egosphere implements visual attention

NN

HR (ISAC)

not necessarily human-like morphology or face, although these two have been proved to facilitate the social interaction between robots and humans. This type of socially interactive robot has been termed in the literature as ‘sociable robot’ [159]. The studies in psychology and neuroscience on social cognition development in humans identify the active interaction with other human as well as with the environment as the key criteria of social cognition development in human child [10, 80, 160]. The AI robotics research on social cognition development in autonomous systems, therefore, is primarily focusing on developing mechanisms used by the robots to interact with humans. The efforts in this area can be categorized into the following two groups. • Joint attention: Shared or joint attention refers to the ability to intentionally attend to an object/region of mutual interest. This is the ability that human infants start to achieve as early as six months [161]. Joint attention is an extremely complex process demanding the capacity to understand other peoples perspective and intention. It is a fundamental requirement for developing social cognition. An impressive body

1 Computational Intelligence Techniques in Bio-inspired Robotics

17

Table 1.5. Research on joint visual attention

Lead research

Synopsis

CI technique

Active learning of Joint attention [149, 150]

Learning joint RBF NN attention through pointing and reaching by robots

Robot HR (NICO)

Learning Passive learning Multilayer joint attention of joint attention NN, [151, 152, 153] through bootstrap Q-learning mechanism

HH

Attention for social communication [124, 154] [155, 156]

HH (Kismet)

Joint attention WTA NN between human (NVT) and robots based on eye contact

Join attention Joint attention Self organizing MR learning for HRI between human map (SOM) [138, 139] and robots based on gaze following Joint attention for imitation [157]

Imitation is Recurrent achieved through NN (RNN) joint attention

HR (Sony QRIO)

of research work has been reported in the past few years on developing the joint attention capability in robotic systems. Table 1.5 provides a brief summary of the research on joint attention. Gaze following technique has been used in a number of works to identify the intent of the partner in a social engagement [124, 138, 139, 154, 155, 156]. Using the embodiment of the robots to attract the attention of partner (e.g. through pointing fingers or waving hand) is another popular way of achieving joint attention [149, 150]. • Social imitation: Learning through imitation has gained much popularity in bioinspired robotics. Imitation provides a powerful way of developing social behavior. More specifically, the sensori-motor skills required to generate different types of social cues (e.g. pointing, reaching, waving) and task sequence learning are the two major areas where imitation based interaction between humans and robots plays a very significant role. The success of imitation based learning, however, depends on the integration with other cognitive abilities of a robotic system, e.g. attention, values system. The task of skilled imitation, therefore, has been termed as a ‘hard problem’ in [162] subjected to the issues ‘what to imitate’, ‘when to imitate’, ‘how to imitate’, and ‘what is the evaluation criteria of imitation’. Table 1.6 presents a brief summary of research on imitation based learning as a means of developing social cognition in autonomous robots.

18

M. Begum and F. Karray Table 1.6. Research on social imitation

Lead research Synopsis

CI technique

Robot

Imitation learning [163]

Motion sequence generated by a human teacher to perform simple tasks are observed and imitated

Bayesian HR learning, (Fujitsu Hidden Markov HOAP-2) model (HMM)

Learning motor skills through imitation [164]

A training phase trains the robot with different motor skills. The robot imitate learned skills in future engagements

RNN

Imitation behavior [165]

Attention guided imitation of hand posture

Reinforcement HR learning, HMM (Kenta)

HR (Sony QRIO)

Learning Manipulation task through is learned through demonstration imitating the task [166, 167] sequence executed by a human teacher

HMM, probabilistic analysis

HR (HOAP-2)

Early imitation ability [168]

Developmental approach of imitation. Learning imitation through self exploration

NN, causal learning

HR (H3)

Gaze shift learning by imitation [169]

Learning of gaze shifting inspired by saliency through imitating caregiver

HMM

MR (Pioneer)

Learning by imitation [170, 171]

Architecture NN for learning motion primitives through viewpoint transformation and visuo-motor mapping

HR

1 Computational Intelligence Techniques in Bio-inspired Robotics

19

Fig. 1.4. The humanoid robot HOAP-2 engaged in an imitation task (adapted from [163])

Similar to value system and visual attention, the research on social cognition is influenced by different types of artificial neural networks [164, 170, 171], although probability-based algorithms have earned much popularity in the design of social imitation capacity [163, 165, 166, 167, 169].

1.5 Conclusion The ultimate goal of bio-inspired robotics is to build human-like machine, in terms of morphology as well as functionality. The solemn purpose of developing human-like machine is the hope that their human-like intelligence will help them to blend naturally with humans and be a great assistance in human’s everyday operations. For instance, intelligent life-like machines will tremendously improve the quality of human life through their assistance in industry, health care, search and rescue, cleaning and household works, space-operation, education and entertainment. Design of human-like machine remains a challenge for the next generation autonomous robotics. Mimicking the mechanism of intelligence development in the biological creatures (specially in human) is currently seen to be a better option in resolving this challenge as compared to the earlier efforts of ‘hand-coding’ the intelligence in machines. In this endeavor one important issue, as mentioned in [172], is mimicking the intelligent behavior of a biological creatures does not guarantee an automatic emergence of intelligence in a target machine. Rather we have to find the underlying principle that governs the emergence of intelligent behavior. Mimicking the principle, rather than the outcome, might lead us to our goal of designing life-like machine. At this point, the CI techniques show us a relatively better hope. Majority of the CI techniques are developed based on different

20

M. Begum and F. Karray

elegant aspects of the biological creatures, e.g. morphological structure, thinking process, evolutionary selection. The CI techniques, therefore, blend naturally with the goal of bio-inspired robotics. This chapter have summarized some efforts to reach the goal of designing humanlike machine mediated by different CI techniques. The major focus was on the research on cognitive modeling for autonomous robots. Majority of the research in cognitive modeling is intended to model the small pieces of cognition, e.g. attention, value system, social interaction. Different kind of neural networks have gained more popularity and acceptability in this endeavor as compared to the other CI tools, e.g. fuzzy reasoning, genetic algorithm and probabilistic reasoning. The integration of neural network with other powerful CI tools might improve the scenario significantly, although such efforts are still in the early stages. After designing the small pieces of cognition, the most crucial part is to integrate them in a single architecture and have them interact with each other in a meaningful way so that we can see human-like manifestations of cognition as their integrated outcome. Such an architecture is yet to be developed but we hope that the CI techniques will contribute in its design, the same way they have been contributing in modeling the discrete cognitive abilities of autonomous robots.

References 1. Karray, F., Silva, C.D.: Soft Computing And Intelligent Systems Design. Addison Wesley, Pearson (2004) 2. Nolfi, S., Floreano, D.: Evolutionary Robotics: The Biology, Intelligence, and Technology of Self-Organizing Machines. Bradford Books (2000) 3. Jain, L.C., Fukuda, T.: Soft Computing for Intelligent Robotic Systems. Physica- Verlag (1998) 4. Watanabe, K., Hashem, M.M.A.: Evolutionary Computations: New Algorithms and Their Applications to Evolutionary Robotics. Springer, Heidelberg (2004) 5. Teshnehlab, M., Watanabe, K.: Intelligent Control Based on Flexible Neural Network. Kluwer Academic Publishers, Dordrecht (1999) 6. Watanabe, K., Izumi, K.: Biomimetic machines: Motivations to the research. In: SICE Annual Conference, pp. 1098–1103 (2004) 7. Haugeland, J.: Artificial Intelligence: The Very Idea. MIT Press, Cambridge (1985) 8. Newell, A.: Unified Theories of Cognition. Harvard University Press (1990) 9. Brooks, R.A.: Intelligence without representation. Artificial Intelligence 47, 139–160 (1991) 10. Johnson, M.H.: Developmental Cognitive Neuroscience. Blackwell Publishing, Malden (1997) 11. Nelson, C.A., Hann, M., Thomas, K.M.: Neuroscience of cognitive development: the role of experience and the developing brain. John Wiley & Sons, New Jersey (2006) 12. Pfeifer, R.: Robots as cognitive tools. International Journal of Cognition and Technology 1, 125–143 (2002) 13. Sporns, O., Alexander, W.H.: Neuro-modulation and plasticity in an autonomous robot. Neural Networks 15, 761–774 (2002) 14. de Castro, L.N., Timmis, J.: Artificial Immune Systems: A New Computational Intelligence Approach. Springer, Heidelberg (2002) 15. Karnik, N.N., Mendel, J.M., Liang, Q.: Type-2 fuzzy logic systems. IEEE Transactions on Fuzzy Systems 7, 643–658 (1999)

1 Computational Intelligence Techniques in Bio-inspired Robotics

21

16. Ziemke, T., Sharkey, N.E.: Artificial neural networks for robot learning. Autonomous Robots 7 (1999) 17. Haykin, S.: Neural Networks: A Comprehensive Foundation. Prentice Hall, Englewood Cliffs (1998) 18. Yen, J., Langari, R.: Fuzzy Logic: Intelligence, Control, and Information. Prentice Hall, United States (1998) 19. Goldberg, D.E.: Genetic Algorithms in Search, Optimization, and Machine Learning. Addison-Wesley Professional, Reading (1989) 20. Engelbrecht, A.P.: Fundamentals of Computational Swarm Intelligence. Wiley, Chichester (2006) 21. Dorigo, M., St¨utzle, T.: Ant Colony Optimization. MIT press, Cambridge (2004) 22. Thrun, S., Burgard, W., Fox, D. (eds.): Probabilistic Robotics. MIT Press, Cambridge (2005) 23. Wrinkler, R.L.: An introduction to Bayesian inference and decision. Wiley, New York (2001) 24. Tanner, M.A.: Tools for statistical inference: methods for the exploration of posterior distributions and likelihood functions. Springer, New York (1996) 25. Pearl, J.: Causality: Models, Reasoning, and Inference. Cambridge University press, UK (2001) 26. Mishkin, A.H., Morrison, J.C., Nguyen, T.T., Stone, H.W., Cooper, B.K., Wilcox, B.H.: Experiences with operations and autonomy of the mars pathfinder microrover. In: IEEE Aerospace Conference, vol. 2, pp. 337–351 (1998) 27. Thrun, S., et al.: Stanley: The robot that won the DARPA grand challenge. Journal of Field Robotics 23, 661–692 (2006) 28. Krupa, A., Gangloff, J., Doignon, C., Mathelin, M., Morel, G., Leroy, J., Soler, L., Marescaux, J.: Autonomous 3-d positioning of surgical instruments in robotized laparoscopic surgery using visual servoing. IEEE Transactions on Robotics and Automation 19, 842–853 (2003) 29. Miyawaki, F., Masamune, K., Suzuki, S., Yoshimitsu, K., Vain, J.: Scrub nurse robot system-intra operative motion analysis of a scrub nurse and timed-automata-based model for surgery. IEEE Transactions on Industrial Electronics 52, 1227–1235 (2005) 30. Brachman, R.J.: More than the sum of its parts. AI Magazine 27, 19–34 (2006) 31. Webb, B., Consi, T.R.: Biorobotic. MIT Press, Cambridge (2001) 32. Krichmar, J.L., Seth, A.K., Nitz, D.A., Fleischer, J.G., Edelman, G.M.: Spatial navigation and causal analysis in a brain-based device modeling cortical-hippocampal interactions. Neuroinformatics 3, 197–222 (2005) 33. Lungarella, M., Metta, G., Pfeifer, R., Sandini, G.: Developmental robotics: a survey. Connection Science 15, 151–190 (2003) 34. Beer, R.D., Chiel, H.J., Quinn, R.D., Ritzmann, R.E.: Biorobotic approaches to the study of the motor system. Current Neurobiology 8, 777–782 (1998) 35. Iida, F., Rummel, J., Seyfarth, A.: Bipedal walking and running with compliant legs. In: IEEE International Conference on Robotics and Automation, pp. 3970–3975 (2007) 36. Bar-Cohen, Y., Breazeal, C.: Biologically Inspired Intelligent Robots. SPIE Press (2003) 37. Drubach, D.: The brain explained. Prentice Hall Health, New Jersey (2000) 38. Wodnicki, R., Roberts, G.W., Levine, M.D.: A log-polar image sensor fabricated in a standard 1.2 ASIC CMOS process. IEEE Journal of Solid-State Circuits 32, 1274–1277 (1997) 39. Payeur, P., Pasca, C., Cretu, A., Petriu, E.M.: Intelligent haptic sensor system for robotic manipulation. IEEE Transactions on Instrumentation and Measurement 54, 1583–1592 (2005)

22

M. Begum and F. Karray

40. Motamed, M., Yan, J.: A review of biological, biomimetic and miniature force sensing for microflight. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 3939–3946 (2005) 41. Bar-Cohen, Y.: Biologically inspired intelligent robots using artificial muscles. In: IEEE International Conference on MEMS, NANO and Smart Systems, pp. 2–8 (2003) 42. Matsuyama, Y., Hirai, S.: Analysis of circular robot jumping by body deformation. In: IEEE International Conference on Robotics and Automation, pp. 1968–1973 (2007) 43. Hirai, K.: The HONDA humanoid robot: development of future perspective. Industrial robots 26, 260–266 (1999) 44. Doi, M., Hmegawa, Y., Fukuda, T.: Passive dynamic autonomous control of bipedal walking. In: IEEE/RAS International Conference on Humanoid Robots, vol. 8, pp. 811–829 (2004) 45. Jenkin, O.C., Mataric, M., Weber, S.: Primitive-based movement classification for humanoid imitation. In: IEEE/RAS International Conference on Humanoid Robots (2000) 46. Sandini, G., Metta, G., Konczak, J.: Human sensori-motor development and artificial systems. In: Proceedings of the Int. Symp. on AI, Robotics, and Intellectual Human Activity Support for Applications, pp. 303–314 (1997) 47. Shen, W., Salemi, B., Will, P.: Hormone-inspired adaptive communication and distributed control for conro self-reconfigurable robots. IEEE Transaction on Robotics and Automation 18, 700–712 (2002) 48. Yim, M., Duff, D.G., Roufas, K.D.: Polybot: A modular reconfigurable robot. In: IEEE International Conference on Robotics and Automation, pp. 514–520 (2000) 49. Sugiyama, Y., Hirai, S.: Crawling and jumping by a deformable robot. International Journal of Robotics Research 25, 603–620 (2006) 50. Sangbae, K., Asbeck, A., Cutkosky, M., Provancher, W.: SpinybotII: climbing hard walls with compliant microspines. In: International Conference on Advanced Robotics, pp. 601– 606 (2005) 51. Spenko, M., Cutkosky, M., Majidi, C., Fearingn, R., Groff, R., Autumn, K.: Foot design and integration for bio-inspired climbing robots. In: Proceedings of SPIE: Unmanned Systems Technology VIII, 6230 (2006) 52. Rosenschein, S., Kaelbling, L.P.: The synthesis of machine with provable epistemic properties, pp. 83–98. Morgan Kaufman, Los Altos (1986) 53. Arkin, R.: Toward the unification of navigational planning and reactive control. In: In Proceedings of AAAI, pp. 1–5 (1989) 54. Brooks, R.A.: A robust layered control system for a mobile robot. IEEE Transaction on Robotics and Automation 47, 139–160 (1991) 55. Weng, J.: The developmental approach to artificial intelligence: Concepts, developmental algorithms and experimental results. In: NSF Design and Manufacturing Grantees Conference (1998) 56. Albus, J.S., Meystel, A.M.: Intelligent Systems: Architecture, Design, and Control. WileyInterscience, Hoboken (2001) 57. Hendler, J.: Computers play chess; humans play go. IEEE Intelligent Systems 21, 2–3 (2006) 58. Weng, J., McClelland, J., Pentland, A., Sporns, O., Stockman, I., Sur, M., Thelen, E.: Autonomous mental development by robots and animals. Science 291, 599–600 (2001) 59. Krichmar, J.L., Edelman, G.M.: Brain-based devices: intelligent systems based on principles of the nervous system. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 940–945 (2003)

1 Computational Intelligence Techniques in Bio-inspired Robotics

23

60. Asada, M., MacDormanb, K.F., Ishigurob, H., Kuniyoshi, Y.: Cognitive developmental robotics as a new paradigm for the design of humanoid robots. Robotics and Autonomous Systems 37, 185–193 (2001) 61. Kawamura, K., Noelle, D.C., Hambuchen, K.A., Rogers, T.E.: A multi-agent approach to self-reflection for cognitive robots. In: IEEE International Workshop on Robot and Human Interactive Communication, pp. 568–575 (2003) 62. Brachman, R.J.: Systems that know what they’re doing. IEEE Intelligent Systems 17, 67–71 (2002) 63. Belliveau, J., Kennedy, D., McKinstry, R., Buchbinder, B., Weisskoff, R., Cohen, M., Vevea, J., Brady, T., Rosen, B.: Functional mapping of the human visual cortex by magnetic resonance imaging. Science 254, 716–719 (1991) 64. Strauss, L.G., Conti, P.: The applications of PET in clinical oncology. Neural Medicine 32, 623–648 (1991) 65. Posner, M.I.: The attention system of the human brain. Annual Reviews, Neuroscience 13, 25–42 (1990) 66. Cohen, S.B., Flusberg, H.T., Cohen, D.J.: Understanding other minds: perspectives from developmental cognitive neuroscience. Oxford University Press, New York (2000) 67. Geary, D.C.: The origin of mind: evolution of brain, cognition and general intelligence. American Psychological Association, Washington (2002) 68. Koch, C.: The quest for consciousness. Roberts & Company, Colorado (2004) 69. Zlatev, J., Balkenius, C.: Introduction: Why epigenetic robotics. In: International Workshop on Epigenetic Robotics (2001) 70. Friston, K.J., Tononi, G., Reeke, G.N., Sporns, O., Edelman, G.M.: Value dependent selection in the brain: Simulation in a synthetic neural model. Neuroscience 59, 229–243 (1994) 71. Huang, X., Weng, J.: Value system development for a robot. In: IEEE International Conference on Neural Networks, pp. 2883–2888 (2004) 72. Huang, X., Weng, J.: Novelty and reinforcement learning in the value system of developmental robots. In: Proceedings of International Workshop on Epigenetic Robotics, pp. 47–55 (2002) 73. Schultz, W.: Multiple reward signal in the brain. Nature Reviews, Neuroscience 1, 199–207 (2000) 74. Edelman, G.M.: Group selection and phasic reentrant signaling: a theory of higher brain function, pp. 51–100. MIT Press, Cambridge (1978) 75. Zhang, Y., Weng, J.: Action chaining by a developmental robot with a value system. In: IEEE International Conference on Development and Learning, pp. 53–60 (2002) 76. Oudeyer, P.-Y., Kaplan, F., Hafner, V., Whyte, A.: The playground experiment: Taskindependent development of a curious robot. In: AAAI Spring Symposium Workshop on Developmental Robotics (2005) 77. Oudeyer, P.-Y., Kaplan, F.: Intelligent adaptive curiosity: a source of self-development. In: International Workshop on Epigenetic Robotics (2004) 78. Sporns, O., Alm´assy, N., Edelman, G.M.: Plasticity in value systems and its role in adaptive behavior. Adaptive Behavior 12, 129–148 (2000) 79. Scheier, C., Lambrinos, D.: Categorization in a real-world agent using haptic exploration and active perception. In: Proceedings of the International Conference on Simulation of Adaptive Behavior, pp. 65–75 (1996) 80. Vygotsky, L. (ed.): Thought and Language. MIT Press, Cambridge (1962) 81. Almassy, N., Edelman, G.M., Sporns, O.: Behavioral constraints in the development of neuronal properties: a cortical model embedded in a real-world device. Cerebral Cortex 8, 346–361 (1998)

24

M. Begum and F. Karray

82. Edelman, G.M., Reeke, G.N., Gall, E., Tononi, G., Williams, D., Sporns, O.: Synthetic neural modeling applied to a real-world arttifact. Proceedings of National Academic Science 89, 7267–7271 (1992) 83. Krichmar, J.L., Snook, J.A., Edelman, G.M., Sporns, O.: Experience dependent perceptual categorization in a behaving real-world device, pp. 41–50. MIT Press, Cambridge (2000) 84. Treisman, A.M., Gelade, G.: A feature intigration theory of attention. Cognitive Psychology 12, 97–136 (1980) 85. Wolfe, J.M.: Visual search in continuous naturalistic stimuli. Vision Research 34, 1187– 1195 (1994) 86. Eriksen, C.W., Yeh, Y.Y.: Allocation of attention in the visual field. Journal of Experimental Psychology: Human Perception and Performance 11, 583–597 (1985) 87. Posner, M.I.: Orienting of attention. Journal of Experimental Psychology 32, 3–25 (1980) 88. Posner, M.I., Rothbart, M.K.: Educating the human brain. American Psychological Association, Washington (2007) 89. Bundense, C.: A theory of visual attention. Psychologycal Review 97, 523–527 (1990) 90. Desimone, R.: Visual attention mediated by biased competition in extrastriate visual cortex. Philosophical transactions of the Royal Society of London, Series B, Biological sciences 353, 1245–1255 (1998) 91. Chelazzi, L., Duncan, J., Miller, E.K., Desimone, R.: Responses of neurons in inferior temporal cortex during memory guided visual search. Journal of Neurophysiology 80(6), 2918– 2940 (1998) 92. Kastner, S., Ungerleider, L.G.: The neural basis of biased competition in human visual cortex. Vision Research 40, 1489–1506 (2000) 93. Duncan, J.: Selective attention and the organization of visual information. Journal of Experimental Psychology 113, 501–513 (1984) 94. Luck, S., Chelazzi, L., Hillyard, S.A., Desimone, R.: Neural mechanisms for directed visual attention. Journal of Neurophysiology 77, 24–42 (1997) 95. Desimone, R., Duncan, J.: Neural mechanisms of selective visual attention. Annual Reviews of Neurscience 18, 193–222 (1995) 96. Carpenter, G.A., Grossberg, S.: ART2: Self-organization of stable category recognition codes for analog input patterns. Applied Optics 26(23), 4919–4930 (1987) 97. Itti, L., Koch, C.: Computational modelling of visual attention. Nature Reviews: Neuroscience 2, 194–203 (2001) 98. Koch, C., Ullman, S.: Shifts in selective visual attention: toward the underlying neural circuitry. Human Neurobiology 4, 219–227 (1985) 99. Itti, L., Koch, C., Niebur, E.: A model of saliency-based visual attention for rapid scene analysis. IEEE Transaction on Pattern Analysis and Machine Intelligence 20, 1254–1259 (1998) 100. Itti, L., Rees, G., Tsotsos, J.K. (eds.): Neurobiology of Attention. Elsevier Academic Press, Amsterdam (2005) 101. Tsotsos, J.K., Culhane, S.M., Winky, Y.K.W., Yuzhong, L., Davis, N., Nuflo, F.: Modeling visual attention via selective tuning. Artificial Intelligence 78, 507–545 (1995) 102. Cave, K.R.: The featuregate model of visual selection. Psychological Research 62, 182–194 (1999) 103. Sun, Y., Fisher, R.: Object-based visual attention for computer vision. Artificial Intelligence 146, 77–123 (2003) 104. Frintrop, S.: VOCUS: A Visual Attention System for Object Detection and Goal-Directed Search. LNCS (LNAI), vol. 3899. Springer, Heidelberg (2006)

1 Computational Intelligence Techniques in Bio-inspired Robotics

25

105. Deco, G.: Biased competition mechanisms for visual attention in a multimodular neurodynamical system. In: Wermter, S., Austin, J., Willshaw, D. (eds.) Emergent Neural Computational Architectures Based on Neuroscience. LNCS (LNAI), vol. 2036, pp. 114–126. Springer, Heidelberg (2001) 106. Deco, G., Rolls, E.T.: A neurodynamical cortical model of visual attention and invariant object recognition. Vision Research 44, 621–642 (2004) 107. Rolls, E.T., Deco, G. (eds.): Computational Neuroscience of Vision. Oxford University Press, Oxford (2002) 108. Lanyon, L.J., Denham, S.L.: A biased competition computational model of spatial and object-based attention mediating active visual search. Neurocomputing 58-60, 655–662 (2004) 109. Lanyon, L.J., Denham, S.L.: A model of active visual search with object-based attention guiding scan paths. Neural Networks 17, 873–897 (2004) 110. Vijayakumar, S., Conrad, J., Shibata, T., Schaal, S.: Overt visual attention for a humanoid robot. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 2332– 2337 (2001) 111. Orabona, F., Metta, G., Sandini, G.: Object-based visual attention: A model for a behaving robot. In: IEEE Conference on Computer Vision and Pattern Recognition (2005) 112. Scassellati, B.M.: Foundations for a Theory of Mind for a Humanoid Robot. PhD thesis, Electrical Engineering and Computer Science, MIT (May 2001) 113. Ude, A., Wyart, V., Lin, L.-H., Cheng, G.: Distributed visual attention on a humanoid robot. In: IEEE-RAS International Conference on Humanoid Robots, pp. 381–386 (2005) 114. Driscoll, J.A., Peters, R.A., Cave, K.R.: A visual attention network for a humanoid robot. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 1968–1974 (1998) 115. Breazeal, C., Edsinger, A., Fitzpatrick, P., Scassellati, B.: Active vision for sociable robots. IEEE Transaction on System, Man, and Cybernetics, Part A 31, 443–453 (2001) 116. Fay, R., Kaufmann, U., Knoblauch, A.: Combining Visual Attention. In: Object Recognition and Associative Information Processing in a Neurobotic System, pp. 117–142. Springer, Heidelberg (2005) 117. Vitay, J., Rougier, N.P., Alexandre, F.: A Distributed Model of Spatial Visual Attention, pp. 54–72. Springer, Heidelberg (2005) 118. Ban, S.-W., Niitsuma, H., Lee, M.: Bilogicaly inspired autonomous mental development model based on visual selective attention mechanism. In: AAAI Symposium on Developmental Robotics (2005) 119. Stasse, O., Kuniyoshi, Y., Cheng, G.: Development of a biologically inspired real-time visual attention system. In: IEEE International Workshop on Biologically Motivated Computer vision, pp. 150–159 (2000) 120. Manfredi, L., Maini, E.S., Dario, P., Laschi, C., Girard, B., Tabareau, N., Berthoz, A.: Implementation of neurophysiological model of saccadic eye movements on an anthropomorphic robotic head. In: IEEE-RAS International Conference on Humanoid Robotics, pp. 438–443 (2006) 121. Aziz, M.Z., Mertsching, B., Shafik, M.S., Stemmer, R.: Evaluation of visual attention models for robots. In: IEEE International Conference on Computer Vision Systems, pp. 20–26 (2006) 122. Itti, L., Koch, C.: A saliency based search mechanism for overt and covert shift of visual attention. Vision Research 40, 1489–1506 (2000) 123. Clark, J.J.: Spatial attention and saccadic camera motion. In: IEEE International Conference on Robotics and Automation, pp. 3247–3252 (1998)

26

M. Begum and F. Karray

124. Breazeal, C.F.: A motivational system for regulating human robot interaction. In: National conference of AAAI, pp. 54–61 (1998) 125. Baccon, J.C., Hafemeister, L., Gaussier, P.: A context and task dependent visual attention system to control a mobile robot. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 238–243 (2002) 126. Kaufmann, U., Fay, R., Schwenker, F., Palm, G.: Neural networks for visual object recognition based on selective attention. In: International SenseMaker Workshop on Life-Like Perception Systems (2005) 127. Fay, R., Kaufmann, U., Schwenker, F., Palm, G.: Learning object recognition in a neurobotic system. In: Workshop on Self Organization of AdaptiveBehavior SOAVE, pp. 198– 204 (2004) 128. Palm, G., Kaufmann, U., Fay, R.: A two-stage scheme for visual object recognition based on selective attention. In: European Conference on Visual Perception (ECVP) (2004) 129. Rougier, N.P., Vitay, J.: Emergence of attention within a neural population. Neural Networks 19, 573–581 (2006) 130. Vitay, J., Rougier, N.P., Alexandre, F.: A Distributed Model of Spatial Visual Attention, pp. 54–72. Springer, Heidelberg (2005) 131. Erlhangen, W., Bicho, E.: The dynamic neural field approach to cognitive robotics. Journal of Neural Engineering 3, 36–54 (2006) 132. Heidemann, G., Rae, R., Bekel, H., Bax, I., Ritter, H.: Integrating context-free and contextdependent attentional mechanisms for gestural object reference. Machine Vision and Applications 16, 64–73 (2002) 133. Scheier, C., Egner, S.: Visual attention in a mobile robot. In: Proceedings of the IEEE International Symposium on Industrial Electronics, vol. 1, pp. 48–52 (1997) 134. Egner, S., Scheier, C.: Feature binding through temporally correlated neural activity in a robot model of visual perception. In: Gerstner, W., Hasler, M., Germond, A., Nicoud, J.-D. (eds.) ICANN 1997. LNCS, vol. 1327, pp. 703–709. Springer, Heidelberg (1997) 135. Kawamura, K.: Cognitive approach to a human adaptive robot development. In: IEEE International Workshop on Robot and Human Interactive Communication, pp. 210–216 (2005) 136. Kawamura, K., Peters, R.A., Bodenheimer, R., Sarkar, N., Park, J., Spratley, A., Hambuchen, K.A.: Multi agent based cognitive robot architecture and its realization. International Journal of Humanoid Robotics 1, 65–93 (2004) 137. Ratanaswasd, P., Gordon, S., Dodd, W.: Cognitive control for robot task execution. In: IEEE International Workshop on Robot and Human Interactive Communication, pp. 440– 445 (2005) 138. Hasoda, K., Sumioh, H., Morita, A., Asada, M.: Acquisition of human-robot joint attention through real-time natural interaction. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 2867–2872 (2004) 139. Nagai, Y., Hosoda, K., Morita, A., Asada, M.: A constructive model for the development of joint attention. IEEE Intelligent Systems 15, 211–229 (2003) 140. Driscoll, J.A., Peters, R.A., Cave, K.R.: A visual attention network for a humanoid robot. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (1998) 141. Amari, S., Arbib, M.A.: Competition and cooperation in neural nets, pp. 119–165. Academic Press, London (1977) 142. Zhang, N., Weng, J.: A developing sensory mapping for robots. In: IEEE International Conference on Development and Learning, pp. 13–20 (2002) 143. Weng, J., Zhang, Y.: Developmental robots: A new paradigm. In: International Workshop on Epigenetic Robotics (2002) 144. Weng, J.: Developmental robotics: Theory and experiments. International Journal of Humanoid Robotics 1, 199–234 (2004)

1 Computational Intelligence Techniques in Bio-inspired Robotics

27

145. Weng, J., Hwang, W.S.: From neural networks to the brain: Autonomous mental development. IEEE Computational Intelligence Magazine 1, 15–31 (2006) 146. Rucci, M., Bajcsy, R.: Learning visuo-tactile coordination in robotic systems. In: IEEE International Conference on Robotics and Automation, pp. 2678–2683 (1995) 147. Colombo, C., Rucci, M., Dario, P.: Attentive behavior in an anthropomorphic robot vision system. Journal of Robotics Autonomous Systems 12, 121–131 (1994) 148. Fleischer, J.G., Szatm´ary, B., Hutson, D.B., Moore, D.A., Snook, J.A., Edelman, G.M., Krichmar, J.L.: A neurally controlled robot competes and cooperates with humans in segway sooccer. In: IEEE International Conference on Robotics and Automation, pp. 3673– 3678 (2006) 149. Doniec, M.W., Sun, G., Scassellati, B.: Active learning of joint attention. In: IEEE/RAS International Conference on Humanoid Robots, pp. 34–39 (2006) 150. Doniec, M.W., Sun, G., Scassellati, B.: A demonstration of the efficiency of developmental learning. In: IEEE International Joint Conference on Neural Networks, pp. 5226–5232 (2006) 151. Nagai, Y., Hosoda, K., Asada, M.: Joint attention emerges through bootstrap learning. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 168–173 (2003) 152. Nagai, Y., Asada, M., Hosoda, K.: A developmental approach accelerates learning of joint attention. In: IEEE International Conference on Development and Learning, pp. 277–282 (2002) 153. Nagai, Y., Hosoda, K., Morita, A., Asada, M.: A constructive model for the development of joint attention. Connection Science 15, 211–229 (2003) 154. Breazeal, C., Fitzpatrick, P., Scassellati, B.: Active vision systems for sociable robots. IEEE Transaction on System, Man, and Cybernetics: Part A 31, 443–453 (2001) 155. Brooks, R.A., Breazeal, C.F., Marjanovic, M., Scassellati, B., Williamson, M.: The cog project: Building a humanoid robot. In: Nehaniv, C.L. (ed.) CMAA 1998. LNCS (LNAI), vol. 1562, pp. 52–87. Springer, Heidelberg (1999) 156. Scassellati, B.: Foundations for theory of mind for a humanoid robot. PhD thesis, MIT, Cambridge, MA (2001) 157. Tani, J., Ito, M.: Joint attention between a humanoid robot and users in imitation game. In: IEEE International Conference on Development and Learning, pp. 277–282 (2004) 158. Fong, T., Nourbakhsh, I., Dautenhahn, K.: A survey of socially interactive robots: concepts design and applications. Technical Report CMU- RI- TR- 02- 29, Robotic Institute, Carnegi Mellon University (2002) 159. Breazeal, C.: Toward sociable robots. Robotics and Autonomous Systems 42, 167–175 (2003) 160. Piaget, J.: The origins of intelligence. Routledge, New York (1953) 161. Butterworth, G., Jarret, B.: What minds have common in space: spatial mechanisms serving joint visual attention in infancy. British Journal of Developmental Psychology 9, 55–72 (1991) 162. Breazeal, C., Scassellati, B.: Robots that imitate humans. Adaptive Behanior 8, 49–74 (2002) 163. Shon, A.P., Storz, J.J., Rao, R.P.N.: Towards a real-time bayesian imitation system for a humanoid robot. In: IEEE International Conference on Robotics and Automation, pp. 2847– 2852 (2007) 164. Ito, M., Tani, J.: On-line imitative interaction with a humanoid robot using a dynamic neural network model of a mirror system. Adaptive Behavior 12, 93–115 (2004)

28

M. Begum and F. Karray

165. Yoshikai, T., Otake, N., Mizuuchi, I., Inaba, M., Inoue, H.: Development of an imitation behavior in humanoid kenta with reinforcement learning algorithm based on the attention during imitation. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, vol. 2, pp. 1192–1197 (2004) 166. Calinon, S., Guenter, F., Billard, A.: On learning, representing and generalizing a task in a humanoid robot. IEEE Transactions on Systems, Man, and Cybernetics–Part B: Cybernetics 36, 286–298 (2007) 167. Calinon, S., Guenter, F., Billard, A.: Goal-directed imitation in a humanoid robot. In: IEEE International Conference on Robotics and Automation, pp. 299–304 (2005) 168. Kuniyoshi, Y., Yorozu, Y., Inaba, M., Inoue, H.: From visuo-motor self learning to early imitation - a neural architecture for humanoid learning. In: IEEE International Conference on Robotics and Automation, pp. 3132–3139 (2003) 169. Belardinelli, A., Pirri, F., Carbone, A.: Bottom-up gaze shifts and fixations learning by imitation. IEEE Transactions on Systems, Man, and Cybernetics–Part B: Cybernetics 37, 256–271 (2007) 170. Lopes, M., Victor, J.S.: Bottom-up gaze shifts and fixations learning by imitation. IEEE Transactions on Systems, Man, and Cybernetics–Part B: Cybernetics 35, 438–449 (2005) 171. Lopes, M., Victor, J.S.: A developmental road map for learning by imitation in robots. IEEE Transactions on Systems, Man, and Cybernetics–Part B: Cybernetics 37, 308–321 (2007) 172. Brooks, R.: Robot: The future of flesh and machines. Penguin Books, London (2003)

2 Swarm Intelligence for Collective Robotic Search Lisa L. Grant1 and Ganesh K. Venayagamoorthy2 1

2

Real-Time Power and Intelligent Systems Laboratory, Missouri University of Science and Technology, Rolla, MO 65409, USA [email protected] Real-Time Power and Intelligent Systems Laboratory, Missouri University of Science and Technology, Rolla, MO 65409, USA [email protected]

Abstract. This chapter presents three strategies for the navigation of a swarm of robots for a target search application in a hazardous environment. The strategies explored include greedy search and two computational intelligence techniques—particle swarm optimization and fuzzy logic. Results for the collective search are presented for simulated environments containing single and multiple targets, with and without obstacles. The proposed navigation strategies can be further developed and applied to real-world applications such as aiding in disaster recovery, detection of hazardous materials, and many other high-risk tasks.

2.1 Collective Robotic Search The use of robots to remove the need for human involvement in dangerous tasks has been an area of great interest [2, 6, 15, 21, 26]. For high-risk applications such as landmine and radioactive material detection, the robots need to be inexpensive and dispensable. A multi-robot search using simple or specialized robots has a higher chance for success than a search utilizing a single more complex and expensive robot attempting the task alone, by reducing the impact of robot loss, equipment failure, and communication failure on the system. One approach is to have a group of robots searching in parallel, but individually with little to no communication or interaction with the group, adding redundancy and fault tolerance to the system. Another common method to control a multi-robot system is the centralized approach where a central server or controller coordinates the robots actions for optimal performance in completing the specified task. This method can prove computationally intensive and vulnerable if the central controller becomes disabled or destroyed. Introducing coordination among the robots in an equally distributed system allows for a more efficient search and the ability to implement cooperative task completion. When performing a group-oriented search, the robot’s independent behavior is mainly based on its locally available sensory information allowing for adaptation to changing and hazardous environments, where utilization of the available swarm information increases the efficiency of the overall search. Performing a multiple robot based search also has the advantage of greater coverage of the search environment and simpler implementation [28]. In Collective Robotic Search (CRS), a group of small, inexpensive, and dispensable mobile robots collectively explore a dangerous environment in order to locate and D. Liu, L. Wang, and K.C. Tan (Eds.): Design and Control of Intel. Robotic Systems, SCI 177, pp. 29–47. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com 

30

L.L. Grant and G.K. Venayagamoorthy

Global Level

Ld Neighborhood level Ld

Pd

Local Swarms

Fig. 2.1. Random deployment of robots in the search space

converge on a known target [7, 9]. For the CRS application in this study, a three-tier architecture is proposed for the implementation of a collective robotic search as shown in Fig. 2.1. The robots are randomly deployed in the search space. Since CRS is targeted to applications where human intervention is undesirable, deployment of the robots into an area of interest can be carried out by dropping the robots from an aircraft. As seen in Fig. 2.1, the search area can be divided into neighborhoods to restrict the robots motion to a smaller area based on a divide and conquer scheme [27]. The robots within each neighborhood form a local team or swarm. Each swarm operates individually of the swarms in the other neighborhoods. In environments with extreme terrain, this division of the search area ensures that the entire search space is explored and the targets are independently pursued. In some cases not all robots are able to access the entire search area due to natural boundary restrictions such as a large body of water or a mountain range. The division allows the robots to perform a fast, efficient and parallel search. The goal of CRS is to identify target locations and the targets can be of different kinds from one application to the other. The simulated environment in this study assumes that most desired targets have an identifiable gradient and uses light sources as the target goal. The robots are equipped with light sensors to detect the total light intensity at their current location relative to the location and number of target light sources. A diagram of the robot’s light intensity sensor is provided in Fig. 2.2. The swarm pursues the target closest to the robot with the highest overall fitness reading. The target fitness is based on the relative intensity, RI, given in Eq. (2.1) which is calculated using the intensity (I) read from each of the n targets in relation to the distance d from the robot to each target. RI =

In I1 + ...+ 2 (d1 ) (dn )2

(2.1)

2 Swarm Intelligence for Collective Robotic Search

31

Target with Intensity In

Target with Intensity I1 dn d1

Robot

Fig. 2.2. Intensity of targets measured at the robot

2.2 Navigation Strategies There are three strategies studied for navigation of the robots in the proposed CRS problem. A traditional approach known as greedy search is compared with two Computational Intelligence (CI) techniques, namely particle swarm optimization and fuzzy logic. These strategies are described in the following subsections. 2.2.1

Greedy Search - A Conventional Method

The greedy search algorithm is a conventional method used to follow a local gradient. When applied to CRS, a robot implementing a greedy search chooses its path based solely on the positions that orient it closest to its goal. The algorithm is called greedy because it grabs the best solution without calculating long-term cost. The robot moves to the position with highest intensity after sampling the intensities at positions around its current location. For the greedy search, essentially only one robot is needed to find the target because each robot acts as an individual agent. In a search utilizing a swarm of greedy robots, no communication between robots is necessary since each robot can locate the target on its own. Since the algorithm relies on little to no communication between robots, it is fairly robust against communication and robot loss [4]. Similar strategies have been proposed to produce agent convergence where global information and communication are not required, but global group behavior is still observed [5, 14, 18, 25]. The version of the greedy search algorithm used in this study is the ordinal greedy search method [24]. In order to find its next position, each robot maps out four possible positions, one in each of the ordinal directions (NE, SE, SW, and NW) from its current position. As the robot visits each of the four search points, it evaluates the target fitness at every position and ultimately chooses the search point with the best fitness or the shortest straight-line distance from the robot to the goal for its next position. The process repeats for every iteration, or position change. The robot is basically used as a mobile sensor. A detailed example of the mapping of one robot’s search points for a number of iterations using the greedy ordinal method is provided in Fig. 2.3. A flowchart of the greedy algorithm applied to a CRS problem is illustrated in Fig. 2.4.

32

L.L. Grant and G.K. Venayagamoorthy

Target (t+4)th move Pos. 4(t+4)

Pos. 1(t+4)

(t+3)th move Pos. 4(t+3) Pos. 1(t+3)

r2

(t+2)th move

(Next Pos. (t+4)) Pos. 4(t+2)

Pos. 1(t+2) Pos. 3(t+3)

(t+1)th move

(Next Pos. (t+3)) Pos. 4(t+1)

Pos. 1(t+1) Pos. 3(t+2)

(Next Pos. (t+2))

tth move

Pos. 4(t) (Next Pos. (t+1))

Pos. 3(t+1)

Pos. 1(t) Original Pos. (t)

Pos. 3(t)

r1 Pos. 2(t)

Fig. 2.3. A robot scanning positions for its next move using the greedy method

Start

Initialize search area. Initialize robots positions and velocities. Move to NE (Northeast) search point.

Check and store fitness of first search point. Move to SE (Southeast), SW (Southwest), and NW (Northwest) search points. Check and store fitness at each point. Compare fitness of search points to determine position of best fitness. Move to search point of best fitness.

Have all robots converged on the target?

No

Yes End

Fig. 2.4. Flowchart of greedy ordinal search as applied to CRS

2 Swarm Intelligence for Collective Robotic Search

2.2.2

33

Particle Swarm Optimization

Particle Swarm Optimization (PSO) is a swarm intelligence based algorithm developed by James Kennedy and Russell Eberhart in 1995 [10, 16, 17, 23]. This algorithm is based on the behavior of a school of fish or flock of birds. In PSO, better solutions are evolved through the collective interactions of the individuals in the swarm. For this study, PSO particles are referred to as robots. The success of the group depends on the success of the individuals in the swarm which occurs due to the social interactions between individual robots in the swarm. Each robot randomly searches the environment by updating its position using its own memory and social information gathered from the other robots. The robots are randomly deployed into a defined search area as previously described in Fig. 2.1. Each robot is given an initial random velocity and ‘flown’ through the problem space. The robots use memory to keep track of the position that resulted in their highest fitness reading, known as the ‘Pbest ’ value. The best value of all the ‘Pbest ’ values is defined as the local best position, ‘Lbest ’ with respect to the target. The velocities and positions are updated every iteration until all of the robots have converged on the target location [28]. PSO only requires four values for memory requirements: ‘Pbest ’, ‘Lbest ’, and the previous iteration’s velocity and position values. The PSO velocity and position update equations are given by Eqs. (2.2) and (2.3). The parameters w, c1 , and c2 are adjusted to determine to what degree the current velocity, ‘Pbest ’, or ‘Lbest ’ values affects a robot’s next position, respectively [22]. ‘rand1’ and ‘rand2’ are random numbers from a uniform distribution. Several versions of PSO are detailed in [8]. Vnew = w × Vcur + c1 × rand1 × (Pbest − Pcur ) + c2 × rand2 × (Lbest − Pcur ) Pnew = Pcur + Vnew

Where, Vnew Vcur Pnew Pcur w c1 and c2 rand

(2.2) (2.3)

- New velocity calculated for each robot - Velocity of the robot from the current iteration - New position calculated for each robot - Position of the robot from the current iteration - Inertia weight - Cognitive and social acceleration constants respectively - Generates a uniform random value in the range [0 1]

A vector representation of the PSO algorithm in a 2-dimensional search space is given in Fig. 2.5. The robots are represented by stars and the target is represented by the labeled circle. The vectors Vpd and Vgd represent the effect of the ‘Pbest ’ and ‘Lbest ’ positions on the robots, respectively. To associate Eq. (2.2) with Fig. 2.5, the term Vpd in

34

L.L. Grant and G.K. Venayagamoorthy

Y T Target

PnewPnew

Vcur Vnew

Vgd Pcur

Vpd X Fig. 2.5. Vector representation of PSO

the figure represents ‘c1 × rand × (Pbest − Pcur )’ and the term Vgd in the figure represents ‘c2 × rand × (Lbest − Pcur )’. The robots use the values of ‘Pbest ’ and ‘Lbest ’ to find a new position closer to the target. The PSO algorithm can be summarized into the following steps: 1. Randomly initialize a swarm of robots in a defined n-dimensional search space with initial random velocities, Pbest , and Lbest values. 2. Evaluate each robot’s current position using the fitness function (the Euclidean distance from the robot to the target(s) based on sensory readings). Select the Lbest from all Pbests . 3. Compute the new velocities and positions using Eqs. (2.2) and (2.3). 4. Check if the robots’ positions are within the problem space and if the velocity exceeds the predefined limits. If not then set the velocity to the maximum velocity (Vmax ) and set the new position to its Pbests . 5. Calculate the new fitness for the robots’ new positions. Determine the new Pbest . Compare with the previous Pbest and update the value if necessary. 6. Calculate the new local best position Lbest among all the new Pbest positions. Compare with the previous best and update the Lbest . 7. Repeat procedure from step (3), until all robots converge on the target(s). A flowchart of the PSO algorithm applied to a CRS problem is illustrated in Fig. 2.6. 2.2.3

Fuzzy Logic

Fuzzy logic was pioneered by Lofti Zadeh in the 1960s [9, 29, 31]. This logic combines human knowledge with sensory measurements and mathematical formulas to try to model commonsense. A system incorporating fuzzy logic utilizes linguistic variables to represent real world parameters. Membership functions define the degree of the input and output variables in the fuzzy system. A rule set developed by heuristics defines

2 Swarm Intelligence for Collective Robotic Search

35

Start

Initialize search area. Initialize robots positions and velocities and set initial Pbest. Evaluate fitness function to determine Lbest. Compute new velocities and positions for all robots. Check new velocities and positions and reset if they are not within defined limits. Calculate new fitness value for each of the robots new positions. Update Pbest and Lbest values.

Have all robots converged on the target?

No

Yes End

Fig. 2.6. Flowchart of the PSO based strategy as applied to a CRS problem

the relationship between the input and output variable to tie the whole system together. During the fuzzification process, the inputs are transferred into the linguistic variables and then formed into fuzzy sets. An inference engine uses the defined rules to develop the fuzzy outputs from the inputs. Finally, in the defuzzification process, the outputs are mapped back to real world variables for use in controlling a real world application. Since the system’s performance depends strongly on the way the membership functions and rules are defined, it is essential to optimize these parameters [28]. A block diagram of a fuzzy control system is shown in Fig. 2.7. In this study, a swarm of robots guided by fuzzy logic controllers is used to locate targets in a CRS problem. Each robot has an identical fuzzy controller whose membership functions and rules are developed based on heuristics. The inputs to the fuzzy controller are the measured light intensities of the targets read by the robot’s sensor and the Euclidean distance of the robot’s current position to the position of the robot with the highest fitness or ‘Lbest ’ value in the swarm. The outputs of the fuzzy controller are displacements in the X-axis and Y-axis directions, which are added to the robot’s current position. Eqs. (2.4) and (2.5) are used to find each robot’s next position [28]. In Eqs. (2.4) and (2.5), Ldx = Lbestx − Px and Ldy = Lbesty − Py , where ‘Px ’ and ‘Py ’ are the X-Y coordinates of the robot’s current position and ‘Lbestx ’ and ‘Lbesty ’ are the X-Y coordinates of the ‘Lbest ’ of the swarm. The ‘Lbest ’ value is the position of the robot in

36

L.L. Grant and G.K. Venayagamoorthy

Real World Inputs

Real World Outputs Fuzzy Logic Controller

Fuzzification

Inference Engine

Defuzzification

Rule base

Fig. 2.7. Block diagram illustrating the stages involved in a fuzzy control system

the swarm with the highest intensity reading. If Ldx is negative then the X coordinate in Eq. (2.5) becomes (Xi − Δ Xi ) and if it is positive then it becomes (Xi + Δ Xi ). The same logic is applied to the Y coordinates as well. The inference engine used for this simulation is Mamdani’s product inference engine and the defuzzification is based on center average. Pi = (Xi ± Δ Xi ,Yi ± Δ Yi )

(2.4)

Δ Xi = f (Ii , Ldx ); Δ Yi = f (Ii , Ldy )

(2.5)

The membership functions of the three input variables to each robot’s fuzzy controller are described below. • The intensity reading of the sensor on each robot (Ii ) has zero (Z), small (S), medium (M) and large (L) triangular membership functions. • The difference of the X coordinates of the current position of the robot from the robot having the best position in the swarm given by ‘Ldx ’ has zero (Z), very small (VS), small (S), medium (M) and large (L) triangular membership functions. • The difference of the Y coordinates of the current position of the robot from the robot having the best position in the swarm given by ‘Ldy ’ has zero (Z), very small (VS), small (S), medium (M) and large (L) triangular membership functions. The membership functions of the two output variables from each fuzzy logic controller are described below. • The displacement amount Δ Xi to be added to the robot’s current X-axis position has very small (VS), small (S), medium (M) and large (L) triangular membership functions. • The displacement amount Δ Yi to be added to the robot’s current Y-axis position has very small (VS), small (S), medium (M) and large (L) triangular membership functions.

2 Swarm Intelligence for Collective Robotic Search

Fuzzy Inputs

A target is identified?

37

Yes Fine Rules

No Fuzzy Outputs

Coarse Rules

Fig. 2.8. Rule base switching logic Table 2.1. Coarse Rule Set

Ldx/Ldy Intensity(I)

Z

VS

S

M

L

Z S M L

M S S VS

M M S VS

L M M S

L L M S

L L M M

Table 2.2. Fine Rule Set

Ldx/Ldy Intensity(I)

Z

VS

S

M

L

Z S M L

VS VS VS VS

VS VS VS VS

S S VS VS

M S S S

M M M S

There are two different rule sets for the fuzzy controllers. The first set of rules is called the ‘coarse rule set’, which makes the robots move in larger steps if they are far from the target. This rule set helps to advance exploration of the search area. Once a target has been identified, the second rule set referred to as the ‘fine rule set’ is used to make the robots move toward the target in smaller steps. Fig. 2.8 shows the relationship between the robot’s position relative to the target, and the coarse and fine rule sets. Tables 2.1 and 2.2 show the coarse and fine rule sets, respectively. Both rule sets have a total of 40 rules. There are 20 rules each for X- and Y-axis. The rules are used in IF-THEN statements to determine the output or which size step the robot should take. An example is given in Eq. (2.6) taken from the coarse rule set [28]. A flowchart of the fuzzy algorithm applied to a CRS problem is illustrated in Fig. 2.9. IF Intensity is small(S) and Ldx is large(L) THEN Δ Xi is Large (L)

(2.6)

38

L.L. Grant and G.K. Venayagamoorthy

Start

Initialize search area. Initialize robots positions and velocities and set initial Lbest. Evaluate fitness function to determine Lbest. Check intensity/fitness of each robot’s current position. Use robot’s fitness and location relative to the Lbest to calculate X- and Y- axis fuzzy displacements. Move to next position based on fuzzy displacements. Evaluate fitness of new position to determine new Lbest. Have all robots converged on the target?

No

Yes End

Fig. 2.9. Flowchart of the fuzzy logic based strategy as applied to a CRS problem

2.3 Single Target Obstacle-Free Environment The PSO algorithm was simulated in a simple single-target, obstacle-free environment to determine the best combination of values to use for the PSO strategy parameters w, c1 , and c2 . Fig. 2.10 shows a search space containing a single target. The target location is fixed and 10 robots are randomly dispersed throughout the search environment. The performance of the swarm based on changes in PSO’s parameter values is shown in Table 2.3. The results are averaged over 100 trials. For the c1 and c2 variables, values of 0.5 and 2 were tested and values of 0.6 and 0.8 were tested for the w parameter. As can be seen from Table 2.3, the values producing the fastest swarm convergence and minimum iterations are w = 0.6, c1 = 0.5, and c2 = 2. The higher value of the desired social acceleration constant, c2 , as compared to the desired cognitive acceleration constant, c1 , indicates the PSO algorithm’s dependency on social interactions among the swarm and the importance of the Lbest parameter. The PSO parameters can be further optimized during the search process by using optimal PSO. For optimal PSO, there is an outer PSO algorithm and an inner PSO algorithm. The inner swarm provides the actual implementation of the target search problem while the outer swarm optimizes the PSO parameter values. Every time a cycle of the

2 Swarm Intelligence for Collective Robotic Search

Y

39

Mobile Robots/ PSO Particles

10

Target

X -10 -10

10

Fig. 2.10. Graphical representation of a single target case Table 2.3. Effect on PSO results in single target case with parameter changes

c1

c2

# of Iterations (w=0.6)

# of Iterations (w=0.8)

2 0.5 2 0.5

2 2 0.5 0.5

135.45 79.63 – 110.29

491.95 348.20 498.14 145.08

Table 2.4. Optimization of PSO parameters using optimal PSO strategy

w

c1

c2

# of Iterations

0.60 0.45 0.45 0.45

0.50 0.60 1.19 0.35

2.00 1.50 1.52 1.45

79.63 39.50 60.80 83.40

algorithm runs, a new set of PSO parameter values are calculated in the outer swarm and plugged into the inner swarm. Fig. 2.11 contains the flowchart for performing an optimal PSO algorithm [9]. Table 2.4 presents the parameter values tested in the optimal PSO trials. It is evident from the table that the combination of 0.45, 0.6, and 1.5 for w, c1 , and c2 respectively, provide the fastest search. The results in Tables 2.3 and 2.4 show that swarm communication has a large influence on the success of the search. The social acceleration constant, c2 , which controls the how much information is shared among the swarm is much higher than the cognitive acceleration constant, c1 which controls the robot’s use of it’s own gathered information for a faster search.

40

L.L. Grant and G.K. Venayagamoorthy

OUTER PSO Initialize PSO parameters for the outer swarm (wout, c1out, & c2out)

Initialize the values of win, c1in, & c2in to be used in the inner swarm

INNER SWARM FOR TARGET SEARCHING Uses the values win, c1in, & c2in as passed into it The target search application program that uses PSO Fitness: lowest number of iterations

Chooses the values of w in, c1in, & c2in that corresponds to the particle with the least number of iterations

Best values of win, c1in, & c2in (Lbest for outer PSO)

TARGET SEARCHING

Fig. 2.11. Flowchart for Optimal PSO

2.4 Multiple Target/Obstacle Environment There are two types of case studies performed for the multi-target/obstacle case. For the first case study, a single swarm performs a global search of the entire environment space. In the second case study, the environment is divided into 4 neighborhoods and the swarm is broken down into four local groups, one searching each neighborhood. With the introduction of obstacles into the search space it is necessary to provide the robots with a method for obstacle avoidance. The following sub-sections present a brief explanation of the obstacle avoidance technique chosen, followed by sections detailing both case studies and simulation results for the greedy, PSO, and fuzzy algorithms. 2.4.1

Obstacle Avoidance Method

Obstacle avoidance is a real-world issue in robot navigation implementation. A variety of methods have been proposed for obstacle avoidance [1, 11, 12, 13, 30]. These methods incorporate various CI and computational techniques, but many of the existing methods are computationally intensive and the necessary coordination scheme amongst the robots diminishes the robot’s ability to effectively explore the search space. The

2 Swarm Intelligence for Collective Robotic Search

41

For (i = 1:N )Do,

For ( j = 1 : m )Do,

dij =|| Pi - Xj || ; where ( j = 1,..., m ) If (dij £ rij )Then , Pi - Xj dij Set Pi = Pi + pij ´ Zij ´ sin (qij ) Set Zij =

End If End For End For

Fig. 2.12. Pseudo code for the repulsion algorithm

obstacle avoidance method considered in this study is a modification of the repulsion algorithm presented in [20]. This algorithm is chosen because of its computational simplicity as compared to other existing methods, and its compatibility with numerous robot coordination strategies. The repulsion strategy is used to repel the robots away from detected obstacles. The pseudocode for the repulsion algorithm is provided in Fig. 2.12 for a swarm of N robots and a search area containing m obstacles. Repulsion from an obstacle occurs if the robot’s next projected position lies within a prescribed boundary area around the obstacle, called the obstacle’s repulsion range - ri j [24]. The swarm of robot’s current positions are represented by Pi at a single iteration, where {i = 1, . . . , N}. The obstacles are represented by Xi for { j = 1, . . . , m}. The robot’s next position relative to the obstacle boundary condition is calculated in di j , if this variable is less than the value of the obstacle’s repulsion range (ri j ), then the robot’s next position will lie within an obstacle. To calculate the robot’s new adjusted position, the following are added to the robots current position: the repulsion strength (pi j ), a unit vector with the direction from the obstacle to the robots position (Zi j ), and a deflection angle (θi j ). For a real-world implementation, the robot would desirably have touch sensors which would activate the repulsion algorithm once a robot made contact with an obstacle. 2.4.2

Case Study 1: Single Swarm

The search environment used for simulation in the single swarm case study is provided in Fig. 2.13. The search area is restricted to a 200 by 300-unit space in the X-Y plane. Four stationary targets and three obstacles reside within the search area. In Fig. 2.13, the targets are represented by circles and each one is assumed to emit a different intensity level. The targets simulate radiation or other volatile sources, depending on the desired application. Obstacles in the environment, such as buildings, impassable mountain ranges and large bodies of water, are approximated by basic geometric shapes that the robots must navigate around in order to converge on the target of interest. The obstacles shown in Fig. 2.13 include: a lake approximated by a circle, a cluster of three small

42

L.L. Grant and G.K. Venayagamoorthy

Fig. 2.13. Graphical representation of multi-target/obstacle case utilizing single swarm. (♦=robots, o=targets)

square buildings approximated by a triangle, and a large irregularly shaped building approximated by a rectangle. The solid figures are the actual obstacles and the shapes enclosing them represent the obstacle approximation boundaries. Thirty robots, represented by stars, are randomly distributed in the search area. For simulation, any number of robots, obstacles, or targets is applicable. The PSO and greedy algorithms are applied to solve the CRS problem in this environment. The simulation results are provided in Table 2.5. The algorithms are analyzed based on the simulation run-time, fitness evaluations until swarm convergence, distance traveled by the robots, and percent convergence on all targets. The results are further broken down by the average value with standard deviation for one trial, and the maximum and minimum values observed out of 100 total trials. One trial involves the convergence of the entire swarm on all targets in the search environment. A single iteration is equal to each of the thirty robots completing one change in position. A fitness evaluation occurs at every instance when a robot checks the fitness of its current position relative to the targets. The distance traveled is the total distance traveled by the swarm in units. The convergence percentage is obtained by tracking whether or not each robot reaches every target by a distance of less than 0.01 units for a single trial within the set Table 2.5. PSO and Greedy Multi-Target/Obstacle Results (Averaged over 100 trials) Mean ± std PSO Greedy Time(sec) 15 ± 0.83 23.8 ± 1 Fitness Evaluations 15,666 ± 711 70,471 ± 2,188 Distance Traveled 19,848 ± 2,054 158,720 ± 6,988

Max PSO 17.91 18,660 31,588

Greedy 28.24 78,536 179,200

Min PSO 14.02 14,580 16,864

Greedy 22.33 65,836 146,200

2 Swarm Intelligence for Collective Robotic Search

Fig. 2.14. Trajectories of a greedy swarm converging on a target

Fig. 2.15. Trajectories of a PSO swarm converging on a target

43

44

L.L. Grant and G.K. Venayagamoorthy

limit of a maximum of 500 iterations per target. If the final fitness of a single robot is greater then 0.01 for any target, then the convergence for that trial is 0%. Both the PSO and greedy algorithms had 100% convergence on all targets over 100 trials. For time, fitness evaluations, and distance traveled, PSO performs better than greedy search. PSO converges around 1.6 times faster and performs around five times fewer fitness evaluations than the greedy search. The robots driven by PSO also covered eight times less distance than the greedy robots in order to locate the targets. Most of the extra distance gained by the greedy robots occurs during the mapping of the search points when the robot tries to locate the next position of best fitness. Also, the greedy algorithm is less compatible with the repulsion algorithm than PSO. The repulsion algorithm is unable to overcome the tendency of the greedy drive to keep the robots moving directly towards the target. The robots try repeatedly to move straight through the obstacles, attempting to reach the target on the other side. In some instances, this causes the robot to become trapped on one side of the obstacle, requiring more time, evaluations, and distance traveled while attempting to maneuver around the obstacle. This disturbance does not happen in PSO due to the swarm communication, which allows for more freedom in exploration of the search space instead of requiring a direct trajectory to the target. This phenomenon can be seen in Figs. 2.14 and 2.15 with robot paths traced for the greedy and PSO algorithms, respectively. The robot’s have the same initial positions for each algorithm indicated by the star symbols. In Fig. 2.14, the greedy robot paths are straighter than the PSO paths in Fig. 2.15 because the greedy robots are driven to take the most direct path to the target, and the PSO robot’s ability to communicate helps them avoid becoming trapped since they are not only drawn to the target but also to other robots with better fitness [3]. From the enlarged portions of the figures it can be seen that the greedy robot must re-position itself using the repulsion algorithm multiple times to get around the triangle shaped obstacle while the PSO robot is pulled toward the robots beside it and smoothly passes around the obstacle. 2.4.3

Case Study 2: Multiple Swarms

For cases where there is a large amount of terrain to explore or boundary restrictions such as large bodies of water exist and restrict explorations, it is necessary to confine the robot’s motion to a smaller area based on a divide and conquer scheme [27]. Multiple swarms are used to search smaller portions of the environment ensuring a fast and efficient parallel search. Fig. 2.16 presents a search area similar to the one in Fig. 2.13, but segregated into four neighborhoods. Two targets reside in swarm area 1 and a single target resides in swarm area 4. There are also obstacles present in swarm areas 1, 2, and 4. The robots are randomly dispersed throughout the search area and assigned to the swarm corresponding to the location of their initial search position. The fuzzy and greedy methods are applied to solve the CRS problem in this environment set-up. The results for both methods obtained over 100 trials are presented in Table 2.6. As seen in Table 2.6, the fuzzy method outperforms the greedy method with around 50% fewer iterations and a much shorter overall run-time. The greedy search also has a lower convergence percentage than the fuzzy method.

2 Swarm Intelligence for Collective Robotic Search

45

Fig. 2.16. Graphical representation of multi-target/obstacle case utilizing multiple swarms. (♦=robots, o=targets) Table 2.6. Multiple Swarm Results for Fuzzy and Greedy Searches

Fuzzy Greedy

Time

Iterations

Convergence

214.04 622.81

256.14 662.56

94% 87%

2.5 Conclusion This chapter has introduced two computational intelligence techniques and compared them with the greedy search approach for application to the navigation of a swarm of robots for a target search in a hazardous environment. All algorithms have been shown to be suitable for CRS applications based on reliability in performance when evaluating convergence. Both CI techniques, fuzzy and PSO, outperform the greedy search in terms of time, target convergence, fitness evaluations, and compatibility with an obstacle avoidance method. Greedy search requires no communication among the robots, in contrast to the other methods, but does not adapt well to the simple obstacle avoidance technique presented. Comparing the results for the greedy search in the single target search versus the multiple target search, convergence drops significantly from 100% to 87% for a more complex environmental setup. The PSO algorithm can be further optimized by the adjustment of its velocity equation parameters for a faster, more efficient search and this optimization can be carried out by PSO.

46

L.L. Grant and G.K. Venayagamoorthy

Future work involves an investigation of the effect of optimizing the membership functions and rules of the fuzzy controllers. Optimization of the membership functions, rule bases, and PSO parameter ‘w’, is expected to improve the performance of the CRS. Also, a hybrid combination of the PSO and greedy algorithms for implementing both a local and global search would be useful for this application. Recently, a collective robotic search platform has been developed for evaluating search algorithms in a laboratory setting and a successful implementation of PSO has been accomplished [19].

References 1. An, D., Wang, H.: VPH: a new laser radar based obstacle avoidance method for intelligent mobile robots. In: 5th World Congress on Intelligent Control and Automation, vol. 5, pp. 4681–4685 (2004) 2. Cassinis, R.: Landmines detection methods using swarms of simple robots. Intelligent Autonomous Systems 6 (2000) 3. Chuang-xin, G., Zhao, B.: A pooled-neighbor swarm intelligence approach to optimal reactive power dispatch. Zhejiang University SCIENCE A 7, 615–622 (2006) 4. Cook, B., Marthaler, D., Topaz, C., Bertozzi, A., Kemp, M.: Fractional bandwidth reacquisition algorithms for VSW-MCM. In: Multi-robot Systems from Swarms to Intelligent Autonoma, vol. 2, pp. 77–86. Kluwer Academic Publishers, Dordrecht (2003) 5. Cortes, J., Martinez, S., Bullo, F.: Robust rendezvous for mobile autonomous agents via proximity graphs in arbitrary dimensions. IEEE Transactions on Automatic Control 51, 1289– 1298 (2004) 6. Cui, X., Hardin, T., Ragade, R.K., Elmaghraby, A.S.: A swarm-based fuzzy logic control mobile sensor network for hazardous contaminants localization. In: IEEE International Conference on Mobile Ad-hoc and Sensor Systems, pp. 194–203 (2004) 7. Dadios, E.P., Maravillas Jr., O.A.: Cooperative mobile robots with obstacle and collision avoidance using fuzzy logic. In: IEEE International Symposium on Intelligent Control, pp. 75–80 (2002) 8. del Valle, Y., Venayagamoorthy, G.K., Mohagheghi, S., Hernandez, J.C., Harley, R.G.: Particle swarm optimization: basic concepts, variants and applications in power systems. IEEE Transactions on Evolutionary Computation (accepted, 2003) 9. Doctor, S., Venayagamoorthy, G.K., Gudise, V.G.: Optimal PSO for collective robotic search applications. In: IEEE Congress on Evolutionary Computation, vol. 2, pp. 1390–1395 (2004) 10. Eberhart, R., Shi, Y.: Particle swarm optimization: developments, applications and resources. In: IEEE Congress on Evolutionary Computation, vol. 1, pp. 81–86 (2001) 11. Er, M.J., Deng, C.: Obstacle avoidance of a mobile robot using hybrid learning approach. IEEE Transactions on Industrial Electronics 52, 898–905 (2005) 12. Fujimori, A., Nikiforuk, P.N., Gupta, M.M.: Adaptive navigation of mobile robots with obstacle avoidance. IEEE Transactions on Robotics and Automation 13, 596–601 (1997) 13. Hu, E., Yang, S.X., Chiu, D.K.Y., Smith, W.R.: Real-time tracking control with obstacle avoidance of multiple mobile robots. In: IEEE International Symposium on Intelligent Control, pp. 87–92 (2002) 14. Jadbabaie, A., Lin, J., Morse, A.S.: Coordination of groups of mobile autonomous agents using nearest neighbor rules. IEEE Transactions on Automatic Control 48, 988–1001 (2003) 15. Jennings, J.S., Whelan, G., Evans, W.F.: Cooperative search and rescue with a team of mobile robots. In: 8th International Conference on Advance Robotics, pp. 193–200 (1997) 16. Kennedy, J., Eberhart, R.: Particle swarm optimization. In: IEEE International Conference on Neural Networks, vol. 4, pp. 1942–1948 (1995)

2 Swarm Intelligence for Collective Robotic Search

47

17. Kennedy, J., Eberhart, R.C., Shi, Y.: Swarm Intelligence. Morgan Kauffman, San Francisco (2001) 18. Lin, Z., Broucke, M., Francis, B.: Local control strategies for groups of mobile autonomous agents. IEEE Transactions on Automatic Control 49, 622–629 (2004) 19. Palangpour, P., Parrott, C., Grant, L.: Collective robotic search in software and hardware implementation. IEEE Industry Applications Magazine 14, 71–73 (2008) 20. Parsopoulos, K.E., Vrahatis, M.N.: On the computation of all global minimizers through particle swarm optimization. IEEE Transactions on Evolutionary Computation 8, 211–214 (2004) 21. Russell, R.A., Thiel, D., Deveza, R., Mackay-Sim, A.: A robotic system to locate hazardous chemical leaks. In: IEEE International Conference on Robotics and Automation 1 (1995) 22. Shi, Y., Eberhart, R.C.: Parameter selection in particle swarm optimization. Evolutionary Programming 7, 611–616 (1998) 23. Shi, Y., Eberhart, R.C.: Empirical study of particle swarm optimization. In: Congress on Evolutionary Computation, vol. 3, p. 1950 (1999) 24. Smith, L., Venayagamoorthy, G.K.: Optimized pso for collective robotic search in an environment with multiple targets and obstacles. In: The International Journal of Engineering Applications of Artificial Intelligence (in preparation, 2007) 25. Tang, Z., Ozguner, U.: Motion planning for multitarget surveillance with mobile sensor agents. IEEE Transactions on Robotics 21, 898–908 (2005) 26. Thayer, S., Digney, B., Dias, M.B., Stentz, A., Nabbe, B., Hebert, M.: Distributed robotic mapping of extreme environments. SPIE Mobile Robots XV and Telemanipulator and Telepresence Technologies VII, p. 4195 (2000) 27. Venayagamoorthy, G.K., Doctor, S.: Navigation of mobile sensors using PSO and embedded PSO in a fuzzy logic controller. In: 39th IEEE IAS Annual Meeting on Industry Applications, vol. 2, pp. 1200–1206 (2004) 28. Venayagamoorthy, G.K., Doctor, S., Smith, L.: Collective robotic search using hybrid techniques: fuzzy logic and Swarm intelligence inspired by nature. The International Journal of Engineering Applications of Artificial Intelligence (submitted, 2007) 29. Wang, L.X.: A course in fuzzy systems and control. Prentice Hall, Inc., Upper Saddle River (1996) 30. Ye, C., Yung, N.H.C., Wang, D.: A fuzzy controller with supervised learning assisted reinforcement learning algorithm for obstacle avoidance. IEEE Transactions on Systems, Man, and Cybernetics 33, 17–27 (2003) 31. Zadeh, L.A.: Fuzzy Logic. CSLI, 83–93 (1988)

3 A Glowworm Swarm Optimization Based Multi-robot System for Signal Source Localization K.N. Krishnanand1 and D. Ghose2 1

2

Mobile Robotics Laboratory (GCDSL), Department of Aerospace Engineering, Indian Institute of Science, Bangalore [email protected] Mobile Robotics Laboratory (GCDSL), Department of Aerospace Engineering, Indian Institute of Science, Bangalore [email protected]

Abstract. We address the problem of multiple signal source localization where robotic swarms are used to locate multiple signal sources like light, sound, heat, leaks in pressurized systems, hazardous plumes/aerosols resulting from nuclear or chemical spills, fire-origins in forest fires, hazardous chemical discharge in water bodies, oil spills, deep-sea hydrothermal vent plumes, etc. In particular, we present a multi-robot system that implements a modified version of the glowworm swarm optimization (GSO) algorithm, which is originally developed to solve multimodal function optimization problems, for this purpose. The GSO algorithm uses a leapfrogging behavior for the basic search capability and an adaptive decision range that enables the agents to partition into disjoint subgroups, simultaneously taxis towards, and rendezvous at, multiple source locations of interest. Transition of agent behaviors from simulation to real-robot-implementation needs modifications to certain algorithmic aspects mainly because of the point-agent model of the basic GSO algorithm and the physical dimensions and dynamics of a real robot. We briefly describe the basic GSO algorithm and the modifications incorporated into the algorithm in order to make it suitable for a robotic implementation. Realization of each sensing-decision-action cycle of the GSO algorithm requires the robots to perform subtasks such as identification and localization of neighbors, selection of a leader among current neighbors, updating of the associated luciferin and local-decision range, and making a step-movement towards the selected leader. Experiments in this regard validate each robot’s capability to perform the above basic algorithmic primitives. Real-robot-experiments are conducted in the context of light source localization in order to validate the GSO approach to localization of signal sources. These experiments constitute a first step toward implementation in multiple robots for detection of multiple sources.

3.1 Introduction Group intelligence found in biological swarms − cooperative foraging of ant colonies, evading of predators by fish schools, flocking of birds, group-hunting found in canids, herons, and several cetaceans − is a result of actions of each individual in a group based on simple interactions with immediate neighbors and perception of the local environment. Swarm robotics draws from these principles in order to devise decentralized algorithms for control and coordination of multiple mobile robots when the groupsize is very large. The topic represents one of the applications of the general area of D. Liu, L. Wang, and K.C. Tan (Eds.): Design and Control of Intel. Robotic Systems, SCI 177, pp. 49–68. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com 

50

K.N. Krishnanand and D. Ghose

computational intelligence (CI) where order and structure emerges from an unstructured beginning [1]. Localization of multiple signal sources using mobile robot swarms has received some attention recently in the collective robotics community. In particular, the problem of multiple source localization involves the deployment of a group of mobile robots that use their sensory perception of signal-signatures at distances that are potentially far from each source, and interaction with their neighbors, as cues to guide their movements towards, and eventually co-locate at, the multiple signal-emitting sources. Examples of such sources include light, sound, heat, leaks in pressurized systems [2], hazardous plumes/aerosols resulting from nuclear/chemical spills [3], fire-origins in forest fires [4], hazardous chemical discharge in water bodies [5], oil spills [6], deep-sea hydrothermal vent plumes [13], etc. Most prior work related to signal source localization was devoted to using either single or multiple robot systems for seeking, detecting, and tracking of a single emission source location [2]−[12]. However, very little attention has been given to the equally important problem of localization of multiple sources [13], [14]. This problem has practical relevance to situations where it is imperative to simultaneously identify and neutralize all the radiating sources before they can cause a great loss to the environment and people in the vicinity. The above problem presents several challenges such as multiplicity of sources, need for simultaneous localization of all sources, time-varying nature of the sources, dynamically changing environment, and perceptional noise. In the above applications, the signal-source profile that spreads in the environment can be represented as a multimodal function. Therefore, the problem of multiple source localization can be modelled as optimization of multimodal functions. Multimodal function optimization has been addressed extensively in the recent literature. The traditional class of problems related to this topic focussed on developing algorithms to find either the global optimum or all the global optima of the given multimodal function, while avoiding local optima. However, the multiple signal source localization problem belongs to the class of optimization problems which has the objective of finding multiple optima having either equal or unequal function values. In this chapter, we present a multi-robot system that implements a modification to an algorithm called the glowworm swarm optimization (GSO) algorithm and demonstrate the GSO based robotic system’s suitability for source localization tasks. The basic GSO algorithm, which is originally developed for multimodal optimization problems, enables a homogeneous swarm of mobile agents to partition into subgroups, exhibit simultaneous taxis towards, and rendezvous at multiple optima (not necessarily equal) of a multimodal function. The underlying principles of the GSO algorithm and numerical experiments to show the algorithm’s efficacy in capturing multiple peaks of a wide range of multimodal functions are presented in [16]. We show the GSO algorithm’s applicability to the problems of hazard sensing in ubiquitous environments in [17] and pursuit of multiple mobile signal sources in [18]. However, in all the above work, the algorithm’s suitablity for multiple source localization tasks is tested using only non-embodied point simulations, where agents are considered as points and agentcollisions are ignored. In this chapter, we conduct real-robot-experiments, using a set of four wheeled robots, in order to assess the GSO algorithm’s suitability for source

3 A Glowworm Swarm Optimization Based Multi-robot System

51

localization tasks. There are mainly three issues that arise during a robotic implementation and are not taken into account in the algorithmic description of GSO: 1. The linear and angular movements of agents in the algorithm are instantaneous in nature. However, embodied robots spend a finite time in performing linear and angular movements based on their prescribed speeds and turn rate capabilities. 2. In the algorithm, agents are considered as points and agent collisions are ignored. These are tenable assumptions for numerical optimization problems. However, robots have a physical shape and foot print of finite size, and cannot have intersecting trajectories. Thus, robots must avoid collisions with other robots in addition to avoiding obstacles in the environment. 3. The GSO algorithm uses a leapfrogging behavior (described in Section 3.3), where agents move over one another, to perform local search. Since robots perform collision avoidance with other robots, a robotic implementation of the leapfrogging effect becomes a critical issue and is not straightforward. The above issues are very important and need a careful consideration as they call for changes in the agent movement models and alter the working of the basic algorithm when it is implemented on a swarm of real robots. For instance, since physical robots cannot move over one another, a direct implementation of the leapfrogging behavior is not possible. However, the obstacle avoidance feature alters the algorithm’s behavior, giving rise to a partial leapfrogging effect, which is described in Section 3.3. Therefore, the robot experiments that we conduct in this chapter serve as means to validate the transition of agent behaviors from simulations to a real world implementation of the GSO algorithm, examine the altered algorithm’s behavior, and study the robustness of the algorithm to implementational constraints. The chapter is organized as follows. Prior work on source localization is given in Section 3.2. An overview of the GSO algorithm and its special suitability for localizing multiple sources are described in Section 3.3. The GSO variant for robotic implementation is presented in Section 3.4. The multi-robot platform is described in Section 3.5. The results of real-robot-experiments are discussed in Section 3.6. Concluding remarks and directions for future work are given in Section 3.7.

3.2 Prior Work Approaches to robotic signal source localization include gradient-based [11], modelbased [12], biomimetic [8]− [10], and occupancy grid-based [13], [15] strategies. Multi-robot networks have been proposed to evaluate the performance of adding cooperation as an additional layer to the above strategies [10], [11]. Over the recent years, swarm robotics is increasingly being used for source detection applications [2], [3], [7], [14]. Fronczek and Prasad [2] propose bio-inspired sensor swarms to detect leaks in pressurized systems where sensory perception of agents is based on the biological response of cockroaches to changes in ambient air-flow patterns, communication architecture is modelled around a bee colony localizing an odor source, and the self-repair abilities emulates human decision making. Zarzhitsky et al. [3] design a swarm robotic network based on the principles of fluid physics for chemical plume tracing tasks. Hayes

52

K.N. Krishnanand and D. Ghose

et al. [7] describe a spiral-surge algorithm in which a collection of autonomous mobile robots use spiral plume finding, surge, and spiral casting behaviors to find the source of an odor plume. However, all these implementations are for single odor source detection and do not explicitly address the problems of simultaneous localization of multiple sources. Cui et al. [14] present a multiple source localization algorithm in which mobile agents use a grid map to represent the unknown environment, collect concentration values from all other agents through ad-hoc communication, and calculate a positive gradient direction using a biasing expansion swarm approach (BESA). According to this approach, each agent moves to one of the neighboring eight cells on which the net influence (called the biasing parameter) of other agents is maximum. Each agent’s influence is proportional to the concentration of the diffused source at its location and inversely proportional to the square of its distance to the cell. The cohesion property of this swarming approach maintains the connectivity of the ad-hoc network. Note that each agent uses global information to decide its movements. Even though the authors consider multiple sources, the algorithm does not explicitly take into account the interference problems that are caused due to multiplicity of sources. However, the GSO algorithm, that was originally designed for multimodal optimization problems [16], is specially suited for multiple source localization tasks, which is described in the next section.

3.3 The GSO Algorithm The glowworm swarm optimization (GSO) algorithm is an optimization technique that is originally developed for simultaneous capture of multiple optima of multimodal functions. The underlying principles of the GSO algorithm and numerical experiments to show the algorithms efficacy in capturing multiple peaks of a wide range of multimodal functions are presented in [16]. Other earlier work on the GSO algorithm can be found in [17] and [18]. As this chapter focusses on the implementation aspects of GSO algorithm for multiple source localization tasks using a multi-robot system, we limit ourselves to a brief description of the GSO algorithm. 3.3.1

GSO Algorithm Description

In the GSO algorithm, the agents are initially deployed randomly, according to a uniform distribution, in the objective function space. Each agent carries a luminescence quantity called luciferin that encodes the information about the function profile at its current location. Agents are thought of as glowworms that emit a light whose intensity is proportional to the associated luciferin. A glowworm considers other glowworms that are located within its local-decision domain and those with a higher luciferin value than that of its own as neighbors. The decision domain is adaptive by nature and is bounded above by a circular sensor range. Each glowworm selects a neighbor using a probabilistic mechanism and moves towards it. That is, each glowworm is attracted by the brighter glow of other neighboring glowworms. These individual agent movements, that are based only on local information, enable the swarm of glowworms to partition into disjoint subgroups, exhibit simultaneous taxis-behavior towards, and rendezvous at multiple optimums of a given multimodal function.

3 A Glowworm Swarm Optimization Based Multi-robot System

53

G LOWWORM S WARM O PTIMIZATION (GSO) A LGORITHM deploy agents randomly; ∀ i, set i (0) = 0 ∀ i, set rdi (0) = r0 { for each glowworm i do: i (t) ← (1 − ρ )i(t) + γ J(xi (t + 1)); for each glowworm i do: { Ni (t) = { j : di j (t) < rdi (t); i (t) <  j (t)}; for each glowworm j ∈ Ni (t) do: p j (t) =

 j (t) − i(t) ; ∑k∈Ni (t) k (t) − i (t)

j = select glowworm j; % (using p j (t))   x j (t) − xi(t) xi (t) ← xi (t) + s ; x j (t) − xi(t) rdi (t) ← min{rs , max{0, rdi (t) + β (nt − |Ni (t)|)}}; }

}

t ← t + 1;

The leapfrogging effect: According to the glowworm algorithm presented in this paper, a glowworm with the maximum luciferin at a particular iteration remains stationary during that iteration. Ideally, the above property leads to a dead-lock situation when all the glowworms are located such that the peak-location lies outside the convex-hull formed by the glowworm positions. Since the agent movements are restricted to the interior region of the convex-hull, all the glowworms converge to a glowworm that attains maximum luciferin value during its movements within the convex-hull. As a result, all the glowworms get co-located away from the peak-location. However, the discrete nature of the movement-update rule automatically takes care of this problem which could be described in the following way. During the movement phase, each glowworm moves a distance of finite step-size s towards a neighbor. Hence, when a glowworm i approaches closer to a neighbor j such that the inter-agent distance becomes less than s, i leapfrogs over the position of j and becomes a leader to j. In the next iteration, i remains

54

K.N. Krishnanand and D. Ghose

stationary and j overtakes the position of i thus regaining its leadership. This process of interchanging of roles between i and j repeats giving rise to a local-search behavior of the glowworm pair along a single ascent direction. A group of glowworms use the same principle to perform an improved local-search and eventually converge to the peak location. The leapfrogging behavior of the agents in the GSO algorithm are supported by simulations in Section 3.6. The various steps of the GSO algorithm are given in the inset box, where i (t) represents the luciferin level associated with glowworm i at time t, ρ is the luciferin decay constant (0 < ρ < 1), γ is the luciferin enhancement constant, J(xi (t)) represents the value of the signal strength at agent i’s location at time t, xi (t) ∈ ℜm is the location of glowworm i, at time t, in the m−dimensional real space ℜm , Ni (t) is the neighborhood of glowworm i at time t, di, j (t) represents the euclidian distance between glowworms i and j at time t, rdi (t) represents the variable local-decision range associated with glowworm i at time t, s (> 0) is the step-size, rs represents the radial range of the luciferin sensor, the glowworm i selects to move towards a glowworm j ∈ Ni (t) with probability p j (t), and nt represents the desired number of neighbors. 3.3.2

A Simulation Example to Illustrate Basic Working of GSO

Simulation results demonstrating the capability of the glowworm algorithm to capture multiple peaks of a number of complex multimodal functions have been reported in [17]−[19]. In this paper, we demonstrate the basic working of the algorithm using the multimodal function given below: Q

J(x, y) =

∑ ai exp(−bi ((x − xi)2 + (y − yi)2 ))

(3.1)

i=1

where, Q represents the number of peaks and (xi , yi ) represents the location of each peak. A workspace of (−5, 5) × (−5, 5) and a set of ten peaks (Q = 10) are considered for the purpose. Initially, we consider equal slopes for all the peaks by choosing bi = 3, i = 1, . . . Q. The values of ai , xi , and yi are generated according to the following equations: ai = 1 + 2 rand() xi = −5 + 10 rand() yi = −5 + 10 rand() where, rand() is a uniform random variable within the interval [0, 1]. The corresponding values of {ai , xi , yi , i = 1, . . . , 10} for the function profile are shown in Table 3.1. Figure 3.1(a) shows the multimodal function profile chosen for the purpose and Figure 3.1(b) shows the emergence of solution when n = 300 and rs = 5. We observe that when the slopes of the peaks are equal, the swarm splits into subgroups according to the Voronoi-partition of the peak locations, that is, agents deployed in the Voronoi-partition of a peak location remain in the same partition, during their movements, and eventually converge to the corresponding Voronoi-center that coincides with the peak location. However, a few agents located near the border region of

3 A Glowworm Swarm Optimization Based Multi-robot System

55

Table 3.1. Values of ai , xi , and yi used to generate the function profile J(x, y) 1 2 3 4 5 6 7 8 9 10 ai 2.6160 2.4270 1.2225 2.3660 1.3380 2.2851 2.3253 1.0020 1.7090 2.9846 xi 2.9329 -4.0835 3.4591 2.0655 -4.8141 4.3812 -1.1106 -1.7107 -1.7282 -3.2330 yi -0.5591 -2.1457 -2.8029 2.7104 4.7894 3.2815 -1.4957 -3.9765 0.2440 -3.1662

5

2.5 4 3

1.5

2

1

1 Y

7

J1

2

0.5

0 −1

5

−2

5 0

0

y

−3 −4

x −5

−5

(a)

−5 −5

−4

−3

−2

−1

0 X

1

2

3

4

5

(b)

Fig. 3.1. a) Multimodal function profile. b) Emergence of agent movements and their eventual co-location at the peaks.

a Voronoi-partition migrate into adjacent partitions and eventually get co-located at the respective peaks. Algorithmic aspects such as usage of a point-agent model with instantaneous linear/angular agent-movements, allowing of collisions between agents, and enabling the agents to move over one another are expected to create problems during implementation on real robots. These aspects that have an effect on the algorithm’s behavior will be discussed next.

3.4 Algorithmic Modifications for Robotic Implementation A robotic implementation of the GSO algorithm would require a collection of mobile robots where each robot has the following capabilities: 1. Sensing and broadcasting of profile-value (luciferin level) at its location. 2. Detection of number of neighbors and their localization with respect to its own position. 3. Receiving profile-values (luciferin levels) from all neighbors. 4. Selecting a neighbor using a probability distribution (based on the relative luciferin levels of its neighbors) and making a step-movement towards it. 5. Obstacle avoidance behavior in order to avoid collisions with obstacles and other robots in the environment. 6. Leapfrogging behavior for performing local search.

56

K.N. Krishnanand and D. Ghose

As we assume perfect sensing and broadcast in our initial experiments, the first three robot capabilities are rather straightforward to implement and do not need a discussion. However, there are a few issues related to resolution of the sensing elements and communication delays/interference which will be discussed later. Implementation details of obstacle avoidance and leapfrogging behaviors are given below: 3.4.1

Obstacle Avoidance Model

The footprint of the robot is considered to be a circle of radius rrobot . When a robot approaches closer than a safe distance dsa f e < s to another robot, it performs a simple collision-avoidance maneuver through a sequence of point-turn and straight-line movements. 3.4.2

Implementation Models for Leapfrogging Behavior

The leapfrogging effect of the GSO algorithm requires a glowworm to move over another glowworm whenever two glowworms are within a step distance s of each other. This is illustrated in Figure 3.2(a) where glowworm A moves over glowworm B to reach the position A(t + 1). However, this is not directly realizable with physical mobile robots. We describe three methods to achieve an explicit robotic implementation of the leapfrogging behavior. The first method (Figure 3.2(b)) involves interchange of agent-roles, through communication, where robot B moves to the desired position of

A(t+1) B(t)

B(t+1)

A(t)

A(t+k)

B(t+1) B(t) A(t)

(a)

A(t) (c)

(b)

A(t+k)

A(t+k) B(t)

B(t+k)

A(t)

B(t+k)

B(t)

A(t+1)

B(t) B(t+k) A(t)

(d)

(e)

Fig. 3.2. a) Leapfrogging behavior according to the GSO algorithm. b), c), and d) Three methods of robotic implementation of explicit leapfrogging behavior. e) Implicit leapfrogging behavior due to obstacle avoidance.

3 A Glowworm Swarm Optimization Based Multi-robot System

57

robot A and robot A replaces robot B’s position. The illustration in Figure 3.2(c) shows another method where the robot A takes a detour about robot B’s position and reaches its desired position. We propose a modification to this method where the detour is achieved using a blend of two circular paths as shown in Figure 3.2(d). Implicit leapfrogging using obstacle avoidance: We observe that the obstacle avoidance behavior automatically gives rise to an implicit leapfrogging effect, which is described using the illustration in Figure 3.2(e). Robot A alternatively performs collision avoidance with and seeks to move towards robot B until it crosses the equi-contour line on which robot B is situated. Thereafter, since robot A becomes a leader to robot B, robot performs similar movements with respect to robot A, thus leading to an implicit leapfrogging behavior of the glowworm-pair.

3.5 Kinbots: Multi-robot Platform for Real-Robot-Experiments We use a set of four wheeled robots, called Kinbots (See Figure 3.3), that were originally built for experiments related to robot formations [19]. Realization of each sensing-decision-action cycle of the GSO algorithm requires the robots to perform subtasks such as identification and localization of neighbors, selection of a leader among current neighbors, updating of the associated luciferin and local-decision range, and making a step-movement toward the selected leader. By making necessary modifications to the Kinbot hardware, the robots are endowed with the capabilities required to implement the various behavioral primitives of the GSO algorithm. A PIC16F877

Fig. 3.3. Multi-robot system built for source localization experiments

58

K.N. Krishnanand and D. Ghose

Left motor

Right motor

Photodiode output Light sensor

Driver stage

+5V

-

C

R

A/D +

PWM module

Bandpass filter

Step motor

Driver stage

Luciferin receiver

PIC 16F877 Micro controller RX (RS 232)

I/O PORT

Home sensor

Step + Direction

1 KHz signal

TX (RS 232)

33 KHz

(center freq. = 1KHz)

Array of 16 IR LEDs (Beacon)

Array of 8 IR leds (Luciferin transmitter)

Fig. 3.4. The hardware architecture of Kinbots

microcontroller is used as the robot’s processing unit. The hardware architecture of Kinbots is shown in Figure 3.4. The various hardware modules of each robot that are used to achieve the above tasks are shown in Figure 3.5. The tether to the robots (refer to Figure 3.3) provide only power to the robots from an external power supply and is not used for data communication purpose. However, the tethered power supply can be replaced by using an onboard battery. Real-robot-experiments are conducted for the purpose of

3 A Glowworm Swarm Optimization Based Multi-robot System

59

Light intensity pick-up sensor

M1

Sweep platform

M2

Luciferin receiver

M3

Luciferin transmitter (Array of 8 IR LEDs)

M4

Bandpass filter

M5

Photodiode (mounted on a rotary platform)

M6

Circular array of 16 LEDs placed radially outward (Serves as a beacon to other robots)

M7

Differential mobile robot base

M8

Fig. 3.5. Various hardware modules used by the Kinbots to implement the GSO behaviors

light source localization in order to validate the GSO approach to localization of signal sources. 3.5.1

Relative Localization of Neighbors

The various modules and sensors are marked as M1, M2, . . . , M8 in the Figure 3.5. The neighbor localization hardware module is achieved in the following way. A circular

60

K.N. Krishnanand and D. Ghose

array of sixteen infrared LEDs (M7 in Figure 3.5) placed radially outward serves as a beacon to obtain a near circular emission pattern around the robot. These IR LEDs are actuated by a 1 KHz square wave signal. A photodiode (M6), mounted on a rotary platform (M2), performs a sweep and records the intensity of the received infrared light as a function of the angle made by the direction of the source with the heading direction of the robot. Local peaks in the intensity pattern indicate the presence of neighbors and serve to compute their location in polar coordinates. In particular, the received intensity of infrared light follows a inverse square law with respect to distance which is used to compute range information to robot-neighbors and the angular position of the local peak is approximated as the relative angle to a neighbor. The photo-diode output is passed through a bandpass filter, centered around a frequency of 1 KHz, in order to make it sensitive only to the frequency of the actuation signal of the infrared-beacons and reject noise due to ambient light conditions and IR signals used to broadcast luciferin information. 3.5.2

Luciferin Glow/Reception

As we test the GSO algorithm in light source localization experiments, a light pick-up sensor (M1) is used to measure the light intensity at the robot’s location. The light sensor output is fed to an in-built analog-to-digital (A/D) convertor of the microcontroller. The output of the A/D module is converted into a luciferin glow that consists of an infrared emission modulated by an 8-bit serial binary signal that is proportional to the robot’s luciferin value at the current sensing-decision-action cycle. Eight IR emitters (M4) that are mounted symmetrically about the robot’s central axis are used to broadcast luciferin data in its neighborhood. An infrared receiver (M3) mounted on a sweep platform (M2) is used as a luciferin receptor. In order to avoid problems due to interference between data signals coming from different neighbors, the receiver sweeps and aligns along the line-of-sight with a neighbor before reading the luciferin data transmitted by it.

3.6 Real-Robot-Experiments A real-robot-implementation of the GSO algorithm would require the robots to have a similar set of capabilities that were used to implement the GSO algorithm in embodied simulations. The movements of the robots are achieved using a differential mobile platform (M8). The obstacle avoidance maneuver is achieved by the movement of the robot along an arc in embodied simulations. However, in real-robot implementation, when a robot approaches closer than a safe distance dsa f e < s to another robot, it performs a simple collision-avoidance maneuver through a discrete sequence of point-turn and straight-line movements. The resulting change in the obstacle avoidance behavior also leads to a change in the evolution of the leapfrogging behavior of the robots during real world experimentation. 3.6.1

Experiment 1: Basic GSO Behavior

According to the GSO algorithm, a glowworm moves toward a neighbor whose luciferin value is relatively higher than that of itself. It remains stationary either when it has

3 A Glowworm Swarm Optimization Based Multi-robot System

61

the highest luciferin value in its neighborhood or when it is isolated. The above basic behavior is implemented with two Kinbots in the following way. The first Kinbot (KA ), which is initially isolated, starts scanning its environment for neighbors using its sweepsensor module. Another Kinbot (KB ) is allowed to move from a far away point and stop once it enters the sensing range of KA . The robots emit constant luciferin values with A < B . Snapshots taken from a video of the above experiment at 2 sec intervals are shown in Figure 3.6. The circular arc imposed on the snapshots represents the sensor range of KA . Note that it shifts its relative position from one snapshot to the other as KA advances toward KB . At t = 0 sec, KA is isolated and hence remains stationary until KB enters its neighborhood at t = 6 sec. At this point of time, KA senses the presence of KB , performs a point-turn until its heading direction is aligned along the line-of-sight with KB , reads the luciferin value broadcast by KB , starts moving toward KB (since A < B ) at t=8 sec, traverses a distance of about 10 cm, and stops moving at t = 10 sec. Note that the sensing-decision-action cycle time is approximately 4 sec. This 4 sec-cycle is repeated twice from t=10 sec to t = 18 sec.

Sensing range

(a) t = 0 sec

(b) t = 2 sec

(c) t = 4 sec

(d) t = 6 sec

(e) t = 8 sec

(f) t = 10 sec

(g) t = 12 sec

(h) t = 14 sec

(i) t = 18 sec

Fig. 3.6. Snapshots taken from a video of the experiment used to demonstrate the basic GSO behavior

62

K.N. Krishnanand and D. Ghose

(a) t = 0 sec

(b) t = 5 sec

(c) t = 10 sec

(d) t = 15 sec

(e) t = 20 sec

(f) t = 25 sec

(g) t = 30 sec

(h) t = 35 sec

(i) t = 40 sec

(j) t = 45 sec

(k) t = 50 sec

(l) t = 55 sec

(m) t = 60 sec

(n) t = 65 sec

(o) t = 70 sec

Fig. 3.7. Snapshots taken from a video of the experiment used to demonstrate the leapfrogging behavior

3 A Glowworm Swarm Optimization Based Multi-robot System

3.6.2

63

Experiment 2: Leapfrogging Behavior

We conduct experiments with two Kinbots KA and KB to demonstrate the implicit leapfrogging behavior caused due to the incorporation of obstacle avoidance behavior into the basic GSO algorithm. Similar to the previous experiment, the robots emit constant luciferin values with A < B and KB is kept stationary. KB is placed such that it is within the sensing range of KA . However, the heading of KA is, not aligned along the line-of-sight with KB and instead, kept at a small acute angle away from it. Snapshots taken from a video of this experiment at 5 sec intervals are shown in Figure 3.7. Initially, KA performs a point turn such that it is oriented toward KB and then starts moving towards KB (Figure 3.7(b)). As KA reaches closer than about 10 cm to KB at t = 20 sec (Figure 3.7(e)), it toggles from the basic GSO behavior of moving toward a leader to an obstacle avoidance behavior at t = 25 sec (Figure 3.7(f)). The collision avoidance maneuver consists of the following sequence of movements: 1. A clock-wise point-turn of about 45o . 2. A straight-line movement of about 10 cm. 3. A counter clockwise point-turn of 45o . The above sequence is repeated for another two times until the inter-robot distance becomes more than 10 cm at t = 40 sec (Figure 3.7(i)). Now, KA switches to its basic GSO behavior and resumes to move toward KB . Thereafter, this process of toggling between basic GSO and obstacle avoidance behaviors is repeated that gives rise to an implicit leap-frogging behavior of KA around KB . The path traced by robot KA as it approaches, and leapfrogs over, robot KB is shown in Figure 3.8.

Fig. 3.8. Paths traced by the robot KA as it approaches, and leapfrogs over KB

64

K.N. Krishnanand and D. Ghose

Light source

(a) t = 0 sec

(b) t = 5 sec

(c) t = 10 sec

(d) t = 15 sec

(e) t = 20 sec

(f) t = 25 sec

(g) t = 30 sec

(h) t = 35 sec

(i) t = 40 sec

(j) t = 45 sec

(k) t = 50 sec

(l) t = 55 sec

(m) t = 60 sec

(n) t = 65 sec

(o) t = 70 sec

Fig. 3.9. Snapshots taken from a video of the light source localization experiment

3 A Glowworm Swarm Optimization Based Multi-robot System

(p) t = 75 sec

(q) t = 80 sec

(s) t = 90 sec

(t) t = 95 sec

(v) t = 105 sec

(w) t = 110 sec

65

(r) t = 85 sec

(u) t = 100 sec

(x) t = 115 sec

Fig. 3.9. (continued)

3.6.3

Experiment 3: Localization of a Single Light Source

In the following experiment, two Kinbots KA and KB implement the GSO algorithm to detect, taxis toward, and co-locate at a light source. The robots provide simple light intensity measurements at their current locations as inputs to the GSO algorithm running on their onboard processors. The robots use a photodiode based light sensor for this purpose. Snapshots from a video of this experiment are shown in Figure 3.9. The robots are initially deployed in such a way that KB is closer to the light source. Both the robots start scanning their respective neighborhoods and sense each other. As A (0) < B (0), KB remains stationary and KA moves toward KB until t = 10 sec (Figure 3.9(c)). However, between t = 10 sec and t = 30 sec (Figure 3.9(g)), KA remains stationary. This can be attributed to the fact that the luciferin value received by KA is corrupted. After t = 30 sec KA resumes to move toward KB until t = 35 sec (Figure 3.9(h)) when it switches to obstacle avoidance behavior. Note that KB starts moving toward KA at t= 45 sec (Figure 3.9(j)). However, at t = 45 sec, KA is still relatively far from the light source than KB . This can be attributed to the fact that when the robots are very close to each other, the difference in luciferin values interferes with the range of sensor noise. KB

66

K.N. Krishnanand and D. Ghose

Fig. 3.10. Paths traced by the robots KA and KB as they taxis toward, and co-locate at, the light source 7

KA Obstacle 6 aoidance GSO− 5 move Idle

4

Obstacle 3 aoidance

KB

GSO− 2 move Idle

1

0

0

20

40

60

80

100

120

t (sec)

Fig. 3.11. Toggling between basic GSO and obstacle avoidance behaviors of the two robots KA and KB as they interact with each other and search for the light source

3 A Glowworm Swarm Optimization Based Multi-robot System

67

stops moving at t = 65 sec (Figure 3.9(n)). However, at t= 110 sec (Figure 3.9(w)), KA is closer to the source and KA (110) > KB (110). Therefore, KB moves toward KA . This results in the robot pair moving further close to the light source. The paths traced by the two robots as the two robots execute the modified GSO algorithm is shown in Figure 3.10. The toggling between the basic GSO and obstacle avoidance behaviors of the two robots as they interact with each other, which eventually leads to their localization at the light source is shown in Figure 3.11. Note from the figure that the robots are idle during certain time intervals, which can be attributed to one of the following reasons: 1. A robot is isolated. 2. A robot is measuring the highest light intensity. 3. Both the robots are measuring almost the same light intensity (A robot is made to move toward another one only when |A − B| > 3). 4. Reception of luciferin data is corrupted. From Figure 3.11, it is also clear that the sensing-decision-action cycles of the robots are asynchronous with respect to each other. In the above experiment, two robots demonstrate localization of a single source. However, the experiment constitutes a first step toward implementation in multiple robots for detection of multiple sources.

3.7 Concluding Remarks We present a multi-robot system that employs glowworm swarm optimization algorithm to address the problem of simultaneous localization of multiple signal sources. We briefly describe the basic GSO algorithm and the modifications incorporated into the algorithm in order to make it suitable for a robotic implementation. We build a set of four robots that are endowed with the capabilities to implement the GSO algorithm and perform source localization tasks. We conduct real-robot-experiments where two robots localize a single light source in order to validate the GSO approach to signal source localization. However, by using more number of these robots, the GSO approach can be extended to multiple signal source localization.

References 1. Craenen BGW Eiben AE Computational Intelligence. Encyclopedia of Life Support Sciences, EOLSS Co. Ltd. 2. Fronczek, J.W., Prasad, N.R.: Bio-inspired sensor swarms to detect leaks in pressurized systems. In: Proceedings of IEEE International Conference on Systems, Man and Cybernetics (2005) 3. Zarzhitsky, D., Spears, D.F., Spears, W.M.: Swarms for chemical plume tracing. In: Proceedings of IEEE Swarm Intelligence Symposium (2005) 4. Casbeer, D.W., Beard, R.W., McLain, T.W., Li, S.-M., Mehra, R.K.: Forest fire monitoring with multiple small UAVs. In: Proceedings of American Control Conference (2005) 5. Farrell, J., Li, W., Pang, S., Arrieta, R.: Chemical plume tracing experimental results with a REMUS AUV. In: Proceedings of Oceans 2003 Marine Technology and Ocean Science Conference (2003)

68

K.N. Krishnanand and D. Ghose

6. Clark, J., Fierro, R.: Cooperative hybrid control of robotic sensors for perimeter detection and tracking. In: Proceedings of American Control Conference (2005) 7. Hayes, A.T., Martinoli, A., Goodman, R.M.: Robotica 21, 427–441 (2003) 8. Dhariwal, A., Sukhatme, G.S., Requicha, A.A.G.: Bacterium-inspired robots for environmental monitoring. In: Proceedings of IEEE International Conference on Robotics and Automation (2004) 9. Grasso, F.W., Consi, T.R., Mountain, D.C., Atema, J.: Robotics and Autonomous Systems 30, 115–131 (2000) 10. Lytridis, C., Virk, G.S., Kadar, E.E.: Co-operative smell-based navigation for mobile robots. In: Proceedings of the 7th International Conference CLAWAR 2004. Springer, Heidelberg (2004) 11. Sandini, G., Lucarini, G., Varoli, M.: Gradient driven self-organizing systems. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (1993) 12. Farrell, J.A., Pang, S., Li, W.: IEEE Transactions on System, Man, and Cybernetics 33, 850– 863 (2003) 13. Jakuba, M.V.: Stochastic mapping for chemical plume source localization with application to autonomous hydrothermal vent discovery. PhD Thesis, Massachusetts Institute of Technology and Woods Hole Oceanographic Institution (2007) 14. Cui, X., Hardin, C.T., Ragade, R.K., Elmaghraby, A.S.: A swarm approach for emission sources localization. In: Proceedings of the 16th International Conference on Tools with Artificial Intelligence (2004) 15. Lilienthal, A., Duckett, T.: Robotics and Autonomous Systems 48, 3–16 (2004) 16. Krishnanand, K.N., Ghose, D.: Multiagent and Grid Systems: Special Issue on Recent Progress in Distributed Intelligence 3, 209–222 (2006) 17. Krishnanand, K.N., Ghose, D.: Glowworm swarm optimization algorithm for hazard sensing in ubiquitous environments using heterogeneous agent swarms. In: Prasad, B. (ed.) Soft Computing Applications in Industry. Studies in Fuzziness and Soft Computing, vol. 226, pp. 165–187. Springer, Berlin (2008) 18. Krishnanand, K.N., Ghose, D.: Chasing multiple mobile signal sources: A glowworm swarm optimization approach. In: 3rd International Conference on Artificial Intelligence (2007) 19. Krishnanand, K.N., Ghose, D.: Robotics and Autonomous Systems 53, 194–213 (2005)

4 Evolutionary Robotics: From Simulation-Based Behavior Learning to Direct Teaching in Real Environments Seiji Yamada1 and Daisuke Katagami2 1 2

National Institute of Informatics [email protected] Tokyo Institute of Technology [email protected]

Abstract. In our work on evolutionary robotics, two different approaches have been developed: simulation-based behavior learning with genetic algorithm and direct teaching with a learning classifier system in real environments. As the first approach, action-based environment recognition for an autonomous mobile robot. The robot moves in various environments by executing behaviors designed by a human designer and obtains different action sequences. These action sequences are automatically classified by self-organizing maps and the structure of the environment is identified from them. We also developed a GA-based behavior learning method in which a robot can learn suitable behaviors to recognize, and conducted simulation experiments to verify the learning ability. However, all the experiments has been done through only computer simulation. Thus we attempted to develop a direct teaching framework in which a real robot learned from human teacher using LCS in real and physical environments. Direct teaching means a human teacher gives adequate actions to a mobile robot by a GUI at work, and this teaching can accelerate a robot to learn classifiers. This framework is important to realize evolutionary robotics which can learn sufficiently fast in real environments, and we confirmed that it is useful by experimenting with a small mobile robot. In this chapter, we describe these two innovative approaches in evolutionary robotics in detail and discuss them in various points of view.

4.1 Introduction We have developed evolutionary robotics with two different approaches: simulationbased behavior learning with genetic algorithm (GA) and direct teaching with a learning classifier system (LCS) in real environments. Regarding the first approach, this chapter describes action-based environment recognition for an autonomous mobile robot (Khepera), which can not perform localization because dead reckoning is very hard. The robot moves in various environments by executing embedded behaviors and obtains the different action sequences. These action sequences are automatically classified by self-organizing maps (SOM)[15] and the structure of the environment is identified from them. For this environment recognition, we developed a GA-based behavior learning method in which a robot can learn suitable behaviors to recognize, and conducted simulation experiments[33, 34]. However, all the experiments has been done through only simulation in the work. Thus we attempted to develop a direct teaching framework in which a real robot learned D. Liu, L. Wang, and K.C. Tan (Eds.): Design and Control of Intel. Robotic Systems, SCI 177, pp. 69–96. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com 

70

S. Yamada and D. Katagami

from human teacher using LCS in real environments. Direct teaching involves a human teacher who gives adequate actions to a mobile robot through a GUI and this teaching can accelerate a robot to learn classifiers. This framework is important to realize evolutionary robotics which can learn sufficiently fast in real environments, and we confirmed that it is useful by experimenting with a small mobile robot. In this chapter, we describe these two innovative approaches in evolutionary robotics in detail and discuss them in various points of view. In the section 4.2, the simulationbased behavior learning with GA is described. Next, in the section 4.3, how we developed direct teaching with LCS in real environments is explained. The section 4.4 summaries and concludes our work.

4.2 Simulation-Based Behavior Learning Using Genetic Algorithm Some of the previous research on autonomous robots that can recognize their environments have tried to build precise geometric maps by using sensitive global sensors, e.g., vision sensor [6]. Since their main aim is to navigate a robot with accuracy, a precise map is necessary. However, a simple task like environment recognition may not require a precise map. Actually, many natural agents (like animals) can recognize their environments by using only low-sensitive local sensors [26]. In engineering, it would be useful to build a mobile robot that can recognize environments with inexpensive sensors. Thus, we tried to build a mobile robot that recognizes environments by using lowsensitive local sensors. Such a robot cannot determine its position by dead reckoning in the environment, and hence it cannot build a global map of the environment. Hence, we proposed a method whereby the robot can recognize its environments by performing a series of actions. We call this approach Action-based Environment Modeling (AEM) [33, 35]. In AEM, a mobile robot is behavior-based and acts using suitable given behaviors like wall-following. The action sequences executed in each environment are recorded and transformed into environment vectors. A robot identifies the current environment by referring to its store of environment vectors on the environments that it has already been trained in. Through research on AEM, we have recognized a significant problem: where do the suitable behaviors to AEM come from? An easy solution is that a human designs them and gives them to the robot. While this common approach works in simple environments, a human designer faces greater difficulties as the variety of environments increases because the search space becomes huge and it becomes difficult to get an intuitive understanding of the suitable behaviors. Hence, an automatic design method will inevitably be necessary [10], and for this we devised an evolutionary robotics approach [9, 18] in which the behaviors of a robot are designed using evolutionary computation. In this section, we applied evolutionary robotics to acquire suitable behaviors for AEM. We devised an evolutionary design method for such behaviors that uses GA and conducted an experimental evaluation [32]. In anticipation of a future implementation on a real mobile robot, we used a Khepera robot simulator in the experiments. The experimental results indicated that our evolutionary robotics approach is a promising way

4 Evolutionary Robotics: From Simulation-Based Behavior Learning

71

to automatically acquire suitable behaviors for AEM, and that the acquired behaviors are more efficient than hand-coded ones. 4.2.1

Related Work

Several studies on robotics [20] and artificial life [24] have taken similar approaches to AEM. Nehmzow and Smithers studied recognizing corners in simple enclosures with a self-organizing network [24]. They used direction-duration pairs, which indicated the lengths of walls and shapes of past corners, as an input vector to a self-organizing network. After learning, the network became able to identify corners. However the recognized structure was very local. Mataric represented an environment with automatons in which nodes correspond to landmarks [20]. Although such a representation is more robust than a geometric one, a mobile robot must segment raw data into landmarks and identify them. Nakamura et al. utilized a sequence of sonar data in order to reduce the uncertainties in discriminating the local structure [23]. Although the sequence consisted of sensor data (not actions), their approach is similar to AEM. Kato et al. used modified random walks to discern the environment in a statistical way. Most of these studies chose wall-following behaviors as suitable [20, 24, 33]. The behaviors were described by human designers and they were independent of the tasks. Hence, despite their success, they all face a significant problem: it is very difficult to design behaviors. There are several studies on using genetic programming (GP) [17] for mobile robots to learn behaviors [13, 16, 25, 27]. Thus far, these studies have concentrated on very few simple behaviors such as obstacle avoidance. In contrast, our aim is to learn suitable behaviors for AEM and these behaviors are complicated ones consisting of several primitive behaviors. Our research is also relevant to automatic generation of pattern recognition procedure because the environment can essentially be described as a pattern. There are studies on evolutionary design of neural networks for pattern recognition [3]. However, they focus on the traditional pattern recognition task, and none deal with the cost of recognition, which is significant in our research. Reinforcement learning is also widely applied to design behaviors of an autonomous robot [28]. In reinforcement learning, behaviors are described as a policy consisting of states in an environment and actions which can be executed in each state, and rewards are assigned to some actions in some states. The purpose of reinforcement learning is to search for the optimal policy with which a robot can obtain the maximum expected reward by executing actions. All studies on reinforcement learning in robotics try develop to the optimal policy with which a robot can trace the shortest path in an environment. Since our objective of behavior learning is to search suitable behaviors for AEM, not the shortest path from a start point to a goal in an environment, the objective of reinforcement learning is significantly different from ours. Furthermore, it is intuitively difficult to design a reward function for reinforcement learning that is equivalent to the fitness function that we use in our evolutionary robotics task. This is because our fitness function evaluates the recognition ability of AEM using behaviors and does not directly evaluate actions in certain states as does the reward function of reinforcement learning.

72

S. Yamada and D. Katagami

Fig. 4.1. Overview of AEM

4.2.2

Task: Action-Based Environment Modeling

In AEM [33], a mobile robot is designed by taking a behavior-based approach [2]. Behaviors are mappings from states to actions. A state is described with conditions on sensed values, and an action is described with motor commands. A human designer describes states, actions, and behaviors so that action sequences can represent the environment’s structure. An AEM procedure consists of two stages: a training phase and a test phase (Fig. 4.1). In the training phase, classifiable training environments are given to a robot. The class is a category in which the environment should be included, and more than one environment may be included in the same class. The mobile robot acts in the training environments by using given behaviors, and obtains sequences of executed actions (called action sequences) for each of them. The action sequences (lists of symbols) are transformed into real-valued vectors (called environment vectors) using a chain coding-like method. The environment vectors are stored as instances, and a training phase finishes.

4 Evolutionary Robotics: From Simulation-Based Behavior Learning

73

In the test phase, a robot is placed in a test environment (which is typically one of the training environments). The robot tries to identify the test environment as one of the training environments it has visited, and we call this task environment recognition. Note that although a test environment may be a training environment, the action sequence of the test environment is significantly different from that of the environment because of noise affecting sensing and actions. Thus, some form of generalization is necessary for environment recognition. We used using the 1-nearest neighbor method [7] for this purpose, i.e., the robot selects the most similar instance to the test environment and considers that the class of the instance is that of the test environment. Similarity is measured as the Euclidean distance between environment vectors. We implemented an unsupervised version of AEM in which SOM [15] classified the training environment without a teacher. Through experiments with a real mobile robot, we determined that AEM was promising for environment recognition and adaptive to noisy environments [33]. However, AEM faces a significant problem: Where do the suitable behaviors come from? Since the suitable behaviors depend on the environment structures that the robot should recognize, they have thus far been described by humans. However, the task is potentially very difficult because the search space for a suitable behavior is huge: the computational complexity is O(as ) where a and s are the number of actions and states. Thus, we propose an evolutionary robotics approach to automatically acquire suitable behaviors. 4.2.3

States, Actions and Environment Vectors

Using real mobile robots as individuals in GA is currently impractical because it is impossible to operate dozens of real robots for more than 100 generations. Thus, we decided to use a simulator for acquiring behaviors and then implement the learned behaviors on a real mobile robot. Note that in this research, on-line evolutionary learning for a real robot like [25] is inapplicable. In on-line evolutionary learning, the behaviors of each individual are executed in a few time steps or in a single time step as in a time-sharing system and a fitness function independently evaluates short sequences of actions. However, to evaluate AEM, a large number of actions need to be continuously executed by an individual. Thus, on-line evolutionary learning cannot be used to acquire suitable behaviors for AEM, and real mobile robots find it very hard to learn. 4.2.3.1 A Simple Mobile Robot: Khepera We used a miniature mobile robot, KheperaTM (Fig. 4.2) as a target real robot. It has a Motorola 68331 micro-processor and 256 Kbytes of RAM. As shown in Fig. 4.3, it has two DC motors as actuators and eight infrared proximity sensors which measure the distance to obstacles and light strength. Since the sensor data is imprecise and local, Khepera cannot know its position in a global map. The later experiments used a simulator build for Khepera.

74

S. Yamada and D. Katagami

55mm

32mm

Fig. 4.2. Khepera

Fig. 4.3. Actuators, sensors on Khepera

4.2.3.2 States and Actions We describe a state as a range of a sensed values. To reduce the search space of behaviors, we restrict the number of states and actions. Each sensor on Khepera returns a 10-bit (0∼1023) value for the distance and light strength, and the value is very noisy. Thus we transform the distance value into binary values of 0 or 1. The value ee0ff means an obstacle exists within 3 cm from a robot. The value ee1ff means that no object exists there. Only three sensors (0, 2 and 5 in Fig. 4.3) are used for reducing states. Four sensors (0, 2, 5 and 7 in Fig. 4.3) are used to measure light strength. We chose for this state the sensor measuring the strongest light and assign binary values which mean a light is eenearff or eefarff. A state in which all sensors measure almost the same value is also considered. As a result, the number of states for light is nine. The total number of states is 72 (= 23 × 9) We describe the following four actions. Through preliminary experiments [33, 35], we found that four actions were sufficient for a mobile robot to do simple behaviors like wall-following. • • • •

A1: Go 5 mm straight on. A2: Turn 30◦ left. A3: Turn 30◦ right. A4: Turn 180◦ left.

A mobile robot acts in an environment by executing the above actions, and consequently it generates an action sequence. Note that we utilize action sequences, not behavior sequences, because we consider that an action describes the environment structure and several behaviors may contain the same action. 4.2.3.3 Environment Vectors The generated action sequence is transformed into an environment vector. Let an action sequence and its environment vector be [a0 , a1 , a2 , · · · , an ](ai ∈ {A1, A2, A3, A4}, a0 = 0) and V =(v1 , v2 , · · · , vm )(m ≥ n), respectively. The vector values of V are determined by the following rules. They change the vector value when the direction of movement changes in a way similar to chain coding [1]. An example of an environment vector is shown in Fig. 4.4.

4 Evolutionary Robotics: From Simulation-Based Behavior Learning

75

Fig. 4.4. Transformation into environment vectors

• • • •

If ai If ai If ai If ai

= A1 then vi = vi−1 . = A2 then vi = vi−1+1 . = A3 then vi = vi−1 − 1. = A4 then vi = −vi−1 .

As mentioned in Section 4.2.2, a robot is put in training environments in the training phase. The robot acts in these environments, and stores the environment vectors transformed from the action sequences. Next, in the test phase, the robot is put in a test environment. The robot identifies which training environment corresponds to the test environment by using the 1-nearest neighbor method and the Euclidean distance as the measure of similarity. 4.2.4

Applying GA to Acquire Behaviors

A behavior is a mapping from a state to an action. Since we have 72 states and 4 actions, the number of possible behaviors is 472 = 2.2 × 1043. We can hardly apply conventional mathematical programming to this search. Hence we use GA [11] to search for suitable behaviors to AEM in such a huge search space. Although other evolutionary computation methods like evolutionary system, genetic programming, may be also applicable to this search, we select GA because of its easy implementation and rich experimental knowledge to control it. 4.2.4.1 Ga Procedure and Coding We use a simple GA procedure and experimentally obtained parameters as follows: Step 1: Initialize population: Randomly generate the initial population I1 , · · · , IN . Step 2: Compute fitness: Compute the fitness f1 , · · · , fN for each individual I1 , · · · , IN . Step 3: If a terminal condition is satisfied, end the procedure. The terminal condition is defined in Section 4.2.5. Step 4: Selection: Using f1 , · · · , fN , select a child population C from the population. Step 5: Crossover: Select pairs randomly from C with probability Pcross . Generate two children by applying a crossover operator to each pair, and exchange the children with the pairs in C.

76

S. Yamada and D. Katagami

Step 6: Mutation: Mutate the individuals in C according to the mutation rate Pmut . Step 7: Go to Step 2. • • • • •

Population size: 50. Crossover operator: Uniform crossover. Selection method: Elite strategy and tournament selection (the tournament size = 2). Crossover rate Pcross: 0.8. Mutation rate Pmut : 0.05.

Since we deal with deterministic, not probabilistic, action selection, the behavior is a mapping from a single state to a single action. Thus we use the coding in which one of actions {A1, · · · , A4} is assigned to each state. The genotype is shown in Fig. 4.5. S1

S2

S3

S4

A4 A1 A3 A2

....

S 71 S 72

A4 A2

Fig. 4.5. A coded behavior

4.2.4.2 Defining a Fitness Function Since fitness is a very important for GA, we have to carefully define the fitness function for AEM. We consider three conditions on suitable behaviors for AEM: termination of actions, accuracy of recognition, and efficiency of recognition. Once defined, the fitness functions for each condition are then integrated. Termination of actions A mobile robot needs to stop its actions by itself. Otherwise it may act forever in an environment and no action sequence would be obtained. Thus, the termination of actions is the most important condition. We use homing as the means of terminating actions: the robot stops when it returns to the neighborhood of the start point. Homing makes the length of an obtained action sequence depend on the size of the environment. In contrast, terminating actions after a fixed number of actions does not have this advantage. Termination is evaluated with the following function g. g=

(Number of E-trials) + (Number of H-trials) 2 × (Total number of trials)

(4.1)

where E-trials and H-trials means trials in which a robot escaped from the neighborhood of the start point and trials in which it succeeded in homing. Hence, g evaluates escape as well as homing. Its range is [0, 1], and it returns 1 when the robot succeeds in homing in all of its environments.

4 Evolutionary Robotics: From Simulation-Based Behavior Learning

77

Recognition accuracy Another important criterion is the accuracy of identifying test environments. Accuracy is evaluated with the following function h. Its range is [0, 1], and h = 0 when g = 1. h=

Number of successful test environments Total number of test environments

(4.2)

Recognition efficiency In AEM, a robot acts by operating physical actuators for recognizing an environment, and the actions have a significant cost. Hence, the actions should be as small as possible for efficiency. To enable this, we use the following evaluation function k. k = 1−

n S Σi=1 i n × Smax

(4.3)

where Si is the size of an action sequence obtained in the ith environment i, Smax is the given size of an action sequence, and k = 0 when h = 1. The function has a range of [0,1] and a higher value means more efficiency. The obstacle avoidance is implicitly evaluated by the function k because a collision increases the length of an action sequence.

Fig. 4.6. Whole procedure

78

S. Yamada and D. Katagami

We integrated these three fitness functions into one function f = g + h + k having the range of [0,3]. Since the function h (or k) takes 0 when the value of g (or h) is less than 1, the function f is phased: the actions are terminated when 1≤ f , the recognition is completely correct when 2 ≤ f , and the recognition efficiency is higher when 2 ≤ f < 3. There were an another way to optimize the fitness function f , in which we assigned weights to three terms like f = α g + β h+ γ k and optimize it by applying multi-objective optimization by GA[8]. However, in such a case, the interpretation of the dominant solutions will be very hard. Fig. 4.6 outlines the whole procedure of evolutionary design. It integrates a simple GA procedure with a Khepera simulation for evaluating an individualsf fitness. Note that the fitness is computed from the simulated behaviors. 4.2.5

Simulation Experiments

We implemented our behavior design method on a Khepera simulator [21]. The parameters were as follows: the whole environment was a square (100cm × 100cm), the neighborhood of the start point for homing was a circle with a 100 mm radius, and the action sequence was limited to 2000 actions. This 2000 actions limitation was set by preliminary experiments. In the simulator, the motor had ±10% random noise in velocity, and the robot’s body had ±5% random noise in rotation. Furthermore, each infrared proximity sensor had 10% random noise added to the distance and ±5% random noise added to the light strength. The added noise made the simulator similar to a real environment. If a robot couldn’t return home within a limited number of actions, the trial resulted in zero fitness. The terminal condition of GA was that the maximum fitness of the population exceeds 2 or the generation number reaches 100. The termination and accuracy conditions satisfied when the fitness exceeds 2. The robot was placed in each of the training environments once. The robot acted in the environments, and the environment vectors (transformed from the action sequences) were stored as instances. Next, the training environments were given to the robot as test environments, and the robot tried to identify them. The start points and directions were fixed (bottom, center, and left). As mention above, although the test environments were the same as the training environments, the action sequences were different because of the added random noise. Hence the simulated robot had to learn behaviors that would be robust to such noise. There were 10 different trials in the initial population, and we computed the averages and standard deviations of the fitness for the generation number at which GA stopped. We employed a PC with a Pentium Pro (200 MHz) and 128 Mbytes of RAM and programmed the system in the C programming language on Linux and FWM window manager. By using this computational resource, each learning needed 3 minutes ∼ 5 hours for convergence. 4.2.5.1 Exp-1: Different Contours First, we tested environments with twelve different contours and a part of the environments is shown as emp, L, L2, iL, s-emp in Fig. 4.7. The experimental results are shown

4 Evolutionary Robotics: From Simulation-Based Behavior Learning

(a) emp

(b) L

(c) L2

(d) iL

(e) s-emp Fig. 4.7. A part of environments and trajectories in Exp-1

79

80

S. Yamada and D. Katagami

Table 4.1. Experimental results in Exp-1

Env. Training env. GN MaxF E1 emp, L 1.0 (0) 2.80 (0.106) E2 E1 + L2 2.6 (1.84) 2.43 (0.131) E3 E2 + iL 2.8 (1.69) 2.44 (0.025) E4 E3 + s-emp 2.8 (1.75) 2.44 (0.093) E5 10 env. 2.1 (0.738) 2.51 (0.024) E6 12 env. 5.2 (3.08) 2.48 (0.07)

Fig. 4.8. Fitness for 12 environments

in Table 4.1. Four subsets (E1, E2, E3 and E4 in Table 4.1) of the five environments were given to a robot. Additionally, 10 and 12 different contour environments were used (E5 and E6 in Table 4.1). The “GN” is the generation number in which GA stopped, and “MaxF” means the maximum fitness value at GN. The numbers in GN and MaxF stand for averages, and the numbers in parentheses are standard deviations. This format is common in all the experimental results. Fig. 4.7 also indicates the trajectories of the best individuals at GN in E4. Fig. 4.8 shows the maximum values and average values of fitness at every generation for E6 of Exp-1. The average and maximum fitness increased monotonously. In such simple environments, suitable behaviors for AEM were obtained within a few generations. The standard deviation was large in GN and small in MaxF, and this tendency was observed throughout all the experiments. Seeing from Fig. 4.7, different action sequences were obtained depending on the contours of the environments.

4 Evolutionary Robotics: From Simulation-Based Behavior Learning

(a) emp

(b) 1-la

(c) 2-la

(d) 3-la

(e) 4-la Fig. 4.9. Environments and trajectories in Exp-2A

81

82

S. Yamada and D. Katagami

(a) emp

(b) 1-la

(c) L

(d) L1-la

(e) iT

(f) iT1-la

Fig. 4.10. Environments and trajectories in Exp-3

4 Evolutionary Robotics: From Simulation-Based Behavior Learning

83

Table 4.2. Experimental results in Exp-2A

Env. Training env. GN MaxF E7 emp, 1-la 1.6 (0.966) 2.68 (0.196) E8 E7 + 2-la 4.8 (2.78) 2.59 (0.131) E9 E8 + 3-la 9.3 (5.19) 2.59 (0.111) E10 E9 + 4-la 10.0 (5.94) 2.62 (0.075)

4.2.5.2 Exp-2: Different Lights and Shape Next, we made five different environments, emp, 1-la, 2-la, 3-la, 4-la in Fig. 4.9, by varying the lighting in terms of number of lamps and their position. The light was strong enough that the robot could detect the light direction anywhere in the environment. The experimental results are listed in Table 4.2. Fig. 4.9 also indicates the trajectories of the best individual at GN in E10. The black circles in the figures indicate the lamp positions. Although the GN values exceeded those in Exp-1, suitable behaviors were obtained. Note that we cannot intuitively understand the behaviors in Fig. 4.9. This means that it is very hard for human to design such behaviors by hand-coding and that our automatic design method is effective. Next, we created six environments, emp, 1-la, L, L1-la, iT, iT1-la in Fig. 4.10 by adding different numbers of lights to an environment. Each environment had different classes and all of them should be distinguished. As a result, GN was 13.2 (7.67) and MaxF was 2.61 (0.091). Fig. 4.10 also shows the trajectories of the best individual at GN. The actions in the different environments differed significantly, and hence manual design seems to be difficult. 4.2.5.3 Exp-3: Classes Including More Than One Training Environment In the environments of Exp-1 and Exp-2, each had different classes. However, in this experiment, plural environments are included in a single class. Generalization is necessary to recognize such environments. We assigned three classes to six environments in Fig. 4.9: {emp, 1-la}, {L, L1-la}, {iT, iT1-la}, and made experiments Exp-3. These classes meant the three environments with different contours, in which the light was turned on or off. As a result, GN was 8.8 (4.24) and the maximum fitness was 2.61 (0.091). This showed that our approach was valid for induction from several instances of a class. 4.2.6

Discussion

4.2.6.1 Comparison with Hand-Coded Behaviors It is very difficult for a human to design the behaviors of the experiments described above. The search spaces were huge and there were few heuristics for efficiency. Note also that the obtained behaviors were different from manually designed ones for wall-following, random walking, etc.

84

S. Yamada and D. Katagami

Fig. 4.11. Comparison with wall-following

We compared the learned behaviors with wall-following. Fig. 4.11 plots the average lengths of action sequences for test environments in Exp-1 (Table 4.1). It shows that the automatically learned behaviors are more efficient than wall-following in all environment sets. Thus our automatic design approach designed more efficient behaviors than typical hand-coded behaviors. We repeated Exp-1 by adding 1% random noise to the initial position and orientation of the robot. Although the noise was very small, it complicated the experiments to such an extent that adequate were only acquired in E1, E2, E3. This difficulty has yes to be resolved[27]. 4.2.6.2 Difficulty of Behavior Acquisition As E1 in Table 4.1 indicates, suitable behaviors are already included in the initial population. This implies that although the search space is huge, a large number of suitable behaviors exist for E1. To verify this expectation, we investigated behaviors of individuals with maximum fitness in 10 different trials. The behaviors could be categorized into the five classes shown in Table 4.3. The numbers in parentheses indicate the number of categorized behaviors in the class, and the actions were described earlier. The intuitive explanations for the states are as follows: • • • •

State S0: No obstacle and light around a robot. State S9: Obstacle to the left and no light. State S18: Obstacle in front and no light. State S36: Obstacle to the right and no light.

As Table 4.3 shows, the 10 individual trials executed only the subsets of {S0, S9, S18, S36}. Thus, the actual combinations of states is 44 = 256. Also, BS-1 and BS-2, each had four suitable behaviors because BS-1 did not care about the action for S9 and there were four actions. BS-2 was similar. BS-3, BS-4, BS-5, each had a single suitable

4 Evolutionary Robotics: From Simulation-Based Behavior Learning

85

Table 4.3. Executed behaviors

behaviors behaviors BS-1 (5) S0 → A1 BS-4 (1) S0 → A1 S18 → A2 S9 → A1 S36 → A2 S18 → A2 BS-2 (1) S0 → A1 S36 → A4 S9 → A3 BS-5 (1) S0 → A1 S18 → A3 S9 → A4 BS-3 (2) 0 → A1 S18 → A3 S9 → A3 S36 → A2 S18 → A2 S36 → A4

behavior. Consequently, the total number of suitable behaviors was 11. Since the population size was 50, the probability that any suitable behavior exists in the initial popula50 tion is 1 − ( 256−11 256 ) = 0.89. This supports the fact that suitable behaviors exist in the initial population. Furthermore, we found that the average number of suitable behavior classes was 3.4∼18.4. Hence we found that a robot actually experienced only a few of the 72 states.

4.3 Directly Teaching for Robots in Real Environments An autonomous robot must accomplish tasks in environments where human are present, but it’s difficult to prepare for some situations. The situations, for example, could be dynamic or involve unexpected interactions, and they would require autonomous behavior acquisition and adaptation to environments. Recently, reinforcement learning and evolutionary computation were used as a framework of learning and adaptation. Moreover, research that enables a robot get a controller autonomously has attracted attention. Needless designer bias should be eliminated from the interaction dynamics, with the robot’s embodiment and environment reflected in construction of a controller. To eliminate bias, an agent can learn by trial and error without using prior knowledge in the framework of reinforcement learning. However, the slowness of trial and error learning is a problem in a real environment. Interaction with humans in an environment could be helpful. Particularly for the robots that do not have a priori knowledge or use trial and error in the initial stage, human instruction is a very effective way of acquiring autonomous behavior. However, once it has reached a certain level, an autonomous robot does not have to follow instructions from humans all the time. At this stage, the robot should demonstrate its autonomy based on the instruction rules it stored through interactions with humans. Therefore, we need a technique of establishing a robot’s autonomy through interaction with humans. The approaches mentioned have a design stage in which the designer ignores possible inappropriate bias. Below, we discuss the design of interactions between humans

86

S. Yamada and D. Katagami

and robots, especially interactive design human teachers, as a design logic. Interactive design should account for the teacher’s load and the gap between of environmental recognition of the teacher and a learner. 4.3.1

Supervised Learning and Interactive Evolutionary Computation

Interactive evolutionary computation (IEC) is a popular way of exploiting human teachers. IEC directly incorporates human evaluations and searches in the evolutionary way. It has been used in various fields, e.g. design, music, and medicine [29]. IEC’s main problem is fatigue; the human teacher has to cooperate with a tireless computer, and such a process cannot be continued for many generations. Lund et. al. proposed an evolutionary robotics approach using IEC for constructing a controller for LEGO robots [19]. They reduced the construction time by merging an interactive genetic algorithm with reinforcement learning and employing neural networks as the robot controller. The IEC realized a programming-less program for children’s games and toys. Their experiment was surprising because only nine moving robots was all it took to generate neural networks for controlling a robot to do interesting movements after only five to seven generations. Generally, in interactive evolutionary learning, the more that is taught, the better the performance is. However, human labor is not unlimited. Evaluating many individuals (or rules) and inputting evaluations for every generation are extremely tiresome tasks in practice. Crandall et al. devised a theoretical approach to achieving interaction efficiency in designing human-robot coordination systems that considered cognitive fatigue [5]. By measuring hours worked, number of answers, and accuracy rate in a paired task, they showed that cognitive fatigue from a direct control system is higher than from a a shared-control system. We consider the following ways to apply their insights on interactive learning to a real robot environment. If a robot does not have an order from a teacher, then it tries to learn automatically. If a teacher intuitively teaches a robot through an input device (like a joystick) with arbitrary timing, the robot makes rules automatically. Fatigue might be reduced in this manner. Horiguchi et al. [12] used the concept of mutual leadership pattern interaction to design human-robot interaction and realized cooperative automation between a mobile robot and a human through power feedback. However, the result of learning had no effect on the behavior acquisition of the robot. 4.3.2

Teacher View and Learner View

There are big differences between a robot’s recognition and a human teacher’s one. Mishima and Asada et al. have proved that the efficiency of learning gets worse by passive teaching for the gap (cross perceptual aliasing) of the environmental recognition between a teacher and a learner [22]. Yet in terms of efficiency, passive teaching is considered to be a good method even though the teacher has to monitor the robot until it needs assistance.

4 Evolutionary Robotics: From Simulation-Based Behavior Learning

87

We dealt with the gap between a teacher and a learner as a design problem[14]. We devised a faster learning system by using a classifier system that allows a teacher teach the appropriate action by directly teaching with a joystick controller. In the experiments, we could select two different view to teach a robot: teacher’s view (TV) and learner’s view (LV). By using TV, a teacher teaches a robot in the view point of a bird, and see a whole environment. In contrast, using LV, a teacher can see just the local scene which a robot actually sees in the environment. We were interested in gap between TV and LV, and their influence to interactive robot learning like interactive classifier system. This issue was experimentally investigated in Section 4.3.4. 4.3.3

Interactive Classifier System

In order to realize this evolutionary robotics by IEC, we developed a robot learning system, called an interactive classifier system (ICS). Fig.4.12 shows the teaching environment that we envision. Including the interactive function of IEC in a learning classifier system (LCS), the ICS is a learning model for a robot that can be instructed by a human or can autonomously learn. XCS [30] is a LCS algorithm that Wilson proposed. XCS adds a parameter called accuracy to ZCS [31]. Although the classifier was generalized by including “don’t care” symbols to the conditional part, the classifier system was not able to utilize it effectively. This problem originates in that LCS does not have a mechanism to advance generalization appropriately, and a phenomenon in which an over-generalized classifier degrades performance has been reported [4]. In XCS, both intensity and accuracy are used to check the validity of an over-generalized classifier. This accuracy is calculated according to the error of the reward received as a result of classification and a predicted value. It is reported that a rule can be appropriately generalized in this way [30]. Fig.4.13 shows an outline of the interactive classifier system.

Operator

Actual Environment

Monitor

Robot

Joystick Sensor Information

Teaching ICS

Interactive Classifire System

Fig. 4.12. Teaching environment

Action

88

S. Yamada and D. Katagami

Environment operator joystick

robot

monitor

sensor information

"left"

ICS SPC

DC 01

[P] p 43 32 14 27 18 24

#011:01 11##:00 #0##:11 001#:01 #0#1:11 1#01:10

e .01 .13 .05 .24 .02 .17

F 99 9 52 3 92 15

Match Set

Prediction Array

Action Set

[M] #011:01 #0##:11 001#:01 #0#1:11

RGC

43 14 27 18

.01 99 .05 52 .24 3 .02 92

nil 42.5 nil 16.6

#011:01 001#:01

43 .01 99 27 .24 3

max discount

GA RC

+

delay=1

Previous Action Set [A] -1

Fig. 4.13. Overview of interactive classifier system

ICS consists of a rule generation component (RGC), a sensor processing component (SPC), a display component (DC) and a reinforcement component (RC). All of them are developed on Linux and programmed by the C language and GTK+. Each module is explained as follows. RGC creates the rule by instruction. A teacher operates a robot by using an input equipment and looking at the robot’s information displayed on a GUI. SPC receives the operation history of a there, and the sensor information of the robot at that time. RGC creates a rule newly from it, and it adds to a rule list. The creation procedure of a rule was improved so that a rule could be created from instruction information (the action to which an operator operated the robot) from a human teacher on the basis of XCS [30]. 1. ICS receives robot’s sensor information X and instruction information from SPC. 2. Some classifiers that matched X are moved from a population [P] to a match set [M]. ICS turns regularly the prediction value of classifier which supports each act ai in [M] with a fitness value, and creates P(ai ). The value of P(ai ) is put on prediction array, and the act of classifier chosen by P(ai ) is chosen by action selection

4 Evolutionary Robotics: From Simulation-Based Behavior Learning

89

methods. The action selection methods are performed by a deterministic selection or a roulette wheel selection. 3. If a j = at to compare act a j chosen by act selection methods and act at obtained by teaching, the action part of the rule which has a j in an action part in [M] will be rewritten to at . A change will not be made if a j = at . 4. The action set [A] which consists of classifiers in [M] which support selected action a j is created. When an action a j or at is sent to an actuator, and in case of at , reward rteach is given immediately. When there is no input of at , remuneration rimm is returned from an environment. RC is a reinforcement learning module in a classifier system. It learns by updating the parameter of classifiers chosen last time step. When there is no operation of a teacher, a robot can act autonomously with the rule created thus far. DC indicates the data processed by SPC. GTK+ is used for development of an interface. SPC performs processing of a robot’s various sensors and processing of teaching information. The information is sent to DC and RGC and the processed data is displayed and ICS creates classifiers from them. 4.3.4

Robot Experiments in the Real World

We conducted experiments in a real world to investigate difference in two teaching methods: TV and LV as mentioned in Section 4.3.2. Fig. 4.14 shows an experimental environment. As an experimental task, ICS reduced the number of steps from any start points to a light source which set up as a goal in a field surrounded with white plastic plates. We compared two teaching methods (TV and LV) with a traditional method in which a robot autonomously learns by simple EC. To investigate influence by gap between operator’s recognition and robot’s one, we prepared experiment-1 without an obstacle and experiment-2 with an obstacle. We considered 40 steps as a trial and began

Fig. 4.14. Experimental environment

90

S. Yamada and D. Katagami

teaching or autonomous exploration for five trials at random start points, and tested a trial at each of five start points as examination of learned rules. It performed this procedure six times, and consequently we had 30 trials for evaluation. 4.3.4.1 Experimental Settings The experiments were made with a standard miniature mobile robot Khepera (Fig.4.2). The mobile robot has cylindrical shape, a diameter of 6 cm and a height of 5 cm. It possesses two motors and on-board power supply. The motors can be independently controlled by a PID controller. It is equipped with eight infrared proximity sensors. The eight infrared sensors are distributed around the robot in a circular pattern. They emit infrared light, receive the reflected light and measure distances in a short range: 2-5 cm. The robot is also equipped with a Motorola 68331 micro-controller which can be connected to a computer via serial cable. Moreover, the system utilizes SONY analog controller DUALSHOCK as a joystick. Task goal was minimization of the number of steps which a robot needed to reach the goal from arbitrary initial locations in the region surrounded by white plastic board. We compared TV, LV and a learning method by using classifier system. An encoding of a classifier was as follows. A classifier was the 20-bits string “#000#100000###100#:01”. The robot’s condition was the left 16-bits string, it represented eight infrared proximity sensors (the left 8-bits) and light sensors (the right 8-bits) around the robot in a circular pattern. The bit was “1” if an input sensor value was larger than a threshold, else “0”. “#” is a “don’t care” symbol. The robot’s previous action was the next 2-bits string, represented as forward “11”, left-turn “01”, right-turn “10” and back “00”. The robot’s current action was a 2-bits string similarly. A fitness function of ICS was defined as follows. Reward F was computed by the sum of eight light sensors through a sigmoid function. 7

u = ∑ lighti i=0

PaymentRange , lightMAX × 8

F=

1 1 + exp(−u)

(4.4)

In the case of TV, the operator could recognize a small obstacles as the robot could perceive although he/she looked a whole environment. To implement TV, an operator taught a robot using a camera by which he/she could look a whole environment. Fig.4.15 shows information through GUI by a camera. In the case of LV, an operator could not look a whole environment although it could recognize directly a small obstacles and a recognition error which the robot perceived. To implement LV, ICS used GUI indicating sensor values. Since an operator hardly recognized an environment by reading only numerical values of sensor data, we developed GUI interface which can represent sensor values graphically. Fig.4.16 shows such sensor information through GUI interface. We considered 40 steps as a trial and began teaching or autonomous exploration for five trials from random start points, and tested a trial at each of five start points as examination of created rules by the teachings or the exploration. It performed this procedure seven times, and consequently we had 35 trials as examination. Table 4.4 shows experimental parameters.

4 Evolutionary Robotics: From Simulation-Based Behavior Learning

91

Fig. 4.15. External Observation Settings: A teacher acquires environmental information by bird’seye view. The teaching is performed from the viewpoint of whole environment.

Fig. 4.16. Internal Observation Settings: A teacher acquired a robot’s sensor information and used it for teaching. The teaching was performed through interface which displayed robot’s sensor information.

92

S. Yamada and D. Katagami Table 4.4. Experimental Parameters

Parameters The number of problems in one experiment The number of experiments The maximum size of the population The probability to do crossover The probability of mutating one bit

Value 35 1 300 0.8 0.04

4.3.4.2 Experimental Results In the experiment-1, we examined the number of reach times to a light source from five start points. Although LV began to success the task at early stage, it didn’t success later. On the other hand, TV improved robot’s learning and achieved the task from all five start points before LV. TV improved a robot’s learning in the simple environment in which cognitive difference is little because an operator looked a whole environment. However, the difference is not so large. In the experiment-2, we introduced an obstacle to the environment for making a task more difficult. The obstacle was made of the transparent plastic board because so that a robot might perceive a direction of a light source. A robot must reach to the light source as avoiding the obstacle. We compared a teaching method by TV with LV in the same way as experiment-1. We examined the number of steps to a light source. Since ICS hardly used information of proximity sensors and acquired effective rules by using only information of light sensors in the experiment-1, the difference of cognition between an operator and a robot was little. For this reason, the difference of two teaching methods was not so large. ICS hardly learns from teaching of an operator in experiment-2 because there is the difference between the situation of the robot which an operator estimated by observation from the outside and the one of the robot in the real world. LV actually created effective rules because an operator performed teaching as he/she checked robot’s internal status. After we conducted experiments for TV and LV each twenty trials, we tested a trial at each of five start points as examination using each learned rules. Table 4.5 shows the experimental results. The robot could not reach to a light source at any start points of five in the examination of TV. On the other hand, the robot reached to a light source by six steps and nine steps respectively at two of five start points in LV. We can see that ICS can create effective rules independently of start points since LV improves the robot’s learning by teaching. Fig.4.17, 4.18 showed typical teaching behaviors of TV and LV respectively. In the case of TV, a teacher acquired locations of obstacles in advance, but could not the Table 4.5. Results of experiment-2

Trial 1 2 3 4 5 TV - - - - LV 6 9 - - -

4 Evolutionary Robotics: From Simulation-Based Behavior Learning

93

Fig. 4.17. TV example of teaching

Fig. 4.18. LV example of teaching

changes of a robot’s sensor information because a teacher can observe a whole environment. On the other hand, a teacher observed the changes of internal situation in LV because a teacher could see robot’s information by using a sensor panel. 4.3.5

Teaching Efficiency

Table 4.6 shows ten rules which have best values of the system prediction in created ones after twenty trials with LV. ICS created effective and commonsense rules like as the robot moves forward when the light source faces in front of the robot and it

94

S. Yamada and D. Katagami

Table 4.6. Created rules by LV method

Condition Action Prediction Teaching 0#0#0#####0#10##1# 10 413.7 yes 0010###0##000#1### 01 364.0 yes 00###0#000#1#0#01# 11 292.0 yes ###000000101001011 11 256.0 yes

moves right when the light source faces right. In this table, “Prediction” is a fitness of a classifier and “Teaching” is whether it is taught by human or its offspring. Its number is the number of which a taught classifier alternate generations. We see that teaching was very effective because all the rules whose the system prediction is high created from teaching. Although an operator did not teach a robot to go back in the both experiments, the robot went back for avoiding an obstacle when the robot collided with a wall and reached to a light source. Because ICS created these rules by which a robot worked in cooperation with human.

4.4 Conclusion We reported evolutionary acquisition of suitable behaviors to Action-based Environment Modeling. GA was applied to search the behaviors, and the simulated mobile robots were used as individuals. States and actions were described for coding chromosomes, and we carefully defined a fitness function. We conducted various experiments using environments with different contour, lights and both of them, and found that our approach was promising to learn suitable behaviors for AEM. Furthermore some of learned behaviors were more efficient than hand-coded ones. Next, we proposed a novel interactive method from the viewpoint of observation and investigated its effects in a real world experiments. The direct teaching method to train a mobile robot in a real environment. It was found that internal observation increased the effects according to arise the difference between the robot’s recognition and operator’s one in a more complicated environment.

References 1. Arkin, E.M., Chew, L.P., Huttenlocher, D.P., Kedem, K., Mitchell, J.S.B.: An efficiently computable metric for comparing polygonal shapes. IEEE Transaction on Pattern Analysis Machine Intelligence 13(3), 209–216 (1991) 2. Brooks, R.A.: A robust layered control system for a mobile robot. IEEE Transaction on Robotics Automation 2(1), 14–23 (1986) 3. Cho, S.B., Shimohara, K.: Emergence of structure and function in evolutionary modular neural networks. In: Proceedings of the Fourth European Conference on Artificial Life, pp. 197–204 (1997) 4. Cliff, D., Ross, S.: Adding temporary memory to ZCS. Adaptive Behavior 3(2), 101–150 (1994)

4 Evolutionary Robotics: From Simulation-Based Behavior Learning

95

5. Crandall, J.W., Goodrich, M.A.: Evolutionary robotics - a children’s game. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 1290–1295 (2002) 6. Crowly, J.L.: Navigation of an intelligent mobile robot. IEEE Transaction on Robotics Automation 1(1), 34–41 (1985) 7. Dasarathy, B.V.: Nearest Neighbor (NN) Norms: NN Pattern Classification Techniques. IEEE Computer Society Press, Los Alamitos (1991) 8. Deb, K.: Multi-Objective Optimization Using Evolutionary Algorithms. John Wiley and Sons, Chichester (2001) 9. Floreano, D., Mondada, F.: Evolution of homing navigation in a real mobile robot. IEEE Trans. Systems, Man, Cybernetics. Part B: Cybernetics 26(3), 396–407 (1996) 10. Harvey, I., Husbands, P., Cliff, D.: Issues in evolutionary robotics. In: Proceedings of the Second International Conference on Simulation of Adaptive Behavior, pp. 364–373 (1992) 11. Holland, J.: Adaptation in Natural and Artificial Systems. University of Michigan Press (1975) 12. Horiguchi, Y., Sawaragi, T., Akashi, G.: Naturalistic human-robot collaboration based upon mixed-initiative interactions in teleoperating environment. In: IEEE International Conference on Systems, Man, and Cybernetics, pp. 876–881 (2000) 13. Ito, T., Iba, H., Kimura, M.: Robustness of robot programs generated by genetic programming. In: Genetic Programming 1996, Proceedings of the First Annual Conference, pp. 321– 326 (1996) 14. Katagami, D., Yamada, S.: Interactive classifier system for real robot learning. In: IEEE International Workshop on Robot and Human Interaction, pp. 258–263 (2000) 15. Kohonen, T.: The self-organizing map. Proceedings of the IEEE, 1464–1480 (1990) 16. Koza, J.R.: Evolution of subsumption using genetic programming. In: Proceedings of the First European Conference on Artificial Life, pp. 110–119 (1991) 17. Koza, J.R.: Genetic Programming. MIT Press, Cambridge (1992) 18. Lund, H.H.: Adaptiverobotics in entertainment. Applied Soft Computing 1(1), 3–20 (2001) 19. Lund, H.H., Miglino, O., Pagliarini, L., Billard, A., Ijspeert, A.: Evolutionary robotics - a children’s game. In: Characterizing Efficiency of Human Robot Interaction: A Case Study of Shared-Control Teleoperation, pp. 148–156 (1998) 20. Mataric, M.J.: Integration of representation into goal-driven behavior-based robot. IEEE Transaction on Robotics Automation 8(3), 14–23 (1992) 21. Michel, O.: Khepera simulator v.2 user manual. University of Nice-Sophia (1996) 22. Mishima, C., Asada, M.: Active learning from cross perceptual aliasing caused by direct teaching. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 1420–1425 (1999) 23. Nakamura, T., Takamura, S., Asada, M.: Behavior-based map representation for a sonarbased mobile robot by statistical methods. In: Proceedings of the 1996 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 276–283 (1996) 24. Nehmzow, U., Smithers, T.: Map-building using self-organizing networks in really useful robots. In: Proceedings of the 1st International Conference on Simulation of Adaptive Behavior, pp. 152–159 (1991) 25. Nordin, P., Banzhaf, W.: A genetic programming system learn-ing obstacle avoiding behavior and controlling a miniature robot in realtime. Technical report, Department of Computer Science, University of Dortmund (1995) 26. Olton, D.S.: Spatial memory. Scientific American 236(6), 82–99 (1977) 27. Reynolds, C.W.: Evolution of obstacle avoidance behavior: using noise to promote robust solutions. In: Kinnear, J.K.E. (ed.) Advances in Genetic Programming, vol. 1, pp. 221–241. MIT Press, Cambridge (1994)

96

S. Yamada and D. Katagami

28. Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (1998) 29. Takagi, H.: Interactive evolutinary computation: Fusion of the capabilities of ec optimization and human evaluation. Proceedings of the IEEE 89(9), 1275–1296 (2001) 30. Wilson, S.W.: Classifier fitness based on accuracy. Evolutionary Computation 3(2), 149–175 (1995) 31. Wilson, S.W.: ZCS: a zeroth order classifier system. Evolutionary Computation 2(1), 1–18 (1995) 32. Yamada, S.: Evolutionary design of behaviors for action-based environment modeling by a mobile robot. In: Proceedings of Genetic and Evolutionary Computation Conference, pp. 957–964 (2000) 33. Yamada, S.: Recognizing environments from action sequences using self-organizing maps. Appllied Soft Computing 4(1), 35–47 (2004) 34. Yamada, S.: Evolutionary behavior learning for action-based environment modeling by a mobile robot. Applied Soft Computing 5(2), 245–257 (2005) 35. Yamada, S., Murota, M.: Unsupervised learning to recognize environments from behavior sequences in a mobile robot. In: Proceedings of the 1998 IEEE International Conference on Robotics and Automation, pp. 1871–1876 (1998)

5 Ranked Pareto Particle Swarm Optimization for Mobile Robot Motion Planning D. Wang, N.M. Kwok, D.K. Liu, and Q.P. Ha ARC Centre of Excellence for Autonomous Systems, Faculty of Engineering, University of Technology, Sydney, Broadway, NSW 2007, Australia {Da-Long.Wang,nmkwok,dkliu,quangha}@eng.uts.edu.au

Abstract. The Force Field (F 2 ) method is a novel approach for multi-robot motion planning and coordination. The setting of parameters in the (F 2 ) method, noticeably, can affect its performance. In this research, we present the Ranked Pareto Particle Swarm Optimization (RPPSO) approach as an extension of the basic idea of Particle Swarm Optimization (PSO), which makes it capable of solving multiobjective optimization problems efficiently. In the RPPSO, particles are initiated randomly in the search space; these particles are then evaluated for their qualities with regard to all objectives. Those particles with highly-ranked qualities have preferences to enter the set of Global Best vectors, which stores many currently best solutions found by particles. Thus, particles in RPPSO will search towards many possible directions and the diversity among solutions is well preserved. Ideally, a set of optimal solutions will be found when the termination criterion is met. The effectiveness of the proposed RPPSO is verified in simulation studies. Satisfactory results are obtained for multiobjective optimization problems of multi-robot motion planning in challenging environments with obstacles.

5.1 Introduction An existing trend in robot applications is the utilization of a team of robots to accomplish complex tasks. Examples may include the deployment of autonomous unmanned vehicles in container terminals and material handling, the exploration of unknown terrain and cooperation in robot soccer, etc. For a team of robots operating in a common work space, motion planning and coordination for multiple robots has arisen as a very important issue to be addressed. Motion planning is one of the fundamental problems in robotics. It can be described as the construction of collision free trajectories that direct robots to their destinations. Researchers have developed a variety of algorithms on robot motion planning [1, 2]. Although many algorithms have been proven to be feasible and efficient for single robot motion planning and collision avoidance, they cannot be transferred directly to multi-robot systems. The existing methods can be broadly categorized into centralized and decentralized approaches. Centralized approaches consider all robots together as if they were forming a high degree-of-freedom system and capable of providing complete and optimal solutions. However, as the number of robots and obstacles in the working environment increases, such approaches may suffer from the exponentially increasing computational complexity. Decentralized approaches generate independent paths for D. Liu, L. Wang, and K.C. Tan (Eds.): Design and Control of Intel. Robotic Systems, SCI 177, pp. 97–118. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com 

98

D. Wang et al.

individual robots and avoid collisions locally. Decentralized approaches are not affected by the number of robots but are usually incomplete, i.e., they may fail to find a solution even if it exists. The Potential Field method has been a popular approach for autonomous mobile robot and manipulator path planning in the past decades because of its mathematical simplicity and computational efficiency [3]. The basic concept of this method is to fill a working environment with an artificial potential field in which the robot is attracted by the goal and repulsed away by obstacles. Researchers have developed a variety of methods based on the potential field concept. For example, Connolly presented a method using Laplace’s Equation to avoid local minima [4]. Ge and Cui developed a potential field method which defines attractive and repulsive potentials by taking account of the relative position and velocity of a robot with respect to obstacles and targets [5]. A potential field model using generalized sigmoid functions is proposed in [6] to meet the requirement of accurate representation of complex objects in applying the artificial potential field in some practical applications. The F 2 (Force Field) method is a decentralized method for multi-robot path planning and collaboration [7]. Instead of generating potential fields or force fields based on the locations of obstacles, a virtual force field is constructed for each robot according to its status, including traveling speed, dimension, task priority, location and environmental factors, etc. The force field of a robot is different from those of other robots due to its different status and varies with the robot during its movement. A robot with larger volume, traveling with higher speed, or with higher task priority than other robots will have priority in collision avoidance. The interactions among the robots’ force fields and obstacles provide a feasible and efficient way for multi-robot motion planning and collaboration. The concept of F 2 is a further refinement of the Potential Field to some extent. Both concepts use the repulsive potential/force field to avoid collision with obstacles and the attractive potential/force to guide a robot to its target. But the differences between the F 2 and Potential Field are distinct. The Potential Field is generated based on environment information. That is to say, the potential value of a point in a potential field is determined by its location with respect to the goal and obstacles nearby. The potential field remains unchanged if environment does not change. In the F 2 method, the repulsive force field of a robot is generated around the robot body, instead of around obstacles as those of potential field based approaches. This force field is continuously changing during this robot’s movement. Collision avoidance is achieved by interaction of this protective field with the environment. This research builds upon the F 2 method proposed in our previous work [7], where the construction of force fields uses a number of parameters. The setting of these parameters can noticeably affect the method’s performance. Since the problem is highly coupled, non-linear and incomplete, closed-form solutions are not always available. To this end, an efficient and implement simple algorithm, such as the Particle Swarm Optimization (PSO) algorithm, is very attractive. A single objective optimization method was presented in our previous work [8], in which PSO is utilized to find suitable parameters for F 2 method. But the PSO-tuned F 2 method only deals with the optimization of a single objective. PSO has been proposed for multi-objective optimization in

5 Ranked Pareto Particle Swarm Optimization

99

tuning control parameters of the virtual robot tracking structure for robot formation [9]. In this research, we present a multiobjective optimization method - Ranked Pareto Particle Swarm Optimization (RPPSO) for motion planning of multiple mobile robots taking into account collision avoidance between robots and obstacles as well as minimum travel lengths. The RPPSO follows the basic idea of PSO but is capable of solving multiobjective optimization problems efficiently. At the beginning of RPPSO, particles are initiated randomly in the search space. These particles are then ranked by their qualities with regard to all objectives. Those particles with good qualities constitute the Global Best, which stores many current best solutions found by particles. In the same time, each particle maintains a memory of its previous best position, denoted by Particle Best. Particles then update their locations based on Global best and Particle best. Thus particles in RPPSO will search towards many possible directions. Ideally, a set of optimal solutions will be found when the termination criterion is met. The rest of the chapter is organized as follows. The proposed RPPSO method for multiobjective optimization is presented in Section 5.2. In Section 5.3, the F 2 method used for robot coordination is discussed. The effectiveness of RPPSO is evaluated through simulations on motion planning problems in Section 5.4. Finally, a conclusion is drawn in Section 5.5.

5.2 Ranked Pareto Particle Swarm Optimization In this section, we first introduce some key concepts in a Multiobjective Optimization Problem (MOP). Since the proposed optimization method builds upon the idea of Particle Swarm Optimization (PSO), we will then give a brief description of PSO. Our multiobjective optimization algorithm, the Ranked Pareto Particle Swarm Optimization (RPPSO) method, will be given in detail. 5.2.1

Key Concepts in MOPs

The concept of Pareto Dominance was proposed by Pareto and has been widely used in the research of MOPs. In general, an objective vector Fa in a minimization problem is said to dominate another objective vector Fb , denoted by Fa ≺ Fb , iff fa,i ≤ fb,i

∀ i ∈ {1, 2, ..., m}

(5.1)

fa, j < fb, j

∃

(5.2)

and j ∈ {1, 2, ..., m}

where fa,i and fa, j are components of Fa , fb,i and fb, j are components of Fb . For a multiobjective problem, a solution is said to be Pareto Optimal if it is not dominated by any other solution in the search space. The set of all Pareto Optimal solutions is called a Pareto Set. The aim of MOP is to derive a set of non-dominated solutions with objective values as close as possible to the objective values of the Pareto Set, denoted by Pareto Front.

100

D. Wang et al.

The so-called Multiobjective Evolutionary Algorithm (MEA) has been shown to be suitable for solving multiobjective problems [10, 11, 12, 13, 14]. At each iteration, fitness values (sometimes referred to as the objective values) are evaluated in order to determine better solutions for the next generation. As a result, this will ideally lead to a population of optimal solutions when some termination condition is satisfied. Following this, MEAs have to overcome two major problems [15]. The first problem is how to get close to the Pareto optimal front. The second is how to keep diversity among the solutions in the obtained set. These two problems have become common criteria for most current algorithmic performance comparisons. 5.2.2

Particle Swarm Optimization

Particle Swarm Optimization (PSO) was proposed by Kennedy and Eberhart in 1995, motivated by the social behaviour of organisms such as bird flocking [16]. In PSO, a problem space is covered with an initial population of random solutions in which they are guided to search for the optimum over a number of generations. The concept of PSO is that each particle randomly searches through the problem space by updating itself with its own memory and the social information gathered from other particles. Due to its simple mechanism and high performance of global optimization, PSO has been successfully applied for many optimization problems, including motion planning problems [17, 18, 19]. Suppose the dimension of the searching space is M and the number of particles is N. The position of the ith particle is represented as Xi = (xi1 , xi2 , ..., xiM ). Each particle maintains a memory of its previous best position denoted by Pbesti = (pi1 , pi2 , ..., piM ). The current best position found by all particles is denoted by Gbest = (g1 , g2 , ..., gM ). Vector V = (vi1 , vi2 , ..., viM ) is the velocity of the ith particle. Each particle updates its position according to the following equations: Vnew = ε × Vold + c1 × rand1() × (Pbest − X) + c2 × rand2() × (Gbest − X) Xnew = Xold + Vnew

(5.3) (5.4)

where ε is the inertia factor, c1 and c2 are two positive constants and rand1() and rand2() are two functions returning random numbers in the range [0, 1]. 5.2.3

Ranked Pareto Particle Swarm Optimization

This section introduces the Ranked Pareto Particle Swarm Optimization method (RPPSO). The flowchart of RPPSO is given in Fig. 5.1 (where equations invoked in calculating the variables are indicated within brackets). Symbols and abbreviations used are listed in Table 5.1. Calculation of the intermediate variables is based on the equations enclosed in brackets in the flowchart. 5.2.3.1 Fitness Assignment Consider a multiobjective minimization problem, min F(X), X ∈ Rn P∈φ

5 Ranked Pareto Particle Swarm Optimization

Start

Initialize particles

(5.7) (5.7)

Calculate Raw Fitness Value (RFV) of each function for particles

(5.8) (5.9) Modified Fiteness Value (MFV) indicates the particle qualities with regard to single objective (5.10) Global Modified Fitness Value (GMFV) indicates the particle qualities with regard to all objectives in this iteration (5.11) Particles are ranked according to their qualities

(5.12) Particle Rank Value (PRV) are rounded to integers Particle Copy Number (PCN)

Particles with good qualities form the Global Best

The best histories of particles form the Particle Best

(5.13) (5.14)

New particle locations

N Termination?

Y

End

Fig. 5.1. Flowchart of RPPSO

101

102

D. Wang et al.

Table 5.1. Nomenclature in RPPSO method

Symbols n m PN RFV RFV M MFV MFVV GMFV PRV PCN Gbest Pbest

Descriptions number of decision variable or parameters number of objectives to be minimized particle numbers raw fitness value raw fitness value matrix modified fitness value modified fitness value vector global modified fitness vector particle rank value particle copy number global best particle best

where X = {x1 , x2 , ....xn } is an n-dimensional vector having n decision variable or parameters and φ defines a feasible set of P. F = { f1 , f2, ....., fm } is a vector with m objective components to be minimized. At the beginning of iterations, some particles are initialized randomly in the search space. Assume particles are denoted by Xi . As stated above, Xi is an n-dimensional vector having n decision variables or parameters: Xi = {x1 , x2 , ......xn } ,

i = 1, 2, ......, PN

(5.5)

where n is the number of parameters, PN is the number of particles used in the optimization problem. For any objective, a raw fitness value (RFV) is obtained for each particle: RFV j,i = f j (Xi ),

j = 1, 2, ......, m

(5.6)

where i is the particle number. Thus we get a raw fitness value matrix (RFVM), which is a m × PN array of raw fitness values of this generation: ⎡ ⎤ RFV1,1 RFV1,2 · · · RFV1,PN ⎢ RFV2,1 RFV2,2 · · · RFV2,PN ⎥ ⎢ ⎥ RFV M = ⎢ (5.7) ⎥ .. .. .. .. ⎣ ⎦ . . . . RFVm,1 RFVm,2 · · · RFVm,PN The raw fitness value of each particle is compared with raw fitness values of all other particles. We define that for a minimization problem f j , particle i1 is said to dominate particle i2 at this problem, iff RFV j,i1 < RFV j,i2 ,

j = 1, 2, ......, m

(5.8)

For each particle, the modified fitness value (MFV) on an objective is the number of other particles which are dominated by this particle. MFV shows the quality of particles

5 Ranked Pareto Particle Swarm Optimization

103

regarding objective f j , i.e., particles with higher MFV are more desirable than those particles with lower MFV. For objective f j , a PN-dimensional vector of modified fitness value (MFVV) of this generation is obtained:  MFVV j = MFV j,1 , MFV j,2 , ......MFV j,PN , j = 1, 2, ...m (5.9) For every objective, the same processes is repeated and a MFVV is obtained. The sum of all MFVVs is called to be a global modified fitness vector (GMFV), which is a PN-dimensional vector: GMFV = ∑ MFVV j (5.10) j=1,...,m

The GMFV can be treated as a benchmark of particle quality, in which particle quality with regard to all objectives are taken into account. Particles with higher GMFVs have higher aggregate qualities than those with lower GMFVs in general. Please note that a particle with higher GMFV does not need to have better MFVs (or RFVs) on each objective than those with lower GMFVs. 5.2.3.2 Global Best Here, we set up rules to decide the Global Best, denoted by Gbest . In traditional PSO, Global Best stores the current best location found by all particles. Global Best in RPPSO does almost the same task, but Global Best in RPPSO is a PN-dimensional vector which stores PN best locations found by all particles. To do this, we first set a particle rank value, denoted by PRV, for each particle. The PRV of ith particle is defined by GMFVi PRVi = × PN, i = 1, 2, ..., PN (5.11) ∑ GMFV j j=1,...,PN

The PRVi is a decimal fraction which is proportional to the GMFVi of a particle. This value is then rounded to an integer, denoted by Particle Copy Number(PCN), which determines how many copies of this particle will enter the Global Best of this generation: PCNi = Round(PRVi),

i = 1, 2, ..., PN

(5.12)

To ensure that there are enough candidates for Global Best, this function rounds to the nearest integer towards infinity. The particles which hold higher PRVs have the priority to enter the Global Best vector. This process continues until the Global Best vector is filled up. Particles with lower PRVs, not contributing to solution improvements, will not enter the Global Best vector if it has been filled by copies of particles with higher PRVs and redundant particles will be abandoned. 5.2.3.3 Particle Best The best histories of particles are stored in a PN-dimensional vector, denoted by Pbest . To achieve this, the algorithm keeps the records of the best history of each particle. For a particle, its current location will replace its Pbest only when its current raw fitness values (RFVs) for all objectives are better than those of current Pbest .

104

D. Wang et al.

5.2.3.4 Update Each particle updates its position according to the following equations. vnew = ε × vold + c1 × rand1() × (Pbest − Xold ) + c2 × rand2() × (Gbest − Xold ) (5.13) Xnew = Xold + vnew

(5.14)

These equations look similar to those of conventional PSO. However Gbest in standard PSO usually only stores one best location for all particles. In RPPSO, Gbest is a PN-dimensional vector which stores best PN positions found by particles. As a consequence, particles in RPPSO will search towards many possible directions. In this way, the diversity among solutions is well preserved. 5.2.3.5 Case Study In this section, a simple case is studied to illustrate the process of RPPSO. This problem consists of two variables (x1 and x2 ) space and two objective functions ( f1 and f2 ). f1 (x1 , x2 ) = x21 − 2 × x2, f2 (x1 , x2 ) =

x1 , x2 ∈ [0, 5]

| x1 − 0.2 | + (2 × x2 + x1 )2 ,

x1 , x2 ∈ [0, 5]

(5.15) (5.16)

There are 5 particles in the search space. Particle 1 to 5 are denoted by star, square, diamond, circle and plus sign, respectively. Figure 5.2 (a) shows the particle locations in the 4th generation. (b) and (c) show the locations of Gbest and Pbest before the 4th generation. (e) and (f) show the locations of Gbest and Pbest after this generation. (d) shows the updated particle locations in the 5th generation. There are two points which should be noted in Fig. 5.2. • Gbest in RPPSO may contains some duplicate particles, e.g., square and star in (b), circle and diamond in (e). That means good particles have higher chances to enter the next generation than others. • From (c) and (f) we can see that some values of Pbest changed in this generation (square and star). This is because particle 1 (star) and 2 (square) have better fitness values than before and replaced corresponding Pbest . See (a), (c) and (f). In order to acquire the Pareto Front, we increase the number of particles to 2000. Fig. 5.3 shows the optimization results. The gray dots in Fig. 5.3 illustrate the particle locations and the black stars show the Pareto Optimal Set. If a third objective function f3 (x1 , x2 ) = x2 is added into this problem, the optimization results are given in Fig. 5.4 again. The gray dots show the particle locations and black stars show the Pareto Optimal Set. The number of particles in this simulation is 2000.

5 Ranked Pareto Particle Swarm Optimization

Fig. 5.2. Snapshots of the progress of RPPSO

105

106

D. Wang et al.

Fig. 5.3. Optimization results - 2 objectives

Fig. 5.4. Optimization results - 3 objectives

5.3 F 2 Method In the problem scenario considered in this work, robots are assumed to travel in a 2D environment and their locations can be precisely known. Each robot is aware of the updated information of all other robots, such as their locations, speeds, etc. For

5 Ranked Pareto Particle Swarm Optimization

107

simplicity of explanation, robots are represented as round discs, but can be easily extended to other complex geometric shapes. For more information regarding the F 2 method, refer to [7, 20, 21]. 5.3.1

Definition of a Force Field

A force field is defined as a virtual field of repulsive force in the vicinity of a robot when it travels in a working space. The magnitude and orientation of a force field are determined by and vary with the robot’s status. This virtual repulsive force increases with the decrease of the distance to the robot. Parameters in the F 2 method are listed in Table 5.2. Table 5.2. Parameters used in F 2 Method

Parameters Rr vr vmax Tp θr

We define:

Descriptions radius of a robot absolute value of a robot’s speed maximum absolute value of a robot’s speed a robot’s task priority robot’s orientation in global coordiantion

θ = ∠(θ0 , θr )

(5.17)

Er = vr /(vmax × C)

(5.18)

Dmax =

k × Er × Rr × Tp 1 − Er × cos θ

Dmin = ρ0 × Dmax

(5.19) (5.20)

For any point (x, y) in the 2-D space, assume θr is the robot’s orientation in global coordinate, θ0 is the orientation of this point in global coordinate, thus θ denotes the relative angle of this point to the robot’s orientation (Fig. 5.5). C is a positive number which assigns the environmental influence to the force field with C > 1. Er is a positive decimal fraction with 0 ≤ Er < 1. k is a positive multiplier which determines the coverage area of the force field. Dmax is the maximum active distance of a robot’s force field and Dmin is the distance at which this robot has maximum repulsive force. Dmax shows how far this robot can affect others in its vicinity. Dmin provides a safe distance for the robot to prevent other objects from moving into this area. ρ0 is a positive fractional number with 0 < ρ0 < 1 and heavily influences how close the robot can be separated from obstacles. Tp represents the priority of a task which is undertaken by the robot with Tp ≥ 1. The repulsive force generated by a robot is defined by:

108

D. Wang et al.

Fig. 5.5. Parameters in F 2 method

| Frepulsive robot

⎧ D > Dmax ⎨0 max −D |= P × DDmax −Dmin Dmax ≤ D < Dmin ⎩ Fmax D < Dmin

(5.21)

where D is the shortest distance from point (x, y) to the perimeter of the robot. P is a positive constant scalar which determines the magnitude of the repulsive force. When D changes from Dmin to Dmax , the magnitude of the repulsive force changes from P to 0 gradually. Fmax is the maximum repulsive force which will cause the maximum deceleration on the robot. P and Fmax should be selected based on the robot’s characteristics, with Fmax  P. Equation (5.21) shows that the magnitude of repulsive force varies with distance. To represent the force field in contours, we further define:

ρ = D/Dmax Then (5.21) can be rewritten in an alternative form as: ⎧ ρ >1 ⎨0 1−ρ | Frepulsive robot |= P × 1−ρ0 ρ0 ≤ ρ < 1 ⎩ ρ < ρ0 Fmax

(5.22)

(5.23)

When ρ changes from ρ0 to 1, the magnitude of the repulsive force changes from P to 0. 5.3.2

Attractive Force

When a robot is allocated a particular task, for example, traveling from start point (Xstart ,Ystart ) to goal point (Xgoal ,Ygoal ), a virtual attractive force which attracts the robot from the start point to the goal point is generated. | F f ield |= Q

(5.24)

5 Ranked Pareto Particle Swarm Optimization

109

where Q is a positive constant scalar which determines the magnitude of the attractive force. The attractive force directs to the goal point from the centre of the robot and can be assumed to be a constant. It drives a robot to its destination (Xgoal ,Ygoal ). 5.3.3

Reaction Force

When a robot is moving, its force field will cover some area in its vicinity to protect it from possible collisions. A robot’s force field will cover more area when it travels with a higher speed and shrink when it slow down. When a robot approaches an obstacle or other robots’ force fields, its force field is suppressed and the robot is repelled away by the virtual reaction forces from obstacles or other robots. A reaction force, Frep , can be calculated by: (5.25) | Frep |=| Frepulsive robot | Frepulsive robot is the virtual repulsive force generated by this robot at the interaction point (x, y) in the 2-D space. The reaction force Frep is a vector whose direction is defined to be the normal line of the interaction contour with respect to the interaction point. 5.3.4

V SF 2 Method

The Variable Speed Force Field method (V SF 2 ) is an approach using the concept of the F 2 method for multi-robot motion planning and collision avoidance [7]. Define a fixed reference frame O(X,Y ) and a moving reference frame o(x, y) attached to the robot’s body. Let x, ˙ y˙ be the longitudinal and lateral velocity of the robot in frame o(x, y), the absolute velocities , in the fixed reference frame O(X,Y ) are:        X˙ cos θr − sin θr x˙ x˙ = θ ) (5.26) = R ( r Y˙ y˙ sin θr cos θr y˙ Differentiation with respect the time gives     ax X¨ θ ) = R ( r ay Y¨

(5.27)

where ax , ay are the absolute accelerations expressed in the moving frame o(x, y). For a robot of mass mrobot and rotational inertia I about its centre of mass, equations of motion in the frame are: (5.28) mrobot ax = Fatt x + ∑ Frep x mrobot ay = Fatt y + ∑ Frep y

(5.29)

I ω˙ = Matt + ∑ Mrep

(5.30)

where Fatt x , Fatt y are the components of the virtual attractive force along the longitudinal direction and the lateral direction respectively. Frep x , Frep y are the components of the virtual repulsive forces along the longitudinal direction and the lateral direction

110

D. Wang et al.

Fig. 5.6. Parameters in V SF 2 method

respectively (see Fig. 5.6). ω is the angular velocity of this robot about its centre of mass in frame o(x, y). Matt and Mrep are the moments generated by the virtual attractive force and repulsive force respectively. The forces and moment are calculated as: Matt = r1 × Fatt

(5.31)

Mrep = r2 × Frep

(5.32)

where r1 is the vector from the centre of robot o to the attractive force (Fatt ) action point A. r2 is the vector from the centre of robot to the repulsive force (Frep ) action point B. Please note that we assume the attractive force acts on the robot’s front edge A, which is the interaction of x coordinate and a robot’s body (see Fig. 5.6). It should be noted that a robot needs to satisfy the limitations of translational speed (vr ), translational acceleration (v˙r ), angular speed (ω ) and angular acceleration (ω˙ ). These constraints are: −vmax ≤ vr ≤ vmax

(5.33)

−amax ≤ v˙r ≤ amax

(5.34)

−ωmax ≤ ω ≤ ωmax

(5.35)

−ω˙ max ≤ ω˙ ≤ ω˙ max

(5.36)

These parameters should be chosen on the basis of the robot’s dynamic and kinematic characteristics.

5 Ranked Pareto Particle Swarm Optimization

111

5.4 Using RPPSO for Multiobjective Optimization of F 2 Method Many parameters are used in F 2 method. The setting of these parameters would noticeably affect the method’s performance. In Fig. 5.7, Paths 1 to 5 are simulation results using randomly selected parameters. Please note that for Path 4, collision occurs when the robot travels close to the obstacle, thus the robot fails to achieve its goal in this case. This is a case of unsuccessful parameter selection. Path 6 (solid line) is the optimized result using single objective optimization approach in [8], which is much shorter than those using nonoptimal parameters.

Fig. 5.7. The influence of parameters in V SF 2 method

It can be found that robot runs close around the obstacle in Path 6. Although the path length is reduced significantly, operating too close to obstacles may be undesirable in real applications for safety consideration. Thus an approach which takes into account both efficiency (e.g. path length) and safety (e.g. sufficient safety distance) is needed. RPPSO provides a suitable solution for this multiobjective optimization problem. For all simulations in this section, we assume that all robots are identical and their characteristics are selected based on those of the Amigo robot [22], which has a radius of 0.18 m and a weight of 3.6 kg. The maximum speed is 0.75 m/s. Each robot has an assigned goal and each robot knows its start and goal positions. Robots are equipped with communication devices such that they know the status of other robots, including task priorities, velocities, positions, etc. Task priorities of all robots are set to Tp = 1, i.e., no robot has priority over other robots during collision avoidance. We start from the simple case used in [8], that is, one mobile robot and one static circular obstacle in the working space (Fig. 5.7). This robot is supposed to travel from locations (2, 3) to (8, 7). An obstacle is located at the centre of the work space.

112

D. Wang et al.

Parameters of the F 2 method included in the optimizations are: k, P, Q, C, ρ0 . Based on past experience, we set ranges for parameters as: k ∈ [2, 10], P ∈ [5, 20], Q ∈ [5, 20], C ∈ [1.5, 3], ρ0 ∈ [0.2, 1]. The definition of the ith particle is: xi = [k, P, Q,C, ρ0 ] ,

i = 1, 2, ..., PN

(5.37)

The optimization objectives are set to be: • to minimize the travel length • to keep away from obstacles and other robots as far as possible Two objective functions are chosen based on the above objectives: f1 (xi ) = FF(xi )

(5.38)

where FF(xi ) is the resultant path length obtained from the F 2 method using parameters xi . When a robot reaches its destination, f1 (xi ) will return the value of the real travel distance. If a robot fails to reach its destination using these parameters or collision occurs, f1 (xi )will return a very high value. Assume that for each time cycle, the shortest distance from this robot to obstacles in the working space is denoted by Dnearest . The safety measurement function is defined as: f2 (xi ) = ∑

1 Dnearest,xi

(5.39)

If a robot travels close to obstacles, f2 (xi ) will give a high value. On the contrary, when robot travel far from obstacle, f2 (xi ) will result in a lower value. 5.4.1

Case 1: Single Robot and One Obstacle

In this simulation, a robot is supposed to travel from locations (2, 3) to (8, 7), as shown in Fig. 5.8. The time cycle of simulation is set to 0.1 second. An obstacle is located at the centre of the work space. The optimization results are shown in Fig. 5.9, in which the optimal set is shown in black stars and other particles are shown in gray dots. Table 5.3 shows some optimization results using RPPSO. In Table 5.3, simulation results in [8] are given at the 1st row. Row 2 to 6 are simulation results using RPPSO, denoted by point A, B, C, D, E (particles in the Pareto Front) in Fig. 5.9. Table 5.3. Case 1: optimization results

Test 1 A B C D E

k

P

Q

C

ρ0

Path Danger length(m) index 4.2380 9.5660 9.2680 1.6230 0.2980 7.1060 43.5490 4.5068 10.9423 15.7974 1.1488 0.4605 7.5916 5.2414 4.5273 10.8845 16.3748 1.1476 0.4594 7.6023 5.2162 4.5280 10.8821 16.5471 1.1479 0.4587 7.6091 5.1895 4.5265 10.9539 16.7658 1.1475 0.4574 7.6177 5.1599 4.7715 10.8570 16.3219 1.1497 0.4588 7.7501 5.1409

5 Ranked Pareto Particle Swarm Optimization

113

Fig. 5.8. Case 1: resultant path

Fig. 5.9. Case 1: Pareto optimal set

The number of particles of the RPPSO process in this simulation is 1000. The termination criterion is set to be 100 iterations. In this case, RPPSO reaches a stable state after 60 generations. Figure 5.10 shows the parameter values of particle B (3rd row in Table 5.3) during the iterations.

114

D. Wang et al.

Fig. 5.10. Case 1: evaluation of optimized parameters

Fig. 5.11. Case 2: resultant paths

As shown in Table 5.3, the resultant parameters fall within the same order of magnitudes. The path lengths are close to, but slightly longer than, the case for the single objective optimization approach. The effectiveness of the proposed multiobjective optimization method is explicitly noticeable from significant reductions in the danger index (last column in Table 5.3). In Fig. 5.9 the path of point B is shown in solid line in

5 Ranked Pareto Particle Swarm Optimization

115

Fig. 5.8. Compared with previous result of [8], which is denoted by a dashed line in Fig. 5.8, the new path follows a larger contour around the circular obstacle. By using RPPSO, we can guarantee simultaneously efficiency and safety. 5.4.2

Case 2: Two Robots Navigating in a Corridor

In this simulation robot 1 is supposed to travel from locations S1 (1, 5.25) to G1 (9, 5) and robot 2 travels from S2 (9, 5) to G2 (1, 4.75). The distance between two walls (black patches) is 2 m. Robots start simultaneously from their start points. The time cycle is set to 0.01 second. Robot 1 and 2 use the same set of parameters in each RPPSO iteration. Fig. 5.12 shows the results of RPPSO after 50 generations using 50 particles. Particles of the optimal set (totally 6) are shown in black stars and other particles are shown in gray dots. Details of these optimal particles are given in Table 5.4. The parameters corresponding to the particle indicated by the circle (Fig. 5.12) are listed in the 3rd row

Fig. 5.12. Case 2: Pareto optimal set Table 5.4. Case 2: optimization results

Test 1 2 3 4 5 6

k

P

Q

C

ρ0

Path Danger length(m) index 2.4208 17.1789 19.3420 1.3274 0.4159 15.7011 1417.4227 2.3219 17.0566 19.7220 1.3278 0.4647 15.7066 1394.4961 2.3407 17.8411 19.6860 1.3381 0.3980 15.7589 1376.8007 2.3524 17.8243 19.9060 1.3340 0.3958 15.7755 1376.3172 2.2883 16.7237 19.4253 1.3278 0.3920 15.7789 1370.6500 2.3047 16.2929 20.0047 1.3304 0.4802 15.7951 1369.6322

116

D. Wang et al.

Fig. 5.13. Case 2: distances to obstacles

of Table 5.4. Its resultant paths are shown in Fig. 5.11. The paths of robot 1 and 2 are shown in dashed and solid lines, respectively. In Fig. 5.13, X coordinate is time and Y coordinate shows the nearest distances from a robot to obstacles (including walls and another robot). The distances to obstacles with regard to Robot 1 and 2 are shown in dashed line and solid line, respectively.

5.5 Conclusion In this chapter, we have proposed the Ranked Pareto Particle Swarm Optimization method (RPPSO) for planning collision free paths for multiple mobile robots. In RPPSO, particles are ranked by their qualities with regard to all optimization objectives. Those particles with higher aggregate qualities constitute the Global Best, which stores many current best solutions found by particles. So unlike conventional Particle Swarm Optimization (PSO) method, particles in RPPSO will search toward many directions while taking into account all optimization objectives. We have illustrated that RPPSO is capable of solving efficiently multiobjective optimization problems. To this end, the RPPSO is successfully utilized in the multiobjective parameter optimization problem for the Force Field (F 2 ) method, which is developed for collision avoidance between mobile robots and obstacles. Simulations and experiments are included to demonstrate satisfactory performance in motion planning of multiple robots taking into account not only collision avoidance but also minimum travel lengths.

5 Ranked Pareto Particle Swarm Optimization

117

References 1. Labtombe, J.C.: Robot motion planning. Kluwer Academic Publisher, Boston (1991) 2. Hwang, Y.K., Ahuja, N.: Gross motion planning - a survey. ACM Computing Surveys 24(3), 219–291 (1992) 3. Khatib, O.: Real-time obstacle avoidance for manipulators and mobile robots. International Journal of Robotics Research 1(5), 90–98 (1986) 4. Connolly, C.I.: Harmonic functions and collision probabilities. In: Proceedings of the IEEE International Conference on Robotics and Automation, pp. 3015–3019 (1994) 5. Ge, S.S., Cui, Y.J.: Dynamic motion planning for mobile robots using potential field method. Autonomous Robot 3(13), 207–222 (2002) 6. Ren, J., McIsaac, K.A., Patel, R.V., Peters, T.M.: A potential field approach using generalized sigmoid functions. IEEE Transactions on Systems, Man, and Cybernetics - Part B: Cybernetics 37(2), 477–484 (2007) 7. Wang, D., Liu, D.K., Dissanayake, G.: A variable speed force field method for multirobot collaboration. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 2697–2702 (2006) 8. Wang, D., Kwok, N.M., Liu, D.K., Lau, H., Dissanayake, G.: PSO-tuned F2 method for multi-robot navigation. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and System, pp. 3765–3770 (2007) 9. Kwok, N.M., Ngo, V.T., Ha, Q.P.: PSO-based cooperative control of multiple robots in parameters-tuned formations. In: Proceedings of the IEEE Conference on Automation and Science Engineering (IEEE CASE 2007), pp. 332–337 (2007) 10. Schaffer, J.D.: Multiple objective optimization with vector evaluated genetic algorithms. In: Proceedings of the First International Conference on Genetic Algorithms: Genetic Algorithms and their Applications, pp. 93–100 (1985) 11. Fonseca, C.M., Fleming, P.J.: Genetic algorithms for multiobjective optimization: Formulation, discussion and generalization. In: Proceedings of the Fifth International Conference on Genetic Algorithms, pp. 416–423 (1993) 12. Horn, J., Nafpliotis, N., Goldberg, D.E.: A niched Pareto genetic algorithm for multiobjective optimization. In: Proceedings of the 1994 Congress on Evolutionary Computation, vol. 1, pp. 82–87 (1994) 13. Zitzler, E., Thiele, L.: Multiobjective evolutionary algorithms: a comparative case study and the strength Pareto approach. IEEE Transactions on Evolutionary Computation 3(4), 257– 271 (1999) 14. Zitzler, E., Laumanns, M., Thiele, L.: SPEA2: Improving the strength Pareto evolutionary algorithm, Technical Report 103, Computer Engineering and Networks Laboratory (TIK), Swiss Federal Institute of Technology (ETH), Zurich, Switzerland (2001) 15. Zitzler, E., Deb, K., Thiele, L.: Comparison of multiobjective evolutionary algorithms: empirical results, evolutionary computation 8(1), 173–195 (2000) 16. Kennedy, J., Eberhart, R.: Particle swarm optimization. In: Proceedings of the IEEE International Conference on Neural Networks, vol. 4, pp. 1942–1948 (1995) 17. Doctor, S., Venayagamoorthy, G.K.: Unmanned vehicle navigation using swarm intelligence. In: Proceedings of International Conference on Intelligent Sensing and Information Processing, pp. 249–253 (2004) 18. Chen, X., Li, Y.: Smooth path planning of a mobile robot using stochastic particle swarm optimization. In: Proceedings of the 2006 IEEE International Conference on Mechatronics and Automation, pp. 1722–1727 (2006)

118

D. Wang et al.

19. Kwok, N.M., Ha, Q.P., Fang, G.: Motion coordination for construction vehicles using swarm intelligence. International Journal of Advanced Robotic Systems 4(4), 469–476 (2007) 20. Wang, D.L., Liu, D.K., Wu, X., Tan, K.C.: A force field method for robot navigation. In: Proceedings of the Third International Conference on Computational Intelligence, Robotics and Autonomous Systems (2005) 21. Liu, D.K., Wang, D., Dissanayake, G.: A force field method based multi-robot collaboration. In: Proceedings of the IEEE International Conference on Robotics, Autonomous & Mechatronics, pp. 662–667 (2006) 22. Mobile Robot Bases Specifications, ActivMedia Robotics, http://www.activrobots.com/ROBOTS/specs.html

6 Path Planning Inspired on Emotional Intelligence R. Barber, V. Egido, M. Malfaz, and M.A. Salichs RoboticsLab. Carlos III University, Madrid, Spain [email protected], [email protected], {mmalfaz,salichs}@ing.uc3m.es

Abstract. This chapter is focused on how emotional intelligence is used in order to plan the most suitable path for a robot when fulfilling a certain task. Classic planners use several criteria such as the shortest path, or the minimum time, but those planners do not take into account some inner factors as human beings do in the decision making process. Those factors are for instance, previous experiences that led to failures or successes as well as inner factors that can determine the fact that sometimes people are more conservative or more brave when selecting a path to reach to some place. In this chapter, a general view of robot planning as well as the role of emotions in robots are exposed first. Moreover, an example of an emotion inspired planner applied to a topological navigation system is shown, as well as the results obtained with a real robot.

6.1 Introduction The path planning in robotics involves the knowledge and modeling of the environment as well as certain criteria for selecting the best path. In order to tackle this problem, some basic concepts related to the robot navigation are going to be described. Next, the role of emotions in the decision making process will be stated and later, the topological modeling of the environment will be introduced as the natural way of representing the environment. Finally, a planning algorithm inspired on emotions will be presented as an example. This algorithm is derived from another known algorithm, the Dijkstra algorithm. Making some modifications in the cost function of the Dijkstra algorithm the proposed “Guided Dijkstra” is obtained. This new algorithm tries to imitate one of the most common human behaviour of selecting longer but more familiar routes, minimizing the chance of getting lost. Differing from other similar research, as that one carried out by Roy [17], the chosen path here will depend on the previous navigation actions (past experiences) and on the successful or failed actions that the robot carried out (positive or negative reinforcement). This fact implies that the selected path will not be established a priori.

6.2 The Path Planning Problem As part of everyday life, people navigate from one place to another using their knowledge about the environment. This is a natural process people learn in their childhood D. Liu, L. Wang, and K.C. Tan (Eds.): Design and Control of Intel. Robotic Systems, SCI 177, pp. 119–132. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com


120

R. Barber et al.

[11] and develop as they grow up. But how do people find their ways? Usually most of the computer models do not simulate the behaviour of human wayfinders. Researches in topological navigation in the last years tend to separate two problems that were before considered as one: path planning and wayfinding. The first one groups together problems related with obstacle avoidance until a defined point is reached. The second one is the task sequencing at higher level. This one can lead in a final event that does not necessarily have to be a specific geometric point. The path planning problem in a first approximation, can be divided in two tendencies. The first one, considered as the classic one, assumes the environment’s total geometric knowledge. The second one, based on the sensorial information, considers the lacks that are linked to the real information obtained of what surrounds such as: partial knowledge, sensor failure, obstacles that do not allow total vision, unexpected situation, etc. Within this field, all the classical avoidance techniques are found, as those based in potential fields or the use of maps based on reticular occupation. In relation to the wayfinding problem, planning movements from an initial place to a final one trying to imitate the human behaviour, researchers in the field of spatial knowledge have the same overall vision of the problem by layers. Timpf and Volta [21] establish 3 levels: The planning level, the instruction one and the guidance behaviour one. Based on the navigation model that focuses on the mental or cognitive map, these researches consider that it is through this mental map how the following task is accomplished: the task of obtaining verbal instructions to indicate paths, navigation and environment description. Some other research lines, such as those of Chatila [6], and Firby [9], are found. Both consider the planning in a high level of deliberative type. Chatila considers a more extended planning hierarchy, in which it is established that the planning concept can be divided into path and movement planning, perception planning, navigation planning, manipulation planning, communication planning, and task planning.

6.3 Emotions in Robotics One of the main objectives in robotics and artificial intelligence research is to imitate the human mind and behaviour. For this purpose the studies of psychologists on the working mind and the factors involved in the decision making are used. Several psychologists have realized the importance of the emotions in the intelligent thinking and behaviour. In fact, it has been proved that two highly cognitive actions are dependant not only on rules and laws, but on emotions: Decision making and perception [16]. Moreover, some authors affirm that emotions are generated through cognitive processes. Therefore emotions depend on ones interpretation, i.e. the same situation can produce different emotions on each agent, such as in a football match [15]. Moreover, emotions can be considered as part of a provision for ensuring and satisfaction of the system’s major goals [10]. Emotions play a very important role in human behaviour, communication and social interaction. Emotions also influence cognitive processes, particularly problem solving and decision making [7].

6 Path Planning Inspired on Emotional Intelligence

121

Based on similar assumptions Minsky conclude that, “the question is not whether the intelligent machines can have any emotions, but whether machines can be intelligent without any emotions” [13]. In this chapter some experiments where the planning process of a real robot is affected by an emotional factor will be shown. The selection of a certain path will be determined by the importance given to previous experiences. Depending on the weight of the negatives past experiences the robot will select or not an known path. This weight will be given by the inner state or personality of the robot.

6.4 Topological Representation of the Environment In the past years, some authors [14] [19] have studied the way to represent the environment and the way in which animals and humans move along it. These authors conclude that the natural method to represent the environment is by topological structures and topological relationships. Autonomous mobile robots need good environment models. Several maps representation have been considered in the literature. The map types used are the geometric and the topological ones. The first one uses exact metrical information and it allows us to work with accuracy in nearby robot environments. Its complexity makes us use complex planners, not being useful for big environments. The second one is a world-wide or an environment representation based on environment characteristic and relationships. This information obtained from this type of map is represented by graphs, allowing a big environment simple representation and facilitating fast planning. Topological maps are made up, in most cases, of nodes and edges. An example of this construction are the maps implemented by Kuipers and Byun [12], Wallner, Kaiser, Friedrich and Dillmann [23], Aycard, Laroche and Charpillet [1] and by Tiche and Hgli [20]. Kuipers and Byun built an environment topological map in which geometric information can be added. The topological map consists of nodes related to different places and links related to routes or paths between nearer nodes. Topological navigation has some advantages over the geometric navigation. Considering the geometrical navigation, if some information related to the robots odometry is lost, for example a small step on the floor, the robot can get lost easily. With topological information, the different sensorial perceptions and the different motor actions are the ones which tell the robot where it is. This obviously demands big accuracy to the sensorial systems. Besides this advantage, the topological maps present some other advantages versus the geometric maps. The topological maps make an easy representation of big environments, facilitating a fast planning, but complex sensorial and sensimotors skills are needed.

6.5 Path Planning Algorithm Inspired on Emotions In this section a path planning algorithm inspired on emotions is going to be introduced. First, the representation of the environment that has been used is going to be described. Moreover, the planning is going to be done based on this representation. Next, the planning algorithm is described in detail.

122

6.5.1

R. Barber et al.

Environment Representation: The Navigation Chart

The environment representation used in this paper is the Navigation Chart. The Navigation Chart is not formed by a succession of places of the environment but by a succession of elementary skills [2]. It is represented by a simple directed parameterized graph G(v,e) formed by nodes v and edges e. Nodes are events regarding sensorial perceptions (be in front of a door) and edges represent sensorimotor skills which lead the robot to the next event (move towards a door). A new situation is being described on the Navigation Chart when detecting an event. The robot finds itself in the specific situation or place in which a specific sensorial event is sensed. An example of this Navigation Chart is shown in Figure 6.1.

B finish A

Go straight

Turn on the right

Turn on the right

start

A on the left

Intersection Intersection

B on the right

Fig. 6.1. Example of Navigation Chart

6.5.1.1 Nodes The Navigation Chart nodes have the structure shown in Figure 6.2. Node identification: It is the digit which allows to identify each node. However, several nodes may have assigned the same sensorial event. This digit will be unique due to the fact that the same two events, for example the event left door, could be assigned to different doors. Sensorial event: Depending on the capacities carried out on the robot, a set of detectable events will be obtained. These events are generated, in the present work by

Id:

Node 0

Event:

left door

Sentinel:

(0,1)

Weight:

[-2...3]

Parameters:

d,α...

Fig. 6.2. Node structure in the Navigation Chart

6 Path Planning Inspired on Emotional Intelligence

123

robot skills. These skills, as it will be explained later, are implemented in the automatic level of a two level control architecture. Sentinel: It is a variable with range (0,1) which will show if the node has been previously visited or not. Weight: It is a variable, which allows obtaining an historic of the sensorial events that have already been tested and those that failed. Its value is of great importance to establish new plans considering past experiences. Parameters: The node can contain data and additional information, obtained from the sensorial skill data object that enables the event. 6.5.1.2 Edges Transitions between nodes are performed by robot actions or robot movements such as turn right, follow a corridor, move towards a door. The Navigation Chart transitions have the structure shown in Figure 6.3:

Skill:

Corridor follow

Planning parameters:

40s 10m

Activation parameters:

vmax, dmin...

Sentinel:

(0,1)

Fig. 6.3. Edge structure in the Navigation Chart

Sensorimotor skill: The transitions are identified by a name. The name indicates the sensorimotor skill that allows the robot to go from one event to another. Planning parameters: The planner considers them when trying to establish a plan. An example of this is the estimated distance between points or the time needed to reach the following event. Activation parameters: To activate motor actions, additional information could be required. This information is stored in those parameters. Sentinel: This parameter is different from the one explained in the nodes. It indicates when an error has occurred during the execution of the sensorimotor skill. 6.5.2

Topological Planning Algorithm: Guided Dijkstra

6.5.2.1 Simple Planning with Dijkstra Algorithm As the first approximation, the Dijkstra algorithm is used to carry out the path planning. This algorithm has been used by other authors, as [5], and it is considered as the most adequate algorithm to establish the path that minimizes a parameter in a graph. Therefore, this algorithm is used for searching for a sequence between nodes of the Navigation Chart which will optimized a certain parameter. Thinking on the way human beings decide which way to take, it was considered that one of the factors that

124

R. Barber et al.

Fig. 6.4. Graph G(v, e) and the sequence from v0 to v f

influences the decision making process is the time needed to get from one place to another. Therefore, the cost function that this algorithm tries to minimize is the time required from the source node to the final node. The Navigation Chart can be considered as a guided parameterized graph, G(v, e) made up of edges and nodes, both including information. To plan consists of establishing from an initial node v0 , the sequence of nodes v0 , ..., vi , ...v f and edges e1 , ..., ei+1 ...e f −1 that leads to the objective node v f , see Figure 6.4. The sequence will be translated in commands to perform, and events to detect until the sequence reaches the event v f of the final node. This algorithm is very efficient when establishing paths that minimizes times and distances. If the objective is to have an autonomous robot, the robot must take into account the time restrictions as well as its inner or emotional state, as humans do. Many times one do not choose the shortest path but the one that seems to be more secure. For this reason, a modification of the Dijkstra algorithm is going to be used. This new approach will allow the robot to consider its inner state when planning a certain path. 6.5.2.2 Path Planning with Guided Dijkstra Algorithm Based on previous works carried out by expert authors in algorithms [22] and observing the human being behaviour reflected in similar works dedicated to its study [18], the Dijkstra algorithm was modified to give the robot a behaviour that will take into account human traces. The psychologist R.Zajonc demonstrated in 1980 at Michigan University that human beings prefer stimulus that they have already experienced over new ones. Zajonc in [24] described that there exists a clear tendency towards what is familiar (more pleasant) over the unknown. In the case of a robot, the more the robot visits a certain place, the more secure the robot will be since the stored parameters will be updated. In order to take into account the previous experiences, the guided Dijkstra algorithm proposes the following modification of the cost function: guidedcost[e(i, j) ] = cost[e(i, j) ] − λ · r(v j , e(i, j) )S(v j )

(6.1)

6 Path Planning Inspired on Emotional Intelligence

125

Where: • cost[e(i, j) ], is the time needed to perform the motor action that takes the robot from event vi to another event v j . • r(v j , e(i, j) ) is the reinforcement parameter. If its value is greater than 0 works as a reward and as a penalization if its value is lower than 0. If the navigation between node i to node j was successful then the reinforcement value increases, on the contrary, if that navigation was a failure then the reinforcement decreases. • S(v j ) is the sentinel correspondent to node j. This parameter indicates if this node has been visited previously or not. • λ is the emotional factor, that gives more or less importance to previous experiences. If this factor is high, then the robot will be fearful and tends to select secure paths. The effect of the guided Dijkstra algorithm is the following: if we consider an initial situation in which no previous planning has been done, all the nodes have its sentinels set to 0 and it will choose the shortest path applying the Dijkstra Algorithm without modification. During navigation, the S(v j ) sentinel will take value 1 if is visited, and 0 if it is not visited or a problem while detecting the event has occurred. If the sentinel is 0 the planner will not consider the previous experiences and therefore, the reinforcements of the nodes in future planning. Once the event is detected, the sentinel will take value 1 again, considering previous navigations. The r value is set by: r(v j , e(i, j) ) = w(v j ) ∗ cost[e(i, j) ]

(6.2)

where w(v j ) is the weight vector. Its value begins at 0 and increases or decreases one by one according to the following criterion: if the node or event is detected at the time set by the Navigation Chart, during the navigation, the node is then considered as visited, its sentinel is set to 1 and w(v j ) is increased in one unit. On the other hand, if the event does not occur, w(v j ) is decreased in one unit and its sentinel is set to 0.

6.6 How the Guided Dijkstra Algorithm Works Next, an example showing how the previously described algorithm works is presented. The implementation has been done in a control robot architecture called AD, that runs on a B12 platform of i-Robot. 6.6.1

Planning in the AD Control Architecture

The control architecture used in this work is the AD architecture [3]. The architecture is made up of two levels: a deliberative level, which contains the reflexive processes and the automatic level, where the processes with sensor and motor capacities are found. Both levels present the same characteristic: they are made up of skills. Skills consist of the different capacities of reasoning or carrying out an action. These skills return data and events to the skills or sequencers which have activated them. In the higher level, see Figure 6.5, the skills related to the capacities of reasoning and decision-taking, can be found. The planner, described in this paper, and the navigator are examples of deliberative skills. The planner uses the Navigation Chart and it

R. Barber et al.

PRINCIPAL SEQUENCER

DELIBERATIVE LEVEL

Long Term Memory

Planner

Navigation Navigator Chart with modifications Sequencer

Events

To the Automatic Level

Short Term Memory

New plan needed event failure Data Flow Performance orders flow

New plan for the navigator Events Flow

Fig. 6.5. Deliberative level in the AD architecture

From the Deliberative Level

AUTOMATIC LEVEL

126

Corridor Travelling

Virtual Laser

Laser

Short term Memory

Cross Door

Virtual Sonar

Sonar

Data Flow Performance orders flow

Contour Following

Virtual Odometry

Odometry

Door Detection

Identify Location

Virtual Base

Base Events Flow

Fig. 6.6. Automatic Level in the AD architecture

6 Path Planning Inspired on Emotional Intelligence

127

establishes the plan as a subgraph of that representation. The navigation skill takes this subgraph and converts it in the sequence of events to be detected (nodes) and sensorimotor actions (edges). Once the process is sequenced, this skill is the one that modifies the Navigation Chart, storing the parameter changes as sentinels and nodes weights. In the automatic level there are skills which interact with sensors and actuators [4]. Examples of automatic skills used in the present work are: door detection, corridor traveling, door crossing, contour following and identifying location, see Figure 6.6. These skills will be associated with the nodes and edges of the Navigation Chart. The architecture considers the exchange of events and data through a Long Term Memory and a Short Term Memory. The Long Term Memory is exclusively used for the deliberative level skills, containing information which needs to be stored for long periods of time. The Long Term Memory contains, in this case, the environment representation as a Navigation Chart. On the other hand, the Short Term Memory stores temporary data and is used for both the deliberative and automatic level. The Short Term Memory contains, in this work, the generated plan for the planning skill. 6.6.2

Deliberative Planner Skill

The planner has been implemented as a deliberative skill in the AD control architecture. This skill has been considered one of the deliberative skills because it has to take decisions about the more suitable path according to some criteria. Its mission is to generate the sequence of nodes and edges which will be used by the Navigator skill to take the robot from an initial event v0 to a final event v f , considering the parameters from previous planning experiences. The basic structure of the Planner deliberative skill is shown in Figure 6.7. The value of the emotional factor λ is determined a priori as part of the personality of the robot. The skill is made up of an active object and a data object [4]. The active object receives as activation parameters the initial and final nodes (v0 ,v f ) which show the first and last planning events and as data input the Navigation Chart from the LongTerm Memory. Its output will be the events which will get to the sequencer from the

Fig. 6.7. The planning deliberative skill

128

R. Barber et al.

Deliberative Level and the plan extracted from the Navigation Chart that will be stored in the data object, that is, in the Short-Term Memory, for further use by the navigation deliberative skill. The navigator will execute the plan and will carry out the plan supervision, considering the possibility of changing weights and sentinels of the Navigation Chart that is found in the Long-Term Memory. 6.6.3

An Example of the Emotional Inspired Planning

To test the planner, an indoor environment formed by corridors and rooms has been considered. Before the planning experiments were carried out, the robot generated autonomously its own model of the environment. In this case, this model was the Navigation Chart. The process of autonomous generation of this Navigation Chart by the robot is explained in [8]. The environment layout during the planning experiments is shown in Figure 6.8 where doors which belong to rooms C8, C4, C12, C13 and the one which communicates C12 with C13 are opened. In this environment several plans are carried

Fig. 6.8. Work environment L.D.Det.

17

Corr.F.

. 24 L. D. Det .

R.D.Det.

c8

C. D o or

I.Ub. ht ig

C.F.

T. Le ft

R T. .

C.

21

Do or

oo r

or

oor

T. Rig ht .

Co rr. F

.

C. D oo r

C. D

Ub. Corr.F.

27

20 22

.

10

I.

I.Ub.

.F

14

15

26

C

.F.

r oo

.

13

C.F.

32 b. I. U C.F.

25

t igh

I.Ub.

T. R

F C.

ht

rr Co

.D C

r. oo 19

18

.

I.Ub.

R.D. Det.

c12

Rig

Corr.F.

L.D.Det.

Corr.F.

23

.D C

T.

7 C.F. I.Ub. 33

F C.

or Do 6

12

11

5

C. 8

or

C.F.

t Lef

ft Le T. 2

4

I.Ub.

t

End of Corridor

R.

r. oo i gh T. r

3

0

Corr.F.

16

Do C.

.D C

C. Pta

R.D.Det.

L.D.Det.

Corr.F.

9

C. Do

Corr.F.

1

C. D

R.D.Det.

c4

Corr.F. Corridor Following. T. Righ. Turn Right T.Left. Turn Left. I. Ub Identify Ubication C.F Contour Following C. Door. Cross Door R.D.Det. Right Door Detected. L.D.Det. Left Door Detected.

Fig. 6.9. Plan obtained from the Navigation Chart

30 29 . C I.Ub.

Corr.F .

c13

28 or Do

31 End of corridor

C.Door

6 Path Planning Inspired on Emotional Intelligence

129

End of corridor – Room C13 Planning

Corridor

1 0 -1

0

2

4

6

8

10

12

Room C13

-2 -3 -4

Room C12

Fig. 6.10. Path followed by the robot during the navigation, the shortest path

Fig. 6.11. Different navigation processes carried out successfully by the robot

out, showing how the planning criteria is modified as a consequence of the weight and sentinel changes during the navigation. At the beginning the robot is commanded to reach, from its current position (end of corridor), the event that identifies room C13. Since this navigation mission is the first one carried out in this environment, sentinels set to zero, the robot chooses the nodes and edges sequence that takes less travel time. Those sequences have been drawn in a thicker line in Figure 6.9, over part of the Navigation Chart. The representation of the path followed by the robot is obtained through its odometry, for this case, as it is shown in Figure 6.10. Afterwards several navigation processes that implied corridor traveling were carried out. This situation made that the nodes associated to events placed in the corridor were reinforced, compared with those that were unknown zones. The generated plans of these missions and the accomplished navigation processes are shown in Figure 6.11.

R. Barber et al.

10

17

R T. .

.

21 I.Ub. ht ig

C. D oor

22

Corr.F.

24 L.D. Det.

R.D.Det.

c8

27

C.F.

30 I.Ub.

28 r 29 . Doo C

31 End of corridor

T. L

C. Do or

Corr.F.

Co rr. F.

26

I.Ub.

20

C. Do or

or

Rig ht .

C. D o

C. D oor

14 I.Ub.

25

F. C.

. .F

I.Ub.

ht Rig

19

C

. rr. F Co

r

T.

15

32 b. I. U C.F.

oo .D C

r. oo .D C

C.F.

Corr.F.

23

13

R.D. Det .

c12

18

. .F

Corr.F.

L.D.Det.

ht Rig

7 C.F. I.Ub. 33

12 I.Ub.

C

r oo .D 6

T.

8

11

5

C

t ef

eft

C.F.

or Do

ht L T.

L R.

ig T. r 2

4

I.Ub.

Corr.F.

16

C.

r. oo .D C 3

0 End of Corridor

C. Pta

R.D.Det.

L.D.Det.

Corr.F.

9

C. Do or

Corr.F.

1

eft

L.D.Det.

R.D.Det.

T.

130

c4

Corr.F .

C.Door

c13

Corr.F. Corridor Following. T. Righ. Turn Right T.Left. Turn Left. I. Ub Identify Ubication C.F Contour Following C. Door. Cross Door R.D.Det. Right Door Detected. L.D.Det. Left Door Detected.

Fig. 6.12. Plan obtained when the robot is fearful and the corridor is a secure path

Fig. 6.13. Path followed by the robot when it is fearful and the corridor is a secure path

Next, a new planning to reach to room C13 again is done. In this occasion, the emotional factor λ is high, this is, the robot is fearful. Now, the robot chooses the path that includes the corridor in order to fulfil the previous task since the corridor has been reinforced and marked as a secure path. The plan is shown in Figure 6.12 and The path followed by the robot is shown in Figure 6.13. Next, a small value is given to the emotional factor and the robot, since is “brave”, selects again the first path, the shortest one, see Figure 6.9 and Figure 6.10. This is because the robot does not consider the previous navigation, so decides to select the path with the smallest time cost instead of the most secure one. Later, the corridor is blocked by us and the robot tries to navigate this area, see Figure6.14. Therefore, during this navigation the value of the reinforcement of the nodes of the corridor decreases. After this experiment, the robot with a high emotional factor, the robot is fearful, tries to reach again the room C13. As a consequence of the negative reinforcements of the nodes of the corridor, the robot selects again the shortest path, see Figure 6.9 and Figure 6.10, since the corridor is marked as an unsecured path.

6 Path Planning Inspired on Emotional Intelligence

131

Fig. 6.14. Path followed by the robot when the corridor is blocked

6.7 Conclusions The role of the emotional factor, as in human beings, is useful as well as necessary for the decision making process. In the case of path planning for mobile robots, this emotional factor is presented as an alternative to the classic navigators. The described planner uses information from previous navigation. Two criteria have been established for the plan generation: one that only minimizes travel time (Dijkstra) and a modification in the cost function that considers a human behaviour, minimizing the chance of getting lost by taking longer but known routes (Guided Dijkstra). As a consequence of this, when planning, the previous experiences make the robot choose a known zone instead of a shortest but unknown path. The robot navigation becomes then safer.

References 1. Aycard, O., Laroche, P., Charpillet, F.: Mobile robot localization in dynamic environments using places recognition. In: Proceedings of the 1998 IEEE International Conference on Robotics and Automation (1998) 2. Barber, R., Salichs, M.A.: A new human based architecture for intelligent autonomous robots. In: The Fourth IFAC Symposium on Intelligent Autonomous Vehicles, pp. 85–90 (2001) 3. Barber, R., Salichs, M.A.: Mobile robot navigation based on event maps. In: Field and Service Robotics, pp. 61–66 (2001) 4. Boada, M.J.L., Barber, R., Salichs, M.A.: Visual approach skill for a mobile robot using learning and fusion of simple skills. Robotics and Autonomous Systems 38, 157–170 (2002)

132

R. Barber et al.

5. Cara, A., Taylorb, G., Brunsdonc, C.: An analysis of the performance of a hierarchical wayfinding computational model using synthetic graphs. Computers, Environment and Urban Systems 25, 69–88 (2001) 6. Alami Chatila, R., Prayoux, R.: An architecture integrating task planning and reactive execution control. In: Workshop on Architectures for Intelligent Control Systems, IEEE Nice, France (1992) 7. Damasio, A.: Descartes’ Error - Emotion, reason and human brain. Picador, London (1994) 8. Egido, V., Barber, R., Boada, M.J.L., Salichs, M.A.: Self-generation by a mobile robot of topological maps of corridors. In: IEEE International Conference on Robotics and Automation, Washington, USA, pp. 2662–2667 (May 2002) 9. Firby, J.R.: Task networks for controlling continuous processes. In: Hammond, K. (ed.) In sensorial International Conference on AI Planning Systems, Menlo Park, California, pp. 49– 54. AAAI Press, Menlo Park (1994) 10. Frijda, N., Swagerman, J.: Can computers feel? theory and design of an emotional model. Cognition and Emotion 1(3), 235–357 (1987) 11. Inhelder, B., Piaget, J.: The child’s conception of space. Norton, New York (1967) 12. Kuipers, Y.T., y Byun, B.J.: A qualitative approach to robot exploration and map-learning. spatial reasoning and multi-sensor fusion. In: Prooceedings of the 1987 Workshop, Morgan Kaufmann, USA (1987) 13. Minsky, M.: The Society of Mind. Simon and Schuster (1986) 14. Nehmzow, U.: Animal and robot navigation. Robotics and Autonomous Systems 15, 71–81 (1995) 15. Ortony, A., Clore, G.L., Collins, A.: The Cognitive Structure of Emotions. Cambridge University Press, Cambridge (1988) 16. Picard, R.W.: Los ordenadores emocionales. Ed. Ariel S.A (1998) 17. Roy, N., Burgard, W., Fox, D., Thrun, S.: Coastal navigation – robot motion with uncertainty. In: Proceedings IEEE Conf. Robotics and Automation (ICRA) (1999) 18. Ruddle, R.A., Payne, S.J., Jones, D.M.: Navigating buildings in desk-top virtual environments: experimental investigations using extended navigational experience. J. Exp. Psychol., 143–159 (1997) 19. Thruiller, O., Meyer, J.A.: Biomimetic navigation models and strategies in animats. AI Communications 10, 79–82 (1997) 20. Ti´eche, F., H¨ugli, H.: From topological knowledge to geometrical map. In: Proceedings of the 3rd IFAC Symposium on Intelligent Autonomous Vehicles (IAV 1998) (1998) 21. Volta, F.S., Pollock, D.W., Timpf, S., Egenhofer, M.J.: A Conceptual Model of Wayfinding Using Multiple Levels of Abstraction. In: Theories and Methods of Spatio-Temporal Reasoning in Geographic Space, pp. 348–367 (1992) 22. Voudouris, C., Tsang, E.: Guided local search and its application to the travelling salesman problem. European Journal of Operational Research 113, 469–499 (1999) 23. Wallner, F., Kaiser, M., Freidrich, H., Dillman, R.: Integration of topological and geometrical planning in a learning mobile robot. In: IROS 1994. Proceedings of the IEEE/RSJ/GI International Conference on Intelligent Robots and Systems. Advanced Robotic Systems and the Real World (1994) 24. Zajonc, R.: Feeling and thinking: Preferences need no inferences. American Psychologist 35 (1980)

7 An Exploration of Online Parallel Learning in Heterogeneous Multi-robot Swarms Jim Pugh and Alcherio Martinoli ´ Distributed Intelligent systems and Algorithms Laboratory, Ecole Polytechnique F´ed´erale de Lausanne, 1015 Lausanne, Switzerland {jim.pugh,alcherio.martinoli}@epfl.ch

Abstract. Designing effective behavioral controllers for mobile robots can be difficult and tedious; this process can be circumvented by using unsupervised learning techniques which allow robots to evolve their own controllers online in an automated fashion. In multi-robot systems, robots learning in parallel can share information to dramatically increase the evolutionary rate. However, manufacturing variations in robotic sensors may result in perceptual differences between robots, which could impact the learning process. In this chapter, we explore how varying sensor offsets and scaling factors affects parallel swarm-robotic learning of obstacle avoidance behavior using both Genetic Algorithms and Particle Swarm Optimization. We also observe the diversity of robotic controllers throughout the learning process using two different metrics in an attempt to better understand the evolutionary process.

7.1 Introduction Designing even simple behaviors for robots that are efficient and robust can be very difficult for humans; it is often not hard to implement a rudimentary controller that accomplishes the task, but achieving near-optimal performance can be very challenging, especially for miniature robotic platforms with severe hardware and computational limitations. Unsupervised robotic learning allows for automated design of efficient, robust controllers, which saves much design time and effort. Unsupervised learning is also useful for allowing robots to adapt to situations where the task/environment is unknown beforehand or is constantly changing. Genetic Algorithms (GAs) are a very common method of accomplishing machine learning and optimization. Candidate solutions to a problem are modeled as members of a population, and breeding (selection and crossover) and mutation are applied to “parents” (high performing solutions) in the population to generate “children” (new candidate solutions). GA can be used to shape an Artificial Neural Network (ANN) controller by using the parameter set as the weights, and the evaluative function as a measure of the performance of a desired robot behavior. Particle Swarm Optimization (PSO) is a promising new optimization technique which models a set of potential problem solutions as a swarm of particles moving about in a virtual search space. The method was inspired by the movement of flocking birds and their interactions with their neighbors in the group. PSO achieves optimization using three primary principles: evaluation, where quantitative fitness can be determined D. Liu, L. Wang, and K.C. Tan (Eds.): Design and Control of Intel. Robotic Systems, SCI 177, pp. 133–151. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com 

134

J. Pugh and A. Martinoli

for some particle location; comparison, where the best performer out of multiple particles can be selected; and imitation, where the qualities of better particles are mimicked by others. The algorithm can also be used to evolve ANN robotic controllers. In robotic learning, in order to evaluate a candidate controller solution, a robot must run the controller for some period of time (typically from several seconds to several minutes) and observe how well it performs over that duration according to a preestablished metric (fitness function). The computational and temporal resources needed for this evaluation are drastically higher than those needed for the processing of the learning algorithm itself in almost all cases (less than 1% for the case study presented here). Therefore, robotic learning can be considered to be a very expensive optimization problem, and speed-up efforts should be focused on decreasing the number/length of evaluations rather than the internal workings of the learning technique. Both GA and PSO use groups of interacting virtual agents in order to achieve their optimization. In distributed robotics, groups of robots interact to accomplish their goals. It is therefore possible to implement these algorithms in a parallel distributed fashion for learning in multi-robot systems. Each robot is responsible for several virtual agents, which it evaluates at each iteration. After each set of evaluations, the robots communicate to share the fitness information needed to progress to the next iteration of the algorithm. By running the algorithms in this fashion, we need no external supervisor to oversee the learning process, and the speed of learning is significantly improved, as many robots evaluating in parallel increase the rate of candidate solution evaluation and therefore decrease the total learning time. On real robots, sensors and actuators may have slightly different performances due to variations in manufacturing. As a result, multiple robots of the same model may actually perceive and interact with their shared environment differently, creating a physically heterogeneous swarm. While robotic controller evaluations are inherently stochastic due to partial, noisy environmental perception by individual robots, this heterogeneity may add a systematic bias between agents. This is a crucial difference; partial perception and noise can be overcome by information exchange between robots (i.e. shared evaluation measures of a controller between multiple robots will give a better estimate of its true performance), while sharing information between robots with different systematic biases may actually hinder the learning process, since biased results from other robots could cause a robot to incorrectly adapt its behavior. The result may be increased difficulty in evolving effective behavioral controllers. In order to model swarm heterogeneity, simulations of swarm-robotic learning can include fixed random variations in, for example, the sensitivity and offsets of on-board sensors. This should reflect a more realistic learning scenario, particularly for miniature robots where low-cost sensors often exhibit higher levels of manufacturer variation (often on the order of 10% or more). In this chapter, we explore the efficacy of parallel robotic learning using GA or PSO on swarms of heterogeneous robots with sensor variations. We observe the diversity of the GA and PSO populations in different instances to attempt to gain insight into the evolutionary process. Section 7.2 provides some background on GA, PSO, unsupervised robotic learning, and multi-robot learning. Section 7.3 details our robotic learning case study and gives results in the case of a homogenous robot swarm. Section 7.4 explores the impact of sensor offsets on the learning process, while section 7.5 studies

7 An Exploration of Online Parallel Learning

135

the impact of different sensor scaling factors. In section 7.6, we analyze the diversity of the GA and PSO populations throughout the learning process in order to determine the cause of differences between the algorithms with and without sensor variations. In section 7.7, we discuss the implications of our results, and section 7.8 concludes and provides outlook on future work.

7.2 Background Genetic algorithms were originally developed in the 1960s by John Holland. The algorithms are inspired by evolution, where the fittest members of a population tend to reproduce more often than the less fit members. Candidate solutions are modeled as a population of “chromosomes”. At each iteration of the algorithm, a new population is generated from the previous one. Selection of the parents of the new generation is implemented using one or more of several schemes, such as elitism (using only the top performing members of the population), Roulette Wheel sampling (stochastically choosing parents with weight proportional to performance), and rank selection (ranking chromosomes from best to worst and stochastically choosing parents with weight proportional to their rank). After parents have been chosen, crossover between the parents can occur with some probability (each chromosome is split into multiple pieces, and children use some parts from one parent and some parts from the other). This allows positive aspects from different chromosomes to be merged into a single chromosome. Last, mutation is applied, where each element of the chromosome may have its value randomly changed with some probability. This provides a random local search, which allows solutions to continue to improve beyond the genetic diversity that was available in the original population ([8], [17]). The original PSO method was developed by James Kennedy and Russel Eberhart [5], [10]. Every particle in the population begins with a randomized position (xi, j ) and randomized velocity (vi, j ) in the n-dimensional search space, where i represents the particle index and j represents the dimension in the search space. Candidate solutions are optimized by flying the particles through the virtual space, with attraction to positions in the space that yielded the best results. Each particle remembers the position at which it achieved its highest performance (x∗i, j ). Each particle is also a member of some neighborhood of particles, and remembers which particle achieved the best overall position in that neighborhood (given by the index i ). This neighborhood can either be a subset of the particles (local neighborhood), or all the particles (global neighborhood). For local neighborhoods, the standard method is to set neighbors in a pre-defined way (such as using particles with the closest array indices modulo the size of the population as neighbors, henceforth known as a “ring topology”) regardless of the particles’ positions in the search space. The equations executed by PSO at each step of the algorithm are vi, j = w · vi, j + pw · rand() · (x∗i, j − xi, j ) + nw · rand() · (x∗i, j − xi, j ) xi, j = xi, j + vi, j

(7.1) (7.2)

where w is the inertia coefficient which slows velocity over time, pw is the weight given to the attraction to the previous best location of the current particle and nw is the weight

136

J. Pugh and A. Martinoli

given to the attraction to the previous best location of the particle neighborhood. rand() is a uniformly-distributed random number in [0, 1]. PSO has been shown to perform as well as or better than GA in several instances. Eberhart and Kennedy found PSO performs on par with GA on the Schaffer f6 function [5, 10]. In work by Kennedy and Spears [11], a version of PSO outperforms GA in a factorial time-series experiment. Fourie showed that PSO appears to outperform GA in optimizing several standard size and shape design problems [7]. Unsupervised learning describes learning scenarios where there is no external entity which decides upon the training set inputs for the learning agent(s). Rather, inputs are generated dynamically as the agents interact with their environment. This is as opposed to supervised learning, where the inputs are generated/collected first and then used repeatedly. In supervised learning, the accuracy of the system at each iteration is usually decided by an external “teacher” evaluating the system output. The pre-defined inputs are split into two separate sets, one for training the system and the other for testing the performance. Supervised learning tends to be easier than unsupervised, as the data does not change between iterations of the algorithm and can be preselected to avoid using unusual or particularly noisy data points. However, supervised learning is not possible in situations where the input data to the system depends on the current state of the learning agent; this is the case for online robotic learning, since the robot’s movements affect what its sensors will perceive. Evolutionary algorithms have been used extensively for unsupervised learning of robotic behavior. A good survey of the work is given in [14] and more recently in [21]. More specifically, standard GA has been shown to be effective in evolving simple robotic controllers [6], and modified noise-resistant versions of both GA and PSO were shown to achieve very good performance on simulated unsupervised robotic learning, outperforming the standard versions of the algorithms [22]. Multi-robot learning has been used and explored in various ways; a survey of work (including learning on other multi-agent systems) can be found in [26]. Matari´c studied mechanisms to encourage individual agents in a group to act in ways to help the group performance [13]. Multi-robot learning using several methods in a wide variety of scenarios has been explored [2], [25]. Specialization in multi-agent systems using reinforcement learning was studied in [19]. Techniques for increasing individual learning speed via multi-robot learning were studied in [9] and [15]. A modified version of a genetic algorithm has been embedded onto a 2-robot system to allow for distributed parallel learning [20]. Pugh and Martinoli found that both GA and PSO could be used for effective distributed parallel multi-robot learning in [23] and applied to heterogeneous robot groups in [24].

7.3 Parallel Swarm-Robotic Learning Case Study: Obstacle Avoidance Following on the work presented in [23] and [24], we use the case study of obstacle avoidance for our swarm-robotic learning task.

7 An Exploration of Online Parallel Learning

Fig. 7.1. Flowcharts for noise-resistant GA (left) and PSO (right)

137

138

7.3.1

J. Pugh and A. Martinoli

Experimental Setup

For the learning techniques, we use the noise-resistant GA and PSO algorithms from [23]. GA uses elitism to select the best half of the population as the parent set, and then applies Roulette Wheel sampling to replenish the missing chromosomes. PSO uses a local neighborhood in a ring topology with one neighbor on each side. At every iteration, these algorithms reevaluate their previous best locations and parent sets for PSO and GA, respectively, combining the new fitness value with previous ones to get a more accurate measure of the actual fitness. Although this requires twice as many fitness evaluations at each iteration as their standard counterparts, this technique prevents noisy fitness evaluations from severely disrupting the learning process and gives much better results given the same number of evaluations of candidate solutions. Flowcharts of both algorithms are shown in Fig. 7.1. For both PSO and GA, initial population member elements are randomly generated in the range [−20, 20] but allowed to change to any value during evolution. Velocity in PSO is also randomly initialized in the range [−20, 20] but prevented from ever going outside this range. The parameters for the algorithms are given in Table 7.1. Table 7.1. GA and PSO Parameters for Unsupervised Learning GA

PSO

Population Size

20

Population Size

20

Mutation Probability

0.15

pw

2.0

Crossover Probability

0.2

nw

2.0

Mutation Range

[-20.0, 20.0]

w

0.6

We use Webots, a realistic simulator, for our robotic simulations [16], using 20 epuck1 robots [4] (this is as opposed to the Khepera robot [18] used in some previous experiments). The robot(s) operate in a 2.0 m x 2.0 m square arena with no additional obstacles (see Fig. 7.2 left). The robotic controller is a single-layer discrete-time artificial neural network of two neurons, one for each wheel speed, with sigmoidal output functions. The inputs are the eight infrared proximity sensors, as well as a recursive connection from the previous output of the neuron, lateral inhibitions and bias values (see Fig. 7.2 right), giving us 22 weights total. Sensors have a maximum range of 5.0 cm, and sensor output varies linearly from 0 at maximum range to 1 at minimum range (0.0 cm) with 3% noise. Slip noise of 10% is applied to the wheel speed. The time step for neural updates is 128 ms. We use the fitness function used originally in [6] and again in [23]. The fitness function is given by: √ (7.3) F = V · (1 − Δ v) · (1 − i) 0 ≤ V ≤ 1, 1

http://www.e-puck.org

0 ≤ Δ v ≤ 1,

0≤i≤1

(7.4)

7 An Exploration of Online Parallel Learning

139

Fig. 7.2. Left: Robot arena with e-puck robots. Right: Depiction of the artificial neural network used for the robot controller. Curved arrows are recurrent connections and lateral inhibitions.

where V is the average absolute wheel speed of both wheels, Δ v is the average of the difference between the wheel speeds, and i is the average activation value of the most active proximity sensor over the evaluation period. These factors reward robots that move quickly, turn as little as possible, and spend little time near obstacles, respectively. The terms are normalized to give a maximum fitness of 1. The evaluation period of the fitness tests for these experiments is 240 steps, or approximately 30 seconds. Between each fitness test, the position and bearing of the robots are randomly set by the simulator to ensure the randomness of the next evaluation. During evolution, each of the 20 robots is responsible for a single member of the GA/PSO population (i.e. they must evaluate that member at each iteration and communicate the resulting fitness measure). With 100 iterations of the learning algorithms, this results in a total simulated learning time of approximately 1 hour 40 minutes (100 iterations comprised of 2 evaluations each lasting 30 seconds). 7.3.2

Results

The average performance over 100 runs of the best evolved controllers in the population can be seen in Fig. 7.3 left. As previously found, PSO is able to achieve superior performance, as GA occasionally produces poorer solutions. The progress of evolution can be seen in Fig. 7.3 right. While GA improves faster in the first few iterations, it quickly changes to a gradually increasing plateau, while PSO continues improving throughout the entire process. Two primary types of obstacle avoidance controllers were observed in successful evolutionary runs of both GA and PSO. In the absence of obstacles, both types caused the robot to move ahead at full speed. The first controller consistently made the robot perform either a right turn or a left turn at the activation of any of its front sensors, thereby avoiding the obstacle in its path. The second controller used the recurrent neural connection to achieve a multi-state behavior; the robot would run forward until the

140

J. Pugh and A. Martinoli 1

Fitness

0.8 0.6 0.4 0.2 0

GA

PSO

0.7

Fitness

0.6

GA PSO

0.5 0.4 0.3 0.2 0

20

40 60 Iteration

80

100

Fig. 7.3. Left: Fitness of best evolved controllers in population after evolution averaged over 100 runs for GA and PSO. Error bars represent standard error. Right: Average fitness of population members throughout evolution averaged over 100 runs for GA and PSO.

front sensors detected an obstacle, at which point it would reverse its wheel speed and run backwards at full speed. When the rear sensors were activated, the robot would again resume forward movement, resulting in a back-and-forth behavior which somewhat resembles bouncing. Depending upon the environment, either of these two types of behavior might be preferable to the other.

7.4 Heterogeneity from Sensor Offsets One potential variation between sensors is different offset values (i.e. if some sensor A detects value x, some other sensor B would detect x + a, where a is the offset value). We can incorporate this variation in our simulation using sensor values of: vi = vi + ai

(7.5)

where vi is the original sensor value, vi is the offset sensor value used by the robot controller, and ai is a Gaussian offset value randomly generated at the start of the simulation with mean of 0 and standard deviation of s.

7 An Exploration of Online Parallel Learning

7.4.1

141

Experimental Setup

We repeat the experiments from section 7.3 using sensor offsets with standard deviation of 0, 0.05, 0.1, 0.2, and 0.5 (corresponding to 0%, 5%, 10%, 20%, and 50% of the maximum sensor value, respectively). In order to accurately evaluate the performance of the controller, the original sensor values are used for the proximity term of the fitness calculation. The same parameters are used for both GA and PSO. 7.4.2

Results

The average performance over 100 runs of the best evolved controllers in the population for different levels of sensor offset variation can be seen in Fig. 7.4 left. While the performance is not highly impacted for a standard deviation of up to 0.1, a significant decrease can be seen at 0.2, and a standard deviation of 0.5 yields quite poor performance. This suggests that neither GA nor PSO are affected by minor variations, but major variations could cause some sensors to be always/never perceived as active, which would cause very poor performance. The same trend can be seen in both GA and PSO, though it appears as though GA performs relatively better with large offsets. 0.8 GA PSO

Fitness

0.6

0.4

0.2

0

0

0.05

0.1

0.2

0.5

Standard Deviation of Sensor Offset

0.8

GA PSO

Fitness

0.6

0.4

0.2

0

0 0.05 0.1 0.2 0.5 Standard Deviation of Sensor Offset

Fig. 7.4. Fitness of best evolved controllers in population after evolution averaged over 100 runs for GA and PSO with different standard deviations (s) of sensor offset. Results shown for final evaluation with both heterogeneous (left) and homogenous (right) robots. Error bars represent standard error.

142

J. Pugh and A. Martinoli

0.5

Fitness

0.4 0.3 s = 0.0 s = 0.05 s = 0.1 s = 0.2 s = 0.5

0.2 0.1 0 0

20

40 60 Iterations

80

100

0.6

Fitness

0.5 0.4 0.3 s = 0.0 s = 0.05 s = 0.1 s = 0.2 s = 0.5

0.2 0.1 0 0

20

40 60 Iterations

80

100

Fig. 7.5. Average fitness of population members throughout evolution averaged over 100 runs for GA (left) and PSO (right) with different standard deviations (s) of sensor offset

While the previous results indicate how well evolved controllers perform on their heterogeneous robot group, we can evaluate the general performance of the controllers evolved on a heterogeneous swarm by testing them on a homogenous set of robots without sensor variations (see Fig. 7.4 right). We observe that while performance still decreases as sensor offsets increase, the effect is noticeably less. This indicates that the previously observed performance drop is due to a combination of suboptimal controllers from impaired learning and degraded performance from operating on a heterogeneous swarm. The relative performance of GA versus PSO is even higher for large sensor offsets, suggesting that learning is less impacted when using GA than when using PSO. The performance of GA throughout learning with different offset levels can be seen in Fig. 7.5 left. We notice that the trend of rapid initial improvement followed by a gradually increasing plateau is not affected, but rather performance seems to be scaled down as the standard deviation of the offset grows. A similar trend can be observed for the PSO results in Fig. 7.5 right.

7.5 Heterogeneity from Sensor Scaling Another variation that can occur between sensors is sensitivity. One sensor may be more sensitive to input, which causes its output to vary more quickly than others. We can model this in our simulation using the equation:

7 An Exploration of Online Parallel Learning

vi = mi vi

143

(7.6)

where vi is the original sensor value, vi is the offset sensor value used by the robot controller, and mi is a Gaussian scaling factor randomly generated at the start of the simulation with mean of 1 and standard deviation of s. 7.5.1

Experimental Setup

We repeat the experiments from section 7.3 using sensor scaling factors with standard deviation of 0, 0.05, 0.1, 0.2, and 0.5 (and mean of 1). In order to accurately evaluate the performance of the controller, the original sensor values are used for the proximity term of the fitness calculation. The same parameters are used for both GA and PSO. 7.5.2

Results

The average performance over 100 runs of the best evolved controllers in the population for different levels of sensor scaling variation can be seen in Fig. 7.6 left. The performance here is impacted less than for the case of sensor offsets; good performance is maintained even for a scaling factor standard deviation of 0.5, though we do see

0.8

GA PSO

Fitness

0.6 0.4 0.2 0 0 0.05 0.1 0.2 0.5 Standard Deviation of Sensor Scaling Factor 0.8

GA PSO

Fitness

0.6 0.4 0.2 0

0 0.05 0.1 0.2 0.5 Standard Deviation of Sensor Scaling Factor

Fig. 7.6. Fitness of best evolved controllers in population after evolution averaged over 100 runs for GA and PSO with different standard deviations (s) of sensor scaling factors. Results shown for final evaluation with both heterogeneous (left) and homogenous (right) robots. Error bars represent standard error.

144

J. Pugh and A. Martinoli

0.55 0.5

Fitness

0.45 0.4

s = 0.0 s = 0.05 s = 0.1 s = 0.2 s = 0.5

0.35 0.3 0.25 0.2 0

20

40 60 Iterations

80

100

0.6

Fitness

0.5 0.4

s = 0.0 s = 0.05 s = 0.1 s = 0.2 s = 0.5

0.3 0.2 0.1 0 0

20

40 60 Iterations

80

100

Fig. 7.7. Average fitness of population members throughout evolution averaged over 100 runs for GA (left) and PSO (right) with different standard deviations (s) of sensor scaling factors

an observable decrease here. The cause of this may be that, although the sensitivity is changed, sensors still react is the same way (i.e. higher for close proximity, lower for far proximity), and therefore the performance of the controller is not significantly impacted. We again see the same trend in both GA and PSO, and GA again performs relatively better with large scaling variations. Observing the performance on a homogenous set of robots (see Fig. 7.6 right), we see that the actual quality of the evolved controller has not been at all affected by sensor scaling variation in almost all cases. The one minor exception is for controllers evolved using GA with a scaling factor standard deviation of 0.5, where we see a very slight performance decrease. Following the theory suggested above, it could be that because sensors respond in a similar way in spite of scaling, the same obstacle avoidance approaches are selected by the learning algorithms, which results in unaffected performance when using a homogenous set of robots. The performance of GA throughout learning with different scaling levels can be seen in Fig. 7.7 left. As a result of the better performance, there is not such an obvious decrease as the standard deviation of the scaling grows, with the exception of a standard deviation of 0.5 where there is a marked performance drop. There is only very minor differences between the curves for PSO for different sensor scaling variation levels (see Fig. 7.7 right).

7 An Exploration of Online Parallel Learning

145

7.6 Diversity in Evolving Robot Swarms Although PSO was able to achieve superior performance to GA in evolving robotic controllers in [22] and [23], the reasons for this were never apparent. A major drawback to using stochastic multi-agent algorithms such as GA and PSO is that they lack transparency about what exactly is happening during the evolutionary process and why they achieve the results that they do. In an attempt to shed some light on this issue, we analyze the diversity of the GA and PSO populations through their evolution using two different metrics. 7.6.1

Euclidean Diversity

For our first diversity metric, we choose one of the most simple: Euclidean distance. We consider each candidate solution as a location in a virtual search space. The pairwise diversity between two members of a population is therefore given by:  (7.7) d(a, b) = ∑(ai − bi )2 i

where d(a, b) is the pairwise diversity between member a and member b, and xi is element i of member x. The diversity of the entire population is given by the average pairwise diversity of all pairs of members in the population, or:   1 (7.8) Deu = ∑ d(a, b) N(N − 1) ∑ a b=a where a and b are members of the population, and N is the total size of the population. 7.6.2

Entropic Diversity

For our second diversity metric, we make use of a diversity measure dubbed the “hierarchic social entropy”, first proposed in [3] and later used in [12]. For this metric, population members are divided into clusters, where clusters are defined as maximal sets of “connected” population members. Two population members are considered directly connected if the measure of some distance metric between them (Euclidean distance in this case) is less than some threshold value h, and two population members are considered connected if there exists a set of direct connections linking them. Each cluster i can then be given a “probability” pi which is calculated as follows: pi =

ni N

(7.9)

where ni is the number of members in cluster i and N is the total population size. Because clusters are maximal, an individual population member will appear in only a single cluster, and therefore:

∑ pi = 1 i

(7.10)

146

J. Pugh and A. Martinoli

We can therefore calculate the entropy of these probabilities as: H = − ∑ pi log pi

(7.11)

i

resulting in an entropy of zero for a single cluster and increasing entropy as the number of clusters increases. To calculate the hierarchic social entropy (or Entropic diversity), we integrate entropy over all values of the threshold distance h:  ∞

Den =

H(h)dh

(7.12)

0

where H(h) is the entropy of cluster probabilities as a function of the threshold distance h. While both Euclidean and Entropic diversities provide a measure of the distance between population members (high distance implies more clusters for the same value of h which yields higher entropy), only Entropic diversities directly reflects whether or not there exist compact groups of members in the virtual search space (which would suggest similar behaviors among these members). We measure diversity using both metrics throughout evolution for parallel learning using GA and PSO for homogenous robotic swarms and for heterogeneous robotic swarms with sensor offsets. We keep the same scenario and parameters as used previously. In order to make valid comparison, we normalize diversity measures to the diversity of the population at the first iteration (i.e. diversity always begins with value1). 7.6.3

Results

The progress of population diversity throughout evolution for GA and PSO on a homogenous swarm can be seen in Fig. 7.8. For both metrics, the diversity in GA drops quickly in the initial iterations and maintains a stable lower value for the rest of the evolution. In contrast, the diversity of PSO actually increases initially and maintains a higher value. This is evidence for the hypothesis given in [23] that by very quickly converging to a good candidate solution, GA sacrifices the ability to further explore for better possible solutions, which causes it to occasionally converge to a mediocre final solution. On the other hand, by maintaining high diversity throughout evolution, PSO is able to continuously discover new and better solutions and continue improving throughout the entire evolution. This also demonstrates why PSO is better able to maintain good performance with a smaller population size, while GA does very poorly with small populations since it requires enough genetic diversity to already include a good solution. Comparing Euclidean versus Entropic diversity, we see that while the trends are the same, the diversity increase in PSO is consistently less than for Entropic diversity (the difference between diversity measures, Deu − Den , can be seen in Fig. 7.9). This suggests that while PSO particles do indeed spread out in the virtual search space, they simultaneously form clusters, with some particles maintaining close proximity to some others. This follows as a result of the local neighborhood topology used by PSO, as neighboring particles should often assume similar values and therefore have a small Euclidean distance between them. For GA, the initial diversity decrease is greater for Entropic diversity than Euclidean, indicating that individuals are simultaneously converging and forming clusters. However, the difference in diversity plateaus and actually

7 An Exploration of Online Parallel Learning

147

Euclidean Diversity

1.5

1

GA PSO

0.5

0 0

20

40 60 Iterations

80

100

80

100

Entropic Diversity

1.5

1

GA PSO

0.5

0 0

20

40 60 Iterations

Fig. 7.8. Diversity of population members throughout evolution averaged over 100 runs for GA and PSO for a homogenous robot swarm for both Euclidean (left) and Entropic (right) diversity measures

Diversity Difference

0.4

0.3 GA PSO

0.2

0.1

0 0

20

40 60 Iterations

80

100

Fig. 7.9. Euclidean Diversity minus Entropic Diversity of population members throughout evolution averaged over 100 runs for GA and PSO for a homogenous robot swarm

148

J. Pugh and A. Martinoli

s = 0.0 s = 0.05 s = 0.1 s = 0.2 s = 0.5

Euclidean Diversity

1 0.9 0.8 0.7 0.6 0

20

40 60 Iterations

100

s = 0.0 s = 0.05 s = 0.1 s = 0.2 s = 0.5

1 Entropic Diversity

80

0.9 0.8 0.7

0

20

40 60 Iterations

80

100

Fig. 7.10. Diversity of population members throughout evolution averaged over 100 runs for GA with different standard deviations (s) of sensor offset for both Euclidean (left) and Entropic (right) diversity measures

decreases slightly in the latter portion of evolution, suggesting that all further diversity decreases are the result of convergence rather than clustering. We can also compare the progression of diversity for evolution of heterogeneous robotic swarms with sensor offsets using GA (see Fig. 7.10) and PSO (see Fig. 7.11). Both diversity metrics yield very similar results here. In the case of GA, sensor offset variation seems to have little impact on diversity, with much overlap between diversity measures for different standard deviations throughout the learning process. For PSO, there is no significant change in diversity for low sensor offset variations; for higher variations, the diversity grows at the same pace initially but achieves higher values towards the end of the evolution. This may be caused by less useful information sharing between robots due to major sensorial differences, promoting individual reliance. This might prevent convergence of particles, which would increase the diversity in the latter evolutionary stages.

7.7 Discussion For parallel swarm-robotic learning of obstacle avoidance behavior, offset variation seemed to have a much higher impact on performance than scaling variation. However, it is highly likely that this result is specific to our case study. As mentioned previously,

7 An Exploration of Online Parallel Learning

149

Euclidean Diversity

1.6 1.5 1.4 s = 0.0 s = 0.05 s = 0.1 s = 0.2 s = 0.5

1.3 1.2 1.1 1 0.9 0

Entropic Diversity

1.6 1.5 1.4

20

40 60 Iterations

80

100

40 60 Iterations

80

100

s = 0.0 s = 0.05 s = 0.1 s = 0.2 s = 0.5

1.3 1.2 1.1 1 0.9 0

20

Fig. 7.11. Diversity of population members throughout evolution averaged over 100 runs for PSO with different standard deviations (s) of sensor offset for both Euclidean (left) and Entropic (right) diversity measures

because of the linear manner in which proximity sensors are used (i.e. higher - there is an obstacle, lower - there is no obstacle), scaling does not have much effect, while offset variation could potentially confuse the sensor as to its current state. One can envision other scenarios where this is not the case: if a sensor has some intermediate optimum value (such as would be the case in wall-following or flocking behavior), both offset and scaling would disrupt its reading; comparing subsequent sensor readings would remove any offset bias but still be affected by scaling. Therefore, parallel learning in heterogeneous robotic swarms needs to be tested on many other case studies before any such generalizations can be made. Although not shown here, we also performed several parallel swarm-robotic learning tests using PSO without any velocity limit (w was kept at 0.6). While the performance of the resulting controllers was not significantly different, the diversity of the population exploded, growing exponentially until the end of the evolution (it was not uncommon to have a final Euclidean diversity on the order of 100,000). The plateau previously observed is therefore only caused by a hard velocity bound. We believe the reason for this explosive behavior is the same as the reason for the low impact of variations in sensor scaling; ANNs continue to function nearly the same if every weight is multiplied by a constant factor. The only change will be the sensitivity of the sigmoid output function, which will approach a step function as the neural weights increase.

150

J. Pugh and A. Martinoli

7.8 Conclusion and Outlook We have explored some of the effects of robot heterogeneity on parallel swarm-robotic learning. In the case of evolving obstacle avoidance, both Genetic Algorithms and Particle Swarm Optimization were able to withstand small variations in sensor offsets and large variations in sensor scaling factors, while showing poor performance with high offset variations. By observing population diversity throughout evolution, we discovered that PSO maintains much higher diversity, which could explain its superior ability to GA in avoiding local optima. Though not explored here, there could be ways in which heterogeneous robotic swarms could identify differences between robots by sharing and comparing the observed fitness values of common controllers. This information could be used to help avoid the fitness degradation previously observed and might even be useful for deliberately pursuing specialization within the swarm in scenarios not limited to evolving one single controller. We have shown that tracking population diversity can yield some insight into the progress of the evolving swarm. One could therefore imagine that these learning algorithms could be improved if they were to track their own diversity. A simple example would be allowing GA to scale the values of some of its population members if the diversity were too low. While calculating the diversity of the entire population requires a global manager, it might be feasible for individual robots to acquire a good estimation using local sensing of nearby teammates, which could allow this change to be implemented in a distributed fashion.

Acknowledgements This work has been mainly sponsored by a Swiss NSF grant (contract Nr. PP002116913).

References 1. Antonsson, E.K., Zhang, Y., Martinoli, A.: Evolving Engineering Design Trade-Offs. In: Proc. of the ASME Fifteenth Int. Conf. on Design Theory and Methodology, Chicago, IL, USA, paper No. DETC2003/DTM–48676 (September 2003) 2. Balch, T.: Behavioral diversity in learning robot teams. PhD Thesis, College of Computing, Georgia Institute of Technology (1998) 3. Balch, T.: Measuring robot group diversity. In: Balch, T., Parker, L.E. (eds.) Robot teams: From diversity to polymorphism, pp. 93–135 (2002) 4. Cianci, C., Raemy, X., Pugh, J., Martinoli, A.: Communication in a swarm of miniature robots: The e-puck as an educational tool for swarm robotics. In: S¸ahin, E., Spears, W.M., Winfield, A.F.T. (eds.) SAB 2006 Ws 2007. LNCS, vol. 4433, pp. 103–115. Springer, Heidelberg (2007) 5. Eberhart, R., Kennedy, J.: A new optimizer using particle swarm theory. In: Proc. of the Sixth Int. Symposium on Micro Machine and Human Science, MHS 1995, October 4–6, pp. 39–43 (1995)

7 An Exploration of Online Parallel Learning

151

6. Floreano, D., Mondada, F.: Evolution of Homing Navigation in a Real Mobile Robot. IEEE Transactions on Systems, Man and Cybernetics, Part B 26(3), 396–407 (1996) 7. Fourie, P.C., Groenwold, A.A.: The particle swarm optimization algorithm in size and shape optimization. Struct. Multidisc. Optim. 23, 259–267 (2002) 8. Goldberg, D.E.: Genetic Algorithms in Search, Optimization & Machine Learning. AddisonWesley, Reading (1989) 9. Kelly, I.D., Keating, D.A.: Faster learning of control parameters through sharing experiences of autonomous mobile robots. Int. Journal of System Science 29(7), 783–793 (1998) 10. Kennedy, J., Eberhart, R.: Particle swarm optimization. In: Proc. of the IEEE Int. Conf. on Neural Networks, vol. 4, pp. 1942–1948 (November/December 1995) 11. Kennedy, J., Spears, W.M.: Matching algorithms to problems: an experimental test of the particle swarm and some genetic algorithms on the multimodal problem generator. In: Proc. of IEEE Int. Conf. on Evolutionary Computation, Anchorage, pp. 78–83 (May 1998) 12. Li, L., Martinoli, A., Abu-Mostafa, Y.: Learning and Measuring Specialization in Collaborative Swarm Systems. Adaptive Behavior 12(3-4), 199–212 (2004) 13. Matari´c, M.J.: Learning to Behave Socially. In: Proc. of the 3rd Int. Conf. on Simulation and Adaptive Behaviors - From animals to animats, vol. 3, pp. 453–462 (1994) 14. Matari´c, M.J., Cliff, D.: Challenges in evolving controllers for physical robots. Robot. and Autonomous Sys. 19(1), 67–83 (1996) 15. Matari´c, M.J.: Learning in behavior-based multi-robot systems: Policies, models, and other agents. Special Issue on Multi-disciplinary studies of multi-agent learning, Ron Sun (ed.) Cognitive Systems Research 2(1), 81–93 (2001) 16. Michel, O.: Webots: Professional Mobile Robot Simulation. Int. J. of Advanced Robotic Systems 1, 39–42 (2004) 17. Mitchell, M.: An Introduction to Genetic Algorithms. MIT Press, Cambridge (1996) 18. Mondada, F., Franzi, E., Ienne, P.: Mobile robot miniaturisation: A tool for investigation in control algorithms. In: Proc. of the Third Int. Symp. on Experimental Robotics, Kyoto, Japan, pp. 501–513 (October 1993) 19. Murciano, A., Mill´an, J.R., Zamora, J.: Specialization in multi-agent systems through learning. Behavioral Cybernetics 76, 375–382 (1997) 20. Nehmzow, U.: Learning in multi-robot scenarios through physically embedded genetic algorithms. In: Proc. of the 7th Int. Conf. on the Simulation of Adaptive Behavior: From animals to animats, pp. 391–392 (2002) 21. Nolfi, S., Floreano, D.: Evolutionary Robotics: The Biology, Intelligence, and Technology. MIT Press, Cambridge (2000) 22. Pugh, J., Zhang, Y., Martinoli, A.: Particle swarm optimization for unsupervised robotic learning. In: IEEE Swarm Intelligence Symposium, Pasadena, CA, pp. 92–99 (June 2005) 23. Pugh, J., Martinoli, A.: Multi-Robot Learning with Particle Swarm Optimization. In: Int. Conf. on Autonomous Agents and Multiagent Systems, Hakodate, Japan, May 8–12, pp. 441–448 (2006) 24. Pugh, J., Martinoli, A.: Parallel Learning in Heterogeneous Multi-Robot Swarms. In: IEEE Congress on Evolutionary Computation, Singapore, September 25-28, pp. 3839–3846 (2007) 25. Stone, P.: Layered Learning in Multi-Agent Systems. PhD Thesis, School of Computer Science, Carnegie Mellon University (1998) 26. Stone, P., Veloso, M.: Multiagent Systems: A Survey from a Machine Learning Perspective. Autonomous Robots 8(3), 345–383 (2000)

8 Robot Control in Dynamic Environments Using Memory-Based Learning Patrick M. McDowell1 , Brian S. Bourgeois2, and Frederick E. Petry2 1 2

Southeastern Louisiana University, Computer Science Department, Hammond, LA 70402 Naval Research Laboratory, Stennis Space Center, MS, USA 39529

Abstract. Learning systems used on robots typically require a-priori knowledge in the form of environmental models or trial-and-error approaches requiring a robot to physically execute multitudes of trial solutions. Neither approach is very suitable for dynamic unstructured environments in which a robot is sent to explore and gather information prior to human entry. This chapter presents a new approach, ‘memory-based learning’, in which a robot is provided with an initial baseline behavior whose performance is linked with a metric explicitly defined by a function whose arguments are sensory inputs and resulting robot actions. A neural network, using sensor inputs and action outputs, has been chosen as the basic controller building block for behaviors as it is very amendable to rapid in-situ generation and adaptation. The use of a neural network controller for a maneuvering task using electronic ‘ears’ for sensory input is demonstrated in this work, and genetic algorithms are shown to be an effective method for rapidly developing the network weights and transfer functions. The concept of memory based learning is introduced and a construct for representing coupled sense/action information is presented and demonstrated using both simple reactive and memory based controllers, whose performance is demonstrated through simulation studies and on land robots.

8.1 Introduction For a robot to work autonomously in a complex, changing environment it must be able to adapt dynamically. The work discussed in this paper develops an approach to this complex problem using a sensor input based metric for success and recently collected memories - a temporal series of sensor/action pairs. In essence this scheme monitors the progress of a robot while it carries out a particular task. If the robot is not achieving its goal based on the metric for success, then a learning process will be triggered that uses recently collected sensor/action pairs to train a new controller. The learning is achieved with a neural network controller utilizing a genetic algorithm to develop both the network weights and node transfer functions. This approach is applied to a formation maneuvering problem [3,28] where robots with ‘ears’ listen for a leader robot and attempt to follow. For this application the sensor input is the intensity of the sound in the left and right ear and the action is either turn left, go straight or turn right; the metric of success is to keep the intensity within a certain range and equal in both earscorresponding to the situation where the robot is successfully following the leader. Conventional robotic control follows basically a functional approach [14,18]. A key aspect is the use of a world model which can be updated by sensor inputs. To develop D. Liu, L. Wang, and K.C. Tan (Eds.): Design and Control of Intel. Robotic Systems, SCI 177, pp. 153–170. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com 

154

P.M. McDowell, B.S. Bourgeois, and F.E. Petry

a set of actions to achieve the desired goals the functional approach utilizes a planning module. The planner uses the world model and current sensor inputs to produce the steps toward a goal that constitute the plan of actions. This approach is very brittle especially in its reliance on a pre-developed world model and is very limited in controlling multiple robots. Learning approaches such as those described in this chapter can avoid the drawbacks of a functional approach. In this chapter we first provide some background on Remote Operated Vehicles (ROVs) and Untethered Unmanned Underwater Vehicles (UUVs) by describing their potential applications and issues involved in their operation. The following section discuses the paradigm of recent memories to generate an adequate training set and collecting memories needed to train an initially ‘blank’ controller. Next we describe the neural network controller which is trained using genetic algorithms. In the following sections we explain the design and training of reactive and memory based controllers using recent memories. Results from simulation tests of these controllers are shown in the experimental results section.

8.2 ROV/UUV Background Remote Operated Vehicles (ROVs) are used extensively for salvage operations, ocean floor surveying and numerous inspection activities that support a wide range of underwater commercial activities. In deep water (greater than 1000 ft) an ROV is the platform of choice because of the depth and endurance limitations for human divers. The key disadvantage to an ROV is the requirement for the long tether. Untethered Unmanned Underwater Vehicles (UUVs) have demonstrated the ability to perform deep-water surveys faster and cheaper than towed vessels [9,10]. UUVs have the potential for supplementing and even replacing ROVs for many deep-water operations because of the cost and problems associated with the tether. One promising scenario for the near future is to use a ROV to control several UUVs in a local work area. Table 1 compares key factors of tethered vessels to single and multiple UUVs. A tethered vessel clearly has the advantage of power and endurance, but this advantage can be lost when multiple UUVs are deployed from the same host ship. The factors of deck gear size, launch and recovery time, maneuverability and coverage rate are distinct disadvantages of tethered vessels and directly impact their cost-effectiveness. Accurate positioning of deep-water vessels remains a difficult and expensive issue. At present this can be accomplished with ultra-short baseline systems on GPS equipped ships that remain directly over the deployed vessels [9]. A tether offers a distinct advantage in the areas of communications and intelligence since it provides high-bandwidth instantaneous communications and allows human intelligence to be in the loop for vehicle operation. The communications, intelligence and multi-vessel navigation capabilities required for deploying multiple UUVs are still in active development and of particular interest for our research approach. Figure 8.1 illustrates a UUV task force that could be used to conduct an oceanographic survey [5]. Data of interest in such an operation would include bathymetry,

8 Robot Control in Dynamic Environments Using Memory-Based Learning

155

Table 8.1. Comparison of UUVs and Tethered Vessels for Deep Water Work

Comparison Factors ROV or TOW Single UUV Multiple UUVs Power/endurance + + Deck gear size + + Launch/recovery time + + Maneuverability + + Coverage Rate + ++ Positioning Communications bandwidth + Intelligence + -

Fig. 8.1. A UUV task force concept for a notional oceanographic mission. The blue lines indicate range/bearing positioning links and the red links are range only positioning links. The black lines indicate communication links.

acoustic imagery, wave characterization, sediment type, water salinity and temperature, currents, bioluminescence, water clarity, etc. An important aspect to note is that a survey of this sort could cover large areas (many square miles) and need to be executed in a fairly short period (a few days). Due to the characteristics of the vessels and the sensors involved, high vessel speeds cannot typically be used. Consequently, a large number of vessels are required to accomplish the mission as is shown in the figure. As with most operations employing a large number of assets, it is not economically feasible or desirable for all vessels to have capabilities of the same type and grade. Bathymetry and acoustic imagery for example, require a vessel that can cruise smoothly at fairly low speeds. Sound velocity profiles (salinity and temperature) need a vessel that

156

P.M. McDowell, B.S. Bourgeois, and F.E. Petry

is good at performing vertical excursions. Obstacle sensors are not necessarily required for every vessel; a maximum safe depth can be set based on a priori knowledge of the operation area and a few vessels equipped with look-ahead sonar can be deployed ahead of the task force to ensure safe navigation for all vessels. Close inspection of an object of interest would require a vessel with a hover capability and a high-resolution imaging system. This will in general require an active coordination of a robot team [4]. A command link to the outside world is required for status monitoring and for task force redirection. Instead of equipping every vessel with this capability, this can be served by a vessel that provides a radio/acoustic communications interface to the task force’s Command Vessel (CV) as shown in Figure 8.1. Communications for status monitoring and asset redirection is also required at the task force level, so a communication capability is required between the CV and the rest of the task force. Asset redirection can typically be handled using low-bandwidth communications and the CV could be equipped with a low frequency long-range broadcast acoustic communications system for this task [6,20,26]. Military and commercial applications that would use multiple UUVs share the common problem of positioning and communications. While a commercial application typically has greater flexibility and could put a fixed infrastructure into place or use more ships, these approaches are not always economically desirable. Here a team concept can be used with lower cost UUVs that are co-coordinated to the higher capability UUV leader with an advanced positioning capability. This allows a group travel formation to and from the mission site where the individual UUVs will then carry out their mission specific goals [16,24].

8.3 Memory Based Learning Relevant related work for this topic includes Q-learning, anytime learning and valuedependent learning. Q-learning [17,25,27] is an algorithm that learns optimal control strategies through the exploration of its environment. The main strength of Q-learning is that it allows a robot or agent to operate in its environment without a priori knowledge. While it has been applied successfully to specific game playing tasks, it is best suited for well defined problems with finite state spaces and not for dynamic environments since it must exhaustively test alternatives. Anytime learning [13] allows a robot to learn without requiring it to physically execute many trial solutions. However, anytime learning’s functionality is derived through extensive simulation that requires significant processing power and a priori knowledge of the environment in which the robot will work. The research effort by Rucci et al. [21] developed ’value-dependent learning’ that uses a sensor based approach similar to this research, but requires significant physical repetition by the robot to converge on a solution. The approach described in this chapter is intended to control autonomous robots in unstructured dynamic environments, such as the undersea environment or in a desert. Environments such as these motivate this approach, which focuses on creating a method for the robot to explore its environment in a non-goal-oriented manner so it can learn what actions lead to success. The environmental exploration routines are intended to provide the robot with memories from which it can learn to achieve its goals, using the

8 Robot Control in Dynamic Environments Using Memory-Based Learning

157

reactive and sensor history driven (memory based) learning methods, without having to physically repeat each movement until it reaches a criterion for success. Learning from recent memories is used so that the robot or agent can avoid regimes of directed testing of trial solutions. The approach records a memory consisting of sensor data and associated action values and learns from this which actions helped to optimize goals. By learning from previous experiences the robot can create a “draft” controller that can be iteratively improved as the robot operates in its environment in fulfillment of its goals. Of primary importance in this strategy is for an agent to be able to identify where in the recent memory the robot’s goals were being realized. So the memory must contain sufficient usable examples to learn all the functions needed to realize its goals. If this information is not available, or cannot be generated from what is available, the robot will learn an incomplete strategy. Thus, for this method to be successful, some assumptions are necessary: 1. A goal must be specifiable in terms of sensor values. 2. The system must be able to observe its progress in order to detect when goals are not being met. 3. The data in the recent memory that the learning system is utilizing must contain sufficient information to form a complete strategy for the situation at hand. 4. The learning system must be able to generate a new controller before the environment changes enough to make it irrelevant. As discussed, the formation following goal can be specified in terms of sensor values in such a way that progress to the goal can be observed. Several approaches have been used to develop a baseline controller (non-adaptive) that would collect an adequate memory for training; the most successful was a random but purposeful algorithm that utilized slightly modified goal criteria to direct a robot during a memory collection process. This controller uses sensor feedback to continue actions that move it closer to its goal state, where the goal is for example an increasing sound intensity. As long as the feedback indicates that the metric is being met it continues the current action, but as soon as this is no longer valid, it randomly chooses another action. Since the selection of the new action is random, examples of correct actions are distributed relatively evenly. That is, there will be about an equal number of examples in which a right turn was the correct action as there will be for going straight or left. The structure of the process for a following robot is described next. Since we specifically have an underwater application context, the sensors that are being utilized are some sort of microphones. Acoustic sensing is typically the only feasible choice for such an environment, but the following discussion is not specific to this. The variables used in the subsequent discussions are ML(t) - Left Microphone (Sensor) value MR(t) - Right Microphone (Sensor) value A(t) - the current action of the robot: left, straight, right Now first, for the following robot, if both sensor values are increasing in intensity and close to each other, the robot will go to a neutral state, that is, it goes straight (A(t)= straight). We can express this case by the condition:

158

P.M. McDowell, B.S. Bourgeois, and F.E. Petry

CSraight = [(ML(t) > ML(t − 1)) ∧ (MR(t) > MR(t − 1)) ∧ (Little ∗ f ≥ Big)] (8.1) The first two parts of the condition represent the gains in intensity. The last term captures the concept of closeness of the sensor values. The values of Big and Little are the larger and smaller of the sensor values respectively and f is a user specifiable parameter (e.g. 1.05). CNochange = [(ML(t) > ML(t − 1)) ∨ (MR(t) > MR(t − 1))]

(8.2)

Lastly if neither of these cases apply then the approach will be to attempt a different action. The new action A(t) is a random choice from the set of possible actions {left, straight, right}.

Fig. 8.2. An example path generated by the random but purposeful controller. The sound source is at the center of the plot and the smiley faces with antenna denote the robot and its orientation as it follows the path.

Figure 8.2 shows a graphic example of the robot trajectory taken using the random but purposeful training controller. It is evident that this controller will generate a generous set of both good (moving towards the source) and bad (moving away from the source) examples for training a new controller. As noted earlier, a sufficient memory is required to generate a properly functioning controller. In the context of this application we could envision a situation in which the good examples predominately consist of left turns; as a result, the generated neural network controller may not know how to turn right to move towards the source. To contend with this situation, and due to the symmetry in this particular application, a mirroring technique was created that generated ‘extrapolated’ memories to complete the training set. Essentially this means that for every example where a left turn was executed to get closer to the source, a right turn

8 Robot Control in Dynamic Environments Using Memory-Based Learning

159

example was manufactured from this memory and vice versa. Also the positive straight example set takes advantage of the strategy, providing more variety in that example set. A high level description of the algorithm is illustrated below, where a positive example is one in which an action results in the situation in which sensor intensity increases the most: For each positive example Swap right and left sensor values if (action = left) then New action ← right; if (action = right) then New action ← left else New action ← straight In our approach the methods of collecting the memories and using that information to create controllers is addressed. Another equally important consideration is the method in which the memories and controllers developed from them are maintained. For instance the goal may remain constant but the environment may change rapidly, necessitating the use of a previously generated controller, or consider the case where the goal changes but the environment remains unchanged. The storage and management of these behaviors and memories is a robot architectural issue and is not addressed in this paper.

8.4 Genetic Algorithm Training of Neural Network Controllers The controllers used in this research are based on feed forward neural networks (FFNNs) with one hidden layer, since their structure readily lends itself to rapid generation on an autonomous robot. We utilize a genetic algorithm (GA) to train the neural network in which the GA is used to find the neural network’s transfer functions along with the weights. A GA is used to train the network for two main reasons: GAs provide an efficient and robust means of training neural networks[2], while avoiding many of the problems with becoming trapped in local minima as experienced with gradient techniques[23]. Since the best transfer function for a particular environment may not be known ahead of time, having it be GA selectable decreases the user’s need for a priori knowledge. The GA was selected over more traditional methods such as Back Propagation so that the need for explicit desired values for each training example could be avoided. In our current experiments involving formation maneuvering, this data could be acquired from a simulator, but for other applications the correct motions at each moment are often not known. In these instances a GA can use a fitness function that promotes overall system performance by rewarding controllers that best fulfill a programmed goal. There are a variety of issues that are involved in the effective and efficient execution of a genetic algorithm. The selection of suitable representation is a crucial component of GA design. Often the particular fitness function which carries out the evaluation of individuals in a population can be the most costly aspect of a GA’s execution. Also other

160

P.M. McDowell, B.S. Bourgeois, and F.E. Petry

parameters such as selection, crossover and mutation schemes must often be adjusted for efficient execution [8]. Although the inclusion of network transfer functions as part of the optimization algorithm has not been widely reported, there is at least one paper describing the use of a GA to optimize the number and placement of radial basis nodes in a modular neural network [15]. That research differs from our effort in that the GA worked with node placement and topology, while the weights were found with more conventional back propagation techniques. In our approach we use an Elite genetic algorithm to train the neural networks. For an Elite GA, the breeding portion of the population (best scoring individuals) is carried through to the next generation with the remainder of the population being derived using individuals from the breeding portion. A high level outline of the Elite GA algorithm used to find the weights and transfer functions of the feed forward neural network controller is described next. The population P = {pi } for the GA consists of n individuals (organisms) pi that represent feed forward neural networks. Each pi = (w,t) where w represents the weight and t the transfer functions of the FFNN respectively: w = (w1 , · · · wq )

− 1.0 ≤ w j ≤ 1.0

where q = [(number of inputs * number of hidden nodes)+(number of hidden nodes * number of outputs)] t = (t1 , · · ·tr ) r =number of hidden nodes where t j = 0 for discrete transfer function: t j = 1 for linear transfer function; t j = 2 for sigmoid transfer function. The fitness function of the GA is Fit(pi ) which measures the cumulative distance between the follower robot and a lead robot. The follower robot is controlled by the neural network described by individual pi , while the simulation system controls the lead robot. We use a notation topPercent(P) to represent the elite percent of the population P that is directly transferred to the next generation’s population. This percentage is a user adjustable parameter with a typical range of 10 to 33 percent. The Elite GA can be outlined as follows: Generate an initial population P0 by randomly initializing for each pi ∈ P0 the values of w and t. Run the simulation for M generations Evaluate the population P0 using the simulation in the fitness function Fit(pi) Order the evaluated individuals pi in P0 from best to worst P1 ← topPercent (P0) Use crossover and mutation on the topPercent portion of P1 to provide individuals for the remaining portion of the new generation’s population P1 P0 ← P1 Return the individual pi from P0 with the highest fitness value after M generations The benefits of using this method are its stable operation and straightforward implementation. This method was originally developed for generating feed-forward neural

8 Robot Control in Dynamic Environments Using Memory-Based Learning

161

networks for use in a formation maneuvering simulator [16]. In a typical simulation session, the user will use this algorithm to grow a controller. Usually the GA will build a working controller within 200 generations. For this research it was integrated into an online learning process that operates on our laboratory robots. In this capacity it trains neural networks based on training sets generated by reactive and memory based training processes. Unlike the brute force methods used in the formation maneuvering simulator, these methods take advantage of positive and negative example sets developed using the goal focused exploration described earlier. The next sections will discuss how the memory captured by the focused exploration is processed so that system can avoid excessive use of a priori knowledge and still use the GA to generate controllers.

8.5 Reactive Controller For a reactive controller, the actions taken at time t are based solely upon the current sensor readings. Consequently, the reactive learning algorithm uses a training set comprised strictly of individual sensor-action values that are marked as good if the action results in an increased sensor value and bad for a decreased value. The recent memory is examined for occurrences of these examples, where the sensor values at time t + 1 are examined to determine if the action at time t was a good or bad example. One known drawback of this type of controller is that it cannot react to trends that occur over time. A classic example for the formation maneuvering task is the forward-backward ambiguity: a following robot with a reactive controller cannot discern if it is heading directly away or towards the leader if the amplitude in both sensors remains the same; without an additional observation to detect this trend a robot could wander away from its leader in the wrong direction. Figure 8.3 and the steps below illustrate the process for creating the reactive controller using recent memories. A robot ‘trainee’ records a memory M consisting of sensor values and actions taken while seeking either a fixed or moving sound source. The robot could be guided by an operator or an algorithm such as the classic logic method, another neural network, or the random but purposeful controller. The sensor values and actions at a given time, denoted SA[t], is a 3-tuple (ML, MR, A) where ML / MR are the left / right microphone values respectively and A the action taken. A recent memory M will be defined as a sequence of sensor action values SA[0] · · ·SA[n − 1]. Now we can organize training sets of positive examples, P, and negative examples, N, from M by using the changes in the total sound energy E[t] where E[t] = ML[t] + MR[t]

(8.3)

The classification of the SA[t] values will be based on the total energy at times t and t + 1. So (8.4) P = P ∪ SA[t] i f E[t + 1] ≥ E[t] or N = N ∪ SA[t] i f E[t + 1] < E[t]

(8.5)

Using the set of positive examples as a new training set, the GA based technique can generate a new neural network. In order to increase the efficiency of the GA, the current best neural network can be used to seed the initial population.

162

P.M. McDowell, B.S. Bourgeois, and F.E. Petry

Fig. 8.3. The reactive learning process. Key to the process is the intensity filter. It ensures that the memory is composed of examples for which actions at time t had a direct influence on sensor values at t + 1. As a part of a robot’s architecture, this process would be called when an observer function notices goals are not being met.

We have described only the use of positive examples for which we present experimental results in section 7. However, it was also demonstrated in our laboratory simulations that negative examples could also be used by rewarding a neural network that is being trained if it takes a different action than the one associated with the example.

8.6 Memory Based Controller Figure 8.4 below shows the neural network configuration used for demonstrating learning with a memory based controller. The memory based controller behavior is achieved by using past sensor inputs in addition to the current input. The choice of the number of past inputs dictates the span of time over which the controller will be able to observe trends and also its ability to react quickly to sudden changes. The motivation for testing learning with this type of controller was to ensure proper handling of the forward-backward ambiguity mentioned earlier. Creating a training set for this type of controller proved to be significantly more challenging and was accomplished by splitting the recent memory into four categories with respect to the average sensor intensity: rising, dropping, stable and fluctuating. These

8 Robot Control in Dynamic Environments Using Memory-Based Learning

163

Fig. 8.4. A feed forward neural network with past inputs

Fig. 8.5. A time series of the goal parameter. The line is drawn in different line styles to illustrate the principal parts that it is made of. The bold part of the line fits in the category of rising parts, the dashed part in the dropping category, the dotted part in remaining stable category, and the thin part near the middle of the time series is the fluctuating category.

cases are illustrated in Figure 8.5, with the 4 categories referred to as the ‘principal parts’ of the memory. Since the controller uses 4 consecutive time inputs, the training set is composed of groups of 4 consecutive sets of sensor-action values obtained from the recent memory M. The classification of the memory into its principal parts will be based on the sound intensity value I ( the average of ML and MR). So now SA[t] becomes a 4-tuple

164

P.M. McDowell, B.S. Bourgeois, and F.E. Petry

including I : (I, ML, MR, A) and a recent memory M of length n is again a sequence: SA[0]..SA[n − 1]. Next we will describe the characteristics of the principal parts categories: RC: rising category where I(t) > I(t − 1). This is further subdivided into subsets in which the action is consistent. RC left - all actions are left turns RC straight - all actions are straight RC right - all actions are right turns Whether any given series of 4 consecutive SA values (referred to as a ‘run’) in the M belong to these categories is determined by applying the following rules time sequentially throughout the memory: SA[t] ∈ RC left if I(t) > I(t − 1) ∧ A(t) =left Membership in RC straight and RC right are defined similarly. SC: stable category where I(t) = I(t − 1) ± ε ; ε is a designer specified small value. This is further subdivided into subsets in which the action is consistent: SC left, SC straight and SC right as above except = is used instead of >, relative to I in the rules. DC: dropping category where I(t) < I(t − 1). Again we have subsets in which the action is consistent: DC left, DC straight and DC right and for which < is used instead of > for the rules. FC: fluctuating category. All series of 4 SA values in the memory that do not fall into the RC, SC, or DC categories are placed in the fluctuating category. The training sets are formed from the RC, SC and DC groups, with a slight variation of the DC components. For the RC and SC groups the action taken during each particular run is assumed to be correct, so the neural networks in training are rewarded when they take this same action. However, when the goal parameter is dropping the correct action is unknown. To determine the action to be used for a run that falls into the dropping category, an adjacent or overlapping rising run must be found. Once the overlapping example is found, its action is used as the correct action for the dropping run. It is used because it is the first correct action that occurs since the negative energy gain. If there is no adjacent or overlapping rising run, the dropping run is not used. Fluctuating runs are not used to create training examples at this time because there is no programmatic method of determining what a correct action is for these states.

8.7 Experimental Results Quantitative testing was conducted with a LabView [19] simulation that measured the distance and the heading differences between the leader and follower robot in a simple maneuvering task. In the test, illustrated in Figure 8.6, the lead robot traverses two laps around a preprogrammed course. The tests are started with the robots in formation, by maneuvering the leader-follower pair to the starting position and then engaging the

8 Robot Control in Dynamic Environments Using Memory-Based Learning

165

Fig. 8.6. Test course for the following robots. The lead robot (gray robot) is controlled by the simulator. The follower robot (clear robot) is controlled by the current controller being tested. Over two laps the inter-robot distance and course differences are recorded.

Fig. 8.7. The leader/follower distance over two laps. The follower robot is controlled by a reactive FFNN. As can be seen in the figure, the robot closed the distance to the leader and maintained it throughout the run except during the turns.

automatic navigation control for the leader robot [11,12,22]. The robots’ turning rates and speeds were set the same for all tests. The tests are started in the upper left hand corner of the track, when the leader/follower combination is in formation and tracking. Tests were also conducted using ActivMedia robots [1] equipped with microphones and speakers. The reactive feed forward neural nets (FFNNs) generated were able to control the robots in these tests although the sensor history network has not yet been tested with the robots. Figure 8.7 shows the average distance between the leader and a follower controlled by a typical reactive neural network trained as previously discussed. The distance between the two robots remains very stable except during the turns. The turns are the eight places in the curve where the distance fluctuates. Figure 8.8 next shows the heading

166

P.M. McDowell, B.S. Bourgeois, and F.E. Petry

Fig. 8.8. The heading difference between the leader and the follower robot controlled by a reactive FFNN during a two lap test in the simulator. Notice that each of the 8 peaks (the peak denotes a course change by the leader) occur at about the same time as the fluctuations occur in the curve in distance plot.

Fig. 8.9. The leader/follower distance for the two lap test where the follower is using the classic logic controller

differences measured during the same test. The heading difference grows large after the leader turns since it takes the follower a finite time to respond to the changing relative position of the leader and effect its own course change to follow. For comparison, Figures 8.9 and 8.10 show a similar test run using a classic logic controller based on Braitenberg’s robots [7,17]. The FFNN controller is seen to hold its distance almost as consistently as the classical controller. While the classic controller has a slight advantage on the robot-to-robot distance, the FFNN handles the turns a little

8 Robot Control in Dynamic Environments Using Memory-Based Learning

167

Fig. 8.10. This figure shows the course difference between the leader and follower where the follower is using the classic logic controller. Each peak corresponds to when the leader is making a turn.

Fig. 8.11. The leader/follower distance for the two lap test when the follower is using the principle parts controller

more smoothly, as can be seen by examining the oscillations in the robot-to-robot distance near the turns in Figures 8.7 and 8.9. Also, it was noted that the classic controller tends to oscillate more about the proper heading, depending upon the operator-specified ‘dead-zone’. This is observed in the thick lines in the course difference figure at the end of each turn, where the following robot oscillated about the desired course. One of the advantages of a learning system is the elimination of heuristically set parameters such as the dead zone size. Results for the principle parts (memory based) controller are shown in Figures 8.11 and 8.12. As can be seen in these graphs the simulated robot was able to follow

168

P.M. McDowell, B.S. Bourgeois, and F.E. Petry

Fig. 8.12. This figure shows the course difference between the leader and follower for the principle parts controller. Each peak corresponds to when the leader is making a turn. Notice that this controller takes more time to complete turns than the reactive controllers.

successfully, although it took much longer than the reactive controller to steady onto a heading. This result is anticipated since the reactive controller responds only to the last set of sensors values while the principle parts controller looks for a trend in the last n values and will typically be slower to respond. Tests were also conducted to demonstrate that the principle parts controller could properly deal with the forward/backward ambiguity problem, which is a known problem for the reactive controller. The outcomes of the tests were mixed. Holding the memorybased robot still (zero velocity), but letting it control its direction and moving the source directly at it quick enough for there to be a sensor gradient present in its input, allowed the robot to track the source. Slowing the source’s approach to where the gradient in the sensor history did not exist caused it to behave as the reactive system.

8.8 Conclusions Learning using recent memories was demonstrated successfully with both a reactive and memory based controller. Simulation testing with the formation following application showed promising results, and the reactive controller was also tested successfully on land robots. The use of the genetic algorithm to train a neural network controller based on selected recent memories gathered from the random but purposeful exploration was observed to be efficient and suitable for implementation in a real-time learning system. This work represents a new approach to in situ learning for an autonomous robot based on experiences (sensor/action pairs), without requiring exhaustive testing of all possible actions or the use of a simulator requiring a substantial amount of a priori knowledge about the operating environment. Future work may involve the development of a graph learning algorithm which can create a controller whose purpose is to achieve a specific goal. A directed graph can be built and updated from memories recorded as the robot explores and operates in the

8 Robot Control in Dynamic Environments Using Memory-Based Learning

169

environment. The graph’s structure can represent the robot’s experiences in a topological manner, preserving the temporal sequencing information. This could be used to train a neural network or by traversing the graph in real time, directly controlling the robot.

Acknowledgement We would like to thank the Office of Naval Research through the Naval Research Laboratory for sponsoring this research.

References 1. ActivMedia Robotics, Pioneer 2 Operations Manual, Version 1-1, Petersborough, NH (2002) 2. Angeline, P., Saunders, G., Pollack, J.: An Evolutionary Algorithm That Constructs Recurrent Neural Networks. IEEE Transactions on Neural Networks 5, 54–65 (1993) 3. Balch, T., Arkin, R.: Behavior-Based Formation Control for Multirobot Teams. IEEE Trans. on Robotics and Automation 14, 926–939 (1998) 4. Balch, T., Parker, L. (eds.): Robot Teams: From Diversity to Polymorphism. AK Peters. Wellesley (2002) 5. Bourgeois, B., Petry, F., Harris, M., Alleman, P.: A GIS Integration Approach for Dynamically Reconfigurable Surveys. Hydrographic Journal 91, 3–11 (1999) 6. Bradley, M.: Environmental Acoustics Handbook, 2nd edn. (1996) 7. Braitenberg, V.: Vehicles: Experiments in Synthetic Psychology. MIT Press, Boston (1986) 8. Buckles, B., Petry, F.: Genetic Algorithms: Introduction and Applications. IEEE Computer Society Press, Los Alamitos (1992) 9. Chance, T., Kleiner, A.: The Autonomous Underwater Vehicle (AUV): A Cost Effective Alternative to Deep-Towed Technology. Integrated Coastal Zone Management 8, 65–69 (2002) 10. Church, R., Warren, D.: New Technologies Rewrite History. Hydro International 6, 56–59 (2002) 11. Desai, J., Ostrowski, J., Kumar, V.: Modeling and Control of Formations of Nonholonomic Mobile Robots. IEEE Trans. on Robotics and Automation 17, 905–908 (2001) 12. Fierro, R., Das, A., Kumar, V., Ostrowski, J.: Hybrid Control of Formations of Robots In: Proc. IEEE Int. Conf. on Robotics and Automation, Seoul, Korea, pp. 157–162 (2001) 13. Grefenstette, J., Schultz, A.: An evolutionary approach to learning in robots. In: Proc. Learning Workshop on Robot Learning, New Brunswick, NJ, pp. 47–56 (1994) 14. Holland, J.: Designing Autonomous Mobile Robots. Elsevier, Amsterdam (2004) 15. Jiang, N., Zhao, Z., Ren, L.: Design of Structural Modular Neural Networks Genetic Algorithm. Advances in Engineering Software 34, 17–24 (2003) 16. McDowell, P., Chen, B., Bourgeois, B.: UUV Teams, Control From A Biological Perspective. In: Proceedings of the Oceans, MTS/IEEE Conference, Biloxi MS, pp. 331–337 (2002) 17. Mitchell, T.: Machine Learning. McGraw-Hill, New York (1997) 18. Nehmzow, U.: Mobile Robots: A Practical Introduction, 2nd edn. Springer, Heidelberg (2003) 19. NI LabVIEW Manual, National Instruments, Austin TX (2000) 20. Proakis, J., Sozer, E., Rice, J., Stojanovic, M.: Shallow Water Acoustic Networks. IEEE Communications Magazine 39, 114–119 (2001) 21. Rucci, M., Wray, J., Edelman, G.: Robust localization of auditory and visual targets in a robotic barn owl. Robots and Autonomous Systems 30, 181–193 (2000)

170

P.M. McDowell, B.S. Bourgeois, and F.E. Petry

22. Sgorbissa, A., Arkin, R.: Local Navigation Strategies for a Team of Robots. Robotica 21, 461–473 (2003) 23. Sofge, D., White, D.: Applied Learning: Optimal Control for Manufacturing. In: White, D., Sofge, D. (eds.) Handbook of Intelligent Control: Neural, Fuzzy, and Adaptive Approaches, Van Nostrand Reinhold, pp. 272–273 (1992) 24. Stroupe, A., Balch, T.: Value-Based Action Selection for Observation with Robot Teams Using Probabilistic Techniques Journal of Robotics and Autonomous Systems 50, 85–97 (2005) 25. Touzet, C.: Distributed Lazy Q-Learning for Cooperative Mobile Robots. Int. Jour. Of Advanced Robotic Systems 1, 5–13 (2004) 26. Urick, R.: Principles of Underwater Sound, 3rd edn. McGraw-Hill, New York (1983) 27. Watkins, C., Dayan, P.: Q-Learning. Machine Learning 8, 279–292 (1992) 28. Yamaguchi, H., Arai, T.: Distributed and Autonomous Control Method for Generating Shape of Multiple Robot Group. In: Proceedings of IEEE Int. Conf. on Intelligent Robots and Systems, pp. 800–807 (1994)

9 A Behavior Based Control and Learning Approach to Real Robots Dongbing Gu1 , Jindong Liu2 , and Huosheng Hu3 1 2 3

Department of Computing and Electronic Systems, University of Essex, United Kingdom [email protected] Department of Computing and Electronic Systems, University of Essex, United Kingdom [email protected] Department of Computing and Electronic Systems, University of Essex, United Kingdom [email protected]

Abstract. Programming a real robot to do a given task in unstructured dynamic environments is very challenge. Incomplete information, large learning space, and uncertainty are major obstacles for control in real robots. When programming a real robot in unstructured dynamic environments, it is impossible to predict all the potential situation robots may encounter and specify all robot behaviors optimally in advance. Robots have to learn from, and adapt to their operating environment. In this chapter, we propose to use fuzzy logic to design robot behaviors and use a Markov decision process to model the coordination mechanism in the control and learning of real autonomous robotic systems. Based on the model, a Q-learning approach can be used to learn the behavior coordination. Two real robot applications are implemented by using such an approach, one is a Sony quadruped robot for soccer playing and another is a robotic fish for entertainment. Real robot testing results are provided to verify the proposed approach.

9.1 Introduction Programming autonomous robots raises many difficulties because they not only have multiple motors for movement, but also most of them contain heterogeneous sensors required to achieve goal-oriented tasks. The program requirements for these robotic systems are not limited to keeping them moving, but include the commitment to the perception of its environment, its autonomous operation in a changing environment, and achieving goal-oriented tasks. This chapter is concerned with programming with two autonomous robotic systems. One is an autonomous quadruped robot, the Sony quadruped robot, which is used as a member of a team to play soccer in the RoboCup challenge, and another a robotic fish, which is used for entertainment in aquarium. Autonomous robots are designed to achieve goal-oriented tasks, such as scoring a goal in RoboCup or randomly swimming in a tank. The major challenges when programming such a robot stem from: i) the dynamic environment in which the robot is located and it is unable to predict when events will occur. (ii) Uncertain sensory information that is inaccurate, noisy, or faulty. (iii) Imperfect actuators that cannot guarantee correct and perfect execution of actions D. Liu, L. Wang, and K.C. Tan (Eds.): Design and Control of Intel. Robotic Systems, SCI 177, pp. 171–186. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com 

172

D. Gu, J. Liu, and H. Hu

due to mechanical, electrical, and servo problems. (iv) Limited time that constrains time intervals needed for sensor information processing, actuator control, and goal-oriented planning. For autonomous mobile robots, early programming approaches followed a sequence: sensing the environment, planning trajectories, and controlling motors to move. These approaches belong to a deliberative control approach that needs rigid design models in each part of the system [3]. Robots employing this type of control architecture are well known for their lack of real-time response and robustness. Reactive control or Behaviorbased robotics provides an alternative solution to the problems described above [3][2]. In behavior-based robotics, behavior design and behavior co-ordination are two essential topics. To design a behavior is to find a mapping function from environment states to motor actions associated to this behavior. Due to less computation in the direct mapping, behaviors possess a real-time response ability. For a specific domain, a set of behaviors defines robot skills in the domain and they constitute the fundamental basis for the control architecture. Each behavior is designed to achieve a specific subtask and their coordination forms the deliberative planning for goal-oriented tasks in the domain. When designing a behavior or behavior co-ordination mechanism, it is impossible to predict all the potential situation robots may encounter and specify all robot behaviors optimally in advance. Robots have to learn from, and adapt to their operating environments. Learning has been recognisee as one of the basic abilities for autonomous robots. To take advantage of the strengths of deliberative and reactive control architectures, and attempt to avoid their weakness, some research has been conducted into hybrid architectures [5][6]. Hybrid architectures possess the planning ability of a deliberative architecture to achieve goal-oriented tasks, but also apply the fast response ability of reactive architectures in order to deal with environment dynamics. In this chapter, we design a layered control architecture for autonomous robots, which includes a cognition layer for planning, a behavior layer for reactive control, and a locomotion layer for basic movement skills. In the cognition layer, a state-based planning method is applied for reasoning via a Markov Decision Process (MDP). The state-based planning can be combined with Q-learning to implement the learning in the cognitive layer. Q-learning is a trial-and-error learning approach that interacts with dynamic environments [1][11]. It explicitly considers the whole problem of a global goal, and its final purpose is to maximize a training signal over a period of time, rather than to maximize instant feedback in each trial. This matches our goal-oriented tasks in the cognitive layer. In the behaviors layer, reactive control is realized based on a group of behaviors. The design methodology of a behavior has a wide spectrum, including heuristic rules, potential fields, biological inspired approaches, fuzzy logic, and many learning algorithms. Using fuzzy logic to design behaviors can enable the behaviors to handle uncertain sensory readings and imperfect motor control and have been successful used for many real robots [10][7][4][9][8]. In fuzzy logic, fuzzy sets model the uncertainties and fuzzy rules work co-operatively to produce final control results. The produced control effect is smooth, reasonable, and robust, as each fuzzy rule makes its contribution to final outputs by its activated degree. Furthermore, using fuzzy logic to design behaviors can make use of prior experiences, which can be expressed as fuzzy rules or employed to

9 A Behavior Based Control and Learning Approach to Real Robots

173

select the parameters of fuzzy membership functions. Optimizing fuzzy behaviors or accurately defining fuzzy rules and their parameters is still a challenge. Learning or adaptation of fuzzy behaviors provides a feasible solution to the challenge, since the learning can modify the parameters of fuzzy logic to improve the behavior performance or autonomously select fuzzy rules to achieve the behavior automatic acquisition. In the following, we will introduce the layered control architecture in section 9.2 first. Then the using of fuzzy logic to design behaviors is described in section 9.3. The model and learning of the cognitive layer are presented in section 9.4. The testing results of both Sony quadruped robots and robotic fish are provided in section 9.5 and section 9.6, respectively. Finally a brief conclusion is summarized in section 9.7.

9.2 Layered Control Architecture A layered architecture is designed for autonomous robotic systems. Figure 9.1 presents the proposed layered control architecture, which consists of three layers: a Cognitive Layer, a Behavior Layer, and a Locomotion Layer. At the center of the architecture is the behavior layer; where most of the instant decisions are made. The locomotion layer is particularly designed for different kind of robotic systems. For example, it is called swim pattern Layer in our robotic fish and gait control Layer in our legged robots. In the behavior layer, states and actions are directly related to robot physical sensors and gaits or swim patterns in the locomotion layer. In the cognitive layer, states relating to high-level activities are extracted from the behavior layer, and actions correspond to behavior coordination.

Internal World States

MDP-based learning schema

Actions

Cognitive Layer Behaviour 1 Sensors Behaviour n Motions Behaviour Layer Motion 1

Motion n

Locomotion Layer Motors Fig. 9.1. The layered control architecture for autonomous robotic systems

174

D. Gu, J. Liu, and H. Hu

From control point of view, programming autonomous robots based on the layered architecture is a feasible and practical scheme. With this scheme, large sensory and action spaces can be separated into a number of small local regions both in the range and dimensions. Searching small local regions to find solutions is easier than operating on a large space. From learning point of view, a bottom-up learning structure can be applied to the control architecture. The locomotion layer establishes the optimal locomotion parameters first, and then provides robust locomotion to the behavior layer. The behavior layer then starts to optimize the parameters of each behavior. Finally, the cognitive layer takes the optimal behaviors as actions, and learns an optimal policy between states and actions. 9.2.1

Locomotion Layer

The Locomotion layer is at the bottom of the three-layer architecture. For our robotic fish, this layer consists of five swim patterns: Cruise-Straight, Ascent/Descent, Coast, Sharp-Left/Right-Turn and Cruise-in-Left/Right-Turn. A swim pattern is defined as a temporal sequence of the associative movement of tail joints. For our quadruped robots, the locomotion layer is to implement basic walking gaits. It provides a group of walking commands to the behavior layer. The walking commands includes Move Forward, Left Forward, Right Forward, Left Turn, Right Turn, and Stop. When the behavior layer finishes behavioral coordination, it produces a locomotion command to be executed. The robotic system inputs this locomotion command into the locomotion layer and implements it until a new command is received from the behavior layer. 9.2.2

Behavior Layer

The behavior layer is located in the middle of the control architecture. It is responsible for the reactive control of the robotic systems. It converts raw sensor data into sensor states using fuzzy membership functions first before inputting these states into each behavior. A fuzzy logic controller is designed for every behavior to process states. For our robotic fish, eight behaviors are created and listed as follows. • Keep-level: This is the most efficient for long distance swimming. • Avoid-obstacle: This behavior is used to move away from other fish, predators, and rocks, etc. A significant feature of this behavior is that a fish changes its swimming direction (pitch, roll, and yaw angles) to avoid obstacles, rather than slow down or come to a rest. This is because changing direction is much easier than braking. • Follow-wall: Fish swim along the surface of rocks to patrol their territory and find food. This behavior helps fish to explore an unknown environment in a safe way, because it decreases the risk of predator attack. • Wandering: Fish employ this behavior for relaxation. Our robotic fish apply this behavior to swim around in order to find potentially interesting objects. • Seek-goal: Food, mate, and home are the main goals for real fish. For the robotic fish, the charging station is one goal, and the tank wall is another when they implement a tank exploration task. This behavior enables robotic fish to find a way to a charging station.

9 A Behavior Based Control and Learning Approach to Real Robots

175

• Rescue: In some dangerous situations, when fish cannot get themselves out of trouble through other behaviors, a rescue behavior can be applied. In most cases, a fish will reduce its movement as much as possible and keep still. The spots and strips on their body confuse a predator, and provide camouflage. Likewise, our robotic fish implement this behavior if they face uncontrollable situations, such as a servo motor malfunction, or extremely high internal temperature, etc. They will stop performing all other behaviors, and empty their water chamber to float up slowly to the water surface. • Feed: This behavior allows fish to find food. For example, a shark chases a small fish and swallows it. For our robotic fish, the feed behavior enables them to adjust their movements to swim towards a charging station, in order to charge their battery. For Sony quadruped robots to play an attacker role, five behaviors are created and listed as follows. • Find-ball: The robot starts to scan the field through its head motion. Once a ball is found, the robot updates the ball’s position and its confident value. Simultaneously the robot rotates its body to increase its view and gradually moves towards the center of the pitch. • Chase-ball: The robot moves its head to track the ball. It moves towards the ball and tries to go behind the ball to face the goal in order to kick the ball. The sensory states of this behavior include ball’s co-ordination, goal’s position, and robot’s position. • Align with goal: The robot slightly adjusts its position to align the ball and its orientation with the goal in order to kick the ball into the goal. The sensory states of this behavior include ball’s co-ordination, goal’s position, and robot’s position. • Kick-ball: some specialized kick skills are applied. • Dribble-ball: The robot kicks the ball by using its feet and keeps its body moving. It’s very useful when the ball is close to edges or corners of the pitch. 9.2.3

Cognitive Layer

The top layer of the architecture is the cognitive layer. This layer first extracts the robot status from the sensor states in the behavior layer. It then performs reasoning and planning based on an internal world which defines the robot constraints and environment information. Finally, it makes decisions on how to coordinate behaviors to achieve a given task. In contrast to a traditional deliberative control architecture, the cognitive layer does not provide a trajectory for robots follow. The only way it affects and supervises the behavior layer, is through the global performance of emergent behavior. The aim of the cognitive layer is to organize actions to accomplish a task. An action can enable a behavior, or set a configuration parameter of a behavior. Actions in the cognitive layer are denoted ca and are related to all behaviors, States in the cognitive layer are extracted from sensor readings. They are the highlevel conditions and reflect significant physical events, which are sensed or recognized by a temporal and spatial combination of several sensors. For example, if the downfacing infrared sensor outputs a higher value and the pressure sensor is larger than a threshold value, they indicate that the fish is near the bottom of the tank. Events such as these constitute the state space. Formally, each event is denoted by a binary

176

D. Gu, J. Liu, and H. Hu

variant bv. Once an event occurs, the related bv is set to 1; otherwise it is cleared to 0. Grouping n events (bv1 , ..., bvn ) in an order, generates a state cs for the cognitive layer, i.e., cs ← bv1 , ..., bvn . The state space consists of 2n combinations of n events. A policy π is a mapping from states to actions. The optimal policy π ∗ maximizes the probability of reaching the goal. In section 9.4, we introduce the Q-learning, in which each policy generates a specify value and maximizing the value function leads to obtaining an optimal policy.

9.3 Behavior Design Using Fuzzy Logic Acquiring behaviors is a fundamental step in the control design of autonomous robots. Fuzzy logic is a widely used approach for the robot behaviors [10][7]. In fuzzy logic, linguistic rather than numerical variables are used (which mimics humans reasoning). Our experience is incorporated into the system in this way. Each behavior receives input signals, i.e. states, from a perception module. It then processes these using a fuzzy logic, and generates a response, i.e. an action. A typical fuzzy controller is constructed from four elements: a Fuzzifier, a Fuzzy Rulebase, a Fuzzy Logic Inference Engine, and a Defuzzifier. The Fuzzifier translates input signals into linguistic values, i.e. fuzzy input variables, using membership functions. The Fuzzy Logic Inference Engine evaluates the set of fuzzy rules defined in the Fuzzy Rulebase, identifies rules that apply to the current input variables, and calculates output linguistic values, i.e. fuzzy output variables. The Defuzzifier translates the output variable to a crisp control signal, i.e. a behavior response. In the following, a follow-wall behavior example is illustrated for behavior design in our robotic fish. This behavior keeps a robotic fish at a certain distance from a wall, based on the information gathered by its IR sensors. Figure 9.2 shows a block diagram of the follow-wall behavior. The left, right and bottom IR sensor data are the inputs of the perception module. One more input Wd , is added to define the relative position of the wall to the robotic fish. It is appointed by the cognitive layer, and defined in (9.1). For instance, if Wd = 1 the robotic fish should follow the wall on its left side. The perception module outputs an IR intensity ds , which is either dl or dr ; depending on Wd using (9.2). Via the fuzzy controller, the output ωd is tweaked using Wd to calculate ωd = Wd ωd ; which is one of the inputs of the Bev-coordination module. Finally, swim pattern S p is generated to be executed on the robotic fish. Another output of the perception module Wd

follow-wall behaviour

db

Fuzzy Logic Inference Engine

Defuzzifier

B_IR

Fuzzy Rulebase

ds Fuzzifier

R_IR

Perception

L_IR

ω 'd

ωd

Pt

Bev Coordination

Wd

Sp

Pt*

Fig. 9.2. A fuzzy logic representation of the follow-wall behavior

Robotic fish

9 A Behavior Based Control and Learning Approach to Real Robots

177

μ far

middle near

1 IR Intensity (ds,db) 0

25% 50% 75% 100% m11 m12 m13 m21 m22 m23

(a)

μ

μ RS

Z

NS

LS

1

-60 -30 n11

0 30 n12 n13

(b)

Z

PS

1

60

ω'd(degree)

-30 -15 n11

0 n12

15 n13

30

P t(mm)

(c)

Fig. 9.3. Membership functions of the input and output variable for follow-wall. mi j , nls : are the centers of the membership functions. μ : is the degree of truth. (a) the IR intensity ds ; (b) the yaw angle ωd relative to the current swimming direction; and (c) the position of the pitch weight Pt .

is db , which is the intensity of the bottom IR (which affects another fuzzy controller output–Pt ).  1 le f tside Wd = (9.1) −1 rightside  dl , if Wd = 1 ds = (9.2) dr , if Wd = −1 Figure 9.3 shows the membership functions of ds , db , ωd and Pt . The parameters (mi j , ni j ) are decided by the features of the IR sensors and the mechanical structure of the fish. For example, when a robotic fish follows a wall at an appointed distance (default 0.3 m), the obtained samples of the side IR intensity are approximately 25% to 75%. Therefore, the center of the middle term of ds is set to 50%, and the range is set from 25% to 75%. ωd is encoded in three terms, because the possible horizontal reactions of the follow-wall behavior are limited to the cruise straight, cruise in left turning, and cruise in right turning swim patterns. These three swim patterns have roughly a 0◦ , 30◦ and −30◦ turning angle. Therefore, the centers of the three terms are set as 0, 30 and −30. The membership of Pt is decided via the mechanical design of the pitch control module, where a pitch weight moves. For example, the robotic fish will dive slowly when the pitch weight moves to a −15 mm position; therefore the center of the negative small diving (NS) term is set to −15.

178

D. Gu, J. Liu, and H. Hu

Table 9.1. Rulebase for the follow-wall fuzzy controller. N: near, M: middle, F: far, RS: right small turn, LS: left small turn, Z: zero. PS: positive small (ascent), NS: negative small (descent), N/A: no input or output for the rule.

Rule Input Output No. ds db ωd Pt 1 N N/A RS N/A 2 M N/A Z N/A 3 F N/A LS N/A 4 N/A N N/A PS 5 N/A M N/A Z 6 N/A F N/A NS The rulebase for the fuzzy inference engine is listed in Table 9.1. It shows that ds and db independently affect ωd and Pt , because heading change is only related to the side IR sensors, and the diving control is only related to the bottom IR. These rules are decided based on our experience and the mechanical configuration of the IR sensors. For example, if Wd = 1 in (9.2), the first rule can be interpreted as: “IF the left IR intensity is near, THEN make the fish turn right a little, i.e. ωd = RS, regardless the bottom IR”. After all the rules are evaluated, a classical defuzzify method, the center of gravity (COG) method, is selected for defuzzification (see (9.3) and (9.4)). This method combines the outputs represented by the implied fuzzy set from all rules in order to generate the gravity centroid of the output distributions for the final output.

ωd =

∑6k=1 v1k qk ∑6k=1 v2k qk , P = t ∑6k=1 qk ∑6k=1 qk

(9.3)

qk = min{ μ1k1 , μ2k2 }

(9.4)

where v1k and v2k denote the estimated value of the kth rule output; which is related to the center of the membership functions of the fuzzy output variables ωd and Pt . qk is the conjunction degree of the IF part of the kth rule, which is calculated using a min(·) function (because the AND fuzzy logic operator is applied). μiki is the degree of the membership for the ith input contributing to the kth rule; ki is the term number of the input variable corresponding to the ith input in the kth rule. μiki is 1 if the corresponding input in Table 9.1 is “N/A”.

9.4 Q-Learning in the Cognitive Layer In the cognitive layer, the correct behavior should be selected to achieve specific tasks, i.e., the robot should reason or plan actions according to states. Reinforcement learning, specifically Q-learning, is used as a planner, which consists of three parts: Action Space - a set of possible actions; State Space - the discrete possible situations of a robot moving from an initial place to a goal; and World Model - a mapping from the state to

9 A Behavior Based Control and Learning Approach to Real Robots

179

action space. A Markov Decision Process (MDP) model can be used to formally model such a planner. Q-learning can be used to learn the mapping function in this model. The state transitions are described by the transition probabilities. P = Pr{cs(t+1) |cs(t) , a(t) }

(9.5)

where Pr{} denotes the transition probability. The expected value of the next reward, given the current state and action, cs(t) and a(t) , together with the next state, cs(t+1) , is expressed as. R = E{rt+1 |cs(t) , a(t) , cs(t+1) }

(9.6)

where E{} denotes the operation of calculating the expected value. A policy π is a mapping from states to actions. The optimal policy π ∗ maximizes the probability of reaching the goal. The value of a state s under a policy π , denoted V π (s), is the expected return when starting in s and following π thereafter. Function V π is referred to as the state-value function for policy π . So V π (cs) is defined as. ∞

V π (cs) = Eπ {Rt |cs} = Eπ { ∑ γ k rt+k+1 |cs}

(9.7)

k=0

where Eπ {} represents the expected value when the robot follows policy π . At the same time, the value of taking an action a in a state cs, is defined under policy π as Qπ (cs, a). ∞

Qπ (cs, a) = Eπ {Rt |cs, a} = Eπ { ∑ γ k rt+k+1 |cs, a}

(9.8)

k=0

The one-step Q-learning updated function is written as follows[12]:   Q(s, a) = Q(s, a) + α r + γ max Q(s , a ) − Q(s, a) a

(9.9)

where Q represents Qπ ; s denotes cs; α and γ are learning ratios; r is the observed reward when taking action a; and s is the succeeded state after taking action a. The ε -greedy method is used to generate a from s using the policy derived from Q. Assume that all Q values continue to be updated during learning, and that the learning rate α satisfies the following two conditions. ∞

∞

t=0

t=0

∑ αt = ∞ and ∑ αt2 < ∞

(9.10)

Then Q has been shown to converge with a probability 1 to optimal Q∗ [11]. Pseudo code for Q-learning is listed below. 1. Initialize: Q(s0 , a0 ) arbitrarily; 2. While t < TE do: 3. choose at from st using the policy derived from Q and ε -greedy; 4. Take action at , observe new   state st+1 and reward rt+1 ; 5. Q(st , at ) = Q(st , at ) + α rt + 1 + γ max Q(st+1 , a) − Q(st , at ) ; a 6. t=t+1; 7. endwhile

180

D. Gu, J. Liu, and H. Hu

9.5 Robotic Fish Experiments Figure 9.4 shows the profile of our robotic fish. They are approximately 52 cm long, and have 3 servo motors, and two DC motors. The servo motors are concatenated together in the tail to act as 3 joints; one DC motor is fixed in the head to change the Center of Gravity (COG) of the fish, while the other controls a micro-pump in order to adjust the weight of robot. The central controller for each robotic fish is based on a 400 MHz Gumstix computer [13]. The Gumstix samples data from sensors, processes data, makes decisions, and sends signals to several PIC micro-controllers. These PIC controllers include three servo control boards, a DC motor control board, and a sensor board. The head of the robotic fish is waterproofed and rigid to counteract water pressure. The tail consists of three joints, which are designed to seal three servo motors independently. Each fish has over ten embedded sensors: a gyroscope, a pressure sensor, two position sensors, two electrical current sensors, a voltage meter, four infrared sensors, and an inclinometer. These embedded sensors enable the fish to detect: the depth in the water, the yaw/roll/pitch angles of its body, and a range of obstacles in front. In addition, the turning angle and electrical current consumption of each servo motor can be obtained. Bluetooth and RS232 serial ports are used as a means of communication with an external PC, which is used to program the Gumstix and PIC controllers. A sensor log can be retrieved via Bluetooth. The robotic fish are fully autonomously controlled to swim underwater in 3D and no supervision is taken during their swimming. On the top of the fish body, a dorsal fin is fixed vertically to avoid the fish wobble to side. Two pectoral fins are attached on two sides of the head to help balance the fish body. Both dorsal fin and pectoral fins are inactive. The propulsion force for the robotic fish swimming comes from an associated undulated movement of three joints, which make the fish body to push the surrounding water and obtain reaction force from the water. The design of the joint movement is the key factor to affect the swimming performance, such as swimming efficiency and head undulation angle, etc. In the cognitive layer, states and actions are discrete variables, and the Q-learning (state based learning) can be implemented here conveniently.

Fig. 9.4. A robotic fish

9 A Behavior Based Control and Learning Approach to Real Robots

181

3

2.5

y−width (m)

2

1.5

1

0.5

0

0

0.5

1

1.5 x−length (m)

2

2.5

3

Fig. 9.5. The fish trajectory during learning. Please note that the dashed line is the desired path, α = 0.5, γ = 0.3, ε = 0.01. 60

Root Meas Square Error (m)

50

40

30

20

10

0

0

50

100

150 Episode

200

250

300

350

Fig. 9.6. The root mean square error between the learning trajectory and desired path, against the number of learning trips. The error is the sum of two parts: position error and heading error. Each trip had 50 steps.

To test the learning algorithm, a tank border exploration task was implemented using our robotic fish. The objective is to make a robotic fish autonomously explore a tank border. The fish should follow tank walls at an appointed distance, be able to avoid corners and other fish, and maintain itself in a desired depth level. Additionally it must keep the wall to its right hand side. The robotic fish is not supposed to know anything about its environment. It will learn to choose a correct action according to its state. The action space is structured according to the objective of the task. Three behaviors

182

D. Gu, J. Liu, and H. Hu

(Wander,Follow-wall, and Avoid-obstacle) are chosen and customized from the behavior layer for the level plane action subspace. There is no vertical action subspace in the cognitive layer for this task because the keep-level behavior is capable enough for the task. Three events are defined to create states (cs): • Is the wall on the right hand side of the robotic fish? (bv1 ): This is determined using the right, front, and left infrared sensors. bv1 = 1 if the recent history of these sensors satisfies some conditions. • Is the robotic fish at a reasonable distance from wall? (bv2 ): This event is similar to bv1 , but has more strict conditions. • Is the wall on the left hand side of the robotic fish? (bv3 ): Like bv1 , it recognizes the situation that the nearest wall is on the left hand side of the fish. In other words, the swimming direction is the reverse of the desired direction in this situation. In this task: r = 1 when the robotic fish in the follow-wall behavior is able to successfully follow the wall; r = −1 when the fish in the follow-wall behavior loses the wall it has been following; r = −3 when there is a bump between the wall and robotic fish; and r = 0 for all other situations. According to the Q-learning algorithm, policy π ∗ of the task is learnt in a 3D simulator, and then applied to the real robotic fish. Figure 9.5 presents fish swimming trajectories during learning. It shows that the robotic fish maintains a required distance from the wall most of the time. Figure 9.6 provides a history of the root mean square (RMS) error between the learning trajectory and desired path over 300 trips. The RMS error is the accumulated error for one learning episode. It decreases and converges to a low value as the number of learning trips increases. Figure 9.7 shows the trajectory of our robotic fish operating in a real tank. The optimized parameters in the simulation were implemented in the fish. The arrows in the figure show heading directions at specific points, while the circle points indicate the position of the robotic fish. The time step between two record points is 3 seconds. The robotic fish was put into the water at point S, with a heading direction B0 . During swimming, the fish encountered an obstacle at point B1 . It implemented a sharp-left-turning swim pattern to avoid it. It then found the wall again using the cruise-in-turning-right swim pattern. Finally, the robotic fish spent 105 seconds swimming around the interior perimeter of the tank and finally reached point E.

y−width (m)

1.5

S(0s) B0

1

0

B1

E(111s)

0.5

(60s) B2 0

1

2

3 x−length (m)

4

Fig. 9.7. A circular trajectory on a level plane

5

9 A Behavior Based Control and Learning Approach to Real Robots

183

9.6 Sony Quadruped Robot Experiments The Sony quadruped robot is equipped with a camera mounted on the head. The camera has approximately 60 horizontal and 40 vertical fields of view. The head has three DOFs, namely, pan, tilt, and roll. The robot has three gyros for measuring body postures, namely, roll, pitch, and yaw. There is an infrared range finder mounted in the head. Its locomotion is implemented by four legs, each of which has three DOFs. The robot has one embedded 64 bits R4000-series microprocessor with over 100 MIPS. To evaluate the performance, the environment provide payoffs to robots with a certain accuracy to improve the learning efficiency. A global monitor is set up in this research (see figure 9.8), which includes an overhead camera, a desktop computer, and a visual tracking software, to provide an external judgement for the interaction between the robot and its environment. The monitor feeds the position information of the robot and the ball to the robot. Then, the robot can autonomously test its control strategies and evaluate the results.

Fig. 9.8. A learning environment for a Sony quadruped robot

The goals are centrad on both ends of the field. Six unique colored beacons are placed around edges of the pitch, with one at each corner and one on each side of the halfway line. The monitor recognizes the robot and the ball according to their colors. Through the image processing, the monitor updates their positions continuously. The robot can ask for the information at any time during its learning process. The communication is achieved through the Internet where the monitor system acts as a server and the robot acts as a client. The server provides the global information to the client when the client has a request. Since the robot only evaluates its performance at the end of one run, there is no significant effect of the communication on the learning process. All five behaviors defined in section 9.2 are used. Four physical events are defined for a robot to play an attacker role, • Is the ball found? (bv1 ): It simply compares the ball confidence value with a threshold.

184

D. Gu, J. Liu, and H. Hu

• Is the ball near enough? (bv2 ): since the robot always tracks the ball, its tilt angle and pan angle show if the ball is just in the front of the robot. • Is the robot behind the ball? (bv3 ): it depends on the distance and the angle between the ball and the robot, and the angle between the robot and the goal. • Is the robot aligned with the goal? (bv4 ): it again depends on the information same as ps3, but with more accurate values. All experiments are conducted in five situations, each of which has different setting for initial ball position and initial robot position. Totally 30 trails are conducted. The initial Q table is randomly initialized. The Q-learning is used to find a final Q-table. Finally the robot is run again based on the final Q table to evaluate the performance. In the Q-learning algorithms, the ε -greedy algorithm used ε = 1% to explore the action space, i.e. 99% of behaviors were selected by maximizing Q values and only 1% of behaviors were selected randomly. The discount factor was selected as γ = 0.9. The larger the discount factor, the more the learning exploits the Q value predictions. The learning rate of the Q values was selected as α = 0.1. Figure 9.9 shows the increasing trend of average rewards received. Several decreasing points in the curve reflected the exploration mechanism. The number of policy changes was also decreased as shown in figure 9.10, indicating that the learning was going toward the steady results. 2.1 1.8

Rewards

1.5 1.2 0.9 0.6 0.3 0 1

3

5

7

9

11

13

15 17 Trials

19

21

23

25

27

29

21

23

25

27

29

Fig. 9.9. Average rewards 100

Policy changes

80 60 40 20 0 1

3

5

7

9

11

13

15 17 Trials

19

Fig. 9.10. Number of policy changes

9 A Behavior Based Control and Learning Approach to Real Robots

(a)

185

(b) Fig. 9.11. Two testing scenarios

The two successful scenarios after the robot learning are displayed in figure 9.11. The dark curves represent the robot’s trajectories and the light curves represent the ball’s trajectories. The two curves met together at some places. It means the robot moved near to the ball at the meeting places. Then, the ball was kicked and moved away from the robot. So the curves were separated after the meeting places. During the whole processes, the robot can move to the ball and manoeuvre the ball into the goal. It demonstrates that the robot acquired the state planning strategy after the learning.

9.7 Conclusion To design a control and learning system for autonomous quadruped robots, this chapter has examined a behavior-based control and learning approach. The methodologies make contributions for building successful autonomous robots with reactive behaviors, deliberative planning and learning abilities. A layered control architecture for autonomous quadruped robots consists of three layers and they are connected hierarchically. The bottom layer is called the locomotion layer that is directly related to the basic movements. Based on the locomotion skills, the functional behaviors are collected in the middle layer - behavior layer. The top layer is called the cognition layer in which the deliberative reasoning for a task is established. Fuzzy logic is well known for its ability to handle uncertainty and inaccuracy. Using Fuzzy logic to design robot behaviors has been explored in the applications. The parameterized fuzzy logic also provide an opportunity to learn robot behaviors. The planning problem is viewed as a behavior co-ordination problem and formulated as a MDP to accommodate the learning ability. The state space in the high-level layer

186

D. Gu, J. Liu, and H. Hu

is based on the definition of the feature states that are physical events of the interaction between robots and their environment. The state space in the low-level is based on the map that directly consists of sensor readings.

References 1. Kaelbling, L.P., Littman, M.L., Moore, A.W.: Reinforcement learning: A survey. Journal of AI Research 4, 237–285 (1996) 2. Arkin, R.C. (ed.): Behavior-based Robotics. MIT Press, Cambridge (1998) 3. Brooks, R.: A robust layered control system for a moible robot. IEEE J. Robotics and Automation 2(1), 14–23 (1986) 4. Saffiotti, A.: Fuzzy logic in autonomous robotics: Behaviour coordination. In: Proceedings of the 6th IEEE Int. Conf. on Fuzzy Systems, pp. 573–578 (1997) 5. Huber, M.: A hybrid architecture for adaptive robot control. Ph.D. dissertation, University of Massachusetts (2000) 6. Stone, P.: Layered Learning in Multi-Agent Systems. PhD Thesis, Carnegie Mellon University (1998) 7. Angelov, P., Filev, D.: An approach to online identification of takagi-sugeno fuzzy models. IEEE Transactions on Systems, Man and Cybernetics- Part B 34(1), 484–498 (2004) 8. Xu, W.L., Tso, S.K.: Sensor-based fuzzy reactive navigation of a mobile robot through local target switching. IEEE Transactions on Systems, Man, and Cybernetics- Part C: Applications and Reviews 29(3), 451–459 (1999) 9. Vadakkepat, P., Miin, O.C., Peng, X., Lee, T.H.: Fuzzy behavior-based control of mobile robots. IEEE Transactions on Fuzzy Systems 12(4), 559–565 (2004) 10. Rusu, P., Petriu, E.M., Whalen, T.E., Cornell, A., Spoelder, H.J.W.: Behavior-based neurofuzzy controller for mobile robot navigation. IEEE Transactions on Instrumentation and Measurement 52(4), 1335–1340 (2003) 11. Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. Dietterich, T. (ed.). The MIT Press, Cambridge (1998) 12. Watkins, C.J.C.H.: Learning from delayed rewards. Ph.D. dissertation, Cambridge University (1989) 13. Gumstix Co., Gumstix-way Small Computing (2005), http://www.gumstix.com

10 Incremental Acquisition of Neural Structures through Evolution Yohannes Kassahun, Jan Hendrik Metzen, Mark Edgington, and Frank Kirchner Robotics Group, University of Bremen, Robert-Hooke-Str. 5, D-28359 Bremen, Germany [email protected], jan [email protected], [email protected], [email protected]

Abstract. In this chapter we present a novel method, called Evolutionary Acquisition of Neural Topologies (EANT), for evolving the structures and weights of neural networks. The method uses an efficient and compact genetic encoding of a neural network into a linear genome that enables a network’s outputs to be computed without the network being decoded. Furthermore, it uses a nature inspired meta-level evolutionary process where new structures are explored at a larger timescale, and existing structures are exploited at a smaller timescale. Because of this, the method is able to find minimal neural structures for solving a given learning task.

10.1 Introduction A meaningful combination of the principles of neural networks, reinforcement learning, and evolutionary computation is important in designing agents that learn and adapt to their environment through interaction [21, 35]. These principles can be combined to form an evolutionary reinforcement learning system in which each component plays a significant role. Neural networks are useful for evolving the control system of an agent [28]. They provide a straightforward mapping between sensors and actuators, which enables them to directly represent the policy (control) or the value function to be learned. Reinforcement learning is useful as a type of learning where the agent is not told directly what to do, but is fed with a signal (i. e. reward) that measures the quality of executing an action in a given state [38]. The purpose of the agent is to act optimally in its environment, such that its rewards are maximized. Learning through interaction, of which reinforcement learning serves as a good example, underlies nearly all the principles of intelligence [30]. Reinforcement learning plays, therefore, a very important part in an evolutionary reinforcement learning system. Like neural networks, evolutionary algorithms [15, 24, 32] are inspired from biology. Populations of organisms adapt to their particular environmental conditions through evolutionary selection (survival of the fittest), and due to variation among the organisms. From these principles of adaptation in nature, it is possible to derive a number of concepts and strategies for solving learning tasks, and to develop optimization strategies for artificially intelligent systems. D. Liu, L. Wang, and K.C. Tan (Eds.): Design and Control of Intel. Robotic Systems, SCI 177, pp. 187–208. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com 

188

Y. Kassahun et al.

Fig. 10.1. Evolutionary reinforcement learning system. The agents (in which the neural networks are embedded) are evaluated in the environment, and their fitness values are returned to the evolutionary algorithm as rewards.

An evolutionary reinforcement learning system combining the principles of neural networks, reinforcement learning, and evolutionary algorithms is shown in Figure 10.1. The evolutionary algorithm works with genotypes representing neural networks to be evaluated in a given environment. Each neural network is evaluated and assigned a given fitness value (i. e. reward). The agents are improved by the evolutionary algorithm through the use of genetic operators. This process is repeated until a given number of generations have passed, or until an agent is found that solves the specified task. The method presented in this chapter uses an evolutionary reinforcement learning system, and is closely related to the works of Angeline et al.[1] and Stanley [35]. As in the work of Angeline et al., the method uses a type of parametric mutation that comes from evolution strategies and evolutionary programming, in which adaptive step sizes are used for optimizing the weights of the neural networks. Similar to the work of Stanley, the method starts with minimal structures, which are complexified along the evolution path. Despite sharing these two properties, the method has the following important features which distinguish it from previous works in the evolution of neural networks: 1. It introduces a genetic encoding called Common Genetic Encoding (CGE) that handles both direct and indirect encoding of neural networks with in the same theoretical framework. The encoding has important properties that makes it suitable for evolving neural networks: (1) It is complete in that it is able to represent all types of valid phenotype networks. (2) It is closed, i.e. every valid genotype represents a valid phenotype. Similarly, the encoding is closed under genetic operators such as structural mutation and crossover. These properties have been formally proven in [19].

10 Incremental Acquisition of Neural Structures through Evolution

189

2. For evolving the structures and weights of neural networks, a nature inspired metalevel evolutionary process is used, where exploration of structures is executed at a larger timescale, and exploitation of existing structures is done at a smaller timescale. The chapter is organized as follows. A summary of previous works in the area of evolution of neural networks is given in section 10.2. A more detailed discussion of the method presented in this work is given in section 10.3. Practical applications and evaluation of the method are presented in section 10.4. Finally, in section 10.5, a summary of the presented method and outlook are given.

10.2 Review of Related Works The evolution of neural networks can be divided into two major categories: the evolution of connection weights, and the evolution of both structure (i. e. network topology) and connection weights. In the first category, the structure of neural networks is fixed and is determined by the domain expert before the evolution begins. In the second category, both the structure and the connection weights are determined automatically by the evolutionary process. The work presented in this chapter falls into the second category. In this section, we review works in the second category that have been applied to reinforcement learning tasks. These works can be divided into two major groups, depending on whether their approach uses direct or indirect genetic encodings. For a detailed review of works in the evolution of neural networks, see [42]. 10.2.1

Direct Genetic Encoding Approaches

Approaches that use a direct encoding scheme specify explicitly every connection and node that will appear in the phenotype. Angeline et al. developed a system called GNARL (GeNeralized Acquisition of Recurrent Links) that uses only structural mutation for modifying the topology, and parametric mutation for modifying the weights [1]. The system is based on evolutionary programming, where the crossover operator is not used as a search operator. The system tries to maintain the behavior of the network, avoiding a radical jump from parent to offspring. This is done by initializing new links with zero weight, leaving the behavior of the modified network unchanged, and by adding hidden nodes to the network without any incident connections. The main problem this method faces is that genomes may end up having many extraneous disconnected structures that have no contribution to the solution. In contrast to GNARL, the Neuroevolution of Augmenting Topologies (NEAT) [35] evolves both the structure and weights of neural networks using both crossover and mutation operators. It starts with networks of minimal structure and increases their complexity along the evolution path. Every node and connection of the phenotype is encoded by the genotype. The algorithm keeps track of the historical origin of every gene that is introduced through structural mutation. The history is used by a specially designed crossover operator to match up genomes encoding different network topologies, and to create a new structure that combines the overlapping parts of the two parents as

190

Y. Kassahun et al.

well as their different parts. Structural discoveries of the evolutionary process are protected by niching (speciation). The speciation in NEAT is achieved by explicit fitness sharing, where organisms in the same species share the fitness of their niche. Unlike GNARL, NEAT does not use self-adaptation of mutation step-sizes. Each connection weight is perturbed with a fixed probability by adding a floating number chosen from a uniform distribution of positive and negative values. 10.2.2

Indirect Genetic Encoding Approaches

In indirect encoding, one specifies rules that are used in constructing the phenotype. Every connection and node in the network can be derived from the genome but is not specified explicitly. Kitano’s grammar based encoding of neural networks uses Lindenmayer systems (L-systems) [25] to describe the morphogenesis of linear and branching structures in plants [23]. L-systems are parallel string rewriting systems that rewrite a starting string into a new string by applying a set of production rules in parallel to all the symbols in the string. Sendhoff et al. extended Kitano’s grammar encoding with their recursive encoding of modular neural networks [33]. Their system provides a means of initialization of the network weights. Networks are trained using the standard back-propagation algorithm, and the encoding itself is variable and optimized on a larger timescale. Both Kitano and Sendhoff have used their systems to evolve feed-forward networks. In Kitano’s grammar based encoding, there is no direct way of representing the connection weights of neural networks in the genome. Gruau’s Cellular Encoding (CE) method is a language for local graph transformations that controls the division of cells which grow into an artificial neural network [13]. Through cell division, one cell, the parent cell, is replaced by two cells, the child cells. During division, a cell must specify how the two child cells will be linked. The genetic representations in CE are compact because genes can be reused multiple times during the development of the network. This saves space in the genome because not every connection and node need to be explicitly specified in the genome. Defining a crossover operator for CE is still difficult, and it is not easy to analyze how crossover affects the subfunctions in CE since they are not represented explicitly. Moreover, CE suffers from the same problem as that of Kitano’s grammar based encoding since there is no direct way of representing the connection weights of neural networks in the genome. Vaario et al. have developed a biologically inspired neural growth, which is based on diffusion field modeling combined with genetic factors for controlling the growth of the network [40]. The neural structures are grown in either a two or a three dimensional grid, resulting in a two or three dimensional tree-based neural structure, respectively. One weak point of this method is that it cannot generate networks with recurrent connections or networks with connections between neurons on different branches of the resulting tree structure. Nolfi and Parisi modelled biological development at the chemical level using a reaction-diffusion model [29]. Diffusion is modelled at the level of axon growth, and reaction is modelled as the interaction between axon and cell bodies. This method utilizes growth to create connectivity without explicitly describing each connection in the phenotype. Axons which do not reach other cells during development are pruned. The

10 Incremental Acquisition of Neural Structures through Evolution

191

complexity represented by the genome is limited since every neuron is directly specified in the genome. Other works in indirect encoding have simulated genetic regulatory networks (GRNs) in biology, where genes produce signals that either activate or inhibit other genes in the genome. The interaction of all genes forms a network that produces a phenotype. Representative works that use GRNs include the works of Dellaert and Beer [7], Jakobi [17], Bongard and Pfeifer [6], and Bentley and Kumar [5].

10.3 Evolutionary Acquisition of Neural Topologies (EANT) This section presents an evolutionary reinforcement learning system named Evolutionary Acquisition of Neural Topologies (EANT). It is a novel method that is suitable for learning and adaptation to the environment through interaction. The system evolves both the structures and weights of neural networks. With respect to the goal of selforganizing learning machines that start from minimal specifications and rise to great sophistication, EANT starts with neural networks of minimal structures, and increases their complexity along the evolution path. A defining characteristic of EANT is its ability to influence the ratio of exploration to exploitation that occurs during the evolutionary process. Although EANT can be used with any arbitrary genetic encoding that can represent neural networks, we have chosen to use the Common Genetic Encoding (CGE), an encoding with features that make it well suited for use in the evolution of neural networks. CGE will be presented in what follows, including a description of three genetic operators defined on CGE-genomes. Building on this, two methods for computing the output values of a CGE-genotype for a given input will be discussed. After this, a method will be presented which can be used to generate an initial population with a high amount of diversity, yet having low structural complexity. Finally, the exploitation/exploration scheme used in EANT will be discussed. 10.3.1

Common Genetic Encoding (CGE)

A flexible encoding method enables one to design an efficient evolutionary method that can evolve both the structures and weights of neural networks. A genome in CGE is designed by taking this fact into consideration. The genome in CGE is a linear genome consisting of genes (nodes) that can take different forms (alleles). The forms that can be taken by a gene can either be a neuron, or an input to the neural network, or a jumper connecting two neurons. The jumper genes are introduced by one of the genetic operators, the structural mutation (compare Subsection 10.3.2), along the evolution path. A jumper gene can either encode a forward or a recurrent connection. Jumper genes encoding a forward connection are not allowed to encode connections starting from a neuron with lower depth1 and ending at a neuron with higher depth. On the other hand, a jumper gene encoding a recurrent connection represents a connection between neurons having the same depth, or a connection starting from a neuron at a lower depth 1

The depth of a vertex is defined as the minimal topological distance (i.e. minimal number of connections to be traversed) from an output vertex of the network to the vertex itself.

192

Y. Kassahun et al.

(a)

(b)

Allele + ID N 0 N 1 N 3 I x I y I y N 2 JF 3 I x I y JR 0 Weight 0.6 0.8 0.9 0.1 0.4 0.5 0.2 0.3 0.7 0.8 0.2 Number of inputs 2 2 2 – – – 4 – – – – (c) Fig. 10.2. An example of encoding a neural network using a CGE. (a) The neural network to be encoded, with one forward and one recurrent jumper connection. (b) The neural network interpreted as a tree structure, where the jumper connections are considered as terminals. (c) The linear genome encoding the neural network shown in (a). In the linear genome, N stands for a neuron, I for an input to the neural network, JF for a forward jumper connection, and JR for a recurrent jumper connection. The numbers beside N represent the global identification numbers of the neurons, and x or y represent the labels of the inputs encoded by the input genes.

and ending at a neuron at a higher depth. Every node in a linear genome has a weight associated with it. The weight encodes the synaptic strength of the connection between the node coded by the gene and the neuron to which it is connected. Moreover, every node can save the results of its current computation. This is useful since the results of signals at recurrent links are available at the next time step. In addition to the synaptic weight, a neuron node has a unique global identification number and stores the number of input connections associated with it. A jumper node has also additionally a global identification number, which stores the neuron to which it is connected. An example of a linear genome encoding a neural network is shown in Figure 10.2. The linear genome has some interesting properties that makes it useful for evolution of the structure of neural networks [19]. Assume that integer values are assigned to the nodes of a linear genome encoding a neural network such that the integer values show the difference between the number of outputs of the nodes and the number of arguments of the nodes (inputs to the nodes). Note that every node in the linear genome has only one output. If a node is an input to the neural network, the integer assigned to it is 1 since an input to a neural network has only one output and no arguments (inputs) at all. An integer value of 1 is also assigned to the forward and recurrent jumper nodes since they are sources of signals from other neurons in the neural network encoded by the linear genome. A neuron node will take an integer value which is one minus the number of inputs to the neuron. In CGE, since there is no neuron without an input, the

10 Incremental Acquisition of Neural Structures through Evolution

193

Fig. 10.3. An example of the use of assigning integer values to the nodes of the linear genome. The linear genome encodes the neural network shown in Figure 10.2. The numbers in the square brackets below the linear genome show the integer values assigned to the nodes of the linear genome. Note that the sum of the integer values is one showing that the neural network encoded by the linear genome has only one output. The shaded nodes form a sub-network. Note also that the sum of the integer values assigned to a sub-network is always one.

maximum value of an integer assigned to a neuron node is zero. This is true for all neurons with only one input. One of the important properties of a linear genome is that the sum of the integer values assigned to the nodes in a linear genome encoding a neural network is the same as the number of outputs of the neural network. After assigning the integer values to the nodes of the linear genome, it is possible to detect a sub-genome (sub-network) of a linear genome. A sub-genome in CGE is defined as a collection of nodes starting from a neuron node and ending at a node where the sum of integer values assigned to the nodes between and including the start neuron node and the end node is one. An example is shown in figure 10.3. In [19], Kassahun et al. have discussed the representation capabilities of CGE. They proved that the linear genome is complete in that it can be used to represent any type of neural network. It is also a compact encoding of neural networks since the length of the linear genome is the same as the number of synaptic weights in the neural network. Moreover, the encoding scheme used is closed. An encoding scheme is said to be closed if all genotypes produced are mapped into a valid set of phenotype networks [3, 18]. 10.3.2

Genetic Operators

In this subsection, we present the three genetic operators defined on CGE genotypes. The first one, the parametric mutation operator perturbs only the synaptic weights, the network structure itself remains unchanged. In contrast, the structural mutation operator changes the network structure by adding genes to the genotype but does not change the network’s weights. The last genetic operator, the structural crossover takes two genomes as inputs (the parents) and produces one new genome (the offspring) which comprises the properties of its parents. Kassahun et al. [19] have proven that linear genomes are closed under parametric and structural mutation. A genome is said to be closed under a genetic operator if the newly generated structure always maps onto a valid phenotype network. Furthermore, CGE is also closed under the structural crossover operator. We continue with a detailed description of the three genetic operators. Parametric Mutation Parametric mutation is used for optimization of the weights of a given structure. It is accomplished by perturbing the synaptic weights of the networks according to the

194

Y. Kassahun et al.

uncorrelated mutation in evolution strategy or evolutionary programming [9, 31]. In addition to the associated weight wi , each node in a linear genome has an associated mutation step size or learning rate2 σi . The mutation mechanism is specified as follows [8]:  σi = σi eτ N(0,1)+τ Ni (0,1) , (10.1) wi = wi + σi Ni (0, 1) , (10.2)  √ √ where τ  = 1/ 2n and τ = 1/ 2 n, and N (0, 1) as well as Ni (0, 1) are random numbers drawn from a Gaussian distribution of zero mean and unity standard deviation. If one wants to employ a global learning rate σ , one can replace Equation 10.1 by

σ  = σ eτ N(0,1)

(10.3)

There are two main advantages of parametrically mutating the synaptic weights of neural networks in the same fashion as is done in evolution strategies or evolutionary programming: 1. Offspring created by mutation remain within a locus of similarity to their parents [1]. 2. Self-adaptation of mutation step sizes of learning rates is inherent in both evolution strategies and evolutionary programming [31, 9]. Structural Mutation The structural mutation used by CGE adds a forward or a recurrent jumper connection between neurons, or adds a whole new subgenome to the linear genome. If a subgenome is added, this genome consists of a neuron node followed by an arbitrary number of input or jumper genes. The initial weight of a newly added jumper connection or the initial weight of the first node of a newly added sub-network are set to zero so as not to disturb the behavior of the neural network. When applying the structural mutation, one neuron node from the genome is chosen randomly and will be mutated. This mutation can be either the insertion of a jumper gene or of a whole subnetwork after the neuron node. Which mutation is actually performed is determined randomly as well as the source nodes for jumper genes. If a forward jumper gene is added to the genome, it is taken care that this does not result in a closed cycle of forward jumper genes since this would yield in problems during the evaluation of the network later on (infinite loops). Figure 10.4 shows an example where the structural property of a neuron node N3 is changed through the structural mutation: A new forward jumper gene is added and the number of input of N3 is increased by one. The new forward jumper stores the id of its source node N2. In the corresponding phenotype, this results in an additional edge going from N2 to N3. 2

It is possible to enforce that all genes of a linear genome have the same learning rate (called global learning rate).

10 Incremental Acquisition of Neural Structures through Evolution

195

Fig. 10.4. An example of structural mutation. A forward jumper gene connecting the node N2 with the node N3 is added and the number of inputs of N3 is increased by one.

Structural Crossover The third genetic operator defined is the Structural Crossover operator. This operator exploits the fact that structures which originate from the same ancestor structure have some parts in common. By aligning the common parts of two randomly selected structures, it is possible to generate a third structure which contains the common and disjoint parts of the two mother structures. The resulting structure formed in this way maps to a valid phenotype network. This type of crossover was introduced and used by Stanley [35]. An example is given in Figure 10.5. The number of inputs to a neuron node which is common to the parent structures is updated using n(s1 × s2 ) = n(s1 ) + n(s2 ) − n(s1 ∩ s2 ),

(10.4)

where n(s1 × s2 ) is the number of inputs to the neuron node in the offspring, n(s1 ∩s2 ) is the number of inputs to the neuron node which are common to both structures, and n(s1 ) and n(s2 ) are the number of inputs to the neuron node in the parent structure s1 and s2 , respectively. Furthermore, if the parents have some genes in common whose weights differ, one of this weights is chosen randomly for the offspring. An other option would be to take the mean of these two values as new weight. 10.3.3

Evaluating a Linear Genome

In CGE, there are two methods of evaluating a linear genome. In the first method, one decodes the linear genome into a neural network and evaluate this neural network

196

Y. Kassahun et al.

Fig. 10.5. Performing crossover between two linear genomes. The genetic encoding is closed under this type of crossover operator since the resulting linear genome maps to a valid phenotype network. The weights of the nodes of the resulting linear genomes are inherited randomly from both parents.

directly. In the second method, it is not necessary to decode the linear genome into a neural network but one can use the linear genome directly to evaluate the neural network represented by it. The second method emphasizes the fact that it is not always necessary to create a separate phenotype structure from genotype by some sort of embryogeny [4]. In other words, it is not always necessary to decode the linear genome into a neural network for the purpose of evaluating the network encoded by it. For computing the output of the neural network without decoding the genome, we use a first in last out stack and the following rules: 1. Start from the right most node of the linear genome. 2. Traverse the linear genome from right to left in computing the output of the neural network:

10 Incremental Acquisition of Neural Structures through Evolution

197

Fig. 10.6. An example of evaluating a linear genome without decoding the neural network encoded by it. The linear genome encodes the neural network shown in Figure 10.2. For this example, the current values of the inputs to the neural network, x and y, are both set to 1. In the example, all neurons have a linear activation function of the form g(a) = a, where a is the weighted linear combination of the inputs to a neuron. The overlapped numbers above the linear genome show the status of the stack after computing the output of a node. The numbers in brackets are the weights associated with the nodes.

a) If the current node is an input node, push its current value and the weight associated with it onto the stack. b) If the current node is a neuron node with n inputs, pop n values with their associated weights from the stack and push the result of computation with its associated weight onto the stack. The output of a neuron node j is computed using   n

aj = g

∑ wi ai

,

(10.5)

i=1

where ai are the values popped from the stack and wi are their associated weights. g (.) is the transfer function of the neuron node. c) If the current node is a recurrent jumper node, get the last value (i. e. the output of the node in the last evaluation of the neural network) of the neuron node identified by the source unique id stored in this recurrent jumper. Then push the value obtained with the weight associated with the jumper node onto the stack. d) If the current node is a forward jumper node, first copy the subgenome starting from the neuron whose global identification number is the same as the global identification number of the forward jumper node. Then compute the response of the subgenome in the same way as that of the linear genome. Finally, push the result of computation with the weight associated with the forward jumper node onto the stack. 3. After traversing the genome from right to left completely, pop the resulting values from the stack. The number of the resulting values is the same as the number of outputs of the neural network encoded by the linear genome. Figure 10.6 shows an example of evaluating a linear genome encoding the neural network shown in Figure 10.2. As can be seen from the figure, one does not need to decode the neural network in order to evaluate it. The evaluation of a linear genome discussed above is equivalent to the evaluation of a decoded neural network represented by the genome: Let xˆi be a neuron with incoming

198

Y. Kassahun et al.

connections from the inputs xˆ1 , ..., xˆk and the neurons xˆk+1 , ..., xˆl , and the incoming recurrent connections from neurons xˆl+1 , ..., xˆm . For an arbitrarily chosen transfer function g, the output ai (t) of the neuron xˆi at time step t is computed using k

ai (t) = g( ∑ w j I j (t) + j=1

l

∑

m

w j a j (t) +

j=k+1

∑

w j a j (t − 1)),

(10.6)

j=l+1

where the values of I j (t) represent the inputs of the neural network and a j (t − 1) is the output of neuron j at time step t − 1. 10.3.4

Generating the Initial Population

Traditionally, there are two methods used for the generation of initial populations of genomes. These are the grow and full methods [4]. In EANT, we used a third approach which enforces that all genomes originate from the same ancestor genome. This “proto” linear genome contains only output neuron nodes and input nodes. It connects all inputs directly with all outputs3 without having neurons in a hidden layer. One requirement of the initial population of a neuroevolutionary method is that it is sufficiently diverse while its members (the individuals in the population) are still fairly simple. To achieve this, the following approach has been applied in EANT: For a population of size n, n copies of the “proto” individual are created. Each of these copies is then separately mutated by the structural and parametric mutation operator m times. The bigger m is, the more diverse the initial population gets while losing more and more the minimality of its members. All n individuals created with these procedure have still some parts in common, since the mutation operators do never erase a part of an ancestor. Note that this procedure can easily be adjusted to work together with techniques like speciation as discussed by Metzen [26]. 10.3.5

Exploitation and Exploration of Structures

As discussed in Subsection 10.3.4, EANT starts with networks of minimal structure whose initial complexity is controlled the by the choice of m, the number of initial mutations to be applied. The evolution itself is divided into two alternating processes: The exploitation process which is occurring at a smaller timescale and the exploration process occurring at larger timescale. In the exploitation process, the weights of existing structures are optimized. The exploitation process itself is subdivided into different generations. In each generation, first of all, the fitness of all individuals is determined. Based on this, a mating pool consisting of n < n individuals is created using well known selection methods like Stochastic Universal Sampling [2]. Based on this mating pool, the population of the next generation is created using the parametric mutation operator as only active genetic operator. Thereby, the underlying network structures remain unchanged while only the individuals whose weights perform good for the given structure survive. After a certain number 3

The number of outputs as well as number of inputs of the neural network needs to be specified for each task anew.

10 Incremental Acquisition of Neural Structures through Evolution

Trajectory in weight space

199

t N0 W=5.0

Ix W=5.0

Iy W=5.0

5.0

t+1

0

w(Iy)

w(Ix)

5.0

5.0

x

y

w(N0)

Fig. 10.7. The weight trajectory of the linear genome shown at the right side while being exploited. The quantities t and t + 1 are time units with respect to the larger timescale. The weights of the existing structures are optimized between two consecutive time units with respect to the larger timescale. The point clouds at t and t + 1 show populations of individuals from the same species.

of generations (usually between five and ten), the weights of the existing structures are fairly well optimized. An example for the change of the weights for a given structure during the exploitation process is shown in Figure 10.7. The exploration process occurs at a larger timescale and is used to introduce new structures. In each step of exploration, first the weights of all existing structures are optimized using the exploitation process. Then, selection is performed on this optimized genomes. Since the weights of all structures were optimized independently, the performance of these individuals in the exploration step should not depend too strong on the actual weights, rather the actual fitness should be a good estimate for the