Reconstruction nstruction of
Chaotic Signals with Applications to Chaos-Based Communications
Jiu Chao Feng South China University ofTechnology, Guangzhou, China
Chi Kong Tse The Hong Kong Polytechnic University, Hong Kong
World Scientific NEW JERSEY· LONDON· SINGAPORE· BEIJING· SHANGHAI· HONG KONG· TAIPEI· CHENNAI
Published by
jl'ii/'!7c"j: W!\&ft (Tsinghua University Press) -I tfJ\ jj!j i/'! A"t"t~3T7c[l A ~ ~ll;A1U: 100084
and World Scientific Publishing Co. Pte. Ltd. 5 Toh Tuck Link. Singapore 596224 USA office: 27 Warren Street, Suite 401-402. Hackensack, NJ 07601 UK office: 57 Shelton Street, Covent Garden, London WC2H 9HE
ill i'i': {1§ ~ fl'J 4l: fiJ Ez ~:tE cljk Till i'i': 81 jffi {151' 81 @. ffi ~ Reconstruction of Chaotic Signals with Applications (0 Chaos-Based Communicacions/lllJ;7\11'lL iMi'I'lxjIJet,-)t.M: lfii/'!:;J(''¥:WhiRH, 2007.11 ISBN 978-7-302-12120-6
British Library Cataloguing-in-Publication Data A catalogue record for this book is available from the British Library.
RECONSTRUCTION OF CHAOTIC SIGNALS WITH APPLICATIONS TO CHAOS-BASED COMMUNICATIONS Copyright © 2008 by Tsinghua University Press and World Scientific Publishing Co. Pte. Ltd. All rights reserved. This book, or parts thereof: may IlOt be reproduced ill anv j(mn or by any meallS, electronic or mechanical, including photocopying, recording or any information storage and retrieval system /lOW kllown or to be invented, without written pemlissionjrom the Publisher.
For photocopying of material in this volume, please pay a copying fee through the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, USA. In this case permission to photocopy is not required from the publisher.
ISBN-13 978-981-277-113-1 ISBN-l0 981-277-113-1
Printed in Singapore by World Scientific Printers
Preface
The study of time series has traditionally been within the realm of statistics. A large number of both theoretical and practical algorithms have been developed for characterizing, modeling, predicting and filtering raw data. Such techniques are widely and successfully used in a broad range of applications, e.g., signal processing and communications. However, statistical approaches use mainly linear models, and are therefore unable to take advantage of recent developments in nonlinear dynamics. In particular, it is now widely accepted that even simple nonlinear deterministic mechanisms can give rise to complex behavior (i.e., chaos) and hence to complex time series. Conventional statistical time-series approaches are unable to model or predict complex time series with a reasonable degree of accuracy. This is because they make no use of the fact that the time series has been generated by a completely deterministic process, and hence ascribe most of the complexity to random noise. Furthermore, such approaches cannot yield much useful information about the properties of the original dynamical system. Fortunately, a remarkable result due to Takens shows that one can reconstruct the dynamics of an unknown deterministic finite-dimensional system from a scalar time series generated by that system. Takens' theorem is actually an extension of the classical Whitney's theorem. It is thus concerned with purely deterministic autonomous dynamical systems and the framework that it provides for time series analysis is unable to incorporate any notion of random behavior. This means that the process of reconstruction is outside the scope of statistical analysis because any such analysis would require a stochastic model of one kind or another as its starting point. This is reflected in common practice, where reconstruction is seen as a straightforward algorithmic procedure that aims to recover properties of an existing, but hidden, system.
Reconstruction of Chaotic Signals with Applications to Chaos-Based Communications
The problem of reconstructing signals from noisy corrupted time series arises in many applications. For example, when measurements are taken from a physical process suspected of being chaotic, the measuring device introduces error in the recorded signal. Alternatively, the actual underlying physical phenomenon may be immersed in a noisy environment, as might be the case if one seeks to detect a low-power (chaotic) signal (possibly used for communications) that has been transmitted over a noisy and distorted channel. In both cases, we have to attempt to purify or detect a (chaotic) signal from noisy and/or distorted samples. Separating a deterministic signal from noise or reducing noise in a noisy corrupted signal is a central problem in signal processing and communications. Conventional methods such as filtering make use of the differences between the spectra of the signal and noise to separate them, or to reduce noise. Most often the noise and the signal do not occupy distinct frequency bands, but the noise energy is distributed over a large frequency interval, while the signal energy is concentrated in a small frequency band. Therefore, applying a filter whose output retains only the signal frequency band reduces the noise considerably. When the signal and noise share the same frequency band, the conventional spectrumbased methods are no longer applicable. Indeed, chaotic signals in the time domain are neither periodic nor quasi-periodic, and appear in the frequency domain as wide "noise-like" power spectra. Conventional techniques used to process classic deterministic signals will not be applicable in this case. Since the property of self-synchronization of chaotic systems was discovered in 1990 by Pecora and Carroll, chaos-based communications have received a great deal of attention. However, despite some inherent and claimed advantages, communicating with chaos remains, in most cases, a difficult problem. The main obstacle that prevents the use of chaotic modulation techniques in real applications is the very high sensitivity of the demodulation process. Coherent receivers which are based on synchronization suffer from high sensitivity to parameter mismatches between the transmitter and the receiver and even more from signal distortion/contamination caused by the communication channel. Non-coherent receivers which use chaotic signals for their good decorrelation properties have been shown to be more robust to channel noise; however, as iv
Preface
they rely mainly on (long-term) decorrelation properties of chaotic carriers, their performances are very close to those of standard non-coherent receivers using pseudo-random or random signals. In both cases, coherent and non-coherent, it has been shown that noise reduction (or signal separation) can considerably improve the performance of chaos-based communication systems. The aforementioned problems can be conveniently tackled if signals can be reconstructed at the receiving end. Motivated by the general requirements for chaotic signal processing and chaos-based communications, this book addresses the fundamental problem of reconstruction of chaotic signals under practical communication conditions. Recently, it has been widely recognized that artificial neural networks endow some unique attributes such as universal approximation (input-output mapping), the ability to learn from and adapt to their environments, and the ability to invoke weak assumptions about the underlying physical phenomena responsible for the generation of the input data. In this book, the technical approach to the reconstruction problem is based on the use of neural networks, and the focus is the engineering applications of signal reconstruction algorithms. We begin in Chapter 1 by introducing some background information about the research in signal reconstruction, chaotic systems and the application of chaotic signals in communications. The main purpose is to connect chaos and communications, and show the potential benefits of using chaos for communications. In Chapter 2 we will review the state of the art in signal reconstruction, with special emphasis laid on deterministic chaotic signals. The Takens' embedding theory will be reviewed in detail, covering the salient concepts of reconstructing the dynamics of a deterministic system from a higher-dimensional reconstruction space. The concepts of embedding dimension, embedding lag (time lag), reconstruction function, etc., will be discussed. Two basic embedding methods, namely topological and differential embeddings, will be described. Some unsolved practical issues will be discussed. A thorough review of the use of neural networks will be given in Chapter 3. Two main types of neural networks will be described in detail, namely, the radial-basis-function neural networks and the recurrent neural networks. The background theory,
v
Reconstruction of Chaotic Signals with Applications to Chaos-Based Communications
network configurations, learning algorithms, etc., will be covered. These networks will be shown suitable for realizing the reconstruction tasks in Chapters 7 and 8. In Chapter 4 we will describe the reconstruction problem when signals are transmitted under ideal channel conditions. The purpose is to extend the Takens' embedding theory to time-varying continuous-time and discrete-time systems, i.e., nonautonomous systems. This prepares the readers for the more advanced discussions given in the next two chapters. In Chapter 5, we will review the Kalman filter (KF) and the extended Kalman filter (EKF) algorithms. In particular, we will study a new filter, i.e., the unscented Kalman filter (UKF). The issue for filtering a chaotic system from noisy observation by using the UKF algorithm will be investigated. A lot of new finding of the UKF algorithm will be demonstrated in this chapter. In Chapter 6, we will apply the UKF algorithm to realize the reconstruction of chaotic signals from noisy and distorted observation signals, respectively. Combining the modeling technique for signal with the UKF algorithm, the original UKF algorithm will be expanded to filter timevarying chaotic signals. We will also address the issues of blind equalization for chaos-based communication systems. Our start point is to make use of the UKF algorithm and the modeling technique for signal, and to realize a blind equalization algorithm. In Chapter 7 we will present novel concepts of using a radial-basisfunction neural network for reconstructing chaotic dynamics under noisy condition. A specific adaptive training algorithm for realizing the reconstruction task will be presented. Results in terms of the mean-squared-error versus the signal-to-noise ratio will be given and compared with those obtained from conventional reconstruction approaches. Also, as a by-product, a non-coherent detection method used in chaos-based digital communication systems will be realized based on the proposed strategy. In Chapter 8 we will continue our discussion of signal reconstruction techniques. Here, we will discuss the use of a recurrent neural network for reconstructing chaotic dynamics when the signals are transmitted through distorting and noisy channels. A special training algorithm for realizing the reconstruction task will be presented. This problem will also be discussed under the conventional viewpoint of channel equalization. In Chapter 9, the reconstruction of chaotic signals will be considered in terms of chaos vi
Preface
synchronization approaches. In particular, the problem of multiple-access spreadspectrum synchronization for single point to multiple point communication is considered in the light of a chaotic network synchronization scheme, which combines the Pecora-Carroll synchronization scheme and the OOY control method. The results indicate that such a network synchronization scheme is applicable to fast synchronization of multiple chaotic systems, which is an essential condition for realizing single point to multiple point spread-spectrum communications. Simulation results indicate that such a network synchronizationbased communication system is effective and reliable for noisy and multi-path channels. The final chapter summarizes the key results in this research. In closing this preface, we hope that this book has provided a readable exposition
of the related theories for signal reconstructions as well as some detailed descriptions of specific applications in communications. We further hope that the materials covered in this book will serve as useful references and starting points for researchers and graduate students who wish to probe further into the general theory of signal reconstruction and applications. Jiuchao Feng, Guangzhou Chi K. Tse, Hong Kong
vii
Acknowledgements
The completion of this book would have been impossible without the help and support of several people and institutions. We wish to express our most grateful thanks to Prof. Ah-Chung Tsoi from University of Wollongong, Australia, and Prof. Michael Wong from Hong Kong University of Science and Technology, Hong Kong, who have read an early version of this book (which was in the form of a Ph.D. thesis) and have offered countless suggestions and comments for improving the technical contents of the book. We also wish to thank Dr. Francis Lau and Dr. Peter Tam from Hong Kong Polytechnic University, Hong Kong, for many useful and stimulating discussions throughout the course of our research. The first author gratefully acknowledges the Hong Kong Polytechnic University for providing financial support during his Ph.D. study. He also thanks the following institutes and departments for their financial support in the course of preparing this book and in its production: The Natural Science Foundation of Guangdong Province, China (Grant Number 05006506); The Research Foundation of Southwest China Normal University, China (Grant Number 413604); The Natural Science Foundation of Guangdong Province, China, for Team Research (Grant Number 04205783); The Program Foundation for New Century Excellent Talents in Chinese University (Grant Number NCET-04-0813); The Key Project Foundation of the Education Ministry of China (Grant Number 105137); The National Natural Science Foundation of China (Grant Number 60572025); The publishing Foundation for Postgraduate's Textbook in South China University of Technology, China. He is also grateful to his graduate student Shiyuan Wang and Hongjuan Fan for their simulation work in partial chapters. Last, but not least, we wish to thank our families for their patience, understanding and encouragement throughout the years.
To our families Yao Tan and Lang Feng Belinda and Eugene
This page intentionally left blank
Contents
Preface .......................................................................................................... iii Acknowledgements ...................................................................................... ix 1 Chaos and Communications ................................................................... 1 1.1
Historical Account ........................................................................... 2
1.2
Chaos ................................................................................................ 5
1.3
Quantifying Chaotic Behavior ......................................................... 5 1.3.1
Lyapunov Exponents for Continuous-Time Nonlinear
1.3.2
Lyapunov Exponent for Discrete-Time Systems ................. 8
1.3.3
Kolmogorov Entropy ............................................................ 8
1.3.4
Attractor Dimension ........................................................... 10
Systems ................................................................................ 6
1.4
Properties of Chaos ........................................................................ 12
1.5
Chaos-Based Communications ...................................................... 14
1.6
1.7
1.5.1
Conventional Spread Spectrum .......................................... 14
1.5.2
Spread Spectrum with Chaos ............................................. 16
1. 5.3
Chaotic Synchronization .................................................... 16
Communications Using Chaos as Carriers ..................................... 19 1.6.1
Chaotic Masking Modulation ............................................. 19
1.6.2
Dynamical Feedback Modulation ...................................... 20
1.6.3
Inverse System Modulation ................................................ 21
1.6.4
Chaotic Modulation ............................................................ 22
1.6.5
Chaos Shift Keying ............................................................ 22
1.6.6
Differential Chaos Shift Keying Modulation ..................... 24
Remarks on Chaos-Based Communications .................................. 25
Reconstruction of Chaotic Signals with Applications to Chaos-Based Communications
2
1.7.2
Engineering Challenges ........................................... ,.......... 25
Reconstruction of System Dynamics ............................................. 28 2.1.1
Topological Embeddings .................................................... 29
2.1.2
Delay Coordinates .............................................................. 30
2.2
Differentiable Embeddings ............................................................ 33
2.3
Phase Space Reconstruction-Example ........................................ 34
2.4
Problems and Research Approaches .............................................. 39
Fundamentals of Neural Networks ..................................................... .41
3.1
Motivation ...................................................................................... 41
3.2
Benefits of Neural Networks ......................................................... .43
3.3
Radial Basis Function Neural Networks ....................................... .46
3.4
4
Security Issues .................................................................... 25
Reconstruction of Signals ................. ,................................................... 27
2.1
3
1.7.1
3.3.1
Background Theory ........................................................... .46
3.3.2
Research Progress in Radial Basis Function Networks ..... .49
Recurrent Neural Networks ........................................................... 56 3.4.1
Introduction ........................................................................ 56
3.4.2
Topology of the Recurrent Networks ................................. 57
3.4.3
Learning Algorithms .......................................................... 58
Signal Reconstruction in Noisefree and Distortionless Channels ...... 60
4.1
Reconstruction of Attractor for Continuous Time-Varying Systems .......................................................................................... 60
4.2
Reconstruction and Observability .................................................. 62
4.3
Communications Based on Reconstruction Approach ................... 63
4.4
4.3.1
Parameter Estimations ........................................................ 64
4.3.2
Information Retrievals ........................................................ 66
Reconstruction of Attractor for Discrete Time-Varying Systems .......................................................................................... 69
4.5 xii
Summary ........................................................................................ 71
Contents
5
Signal Reconstruction from a Filtering Viewpoint: Theory .... ............ '" 72
5.1
5.2
5.3 6
The Kalman Filter and Extended Kalman Filter ............................ 72 5.1.1 The Kalman Filter .............................................................. 72 5.1.2 Extended Kalman Filter ..................................................... 76 The Unscented Kalman Filter ........................................................ 77 5.2.1 The Unscented Kalman Filtering Algorithm ...................... 78 5.2.2 Convergence Analysis for the UKF Algorithm .................. 82 5.2.3 Computer Simulations ........................................................ 86 Summary ........................................................................................ 89
Signal Reconstruction from a Filtering Viewpoint: Application ............ 91 6.1 Introduction .................................................................................... 91
6.2
Filtering of Noisy Chaotic Signals ................................................. 92 6.2.1 Filtering Algorithm ............................................................ 92 6.2.2 Computer Simulation ......................................................... 94
Blind Equalization for Fading Channels ...................................... 101 6.3.1 Modeling of Wireless Communication Channels ............. 101 6.3.2 Blind Equalization of Fading Channels with Fixed Channel Coefficients ........................................................ 103 6.3.3 Blind Equalization for Time-Varying Fading Channels .... 106 6.4 Summary ...................................................................................... 109
6.3
7
Signal Reconstruction in Noisy Channels .......................................... l1O 7.1 Review of Chaotic Modulation .................................................... 11 0
7.2 7.3
7.4 7.5
F onnulation of Chaotic Modulation and Demodulation .............. 112 On-Line Adaptive Learning Algorithm and Demodulation ......... 116 7.3.1 Description ofthe Network .............................................. 116 7.3.2 Network Growth ............................................................... 118 7.3.3 Network Update with Extended Kalman Filter ................ 119 7.3.4 Pruning of Hidden Units .................................................. 121 7.3.5 Summary of the Flow of Algorithm ................................. 121 Computer Simulation and Evaluation .......................................... 123 Application to Non-coherent Detection in Chaos-Based Communication ............................................................................ 131 xiii
Reconstruction of Chaotic Signals with Applications to Chaos-Based Communications
7.6
Summary ...................................................................................... 139
8
Signal Reconstruction in Noisy Distorted Channels ......................... 140 8.1 Preliminaries ................................................................................ 141 8.1.1 Conventional Equalizers .......................... ,....................... 142 8.1.2 Reconstruction of Chaotic Signals and Equalization ....... 143 8.1.3 Recurrent Neural Network and Equalization ................... 144 8.2 Training Algorithm ...................................................................... 148 8.3 Simulation Study .......................................................................... 151 8.3.1 Chaotic Signal Transmission ............................................ 151 8.3.2 Filtering Effect of Communication Channels ................... 152 8.3.3 Results .............................................................................. 156 8.4 Comparisons and Discussions ...................................................... 161 8.5 Summary ...................................................................................... 164
9
Chaotic Network Synchronization and Its Applications in Communications .................................................................................. 165 9.1
9.2
9.3 10
Chaotic Network Synchronization ............................................... 166 9.1.1 Network Synchronization ................................................. 167 9.1.2 Chaos Contro1. .................................................................. 168 9.1.3 Implementation of the Synchronization Scheme ............. .172 Implementation of Spread-Spectrum Communications ............... 175 9.2.1 Encoding and Decoding ................................................... 175 9.2.2 Test Results for Communications .................................... 178 Summary ...................................................................................... 181
Conclusions ....................................................................................... 183 10.1 Summary of Methods .............................................................. 183 10.2 Further Research ..................................................................... 185
Bibliography ............................................................................................. 188 Index .......................................................................................................... 214 xiv
Chapter 1
Chaos and Communications
Traditionally, signals (encompassing desired signals as well as interfering signals) have been partitioned into two broadly defined classes, i.e., stochastic and deterministic. Stochastic signals are compositions of random waveforms with each component being defined by an underlying probability distribution, whereas deterministic signals are resulted from deterministic dynamical systems which can produce a number of different steady state behaviors including DC, periodic, and chaotic solutions. Deterministic signals may be described mathematically by differential or difference equations, depending on whether they evolve in continuous or discrete-time. DC is a nonoscillatory state. Periodic behavior is the simplest type of steady state oscillatory motion. Sinusoidal signals, which are universally used as carriers in analog and digital communication systems, are periodic solutions of continuous-time deterministic dynamical systems. Deterministic dynamical systems also admit a class of nonperiodic signals, which are characterized by a continuous "noiselike" broad power spectrum. This is called chaos. Historically, at least three achievements were fundamental to the acceptance of communication using chaos as a field worthy of attention and exploitation. The first was the implementation and characterization of several electronic circuits exhibiting chaotic behavior in early 1980's. This brought chaotic systems from mathematical abstraction into application in electronic engineering. The second historical event in the path leading to exploitation for chaosbased communication was the observation make by Pecora and Carroll in 1990 that two chaotic systems can synchronize under suitable coupling or driving conditions. This suggested that chaotic signals could be used for
Historical Account
1.1
communication, where their noise like broadband nature could Improve disturbance rejection as well as security. A third, and fundamental, step was the awareness of the nonlinear (chaos) community that chaotic systems enjoy a mixed deterministic / stochastic nature [1 - 4]. This had been known to mathematicians since at least the early 1970's, and advanced methods from that theory have been recently incorporated in the tools of chaos-based communication engineering. These tools were also of paramount importance in developing the quantitative models needed to design chaotic systems that comply with the usual engineering specifications. The aim of this chapter is to give a brief review of the background theory for chaos-based communications. Based on several dynamical invariants, we will quantitatively describe the chaotic systems, and summarize the fundamental properties of chaos that make it useful in serving as a spreadspectrum carrier for communication applications. Furthermore, chaotic synchronization makes it possible for chaos-based communication using the conventional coherent approach. In the remaining part of this chapter, several fundamental chaotic synchronization schemes, and several chaosbased communication schemes will be reviewed. Finally, some open issues for chaos-based communications will be discussed.
1.1
Historical Account
In 1831, Faraday studied shallow water waves
III
a container vibrating
vertically with a given frequency OJ. In the experiment, he observed the sudden appearance of sub harmonic motion at half the vibrating frequency (OJ / 2) under certain conditions. This experiment was later repeated by Lord
Rayleigh who discussed this experiment in the classic paper Theory of Sound, published in 1877. This experiment has been repeatedly studied since
1960's. The reason why researchers have returned to this experiment is that the sudden appearance of sub harmonic motion often prophesies the prelude to chaos. 2
Chapter 1
Chaos and Commnnications
Poincare discovered what is today known as homo clinic trajectories in the state space. In 1892, this was published in his three-volume work on Celestial Mechanics. Only in 1962 did Smale prove that Poincare's homoclinic trajectories are chaotic limit sets [5]. In 1927, Van der Pol and Van der Mark studied the behavior of a neon bulb RC oscillator driven by a sinusoidal voltage source [6]. They discovered that by increasing the capacitance in the circuit, sudden jumps from the drive frequency, say w to w/2, then to w/3, etc., occurred in the response. These frequency jumps were observed, or more accurately heard, with a telephone receiver. They found that this process of frequency demultiplication (as they called it) eventually led to irregular noise. In fact, what they observed, in today's language, turned out to be caused by bifurcations and chaos. In 1944, Levinson conjectured that Birkhoffs remarkable curves might occur in the behavior of some third-order systems. This conjecture was answered affirmatively in 1949 by Levinson [7]. Birkhoff proved his famous Ergodic Theorem in 1931 [8]. He also discovered what he termed remarkable curves or thick curves, which were also studied by Charpentier in 1935 [9]. Later, these turned out to be a chaotic attractor of a discrete-time system. These curves have also been found to be fractal with dimension between 1 and 2. In 1936, Chaundy and Phillips [10] studied the convergence of sequences defined by quadratic recurrence formulae. Essentially, they investigated the logistic map. They introduced the terminology that a sequence oscillates irrationally. Today this is known as chaotic oscillation. Inspired by the discovery made by Van der Pol and Van der Mark, two mathematicians, Cartwright and Littlewood [11] embarked on a theoretical study of the system studied earlier by Van der Pol and Van der Mark. In 1945, they published a proof of the result that the driven Van der Pol system can exhibit nonperiodic solutions. Later, Levinson [7] referred to these solutions as singular behavior. Melnikov [12] introduced his perturbation method for chaotic systems in 1963. This method is mainly applied to driven dynamical systems. 3
1.1
Historical Accouut
In 1963, Lorenz [13], a meteorologist, studied a simplified model for thermal convection numerically. The model (today called the Lorenz model) consisted of a completely deterministic system of three nonlinearly coupled ordinary differential equations. He discovered that this simple deterministic system exhibited irregular fluctuations in its response without any external element of randomness being introduced into the system. Cook and Roberts [14] discovered chaotic behavior exhibited by the Rikitake two-disc dynamo system in 1970. This is a model for the time evolution of the earth magnetic field. In 1971, Ruelle and Takens [15] introduced the term strange attractor for dissipative dynamical systems. They also proposed a new mechanism for the on-set of turbulence in the dynamics offluids. It was in 1975 that chaos was formally defined for one-dimensional
transformations by Li and Yorke [16]. They went further and presented sufficient conditions for so-called Li -Yorke chaos to be exhibited by a certain class of one-dimensional mappings. In 1976, May called attention to the very complicated behavior which included period-doubling bifurcations and chaos exhibited by some very simple population models [17]. In 1978, Feigenbaum discovered scaling properties and universal constants (Feigenhaum's number) in one-dimensional mappings [18]. Thereafter, the idea of a renormalization group was introduced for studying chaotic systems. In 1980, Packard et al. [19] introduced the technique of state-space reconstruction using the so-called delay coordinates. This technique was later placed on a firm mathematical foundation by Takens. In 1983, Chua [20] discovered a simple electronic circuit for synthesizing the specific third-order piecewise-linear ordinary differential equations. This circuit became known as Chua's circuit (see the following chapter). What makes this circuit so remarkable is that its dynamical equations have been proven to exhibit chaos in a rigorous sense. Ott, Grebogi and Yorke, in 1990 [21], presented a method for controlling unstable trajectories embedded in a chaotic attractor. At the same time, there was another course of events leading to the field of chaos. This was the study of nonintegrable Hamiltonian systems in classical 4
Chapter 1
Chaos and Commnnications
mechanics. Research in this field has led to the formulation and proof of the Kolmogorov-Amold-Moser (KAM) theorem in the early 1960's. Numerical studies have shown that when the conditions stated by the KAM theorem fails, then stochastic behavior is exhibited by nonintegrable Hamiltonian systems. Remarks: Today, chaos has been discovered in bio-systems, meteorology, cosmology, economics, population dynamics, chemistry, physics, mechanical and electrical engineering, and many other areas. The research direction has been transferring from fmding the evidence of chaos existence into applications and deep theoretical study.
1.2
Chaos
There are many possible definitions of chaos for dynamical systems, among which Devaney's definition (for discrete-time systems) is a very popular one because it applies to a large number of important examples.
Theorem 1.1 [22] Let Q be a set.f: Q---+ Q is said to be chaotic on Q if: (1) fhas sensitive dependence on initial conditions, i.e., there exists
(» 0
such that, for any x EQ and any neighborhood U of x, there exists Y E U and
n?oO such that IF(x)-F(Y)1 >0; (2) f is topological transitive, i.e., for any pair of open sets V, We Q there exists k> 0 such that fk (V) n
w
-:j:.
0;
(3) periodic points are dense in Q.
1.3
Quantifying Chaotic Behavior
Lyapunov exponents, entropy and dimensionality are usually used to quantify (characterize) a chaotic attractor's behavior. Lyapunov exponents indicate the average rates of convergence and divergence of a chaotic atiractor in the state space. Kolmogorov Entropy (KE) is used to reveal the rate of information 5
1.3
Quantifying Chaotic Behavior
loss along the attractor. Dimensionality is used to quantify the geometric structure of a chaotic attractor.
1.3.1 Lyapunov Exponents for Continuous-Time Nonlinear Systems The determination of Lyapunov exponents is important in the analysis of a possibly chaotic system, since Lyapunov exponents not only show qualitatively the sensitive dependence on initial conditions but also give a quantitative measure of the average rate of separation or attraction of nearby trajectories on the attractor. Here, we only state the definitions of Lyapunov exponents for continuous-time and discrete-time (see next subsection) nonlinear systems. More detailed descriptions of various algorithms for calculating Lyapunov exponents can be found in references [23 - 33]. In the direction of stretching, two nearby trajectories diverge; while in the directions of squeezing, nearby trajectories converge. Ifwe approximate this divergence and convergence by exponential functions, the rates of stretching and squeezing would be quantified by the exponents. These are called the Lyapunov exponents. Since the exponents vary over the state space, one has to take the long time average exponential rates of divergence (or convergence) of nearby orbits. The total number of Lyapunov exponents is equal to the degree of freedom of the system. If the system trajectories have at least one positive Lyapunov exponent, then those trajectories are either unstable or chaotic. If the trajectories are bounded and have positive Lyapunov exponents, the system definitely includes chaotic behavior. The larger the positive exponent is, the shorter the predictable time scale of system. The estimation of the largest exponent therefore assumes a special importance. For a given continuous-time dynamical system in an m-dimensional phase space, we monitor the long-term evolution of an infinitesimal m-sphere of initial conditions. The sphere will evolve into an m-ellipsoid due to the locally
6
Chapter 1
Chaos and Communications
deforming nature of the flow. The ith one-dimensional Lyapunov exponent can be defined in terms of the length of the ellipsoidal principal axis Ii (I):
A;
=
lim!lnI/JI) I, Hoo 1 1;(0)
(1.1)
whenever the limit exists [24,25]. Thus, the Lyapunov exponents are related to the expanding or the contracting nature of the principal axes in phase space. A positive Lyapunov exponent describes an exponentially increasing separation of nearby trajectories in a certain direction. This property, in tum, leads to the sensitive dependence of the dynamics on the initial conditions, which is a necessary condition for chaotic behavior. Since the orientation of the ellipsoid changes continuously as it evolves, the directions associated with a given exponent vary in a complicated way through the attractor. We cannot therefore speak of a well defined direction associated with a given exponent. For systems whose equations of motion are known explicitly, Benettin el
al. [25] have proposed a straightforward technique for computing the complete Lyapunov spectrum. This method can be described in principle as follows. Let an m-dimensional compact manifold M
be the state space of a
dynamical system. The system on M is a nonlinear differentiable map ([J:
M ---+M, which can be conveniently described by the following difference equation: x(n)
= ([J(x(n
-1»
= ([In (x(o».
(1.2)
Let v(O) denote an initial perturbation of a generic point x(O), and & be a sufficiently small constant. Consider the separation of trajectories of the unperturbed and perturbed points after n iterations:
II ([In (x(O»
- ([In (x(O) + &v(O»
a
II = II D([Jn (x(O»v(O)& II 0(&2) = II(
D([Jn (x(k» }(O)&II + 0(&2),
(1.3) 7
1.3
Quautifying Chaotic Behavior
where D
11'11
represents the Euclidean norm. Let the distinct eigenvalues D
Unfortunately, this method cannot be applied directly to experimental data, since we do not usually know the underlying dynamical equations. For experimental data, one has to resort to Eq. (1.1).
1.3.2
Lyapunov Exponent for Discrete-Time Systems
Consider any initial condition xo, and let {Xk };~o be the corresponding orbit of a p-dimensional, discrete-time map 1Jf. Let mI(k), m2(k),'" , mik) be the eigenvalues of DlJfk(xo). The ith one-dimensional Lyapunov exponent of IJf with respect to Xo can be defined as (1.5) whenever the limit exists [29].
1.3.3
Kolmogorov Entropy
The Kolmogorov entropy (KE), also known as metric entropy or KolmogorovSinai entropy, is also an important measure by which a chaotic motion in an arbitrary dimensional phase space can be characterized (quantified) [34 - 36]. The Kolmogorov entropy of an attractor can be considered as a measure of the rate of information loss along the attractor or as a measure of the degree of predictability of points along the attractor, given an arbitrary initial point.
8
Chapter 1
Chaos and Communications
Consider the trajectory of a dynamical system on a strange attractor and divide the phase space into L-dimensional hypercubes of volume SL. Let P;o,ij,.,i, be the probability that a trajectory is in hypercube io at t= 0, i 1 at t= T,
i2 at t= 2T, and so on. Then the quantity
Kn= -
I,
P;o,ij, ·.,i, In(P;o,ij,... ,i)
(1.6)
io,i1 .. ··,in
is proportional to the information needed to locate the system on a special trajectory with precision s [37, 38]. Therefore, K N+1 - KN is the additional information needed to predict which cube the trajectory will be in at (n + 1)T, given trajectories up to nT. This means that K N+1 - KN measures the loss of information about the system from time instant n to the next n + 1. The KE is then defined as the average rate of information loss in the following way: K=lim T -->0
1 N-l lim lim -I,(Kn +1 -KJ. &-->0+ N -->00 NT n=O
(1.7)
The order in which the limits in the above expression are taken is immaterial. The limit s ----> 0 makes KE independent of the particular partition. The main properties ofKE are as follows: (1) The entropy K (averaged) determines the rate of change in information entropy (i.e., Eq. (1.6)) as a result of a purely dynamical process of mixing of trajectories in phase space. (2) The entropy is a metric invariant of the system, i.e., its value being independent of the way that the phase space is divided into cells and coarsened. (3) Systems with identical values of entropy are in a certain sense isomorphic to each other [39, 40], i.e., these systems must have identical statistical laws of motion. (4) When applied to prediction, KE can be interpreted as the average rate at which the accuracy of a prediction decays as prediction time increases, i.e., the rate at which predictability will be lost [37].
9
1.3
Quantifying Chaotic Behavior
1.3.4
Attractor Dimension
Long-term chaotic motion in dissipative dynamical systems is confined to a strange attractor whose geometric structure is invariant to the evolution of the dynamics. Typically, a strange attractor is a fractal object and, consequently, there are many possible notions of the dimension for strange attractor. Here, we discuss some well-known and widely accepted definitions of attractor dimension. We also discuss the simple relationships to Lyapunov exponents and entropy. Dissipative chaotic systems are typically ergodic. All initial conditions within the system's basin of attraction lead to a chaotic attractor in the state space, which can be associated with a time-invariant probability measure. Intuitively, the dimension of the chaotic attractor should reflect the amount of information required to specify a location on the attractor with a certain precision. This intuition is formalized by defining the information dimension, d" of the chaotic attractor as -1· In p[Bx (&)] d[ - 1m , <-->0 In&
(1.8)
where p[Bx(&)] denotes the mass of the measure p contained in a ball of radius
&
centered at the point x in the state space [2, 41]. Information dimen-
sion is important from an experimental viewpoint because it is straightforward to estimate. The mass, p[Bx(&)] ' can be estimated by (1.9) where U(-) is the Heaviside function, and M is the number of points in the phase space. In typical experiments, the state vector x is estimated from a delay embedding of an observed time-series [42]. The information dimension as defmed above, however, depends on the particular point x in the state space being considered. Grassberger and Procaccia's approach eliminates this dependence [43] by defming the quantity 10
Chapter 1 Chaos and Communications
(1.10) and then defining the correlation dimension de as de = lim In C(l'). B--+O
In practice, one usually plots In C( £
)
(1.11 )
Inl'
as a function of In £ and then measures
the slope of the curve to obtain an estimate of de [44 - 66]. It is often the case that d, and de are approximately equal. The box-counting dimension (capacity), do, of a set is defined as -1· 1n.ll/(c) d0 - 1m &-->0 1n(l/ c)
(1.12)
,
where .11/(.0) is the minimum number of N-dimensional cube of side length £
needed to cover the set. There is a meaningful relationship between the Lyapunov dimension and
Lyapunov exponents for chaotic systems [41, 67, 68]. If AI, ... , AN are the Lyapunov exponents of a chaotic system, then the Lyapunov dimension, dL , is defined as (1.13) where k
=
max {i : ~ + ... + Ai > O}. It has been shown that do ?cd"
d,~dc
[43, 69].
Equation (1.14) suggests that only the first k + 1 Lyapunov exponents are important for specifying the dimensionality of the chaotic attractor. Kaplan et
al. [67, 68] conjectured that d, = dL in "almost" all cases. Clearly, if this is correct, then Eq. (1.13) provides a straightforward way to estimate the attractor dimension when the dynamical equations of motion are known. The relation between KE and Lyapunov exponents is also available. In one-dimensional maps, KE is just the Lyapunov exponent [70]. In higherdimensional systems, we lose information about the system because the cell in which it was previously located spreads over new cells in phase space at a 11
1.4
Properties of Chaos
rate determined by the Lyapunov exponents. The rate K at which the information about the system is lost is equal to the (averaged) sum of the positive Lyapunov exponents [71], as shown by (1.14) where A's are the positive Lyapunov exponents of the dynamical system being considered.
1.4
Properties of Chaos
It is now well-known that a deterministic dynamical system is one whose
state evolves with time according to a deterministic evolution rule. The time evolution of the state is completely determined by the initial state of the system, the input, and the rule. For example, the state of a digital filter is determined by the initial state of the filter, the input, and a difference equation which describes the evolution of the state from one time instant to the next. In contrast to a stochastic dynamical system, which may follow any number of different trajectories from a given state according to some probabilistic rule, trajectories of a deterministic dynamical system are unique. From any given state, there is only one "next" state. Therefore, the same system started twice from the same initial state with the same input will follow precisely the same trajectory through the state space. Deterministic dynamical systems can produce a variety of steady-state behaviors, the most familiar of which are stationary, periodic, and quasi-periodic solutions. These solutions are "predictable" in the sense that a small piece of a trajectory enables one to predict the future behavior along that trajectory. Chaos refers to solutions of deterministic dynamical systems which, while predictable in the short-term, exhibit long-term unpredictability. Since the initial state, input, and rule uniquely determine the behavior of a deterministic dynamical system, it is not obvious that any "unpredictability" 12
Chapter 1
Chaos and Communications
is possible. Long-term unpredictability arises because the dynamics of a chaotic system persistently amplifies errors in specifying the state. Thus, two trajectories starting from nearby initial conditions quickly become uncorrelated. This is because in a chaotic system, the precision with which the initial conditions must be specified in order to predict the behavior over some specified time interval grows exponentially with the length of the prediction interval. As a result, long-term prediction becomes impossible. This long-term unpredictability manifests itself in the frequency domain as a continuous power spectrum, and in the time domain as random "noiselike" signal. To get a better idea of what chaos is, here is a list of its main characteristics (properties) : (1) Chaos results from a deterministic nonlinear process. (2) The motion looks disorganized and erratic, although sustained. In fact, it can usually pass all statistical tests for randorrmess (thereby we cannot distinguish chaotic data from random data easily), and has an invariant probability distribution. The Fourier spectrum (power spectrum) is "broad" (noiselike) but with some periodicities sticking up here and there [72, 73]. (3) Details of the chaotic behavior are hypersensitive to changes in initial conditions (minor changes in the starting values of the variables). Equivalently, chaotic signals rapidly decorrelate with themselves. The autocorrelation function of a chaotic signal has a large peak at zero and decays rapidly. (4) It can result from relatively simple systems. In nonautonomous system, chaos can take place even when the system has only one state variable. In autonomous systems, it can happen with as few as three state variables. (5) For given conditions or control parameters, chaos is entirely selfgenerated. In other words, changes in other (i.e., external) variables or parameters are not necessary. (6) It is not the result of data inaccuracies, such as sampling error or measurement error. Any specific initial conditions (right or wrong), as long as the control parameter is within an appropriate range, can lead to chaos. (7) In spite of its disjointed appearance, chaos includes one or more types 13
1.5
Chaos-Based Commuuicatious
of order or structure. The phase space trajectory may have fractal property (self-similarity). (8) The ranges of the variables have finite bounds, which restrict the attractor to a certain finite region in the phase space. (9) Forecasts of long-term behavior are meaningless. The reasons are sensitivity to initial conditions and the impossibility of measuring a variable to absolute accuracy. Short-term predictions, however, can be relatively accurate. (10) As a control parameter changes systematically, an initially nonchaotic system follows one of a few typical scenarios, called routes to chaos.
1.5 1.5.1
Chaos-Based Communications Conventional Spread Spectrum
In recent years, there has been explosive growth in personal communications, the aim of which is to guarantee the availability of voice and/or data services between mobile communication terminals. In order to provide these services, radio links are required for a large number of compact terminals in densely populated areas. As a result, there is a need to provide highfrequency, low-power, low-voltage circuitry. The huge demand for communications results in a large number of users; therefore, today's communication systems are limited primarily by interference from other users. In some applications, the efficient use of available bandwidth is extremely important, but in other applications, where the exploitation of communication channels is relatively low, a wideband communication technique having limited bandwidth effciency can also be used. Often, many users must be provided with simultaneous access to the same or neighboring frequency bands. The optimum strategy in this situation, where every user appears as interference to every other user, is for each communicator's signal to look like white noise which is as wideband as possible. 14
Chapter 1
Chaos and Communications
There are two ways in which a communicator's signal can be made to appear like wideband noise: (1) spreading each symbol using a pseudo-random sequence to increase the bandwidth of the transmitted signal; (2) representing each symbol by a piece of "noiselike" waveform [74]. The conventional solution to this problem is the first approach: to use a synchronizable pseudo-random sequence to distribute the energy of the information signal over a much larger bandwidth to transmit the baseband information. The transmitted signal appears similar to noise and is therefore diffcult to detect by eavesdroppers. In addition, the signal is difficult to jam because its power spectral density is low. By using orthogonal spreading sequences, multiple users may communicate simultaneously on the same channel, which is termed Direct Sequence Code Division Multiple Access (DS/CDMA). Therefore, the conventional solution can: (1) combat the effects of interference due to jamming, other users, and multipath effects; (2) hide a signal "in the noise" by transmitting it at low power; and (3) have some message privacy in the presence of eavesdroppers. With rapidly increasing requirements for some new communication services, such as wideband data and video, which are much more spectrumintensive than voice service, communication networks are already reaching their available resource limitation. Some intrinsic shortcomings of the convenient DS/CDMA have been known. For example, the periodic nature of the spreading sequences, the limited number of available orthogonal sequences, and the periodic nature of the carrier, are imposed to DS/CDMA systems in order to achieve and maintain carrier and symbol synchronization. One further problem is that the orthogonality of the spreading sequences requires the synchronization of all spreading sequences used in the same frequency band, i.e., the whole system must be synchronized. Due to different propagation times for different users, perfect synchronization can never be achieved in real systems [75]. In addition, DS/CDMA systems using binary spreading sequences do not provide much protection against 15
1.5
Chaos-Based Communications
two particular interception methods: the carrier regeneration and the code clock regeneration detectors [76]. This is due to the binary nature of the spreading sequences used in binary waveforms. The intrinsic properties of chaotic signals stated previously provide an alternative approach to making a transmission "noiselike". Specifically, the transmitted symbols are not represented as weighted sums of periodic basis functions but as inherently nonperiodic chaotic signals, which will be described in the following subsections.
1.5.2
Spread Spectrum with Chaos
The properties of chaotic signals resemble in many ways those of the stochastic ones. Chaotic signals also possess a deterministic nature, which makes it possible to generate "noiselike" chaotic signals in a theoretically reproducible manner. Therefore, a pseudo-random sequence generator is a "practical" case of a chaotic system, the principal difference being that the chaotic system has an infinite number of (analog) states, while pseudo-random generator has a finite number (of digital states). A pseudo-random sequence is produced by visiting each state of the system once in a deterministic manner. With only a finite number of states to visit, the output sequence is necessarily periodic. By contrast, an analog chaos generator can visit an infinite number of states in a deterministic manner and therefore produces an output sequence, which never repeats itself. With appropriate modulation and demodulation techniques, the "random" nature and "noiselike" spectral properties of chaotic electronic circuits can be used to provide simultaneous spreading and modulation of a transmission.
1.5.3
Chaotic Synchronization
How then would one use a chaotic signal in communication? A first approach would be to hide a message in a chaotic carrier and then extract it 16
Chapter 1
Chaos and Communications
by some nonlinear, dynamical means at the receiver. If we do this in realtime, we immediately lead to the requirement that somehow the receiver must have a duplicate of the transmitter's chaotic signal or, better yet, synchronize with the transmitter. In fact, synchronization is a requirement of many types of communication, not only chaotic possibilities. Early work on synchronous coupled chaotic systems was done by Yamada and Fujisaka [77, 78]. In that work, some sense of how the dynamics might change was brought out by a study of the Lyapunov exponents of synchronized coupled systems. Later, Afraimovich et al. [79] exposed many of the concepts necessary for analyzing synchronous chaos. A crucial progress was made by Pecora and Carroll [80 - 84], who have shown theoretically and experimentally that two chaotic systems can be synchronized. This discovery bridges between chaos theories and communications, and opens up a new research area in communications using chaos. The driving response synchronization configuration proposed by Pecora and Carroll is shown in Fig. 1.1, in which the Lorenz system is used in the transmitter and the receiver, where Xi or X; (i = 1,2,3, r standing for the response system) is the state variable of the Lorenz system [13], Ii is the ith state equation, and 11 (t) is additive channel noise. The drive-response synchronization method indicates that if a chaotic system can be decomposed into subsystems, a drive system system
(X2' X3)
Xl
and a conditionally stable response
in this example [82], then the identical chaotic system at the
receiver can be synchronized when driven with a common signal. The output signals
x; and x; will follow the signals
X2
and
X3.
For more discussions on
chaotic synchronization, see [83]. ~i:l(t)~ll(Xl(t), X2(t), x3(t)) ~icit)~lixl(t),
x 1(t)
y(l)
xf (t)~ll (Y (f). x~(t), x3(t)) x2(t)~12( Y (I),
x 2 (t), x 3(t))
x2(t), x{(I))
x;(t)
xj (t)~13( Y (I), x~(t), xl(tll
xJ(I)~13(X 1(I), xit), x 3(t))
11(1)
Drive system
Response system
Figure 1.1 Drive-response synchronization schematic diagram, in which
Xi
or
x;
(i = 1,2,3, r stands for the response system) is the state variable of the Lorenz system [13], Ii is the ith state equation, and 17 (t) is additive channel noise 17
1.5
Chaos-Based Communications
Based on the self-synchronization properties of chaotic systems, some chaotic communication systems using chaotic carriers have been proposed. Since the performance of such communication systems will strongly depend on the synchronization capability of chaotic systems, the robustness of se1fsynchronization in the presence of white noise needs to be explored [85]. Inspired by Pecora and Carroll's work, many other synchronization schemes have been proposed, including error feedback synchronization [87], inverse system synchronization [88], adaptive synchronization [89], generalized synchronization [90], etc. The error feedback synchronization is borrowed from the field of automatic control. An error signal is derived from the difference between the output of the receiver system and that received from the transmitter. The error signal is then used to modify the state of the receiver such that it can be synchronized with the transmitter. The operating theory of the inverse system synchronization scheme is as follows. If a system L with state x(t) and input set) produces an output yet), 1
then its inverse L- produces an output Yr(t) = s(t) and its state xr(t) has synchronized with x(t). Adaptive synchronization scheme makes use of the procedure of adaptive control and introduces the time dependent changes in a set of available system parameters. This scheme is realized by perturbing the system parameters whose increments depend on the deviations of the system variables from the desired values and also on the deviations of the parameters from their correct values corresponding to the desired state. Generalized synchronization of the uni-directionally coupled systems
x = F(x)
(x
y = H(y, x)
E
ffi.", drive)
(y
E
ffi.", response)
(1.15) (1.16)
occurs for the attractor Ax c lR of the drive system if an attracting synchronization set
18
Chapter 1 Chaos and Commnnications
M = {(X,y) E A/]Rm : y = H(x)}
(1.17)
exists and is given by some function H:Ar~Ayc]Rm. Also, M possesses an open basin 13 ::J M such that: lim II y(t) - H(x(t)) 11= 0,
\t(x(O), y(O))
E
13.
1-'>00
(1.18)
It was reported in [91] that for a linear bandpass communication channel with
additive white Gaussian noise (AWGN), drive-response synchronization is not robust (signal-to-noise ratio, > 30 dB is required) and the continuoustime analog inverse system exhibits extreme noise sensitivity (SNR > 40 dB is required to maintain synchronization). Further, recent studies of chaotic synchronization, where significant noise and filtering have been introduced to the channel, indicate that the performance of chaotic synchronization schemes is worse, at low SNR, than that of the best synchronization schemes for sinusoids [92].
1.6
Communications Using Chaos as Carriers
The use of modulating an aperiodic or nonperiodic chaotic waveform, instead of a periodic sinusoidal signal, for carrying information messages has been proposed since chaotic synchronization phenomenon was discovered. In particular, chaotic masking [85,93], dynamical feedback modulation [94], inverse system modulation [95], chaotic modulation [88, 96 - 10 1], chaoticshift-keying (CSK) [102 - 112], and differential chaos shift keying (DCSK) [113], have been proposed. In the following, we will provide a brief summary of these schemes.
1.6.1
Chaotic Masking Modulation
The basic idea of a chaotic masking modulation scheme is based on chaotic signal masking and recovery. As shown in Fig. 1.2, in which the Lorenz 19
1.6
Communications Using Chaos as Carriers
system is also used as the chaotic generator, we add a noiselike chaotic signal
Xl (t)
to the information signal met) at the transmitter, and at the
receiver the masking signal is removed [85, 86,93]. The received signaly(t), consisting of masking, information and noise signals, is used to regenerate the masking signal at the receiver. The masking signal is then subtracted from the received signal to retrieve the information signal denoted by
m(t).
The regeneration of the masking signal can be done by synchronizing the receiver chaotic system with the transmitter chaotic system. This communication system could be used for analog and digital information data. Cuomo et al. [85] built a Lorenz system circuit and demonstrated the performance
of chaotic masking modulation system with a segment of speech from a sentence. The performance of the communication system greatly relies on the synchronization ability of chaotic systems. The masking scheme works only when the amplitudes of the information signals are much smaller than the masking chaotic signals.
.i:[(I)=I[(x[(t), x 2(1), "3(1»
.i:[ (1)=/[ (JJ (I), x~(t), xj(l))
xil)=Iz{x[(I),
x2(1), x3(1»
.i:2(t)=iz( Y (f), x;[(t), x 3(1))
·i:3(t)=ll~[(I),
x 2(1), x3(1»
,iej(t)=i 3( Y (f), x 2(t), x 3(1» l1(t)
Drive system
in (I)
Response system
Figure 1.2 Block diagram of a chaotic masking modulation communication system, in which
Xi
or
X; (i = 1,2, 3, r stands for the response system) is the state variable
of the Lorenz system [13], Ii is the ith state equation, 77 (t) is additive channel noise, met) and
1.6.2
m(t) are the injected message signal and the recovered message signal
Dynamical Feedback Modulation
To avoid the restriction of the small amplitude of the information signal, another modulation scheme, called dynamical feedback modulation, has been proposed in [94]. As shown in Fig. 1.3, in which the Lorenz system is used again as the chaotic generator, the basic idea is to feedback the information signal into the chaotic transmitter in order to have identical 20
Chapter 1
Chaos and Communications
input signals for both the chaotic transmitter and the receiver. Specifically, the transmitted signal, consisting of the information signal met) and the chaotic signal Xl (t), is communicated to the receiver which is identical to the chaotic transmitter. Since the reconstructed signal
x; (t) will be identical to
x(t) in the absence of noise '7(t), the information signal met) can be decoded from the received signal. m(t) ~i:/t)~ll(xl(t)+m(t), xit), X3(t»
xl(t)~ll (y(t), -'2(t), x](t))
x2(t)~lixl(t)+m(t), xit), x3(t))
xM~liy(t), xW), xj(t))
x3(t)~lixl(t)+m(t),
xW)~13(
x 2(t), x3(t»
y(t), x2(t), xj(t)
met) Drive system
Response system
Figure 1.3 Block diagram of a dynamical feedback modulation communication system, in which
Xi
or X; (i = 1, 2, 3, r stands for the response system) is the state
variable of the Lorenz system [13], (is the ith state equation, noise, met) and
1]
(t) is additive channel
m(t) are the injected message signal and the recovered message signal
This analog communication technique can be applied to binary data communication by setting met) = C if the binary information data is one, and met) = - C if the binary data is zero. Since the feedback information will
affect the chaotic property, the information level C should be scaled carefully to make the transmitter chaotic to maintain the desired communication security.
1.6.3
Inverse System Modulation
In the inverse system approach [95], the transmitter consists of a chaotic system which is excited by the information signal set). The output yet) of the transmitter is chaotic. The receiver is simply the inverse system, i.e., a system which produces ret) = set) as output when excited by yet) and started from the same initial condition. If the system is properly designed, the output ret) will approach set), regardless of the initial conditions. 21
1.6
Communications Using Chaos as Carriers
1.6.4
Chaotic Modulation
In chaotic modulation [88,96-101], the message signal is injected into a chaotic system as a bifurcation "parameter,,(j), with the range of the bifurcation parameter chosen to guarantee motion in a chaotic region (for more details, see Sec. 7.2). The main advantage of the chaotic modulation scheme is that it does not require any code synchronization, which is necessary in traditional spread spectrum communication systems using coherent demodulation techniques. The crucial design factor is, however, the retrieval of the bifurcation "parameter" variation from the receiving spread spectrum signal, which may be distorted by nonideal channel and contaminated by noise (one of the goals of this book is to investigate signal reconstruction techniques at the receiving end such that the bifurcation parameter and hence the injected message can be recovered).
1.6.5
Chaos Shift Keying
In binary CKS [102-112] as shown in Fig. 1.4 (a), an information signal is encoded by transmitting one chaotic signal
Zj (t)
for a "1" and another
chaotic signal zo(t) to represent the binary symbol "0". The two chaotic signals come from two different systems (or the same system with different parameters); these signals are chosen to have similar statistical properties. Two demodulation schemes are possible: coherent and non-coherent. The coherent receiver contains copies of the systems corresponding to "0" and "1". Depending on the transmitted signal, one of these copies will synchronize with the incoming signal and the other will desynchronize at the receiver. Thus, one may determine which bit is being transmitted. A coherent demodulator is shown in Fig. 1.4 (b), in which Zt(t) and zo(t) are the regenerated chaotic signals at the receiver.
(j)
Bifurcation parameters determine the dynamical behavior of a dynamical system. For some selected range of the parameter values, the system can demonstrate chaotic behavior [22].
22
Chapter 1
Chaos and Communications
Transmitter x(t)
t
"0" Digital infomlation to be transmitted
I I
: - - - - ________________ Receiver I I Channel ~L ______ _ (al Correlator
yet) - - , - - - - - - - - - - 1 Synchronization circuit Symbol duration
Synchronization circuit
Correlator
Digital infomlation received Threshold detector (b)
Figure 1.4 Chaos shift keying digital communication system. Block diagrams of (a) the system, and (b) a coherent CSK demodulator
One type of non-coherent receivers requires the transmitted chaotic signals having different bit energies for "1" and "0". By comparing the bit energy with a decision threshold, one can retrieve the transmitted source information signal. Moreover, other non-coherent schemes exploit the distinguishable property of the chaotic attractors for demodulation, such as in Tse et al. [114]. In particular, if the two chaotic signals come from the same system
23
1.6
Communications Using Chaos as Carriers
with different bifurcation parameters, demodulation can be performed by estimating the bifurcation parameter of the "reconstructed" chaotic signals.
1.6.6
Differential Chaos Shift Keying Modulation
When the channel condition is so poor that it is impossible to achieve chaotic synchronization, a differential chaotic modulation technique for digital communication, called DCSK, has been introduced [113]. This modulation scheme is similar to that of the differential phase shift keying (DPSK) in the conventional digital communication except that the transmitted signal is chaotic. That is, in DCSK, every symbol to be transmitted is represented by two sample functions. For bit" 1", the same chaotic signal are transmitted twice in succession while bit "0" is sent by transmitting the reference chaotic signal followed by an inverted copy of the same signal. At the receiver the two received signals are correlated and the decision is made by a zero-threshold comparator. The DCSK technique offers additional advantages over the CSK: (l) The noise performance of a DSCK communication system in terms of
bit error rate (BER) versus EblNo (Eb is the energy per bit and No is the power spectral density of the noise introduced in the channel) outperforms the BER of a standard non-coherent CSK system. For sufficiently large bit duration, the noise performance of DCSK is comparable to that of a conventional sinusoid-based modulation scheme. In particular, EblNo = 13.5 dB is required for BER= 10-3 [115]. (2) Because synchronization is not required, a DCSK receiver can be implemented using very simple circuitry. (3) DCSK is not as sensitive to channel distortion as coherent methods since both the reference and the information-bearing signal pass through the same channel. The main disadvantage of DCSK results from differential coding: Eb is doubled and the symbol rate is halved. 24
Chapter 1
1.7 1.7.1
Chaos and Communications
Remarks on Chaos-Based Communications Security Issues
Recent studies [116 - 118] have shown that communication schemes using chaotic or hyperchaotic sources have limited security. Therefore, most of the chaos-based communication schemes are based on the viewpoint that security is an added feature in a communication system, which may be implemented by adding encryption/decryption hardware at each end of the system.
1. 7.2
Engineering Challenges
The field of "communications with chaos" presents many challenging research and development problems at the basic, strategic, and applied levels. The building blocks with which to construct a practical chaos-based spread spectrum communication system already exist: chaos generators, modulation schemes, and demodulators. Nevertheless, further research and development are required in all of these subsystems in order to improve robustness to a level that can be comparable to existing conventional system. Synchronization schemes for chaotic spreading signals are not yet sufficiently robust to be competitive with pseudo-random spreading sequences. Nevertheless, they do offer theoretical advantages in terms of basic security level. Furthermore, an analog implementation of chaotic spreading may permit the use of simple low power, high-frequency circuitry. Although an improved scheme, called frequency modulation DCSK (FMDCSK) [115], shows a better performance under multipath environment, channel characteristics are not fully taken into account yet, which limits its realizability in practical environments. Finally, there are still many practical problems that need to be solved, for example, the extension of multiple access design is a practical challenging 25
1.7
Remarks on Chaos-Based Communications
issue involving both system level and basic research. The effects of bandwidth limitation also presents different problems to the practical imple-mentation of such systems. In summary, chaos provides a promising approach for communications. It should be emphasized here that the field of chaos communications is very young: much fundamental work as well as practical problems need to be addressed before high-performance robust chaos-based communication systems can be generally available.
26
Chapter 2
Reconstruction of Signals
In most experimental situations we have to measure some variables from the system under consideration. In many cases these variables are often converted to electrical signals, which can be easily measured by precision instrumentation and later processed by computers. Electrical sensors are widely used to measure temperature, velocity, acceleration, light intensity, activity of human organs, etc. Modem techniques of data acquisition measure the signals in a specific way-they are sampled both in time and space (AID conversion, finite word-length effects, quantization, roundoff, overflow), making it possible to store and process the data using computers. When signals are collected from an experimental setup, several questions are often asked. What kind of information do the measured signals reflect about the system? How do we know if the signals are of sufficient integrity to represent the system in view of possible distortions and noise contamination? What conclusions can be drawn about the nature of the system and its dynamics? These are the basic questions to ask when we attempt to derive useful information from the measured signals. Before we can answer these questions, an important process to be considered is a faithful reconstruction of the signals. In this chapter, we consider the theoretical foundation of signal reconstruction and illustrate how signals can be reconstructed in practice even though the signals may appear "disordered" or "random-like", such as those produced from chaotic systems.
2.1
Reconstruction of System Dynamics
2.1
Reconstruction of System Dynamics
It is seldom the case that all relevant dynamical variables can be measured
in an experiment. Suppose some of the variables are available from measurements. How can we proceed to study the dynamics in such a situation? A key element in solving this general class of problems is provided by the embedding theory [1]. In typical situations, points on a dynamical attractor in the full system phase space have a one-to-one correspondence with measurements of a limited number of variables. This fact is useful for signal reconstruction. By definition, a point in the state space carries complete information about the current state of the system. If the equations defining the system dynamics are not explicitly known, this phase space is not directly accessible to the observer. A one-to-one correspondence means that the state space can be identified by measurements. Assume that we can simultaneously measure m variables YI(t), Y2(t),'" , Ym(t), which can be denoted by the vector y. This m-dimensional vector can
be viewed as a function of the system state x(t) in the full system phase space: y = F(x) =[J;(x) h(x) ... fm(x)]T.
(2.1)
We call the function F the measurement function and the m-dimensional vector space in which the vector y lies the reconstruction space. We have grouped the measurements as a vector-valued function F of x. The fact that F is a function is a consequence of the definition of the state. Specifically, since information about the system is determined uniquely by the state, each measurement is a well-defined function of x. As long as m is sufficiently large, the measurement function F generally defines a one-to-one correspondence between the attractor states in the full state space and the m-dimensional vector y. By "one-to-one" we mean that for a given y there is a unique x on the attractor such that y==F(x). When there is a one-to-one correspondence, each vector from m measurements is a 28
Chapter 2
Reconstruction of Signals
proxy for a single state of the system, and the fact that the information of the entire system is determined by a state x and F(x). In order for this to be valid, it suffices to set m to a value larger than twice the box-counting dimension of the attractor. The one-to-one property is useful because the state of a deterministic dynamical system is unique, and therefore its future evolution is completely determined by a point in the full state space. There are two types of embedding on the basis of measurements, which are relevant to system identification, namely, the topological embedding and differentiable embedding.
2.1.1
Topological Embeddings
Consider an n-dimensional Euclidean space
]Rn.
Points in this space are
defined by n real coordinate values, say Xl, X2,··· , Xn . Let us represent these elements by a vector Xoo [Xl X2··· xn]T. Let F be a continuous function from ]Rn
to
]Rm ,
where
]Rm
is an m-dimensional Euclidean space. The mapping
can be represented in the following way:
yooF(x),
(2.2)
whereYElRm. Let AclRn be an attractor ofa dynamical system. F(A) is an image of the attractor A in
]Rn via
the observation (measurement) function F.
F is bijective (one-to-one) if for any Xl, X2 C A, F(X1) = F(X2) implies Xl ooX2. For a bijective function F, there exists an inverse function F
-1.
A bijective
map on A which is continuous and has a continuous inverse is called topological embedding. In a typical experimental situation, the set A we are interested in is an attractor, which is a compact subset of
]R n
and is invariant
under the dynamical system. The goal is to use the measurements to construct the function F so that F(A) is a copy of A that can be analyzed. A finite time-series of measurements will produce a finite set of points that make up F(A). If enough points are present, we may hope to discern some of the properties of F(A) and therefore those of A. For topological embeddings we have the following theorem:
29
2.1
Reconstruction of System Dynamics
Theorem 2.1 [119]
A is a compact subset of JR" of box-counting dimension do. If m > 2do, then almost every d function F = [Ii h ... fm] T from JR nto JR mis a Assume that
topological embedding of A into JRm . The intuitive reason for the condition m > 2do can be seen by considering generic intersections of smooth surfaces in m-dimensional Euclidean space JRm . Two sets of dimensions d] and d2 in JRm mayor may not intersect. If they do intersect and the intersection is generic, then they will meet in a surface of dimension (2.3) If this number is negative, generic intersections do not occur. If the surface lies in a special position relative to one another, the intersection may be special and have a different dimension. In particular the delay coordinates can be used for constructing the topological embedding.
2.1.2
Delay Coordinates
Assume that our ability to make measurements of independent components is limited. In the worst case, we may be able to make measurements of a single scalar variable, say yet). Since the measurement depends only on the system state, we can represent such a situation by yet) = f(x(t)), where f is the single measurement function, evaluated when the system is in state xU). We assign to x(t) the delay coordinate vector H(xU)) = [yet - r)··· yet - mr)f =
[f(x(t - r))··· f(x(t - mr))f.
(2.4)
Note that for an invertible dynamical system, given x(t), we can view the state x (t - r) at any previous time t - r as being a function of x(t). This is true because we can start at x(t) and use the dynamical system to follow the 30
Chapter 2
Reconstruction of Signals
trajectory backward in time to the instant t - r. Hence, xCt) uniquely determines x (t - r), and thereby m is called embedding dimension. To emphasize this, we define hI (x(t»
= f(x(
t - r» , ... , hm(x(t» =f(x( t - m r» . Then, if
we writey(t) for [Yet - r) ···y(t- mr)f, we can express Eq. (2.1) as y=H(x),
(2.5)
where H(x) = [h](x) ... hm(X)]T is the delay coordinate function. The function H can be viewed as a special choice of the measurement function in Eq. (2.1). For this special choice, the requirement that the measurements do not lie in a special position is brought into question, since the components of the delay coordinate vector in Eq. (2.4) are constrained: They are simply time-delayed versions of the same measurement functionf This is relevant when considering small perturbations of H. Small perturbations in the measuring process are introduced by the scalar measurement function f, and they influence coordinates of the delay coordinate function H in an interdependent way. Although it was determined in the simultaneous measurement case that almost every perturbation of hI,···, hm leads to a one-to-one correspondence, these independent perturbations may not be achievable by perturbing the single measurement function
f
A simple example will illustrate this point. Suppose the set A contains a single periodic orbit whose period is equal to the delay time r . This turns out to be a bad choice of r, since each delay coordinate vector from the periodic orbit will have the form [h(x)··· h(x)]T for some x and lie along the straight line y] = ... =Ym in ]Rm . However, a circle cannot be continuously mapped to a line without points overlapping, violating the one-to-one property. Notice that this problem will affect all measurement functions h, so that perturbing h will not help. In this case, we cannot eliminate the problem by making the measurement function more generic; the problem is built-in. Although this case is a particularly bad one because of a poor choice of the time delay r , it shows that the reasoning for the simultaneous measurement case does not extend to delay coordinates, since it gives an obviously wrong conclusion in this case. 31
2.1
Reconstruction of System Dynamics
This problem can be avoided, for example, by perturbing the time delay 7 (if indeed that is possible in the experimental settings). In any case, some extra analysis beyond the geometric arguments we made for the simultaneous measurements case needs to be done. This analysis was tackled by Takens [1] and extended later by Sauer et al. [120]. The result can be stated as follows: Theorem 2.2
(Takens' embedding theorem with extensions by Sauer et al. [120])
Assume that a continuous-time dynamical system has a compact invariant set A (e.g., a chaotic attractor) of box-counting dimension do, and let m > 2do. Let
7
be the time delay. Assume that A contains only a finite number of
equilibria and a finite number of periodic orbits of period P T for 3 ~ P ~ m and that there are no periodic orbits of period 7 and 27. Then, with probability one, a choice of the measurement function h yields a delaycoordinate function H which is bijective from A to H(A). The one-to-one property is guaranteed to fail not only when the sampling rate is equal to the frequency of a periodic orbit, as discussed above, but also when the sampling rate is twice the frequency of a periodic orbit, that is, when A contains a periodic orbit of minimum period 27. To see why this is so, define the function ((x) = hex) - h( ¢-r (x)) on the periodic orbit, where ¢r denotes the action of the dynamics over time t. The function (is either identically zero or is nonzero for some x on the periodic orbit, in which case it has the opposite sign at the image point ¢-r (x) and changes sign on the periodic orbit. In any case, ((x) has a root Xo. Since the period
»
»
is 27, we have h( x o) = h(¢-r (x o = h( ¢-2r (x o = .. '. Then the delay coordinate map Xo and ¢-r (xo) are distinct, and H is not one-to-one for any observation function h. This problem may be eliminated by a proper choice of 7. Unfortunately, the choice of
7
that is not ruled out by the theory includes
those that are unnaturally small or large in comparison to the time constant of the system. Such values of
7
will cause the correlation between
successive measurements to be excessively large or small, causing the effectiveness of the reconstruction to degrade in real-world cases, where 32
Chapter 2
Reconstruction of Signals
noise is present. The choice of optimal time delay for unfolding the reconstructed attractor is an important and largely unresolved problem. A commonly used rule of thumb is to set the delay to be the time lag required for the autocorrelation function to become negative (zero crossing) or, alternatively, the time lag required for the autocorrelation function to decrease by a factor of expo Another approach, that of Fraser and Swinney [121], incorporates the concept of mutual information, borrowed from Shannon's information theory [122]. This approach provides a measure of the general independence of two variables, and chooses a time delay that produces the first local minimum of the mutual information of the observed quantity and its delayed value.
2.2
Differentiable Embeddings
Assume that A is a compact smooth d-dimensional submanifold of IRk. A circle is an example of a smooth one-dimensional manifold; a sphere and torus are examples of two-dimensional tangent space at each point. If F is a smooth function from one manifold to another, then the Jacobian matrix DF maps tangent vectors to tangent vectors. More precisely, for each point x on
A, the map DF(x) is a linear map from the tangent space at x to the tangent space at F(x). If, for all x in A, no nonzero tangent vectors map to zero under DF(x), then F is called an immersion. A smooth function F on a smooth manifold is called a differentiable embedding if F and F- 1 are one-to-one immersions. In particular, a differentiable embedding is automatically a topological embedding. In addition, the tangent spaces of A and F(A) are isomorphic. In particular, the image F(A) is also a smooth manifold of the same dimension as A. In 1936, Whitney [123] proved that if A is a smooth d-manifold in IRk and m> 2d, then a typical map from IRk to IR m is a differentiable embedding when restricted to A. Takens [1] proved a result in this context for delay coordinate functions. The following is a version of Takens' theorem by Sauer et al. [120]. 33
2.3
Phase Space Reconstruction - Example
Theorem 2.3 (Sauer et al. [120]) Assume that a continuous time dynamical system has a compact invariant smooth manifold A of dimension d, and let m > 2d. Let , be the time delay. Assume that A contains only a finite number of equilibria, a finite number of periodic orbits of period p, for 3 :( p :( m, and that there are no periodic orbits of period, and 2, . Assume that the Jacobians of the return maps of those periodic orbits have distinct eigenvalues. Then, with probability one, a choice of the measurement function h yields a delay coordinate function H which is bijective from A to H(A).
From the point of view of extracting information from an observed dynamical system, there are some advantages of having a differentiable embedding, as compared to a topological embedding. These advantages stem from the fact that metric properties are preserved by the reconstruction. Differentiable embeddings offer two advantages compared to topological ones [119]. First, there is a uniform upper bound on the stretching done by H 1 and H- • Such H functions are referred to as bi-Lipschitz. The dimensions are preserved under bi-Lipschitz maps. Second, all Lyapunov exponents on an attractor are reproduced in the reconstruction. For the purpose of the present study, we are interested in the consequences of the embedding theorems for the synchronization/transmission problem. These consequences can be summarized in the following result. (Main existence theorem [124]) If the assumptions of the embedding theorems are satisfied, it is always possible to reconstruct the state of the system and synchronize (e.g., by forcing the states) an exact copy of it on the basis of the measurements of a single (scalar) output variable.
Theorem 2.4
2.3
Phase Space Reconstruction-Example
From a signal processing perspective, an issue of paramount importance is the reconstruction of dynamics from measurements made on a single coordinate of the system. The motivation here is to make "physical sense" from the 34
Chapter 2
Reconstrnction of Signals
resulting time-series, by passing a detailed mathematical knowledge of the underlying dynamics. Let the time-series be denoted by {s(nT)}~~o' where N is the total number of samples and T is the sampling period. To reconstruct the dynamics of the original attractor that gives rise to the observed time-series, we seek an embedding space where we may reconstruct an attractor from the scalar data so as to preserve the invariant characteristics of the original unknown attractor [125, 126]. For simplicity we set T= 1. By applying the delay coordinate method, the dEx 1 phase space vector sen) is constructed by assigning coordinates: SI
(n) = sen),
S2 (n) =
sen - r),
(2.6)
where dE is the embedding dimension and r is the normalized embedding delay. These parameters are not chosen arbitrarily; rather they have to be determined experimentally. Procedures for estimating the normalized embedding delay r based on mutual information are described in the literature [121 - 129]. To estimate the embedding dimension dE,we may use the method of global false nearest neighbor [130 - 133]. Another important parameter that needs to be determined in the analysis of an observed process is the local or dynamical dimension [130]. The local dimension represents the number of dynamical degrees of freedom that are active in determining the evolution of the system as it moves around the attractor [130, 131]. This integer dimension gives the number of true Lyapunov exponents of the system under investigation. It is less than or equal to the global embedding dimension; that is, dL ~dE' The parameter dLcan be determined by using the local false-nearest-neighbor (LFNN) method [130, 131]. The notion offalse nearest neighbors refers to nearest neighbors as attractors that have become near one another through projection when the attractors are viewed in a dimension too low to unfold the attractors completely. Let us analyse an example of reconstruction of an attractor from a 35
2.3
Phase Space Reconstruction - Example
measured time-series obtained from the numerical integration of Chua's circuit (see Fig. 2.l) [134 - 138]. The circuit will also be used in subsequent chapters of this book. R
L
(a)
v
(b)
Figure 2.1 Chua's Circuit. (a) Circuit topology and (b) Chua's diode characteristic
The circuit is certainly one of the most widely studied nonlinear circuits and a great number of papers ensure that the dynamics of this circuit are also well documented. The normalized equations of the circuit are
X2 = XI - x 2 + x3' X3 = a 2 x2 , where
K(XI )
(2.7)
is a piecewise-linear function given by mIXI K(XI ) =
+ (rna
- ml )
for
XI ~ 1
maxI
for 1XI
mlxl - (rna - m l )
for
{
1< 1
(2.8)
xI::::;-1
with mo= -1/7 and ml = 2/7. For different values of a l and a 2 , the system operates in different regimes, e.g., periodic and chaotic regimes. The wellknown double scroll Chua's attractor, for example, is obtained fora l = 9 and a2 = -100/7. The largest Lyapunov exponent and the Lyapunov dimension 36
Chapter 2
Reconstruction of Signals
of the attractor are equal to 0.23 and 2.13, respectively [139,140]. Equations (2.7) and (2.8) are simulated using a fourth-order Runge-Kutta algorithm with integration step size equal to 10-2 • Figures 2.2 (a) and 2.2(b) 0.4 0.3 0.2 0.1 0 ~
-0.1 -0.2 -0.3 -0.4 -2.5
-2
-1.5
-I
-0.5
2.5
0 Xl
(a)
4 3 2
N
0 -1
-2 -3
-4 5
0.5 0 X
0 -5 -0.5
Y
(b)
Figure 2.2 Projection of the double scroll attractor observed in Chua's circuit on (a) x - y (i.e., Xl -
X2)
plane, and (b) X - Y - z (i.e., Xl -
X2 - X3)
space 37
2.3
Phase Space Reconstruction - Example
show the two-dimensional projection of the double scroll attractor on the x - y (or x) - X2) plane and the three-dimensional attractor in ]R3 . To be able to reconstruct the dynamics we further use only recording of i L , i.e., X3 (normalized current). To be able to apply the delay coordinate method we need to find a suitable time delay. In this example we follow the mutual information criteria to choose the delay T . It is found that T = 17 [124]. Figure 2.3 shows the reconstructed attractors on the x - y plane for different delays by using dE = 6. When the delay is too small, the reconstructed attractor becomes compacted and the dynamics is not well visualized. Figure 2.3(a) gives an example with the delay chosen as 4,------------------------,
T
= 3. Figure 2.3(b) shows the
4
2
-,
0
"-,
0
-)
-)
-2
-2
-3
-3
-4
-2
0 x
2
-4
4
-2
(a)
2
4
(b)
4
4
"-,
"
-)
-)
-2
-2
-3
-3
-4
0 x
-2
0 x
(c)
2
4
-4
-2
4 x
(d)
Figure 2.3 Reconstructed double scroll attractors using measured iL or X3 (i.e., z) time-series. Successive figures show attractors reconstructed using time delays of 3, 17, 30, 50, respectively. When the time delay is too small, the reconstructed attractor is squeezed. When it is too large, the reconstruction is bad - the geometric structure of the attractor is lost. With a proper choice of the delay time, as seen in (b), the double scroll structure is well reproduced
38
Chapter 2
reconstructions with
T
=
Reconstruction of Signals
17 as detennined from the mutual infonnation
criteria. In this case the attractor structure is easily seen. When the delay used for reconstruction becomes too large, the delayed copies of the signal become more and more uncorrelated. This situation is indicated in Figs. 2.3 (c) and (d). The trajectories shown in these two figures do not resemble at all the original double scroll structure. Figure 2.4 shows the reconstructed attractor in the reconstruction space by using attractor structure can be seen clearly.
T
= 17 and dE = 6. In this case, the
4 3 2
:;. 0 -1
-2 -3
-4 -5
o Figure 2.4 Reconstructed Chua's double scroll attractor in the reconstruction space by using T = 17 and dE = 6
2.4
Problems and Research Approaches
Embedding theorems offer only the existence result. Construction of an inverse of an embedding function is an open problem-no general solution or algorithm is available [141]. In principle, any choice of the time delay
T
is acceptable in the limitation
of an infinite amount of noiseless data. In the more likely situation of a finite amount of data, the choice of
T
is of considerable practical importance in
trying to reconstruct the attractor that represents the dynamical system 39
2.4
Problems and Research Approaches
which has generated the data. As mentioned above, if the time delay r is too short, the coordinates sen) and sen + r) that we wish to use in our reconstructed data vector sen) will not be independent enough. That is to say, not enough time will have evolved for the system to explore enough of its state space to produce, in a practical numerical sense, new information about that state space. On the other hand, since chaotic systems are intrinsically unstable, if r is too large, any connection between the measurements sen) and sen + r) is numerically equivalent to being random with respect to each other, which makes the reconstruction of the system dynamics very difficult. Even though it has not been stated clearly in the embedding theory, the reconstructions in the context of the Takens' embeddings are valid for autonomous systems only. In addition, the results stated in the previous sections are closely related to the observability problem known from control theory. Observability issues are well developed for linear systems, but only a limited number of results for nonlinear cases exist. Observers provide "the missing tool" for reconstruction. In Chapter 4, we will expand the results here to include non-autonomous systems based on the observer approach. Measurement noise, which always companies the observed chaotic data, contaminates the original chaotic signal and destroys the basic dynamics of the system of interest. Identifying and separating chaotic signal from noisy contaminated signal is still a fundamental issue in the context of chaotic signal processing and chaos-based communications [151]. Chapter 5 - 8 will discuss the modeling of chaotic systems via noisy measured data based on neural network approaches.
40
Chapter 3
3.1
Fundamentals of Neural Networks
Motivation
Artificial neural networks, commonly referred to as "neural networks", have been motivated right from their inception by the recognition that the human brain computes in an entirely different way from the conventional digital computer. The brain is a highly complex, nonlinear and parallel computer (information-processing system). It has the capability to organize its structural constituents, known as neurons, so as to perform certain computations (e.g., pattern recognition, perception, motor control, etc) many times faster than the fastest digital computer in existence today. Consider, for example, human vision, which is an information-processing task [175]. It is the function of the visual system to provide a representation of the environment around us and more importantly, to supply the information we need to interact with the environment. To be specific, the brain routinely accomplishes perceptual recognition tasks (e.g., recognizing a familiar face embedded in an unfamiliar scene) in approximately lOO - 200 ms, whereas tasks of much less complexity may take days on a conventional computer. How, then, does a human brain do it? At birth, a brain has great structure and the ability to build up its own rules through what we usually refer to as "experience". Indeed, experience is built up over time, with the most dramatic development (i.e., hard-wiring) of the human brain taking place during the first two years from birth, but the development continues well beyond that stage. A "developing" neuron is synonymous with a plastic brain: plasticity permits the developing nervous system to adapt to its surrounding
3.1
Motivation
environment. Just as plasticity appears to be essential to the functioning of neurons as information-processing units in the human brain, so it is with neural networks made up of artificial neurons. In its most general form, a neural network is a machine that is designed to model the way in which the brain performs a particular task or function of interest; the network is usually implemented by using electronic components or is simulated in software on a digital computer. In this book we confine ourselves to an important class of neural networks that perform useful computations through a process of learning (adaption). To achieve good performance, neural networks employ a massive interconnection of simple computing cells referred to as "neurons" or "processing units". We may thus offer the following definition of a neural network viewed as an adaptive machine: Definition 3.1 [176] A neural network is a massively parallel distributed processor made up of simple processing units, which has a natural propensity for storing experiential knowledge and making it available for use. It resembles the brain in two respects: (1) Knowledge is acquired by the network from its environment through a learning process. (2) Interneuron connection strengths, known as synaptic weights, are used to store the acquired knowledge. The procedure used to perform the learning process is called a learning algorithm, the function of which is to modify the synaptic weights of the network in an orderly fashion to attain a desired design objective. The modification of synaptic weights provides the traditional method for the design of neural networks. Such an approach is the closest to linear adaptive filter theory, which is already well established and successfully applied in many diverse fields [177, 178], However, it is also possible for a neural network to modify its own topology, which is motivated by the fact that neurons in the human brain can die and that new synaptic connections can grow.
42
Chapter 3
3.2
Fundamentals of Neural Networks
Benefits of Neural Networks
It is apparent that a neural network derives its computing power through,
first, its massively parallel distributed structure, and second, its ability to learn and therefore generalize. Generalization refers to the neural network producing reasonable outputs for inputs not encountered during training (learning). These two information-processing capabilities make it possible for neural networks to solve complex (large-scale) problems that are currently intractable. In practice, however, neural networks cannot provide the solution by working individually. Rather, they need to be integrated into a consistent system engineering approach. Specifically, a complex problem of interest is decomposed into a number of relatively simple tasks, and neural networks are assigned a subset of the tasks that match their inherent capabilities. The use of neural networks offers the following useful properties and capabilities [179]: (1) Nonlinearity
An artificial neuron can be linear or nonlinear. A
neural network, made up of an interconnection of nonlinear neurons, is itself nonlinear. Moreover, the nonlinearity is special in that it is distributed throughout the network. Nonlinearity is a crucial property, particularly if the underlying physical mechanism being responsible for generation of the input signal (e.g., speech signal) is inherently nonlinear. (2) Input-Output
Mapping A popular paradigm of learning called
learning with a teacher or supervised learning involves modification of the synaptic weights of a neural network by applying a set of labeled training samples or task examples. Each example consists of a unique input signal and a corresponding desired response. The network is presented with an example picked at random from the set, and the synaptic weights (free parameters) of the network are modified to minimize the difference between the desired response and the actual response of the network produced by the input signal in accordance with an appropriate statistical criterion. The training of the network is repeated for many examples in the set until the 43
3.2
Benefits of Nenral Networks
network reaches a steady state where there are no further significant changes in the synaptic weights. The previously applied training examples may be reapplied during the training session but in a different order. Thus, the network learns from the examples by constructing an input-output mapping for the problem at hand. Such an approach brings to mind the study of nonparametric statistical inference, which is a branch of statistics dealing with model-free estimation. The term "nonparametric" is used here to signify the fact that no prior assumptions are made on a statistical model for the input data. (3) Adaptivity
Neural networks have a built-in capability to adapt their
synaptic weights to changes in the surrounding environment. In particular, a neural network trained to operate in a specific environment can be easily retrained to deal with minor changes in operating environmental conditions. Moreover, when it is operating in a nonstationary environment (i.e., one where statistics change with time), a neural network can be designed to change its synaptic weights in real time. The natural architecture of a neural network for pattern classification, signal processing, and control applications, coupled with the adaptive capability ofthe network, makes it a useful tool in adaptive pattern classification, adaptive signal processing, and adaptive control. As a general rule, it may be said that the more adaptive we make a system, all the time ensuring that the system remains stable, the more robust its performance will likely be when the system is required to operate in a non stationary environment. It should be emphasized, however, that adaptivity does not always lead to robustness; indeed, the opposite may occur. For example, an adaptive system with short time constants may change rapidly and therefore tend to respond to spurious disturbances, causing a drastic degradation in system performance. To realize the full benefits of adaptivity, the principal time constants of the system should be long enough for the system to ignore spurious disturbances and yet short enough to respond to meaningful changes in the environment. The problem described here is referred to as the stability-plasticity dilemma [180].
44
Chapter 3
(4) Evidential Response
Fundamentals of Neural Networks
In the context of pattern classification, a neural
network can be designed to provide infonnation not only about which particular pattern to select, but also about the confidence in the decision made. This latter infonnation may be used to reject ambiguous patterns, should they arise, and thereby improve the classification perfonnance of the network. (5) Contextual Information
Knowledge is represented by the very
structure and activation state of a neural network. Every neuron in the network is potentially affected by the global activity of all other neurons in the network. Consequently, contextual infonnation is dealt with naturally by a neural network. (6) Fault Tolerance
A neural network, implemented in hardware fonn,
has the potential to be inherently fault tolerant, or capable of robust computation, in the sense that its perfonnance degrades gracefully under adverse operating conditions. For example, if a neuron or its connecting links are damaged, recall of a stored pattern is impaired in quality. However, due to the distributed nature of infonnation stored in the network, the damage has to be extensive before the overall response of the network is seriously degraded. Thus, in principle, a neural network exhibits a graceful degradation in perfonnance rather than catastrophic failure. There is some empirical evidence for robust computation, but usually it is uncontrolled. In order to be assured that the neural network is in fact fault tolerant, it may be necessary to take corrective measures in designing the algorithm used to train the network. (7) VLSI Implementability
The massively parallel nature of a neural
network makes it potentially fast for the computation of certain tasks. This same feature makes a neural network well suited for implementation using very-Iarge-scale-integration (VLSI) technology. One particular beneficial virtue of VLSI is that it provides a means of capturing truly complex behavior in a highly hierarchical fashion. (8) Uniformity of Analysis and Design
Basically, neural networks enjoy
universality as infonnation processors. We say this in the sense that the 45
3.3
Radial Basis Function Neural Networks
same notation is used in all domains involving the application of neural networks. This feature manifests itself in different ways:
CD
Neurons, in one form or another, represent an ingredient common to
all neural networks.
®
This commonality makes it possible to share theories and learning
algorithms in different applications of neural networks.
®
Modular networks can be built through a seamless integration of
modules. (9) Neurobiological Analogy
The design of a neural network is moti-
vated by analogy with the brain, which is a living proof that fault tolerant parallel processing is not only physically possible but also fast and powerful. Neurobiologists look to (artificial) neural networks as a research tool for the interpretation of neurobiological phenomena. On the other hand, engineers look to neurobiology for new ideas to solve problems more complex than those based on conventional hard-wired design techniques. The field of neural networks is now extremely vast and interdisciplinary, drawing interest from researchers in many different areas such as engineering (including biomedical engineering), physics, neurology, psychology, medicine, mathematics, computer science, chemistry, economics, etc. Artificial neural networks provide a neurocomputing approach for solving complex problems that might otherwise not have tractable solutions. In the following, we will briefly review the fundamentals of the two classes of neural networks, i.e., radial basis function (REF) neural networks, and recurrent neural networks (RNNs).
3.3 3.3.1
Radial Basis Function Neural Networks Background Theory
Originally, the REF method has been traditionally used for data interpolation in multi-dimensional space [181- 183]. The interpolation problem can be 46
Chapter 3
Fundamentals of Neural Networks
stated as follows: Given a set of N different points {z;
E
IR M Ii = 1, ... , P} and a corres-
IRI Ii = 1,,,,, P}, find a function h: IR M ---+ IRI that satisfies the interpolation condition:
ponding set of P real numbers {d;
E
(3.1)
h(zJ=di'i=l, .. ·,P.
Note that for strict interpolation as specified here, the interpolation surface (i.e., function h) is constrained to pass through all the training data points. The RBF technique consists of choosing a function h that has the following form: p
h(z) =
L w;qJ(llz - z;ll),
(3.2)
i=l
where {qJ(llz-z;II)Ii=l, ... ,P} is a set of P arbitrary (generally nonlinear) functions, known as RBFs, and 11'11 denotes here a norm that is usually taken to be Euclidean [152]. The known data points Z; EIR M ,i=1,2, .. ·,P are taken to be the centers of the RBF (see Fig. 3.1). Inserting the interpolation conditions of Eq.(3.1) in Eq.(3.2), we obtain the following set of simultaneous linear equations for the unknown coefficients (weights) of the expansion {Wi}:
qJIP] [WI qJ2P w2
·· ·
.. .
qJpp
Wp
_
-
[dl] d2 . , ..
(3.3)
dp
where
qJij=qJ(llz;-zjID, j,i=1,2, .. ·,P.
(3.4)
Let
d=[dl d 2
w=
[WI
...
w2
...
dpf, Wp]T.
(3.5) 47
3.3
Radial Basis Function Neural Networks
The P-dim vector d and w represent the desired response vector and linear weight vector, respectively. Let (/J denote an P x P matrix with elements rpji: (3.6)
(/J = {rpji I j, i = 1,2, "', P}.
We call this matrix the interpolation matrix. We may then rewrite Eq. (3.3) in the compact form
(/Jw
=
d.
(3.7)
Assuming that (/J is nonsingular and therefore that the inverse matrix (/J-l exists, we may go on to solve Eq. (3.7) for the weight vector w, obtaining (3.8) The vital question is: How can we be sure that the interpolation matrix (/J is nonsingular? It turns out that for a large number of radial-basis functions and under certain conditions, the answer to this question is given in the following important theorem. be distinct inIRM. Then the P-by-P
Theorem 3.1 [181] Let
Z\'Z2, "',Zp
interpolation matrix
whose ji-th element is CfJJi
(Jj
=
rp (liz) - zilD is
nonsingular. There are a class of radial basis functions that are covered by the above theorem, which includes the following functions that are of particular interest in the study of radial-basis function networks:
(1) Thin plate spline function 2
x log (x).
(3.9)
(2) Multiquadrics [184]
rp(r) = (r2 + C2)1/2
for some c> 0,
and
rE JR.
(3.10)
for some c> 0,
and
rER
(3.11 )
(3) Inverse multiquadrics [184]
rp(r)
48
=
(2 2)1/2 r +c
Chapter 3
Fuudamentals of Neural Networks
(4) Gaussian functions rp(r) = exp ( -
;;2 J
for some
(J>
0,
and
rE
R
(3.12)
where r represents the Euclidean distance between a center and a data point. The parameter
(J,
termed spread parameter, controls the radius of influence
of each basis function. The multi-quadratic and the Gaussian kernel functions share a common property-they are both localized functions, in the sense that rp --+
°
as r --+
00.
For the radial basis functions listed in
Eq. (3.9) to Eq. (3.12) to be nonsingular, the points {ZJ:l must all be different (i.e., distinct). This is all that is required for non singularity of the interpolation matrix (/J, regardless of the number N of the data points or dimensionality M of the vectors (points) Zi.
3.3.2
Research Progress in Radial Basis Function Networks
Broomhead and Lowe [185] were the first to exploit the use of the radial basis function (RBF) in the design of neural networks. They pointed out that the procedure of the RBF technique for strict interpolation may not be a good strategy for the training of RBF networks for certain classes of tasks because of poor generalization to new data. Specifically, when the number of data points in the training set is much larger than the number of degrees of freedom of the underlying process, and we are constrained to have as many RBFs as the data points presented, the problem is overdetermined. Consequently, the network may end up fitting misleading variations due to noise in the input data, thereby resulting in a degraded generalization performance. Broomhead and Lowe [185] removed the restriction for strict function interpolation and set up a two-layer network structure where the RBFs are employed as computation units in the hidden layer. They considered that the training phase of the network learning process constitutes the optimization of a fitting procedure for a desired surface, based on the known data points presented to the network in the form of input-output examples 49
3.3
Radial Basis Function Neural Networks
(patterns). The generalization phase of the network learning process was considered synonymous with interpolation between the data points. The interpolation being performed along the constrained surface generated by the fitting procedure was viewed as an optimal approximation to the true surface. Shown in Fig. 3.1 is the prototype of the basic RBF neural network, in which the computational units (referred to here as hidden neurons or hidden units) provide a set of "functions" that constitutes a "basis" for the network input vectors when they are expanded into the hidden unit space. The output of the RBF network is a linear combination of the outputs from its hidden units. z 21
CPi ""
cp(lz-zil)
CPI
WI Z2
I:
11'}
h(z)
zM-1
Output layer WN
zM
Input layer
Hidden layer
Figure 3.1 The architecture of an RBF network. The hidden unit
CPi
is centered at Zi'
The dimensionality of the input space is M and that of the hidden unit space is N
Poggio and Girosi [186] treated the problem of neural networks for approximation in a theoretical framework, based on regularization techniques. They highlighted the application of the traditional regularization techniques and derived the network structures which they called regularization networks. They considered an RBF neural network as a special case in the realm of regularization networks. By this approach, it is also possible to reveal what happens inside RBF networks. The network of Poggio and Girosi has two layers and the number of hidden units is fixed apriori. The centers of the hidden units are a subset of the input samples. The number of the network 50
Chapter 3
Fundamentals of Neural Networks
input is equal to the number of independent variables of the problem. The approximation of a continuous function depends on finding the linear weights connecting the hidden layer and output layer. The distance between the function constructed by the neural network and the desired function is measured by a cost function. The gradient descent algorithm can be used to find suitable values for the weights in order to obtain good approximation of the underlying function. Before applying the gradient algorithm, a suitable initialization is given for the output weights. In order to decrease the computational complexity required for finding exact solutions when the sample size is large, an approximation to the regularized solution was introduced by Poggio and Girosi [186]. Based on the "locally-tuned" concept of neurons in various biological nervous systems, Moody and Darken [187] presented a network containing computational units with locally-tuned response functions. This network model takes the advantages of local methods, which have been originally used for density estimation classification, interpolation and approximation. According to Moody and Darken [187], the local methods have attractive computational properties including parallelizability and rapid convergence. Due to the locality of the unit response, for any given input, only a small fraction of computational units with centers very close to the input (in the input space) will respond with activation which differs significantly from zero. Thus, only those neurons with centers close enough to the input need to be evaluated and trained. The structure of the proposed network is the same as an RBF network. A radially symmetric function, which has a single maximum at the origin and drops rapidly to zero at large radii is chosen as the RBF in the sole hidden layer of the network. For this network, the number of hidden units is chosen apriori. The RBF hidden neuron centers are chosen as a random subset of the available training patterns and no special assumptions are made about this selection such as orthogonality or being uniformly distributed over the input space. Two types of training algorithms were studied by Moody and Darken [187]. One is a fully supervised method and the other is a hybrid method combining the 51
3.3
Radial Basis Function Neural Networks
supervised and self-organizing methods. Comparison of the performance of the fully supervised method and the back propagation network shows that the supervised method does not learn appreciably faster than the back propagation network because it casts the learning as a nonlinear optimization problem which results in slow convergence. Furthermore, since this method assumes large width values for the Gaussian functions, it loses the locality desired for the computational units in the network. In the hybrid method, a self-organized learning scheme, which is the standard k-means clustering algorithm, is used to estimate the centers of the basis functions, and the width values are computed using various "P nearest-neighbor" heuristics. A linear supervised learning method (the standard least mean square (LMS) algorithm) is employed to estimate the output weights. Because the local response of the network is ensured by this hybrid algorithm, only a few hidden units respond to any given input, thus reducing the computational overhead. In addition, adaptation of the network parameters in this algorithm is linear rather than nonlinear, leading to fast convergence. Clearly, this hybrid learning algorithm is faster than the back propagation method. As pointed out by Poggio and Girosi [186], there are two basic problems associated with the k-means clustering based algorithm. The first problem is that there is still an element of chance in getting the right hidden neuron centers. Second, since clustering is more probability density oriented, this method is suitable for the problem of pattern classification, whereas for function approximation, clustering may not guarantee good results, because two samples close to each other in the input space do not necessarily have similar outputs. To overcome the problems, Chen et af. [188] proposed a more systematic procedure, known as orthogonal forward regression (OFR) to select the RBF hidden neuron centers. This iterative method was derived by augmenting and modifying the well-known orthogonal least squares (OLS) algorithm. In this procedure, a Gram-Schmidt type of orthogonal projection is employed to select the best center one at a time. The selection process terminates when 52
Chapter 3
Fundamentals of Neural Networks
the number of predetermined centers has been filled or the benefit of adding more hidden neurons becomes diminishingly small. So far, the learning algorithms we have described are based on the assumption that the number of hidden units is chosen apriori. During the application of the above algorithms, users are often puzzled by the problem of choosing the right number of hidden units. It may not be possible to train an RBF neural network to reach a desired level of performance if the network does not have enough hidden neurons, or the learning algorithm fails to fmd the optimal network parameters. Therefore, it is useful to develop a new type of learning algorithm capable of automatically recruiting new hidden units whenever necessary for improving the network performance. A hierarchically self-organizing learning (HSOL) algorithm for RBF was developed in [189]. This algorithm was capable of automatically recruiting new computational units whenever necessary for improving the network performance. In the HSOL algorithm, the hidden units are associated with the accommodation boundaries defined in the input space and the class representation defined in the output space. The accommodation boundary of a hidden unit defines a region of input space upon which the corresponding hidden unit has an influence. If a new sample falls within the accommodation boundary of one of the currently existing hidden units, which has the same class representation as that of the new sample, then the network will not generate a new hidden unit but accommodate the new sample by updating the parameters of the existing hidden units. Otherwise, the network will recruit a new hidden unit. Furthermore, in HSOL, the accommodation boundaries of individual hidden units are not fixed but adjusted dynamically in such a way as to achieve hierarchical learning. Initially, the accommodation boundaries are set large for achieving rough but global learning, and gradually reduced to a smaller size for fine learning. In summary, the HSOL starts with learning global mapping features based on a small number of computational units with larger accommodation boundaries and then proceeds to learn finer mapping details with increasing number of computational units and diminishing accommodation boundaries. The HSOL algorithm starts 53
3.3
Radial Basis Function Neural Networks
from no hidden units and then builds up the number of hidden units from the input data. An alternative approach is to start with as many hidden units as the number of inputs and then reduce them using a clustering algorithm which essentially puts close patterns in the input space into a cluster to remove the unnecessary hidden units. Such an approach has been developed by Musavi et al. [190] and it essentially consists of an iterative clustering algorithm that takes class membership into consideration and provides an approach to remove the unnecessary hidden units. This approach also provides for the width estimation of the basis functions. After the training patterns have been clustered and their centers are found, the algorithm selects the width parameters or the Gaussian functions in the hidden layer. This is done by fmding the eigenvalues inside the variance matrix of each Gaussian function. The Gram-Schmidt orthogonalization procedure is utilized to determine the normalized axes of the contour or the constant potential surface of the corresponding Gaussian function and the projections of the nearest neighbor of the opposite class on these axes. The eigenvalues can then be derived from these projections. In practical applications of an RBF neural network, learning results from many presentations of a prescribed set of training examples to the network. One complete presentation of the entire training set during the network learning process is called an epoch. There are two types of learning. One is called batch learning. To perform batch learning, the network parameters are updated after the presentation of all training patterns which constitute an epoch. So the batch learning process is maintained on an epoch-by-epoch basis until the parameters of the network stabilize and the average network output error over the entire training set converges to some minimum value. All the learning algorithms mentioned above belong to batch learning. The other type oflearning is sequential (on-line/recursive) learning, in which the network parameters are adjusted after the presentation of each training pattern. To be specific, consider an epoch consisting of N training examples (patterns) arranged in the order [ZI' d l ],
... , [ZN'
dN ]. The first example [z], d l ] in the
epoch is presented to network and the network output is calculated. The 54
Chapter 3
Fundamentals of Neural Networks
network parameters are then updated. After this, the second example
[Z2,
d2 ]
is presented and the above procedure is repeated resulting in further adaptation of the network parameters. This process is continued until the last example
[ZN'
dN ] is accounted for. In the rest of the section, we look at some
sequential learning algorithms. In addition to OFR algorithm mentioned above, Chen et al. have proposed a recursive hybrid learning algorithm for RBF neural networks [191]. They applied such an RBF neural network for on-line identification of nonlinear dynamical systems. Their algorithm employs a hybrid clustering and least squares algorithm. The recursive clustering algorithm adjusts the hidden neuron centers for the RBF network while the recursive least squares algorithm estimates the connection weights. Thin-plate-spline functions are chosen as the RBFs in the network and only the hidden unit center values have to be determined in the network hidden layer. However, in this algorithm, the number of hidden units has to be determined before training commences, and this number varies from application to application. To remedy this problem, Platt proposed a sequential learning algorithm for a resource allocation network (RAN) [192]. This algorithm is inspired by the method of a harsh table lookup, which was proposed by Moody [187]. Based on the idea of adjusting the number of hidden units to reflect the complexity of the function to be interpolated, hidden neurons are added based on the "novelty" (referred to as "innovations" in the estimation literature) of the network input data. The network parameters are then estimated using the well-known least mean square (LMS) algorithm. A new pattern is considered novel if that input pattern is far away from the existing centers and if the error between the network output and the desired output is large. If no additional hidden neuron is added, the parameters of the existing hidden neurons, such as the centers, widths and weights, are updated by the LMS algorithm. Kadirkamanathan and Niranjan interpreted Platt's RAN from the viewpoint of function space [193]. However, it should be noted that even with the learning algorithm developed by Platt [193], it is still difficult to achieve a minimal RBF network. This is due to the following senous 55
3.4
Recurreut Neural Networks
drawback of REF networks. Once a hidden unit is created, it can never be removed. Because of this, REF networks could produce networks in which some hidden units, although active initially, may subsequently end up contributing little to the network output. Thus, pruning becomes imperative for the identification problems of nonlinear systems with changing dynamics, because failing to prune the network in such cases will result in the presence of numerous inactive hidden neurons. If inactive hidden units can be detected and removed as learning progresses, a more parsimonious network topology can be realized. In this book, we will consider a modified REF network with a learning algorithm that combines the pruning strategy to minimize the network. In order to ensure that transition in the number of hidden neurons is smooth, the novelty criteria in RAN are augmented with an additional growth criterion based on the root mean square value of the output error over a sliding data window. In brief, the new algorithm developed in this book improves the basic method of hidden neuron growth and network parameter adaptation by adding a pruning strategy to achieve a minimal REF network. Details will be given in Chapter 7.
3.4 3.4.1
Recurrent Neural Networks Introduction
Feedforward neural networks (FNNs) have been known to have powerful capability for processing static information [194], function approximation [176], etc. When temporal information is concerned, we can either add time delay steps to spatialize the temporal signal or use recurrent networks to process the temporal information. Recurrent neural networks (RNN s) are characterized by adding recurrent connections to the feedforward networks. The feedback links provide extra capability to process temporal information [195]. Under certain conditions, the RNN s can also make generalization from the training data to produce smooth and consistent dynamical behavior for 56
Chapter 3
Fundamentals of Neural Networks
entirely new inputs or new regions of the state space (i.e., inputs or regions of the state space not encountered during training). RNN network techniques have been applied to a wide variety of problems. Simple partially RNNs were introduced in the late 1980's by Rumelhart et al. [196] to learn strings of characters. Recently, many other applications have been reported involving dynamical systems with time sequences of events. For example, the dynamics of tracking the human head for virtual reality systems was studied by Saad [197]. The forecasting of financial data and of electric power demand were the objects of other studies [198, 199]. RNNs were also used to track water quality and to minimize the additives needed for filtering water [200].
3.4.2
Topology of the Recurrent Networks
On the research of the topology of the recurrent networks, much work has been done on the analysis and comparison of the location of the feedback connections. In general, the treatment on the feedback terms can be categorized into three approaches: the locally recurrent globally feedforward approach, the context unit approach and the global feedback approach. In the first approach, Back and Tsoi [201] suggested an architecture with the time delays and the feedback terms similar to those in infinitive impulse response (UR) and finitive impulse response (FIR) filters, all occurring inside a synapse. Frasconi et al. [202] allow self-feedback connections in the dynamical neurons only. There are no inter-neuron feedback terms in these models but multiple delays are allowed in each feedback. Tsoi and Back [203] named this approach local recurrent and global feedback architecture. Alternatively, inter-neural feedbacks are present in the other two approaches. Examples of the context unit approach are the Jordan network shown in Fig. 3.2 and the Elman network in Fig. 3.3. Jordan [204] copied the activation of the output units into the context units. The context units, together with self-feedback loops, link feedforwardly to the hidden units. These feedback links let the hidden layer interpret the previous state of the network and thus 57
3.4
Recurrent Neural Networks
provide the network with memory. Elman [205], on the other hand, copied the activation of the hidden units to form the context units. The hidden units appear to encode both the present input information and the previous states represented by the hidden units. The feedback connections to the context units are non-trainable and thus the generalized delta rule can be used for the training of the whole network. As errors are not backpropagating through time, long term dependence on time is difficult to model. Output
Output
Input units
Input units
Input
Input
Figure 3.2 A Jordan network
t
Figure 3.3 An Elman network
Finally, the global feedback approach allows inter-neuron feedback. William and Zipser [206] derived the real-time recurrent learning (RTRL) algorithm using a fully connected structure as shown in Fig. 3.4. All noninput units, including the output units and hidden units, are fully feedbacked by the recurrent weights. Robinson and Fallside [207] used a similar architecture but only the hidden units are fully connected with the recurrent weights.
3.4.3
Learning Algorithms
The learning algorithm defmes how the connection weights are being updated. The generalized delta rule for back propagation network is generally not applicable directly as the output activations are recursively dependent on the 58
Chapter 3
Fuudamentals of Neural Networks
Output
v
Figure 3.4 Architectural graph of a real-time recurrent network
recurrent connections. A simple modification is the back propagation through time (BPTT) by Rumelhart et al. [196]. Unfortunately this learning algorithm is not local in time and the amount of storage needed varies with the length of the training samples. The BPTT will be a good choice if the training sequences are known, in advance, to be short in length. Another well-known learning rule for the recurrent network is the RTRL by Williams and Zipser [206] which uses the sensitivity matrix to provide an on-line learning algorithm of the recurrent networks. There are some other learning rules, such as the one by Schmidhuber [208], which combines the BPTT and the RTRL to form a learning algorithm. Another variation is the subgrouped RTRL proposed by Zipser [209], which divides the fully recurrent network into sub-groups. The connections between different subnets are fixed and nontrainable. In Chapter 8, we will use a modified RNN to realize the equalization task in chaos-based communication systems.
59
Chapter 4
Signal Reconstruction in N oisefree and Distortionless Channels
In Chapter 2, it has been shown that embedding reconstructions in the context of the Takens' embedding theory are applicable only for timeinvariant systems. It is shown in this chapter that embedding reconstructions can be applied to time varying systems based on an observer approach. In particular, we consider the Lur' e system. As an application example, we discuss the information retrieval in chaos-based communication systems in the absence of noise and distortion.
4.1
Reconstruction of Attractor for Continuous TimeVarying Systems
Consider the chaotic system
x = g(x,t), where
X= [Xl Xz ... XD]T
(4.l)
is the D-dimensional state vector of the system, and
g is a smooth nonlinear function defined on ffi. D X R The output (observed) signal of the system s is generally given by s = rfJ(x(t)),
where
rfJO
(4.2)
is a smooth continuous scalar real-valued function. The goal is to
reconstruct the attractor using only a function of a subset of the state variables. In the Euclidean space ffi.M, the chaotic attractor of Eq. (4.l) can be reconstructed
from
s=[s 2
s s ... dM-l
S(M-l)f,
where
M~(2D+l), and
ds d s . s . 1 1 s,. .s, ... , s (M-l) denote - , - 2 ,. .. , ~, respectlve y [ 9]. In other words, dt dt dt-
Chapter 4
there exists a function
I]'
Signal Reconstruction in Noisefree and Distortionless Channels
such tha x = 1]'(S,t),
(4.3)
where 1]'=['1/1 'l/2···'l/D]T.It should be noted that s=¢J(x(t)) Vx¢J(x(t)).g(x,t)
def
def
J;(x,t),s=
12(x,t),and ds(i-I)
s(i)=-dt =
VxJ;(x,t). g(x,t) + oJ;(x,t)
at
(4.4) whereVxJ;(x,t) is the gradient ofJ;(x,t) with respect to x,and "." denotes the vector dot (inner) product. Also,
s = f(x,t), where f
= [J; h ... 1M t ,and
(4.5)
J; (i = 1,2" . " M) is a smooth function. Combining
Eq. (4.3) and Eq. (4.5), we have x = I]'(f(x,t),t).
(4.6)
By taking the derivative with respect to t of S(M-l) = 1M (x, t) and using the above equations, we obtain
(4.7) Defining the variablesYI =S'Y2 =S'Y3 =s,""YM =S(M-ll, we have another form ofEq. (4.1) in a higher dimensional space: Yl=Y2'
Y2 = Y3' YM
1
(4.8)
~ h(Yph,···,YM,t)· 61
4.2
Reconstruction and Observability
Suppose that h is a polynomial that does not depend on t. Then we can determine the parameters of h. The function h can be written as:
(4.9) The parameters ak\,k"k3 , .. ,kM can be determined from set) by finding sand S(M)
at a finite number of time points and plugging into Eq. (4.9) to obtain a
set of linear equations which we can solve if s has not degenerate. Thus, it is generally possible to reconstruct the chaotic attractor of Eq. (4.1) on the higher-dimensional space from given s(t). It is also possible to estimate unknown parameters using the above method. CD The requirement of M? 2D + I is a suffcient condition on the dimension of the reconstructed phase space for this method to work. In certain cases the attractor can be reconstructed using a smaller M. Specifically, we consider the reconstruction of a class of Lure system in the next section, where M can be set to D.
4.2
Reconstruction and Observability
The pair (A, w
T
)
is said to be observable if the matrix
wT A=A(A,WT)=
wTA wTA2
(4.10)
is nonsingular, where A is a D x D matrix and w is a D x I vector. In
linear
system
eD = [0 0 ...
If E jRD,
theory,
the
statement
that
(A,e~), where
is observable means that we can reconstruct the state
vector x in the system governed by the dynamics
x =Ax
via observing
CD The same conclusion holds for a chaotic system with a hysteresis loop if the system can be considered to be smooth.
62
Chapter 4
Signal Reconstruction in Noisefree and Distortionless Channels
only xD for some time [210]. We consider now the nonlinear Lure system defined by (4.11 ) where el = [1 0 0
Of E]RD,
U
is a continuous real-valued function, and
A is given by 0
0
-bo
1 0
0
-bl
0
0
-b2
0
0 where bi, i
=
0
(4.12)
1 -bD _ l
0, 1, 2,"', (D -1), is the coefficient of the characteristic
polynomial of matrix A, i.e., (4.13) When (A,e~) is observable, we can reconstruct the system dynamics via (XD,XD,X D,.·
·,xb
D l - )).
In fact, if we defmec( as the ith row of Aj, it is easy to
see thatcbel =0 for j=O, 1,.··,(D-2). Then, x D =e~x=c~x,XD =c1x, and xD=c1Ax+c1elu(x,t)=c~x,.··,x~-1)=c~-1)x. This means that s=
xf = A(A,e~)x. SinceA(A,e~) is invertible, there exists an invertible linear mapping between s sand x. We can write that Xl as [e~x c1x ... C~-l)
(4.14) Therefore, we can reconstruct the system dynamics of the Lure system.
4.3
Communications Based on Reconstruction Approach
Consider the well-known Duffing's equations (4.15)
63
4.3
Communications Based on Reconstruction Approach
where
rj
and
r2
are some constant parameters. Shown in Fig. 4.1 is the
two-dimensional phase space trajectory of the Duffing's equations with rj
= 0.15 and r2 = 0.3. 1.5
1.0
0.5
0
>< -0.5
-1.0
-1.5
-1.5
-1.0
-0.5
0
0.5
1.0
1.5
XI
Figure 4.1 The two-dimensional phase space trajectory of the Duffing equations with the parameter values rl
= 0.15 and r2 = 0.3
The above equations are equivalent to the following scalar differential equation (4.16) Therefore, if the signal Xj(t) is given, we can easily reconstruct a chaotic attractor ofEq. (4.15), by using
(XI (t),
Xl (t)) , due to the following relation:
(4.17)
i.e., the system being observable.
4.3.1
Parameter Estimations
The parameters in Eq. (4.15) can be estimated from a given signal. If Xj (t) is 64
Chapter 4
Signal Reconstruction in Noisefree and Distortionless Channels
given, then we can find the parameters rl and r2 from the following relation: (ll) - COS(ll) ] ['i 1 [Xl (ll) - X;3 (ll) - ~~ (ll) ]. [ Xl~l (l2) - COS(l2) r2 XJ(2) - Xl (t2) - Xl (lJ =
Shown in Figs. 4.2 and 4.3 are the errors of the estimated
(4.18) r]
and
r2
as a
-76
-77
S
~
,,"
"0
'"
.~
"0
'-
g P.l
-S2 -S3
0
100
200
300
400
500
Number of timesteps
Figure 4.2 Error curve of the estimated system parameter rl versus the time step -]07
-lOS
iii:
~
"0
'"
'"
.~
-III
'"0 '....0 t:
P.l
-113 -114
0
100
200
300
400
500
Number of time steps
Figure 4.3 Error curve of the estimated system parameter r2 versus the timestep
65
4.3
Communications Based on Reconstruction Approach
function of the timestep, when the real parameter values in Eq. (4.15) are set to
rl
= 0.15 and r2 = 0.3.
4.3.2
Information Retrievals
An information signal v(t) can be injected into the system Eq. (4.15) (4.19) or equivalently
X1 =x2 ' x2 = -fix2 + x 1 - x~ + r2 cos(t) + vet).
(4.20)
Then, vet) can be reconstructed by using (X1 (t),X1 (t),x1(t)): (4.21 ) Figure 4.4 shows the retrieved information signal (circled line) when vet) to be injected into Eq. (4.15) is a sinusoidal function (solid line) of frequency 1/ 811: and amplitude 0.1. 0.15
o
0.10
0.05
s
0
'" -0.05
-0.10 0
Original Retrieved
AI ' J
f\\
\ ~ 20
40
\
\\ t \;
60
80
100
Time
Figure 4.4 The retrieved information signal and the original one injected into Eq. (4.20)
66
Chapter 4
Signal Reconstruction in Noisefree and Distortionless Channels
Another system to be considered here is the Chua's circuit studied in Chapter 2. Let us rewrite the dynamical equations as follows
(4.22)
X2 =xI -X2 +X3' X3 = a l x 2 ,
where K(-) is a piecewise-linear function as described in Eq. (2.8). We can reco-
x
nstruct the chaotic attractor by using (X3' 3' x 3) when a2 -:t o. In fact, we have
X3
=
a 2 x2 ,
X3
=
a 2(xl -X2 + xJ.
(4.23)
Thus, we obtain
(4.24)
Furthermore, putting Eq. (4.23) and Eq. (4.24) in Eq. (4.22), we have (4.25) If K(·) is a cubic polynomial K(X) = d 1X3+ d2x [138], then we get
a;xi3) + dial (x; + 3X3X~ - 6a2x 3x3x 3 + 3a;x3x~ ··2 X3 - 3a 2x··23x 3 +X3·3 - 3a 2x·23x 3 + 3a 22·x3x 32 -a23 x33) + 3X3 + a; [(1 + d 2a l )x3 + (-a 2 -al +d2a l )x3 (4.26)
+dpI(al -a2)x3]=O.
We can estimate the parameters in Eq. (4.22) from the given signal X3 by using (X3 (t), X3 (t), X3 (t), xi 3)(t)). Furthermore, we inject the information signal vet) into the system:
J- ()
•. X3(3) +X3 -(al +a2)x. 3 +ap2 K [X3 +X3a -a2X3 -a2v t , 2
(4.27)
67
4.3 Communications Based on Reconstruction Approach
or equivalently
(4.28)
Xl =Xj -Xl +X3'
X3 = alxl ·
For a l (x],
-:f-
0 we can reconstruct (x\,
X2, X3).
X2, X3)
from
X3
and then recover vet) from
In other words, the chaotic attractor and v(t) are reconstructed by
(4.30)
Figure 4.5 shows the retrieved information signal (circled line) when vet) to -
0.25
o
Original Retrieved
0.20 0.15 0.10
S
"
0.05 0
1\ \
. I
I
\f
1\ \
-0.05 -0.10 -0.15 -0.20 0
5
10
15
20 Time
25
30
35
40
Figure 4.5 The retrieved information signal and the original one is injected into Eq. (4.28) with a j = 9 and a 2 = -100/7
68
Chapter 4 Signal Reconstruction in Noisefree and Distortionless Channels
be injected into Eq. (4.28) with
a]
= 9 and
a2
100
= -""7' is a sinusoidal
function (solid line) of frequency lin and amplitude 0.2.
4.4
Reconstruction of Attractor for Discrete Time-Varying Systems
The reconstruction approach can be applied not only to continuous-time dynamical systems, but also to discrete-time dynamical systems. Only a slight change is needed for application to discrete-time systems. The results obtained for continuous systems are transformed to those for discrete-time dynamical systems by replacing the derivatives with time-advanced state variables: (x(t), x(t),x(t),"') ~ (x(t),x(t + 1), x(t + 2)", J
(4.31 )
The advantage of using discrete-time systems is that there are no errors introduced by numerical differentiation, unlike in the continuous-time case. Consider the HEmon map governed by the dynamical equations: x\(t + 1) = 1- r\x~(t) + x2(t),
(4.32)
x 2(t + 1) = r2x\ (t),
where r\ and r2 are the two bifurcation parameters of the system, which determine the dynamical behavior of the system(]).For r2
-:j:.
0, the attractor
can be reconstructed from X2(t) and X2 (t + 1) as
The information signal vet) is then injected into the system
CD For some selected range of parameter values, the system can show the chaotic behavior (see Sec. 4.2).
69
4.4
Reconstruction of Attractor for Discrete Time-Varying Systems
XI (t + 1) = 1- rlxlz (t) + Xz (t) + vet), X z (t
(4.33)
+ 1) = rzxI (t),
or equivalently (4.34) Shown in Fig. 4.6 is the chaotic atiractor of the Henon map when the system parameters are Yl = 1.4 and Y2 = 0.3 and the inj ected signal is a sinusoidal function of amplitude 0.001 and frequency 1/20 . 0.4 0.3 0.2 0.1
k'
0 -0.1 -0.2 ..•....
-0.3 -0.4 -1.5
-1.0
-0.5
0.5
1.0
1.5
Figure 4.6 Chaotic attractor of the Henon map with YI = 1.4 and Yz = 0.3. The
injected signal vet) in Eq. (4.33) is a sinusoidal function of amplitude 0.001 and period 1/20
Again, when Y2;j:. 0, set) can be recovered from X2(t), X2(t +1), and
x2(t+2) as: vet)
X
z
(t + 2) rz
r
+-+x~(t+l)-xz(t)-1. rz
(4.35)
Shown in Fig. 4.7 are the recovered information signal (circled line) and the original one (solid line) injected into Eq. (4.33).
70
Chapter 4
Signal Reconstruction in Noisefree and Distortionless Channels
-Original o Retrieved
0.5
S
0
;:.
-0.5
-\.O 0
25
50
75
100
125
ISO
Time Figure 4.7 The retrieved information signal and the original one injected into
Eq. (4.33)
4.5
Summary
In this chapter, we have extended the Takens' embedding theory to the reconstruction of continuous and discrete-time varying systems based on the observer approach. In particular, we have studied the Lure system, which can be reconstructed in a state space whose dimension is equal to the degree of freedom of the system. Also, the information signals injected into the so-constructed chaos-based communication systems in the absence of noise and distortion can be retrieved.
71
Chapter 5
5.1 5.1.1
Signal Reconstruction from a Filtering Viewpoint: Theory
The Kalman Filter and Extended Kalman Filter The Kalman Filter
The Kalman filter (KF) is a recursIve filtering tool which has been developed for estimating the trajectory of a system from a series of noisy and/or incomplete observations of the system's state [211- 231]. It has the following characteristics. First, the estimation process is fonnulated in the system's state space; Second, the solution is obtained by recursive computation; Third, it uses an adaptive algorithm, which can be directly applied to stationary and non-stationary environment [214]. In the Kalman filtering algorithm, every new estimate of the state is retrieved from the previous one and the new input so that only the previous estimated result need to be stored, and all those before the previous one can be discarded. Thus, the Kalman filter is more effective in computation than those which use all or considerable amount of the previous data directly in each estimation [214, 215]. For the purpose of our discussion, it suffces to consider the state of a system as the minimum collection of data which can describe the dynamical behavior of the system. Knowledge of the state is necessary for the prediction of the system's future trajectory, and is also relevant to the past trajectory of the system. In discrete time, a dynamical system can be described by a process equation that essentially defines the system dynamics in tenns of the state, and an observation equation that gives the observed signal, i.e.,
Chapter 5
Signal Reconstruction from a Filtering Viewpoint: Theory
• Pcrocess equation The process equation of a dynamical system can be generally written as (5.1) where xn is an N-dim state vector of the system at discrete time n; F is an N x N state transition matrix representing the movement of the dynamical system from time n to n + 1 and is usually given; wn is an N-dim process noise vector that describes the additive noise modelled as a zero-mean, white noise process whose correlation matrix is defined as: (5.2) where
Q;
is a diagonal matrix and 0 is a zero matrix.
• Measurement equation The measurement equation provides a linkage between the observed output and the system's state, and can be described as (5.3) where Y n is an M-dim vector representing the observed or measured output of the dynamical system at instant n;
en
is an M x N observation matrix,
which makes the state observable and is usually required to be given; vn is an M-dim observation noise vector, which is modelled as a zero-mean white noise process whose correlation matrix is defined as:
{Q0,n' V
E[vnvJ] = where
Q;
n- k -
n"#
k
(5.4)
is a diagonal matrix and 0 is a zero matrix.
Suppose wn and vn are statistically independent. Then, we have (5.5)
73
5.1
The Kalman Filter and Extended Kalman Filter
Therefore, the filtering problem, namely solving the state process equation and the observation equation by means of optimization, can be described as follows. From the observed dataYpY2" .. ,Yn' for all n"? 1, the problem is to find the least squared estimate of each component of the state variable Xi' If
n = i, the problem is a filtering one; if i > n , it belongs to prediction; and if 1 ,,;; i < n, , then it would be a process of smoothing.
We introduce an innovation process (innovation in short) to Kalman filtering, which represents the update of new information of the observation vector yen). In effect, innovation can be regarded as one-step forward prediction. With the observed values yp Y2' ... , Yn-P we can obtain the least A f0 b ' servatIon vector Y n as YAdef n=Yn IYp ... ,Yn- 1• So the M-dim innovated vector, an , is generally given as
· squared estImate
0
(5.6) Note that an is a Gaussian noise process which is uncorrelated with the observed data YP""Y n-1 before the instant n [219]. Nonetheless, it provides new information about Y n • This is why we name it as innovation. Basically, the Kalman filter algorithm includes two stages: prediction and correction. We define the one-step predictive value and the estimate of the state vector, respectively, as (5.7) (5.8) A summary of the main parameters and equations in the Kalman filter algorithm is as follows.
• One-step prediction of initial state Correlation matrix of predicted state errors:
74
Chapter 5
Signal Reconstruction from a Filtering Viewpoint: Theory
• Input vector Observed vector sequence: {YPY2, ... ,Yn}
• Given parameters State transfer matrix: F(n + 1,n) Observation matrix: Cn Correlation matrix of process noise vector:
Q:
Correlation matrix of observation noise vector:
Q;
• Prediction One-step state prediction
xn : Xn =F(n,n-1)xn_1
One-step prediction for observed vector Yn
(5.9)
:
Yn =CnXn
(5.10)
A
Correlation matrix of state prediction error Pn1n -1 :
Pn1n -1 = F(n,n -1)Pn_1F(n,n-1) +
Q:
(5.11)
• Error correction Kalman gain matrix Kn : (5.12) Innovation a(n) : (5.13) Error correction for state estimate
xn: (5.14)
Error correction for correlation matrix of state estimate errors: (5.15)
75
5.1
The Kalman Filter and Extended Kalman Filter
In conclusion, the Kalman filter is essentially a finite-dimensional linear discrete-time recursive algorithm. The core of the algorithm is to minimize the trace of correlation matrix of the state estimate error Pn • In other words, the Kalman filter performs a linear least squared estimation for the state vector xn [221, 223].
5.1.2
Extended Kalman Filter
The Kalman filter algorithm is a stochastic estimation tool for linear systems. If the system is nonlinear, the Kalman filter cannot be applied directly, and an additional linearization process is required. Incorporating a linearization process in the Kalman filter, we have the extended Kalman filter (EKF) [232 - 247]. Consider a nonlinear system whose state-space description is given as (S.16) (S.17)
where wn and vn are uncorrelated zero-mean white nOIse processes with correlation matrix Q: and
Q~,
respectively, and F(n, xJ is a nonlinear
transfer function that may vary with time. Moreover, if F is a linear function, we have (S.18)
The fundamental idea of the EKF lies in linearizing the state-space model given in the form of Eqs. (S.16) and (S.17) at each time instant by using a first-order Taylor's expansion. Once the linearized system is obtained, the Kalman filter algorithm described in the previous section can be applied. The linearization process is described mathematically as follows: First, we construct two matrices as follows. F(n + 1,n) = of(n,x)1 ' ox X~X"
76
(S.19)
Chapter 5
Signal Reconstruction from a Filtering Viewpoint: Theory
(5.20) Next, we obtain F(n+ l,n)and C(n) by using F(n,x n ) and C(n,x n ) , and expand them according to first-order Taylor expansion at
x
n
•
This gives (5.21) (5.22)
Then, we linearize the nonlinear state-space representation (i.e., Eq. (5.16) and Eq. (5.17)), with the help ofEqs. (5.21) and (5.22). Finally, we obtain (5.23) (5.24) which correspond to Eq. (5.9) and Eq. (5.13) in the basic Kalman filter formulation. The complete EKF algorithm can be formulated by putting the linearized terms described above into the Kalman filter algorithm (from Eqs. (5.9 - 5.15)). It should be noted that the EKF algorithm is, in fact, a direct application
of the original Kalman filter algorithm with an additional linearization process. Moreover, since the EKF uses a first-order approximation, when the system has a higher degree of nonlinearity, the error caused by the higherorder terms of the Taylor's expansion might cause the algorithm to diverge [242,245,247], as will be demonstrated in the simulation experiments to be reported in the next chapter.
5.2
The Unscented Kalman Filter
Recently, a new type of filters, known as the unscented Kalman filter (UKF), has been proposed for noise cleaning applications [248,249,250]. The fundamental difference between EKF and UKF lies in the way in which 77
5.2
The Unscented Kalman Filter
Gaussian random variables (GRV) are represented in the process of propagating through the system dynamics. Basically, the UKF captures the posterior mean and covariance of GRV accurately to the third order (in terms of Taylor series expansion) for any form of nonlinearity, whereas the EKF only achieves first-order accuracy. Moreover, since no explicit Jacobian or Hession calculations are necessary in the UKF algorithm, the computational complexity of UKF is comparable to EKF. Very recently, Feng and Xie [260] applied the UKF algorithm to filter noisy chaotic signals and equalize blind channels for chaos-based communication systems. The results indicate that the UKF algorithm outperforms conventional adaptive filter algorithms including the EKF algorithm. In this section, we evaluate the performance of UKF in filtering chaotic signals generated from one-dimensional discrete-time dynamical systems which basically include most behaviors and characteristics in many multidimensional systems, and are widely used in signal processing and communications with chaos.
5.2.1
The Unscented Kalman Filtering Algorithm
Consider a nonlinear dynamical system, which is given by (5.25) where f: lR N ~ lR N is a smooth function, and the measurement equation is given by (5.26) where
vn
is a zero-mean white Gaussian noise process, E
[VjVn]
= Qbjn
> 0,
and b jn is the Kronecker delta function, Q is a matrix with a suitable dimensionality. When the global Lyapunov exponent of Eq. (5.25) is positive, the system is chaotic. We will employ the the UKF algorithm
78
Chapter 5
Signal Reconstruction from a Filtering Viewpoint: Theory
developed in [248,249,250] to filter the noisy chaotic time series, i.e., to estimate the state xn fromy n . Assume that the statistics of random variable x (dimension L) has mean x and covariance Px • To calculate the statistics of y, we form a matrix X of 2L + 1 sigma vector Xi according to the following:
Xo
(5.27)
=X
Xi =x+(~(L+A)Px)i' i=I,2,. .. ,L Xi
(5.28)
= x -(~(L + A)Px )i-L' i = L + 1,L + 2,. .. ,2L
(5.29)
where A = a 2 (L + K) - L is a scaling parameter. The constant a determines the spread of the sigma points around x and is usually set to a small positive value (0.0001 < a < 1). The constant K is a secondary scaling parameter which is usually set to Also, (~(L + A)PX)i
def
K
= 0 (for parameter estimation,
K
=3 -
L).
ei is the vector of the ith column of the matrix square
root. These sigma vectors are propagated through the nonlinear function (5.30) The mean and the covariance of z can be approximated by using the weighted sample mean and covariance of the posterior sigma points, i.e., (5.31 ) i=O
2L
P"z = LWY)(z-z)(z-zY
(5.32)
i~O
where weights
Wi
are given by m
A
(5.33)
w =-o L+A
A L+A
w~ = - - + (1- a + /3) 2
(5.34)
79
5.2
The Unscented Kalman Filter
wm
= w~ =
I
I
1 . , 1 = 1,2, .. ·,2L 2(L+A)
(5.35)
f3 is used to incorporate prior knowledge of the distribution of x Gaussian distributions, f3 = 2 is optimal) [248, 249].
where
(for
The transform process described above is known as unscented transform (UT). Now the algorithm can be generalized as follows:
Time update: (5.36) (5.37) 2L
~In-l
=
L w; (Xi,nln-l - xn1n - )(Xi.nln-l - xn1n 1
(5.38)
1) T
i~O
2L
Px, y, =
L w; Cri,nln-l - xn1n - )(Yi,nln-l - Ynln- f 1
(5.39)
1
i=O
where ~-1
%nln-1'
xn1n -
1
and Pn1n - 1 is the predicted estimation for
%n-l'
x n_1 and
respectively. Px, y, is the covariance matrix of x and Y at instant n.
Measurement update: (5.40) (5.41) Pn =Pnln-l -Kn PYn Yn KTn
where Py ,
y,
is the covariance matrix of y at instant n,
(5.42)
Ynln-l
is the estimation
of the observed signal, and Kn is the Kalman gain matrix at time instant n. Using Eqs. (5.26), (5.32), (5.38) and (5.39), we can get Pnln-l =PXn Yn =PYn Yn -Qn'
80
(5.43)
Chapter 5
Signal Reconstruction from a Filtering Viewpoint: Theory
which then gives
P=KQ n n n
(5.44)
where Qn is the covariance matrix of the measurement noise at instant n. Furthermore, Pn1n -1 can be calculated as follows. Ifwe consider the prior variable x by a zero-mean disturbance ~ = x - X, then Px
~x,
where
= ~x ~XT, so that the Taylor series expansion of
Eq. (5.25) is given by
I(x) = I(x) + f'(x)(~x)+ .!.(~xf rex) + 001 ~xI12) 2 where
0 (-)
(5.45)
denotes infinite smallness, and II . II denotes the Euclidean norm.
From Eqs. (5.27 - 5.29), we know that Xn-l is symmetrically distributed L
around xn_l' Since ~>i.n-l
as 2L
m[
x n1n - 1 = ~ wi I(x n- l ) + f'(xn-l)(ei,n-l)
+~(ei,n_I)T r(ei,n_I)+oOlei,n_lln]
(5.46)
Since 2L
2L
L w; = L w; + (1- a i~O
2
+ /3)
(5.47)
i~O
we can express Pn1n -1 as 2L
" m Pn1n -1 = "~ Wi (Xi,nln-l
-
Xn1n -1)(Xi,nln-l A
-
Xn1n -1 )T A
i=O
2
+ (1- a + /3)(XO,nln-1 )T . XO,nln-1 - Xn1n - 1
-
Xn1n - 1 A
)
A
(
Again, Px
(5.48)
= E[(x - x)(x - X)T] = E(XXT)- xxT. We define
81
5.2
The Unscented Kalman Filter 2L
U=
L w;m (Z;,nln-I - xn1n - 1)(Z;,nln-I - xn1n _
l )T
(5.49)
i=O
V = (1- a 2 + fJ)(ZO,nln-1 We also define f' = F and
f" = H.
~)(
X n1n _1
ZO,nln-1 -
~)T
X n1n _1
(5.50)
Now, since
2L
LW~ =1,
(5.51)
;~O
we get U=
Fn~IP,,_IFn_1 -~[(Hn_IPn_Y (Hn_IPn_I )] +0(11 en _ 1 W)
(5.52)
(5,53) Since a is small, so is en_I' and the higher-order terms can be truncated. Thus, P n1n - 1 can be expressed as
(5.54) where F n _ 1 and H n _ 1 are the Jacobian and Hession matrices of Eq, (5,25), respectively.
5.2.2
Convergence Analysis for the UKF Algorithm
When we consider a one-dimensional system, Eq. (5.54) can be simplified as 2 2 Pnln-I =F2n-I Pn +'!"(fJ-a )Hn-I p2 4 n-I
82
(5.55)
Chapter 5
Signal Reconstruction from a Filtering Viewpoint: Theory
By using Eqs.(5.44), (5.54) and (5.55), we can calculate the Kalman gain as k
= n
l
F2 +l(p_a2)H2 p2 n-I
4
n-I n-I
]k
n-I
(5.56)
[F2n-I + 4(P 1 - a 2 )Hn-1P"_1 2 ] kn_ 1 + 1
Note that both p" and kn can be used to measure the filter's performance. However, since they are linearly dependent, as shown in Eq. (5.44), it suffces to analyze the behavior of any one of them. For simplicity, we focus on kn • In general, four types of behavior can be identified in a nonlinear dynamical system: • stable fixed point; • periodic motion; •
quasiperiodic motion;
• chaotic state. In a one-dimensional dissipative system, a quasi-periodic state corresponds to a zero Lyapunov exponent, and it is a critical state with zero measure of parameters. It is therefore not of interest to our present study. Here, we consider two classes of dynamical behavior, namely, chaotic and periodic, as the stable fixed point can be regarded as a special periodic state. Our analysis for the UKF algorithm used to filter a chaotic system is summarized in the following theorems. In particular, we consider three types of nonlinear dynamical systems which determine the behavior ofUKF. • Type 1: Fn~1 is independent of x n _ l • • Type 2: Fn~1 is dependent upon x n _ 1 , but • Type 3:
H;_I
H;_I
is independent of x n _ l •
is dependent upon x n _ l .
Theorem 5.1 For Type 1 systems, the Kalman gain k n given in Eq. (5.56) converges to zero when the systems being dealt with are periodic, and converges to a non-zero fixed point when the systems are chaotic. Proof: As Fn~1 is independent of X n _ I ' Fn~1 can be regarded as a constant i;. Thus, Eq. (5.56) becomes 83
5.2
The Unscented Kalman Filter
k = n
qkn _ 1 def J:k +1
'='
do 'f/
(k
J:)
n-P':> ,
(5.57)
n-l
which includes two fixed points, i.e., (5.58) The stability of the two fixed points can be determined by the derivative of ¢ at the corresponding fixed points. If the derivative has a magnitude larger than one, the fixed point is unstable, and the Kalman gain cannot converge. Otherwise, the fixed point is stable. By evaluating the derivative ofEq. (5.57) at the two fixed points, we have (5.59) When q < 1, k n = 0 is a stable fixed point, and k n = (q -1) / q is an unstable fixed point. Conversely, when kn
=
(q -1) / q
q > l,kn
=0
is an unstable fixed point, and
is a stable fixed point.
The global Lyapunov exponent of the one-dimensional system can be written as
s = lim -N1 L In 1F
1
N
N->oo
n=1
n
1= -InC q). 2
(5.60)
Thus, when Eq. (5.25) is periodic, ¢'< 1, and kn converges to k(J) . Moreover, when Eq. (5.25) is chaotic, k n converges to
k(2) •
Theorem 5.2 For Type 2 systems, if H~_I = q
oF-
0 (q is a constant), the
Kalman gain kn in Eq. (5.56) converges to zero when the systems being dealt with are periodic and k n oscillates aperiodically when the systems are chaotic. Proof: From Eq. (5.56), we obtain
84
Chapter 5
Signal Reconstruction from a Filtering Viewpoint: Theory
(5.61)
IfEq. (5.25) is chaotic and the square of the gradient Fn2 is aperiodic, then 2 Fn~l + (114 )q(P - a ) is also aperiodic. Hence, the result follows according to Theeorem 2 in [251]. If we define en_le~_l
= p"-p then Hn_le n_l = Fn - Fn_l . By using Eq. (5.56),
we can obtain
(5.62)
2 Lemma 1: Suppose Gn = Fn2_1 + (1/4)(p-a )(Fn -Fn_I)2. Then, Gn is aperiodic. Proof: Assuming that Gn has a period of N, without loss of generality, we have
Gn+N -Gn = 0.
(5.63)
Define
ml,n
= F;+N_1 - Fn~l
2 2 m2.n = Fn +N - Fn I],n
= Fn+N
r2,n
= F'" - F',,-l
- Fn+N- 1
(5.64) (5.65) (5.66) (5.67)
and consider Fn~l being aperiodic. Then, when l],nFn+N - r2,nFn = 0, 2 Gn+N - Gn = m\,n + (114) (P - a ) (ml,n + m2,n - 2m 2,J is not always equal to 2 zero. Also, when 2 (l],nF',,+N-r2,nF',,) =In-:F-O,Gn+N-Gn =m\,n+(1I4)(p-a ) 85
5.2
The Unscented Kalman Filter
(mI,n + m2 ,n - 2m2 ,n + IJ is not always equal to zero. Therefore, the above
assumption about the periodicity of Gn is not valid. In other words, Gn is aperiodic. Theorem 5.3 For Type 3 systems, the Kalman gain k n in Eq. (5.56) converges to zero when the systems being dealt with are periodic, and k n oscillates aperiodically when the systems are chaotic. Proof: Since Gn is aperiodic, Type 3 systems can be arranged to Type 2 systems, and the result follows from Lemma 1.
5.2.3
Computer Simulations
To illustrate the theoretical results developed in the foregoing section, we consider three systems, i.e., the tent map and two logistic maps. 5.2.3.1
Type 1
We consider the following class oftent maps: Xn+I = a(1-12xn -11)
where a E [0, 1] and xn
E
(5.68)
[0, 1]. The global Lyapunov exponent and local
exponent of the system are identical, and equal to In 2a. The map is chaotic for a > .!.-, and is periodic otherwise. The UKF algorithm for this map can 2
be realized by first using Eq. (5.64) to get the Jacobian matrix, i.e., 2a,
1 x
{ -2a,
1 x >-. n 2
F= n
2
Then, we have Fn
=
4a
2 ,
which is independent of x n' According to
Theorem 1, the Kalman gain converges to
86
(5.69)
Chapter 5
Signal Reconstruction from a Filtering Viewpoint: Theory
limk = n-->oo
n
o,
1 a<2
t
1 a>-.
4a 2 -1 4a 2
'
(5.70)
2
Figure 5.1 shows the typical convergence behavior of the Kalman gain for this map, in which the dotted lines denote the steady states given by Eq.(5.70). We can see from Fig. 5.1 that the magnitude of the fixed point increases as the Lyapunov exponent gets larger when the system is chaotic, as given in Theorem 5.1. 1
0.9 0.8 0.7
IV
0.6
"'c"
'0:;
0.5
01)
c
"E ""OJ ~
"
a=0.8
0.4 0.3 0.2 1
l / a =0.3
0.1 0
\a=O.4
-0.1 0
10
20
30 Iteration
40
50
60
Figure 5.1 Convergence characteristic of the Kalman gain for tent map with different
values of a corresponding to periodic signals (a = 0.3 and a = 0.4) and chaotic signals (a = 0.8 and a = 0.9). Dotted lines denote the theoretical steady states
5.2.3.2
Type 2
We consider the logistic map given by (5.71)
87
5.2
The Unscented Kalman Filter
where
aE
[0,4] and
Xn E
[0, 1]. The Jacabian matrix of the map is
Fn =
a - 2axn , and the Hession matrx is - 2a. Thus, this map is a Type 2 system.
Figure 5.2 shows the convergence behavior of kn' in which the dotted, dashed, and solid lines describe the kn 's convergence behavior for the system with parameter a= 3.4, a= 3.84 and a= 4.0, respectively. The former two are periodic, while the latter gives a chaotic time-series. When the timeseries is periodic, k n converges to the fixed zero state. However, when the time series is chaotic, k n exhibits an aperiodic behavior, as described in Theorem 5.2. a=3.4 - -- a=3.84 a=4.0
0.9 0.8
1\
A
A
A
"'c"
~
1\
0.7
n
Ii
0.6
'0;;
bIJ
c 0.5 co
E
(ii
::.c 0.4 0.3 0.2 0.1
'i
. ,t" 1\
o o
j
l,' '\ 20
40
60
80
100
Iteration
Figure 5.2 Convergence characteristic of the Kalman gain of the logistic map
(Type 2). Dotted, dashed, and solid lines describes the k n 's convergence behavior for the system with parameter a=3.4 (periodic), a=3.84 (periodic), and a=4.0 (chaotic), respectively
5.2.3.3
Type 3
We consider another form oflogistic map, which is defined as (5.72) 88
Chapter 5
where a E [0, 1.45] and xn
Signal Reconstruction from a Filtering Viewpoint: Theory
1.5, 1.5]. The Jacobian matrix of the map is Fn = an cos (nXn) , and the Hession matrix is Hn = -an2 sin (nxJ, which E [ -
are dependent on x n • This map is therefore a Type 3 system. Figure 5.3 shows the convergence behavior of kn' in which the dotted, dashed, and solid lines correspond to the results of a=
a=
0.8,
a=
0.94 and
1.0, respectively. The former two are periodic, while the latter gives a
chaotic time series. When the time series is periodic, k n converges to the fixed zero state. However, when the time-series is chaotic, k n exhibits an aperiodic behavior, as described in Theorem 5.3.
0.9 0.8
- _. a=0.8 a=0.94 a=1.0 -
(' ( n ~
0.7 0.6 c
.~
0.5
B
E 0.4
" ::L
0.3 0.2 0.1
o ~--------------------------------
-0.05 L-_-'--_--'--_-'-_----'_ _'---_-'--_--'---_---'--_----'_ _ o 10 20 30 40 50 60 70 80 90 100
Iteration Figure 5.3 Convergence characteristic of the Kalman gain for the logistic map (Type 3). Dotted, dashed and solid lines describes k n 's behavior for the system with parameter a == 0.8 (periodic), a == 0.94 (periodic) and a == 1.0 (chaotic), respectively
5.3
Summary
In this chapter, we revisited the KF, EKF and UKF algorithms. In particular, the problem for filtering noisy signals arising from chaotic systems using the 89
5.3
Summary
UKF algorithm has been investigated. It has been shown that when a nonlinear system is chaotic, the Kalman gain of the UKF does not converge to zero, but either converges to a fixed point with magnitude larger than zero or oscillates aperiodically. The dynamical behavior of the error covariance matrix can be readily found since it is linearly related to the Kalman gain.
90
Chapter 6
6.1
Signal Reconstruction from a Filtering Viewpoint: Application
Introduction
As mentioned previously, separating a deterministic signal from noise or reducing noise for a noisy corrupted signal is a central problem in signal processing and communication. Conventional methods such as filtering make use of the differences between the spectra of the signal and noise to separate them, or to reduce noise. Most often the noise and the signal do not occupy distinct frequency bands, but the noise energy is distributed over a large frequency interval, while the signal energy is concentrated in a small frequency band. Therefore, applying a filter whose output retains only the signal frequency band reduces the noise considerably. When the signal and noise share the same frequency band, the conventional spectrum-based methods are no longer applicable. Indeed, chaotic signals in the time domain are neither periodic nor quasi-periodic, and are unpredictable in the long-term. This long-term unpredictability manifests itself in the frequency domain as a wide "noiselike" power spectrum [2, 151]. Conventional techniques used to process classic deterministic signals will not be applicable in this case. A number of noise reduction methods for noisy chaotic time series have been proposed [252 - 254]. These methods usually treat noise reduction for noisy delay-embedded scalar time series in a reconstruction space based on the embedding theory [1], or for observed noisy time series by using some filtering technique to obtain an optimal estimation of the original dynamical system [101, 255 - 257]. A popular technique for filtering noisy chaotic signals is to use an extended Kalman filter (EKF) algorithm, as described in the previous chapter. However, the side effects from using EKF for filtering
6.2
Filtering of Noisy Chaotic Signals
chaotic signals are lowered estimation precision due to the first-order approximation and higher complexity due to the need for computing the Jacabian matrix [178,258]. In this chapter, we describe a new method for filtering noisy chaotic signals and retrieving the original dynamics of a chaotic system.
6.2
Filtering of Noisy Chaotic Signals
6.2.1
Filtering Algorithm
Figure 6.1 shows a block diagram of the problem of filtering a noisy chaotic signal. Suppose the dynamics of the chaos generator can be described in the state space as (6.1) (6.2) where f: lR N ~ lR N is the nonlinear (chaotic) function, xn
E
lR N is the state
vector of the system at time instant n, xn is the output (scalar) signal of the chaos generator, Yn is the observed signal, and
Vn
is the measurement noise.
By using a filter, we can, from Y n , get the estimation of the source signal x n ' l.e., x n '
Figure 6.1 Block diagram of filtering a noisy chaotic signal
In estimating the state, the EKF algorithm is widely used to achieve a recursive maximum-likelihood estimation of the state xn [178,259]. The basic idea of the algorithm is summarized as follows. Given a noisy observation Y n , a recursive estimation for 92
xn
can be expressed as
Chapter 6
Signal Reconstruction from a Filtering Viewpoint: Application
(6.3) where k is the Kalman gain vector. The recursion provides the optimal minimal mean squared error (MSE) estimation for xn if the prior estimation
xn
and current observation Yn, are Gaussian random variables. In the
algorithm, each state distribution is approximated by a Gaussian random variable, which is propagated analytically through the first-order linearized model of the nonlinear system. This approximation, however, can produce large errors in the true posterior mean and covariance of the transformed random variables, which may even lead to the divergence of the algorithm [ 178]. Recently, the UKF has been proposed for solving the filtering problem [248,249], [260 - 262]. When the parameter of Eq. (6.1) varies with time, Eq. (6.1) can be rewritten as (6.4) where () is the parameter, which can be modeled by using the autoregressive (AR) model as [178] p
()n = L>J()n-J
+.9n
(6.5)
J~1
where a.) is the coeffcient of the AR model, p is the order of the AR model, and .9n is additive white Gaussian noise. The problem is now changed into the estimation of the mixed state and parameter in the state space. Essentially, the augmented state-space vector is [xn ()n ()~2) ... ()~p) an a~2) ... a~p)
r.The UKF algorithm can then be applied to this augmented state-
space model. The general algorithm for estimating the system can be summarized in two main stages.
Prediction Stage The purpose of the prediction stage is to calculate the following:
93
6.2
Filtering of Noisy Chaotic Signals
Xnln-1 =
Xnln-l
(6.6)
1(1'.) H
= "W(m)X
L....;
(6.7)
;,nln-l
i=O
H
Px,nln-l = "L.... w;(c) (X;,nln-l
')(
- X n1n - 1
')T
(6.8)
X;,nln-l - X n1n- 1
i=O
where
X nln-l
is the estimation for
Xn
at time instant n,
xn1n -
1
is the
estimation for x's at instant n, and Px ,nln-l is the estimation for Px at instant n.
Correction Stage In the correction stage, we perform the following calculations: 1 k=P-'Y py
(6.9) (6.10)
P=P n nln-l -kPk n y n
(6.11)
where P-'Y is the covariance matrix ofx andy, Py and
Ynln-l
are the variance
and the estimation ofthe observed signal. It should be noted that it is unnecessary to perform explicit computation
of the Jacobian matrix (partial derivatives) in this algorithm. In addition, the complexity of the UKF is, in general, O( A3 ) , where A is the dimension of the state [178, 248, 249].
6.2.2
Computer Simulation
In performing simulation, the output signal
Xn
of the chaotic generator is
controlled to reach the required signal-to-noise ratio (SNR) in a A WGN channel. For the retrieved signal Xn , the MSE is used to evaluate the algorithm's performance. The MSE is defined as:
94
Chapter 6
Signal Reconstruction from a Filtering Viewpoint: Application
(6.12) where Mis the number of the retrieved sample points. Three chaos generators, namely, logistic map, HEmon-map and Chua's circuit, will be used to demonstrate the performance of the algorithm in this chapter. The logistic map is described by the following equation (6.l3)
where the parameter
Ii (Ii E
(0,4]) is fixed at 3.8, which ensures the chaotic
motion ofthe system. Figure 6.2 shows the retrieved return map ofEq. (6.13) from xn with an SNR of 30 dB. Figure 6.3 shows the MSE of the retrieved signal versus SNR, in which the solid line and the dotted line correspond to the results by using UKF and EKF, respectively. It can be seen from Fig. 6.3 that the MSE by usingUKF is approximately - 45 dB when SNR is 30 dB, which slightly outperforms the EKF algorithm.
0.8 0.6 0.4 0.2 ~
.j.
I·
0
I
-0.2 -0.4 -0.6 -0.8 -1
I
-1
-0.5
0 Xn-l
Figure 6.2 Return map of the logistic map retrieved from noisy data (SNR= 30 dB)
by using the UKF algorithm
95
6.2
Filtering of Noisy Chaotic Signals
-10
-20
I--EKFI -UKF
,
-30 -40
EO -50
::s,
~ -60
2
-70 -80 -90 -100
10
20
30
40
50
60
70
80
SNR (dB)
Figure 6.3 MSE of the retrieved signal versus SNR for the logistic map (solid line
for UKF and dotted line for EKF)
When c varies with time, e.g., (6.14) we should model the system by using the second AR model as described previously and expand the system into the augmented state-space model. Then, the UKF algorithm can be used to estimate the states of the augmented model. Figure 6.4 shows the MSE of the retrieved signal xn versus SNR, in which the solid line and the dotted line represent the results by using UKF and EKF, respectively. Figure 6.5 shows the MSE of the retrieved parameter
cn versus SNR. We can see from Figs. 6.4 and 6.5 that the two filtering methods have the same performance when SNR is high, and the UKF outperforms the EKF when SNR is low. Next we examine a chaotic modulation system as described in [101], in which the logistic map Eq. (6.13) is again used as the chaos generator, and the value of each pixel point of the standard Lenna portrait (256 x 256 pixels) shown in Fig. 6.6 is used as the modulating signal. The modulation signal is
96
Chapter 6 Signal Reconstruction from a Filtering Viewpoint: Application
-10 -20
1--
EKFI -UKF
,
-30 -40
ill
~-50 w
ifJ
~
-60 -70 -80 -90 10
20
30
40
50
60
70
80
SNR (dB)
Figure 6.4 MSE of the retrieved signal xn versus SNR for the logistic map when li varies with time according to Eq. (6.14) (solid line for UKF and dotted line for EKF) 10 5 0
ill -5 ~
~-1O
2 -15 -20 -25
UKF
10
20
30
40
50
60
70
80
SNR (dB)
Figure 6.5 MSE of the retrieved parameter li versus SNR for the logistic map when li varies with time according to Eq. (6.14)
injected into the system according to the following equation: lin =
3.57 + ¢(i,j)
(6.15)
where Ifi(i, j) is the pixel value of point (i, j). The quality of the retrieved portrait is evaluated by peak signa1-to-noise ratio (PSNR), which is defined as 97
6.2
Filtering of Noisy Chaotic Signals
PSNR(dB) = 1010glO(2552 I E)
(6.16)
where 256
E=
L
[r/J(i,j)-¢(i,j)]2/256
2
(6.17)
i,j=1,.··,256
and r/J(i,j) is the estimated value of r/J(i,j). When the SNR ofthe received signal xn is 50 dB, the PSNR of the retrieved portrait (shown in Fig. 6.7) using the UKF algorithm is 35.1 dB, which shows a 7 dB improvement in performance compared to that using the EKF algorithm.
Figure 6.6 The standard Lenna portrait with 256 x 256 pixels
Figure 6.7 The retrieved Lenna portrait
98
Chapter 6
Signal Reconstruction from a Filtering Viewpoint: Application
The second system we study here is the Henon-map, which is described by the following 2-dim iterative map: (6.18) (6.19) where a l is a time-varying bifurcation parameter (al
E
(1.32, 1.42]) whose
variation ensures chaotic motion of the system, and a 2 is fixed at 0.3. When
al
=
1.4, we evaluate the performance of the two algorithms. Figure 6.8
shows the MSE of the retrieved signal xn versus SNR, in which the solid line and the dotted line represent the results using UKF and EKF, respectively. We can see from Fig. 6.8 that the performance of UKF outperforms EKF, and the MSE of the retrieved signal using UKF is reduced by 17 dB when SNR is equal to 30 dB. 0 -10
1-EKF 1 - UKF
-
-20 -30
~ -40 W
~ -50 -60 -70 -80 -90
10
20
40
30
50
60
70
80
SNR(d8)
Figure 6.8 MSE of the retrieved signal xn versus SNR for the Henon-map (solid
line for UKF and dotted line for EKF)
The third chaos generator to be used is the Chua's circuit, which is described by the following dimensionless equations [263]: x=
a 3(y - K(X))
(6.20) 99
6.2
Filtering of Noisy Chaotic Signals
(6.21)
y=x-y+z
(6.22) where a 3 is fixed at 9, and a 4 at -10017. This choice of parameters ensures the chaotic motion of the system. Also, KO is a piecewise linear function which is given by _{mjx+(mo-m j ), x;?l K(X)- mox, Ixl
(6.23)
mjx-(mo-mj), x:(-l
where mo =-ll7andmj =217. Equations (6.20) to (6.22) are first discretized by using the Euclidean method, and z is then selected as the output signal. Figure 6.9 shows the MSE of the retrieved signal z versus SNR, in which the solid line and the dotted line correspond to the results using UKF and EKF, respectively. We can see from Fig. 6.9 that the performance of UKF outperforms EKF in the case of low SNR, and the MSE of the retrieved signal using UKF is reduced by 5 dB when SNR the is equal to 30 dB. 20'---~----~--~----~----~--~----~
o '
1--EKF I -UKF ,
,
-20
,
, ,
,
-60
-80
-100~--~----~---7,~--~--~~--~~~~
10
20
30
40 50 SNR(dB)
60
70
80
Figure 6.9 MSE of the retrieved signal z versus SNR for the Chua's circuit (solid
line for UKF and dotted line for EKF)
100
Chapter 6
6.3
Signal Reconstrnction from a Filtering Viewpoint: Application
Blind Equalization for Fading Channels
A simplified chaos-based communication system is shown in Fig. 6.10, where xn is the output signal of the chaotic modulator at time instant n, h is the transfer function of the non-ideal channel, signal of the receiver, and
xn
Wn
is AWGN, Y n is the input
is the estimated source signal. In the following,
we will focus our attention on the fading channels with fixed or time-varying channel coefficients.
Transmitter
Channel
receiver
Figure 6.10 A simplified block diagram of a chaos-based communication system
6.3.1
Modeling of Wireless Communication Channels
A communication channel refers to the entire medium between a sending end and a receiving end, and it is an inevitable component of any communication system. Physical channels can be classified into cable channels and wireless channels according to the type of transmission medium. A cable channel is stationary and predictable. However, a wireless channel is generally random and is hard to analyze [264, 265]. A wireless channel realizes information transmission based on the spatial transmission of electromagnetic radiation, and thus the transmission is open-ended. On the other hand, the geographical environment of the receiving end can be very complicated and diversified, and the users at the receiving end may take on random mobility. Under practical communication scenarios, there exist the surface waves' propagation of middle and long waves, the ionosphere's emission propagation of the short wave, the direct propagation of the ultra-short wave and microwave, and various diffraction propagation and scattering propagation, etc. These result in the performance degradation and losses for the receiving signals. 101
6.3
Blind Equalization for Fading Channels
Degradation • Shadow effect: This refers to the presence of semi-blind scope in the signal's receiving area, which is generally caused by blocking of transmission signal by large obstacles, such as large buildings. • Near-far effect: Due to the mobility of the receiver, the distance between the receiver and the base station varies randomly. Signals with the same transmission power arrive at the base station with different intensities. Typically, the receiving signal is stronger if its transmission signal is near a base station and vice versa. This manifests as a non-linear effect ofthe communication system. • Doppler effect: When the user is moving at a high speed, frequency diffusion occurs causing degrading in the performance of the communication. Doppler effect is generally related to the relative speed of the moving user to the frequency of the carrier. Losses • Propagation loss: Loss is inevitable as signals travel through spatial propagation path. Usually such loss becomes significant when the distance travelled is relatively long, in the order of kilometers. • Slow fading loss: This refers to the loss caused by the shadow effect resulting from buildings or mountains blocking the transmission path. The nature of such loss is random and Gaussian distributed. • Fast fading loss: Under a multipath propagation environment, which is usually modeled by Rayleigh distribution or Rician distribution, fast fading loss occurs as the rate of change of the multipath environment can be relatively high. Three types of fast fading loss can be identified, namely, space selective fading, frequency selective fading and time selective fading. They correspond to different fading features in space, frequency and time. In a real communication environment, there exists all three types of the selective fadings caused by multipath propagation. Generally, a fast fading channel can be modeled as: L
h(xn) = ~»n-i + Wn
(6.24)
i=l
where L is the number of channel paths, Wn is AWGN, 102
a! (i = 1,2,' ", L)
Chapter 6
Signal Reconstruction from a Filtering Viewpoint: Application
is the time-varying channel coefficients. We denote an = [a~, a~,· .. , a~], which stands for the vector of channel coefficients at time instant n [266]. Using the AR model, we can express the time-varying channel coefficient dn as: i = 1, 2, ... , L
(6.25)
where pi is the order of the AR model, C~~l is the corresponding coefficient and is AWGN. As mentioned previously, the task of an equalizer is to restore the transmitted source signal. Generally, in the state space, we can formulize the
v:
problem of blind equalization for a wireless communication system (as shown in Fig. 6.10) with a fast fading channel in terms of a set of state and measurement equations. The state equation is the same as Eq. (6.1), and the measurement equation is (6.26) where h represents the channel's transfer function. Thus, the channel equalization problem can be translated to an estimation problem for an extended state space model, in which each time-varying channel coefficient can be modeled by using the AR model.
6.3.2
Blind Equalization of Fading Channels with Fixed Channel Coefficients
In general, the shortest distance between a transmitter and its receiver is the major signal propagation path, and all other paths carry weaker signals. Therefore, it is reasonable for us to assume that a~ = 1. Since all channel coefficients are constant, the AR model can be ignored. We again resort to the MSE defined in Eq. (6.12) for evaluating the performance of an equalizer. Figure 6.11 shows the equalization performance for a channel with an = [1, 0.45, - 0.22] at SNR = 10 dB, in which the dashed lines stand for the estimated values and the horizontal axis is the number of iteration. 103
6.3
Blind Equalization for Fading Channels
Figures 6.11(a) and (b) show the results using the EKF and UKF algorithms, respectively. As shown in the figures, the UKF-based equalizer converges faster than the EKF-based equalizer, and the UKF-based equalizer also shows an improved MSE over the EKF-based equalizer, which is about 9.5 dB. Hence, the equalization method based on UKF outperforms that based on EKF.
0.8 ~ v
0.6
'(3
~.
1.. ''Jl'r ____ _
~ 0.4 o
u
"0
~
0.2
E
.~
w
o
1
I\~I
\
-
---~-~--------------
_____ _
-0.2[\-',....;-,_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _-----.:j II,'
!
-0.4'-------'-----'-------'-----'-------.J o 200 400 600 800 1000
n
(a)
II. ,
/\,~ ..
v'"
/---
0.8 '" en
C v
0.6
'()
fEv 0.4 0
U
"0
v
"'
.§ ~
0.2 0'.,
I,:
-0.2 -O.4,L-_ _.,...,...,,_ _ _-'--_ _--'-_ _---:--'--_ _---1 o 200 400 600 800 1000 n (b)
Figure 6.11 Blind equalization of a channel with the channel coefficients being an = [\,0.45, - 0.22] when SNR = 10 dB. (a) EKF-based equalizer is used; (b) UKF-
based equalizer is used
104
Chapter 6
Signal Reconstruction from a Filtering Viewpoint: Application
We further consider four other channel models with fixed coefficients [267, 268], i.e., Channel I :
an=[l, 0.7, -0.3,0.5, -0.1],
(6.27)
Channel II:
an = [1, - 0.5,0.7,0.2, - 0.3],
(6.28)
Channel III:
an=[l, 0.24, 0.32, -0.52, -0.12],
(6.29)
Channel N:
a n = [1,0.64, - 0.52, - 0.48, - 0.24].
(6.30)
Using the UKF-based equalizer, the equalization performances for the four channels are shown, respectively, in Fig. 6.12 (a), Fig. 6.12 (b), Fig. 6.12 (c) and Fig. 6.12 (d), in which the SNR of each channel is controlled to 15 dB, and the dashed line stands for the estimated value. We can observe from ].4~---~-~--~--.
1.2
~
1~"~'~------~---------~
" S "o
'(3
0.8",.-- .. "" ... 0.6t.-______~__________~
0.4 ..... -. ..... ".
()
"d
~E 0.2 ·tl 0 ~
..
~~-~.~.~,,~.~ .. ~--~--------~
-0.2 .'
t
-O.4o==2=0::'0==4=0=0==6=00==8=00==]ojoo n (a) Channell 1.2~------~----------
1.2~--------------------
1',--· '-
I~---------------------
°0
0.8 k--------------------~ 0.6
8
0.2~"c..·---~~------------
"E
~ 0.4
1l 0 ,8 -0.21,c.~__~_______________
"
~ 0.8
G
O,6f-,,,~··~-~~-----------j
S" 0.4 8 0.2~--------------.-j ]
0
.8" -0,2,,
Jl-O.4k__~--~------------- Jl-O.4',·." '- ... -0,6 ,0' -0,6 ' 200
400
n
600
(b) Channel 11
800
1000
-0.8 0
200
400
n
600
800
] 000
(d) Channel IV
Figure 6.12 Equalization performance for the four channels (Eqs, (6.27 - 6.30)) by using UKF-based algorithm, in which SNR of each channel is controlled to 15 dB
105
6.3
Blind Equalization for Fading Channels
Fig. 6.12 that the equalization is realized after about 300 iterations. Also, for comparison, we realize the same equalization task for the four channels using the EKF-based equalizer, and the results are shown in Fig. 6.13 (a), Fig. 6.13 (b), Fig. 6.13 (c) and Fig. 6.13 (d). 1.5
1.2~---~----~------'
1 ". ~
en
~ O'S~~~_ _ _ _ _ _ _ _-----j 'u" ·u \~ .. ---S ~8 0.6"-"",--, --~~--~---C~~ 0.4 . - --- -8 '0 0)
'0
I~;~'-----~~~~~~
0.5
~~--~.~--=-========~======~
0 ",
~ 0.2
.~
~
Jl-0.5~:~~~-~~~~~=='-'I
.~
0)
O~
~~~.~~-~--~~.~-=--~--~--~--=--=-~--
~--=--=--~-~--~-~~~-~----I
-0.2111:'1""";~"~~~~~~~~--~-'::":--'::":--~-'::":--'::":--=-=i'
-O.4L'_~_ _~_~_ _~_..J o 200 400 n 600 SOO 1000 (a)
__ 400 600
-1L--~_~
o
200
~_~_~
n
SOO
1000
(e)
1.2[, I~"~----.------------------------------j ~
~
O.S
.~
0.6l-..~,~--~--=--~-=--=--~-~·~--=-~.=-=--=--~-~.~--~--=-cccd
0.4 ~)._
~
0.4
0
]
.~
O.S '. 0.6
] C)
. _ ___ __ __ __
0.2~~~~------~---i
'2
8
0.2F...~~----~-----l
0
'E ro I S - 0 . 2 § -0.2 , ~ -0.4 ,T Jl-O.4 '.. , ·----cc::: -0.6 '",,-'--- -0.6 200
400
n
(b)
600
SOO
1000
-O.SO
200
_:0_ ----Co: --
400
n
600
SOO
1000
(d)
Figure 6.13 Equalization perfonnance for the four channels (Eqs. (6.27 - 6.30))
using EKF-based equalizer, in which SNR of each channel is controlled to 15 dB
6.3.3
Blind Equalization for Time-Varying Fading Channels
Finally, we consider the blind equalization of time-varying fading channels. For these channels, we have to model each time-varying channel coefficient using the AR model. As a compromise between computational complexity and tracking capability, we use the zero-order AR model to model these coefficients. The following two channels will be used to demonstrate the 106
Chapter 6
Signal Reconstruction from a Filtering Viewpoint: Application
1.2,---~--~---~--~------,
en
E
0.8
'"
0.6
0
0.4
!+= 4-
(j
"0
%i 0.2 2
.~
U.l
0, -0.2 200
400
n
600
800
1000
800
1000
(a) Channel V
1.2
en
0.8
E
'"tE'"
'"0
(j
0.6 0.4
"0
~
2 0.2
.~
U.l
0', -0.2 600 n (b) Channel VI
Figure 6.14 Equalization perfonnance for the two time-varying fading channels
(Eqs. (6.31) and (6.32)) using UKF-based equalizer, in which the SNR of each channel is controlled to 30 dB, and the dotted line corresponds to the estimation
107
6.3
Blind Equalization for Fading Channels
Figure 6.15 Equalization performance for the two time-varying fading channels (Eqs. (6.31) and (6.32)) using EKF-based equalizer, in which SNR of each channel is controlled to 30 dB, and the dotted line corresponds to the estimation
108
Chapter 6
Signal Reconstruction from a Filtering Viewpoint: Application
equalization performance of the UKF-based equalization algorithm [268]: Channel V:
an = [1,0.7(1 + 0.1 cos(n /1 00)), - 0.3(1 + O.lsin(n / 50))],
(6.31 )
Channel VI:
an = [1 + 0.05 cos(n 1100),0.7(1 + O.lcos(n 11 00)), - 0.3(1 + 0.1 sin(n / 50))]. (6.32) The second model diffiers from the first one only in the main path. Figures 6.14 and 6.15 show the equalization performance for the two time-varying channels using the UKF-based and the EKF-based equalizers, respectively. In the simulations, the SNR of each channel is controlled to 30 dB. We can see that both equalization algorithms are able to track the time-varying fading channels. Though some local divergence occurs in EKF-based equalizer, the timevarying channels can eventually be tracked. Moreover, the UKF-based blind equalizer can accurately track the time-varying channels and no local divergence has been observed.
6.4
Summary
Application of the original UKF algorithm has been extended to filtering signals arising from time-varying chaotic systems. It has been shown by computer simulation that this UKF-based algorithm can reduce noise in chaotic signals even if the parameters of the chaos generators are varying with time. In comparison with the EKF algorithm, this algorithm has a better filtering performance in the case of low SNR, and has a similar performance in the case of high SNR. In addressing the problem of blind equalization for practical communication channels, we apply the UKF algorithm and have shown by computer simulation that this algorithm can help combat various forms of distortions and reduce noise in a communication channel. In comparison with the EKF-based algorithm, the UKF-based algorithm has faster convergence speed and better equalization performance under low SNR conditions. 109
Chapter 7 Signal Reconstruction in Noisy Channels
As discussed in Chapter 2, a message signal in a chaotic modulation communication system is injected into the transmitter as a "bifurcation parameter", with the range of variation of the bifurcation parameter chosen to guarantee motion in a chaotic region. The main advantage of the chaotic modulation scheme is that it does not require any code synchronization, which is necessary in traditional spread-spectrum communication systems employing coherent dispread and demodulation techniques.
7.1
Review of Chaotic Modulation
Recently, a number of schemes for chaotic modulation communication systems have been proposed. Corron and Hahs [269] proposed a drivingresponse (master-slave) based chaotic modulation communication system for noisefree channels. The demodulation task was realized by a nonlinear filter. Another drive-response based demodulator that can operate under a noise-free channel was also suggested by Sharma and Poonacha [270]. This system estimates the transmitter's parameters via minimization of a cost function through a gradient search algorithm. Anishchenko and Pavlov [271] proposed a global reconstruction approach for extracting bifurcation parameters of a class of chaotic modulation communication systems in the absence of channel noise. This modulation-demodulation strategy is effective and reliable if, and only if the motion equation of the transmitter can be rewritten in the following form: (7.1)
Chapter 7
where x = [Xl
X 2 ••• XE
Signal Reconstruction in Noisy Channels
r is the vector of equivalent state variables of a chaotic
system, and r is the parameter vector of the system. When deriving Eq. (7.1) from the original set of state variables of the transmitter, one usually encounters singularity due to the existence of a zero denominator [271]. Such singularity may lead to an undesired high frequency component in the retrieved message signal. It has been demonstrated in [272] that lowdimensional chaotic systems are structurally identifiable via an optimization technique based on minimizing a chosen cost function, e.g., the secondorder quasi-Newton algorithm. In the identification process, however, one has to design an effective adaptive algorithm in order to avoid being trapped in local minima of the cost function during the optimal process. In Muller and Elmirghani [273], an artificial-neural-network-based chaotic transmission strategy for the one-dimensional logistic map was proposed. This approach employs an REF neural network with a fixed number of hidden units to approximate spread-spectrum signals. However, in reconstructing a chaotic system, there is no systematic method to select suitable number of hidden layer units, especially for high-dimensional chaotic systems [256,274]. Thus, the design of an appropriate REF network for a general class of signals can be a complicated task. Because of their ability in modeling any arbitrary nonlinear real-valued map defined on compact real sets [193], REF neural networks have been employed for the identification of nonlinear dynamical systems [275 - 278]. In this chapter, we specifically describe an on-line adaptive demodulator for a chaotic modulation communication system based on an radial-basisfunction (REF) network. This demodulator can adaptively retrieve message signals from receiving spread-spectrum signals which are contaminated by channel noise. Two assumptions are made in our design. Firstly, the transmitter's parameters vary slowly with time so that the modulation system in the transmitter can be seen as an autonomous system in a time interval. Secondly, the communication channel is distorted by additive white Gaussian noise (AWGN). This chapter is organized as follows. In Sec. 7.2, a Henon-map 111
7.2
Formulation of Chaotic Modulation and Demodulation
based demodulation process is formulated. Using an REF network with an adaptive learning algorithm, a new demodulator for extracting message signals from receiving signals is described in Sec. 7.3. Simulation results are presented in Sec. 7.4 for verification. Finally, in Sec. 7.5, as an application example of this tracking algorithm, we consider the message retrieval of a chaotic-shift-keying digital communication system by using non-coherent detection technique.
7.2
Formulation of Chaotic Modulation and Demodulation
Let us rewrite the HEmon map as Xl
(k + 1) = 1- ax; (k) + X z(k),}
X z (k
+ 1) = bXl (k),
(7.2)
where a and b are the bifurcation parameters CD • In this case, we fix b at 0.3 [101]. The broadband feature of this map can be verified by inspecting the autocorrelation function and power spectrum of xJ,when a is fixed or
time-varying. If a(k) is time-varying, we choose its range to ensure chaotic motion of the system. Figure 7.1 shows the autocorrelation function and power spectrum of Xl (k) when a is fixed at 1.4. Also, Fig. 7.2 shows the autocorrelation function and power spectrum of xl(k)when a is varied according to a(k) = 1.37 + 0.05 sin (k/5),
for
k= 1, 2, 3""
(7.3)
We clearly see that Xl is a highly uncorrelated signal and occupies a broad spectrum, for both cases of fixed and time-varying a. This property is desirable for spread-spectrum communication, and hence we may choose Xl as a transmission signal with a(k) being the message signal.
CD Bifurcation parameters determine the dynamical behavior of a dynamical system. range of the parameter values, the system can demonstrate chaotic behavior [22].
112
For some selected
Chapter 7
Signal Reconstruction in Noisy Channels
5,500~---------------~
._3 u
"
~
.~ ] o
5,000 4,500 4,000 3,500 3,000 2,500 2,000 1,500 1,000 500
a=l.4
b=0.3
B -500Or-~~"""""""""""""~ ::l
-< -1,000
-1,500 -2,000 -2,500 "--:-~_ _-L-_ _ _--"---_ _ _-,---_ _ _~ -10,000 - 5,000 o 5,000 10,000 Time delay (a)
5o,-------------------------______~ a(I1)=I.4
b=0.3
30
25L_73----~2----~1--~0~-~---2L--~3
Nonnalized angular frequency (b)
Figure 7.1 (a) Auto-correlation function; and (b) power spectrum of Xl for the HEmon map with a = 1.4 and b = 0.3
In general, when the spread-spectrum signal Xl passes through a practical channel with A WGN, the receiving signal y(k) is y(k) = Xl (k) + lJ(k),
(7.4)
where lJ(k) is A WGN. One key function of the receiver is to estimate or track Xl from the receiving signal y(k). It has been demonstrated [279] that for 113
7.2
Formulation of Chaotic Modulation and Demodulation
S,SOO S,OOO 4,SOO 4,000 ._S 3,SOO 3,000 ~ 2,SOO '" 2,000 .~ I,SOO ] 1,000 o soo
a(n)= 1.3 7+0.0S * sin(n/S) b=0.3
'" -SOO0~"""~"~"""~"""""--1 «-1,000 - 1,SOO -2,000
E
-2,Soo'=---------'----------'----------'---~
-10,000
-S,OOO
0 Time delay (a)
S,OOO
10,000
SOr--------------------------------, a(n)= 1.37+0.0S*sin(n/S) b=O.3
4S
30
2Su_3----~2----~I---L0------'-----2L--~3
Normalized angular frequency (b)
Figure 7.2 (a) Auto-correlation function; and (b) power spectrum of Xl for the Henon map with a(k) = 1.37 + 0.05 sin (k/5) and b = 0.3
a low dimensional noisy chaotic attractor, noise can be removed or reduced by projecting the chaotic artractor onto a higher dimensional subspace. Such a subspace corresponds to the input space of an REF network, and according to the Takens' embedding theory [1], the dimension of this subspace is as low as 5 for the two-dimensional Henon attractor. 114
Chapter 7
Signal Reconstruction in Noisy Channels
Specifically, in our approach, x,(k) will be estimated from previously observed data y(k-l), y(k-2), ... , y(k-M), where M=5. For brevity, we define
z(1) =
Y(~~l)1 :
y(2(M + 1) -1)
y(1)
zen) =
y(n(M +1)-1) y(n(M + 1) - 2)
.
y(2(M + 1) - 2)
I
,z(2) =
y(M +2)
,
y(n(M + 1) - M)
where z (-) also stands for [z,
Z2 ... ZM (
Note that z (n)and y (n (M + 1))
together form one complete observation. To avoid confusion, we define an observation step as the duration for one complete observation, i.e., the time for reading (M + 1) data points. The problem is effectively reduced to a one-step-ahead prediction which can be formulated as
Xl (n(M + 1)) = h(z(n)),
(7.5)
where Xl (n(M+ 1)) is the estimate for x,(n(M+ 1)) and h (-) is a nonlinear function that can be realized by an RBF neural network with an adaptive learning algorithm, as will be described in the next section. After x, is tracked, the second equation in Eq. (7.2) can be used to estimate X2. We will call (Xl' X2 ) an estimate point pair, which is available in every observation step, i.e., (M + 1) time steps. To estimate a, we will make use of the first equation of Eq. (7.2), which can be re-arranged as a(k)g(k) -1 =
x (k) - Xl (k + 1). 2
(7.6)
If a (k) is a constant within a window of T, observation steps (i.e., T,(M + 1) time steps), then the REmon-map can be seen as an autonomous system in 115
7.3
On-Line Adaptive Learning Algorithm and Demodulation
the window, and
a
Specifically, to find
can be estimated by a least-square-fit approach.
a,
we use the following formula which requires L
samples of (Xl' X2 ), at intervals of Tz observation steps:
a=
{t[ x~(nI;(M x
+1))-X12 ]
[x2(nI;(M +1))-xl(nI;(M +1)+1)-X2 +XlJ}
It[ x~(nI;(M
+ 1))
-x~
J'
(7.7)
where T j > LTz, and x~ ,x2 and Xl are respectively the mean of estimated values x~(nTz(M+1)), x 2 (nTz (M+1)) and Xl (nT2(M+1)+1),for n=1, 2, ... , L. Thus, in LTz observation steps, we will make available one estimate of a which is given by Eq. (7.7). A block diagram representation of this chaotic modulation communication system is shown in Fig. 7.3. r----------------. I
:
1----','"' '--------.-------'
y( n)
:
RBF-based adaptive demodulator
'
: u(n)
,L______________ -.!' Transmitter
T/(n)
Receiver
Figure 7.3 Block diagram representation of the chaotic modulation communication system
7.3
7.3.1
On-Line Adaptive Learning Algorithm and Demodulation Description ofthe Network
The REF network is a three-layer neural network [275,179], comprising an input layer, a hidden layer and an output layer, as shown in Fig. 7.4. The input layer consists of M units, connecting the input vector, for example z (n) 116
Chapter 7
Signal Reconstruction in Noisy Channels
in our application. The ith input unit is directly connected to the output unit through a gain factor
Ci,
and the ith hidden layer is connected to the output
unit through a weight factor
Wi.
Effectively, the network performs a
nonlinear mapping from the input space ffi. M to the output space ffi., which is described by h(z(n)) =
Input layer
Wo
+
M
N
;:::1
i=1
L>;z; + L w;91Jz(n))
Hidden layer
(7.8)
Output layer
Figure 7.4 RBF network configuration
where
Wo
is the bias term. The function 91;: ffi. M ~ ffi. is called the activation
function and is given generally by 91;(z) = 91(11 z - Q 11),
where
QiE ffi.M
(7.9)
is known as the REF center, and 11·11 denotes a distance
measurement. The Euclidean distance is adopted in this book. Moreover, it has been shown [276] that the choice of the nonlinear function 91 is not critical to the performance of the REF network. Typical choices include (see Chapter 3), for instance, the thin plate spline function x2log(x), the Gaussian function exp (- x2/2(i), the multi-quadric function (x2 +
li)1I2
and the
2
inverse multi-quadric function (x + (i) -1 / 2 • All these activation functions have proven to have good approximation capability regardless of their asymptotic properties [179, 276]. In the following, we will use the Gaussian function 117
7.3
On-Line Adaptive Learning Algorithm and Demodulation
_
rpj (z(n)) - exp
where
(Ji
2 ) 2 '
(1Iz(n)-Q(n)11
(7.10)
2CJj
is the width of the Gaussian activation function of the ith hidden
unit. By putting Eq. (7.10) in Eq. (7.8), we have (7.11)
7.3.2
Network Growth
The network begins with no hidden layer unit. As signal y is received, the network grows by creating new hidden units and connecting the received data to the new hidden units. Precisely, given an observation {zen), y(n(M+ 1))}, the criteria for creating a new hidden unit are
(7.12) e(n) = y(n(M + 1)) - h(z(n)) > '72'
enrms
=
(7.13)
L ;~n-1]+1 [y(i(M + 1)) - Xl (i(M + 1))]2 > '7
=---"-'--------------
T;
3'
(7.14)
where Qnr is the center of the hidden unit which is nearest to zen), T3 is the number of observation steps of a sliding data window covering a number of latest observations for computing the output error, and '7), '72 and '73 are thresholds. Specifically, '71
=
max('7max
pk,
'7rnin) , where
p is
a decaying
factor, Y/max and '7rnin are the maximum and minimum of '71, and k is a counter representing the number of the observation pairs input to the network. The first criterion essentially requires that the input be far away from stored patterns, the second criterion requires that the error signal be significant, and the third criterion specifies that within the sliding data window of T3 observation steps, the root-mean-square (RMS) error is also
118
Chapter 7
Signal Reconstruction in Noisy Channels
significant. Now suppose the (N + 1) th hidden unit is to be added to the network. The parameters associated with this new unit are assigned as follows: WN+I = 8(n),
(7.15)
QN+I = z(n),
(7.16)
0"N+I =
P
II z(n) -
Qnr II,
(7.17)
where P (p< 1) is an overlap factor which controls the extent of overlap of the responses of the hidden units for an input.
7.3.3
Network Update with Extended Kalman Filter
When the observation (z(n), y(n(M + 1») does not satisfy the criteria Eq. (7.12) to Eq. (7.14), no hidden unit will be added, and the extended Kalman filter (EKF) is then used to adjust the parameters of the networkCD. These parameters define the state vector, v, of the network, (7.18) Thus, we can write the gradient vector of h (-) with respect to v as B(z(n» = 8hO 8v WI
T
= zpz2,",zM,I,IPI(z(n»,IP1(z(n»-2 (z(n)-Q1) , [
IP1 (z(n»
0"1
w~ II (z(n) - Q1 11 2 ,
•• ,
IPN (z(n»,
0"1 WN
IPN (z(n»-2 (z(n) - QN
)T
'
O"N
IPN(z(n»
W~ II (z(n)-QN w].
(7.19)
O"N
CD It is reported that in the case of sequential learning, a standard RBF neural network using EKF will be more compact than without using EKF [193].
119
7.3
On-Line Adaptive Learning Algorithm and Demodulation
Now, denote the corrected error covariance matrix of v at instant (n -1) by
P (n -1), n -1). Then, the current estimate of the error covariance matrix can be found from the following relation: P(n,n -1) = IP(n -1,n -1)IT = pen -1,n -1),
(7.20)
where 1 is an identity matrix. Other parameters used in the EKF algorithm are the variance R(n) of the observed signal and the Kalman gain vect0r K(n), whose propagation equations at time instant n satisfY with R(n) = B(z(n))P(n,n -1)BT (z(n)) + RD
(7.21)
K(n) = P(n,n _1)BT (z(n))/ R(n),
(7.22)
where RD is the variance of the measured noise. Having computed K(n), we can then update the state vector according to v(n) = v(n -1) + K(n)&(n),
(7.23)
where v(n) and v(n -1) are respectively the state vector of the present and previous observation step. Finally, the error covariance matrix is corrected according to P(n,n) = [I - K(n)B(z(n))]P(n,n -1) + rl,
where
(7.24)
r is a small scaling factor introduced to improve the RBF network's
adaptability to future input observations in the case of very rapid convergence of the EKF algorithm [193]. Finally, it is worth noting that when a new unit is added to the hidden layer, the dimension of pen, n) changes, as can be seen from the following relation. _ [pen -1, n -1) P(n,n) -
O2
1
01 , pol
(7.25)
where 0 1 and O2 are zero matrices of appropriate dimension, and Po is a constant representing an estimate of the uncertainty in the initial values assigned to the network parameters, which in this algorithm is also the variance of the observation {z(n),y(n(M+l))}. 120
Chapter 7
7.3.4
Signal Reconstruction in Noisy Channels
Pruning of Hidden Units
As the network grows, the number of hidden units increases, and so will the computing complexity. Moreover, some added hidden units may subsequently end up contributing very little to the network output. The network will only benefit from those hidden units in which the input patterns are close to the stored patterns. Thus, pruning redundant units in the hidden layer becomes imperative. We denote the weighted response of the ith hidden unit for input zen) as (7.26) Suppose the largest absolute output value for the nth input zen) among all hidden units' weighted outputs is [umax(n)[. Also denote the normalized output of the ith hidden unit for the nth input as ';/n) =
uJn) I· umax(n)
1
(7.27)
In order to keep the size of the network small, we need to remove hidden units when they are found non-contributing. Essentially, for each observation, each normalized output value ';i(n) is evaluated. If ';i(n) is less than a threshold () for T3 consecutive observations, then the ith hidden unit should be removed, thereby keeping the network size and the computing complexity to a minimum.
7.3.5
Summary of the Flow of Algorithm
Basically, the above adaptive algorithm aims to retrieve a value of the bifurcation parameter (e.g., a in Eq. (7.2» during each time window of Tj observation steps. Two types of estimation are performed by this algorithm. In one observation step (i.e., (M+l) time steps), an estimate of (Xj,X2) is produced. This reconstructs the spread-spectrum signal. Then, in LT2 observation steps, an estimate of the bifurcation parameter is produced. This 121
7.3
On-Line Adaptive Learning Algorithm and Demodulation
retrieves the message. Moreover, the latter estimation requires knowledge of the former, and in practice, the REF network needs a number of observation steps to converge its weights and coefficients. Thus, in our algorithm, we allow a sub-window of T4 observation steps (T4 < T I ), during which estimation of the bifurcation parameter is omitted. In the remaining sub-window of (TI - T4) observation steps, estimation of a is then performed to retrieve the message signal. Specifically, the purpose in the first sub-window of T4 observation steps is mainly to train the network to track the dynamics, and in the next sub-window of (TI - T4) observation steps, the "trained" network estimates XI
using
Xl (nI; (M + 1) + 1) == h(Z'(n + 1))
(7.28)
Xl (nI; (M + 1) + 2) == h(z"(n + 1)),
(7.29)
where
'
Z (n
Xl (nI; (M + 1)) y(nI;(M +1)-1)
+ 1) =
.
[ y(nI; (M
I ,
(7.30)
+~) - (M -1))
Xl (nI; (M + 1) + 1) Xl (nI; (M + 1)) y(nI; (M + 1) -1)
z"(n+l)=
(7.31)
y(nI; (M + 1) - (M - 2)) and the least-square-fit is used to retrieve the message a, i.e., using Eq. (7.7). The following pseudocodes summarize the demodulation algorithm: initialize the networks (assign small random numbers for each message signal a(k) for each observation {zen), y(n(M + I))} do
122
«
0.05) to Ci),
Chapter 7 Signal Reconstruction in Noisy Channels
compute networks output subject to Eq. (7.11), determine whether or not a hidden unit should be added, if conditions Eq. (7.12) to Eq. (7.14) hold add a new hidden unit, assign relevant parameters, adjust covariance matrice Eq. (7.25), else adjust the networks parameters according to Eq. (7.18) to Eq. (7.24) endif; check the criterion for pruning a hidden unit, if; i (n) < Bfor T3 continuous observations delete the ith hidden unit, reduce the network size, endif; ifn?: T4 estimate xl(nT2(M + I) + I) and Xl (nT2(M + I) + 2) with the RBF network, perform demodulation with Eq. (7.7), endif; end for; end for.
In the next section, we will implement the above demodulation algorithm in an REF network and apply the network to extract messages from broadband signals.
7.4
Computer Simulation and Evaluation
Four different kinds of message signals will be employed to test the aforedescribed demodulation scheme, namely, square-wave, sine-wave, speech, and image signals. The square-wave signal is defined by the following piecewise-linear function: 123
7.4 Computer Simulation and Evaluation
1.37, kE[I,551]. 1.42, k
E
[552,1102].
1.35,
k
E
[1103,1653].
a(k) = 1.39,
k
E
[1654,2204].
1.32, k
E
[2205,2755].
1.36, k
E
[2756,3306].
1.41,
E
[3307,3857].
k
The sine-wave signal is as defined in Eg. (7.3) for k
(7.32)
=
1 to 200. The speech
signal used in the test contains a male speech signal "CHAOS COMMUNICATIONS", as shown in Fig. 7.5, which is sampled at 11 kHz and 8 bit precision. Finally, the image signal is from an Einstein portrait with 192 x213 pixels, each pixel having 256 gray levels, as shown in Fig. 7.6. In this proposed demodulating algorithm, each message signal a(k) is constant in a time window of TJ observation steps. For example, each pixel value of the image signal is constant in its TJ observation steps, and each sampling value of the speech signal is also constant in its TJ observation steps. 250
200
"
'0
150
.g
0E -
50
5,000
10,000 15,000 20,000 Time (sampling period)
25,000
Figure 7.5 A male speech waveform: "CHAOS COMMUNICATIONS", sampled at 11 kHz and 8 bit precision
124
Chapter 7
Signal Reconstruction in Noisy Channels
Figure 7.6 Einstein portrait (192 x213 pixels)
In the simulation, the transmitted spread-spectrum signal is controlled to reach the required signal-to-noise ratio (SNR) value in the AWGN channel. For the retrieved signal, the mean-square-error (MSE) is used to evaluate the demodulator's performance. The following definition for MSE is adopted. ITS
}
MSE = 10 10glO { - 'L[a(n) - a(n)]2 ,
1:;
(7.33)
n=1
where Ts is the number of the sampled message signals. In our simulation, the parameters of the RBF network and the EKF are assigned as follows:
T1 =551, T2 =5, T3=40, T4 250, L=60, 172=0.05, 173=0.07, 17rnax=2.0, 17min = 0.02, p= 0.973,po= 15.0, r = 0.01, P = 0.997, and e = 0.001. We will use the square-wave message example to illustrate a few important performance areas, i.e., the error propagation, the network growth profile, and the adaptive movement of the hidden units' centers. (1) For the square-wave message and an SNR of 20 dB in the received spread-spectrum signal, the error waveform of Xl is shown in Fig. 7.7, which shows that in the window of the first Tl observation steps, the error is the largest among all subsequent windows of TJ observation steps. This is because the tracking of the dynamics is mainly done in the first window. In each subsequent window of Tl observation steps, it is found that the error signal in the sub-window of the first T4 observation steps is larger than that in the rest 125
7.4
Computer Simulation and Evaluation
of the window. The sub-window from (T4+l) to TJ is, in fact, the estimation window during which the message signal is evaluated according to Eg. (7.7). 1.5~~~~~~~~-------------,
Message signal a(l1) square-wave
1.0 0.5
-;;;
51
.0;;
0.0
~ -0.5 -1.0 -1.5 500
1,000 1,500 2,000 2,500 3,000 3,500 Time (observation step)
Figure 7.7 Error wavefonn ofxJ for the case of square-wave message
(2) Illustrated in Fig. 7.8 is the growth of the hidden layer. It can be seen that the variation of the number of hidden units in the first window is 35 30
I 5
Message signal a(I1): square-wave
IV
OL---L---~---L
500
__- L_ _~_ _ _ _L -__L-~
1,000 1,500 2,000 2,500 3,000 3,500 Time (observation step)
Figure 7.8 Number of hidden units showing growth profile for the case of square-wave
126
Chapter 7
Signal Reconstruction in Noisy Channels
most drastic. Specifically, in the window of the first TI observation steps, the number of the hidden units adaptively adds and drop with time as well as with the input pattern. In the subsequent windows, the number of the hidden units varies with the input pattern adaptively. (3) Figure 7.9 shows the variation of the first and second components of the first unpruned hidden unit's center vector. From this, we can again see that the first window experiences the most rapid change. 2.4 2.2 . 2.0
.... : First component - - : Second component
·c;::l
"'"
.,,; .,,;
:E
1.0
~
0.8
-='"
0.6
'+0 en
0.4
'" c
0.2 0.0 g -0.2
';;j
'6 ...
u
-0.4 -0.6 -0.8
0
500
1,000 1,500 2,000 2,500 3,000 Time (observation step)
3,500
Figure 7.9 Variation of the first component (dotted line) and the second com-
ponent (solid line) of the first unpruned hidden unit's center
It should be clear that the REF-based demodulator can track the time-
varying chaotic system by adaptively adjusting both the number and center positions of the hidden layer units. The message signal and retrieved message signal are shown in Figs. 7.10 and 7.11, for the cases ofthe square-wave and sine-wave test messages, with SNR = 15 dB. Also, the retrieved speech signal and Einstein portrait are shown in Figs. 7.12 and 7.13 respectively.
127
7.4
Computer Simulation and Evaluation
1.42
- - : Message signal ------: Retrieved signal
~
1.40
1.38
s <:!
= 1.36
1.34 ~
1.32
o
500
1,000
1,500 2,000 2,500 3,000 3,500 Time (observation step)
Figure 7.10 Square-wave signal (solid line) and retrieved square-wave signal (dotted line, SNR = 15 dB)
- - : Message signal ...... : Retrieved signal
A A 1.40
n
A A
~
1.38
1.36 1.34 1.32
o
v v 20
40
v
60 80 100 120 140 160 180 200 Time (T] observation steps)
Figure 7.11 Sine wave signal (solid line) and retrieved sine wave signal (dotted line, SNR= 15 dB)
128
Chapter 7 Signal Reconstruction in Noisy Channels
5,000
10,000 15,000 20,000 Time (sampling period)
25,000
Figure 7.12 Retrieved speech signal (MSE = -22.9 dB, SNR= 15 dB)
7.13 Retrieved Einstein portrait (SNR
15 dB,
difference can be seen from the right hand side)
Finally, we measure the MSE performance for the test message signals. Results are shown in Fig. 7.14, from which we can see that the MSE descreases as the channel SNR increases. At an SNR of 15 dB, the MSE of the four retrieved signals are respectively, -21.1 dB, -19.1 dB, -22.9 dB and
-16.8 dB for the square-wave, sine-wave, speech signal and image signal. Remarks-It is of interest to note that the estimation of the parameter a(k) can be performed using direct computation based on Eq. (7.2), similar to 129
7.4
Computer Simulation and Evaluation
o·
- : Sine signal ...... : Image signal ---: Speech signal
~-20
ill
"'~-30 " VJ
~
-40
-50 10
20
30
40
50
60
70
SNR (dB)
Figure 7.14 MSE of retrieved source signals versus SNR using the proposed method
the method described in Anishchenko and Pavlov [271]. However, direct computation achieves reasonable performance only when the noise level is very low. Our method shows significant improvement over direct computation in the presence of noise. As a comparison, we present here the MSE's corresponding to the above same examples using the method of direct computation as in [271]. In brief, the method of direct computation [271] involves using the noisy data to solve a from Eq. (7.2) at each observation step. Then, the average value of a over TJ observation steps gives an estimate for the message signal. From Fig. 7.15, we note that when the SNR of the received signal is 15 dB, the MSE of the four retrieved signals are 69.5 dB, 89.9 dB, 139.2 dB and 145.8 dB for the square-wave, sine-wave, speech signal and image signal, respectively. Thus, we can see the significant improvement that can be gained from using the proposed method, especially for applications in noisy environment. In fact, at a high noise level (low SNR), the chaotic dynamics ofEq. (7.2) will be drastically altered, invalidating the conventional demodulation approaches for parameter estimation [280]. It should be noted that since our interest is to deal with noisy signals, the variance of the added random noise used in our study 1
(:::;10- ) is much higher than that used by Anishchenko and Pavlov [271] (:::;10-5 ) •
130
Chapter 7
o
5
10
Signal Reconstruction in Noisy Channels
15 20 25 30 35 40 45 50 55 60 65 70 SNR (dB)
Figure 7.15 MSE of retrieved source signals versus SNR using direct computation (for comparison) [155].
7.5
Application to Non-coherent Detection in Chaos-Based Communication
One useful application of the proposed demodulation (tracking) algorithm is to reconstruct the transmitted chaotic signal in a chaos-based communication system. In particular, we consider a simple digital communication system, in which the Chua's circuit and the hyperchaos oscillator [281] are respectively employed as chaos generators. Let us rewrite the circuit equations as follows
~l =al (x2 -Kl(Xl))'} x 2 = Xl - x 2 + X3 ,
X3
=
(7.34)
a 2(t)X2'
where al is fixed at 9, and a2 (t) varies here according to (7.35) which ensures the chaotic motion of the system. Also,
KIO
is the piecewise-
linear function as described in Eq. (2.8). The hyperchaos circuit [281] can be expressed by the following dimensionless equations: 131
7.5
Application to Non-coherent Detection in Chaos-Based Communication
(7.36) where x = [x],
X2, X3, X4]
T
,
T
d = [ -1, 0,10,0] ,
0 -a3 (t)
A=
and
K2 (Xl, X3)
0
1
0.7
0
0
0
0
0
0
1.5
-~oJ.
(7.37)
is the piecewise-linear function given by
-0.2+ 3(Xl
K2 (xl'x 3 )
=
j
-0.2(XI -
-X3
+1),
I Xl -
x 3 ),
-0.2 + 3(Xl
Xl -X3
- X3
-1),
X3
Xl - X3
<-1 l~ 1
(7.38)
> 1.
a3 (t) is a time-varying bifurcation parameter assigned in this study as (7.39) which guarantees the motion of the system is chaotic. In general, the signal to be transmitted is first normalized to the range [0, 1], and its average, a , is then removed, i.e.,
s=S(x)=
X-X. mm
-a,
(7.40)
Xmax -xmin
where X is the output signal of the transmitter. Xmin and Xmax are the minimum and the maximum of x. Figure 7.16 (a) shows a wave-form of s, as defined above, where X3 in Eq. (7.34) is selected as the output signal. For the hyperchaos oscillator, we will guarantee the transmitted (observed) signal s such that the system Eq. (7.36) is observable. In this study, (AT, AT) is observable, i.e.,
1i=
132
(7.41 )
Chapter 7
Signal Reconstruction in Noisy Channels
0.8 0.6
"'"
.~
'"0
'"
0.4
~
0.2
..c: u
"0
~
~
0.0
v
'"
::§
~
-0.2
0
E
:5 -0.4
I~
-0.6 -0.8 0
4,000
8,000
12,000 16,000 Time (a)
20,000 24,000 28,000
0.8 0.6 0.4
"'"
.~
0.2
~I
en 0
oj
..c: u
0.0
"0
v
1-0.2 "0 0
§ -04 .
~~I!.II
~
-0.6 -0.8 0
1,000
2,000
3,000
4,000 Time (b)
5,000
6,000
7,000
Figure 7.16 Normalized waveform of s of (a) Chua's circuit and (b) the hyperchaos oscillator
is invertible. Let (7.42)
where e4 = [0, 0, 0, 1] T , 1-£-1 is the inverse matrix of 1-£. The transmitted signal s is then 133
Application to Non-coherent Detection in Chaos-Based Communication
7.5
(7.43) Because (AT, rT(t)) is also observable in this study, there exist functions f and IfF such that Eq. (4.6) holds [283]. A normalized transmission signal s of the hyperchaos oscillator is shown in Fig. 7.16 (b). In particular, we now consider a simple digital communication system as shown in Fig. 7.17, which effectively employs a chaos-shift-keying (CSK) modulation [282]. Here, {b;} is the source message bit train which is shown in Fig. 7.18. Within each symbol duration of window size TJ, the signal to be transmitted is given by t E [1,T4 ]; for all bi' SO, t E [T4 + 1,1~]; if bi = 1. s= SO, { -SO, tE[T4+1,~]; ifbi=O.
(7.44)
where S(-) is the normalization function defined in Eq.(7.40) and the unit of t is observation step. y
Transmitter
,---------------------------------, I
bI ,
l~
0
Symbol duratIOn T1
1
lee I
fT +1dt T
Decision
4
£
Proposed tracker
L _________________________________
I I I I I
~
Detector
Figure 7.17 Block diagram of a digital chaos-based communication system
At the receiver, an integrator is used to compute the energy of the error signal, e, in the interval T4 +1 to TJ for each TJ window. The decoded symbol is then given by
{I,
if e
r. where 134
r is the threshold of the decision circuit.
(7.45)
Chapter 7
Ie-
r--
o o
-r--
L-
1,000
-
-'--
2,000
3,000
Signal Reconstruction in Noisy Channels
-;-
__
4,000 5,000 Time
'--
6,000
r-
_
'-
7,000 8,000
Figure 7.18 Source bit stream
The above tracking algorithm is used to track the transmitted signal during the first T4 observation steps of the TJ time window. The EKF algorithm, i.e., from Eq. (7.18) to Eq. (7.24), helps to find a suitable state vector for the reconstructed system, i.e., v in Eq. (7.18). However, once this state vector is found, typically within the T4 observation steps, this EKF algorithm can be modified to improve convergence of the network by enlarging the attracting domain [284]. Specifically, for (T4 + 1) to T J observation steps, we replace Eq. (7.21) and Eq. (7.24) by
R(n) =- a4B(z(n»P(n, n -1) BT (z(n» + as,
(7.46)
P(n, n) =- [1 - K(n) B(z(n»] P(n, n -1),
(7.47)
Where a4 and as are two parameters used to guarantee the convergence of this algorithm [284]. Also, if a zero bit is retrieved, -S (-) in Eq. (7.44) will be restored to S(-) in the duration of [T4+1, T J], and the algorithm (Eqs.(7.187.20), Eq. (7.22), Eq. (7.23), Eq. (7.46), and Eq. (7.47», will be used again in this duration until the beginning of the next T J• In our simulation, the parameters of the RBF network and the EKF are assigned as follows. T J=-420, T3=-65, T4=-220, 1]2=-0.01, 1]3=-0.055, 1]max 4 =- 2.2, 1]min =- 0.06, p =- 0.994, Po =- 10.0, = 0.01, j3 = 0.997, () = 10- , a4 = 1.5,
r
as = 0.4, are used for the simulation of the Chua's-circuit-based commu-
= 370, T3 = 75, T4 = 215, 1]2 = 0.005, 1]3 = 0.09, 1]max = 5.3, 1]min = 0.05, P = 0.982, po = 10.0, r = 0.01, j3 = 0.975, () = 1.1 X 10-4,
nication system. Similarly, TJ
135
7.5
Application to Non-coherent Detection in Chaos-Based Communication
a4 = 2.3, as = 0.51, are used for the simulation of the hyperchaos-circuitbased communication system. Some waveforms of interest are shown here. When the Chua's circuit is used as a chaos generator, Fig. 7.19 (a) shows the transmitted chaotic signal s, as defined by Eq. (7.44), Fig. 7.19 (b) shows the AWGN for SNR= 20 dB and Fig. 7.19 (c) shows the received signal at the receiver. 0.8 ,-----------------------------------, 0.6 0.4
~
OIl .;;;
0.2
.~ 0.0
11111111\1
'" -0.2 .~
e'"
f-<
-0.4 -0.6 -0.8 '--__-'-____'--__-'-__--.1_ _ _ _- ' -_ _- - '_ _ _ _--'---' o 4,000 8,000 12,000 16,000 20,000 24,000 28,000 Time (a)
0.8 , - - - - - - - - - - - - - - - - - - - - - , 0.6 0.4
-0.8 '-----'-____'--__-'-__- - . 1 ' - -_ _- ' -_ _--.1_ _ _ _-'--1 o 4,000 8,000 12,000 16,000 20,000 24,000 28,000 Time (b)
Figure 7.19 Wavefonn of (a) the transmitted chaotic signal generated from the Chua's circuit, (b) the noise of the channel at SNR = 20 dB, and (c) the received signal at the receiver 136
Chapter 7
Signal Reconstruction in Noisy Channels
0.6 0.4 '" c:
0.2
Ui -0
0.0
b!)
">
8-0.2
"
0::
-0.4 -0.6 -0.8 '---_---'---_ _'---_---'---_-----'_ _---'---_-----'-_ _--'-----J o 4,000 8,000 12,000 16,000 20,000 24,000 28,000 Time (e)
Figure 7.19 (Continued)
Similarly, Fig. 7.20 (a) and Fig. 7.20 (b) show the waveforms of the transmission signal and the received signal (SNR = 20 dB) of the hyperchaoscircuit-based communication system. 0.8 , - - - - - - - - - - - - - - - - - - - - - - , 0.6 0.4
"2
0.2
b!)
.;;;
.~
'"
0.0
a ~
f': f-<
0.2
-0.4 -0.6 -0.8
' - - _ - - - - L_ _--'---_-----'_ _----L_ _--'---_-----'_ _---"
o
\,000
2,000
3,000
4,000
5,000
6,000
7,000
Time (a)
Figure 7.20 Waveform of (a) the transmitted hyperchaotic signal, and (b) the received signal at an SNR of 20 dB
137
Application to Non-coherent Detection in Chaos-Based Communication
7.5
0.8 - - - - : - - - - - - - - - - - - - - - - - - , 0.6 0.4 0.2 0;
"
.~ 0.0 "0 OJ
.~
0.2
u
OJ
~
-0.4 -0.6 -0.8
L-_---'...._ _L-_---'--_ _L-_----'--_ _..L-_----'
o
1,000
2,000
3,000
4,000
5,000
6,000
7,000
Time (b)
Figure 7.20 (Continued) 10-1~------------------------------~ ..... : Hyperchaos --: Chua
til
co
10-4
10-7 '-----,-L-----~--------:":----____:L------:'-::------_::'::_----__= 10 20 30 40 50 60 70 SNR (dB)
Figure 7.21 Bit error rate of the retrieved digital message vs SNR (solid line for Chus's circuit and dotted line for the hyperchaos oscillator)
The above detection algorithm is used to evaluate the performance of the chaos-based digital communication system via 40 independent realizations, 138
Chapter 7
Signal Reconstruction in Noisy Channels
m which the two chaos generators mentioned above are used in the transmitter respectively. The Bit Error Rates (BER) of the retrieved digital message signals of the two systems via different SNRs are shown in Fig. 7.21, where the solid line is for the Chua's circuit, and the dotted line for the hyperchaos oscillator. When SNRs are equal to 10 dB and 15 dB, 3
BERs are 1.047xl0- and 1.288 X 10--4 for the Chua's-circuit-based system, respectively; and BERs are 2.262xl0-3 and 5.321 X 10--4 for the hyperchaososcillator-based system, respectively. Therefore, we can see from the figure that the digital message signal can be successfully retrieved from an AWGN channel by the proposed detector.
7.6
Summary
In this chapter, we have designed an on-line adaptive demodulator for recovenng message signals that are transmitted through chaotic carriers contaminated by additive white Gaussian noise. The message is injected into a Henon-map-based transmitter system as variation of a bifurcation parameter. The demodulation process essentially aims to recover the embedded parameter variation. The essential component of the proposed demodulator is a RBF neural network, which incorporates an adaptive learning algorithm to track the dynamics of the Henon map. The least square fit is used to estimate message signals. The purpose of the adaptive learning algorithm is to adaptively configure hidden layer size and adjusts the relevant parameters with an EKF algorithm. The system is tested with square-wave, sine-wave, image, and speech signals serving as messages. Results have demonstrated that the demodulator is capable of performing the required demodulation task for a noisy communication channel. As an application example of the proposed tracking algorithm, we have studied the information retrieval in a simplified CSK digital communication system employing the proposed tracker for non-coherent detection, in which Chua's circuit and the hyperchaos oscillator are respectively used as chaos generators. 139
Chapter 8
Signal Reconstruction in Noisy Distorted Channels
Most chaos-based communication systems proposed and analyzed are based on the assumption of a rather ideal communication environment, in which signals are transmitted without distortion and with only a moderate amount of added noise. However, in reality, the performance of a communication system can be seriously impaired by channel effects and noise, especially for coherent-type systems where the unfaithful chaos synchronization can make detection rather unreliable [280,285]. Theoretically, if a wideband chaotic signal is transmitted through a band-limited channel, the inevitable loss of spectral components in the received signal may cause the transmitted signal of one symbol to spread over and overlap successive symbol intervals, and this effect is commonly termed inter-symbol interference (lSI) [264]. It has been demonstrated that even simple variations of channel gain or pure phase distortions in the channel may adversely affect the transmitted signal [101, 286].
In addition to linear distortion, the transmitted signal is subject to other contaminations such as thermal noise, impulse noise and nonlinear distortions arising from the modulation process. The idea of channel equalization is to combat the unfavorable channel effects such that the transmitted signal can be preserved with highest integrity [287]. Recently, some approaches have been proposed for combating the effect of channel distortions in chaos-based communication systems. For example, the synchronization-based method [288 - 290] takes advantage of synchronization between the transmitter and receiver to estimate channel distortion. However, it is not easy to choose a suitable coupling parameter (or adaptive coupling parameter) to ensure that all of the conditional Lyapunov exponents
Chapter 8
Signal Reconstruction in Noisy Distorted Channels
in the demodulator are less than zero so that the receiver can approximately synchronize with the transmitter. Motivated by the lack of effective channel equalization methods for chaosbased communication systems, we attempt to design a channel equalizer for chaotic signals. Linear and nonlinear distortions are considered, in addition to additive white Gaussian noise (AWGN). Specifically, we will employ a modified RNN to realize the equalization task [291]. This chapter is organized as follows. In Sec. 8.1, we review the background theory of channel equalization in both chaos-based communication systems and conventional communication systems. Details of the proposed equalizing (learning) algorithm are described in Sec. 8.2. The simulation performances of the proposed equalizer are demonstrated in Sec. 8.3, where three wellknown channel models (linear and nonlinear) will be used to evaluate the performance of the equalizer. Finally, in Sec. 8.4, the performance of the proposed equalizer is compared with conventional linear transversal equalizers (LTEs) and those based on feedforward neural networks (FNNs).
8.1
Preliminaries
Shown in Fig. 8.1 is the block diagram of part of a chaos-based communication system, where x is the transmitted signal produced by the chaotic modulator and h is the transformation function of the channel. The output of the channel s is corrupted by noise '7, which is usually modelled as an additive white Gaussian noise (AWGN) process with a zero mean. At the receiver, the received signal, y, first goes through an equalizer, which cancels ,-------, ,----------, ,--------, 1
C I'I~II laotlc 1 xis @ 1 Y
11 transmltte . d 1 L
signal
______
h
L
1 1 l i t 1]
..J
L
__________
1 1
J
.11 1 L
RNN-based equalizer ________
x
1 1 ----1
J1
Channel
Figure 8.1 Block diagram of part of a chaos-based communication system showing the channel and equalizer
141
8.1
Preliminaries
the channel effects and estimates the transmitted signal. In this study we do not consider any specific modulation (demodulation) strategy, and our specific focus here is on canceling the channel effects. If the channel h is modeled as a linear operator, the output of the channel, s, is simply the convolution of the input sequence x with h, i.e., s = h x x. Alternatively, h can be modeled as a nonlinear operator, which is given generally by s = h(x) [264]. The input to the equalizer is then y = s + 77.
(8.1)
The problem addressed in this chapter may be summarized as follows. Given the noisy and distorted sequence y, the problem is to find an equalizer such that the originally transmitted sequence x or at least a delayed and/or phase-shifted version of it, can be reconstructed. Therefore, the ideal equalization requires
x = btL
ej(J x
be achieved, where t is time instant, L is a
time delay, Bis a constant phase shift, and bis the Kronecker delta function.
8.1.1
Conventional Equalizers
The problem of equalization can be interpreted as one of inverse modeling [292], i.e., deconvoluting the received sequence in order to reconstruct the original message. The conventional equalizer architecture is shown in Fig. 8.2. The received training sequence, {y(t)} , is filtered by a linear transversal equalizer (LTE) which produces an output for the sample at time t-L based on m most recent channel observations made at time t, where the integers m and L are the equalizer's order (tap) and delay, respectively. In the training stage (Fig. 8.2 (a)), some preset sample signals are transmitted via the channel, and the received samples are used to train the LTE. Usually, a stochastic gradient algorithm, such as the LMS algorithm [179], is used to adjust the tap weight
Wi,
i = 0, 1, 2, ... , m -1, in the light of the error signal
e(t-L), which is given by e(t - L) = d (t - L) - x(t - L), 142
(8.2)
Chapter 8
Signal Reconstruction in Noisy Distorted Channels
where d(t-L) is the desired signal at time t-L, and
x(t-L)
is the estimate
of x(t-L). In the equalizing stage (Fig. 8.2 (b)), the trained LTE gives the estimate ofx(t-L) from the unknown received sequence when communication commences. y(t)
Adder
X(t-L) L--_ _ _ _ _ _
~
2:
e(t-L)
+ d (Desired) (a)
y(t)
y(t-I)
y(t-2)
y(t-m+J)
(b)
Figure 8.2 Linear transversal equalizer used in (a) training stage, and
(b) equalizing stage
8.1.2
Reconstruction of Chaotic Signals and Equalization
From the viewpoint of signal processing, the equalization issue of the chaosbased communication system can be described in terms of signal reconstruction. Specifically, let us consider a chaotic system whose dynamics is governed by the following state equation
x = g(x,t),
(8.3) 143
8.1
Preliminaries
where X==
[Xl
X2 ... XD]T is the D-dimensional state vector of the system, and
g is a smooth nonlinear function defined on
]RD
x R When an observer in
another space makes some measurements of the system, some observed samples {s} can be generally obtained from s
(8.4)
== ¢(x(t)),
where ¢ (-) is a scalar real-valued function. Now, the task is to reconstruct the original chaotic dynamics ofEq. (8.3) based on the observed samples. As studied in Chapter 5, the chaotic attractor of Eq. (8.3) can be reconstructed as:
x where 'F == [If/l If/z''' If/D f
(8.5)
== 'F(s,t),
, and
s==[s ==
ss
"'S(M-11f
(8.6)
I(x,t).
Thus, it is generally possible to reconstruct the chaotic attractor ofEq. (8.3) in a higher dimensional space, ifI
(-) and 'F
are known. Similar conclusion
can be obtained for discrete-time chaotic systems by simply replacing the derivatives with time-advanced state variables, i.e., (X(t),x(t),x(t),"')
~
(x(t),x(t + l),x(t + 2),,, .),
(8.7)
where x is one state variable or one observed scalar of a discrete-time chaotic system. However, the receiver usually has no exact knowledge of I and 'F in practice even when the system is noise free. Thus, the crucial step is to find
I
and 'F under noisy environment in order to realize the recon-
struction task. This problem is tackled in this chapter by a modified recurrent neural network (RNN), which will be described in the next section.
8.1.3
Recurrent Neural Network and Equalization
The aforementioned equalization problem can be regarded as a nonlinear 144
Chapter 8
Signal Reconstruction in Noisy Distorted Channels
modeling problem. The nonlinear autoregressive moving average model (NARMA), which is a widely used tool for modeling nonlinear dynamical system (e.g., in time-series processing, nonlinear signal reconstruction, etc.), can be used to describe the said system [293]. Typically, we write
x(t) =
w(y(t- 1), y(t- 2),··· ,y(t- M), e(t- 1), e(t - 2),··· , e(t - N» + e(t),
(8.8)
where e(t) is the error signal at time instant t between the original and the estimated signal, wis an unknown function, M and N are the time delays of the input signal and the error signal, respectively. The conditional mean of x based on the infinite past observations is
x(t)
=
E [w (y(t- 1), y(t- 2),···, y(t- M), e(t- 1), e(t- 2), ... , e(t-N» Iy(t-l), y(t-2),··· ,],
(8.9)
where E denotes expectation. Suppose that the NARMA model is invertible in the sense that there exists a function
x(t) =
fj/ (y(t -
fj/ such
that
1), y(t - 2), ... ,) + e(t).
(8.10)
Then, given the infinite past observations y(t- 1), y(t- 2),···, one can in principle use the above equation to estimate e(t- j) in Eq. (8.8). In this case the conditional mean estimate is
x(t) = w(y(t- 1), y(t- 2),··· ,y(t- M), e(t- 1), e(t- 2),···, e(t- N».
(8.11 )
Since in practice only a finite observation record is available, one cannot perform computation using Eq. (8.11). However, it is possible to approximate Eq. (8.11) by the recursive algorithm [293]
x(t) = w(y(t -
1), y(t - 2), ... ,y(t - M),
e(t-l), e(t-2),···, e(t-N»,
(8.12)
145
8.1
Preliminaries
where eU) = xU) - xU), j
=
t - 1, t - 2, ... , t - N. The above equation can
be approximated by the following recurrent model [293, 206]
1
N (M N x(t) = ~Ui~ ~Wyy(t- j)+ ~w~(X(tj)-x(t- j)) + ()i '
(8.13) which is actually a special case of a general RNN to be described in the following. Here,
Ui,
Wy and w~ are coefficients.
~
is a parameter and
~
is a
nonlinear function.
An RNN is shown in Fig. 8.3, which is a three-layer network consisting of the input layer, the hidden layer (processing layer) and the output layer. The input vector of the input layer at time instant t is v(t), which is defined as (8.14) where Vi (t), 2 ~ i ~ M + 1, is the external input which is the delayed version of y, i.e., vi(t)=y[t-(i-1)], and Vi, M+2
~i~M+N+1,
is the feedback
input of the ith input unit at time instant t. Also, N is the number of hidden layer units and
VI
is the bias input which has been fixed at" +1 " in this
chapter.
v Figure 8.3 Block diagram of a modified recurrent neural network
146
Chapter 8
Signal Reconstruction in Noisy Distorted Channels
The internal activity of the ith hidden unit at time instant t is given by M+N+i
lj (t) =
L
wij
(8.15)
(t)v j (t),
j=i
where wij (t) is the connection weight between the ith hidden unit and the jth input unit at time instant t. At the next time step 1+ 1, the output of the ith neuron, qi (/+ 1), is computed using a nonlinear activation function, tp (.), yielding (8.16) In this study, we choose (8.17)
tp(x) = tanh (ex),
Where e is constant. Let u;(t) be the connection weight between the ith hidden unit and the output unit. The output of the output unit is given by N
(8.18)
£(t+1)= L[ui (t)qi (t + 1)]. i=1
For 1,;;; i,;;; N, and letting (8.19) and w (if M > N) = lj
w + W' { wij lj
lj
for 2,;;; j ,;;; N + 1 for N + 2 ,;;; j ,;;; M + 1,
(8.20)
we can easily see that Eq. (8.13) is a special case of Eq. (8.18) for M> N. Similarly, for the cases where M < Nand M
=
N, the same conclusion
holds. The above estimation procedures, together with the training algorithm described in the following section, can be used to realize the equalization task.
147
8.2
8.2
Training Algorithm
Training Algorithm
Let d(t+l) be the desired output of the output unit at time instant t+1. The error signal e(t +1) is
e(t + 1) = d(t + 1) - x(t + 1).
(8.21)
The weight between the hidden layer and the output unit is then updated by a least-mean-square (LMS) algorithm [178], i.e.,
Ui(t + 1) = ui (t) + fJ1e(t + 1) qi(t + 1),
(8.22)
where fJI is the learning rate. The instantaneous sum of squared errors of the network is defined as &O(t+l)=(l/2)e 2 (t+l). Also, we defme the local gradient of the ith hidden unit at time instant t + 1, y; (t+ I), as
y(t+l)= ,
cl&°(t+l) Olj (t)
oq.(t + 1) = e(t + l)u i (t)--"-'-'-'--....:.. olj (I) =
e(1 + l)u i (t)tp'(lj (t)),
(8.23)
where tp' (-) is the derivative of tp with respect to its argument. According to the delta learning law, the weight
wij
(i = 1,2,.··, N, j
= 1,2,.··, M + N + 1) can
be updated as follows:
Wij (t + 1) =
wij (t)
+ fJ2Y i (t + l)vj (t),
(8.24)
where fh is the learning rate. Now, define the instantaneous sum of squared errors for the hidden layer units as 1 N &(t) = e; (t), 2 k~l
L
(8.25)
where ek(t) is the difference (error) in the output of the kth hidden unit before and after the weight updated as 148
wij
is updated. Then, the instantaneous weight is
Chapter 8
Signal Reconstruction in Noisy Distorted Channels
_ wij(t+1) = wij(t+1)-J33
O&(t) ow~(t+1)
,
(8.26)
where /33 is learning rate. From Eq. (8.16) and Eq. (8.25), we have
= _~
() oq; (t + 1)
k=1
ow~ (t + 1)
L..ek t
where
(8.27)
,
q; (t + 1) is the output of the kth hidden unit after the weight
wij
is
updated to w~ (t + 1). To determine the partial derivative oq; (t + 1) / ow~ (t + 1), we differentiate Eq. (8.15) and Eq. (8.16) with respect to
By applying the
wij.
chain rule, we obtain oq;(t + 1)
oq; (t + 1) ork- (t + 1)
ow~(t + 1)
ork- (t + 1) ow~ (t + 1) = m'(r - (t
rk-.....:(t_+_1-,---) + 1)) _0--2
(8.28)
ow~(t+1)'
k
'f'
where rk- (t + 1) corresponds to the updated internal state of hidden unit k. By using Eq.(8.16), we get ork-(t+1) ow~(t+1)
=
=
I
M +1 0 [w- (t+1)v (t)J n=1 ow~(t+1) kn n
M~+I[ _( 1) L.. w t + n=1
kn
ovn(t) ow~(t + 1)
()OW;n(t+1)~
+v t . n ow~(t + 1) (8.29)
· . ow;;;, (t + 1) IS . equato I one 1'f k = 1. an d n =j,. an d'IS Note that t h e d envatlve ow~(t+1)
zero otherwise. Thus, we may rewrite Eq. (8.29) as
ork-(t+1)_M~+I[ -( 1) - L.. W t +
ow~(t+1)
n=1
kn
ovn(t) ow~(t+1)
1
s: (t) • + Uk V. I
(8.30)
}
149
8.2
Training Algorithm
From Eg. (8.14), we have
o
vn (I)
=
{
ow::(/+l) lj
oq;(t) ow:: (t + 1)
0
for n = M + 2, ... , M + N + 1.
(8.31 )
lj
otherwise.
We may then combine Eg. (8.28), Eg. (8.30) and Eg. (8.31) to yield
(8.32)
where J is the Kronecker delta function. We now defme a dynamical system described by a triply indexed set of variables {Q; (I + I)}, where
Q~(t+l)= lj
oq;(t) , Owij(1 + 1)
(8.33)
for } = 1, 2,··· , M + N + 1, i = 1, 2,··· , N, and n = M + 2,··· , M + N + 1. For each time step 1 and all appropriate n, i and}, the dynamics of the system so defined is governed by
(8.34)
Finally, the weight between the input layer and the hidden layer is updated by N
wij (I + 1) = wij (I + 1) + /33
L e (I + I)Q~ (t + 1). k
(8.35)
k~l
The above procedure is repeatedly applied to all input sample pairs during the training stage. 150
Chapter 8
8.3
Signal Reconstruction in Noisy Distorted Channels
Simulation Study
In this section we simulate a chaos-based communication system, which is subject to channel distortion and additive white Gaussian noise. Our purpose is to test the ability of the proposed equalizer in combating the channel effects and noise.
8.3.1
Chaotic Signal Transmission
Two chaotic systems will be used to evaluate the performance of the proposed equalizer in this chapter. The first system is based on the HEmon map: Xl (t
+ 1) = 1- alx~ (t) + x2(t),}
(8.36)
x2 (t + 1) = a 2 xI (t),
where al and az are the bifurcation parameters fixed at 1.4 and 0.3, respectively. In particular, we select Xz as the transmitted signal, which guarantees that Eq. (8.5) holds. The second system is based on the Chua's circuit [138], which is rewritten as follows:
~l :a3~2 -K(XI )),}
where
a3
and
a4
x2
-
Xl
X2
X3
=
a 4 x2 ,
(8.37)
+ X3 '
are respectively fixed at 9 and -100/7, and
K (-)
is a
piecewise-linear function given by _{mIXI +(mo -ml ) K(XI ) - mOxl mixi - (mo - ml )
for
Xl:;:: 1
for for
1Xl 1< 1
(8.38)
Xl (-1
with mo = -117 and ml = -217. The attractor has the largest Lyapunov exponent and the Lyapunov dimension equal to 0.23 and 2.13, 151
8.3
Simulation Study
respectively [139]. In this case, we select
X3
as the transmitted signal. This
choice of the transmitted signal guarantees that the dynamics of the transmission system can be reconstructed, i.e., there exist functions/and 'P such that Eq. (8.5) holds.
8.3.2
Filtering Effect of Communication Channels
Three channel models will be used to test the performance of the proposed equalizer in this chapter. The first two channels are linear channels which can be described in the z domain by the following transformation functions: channel I :
HI (z) = 1+ 0.5z- 1
(8.39)
channel II:
H 2 (z)=0.3+0.5z- 1 +0.3z- 2
(8.40)
These two channel models are widely used to evaluate the performance of equalizers in communication systems [294]. Let us consider the frequency responses of the channels. The amplitude-frequency and phase-frequency responses of the channels are shown in Fig. 8.4. It is worth noting that from Fig. 8.4 (a), channel II has a deep spectrum null at a normalized angular frequency of2.56, which is difficult to equalize by the usual LTE [264]. The third channel to be studied is a nonlinear channel, which is shown in Fig. 8.5. This channel is also widely used for testing the performance of equalizers [295]. The model can be described by (8.41 ) where a1 and a2 are channel parameters which are fixed at 0.2 and -0.1, respectively, ands is the output of the linear part of the channel which is given by set) = 0.3482 x(t) + 0.8704 x(t -1) + 0.3482 x(t - 2).
(8.42)
Thus, the transformation function of the linear component can be expressed as H(z) = 0.3482 + 0.8704 Z-l + 0.3482 152
Z-2.
(8.43)
Chapter 8
Signal Reconstruction in Noisy Distorted Channels
10 0 -10 -20
EO
'S -30
"
@ (J
-40 -50 -60 -70 -3
-2
-1 0 Nonnalized angular frequency
2
3
(a)
3,----------------------------------, 2
-2 -3_u3~----72-----~1~---07---~-----27---~3
Normalized angular frequency (b)
Figure 8.4 Frequency responses of Channel I (dotted line) and ChannelII (solid line) according to Eq. (S.39) and Eq. (S.40). (a) Magnitude responses; (b) Phase responses y
Figure 8.5 Nonlinear channel model studied in this book
153
8.3
Simulation Study
20 18 16 14 ~
Z
.~
::s'"
12 10
8 6 4 2 0_
3 Nonnalized angular frequency (a)
16 14 12 0)
"c:l
10
Z
a
bJJ
::s'"
8
6 4 2
o Nonnalized angular frequency (b)
Figure 8.6 Illustration of channel effects. FFT magnitude spectrum of signal versus normalized angular frequency for (a) transmitted signal
Xl
from Eq. (8.36), and
(b) received signal y after passing through Channel I and contaminated by noise at SNR= 10 dB
154
ChapterS
Signal Reconstruction in Noisy Distorted Channels
004
IFf' '",
0.3
0.1
~ ~.
I'
/
0.2
.-~
,
/;-:---...~
"i
0 -0.1 -0.2 -0.3
.I /
/
-004
-0.3
-0.2
-0.1
0.1
0 x,(t)
0.2
\ 0.3
004
(a) 004 0.3 0.2 0.1 0 +
~
"
,/ !
-0.1 -0.2 -0.3
/
!
-004 -0.5 -0.6
-004
-0.2
0.2
004
sit)
(b) 0.6
004 0.2
I
"
.
0
. , .,..-.....
"-,
:
-0.2
"',:
"
-004 '"
-0.6
-0.8
-0.6
-004
-0.2
0 y(t)
0.2
OA
0.6
(e)
Figure 8.7 Illustration of channel effects. Return maps reconstructed from (a) the transmitted Xl from Eq. (8,36), (b) the distorted s(t), and (c) the received signal yet) after passing through Channel I and contaminated by noise at SNR = lO dB
155
8.3
Simulation Study
As an example to illustrate the channel effects, we consider a communication event, in which the transmitted signal
X2
generated from Eq. (8.36)
passes through Channel I. When the SNR is 10 dB, the fast Fourier transform spectra of the transmitted signal and the received signal are shown in Figs. 8.6 (a) and (b), from which we clearly observe the wideband property of the transmitted signal and the distortion caused by the channel. Furthermore, the return maps reconstructed from the transmitted signal, the distorted signal due to Channel I, and the noisy received signal are shown in Fig. 8.7. In our previous study [101], it has been shown that without an equalizer, the simple inverse system approach will give unacceptable performance even when the channel, besides AWGN, is an ideal allpass filter, i.e., h = 1.
8.3.3
Results
The equalization for each channel model consists of two stages. The first is the training or adaptation stage in which the equalizer makes use of some partially known sample pairs to adapt to the communication environment. When the training or adaptation is completed, actual communication commences. Determining the size of training sample sets is important. If the sets are too small, the equalizer cannot experience all states of the system in the given communication environment, leading to poor reconstruction. However, if the sets are too large, the training duration will be excessively long. In our simulations, we consider different sizes of the training sample sets, and examine the results in terms of the mean-square-error (MSE) as a function of the number of iterations (in this book, one iteration means that the equalizer is trained once with all training samples), where the MSE is defined as <( x- X )2), in which " <) " is an averaging operator. According to the requirement for the embedding dimension in Takens' embedding theory, we set each training sample pair consists of 8 elements (7 transmitted signals 156
Chapter 8
Signal Reconstruction in Noisy Distorted Channels
and 1 known signal at the receiver). Also, noise is added to the training samples at an SNR of 10 dB during the training stage, and the RNN is assigned with M=7 and N=6. Figs. 8.8 (a) and (b) show the MSE of the equalized samples, averaged over 40 independent realizations, versus the number of iterations for the HEmon-map system and for the Chua's system. o.-------------------------------------~
300 samples
650 samples I
600 samples
600 samples
-80L---~--~--~~--~~~~~~---L--~~
o
100
200
300
400 500 Iterations
600
700
800
(a) o,-----------------------------------~
300 samples
600 samples I -80 0
100
200
300
400 500 Iterations
600
700
800
(b)
Figure 8.8 MSE of the equalized samples for channel I , averaged over 40 independent realizations, versus the number of iterations for (a) the Henon-map system and (b) the Chua's system
157
8.3
Simulation Study
We can see that the smaller the training size is, the more uncertain the equalizer will be; and when the size of the training sets is about 600, the MSE is below -70 dB after some iterations. According to the above results, we select the size of the training sets as 600 in this study, and stop the training when the MSE for all samples is less than 10-7 . Results are summarized as follows: (1) When the RNN-based equalizer is applied to equalize channels I and II , the following results are obtained:
CD
For the Henon-map system, it is found that the equalizer completes
the training in approximately 500 iterations for channel I and 600 iterations for channel II (i.e., MSE < -70 dB). The trained equalizers are then used to test their performances when actual communication takes place. Figure 8.9 shows the MSE versus SNR for channels I and II. When the SNR exceeds 14 dB, the MSE of the equalized signal for the two channels is less than -80 dB. o,----------------------------------, - : Channell ...... : Channel II
-80 -90L-4L---~6--~8--~lLO---lL2---1~4--~16~--IL8--~20
SNR (dB)
Figure 8.9 MSE performance of the recovered
x of the Henon-map system versus 2
SNR for Channel I (solid line) and Channel II (dotted line)
®
For the Chua's system, it is found that the equalizer completes the
training after approximately 500 iterations for channel I and 700 iterations 158
Chapter 8
Signal Reconstruction in Noisy Distorted Channels
for channel II. Again, the trained equalizers are then used to test their perfonnances when actual communication takes place. Figure 8.10 shows the MSE versus SNR for channels I and II. When the SNR is more than 14 dB , the MSE of the equalized signal for the two channels is less than -76.3 dB. 0 -10
- : Channel I ...... : Channel II
-20 -30 ~ -40
::s Bl ;;S
-50 -60 -70 -80 -90
4
6
8
10
12
14
16
18
20
SNR (dB)
Figure 8.10 MSE performance of the recovered X3 of the Chua's system versus SNR for Channel I (solid line) and Channel II (dotted line)
(2) When the RNN-based equalizer is applied to equalize the nonlinear channel described in Sec. 8.3.2, the following results are obtained:
CD
For the HEmon-map system, Fig. 8.11(a) shows the MSE versus the
number of iterations during training, and Fig. 8.11 (b) shows the equalization perfonnance of the trained equalizer. We can see that from Fig. 8.11 the equalizer completes its training after about 800 iterations in the training stage, and the MSE is -73.8 dB when SNR is equal to 14 dB.
® For the Chua's system, Fig. 8.12(a) shows the MSE versus the number of iterations during training, and Fig. 8.12(b) shows the equalization perfonnance of the trained equalizer. It can be seen that the equalizer completes its training after about 900 iterations, and the MSE of the equalized signal is approximately -76 dB when the SNR is equal to 14 dB. 159
8.3
Simulation Study
-10 -20 ~
co
-30
~
i;J -40 ::;: -50 -60
-70 -80L-__
o
~L-
____L -_ _ _ _L -_ _ _ _L -_ _ _ _L-~
200
400
600 Iterations
800
\,000
(a) -IO~------------------------------~
-20 -30
i:Q -40 ~
i;J -50 ::;:
-60
-70 -80L------L----__~~------~------~
o
5
10 SNR (dB) (b)
15
20
Figure 8.11 Equalization of the nonlinear channel for the Henon-map system in
(a) training stage, and (b) test stage
160
Chapter 8
Signal Reconstruction in Noisy Distorted Channels
-10
-30
$' ~ w..l
-40
Vl
:E -50 -60 -70 -80
0
200
400
600
800
1,000
1,200
Iterations (a)
-10 -20 -30
$'
~
-40
w..l
Vl
:E -50 -60 -70 -80
0
5
10
15
20
SNR (dB) (b)
Figure 8.12 Equalization of the nonlinear channel for the Chua's system in (a) training stage, and (b) test stage
8.4
Comparisons and Discussions
Firstly, we compare the proposed RNN-based equalizer with conventional LTEs. In our study, 13-tap and I5-tap LTEs (see Sec. 8.1.1) are applied to equalize Channels I and II. The results are summarized as follows. 161
8.4 Comparisons and Discnssions
Figs. 8.13 (a) and (b) show the MSE performance of the equalized signal versus the SNR of the channel for the communication systems based on the HEmon-map and the Chua's circuit, respectively. When the SNR is 14 dB, the MSE is -19.2 dB and -16.7 dB for channels I and II, respectively, for the Henon-map system. Here, the RNN-based equalizer outperforms these LTEs by 60 dB and 63.3 dB for channels I and II, respectively. Likewise, -12,---------------------------------~
- : 13 taps ...... : 15 taps
£'
""~-16
Channel II
if)
~
-17 -18 Channel I -19 -20
0
5
10
15 SNR(dB) (a)
20
25
30
-20 - : 13 taps ··0··: 15 taps ....... : 15 taps
-22 -24 -26
.
~ -28 ~
~ -30
Channel II -32 -34
-38
.......... ....... ...•.
Channell
-36
'
0
5
10
15 SNR (dB) (b)
20
25
30
Figure 8.13 MSE performance of LTEs versus SNR for (a) Henon-map, and (b) Chua's system. 13-tap equalizer (solid line) and I5-tap equalizer (dotted line)
162
Chapter 8 Signal Reconstruction in Noisy Distorted Channels
for the Chua's system, the MSE is -32.8 dB and -31.8 dB for channels I and II, respectively. In this case, the RNN-based equalizer outperforms the LTEs by 45.5 dB and 44.5 dB for Channels I and II, respectively. Also, the LTEs are found completely inadequate for equalizing the nonlinear channel, and results are omitted here. Secondly, we discuss the performance of the proposed equalizer based on the modified RNN in this chapter. As described in Sec. 8.1.3 the equalization task can be modelled as a general RNN. Basically, artificial neural networks (ANN) can perform complex mapping between its input and output space. Specifically, unlike static networks, for example, feedforward neural networks, RNN can perform temporally extended tasks, for which static networks have serious limitations [296]. The salient property of the RNN is that the outputs of the hidden units are fed back at every time step to provide an additional input. This recurrence enables the filtered data of the previous period to be used as an additional input in the current period. In other words, in each time period the network is subject to not only the new noisy inputs data but also the past history of all noisy inputs as well as their filtered counterparts. This additional information of filtered input history acts as an additional guidance to evaluate the current noisy input and its signal component. Therefore, RNN provides a suitable platform to realize the above equalization task. Here, as an example, we use a FNN without recurrent input, which is a special case of the one shown in Fig. 8.3 (i.e., N = 0 in the input layer), to realize the same equalization task for Channels I
and II. In the adaption stage, the back propagation algorithm [179] is used to train the equalizer. Figure 8.14 shows the MSE performance of the equalized signal versus the SNR of the channel. We can see by comparing Figs. 8.9 and 8.14 that when SNR is 14 dB, the RNN-based equalizer outperforms the static feedforward network by 48.9 dB and 51.8 dB for Channels I and II, respectively.
163
8.5
Summary -5,-------------------------------,
-10 -15
Channel II
.......~
r:o -20
....
~
w
~ -25
-30
-35 -40L-~~--~L---~~----~----~--~
o
5
10
15
20
25
30
SNR (dB)
Figure 8.14 The equalization performance by using an equalizer based on the feedforward neural network without recurrency using the back propagation learning algorithm, in which the Henon-map is employed in the transmitter
8.5
Summary
Channel equalization in chaos-based communication systems has been studied in this chapter. The main focus is the kind of channel distortion arising from linear delays as well as nonlinearity. The aim is to derive an effective equalization method such that a transmitted chaotic signal can be preserved with minimal distortion as it passes through a communication channel. Our design approach is based on the use of a RNN, which has the memory ability to combat dynamical changes, for "probing" the channel characteristics. The equalizer essentially consists of a modified RNN which incorporates a specific learning algorithm. The Henon-map system and Chua's system have been used as chaos generating systems in the simulation study, and three typical linear and nonlinear channels are considered. It has been found that the proposed equalizer can effectively "undo" the channel effects, permitting the chaotic signal to be reconstructed at the receiver.
164
Chapter 9 Chaotic Network Synchronization and Its Applications in Communications
As described in Chapters 1, 4 - 8, studies of chaos in nonlinear electrical circuits have shown that chaotic signals generated in these systems can potentially be used as carriers for information transmission. Thanks to the deterministic origin of chaos, two coupled chaotic systems can be se1fsynchronized [80]. Synchronization is a key process for coherent detection in the recovery of information signal encoded in the received chaotic signa1. However, all practical communication channels introduce signal distortions. As the result, the received chaotic signals do not precisely represent the transmitted signals. Channel noise, filtering, attenuation variability and other distortions in the channel corrupt the chaotic carrier and information signa1. The presence of these channel distortions significantly hamper the onset of identical synchronization of the chaotic systems. When signal distortions in the channel exceed a certain level, self-synchronizing fails resulting in failure of the communication link. To the best of our knowledge, the sensitivity to chaotic signal waveform distortions and the resulting problems in chaos synchronization in coherent chaos-based communication systems remain the major unresolved problems [280,297]. In this chapter, we describe a chaotic network synchronization scheme that can tackle the problem of multiple-system synchronization [298]. The proposed synchronization scheme enables the realization of a fast synchronization of multi-chaotic systems by using the OGY control method [21]. The proposed system is applied to a direct-sequence code-division-multipleaccess (DS/CDMA) spread-spectrum communication (SSC) system. This
9.1
Chaotic Network Synchronization
chapter is organized as follows. Based on Pecora-Carroll synchronization scheme and the OGY control method, a chaotic network synchronization scheme is realized by using the Lorenz chaotic system in Sec. 9.1. A DS/CDMA system using this chaotic network synchronization scheme is evaluated in Sec. 9.2.
9.1
Chaotic Network Synchronization
Let us consider two identical M-dimensional chaotic systems: x=F(x),
(9.1)
x' =F(x').
(9.2)
The attracting basin of Eq.(9.1) and Eq.(9.2) is a set Ae;;;;]RM. By decomposing the state vector of Eq.(9.2) into two parts w'=[x;,x;, .. ·,x~f and v' = [x~+l' <+2'"'' X~]T , we can rewrite Eq. (9.2) as
., F ( I ' ) w=ww,v, ., F( , ') V=vW'V.
(9.3)
Substituting w' with w in Eq. (9.3) leads to a nonautonomous system (9.4) where wet) is a driving signal (variable). Let us assume that the driven system Eq. (9.4) is asymptotically stable with respect to the driving signal wet). This implies that there exists a set Be;;;; ]RM-m, which is the region of asymptotic stability of Eq. (9.4), such that II v'(t, v'(t~» - v'et, v'(t~))I1 converges to zero as t ~ 00 for all initial conditions v' (t~), v' (t~) e;;;; B, where t~ and t~ are two initial time instants. The synchronization of the two systems depends on the conditional Lyapunov exponents (CLE) of the response system driven by the driving signal w, and the two systems are said to be synchronized when all the CLEs are negative [82].
166
Chapter 9
Chaotic Network Synchronization and Its Applications in Communications
It has been shown that the sensitivity of a chaotic system to small perturbations can be exploited to direct the trajectory from an initial state to a desired target state [21]. This is accomplished by stabilizing one of the unstable periodic trajectories embedded in a chaotic attractor, through applying some small time-dependent perturbations to the chaotic system. Based on the OGY method and the Pecora and Carroll's driving-response method, a chaotic network synchronization scheme can be realized, as detailed in the following section.
9.1.1
Network Synchronization
In the context of coherent multiple-access chaos-based communications, the problem of synchronization among a transmitter and a number of receivers that share the same channel is critical. Let us now consider the synchronization of a chaotic system in a transmitter with n - 1 identical chaotic systems in n - 1 receivers (users). This synchronization system is shown in Fig. 9.1, in which the three-dimensional chaotic systems (Xi, Yi, Zi) (i = 1, 2, ... , n - 1) are contained in the n - 1 receivers (response systems), and (xo, Yo, zo) is employed in the drive system (transmitter). The ith response system is composed of the ith (Xi, Yi, Zi) system and its duplicated system (x;, i, z;) which is driven by Yi. All response systems are driven by the driving signal xo. Here, we only consider
------i
I'---:""-r--~_
I I I I
I '---------"----,--L------.J I I ~~-'---~_ I
Figure 9.1 Network synchronization of three-dimensional chaotic systems
167
9.1
Chaotic Network Synchronization
the asymptotically stable solutions that lead to synchronized motion. Without loss of generality, we assume that the decomposition for the chaotic system meets the requirements for synchronization when the system is driven accordingly by variable x or y. In the process of network synchronization, the driving system is first controlled by using OGY method, which is different from the original Pecora and Carroll method. The synchronous signal Xo drives all response systems, enabling the parallel subsystems to enter the controllable regions. As a result, the OGY method can be employed to control chaotic motions in the regions. In this synchronization scheme, all response systems can escape from the driving-response relation and activate their own control mechanism by using the OGY method. Thus, all response systems and the drive system may synchronize quickly. The control for the chaotic system is derived in the next section.
9.1.2
Chaos Control
Consider the Lorenz system:
x = a(y -x), y
=
(r - z)x - y + pet),
(9.5)
z=xy-bz, where
(j
= 10, r = 28, b = 8/3, and pet) is an external perturbation. This choice
of parameters guarantees chaotic behavior [13]. When the Lorenz system is driven by x or y, the corresponding response subsystem is stable [82], thus meeting the requirements for network synchronization. Figure 9.2 is the period-l orbit computed from Eq. (9.5) (P(t) = 0) by the fourth-order Rungekutta algorithm with step-size h = 0.001. This periodic orbit has approximately 1,570 steps. An optimal fixed point of the periodic orbit on the Poincare section (zf=26.92l0), (xf' Yf)=(13.7836, 19.6659), is evaluated with the least squares fit. 168
Chapter 9
Chaotic Network Synchronization and Its Applications in Communications
20 10 >-<
0 -10 -20 -30
-20
-10
0
10
20
30
10
20
30
10
20
X (a)
50 40 N 30 20 10 0
-30
-20
-10
0 X (b)
50 40 N 30 20 10 -30 -20
-10
0 Y
30
(e)
Figure 9.2 Projections of the period-1 orbit of the Lorenz system that goes through
a fIxed point (13.7836,19.6659,26.9210), on planes (a) x - y, (b) x - z, and (c) y - z. Each period has approximately 1,570 steps 169
9.1
Chaotic Network Synchronization
On the Poincare section, Eq. (9.5)
IS
reduced to the following two-
dimensional discrete time equation: (9.6) where
9 is an unknown 2 x 2 transformation matrix. When pet) in Eq. (9.5)
is perturbed by a small !1p(t), a new fixed point (x~, Y~) can be approximated by
(9.7)
In the neighborhood of the fixed point, the vectors that are already pointing along the stable or unstable manifold keep their respective directions under the transformation of
g,
and these vectors are just the eigen-
vectors of g. The eigenvalues of 9 are denoted by As and Au, and their respective eigenvectors are es and eu . Another pair of contravariant basis vectors,
e; and <, satisfying the following equations e~
·es
= 0,
e; .e = 0, u
e~
. eu
=
1,
(9.8)
e: .e = 1, s
are also evaluated. Note that
(9.9)
where e: is the transpose of eu, e and e (i = 1, 2) are the elements of es and Sj
170
Uj
Chapter 9
Chaotic Network Synchronization and Its Applications in Commuuications
eu , respectively. Then, we have (9.10) and the next iterative point
«+1' Y~+l)
becomes (9.11 )
where (9.12) Ifwe denote
(9.13)
then
(9.14)
As point
«+1' Y~+l)
is required to move towards the stable manifold in
order to meet the control requirement, the vector (Ax~+1' L1Y~+l) should be adjusted towards the stable manifold. L1p(t) should then be adjusted to make the inner product ofthe vectors (Ax~+1' L1Y~+l) and e~ equal to zero, i.e., (9.15) 171
9.1
Chaotic Network Synchronization
This leads to
(9.16)
Equation (9.16) gives the variation of the perturbation parameter, pet), such that a chaotic orbit intersecting with the Poincare section at the next iteration step is close to the stable manifold of the fixed point.
9.1.3
Implementation of the Synchronization Scheme
Figure 9.3 shows the trajectories of a response subsystem in three different planes from the initial state (10.0, 10.0, 10.0) to the selected period-l orbit. The subsystem is driven by a synchronous driving signal Xo that has already entered the period-1 orbit, as shown in Fig. 9.2. The results indicate that this process takes approximately a duration of two periods, and all response subsystems will synchronize in less than three periods with their initial states varying in the range of x E (-20.0,30.0), Y
E
(-20.0,30.0) and
ZE
(-10.0,40.0). It is worth noting that these initial states actually overflow the
variation ranges of the state variables. Figure 9.4 shows the moving process of the same subsystem driven by the non-synchronous signal from the initial state (10.0, 10.0, 10.0) to the fixed point. This procedure takes a duration of 5.8 periods for synchronization. Obviously, the proposed synchronization scheme can efficiently achieve synchronization of multiple chaotic systems, and hence satisfies the requirements for synchronous multiple-access chaos communication. On the basis of the aforedescribed network synchronization, a DS/CDMA communication scheme will be realized the next section. 172
Chapter 9
Chaotic Network Synchronization and Its Applications in Communications
20 10 >-,
0 -10 -20 s: start
-30 -20 -10
0
10
20
30
X
(a)
50 40 N
30 20 10 s: start
0
-30 -20 -10
0
10
20
10
20
30
X (b)
50 40 N
30 20 10 s: start
0
-30 -20 -10
0
30
Y (c)
Figure 9.3 Trajectories of a response Lorenz subsystem driven by the syn-
chronized signal from an initial point (10.0, lD.O, lD.O) to the fixed point's neighbourhood 173
9.1
Chaotic Network Synchronization
20 ]0
,..,
0 -]0
-20 s: start
-30
-20 -]0
o
]0
20
30
X (a)
50 40 N
30 20 ]0 s: start oL--_J3o----J2o----]Jo--~0---]~0---2~0---3~0--
X (b)
50 40 N
30 20 ]0 s: start
OL---'----L--L--L---'------'------'----------' -30
-20 -]0
o
]0
20
30
y
(e)
Figure 9.4 Trajectories of a response Lorenz subsystem driven by the nonsynchronous signal of the driving system from the initial point (10.0, 10.0, 10.0) to the fixed point's neighbourhood
174
Chapter 9
9.2
Chaotic Network Synchronization and Its Applications in Communications
Implementation of Spread-Spectrum Communications
The remainder of this chapter is devoted to the realization of DS/CDMA spread-spectrum communication (SSC). One of the key processes of SSC lies in the synchronization of multiple signals, which has been studied in the previous section. Another key issue is the effective and reliable implementation of spreading and de spreading of the information source. Figure 9.5 shows a block diagram of a simplified DS/CDMA chaos-based communication system, in which bel) and b(2) .. · beL) are, respectively, the source and recovered bit streams of the ith user, PN i is the pseudo-random code (chaotic sequence) of the ith user, and wen) is the channel noise.
I
---------------------~
Figure 9.5 Block diagram of a simplified multi-user communication system
9.2.1
Encoding and Decoding
The fourth-order Runge-Kutta algorithm is used to integrate Eq. (9.5) (P(t) = 0), and the iterative step-size is h = 0.001. Figures 9.6 and 9.7 show, respectively, the auto correlation and cross-correlation ofx(t) computed from Eq. (9.5). The initial deviation of the x(t) series used to compute the cross-correlation is of the order of 10- 4 • It can be seen from Figs. 9.6 and 9.7 that the x(t) series has a characteristic similar to the pseudo-random sequence used in the 175
9.2
Implementation of Spread-Spectrum Communications
traditional
sse system. In addition, the extreme sensitivity of the system to
initial conditions can be advantageously used for assigning pseudo-random codes for different users. Apparently, the chaotic series is a good candidate for the pseudo-random code used in the traditional
sse
system. Also,
1.0.----------------------,
"
0.5
0
.~
vt:: 0
<.)
-
'--
0
S
0.0
-0.5'--_ _ _'--_ _ _-'----_ _ _...l.......-_ _ _.LJ 1,200 o 1,600 400 800 Time delay
Figure 9.6 Auto-correlation function of the x(n) series from the Lorenz Eq. (9.6) (P(t) = 0) with step-size h = 0.001 X
10-2 1.5,--------------------, 1.0
§
0.5
.~
-go
0
~
8 -0.5 -1.0
Time delay
Figure 9.7 Cross-correlation for two x(n) series of the Lorenz equation (P(t) = 0), with initial deviation of the order of 10-
176
4
Chapter 9
Chaotic Network Synchronization and Its Applications in Communications
Morantes et al. [299] have shown that the x(t) series of the Lorenz system meets the desired orthogonality for DS/CDMA. Based on the orthogonality of the series, an encoding and decoding scheme can be developed as follows. The pseudo-random sequence used in the traditional DS/CDMA system is now replaced by the x(n) series generated from the Lorenz Eq. (9.5) (P(t) = 0) generated from the fourth-order Runge-Kutta algorithm with step-size h = 0.001 (the sampling period is T= 50). For information security and
simplicity of the coding and decoding procedures, x(n) is normalized in the range of [ -1, 1]. Each bit symbol b(k) E {O, I} is encoded by N elements of the normalized x(n) sequence (n
=
1+ (k -l)N,. ··,kN). In other words, the
transmitted signal is given by sen) =
if b(k) = 1,
x(n) ' { -x(n),
(9.17)
if b(k) = O.
Suppose b'(k) = 2b(k) -1. Then, (9.18)
sen) = b'(k)x(n),
where n = 1 + (k -l)N,.· ',kN(k = 1,2,. .. ). The transmitted signal sen) is distorted by some additive external and / or internal interference wen). The received signal r(n) at a receiver is r(n) = sen) + wen).
(9.19)
Ifwe define the correlation function c(k, N) as kN
c(k,N) =
I
(9.20)
r(n)x(n),
l+(k-l)N
the recovered binary source information b" is b"(k) =
{I,
0,
if c(k,N) > 0 if c(k,N)
~
(9.21)
0
which is the decoding rule for the recovered information at the receiving end. The encoding and decoding rules are based on the fast decay of the 177
9.2
Implementation of Spread-Spectrum Communications
correlation function of the chaotic signals, which guarantees a robust recovery of the source signal against external and / or internal (user) interferences.
9.2.2
Test Results for Communications
To show the feasibility of the foregoing synchronization method under additive noise condition, we consider three source signals, as shown in Fig. 9.8, to simulate the CDMA SSC: (a) part waveform of Chinese vowel "a"; (b) the pixel array diagram of Chinese characters "SPREAD SPECTRUM COM- MUNICATIONS"; and (c) the standard Lenna image (256 x 256). In the following implementations, each receiver simultaneously detects and decodes its own source signal from the received signals. Also, each user makes use of its own local synchronized chaotic series to realize the despreading. Thus, network synchronization is a precondition for the communication to be established. Firstly, the de spreading ability of the test communication system is evaluated with the spread signals passing through an A WGN channel. The channel has an SNR of - 10.0 dB. Note that SNR is here defined as
(9.22) where sand ware the mean values of the transmitted signal and the noise signal. To evaluate the performance of the proposed communication system, the bit error probability is used here, which is defmed as
p. = P(b(k) *-
b"(k)). In
our study, the x(t) sequences generated from Eq. (9.5) by using different initial conditions are assigned to different users (code signatures), and are normalized in the range of [-1,1]. At N= 85, P e is equal to 1.356IxlO-4, 1.3565 X 10-4 and 1.3563 X 10-4 for speech, character and image signals, respectively. Then, the source signals can be recovered correctly from the x(n) sequences with a length of N = 100. The results indicate that the bit error 178
Chapter 9
Chaotic Network Synchronization and Its Applications in Communications
40,-------------------------------, 30
20
(a)
(b)
(c)
9.8 Three source signals: (a) part waveform of Chinese vowel "a", (b) pixel array diagram of Chinese characters "SPREAD-SPECTRUM COMMUNICATIONS" and (c) standard Lenna image (256 x256) 179
9.2
Implementation of Spread-Spectrum Communications
probability of the source information decreases with the length of the x(n) series used for encoding and decoding the information signal. To explore the relation between the length of the x(n) series and the bit error probability, the channel SNR is fixed at -5.32 dB. Figure 9.9 shows the bit error probability versus N. By direct data fitting, we find that the relation between P e and N satisfies ~ (N) =
where
fJ= 0.3326 and
& =
j3 exp(-&N),
(9.23)
0.1598. This indicates that bit error probability
decays exponentially with the length of chaotic encoding series for a fixed SNR of the channel. The reason is that the transmission procedure may be interpreted as a matched filter receiver with filtering coefficients being time varying in accordance with the chaotic time sequence. 10°,-------------------------------, -: For the image --: For the speech + : For the characters
10- 1
10-2 t:!.: bJJ
..3
10-3
10-4 IO- j
0
10
20
40
70
Code length
Figure 9.9 Bit error probability pe versus the length of chaotic encoding series for the recovered source information signals at SNR == -5.32 dB
From the geometrical viewpoint, the received sequence r(n) (n = 1+ (k -l)N, ... , kN) and the chaotic series may be considered as N-dimensional
vectors in an Euclidean space, i.e., r(k) = [r(1+(k-l)N), .. ·,r(kN)]T and x(k) = [x(1 + (k -l)N), ··,x(kN)f, respectively. The correlation function
equals the dot product of x(k) and r(k), i.e., <x(k) . r(k), which defines the 180
Chapter 9
Chaotic Network Synchronization and Its Applications in Communications
projection coefficient of the received vector with respect to the vector generated from the dynamical system. Accordingly we can express the correlation function as c(k,N) = b'(k)(x(k)· x(k) + (x(k)· w(k),
(9.24)
where w(k) is the corresponding noise vector. The source information signal can be recovered reliably if (x(k)· x(k) > 1(x(k)· w(k) I. In general, the higher the dimension N, the lower the probability that the vector describing the noise or interference also possesses strong components in the direction of x(k). This clearly shows that our proposed system can be controlled to achieve an interference-free operation and hence suitable for communications under very noisy conditions. Second, the effect caused by multipath interferences is studied. Here, the noise from the multipath interferences is modeled as a convolution with the Gaussian kernel, namely w(n) = fs(n_k)exp-(k
2
2
Ia
),
(9.25)
k~l
where a is a parameter that is used to control the amplitude of multipath interferences. At a = 75, the channel SNR is equal to -11.34 dB, and the bit error probabilities of the recovered speech, character and image signals are 1.7564 x 10-5 ,1.7493 X 10-5 and 1.7572xlO-5 , respectively, for N= 117. The source information can be reliably recovered by using chaotic series of N = 129 under this condition. This result can also be interpreted as follows. Due to the fast decay of the correlation of chaotic sequences, echoes of transmission signals become uncorrelated with the signals. This suggests that the system can be controlled to achieve an interference-free operation in a multipath environment.
9.3
Summary
The problem of multiple-access spread-spectrum single-point to multiplepoint communication system can be resolved by a chaotic network 181
9.3
Summary
synchronization scheme, which combines the Pecora-Carroll synchronization scheme and the OGY controlling method. Simulation results indicate that this synchronization scheme is applicable to fast synchronization of multiple chaotic systems. This is an essential condition for realizing spread-spectrum single-point to multiple-point communications. In the implementation of DS/CDMA systems, the traditional pseudo-random series can be replaced by the normalized sampled chaotic sequences generated from the Lorenz equation. Simulation results indicate that such a spread-spectrum communication system is effective and reliable in AWGN and mUltipath channels.
182
Chapter 10
Conclusions
The previous chapters have presented the study of the reconstruction of chaotic signals under realistic communication conditions. This study has been motivated by the requirements of chaotic signal processing in chaosbased communications. In this chapter, we conclude our discussion of signal reconstruction techniques by reiterating the key techniques that have been introduced in this book, along with some suggestions for further research.
10.1
Summary of Methods
The main techniques developed in this book for the reconstruction of chaotic signals can be categorized according to the channel conditions under which the signals have been transmitted. (1) Signal reconstruction in noisefree and distortionless channels
(2) Signal reconstruction from a filtering viewpoint (3) Signal reconstruction in noisy channels (4) Signal reconstruction in noisy distorted channels (5) Chaotic network synchronization for multiuser communications In its original form, the Taken's embedding theory expresses the result for reconstructing autonomous systems. One of our work is to extend the Takens' embedding theory to the reconstruction of continuous-time and discrete-time time-varying systems based on the observer approach, i.e., the systems under consideration are nonautonomous. In particular, we have studied the Lure system, which can be reconstructed in a state space whose dimension is equal to the degree of freedom of the system. Also, the information signals injected into the so-constructed chaos-based communication
10.1
Summary of Methods
systems in the absence of noise and distortion can be retrieved. The relevant conclusions have been demonstrated by using the Duffing's equations, the Chua's circuit and the Henon-map as chaos generators. When the observed samples are corrupted by channel noise, the Takens' theory is no longer applicable, at least in the strict sense. Another aspect of our work is to design an on-line adaptive demodulator for recovering message signals that are transmitted through chaotic carriers contaminated by additive white Gaussian noise. The message is injected into a Henon-mapbased transmitter system as variation of a bifurcation parameter. The demodulation process essentially aims to recover the embedded parameter variation. The essential component of the proposed demodulator is a radial-basisfunction neural network which incorporates an adaptive learning algorithm to track the dynamics of the Henon-map in the adaptive stage. The least square fit is used to estimate the message signal. The purpose of the adaptive learning algorithm is to adaptively configure the hidden layer size and to adjust the relevant parameters with an extended Kalman filter algorithm. The system is tested with square-wave, sine-wave, image, and speech signals serving as messages. Results have demonstrated that the demodulator is capable of performing the required demodulation task for a noisy communication channel. As a by-product, we have studied the information retrieval in a simplified CSK digital communication system employing the proposed tracker for non-coherent detection, in which the Chua's circuit has been used as a chaos generator. It has been shown that the digital source message can be retrieved by using the proposed demodulation algorithm. Channel equalization in chaos-based communication systems has been studied in this book. The main focus is the kind of channel distortion arising from linear delays as well as nonlinearity. The aim is to derive an effective equalization method such that a transmitted chaotic signal can be preserved with minimal distortion as it passes through a communication channel. We have shown that the equalization problem can be viewed as a special case of an approximation problem, solvable by a generalized recurrent neural network(RNN), and hence our design approach is based on the use of a RNN, 184
Chapter 10
Conclusions
which has the memory ability to combat dynamical channel distortion. The equalizer essentially consists of a modified RNN which incorporates a specific learning algorithm. The Henon-map system and Chua's system have been used as chaos generating systems in the simulation study, and three typical linear and nonlinear channels have been considered. It has been found that the proposed equalizer can effectively "undo" the channel effects, permitting the chaotic signal to be faithfully reconstructed at the receiver. Synchronization of chaotic systems can also be viewed as a kind of reconstruction method. In particular, we have attempted to tackle the problem of synchronization and to apply it in multiple access communication system. Specifically we have proposed a chaotic network synchronization scheme, which combines the Pecora-Carroll synchronization scheme and the OOY controlling method. The results indicate that this scheme can achieve fast synchronization of multiple chaotic systems, which is an essential condition for realizing multiple access communications. The method has been applied to a DS/CDMA system, in which normalized and sampled chaotic sequences generated from the Lorenz equations are used in lieu of the conventional pseudo-random series. Simulation results indicate that the proposed method is effective under practical channel conditions.
10.2
Further Research
Distinguishing chaos from highly nOISY corrupted signals or effectively reducing noise in chaotic signals is an important issue in chaos-based communication engineering. The reconstruction of chaotic signals is a basic problem in this area. Our work began from an artificial neural network approach, and has considered the reconstruction of chaotic signals under realistic communication conditions. In concluding, we attempt to list out a few possible directions in which the present work can be further extended. Basically, reconstruction is a process of capturing the dynamics of a system by building a model in a high-dimensional space. In this aspect, a 185
10.2
Further Research
unified framework and theory for the reconstruction of any time-varying chaotic system should be further developed, and the observer-based approach can be used as a starting point. The neural-network-based approach to reconstructing the chaotic signals can also be further investigated. Along this line, some further works include: (1) Applying other types of neural networks (NNs), such as multilayer perceptron NN, time delay NN, local feedback and globally feedforward network, to the signal reconstruction in a distorted channel; (2) Developing techniques to grow and prune the hidden layers; (3) Developing stopping algorithms for the growth of the network; (4) Exploring the use of other training algorithms. Furthermore, due to the deterministic nature of chaotic signals and the stochastic nature of noise, some optimization-based theories and techniques, nonlinear adaptive filtering theories and techniques may be considered for separating deterministic chaos from noise or reducing noise. Finally, the challenging problem of separating multiple chaotic signals in a highly noisy background still remains relatively unexplored. This problem is directly relevant to the multiple access capability of communication systems. On the application side, our work on chaos-based communications can be extended to multiple access applications. Although the basis functions generated from a chaotic oscillator are weakly correlated with each other, the correlation property is generally not as good as that of the pseudo-random sequence. Therefore, to optimally select the multiple access signals from a chaotic oscillator or to design chaotic generators for multiple access applications is still very much an open problem. In addition, the source! channel coding and the source! channel decoding in chaos-based communication systems are still rarely studied despite of their importance in achieving security and bandwidth efficiency. Finally, in the study of channel distortions, we have only considered linear and some simple nonlinear channels. However, in practical communication channels, the types of distortion may arise from a wide range of nonlinear phenomena, which make the problem much harder to tackle. More 186
Chapter 10
Conclusions
difficult jobs still lie ahead in tackling real systems with more general distortion characteristics, including bandwidth limitation, existence of coloured multiplicative noise, and complex fadings. Also, the development of a fast adaptive learning algorithm for the reconstruction of chaotic signals would enable the receiver to cope with time-varying distorted channels as well as allowing high data rate transmission in real-time chaos-based communication systems.
187
Bibliography
Takens, F. "Detecting strange attractors in turbulence." In: Dynamical Systems and Turbulence, D. Rand and I. Young (Ed.), pp. 366 - 381, Springer-Verlag, Berlin(1981) 2
Eckmann, J. P. and D. Ruelle. "Ergodic theory of chaos and strange attractors." Rev. Modern Phys .. Vol. 57, pp. 617 - 656(1985)
3
Parker, T. S. and L. O. Chua. "Chaos: a tutorial for engineers." Proc. IEEE. Vol. 75, pp. 982 - 1008(1987)
4
Oppenheim, A. V., G. W. Wornell, S. W. Isabelle and K. M. Cuomo. "Signal processing in the context of chaotic signals." Proc. IEEE ICCASP. Vol. 4, pp. 117 - 120 (1992)
5
Smale, S. "On the structure of manifolds." Amer. J Math .. Vol. 84, pp. 387 - 399 (1962)
6
Van der Pol, B. and J. Van der Mark. "Frequency demultiplication." Nature. Vol. 120, pp. 363 - 364(1927)
7
Levinson, N. "A second order differential equation with singular solutions." Annals ofMathematics. Vol. 50, No.1, pp. 127 - 153(1949)
8
Birkhoff, G. D. "Proof of the ergodic theorem." Proc. Nat!. A cad. Sci. USA. Vol. 17, pp.656-660(l931)
9
Birkhoff, G. D. "Nouvelles recherches sur les systemes dynamiques." Mem. Point. Acad. Sci. Novi. Lyncaei. Vol. 1, pp. 85 - 216(1935)
10
Chaundy, T. W. and E. Phillips. "The convergence of sequences defined by quadratic recurrence-formulae." Quart. J of Math. (Oxford). pp. 74 - 80(1936)
11
Cartwright, M. L. and J. E. Littlewood. "On non-linear differential equations of the
Bibliography
second order: 1. The equation ji - k(I-/)y + y = bAkcoS(At + a),k large." J. London Math. Soc .. Vol. 180 - 189(1945)
12
Melnikov, V. K. "On the stability of the center for time periodic perturbations." Trans Moscow Math. Soc .. Vol. 12, pp. 1- 57(1963)
13
Lorenz, E. N. "Deterministic nonperiodic flow." J. Atmospheric Sciences. Vol. 20, No.2, pp. 130 - 141(1963)
14
Cook, A. E. and P. H. Roberts. "The Rikitake two-disc dynamo system." Proc. Camb. Phil. Soc .. Vol. 68, pp. 547 - 569(1970)
15
Ruelle, D. and F. Takens. "On the nature of turbulence." Comm. Math. Phys.. Vol. 20, pp. 167 - 192(1971)
16
Li, T. Y. and J. A. Yorke. "Period three implies chaos." Amer. Math. Monthly. Vol. 82, pp. 985 - 992(1975)
17
May, R. M. "Simple mathematical models with very complicated dynamics." Nature. Vol. 261, pp. 459 - 467(1976)
18
Feigenbaum, M. J. "Quantitative universality for a class of nonlinear transformations." J. Stat. Phys .. Vol. 19, pp. 1- 25(1978)
19
Packard, N. H., J. P. Crutchfield, J. D. Farmer and R. S. Shaw. "Geometry from a time series," Phys. Rev. Lett.. Vol. 45, pp. 712 -716(1980)
20
Chua, L. 0., J. B. Yu and Y. Y. Yu. "Negative resistance devices." Int. J. Circ. Theory Appl.. Vol. 11, pp. 161- 186(1983)
21
Ott, E., C. Grebogi and J. A. Yorke. "Controlling chaos." Phys. Rev. Lett.. Vol. 64, pp. 1196 - 1199(1990)
22
Devaney, R. L. An Introduction to Chaotic Dynamical Systems. Addison-Wesley, New York(1987)
23
Kennedy, M. P. "Basic Concepts of Nonlinear Dynamics and Chaos." In: C. Toumazou (Ed.), Circuits and Systems Tutorials. pp. 289 - 313, IEEE Press, London (1994)
189
Bibliography
24
Wolf, A. "Quantity Chaos with Lyapunov Exponents," In: A. V. Holden (Ed.),
Chaos, Princeton University Press, New Jersey, pp. 273 - 290(1986) 25
Benettin, B., L. Galgani, A. Giorgilli and J. Strelcyn. "Lyapunov characteristic exponents for smooth dynamical systems and for hamiltonian systems: a method for computing all of them. Part I, II." Meccanica. Vol. 15, No. I, pp. 9 - 30(1980)
26
Wolf, A., J. B. Swift, H. L. Swinney and J. A. Vastano. "Detennining Lyapunov exponents from a time series." Physica D. Vol. 16, pp. 285 - 317(1985)
27
Eckmann, J. P., Kamphorst S. 0., Ruelle D. and Ciliberto S .. "Lyapunov exponents from time series." Phys. Rev. A. Vol. 34, pp. 4971 - 4979(1986)
28
Sano, M. and Y. Sawada. "Measurement of the Lyapunov spectrum from a chaotic time series." Phys. Rev. Lett.. Vol. 55, pp. 1082 - 1085(1985)
29
Peitgen, H., H. Jurgens and D. Saupe. Chaos and Fractals-New Frontiers of
Science . Springer-Verlag, New York(1996) 30
Banbrook, M., G. Ushaw and S. McLaughlin. "How to extract Lyapunov exponents from short and noisy time series." IEEE Trans. Signal Processing. Vol. 45, pp. 1378 - 1382(1997)
31
Geist, K., Parlitz
u.,
and Lauterbom W., "Comparison of different methods for
computing Lyapunov exponents." Prog. Thoeor. Phys., Vol. 83, (1990) 32
Rosenstein, M. T., Collins J. J., Luca C. J. D., "Reconstruction expansion as a geometry-based framework for choosing proper delay times." Physica D. Vol. 65, (1993)
33
Kantz, H. "A robust method to estimate the maximal Lyapunov exponent of a time series." Phys. Lett.
34
Kolmogorov, A. N. "A new metric invariant of transitive dynamical systems and automophisms in Lebesgue spaces." Doklady Akademii Nauk SSSR. Vol. 119, pp. 861 - 864(1958)
35
190
Sinai, Y. G. "On the concept of entropy of a dynamical system." Doklady Akademii Nauk SSSR. Vol. 124, pp. 768 -771(1959)
Bibliography
36
Sinai, Y. G. Introduction to Ergodic Theory. Princeton University Press, Princeton, New Jersey(l976)
37
Farmer, D. 1. "Information dimension and the probability structure of chaos." Zeitschriftfiir Natureforschung. Vol. 37, pp. 1304 - 1314(1982)
38
Farmer, D. J. "Dimension, Fractal Measure and Chaotic Dynamics." In: H. Haken (Ed.), Evolution of order and chaos, Springer-Verlag, Heidelberg, New York (1982)
39
Billingsley, P. Ergodic Theory and Information. John Wiley and Sons, New York (1965)
40
Ornstein, D. S. Ergodic Theory, Randomness and Dynamical Systems. Yale University, New Haven, Connecticut(1974)
41
Ruelle, D. Chaotic Evolution and Strange Attractors. Cambridge Press, Cambridge (1989)
42
Broomhead, D. S. and G. P. King. "Extracting qualitative dynamics from experimental data." Physica D. Vol. 20, pp. 217 - 236(1986)
43
Grassberger, P. and I. Procaccia. "Measuring the strangeness of strange attractors." Physica D. Vol. 9, pp. 189 - 207(1983)
44
Theiler, J. "Statistical precision of dimension estimators." Phys. Rev. A. Vol. 41 (1990)
45
Smith, R. L. "Estimating dimension in noisy chaotic time-series." J R. Statist. Soc. B. Vol. 54(1992)
46
Takens, F. "On the numerical determination of the dimension of an attractor." Braaksma, B. L. 1., H. W. Broer, and F. Takens, eds., In: Dynamical systems and bifurcations, Lecture notes in mathematics. Vol. 1125, Springer, Heidelberg (1985)
47
Cutler, C. D. "Some results on the behavior and estimation of the fractal dimensions of distributions on attractors." J Stat. Phys. Vol. 62(1991)
48
Culter, C. D. "A theory of correlation dimension for stationary time series." Philosoph. Trans. Royal Soc. London A. Vol. 348(1995)
191
Bibliography
49
50
Osborne, A. R. and A. Provenzale. "Finite correlation dimension for stochastic systems with power-law spectra." Physica D. Vol. 35(1989) Theiler, J. "Some comments on the correlation dimension of 11 fa noise." Phys. Lett. A. Vol. 155(1991)
51
Grassberger, P. "Do climatic attractors exist?" Nature. Vol. 323(1986)
52
Grassberger, P. "Evidence or climatic attractors: Grassberger replies." Nature. Vol. 326(1987)
53
Ruelle, D. "Deterministic chaos: The science and the fiction." Proc. R. Soc. London A. Vol. 427(1990)
54
Theiler, J. "Estimating fractal dimension." 1. Opt. Soc. Amer. A. Vol. 7(1990)
55
Kantz, H. and T. Schreiber. "Dimension estimates and physiological data." Chaos. Vol. 5(1995)
56
Grassberger, P. "Finite sample corrections to entropy and dimension estimates." Phys. Lett. A. Vol. 128(1988)
57
Provenzale, A., L. A. Smith, R. Vio, and G. Murante. "Distinguishing between low-dimensional dynamics and randomness in measured time series." Physica D. Vol. 58(1992)
58
M. B. Kennel and H. D. I. Abarbanel, "False neighbors and false strands: A reliable minimum embedding dimension algorithm." PhysRev E, Vol. 66, No.8, pp. 026209: 1- 18(1994)
59
Theiler, J. "Lacunarity in a best estimator of fractal dimension." Phys. Lett. A. Vol. 135(1988)
60
Diks, C. "Estimating invariants of noisy attractors." Phys. Rev. E, Vol. 53(1996)
61
Kugiumtzis, D. "Correction of the correlation dimension for noisy time series." Int. 1. Bifurcation and Chaos. Vol. 7, No.6, pp. 1283 - 1294(1997)
62
192
Oltmans, H. and P. J. T. Verheijen. "The influence of noise on power law scaling
Bibliography
functions and an algorithm for dimension estimations." Phys. Rev. E. Vol. 56(1997) 63
Schreiber, T. "Determination of the noise level of chaotic time series." Phys. Rev. E. Vol. 48(1993)
64
Schreiber, T. "Influence of Gaussian noise on the correlation exponent." Phys. Rev. E. Vol. 56(1997)
65
Olofsen, E., J. Degoede, and R. Heijungs. "A maximum likelihood approach to correlation dimension and entropy estimation." Bull. Math. Bioi. Vol. 54(1992)
66
Elger, C. and K. Lehnertz. "Seizure prediction by nonlinear time series analysis of brain electrical activity." European J Neuroscience. Vol. 10, No.2, pp. 786 -789 (1998)
67
Kaplan, 1. L. and J. A. Yorke. "Preturbulence: a regime observed in a fluid flow model of Lorenz." Comm. Math. Phys., pp. 67 - 93(1979)
68
Frederickson, P., J. L. Kaplan, E. D. Yorke and J. A. Yorke. "The Lyapunov dimension of strange attractors." J of Differential Equations. Vol. 49, pp. 185 - 207 (1983)
69
Grassberger, P. and 1. Procaccia. "Characterization of strange attractors," Phys. Rev. Lett.. Vol. 50, No.5, pp. 346 - 349(1983)
70
Schouten,1. C., F. Takens, C. M. Van den Bleek. "Maximumlikelihood estimation of the entropy of an attractor." Phys. Rev. E. Vol. 49, pp. 126 - 129(1994)
71
Pesin, Y. B. "Characteristic Lyapunov exponents and smooth ergodic theory." Uspeki Matamaticheskikh Nauk. Vol. 32, pp. 55 -71(1977)
72
Williams, G. P. Chaos Theory Tamed. Taylor and Francis, London(1997)
73
Kohda, T. and A. Tsuneda. "Statistics of chaotic binary sequences." IEEE Trans. Information Theory. Vol. 43, pp. 104 - 112(1997)
74
Dixon, R. C. Spread Spectrum Systems: with Commercial Applications (3rd ed.). John Wiley and Sons, New York(l994)
193
Bibliography
75
Kennedy, M. P. and G. Ko1umban. "Digital Communication Using Chaos." In: G. Chen (Ed.), Controlling Chaos and Bifurcations in Engineering Systems, pp. 477 - 500, CRC Press(1999)
76
Heidari-bateni, G. and C. D. McGillem. "A chaotic direct-sequence spreadspectrum communication system." IEEE Trans. Communication. Vol. 42, No. (2,3,4), pp. 1524 - 1527(1994)
77
Yamada, T., and H. Fujisaka. "Stability theory of synchronized motion in coup1edoscillator systems I." Prog. Theor. Phys .. Vol. 69, pp. 32 - 47(1983)
78
Yamada, T. and H. Fujisaka. "Stability theory of synchronized motion in coup1edoscillator systems II." Prog. Theor. Phys .. Vol. 70, pp. 1240 - 1248(1983)
79
Afraimovich, V. S., N. N. Verichev and M. I. Rabinovich. "Stochastic synchronization of oscillations in dissipative systems." Inv.
vuz. Rasiojiz. RPQAEC. Vol. 29,
pp. 795 - 803(1986) 80
Pecora, L. M., and T. L. Carroll. "Synchronization in chaotic systems." Phys. Rev.
Lett.. Vol. 64, pp. 821 - 824(1990) 81
Carroll, T. L. and L. M. Pecora. "Synchronizing chaotic circuits." IEEE Trans. Circ.
Syst.. Vol. 38, pp. 453 - 456(1991) 82
Pecora, L. M. and T. L. Carroll. "Driving systems with chaotic signals." Phys. Rev. A. Vol. 44, pp. 2374 - 2383(1991)
83
Carroll, T. L., and L. M. Pecora. "Cascading synchronized chaotic systems."
Physica D. Vol. 67, pp. 126 - 140(1993) 84
Pecora, L. M., T. L. Carroll, G. A. Johnson and D. J. Mar. "Fundamentals of synchronization in chaotic systems, concept, and applications." Chaos. Vol. 7, pp. 520 - 543(1997)
85
Cuomo, K. M., A. V. Oppenheim, and R. J. Barron. "Synchronization of Lorenzbased chaotic circuits with applications to communications." IEEE Trans. Circ. Syst.
II. Vol. 40, pp. 626 - 633(1993) 86 194
Cuomo, K. M., and A. V. Oppenheim. "Circuit implementation of synchronized
Bibliography
chaos with applications to communications." Phys. Rev. Lett.. Vol. 71, pp. 65 - 68 (1993) 87
Chen, G. and X. Dong. "Controlling Chua's circuit." J. Circ. Syst. Computers. Vol. 3, pp. 139 - 149(1993)
88
Halle, K. S., C. W. Wu, M. Itoh and L.
o.
Chua. "Spread spectrum communication
through modulation of chaos." Int. J. Bifurcation Chaos. Vol. 3, pp. 469 - 477 (1993) 89
John, J. K. and R. E. Amritkar. "Synchronization of unstable orbits using adaptive control." Phys. Rev. E. Vol. 49, pp. 4843 - 4848(1994)
90
Ru1kov, N. F., M. M. Sushchik, L. S. Tsimrin and H. D. I. Abarbanel, "Generalized synchronization of chaos in directionally coupled chaotic systems." Phys. Rev. E. Vol. 51,pp. 980-994(1995)
91
Kolumbcin, G., H. Dedieu, J. Schweizer, J. Ennitis and B. Vizvciri, "Perfonnance evaluation and comparison of chaotic communication systems." Proc. of the Fourth Int. Workshop on Nonlinear Dynamics of Electronic Systems. pp. 105 - 110, Sevilla
(1996) 92
Kolumban, G., M. P. Kennedy and L.
o. Chua. "The role of synchronization in
digital communications using chaos-Part II: Chaotic modulation and chaotic synchronization." IEEE Trans. Circ. Syst. 1. Vol. 45, pp. 1129 - 1140(1998) 93
Kocarev, L. J., K. S. Halle, K. Eckert, L. o. Chua and D. Parlitz. "Experimental demonstration of secure communications via chaotic synchronization." Int. J. Bifurcation Chaos. Vol. 2, pp. 709 -713(1992)
94
Milanovic, V. and M. Z. Zaghloul. "Improved masking algorithm for chaotic communications systems." lEE Electronics Lett.. Vol. 32, pp. 11 - 12(1996)
95
Feldmann, D., M. Hasler and W. Schwarz. "Communication by chaotic signal: the inverse systems approach." Int. J. Circ. Theory Appl.. Vol. 24, pp. 551- 579(1996)
96
Hoh, M., H. Murakami and L.
o. Chua. "Communication system via chaotic
modulations." IEICE Trans. Fundament.. Vol. E77-A, pp. 1000 - 1006(1994)
195
Bibliography
97
Kocarev, L. J., and U. Parlitz. "General approach for chaotic synchronization with applications to communication." Phys. Rev. Lett.. Vol. 74, pp. 5028 - 5031(1995)
98
Parlitz, U. L. J. Kocarev, T. Stojanovski and H. Preckel. "Encoding messages using chaotic synchronization." Phys. Rev. E. Vol. 53, pp. 4351 - 4361(1996)
99
Leung, H. and J. Lam. "Design of demodulator for the chaotic modulation communication system." IEEE Trans. Circ. Syst. I. Vol. 44, pp. 262 - 267(1997)
100
Feng, J. C. and C. K. Tse. "Observer-based tracking and identification of chaotic systems with application to chaotic communication through noisy channel." Proc. of Int. Symposium on Nonlinear Theory and Its Applications (NOLTA'OO). pp. 91 - 94(2000)
101
Feng, J. C. and C. K. Tse. "On-line adaptive chaotic demodulator based on radialbasis-function neural networks." Phys. Rev. E. Vol. 63, pp. 026202: 1 - 10(2001)
102
Parlitz, U., L.
o. Chua, L.
J. Kocarev, K. S. Halle and A. Shang. "Transmission of
digital signals by chaotic synchronization." Int. J. Bifurcation Chaos. Vol. 2, pp. 973 - 977(1992) 103
Heidari-bateni, G. and C. D. McGillem. "Chaotic sequences for spread spectrum: an alternative to PN-sequences." Proc. of IEEE Int. Con! on selected Topics in
Wreless Communications. pp. 437 - 440(1992) 104
Dedieu, H., M. P. Kennedy and M. Hasler. "Chaos shift keying: modulation and demodulation of a chaotic carrier using self-synchronizing Chua's circuits." IEEE
Trans. Circ. Syst. If. Vol. 40, pp. 634 - 642(1993) 105
Ogorzalek, M. J., "Taming chaos-Part I : Synchronization." IEEE Trans. Circ. Syst. I. Vol. 40, pp. 693 - 699(1993)
106
Parlitz, U. and S. Ergezinger. "Robust communication based chaotic spreading sequences." Phys. Lett. A. Vol. 188, pp. 146-150(1994)
107
Boehme, F., U. Feldmann, W. Schwarz, and A. Bauer. "Information transmission by chaotizing." Proc. of the second Int. Workshop on Nonlinear Dynamics of
Electronic Systems. pp. 163 - 168, Crakow, Po1and(1994)
196
Bibliography
108
Mazzini, G., G. Setti and R. Rovatti. "Chaotic complex spreading sequences for asynchronous DS-CDMA-Part I : Sysytem modelling and Results." IEEE Trans
Circ. and Syst. I. Vol. 44, pp. 937 - 947(1997) 109
Kolumban, G., M. P. Kennedy and L.
o.
Chua. "The role of synchronization in
digital communications using chaos-Part I : fundamentals of digital communications." IEEE Trans. Circ. Syst. I. Vol. 44, pp. 927 - 936(1997) 110
Rovatti, R., G. Setti and G. Mazzini. "Chaotic complex spreading sequences for asynchronous DS-CDMA-Part II: Some theoretical performance bounds." IEEE
Trans Circ. and Syst. I. Vol. 45, pp. 496 - 506(1998) III
Feng, J.
c., Y.
Yu and S. Zhou. "Chaos-based spread spectrum communications."
J. of China Institute of communications. Vol. 9, No.6, pp. 76- 83(1998)
112
Umeno, K. and K. I. Kitayama. "Improvement of SNR with chaotic spreading spectrum sequences for CDMA." Proc. of IEEE Information Theory Workshop. South Africa, June(1999)
113
Kolumban, G., B. Vizvari, W. Schwarz and A. Abel. "Differential chaos shift keying: A robust coding for chaotic communication." Proc. of the Fourth Int.
Workshop on Nonlinear Dynamics ofElectronic Systems. pp. 87 - 92, Sevilla (1996) 114
Tse, C. K., K. Y. Cheong, F. C. M. Lau and S. F. Hau. "An approach for CSK detection based on return maps." Proc. of Int. Symp. on Nonlinear Theory and Its
Appl.. pp. 637 - 640(2001) 115
Kennedy, M. P., R. Rovatti and G. Setti. Chaotic Electronics in Telecommunications. CRC Press, Boca Raton, Florida (2000)
116
Short, K. M., "Steps towards unmasking secure communications." Int. J Bifurcation
Chaos. Vol. 4, pp. 957 - 977(1994) 117
Perez, G. and H. A. Cerdeira. "Extracting messages masked by chaos." Phys. Lett. A. Vol. 74, pp. 1970-1973(1995)
118
Short, K. M., and A. T. Parker. "Unmasking a hyperchaotic communication scheme." Phys. Rev. E. Vol. 58, pp. 1159 - 1162(1998)
197
Bihliography
119
Ott, E., T. Sauer and J. A. Yorke. Copying with Chaos: Analysis of Chaotic Data
and the Exploration of Chaotic Systems. Wiley Series in Nonlinear Science, John Wiley and Sons, New York (1994) 120
Sauer, T., J. A. Yorke and M. Casdagli. "Embedology." J Stat. Phys.. Vol. 65, pp. 579 - 616(1991)
121
Fraser, A. M. and H. L. Swinney. "Independent coordinates for strange attractors from mutual information." Phys. Rev. A. Vol. 33, pp. 1134- 1140(1986)
122
Shannon, C. E. and W. Weaver. "A mathematical theory of communications." Bell
Systems Technical Journal. Vol. 27, pp. 379 - 423, pp. 623 - 656(1948) 123
Whitney, H. "Differentiable manifolds." Annals ofMathemitics. Vol. 37, pp. 645 - 680 (1936)
124
Ogorzalek, M. J. Chaos and Complexity in Nonlinear Electronic Circuits. World Scientific, Singapore (1997)
125
Liebert, W. and H. G. Schuster. "Proper choice of the time-delay for the analysis of chaotic time series." Phys. Lett. A. Vol. 142, pp. 107 - 111(1989)
126
Schuster, H. G. "Deterministic chaos". VCH, Verlagsgessellschaft, mbf Weinheim, Germany (1988)
127
Fraser, A. M. "Information and entropy in strange attractors." IEEE Trans.
Information Theory. Vol. 35, pp. 245 - 262(1989) 128
Pineda, F. J. and J. C. Sommerer. "A Fast Algorithm for Estimating the Generalized Dimension and Choosing Time Delays." In: A. S. Weigend and N. A. Gershenfeld (Ed.), Time Series Prediction: Forecasting the Future and Unders-
tanding the Past, pp. 367 - 385(1994) 129
Haykin, S. and S. Puthusserypady. Chaotic Dynamics of Sea Clutter. John Wiley and Sons, New York(1999)
130
Abarbanel, H. D. I. and M. B. Kennel. "Local false nearest neighbors and dynamical dimensions from observed chaotic data." Phys. Rev. E. Vol. 47, pp. 3057 - 3068(1993)
198
Bibliography
131
Abarbanel, H. D. 1. Analysis of Observed Chaotic Data. Springerverlag, New York(1996)
132
Kennel, M. B., R. Brown and H. D. 1. Abarbanel. "Determining embedding dimension for phase-space reconstruction using a geometrical construction." Phys. Rev. E. Vol. 45, pp. 3403 - 3411(1992)
133
Liebert, W., K. Pawelzik and H. G. Schuster. "Optimal embedding of chaotic attractors from topological considerations." Europhysics Lett.. Vol. 14, pp. 521 - 526 (1991)
134
Matsumoto, T. "A chaotic attractor from Chua's circuit." IEEE Trans. Circ. Syst.. Vol. 31, pp. 1055 - 1058(1984)
135
Zhong, G. Q., and F. Ayrom. "Experimental confirmation of chaos from Chua's circuit." Int. J Circ. Theory Appl.. Vol. 13, pp. 93 - 98(1985)
136
Matsumoto, T., L. o. Chua and M. Komuro. "The double scroll." IEEE Trans. Circ. Syst.. Vol. 32, pp. 798 - 817(1985)
137
Chua, L. 0., M. Komuro and T. Matsumoto. "The double scroll family." IEEE
Trans. Circ. Syst.. Vol. 33, pp. 1073 - 1117(1986) 138
Madan (Ed.), R. N., Chua's Circuit: A Paradigm for Chaos. World Scientific, Singapore (1993)
139
Chialina, S., M. Hasler and A. Premoli. "Fast and accurate calculation of Lyapunov exponents for piece-wise linear systems." Int. J Bifurcation Chaos. Vol. 4, pp. 127 - 136(1994)
140
Aguirre, L. A., and S. A. Billings. "Discrete reconstruction of strange attractors of Chua's circuit." Int. J Bifurcation Chaos, Vol. 4, pp. 853 - 864(1994)
141
Schreiber, T. "Interdisciplinary application of nonlinear time series methods." Physics Reports. Vol. 308, No.1, pp. 1- 64(1999)
142
Ding, M., C. Grebogi, E. Ott, T. Sauer, and J. A. Yorke. "Plateauonset for correlation dimension: When does it occur?" Phys. Rev. Lett. Vol. 70(1993)
199
Bibliography
143
Malinetskii, G. G., A. B. Potapov, A. 1. Rakhmanov, and E. B. Rodichev. "Limitations of delay reconstruction for chaotic systems with a broad spectrum."
Phys. Lett. A. Vol. 179(1993) 144
Fraser, A. M. and H. L. Swinney. "Independant coordinates for strange attractors from mutual information." Phys. Rev. A. Vol. 33(1986)
145
Liebert, W., H. G. Schuster. "Proper choice of the time delays for the analysis of chaotic time series." Phys. Lett. A. Vol. 142(1989)
146
Liebert, W., K. Pawelzik, and H. G. Schuster. "Optimal embeddings of chaotic attractors from topological considerations." Europhys. Lett. Vol. 14(1991)
147
Kennel, M. B. and S. Isabelle. "Method to distinguish possible chaos from colored noise and to determine embedding parameters." Phys. Rev. A. Vol. 46(1992)
148
Buzug, T. and G. Pfister. "Comparison of algorithms calculating optimal parameters for delay time coordinates." Physico D. Vol. 58(1992)
149
Buzug, T., T. Reimers, and G. Pfister. "Optimal reconstruction of strange attractors from purely geometrical arguments." Europhys. Lett. Vol. 13(1990)
150
Kugiumtzis, D. "State space reconstruction parameters in the analysis of chaotic time series - the role of the time window length." PhYSico D. Vol. 95(1996)
151
Cencini, M., M. Falcioni, E. Olbrich, H. Kantz and A. Vulpiani. "Chaos or noise: Difficulties of a distinction." Phys. Rev. E. Vol. 62, pp. 427 - 437(2000)
152
Powell, M. J. D. "Radial basis functions for multivariable interpolation: A review."
Proc. IMA Con! Algorithms for the approximation offunctions and data, RMCS, Shrivenham(1985) 153
Broornhead, D. and Lowe, D. "Multivariable function interpolation and adaptive networks," Complex Systems, Vol. 2(1988)
154
Casdagli, M. "Nonlinear prediction of chaotic time series," PhYSico D, Vol. 35(1989)
155
Smith, L. A. "Identification and prediction oflow-dimensional dynamics," Physico D , Vol. 58(1992)
200
Bibliography
156
Breiman, L., Friedman, J. H., Olshen, R. A. and Stone, C. J. Classification and regression trees. Chapman and Hall, New York (1993)
157
Akaike, H. "A new look at the statistical model identification," IEEE Trans. Automat. Contr. Vol. 19(1974)
158
Rissanen, J. "Consistent order estimates of autoregressive processes by shortest description of data." In: Jacobs, O. et al., eds., Analysis and optimization of stochastic systems. Academic Press, New York (1980)
159
Theiler, J. and Prichard, D. "Using 'Surrogate Surrogate Data' to calibrate the actual rate of false positives in tests for nonlinearity in time-series." Fields Inst. Comm. Vol. 11(1997)
160
Kantz, H. and Schreiber, T. Nonlinear time series analysis. Cambridge University Press, Cambridge(1997)
161
Bowen, R. "Symbolic dynamics for hyperbolic flows." Amer. J. Math., Vol. 95 (1973)
162
Christiansen, F. and Politi, A. "Symbolic encoding in symplectic maps." Nonlinearity. Vol. 9(1996)
163
Grassberger, P. and Kantz, H. "Generating partitions for the dissipative Henon map." Phys. Lett. A. Vol. 113(1985)
164
Herzel, H. "Compelxity of symbol sequences." Syst. Anal. ModI. Simul., Vol. 5 (1988)
165
Ebeling, W., Poschel, Th. and Albrecht, K. F. "Entropy, transinformation and word distribution of information-carrying sequences." Int. J. Bifurcation Chaos. Vol. 5 (1995)
166
Schfumann, T. and Grassberger, P. "Entropy estimation of symbol sequences." Chaos. Vol. 6(1996)
167
Hao, B. L. Elementary symbolic dynamics. World Scientific, Singapore(1989)
201
Bibliography
168
Wackerbauer, R., Witt, A., Atmanspacher, H., Kurths, J. and Scheingraber, H. "A comparative classification of complexity measures." Chaos, Solitons and Fractals. Vol. 4(1994)
169
Kurths, J., Voss, A, Saparin, P., Witt, A, Kleiner, H. J. and Wessel, N. "Complexity measures for the analysis of heart rate variability." Chaos. Vol. 5(1995)
170
Brock, W. A, Dechert, W. D., Scheinkman, J. A and LeBaron, B. Test for
independence based on the correlation dimension. University of Wisconsin Press, Madison( 198 8) 171
Brock, W. A, Hseih, D. A and LeBaron, B. Nonlinear dynamics, chaos, and instability:
Statistical theory and economic evidence. MIT Press, Cambridge, MA(1991) 172
Sugihara, G., Grenfell, B. and May, R. M. "Distinguishing error from chaos in ecological time series." Phil. Trans. R. Soc. Lond. B. Vol. 330(1990)
173
Barahona, M. and Poon, C. S. "Detection of nonlinear dynamics in short, noisy time series." Nature. Vol. 381(1996)
174
Schreiber, T. and Schmitz, A "Discrimination power of measures for nonlinearity in a time series." Phys. Rev. E. Vol. 55(1997)
175
Levine, M. Man and Mechine Vision. McGraw-Hill, New York(1985)
176
Aleksander, I. and H. Morton. An Introduction to Neural Computing. Chapman and Hall, London(l990)
177
Widrow, B. and S. D. Steams. Adaptive Signal Processing. Prentice-Hall, New Jersey(l985)
178 179
Haykin, S. Adaptive Filter Theory (3rd ed.). Prentice Hall, New Jersey(1996) Haykin, S. Neural Networks: A Comprehensive Foundation, Prentice Hall, New Jersey( 1994)
180
181
Grossberg, S. Neural Networks and Natural Intelligence. MIT press, MA, Cambridge(l988) Micchelli, C. A "Interpolation of scatter data: distance matrices and conditionally positive definite functions." Constructive Approximation. Vol. 2, pp. 11- 22(1986)
202
Bibliography
182
Dyn, N. "Interpolation of Scattered Data by Radial Functions." In: C. K. Chui, L. L. Schumaker and F. 1. Uteras (Ed.), Topics in Multivariate Approximation. Academic Press, Orlando, FL(1987)
183
Light, W. A. "Some Aspects of Radial Basis Function Approximation." In: S. P. Singh (Ed.), Approximation Theory, Spline Functions and Applications. NATO ASI Series, Vol. 256, pp. 163 - 190, Kluwer Academic Publishers, Boston(1992)
184 Hardy, R. L. "Multiquadric equations of topography and other irregular surfaces." J. Geophysics Research. Vol. 76, pp. 1905 - 1915(1971) 185
Broomhead, D. and D. Lowe. "Mutivariable functional interpolation and adaptive networks." Complex System. Vol. 2, pp. 321 - 355(1988)
186
Poggio, T. and F. Girosi. "Networks for approximation and learning." Proc. IEEE. Vol. 78, pp. 1481- 1497(1990)
187
Moody, J. and C. J. Darken. "Fast learning in network of locally-turned processing units." Neural Computation. Vol. 1, pp. 281 - 294(1989)
188
Chen, S., C. F. N. Cowman and P. Grant. "Orthogonal least squares learning algorithm for radial basis function networks." IEEE Trans. Neural Networks. Vol. 2,pp.302-309(1991)
189
Lee, S. and R. M. Kil. "A gaussian potential function network with hierachically self-organizing learning." Neural Networks.Vol. 4, pp. 207 - 224(1991)
190
Musavin, M., W. Ahmed and K. Chan. "On training of radial basis function classifiers." Neural Networks. Vol. 5, pp. 595 - 603(1992)
191
Chen, S., S. Billings and P. Grants. "Recursive hybrid algorithm for nonlinear system identification using radial basis function networks." Int. J. of Control. Vol. 55, pp. 1051- 1070(1992)
192
Platt, J. "A resource allocating network for function interpolation." Neural Computation. Vol. 3, pp. 213 - 225(1991)
193
Kadirkamanathan, V. and M. Niranjan. "A function estimation approach to sequential learning with neural network." Neural Computation. Vol. 5, pp. 954 - 975(1993)
203
Bibliography
194
Hecht-Nielsen, R. Neurocomputing. Addison-Wesley, Reading, PA(1990)
195
Fausett, L. Fundamentals of Neural Networks. Prentice Hall, Englewood Cliffs, New Jersey(1994)
196
Rumelhart, D. E., G. E. Hinton and R. J. Williams. "Learning Internal Representations by Error Propagation." In: D. E. Rumelhart and J. L. McClelland (Ed.), Parallel Distributed Processing: Explorations in the Microstructure of Cognition, MIT Press, Cambridge, Vol. 45(1986)
197
Saad, E. W., T. P. Caudell and D. C. Wunsch II. "Predictive head tracking for virtual reality." Proc. of the Int. Joint Conference on Neural Networks(1999)
198
Giles, C. L., S. Lawrence and A. C. Tsoi. "Rule inference for financial prediction using recurrent neural networks." Proc. of IEEE Conference on Computational Intelligence for Financial Engineering. IEEE Press(1997)
199
Costa, M., E. Pasero, F. Piglione and D. Radasanu. "Short term load forecasting using a synchronously operated recurrent neural network." Proc. of the Int. Joint Conference on Neural Networks(1987)
200
Coulibay, P., F. Anctil and J. Rousselle. "Real-time short-term water inflows forecasting using recurrent neural networks." Proc. of the Int. Joint Conference on Neural Networks(1999)
201
Back, A. D. and A. C. Tsoi. "FIR and IIR synapses, a new neural network architecture for time series modelling." Neural Computation. Vol. 3, pp. 337 - 350 (1991)
202
203
Frasconi, P., M. Gori and G. Soda. "Local feedback multilayered networks." Neural Computation. Vol. 4, pp. 120 - 130(1992) Tsoi, A. C. and A. D. Back. "Locally recurrent globally feedforward networks, a critical review of architectures." IEEE Trans. on Neural Networks. Vol. 5, pp. 229 - 239(1994)
204
204
Jordan, M. I. "Supervised learning and systems with excess degrees of freedom." COINS Technical Report. Massachusetts Institute of Technology. pp. 88 - 27, May (1988)
Bibliography
205
Elman, J. L. "Distributed representations: simple recurrent networks and grammatical structure." Machine Learning. Vol. 7, pp. 195 - 225(1991)
206
Williams, R. J. and D. Zipser. "A learning algorithm for continually running fully recurrent neural networks." Neural Computation. Vol. 1, pp. 270 - 280(1989)
207
Robinson, A. J. Dynamic Error Propagation Networks. PhD thesis, Cambridge University Engineering Department, Cambridge, England(1989)
208
Schmidhuber, J. H. "A fixed size storage 0 [N 3] time complexity learning algorithm for fully recurrent continually running network." Neural Computation. Vol. 4, pp. 243 - 248(1992)
209
Zipser, D. "Subgrouping Reduces Complexity and Speeds up Learning in Recurrent Networks." In: Advances in Neural Information Processing Systems. Vol. 2, pp. 638 - 641(1990)
210
Kailath, T. Linear Systems. Prentice Hall, New Jersey(1980)
211
Kalman, R. E. "A new approach to linear filtering and prediction problems." Trans.
of the ASME-D, Journal of Basic Engineering. Vol. 82, pp. 34 - 45(1960) 212
McGarty, T. P. Stochastic Systems and State Estimation. John Wiley & Sons Inc., New York(1973)
213
Maybeck, P. S. Stochastic Models, Estimation and Control, Vol. 1. Academic Press, New York(1979)
214
Maybeck, P. S. Stochastic Models, Estimation, and Control, Vol. 2. Academic Press, New York(1982)
215
Maybeck, P. S. Stochastic Models, Estimation, and Control, Vol. 3. Academic Press, New York(1982)
216
Krener, A. J. and Isidori, A. "Linearization by output injection and nonlinear observers." System Control Letters. Vol. 3, pp. 47 - 52(1983)
217
Trees, H. L. V. Detection, Estimation, and Modulation Theory - Part I. John
205
Bibliography
Wiley & Sons Inc. (1968) 218
Jaswinski, A. H. Stochastic Processes and Filtering Theory. Academic Press, New York(1970)
219
Special issue on applications of Kalman filtering. IEEE Trans. On Automatic Control. Vol. 28, No. 3(1983)
220
Vidyasagar, M. Nonlinear Systems Analysis (2nd Ed.). Prentice-Hall, Englewood Cliffs, NJ.(1993)
221
Gelb, A. Applied Optimal Estimation. MIT Press, Cambridge, MA(1974)
222
Goodwin, G. C. and Sin, K. S. Adaptive Filtering Prediction and Control, Prentice-Hall, London, U.K(1984)
223
Grewal, M. S. and Andrews, A. P. Kalman Filtering: Theory and practice Using MATLAB (2nd Ed.). John Wi ely & Sons Inc., New York(200l)
224
Jones, RH. Longitudinal Data with Serial Correlation: A State Space Approach. London: Chapman & Hall(1993)
225
Brown, R G. and Hwang, P. Y. C. Introduction to Random Signals and Applied Kalman Filtering (2nd Ed.). John Wiley & Sons, Inc.(1992)
226
Ham, F. C. and Brown, R G. "Observability, eigenvalues, and Kalman filtering." IEEE Trans. Aerospace and Electronic Syst. Vol. 19, No.2, pp. 269 - 273(1983)
227
Harvey, A. C. Forecasting, structural time series models and the Bibliography Kalman Filter. Cambridge University Press, London(1989)
228
Ansley, C. F. and Kohn, R. "Estimation, filtering and smoothing in state space models with incompletely specified initial conditions." Annals of Statistics. Vol. 13, pp. 1286 - 1316(1985)
229
De J. P. "Smoothing and interpolation with the state space model." J. American Statistical Association. Vol. 85, pp. 1085 - 1088(1989)
206
Bibliography
230
De J. P. "The diffuse Kalman filter." Annals of Statistics. Vol. 19, pp. 1073 - 1083 (1991)
231
Durbin J. and Koopman, S. J. Time Series Analysis by State Space Methods. Oxford University Press, London(200 1)
232
Reif, K., Gunther, S., Yaz, E. and Unbehauen, R. "Stochastic stability of the discrete-time extended Kalman filter." IEEE Trans. On Automatic Control. Vol. 44, pp.714-728(1999)
233
Chui, C. K. and Chen, G. Kalman Filtering with Real-time Applications. Springer-Verlag, New York(1984)
234
Grewal, M. S. and Andrews, A. P. Kalman Filtering: Theory and Practice. Prentice-Hall, Englewood Cliffs, NJ(1993)
235
Baras, J. S., Bensoussan, A. and James, M. R. "Dynamic observers as asymptotic limits of recursive filters: Special cases." SIAM J. Appl. Math. Vol. 48, No.5, pp. 1147 - 1158(1988)
236
Bestle, D., Zeitz, M. "Canonical form observer design for nonlinear time-variable systems." Int. J. Control. Vol. 38, pp. 419 - 431(1983)
237
Boutayeb, M. and Darouach, M. "Recursive identification method for Hammerstein modelextension to the nonlinear MlSO case." Control Theory Advanced Technology. Vol. 10, No.1, pp. 57 -72(1994)
238
Boutayeb, M. and Darouach, M. "Recursive identification method for MISO Wiener-Hammerstein model." IEEE Trans. on Automatic Control. Vol. 40, No.2, pp. 287 - 291(1995)
239
Deyst, J. J. and Price, C. F. "Conditions for asymptotic stability of the discrete minimum-variance linear estimator." IEEE Trans. On Automatic Control, pp. 702 - 705(1968)
240
Krener, A. J. and Respondek, W. "Nonlinear observers with 1inearizable error dynamics." SIAM J. Contr. Optim., Vol. 23, pp. 197 - 216(1985)
207
Bibliography
241
Kushner, H. J. "Approximations to optimal nonlinear filters." IEEE Trans. on Automatic Control. Vol. 12, No.5, pp. 546 - 556(1967)
242
Ljung, L. "Asymptotic behavior of the extended Kalman filter as a parameter estimator for linear systems." IEEE Trans. on Automatic Control. Vol. 24, pp. 36 - 50(1979)
243
Mehra, R. K. "A comparison of several nonlinear filters for reentry vehicle tracking." IEEE Trans. on Automatic Control. Vol. 16, pp. 307 - 319(1971)
244
Nijmeijer, H. "Observability of autonomous discrete-time nonlinear systems: A geometric approach." Int. J. Control. Vol. 36, pp. 867 - 874(1982)
245
Nelson. Song, Y. and Grizzle, J. W. "The extended Kalman filter as a local asymptotic observer for nonlinear discrete-time systems." In: Proc. Amer. Contr. Can! pp. 3365 - 3369(1992)
246
Ciccarella, G., Mora, M. D. and Germani, A. "A robust observer for discrete time nonlinear systems." System Control Letters. Vol. 24, pp. 291 - 300(1995)
247
Moraal, P. E. and Grizzle, J. W. "Observer design for nonlinear systems with discrete-time measurements." IEEE Trans. on Automatic Control. Vol. 40, No.3 (1995)
248
Julier, S. J., Uhlmann, J. K. and Whyte, H. D. "A new approach for filtering nonlinear systems." Proc. American Control Conference. pp. 1628 - 1632(1995)
249
Julier, S. J. and Uhlmann, J. K. "Unscented filtering and nonlinear estimation." Proceedings of the IEEE. Vol. 92, No.3, pp. 401- 422(2004)
250
Wan, E. A., Merwe, R. V. D. and Nelson, A. T. "Dual estimation and the unscented transformation." Advances in Neural Information Processing Systems, MIT Press, Cambridge(1999)
251
Leung H., Zhu, Z. and Ding, Z. "An aperiodic phenomenon of the extended Kalman filter in filtering noisy chaotic signals." IEEE Trans. on Signal Processing, Vol. 48, No.6, pp. 1807 - 1810(2000)
208
Bibliography
252
Schreiber T. "Extremely simple nonlinear noise-reduction method." Phys. Rev. E. Vol. 47, pp. 2401 - 2404(1993)
253
Schouten, J. C., Takens, F. and Bleek, C. "Estimation of the dimension of a noisy attractor." Phys. Rev. E. Vol. 50, pp. 1851- 1861(1994)
254
Leontitsis, A., Bountis, T. and Pagge, J. "An adaptive way for improving noise reduction using local geometric projection." Chaos, Vol. 14, No.1, pp. 106 - 110 (2004)
255
Paul, S., Ott E. and Dayawansa, W. P. "Observing chaos: Deducing and tracking the state ofa chaotic system from limited observation." Phys. Rev. E. Vol. 49, pp. 2650 - 2660(1994)
256
Walker, D. M. and Mees, A. I. "Reconstructing nonlinear dynamics by extended Kalman filtering." Int. J Bifurcation Chaos Appl. Sci. Eng. Vol. 8, No.3, pp. 557 - 570(1998)
257
Boutayeb M., Darouach, M. and Rafaralahy, H. "Generalized statespace observers for chaotic synchronization and secure communication." IEEE Trans. Circ. Syst. J. Vol. 49, No.3, pp. 345 - 349(2002)
258
Wang, S. Y. and Feng, J. C. "A noise reduction method for noisy chaotic signal."
Journal of Circuits and Systems (in Chinese). Vol. 9, pp. 58 - 62(2004) 259
Howard, S. I. and Pahlavan, K. "Autoregressive modelling of wideband indoor radio propagation." IEEE Trans. on Communications. Vol. 40, pp. 1540 - 1552 (1992)
260
Feng, J. and Xie, S. "An unscented-transform-based filtering algorithm for noisy contaminated chaotic signals." Proceedings of 2006 IEEE International Symposium
on Circuits and Systems. pp. 2245 - 2248(2006) 261
Feng, J. and Xie, S. "A noise reduction method for noisy contaminated chaotic signal." Proceedings of 2005 International Conference on Communications, Circuits
and Systems. Vol. 2, pp. 1173 - 1176, Hong Kong, China, June(2005) 262
Feng, J. "A noise cleaning method for chaotic time series and its application in
209
Bibliograpby
communication." Chinese Physics Letters. Vol. 22, No.8, pp. 1851- 1854(2005) 263
Madan, R. N. (Ed.). Chua's Circuit: A Paradigm for Chaos. World Scientific, Singapore(1993)
264
Proakis, J. G. Digital Communication (3rd. Ed.). Mc-GrawHill, New York(l995)
265
Bertoni, H. L. Radio Propagation for Modern Wireless Systems. Prentice Hall, New Jersey(2000)
266
Rappaport, T. S. Wireless Communications Principles and Practice. Prentice Hall, New Jersey(l998)
267
Iltis, R. A. and Fuxjaeger, A. W. "A digital DS spread-spectrum receiver with joint channel and dopple shift estimation." IEEE Trans. on Communications. Vol. 39, pp. 1255 - 1267(1991)
268
Zhu, Z. and Leung, H. "Adaptive blind equalization for chaotic communication systems using extended-kalman filter." IEEE Trans. Circ. Syst. !. Vol. 48, pp. 979 - 989(2001)
269
Corron, N. J. and D. W. Hahs. "A new approach to communications using chaotic signals." IEEE Trans. Circ. Syst. !. Vol. 44, pp. 373 - 382(1997)
270
Sharma, N. and P. G. Poonacha. "Tracking of synchronized chaotic systems with applications to communications." Phys. Rev. E. Vol. 56, pp. 1242 - 1245(1997)
271
Anishchenko, V. S. and A. N. Pavlov. "Global reconstruction in application to multichannel communication." Phys. Rev. E. Vol. 57, pp. 2455 - 2457(1998)
272
Dedieu, H. and M. J. Ogorzalek. "Identifiability and identification of chaotic systems based on adaptive synchronization." IEEE Trans. Circ. Syst. !. Vol. 44, pp. 948 - 962(1997)
273
Miiller, A. and J. M. H. Elmirghani. "Chaotic transmission strategies employing artificial neural networks." IEEE Communication Lett.. Vol. 2, pp. 241- 243(1998)
274
Papadimitriou, S., A. Bezerianos and T. Bountis. "Radial basis function networks
210
Bibliography
as chaotic generators for secure communication systems." Int. J Bifurcation Chaos. Vol. 9, pp. 221 - 232(1999) 275
Chen, C. L., W. C. Chen and F. Y. Chang. "Hybrid learning algorithm for Gaussian potential function networks." lEE Proc. Control Theory and Appl.. Vol. 140, pp. 442 - 448(1993)
276
Chang, P. R. and W. H. Yang. "Environment-adaptation mobile radio propagation prediction using radial basis function neural networks." IEEE Trans.
Vehicular Tech .. Vol. 46, pp. 155 - 160(1997) 277
Yingwei, L., N. Sundararajan and P. Saratchandra. "A sequential learning scheme for function approximation using minimal radial basis function neural networks." Neural Computation. Vol. 9, pp. 461- 478(1997)
278
Girosi, F. and T. Poggio. "Neural networks and the best approximation property."
Biological Cybernetics. Vol. 63, pp. 169 - 176(1990) 279
Principe, J. C. and J. M. Kuo. "Noise Reduction in State Space Using the Focused Gamma Neural Network." In: L. M. Pecora (Ed.), Chaos in Communications, Proc. SPIE, Vol. 2038, pp. 326 - 332(1993)
280
Johnson, G. A., D. J. Mar, T. L. Carroll and L. M. Pecora. "Synchronization and parameter tracking in chaotic systems." Proc. of the fourth experimental chaotic
conference. pp. 407 - 412, World Scientific, Singapore(1998) 281
Matsumoto, T., L. O. Chua and K. Kobayashi. "Hyperchaos: laboratory experiment and numerical confirmation." IEEE Trans. Circ. and Syst.. Vol. 33, pp. 1143 - 1147(1986)
282
Feng, J. C., C. K. Tse and Francis C. M. Lau. "A chaos tracker applied to noncoherent detection in chaos-based digital communication systems." Proc. of IEEE Int. Symp. on Circ. and Syst.. Vol. 3, pp. 795 -798(2001)
283
Itoh, M., C. W. Wu and L. O. Chua. "Communication systems via chaotic signals from a reconstruction viewpoint," Int. J. Bifurcation Chaos. Vol. 7, pp. 275 - 286, 1997
284
Reif, K. and R. Unbehauen. "The extended Kalman filter as an exponential 211
Bibliography
observer for nonlinear systems." IEEE Trans. Signal Processing. Vol. 47, pp. 2324 - 2328(1999) 285
Kisel, A., H. Dedieu and T. Schimming. "Maximum likelihood approaches for non-coherent communications with chaotic carriers." IEEE Trans. Circ. Syst. 1. Vol. 48, pp. 533 - 542(2001)
286
Oppenheim, A. V., K. M. Cuomo, R. J. Baron and A. E. Fredman. "Channel Equalization for Communication with Chaotic Signals." In: R. A.Katz (Ed.), Chaotic,
Fractal and Nonlinear Signal Processing. AlP Press, pp. 289 - 301 (1996) 287
Ciftci, M. and D. B. Williams. "A novel channel equalizer for chaotic digital communications systems." Proc. IEEE ICASSP. Vo1.3, pp. 1301 - 1304(1999)
288
Cuomo, K. M., A. V. Oppenheim and R. 1. Barron. "Channel equalization for self-synchronizing chaotic systems." Proc. IEEE ICASSP. Vol. 3, pp. 1605 - 1608 (1996)
289
Sharma, N. and E. Ott. "Combating channel distortions in communication with chaotic systems." Phys. Lett. A. Vol. 248, pp. 347 - 352(1998)
290
Chua, L. 0., T. Yang, G. Q. Zhong and C. W. Wu. "Synchronization of Chua's circuits with time-varying channels and parameters." IEEE Trans. Circ. Syst. 1. Vol. 43, pp. 862 - 868(1996)
291
Feng, J.
c.,
C. K. Tse and F. C. M. Lau. "Channel equalization for chaos-based
communication systems." IEICE Trans. on Fundament. of Electronics, Communi-
cations and Computers Sciences. Vol. E85A, No.9, pp. 2015 - 2024(2002) 292
Qureshi, S. U. H. "Adaptive equalization." Proc. IEEE. Vol. 73, pp. 1349 - 1387 (1985)
293
Box, G. E. and G. M. Jenkins. Time Series Analysis: Forecasting and Control (3rd ed.). Prentice Hall, New Jersey(1994)
294
Khalid, A. A. and S. R. Irving. "The use of neural nets to combine equalization with decoding for severe intersymbol interference channels." IEEE Trans. Neural Networks. Vol. 5, pp. 982 - 988(1994)
212
Bibliography
295
Chen, S., B. Mulgrew and P. M. Grant. "A clustering teaching for digital communications channel equalization using radial basis function networks." IEEE Trans. Neural Networks. Vol. 4, pp. 570 - 590(1993)
296
Gencay, R. and T. Liu. "Nonlinear modelling and prediction with feedforward and recurrent networks." Physica D. Vol. 108, pp. 119 - 134(1997)
297
Zhou, C. and C. H. Lai. "Decoding information by following parameter modulation with parameter adaptive control." Phys. Rev. E. Vo1.59, pp. 6629 - 6636(1999)
298
Chow, T., J. C. Feng and K. T. Ng. "Chaotic network synchronization with application to communications." Int. J. of Communication Syst.. Vol. 14, No.2, pp. 217 - 230(2001)
299
Morantes, D. S. and D. M. Rodriguez. "Chaotic sequences for multiple access." Elec. Lett .. Vol. 34, pp. 235 - 237(1998)
213
Index
k-means clustering algorithm, 52 adaptability, 120 adaptive algorithm, 72,111,121 adaptive learning algorithm, 112, 115, 139, 184, 187 additive white Gaussian noise, 19, 93, 111,139,141, 151, 184 AR model, 93, 96,103, 106 artificial neural networks, 41, 46,163 connection weight, 55, 58, 147 neurons, 41, 42, 43,45, 50, 51, 53, 55, 57 processing units, 42 synaptic weights, 42, 43, 44 attracting domain, 135 autonomous system, 13, 40, 111, 115, 166, 183 autoregressive model, 93 back propagation algorithm, 163 back propagation network, 52, 58 bi-Lipschitz,34 bifurcation, 3, 4, 22, 24, 69, 99, 110, 112, 121, 132, 139, 151, 184 bifurcation parameter, 22, 24,110,121 bit error probability, 178, 180 box-counting dimension, 11,29,30,32 channel distortion, 24, 140, 151, 164, 165, 184, 186
channel effects, 140, 142, 151, 154, 155, 156, 164, 185 channel equalization, 103, 140, 141 channel equalizer, 141 chaos, 1,5, 12, 13, 14, 16, 19,22,23,24, 25,131,168 control, 4, 13, 14, 18, 40, 41, 44, 45, 49,94,105,106, 107, 108, 109, 119, 125,168,171,181 defmition, 5, 6, 10,28,42, 125 deterministic, 1, 2, 4, 12, 13, 16, 29, 91,165, 186 hypersensitive, 13 irregular fluctuation, 4 Li-Yorke chaos, 4 nonperiodic solutions, 3 period-doubling, 4 properties, 2, 4, 9, 13, 16, 18, 22, 29, 34,43,51,117 sensitive dependence, 5, 6, 7 strange attractor, 4, 9, 10 unpredictability, 12,91 chaos-based communication, 2, 25, 26, 40, 59, 60, 71, 78, 101, 134, 140, 141, 151, 164, 165, 167, 183, 184, 186, 187 challenges, 25 security, 2, 21, 25, 177, 186 chaos-shift-keying, 134 chaotic attractor, 3, 4, 5, 10, 11, 23, 32, 60,62,64,67,68,70,114,144,167 chaotic masking modulation, 19
Index
chaotic modulation, 19, 22, 24, 96, 1l0, lll, ll6 chaotic oscillation, 3 characteristic polynomial, 63 Chua's circuit, 4, 36, 37, 67, 95, 99, 100, 131,133,136,139,151,162,184 coherent demodulation, 22 compact real set, 111 conditional Lyapunov exponents, 140, 166 contextual information, 45 correlation function, 177, 178, 180 auto-correlation, 113, ll4, 176 crosscorrelation, 175 cost function, 51, II 0, III cubic polynomial, 67 decision circuit, 134 delta learning law, 148 demodulation algorithm, 122, 123, 184 demodulator, 22, 23, 25, 1l0, lll, 125, 127, 139, 141, 184 adaptive demodulator, Ill, 139, 184 differential chaos shift keying, 19 differential phase shift keying, 24 digital communication, 1, 23, 24, ll2, 131, 134, 138, 139, 184 dissipative dynamical systems, 4, 10 doppler effect, 102 driving-response, 167, 168 DS/CDMA, 15, 165, 166, 172, 175, 177, 182, 185 Duffing's equations, 63, 64, 184 dynamical feedback modulation, 19, 20, 21 Elman network, 57, 58 embedding, 10,28,29,30,31,32,33,34, 35,3~40,6~71,91, 114, 156, 183
delay coordinate, 4, 30, 31, 32, 33, 34, 35,38 delay coordinate function, 31 differentiable embedding, 29, 33 embedding dimension, 35 embedding space, 35 embedding theorem, 34 embedding theory, 28, 40, 60, 71, 91, 114, 156, 183 global false nearest neighbor, 35 immersion, 33 local false-nearest-neighbor, 35 time lag, 33 topological embedding, 29, 33, 34 entropy, 5, 8,9,10 Kolmogorov, 5, 8 Kolmogorov-Sinai, 8 metric, 6, 8, 9, 10, 32, 34, 38, 51, 81, 180 Ergodic Theorem, 3 error covariance matrix, 90, 120 error propagation, 125 Euclidean norm, 8, 81 Euclidean space, 29, 30, 60,180 distance, 49,51,102,103, ll7 extended Kalman filter, 72, 76, ll9 fast, 41, 45, 46, 52, 102, 103, 104, 109, 156,165,177,181,182,185,187 fading, 101, 103, 106 fault tolerant, 45, 46 feedforward neural networks, 56 Feigenhaum's number, 4 filter, 12, 19, 42, 57, 96, 1l0, 119, 142, 156,163,165, 180, 183, 184, 186 FIR, 57 IIR,57 fixed point, 168, 169, 170, 172, 173, 174 fractal, 3, 10, 14 215
Index
frequency demultiplication, 3 frequency modulation DeSK, 25 frequency selective fading, 102 Gaussian activation function, 118 Gaussian kernel, 49, 181 Gaussian random variable, 78, 93 gradient vector, 119 Gram-Schmidt orthogonalization, 54 Henon map, 69, 70, 112, 113, 114, 139, 151 Hamiltonian systems, 4 Heaviside function, 10 hidden layer, 49,51,54,55,57,111,116, 118, 120, 121, 126, 127, 139, 146,148, 150, 184, 186 hidden unit, 50, 51, 53, 55, 57, 58, 111, 118, 119, 121, 123, 125, 126, 127, 147, 148, 149, 163 hierarchi call y self-organizing learning, 53 homoc1inic trajectories, 3 hyperchaos oscillator, 131, 132, 134, 138, 139 information dimension, 10 information theory, 33 input-output mapping, 44 inter-symbol interference, 140 interpolation, 46, 47, 48, 49,51 inverse multiquadrics, 48 inverse system approach, 21, 156 inverse system modulation, 19 isomorphic, 9, 33 Jacabian matrix, 88, 92 jamming, 15
216
Jordan network, 57, 58 Kalman filter, 72 Kalman gain, 75, 80, 83, 84, 86, 87, 88, 89,90 Kalman gain vector, 93, 120 KAM theorem, 5 largest Lyapunov exponent, 36 learning algorithm, 42, 46, 52, 53, 54, 55, 58,59, 139, 164, 184, 185 learning rate, 148, 149 least-mean-square algorithm, 52, 55 linear transversal equalizer, 141, 142 LMS algorithm, 55, 142 local minima, 111 local minimum, 33 logistic map, 3, 86, 87, 88, 89, 95, 96, 97, 111 Lorenz system, 17,20,21, 168, 169, 177 Lure system, 62, 63, 71,183 Lyapunov dimension, 11,36, 151 Lyapunov exponent, 5, 6, 7, 8, 10, 11, 12, 17,34,35,78,83,84,86,87,140,166 continuous-time system, 6 discrete-time system, 3, 5, 69 Lyapunov spectrum, 7 mean squared error, 93 mean-square-error, 125, 156 measurement function, 28, 30, 31, 32, 34 multipath interferences, 181 mutual information, 33, 35, 38, 39 near-far effect, 102 network growth, 118 noise reduction, 91 non-autonomous system, 40 non-coherent detection, 139
Index
nonlinear auto-regressive moving average model, 145 nonlinear channel, 152, 159, 160, 161, 163, 164, 185, 186 nonlinearity, 43 nonparametric statistical inference, 44 nonsingularity, 49 observability, 40 observation step, 115, 116, 118, 120, 121, 122, 124, 125, 127, 130, 134, 135 OGY control method, 182, 185, 165, 166 small perturbations, 31, 167 one-step-ahead prediction, 115 one-to-one correspondence, 28, 31 bijective, 29, 32, 34 orthogonal forward regression, 52 orthogonal least squares, 52 orthogonality, 15,51,177 parallel processing, 46 piecewise-linear function, 36, 67, 123, 132, 151 Poincare section, 168, 170, 172 power spectrum, 1, 13,91,112,113,114 propagation equation, 120 propagation loss, 102 pruning strategy, 56 pseudo-random sequence, 15, 16, 175, 177, 186 pseudo-random code, 175, 176 radial basis function neural network, 46 reconstruction, 22, 27, 28, 32, 34, 35, 38, 39, 40, 60, 62, 69, 71, 91, 110, 145, 15~ 183, 185, 18~ 187 phase space, 6, 7, 8, 9, 10, 11, 14,28, 35,62,64 reconstruction space, 39
state space, 3, 5, 6, 7, 10, 12, 28, 40, 57,71,72,92,93,103, 183 trajectory, 9, 12, 14,31,64, 72, 167 recurrent neural network, 46, 146 context unit approach, 57 global feedback approach, 57, 58 locally recurrent globally feedforward approach, 57 recursive hybrid learning algorithm, 55 redundant units, 121 regularization, 50 renormalization group, 4 resource allocation network, 55 return maps, 34, 156 robust computation, 45 root-mean-square, 118 Runge-kutta algorithm, 37, 175, 177 scaling factor, 120 self-organized learning, 52 self-similarity, 14 sequential learning, 55, 119 Shadow effect, 102 signal, 1, 13, 14, 15, 16, 17, 21, 22, 23, 24, 25, 27, 28, 43, 44, 56, 60, 64, 66, 67, 71, 72, 78, 80, 87, 89, 91,
18, 34, 68, 92,
19,20, 39, 4~ 69, 70, 94, 95,
96,97,98,99,100, 101, 102, 103, 109, 110, lll, 112, 113, 118, 120, 121, 122, 123, 124, 125, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 139, 140, 141,142, 143, 145, 148, 151, 152, 154, 155, 156, 158, 159, 162, 163, 164, 165, 166,167,168,172,173,174,175,177, 178,179,180,181,183,184,185,186, 187 deterministic, 1 stochastic, 1,2,5, 12, 16,76, 186
217
Index
signal-to-noise ratio, 94, 125 signal-to-noise ration, 97 singular behaviour, 3 sliding data window, 56, 118 slow fading, 102 space selective fading, 102 spread-spectrum communication, 112, 165, 175, 182 stable manifold, 170, 171, 172 state-space model, 76, 96 steady state, 1,44,87 stochastic gradient algorithm, 142
adaptive synchronization, 18 attracting synchronization set, 18 network synchronization, 165, 166, 167 self-synchronization, 18 Takens' embedding theory, 60, 156, 183 110,
sub harmonic motion, 2 supervised learning, 43, 52 synchronization, 2,15,17,18,19,20,22, 24, 34, 110, 140, 165, 166, 167, 168, 172,175,178,182,183,185
218
tangent space, 33 thin plate spline function, 48 time selective fading, 102 tracking algorithm, 112, 135, 139 training algorithm, 51,147,186 transfer function, 76,101, 103 uncertainty, 120 unstable kalman filter, 78 Van der Pol system, 3
This page intentionally left blank
