Pattern Recognition in Bioinformatics: 4th IAPR International Conference, PRIB 2009, Sheffield, UK, September 7-9, 2009, Proceedings (Lecture Notes in ... Science Lecture Notes in Bioinformatics)

Lecture Notes in Bioinformatics 5780 Edited by S. Istrail, P. Pevzner, and M. Waterman Editorial Board: A. Apostolico ...

Author: Visakan Kadirkamanathan | Guido Sanguinetti | Mark Girolami | Mahesan Niranjan | Josselin Noirel

49 downloads 1719 Views 8MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!

Report copyright / DMCA form

DOWNLOAD PDF

Lecture Notes in Bioinformatics

5780

Edited by S. Istrail, P. Pevzner, and M. Waterman Editorial Board: A. Apostolico S. Brunak M. Gelfand T. Lengauer S. Miyano G. Myers M.-F. Sagot D. Sankoff R. Shamir T. Speed M. Vingron W. Wong

Subseries of Lecture Notes in Computer Science

Visakan Kadirkamanathan Guido Sanguinetti Mark Girolami Mahesan Niranjan Josselin Noirel (Eds.)

Pattern Recognition in Bioinformatics 4th IAPR International Conference, PRIB 2009 Sheffield, UK, September 7-9, 2009 Proceedings

13

Series Editors Sorin Istrail, Brown University, Providence, RI, USA Pavel Pevzner, University of California, San Diego, CA, USA Michael Waterman, University of Southern California, Los Angeles, CA, USA Volume Editors Visakan Kadirkamanathan Guido Sanguinetti Josselin Noirel University of Sheffield Mappin Street, Sheffield, S1 3JD, UK E-mail: {visakan,g.sanguinetti,j.noirel}@sheffield.ac.uk Mark Girolami University of Glasgow Glasgow, G12 8QQ, UK E-mail: [email protected] Mahesan Niranjan University of Southampton Southampton SO17 1BJ, UK E-mail: [email protected]

Library of Congress Control Number: 2009933351

CR Subject Classification (1998): J.3, I.5, F.2.2, I.2, H.3.3, H.2.8 LNCS Sublibrary: SL 8 – Bioinformatics ISSN ISBN-10 ISBN-13

0302-9743 3-642-04030-6 Springer Berlin Heidelberg New York 978-3-642-04030-6 Springer Berlin Heidelberg New York

This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. springer.com © Springer-Verlag Berlin Heidelberg 2009 Printed in Germany Typesetting: Camera-ready by author, data conversion by Scientific Publishing Services, Chennai, India Printed on acid-free paper SPIN: 12747078 06/3180 543210

Preface

The Pattern Recognition in Bioinformatics (PRIB) meeting was established in 2006 under the auspices of the International Association for Pattern Recognition (IAPR) to create a focus for the development and application of pattern recognition techniques in the biological domain. PRIB’s aim to explore the full spectrum of pattern recognition application was reflected in the breadth of techniques represented in this year’s submissions and in this book. These range from image analysis for biomedical data to systems biology. We were fortunate to have invited speakers of the highest calibre delivering keynotes at the conference. These were Pierre Baldi (UC Irvine), Alvis Brazma (EMBL-EBI), Gunnar R¨atsch (MPI T¨ubingen) and Michael Unser (EPFL). We acknowledge support of the EU FP7 Network of Excellence PASCAL2 for partially funding the invited speakers. Immediately prior to the conference, we hosted half day of tutorial lectures, while a special session on “Machine Learning for Integrative Genomics” was held immediately after the main conference. During the conference, a poster session was held with further discussion. We would like once again to thank all the authors for the high quality of submissions, as well as Yorkshire South and the University of Sheffield for providing logistical help in organizing the conference. Finally, we would like to thank Springer for their help in assembling this proceedings volume and for the continued support of PRIB. July 2009

Mark Girolami Visakan Kadirkamanathan Mahesan Niranjan Josselin Noirel Guido Sanguinetti

Organization

International Program Committee Shandar Ahmed

National Institute of Biomedical Innovation, Japan Jes´us S. Aguilar-Ruiz Escuela Polit´ecnica Superior, Seville, Spain Tatsuya Akutsu Kyoto University, Japan Sanghamitra Bandyopadhyay Indian Statistical Institute, India Sebastian B¨ocker Friedrich-Schiller-Universit¨at, Jena, Germany Rainer Breitling University of Groningen, The Netherlands Nicolas Brunel CNRS, Paris, France Colin Campbell University of Bristol, UK Frederic Cazals Sophia Antipolis, France CQ Chang University of Hong Kong, China Marco Chierici Bruno Kessler Foundation, Trento, Italy Theo Damoulas University of Glasgow, UK Richard Edwards University of Southampton, UK Maurizio Filippone University of Sheffield, UK Alexandru Floares Oncological Institute Cluj, Romania Jennifer Hallinan University of Newcastle, UK Jin-Kao Hao University of Angers, France Jaap Heringa VU University Amsterdam, The Netherlands Antti Honkela Helsinki University of Technology, Finland Giuseppe Jurman Bruno Kessler Foundation, Trento, Italy R. Krishna Murthy Karuturi Genome Institute of Singapore Samuel Kaski Helsinki University of Technology, Finland Alex Kochetov Russian Academy of Sciences, Russia Mehmet Koyuturk Case Western Reserve University, Cleveland, USA Zoe Lacroix Arizona State University, USA Tak-Wah Lam University of Hong Kong, China Kee Khoon Lee Institute of High Performance Computing, Singapore Pietro Li`o University of Cambridge, UK Xuejun Liu Nanjing University of Aeronaoutics and Astronautics, China Francesco Masulli University of Genova, Italy Mariofanna Milanova Donaghey College of Engineering and Information Technology, USA Sach Mukherjee University of Warwick, UK

VIII

Organization

Alioune Ngom Carlotta Orsenigo Nikhil Pal Magnus Rattray Simon Rogers Juho Rousu Anastasia Samsonova Alexander Schliep Roberto Tagliaferri Gwenn Volkert David Wild Hong Yan Jing Yang Yan-Qing Zhang

University of Windsor, Canada Politecnico di Milano, Italy Indian Statistical Institute, India University of Manchester, UK University of Glasgow, UK University of Helsinki, Finland Harvard University, USA Max Planck Institute for Molecular Genetics, Berlin, Germany University of Salerno, Italy Kent State University, USA University of Warwick, UK City University of Hong Kong, China Qingdao Institute of Bioenery and Bioprocess Technology, China Georgia State University, USA

Conference Organizing Committee Conference Chairs Visakan Kadirkamanathan Guido Sanguinetti

University of Sheffield, UK University of Sheffield, UK

General Co-chairs Raj Acharya Madhu Chetty Jagath Rajapakse

PennState, USA Monash University, Australia Nanyang Technological University, Singapore

Program Chairs Mahesan Niranjan Mark Girolami

University of Southampton, UK University of Glasgow, UK

Tutorial Chair Florence d’Alch´e-Buc

´ University of Evry, France

Special Sessions Chair Cesare Furlanello

Fondazione Bruno Kessler, Italy

Publicity Chair Elena Marchiori

Radboud University of Nijmegen, The Netherlands

Organization

Publications Chair Josselin Noirel

University of Sheffield, UK

Local Organization Chair Daniel Coca

University of Sheffield, UK

Finance Chair Andrew Zammit Mangion

University of Sheffield, UK

Webmaster Maurizio Filippone

University of Sheffield, UK

IX

Table of Contents

Evolutionary Parameters in Sequence Families: Cold Adaptation of Enzymes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Said Hassan Ahmed and Tor Fl˚ a MProﬁler: A Proﬁle-Based Method for DNA Motif Discovery . . . . . . . . . . Doaa Altarawy, Mohamed A. Ismail, and Sahar M. Ghanem On Utilizing Optimal and Information Theoretic Syntactic Modeling for Peptide Classiﬁcation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Eser Ayg¨ un, B. John Oommen, and Zehra Cataltepe Joint Tracking of Cell Morphology and Motion . . . . . . . . . . . . . . . . . . . . . . Jierong Cheng, Esther G.L. Koh, Sohail Ahmed, and Jagath C. Rajapakse

1

13

24

36

Multiclass Microarray Gene Expression Analysis Based on Mutual Dependency Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Girija Chetty and Madhu Chetty

46

An Eﬃcient Convex Nonnegative Network Component Analysis for Gene Regulatory Network Reconstruction . . . . . . . . . . . . . . . . . . . . . . . . . . . Jisheng Dai, Chunqi Chang, Zhongfu Ye, and Yeung Sam Hung

56

Using Higher-Order Dynamic Bayesian Networks to Model Periodic Data from the Circadian Clock of Arabidopsis Thaliana . . . . . . . . . . . . . . . R´ on´ an Daly, Kieron D. Edwards, John S. O’Neill, Stuart Aitken, Andrew J. Millar, and Mark Girolami

67

Sequential Hierarchical Pattern Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . Bassam Farran, Amirthalingam Ramanan, and Mahesan Niranjan

79

Syntactic Pattern Recognition Using Finite Inductive Strings . . . . . . . . . . Paul Fisher, Howard Fisher, Jinsuk Baek, and Cleopas Angaye

89

Evidence-Based Clustering of Reads and Taxonomic Analysis of Metagenomic Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Gianluigi Folino, Fabio Gori, Mike S.M. Jetten, and Elena Marchiori

102

Avoiding Spurious Feedback Loops in the Reconstruction of Gene Regulatory Networks with Dynamic Bayesian Networks . . . . . . . . . . . . . . . Marco Grzegorczyk and Dirk Husmeier

113

XII

Table of Contents

Ligand Electron Density Shape Recognition Using 3D Zernike Descriptors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Prasad Gunasekaran, Scott Grandison, Kevin Cowtan, Lora Mak, David M. Lawson, and Richard J. Morris

125

Deﬁnition of Valid Proteomic Biomarkers: A Bayesian Solution . . . . . . . . Keith Harris, Mark Girolami, and Harald Mischak

137

Inferring Meta-covariates in Classiﬁcation . . . . . . . . . . . . . . . . . . . . . . . . . . . Keith Harris, Lisa McMillan, and Mark Girolami

150

A Multiobjective Evolutionary Algorithm for Numerical Parameter Space Characterization of Reaction Diﬀusion Systems . . . . . . . . . . . . . . . . Tim Hohm and Eckart Zitzler Knowledge-Guided Docking of WW Domain Proteins and Flexible Ligands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Haiyun Lu, Hao Li, Shamima Banu Bte Sm Rashid, Wee Kheng Leow, and Yih-Cherng Liou

162

175

Distinguishing Regional from Within-Codon Rate Heterogeneity in DNA Sequence Alignments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Alexander V. Mantzaris and Dirk Husmeier

187

A Hybrid Metaheuristic for Biclustering Based on Scatter Search and Genetic Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Juan A. Nepomuceno, Alicia Troncoso, and Jes´ us S. Aguilar–Ruiz

199

Di-codon Usage for Gene Classiﬁcation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Minh N. Nguyen, Jianmin Ma, Gary B. Fogel, and Jagath C. Rajapakse

211

Counting Patterns in Degenerated Sequences . . . . . . . . . . . . . . . . . . . . . . . . Gr´egory Nuel

222

Modelling Stem Cells Lineages with Markov Trees . . . . . . . . . . . . . . . . . . . . Victor Olariu, Daniel Coca, Stephen A. Billings, and Visakan Kadirkamanathan

233

Bi-clustering of Gene Expression Data Using Conditional Entropy . . . . . . Afolabi Olomola and Sumeet Dua

244

c-GAMMA: Comparative Genome Analysis of Molecular Markers . . . . . . Pierre Peterlongo, Jacques Nicolas, Dominique Lavenier, Raoul Vorc’h, and Jo¨el Querellou

255

Semi-supervised Prediction of Protein Interaction Sentences Exploiting Semantically Encoded Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tamara Polajnar and Mark Girolami

270

Table of Contents

Classiﬁcation of Protein Interaction Sentences via Gaussian Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tamara Polajnar, Simon Rogers, and Mark Girolami MCMC Based Bayesian Inference for Modeling Gene Networks . . . . . . . . Ramesh Ram and Madhu Chetty

XIII

282 293

Eﬃcient Optimal Multi-level Thresholding for Bioﬁlm Image Segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Dar´ıo Rojas, Luis Rueda, Homero Urrutia, and Alioune Ngom

307

A Pattern Classiﬁcation Approach to DNA Microarray Image Segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Luis Rueda and Juan Carlos Rojas

319

Drugs and Drug-Like Compounds: Discriminating Approved Pharmaceuticals from Screening-Library Compounds . . . . . . . . . . . . . . . . . Amanda C. Schierz and Ross D. King

331

Fast SCOP Classiﬁcation of Structural Class and Fold Using Secondary Structure Mining in Distance Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jian-Yu Shi and Yan-Ning Zhang

344

Short Segment Frequency Equalization: A Simple and Eﬀective Alternative Treatment of Background Models in Motif Discovery . . . . . . . Kazuhito Shida

354

Bayesian Optimization Algorithm for the Non-unique Oligonucleotide Probe Selection Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Laleh Soltan Ghoraie, Robin Gras, Lili Wang, and Alioune Ngom

365

Microarray Time-Series Data Clustering via Multiple Alignment of Gene Expression Proﬁles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Numanul Subhani, Alioune Ngom, Luis Rueda, and Conrad Burden

377

Recursive Neural Networks for Undirected Graphs for Learning Molecular Endpoints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ian Walsh, Alessandro Vullo, and Gianluca Pollastri

391

Enhancing the Eﬀectiveness of Fingerprint-Based Virtual Screening: Use of Turbo Similarity Searching and of Fragment Frequencies of Occurrence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Shereena M. Arif, J´erˆ ome Hert, John D. Holliday, Nurul Malim, and Peter Willett

404

Patterns, Movement and Clinical Diagnosis of Abdominal Adhesions . . . Benjamin Wright, John Fenner, Richard Gillott, Paul Spencer, Patricia Lawford, and Karna Dev Bardhan

415

XIV

Table of Contents

Class Prediction from Disparate Biological Data Sources Using an Iterative Multi-Kernel Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yiming Ying, Colin Campbell, Theodoros Damoulas, and Mark Girolami

427

Cross-Platform Analysis with Binarized Gene Expression Data . . . . . . . . . Salih Tuna and Mahesan Niranjan

439

Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

451

Evolutionary Parameters in Sequence Families Cold Adaptation of Enzymes Said Hassan Ahmed and Tor Fl˚ a Dept of Mathematics and Statistics, University of Tromsø, 9037 Tromsø, Norway

Abstract. In an attempt to incorporate environmental eﬀects like coldadaptation into models of sequence evolution on a phylogenetic tree, we present a viable way of representing descriptive statistics of sequence observables under reversible Markov models of sequence evolution. Local variation in amino acid distribution along and across the sequence family can be connected to enzymatic adaptation to diﬀerent temperatures. Here, we estimate a few amino acid properties and how the variations of these properties both with respect excess mean values (EMVs) and covariance classify the protein family into clusters. Application of a multiscale and multivariate method to an aligned family of distinct trypsin and elastase sequences shows drift of centroid mean sequences of cold adapted enzymes compared to their warm-active counterparts.

1

Introduction

Phylogenetic tree-building methods presume particular evolutionary models [2]. Current evolutionary models of amino acid sequence evolution generally depend on mathematical models based on empirical observations using either comparisons of the observed amino acid sequences or their physical-chemical properties [1, 2]. These models estimate evolutionary distances in terms of expected number of substitutions per site by assuming evolution with independent sites and that the sequences in each site are assumed to evolve according to a single stochastic process, and that this process is ﬁxed across all sites. For instance, in Markov models of amino acid replacement, the Markov process is assumed to be stationary, homogeneous and reversible so that the amino acid distribution and the rate of replacement are assumed to be ﬁxed in time and positions, and that the forward and reverse substitution rates are assumed to be the same [15, 16]. We will be interested in the possibility to parameterize environmental eﬀects like cold adaptation into Markov transition and the corresponding rate matrices. In particular, we are interested in amino acid distribution proﬁle in an aligned family with cold-adapted representatives. Cold-adapted enzymes are characterized by clusters of glycine residues, a reduced number of proline residues in loop regions, a general reduction in charged residues on the surface and exposure of hydrophobic residues to solvent [19, 20]. All these features are thought to give rise to the increased structural ﬂexibility observed in some of the regions of the enzyme. Flexibility seems to be a strategy for cold-adapted enzyme to maintain high catalytic activity at low temeperatures [18, 19]. Often a few conserved V. Kadirkamanathan et al. (Eds.): PRIB 2009, LNBI 5780, pp. 1–12, 2009. c Springer-Verlag Berlin Heidelberg 2009

2

S.H. Ahmed and T. Fl˚ a

residues within each temperature class sequence positions are detemining factors of their strategies to adapt to cold/warm temperature. Here, we study how approximations on Markov models for standard phylogeny will give us an opportunity to obtain ﬁrst hand insight into the statistics of observables related to index and counting variables. Based on parameterized sequence features, we will carry out a multiscale and multivariate data analysis on an aligned family of distinct trypsin and elastase sequences. The basis of this multivariate analysis is that covariation of residue sites in evolution is related mainly to structural or functional site ﬁtness as parameterized by models of mean amino acid distributions and certain amino acid properties. These correlated residues based on amino acid properties (property sequences) show deviation from a common position dependent mean value. Such mean deviations, which we refer to as excess mean values (EMVs), are due to species dependent variations in local and global ﬁtness without aﬀecting the overall 3D fold and ﬁtness with respect to protein (enzyme) function. Our goal is to extract these EMVs from evolutionary noise both along and across the sequence family. On application, the method revealed drift of centroids due to features of cold adaptation. Such deviations could be used as measures of evolutionary adapted ﬁtness landscapes corresponding both to the folding rate as parameterized by the global relative energy gap/energy standard deviation ratio (funneling picture) and the local ﬁtness adaptations at the active site measuring the binding activity eﬀects.

2 2.1

Parameterization of Sequence Features Statistics Based on Amino Acid Unit Count Vectors

We assume an aligned family of L homologous protein sequences of length N . Let α(l, s) be the residue at position s and species l, where l ∈ {1, . . . , L}, s ∈ {1, . . . , N }. Then we describe α(l, s) numerically in the vector space of amino acid unit counts, denoted as Yl,s = Yα(l,s) , where Yα(l,s) = (δα, α(l,s) ) ∈ IR20 . Here, α ∈ A, is one of the 20 amino acid categories, δα, α(l,s) is 1 if one of the amino acids α equals the amino acid at (l, s), α(l, s), or 0 otherwise (the Kronecker delta function). With this representation, we have that the average over the observed present time leaf distribution of the protein amino acids at (l, s) is given by the amino acid distribution < Yl,s >= pl,s = (pl,s α ) ,

(1)

where . is the expectation operator (with respect to phylogenetic distributions). For completness, we have taken into account that pl,s will vary both on subset of species and positions due to diﬀerent species clusters and functional (or structural) constraints, in our case, clusters and residue determinants of cold-adapted enzymes. We are interested in the amino acid distribution given in (1), in terms of two sequence ensembles (Yl,s , Yl ,s ), namely, the ﬁrst-and second-order marginals, p(1)l,s (we will supress the superscript (1)) and P(2)l,l ;s,s , respectively. Since the

Evolutionary Parameters in Sequence Families

3

l,s ﬁrst-order marginal, pl,s = L l=1 pα , is a product of single-site multinomial type probability with no information about the sequence pair probabilites necessary to describe standard phylogenetic tree parameters, we consider correlation of unit count vectors. For simplicity, we look at the two-point covariation, which is given by (2)l,l ;s,s l ,s l ,s (Yα(l,s) − pl,s − pl,s , (2) α )(Yβ(l ,s ) − pβ ) =Pα,β α pβ (2)l,l ;s,s

=ραβ

l ,s pl,s , α pβ

(2)l,l ,s

where ραβ is the pair (α at (l, s) and β at (l , s )) dependent correction. Based on a reversible Markov model of amino acid replacement, with the instantaneous rate of replacement of amino acid α by amino acid β deﬁned by the rate matrix Q = (Qαβ ), as described in Sect. 1, the two point correlation could, for relatively short evolutionary times compared to the mutation rate Qαα and within the same cluster c ∈ {1, 2, . . . , K}, such that the mean amino acid distribution is ﬁxed, pl,s = pc,s , c(l) = c, be modelled as1 (2)l,l ;s

ραβ

psα psβ psα δα,β − psα psβ + (Tl + Tl ) Λsαβ psα psβ ,

(3)

Here Tl is the edge length corresponding to species l, and Λαβ is constrained so that the row sums are all zero: Qαα = Λsαα psα = − β=α = Λsβα , α = β (symmetry), which ensures reversibility of the Markov process. Notice that for long evolutionary times between leaf nodes l and l , the process will eﬀectively (2)l,l ;s be independent and ρα,β 0. This eﬀect will tend to divide our protein sequences into clusters of close neighbors in evolutionary time Tl + Tl where the above model will be used within each cluster. We can extend the above model of the two-point correlation for all sites s = 1, 2, . . . , N and ﬁnd covariance of two unit count vectors between two protein sequences (Yl,s − pl,s )(Yl,s − pl ,s ), for short evolutionary times as (2)l,l ;s,s s s ss s ραβ pα pβ psα δα,β δs,s − psα psβ + (Tl + Tl )Λsαβ psα Jαβ pβ , ∀αβ , (4)

s,s where Jαβ is the pair dependent correlation that ensures symmetric reversible s,s s substitution matrix ∀(α, β) pairs, Λsαα psα − β=α Λsalphaβ s = sJαβ pβ . We are interested in physical-chemical observables and how they are reﬂected in the amino acid distribution of the family. As they are linearly dependent on the unit count vectors, parameterized features (covariance) based on physicalchemical observables can be derived by unit count vector projections as described below. Thus, we consider the unit count vectors as our basic observables.

2.2

Statistics Based on Physico-Chemical Properties

Given a vector of amino acid properties C ∈ IR|A| , we ﬁnd that for a family of sequences with unit count vectors Yl,s , there is a family of property sequences given as 1

For simplicity, we assume one cluster and skip the cluster index c in this section.

4

S.H. Ahmed and T. Fl˚ a (N )

Cl

= (Ci(l,s) )s∈{1,...,N } = CT Yl,s s∈{1,...,N } ,

l = 1, . . . , L ,

(5)

where the superscript (N ) indicates the length of the sequence. Since (5) a linear mapping of the unit count vectors, the mean of the property sequences can be expressed as ¯ (N ) = (C¯c(l),s ) = CT pl,s C , (6) l s∈{1,...,N } where C¯c(l)s is the mean property in cluster c(l) = c and we assume a ﬁxed ¯ (N ) could be the amino acid distribution for each cluster pl,s = pc,s . Here, C l mean property of the whole family or as above of a cluster c(l) = c within the (N ) = C− C ¯ (N ) be the mean subtracted property sequences protein family. Let C l l (this subtraction to be explained below). Then a similar model of covariation as in (4) based on property sequences can be derived by projecting mean subtracted property vector on the parameterized covariance of the unit count vectors in ﬁxed cluster c: 1 (N ) (N ) 1 s Σcll = Cl , Cl (c(l),c(l ))=c (τ + (Tl + Tl )S¯cs , (7) N −1 N −1 s c ¯s 2 τcs = (Cα − C¯cs )2 pc,s Cα2 pc,s α = α − (Cc ) , α

S¯cs =

α,β

α

c,s c,s ¯ (Cα − C¯c )pc,s α Λαβ pβ (Cβ − Cc ) ,

c(l),s s where C¯cs = {α} C¯c(l),α = {α}Cα pα , is the average amino acid property2 for proteins in species l in cluster c = c(1) ∈ {1, 2, . . . , K} since we assume that proteins come in say K groups with more or less the same properties within a group and are independent between groups.3 . The logic of the mean subtraction prior to parameterization, which isalso the basis for our data analysis, is the rec,s lation for the substitution matrix, α Λc,s αβ pβ = 0, which is valid for equilibrium amino acid distributions and symmetric substitution matrix. This would lead to that correlation within a subfamily will be described by a simple variance. If some cold-adapted representatives are present within the family or subfamily, it will imply that the mean amino acid distribution will change. Consequently, both the center of the cluster as described by the mean properties and covariance matrix will move relative to that of standard mesophilic temperature class. Excess mean values (EMVs). When there are more than one cluster in the sequence family or subfamily, each cluster might have a diﬀerent mean amino acid distribution pc,s . Often the size of the data contained in each cluster is not suﬃcient enough to estimate the sequence position dependent mean necessary to observe cluster deviations. In this case, one would have to be satisﬁed with 2 3

τcs is the local property variance obtained for short evolutionary times compared to the local mutation rate, i.e. T μs 1, μs = α Qsαα . The formula for covariance clusters is simply obtained by summing the covriance of each cluster with respect to the cluster prior probability p(c), c = 1, 2, . . . , K.

Evolutionary Parameters in Sequence Families

5

a common mean ps . This analysis lead to artiﬁcial, extrinsic correlations which s c,s c,s we attribute to the EMVs δpc,s − C¯ s . Additional average e = p and δCe = C might come due to intrinsic correlations, interactions /and dependencies along the protein sequence as discussed above and this would lead to cavity ﬁlter (the coeﬃcients J s,s in 4)) averaged ﬁelds pscav . Both extrinsic and intrinsic excess mean values lead to extra correlations to linear and quadratic order (or higher) in our measurements and theoretical model of covariance might indeed give a substantial contribution to the clustering we study below. The theory and detailed discussion of these eﬀects is beyond the scope of this paper. Still we will refer to EMVs in our discussion of the results of the data analysis below.

3

Multivariate Data Analysis

The goal of the data analysis to ﬁnd excess mean values which maximize covariations between clusters of cold-and warm adapted enzymes in an aliged family of homologs of diﬀerently adapted enzymes. 3.1

Data Sets

As a benchmark data, we used trypsin (a well studies enzyme w.r.t. cold adaptation) sequences studied by Nils Peder Willassen and coworkers [20]. The sequences are divided into 3 groups: trypsins from the higher vertebrate, the cold-adapted ﬁsh and the other ﬁsh. Additionally, we used 29 elastase sequences (though could not show all results due to small space), composed of the three types of elastases, namely, elastase type-I (with 3 cold-adapted representatives), II (with 5 cold-adapted elastases) and III. The elastases were collected from homologous search by blast at the data bases available at NCBI (http://www.ncbi. nml.nih.gov/blast) and SiB (http://au-expasy-org/tools/blast). Multiple alignment was performed using Geneious, version 3.7.1 (Biomatters Ltd.). In this paper, the property sequences are based on hydrophobicity (Kyte-Doolittle, 1983) and polarity (Zimmermann, 1968). All analyses subsequently described were implemented in MATLAB 7.0. 3.2

Discrete Wavelet Transform (DWT)

We begin with a brief review of orthogonal forward discrete wavelet transform (DWT). An important concept in wavelets is the multiresolution analysis (MA) [8], which decomposes the property sequences as coeﬃcients Aj+1 at n reference level 0 (unit scale) in orthonormal basis functions {φj+1 (t)} in the k (j) (j) space Vj+1 into approximation and detail coeﬃcients, Ak and Dk , at level 1 in orthonormal basis functions {φjk (t), wkj (t)} in the nested spaces Vj and Wj (Vj+1 = Vj ⊕ Wj ), respectively: f j+1 (t) =

n

A(j+1) ψnj+1 (t) = n

k

Ak ψkj (t) + (j)

k

Dk ψkj (t) , (j)

(8)

6

S.H. Ahmed and T. Fl˚ a

where, qkj (t) = 2j/2 q(2j t−k). That is, the scaling and the wavelet basis functions, ψkj (t) and wkj (t), are dyadic dilations ((1/2)j ) and integer translations ((1/2)j k) of the father and mother functions ψ(t) and w(t), which connect DWT to subband ﬁltering (see the footnote below). As the basis functions are orthonormal at each level j, the corresponding coeﬃcients can be obtained by taking the inner products f j+1 (t), ψkj (t) and f j+1 (t), wkj (t) to yield4 (j)

Ak =

n

A(0) n g(n − 2k),

(j)

Dk =

n

A(0) n h(n − 2k) .

(9)

This is ﬁltering and downsampling operations in the analysis ﬁlter bank [10]. It is convolution with time reversed lowpass ( g(−n) ) and highpass ﬁlters ( h(−n) ). (N ) We performed a 4-level wavelet decomposition of each Cl using Symlet (sym4), a near symmetric, orthogonal wavelet with 4 vanishing moments. In DWT [8] starting with the approximation coeﬃcient at reference level j = 0 (unit (j) scale), (8) was recursively applied on Ak at coaser levels, i.e. levels j = 1, . . . , J, up to the desired level J = 4. The detail (diﬀerences) coeﬃcients at each level and the approximation (averages) coeﬃcients at ﬁnal level J were extracted. For l = 1, 2, . . . , L, the set of L detail coeﬃcients were arranged as L × Nj matrices, (j) denoted by D(j) = (Dlk ), and the approximation coeﬃcients at the ﬁnal level (j) as L × NJ matrix, denoted by A(J) = (Alk ), where Nj ≈ N (1/2)j is the number of coeﬃcients at level j. The goal with using orthogonal DWT is that it produces uncorrelated ensem(N ) bles (due to orthonormality of the basis functions) of Cl along the sequence (j) family based on Dl across k, creating a sparse representation5 . An important feature of this representation is that most of the energy of the ensembles is (j) concentrated on a few number of large coeﬃcients Dlk that contain correlated features, partly due to intrinsic and mostly due to extrinsic correlations (see Sect. 2.2) across the species labels, that could be associated with “EMV variation” (ﬁtness w.r.t. environmental eﬀects like cold adaptation). In other words, most of the coeﬃcients at ﬁner levels are attributed to evolutionary noise (background noise) across the sequence family, with small energy spread out equally across the scales. Additionally, orthogonal DWT represents these ensembles at localspace (1/2)j k and at scales (1/2)j , hence giving an accurate local description and separations of the high-frequency-features (small j) at diﬀerent resolutions. Figure 1 shows histograms of D(j) at levels 1 to 3 (4 not shown but displays a similar form). From the ﬁgure, it is clear that the multivariate distributions of (j) Dl , ∀l has small variance with mean close to zero. Thus, a Gaussian distribution is a reasonable probabilistic model for the multivariate distributions at these 4

5

Use the √ reﬁnement equations of qkj (t), i.e. shift √ the dilation and wavelet equations ψ(t) = 2 m g(m)ψ(2t − m) and w(t) = 2 m h(m)ψ(2t − m) by k and set m = n − 2k. Since A(j) contain few data points of low-frequency features, we concetrate on the details (variations), that is, D(j) .

Evolutionary Parameters in Sequence Families Level 1

Level 2

0.7

0.7

0.6

0.6

0.5

0.5

0.4

0.4

0.3

0.3

0.2

0.2

0.1

0.1

7

Level 3 1

0.8

0.6

0.4

0.2

0 −3

−2

−1

0 a)

1

2

3

0 −3

−2

−1

0 b)

1

2

3

0 −3

−2

−1

0 c)

1

2

3

Fig. 1. Histograms showing multivariate distribution of the set of L detail coeﬃcients corresonding to the 27 hydrophobicity sequences based on trypsin sequence data at levels 1 (a), 2 (b), and 3 (c). The superimposed curves correspond to theoretical probability density functions.

levels (see also the transposed curves). Consequently, we can use diagonalizing transformations to elimate the eﬀects of correlation [7]. 3.3

Diagonalization

Let the energy (correlation) of the ensembles of the property sequences based on D(j) at level j be given by

1 (j) (j) T R = (D(j) )(D(j) )T = Rll = D ) Dl ) , (10) Nj − 1 l where Rll , l, l ∈ {1, 2, . . . , L} is the l − l -th element of the symmetric matrix (j) (j) R, a measure of correlation between Dl and Dl . Since R is symmetric and positive deﬁnite (in our case), there exists an L × L orthogonal matrix U such that R = UD(σi )UT =⇒ UT RU = D(σi ) , (11) where the columns of U is given by a set of L orthonormal eigenvectors (ui ) and (σi ) are the corresponding eigenvariances, ordered from high-to-low species variations, (σ1 > σ2 > . . . σi > . . . σL ), and D(.) is a diagonal matrix with (j) eigenvariances as elements in the bracket. Then projection of Dl along the L (j) orthogonal directions (ui ) will create a set of L uncorrelated coeﬃcients in i D i (j) that are normally distributed with mean D 1 (the subscript 1 indicates the 1st row of the transformed sequences): (j) (j) = UT D(j) ∼ N D , D(σi ) , D 1

(12)

(j) indexed by i are arranged from high-to-low species variWhere the rows of D i ation, as described by (σi ). The eﬀect of the diagonalizing transformation in (11) (j) and D ˜ (j) is that two highly correlated transformed detail coeﬃcients D i i will contribute less that two nearly correlated transformed sequence of detail coefﬁcients (D(σi )), thus eliminating the eﬀect of such correlation. Figure 2 shows

8

S.H. Ahmed and T. Fl˚ a Species variation at level 2

0.4

0.4

0.3

0.3

0.2

0.2 3

0.5

0.1

u

u

3

Species variation at level 1 0.5

0.1

0

0

−0.1

−0.1

−0.2

−0.2

−0.3 −0.2

0

0.2 u 2

0.4

0.6

−0.3 −0.2

−0.1

0

0.1 u

0.2

0.3

0.4

2

Fig. 2. Species variation at decomposition levels (from left) 1 and 2 based on the transformed detail coeﬃcients along the 2nd and 3rd eigenvectors obtained from energy of the ensembles across the sequence family, based on detail coeﬃcients at these levels

species variation along 2nd and 3rd eigenvectors (u2 , u3 ) based on 27 hydrophobicity sequences (trypsin). We see that the extrinsic variations, especially due to cold-and warm adapted trypsins, are represented by the detail coeﬃcients at level 1 and 2, while correlated variations at level 3 and 4 (not shown) are mostly due to within cluster variations. We could use the information at the two ﬁnest level to extract extrinsic correlations associated to environmental eﬀects like cold adaptation, instead we chose to remove redundancy from the L set of transformed sequences. Since each of the L transformed detail coeﬃcients are uncorrelated, we performed one-dimensional Hard-thresholding (”keep or kill” approach) using a universal threshold [11] based on √ eigenvariances σi derived from diagonalization of R at level 1. That is, εi = 2σi logN for the i-th trans (j) , i = 1, . . . , L, j = 1, . . . , 4. We chose formed sequence of detail coeﬃcients, D i the eigenvariances derived at level 1 for two main reasons: (1) a better estimate of σi can be obtained due to high noise level, (2) σi determined from coarser level with more large coeﬃcients can eliminate signiﬁcant coeﬃcients. Finally, we performed diagonalization on the covariance matrix based on approximation coeﬃcients at the ﬁnal level A(4) . In this case, we removed redundancy by keep (j) , i = 1, 2 (using a scree-plot). The ing the ﬁrst two signiﬁcant components of A i output of the inverse transformed coeﬃcients (by transpose UT and transpose of analysis ﬁlter bank due to orthogonality) corresponds to a smoothed version (N ) of the original property sequences Cl . For visualization and extraction of extrinsic variations both across and along ¯ (N ) ˆ (N ) − C ˆ the sequence family, we performed mean subtraction, that is , (C ) l l before computing the covariance. Components of the ﬁrst two eigenvectors and the corresponding eigensequences (projections on the ﬁrst two orthogonal directions), obtained from diagonalization of the covariance matrix based on the mean subtracted smoothed property sequences, were used to visualize species variation and the underlying residue positions responsible for this variation.

Evolutionary Parameters in Sequence Families

4

9

Results and Discussion

We presented a viable way of representing an aligned family of protein sequences through evolutionary parameterization of features. Based on these parameterized 2

2

1.5

1.5 1 Hydrophobicity

Hydrophobicity

1 0.5 0 −0.5

0.5 0 −0.5

−1

−1

−1.5

−1.5

−2 0

−2 0

20 40 60 80 100 120 140 160 180 200 220 Residue number

20 40 60 80 100 120 140 160 180 200 220 Residue number

Fig. 3. Pattern of original hydrophobicity values along the 27 sequences and their smoothed version based on trypsin sequence data

Polarity 0.4

0.2

0.2 2nd eigenvector

2nd eigenvector

Hydrophobicity 0.4

0

−0.2

−0.4

−0.6

0

−0.2

−0.4

−0.2 −0.1

0 0.1 0.2 Ist eigenvector

0.3

−0.6 −0.4

0.4

4

−0.2

0 0.2 1st eigenvector

0.4

0.6

4 3 2

0

Polarity

Hydrophobicity

2

−2

1 0 −1

−4 −2 −6 0

25

50

75 100 125 150 175 200 225 Sequence position

−3 0

25

50

75 100 125 150 175 200 225 Sequence position

Fig. 4. Species variation based on trypsin sequence data in the space spanned by the ﬁrst two eigenvectors (1st row) and the corresponding excess eigensequences, 1st (solid line) and 2nd (dashed line). The sequenes are based on hydrophobicity (on the left side) and poliarity (right).

10

S.H. Ahmed and T. Fl˚ a

features under reversible Markov models, we carried out a multiscale and multivariable data analysis on distinct alignment of L trypsin and elastase sequences of length N (L is 27/29, N is 221/240 for trypsin/elastase) based on hydrophobicity (N ) and polarity sequences Cl . Since the sequences in both groups of enzymes are closely related, for simplicity, we removed the few columns containing gaps. The basis of the data analysis is that covariation of residue sites in evolution is related to mainly structural and functional site ﬁtness as parameterized by models of amino acid distributions and certain amino acid properties. These correlated residues based on property sequences show deviation from a common position dependent mean value. In principle, this requires a sequence data of suﬃcient size and diversity (at each site) to compute such position dependent mean values for each cluster. Therefore, we used multivariate method to remove background noise and extract extrinsic correlations due to environmental eﬀects like cold adaptation. Description of the method is given in details with some illustrations in Sect. 3. The idea is to use orthogonal wavelets to obtain a sparse representation of the property sequences based on detail coeﬃcients and perform diagonalizing transformation in the wavelet domain to decorrelate the small number of large detail coeﬃcients (representing the sequence ensembles) with high energy. Onedimensional thresholding, in this case, Hard thresholding can then be applied on the uncorrelated wavelet coeﬃcients in order to separate out the larger coeﬃcients that are associated with variations due to environmental eﬀects like cold cold adaptation. The resulting backward transformed, denoised property sequences are smoothed version of the original property sequences as shown in Fig. 3. In this ﬁgure (to the right), the thick horizontal curve represents the Elastases (Hydrophobicity)

0.2

1st eigensequence (elastases) 3

0

2 Hydrophobicity

3rd eigensquence

4

0.1

−0.1 −0.2 −0.3 −0.4

0.2 0.2

0

−0.2

0 −0.2

1 0 −1 −2 −3 −4 0

25

50

75 100 125 150 175 200 225 Sequence position

Fig. 5. Drift of centroids, as described by the species covariance matrix (ﬁgure on the left) and centroid sequence (ﬁgure on the right) based on hydrophobicity. The solid triangles and circles represent cold-adapted elastases of type I and II, respectively. The corresponing open triangles and circles represent their warm-active counterparts. The squares represent elastase type-III with no cold representatives.

Evolutionary Parameters in Sequence Families

11

Fig. 6. 3D structure of trysin from Bovine (PDB:3PTB) showing surpport of the components of the excess eigensequences based on hydrophobicity (green) and polarity (red). Region of active site is shown in yellow.

centroid sequence of the trypsin family based on 27 hydrophobicity sequences. The larger spikes are due to extrinsic variations that are associated with residue (hydrophobic) determinants of cold-adapted trypsins. The smaller spikes are due to intrinsic variations, partly due to that there are several clusters, in the trypsin case, the warm-active higher vertebrates and other ﬁshw, and partly due to assymmetries in covariance induced by evolutionary time since two leaf nodes were merged. Such drift of centroids, in terms of species and position variations can be clearly observed in the subspace spanned by the ﬁrst two eigenvectors with largest variances, derived from diagonalization based on the smoothed property sequences after subtracting the mean proﬁle in Fig 3, and projecting the excess variations from th mean proﬁle along the two eigenvectors. Drift of centroids in terms of covariation and mean sequences are shown in Fig. 4 and Fig. 5. Fig. 6 shows support of the excess eigensequences, namely, in the N-and C terminals for distability and around active site [20].

References [1] Goldman, N., Yang, Z.: A codon based model of nucleotide substitution for protein-coding DNA sequences. Mol. Biol. Evol. 11(5), 725–736 (1994) [2] Felsenstein, J.: Evolutionary trees from DNA sequences: A maximum likelihood approach. J. Mol. Evol. 17, 368–376 (1981) [3] Pollock, D., Taylor, W., Goldman, N.: Coevolving protein residues: Maximum likelihood identiﬁcation and relationship to structure. J. Mol. Biol. 287, 187–198 (1999) [4] Jones, D.T., Taylor, W.R., Thornton, J.M.: The rapid generation of muatation data matrices from protein sequences. Comp. Appl. Biosci. 8, 275–282 (1992)

12

S.H. Ahmed and T. Fl˚ a

[5] Kishino, H., Miyata, T., Hasegawa, M.: Maximum likelihood inference of protein phylogenies and the origin of chloroplasts. J. Mol. Evol. 31, 151–160 (1990) [6] Hasegawa, M., Fujiwara, M.: Relative eﬃciencies of the maximum likelihood, maximum parsimony, and neighbor joining methods for estimating protein phylogeny. Mol. Phylog. and Evol. 2, 1–5 (1993) [7] Johnson, R.A., Wichern, D.W.: Applied Multivariate Statistical Analysis, 5th edn. Prentice Hall, Upper Saddler River (2002) [8] Mallat, S.G.: A theory for multiresolution signal decomposition: the wavelet representation. IEEE Trans. Pattern Anal. Mach. Intell. 11(7), 674–693 (1989) [9] Suzuki, T., Srivastava, A., Kurokawa, T.: cDNA cloning and phylogenetic analysis of pancreatic serine proteases from Japanese ﬂounder, Paralichthys olivaceus. Comp. Biochem. and Physiol. Part B 131, 63–70 (2001) [10] Strang, G., Nguyen, T.: Wavelets and Filter Banks. Wellesley-Cambridge Press (1997) [11] Donoho, D., Johnstone, I.: Ideal spatial adaptation by wavelet shrinkage. Biometrika 81(3), 425–455 (1994) [12] Koshi, J.M., Mindell, D.P., Goldstein, R.A.: Beyond mutation matrices: physicalchemistry based evolutionary models. In: Miyano, S., Takagi, T. (eds.) Genome informatics, pp. 80–89. Universal Academy Press, Tokyo (1997) [13] Casari, G., Sander, C., Valencia, A.: A method to predict functional residues in proteins. Nat. Struc. Biol. 2(2), 171–178 (1995) [14] Durbin, R., Eddy, S., Krogh, A., Mitchison, G.: Biological sequence analysis: Probabilistic models of proteins and nucleic acids. Cambridge University Press, Cambridge (1998) [15] Whelan, S., Goldman, N.: A general empirical model of protein evolution derived from multiple protein famililies using a maximum-likelihood approach. Mol. Biol. and Evol. 18, 691–699 (2001) [16] Goldman, N., Whelan, S.: A novel use of equillibrium frequencies in models of sequence evolution. Mol. Evol. 11, 1821–1831 (2002) [17] Ahmed, S.H., Fl˚ a, T.: Estimation of evolutionary average hydrophobicity proﬁle from a family of protein sequences. In: Rajapakse, J.C., Schmidt, B., Volkert, L.G. (eds.) PRIB 2007. LNCS (LNBI), vol. 4774, pp. 158–165. Springer, Heidelberg (2007) [18] Feller, G., Gerday, C.: Psychrophilic enzymes: molecular basis of cold-adaptation. Cell Mol. Life Sci. 53, 830–841 (1997) [19] Georlette, D., Blaise, V., Collins, T., D’Amico, S., Gratia, E., Hoyoux, A., Marx, J.C., Sonan, G., Feller, G., Gerday, C.: Some like it cold: biocatalysis at low temperatures. FEMS icrobiol. Rev. 28, 25–52 (2004) [20] Schrøder, H.-K., Willassen, N.P., Smal˚ as, A.O.: Residue determinants and sequence analysis of cold-adapted trypsins. Extremophiles (2), 5–219 (1999)

MProfiler: A Profile-Based Method for DNA Motif Discovery Doaa Altarawy, Mohamed A. Ismail, and Sahar M. Ghanem Computer and Systems Engineering Dept. Faculty of Engineering, Alexandria University Alexandria 21544, Egypt {doaa.altarawy,maismail,sghanem}@alex.edu.eg

Abstract. Motif Finding is one of the most important tasks in gene regulation which is essential in understanding biological cell functions. Based on recent studies, the performance of current motif ﬁnders is not satisfactory. A number of ensemble methods have been proposed to enhance the accuracy of the results. Existing ensemble methods overall performance is better than stand-alone motif ﬁnders. A recent ensemble method, MotifVoter, signiﬁcantly outperforms all existing stand-alone and ensemble methods. In this paper, we propose a method, MProﬁler, to increase the accuracy of MotifVoter without increasing the run time by introducing an idea called center proﬁling. Our experiments show improvement in the quality of generated clusters over MotifVoter in both accuracy and cluster compactness. Using 56 datasets, the accuracy of the ﬁnal results using our method achieves 80% improvement in correlation coeﬃcient nCC, and 93% improvement in performance coeﬃcient nP C over MotifVoter. Keywords: Bioinformatics, DNA Motif Finding, Clustering.

1

Introduction

Computational identiﬁcation of overrepresented patterns (motifs) in DNA sequences is a long-standing problem in Bioinformatics. Identiﬁcation of those patterns is one of the most important tasks in gene regulation which is essential in understanding biological cell functions. Over the last few years, the sequencing of the complete genome of large variety of species (including human) has accelerated the advance in the ﬁled of Bioinformatics [1]. The problem of DNA motif ﬁnding is to locate common short patterns in a set of co-regulated gene promoters (DNA sequences). Those patterns are conserved but still tend to vary slightly [2]. Normally the patterns (motifs) are fairly short (5 to 20 base pair long) [3]. Those motifs are the locations where transcription factors (TF) bind to in order to control protein production in cells. DNA motifs are also called transcription factor binding sites (TFBS). Many computational methods are being proposed to solve this problem. Their strategies can be divided into two main classes: exhaustive enumeration and probabilistic methods [4]. V. Kadirkamanathan et al. (Eds.): PRIB 2009, LNBI 5780, pp. 13–23, 2009. c Springer-Verlag Berlin Heidelberg 2009

14

D. Altarawy, M.A. Ismail, and S.M. Ghanem

Review of the ﬁeld and description of some motif ﬁnding methods can be found in [1,2,3,4,5,6]. Several studies show that current motif ﬁnding methods are unsatisfactory [3,5,7]. In Tompa et al.’s [7] assessment, 13 motif ﬁnding methods were examined. Their study shows that the accuracy of those methods in terms of sensitivity and speciﬁcity is low. Despite the large number of methods being proposed for motif ﬁnding, it is still a challenging problem. Motifs found by diﬀerent methods are not always the same, meaning that their results can be complementary [7,8]. Although the accuracy of a single motif ﬁnder method is low, ensemble methods are promising. Ensemble methods are compound algorithms that combine the results of multiple predictions from multiple algorithms. Thus, combing more than one stand-alone method can increase the sensitivity (more true positives), but without a good ﬁltering method it will reduce the speciﬁcity (more false positives) [7]. In the last few years, some ensemble methods have been proposed such as SCOPE [9], BEST [10], EMD [11], and more recently, MotifVoter [8]. MotifVoter signiﬁcantly outperforms all existing stand-alone and ensemble methods. For example, in Tompa’s benchmark MotifVoter increased the accuracy of the results (correlation coeﬃcient nCC) over the best stand-alone method by more than 100%. MotifVoter formulates the ensemble motif ﬁnding as an optimization search problem and uses a heuristic to generate a search space consisting of clusters of motifs. It uses a variance-based objective function to select the best cluster among the generated search space. In this paper, we propose a method called MProﬁler to increase the accuracy of MotifVoter by using a new heuristic to generate the search space. Enhancing the search space in both accuracy and quality improves the ﬁnal results and reduces the chances of falling in local maximums. A more accurate search space is the one that has higher percentage of motifs with higher collaboration coeﬃcient with the true motifs. The quality of the search space is the compactness of its higher accuracy clusters, since the selection function is variance-based. The proposed technique for search space generation has more than 200% improvement over MotifVoter’s in terms of percentage of generated sets having nCC greater than 0.5. In addition, the generated sets are having higher mean and lower variance (i.e more compact) when compared to the sets generated by MotifVoter’s approach. Having compact sets is a desirable feature for the objective function because it is variance-based. In our experiments, we compare the proposed MProﬁler technique with MotifVoter on 56 diﬀerent datasets that are proposed on Tompa’s benchmark [7]. The correlation coeﬃcient nCC and performance coeﬃcient nP C are used as measures of accuracy for Motif ﬁnding methods. Our experimental results show that MProﬁler increases correlation coeﬃcient by 80% over MotifVoter on the same benchmark. In addition, MProﬁler increases the performance coeﬃcient by 93%. The rest of the paper is organized as follows: Section 2 provides an overview of related word and the motivation of our proposed algorithm. Section 3 introduces

MProﬁler: A Proﬁle-Based Method for DNA Motif Discovery

15

the MProﬁler algorithm. Section 4 presents experimental results and discussions. Section 5 concludes the paper along with future work.

2

Motivation

Many ensemble motif ﬁnding methods make use of the observation that true motifs are shared by multiple motif ﬁnders. In addition, MotifVoter in addition proposed the use of the negative information from the false positive motifs that are usually predicted by a few or even one motif ﬁnder. MotifVoter performs two selective criteria to ﬁnd an optimal cluster [8]: 1. Discriminative criterion: select a cluster of motifs that are not only similar, but also have the property that motifs outside the cluster are distant from each other. This is done with a variance-based objective function (see equation (6) in methods section). 2. Consensus criterion: the selected cluster must be predicted by as many motif ﬁnders as possible. After the cluster is chosen, one representative motif is extracted from the cluster (i.e. a cluster center) using a process called site extraction. The enumeration technique is unfeasible since it takes exponential time. Instead MotifVoter uses a simple heuristic to generate the search space. Let P be the set of all input motifs. MotifVoter only considers subsets Xz,j ={z, p1, ....., pj } for every z ∈ P and for every 1 < j < |P | − 1, where pi ’s are sorted descending according to its similarity to z, i.e. sim(z, pi ) > sim(z, pi+1 ) and pi ∈ P . The heuristic used by MotifVoter to generate the search space produces good search space for the motif ﬁnding problem. MotiVoter outperform all stand-alone and ensemble motif ﬁnding methods in terms of accuracy [8]. Because the objective function is variance-based, it favors compact clusters even if they are not the optimal ones. In addition, using a variance-based function with diﬀerent size sets can mislead the selection to smaller clusters, since smaller clusters appear more compact. Therefore, the capability of the objective function to select a more accurate cluster can be improved by making the clustered sets of nearly equal size. In this paper, an ensemble motif ﬁnding method, MProﬁler, is proposed that improves MotifVoter search space in three desirable features: 1. Increase the percentage of higher accuracy sets. 2. The generated sets are more compact, i.e. having higher mean and lower variance. 3. Clusters examined by the objective function are nearly of equal size. The proposed MProﬁler technique constructs proﬁles of similar motifs predicted by diﬀerent ﬁnders. Then, the proﬁles are used to generate the search space. The constructed proﬁles increase the similarities between motifs if they exist, thus giving them higher score from the variance-based function. Details of how to generate and use the proﬁles are described in the following section.

16

3 3.1

D. Altarawy, M.A. Ismail, and S.M. Ghanem

MProfiler Methods Definitions

Problem Statement: Ensemble DNA Motif ﬁnding problem can be formalized as follows: given a set of DNA sequences, and the output of diﬀerent motif ﬁnding methods, each output is a set of motifs (i.e n ∗ m motifs in total), it is required to construct a representative motif that is the best approximate of the real motif which is shared in the input sequences. Motif: A motif is a set of sites where each site is a continuous range of positions representing a subsequence from a DNA sequence. Motif Similarity: In [8], the similarity between two motifs is deﬁned by (1), where cov(xi ) is all positions covered by motif x. From that motif similarity deﬁnition 0 ≤ sim(xi , xj ) ≤ 1, and sim(xi , xi ) = 1. sim(xi , xj ) =

cov(xi ) ∩ cov(xj ) cov(xi ) ∪ cov(xj )

(1)

Cluster Similarity: The similarity among a cluster X of motifs is deﬁned as the mean pairwise similarity among its members given by (2), where |X| is the number of motifs in the set X. sim(xi , xj ) sim(X) =

xi ,xj ∈X

2

|X|

(2)

Cluster Center: We deﬁne the center of a cluster as the motif that consists of all positions covered by two or more motifs in the cluster, i.e. it is the pairwise intersection of its members, and can be calculated using (3). center(X) = [cov(xi ) ∩ cov(xj )] (3) xi ,xj ∈X xi =xj

Consensus Cluster Center (Proﬁle): We deﬁne consensus center of a cluster as the motif that consists of all positions covered by at least two motifs, such that the intersecting motifs are predicted by two diﬀerent motif ﬁnders and can be calculated using (4). An extra reﬁnement is added by removing sites (continuous positions) that has only two contributing ﬁnding methods. consCenter(X) = [cov(xi ) ∩ cov(xj )] (4) xi ,xj ∈X f inder(xi )= f inder(xj )

Cluster Weight: There are several weighing functions that can be used to give a score to a set of motifs. In this paper, we compare our technique to MotifVoter [8], and apply the same weight used by MotifVoter as deﬁned by (5).

MProﬁler: A Proﬁle-Based Method for DNA Motif Discovery

weight(X) =

sim(X)

2

xi ,xj ∈X

(sim(xi , xj ) − sim(X))

17

(5)

Objective Function: The objective function is deﬁned in [8] as the ratio between the weight of a chosen set X, and the weight of remaining motifs not ¯ as shown in (6). belonging to X (i.e. X) A(X) =

weight(X) ¯ weight(X)

(6)

Accuracy Measures: Following Tompa et al. [7] and others, the following accuracy measures are considered. Sensitivity is the percentage of known sites that the algorithm was able to ﬁnd correctly. Specificity is the percentage of the predicted sites that are correct. – Nucleotide Correlation Coeﬃcient (nCC): Nucleotide Correlation Coeﬃcient combines both sensitivity and speciﬁcity (Positive predictive value). As nCC calculated by (7), if the predicted motif perfectly coincide with the known motif, then the value of nCC is 1. If they are independent, then the value of nCC is 0. TP, FP, TN and FN are nucleotide true positive, false positive, true negative and false negative respectively [7]. nCC =

T P.T N − F N.F P (T P + F N )(T N + F P )(T P + F P )(T N + F N )

(7)

– Performance Coeﬃcient (nP C): Performance coeﬃcient captures both speciﬁcity and sensitivity in a single accuracy measure using a simple equation. It is the ratio between true positives (true motifs) and all regions that is marked as motifs either correctly or incorrectly. Nucleotide level performance coeﬃcient (nP C) is deﬁned in 8. It ranges from 0 (worst) to 1 (best). nP C =

3.2

TP TP + FN + FP

(8)

MProfiler Algorithm

Given the output of m stand-alone motif ﬁnding methods, it is desirable to produce a motif that best approximates the real motif. MotifVoter algorithm has three steps. First, a search space consisting of sets of motifs is generated using the heuristic described in section 2. Second, a set is chosen from the generated search space the maximizes the variance-based objective function in equation (6) with consensus criterion satisﬁed. Finally, the ﬁnal motif is extracted from the chosen set as described in MotifVoter [8]. Instead of using n ∗ m motifs given by the stand-alone ﬁnders in MotifVoter, the proposed MProﬁler technique uses a set of generated motif proﬁles called consensus cluster centers as deﬁned in section 3.1. Using those proﬁles helps

18

D. Altarawy, M.A. Ismail, and S.M. Ghanem

increase similarities between motifs in the same cluster, if it exists, thus giving them higher score from the variance-based function in equation (6). A proﬁle has at least 3 intersecting motifs predicted by 3 diﬀerent motif ﬁnders. The generation of the proﬁles is described in Algorithm (1).

Input : set P contains n ∗ m Motifs. Output: one Motif and PWM for its aligned sites. 1 2 3 4 5 6 7 8 9 10 11 12

foreach xi , xj ∈ P do compute sim(xi , xj ); proﬁles ← ∅; foreach motif z ∈ P do X ← ∅; sortedP ← sort P according to sim(z,pi ); for top n pi ∈ sortedP do X ← X + pi ; if (sim(profiles.lastElement,consCenter(X))< ) then proﬁles ← proﬁles + consCenter(X); end end acceptedCluster ← MotifVoter(profiles); extractSites and generate PWM ; Algorithm 1. MProﬁler pseudo code

The condition in line 8 avoids obtaining very similar proﬁles from the same group which actually represent the same proﬁle. A new proﬁle is generated only if it diﬀers by at least within its group where is any similarity value between 0 and 1. Small values generate larger number of proﬁles, which will be merged in line 11. In line 11, MotifVoter algorithm is used to ﬁnd the cluster X using the objective function in equation (6). Consensus criterion is not needed in this step because it is already applied in generating the proﬁles. 3.3

Site Extraction

Final sites are extracted from the selected cluster of motifs as in equation (3). Accepted positions are the positions covered by more than one motif in the cluster. The sites are then aligned using MUSCLE [12] and a Position Weight Matrix (PWM) is generated. PWM is a common representation of motifs. A position weight matrix is a matrix of score values that gives a weighted match to any given substring of ﬁxed length. It has one row for each symbol of the alphabet (A, C, G, T), and one column for each position in the motif. 3.4

Time Complexity

Given m motif ﬁnders, each with n predicted motifs, the time complexity of our method is O(m2 n2 ), which is the same for MotifVoter. First, at most mn2

MProﬁler: A Proﬁle-Based Method for DNA Motif Discovery

19

proﬁles are generated. Then, for each proﬁle, the objective function is calculated for m subsets. As in MotifVoter, since motifs are added one by one, the objective function can be calculated in a constant time from the previous value. Unlike MotifVoter, for each proﬁle MProﬁler algorithm did not need to add all other proﬁles to the growing clusters of motifs because sets are more compact. Instead the ﬁrst most similar m proﬁles are examined. Thus the ﬁnal running time is O(m2 n2 ).

4 4.1

Results and Discussion Stand-Alone Motif Finders

We used the same 10 ﬁnders used by MotifVoter with the same parameters described in [8]. The stand-alone motif ﬁnders are: MEME [13], Weeder [14], Bioprospector [15], SPACE [16], MDScan [17], ANN-Spec [18], MotifSampler [19], MITRA [20], AlignACE [21], and Improbizer [22]. Any other DNA motif ﬁnder can be used. For each ﬁnder, the ﬁrst 30 predicted motifs are taken. The top 30 motifs achieve maximum sensitivity (nSn) on Tompa’s benchmark [8]. Since Tompa’s benchmark is a good representative of real motifs, using top 30 motifs for other datasets is a quite reasonable approximation. 4.2

Datasets

Datasets used in the comparison are the Tompa et al. [7] benchmark consisting of 56 diﬀerent datasets, which cover 4 diﬀerent species (Mouse, Fruit ﬂy, Human and Yeast). The datasets are constructed based on real transcription factor binding sites (TFBS). 4.3

Improvement in Search Space

Accuracy: Using performance coeﬃcient nPC as a measure of accuracy, MProﬁler has 380% improvement over MotifVoter in percent of generated sets having accuracy nP C > 0.5. Fig. 1 shows the total improvement in nPC for all 56 datasets. Improvement of nPC over MotifVoter is calculated using (9). Improvement(nP C) =

nP CMP rof iler − nP CMotif V oter nP CMotif V oter

(9)

Also MProﬁler’s search space (generated sets) has more than 200% improvement over MotifVoter in percentage of generated sets having higher correlation coeﬃcient, i.e nCC > 0.5 as shown in Fig. 2. The ﬁgure shows the combined nCC for all 56 datasets. Improvement of nCC over MotifVoter is calculated using (10). Improvement(nCC) =

nCCMP rof iler − nCCMotif V oter nCCMotif V oter

(10)

20

D. Altarawy, M.A. Ismail, and S.M. Ghanem

Fig. 1. The overall improvement in accuracy (nPC) of MProﬁler over MotifVoter. Yaxis: percent improvement of number of generated clusters having nPC greater than or equal to x. MProﬁler’s search space (generated sets) has 380% improvement over MotifVoter in percentage of generated sets having higher accuracy (i.e with nP C > 0.5).

Fig. 2. The overall improvement in accuracy (nCC) of MProﬁler over MotifVoter. Y-axis: percent improvement of number of generated clusters having nCC greater than or equal to x. MProﬁler’s search space (generated sets) has more than 200% improvement over MotifVoter in percentage of generated sets having higher accuracy (i.e with nCC > 0.5).

More accurate clusters mean a higher probability to ﬁnd the correct set, given that their quality are better. Notice that MProﬁler has more improvement in higher nCC and nP C values than lower ones which is a desirable feature (i.e. it increases the percentage of higher quality sets more than lower quality sets). Average Mean and Variance: Since the objective function is based on the mean and the variance of cluster similarity (see equation (6)), it is desirable

MProﬁler: A Proﬁle-Based Method for DNA Motif Discovery

21

Fig. 3. Average mean and variance of similarity in generated clusters. y-axis is the average mean/variance of similarity for sets having nCC greater than or equal to x.

to make higher accuracy clusters more compact, i.e. with higher mean and lower variance. MProﬁler improves both mean (higher value) and variance (lower value) over MotifVoter which led to the improvement in the optimal cluster selected. Fig. 3 shows the improvement of mean and variance of MProﬁler generated sets over MotifVoter for all 56 datasets. 4.4

Comparison of Final Results

On 56 diﬀerent datasets, MProﬁler has 80% improvement in accuracy (nucleotide correlation coeﬃcient nCC) over MotifVoter results using the same input and

22

D. Altarawy, M.A. Ismail, and S.M. Ghanem

the same objective function implemented as described by Wijaya et al. [8]. Also MProﬁler has 93% improvement in accuracy using performance coeﬃcient nP C as a measure of accuracy. Comparison with the results stated in [8] was not possible because the exact implementation of the objective function is not described in their paper and the source code is not available.

5

Conclusion

Ensemble methods provide improvement in motif ﬁnding accuracy without the need to use additional data (such as phylogenetic information or characterization of the domain structure of the transcription factor), which are not always available. Our proposed method, MProﬁler, improves the best existing motif ﬁnding ensemble method, MotifVoter, in terms of accuracy without increasing time complexity. On the widely used Tompa’s benchmark with 56 diﬀerent datasets, MProﬁler’s search space has 200% improvement over MotifVoter in percentage of generated sets having higher accuracy (i.e with nCC > 0.5), and 380% improvement for sets having performance coeﬃcient nP C > 0.5. For ﬁnal motif results, our method achieves 80% improvement in ﬁnal accuracy using correlation coeﬃcient, and 93% improvement using performance coeﬃcient over MotifVoter.

6

Future Work

The problem of computational motif ﬁnding is still standing in bioinformatics. Even with ensemble methods the accuracy is low. The upper-bound of ensemble methods is limited by the underlying stand-alone ﬁnders. Thus, using better stand-alone ﬁnders will raise the maximum possible senstivity for ensemble methods. Moreover, other objective functions can be suggested to enhance the accuracy. The idea of generating the proﬁles can also be used with other standalone or ensemble methods.

References 1. Qiu, P.: Recent advances in computational promoter analysis in understanding the transcriptional regulatory network. Biochemical and Biophysical Research Communications 309(3), 495–501 (2003) 2. Wei, W., Yu, X.D.: Comparative analysis of regulatory motif discovery tools for transcription factor binding sites. Genomics Proteomics Bioinformatics 5(2), 131– 142 (2007) 3. Das, M., Dai, H.K.: A survey of DNA motif ﬁnding algorithms. BMC Bioinformatics 8(suppl. 7) (2007) 4. Li, N., Tompa, M.: Analysis of computational approaches for motif discovery. Algorithms for Molecular Biology 1(1), 8–15 (2006) 5. Hu, J., Li, B., Kihara, D.: Limitations and potentials of current motif discovery algorithms. Nucleic Acids Res. 33(15), 4899–4913 (2005)

MProﬁler: A Proﬁle-Based Method for DNA Motif Discovery

23

6. Stormo, G.D.: DNA binding sites: representation and discovery. Bioinformatics 16(1), 16–23 (2000) 7. Tompa, M., et al.: Assessing computational tools for the discovery of transcription factor binding sites. Nature Biotechnology 23, 137–144 (2005) 8. Wijaya, E., Yiu, S., Son, N.T., Kanagasabai, R., Sung, W.: Motifvoter: a novel ensemble method for ﬁne-grained integration of generic motif ﬁnders. Bioinformatics 24, 2288–2295 (2008) 9. Chakravarty, A., Carlson, J.M., Khetani, R.S., Gross, R.H.: A novel ensemble learning method for de novo computational identiﬁcation of DNA binding sites. BMC Bioinformatics 8, 249–263 (2007) 10. Che, D., Jensen, S., Cai, L., Liu, J.S.: BEST: Binding-site estimation suite of tools. Bioinformatics 21(12), 2909–2911 (2005) 11. Hu, J., Yang, Y.D., Kihara, D.: EMD: an ensemble algorithm for discovering regulatory motifs in DNA sequences. BMC Bioinformatics 7, 342–454 (2006) 12. Edgar, R.C.: MUSCLE: multiple sequence alignment with high accuracy and high throughput. Nucleic Acids Res. 32(5), 1792–1797 (2004) 13. Bailey, T.L., Elkan, C.: Unsupervised learning of multiple motifs in biopolymers using expectation maximization. Machine Learning 21, 51–80 (1995) 14. Pavesi, G., Mereghetti, P., Mauri, G., Pesole, G.: Weeder Web: discovery of transcription factor binding sites in a set of sequences from co-regulated genes. Nucleic Acids Res. 32(Web Server issue) (July 2004) 15. Liu, X., Brutlag, D.L., Liu, J.S.: Bioprospector: discovering conserved DNA motifs in upstream regulatory regions of co-expressed genes. In: Pac. Symp. Biocomput., pp. 127–138 (2001) 16. Wijaya, E., Kanagasabai, R., Yiu, S.-M.M., Sung, W.-K.K.: Detection of generic spaced motifs using submotif pattern mining. Bioinformatics 23(12), 1476–1485 (2007) 17. Liu, X.S., Brutlag, D.L., Liu, J.S.: An algorithm for ﬁnding protein-DNA binding sites with applications to chromatin-immunoprecipitation microarray experiments. Nat. Biotechnol. 20(8), 835–839 (2002) 18. Workman, C.T., Stormo, G.D.: ANN-Spec: a method for discovering transcription factor binding sites with improved speciﬁcity. In: Pac. Symp. Biocomput., pp. 467– 478 (2000) 19. Thijs, G., et al.: A higher-order background model improves the detection of promoter regulatory elements by gibbs sampling. Bioinformatics 17(12), 1113–1122 (2001) 20. Eskin, E., Pevzner, P.A.: Finding composite regulatory patterns in DNA sequences. Bioinformatics 18(suppl. 1) (2002) 21. Huang, H.-D., Horng, J.-T., Sun1, Y.-M., Tsou, A.-P., Huang, S.-L.: Identifying transcriptional regulatory sites in the human genome using an integrated system. Nucleic Acids Res. 32(6), 1948–1956 (2004) 22. Ao, W., Gaudet, J., Kent, W.J., Muttumu, S., Mango, S.E.: Environmentally induced foregut remodeling by PHA-4/FoxA and DAF-12/NHR. Science 305, 1743– 1746 (2004)

On Utilizing Optimal and Information Theoretic Syntactic Modeling for Peptide Classification Eser Ayg¨ un1 , B. John Oommen2 , and Zehra Cataltepe3 1 2

Department of Computer Eng., Istanbul Technical University, Istanbul, Turkey School of Computer Science, Carleton University, Ottawa, Canada : K1S 5B6, and Adjunct Professor at the University of Agder in Grimstad, Norway 3 Department of Computer Eng., Istanbul Technical University, Istanbul, Turkey [email protected]

Abstract. Syntactic methods in pattern recognition have been used extensively in bioinformatics, and in particular, in the analysis of gene and protein expressions, and in the recognition and classiﬁcation of biosequences. These methods are almost universally distance-based. This paper concerns the use of an Optimal and Information Theoretic (OIT) probabilistic model [11] to achieve peptide classiﬁcation using the information residing in their syntactic representations. The latter has traditionally been achieved using the edit distances required in the respective peptide comparisons. We advocate that one can model the diﬀerences between compared strings as a mutation model consisting of random Substitutions, Insertions and Deletions (SID) obeying the OIT model. Thus, in this paper, we show that the probability measure obtained from the OIT model can be perceived as a sequence similarity metric, using which a Support Vector Machine (SVM)-based peptide classiﬁer, referred to as OIT SVM, can be devised. The classiﬁer, which we have built has been tested for eight diﬀerent “substitution” matrices and for two diﬀerent data sets, namely, the HIV1 Protease Cleavage sites and the T-cell Epitopes. The results show that the OIT model performs signiﬁcantly better than the one which uses a Needleman-Wunsch sequence alignment score, and the peptide classiﬁcation methods that previously experimented with the same two datasets. Keywords: Biological Sequence Analysis, Optimal and Information Theoretic Syntactic Classifcation, Peptide Classiﬁcation, Sequence Processing, Syntactic Pattern Recognition.

1

Introduction

The syntactic methods that have been traditionally used in the analysis, recognition and classiﬁcation of bioinformatic data include distance-based methods, and probabilistic schemes which are, for example, Markovian. A probabilistic model, distinct from these, is the one proposed by Oommen and Kashyap [11]. The model, referred to as the OIT model, attains the optimal and information theoretic bound. This paper reports the ﬁrst known results in which the OIT model has been applied in any bioinformatic application. V. Kadirkamanathan et al. (Eds.): PRIB 2009, LNBI 5780, pp. 24–35, 2009. c Springer-Verlag Berlin Heidelberg 2009

On Utilizing Optimal and Information Theoretic Syntactic Modeling

25

Peptides are relatively short amino acid chains that occur either as separate molecules or as building blocks for proteins. Apart from their signiﬁcance in analyzing proteins, peptides themselves may have various distinct chemical structures that are themselves related to diﬀerent molecular functions. These functions, such as cleavage or binding, while being interesting in their own right, have also been shown to be important in areas such as biology, medicine, drug design, disease pathology, and nanotechnology Indeed, for more than a decade, researchers have sought computational techniques to rapidly identify peptides that are known to be, or can be, related to certain molecular functions. The research in peptide classiﬁcation is not new –indeed, a host of techniques have been proposed for in silico peptide classiﬁcation1 . In 1998, Cai and Chou [3] presented one of the pioneering works in this area. They classiﬁed 8residue peptides and used artiﬁcial neural networks with 20 input nodes per residue, thus involving a total of 160 input nodes. In their work, each amino acid was encoded using 20 bits so that the 20 amino acids were encoded as A = 100 . . . 00, B = 010 . . . 00, . . . , Y = 000 . . . 01. Similarly, Zhao et al. [15] mapped the amino acid sequences of peptides directly into feature vectors and fed them into a Support Vector Machine (SVM). They, however, represented the amino acids by a set (more speciﬁcally, ten) of their biophysical properties, such as hydrophobicity or beta-structure preference, instead of an orthonormal representation, as advocated by [3]. By resorting to such a representation, they were eventually able to reduce the dimensionality of the input space by 50%. To further increase the information density of input vectors, Thomson et al. [13] used bio-basis artiﬁcial neural networks, which are a revision of radial-basis function networks, that use biological similarities rather than spatial distances. This work was subsequently enhanced by Trudgian and Yang [14] by optimizing the substitution matrices that are used to compute the latter biological similarities. Kim et al. [8] followed a rule-based approach to achieve results which were interpretable. It should be mentioned that there were also earlier studies based on the properties of quantitative matrices, binding motifs and hidden Markov models, which should really be treated as precursors to the results cited above. The diﬀerences between our results and those which use Hidden Markov Models (HMMs) will be clariﬁed presently. A completely diﬀerent sequence representation technique was introduced in the area of protein fold recognition by Liao and Noble [9]. Liao and Noble represented protein sequences by their pairwise biological similarities, which were measured by ordinary sequence alignment algorithms. Subsequently, by considering these similarities as feature vectors, relatively simple classiﬁers were trained and successfully utilized for classifying and discriminating between diﬀerent protein folds. The primary intention in this study is to use a SVM-based classiﬁer in achieving the classiﬁcation and discrimination. However, rather than use distances, we shall advocate the use of a rigorous probabilistic model, namely one which has 1

The review and bibliography presented here is necessarily brief. A more detailed review is found in [1].

26

E. Ayg¨ un, B.J. Oommen, and Z. Cataltepe

been proven to be both optimal and to attain the information theoretic bound. Indeed, in this study, we combine the strategy of Liao and Noble (i.e., to use pairwise SVM classiﬁers) with a probabilistic similarity metric, and to successfully classify peptides. Observe that, instead of resorting to the alignment scores, we quantify the similarity by means of their Optimal and Information Theoretic (OIT) garbling probabilities as described by Oommen and Kashyap [11]. The latter OIT garbling probability is the probability of obtaining a sequence Y from a sequence U based on the OIT mutation model, whose properties will be clariﬁed later. One clear diﬀerence between the alignment scores and OIT garbling probabilities is that whereas an alignment score considers only the shortest path between two sequences, the OIT garbling probabilities covers all possible paths. Furthermore, since it assigns a probability mass to every possible path (i.e., possible garbling operations), it contains more information about the similarity between the two sequences. It is pertinent to mention that a similar transition probability measurement based on HMMs was earlier proposed by Bucher and Hofman [2]. Indeed, since then, HMM-based similarity metrics have been used in many biological applications. The diﬀerence between our work and the ones which use HMMs can be, in all brevity stated as follows: Unlike the latter, the OIT model permits non-Geometric-based distributions for the number of insertions occurring in any sequence of mutations [1,11]. Additionally, the superiority of OIT model, say Π ∗ , to “distance-based” approaches are (a) Π ∗ is Functionally Complete because it comprehensively considers all the ways by which U can be mutated into Y using the three elementary Substitutions, Insertions and Deletions (SID) operations, (b) The distributions and the parameters involved for the various garbling operations in Π ∗ can be completely arbitrary, (c) Π ∗ captures the scenarios in which the probability of a particular string U being transformed into another string Y , is arbitrarily small, (d) For a given U , the length of Y is a random variable whose distribution does not necessarily have to be a mixture of Geometric distributions, and (e) If the input U is itself an element of a dictionary, and the OIT channel is used to model the noisy channel, the technique for computing the probability Pr [Y |U ] can be utilized in a Bayesian way to compute the a posteriori probabilities, and thus yield an optimal, minimum probability of error pattern classiﬁcation rule. Most importantly, however, in both the Bayesian and non-Bayesian approaches, the OIT model actually attains the information theoretic bound for recognition accuracy when compared with all the other models which have the same underlying garbling philosophy These issues are also clariﬁed in greater detail in [1,11]. We have tested our solution, the OIT SVM, which involves the combination of the SVM-pairwise and the OIT model, on two peptide classiﬁcation problems, namely the HIV-1 Protease Cleavage site and the T-cell Epitope prediction problems. Both of these problems are closely related to pharmacological research work that has been the focus of a variety of computational approaches [3,8,13,14,15]. The results, which we present in a subsequent section, indicate that our solution paradigm leads to an extremely good classiﬁcation performance.

On Utilizing Optimal and Information Theoretic Syntactic Modeling

2

27

Modeling – The String Generation Process

We now describe the model by which a string Y is generated given an input string U ∈ A∗ , where A is the alphabet under consideration, and ξ and λ are the input and output null symbols, respectively. First of all, we assume that the model utilizes a probability distribution G over the set of positive integers. The random variable in this case is referred to as Z, and is the number of insertions that are performed in the mutating process. G is called the Quantified Insertion Distribution, and in the most general case, can be conditioned on the input string U . The quantity G (z|U ) is the probability that Z = z given that U is the input word. Thus, G has to satisfy the following constraint: G (z|U ) = 1. (1) z≥0

The second distribution that the model utilizes is the probability distribution Q over the alphabet under consideration. Q is called the Qualified Insertion Distribution. The quantity Q (a) is the probability that a ∈ A will be the inserted symbol conditioned on the fact that an insertion operation is to be performed. Note that Q has to satisfy the following constraint: Q (a) = 1. (2) a∈A

Apart from G and Q, another distribution that the model utilizes is a probability distribution S over A × (A ∪ {λ}), where λ is the output null symbol. S is called the Substitution and Deletion Distribution. The quantity S (b|a) is the conditional probability that the given symbol a ∈ A in the input string is mutated by a stochastic substitution or deletion –in which case it will be transformed into a symbol b ∈ (A ∪ {λ}). Hence, S (c|a) is the conditional probability of a ∈ A being substituted for by c ∈ A, and analogously, S (λ|a) is the conditional probability of a ∈ A being deleted. Observe that S has to satisfy the following constraint for all a ∈ A: S (b|a) = 1. (3) b∈(A∪{λ})

Using the above distributions we now informally describe the OIT model for the garbling mechanism (or equivalently, the noisy string generation process). Let |U | = N . Using the distribution G, the generatorﬁrst randomly determines the number of symbols to be inserted. Let Z be random variable denoting the number of insertions that are to be inserted in the mutation. Based on the output of the random number generator, let us assume that Z takes the value z. The algorithm then determines the position of the insertions among the individual symbols of ∗ U . This is done by randomly generating an input edit sequence U ∈ (A ∪ {ξ}) . We assume that all the possible strings are equally likely. Note that the positions of the symbol ξ in U represents the positions where symbols will be inserted into U . The non-ξ symbols in U are now substituted for

28

E. Ayg¨ un, B.J. Oommen, and Z. Cataltepe

or deleted using the distribution S. Finally, the occurrences of ξ are transformed independently into the individual symbols of the alphabet using the distribution Q. This deﬁnes the model completely. The process followed by the model, and its graphical display, are formally included in the unabridged version of this paper, and omitted here in the interest of brevity [1]. The theoretical properties of the OIT model can be found in [11].

3

Proposed Methodology

In this section, we provide the explicit details of the syntactic probabilities of the OIT model, and also explain the way by which we utilize it together with the SVM-pairwise scheme for peptide classiﬁcation. For a mutation consisting of random SID operations as per the OIT model, Oommen and Kashyap [11] have derived the syntactic probability of obtaining the sequence Y = y1 y2 . . . yM , from the sequence U = u1 u2 . . . uN as: P (Y | U ) =

M z=max{0,M−N }

N +z G (z) N ! z! p (yi | ui ) , (N + z)! i=1 U

Y

where G(z) is the probability of inserting z elements into U , and p (yi | ui ) is the probability of substituting the symbol element ui with the symbol element yi . Observe that in the above, ui = ξ ⇒ yi = λ, and yi = λ ⇒ ui = ξ. The sum over the strings U = u1 u2 . . . uN +z and Y = y1 y2 . . . yN +z (of the same length), represent the sum over all possible pairs of strings U and Y of equal length N + z, generated by inserting ξ’s into random positions in string U , and λ’s into random positions in strings Y respectively, and which are to represent the insertion and the deletion operations respectively. Although this requires a summation over a combinatorially large number of elements (represented by U and Y ), Oommen and Kashyap [11] have shown that this can be computed in an extremely eﬃcient manner in cubic time, i.e., with complexity O (M · N · min {M, N }). Based on the work of Oommen and Kashyap [11], we have programmed our own toolkit to eﬃciently compute the syntactic probabilities between two arbitrary sequences, and adapted it to this particular domain. Since the OIT model essentially requires three “parameters” namely, S for the Substitution/Deletion probabilities, Q, for the insertion distribution, and G, we list the issues crucial to our solution:

1. The input and output alphabets in our application domain consist of twenty amino acids and one gap element, which for the input strings is the null symbol, ξ, representing an inserted element, and for output strings is the null symbol, λ, representing a deleted element. 2. The substitution of an amino acid with another corresponds to a series of mutations in the biological context. Based on this premise, we have computed

On Utilizing Optimal and Information Theoretic Syntactic Modeling

29

our substitution probabilities on the mutation probability matrix referred to as PAM1 derived by Dayhoﬀ et al. [5]. PAM1 is a 20 × 20 matrix, M, where each cell mij corresponds to the probability of replacing amino acid i with amino acid j after 1% of the amino acids are replaced. It is possible to generate matrices for a series of longer mutations using successive multiplications of PAM1, and thus, for example, PAM250 is equal to PAM249×PAM1 [5]. 3. The ﬁrst major deviation from the traditional PAM matrices involves the operation of deletion. Observe that PAM matrices generally do not specify deletion probabilities for amino acids. As opposed to this, the OIT model of Oommen and Kashyap [11] suggests that an element can be deleted (substituted by λ) as well as substituted by another element. In this vein, we advocate that the matrix PAM1 be extended by appending another column for λ, where the value Δ is assigned to the deletion probabilities of amino acids, and where each row is normalized to satisfy the probability constraint: p (y | u) = 1, (4) y∈A∪{λ}

where A is the set of all amino acids, and u is the amino acid corresponding to the row. 4. There is no standard method of determining the deletion probabilities of amino acids. Comparing the widely-used gap penalties as per [12] to the log − odd PAM matrices, we opted to use Δ = 0.0001. The question of how to optimally determine Δ is open, and we are currently considering how it can be obtained from a training phase using known Input/Output patterns. 5. The second major deviation from utilizing the traditional PAM matrices involves the operation of insertion. As in the case of deletion, we propose to extend the new PAM matrix by appending a row for ξ and assigned to p (y | ξ) (i.e. the probability that a newly inserted amino acid is y) the relative frequency of observing y, f (y). In our experiments, the relative frequencies were computed in a maximum likelihood manner by evaluating the limit of the PAMn matrix as n goes to inﬁnity, i.e., as each row of the limiting matrix converges to f (y). Finally, the remaining cell of our extended PAM matrix, p (λ | ξ), is, by deﬁnition, equal to zero. The resulting matrix has been referred to as the OIT PAM matrix, and is a 21 × 21 matrix. Table 1 gives a typical OIT PAM matrix for the amino acid application domain. Observe that as in the case of the traditional PAM matrices, it is possible to derive higher order OIT PAM matrices for longer mutation sequences by multiplying OIT PAM1 by itself. In our work, we have experimented with OIT PAM matrices of diﬀerent orders to observe the eﬀect of diﬀerent assumptions that concern evolutionary distances. 6. The ﬁnal parameter of the OIT model involves the Quantiﬁed Insertion distribution, G (z), which speciﬁes the probability that the number of insertions during the mutation is z. In our experiments, we have assumed that the probability of inserting an amino acid during a single PAM mutation is equal to the deletion probability of an amino acid, Δ. This assumption leads

30

E. Ayg¨ un, B.J. Oommen, and Z. Cataltepe

to the conclusion that for longer mutation series, the insertion distribution converges to a Poisson distribution such that G (z) = Poisson (z; nΔ) =

z

(nΔ) e−Δn , z!

(5)

where n is the number of PAMs (i.e. the length of the mutation series). In other words, we have currently used Poisson (z; nΔ) as the insertion distribution whenever we use OIT PAMn as the substitution probability matrix. 7. Using the OIT model and the parameters assigned as described above, a classiﬁcation methodology based on the SVM-pairwise scheme proposed by Liao and Noble [9] was devised. This will be explained in the next subsection. Having explained how the OIT-based scheme works, we shall now also present the results obtained from our experiments.

4 4.1

Experimental Results and Discussions Experimental Setup

In our experiments, we used two peptide classiﬁcation data sets, which are accepted as benchmark sets. The ﬁrst one, referred to as HIV, was produced for the HIV-1 Protease Cleavage sites prediction problem by Kim et al. [8]. This set contains 754 8-residue peptides with 396 positives and 358 negatives. The second data set, referred to as TCL, was produced for the T-cell Epitope prediction problem by Zhao et al. [15], and it contains 203 10-residue peptides of which 36 were positives and 167 were negatives. As mentioned earlier, our classiﬁcation scheme was based on the SVM-pairwise scheme proposed by Liao and Noble [9] to detect remote evolutionary relationships between proteins. According to our scheme, m representative peptides were chosen a priori from the training set. Subsequently, for each instance, an m-dimensional vector of scores was computed by comparing the instance to the representatives. The classiﬁers were trained and tested with these feature vectors. As a computational convenience, we used the logarithm of the OIT probability as the measure of similarity because the logarithm is a monotonic function, and it turns out that this can be computed more eﬃciently than the original OIT probabilities. To compare the performance of the OIT SVM to the standard measures, we have also used the Needleman-Wunsch (NW) alignment score [10], which is a commonly used sequence comparison method in bioinformatics, to achieve an analogous classiﬁcation. Our representative peptides were chosen to be the positive training instances, and in each case, we used eight diﬀerent substitution matrices with mutation lengths 10, 50, 100, 200, 250, 300, 400 and 500. Each feature set was tested on a SVM classiﬁer with a linear kernel. A preliminary evaluation showed that the SVM with a linear kernel performs slightly better than the SVM with a radial -basis kernel on all the feature sets. Based

On Utilizing Optimal and Information Theoretic Syntactic Modeling

31

on this observation, we ﬁxed the classiﬁer prior to the experiments, and merely focused on the comparison of feature sets themselves. In the testing phase, we estimated the performance of diﬀerent methods by means of a cross-validation process. To do this, we divided the HIV data set into ten partitions and the TCL data set, which is rather small, into ﬁve partitions as was done in [8] and [15] respectively. In our case, we chose to not divide the TCL data set into more than ﬁve partitions because the number of positive examples was too low, and it consequently prevented us from providing the necessary variation across the partitions. This choice also rendered our results to be compatible with the results of [15]. Finally, we also ensured the preservation of the ratio of positive and negative instances across the partitions. All the classiﬁcation and performance estimations were performed on the Mathworks MATLAB [7] system with the help of the PRTools 4.1 pattern recognition toolbox [6] and the LIBSVM 2.88 support vector machine library [4]. 4.2

Experimental Results and Discussions

The performance of the OIT-based features were compared to the scores obtained by a Needleman-Wunsch (NW) alignment strategy. In each case, and for each of the experiments, we recorded the area under the ROC (AUC), the Accuracy (Acc), the Sensitivity (Sens) and the Positive Predictive Value (PPV). Tables 2 and 3 show the averaged values and average widths of the 95% conﬁdence intervals for the HIV and TCL data sets, respectively. It is worth mentioning that the OIT-based scheme is uniformly superior to the NW-based scheme, and in some cases the superiority is categorically marked –for example, whereas the best accuracy for the NW-based method is 85.7%, the corresponding best accuracy for the OIT-based scheme is 91.7%. Also note that the 95% conﬁdence intervals are generally wider for the TCL dataset than they are for the HIV dataset. This is because the cross validation was performed through a ﬁve-fold strategy on the former, and through a ten-fold strategy on the latter. For the HIV data set, [8] report accuracies for ten diﬀerent methods, and our OIT-based method outperforms nine of them, while the accuracy of the tenth is marginally better. With regard to the TCL data set, it should be mentioned that the OIT SVM leads to better results than those reported by [15] –when it concerns any performance criterion. The behaviors of the two methods for diﬀerent score matrices can be seen in Figures 1.These two ﬁgures display how the AUCs vary as the assumption of the mutation lengths increases from 10 PAMs to 500 PAMs. The reader will observe that for the HIV data set, both the OIT and the NW reach their highest performances between 100 and 300 PAMs. For the TCL data set, however, the NW prefers PAM400. When it concerns the means of the average AUCs, it should be mentioned that the OIT outperforms the NW even in its worst cases. Table 4 records the t-test results that validate this observation. Also, the average widths of the conﬁdence intervals point to the conclusion that the OIT leads to more robust classiﬁcations than the NW.

A R N D C Q E G H I L K M F P S T W Y V ξ

A -0.01 -8.52 -7.01 -6.91 -8.11 -7.13 -6.38 -6.17 -8.52 -7.42 -7.82 -8.52 -7.42 -8.52 -6.12 -5.66 -5.74 -36.04 -8.52 -6.32 -2.43

R -9.21 -0.01 -9.21 -36.04 -9.21 -6.91 -36.04 -36.04 -6.91 -8.11 -9.21 -6.27 -7.82 -9.21 -7.82 -7.42 -9.21 -7.13 -36.04 -9.21 -3.21

N -7.82 -9.21 -0.02 -5.63 -36.04 -7.82 -7.42 -7.42 -6.17 -8.11 -9.21 -6.65 -36.04 -9.21 -8.52 -6.21 -7.01 -9.21 -7.82 -9.21 -3.20

D -7.42 -36.04 -5.47 -0.01 -36.04 -7.42 -5.24 -7.42 -7.82 -9.21 -36.04 -8.11 -36.04 -36.04 -9.21 -7.60 -8.11 -36.04 -36.04 -9.21 -3.04

C -9.21 -9.21 -36.04 -36.04 0.00 -36.04 -36.04 -36.04 -9.21 -9.21 -36.04 -36.04 -36.04 -36.04 -9.21 -7.60 -9.21 -36.04 -8.11 -8.52 -3.43

Q -8.11 -7.01 -7.82 -7.60 -36.04 -0.01 -5.91 -9.21 -6.07 -9.21 -8.11 -7.42 -7.82 -36.04 -7.42 -8.52 -8.52 -36.04 -36.04 -9.21 -3.27

E -6.91 -36.04 -7.26 -5.19 -36.04 -5.66 -0.01 -7.82 -8.52 -8.11 -9.21 -7.82 -9.21 -36.04 -8.11 -7.82 -8.52 -36.04 -9.21 -8.52 -2.99

G -6.17 -9.21 -6.73 -6.81 -9.21 -8.11 -7.26 -0.01 -9.21 -36.04 -9.21 -8.52 -9.21 -9.21 -8.11 -6.17 -8.11 -36.04 -36.04 -7.60 -2.41

H -9.21 -7.13 -6.32 -8.11 -9.21 -6.21 -9.21 -36.04 -0.01 -36.04 -9.21 -9.21 -36.04 -8.52 -8.11 -9.21 -9.21 -9.21 -7.82 -9.21 -3.42

I -8.52 -8.52 -8.11 -9.21 -8.52 -9.21 -8.52 -36.04 -36.04 -0.01 -7.01 -8.52 -6.73 -7.26 -36.04 -9.21 -7.26 -36.04 -9.21 -5.71 -3.33

L -8.11 -9.21 -8.11 -36.04 -36.04 -7.42 -9.21 -9.21 -7.82 -6.12 -0.01 -8.52 -5.40 -6.65 -8.11 -9.21 -8.11 -7.82 -8.52 -6.50 -2.46

K -8.52 -5.60 -5.99 -7.42 -36.04 -6.73 -7.26 -8.52 -8.52 -7.82 -9.21 -0.01 -6.21 -36.04 -8.11 -7.13 -6.81 -36.04 -9.21 -9.21 -2.54

M -9.21 -9.21 -36.04 -36.04 -36.04 -8.52 -36.04 -36.04 -36.04 -7.60 -7.13 -7.82 -0.01 -9.21 -36.04 -9.21 -8.52 -36.04 -36.04 -7.82 -4.21

F -9.21 -9.21 -9.21 -36.04 -36.04 -36.04 -36.04 -9.21 -8.52 -7.13 -7.42 -36.04 -7.82 -0.01 -36.04 -8.52 -9.21 -8.11 -5.88 -36.04 -3.19

P -6.65 -7.60 -8.52 -9.21 -9.21 -7.13 -8.11 -8.52 -7.60 -9.21 -8.52 -8.52 -9.21 -9.21 -0.01 -6.73 -7.82 -36.04 -36.04 -8.52 -2.96

S -5.88 -6.81 -5.68 -7.26 -6.81 -7.82 -7.42 -6.44 -8.52 -8.52 -9.21 -7.26 -7.82 -8.11 -6.38 -0.02 -5.57 -7.60 -8.52 -8.52 -2.66

T -6.12 -8.52 -6.65 -7.82 -9.21 -8.11 -8.52 -8.52 -9.21 -6.81 -8.52 -7.13 -7.42 -9.21 -7.60 -5.74 -0.01 -36.04 -8.52 -7.01 -2.84

W -36.04 -8.52 -36.04 -36.04 -36.04 -36.04 -36.04 -36.04 -36.04 -36.04 -36.04 -36.04 -36.04 -9.21 -36.04 -9.21 -36.04 0.00 -9.21 -36.04 -4.68

Y -9.21 -36.04 -8.11 -36.04 -8.11 -36.04 -9.21 -36.04 -7.82 -9.21 -9.21 -36.04 -36.04 -6.17 -36.04 -9.21 -9.21 -8.52 -0.01 -9.21 -3.45

V -6.65 -8.52 -9.21 -9.21 -8.11 -8.52 -8.52 -8.11 -8.11 -5.17 -6.81 -9.21 -6.38 -9.21 -8.11 -8.52 -6.91 -36.04 -8.52 -0.01 -2.76

λ -9.21 -9.21 -9.21 -9.21 -9.21 -9.21 -9.21 -9.21 -9.21 -9.21 -9.21 -9.21 -9.21 -9.21 -9.21 -9.21 -9.21 -9.21 -9.21 -9.21 -∞

Table 1. The Log-OIT PAM1 matrix used for the OIT model. Each element Mi,j is equal to the logarithm of the probability associated with the event of replacing the ith element with j th element. The symbols ξ and λ represent the insertion and deletion of an element of the alphabet, respectively. Please see Section 3 for more details on the Log-OIT PAM1 matrix.

32 E. Ayg¨ un, B.J. Oommen, and Z. Cataltepe

On Utilizing Optimal and Information Theoretic Syntactic Modeling

33

Table 2. The performance measurements for the HIV data set using the OIT and NW metrics. The highest value over each column is shown in bold. The last row displays the average widths of the 95% conﬁdence intervals (Avg. w) for each measurement. (O)PAM 10 50 100 200 250 300 400 500 Avg. w

AUC 0.948 0.962 0.968 0.969 0.965 0.965 0.958 0.949 0.011

OIT Acc Sens 0.887 0.863 0.902 0.891 0.917 0.897 0.911 0.877 0.913 0.874 0.911 0.863 0.901 0.849 0.893 0.830 0.018 0.037

PPV 0.884 0.904 0.927 0.932 0.938 0.948 0.937 0.938 0.021

AUC 0.906 0.909 0.917 0.927 0.925 0.921 0.912 0.924 0.019

NW Acc Sens 0.839 0.821 0.849 0.841 0.846 0.846 0.857 0.833 0.853 0.830 0.849 0.829 0.849 0.838 0.846 0.813 0.025 0.040

PPV 0.837 0.843 0.833 0.862 0.857 0.852 0.848 0.859 0.029

Table 3. The performance measurements for the TCL data set using the OIT and NW metrics. The highest value over each column is shown in bold. The last row displays the average widths of the 95% conﬁdence intervals (Avg. w) for each measurement. (O)PAM 10 50 100 200 250 300 400 500 Avg. w

AUC 0.918 0.937 0.943 0.947 0.944 0.945 0.939 0.936 0.016

OIT Acc Sens 0.852 0.922 0.872 0.934 0.882 0.929 0.897 0.940 0.902 0.946 0.887 0.940 0.887 0.946 0.882 0.929 0.023 0.022

PPV 0.901 0.912 0.928 0.935 0.936 0.924 0.919 0.928 0.020

AUC 0.883 0.892 0.889 0.889 0.885 0.895 0.904 0.819 0.028

NW Acc Sens 0.837 0.928 0.842 0.922 0.847 0.922 0.853 0.905 0.853 0.893 0.852 0.916 0.867 0.911 0.793 0.881 0.030 0.041

PPV 0.882 0.891 0.895 0.917 0.927 0.905 0.928 0.871 0.021

Table 4. The t-test results for the 1% signiﬁcance level comparing the AUC values of the OIT and NW based schemes (O)PAM 10 50 100 200 250 300 400 500

HIV TCL OIT > NW p-value OIT > NW p-value no 0.013 no 0.018 yes 0.001 no 0.025 yes <0.001 no 0.047 yes <0.001 no 0.014 yes <0.001 yes <0.001 yes <0.001 no 0.015 no 0.012 yes 0.001 no 0.014 yes 0.001

34

E. Ayg¨ un, B.J. Oommen, and Z. Cataltepe 1

1 OIT NW

OIT NW

0.98 0.95

Area Under ROC

Area Under ROC

0.96

0.94

0.92

0.9

0.85

0.9 0.8

0.88 0.86 0

50

100

150

200

250 300 PAMs

350

400

450

500

0

50

100

150

200

250 300 PAMs

350

400

450

500

Fig. 1. The ﬁgure on the left displays the behavior of the OIT and NW similarity metrics on the HIV data set when the mutation length assumption changes between 10 PAMs and 500 PAMs. The ﬁgure on the right displays the corresponding behavior of the OIT and NW similarity metrics for the TCL data set. In each case, the error bars display the respective 95% conﬁdence intervals.

5

Conclusions

In this paper, we have considered the problem of classifying peptides using syntactic pattern recognition methodologies. Unlike the traditional distance-based or Markovian methods, we have considered how the pattern recognition can be achieved by using the Optimal and Information Theoretic (OIT) model of Oommen and Kashyap [11]. We have shown that one can model the diﬀerences between the compared strings as a mutation model consisting of random SID operations which obeys a OIT model. Consequently, by using the probability measure obtained from the OIT model as a pairwise similarity metric, we have devised a Support Vector Machine (SVM)-based peptide classiﬁer, referred to as OIT SVM. The classiﬁer has been tested for eight diﬀerent “substitution” matrices and for two diﬀerent data sets, namely, the HIV-1 Protease Cleavage sites and the T-cell Epitopes, and the results obtained categorically demonstrate that the OIT model performs signiﬁcantly better than the one which uses a Needleman-Wunsch sequence alignment score. Further, when combined with a SVM, it leads to, probably, the best peptide classiﬁcation method available. The main drawback of of the OIT method is its higher time complexity. Otherwise, the avenues for future work include the learning of the PAM matrices using maximum likelihood or Bayesian methods. The use of an OIT model for other bioinformatic pattern recognition problems remains open.

Acknowledgments The ﬁrst and third authors, Ayg¨ un and Cataltepe, were partially supported ¨ ITAK, ˙ by TUB The Scientiﬁc and Technological Research Council of Turkey. The second author, Oommen, was partially supported by NSERC, the Natural Sciences and Engineering Research Council of Canada.

On Utilizing Optimal and Information Theoretic Syntactic Modeling

35

References 1. Ayg¨ un, E., Oommen, B.J., Cataltepe, Z.: Peptide Classiﬁcation Using Optimal and Information Theoretic Syntactic Modeling (submitted for publication) 2. Bucher, P., Hofmann, K.: A sequence similarity search algorithm based on a probabilistic interpretation of an alignment scoring system. In: Proceedings of the Conference on Intelligent Systems for Molecular Biology, pp. 44–51 (1996) 3. Cai, Y.D., Chou, K.C.: Artiﬁcial neural network model for predicting HIV protease cleavage sites in protein. Advances in Engineering Software 29(2), 119–128 (1998) 4. Chang, C.C., Lin, C.J.: LIBSVM: a library for support vector machines (2001), http://www.csie.ntu.edu.tw/~ cjlin/libsvm 5. Dayhoﬀ, M., Schwartz, R., Orcutt, B.: A model of evolutionary change in proteins. Atlas of Protein Sequence and Structure 5(suppl. 3), 345–352 (1978) 6. Duin, R.P.W., Juszczak, P., Paclik, P., Pekalska, E., de Ridder, D., Tax, D.M.J.: PRTools, a Matlab Toolbox for Pattern Recognition. Delft University of Technology (2004) 7. Guide, M.R.: The MathWorks. Inc., Natick, MA (1998) 8. Kim, H., Zhang, Y., Heo, Y.S., Oh, H.B., Chen, S.S.: Speciﬁcity rule discovery in HIV-1 protease cleavage site analysis. Computational Biology and Chemistry 32(1), 71–78 (2008) 9. Liao, L., Noble, W.S.: Combining Pairwise Sequence Similarity and Support Vector Machines for Detecting Remote Protein Evolutionary and Structural Relationships. Journal of Computational Biology 10(6), 857–868 (2003) 10. Needleman, S.B., Wunsch, C.D.: A general method applicable to the search for similarities in the ammo acid sequence of two proteins. J. Mol. Biol. 48(3), 443– 453 (1970) 11. Oommen, B.J., Kashyap, R.L.: A formal theory for optimal and information theoretic syntactic pattern recognition. Pattern Recognition 31(8), 1159–1177 (1998) 12. Tatusova, T.A., Madden, T.L.: BLAST 2 Sequences, a new tool for comparing protein and nucleotide sequences. FEMS Microbiology Letters 174(2), 247–250 (1999) 13. Thomson, R., Hodgman, T.C., Yang, Z.R., Doyle, A.K.: Characterizing proteolytic cleavage site activity using bio-basis function neural networks. Bioinformatics 19(14), 1741–1747 (2003) 14. Trudgian, D.C., Yang, Z.R.: Substitution Matrix Optimisation for Peptide Classiﬁcation. In: Marchiori, E., Moore, J.H., Rajapakse, J.C. (eds.) EvoBIO 2007. LNCS, vol. 4447, pp. 291–300. Springer, Heidelberg (2007) 15. Zhao, Y., Pinilla, C., Valmori, D., Martin, R., Simon, R.: Application of support vector machines for T-cell epitopes prediction. Bioinformatics 19(15), 1978–1984 (2003)

Joint Tracking of Cell Morphology and Motion Jierong Cheng1,2, , Esther G.L. Koh3, , Sohail Ahmed3 , and Jagath C. Rajapakse1,2,4 1

BioInformatics Research Center and School of Computer Engineering, Nanyang Technological University, Singapore [email protected] 2 Singapore-MIT Alliance, Singapore 3 Institute of Medical Biology, Singapore 4 Department of Biological Engineering, Massachusetts Institute of Technology, USA

Abstract. A new method is proposed for joint tracking of cell morphology and motion from 3D temporal cellular images. We adopt the framework of region-based active contours for segmentation, which is able to cope with objects having blurred boundaries. Motion estimation is performed by optical ﬂow to increase the robustness and accuracy. Cell morphology and motion are modelled via a uniﬁed energy formulation and estimated iteratively searching for the minimum energy conﬁguration. Experiments are carried out on synthetic and real cellular images to demonstrate the performance of the method. Keywords: Cell segmentation, tracking, motion estimation, optical ﬂow, active contours, level sets.

1

Introduction

Three-dimensional (3D) molecular imaging techniques are now capable of generating large time-lapsed image datasets rapidly. Manual segmentation of cells in these images is almost impossible. Automatic segmentation and tracking methods are therefore necessary to enable quantitative understanding and analysis of cellular images. In recent years, many image analysis approaches have been adopted for cell segmentation from microscopic images. Parametric active contours, or snakes, are widely applied in boundary detection and motion tracking [1],[2]. Geometric active contours based on level sets are becoming increasingly popular because they are not constrained by the topology and thus are capable of detecting an arbitrary number of cells from an arbitrary initial front. In the model of active contours without edges [3], the image energy terms are computed using intensity variances inside and outside the contour. This region-based approach provides strong robustness to noise and allows segmentation of cells with blurred edges.

Corresponding author. Esther G.L. Koh is now with the Singapore Immunology Network, Singapore.

V. Kadirkamanathan et al. (Eds.): PRIB 2009, LNBI 5780, pp. 36–45, 2009. c Springer-Verlag Berlin Heidelberg 2009

Joint Tracking of Cell Morphology and Motion

37

The approach has been used for segmenting and tracking cells in 2D images [4], [5], [6] and 3D images [7]. Much work has been devoted to the data analysis on time-lapsed cellular images. Existing cell tracking methods can be roughly divided into frame-toframe linking and mathematical shape modelling methods. Promising methods for cell tracking include hidden Markov modelling [8], dynamic programming optimization [9], Kalman ﬁlter [10], and optical ﬂow [11]. In an overview of cell motility analysis and particle tracking methods [12], it is concluded that regional oriented models are far more robust than purely edge oriented models. Optical ﬂow is a convenient and useful representation of image motion. Discontinuities in the optical ﬂow can help in segmenting images into regions of diﬀerent motions. By direct coupling of a motion ﬁeld and a segmentation map, a dense optical ﬂow ﬁeld which is boundary-preserving can be computed, without relying on speciﬁc constraints of motion boundary. Joint motion estimation and segmentation by combining optical ﬂow with active contours has been applied on photographic images in [13], [14], [15], and [16]. The energy functional proposed by Feghali [14] embeds the constraints of both the motion estimation and segmentation. The spatial-temporal surface evolution is entirely driven by the magnitude of the motion ﬁeld. However, 3D temporal images are often of low resolution and the cell boundaries are blurred and noisy. The existing gradientbased methods have diﬃculty in dealing with these images. Based on the method proposed by Feghali, we introduce regional force in the curve evolution which proved eﬃcient in segmenting cells with blurred edges. This work is motivated by the study of neuronal cell growth. In order to obtain basic understanding of cell structure development, cell morphology, and motion need to be tracked accurately from 3D temporal cellular images. It is of great interest to know how neuronal stem cells grow into mature neurons. The quantitative measurements are important as cells are allowed to grow. We propose an automatic technique to quantify the growth of cell by simultaneously tracking cell morphology and motion. The proposed method is more robust and accurate for 3D cell segmentation than the existing methods, by combing optical ﬂow and region-based active contours. The paper is organized as follows. In Section 2, a short overview of active contours and optical ﬂow is introduced and our combining model is described. Experimental results are shown in Section 3 and the conclusion is given in Section 4.

2 2.1

Method Region-Based Active Contours

Active contours compute segmentations of a given image by evolving contours in the direction of the negative gradient of image energy. Traditional active contours make use of image gradient to localize object boundaries. For cell segmentation from 3D microscopic images, we choose to adopt the model of active contours without edges [3]. In this region-based model, image energy is computed from

38

J. Cheng et al.

surface integrals over the entire image. Therefore, region-based models are robust to noise and allow segmentation of objects with blurred edges. The active contours are implicitly represented by a single level set function and the topology of objects changes automatically as the level set function evolves. This enables automatic detection of an arbitrary number of cells from an arbitrary initial front. Let I(ω, t) be the image intensity at location ω = (x, y, z) ∈ Ω at time t, Ω ⊂ R3 being the 3D image domain. φ(ω, t) is a level set function deﬁned on Ω, whose zero-level set {ω ∈ Ω|φ(ω, t) = 0, ∀ t} deﬁnes the segmentation such that the value of φ > 0 inside the segmented objects and φ < 0 outside. The energy function of the active contour is based on a reduced form of the Mumford-Shah function for image segmentation (we drop ω for notational simplicity): E(φ, I) = α δ(φ)|∇φ|dω + λI H(φ)(I − μI )2 dω Ω Ω +λO (1 − H(φ))(I − μO )2 dω (1) Ω

where dω is the elementary volume; μI and μO are respectively the mean intensity of pixels inside and outside the zero level-set; H and δ are the Heaviside and Dirac functions; λI , λO , and α are positive constants. The minimization of image energy is accomplished by letting the level set function evolve as a function of an abstract time t, starting from an initialization φ(ω, t = 0) according to: ∂φ ∇φ = α∇ · − λI (I − μI )2 + λO (I − μO )2 δ(φ). ∂t |∇φ|

(2)

∇φ Here ∇ · |∇φ| is the (mean) curvature of the level set, generating a regulating force which smoothes the contours. The other two forces on the righthand side move the contour toward the actual boundary of objects.

2.2

Optical Flow Field

Let u(ω, t) and v(ω, t) be the x and y velocity component of the apparent motion of brightness patterns of the image, respectively. The classical regularization of the optical ﬂow ﬁeld used by Horn and Schunk [17] has a quadratic functional: min (u2x + u2y + vx2 + vy2 )dω (3) (u,v)

Ω

subject to the optical ﬂow constraint Ix u + Iy v + It = 0.

(4)

∂u ∂v ∂v where ux = ∂u ∂x , uy = ∂y , vx = ∂x , and vy = ∂y are optical ﬂow gradients; Ix , Iy , and It are the intensity gradients in the x, y, and the temporal directions,

Joint Tracking of Cell Morphology and Motion

39

respectively. Horn and Schunk’s smoothness constraint seeks to improve some measure of consistency on the ﬂow vectors that are close to each other. The energy function combining the smoothing error and image motion error is E(I, u, v) = β(u2x + u2y + vx2 + vy2 ) + (Ix u + Iy v + It )2 dω (5) Ω

2.3

Joint Tracking of Cell Morphology and Motion

We propose the energy formulation for joint tracking of cell morphology and motion. E(φ, I, u, v) = α δ(φ)|∇φ|dω + λI H(φ)(I − μI )2 dω Ω Ω +λO (1 − H(φ))(I − μO )2 dω Ω 2 2 2 2 + β(ux + uy + vx + vy ) + (Ix u + Iy v + It )2 dω Ω 1 +γI H(φ) dω + γ (1 − H(φ))(u2 + v 2 )dω (6) O 1 + u2 + v 2 Ω Ω where γI and γO are positive constants. The last two terms of the right hand side of (6) correspond to motion segmentation constraints: large motion ﬁeld is expected in the spatiotemporal region that encloses moving objects; small motion ﬁeld is expected in the spatiotemporal region that encloses background. The evolution equations for (φ, u, v) is derived by applying Euler-Lagrange equations on (6). The level set evolution is given by the following descent equation ∂φ ∇φ = α∇ · − λI (I − μI )2 + λO (I − μO )2 ∂t |∇φ| 1 2 2 −γI + γ (u + v ) δ(φ). (7) O 1 + u2 + v 2 The minimization of the energy function with respect to the motion ﬁeld (u, v) yields a set of two descent equations that can be solved iteratively by

where

u=

β(Ix2 + β + ε)¯ u − βIx Iy v¯ − (β + ε)Ix It 2 (β + ε) + (β + ε)(Ix2 + Iy2 )

v=

β(Ix2 + β + ε)¯ v − βIx Iy u ¯ − (β + ε)Iy It (β + ε)2 + (β + ε)(Ix2 + Iy2 ) 2γI H(φ) + 2γO (1 − H(φ)) 1+u ¯2 + v¯2 (ωi ,t)∈Nφ (ω,t) u(ωi , t) u ¯(ω, t) = |Nφ (ω, t)|

ε=− and

(8)

(9)

(10)

40

J. Cheng et al.

with Nφ (ω, t) = {w ∈ N (ω, t)|sign[φ(w, t)] = sign[φ(w , t)]}, N (ω, t) being the neighborhood of (ω, t). In other words, u ¯(ω, t) is the average take only in the region of the moving objects or that of the static background. Similarly, the expression for v¯ is deﬁned. The proposed algorithm is summarized in Algorithm 1. Algorithm 1. Joint tracking of cell morphology and motion begin t=0 φ(ω, t) as a small circle at the center of the image u(ω, t) = v(ω, t) = 0, ∀ω ∈ Ω repeat for all ω ∈ Ω do Update u(ω, t) and v(ω, t) according to (8) Update φ(ω, t) according to (7) end for t = t+1 until convergence end

3

Experimental Results

Experiment is performed on synthetic images as well as on real 3D temporal cellular images. Our results are compared with the method proposed by Feghali [14]. 3.1

Synthetic Data

In the original synthetic image shown in Fig. 1(a), the intensity on the squareshaped object is set to be the Euclidean distance to the object boundary. This is aimed to simulate real microscopic images where the intensity contrast at the cell boundary is relatively low. The image at the next time point is generated by moving the same object one pixel to the bottom-right. An optical ﬂow ﬁeld (Fig. 1(d)) is estimated from two noise-free images. The noise-free image and the corresponding optical ﬂow ﬁeld are regarded as ground-truth for object segmentation and motion estimation. The images are added with Gaussian noise with mean of 0 and standard deviation of 0.02. The parameter values were taken heuristically for our experiments on synthetic image are: α = 0, λI = 1, λO = 1, β = 0.01, γI = 0.05, and γO = 0.27. From Fig. 1 we found that the segmentation by the proposed method is smoother and closer to the ground-truth than by the method by Feghali, so as the estimated optical ﬂow ﬁeld. Two error metrics are adopted to compare the segmentation results to the ground-truth: the mean absolute diﬀerence estimates the averaged disagreement between the object boundaries. Another measure is the Jaccard similarity coeﬃcient (Intersection/Union), which compares the areas of the segmented objects.

Joint Tracking of Cell Morphology and Motion

(a)

(b)

(c)

(d)

(e)

(f)

41

Fig. 1. Comparison results on synthetic image. (a) Original intensity image, (b) segmentation on noisy image by the method by Feghali, (c) segmentation on noisy image by the proposed method, (d) magnitude of motion ﬁeld estimated from original intensity image, (e) magnitude of motion ﬁeld estimated from noisy image by the method by Feghali, (f) magnitude of motion ﬁeld estimated from noisy image by the proposed method.

The value ranges from 0 (no overlap) to 1 (total overlap), thus the higher value the better correlation. The measurement results against diﬀerent level of noise are plotted in Fig. 2 and 3. 3.2

Neuronal Data

The cells in our neuronal images are ND7 cells, hybrid of mouse neuroblastoma and rat dorsal root ganglia. The purpose of the experiment is to characterize the dynamic properties of the ﬁlopodia. These highly mobile projections are continuously extended and retracted from the cell body. Cells were seeded on plastic petridishes and transfected with an expression plasmid for cherry-actin. The next day, cells were harvested and embedded in a collagen gel matrix (0.4mg/mL collagen) in a glass-bottomed petridish of 35mm diameter. One mL of DMEM (without phenol red) supplemented with 5% fetal calf serum was added to the dish after the gel has solidiﬁed. The petridish of cells was incubated on the

42

J. Cheng et al. 4.5

Mean absolute difference (pixels)

4

The method by Feghali The proposed method

3.5 3 2.5 2 1.5 1 0.5 0 20

22

24

26

28

30

32

34

36

38

40

Signal−to−noise ratio (dB)

Fig. 2. Mean absolute diﬀerences at diﬀerent noise levels on synthetic images

Jaccard similarity coefficients (%)

100 The method by Feghali The proposed method

99

98

97

96

95

94 20

22

24

26

28

30

32

34

36

38

40

Signal−to−noise ratio (dB)

Fig. 3. Jaccard similarity coeﬃcients at diﬀerent noise levels on synthetic images

microscope for 30 minutes to stabilize the temperature and CO2 before images were captured. The imaging was performed on an Olympus IX81 equipped with a 60× 1.20 UPLANSAPO objective. The images were acquired by a Photometrics QuantEM:512SC and the MetaMorph Imaging Software from Molecular Devices. The stack size of our 3D neuronal images is 281 × 233 × 10 voxels and the spatial

Joint Tracking of Cell Morphology and Motion

(a)

(b)

(c)

(d)

(e)

(f)

43

Fig. 4. Comparison results on cellular image. (a) First image of the temporal sequence, (b) segmentation by the method by Feghali, (c) segmentation by the proposed method, (d) second image of the temporal sequence, (e) magnitude of motion ﬁeld estimated by the method by Feghali, (f) magnitude of motion ﬁeld estimated by the proposed method.

resolution is 0.275 × 0.275 × 0.5 μm/pixel. The temporal sequence contains 31 stacks, taken at an interval of 10 seconds. The position of the cell is ﬁxed but the cell morphology changes due to the dendrite growth and movement. In this case, the estimated motion ﬁeld indicates the movement of each voxel within the cell. The parameter values for experiments on neuronal image are: α = 0, λI = 1, λO = 1, β = 0.01, γI = 0.02 and γO = 0.025. Fig. 4(a) and Fig. 4(d) show the middle sections from two successive stacks. The segmentation by the method by Feghali misses the cell boundary sometimes and contains singularities inside the cell. In contrast, the proposed method is more robust to the noise and ﬁnds the cell boundary accurately in spite of the low contrast; meanwhile, an accurate motion ﬁeld is obtained.

44

4

J. Cheng et al.

Conclusion

We proposed a new method of joint tracking of cell morphology and motion from 3D temporal cellular images. Optical ﬂow and region-based active contours methods are combined via a united energy formulation and solved iteratively. Motion estimation and segmentation are coupled directly so that the temporal dimension is taken into account. In addition, we make use of regional information in additional energy terms. The advantage of such method is that the segmentation relies on the average intensity of diﬀerent regions rather than the presence of strong edge around object boundary. This increases the robustness and accuracy of our method. Experimental results on synthetic and real images demonstrate that the proposed method is able to give smooth and precise segmentation on noisy images with blurred boundary. Another advantage of our method is that it is highly automatic. No human interference is required during the process, except that some parameter adjustments may be needed for the processing of a new dataset. Our method enables the quantitative analysis of cell morphology and motion from a large amount of 3D temporal cellular images. The method can help the automatic detection and separation of dendrites from cell body. The lengths of dendrites, the number of branches, and other quantitative features can be measured further from the segmentation results. The robustness and the accuracy of the method are vital for biological studies that make inferences on cell morphology and growth. In our method, we assume that the motion of voxels within the cell is always larger than the motion of background. This assumption is roughly true since the motion is large near the cell boundary where growth is likely to occur. However, there could be near zero-motion regions inside the cell and this will cause a hole in the segmentation. In future studies, we will incorporate adaptive hole-ﬁlling algorithm in our method.

References 1. Kass, M., Witkin, A., Terzopoulos, D.: Snakes: active contour models. International Journal of Computer Vision 1(4), 321–331 (1987) 2. Meegama, R.G.N., Rajapakse, J.C.: Nurbs snakes. Image and Vision Computing 21, 551–562 (2003) 3. Chan, T.F., Vese, L.A.: Active contours without edges. IEEE Trans. Image Processing 10(2), 266–277 (2001) 4. Zhang, B., Zimmer, C., Olivo-Marin, J.-C.: Tracking ﬂuorescent cells with coupled geometric active contours. In: Proc. IEEE Int’l Symp. Biomedical Imaging (ISBI), pp. 476–479 (2004) 5. Cheng, J., Rajapakse, J.C.: Segmentation of clustered nuclei with shape markers and marking function. IEEE Trans. Biomedical Engineering 53(3) (2009) 6. Yu, W., Lee, H.K., Hariharan, S., Bu, W., Ahmed, S.: Quantitative neurite outgrowth measurement based on image segmentation with topological dependence. Cytometry Part A 75A(4), 289–297 (2008) 7. Dufour, A., Shinin, V., Tajbakhsh, S., Guillen, N., Olivo-Marin, J.-C., Zimmer, C.: Segmenting and tracking ﬂuorescent cells in dynamic 3-D microscopy with coupled active surfaces. IEEE Trans. Image Processing 14(9), 1396–1410 (2005)

Joint Tracking of Cell Morphology and Motion

45

8. Althoﬀ, K., Degerman, J., W¨ ahlby, C., Thorlin, T., Faijerson, J., Eriksson, P.S., Gustavsson, T.: Time-lapse microscopy and classiﬁcation of in vitro cell migration using hidden markov modeling. In: Proc. IEEE Int’l Conference on Acoustics, Speech and Signal Processing (ICASSP), vol. 5, pp. 1165–1168 (2006) 9. Sage, D., Neumann, F.R., Hediger, F., Gasser, S.M., Unser, M.: Automatic tracking of individual ﬂuorescence particles: Application to the study of chromosome dynamics. IEEE Trans. Image Processing 14(9), 1372–1383 (2005) 10. Yang, X., Li, H., Zhou, X.: Nuclei segmentation using marker-controlled watershed, tracking using mean-shift, and kalman ﬁlter in time-lapse microscopy. IEEE Trans. Circ. Sys. -I 53(11), 2405–2414 (2006) 11. Melani, C., Campana, M., Lombardot, B., Rizzi, B., Veronesi, F., Zanella, C., Bourgine, P., Mikula, K., Peyrieras, N., Sarti, A.: Cells tracking in a live zebraﬁsh embryo. In: Proc. 29th Annual International Conference of IEEE Engineering in Medicine and Biology Society, pp. 1631–1634 (2007) 12. Olivo-Marin, J.-C.: An overview of image analysis in multidimensional biological microscopy. In: Proc. IEEE Int’l Conference on Acoustics, Speech and Signal Processing (ICASSP), vol. 5, pp. 1173–1176 (2006) 13. Mitiche, A., Feghali, R., Mansouri, A.: Motion tracking as spatio-temporal motion boundary detection. Robotics and Autonomous Systems 43, 39–50 (2003) 14. Feghali, R.: Multi-frame simultaneous motion estmation and segmentation. IEEE Trans. Consumer Electronics 51(1), 245–248 (2005) 15. Mitiche, A., Sekkati, H.: Optical ﬂow 3d segmentation and interpretation: A variational method with active curve evolution and level sets. IEEE Trans. Pattern Anal. Machine Intell. 28(11), 1818–1829 (2006) 16. Sekkati, H., Mitiche, A.: Joint optical ﬂow estimation, segmentation, and 3d interpretation with level sets. Computer Vision and Image Understanding 103(2), 89–154 (2006) 17. Horn, B.K.P., Schunck, B.G.: Determining optical ﬂow. Artif. Intell. 23, 185–203 (1981)

Multiclass Microarray Gene Expression Analysis Based on Mutual Dependency Models Girija Chetty1 and Madhu Chetty2 1

Faculty of Information Sciences and Engineering, University of Canberra, ACT, Australia 2 Faculty of Information Technology, Monash University, Victoria, Australia [email protected], [email protected]

Abstract. In this paper a novel feature selection technique based on mutual dependency modelling between genes is proposed for multiclass microarray gene expression classification. Several studies on analysis of gene expression data has shown that the genes (whether or not they belong to the same gene group) get co-expressed via a variety of pathways. Further, a gene may participate in multiple pathways that may or may not be co-active for all samples. It is therefore biologically meaningful to simultaneously divide genes into functional groups and samples into co-active categories. This can be done by modeling gene profiles for multiclass microarray gene data sets based on mutual dependency models, which model complex gene interactions. Most of the current works in multiclass microarray gene expression studies are based on statistical models with little or no consideration of gene interactions. This has led to lack of robustness and overly optimistic estimates of accuracy and noise reduction. In this paper, we propose multivariate analysis techniques which model the mutual dependency between the features and take into account complex interactions for extracting a subset of genes. The two techniques, the cross modal factor analysis (CFA) and canonical correlation analysis(CCA) show a significant reduction in dimensionality and class-prediction error, and improvement in classification accuracy for multiclass microarray gene expression datasets.

1 Introduction Molecular classification involves the classification of tumour samples into groups of biological phenotypes [1, 2, 3, 4]. Studies on molecular classification have great significance for cancer diagnosis. Molecular classification of tumour samples from patients into different molecular types or subtypes is vital for diagnosis, prognosis, and effective treatment of cancer [3, 4, 5]. Traditionally, such classification relies on observations regarding the location and microscopic appearance of the cancerous cells. These methods have proven to be slow and ineffective; and there is no way of predicting with reliable accuracy the progress of the disease, since tumours of similar appearance have been known to take different paths in the course of time. Some tumours may grow aggressively after the point of the previous observations, and hence require equally aggressive treatment regimes. Other tumours may stay inactive and thus require no treatment at all [2,3,5]. Since cancer treatment often V. Kadirkamanathan et al. (Eds.): PRIB 2009, LNBI 5780, pp. 46–55, 2009. © Springer-Verlag Berlin Heidelberg 2009

Multiclass Microarray Gene Expression Analysis Based on Mutual Dependency Models

47

produces adverse side effects on patients, patients whose tumours are predicted to stay inactive should be spared the unnecessary treatment. The problem is the risks involved in withholding treatment when the classification method (used to predict the aggressiveness of the tumour) is not reliable. Some tumours are particularly resistant to the more commonly prescribed anticancer drugs, while others are not. Predicting resistance to anticancer drugs will also ensure the optimal treatment regime for each patient. A patient predicted to be resistant to the more commonly prescribed anticancer drugs can then be prescribed alternative anticancer drugs, or can be recommended as a potential candidate for clinical trials of new anticancer drugs. As a genome is not just a collection of genes working in isolation, but rather it encompasses the global and highly coordinated control of information to carry out a range of cellular functions [1], any cellular activity requires elaborate patterns of gene interaction to marshal appropriate processes. In addition, the genome also incorporates information that controls when and where the parts of living organisms should be made. Therefore, it is imperative to conduct proper genome-wide studies so as to facilitate: 1. An effective identification of correlated genes and 2. A better understanding of the mechanisms underlying gene transcription and regulation. Expression of several thousands of genes can be measured simultaneously by DNA microarrays. With the advent of the microarray technology, data regarding the gene expression levels in each tumour sample proved to be a useful tool in molecular classification [2,3,4]. Microarrays have been effectively used to classify clinical samples, to investigate the mechanism of drug action and to examine the effects of drugs on gene expression in various organisms. The advantage of microarrays is that gene expression analysis is computationally less demanding than sequencing. Furthermore, recent advances in machine learning and statistical analysis tools for expression profiling have become more mature and cost effective. However, microarrays also have their own limitations. In particular, when the data is very noisy and contain artefacts, .gene prediction can be very difficult. Moreover, the feature dimensions of the genes are usually too large (causing large search space) while the dimensions of samples are too small (causing statistical errors). The problem of high dimensionality of features and small sample size dimensions has been addressed by several feature selection techniques in literature [1,2]. Due to the large number of genes for a typical microarray data, feature selection plays an important role in reducing noise and computational cost in gene expression based tissue classification while improving accuracy at the same time. However, the current feature selection techniques have not quite resulted in an appreciable noise reduction or accuracy improvement, particularly for multiclass microarray data sets. This could be due to the reason that many current feature selection techniques applied on microarray datasets do not take into account accurate feature dependencies or complex interactions between the genes, resulting in incorrect and overly optimistic estimates of accuracy. Only a relatively small number of genes (out of the thousands) monitored in microarray experiments actually influence the biological state of interest (such as tumour type or subtype, or resistance to anticancer drugs). Since majority of genes are not relevant (i.e., they do not supply useful information in distinguishing among

48

G. Chetty and M. Chetty

samples of different classes [2, 3, 4, 5]), adding them to the reduced feature set ot predictor set will not increase the multiclass classification accuracy. In fact, doing so will increase classifier complexity, and will also increase the noise in the classifier (and therefore decrease accuracy).

2 Role of Optimal Feature Selection Techniques In general, the objectives of feature selection techniques are to find from an overall set of N features, an optimal subset of features, S, that gives the best classification accuracy. This reduced feature subset is also known as the predictor set, |S| and is generally << N. Some of the important reasons for using good feature selection approaches are [2, 3, 10, 11]: a) b)

To gain a better understanding of the data and an insight into the way the selected features or genes affect the phenotypes of the samples. To reduce noise, over fitting and classifier complexity.

Identifying the members of the predictor set can indicate the genes involved in biological pathways which are responsible for the observed biological state of the sample (i.e., the class membership of the sample). This information is important to the field of pharmacological gene therapy, where drugs are designed to target specific genes in order to achieve the desired biological state (e.g., from highly aggressive tumour to less aggressive tumour). Only a relatively small number of genes out of the thousands that are monitored in microarray experiments actually influence the biological state of interest (such as tumour type or subtype, or resistance to anticancer drugs). One of the significant work in extracting a predictor set using different feature selection techniques for multiclass microarray gene expression problem is reported in [2, 12, 13, 14]. Here the authors proposed the simple-correlation based criteria such as relevance and redundancy in the formation of the predictor set. For addressing multiclass scenarios, a third criterion called differential prioritization criterion was used which assigned higher priority to maximizing relevance as compared to the priority of minimizing redundancy. The degree of differential prioritization (DDP) measure in this criteria ascertained that the optimal balance between relevance and redundancy is achieved in the multiclass micoarray gene expression classification problem [2, 12, 14]. The DDP measure also allowed increasing the importance of minimizing redundancy as the number of classes increase. For instance, in order to achieve the best accuracy, minimizing redundancy in a 14-class problem can be considered more important than minimizing redundancy in a two-class problem. Further, this technique was extended by developing another measure called antiredundancy, which was used in conjunction with DDP measure. The degree of differential prioritization (DDP) criteria along with antiredundancy measure, resulted in an unique ability to differentially prioritize the optimization of relevance against redundancy (and vice versa), ensuing optimal accuracy for multiclass microarray data analysis problem. However, the authors in this work [2] did not consider feature dependencies or complex gene interactions in extracting the predictor set. Hence, the evaluation results reported for the joint DDP-antiredundancy technique in [2] seemed

Multiclass Microarray Gene Expression Analysis Based on Mutual Dependency Models

49

to be overly optimistic, though it provided a good insight into the multiclass microarray problem. In this paper, we propose new feature selection techniques for extracting the predictor set based on mutual dependency models, which model the feature dependencies and complex gene interactions using multivariate analysis techniques. The proposed technique is expected to enhance the performance of multiclass microarray gene expression classification. An evaluation of the proposed mutual dependency modeling techniques for several multiclass Microarray gene expression datasets showed a significant improvement in dimensionality reduction, deviation error and classification accuracy. The rest of the paper is organized as follows. Next section describes the approach used for feature dependency modeling and the proposed multivariate analysis techniques. The details of the experiments and performance evaluation for different multiclass microarray datasets is discussed in Section 4, and the paper concludes with some conclusions and plan for further work in Section 5.

3 Mutual Dependency Models We examine two different cross modal analysis (CMA) techniques based on multivariate statistical analysis for modelling the feature dependencies: Cross modal Factor Analysis (CFA) and Canonical Correlation Analysis(CFA). Our contribution in this paper is to point out that CMA (CFA/CCA) can be used to extract the optimal predictor set that take into consideration gene dependencies and complex interactions. Commonalities in data sources or genes is exploited by these methods that search for statistical dependencies between them. Methods that model mutual dependencies tend to find the optimal transformations that can best represent or identify the coupled patterns between the features of the two different subsets. For CFA technique, following optimization criterion can be used to obtain the optimal transformations: Given two mean-centred matrices X and Y, which consist of row-by-row coupled samples from two subsets of features, we want orthogonal transformation matrices A and B that can minimise the expression:

XA − YB where

M

F

2 F

A A = I and B B = I . T

T

denotes the Frobenius norm of the matrix M and can be expressed as:

M

F

⎛ = ⎜⎜ ∑∑ mij ⎝ i j

2

⎞ ⎟ ⎟ ⎠

1/ 2

The earliest method was classical linear Canonical Correlation Analysis (CCA) [15], which has later been extended to nonlinear variants and more general methods that maximize mutual information instead of correlation. In other words, A and B define two orthogonal transformation spaces where coupled data in X and Y can be projected as close to each other as possible.

50

G. Chetty and M. Chetty

Since we have:

XA − YB

2 F

(

= trace ( XA − XB ) . (YA − YB) T

( = trace(( XX

)

= trace XAAT X T + YBBT Y T − XAB T Y T − YBAT X T T

) + trace(YY T ) − 2 ⋅ trace( XAB T Y T ) )

)

where the trace of a matrix is defined to be the sum of the diagonal elements. We can easily see from above that matrices A and B which maximise trace (XABTYT)) will minimise the equation above. It can be shown [284] that such matrices are given by:

⎧ A = S xy ⎨ ⎩ B = Dxy

where

X T Y = S xy ⋅ V xy ⋅ D xy

With the optimal transformation matrices A and B, we can calculate the transformed version of X and Y as follows

~ ⎧X = X ⋅ A ⎨~ ⎩Y = Y ⋅B ~ ~ Corresponding vectors in X and Y are thus optimised to represent the coupled relationships between the two feature subsets without being affected by distribution patterns within each subset. Traditional Pearson correlation or mutual information calculation [15] can then be performed on the first and most important k correspond-

~

~

ing vectors in X and Y , which preserve the principal coupled patterns in much lower dimensions. In addition to feature dimension reduction, feature selection capability is another advantage of CFA. The weights in A and B automatically reflect the significance of individual features. Following the development of the CFA technique, we can adopt a different optimization criterion for Canonical Correlation Analysis (CCA) method: Instead of minimizing the projected distance, we attempt to find transformation matrices A and B that maximise the correlation between XA and YB. This can be described more specifically using the following mathematical formulations: Given two mean centred matrices X and Y as defined in the previous section, we seek matrices A and B such that

~ ~ correlation( XA, XB) = correlation( X , Y ) = diag (λ1 ,L λi ,L, λl ) ~ Where X = Y ⋅ B, and 1 ≥ λ1 ≥, L , λi , L , ≥ λl ≥ 0 , λi represents the largest ~ ~ possible correlation between the ith translated features in X and Y . Note that A and B are only determined up to a constant factor (shown in equation above). A statistical method called canonical correlation analysis [15] can solve the above problem with additional norm and orthogonal constraints on translated features:

Multiclass Microarray Gene Expression Analysis Based on Mutual Dependency Models

{

}

{

51

}

~ ~ ~ ~ E X T ⋅ X = I and E Y ⋅ Y = I In CCA, A and B are calculated as follows:

B = ∑ yy . DK −1 / 2

A = ∑xx . S K −1 / 2

and

where

∑

xx

= E {X T X },

∑

yy

= E {Y T Y },

∑

xy

= E {X T Y }

and

L = ∑ xx

−1 / 2

∑ ∑ xy

−1 / 2

yy

= S K .VK . DKT

The calculation of the inverse matrix requires that no linear dependency exists between any two vectors within X or Y. The major differences between CCA and CFA include: •

The transformations provided by CFA are orthogonal, while this is not necessary true for CCA. A and B given by CFA satisfy ATA=I and BTB=I, where I is the identity matrix. CCA, however, does not provide such orthogonal transformations in most cases. CFA is in favour of coupled patterns with high variations (i.e. large amplitude changes), while CCA is more sensitive to highly coupled, but low variation patterns. This is mainly due to the whitening of X and Y in CCA by calculating

•

∑

−1 / 2

xx

and

∑

−1 / 2 yy

.

The optimization criteria used for all both CFA and CCA exhibit a high degree of noise tolerance. Hence the correlation features extracted perform better as compared to normal Pearson correlation analysis against noisy environmental conditions.

The CCA or CFA transformed features form the predictor set or the reduced feature set for training and testing the multiclass microarray datasets. The dimensionality of predictors set should be high enough to preserve most of the shared variation and yet low enough to avoid over fitting. Ideally an optimal dimensionality should be sought. Though a sophisticated optimization criteria could be used for finding the optimal dimensionality of the predictor set, we found that an approach based on empirical and experimental observations was quite satisfactory. This is because, the first few CCA or CFA components normally contain most of the reliable shared variation among the data sets, while the last components may actually represent just noise, and thus dropping some of the dimensions makes the method more robust. The detail of extracting the predictor set is described in the next section.

52

G. Chetty and M. Chetty

4 Experiments and Results The performance of mutual dependency models based on cross modal analysis techniques (CFA/CCA) was carried out on five different multiclass microarray datasets. Further as a baseline comparison, all the experiments were carried out with predictor set obtained by standard principal component analysis technique which is one of the most popular multivariate analysis technique for dimensionality reduction [2, 15]. The five multiclass microarray datasets used as bench- mark datasets were: • • • • •

The PDL dataset [10], which consists of 6 classes, each class representing a diagnostic group of childhood leukemia. The SRBC dataset [11] consisting 4 subtypes of small, round, blue cell tumors (SRBCTs). The Lung dataset [12], which is a 5-class dataset, with 4 classes as subtypes of lung cancer; and the fifth class consisting of normal samples. The MLL dataset [13], which contains 3 subtypes of leukemia: ALL, MLL and AML. The AML/ALL dataset [14] , which also contains 3 subtypes of leukemia: AML, B-cell and T-cell ALL.

The dimensionality of the predictor sets ranging from size P = 2 to P = Pmax was obtained by an empirical experimental technique. In this technique, we increase the dimensionality one at a time, testing with a randomization test that the new dimension captures shared variation. To protect from over fitting, all estimates of captured variation was computed using a validation set, i.e., for data that has not been used when computing the components (dimensions). The randomization test essentially compares the shared variance along the new dimension to the shared variance we would get under the null-hypothesis of mutual independency. When the shared variance does not differ significantly from the null-hypothesis, the final dimensionality has been reached. It was observed that this technique works quite well and infact the dimensionality of the predictor set is significantly better as compared to some of the previous studies, [2]. This could be due to inherently superior modelling and dimensionality reduction capability of mutual dependency models (CCA/CFA method). As can be seen in Table 1, as the number of classes in the datasets increases (from K = 3 to K = 5), it was possible to get better estimates of accuracy with a maximum predictor set dimension of Pmax = 10 to Pmax = 20, which is a significant reduction in the predictor set dimensionality as compared to previous studies reported[2, 3]. Two feature selection experiments were run on each data-set: one using the predictor set based on CFA features (cross modal factor analysis features) and the other using predictor set based on CCA features (canonical correlation analysis features). The DAGSVM classifier was used in evaluating the performance of all resulting predictor sets from both experiments. The DAGSVM is an all-pairs SVM-based multiclassifier which uses substantially less training time compared to neural networks, and has been shown to produce accuracy in some of the previous studies [2, 16].

Multiclass Microarray Gene Expression Analysis Based on Mutual Dependency Models

53

Table 1. Dimensionality of Predictor set for different benchmark datasets

Dataset PDL Lung SRBC MLL ALL

Type Affymetrix Affymetrix cDNA Affymetrix Affymetrix

N 12011 1741 2308 8681 3571

K 5 5 4 3 3

Pmax 25 20 15 12 10

N is the number of CMF(CCA/CFA) features. K is the number of classes in the dataset.

To evaluate the performance of the proposed CFA/CCA based predictors sets, we used two measures: classification accuracy and class-prediction error in class accuracy. For each class, class accuracy denotes the ratio of correctly classified samples of that class to the class size in the test set. The class-prediction error in class accuracies is the difference between the best class accuracy and the worst class accuracy among the K class accuracies in a K-class dataset. In an ideal situation, overall accuracy being exactly 1, each class accuracy is 1, so the perfect range of class accuracies is 0. Hence, lower the class-prediction error, better the classifier performance. Table 2. Classification Accuracy at Predictor set size of Pmax

Dataset NC160 PDL Lung SRBC MLL ALL

CFA 65% 92.6% 88.3% 91.6% 94.8% 94.9%

CCA 63% 90.4% 82.2% 89.8% 92.3% 93.7^

PCA 61% 86% 79.1% 81.5% 88.8 91.4%

As can be seen in Table 2, predictor set obtained by CFA method outperforms the CCA method by yielding better classifier accuracy for all the five datasets. Further, the class-prediction error in class accuracies shown in Table 3 also depicts a better performance of CFA method over CCA method. Finally, both the cross modal methods significantly outperform the PCA method. Table 3. Class-prediction error in Classification Accuracy at Predictor set size of Pmax

Dataset NC160 PDL Lung SRBC MLL ALL

CFA 0.69 0.34 0.53 0.12 0.18 0.16

CCA 0.71 0.38 0.59 0.18 0.20 0.21

PCA 0.78 0.41 0.62 0.21 0.28 0.24

54

G. Chetty and M. Chetty

5 Conclusions and Further Plan In this paper a novel feature selection technique based on mutual dependency modelling between genes is proposed for multiclass microarray gene expression classification. The two techniques based on cross modal factor analysis and canonical correlation analysis show a significant reduction in dimensionality and improvement in classification accuracy and deviation error for multiclass microarray gene expression datasets. Further research work will focus on feature selection techniques based on other multivariate analysis techniques such as co-inertia analysis and latent semantic analysis, and the fusion of predictor sets obtained from different mutual dependency models for large multiclass microarray datasets.

References 1. Dudoit, S., Fridly, J., Speed, T.P.: Comparison of Discrimination Methods for the Classification of Tumors Using Gene Expression Data (June 2000), http://www.stat.berkeley.edu/tech-reports/576.pdf 2. Ooi, C.H., Chetty, M., Teng, S.W.: Differential prioritization between relevance and redundancy in correlation-based feature selection techniques for multiclass gene expression data. BMVC Journal 47, 1–19 (2006) 3. Tripathi, A., Klami, A., Kaski, S.: Simple integrative preprocessing preserves what is shared in data sources. BMC Bioinformatics 9, 111 (2008) 4. Bittner, M., et al.: Molecular classification of cutaneous malignant melanoma by gene expression profiling. Nature 406(3), 536–540 (2000) 5. Cheng, Y., Church, G.M.: Biclustering of expression data. In: Proc. Eighth Int’l Conf. Intelligent Systems for Molecular Biology (ISMB), vol. 8, pp. 93–103 (2000) 6. Duggan, D.J., Bittner, M.L., Chen, Y., Meltzer, P., Trent, J.M.: Expression profiling using cDNA microarrays. Nature Genetics 21, 10–14 (1999) 7. Munagala, K., Tibshirani, R., Brown, P.: Cancer characterization and feature set extraction by discriminative margin clustering. BMC Bioinformatics 5, 21 (2004) 8. Ramaswamy, S., Tamayo, P., Rifkin, R., Mukherjee, S., Yeang, C.H., Angelo, M., Ladd, C., Reich, M., Latulippe, E., Mesirov, J.P., et al.: Multi-class cancer diagnosis using tumor gene expression signatures. Proc. Natl. Acad. Sci. USA 98, 15149–15154 (2001) 9. Ross, D.T., Scherf, U., Eisen, M.B., Perou, C.M., Rees, C., Spellman, P., Iyer, V., Jeffrey, S.S., Van de Rijn, M., Waltham, M., et al.: Systematic variation in gene expression patterns in human cance cell lines. Nat. Genet. 24, 227–235 (2000) 10. Yeoh, E.-J., Ross, M.E., Shurtleff, S.A., Williams, W.K., Patel, D., Mahfouz, R., Behm, F.G., Raimondi, S.C., Relling, M.V., Patel, A., et al.: Classification, subtype discovery, and prediction of outcome in pediatric lymphoblastic leukemia by gene expression profiling. Cancer Cell 1(2), 133–143 (2002) 11. Khan, J., Wei, J.S., Ringner, M., Saal, L.H., Ladanyi, M., Westermann, F., Berthold, F., Schwab, M., Antonescu, C.R., Peterson, C., et al.: Classification and diagnostic prediction of cancers using expression profiling and artificial neural networks. Nat. Med. 7, 673–679 (2001) 12. Bhattacharjee, A., Richards, W.G., Staunton, J.E., Li, C., Monti, S., Vasa, P., Ladd, C., Beheshti, J., Bueno, R., Gillette, M., et al.: Classification of human lung carcinomas by mRNA expression profiling reveals distinct adenocarcinoma subclasses. Proc. Natl. Acad. Sci. USA 98, 13790–13795 (2001)

Multiclass Microarray Gene Expression Analysis Based on Mutual Dependency Models

55

13. Armstrong, S.A., Staunton, J.E., Silverman, L.B., Pieters, R., den Boer, M.L., Minden, M.D., Sallan, S.E., Lander, E.S., Golub, T.R., Korsmeyer, S.J.: MLL translocations specify adistinct gene expression profile that distinguishes a unique leukemia. Nat. Genet. 30, 41– 47 (2002) 14. Golub, T.R., Slonim, D.K., Tamayo, P., Huard, C., Gaasenbeek, M., Mesirov, J.P., Coller, H., Loh, M.L., Downing, J.R., Caligiuri, M.A., et al.: Molecular classification of cancer: Class discovery and class prediction by gene expression monitoring. Science 286, 531–537 (1999) 15. Borga, M.: Canonical correlation a tutorial (1999), http://www.imt.liu.se/mi/Publications/magnus.html 16. Hsu, C.-W., Lin, C.-J.: A comparison of methods for multi-class support vector machines. IEEE Transactions on Neural Networks 13, 415–425 (2002)

An Eﬃcient Convex Nonnegative Network Component Analysis for Gene Regulatory Network Reconstruction Jisheng Dai1,2 , Chunqi Chang1 , Zhongfu Ye2 , and Yeung Sam Hung1 1 Department of Electrical and Electronic Engineering The University of Hong Kong, Pokfulam Road, Hong Kong 2 Department of Electronic Engineering and Information Science University of Science and Technology of China, Hefei, Anhui, P.R. China [email protected], [email protected], [email protected], [email protected]

Abstract. A systems biology problem of reconstructing gene regulatory network from time-course gene expression microarray data via network component analysis (NCA) is investigated in this paper. Inspired by the idea that each column of the connectivity matrix can be estimated independently, we try to propose a fast and stable convex approach for nonnegative NCA (nnNCA). Compared with the existing method, our new method reduces the computational cost substantially, whereas maintains a reasonable accuracy. Both the simulation results and experimental results demonstrate the eﬀectiveness of our method. Keywords: Gene regulatory network, microarray, network component analysis, convex programming, positivity constraints.

1

Introduction

Data from high-throughput DNA microarrays and ChIP-chip binding assays have become the basis of transcriptional regulatory analysis in the post-genome period. The study of gene regulatory network is important to understand the information embedded in these data. To reveal the underlying inter-dependency and cause-and eﬀect relationship between various cellular functions, much eﬀort has been devoted to the research of gene regulatory network reconstruction from microarray data. Developing tractable system identiﬁcation techniques capable of reconstructing gen regulatory networks of a large size from small microarray datasets is a key challenge for gene regulatory network reconstruction. Several diﬀerent approaches have been proposed to modeling gene regulatory network in early studies, including dynamic Bayesian networks (DBN) [1], probabilistic Boolean network (PBN) [2], and diﬀerential/diﬀerence equation [3]. More recently, with the assumption of linear instantaneous signal models, statistical techniques such as the principal component analysis (PCA) [4] and independent component analysis (ICA) [5] have been applied successfully to deduce biologically signiﬁcant information from V. Kadirkamanathan et al. (Eds.): PRIB 2009, LNBI 5780, pp. 56–66, 2009. c Springer-Verlag Berlin Heidelberg 2009

Convex Nonnegative NCA for Gene Regulatory Network Reconstruction

57

microarray datasets. However, neither the reconstructed networks nor the reconstructed regulatory signals is consistent with real biological systems in general, because in these approaches the assumption of mutually orthogonality or statistical independence for the regulatory signals are not usually valid. The network component analysis (NCA) has been shown to be a very eﬀective approach because it incorporates helpful and biologically sound assumptions [6, 7, 8]. It is well known that a real gene regulatory network always has a sparse structure, and sometimes the non-zero entries of its (sparse) connectivity matrix may be obtained from ChIP-chip experiments [9]. Even when experimental data on the structure of the network are unavailable, the structural information sometimes may be extracted partially from the literature or predicted by bioinformatics methods [10]. The NCA approach can make use of this (sparse) structure information to fully reconstruct the network (including the connectivity matrix and the regulatory signals) in the case of noise-free, if some mild conditions, called NCA criteria, are satisﬁed. Most recently, the method was further improved in [11], where a closed-form solution to the NCA problem is obtained through ﬁtting the model by a series of subspace projections. So far, NCA is one of the most eﬀective approaches to gene regulatory network reconstruction, existing algorithms, however, lack accuracy and consistency. To improve the performance of NCA and develop a more robust network reconstruction method, convex programming methods incorporating the nonnegativity constraints on the connectivity matrix are proposed in [12, 13]. Although this convex optimization problem can be solved by an inner-point algorithm, the required complexity is still very high, if the number of the entries in the connectivity matrix is large which is often happened in biological environments. This motivates us to ﬁnd an eﬃcient nonnegative network component analysis (nnNCA) approach for gene regulatory network reconstruction subject to the nonnegativity constraints. Inspired by the idea that each column of the connectivity matrix may be solved independently [11], we successfully derive a very fast and stable approach to nnNCA in this paper. Compared with the existing method, the computational cost is reduced substantially. Both the simulation results and experimental results demonstrate the eﬀectiveness of our method.

2

Network Component Analysis

Figure 1 illustrates the instantaneous linear regulation model we are addressing, which is a bipartite network. Gene expressions are regulated by transcription factors. The upper layer in the ﬁgure represents the expression level of activated transcription factors (TF) or transcription factor activities (TFA), and the lower layer represents the microarray gene expression data. In general, gene regulation processes are dynamic and nonlinear, but are approximately log-linear [14]. Assuming that in the network we have N genes and M transcription factors, and the length of time series is K. Then if the gene expression are represented as log-ratios, the network can be modeled as X = AS + Γ

(1)

58

J. Dai et al.

TF1: s1

x1

x2

TF2:s2

x3

x4

TF3: s3

x5

x6

x7

Fig. 1. Transcriptional regulatory model. The expression of genes (lower layer) are regulated by transcription factors (upper layer) through a bipartite network.

where X ∈ RN ×K , A ∈ RN ×M , S ∈ RM×K and Γ ∈ RN ×K are the gene expression levels which can be measured by microarray data, the connectivity matrix of the network, the unknown TFAs, and the measurement noise, respectively. It should be noted that the connectivity matrix A are assumed sparse, which agrees well with observations in biological environments. Our goal is to recover the connectivity matrix A and the TFAs S from the time-course microarray data X. It is proven in [6] that A and S can be estimated up to possible scaling factors, which is the inherent indeterminacy for any blind source separation problem, if the following NCA criteria are satisﬁed: 1. the connectivity matrix A must be full-column rank; 2. when an element in the regulatory domain is removed together with the rows corresponding to the nonzero entries of this column, the connectivity matrix of the resulting network is still of full-column rank; 3. the TFAs S must have full-row rank. Under the constraint that the non-zero pattern of A has the structure corresponding to the known a priori network topology, the A and S can be uniquely determined by solving the optimization problem as follows. min X − ASF A,S

(2)

subject to : A(I) = 0 where · F is the Frobenius norm of a matrix, and I contains the indices where the entries of A are zeros, which is induced by the network structure. Since the actual estimation of A and S is performed by a two-step alternating leastsquares algorithm, very high computational complexity is required. To reduce the computational cost, [11] makes a eﬀort to develop a fast, stable and globally convergent algorithm for reconstructing the gene-regulatory networks, referred as FastNCA algorithm, where a closed-form solution to the NCA problem is given. Interestingly, it was shown that the connectivity matrix A can be estimated

Convex Nonnegative NCA for Gene Regulatory Network Reconstruction

59

ﬁrst, especially each column of A can be obtained independently, then the TFA can be obtained simply by S = A† X (3) where A† is the pseudo inverse of A. However, all the NCA approaches only use the structure information about the connectivity matrix. In attempt to improve the performance of NCA method, as well as develop a more accurate and robust method for the network reconstruction, a convex algorithm incorporating another new prior information (the entries of connectivity matrix A are all nonnegative, also referred as nnNCA) is proposed in [12]. It is reasonable to put the nonnegative constraint on the entries of the connectivity matrix A, as long as any speciﬁc transcription factor has the same eﬀect to all genes. The sound biological support for this assumption can be found in [15]. With this nonnegative constraint, in [12], it is shown that the connectivity matrix A can be estimated by solving the following optimization problem min CT AF (4) A,S

subject to : A(I) = 0 A(J) ≥ c where C is the null space of matrix X, J contains the indices where the entries of A are nonzero (positive), and c is a constant small positive value. Although the above convex optimization problem can be solved by inner-point algorithm [16], the required computational complexity is still very high, if the number of the entries in matrix A is large which is often happened in real biological circumstance. Inspired by the idea that each column of the connectivity matrix A may be solved independently [11], we try to propose a very eﬃcient approach to nnNCA based on individual estimation of column of A in the next section.

3

Eﬃcient Approach to nnNCA

It is well known that if there is noise in the model Eq. (1), the eﬀect from noise can be coped with by applying the singular value decomposition (SVD) [17] to matrix X. For simplicity of analysis, we ﬁrst write X (assuming no noise) in the standard SVD form as follows. Σ0 Σ0 T X=U V = Us Un VT = Us Σ 0 VT (5) 0 0 0 0 where U ∈ RN ×N and V ∈ RK×K are the singular-vector matrices, Us ∈ RN ×M and Un ∈ RN ×(N −M) are the singular-vector matrices corresponding to M nonzero singular-values and N −M zero singular-values, respectively, and Σ ∈ RM×M is the singular-value matrix containing nonzero singular-values arranged

60

J. Dai et al.

in a decreasing order from top-left to bottom-right. From the subspace principle [18], we know that there always exists a matrix T ∈ RM×M such that Us = AT

(6)

Notice that the above equation only holds if there is no noise in Eq. (1). Nevertheless, AT can provide a robust estimate of X in the existence of noise. Therefore, we still assume that Eq. 6 holds in noise case. Since we can always re-order the rows of X and A, we assume without loss of generality that the ﬁrst column of A can be partitioned as ˜ a a1 = (7) 0L×1 ˜ ∈ R(N −L)×1 . where L is number of zero entries in the vector a1 , and a To show that each column of the connectivity matrix A can be solved independently for nnNCA, we need the following lemma, which forms the basis of our method. Lemma 1. If there is a vector t ∈ RM×M satisfying b Us t = 0L×1

(8)

˜ up to a scaling ambiguity where b ∈ R(N −L)×1 , then b must be an estimate of a η, i.e., b = η˜ a (9) Proof: Using Eq. (6) and Eq. (8), we have

b ˆ Us t = ATt = At = 0

where ˆt Tt. We partition the above equation accordingly as ˜ A12 a η b ˆ At = = 0 A22 t2 0

(10)

(11)

or, equivalently, b = η˜ a + A12 t2

(12)

0 = A22 t2

(13)

Since A22 is of full-column rank determined by NCA criterion (2), according to Eq. (13) we are able to obtain t2 = 0 (14) Then substituting the above equation into Eq. (12), we achieve the desired result b = η˜ a

(15) End of proof

Convex Nonnegative NCA for Gene Regulatory Network Reconstruction

61

Relying on Lemma 1, we are able to derive a very simply approach to estimate a ˜1 . Since we can always re-order the columns of A and the rows of T accordingly while keeping the equality in Eq. (6), the other columns of A can be estimated in the same way. Keeping these results in mind, we now proceed to solve the NCA problem with positivity constraints on the nonzero entries of the connectivity matrix. Unlike the strategy employed in [12], we follow a diﬀerent approach where each column of the connectivity matrix A is estimated independently, so that the computational cost will be reduced substantially. The optimization problem for estimating the i-th column (denoted as ai ) with positivity constraints can be formulated as min Us (Ii , :) · ti F ti

(16)

subject to : Us (Ji , :) · ti ≥ c where Ii contains the indices where the entries of ai are zeros, Ji contains the indices where the entries of ai are nonzero, and Us (Ii , :) denotes the rows of Us corresponding to indices Ii . Eq. (16) is convex and can be solved by eﬀective algorithms [16]. Recall Lemma 1, it is not diﬃcult to see that the estimate of ai can be given by ai = Us ti (17) From the above discussions, now we summarize the proposed algorithm as follows. Eﬃcient Algorithm for nnNCA: 1. Perform a standard SVD on X and obtain the Us as in Eq. (5); 2. Estimation of A For i = 1, 2, · · · , M a) solve the optimization problem (16) and get the optimal ti ; b) let ai = Us ti . End 3. Estimate the TFA matrix by S = A† X.

4 4.1

Results Simulation Results

In order to test the proposed method, we use the simulation data described in [6, 12]. This is a hemoglobin spectroscopy data set with a 7 × 3 connectivity matrix and a 7 × 321 measurement data. If positivity constraints on the nonzero entries of the connectivity matrix and the noise are not considered, the same performances are achieved by the FastNCA, the method in [12], and our proposed

62

J. Dai et al.

Table 1. True connectivity matrix and its estimation by FastNCA, the method in [12], and our proposed method without positivity constraints and noise True 0.417 2.083 0 0.417 1.25 0.833 0

FastNCA

1.184 0 1.184 1.053 0.921 0 0.658

0 1.25 0.25 0.25 0 2.00 1.25

0.435 2.12 0 0.429 1.176 0.841 0

1.196 0 1.164 1.056 0.904 0 0.681

Original

Method in [12]

0 1.267 0.202 0.236 0 2.015 1.281

0.435 2.12 0 0.429 1.176 0.841 0

FastNCA

0.4

0 1.267 0.202 0.236 0 2.015 1.281

0.435 2.12 0 0.429 1.176 0.841 0

Method in [12]

0.2

0.3

1.196 0 1.164 1.056 0.904 0 0.681

Proposed

0.1

0.2

1.196 0 1.164 1.056 0.904 0 0.681

0 1.267 0.202 0.236 0 2.015 1.281

Proposed

0.6

0.6

0.4

0.4

0.2

0.2

0 0.1 −0.1

0 −0.1

0

100

200

300

0.6

−0.2

0

100

200

300

0.8 0.6

0.4

0

0

−0.2

−0.2

−0.4

0

100

200

300

−0.4

0.6

0.6

0.4

0.4

0.2

0.2

0

100

200

300

0

100

200

300

0

100

200

300

0.4 0.2 0.2 0 −0.2

0

0 0

100

200

300

0.6 0.4

−0.2

0

100

200

300

−0.2

0

0

100

200

300

−0.2

0.8

0.8

0.8

0.6

0.6

0.6

0.4

0.4

0.4

0.2

0.2

0.2

0

0

0.2 0 −0.2

0

100

200

300

−0.2

0

100

200

300

−0.2

0 0

100

200

300

−0.2

Fig. 2. True sources and their corresponding estimates by FastNCA, the method in [12], and our proposed method subject to positivity constraints under noisy case

method. The normalized true (original) and estimated connectivity matrices are shown in Table 1. In the next simulation, we study the performance of the proposed algorithm subject to positivity constraints in the noisy case, where the signal to noise ratio (SNR) is set to 9 dB. The true and the estimated source spectra signals are shown in Figure 2, and the normalized true and estimated connectivity matrices are shown in Table 3. The simulation results in the table shows that the connectivity matrix estimated by FastNCA contains both positive and negative entries. Since the true entries connectivity matrix are either nonnegative or nonpositive, such an estimate is unfavorable. The simulation results in the ﬁgure shows that the estimated connectivity matrix achieved by the method in [12] and our method are almost the same. This well demonstrates the optimality of our method.

Convex Nonnegative NCA for Gene Regulatory Network Reconstruction

63

Table 2. True connectivity matrix and its estimation by FastNCA, the method in [12], and our proposed method subject to positivity constraints under noisy case True 0.417 2.083 0 0.417 1.25 0.833 0

FastNCA

1.184 0 1.184 1.053 0.921 0 0.658

0 1.25 0.25 0.25 0 2.00 1.25

-1.785 -0.202 0 -0.947 -0.279 1.788 0

Method in [12]

Proposed

1.028 0 0.985 1.029 0 1.12 1.028 0 0 2.065 1.505 0 1.236 1.485 0 1.358 1.244 -0.468 0 1.245 0.67 0 1.244 0.604 0.966 0.096 1.027 0.965 0.67 0.917 0.966 0.604 1.087 0 0.741 1.087 0 0.74 1.087 0 0 1.782 0.741 0 1.754 0.74 0 1.537 0.675 0.59 0 0.674 0.67 0 0.675 0.898

Table 3. The running time comparison between our method and the method in [12] Proposed

Method in [12]

Running Time 6.71 (seconds) 1144.62 (seconds)

4.2

Experimental Results

To test the proposed method in the real biological networks, we apply all the methods to analyze the time-course Yeast cell cycle microarray data used in [19]. Note that the network topology data required by the algorithm are from [9]. The microarray data contains 441 genes and 56 time points. In this analysis, we are interested in the 11 transcription factors that are known to regulate the expression of genes that are involved in the cell cycle process. Three experiments with diﬀerent synchronization methods represented by alpha, cdc15, and elutriation Method in [12] [7]

FastNCA 0.1 0 −0.1 0 0.05 0 −0.05 0 0.1 0 −0.1 0 0.1 0 −0.1 0 0.05 0 −0.05 0 0.1 0 −0.1 0 0.05 0 −0.05 0 0.2 0 −0.2 0 0.2 0 −0.2 0 0.2 0 −0.2 0 0.1 0 −0.1 0

1

2

3

4

5

6

1

2

3

4

5

6

1

2

3

4

5

6

1

2

3

4

5

6

1

2

3

4

5

6

1

2

3

4

5

6

1

2

3

4

5

6

1

2

3

4

5

6

1

2

3

4

5

6

1

2

3

4

5

6

1

2

3

4

5

6

0.1 0 −0.1 0 0.05 0 −0.05 0 0.1 0 −0.1 0 0.1 0 −0.1 0 0.05 0 −0.05 0 0.1 0 −0.1 0 0.05 0 −0.05 0 0.2 0 −0.2 0 0.2 0 −0.2 0 0.2 0 −0.2 0 0.1 0 −0.1 0

Proposed

1

2

3

4

5

6

1

2

3

4

5

6

1

2

3

4

5

6

1

2

3

4

5

6

1

2

3

4

5

6

1

2

3

4

5

6

1

2

3

4

5

6

1

2

3

4

5

6

1

2

3

4

5

6

1

2

3

4

5

6

1

2

3

4

5

6

0.1 0 −0.1 0 0.05 0 −0.05 0 0.1 0 −0.1 0 0.1 0 −0.1 0 0.05 0 −0.05 0 0.1 0 −0.1 0 0.05 0 −0.05 0 0.2 0 −0.2 0 0.2 0 −0.2 0 0.2 0 −0.2 0 0.1 0 −0.1 0

1

2

3

4

5

6

1

2

3

4

5

6

1

2

3

4

5

6

1

2

3

4

5

6

1

2

3

4

5

6

1

2

3

4

5

6

1

2

3

4

5

6

1

2

3

4

5

6

1

2

3

4

5

6

1

2

3

4

5

6

1

2

3

4

5

6

Fig. 3. Estimated yeast cell cycle related TFAs for alpha synchronization by FastNCA, the method in [12], and our proposed method

64

J. Dai et al.

are studied. The estimated TFAs for these 11 cell cycle related transcription factors with diﬀerent synchronization methods are shown in Figure 3, Figure 4 and Figure 5 for FastNCA, the method in [12], and our proposed method, respectively. It can be seen that the method in [12] and our proposed method produce almost the same results, and it is obvious that the estimated TFAs subject to positivity constraints show more cyclic behavior than that achieved by FastNCA. Cyclic behavior is expected in this case since all these transcription factors are cell cycle related. The results demonstrate the superior robustness of the nnNCA methods and the equivalence between our method and the existing method. FastNCA 0.1 0 −0.1 0 0.1 0 −0.1 0 0.1 0 −0.1 0 0.2 0 −0.2 0 0.2 0 −0.2 0 0.2 0 −0.2 0 0.2 0 −0.2 0 0.2 0 −0.2 0 0.1 0 −0.1 0 0.2 0 −0.2 0 0.1 0 −0.1 0

Method in [12]

0.5

1

1.5

2

0.5

1

1.5

2

0.5

1

1.5

2

0.5

1

1.5

2

0.5

1

1.5

2

0.5

1

1.5

2

0.5

1

1.5

2

0.5

1

1.5

2

0.5

1

1.5

2

0.5

1

1.5

2

0.5

1

1.5

2

0.1 0 −0.1 0 0.1 0 −0.1 0 0.1 0 −0.1 0 0.2 0 −0.2 0 0.2 0 −0.2 0 0.2 0 −0.2 0 0.2 0 −0.2 0 0.2 0 −0.2 0 0.1 0 −0.1 0 0.2 0 −0.2 0 0.1 0 −0.1 0

Proposed

0.5

1

1.5

2

0.5

1

1.5

2

0.5

1

1.5

2

0.5

1

1.5

2

0.5

1

1.5

2

0.5

1

1.5

2

0.5

1

1.5

2

0.5

1

1.5

2

0.5

1

1.5

2

0.5

1

1.5

2

0.5

1

1.5

2

0.1 0 −0.1 0 0.1 0 −0.1 0 0.1 0 −0.1 0 0.2 0 −0.2 0 0.2 0 −0.2 0 0.2 0 −0.2 0 0.2 0 −0.2 0 0.2 0 −0.2 0 0.1 0 −0.1 0 0.2 0 −0.2 0 0.1 0 −0.1 0

0.5

1

1.5

2

0.5

1

1.5

2

0.5

1

1.5

2

0.5

1

1.5

2

0.5

1

1.5

2

0.5

1

1.5

2

0.5

1

1.5

2

0.5

1

1.5

2

0.5

1

1.5

2

0.5

1

1.5

2

0.5

1

1.5

2

Fig. 4. Estimated yeast cell cycle related TFAs for cdc15 synchronization by FastNCA, the method in [12], and our proposed method Method in [12] [7]

FastNCA 0.1 0 −0.1 0 0.1 0 −0.1 0 0.1 0 −0.1 0 0.1 0 −0.1 0 0.1 0 −0.1 0 0.2 0 −0.2 0 0.1 0 −0.1 0 0.1 0 −0.1 0 0.1 0 −0.1 0 0.2 0 −0.2 0 0.1 0 −0.1 0

1

2

3

4

5

1

2

3

4

5

1

2

3

4

5

1

2

3

4

5

1

2

3

4

5

1

2

3

4

5

1

2

3

4

5

1

2

3

4

5

1

2

3

4

5

1

2

3

4

5

1

2

3

4

5

0.1 0 −0.1 0 0.1 0 −0.1 0 0.1 0 −0.1 0 0.1 0 −0.1 0 0.1 0 −0.1 0 0.2 0 −0.2 0 0.1 0 −0.1 0 0.1 0 −0.1 0 0.1 0 −0.1 0 0.2 0 −0.2 0 0.1 0 −0.1 0

Proposed

1

2

3

4

5

1

2

3

4

5

1

2

3

4

5

1

2

3

4

5

1

2

3

4

5

1

2

3

4

5

1

2

3

4

5

1

2

3

4

5

1

2

3

4

5

1

2

3

4

5

1

2

3

4

5

0.1 0 −0.1 0 0.1 0 −0.1 0 0.1 0 −0.1 0 0.1 0 −0.1 0 0.1 0 −0.1 0 0.2 0 −0.2 0 0.1 0 −0.1 0 0.1 0 −0.1 0 0.1 0 −0.1 0 0.2 0 −0.2 0 0.1 0 −0.1 0

1

2

3

4

5

1

2

3

4

5

1

2

3

4

5

1

2

3

4

5

1

2

3

4

5

1

2

3

4

5

1

2

3

4

5

1

2

3

4

5

1

2

3

4

5

1

2

3

4

5

1

2

3

4

5

Fig. 5. Estimated yeast cell cycle related TFAs for elutriation synchronization by FastNCA, the method in [12], and our proposed method

Convex Nonnegative NCA for Gene Regulatory Network Reconstruction

65

Finally, to testify the improvement in computational cost, Table 3 gives the running time for our method and the method in [12]. All the computer codes are developed in MATLAB 7.7.0 on a Intel Core2 2.66GHz with 2GB of RAM. The result in the table shows that our method is much faster than the method in [12]. Both the simulation results and experimental results demonstrate the eﬀectiveness of our proposed method. Compared with the existing method, our new method reduces the computational cost substantially, whereas maintains a reasonable accuracy.

5

Conclusion

Both the simulation results and experimental results shows the superior robustness of the nnNCA methods, and demonstrate that the performances of our method and the method in [12] are very similar. The contribution of this paper is to derive a powerful convex approach for nnNCA, which reduces the computational cost substantially, whereas maintains a reasonable accuracy.

Acknowledgements This work is supported by a Seed Funding for Basic Research of the University of Hong Kong.

References 1. Friedman, N., Linial, M., Nachman, I., Peer, D.: Using bayesian networks to analyze expression data. Journal of Computational Biology 7(3), 601–620 (2000) 2. Shmulevich, I., Dougherty, E.R., Kim, S., Zhang, W.: Probabilistic boolean networks: a rule-based uncertainty model for gene regulatory networks. Bioinformatics 18(2), 261–274 (2002) 3. Dougherty, E.R., Shmulevich, I., Chen, J., Wang, Z.J.: Genomic Signal Processing and Statistics. Hindawi Publishing Corporation (2005) 4. Alter, O., Brown, P.O.: Singular value decomposition for genome-wide expression data processing and modeling. Proceedings of the National Academy of Sciences of the United States of America 97(18), 10101–10106 (2000) 5. Lee, S.I., Batzoglou, S.: Application of independent component analysis to microarrays. Genome Biology 4(11), R76 (2003) 6. Liao, J.C., Boscolo, R., Yang, Y.L., Tran, L.M., Sabatti, C., Roychowdhury, V.P.: Network component analysis: Reconstruction of regulatory signals in biological systems. Proceedings of the National Academy of Sciences of the United States of America 100(26), 15522–15527 (2003) 7. Chang, C.Q., Hung, Y.S., Fung, P.C.W., Ding, Z.: Network component analysis for blind source separation. In: Proc. 2006 International Conference on Communications, Circuits and Systems, Guilin, China, pp. 323–326 (2006) 8. Chang, C.Q., Ding, Z., Hung, Y.S., Fung, P.C.W.: Fast network component analysis for gene regulation networks. In: Proc. 2007 IEEE International Workshop on Machine Learning for Signal Processing, Thesaloniki, Greece, pp. 21–26 (2007)

66

J. Dai et al.

9. Lee, T.I., Rinaldi, N.J., Robert, F., Odom, D.T., Bar-Joseph, Z., Gerber, G.K., Hannett, N.M., Harbison, C.T., Thompson, C.M., Simon, I., Zeitlinger, J., Jennings, E.G., Murray, H.L., Gordon, D.B., Ren, B., Wyrick, J.J., Tagne, J.B., Volkert, T.L., Fraenkel, E., Giﬀord, D.K., Young, R.A.: Transcriptional regulatory networks in saccharomyces cerevisiae. Science 298(5594), 799–804 (2002) 10. Wang, C., Xuan, J., Chen, L., Zhao, P., Wang, Y., Clarke, R., Hoﬀman, E.: Motifdirected network component analysis for regulatory network inference. BMC Bioinformatics 9(suppl. 1), S21 (2008) 11. Chang, C.Q., Ding, Z., Hung, Y.S., Fung, P.C.W.: Fast network component analysis (FastNCA) for gene regulatory network reconstruction from microarray data. Bioinformatics 24(11), 1349–1358 (2008) 12. Chang, C.Q., Ding, Z., Hung, Y.S.: A new optimization algorithm for network component analysis based on convex programming. In: Proc. 2009 IEEE International Conference on Acoustics, Speech and Signal Processing, Taipei, Taiwan (2009) 13. Chang, C.Q., Ding, Z., Hung, Y.S.: Nonnegative network component analysis by linear programming for gene regulatory network reconstruction. In: Adali, T., et al. (eds.) ICA 2009. LNCS, vol. 5441, pp. 395–402. Springer, Heidelberg (2009) 14. Savageau, M.A.: Biochemical Systems Analysis: A Study of Function and Design in Molecular Biology. Addison-Wesley, Reading (1976) 15. Alon, U.: An Introduction to Systems Biology: Design Principles of Biological Circuits. Chapman & Hall/CRC, Boca Raton (2007) 16. Boyd, S., Vandenberghe, L.: Convex Optimization. Cambridge University Press, Cambridge (2004) 17. Golub, G.H., van Loan, C.F.: Matrix Computation, 3rd edn. The Johns Hopkins University Press (1996) 18. Lay, D.C.: Linear Algebra and Its Applications, 2nd edn. Addison-Wesley, New York (2000) 19. Spellman, P.T., Sherlock, G., Zhang, M.Q., Iyer, V.R., Anders, K., Eisen, M.B., Brown, P.O., Botsein, D., Futcher, B.: Comprehensive identiﬁcation of cell cycleregulated genes of the yeast saccharomyces cerevisiae by microarray hybridization. Molecular Biology of the Cell 9(12), 3273–3297 (1998)

Using Higher-Order Dynamic Bayesian Networks to Model Periodic Data from the Circadian Clock of Arabidopsis Thaliana Rónán Daly1 , Kieron D. Edwards2 , John S. O’Neill3 , Stuart Aitken4 , Andrew J. Millar4 , and Mark Girolami1 1

Inference Group, Department of Computing Science, University of Glasgow {rdaly,girolami}@dcs.gla.ac.uk 2 Advanced Technologies (Cambridge) Limited [email protected] 3 Institute of Metabolic Science, Metabolic Research Laboratories, University of Cambridge [email protected] 4 Centre for Systems Biology at Edinburgh, The University of Edinburgh {s.aitken,andrew.millar}@ed.ac.uk

Abstract. Modelling gene regulatory networks in organisms is an important task that has recently become possible due to large scale assays using technologies such as microarrays. In this paper, the circadian clock of Arabidopsis thaliana is modelled by ﬁtting dynamic Bayesian networks to luminescence data gathered from experiments. This work diﬀers from previous modelling attempts by using higher-order dynamic Bayesian networks to explicitly model the time lag between the various genes being expressed. In order to achieve this goal, new techniques in preprocessing the data and in evaluating a learned model are proposed. It is shown that it is possible, to some extent, to model these time delays using a higher-order dynamic Bayesian network. Keywords: Dynamic Bayesian Network, Gene Regulatory Network, Gene Expression, Arabidopsis Thaliana.

1

Introduction

The analysis of data obtained from experiments looking at the expression of genes has become an important topic in the realm of bioinformatics. Whilst other bioinformatics tasks such as sequence alignment, gene ﬁnding, genome assembly and genome annotation have proven very amenable to computational analysis, there are tasks which are intrinsically diﬃcult at the current level of knowledge. These include protein structure prediction and modelling gene regulatory networks. The latter will be examined in this paper. It is well known that genes code for proteins and that certain proteins can aﬀect how certain genes are expressed. Gene regulatory networks encode these dependencies between genes in an organism, abstracting away the inﬂuence of V. Kadirkamanathan et al. (Eds.): PRIB 2009, LNBI 5780, pp. 67–78, 2009. c Springer-Verlag Berlin Heidelberg 2009

68

R. Daly et al.

proteins so that nodes correspond to genes and directed arcs correspond to causal gene-expression relations. In this paper, an attempt will be made to model these gene regulatory networks and in particular, the time lag between the expression of a gene and its eﬀect on the expression of another gene. This modelling will proceed by using an algorithm based on ant colony optimisation known as ACO-E [1] to build dynamic Bayesian networks (DBNs). DBNs are an extension of Bayesian networks, graphical models that can be useful in modelling probabilistic relations between variables [2]. They consist of a prior network, used to model initial conditions, and a transition network, used to model the eﬀect of interactions over time. Whilst DBNs have been used to model gene regulatory networks [3,4,5,6,7], the method presented here diﬀers in how the data are treated and in how it can infer interactions at multiple time periods. To test the methods proposed, a set of gene expression data obtained from experiments observing particular genes of the plant Arabidopsis thaliana will be used. These genes are known to behave in a clock-like fashion that regulates the function of the plant [8,9]. The data will be preprocessed and experiments will be conducted to learn the structure of DBNs using these preprocessed data. Using a new technique, these networks will be compared to a standard network developed by an expert in the ﬁeld being analysed. Using these results, the ability of DBNs to reconstruct gene-regulatory networks will be investigated.

2

Background

The analysis of gene regulatory networks is a hard task. In order to see how various genes interact, experiments must be performed which measure the expression levels of these genes. Using this data, it is possible to develop models of genetic systems behaviour. One of the earliest and still commonly used techniques is to use diﬀerential equations that show how expression levels vary with respect to other expression levels. However, recent advances in measurement of gene expression levels, using techniques such as DNA microarrays, have allowed the measurements of tens of thousands of genes to be performed simultaneously. With this amount of genes in an organism, all possible interactions between all the genes cannot be looked at. Therefore, methods involving graphical representations have become more widely used. These include Boolean networks, networks similar to neural networks, stochastic process calculi and Bayesian networks [10]. 2.1

Data from Arabidopsis Thaliana Experiments

The experimental study in this chapter will involve luminescence data obtained from experiments on Arabidopsis thaliana. The luminescence is an indirect measure of the synthesis of new RNA transcripts from each gene, originating from a copy of the cognate promoter and 5 untranslated region that are fused to the ﬁreﬂy luciferase reporter gene and integrated into the plant genome at a random location. The particular experiments in question were designed to study

Using Higher-Order Dynamic Bayesian Networks to Model Periodic Data

69

Table 1. Ratio of light to darkness for each experimental condition Condition AT0029 AT0030 AT0031b AT0032 AT0033 AT0047 Ratio 6:18 9:15 12:12 15:9 18:6 3:21

the ‘plant circadian rhythms’ of this organism, i.e. the oscillating behaviour of plants as they respond to the change in sunlight [11]. Brieﬂy, the expression levels of the genes in plants tend to synchronise with the rising and setting of the sun, oscillating between high and low levels. However, when the light source is removed, the expression levels continue to oscillate with the same frequency, with the behaviour decaying over time. In the experiments, there were in general two phases. The ﬁrst phase, called the entrainment, switched the light on and oﬀ at regular intervals and lasted three days. The second phase had constant light and also lasted three days. At intervals of 1.5 hours, readings were taken of the expression levels of the genes being investigated. The experiments involved ten diﬀerent genes: CAB, GI, CCA1, CCR2, LHY, CAT3, ELF3, PRR9, TOC1 and ELF4. Further information about these genes can be found elsewhere [11]. There were also six diﬀerent conditions under which the experiments were conducted; each condition had a diﬀerent light period for the entrainment phase of the experiment. Table 1 shows, for each experimental condition, the ratio of light to dark for that condition. E.g. for condition AT0029, there are 6 hours of light followed by 18 hours of darkness in the entrainment phase. Because the experiments lasted six days, and a sample was taken every 1.5 hours, there were a total of 96 samples taken for each condition. The ﬁrst of these was taken as a reference datum and subtracted from each of the following samples, leading to 95 samples with which to work.

3

Modelling Gene Expression Data Using DBNs

It is the stochastic nature of the gene expression process that makes probabilistic models an appropriate choice in modelling them. One interesting approach is in using Bayesian networks as a probabilistic model. This enables large scale, non-linear interactions between many genes to be represented and simulated. E.g., recent studies have looked at learning Bayesian networks with thousands of genes [12]. The large amounts of data present in microarray studies can be used to learn both the structure and parameters of the network – missing values and noise can also be taken care of. Also, Bayesian networks can be given a causal interpretation. A problem arises however in that loops are not permitted in Bayesian networks. Therefore, feedback processes cannot be represented. A solution to this problem is to use dynamic Bayesian networks (DBNs). With DBNs, it is possible to ‘unroll’ what would be a loop in a Bayesian network across

70

R. Daly et al.

Fig. 1. A 4-layer dynamic Bayesian network

time. In doing so, a temporal order is brought in amongst the various variables. In this respect, DBNs are a natural way to model temporal data. 3.1

Learning a Higher-Order Dynamic Bayesian Network

Most Bayesian network structure learning algorithms are designed to learn a static structure. Whilst dynamic Bayesian networks are very similar to static networks, there are slight diﬀerences that mean extra care has to be taken when trying to learn them. Bearing in mind that a DBN has two parts, the prior and transition network, this section will focus on learning the transition network; the prior network is exactly the same as a normal Bayesian network. In a transition network, the nodes are grouped into multiple layers that designate diﬀerent timepoints. Arcs can never go back in time and the head of any arc can only be in a single layer, normally that layer at time t. It is very normal to have two layers, such that they correspond to variables at time t and t − 1. In fact, in all the general literature on Bayesian networks surveyed, learning DBNs amounted to learning DBNs with two layers. However, in general, n + 1 layers can be speciﬁed, from t − n to t. An example of a 4-layer DBN is shown in Fig. 1. When learning such a DBN, it is suﬃcient to add an extra constraint to the learning algorithm, i.e. the head of an arc must always be in the layer at time t. By following this, any Bayesian network structure learning algorithm that searches through the space of DAGs can be used and will produce a valid network.

4

Experimental Methodology

In order to test the ability of multi-layered DBNs to model a genetic regulatory network, experiments were conducted using the Arabidopsis thaliana data discussed in Sect. 2.1. This section will describe the steps taken to achieve this. This includes preprocessing the data, formulating prior knowledge, designing a methodology to run the algorithm and testing the results obtained against expert knowledge.

Using Higher-Order Dynamic Bayesian Networks to Model Periodic Data

4.1

71

Preprocessing the Data

The data obtained from the experiments described in Sect. 2.1 are continuous in nature, i.e. they show the level of gene expression as measured. However, most scoring functions for use with structure learning are based on nominal data, i.e. data that is of a discrete nature. For the purposes of these experiments, the BDeu scoring function was used [13]. For this to be utilised, the data has to be discretised. Whilst binary discretisation is an easy option, there is a problem with this as shown by Fig. 2a. It can be seen that as the expression level decays after entrainment, the discretised expression level ﬂatlines, even though there is still oscillatory action occurring. One way to counter this is to discretise the data into multiple levels. However, in doing so, the amount of data at each level is reduced. This can lead to less support for dependencies when a model is ﬁtted to the data, something that should be avoided when there is not much data to begin with. In order to avoid this problem the ﬁrst derivative of the data was taken and the data discretised as to whether this derivative was greater than or less than zero. The results of this procedure are shown in Fig. 2b. It can be seen that this discretisation captures the oscillations present in the original data. In a sense, instead of looking for correlations in the mRNA level, we are looking for correlations in the rising and falling of the mRNA level. To the authors’ knowledge, this is the ﬁrst time this has been done when learning Bayesian networks that model gene regulatory networks. Finally due to the expert being able to provide knowledge on the behaviour of ﬁve of the genes and the light source, these were selected as the variables that would be used in the construction of the DBN. The genes in question were GI, CCA1, LHY, PRR9 and TOC1. Whilst the other genes could have been included in the analysis, any connections among them would be unveriﬁable by expert knowledge in this study. 4

6

4

x 10

x 10 Expression Level Discretised Level 1

Expression Level Difference Discretised Level 1

Expression Level Difference

Expression Level

5

4

3

0

2 0 1

0

10

20

30

40

50 60 Time Steps

70

(a) Expression level

80

90

100

0 0

10

20

30

40

50 60 Time Steps

70

80

90

(b) Expression level derivative

Fig. 2. Expression levels for PRR9 in AT0029 condition

100

72

4.2

R. Daly et al.

Prior Knowledge

In order to provide more meaningful results and to compensate for the absence of much data, prior knowledge was incorporated into the learning algorithm in the form of constraints as to the allowable edges. These constraints are summarised below: – Connections from a variable to the same variable at a later time point were disallowed. This type of information is trivial and does not add anything interesting to the model. – No connection was allowed to the light variable, as this was causally independent of the gene expression values. – Since interactions at this genetic level took time in the region of at least one time step, no connections were allowed among the variables at time t. – A dynamic constraint was introduced that speciﬁed that if a connection was made from a variable X at time t − i to a variable Y at time t, then no other connection could be made from X to Y . This is equivalent to saying that variables only aﬀect each other temporally in a single manner, i.e. from X to Y only ever takes a set amount of time. Without a constraint such as this, in the absence of much data, multiple interactions between the same genes at diﬀerent time points could easily be inferred. – The number of slices of a possible dynamic Bayesian network was set to 9, i.e. the time slice at t and 8 slices back. Other authors show using DBNs with multiple layers to model gene regulatory networks with multiple time lags [14]. However, their methods consider arcs between arbitrary layers that are not at time t. This method is incorrect, as it means that arcs could be added between nodes without taking into account the other parents these nodes have. E.g. a local search could add an arc between Xt−2 and Xt−1 and between Yt−1 and Xt . This implies that Xt has parents Xt−1 and Yt−1 , but both arcs were added without considering the other. 4.3

Testing Methodology

In order to test the ability of DBNs to recreate the expert’s knowledge, a series of experiments were conducted. Each experiment used the data described in the preprocessing section, with the prior knowledge as described above. Also, the BDeu scoring criterion was used with an uniform structure prior. The parameters of the structure learning algorithm (ACO-E) were set as ρ = 0.3, q0 = 0.8 and β = 1. These were selected as reasonable values that should perform well in most cases [1]. It has been known for some time that the value of the equivalent sample size parameter N for the BDeu scoring function has a large eﬀect on learning structure [15,16,17]. In a sense, the value of N can be seen as a regularisation parameter - the larger the value of N , the more edges are supported in the learned graph [18]. This can be contrasted to scores like the BIC and AIC, where regularisation is implicit in the function and cannot be adjusted. Therefore, in

Using Higher-Order Dynamic Bayesian Networks to Model Periodic Data

73

learning a Bayesian network structure from data, it is important to see how the N parameter will aﬀect the learned graph. Because of this, for each condition described in Table 1, experiments were performed that varied the equivalent sample size N over 15 values ranging from 0.0001 to 500. These values were 0.0001, 0.001, 0.01, 0.1, 1, 3, 7, 10, 20, 30, 50, 80, 100, 300, 500. At each level of N , 10 experiments were performed. Also, the data from all the conditions was concatenated and experiments performed, varying N over 20 values ranging from 0.01 to 8000. These values were 0.01, 0.1, 1, 3, 7, 10, 20, 30, 50, 80, 100, 300, 500, 750, 1000, 2000, 4000, 6000, 7500, 8000. Again, at each level of N , 10 experiments were performed. For each experiment, the resulting DBN was saved for later comparison against the expert supplied knowledge. 4.4

Evaluation Criteria

Evaluating the learned DBNs was achieved by comparing them against the expert supplied knowledge. For this task, the knowledge was transformed into a DBN representing the state of the expert’s knowledge on the domain. The DBN was validated by the expert and is shown in Fig. 3. Note that although the DBN looks diﬀerent to published networks in the literature, it captures the same semantics. Please also note that nodes are not shown if they have no connection to any other node. Also, although the arcs in the network are deﬁnite, the knowledge of the domain expert was not as deﬁnite. Often the time points supplied were of the order of six hours long, i.e. instead of a deﬁnite time lag being given, the lag was bounded by two times roughly six hours apart. The eﬀect of this would be to reduce the measured accuracy of any learned network. To compare a learned network against the standard network, two measures were used: the true positive rate (TPR) and the false positive rate (FPR). These are deﬁned as TP TPR = TP + FN and FP FPR = FP + TN where TP is the number of true positives, i.e. the number of arcs computed present, that are present. For the evaluation of the networks, this was calculated in two diﬀerent ways: Time-Independent True Positive. If there was an arc from gene X to gene Y in the standard network and also an arc from gene X to gene Y in the testing network, irrespective of the time slice that X is in, then this counts as a true positive. Time-Dependent True Positive. If there was an arc from gene X to gene Y in the standard network and also an arc from gene X to gene Y in the testing network, then this counts as a time-dependent true positive. The

74

R. Daly et al.

Fig. 3. Expert developed DBN for Arabidopsis thaliana circadian clock

degree to which it is a true positive depends on the diﬀerence of time slices that the tail of each arc comes from. A diﬀerence of 0 gives a true positive of 1 and a diﬀerence of 8 gives a true positive of 0. Other values are calculated linearly between these two values. E.g. with an arc in the standard network having the tail in slice 4 and an arc in the testing network having a tail in slice 2, this leads to a diﬀerence of 2, which gives a time dependent true positive of 1 − 2/8 = 6/8. FP is the number of false positives, i.e. the number of arcs computed present that are absent. If there exists an arc from gene X to gene Y in the testing network and there does not exist an arc in the standard network, then this counts as a false positive. TN is the number of true negatives, i.e. the number of arcs computed absent that are absent. If there does not exist an arc from gene X to gene Y in the testing network and there does not exist an arc in the standard network, then this counts as a true negative. FN is the number of false negatives, i.e. the number of arcs computed absent that are present. If there does not exist an arc from gene X to gene Y in the testing network and there does exist an arc in the standard network, then this counts as a false negative. In previous work of a similar nature, evaluating the learned network proceeds in a diﬀerent manner than above [14]. Some problems in their methods include: – Firstly, as mentioned in Sect. 4.2, the method they use to add arcs can lead to incorrect conclusions about the structure of the dynamic network;

Using Higher-Order Dynamic Bayesian Networks to Model Periodic Data

75

– Secondly, they don’t take into account the time delay when evaluating the learned network; and – Thirdly, they don’t take any false positive rate into account.

5

Results and Discussion

For each condition and each level of N , the mean of the TPR and that of the FPR were taken over the 10 experiments. This was done for the TimeIndependent TP (TITP) and the Time-Dependent TP (TDTP). With these results, two diﬀerent types of graphs were plotted. The ﬁrst was the receiver operating characteristic (ROC) curve, which plots the FPR against the TPR. The second plots the FPR and TPR as a function of N . The results of these plots for the concatenated data are shown in Fig. 4. Due to reasons of space it was not possible to show the graphs for the individual conditions. To compensate, the area under the curve (AUC) statistic is given in Table 2. 5.1

Discussion of Experimental Results

The value of the equivalent sample size N has a large eﬀect on learning structure [15,16,17]. Therefore, in learning a Bayesian network structure from data, it is important to see how this parameter will aﬀect the learned graph. This is the reason why the experiments described above were conducted over various values of N . Looking ﬁrst at Figs. 4a and 4b, it can be seen that the learned DBN structures do a good job of identifying all of the connections between the genes as supplied by the domain expert. In this case, the best value of N is at 1, where the true positive rate is 1 and the false positive rate at just over 0.3. With the time dependent TPRs as seen in Figs. 4c and 4d, again the best value of N is at 1, with a time dependent TPR of around 0.8 and a false positive rate of 0.3. In this case, ‘best’ is being taken as the absolute diﬀerence between the TPR and FPR. These results show that the learned models are good at showing all the connections between the genes, but not as proﬁcient at keeping out bad connections. Table 2. Area under the ROC curve for both the time-independent and time-dependent true positive cases Condition All AT0029 AT0030 AT0031b AT0032 AT0033 AT0047

Time-Independent AUC Time-Dependent AUC Light Ratio 0.7913 0.6943 0.6597 0.8234 0.7135 0.8156 0.5200

0.6749 0.5749 0.5942 0.7046 0.5696 0.7155 0.4326

— 6:18 9:15 12:12 15:9 18:6 3:21

R. Daly et al.

1

1

0.9

0.9

0.8

0.8

0.7

0.7

True/False Positive Rate

True Positive Rate

76

0.6 0.5 0.4 0.3 0.2

0.5 0.4 0.3

0.1

0

0.1

0.2

0.3

0.4 0.5 0.6 False Positive Rate

0.7

0.8

0.9

0 −2 10

1

(a) ROC for the time-independent TPR and FPR 1

1

0.9

0.9

0.8

0.8

0.7

0.7

0.6 0.5 0.4 0.3 0.2

0

10

1

2

10 10 Equivalent Sample Size

3

10

4

10

True Positive Rate False Positive Rate

0.6 0.5 0.4 0.3 0.2

0.1 0

−1

10

(b) The time-independent TPR and FPR as a function of N

True/False Positive Rate

True Positive Rate

0.6

0.2

0.1 0

True Positive Rate False Positive Rate

0.1

0

0.1

0.2

0.3

0.4 0.5 0.6 False Positive Rate

0.7

0.8

(c) ROC for the time dependent TPR and FPR

0.9

1

0 −2 10

−1

10

0

10

1

2

10 10 Equivalent Sample Size

3

10

4

10

(d) The time dependent TPR and FPR as a function of N

Fig. 4. Results for concatenated data

Indeed, when examining a trace of the algorithm it can be seen that spurious connections between highly synchronised genes are often inserted. E.g. if X causes Y and X causes Z, then a connection between Y and Z can easily appear as well as the connections from X to Y and Z. Problems such as these often appear with a small amount of data and prior knowledge becomes increasingly important in these situations. With the results of the individual conditions, diﬀerent behaviours can be observed. The value of N for which best results are obtained diﬀers widely depending on the condition used, ranging from 1 to 100. The diﬀerent conditions also diﬀer in how good the results are. E.g. the best performance came from the results with the larger light to dark ratio, i.e. AT0031b, AT0032 and AT0033. The worst performance came from the other conditions, i.e. AT0029, AT0030 and AT0047. Results such as these are plausible, as having more light in the entrainment phase equates to higher expression levels, which are less likely to be aﬀected by noise. It should be noted that not all the expert supplied knowledge is as accurate as may seem from the standard network. Whilst the time in that network was

Using Higher-Order Dynamic Bayesian Networks to Model Periodic Data

77

given in time steps of 1.5 hours, the knowledge of the expert was often given in terms of much larger grained steps (e.g. ‘the morning’), as opposed to a deﬁnite time period. This is an artifact of the domain in question, as it is not completely understood at the present time. In turn, it has an eﬀect on the time dependent true positive values obtained from comparing two networks; with better prior knowledge a more accurate comparison could be used that bounds what the correct time lags might be for an arc. It should also be noted that the layers in the network impose a discretisation in time that might not be accurately reﬂected in the domain.

6

Conclusions

This paper asked the question on whether it is possible to use multi-layered dynamic Bayesian networks to model gene regulatory networks using gene expression data. To help answer this question new techniques were devised. Firstly, the gene expression data was preprocessed by a novel method in relation to learning gene regulatory networks; the ﬁrst diﬀerence was obtained and this diﬀerence discretised into two bins – rising or falling. Secondly, because of the lack of much previous work on learning higher-order DBNs, new evaluation criteria were deﬁned to judge how well the learned network reconstructed the expert supplied network. From the results it can be seen that it is possible, to some extent, to model both the connections between various genes and the time lags associated with these connections. Acknowledgements. We thank Adrian Thompson for his contribution to the experimental work. Kieron Edwards and Adrian Thompson performed the experimental work. John O’Neill helped develop the reference model of the circadian clock. The experimental work was funded by BBSRC awards 88/G19886 and BB/E015263/1 and by the EU FP6 award EUClock to AJM. The Centre for Systems Biology at Edinburgh (CSBE) is a Centre for Integrative Systems Biology (CISB) funded by BBSRC and EPSRC, reference BB/D019621/1. JSO, SA and AJM are funded by CSBE. SA is also partially funded by BBSRC grant BB/F015976/1.

References 1. Daly, R., Shen, Q.: Learning Bayesian network equivalence classes with ant colony optimization. Journal of Artiﬁcial Intelligence Research 35, 391–447 (2009) 2. Daly, R., Shen, Q., Aitken, S.: Learning Bayesian networks: Approaches and issues. The Knowledge Engineering Review (in press, 2009) 3. Husmeier, D.: Sensitivity and speciﬁcity of inferring genetic regulatory interactions from microarray experiments with dynamic Bayesian networks. Bioinformatics 19(17), 2271–2282 (2003) 4. Kim, S.Y., Imoto, S., Miyano, S.: Inferring gene networks from time series microarray data using dynamic Bayesian networks. Brieﬁngs in Bioinformatics 4(3), 228–235 (2003)

78

R. Daly et al.

5. Perrin, B.E., Ralaivola, L., Mazurie, A., Bottani, S., Mallet, J., d’Alché Buc, F.: Gene networks inference using dynamic Bayesian networks. Bioinformatics 19(suppl. 2), ii138–ii148 (2003) 6. Zou, M., Conzen, S.D.: A new dynamic Bayesian network (DBN) approach for identifying gene regulatory networks from time course microarray data. Bioinformatics 21(1), 71–79 (2005) 7. Geier, F., Timmer, J., Fleck, C.: Reconstructing gene-regulatory networks from time series, knock-out data, and prior knowledge. BMC Systems Biology 1, 11 (2007) 8. Edwards, K.D., Anderson, P.E., Hall, A., Salathia, N.S., Locke, J.C., Lynn, J.R., Straume, M., Smith, J.Q., Millar, A.J.: FLOWERING LOCUS C mediates natural variation in the high-temperature response of the Arabidopsis circadian clock. The Plant Cell 18, 639–650 (2006) 9. Locke, J.C.W., Kozma-Bognár, L., Gould, P.D., Fehér, B., Kevei, É., Nagy, F., Turner, M.S., Hall, A., Millar, A.J.: Experimental validation of a predicted feedback loop in the multi-oscillator clock of Arabidopsis thaliana. Molecular Systems Biology 2(59) (2006) 10. Schlitt, T., Brazma, A.: Current approaches to gene regulatory network modelling. BMC Bioinformatics 8(suppl. 6), S9 (2007) 11. McClung, C.R.: Plant circadian rhythms. The Plant Cell 18, 792–803 (2006) 12. Huang, Z., Lib, J., Su, H., Watts, G.S., Chen, H.: Large-scale regulatory network analysis from microarray data: modiﬁed Bayesian network learning and association rule mining. Decision Support Systems 43(4), 1207–1225 (2007) 13. Heckerman, D., Geiger, D., Chickering, D.M.: Learning Bayesian networks: The combination of knowledge and statistical data. Machine Learning 20(3), 197–243 (1995) 14. Xing, Z., Wu, D.: Modeling multiple time units delayed gene regulatory network using dynamic Bayesian network. In: Proceedings of the Sixth IEEE International Conference on Data Mining - Workshops (ICDMW 2006), pp. 190–195 (2006) 15. Steck, H., Jaakkola, T.S.: On the Dirichlet prior and Bayesian regularization. In: Becker, S., Thrun, S., Obermayer, K. (eds.) Advances in Neural Information Processing Systems 15 (NIPS 2002), pp. 697–704. MIT Press, Cambridge (2003) 16. Silander, T., Kontkanen, P., Myllymaki, P.: On sensitivity of the MAP Bayesian network structure to the equivalent sample size parameter. In: Proceedings of the Twenty-Third Conference on Uncertainty in Artiﬁcial Intelligence, UAI 2007 (2007) 17. Kayaalp, M., Cooper, G.F.: A Bayesian network scoring metric that is based on globally uniform parameter priors. In: Darwiche, A., Friedman, N. (eds.) Proceedings of the Eighteenth Conference on Uncertainty in Artiﬁcial Intelligence (UAI 2002), pp. 251–258. Morgan Kaufmann, San Francisco (2002) 18. Steck, H.: Learning the Bayesian network structure: Dirichlet prior vs data. In: McAllester, D.A., Myllymäki, P. (eds.) Proceedings of the Twenty-Fourth Conference on Uncertainty in Artiﬁcial Intelligence (UAI 2008), pp. 511–518. AUAI Press (2008)

Sequential Hierarchical Pattern Clustering Bassam Farran, Amirthalingam Ramanan, and Mahesan Niranjan School of Electronics and Computer Science, University of Southampton Southampton, SO17 1BJ, United Kingdom {bf06r,ar07r,mn}@ecs.soton.ac.uk

Abstract. Clustering is a widely used unsupervised data analysis technique in machine learning. However, a common requirement amongst many existing clustering methods is that all pairwise distances between patterns must be computed in advance. This makes it computationally expensive and diﬃcult to cope with large scale data used in several applications, such as in bioinformatics. In this paper we propose a novel sequential hierarchical clustering technique that initially builds a hierarchical tree from a small fraction of the entire data, while the remaining data is processed sequentially and the tree adapted constructively. Preliminary results using this approach show that the quality of the clusters obtained does not degrade while reducing the computational needs. Keywords: On-line clustering, Hierarchical clustering, Large scale data, Gene expression.

1

Introduction

Clustering as a tool in pattern analysis has a wide spectrum of applications: mining in large data warehouse environments [1], dynamic routing in optical networks [6], text classiﬁcation [19] and codebook construction for bag-of-keypoint visual scene analysis problems [20] are examples of this. For pattern recognition in bioinformatics, the popular use of clustering is in gene expression analysis [4], where expression proﬁles of genes measured across diﬀerent biological conditions are clustered. Genes that fall into the same cluster may be assumed to have common functional properties, such as acting under control of the same regulatory mechanism, or acting in tandem along a signalling pathway. Another example of clustering in bioinformatics is the analysis of protein sequences to assign putative function [5]. Repositories of protein sequences have seen massive growth in recent years [18]. As the number of sequenced proteins grows at a much faster rate than those whose structure is determined, or function characterised, automatically predicting function by clustering the sequence space is an active topic of interest. While clustering algorithms and their performance characteristics have been studied extensively over recent years, a particular property of several of the new problems, including the bioinformatics problems, is their massive scale. In other areas, too, data mining examples with a million or more data points are V. Kadirkamanathan et al. (Eds.): PRIB 2009, LNBI 5780, pp. 79–88, 2009. c Springer-Verlag Berlin Heidelberg 2009

80

B. Farran, A. Ramanan, and M. Niranjan

becoming available (e.g. KDD Cup1 ). The UniProt database of proteins [18] now consists of over six million sequences. A matrix of pairwise similarity scores of all these proteins has a ﬁle size of 2.6GB. At these scales, classical clustering algorithms such as K-means or hierarchical clustering aren’t straightforward to apply and a need for novel approaches arises. Our approach to such very large scale clustering algorithms is to study sequential and constructive algorithms, much in the spirit of the resource allocating network by Platt [14] and its variants by Kadirkamanathan et al. [7], and Molina et al. [11]. Another online approach for large scale learning (> 4M datapoints) was proposed by Farran et al. [2]. In recent work Ramanan et al. [15] developed a codebook design strategy for visual object categorization which uses a resource-allocating clustering approach. In this paper we investigate whether a formulation for hierarchical clustering by sequentially processing the data in a one-pass setting can be designed. Computational saving in such an approach will come from not having to evaluate all pairwise similarities between data items. This clearly is not possible for all the data as a measure of the scale of the distribution is required. Thus, the approach we take involves the construction of an initial tree of hierarchical clustering by processing a random subset of the data in batch mode. To a cluster structure formulated in this way, any further data may be sequentially included, adaptively changing the structure of the cluster tree, at a heavily reduced cost of similarity computations.

2

Previous Work

Clustering has been widely applied in bioinformatics in the past years. Hierarchical clustering techniques are useful for representing protein sequence family relationships [8]. Eisen et al. [4] applied a variant of the hierarchical average-linkage clustering algorithm to identify groups of co-regulated genome-wide expression patterns. Loewenstein et al. [10] applied agglomerative clustering to protein sequences that automatically builds a comprehensive evolutionary driven hierarchy of proteins from sequence alone. Frey et al. [5] applied an aﬃnity propagation clustering technique to detect putative exons comprising genes from mouse chromosomes. While they claim lower computational cost in comparison to other algorithms, they do not include the cost of pairwise similarity computations. Since this is the most expensive stage in large scale problems, the claimed advantage is exaggerated. Our focus is on reducing the computational complexity arising from this cost. El-Sonbaty et al. [17] proposed an on-line hierarchical clustering algorithm based on the single-linkage method that ﬁnds at each step the nearest k patterns with the arrival of a new pattern and updates the nearest seen k patterns so far. Finally the patterns and the nearest k patterns are sorted to construct the hierarchical dendrogram. While they claim their method is sequential, at the arrival of each data item they compute similarity to all the data seen previously. Thus there is little computational saving in their method, and it is equivalent to 1

http://www.sigkdd.org/kddcup/index.php

Sequential Hierarchical Pattern Clustering

81

re-training a new model at the arrival of new data. Contrast that with a truly sequential algorithm, where it is the model that is adapted, similar in fashion to the Kalman ﬁlter.

3

Methodology

Our algorithm for sequentially updating a hierarchical tree is shown as pseudo code in Algorithm 1. We construct an initial hierarchical tree by computing all pairwise similarities between a small subset of the data, and then passing these to the Single-Round-MC-UPGMA algorithm [10]. Following the construction of the initial tree using Single-Round-MC-UPGMA, the remaining data is sequentially processed using Algorithm 1. Whenever a new pattern (xi ) arrives for clustering, its similarity distance d to the root of the current hierarchical tree is computed. If d is greater than a predeﬁned threshold (θ), a new root is created having the current pattern (xi ) and the previous root as its children, and as a consequence the depth of the tree increases by one. The value of the new root is assigned with the arithmetic mean of all the leaf nodes. However, if d is less than θ, the nearest child of the current node is retrieved. If the distance of xi to this child node is also smaller than θ then we continue to repeat ﬁnding the closest child until either the distance to the current node is greater than θ or we reach a leaf node. In either of the two cases, xi is created as a sibling to the node under consideration, and xi ’s value is propagated up the tree to update its ancestors. Algorithm 1. Update the initial hierarchical tree in an online fashion Input: Root of the initial tree (CurNode), the new pattern (NewNode), and the novelty threshold (θ) Output: Updated hierarchical tree simdist CN ← similarity distance(CurNode, NewNode) if (simdist CN ≤ θ) then () Children ← getChildrenOf(CurNode) if (Children == NULL) then Make NewNode as a sibling of CurNode and update ancestors else {CurNode has children} nearestNode ← min (similarity distance(Children, NewNode)) if (nearestNode ≤ thresh) then CurNode ← nearestNode Go to () else Make NewNode as sibling of CurNode and update ancestors end if end if else Make NewNode as sibling of CurNode by creating a new root end if

82

B. Farran, A. Ramanan, and M. Niranjan

Fig. 1. Hierarchical tree constructed using Single-Round-MC-UPGMA on the entire capitals dataset

Fig. 2. Hierarchical tree constructed by our approach in an online fashion with the aid of an initial tree constructed by Single-Round-MC-UPGMA. The initial tree was constructed with the ﬁrst 15 capitals in the dataset. Capitals Stockholm and Washington (depicted in dotted hexagons) were sequentially inserted using Algorithm 1.

Fig. 3. Hierarchical tree constructed by the approach proposed in [17] on the entire capitals dataset

By adjusting θ, one can obtain diﬀerent numbers of clusters at diﬀerent levels of granularity. This threshold is seen as a hyperparameter in the algorithm and can be tuned in diﬀerent ways. The speciﬁc way in which we set this is described in section 4. The leaf nodes of the hierarchical tree are the input patterns, and

Sequential Hierarchical Pattern Clustering

83

Fig. 4. Hierarchical tree constructed by our approach with the aid of an initial tree constructed using the method in [17] on the capitals dataset

each intermediate node (up to and including the root) contains the arithmetic mean of every leaf node it represents. Because of this, we need not traverse the entire tree during the update process. We use the capitals dataset2 for illustration and evaluation purposes of the proposed algorithm, as the structure of the clusters is known. We numbered the capital cities in the following way: Tallinn (1), Beijing (2), Berlin (3), Buenos Aires (4), Cairo (5), Canberra (6), Cape Town (7), Helsinki (8), London (9), Moscow (10), Ottawa (11), Paris (12), Riga (13), Rome (14), Singapore (15), Stockholm (16), Washington (17). In Fig. 1, we show the tree constructed using Single-Round-MC-UPGMA using the enire capitals data set, which is identical to the tree depicted on the data’s website. Fig. 2 shows how our approach inserts the last two points into the tree, given that the ﬁrst 15 points were used for the construction of the initial tree. Cutting at the second level gives us the exact same 4 clusters obtained by cutting the tree in Fig. 1 at the same level. To illustrate the importance of having a good initial tree, we use El-Sonbaty et al. [17]’s method on the entire capitals data set (shown in Fig. 3). Fig. 4 shows how Algorithm 1 inserts the last 2 points, given that the ﬁrst 15 points were used for the initial tree using El-Sonbaty et al.’s method. Even though Algorithm 1 inserted the last 2 points correctly with respect to Fig. 3, the same 4 clusters obtained by the trees in Fig.1 and Fig.2 are not attainable due to the incorrect initial tree.

4

Experiments and Results

Evaluation of clustering algorithms is not straightforward. Unlike in supervised learning problems, there is no ground truth information available to verify whether the clusters obtained are correct or not. To evaluate our algorithm we used two bioinformatic datasets. The ﬁrst is Eisen et al. [4]’s gene expression clusters consisting of ten clusters formed by their clustering algorithm. We make 2

http://www.quretec.com/HappieClust/

84

B. Farran, A. Ramanan, and M. Niranjan

Fig. 5. Hierarchical tree constructed by the Single-Round-MC-UPGMA scheme on Eisen’s data clusters labelled as B and D [4]. The tree was constructed by using the whole data of the selected two clusters. The dendrogram was cut at the root node (shown in dotted lines) to obtain two clusters.

Fig. 6. Tree constructed by the proposed approach with the aid of an initial tree constructed by Single-Round-MC-UPGMA on Eisen’s clusters labelled as B and D [4]. The initial tree was constructed with 20% of the data and the rest was clustered using Algorithm 1. The dendrogram was cut at the root node (shown in dotted lines) to obtain two clusters.

the weak assumption that the clusters published by these authors are perfect associations and quantify how close our approach gets to this solution. The second dataset is from a protein fold classiﬁcation problem, constructed by Ding & Dubchak [3] on a subset of the Structural Classiﬁcation of Proteins (SCOP) [12] database. This is essentially a classiﬁcation problem, but we apply clustering to it (without using the class labels) and evaluate how well the resulting cluster membership matches the clusters returned by Single-Round-MC-UPGMA applied on the entire subset. We combined both training and testing sets provided in [3]. To quantify the quality of clusters, we use the F1 measure, widely used in information retrieval literature, and the results given in Table 1 are the average of the 10 runs where we randomised the initial subset and the order of presentation of the remaining data. precision =

TP TP + FP

F1 =

recall =

2 × precision × recall precision + recall

TP TP + FN

Sequential Hierarchical Pattern Clustering

85

Fig. 7. Tree constructed by Single-Round-MC-UPGMA on the two selected folds Alpha: four-helical up-and-down bundle (depicted as ﬁlled circles) and Beta: ConA-like lectins (depicted as diamonds) of the SCOP data subset [3]. We used the whole 28 data points for the construction of this tree.

Fig. 8. Tree constructed by the proposed approach with the aid of an initial tree constructed by Single-Round-MC-UPGMA on the two selected folds Alpha: four-helical up-and-down bundle (depicted as ﬁlled circles) and Beta: ConA-like lectins (depicted as diamonds) of the SCOP data subset [3]. We used 20 out of 28 data for the construction of initial tree and used the rest in an online manner. Table 1. Preliminary results of the hierarchical clustering performed on a subset of SCOP and Eisen’s data. (1) Beta: ConA-like lectins, (2) Alpha: Four-helical up-anddown bundle, (3) Beta: Immunoglobulin-like beta-sandwich, and (4) A/B: beta/alpha (TIM)-barrel. Dataset SCOP (1) & (2) SCOP (3) & (4) Eisen (B & D) Eisen (C & I) Eisen (C, B & I) Eisen (C, B, D & I)

No. of data 28 151 20 104 113 124

No. of data used F1-measure or the initial tree 20 1.0 100 0.9921 ± 0.0086 4 1.0 26 1.0 25 1.0 30 0.9247 ± 0.0652

The threshold θ used in this paper was determined from the data sample used to construct the initial tree. θ was set as the sum of the pairwise Euclidean distances between the patterns in this data sample.

86

B. Farran, A. Ramanan, and M. Niranjan

(a)

(b)

Fig. 9. Tree constructed by our approach on the selected clusters of Eisen’s data (a) Clusters B, C, D and I; 30 out of 124 data were used for the construction of the initial tree. The tree was cut at the level indicated by the dotted line to yield four perfect clusters. (b) Clusters B, C, and I; 25 out of 113 data were used for the construction of the intial tree. The tree was cut at the level indicated by dotted line to yield three perfect clusters.

Fig. 5 shows the tree constructed by the Single-Round-MC-UPGMA scheme on the Eisen’s clusters labelled as B and D in [4]. Fig. 6 shows the tree obtained by using 4 points for the initial tree, and the remaining 16 points inserted using our approach. Cutting at depth 1 to obtain 2 clusters shows that we get the exact same 2 clusters as in Fig. 5. However when we used our approach on the protein fold data of Ding & Dubchak [3] (see Fig. 8), we obtained a better separation than when the entire data was used with Single-Round-MC-UPGMA (see Fig. 7). Finally, Fig. 9 shows the trees obtained by using Eisen’s data clusters (C,B,I and C,B,D, & I), and the cuts that return the desired clusters.

5

Dealing with Categorical or Symbolic Patterns

In the previous section, the illustration of the proposed algorithm was restricted to the Euclidean space. Here, we discuss the capability of our algorithm to handle categorical or symbolic patterns. Instead of updating the parent nodes using the arithmetic means of their leaf nodes, the most informative child node can be selected to act as a parent. However, for simplicity in this paper, we choose a random child to act as the parent node in Algorithm 1. Also, the measure of similarity used depends on the application of interest, and is not limited to numerical data. For example, if we are clustering protein sequences, then Smith-Waterman local alignment [16] or Needleman-Wunsch global alignment [13] measures can be used instead of Euclidean distances. For immediate comparison purposes with the previously shown cluster trees in section 3 (Capitals data and the selected subsets of Eisen’s data), we used the Euclidean distances with the selective node approach. Our preliminary results gave us exactly the same clusters as in the Euclidean case, and hence the same F1-measure.

Sequential Hierarchical Pattern Clustering

6

87

Conclusion and Future Work

In this paper we present an algorithm for on-line hierarchical clustering. The approach depends on the construction of an initial clustering stage using a random subset of the data. This establishes a scale structure of the input space. Subsequent data can be processed sequentially and the tree adapted constructively. We have shown that on small bioinformatics problems such an approach does not degrade the quality of the clusters obtained while saving computational cost. The proposed technique could be signiﬁcantly improved with an appropriate choice of the novelty threshold (θ). θ can be better estimated by taking into account the inter-cluster and/or intra-cluster information of the initial tree. This can be subsequently updated after the insertion of a newly arrived pattern. Another way of better estimating θ might be to use local thresholds associated with each parent or level of the tree, instead of a global threshold. The greatest beneﬁt of the proposed technique lies in its application on very-large datasets such as UniRef90 proteins [18]. Acknowledgments. We thank the anonymous reviewers for their useful comments on this paper. AR is partially supported by a grant from the School of Electronics and Computer Science, University of Southampton, United Kingdom, and University of Jaﬀna, Sri Lanka, under the IRQUE Project funded by the World Bank.

References 1. Achtert, E., Bohm, C., Kriegel, H.-P., Kr¨ oger, P.: Online Hierarchical Clustering in a Data Warehouse Environment Data Mining. In: Proceedings of the Fifth IEEE International Conference on Data Mining, pp. 10–17 (2005) 2. Farran, B., Saunders, C.: Voted Spheres: An online Fast Approach to Large Scale Learning. In: IEEE International Symposium on Mining and Web (2009) 3. Ding, C.H., Dubchak, I.: Multi-class protein fold recognition using support vector machines and neural networks. Bioinformatics 17(4), 349–358 (2001) 4. Eisen, M.B., Spellman, P.T., Brown, P.O., Botstein, D.: Cluster analysis and display of genome-wide expression patterns. Proceedings of the National Academy of Sciences USA 95(25), 14863–14868 (1998) 5. Frey, B.J., Dueck, D.: Clustering by Passing Messages between Data Points. Science AAAS 315, 972–976 (2007) 6. Hasan, M., Jue, J.: Online Clustering for Hierarchical WDM Networks. In: IEEE/OSA Conference on Optical Fiber Communication, San Diego, CA, pp. 1–3 (2008) 7. Kadirkamanathan, V., Niranjan, M.: A Function Estimation Approach to Sequential Learning with Neural Networks. Neural Computation 5, 954–975 (1993) 8. Kaplan, N., Friedlich, M., Fromer, M., Linial, M.: A functional hierarchical organization of the protein sequence space. BMC Bioinformatics 5, 196 (2004) 9. Kull, M., Vilo, J.: Fast approximate hierarchical clustering using similarity heuristics. BioData Mining 1, 9 (2008)

88

B. Farran, A. Ramanan, and M. Niranjan

10. Loewenstein, Y., Portugaly, E., Fromer, M., Linial, M.: Eﬃcient algorithms for accurate hierarchical clustering of huge datasets: tackling the entire protein space. Bioinformatics 24(13), 41–49 (2008) 11. Molina, C., Niranjan, M.: Pruning with replacement on limited resource allocating networks by f-projections. Neural Computation 8(4), 855–868 (1996) 12. Lo Conte, L., Ailey, B., Hubbard, T.J., Brenner, S.E., Murzin, A.G., Chothia, C.: SCOP: a Structural Classiﬁcation Of Proteins database. Nucleic Acids Research 28(1), 257–259 (2000) 13. Needleman, S.B., Wunsch, C.D.: A General Method Applicable to the Search for Similarities in the Amino Acid Sequence of two Proteins. Journal of Molecular Biology 48(3), 443–453 (1970) 14. Platt, J.C.: A Resource-Allocating Network for Function Interpolation. Neural Computation 3, 213–225 (1991) 15. Ramanan, A., Niranjan, M.: Designing a Resource-Allocating Discriminant Codebook for Visual Object Recognition. Neural Computation (2009) (under review) 16. Smith, T.F., Waterman, M.S.: Identiﬁcation of Common Molecular Subsequences. Journal of Molecular Biology 147, 195–197 (1981) 17. El-Sonbaty, Y., Ismail, M.A.: On-line hierarchical clustering. Pattern Recognition Letters 19, 1285–1291 (1998) 18. Wu, C.H., Apweiler, R., Bairoch, A., Natale, D.A., Barker, W.C., Boeckmann, B., Ferro, S., Gasteiger, E., Huang, H., Lopez, R., et al.: The Universal Protein Resource (UniProt): an expanding universe of protein information. Nucleic Acids Research 34, D187–D191 (2006) 19. Zhao, Y., Karypis, G., Fayyad, U.: Hierarchical Clustering Algorithms for Document Datasets. Data Mining and Knowledge Discovery 10(2), 141–168 (2005) 20. Zhang, J., Marszalek, M., Lazebnik, S., Schmid, C.: Local Features and Kernels for Classiﬁcation of Texture and Object Categories: A Comprehensive Study. International Journal of Computer Vision 73, 213–238 (2007)

Syntactic Pattern Recognition Using Finite Inductive Strings Paul Fisher1, Howard Fisher2, Jinsuk Baek1, and Cleopas Angaye3 1

Department of Computer Science, Winston-Salem State University, Winston-Salem, North Carolina, United States 2 Fisher Company, Salt Lake City, Utah, United States 3 National Information Technology Development Agency, Nigeria, Africa

Abstract. A syntactic pattern recognition technique is described based upon a mathematical principle associated with finite sequences of symbols. The technique allows for fast recognition of patterns within strings, including the ability to recognize expected symbols that are close to the desired symbols, and mutations as well as both local and global substring matching. This allowance of deviation permits sequences to be subject to error and still be recognized. Some examples are provided illustrating the technique. Keywords: Pattern recognition, finite inductive sequences, syntactic pattern recognition, genome recognition.

1 Introduction We provide a description including the properties of finite inductive sequences, and two programs which utilize this formulation to process strings of unknown patterns for recognition of subsequences of symbols that occur within other sequences. There have been and continue to be syntax pattern recognition systems for application within the biomedical field, as well as utilization in many other fields. The basis for this work began many years ago with the consideration of sequences of control structures for autonomous robots. Needing a concise and simple way of representing learned activities, and using that learned experience to drive the next steps, we proposed the concept of Finite Inductive Sequences. When experienced as next activities, the results would provide some measure of new data that would be incorporated into the ‘learned’ model. So in either stationary models where the rules do not evolve, thus less interesting, or non-stationary models where the rules would be altered to adapt to the changing conditions, provided the impetus to find a solution [3]. The need for more efficient pattern detection algorithms is especially urgent in bioinformatics where genome-wide association studies (GWASs) are becoming increasingly popular in determining the relationship between genetic variations such as single nucleotide polymorphisms (SNPs) and copy-number variations (CVNs) and disease. Current software tools used to analyze these large datasets are computationally intensive. The strength of the finite inductive method is in its ability to determine small variations over a large data set. We believe that this algorithm is ideally suited to improve the speed of GWASs. We note that many tools exist for processing the V. Kadirkamanathan et al. (Eds.): PRIB 2009, LNBI 5780, pp. 89–101, 2009. © Springer-Verlag Berlin Heidelberg 2009

90

P. Fisher et al.

genome sequences, and new applications using graphical display, color, 3D and other extensions have and are being developed [2]. At issue is both speed and accuracy with sufficient flexibility to deal with insertions and deletions in the sequence comparison processes. This has given birth to extensions of the dynamic programming approach first developed for the longest common subsequence (LCS) problem using modifications to the Smith-Waterman algorithm using both hardware and software systems [1], [5], [6], [7], [8], [10]. These systems utilize algorithms ranging over neural nets, Bayesian statistics, tree structures and many other approaches incorporating video algorithms, visualization, data mining, etc., and each has been used in the analysis and recognition process for both protein and amino acid sequences. One only need look at any conference or journal to find articles detailing such research and development activities. All implemented systems must incorporate work from many aspects of science to accomplish the continued analysis of the data which arrives at a rate estimated to be two-million DNA bases each day [8]. We take some space to describe this algorithm which we propose for the purpose of subsequence processing, and indicate some extensions this approach provides. We also provide a measure of the storage and processing complexity of this approach, and lastly, we provide some small examples illustrating the application of FI to the genomic problems.

2 Finite Inductive Sequences In the next sections we describe first a sequence of symbols (‘string’) which is finitely inductive. Following, that we provide a description of the algorithm for building a set of rules from exemplar sequences together with their associated storage structure called a ruling. Lastly, we show how these rules can be used to recognize patterns in similar types of sequences. Here similar implies that all sequences are formed from identical alphabets. 2.1 Definition of Finite Inductive Sequences In this section, we will describe a particular kind of sequence composed of symbols coming from some alphabet. Consider a sequence of symbols made up from some alphabet such as the representation of the genome with alphabet A, C, G and T. The technique which we shall describe resembles neural net capabilities [9] and emulates Markov models, but with some important differences to both approaches. Consider one or more sequences of symbols, and we can be process them as a single entity or as multiple strings representing data from several sources to be processed in parallel. These sequences we have termed Finitely Inductive Sequences (FI), and such sequences form a countable collection of sequences of finite length. (We will not address infinite sequences, although there is a class of such infinite sequences that are amenable to this approach.) FI has a formal background in mathematics and lies between a mathematical characterization, on the one hand, and a statistical characterization, on the other. The idea is generally to deal with sequences or subsequences observed previously—such as might have been seen previously or learned through exemplar sequences—then such

Syntactic Pattern Recognition Using Finite Inductive Strings

91

sequences can be used to recognize new, similar subsequences from other strings that one desires to process. Unlike neural nets, it is additionally possible to dynamically modify the data structure formed from exemplar sequences so that new experience can be acquired in near real time from new data sequences. The primary data structure of FI, called a ruling, is a finite state machine that can use a short driving sequence to generate another sequence that may be much longer, or it can be used in reverse to produce a residual short sequence from a presented long sequence. The residual is the ‘left over’ symbols from the string that are not part of the experience base encapsulated within the ruling. In a sense, the size of the residual is a measure of the common experience between an unknown event sequence and the string(s) placed in the ruling. The ruling embodies patterns of interest, and acts as a filter through which unknown strings can be filtered looking for identical or similar subsequences found within the ruling and the unknown string. Finite sequences of symbols are said to be finitely inductive (FI) over a finite alphabet if the choice of any symbol at any particular position in a sequence depends immediately upon only the choices of symbols at the previous n points (the similarity to Markov models except this is generally deterministic and not probablistic.) Given an FI sequence (deterministic), an implicant is a pair (w, p) consisting of a word w over the alphabet and a symbol p of the alphabet such that w occurs at least once as a substring of the sequence; and whenever w occurs as a sub-string of the sequence there is a succeeding entry and it is p. For non-deterministic implicants, there might be two or more values for p with a probability associated with each such value. The antecedent is w while p is called the consequent. A reduced form implicant has no proper segment of its antecedent in another implicant. We note the following: a. b. c.

Every finite sequence is finitely inductive. For any finitely inductive sequence, the inductive base is the maximal length of the antecedents when considering all of its reduced form implicants. If an FI sequence has inductive base n and an alphabet of k symbols, then kn is an upper bound for the number of its reduced form implicants.

2.2 Description of the Storage Structure for Finite Inductive Sequences Figure 1 is an example of function table (called a Ruling) resulting from an FI sequence(s) meeting the requirement for an optimal storage system for streams of symbols. The system consists of a structure with k levels. The rules are defined from the exemplar patterns according to a push up formulation of the strings. The inductive base in Figure 1 is as defined above and is equivalent to the longest antecedent in reduced form. The rules are formulated much like those characterizing a Markov process. (We point out another difference between Markov Models and FI sequences is that within the theory of FI sequences; the order of the sequence, i.e., its inductive base, can be altered to any a priori value desired, unlike the order of the Markov Model.) Given some number n, the inductive base, and a sequence of symbols, then we can say that the previous n characters uniquely determine the next character in the string. As the inductive base is reduced, the number of levels will increase. The number of rules remains nearly invariant as the number of levels change. The last three elements in Figure 1, the rules, starting sequence, and the shift register will be explained by an example.

92

P. Fisher et al. Level k Structure Rules Starting Sequence Shift Register Appeal

1

2

Inductive Base

Output

Level 1 Structure Appeal

Rules Starting Sequence

Output

Shift Register 1

2

Inductive Base Output

Fig. 1. The FI Ruling Structure for Representing a Sequence(s) in k-Levels

Consider the following string to build the structure of Figure 1. We note that the starting symbol ‘S’ is not part of the sequence, but it allows a simpler representation. S2446664882443

(1)

This sequence comes from an alphabet of eight symbols: 1, 2, 3, 4, 5, 6, 7, 8. It is finite in length, and therefore is finitely inductive. We can form rules for this sequence as a function of the inductive base. However, since the inductive base must be large enough to uniquely guarantee the next symbol, an inductive base of two or less would be unsuitable since the occurrence of ‘66’ does not uniquely specify the next symbol (‘66’ precedes both a ‘6’ and an ‘4’). We can easily extend this representation to the non-deterministic case and to a probabilistic case by specifying that the sequence 66 specifies both a ‘6’ and a ‘4’, and if we associate a probability with the two alternatives, then we have the probabilistic representation. We illustrate the table that is generated for the reduced form rules. The first rule in (2) states that each time we see the antecedent ‘S’ then a ‘2’ follows, without ambiguity, as the next symbol in the string. S Æ 2, 466 Æ 6,

S2 Æ 4, 24 Æ 4, S244 Æ 6, 8244 Æ 3 666 Æ 4, 64 Æ 8,

48 Æ 8

46 Æ 6,

88 Æ 2, 82 Æ 4

(2)

From (2) the inductive base is 4 (the largest of the antecedent values). Some of the rules are inductive base 1 and 2; when this variability occurs, we can reduce the inductive base (the 3) to any value less than the 3. Suppose we wish to use an inductive

Syntactic Pattern Recognition Using Finite Inductive Strings

93

base of 2, we would get the FI structure of Figure 1, and the process (called Factoring) for deriving this new FI ruling structure is shown in Example 1. Example 1. Simple String Factoring and Regeneration For the string: S2446664882443 using the reduced rules (2), define the ruling as follows: Step 1: Keep all rules shown in (2) with inductive base 2 or less. For everything else push the consequent symbol up out of the string to the next level, which we designate Level 1. This results in the following pattern: Level 1 Level 0

S 6 64 3 S2446664882443

The Rules for Level 0 are those representing all symbols not pushed up: SÆ2

S2Æ 4

24 Æ 4

46 Æ 6 64 Æ 8

48 Æ 8

88 Æ 2

82 Æ 4

Step 2: For the string of Level 1, we find the reduced rules. These are: SÆ6

S6 Æ 6

66 Æ 4

4Æ3

This process has produced rules of inductive base less than or equal to 2, in a ruling of two levels, and since there are no symbols that need to be processed further, we stop the process. We also point out the symbols in Level 1 are called the residual from Level 0. As noted, there is no residual from Level 1. We also point out that every FI string of finite length produces a ruling where the last level has no residual, when that string is processed independently of any other string. The inductive base for the sequence is 2. The starting sequences are loaded in the shift registers, and the system is ready to regenerate the original sequence. This is done in the following steps: Step 1: The S in the shift register of Level 0 is matched against the antecedents of the rules in that level. In this case the antecedent of the string S Æ 2 matches the character in the Shift Register. Step 2: Push the consequent ‘2’ into the right side of the shift register. Step 3: Since the shift register now contains ‘S2’, see whether this string or the symbol ‘2’ (top of the shift register) matches any antecedent in this level. There is a match, so the consequent ‘4’ is placed in the top of the Shift Register, and the ‘S’ is output. Step 4: Again the Shift Register content of ‘24’ is matched to the antecedents of Level 0. A match is found, and the consequent ‘4’ is pushed into the shift register while the ‘2’ is pushed out to the output. Step 5: The Level 0 Shift Register now contains ‘44’, and this does not match any of the antecedents in Level 0, so an appeal is made to Level 1. The current status is now: Level 1: shift register: S Level 0: shift register: 44 and the output S2 has been generated from level 1.

94

P. Fisher et al.

Step 6: In Level 1, the contents of the Shift Register are matched, and the consequent ‘6’ is placed in the shift register, and the ‘6’ is pushed down to Level 0 and placed in the top of the Shift Register for that level. Now we have the following configuration: Level 1: Shift Register: ‘6’ Level 0: Shift Register: ‘46’ Output: ‘S24’ Step 7: Continue with processing in Level 0 until no processing can continue, and then output the Shift Register of Level 0 to complete the original sequence regeneration. From this simple example, we see the FI ruling structure: how the rules are fashioned and processed using Figure 1, and how the string is regenerated. We have seen that the inductive base was reduced to a smaller value for the example. Non FI strings are those that are pseudo random, and in this case the antecedents of all rules will have equal length, or they grow at the same rate as the string being processsed. Examples are strings that have no pattern, and these include such things as the expansion of PI or other non-repeating, non terminating sequences (transcendental numbers). We have looked [3] at the occurrence of non-FI strings in the case where binary symbols are used as the alphabet, and their occurrence is rare. We have also shown [3] when the alphabet is richer than binary, then the occurrence of non-FI sequences is even less frequent. 2.3 Application of a Ruling in Recognizing an Unknown Pattern The next example uses the Ruling shown in Table 1 resulting from the simultaneous factoring of the four strings. Each string uses the same alphabet, which is not a hard requirement. The next example uses the Ruling shown in Table 1 for the factoring of the four event strings. We also state the obvious: the more data one has about a pattern, the more accurately one can identify that pattern in unknown sequences. In order to show how rulings apply in terms of recognition of subsequences, we now provide some patterns. In this case, we are no longer concerned about regenerating the original sequences from the rulings. Example 2. Ruling Built from Four Sequences (Simultaneous Factoring) Suppose there are four strings designated a, b, c and d. We have added the start symbol ‘S’ to each string for convenience. Consider the following four strings: a : S2446664882443

b : S66666254322187 d : S7765442218

c : S666662225442217

Since we have started each string with the symbol S, this will shorten the residuals from each string. As can be seen from the four strings the starting implicants are: S Æ 2; S Æ 6; S Æ 7; however, since this is simultaneous factoring, and these implicants are contradictory, they are not acceptable in the deterministic case. We require in simultaneous factoring that the implicants be consistent intra- as well as inter-string. The implicants S2 Æ 4; S6 Æ 6; S7 Æ 7 are consistent for all four

Syntactic Pattern Recognition Using Finite Inductive Strings

95

sequences. The final residuals shown in (3) for each of the strings are those symbols that remain due to intra- or inter-string conflicts. The resulting table of implicants augmented with their source string, for each of the levels in the FI ruling table is given in Table 1 with the inductive base held to 2 a: s Æ 2

b: s 6 Æ 5

c: s 6 Æ 2

d: s Æ 7

(3)

Table 1. FI Ruling Table for the Strings a, b, c and d with an Inductive Base of 2

Level 1

Level 2

Level 3

Level 4

Level 5

S2 Æ 4 (a)

S2 Æ 6 (a)

S2 Æ 4 (a)

S6 Æ 6 (b, c)

62 Æ 2 (c)

S6 Æ 6 (b, c)

S6 Æ 6 (b, c)

S6 Æ 2 (b, c)

22 Æ 5 (c)

S7 Æ 7 (d)

S6 Æ 6 (b, c) S7 Æ 4 (d)

S7 Æ 8 (d)

65 Æ 3 (b)

82 Æ 4 (a)

32 Æ 1 (b)

3 Æ 8 (b)

25 Æ 4 (c)

32 Æ 2 (b)

42 Æ 1 (d, c)

54 Æ 7 (c)

42 Æ 2 (d, c)

53 Æ 2 (b)

38 Æ 7 (b)

24 Æ 4 (a)

64 Æ 3 (a)

64 Æ 8 (a)

54 Æ 2 (c)

5 Æ 4 (b, c, d)

74 Æ 2 (d)

46 Æ 6 (a)

26 Æ 6 (a)

76 Æ 5 (d) 77 Æ 6 (d) 48 Æ 8 (a) 88 Æ 2 (a)

The processing that takes place in this example shows how the incoming string is matched with the known data as represented by the implicants contained within the ruling. If the antecedents match, then we can delete the consequent symbol at the end of the process for this level from the unknown string. We can match the position of this deleted symbol for later identification. The residual is the string remaining after all levels have removed the consequents matching the antecedents for the appropriate implicants in each level. The algorithm used in this matching process is called Following. As was indicated above, the residual is important as its length is a percentage of the original string suggesting an overall measure of similarity between the ruling and the string. Residuals also represent a measure of similarity based upon differences between multiple strings with the ruling acting as the adjudicator. These residuals in a sense represent the entropy of the string when the universe of experience or knowledge is contained within the ruling.

96

P. Fisher et al.

Example 3. Recognition of Patterns in a New Sequence Consider the following two strings formed from the new sampled events. We will designate them by e and f. The strings are: e: s666662278122345

f: s666662543443244666488244187

(4)

The Following results are shown in Table 2, when the Ruling of Table 1 is applied to the two strings. In addition, the implicants used from Table 1 have their originating string annotation, and this annotation is copied under the bottom row of Table 2 in the line called ‘Source’. In order to represent implicants with multiple sources, we encode the sources as: (a) = 1, (b) = 2, (c) = 3, (d) = 4, (b, c) = 5, (d, c) = 6, (b, c, d) = 7. Table 2. Following Applied to the Two Patterns e and f Respectively Using Table 1

Following Level For String e 5: s 6 2278122345 4: s 6 62278122345 3: s 6 662278122345 2: s 6 6 6 6 2 2 7 8 1 2 2 3 4 5 1: s 6 6 6 6 6 2 2 7 8 1 2 2 3 4 5 Source 5555 3 Level For String f 5: s 6 25 344324 6664 221 4: s 6 625 344324 6664 221 3: s 6 6625 344324 6664 221 2: s 6 6 6 6 2 5 3 4 4 3 2 4 6 6 6 4 221 1: s 6 6 6 6 6 2 5 4 3 4 4 3 2 4 4 6 6 6 4 8 8 2 2 2 1 Source 5 5 5 5 7 1 111 From the results shown in Table 2 of the Following over e and f we conclude: • • • • • •

Neither string matches very well the data contained in the ruling. Best matches come from a combination of b and c for string e. The best match for string f comes initially from b or c, and then evolves into the string of a. This would indicate in both sequences e and f, that the data in the ruling is not sufficiently rich to represent the sequence, so this data should be added to the ruling. Modification of the ruling on the fly can be accommodated, and depending upon storage space, one can add strings at will. In order to control growth, the implicants can be eliminated based upon a usage criteria. A counter can be added to each rule, and when the counter drops beneath some usage limit, the rule can be deleted, or the sequence containing the rule can be deleted.

Syntactic Pattern Recognition Using Finite Inductive Strings

•

97

When a symbol has not been recognized, but the antecedent was matched, then that symbol represents a possible single symbol mutation. In this case, the system will substitute the consequent in the string, and mark the substitution, and continue recognition with the altered symbol. This is called Replacement Following. For sequence substitutions, the sequence will be processed with the ruling, and the substituted sequence will not be recognized, and except for accidental similarities, replacement will not solve the similarity problem. Thus there will be a block of symbols that will remain in the residual. These symbols can be used to form a new ruling, linked to the previous ruling, so that over time, an assemblage of rulings will be formed that represent the evolution of the original string.

•

3 Finite Inductive Sequences and Biometric Pattern Examples There are several applications that are of interest using the model that we have proposed. The first just deals with recognition of a gene from one source to another. If we extend the recognition technique described in Section 2.3, we can deal with the mutation issues within a gene. Besides the Exact Following of Section 2.3, there are several more extensions. The first allows recognition of genes that have changed from the exemplars stored in the Ruling by mutations of deletion, duplication, inversion, insertion or translocation. Consider the issue of insertion and the Skip Following process. In this process the user defines the size of a skip region, and in the gene, this involves a region that potentially has been inserted and is not in the exemplar set of the ruling. We now provide some examples to show how FI can be used in the genomic pattern matching world when various kinds of mutations are allowed. Example 4. Mutation by insertion. Consider sequence (5) as an exemplar sequence for the ruling content. GTGAGTGGTCTTAGGTGAGTCAGTGCAG

(5)

The ruling is given in Table 3 where the levels are divided by the horizontal lines, and the Order column is the order in which the implicants were formed in processing (5). Table 3. Ruling for Sequence (5)

Antecedent

S G

G T G G G

S G T

S G T G A

A G G T C T

G G T C T T

Consequent Order G 1 T 2 G 3 A 4 G 5, 14, 19, 23, 28 T 6, 20 T 9, 16 C 10 T 11 T 12 A 13

T A G G G A G C C A

A G T T A G G G S G

G T G C G T C S C G G

G G A A T G A G C G C

15 17 18 22 24 25 27 7 26 8 21

98

P. Fisher et al.

Now consider the new sequence and their associated positions: G T G A G T G G T C c g t a t T T A G G T G A G (6) Loc: 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 20 1 2 3 4 We have defined a variable called Skip in the system to permit ignoring some number of symbols in the recognition (following) process. Processing the string of (6) with the ruling of Table 3, we get the results shown in (7) when we process left to right: Residual: g t a t T T Lev 1: S G G g t a t T T Lev 0: S G T G A G T G G T C c g t a t T T A G... (7) Rule: 1 2 3 4 5 6 7 8 9 10 21 13 14... We now know that there is a difference between the ruling and the unknown sequence because of the residual. If we set the skip value to 6 for example, and starting to match from the first symbol that occurs in the residual (being the g) plus the skip value gives the next starting symbol A in position 18. Looking for a rule in Table 3 after 10, maintaining rule order (another parameter setting in the system), we find for the A, rules 4, 13, 18, and 27. All rules are unacceptable except rule 13 due to the antecedent TT and its relative order. Since the skip value is a guess only, we need to see if backing up prior to location 18 can trim some off of the mutation region. We also note that rule 21 is not in the correct place, so removing 21, and backing up from location 18 we get the final insertion mutation candidate as: c g t a t The skip value being arbitrary for this example can be set by the user to an expected value, or just set to some maximal value. The only difference in reality is the length of time it requires to search backwards from the skipped location. If rule order is main-tained, then the processing will be much faster, since all previous rules used up to the first location where a nucleotide appears in the residual will not need to be considered. Example 5. SNP Processing Suppose we have an exemplar sequence around a SNP loci where an allele exists as shown in Table 4. The problem is to determine the frequency of SNPs over a set of loci to allow likelihood patterns for determining both missing nucleotides as well as determining the predominate nucleotide in a chromosome. Table 4. SNP Patterns from Individuals (Rows) and from a Chromosome (Column)

85 91 93 94 83

M CC CC CG CG CC

22 29 34 48 55

M CG CC CG CC CG

45 41 57 61 27

M CG CC CC CG CC

Syntactic Pattern Recognition Using Finite Inductive Strings

99

The surrounding symbols to these alleles are identical in the chromosome and suppose that we define the precursor symbols to be CACC with the two possible sequences for M to be CACCC or CACCG with a frequency of eight and seven. If one doesn’t know where the SNP happens to reside, then one can follow one gene in one chromosome from the ruling of a gene in the exemplar chromosome with the SKIP value set to one, or one can use Replacement Following and keep track of the loci and type of replacements done. Consider the rules derived for the string CACCC and CACCG. The rules, permitting non-determinism are shown in (8). The current, internal form of the Ruling is a Finite State Machine (FSM), and to add non-determinism, we simply add another state for the additional consequent, and a counter to record the number of times each rule was used. For the last two rules of (8), a single rule of the form CC Æ C(8)/G(7) would suffice in the FSM where the 8 and 7 are the frequency counts for occurrences. S Æ C, SC Æ A, A Æ C, AC Æ C, CC Æ C(8), CC Æ G(7)

(8)

For Replacement following, suppose the exemplar sequence was the first five rules implying that the symbol pair CC always produced a C. With replacement suppose we found the string CACCG, the result would be to delete the G and replace it with the C. If counting is turned on, then the number for each of the replaced symbols would be counted. Additionally a position marker can be output to identify to the location of each replacement. Example 6. Trace for Following Suppose we have the ruling of Table 3, and the unknown sequence shown in (9). Here the sequence is only similar to that used as the exemplar in Table 3. One of the things which the system will do is provide a trace of the implicant usage. SGTTGGCCCGTGGTTTTA G G T G AATGGG TGCA G 12

9

S TGGC CCGTGG

13 14 15 16 17 18 TTT

16 ATGGG

21 26

(9)

27 28 (9a) GC

(9b) 21 (9c)

(9a) represents Level 0 implicant application to the sequence of (9), and (9b) represents the residual from Level 0. (9b) and (9c) represent the processing from Level 1. The final residual is given in (10). S TGG

CGTGG

TTT

ATGGG

G

(10)

The residual contains seventeen symbols implying that fifteen symbols were deleted. The more interesting observation is the subsequence of symbols beginning with rule number 13 and running consecutively through rule number 18 in a cluster with no skipped nucleotides indicating the presence of a subsequence of potential interest.

100

P. Fisher et al.

4 Conclusion The need for exact matching in a fast algorithm is important for continued research and development for genomic understanding and disease characterization. We have implemented the algorithms for both the Factoring and Following, and they operate in their basic mode in near O(n) [4] time where n is the length of the sequence to follow. With the newest version of the algorithm based upon a direct addressing scheme for the antecedents in an implicant, we have yet to actually obtain real performance data. It should be clear based upon how the ruling is structured, that one can achieve faster processing by placing levels of a ruling on several independent processors and connect the processors in a pipeline configuration. This will provide a simple, pipelined, parallel algorithm with speed of O(n/p) where p is the number of processors. Alternatively, one can place the entire ruling on each of several processors in a cluster or network, and then supplying overlapping (by the inductive base) subsets of the unknown string to each processor. Again the process complexity becomes O(n/p). In order to achieve these speeds, we first used a finite state machine model indicated earlier as the ruling structure. Storage space has an upper limit of the inductive base*n again where n is the length of the string to be factored. With the new algorithm we are using a large amount of internal memory to hold the ruling, and the following process is simply a direct access into that storage structure. This has limited the inductive base to approximately three codons, but access for any nucleotide would be then, if found ruling-levels/2, and if not found then simply the number of levels in the ruling. Since this number is small, the complexity for following would be at most O(n*number-of-levels-in-Ruling), which is O(n*k) or just O(n). This is the more important time constraint, due to the fact that a ruling is created once and used many times in identifying patterns.

References 1. Altschul, S.F., Gish, W., Miller, W., Myers, E.W., Lipman, D.J.: Basic local alignment search tool. Journal of Molecular Biology 215(3), 403–410 (1990) 2. Buckingham, S.D.: Scientific software: seeing the SNP’s between us. Nature Methods 5, 903–908 (2008) 3. Case, J., Fisher, P.S.: Long Term Memory Modules. Bulletin of Mathematical Biology 46(2) (1984) 4. Das, S., Fisher, P.S., Zhang, H.: Efficient Parallel Algorithms for Pattern Recognition. In: Proceedings of Twenty-sixth Hawaii International Conference in Systems Sciences (January 1993) 5. Pearson, W.: Flexible sequence similarity searching with FASTA3 program package. In: Misener, S., Krawety, S.A. (eds.) Bioinformatics Methods and Protocols, pp. 185–219. Humana Press, Inc., Totowa (2000) 6. Pearson, W.R., Lipman, D.J.: Improved tools for biological sequence comparison. Proc. National Academy of Science 85, 2444–2448 (1988) 7. Smith, T.F., Waterman, M.S.: Identification of Common Molecular Subsequences. Journal of Molecular Biology 147, 195–197 (1981)

Syntactic Pattern Recognition Using Finite Inductive Strings

101

8. Uberbacher, E.: Computing the Genome, http://www.ornl.gov/info/ornlreview/v30n3-4/genome.htm 9. Wang, G.Y., Fisher, P.: Knowledge Acquisition: Neural Network Learning. In: Data Mining and Knowledge Discovery: Theory, Tools, and Technology. II SPIE, vol. 4057, pp. 117–128 (2000) 10. Xu, Y., Mural, R.J., Einstein, J.R., Shah, M., Uberbacher, E.C.: GRAIL: A Multi-Agent Neural Network System for Gene Identification. Proceedings of The IEEE 84(10), 1544– 1552 (1996)

Evidence-Based Clustering of Reads and Taxonomic Analysis of Metagenomic Data Gianluigi Folino1 , Fabio Gori2 , Mike S.M. Jetten2 , and Elena Marchiori2, 1

2

ICAR-CNR, Rende, Italy Radboud University, Nijmegen, The Netherlands [email protected]

Abstract. The rapidly emerging ﬁeld of metagenomics seeks to examine the genomic content of communities of organisms to understand their roles and interactions in an ecosystem. In this paper we focus on clustering methods and their application to taxonomic analysis of metagenomic data. Clustering analysis for metagenomics amounts to group similar partial sequences, such as raw sequence reads, into clusters in order to discover information about the internal structure of the considered dataset, or the relative abundance of protein families. Diﬀerent methods for clustering analysis of metagenomic datasets have been proposed. Here we focus on evidence-based methods for clustering that employ knowledge extracted from proteins identiﬁed by a BLASTx search (proxygenes). We consider two clustering algorithms introduced in previous works and a new one. We discuss advantages and drawbacks of the algorithms, and use them to perform taxonomic analysis of metagenomic data. To this aim, three real-life benchmark datasets used in previous work on metagenomic data analysis are used. Comparison of the results indicates satisfactory coherence of the taxonomies output by the three algorithms, with respect to phylogenetic content at the class level and taxonomic distribution at phylum level. In general, the experimental comparative analysis substantiates the eﬀectiveness of evidence-based clustering methods for taxonomic analysis of metagenomic data.

1

Introduction

The rapidly emerging ﬁeld of metagenomics seeks to examine the genomic content of communities of organisms to understand their roles and interactions in an ecosystem. Given the wide-ranging roles microbes play in many ecosystems, metagenomics studies of microbial communities will reveal insights into protein families and their evolution. Because most microbes cannot grow in the laboratory using current cultivation techniques, scientists have turned to cultivationindependent techniques to study microbial diversity. At ﬁrst shotgun Sanger sequencing was used to survey the metagenomic content, but nowadays massive parallel sequencing technology like 454 or Illumina, allow random sampling of DNA sequences to examine the genomic material present in a microbial community [1].

Corresponding author.

V. Kadirkamanathan et al. (Eds.): PRIB 2009, LNBI 5780, pp. 102–112, 2009. c Springer-Verlag Berlin Heidelberg 2009

Evidence-Based Clustering of Reads and Taxonomic Analysis

103

For a given sample, one would like to determine the phylogenetic provenance of the obtained fragments, the relative abundance of its diﬀerent members, their metabolic capabilities, and the functional properties of the community as a whole. To this end, computational analysis is becoming increasingly indispensable [2,3]. In particular, clustering methods are used for rapid analysis of sequence diversity and internal structure of the sample [4], for discovering protein families present in the sample [5], and as a pre-processing set for performing comparative genome assembly [6], where a reference closely related organism is employed to guide the assembly process. In this paper we focus on clustering methods and their application to taxonomic analysis of metagenomic data. Clustering analysis for metagenomics amounts to group similar partial sequences, such as raw sequence reads, or candidate ORF (Open Reading Frame) sequences generated by an assembly program into clusters in order to discover information about the internal structure of the considered dataset, or the relative abundance of protein families. Diﬀerent methods for clustering analysis of metagenomic datasets have been proposed, which can be divided into two main approaches. Sequence- and evidence-based methods. Sequence-based methods compare directly sequences using a similarity measure either based on sequence overlapping [4] or on extracted features such as oligonucleotide frequency [7]. Evidence-based methods employ knowledge extracted from external sources in the clustering process, like proteins identiﬁed by a BLASTx search (proxygenes) [5]. In this paper we focus on the latter approach for clustering short reads. We consider two clustering algorithms introduced in previous works [5,8] and a reﬁnement of the latter one based on ensemble techniques. These algorithms cluster reads using weighted proteins as evidence. Such proteins are obtained by a specialized version of BLAST (Basic Local Alignment Search Tool), called BLASTx, which associates a list of hits to one read. Each hit consists of one protein, two score values, called bit and identities, which measure the quality of the read-protein matching, and one conﬁdence value, called E-value, which amounts to a conﬁdence measure of the matching between the read and the protein. Speciﬁcally, in [5] an algorithm, here called LWproxy (Local Weight proxy), is introduced, that clusters reads and those proteins in their sets of hits simultaneously, in such a way that one cluster of proteins is associated to one cluster of reads. Then it assigns one local weight to each protein of a cluster, using the cumulative BLASTx bit score of those reads in corresponding cluster having that protein as one of their hits. The protein with best weight (highest cumulative bit score) is selected as proxygene of the cluster of reads. In [8], an alternative method for clustering metagenome short reads based on weighted proteins is proposed, here called GWproxy (Global Weight proxy). The method ﬁrst assigns global weights to each protein using the BLASTx identity and bit score of those reads having that protein as one of their hits. Next, the method groups reads into clusters using an instance of the weighted set covering

104

G. Folino et al.

problem, with reads as rows and proteins as columns. It seeks the smallest set of columns covering all rows and having best total weight. A solution corresponds to a clustering of reads and one protein (proxygene) associated to each cluster. While in [5] the proxygene of a cluster is selected within a set of proteins associated to that cluster, in GWproxy clustering and proxygene selection are performed at the same time. In this paper we introduce a reﬁnement of GWproxy based on the following ensemble technique, called EGWproxy (Ensemble Global Weight proxy). The algorithm associates a list of proteins to each cluster resulting from application of GWproxy, such that each protein occurs as hit of each of the reads of that cluster. Such a list is used for reﬁning the biological analysis of the cluster, for instance by assigning a taxonomic identiﬁer (taxID) by means of weighted majority vote among the taxID’s of the proteins in the associated list. We discuss advantages and drawbacks of the above clustering algorithms, and use them to perform taxonomic analysis of metagenomic data. To this aim, three real-life benchmark datasets used in previous work on metagenomic data analysis are used. These datasets were introduced in [5] and used to perform a thorough analysis of evidence-based direct and indirect (that is, using proxygenes) annotation methods for short metagenomic reads. The results of such analysis substantiated advantages and eﬀectiveness of indirect methods over direct ones. The results of the three considered evidence-based clustering algorithms indicate satisfactory coherence of the taxonomies output by the algorithms. In general, the experimental comparative analysis substantiates the eﬀectiveness of evidence-based methods for taxonomic analysis of metagenomic data.

2

Clustering Metagenome Short Reads Using Proxygenes

Diﬀerent methods for clustering analysis of metagenomic datasets have been proposed, which can be divided into two main approaches. Sequence- and evidencebased methods. Sequence-based methods compare directly sequences using a similarity measure either based on sequence overlapping [4] or on extracted features such as oligonucleotide frequency [7]. Evidence-based methods employ knowledge extracted from external sources in the clustering process, like proteins identiﬁed by a BLASTx search (proxygenes) [5]. Here we consider the latter approach for clustering short reads. The knowledge used by the clustering algorithms here considered is extracted by a reference proteome database by matching reads to that database by means of BLASTx, a powerful search program. BLASTx belongs to the BLAST (Basic Local Alignment Search Tool) family, a set of similarity search programs designed to explore all of the available sequence databases regardless of whether the query is protein or DNA [9,10]. BLASTx is the BLAST program designed to evaluate the similarities between DNA sequences and proteins; it compares nucleotide sequence queries dynamically translated in all six reading frames to peptide sequence databases. The scores assigned in a BLAST search have a statistical interpretation, making real matches easier to distinguish from random background hits. In the following we summarize the main features of BLAST.

Evidence-Based Clustering of Reads and Taxonomic Analysis

2.1

105

The BLAST Alignment Method

BLAST uses a heuristic algorithm that seeks local as opposed to global alignments and is therefore able to detect relationships among sequences that share only isolated regions of similarity [11]. When a query is submitted, BLAST works by ﬁrst making a look-up table of all the words (short subsequences, three letters in our case) and neighboring words, i.e., similar words in the query sequence. The sequence database is then scanned for these strings; the locations in the databases of all these words are called word hits. Only those regions with word hits will be used as alignment seeds. When one of these matches is identiﬁed, it is used to initiate gap-free and gapped extensions of the word. After the algorithm has looked up all possible words from the query sequence and extended them maximally, it assembles the statistically signiﬁcant alignment for each query-sequence pair, called High-scoring Segment Pair (HSP). The matching reliability of read r and protein p is evaluated trough Bit Score, denoted by SB (r, p), and E-value, denoted by E(r, p). The bit score of one HSP is computed as the sum of the scoring matrix values for that segment pair. The E-value is the number of times one might expect to see such a query-sequence match (or a better one) merely by chance. Another important BLASTx score of matching between r and p is Identities score, denoted by Id (r, p), deﬁned as the proportion of the amino-acids in the database sequence p that are matched by the amino-acids translation of the current query frame r. We refer to [9] for a formal description of these measures. We turn now to describe the three methods here used for taxonomic analysis of metagenomic data. Here and in the sequel we assume the BLASTx has been applied to a metagenomic data set with a given Evalue cutoﬀ value. We denote by R = {r1 , . . . , rm } the resulting set of reads having at least one BLASTx hit for the given cutoﬀ, and by P = {p1 , . . . , pn } the set of proteins occurring in the hit of at least one read of R. 2.2

LWproxy

LWproxy generates a collection of pairs (Ci , Pi ), where Ci is a set of reads and Pi a set of proteins. The algorithm can be summarized as follows. Set i = 0. Set X = R. If X is empty then terminate, otherwise set i = i + 1. Select randomly1 one read ri from X as seed of cluster Ci = {ri }. Set Pi to the set of hits of ri . Remove ri from X. Add to Ci all the reads having one element of Pi as a best hit, and remove them from X. 8. Add to Pi all hits of those reads added to Ci in the previous step. 9. If no reads are added then go to step 3, otherwise go to step 7.

1. 2. 3. 4. 5. 6. 7.

1

We consider here random seed selection. However, in [5] the criterion for selecting a seed is not speciﬁed.

106

G. Folino et al.

When the clustering process is terminated, the method assigns one proxygene to each Ci by selecting from Pi the protein having highest cumulative bit-score. Example 1. Suppose given a set of ﬁve reads {r1 , . . . , r5 } and suppose that the proteins occurring in their hits: – – – – –

{p1 , p3 , p5 } for read r1 , with best hit p3 . {p1 , p3 , p5 } for read r2 , with best hit p3 . {p2 , p4 } for read r3 , with best hit p2 . {p2 } for read r4 , with best hit p2 . {p2 , p3 , p5 } for read r5 , with best hit p2 .

If LWproxy starts from r1 as seed for C1 then only r2 is added to C1 , since p2 (best hit of each of the other reads) does not occur in the list of protein associated to C1 . Then construction of a second cluster, say C2 begins. C2 is ﬁlled with the rest of the reads {r3 , r4 , r5 }. 2.3

GWproxy

While LWproxy constructs clusters incrementally, GWproxy searches for clusters in a given search space, consisting of clusters characterized by the proteins as follows. We say that a protein covers a read if the protein occurs as one of the hits of that read. Then each protein characterizes one cluster, consisting of the reads it covers. Moreover, we can assign to each protein a global weight, representing the cost of selecting that protein as cluster representative. The weight of protein p is deﬁned as ⎡ ⎤ max 1 S − S (r, p) B B w(p) = 1 + ⎢ (100 + 100 − Id (r, p))⎥ ⎢ Np ⎥, max min S − S B B ⎢ ⎥ r|p hit of r where v denotes the smallest integer bigger or equal than v, and Np is the number of reads having p as one of their hits. The maximum and minimum value of SB over the considered pairs of reads and proteins, SB max and SB min , respectively, are used to scale SB (r, p). The weight w(p) is in inverse proportion to the average quality of the matchings between p and the reads having p as a hit, so that better proteins have smaller w(p) value (smaller cost). Clustering then amount at ﬁnding a minimum set of proteins in R that, together, cover all the reads in R and have minimum total cost. Formally, consider the vector of protein weights w ∈ Nn and the matrix A ∈ {0, 1}m×n whose elements aij are such that 1, if pj covers ri , aij = 0, otherwise. We want to solve the following constrained optimization problem (weighted set covering problem (WSC, in short)). min

x∈{0,1}n

n j=1

xj wj ,

such that

n j=1

aij xj ≥ 1,

for i = 1, . . . , m .

(WSC)

Evidence-Based Clustering of Reads and Taxonomic Analysis

107

Table 1. Left: input covering matrix; position (i, j) contains a 1 if protein pj occurs in the set of selected hits of read ri , otherwise it contains a 0. Right: proxygenes selected by the GWproxy are indicated with bigger fonts. p1 1 1 A= 0 0 0 w = 15

p2 0 0 1 1 1 10

p3 1 1 0 0 1 10

p4 1 0 1 0 0 20

p5 0 1 0 0 1 5

r1 r2 r3 r4 r5

p1 1 1 A= 0 0 0 w = 15 x= 0

p2 p3 p4 0 1 1 0 1 0 1 0 1 1 0 0 1 1 0 10 10 20 1 1 0

p5 0 1 0 0 1 5 0

r1 r2 r3 r4 r5

The variable xj indicates whether pj belongs to the solution (xj = 1) or not (xj = 0). The m constraint inequalities are used to express the requirement that each read ri be covered by at least one protein. The weight wj speciﬁes the cost of protein pj . Here a fast heuristic algorithm2 for WSC [12] is applied to ﬁnd a solution in an acceptable time. A solution corresponds to a subset of P consisting of those proteins pj such that xj = 1. Each of the selected proteins is a proxygene. It represents the cluster consisting of those reads covered by that protein. Example 2. We illustrate the application of GWproxy on the toy problem of Example 1. Let w = (15 10 10 20 5) be the vector of protein weights. Then Table 1 (left part) shows the corresponding matrix A ∈ {0, 1}5×5. Application of GWproxy outputs proteins p2 , p3 (see Table 1 right part). The selected proteins correspond to the two clusters of reads {r3 , r4 , r5 } and {r1 , r2 , r5 }, with p2 and p3 as associated proxygenes, respectively. 2.4

EGWproxy

The aim of the EGWproxy algorithm is to reﬁne the clustering produced by GWproxy as follows. As we described before, each cluster that GWproxy outputs is represented by one protein. However, because of the short length of reads, and because in general the size of clusters is not very big (see analysis in [8]), it may well happen that more than one protein cover all the reads of a cluster. Despite GWproxy selects only one of such proteins, among those having best score, each of the proteins covering a cluster can be considered as an equivalent representative of that cluster. Taking into account this fact, we can increase the robustness of GWproxy proxygenes choice method. For instance, for performing taxonomic analysis of cluster C, the following criterion can be used in order to decide which taxID to associate to C. The set T of taxID’s of the list of proteins that EGWproxy associates to C is considered. Then the ﬁnal taxID tf in of C is computed as Np tf in = arg min , t∈T wp p with taxID equal to t 2

Publicly available at http://www.cs.ru.nl/~ elenam

108

G. Folino et al.

where the operator arg min gives the element at which the objective function takes its minimum. 2.5

Comparison of Algorithms

GWproxy and LWproxy use diﬀerent clustering heuristics: the ﬁrst algorithm searches a clustering in a ﬁxed searching space characterized by the sets of reads covered by each protein in R, while LWproxy constructs incrementally clusters of reads and of proteins. Furthermore, GWproxy scores proteins using bit and identities score, while LWproxy uses only bit score. Finally, GWproxy scores each protein globally, that is, using all the reads it covers, while LWproxy scores only the proteins of a cluster, where each protein is scored locally using the reads it covers that belong to that cluster. Both EGWproxy and LWproxy associates to each cluster of reads one set of proteins. However, while LWproxy selects one protein as ﬁnal representative of a cluster, EGWproxy employs an ensemble technique in order to exploit information of all the proteins of that set. A drawback of LWproxy is that results may be aﬀected by the choice of the read used in the ﬁrst step of the algorithm, as illustrated by the following example. Example 3. Consider the toy problem in Example 1. Suppose LWproxy selects r5 as seed of the ﬁrst cluster C1 . Then it adds all the other reads to C1 , since their best hit is in the list of hits of r5 . P1 becomes equal to the entire set of proteins. Suppose for simplicity that all proteins have equal bit-score. Then LWproxy selects p2 as proxygene, since it has highest cumulative bit-score. Hence, this clustering While in the experiments here conducted this drawback does not seem to aﬀect the results, it remains to be investigated whether this drawback does not aﬀect results in general. A drawback of GWproxy is that it outputs only one solution, while in general there may be more ”optimal” clustering of reads. This is because the weighted set covering problem seeks one optimal solution, not the set of all optimal solutions. EGWproxy tries to overcome this drawback by using a post-processing step, followed by the application of an ensemble technique for merging multiple solutions. However, the post-processing step acts only on the set of proteins, while the clusters of reads remain those produced by GWproxy. It remains to be investigated whether application of ensemble techniques also at the level of clusters of reads can improve the performance of the method.

3

Taxonomic Analysis of Metagenome Data

We consider three complex metagenome datasets introduced in [5], called in the following M1, M2 and M3. These datasets were generated, respectively, from 9, 5 and 8 genome projects, sequenced at the Joint Genome Institute (JGI) using the 454 GS20 pyrosequencing platform that produces ∼ 100 bp reads. From

Evidence-Based Clustering of Reads and Taxonomic Analysis

109

Table 2. Characteristics of the organisms used in the experiments: the identiﬁer and name of the organism, the size of its genome and the total number of reads sampled (M1 dataset) Id. a b c d e f g h i

Organism genome size (bp) reads sampled Clostridium phytofermentans ISDg 4 533 512 4638 Prochlorococcus marinus NATL2A 1 842 899 1866 Lactobacillus reuteri 100-23 2 174 299 2371 Caldicellulosiruptor saccharolyticus DSM 8903 2 970 275 2950 Clostridium sp. OhILAs 2 997 608 2934 Herpetosiphon aurantiacus ATCC 23779 6 605 151 6937 Bacillus weihenstephanensis KBAB4 5 602 503 4158 Halothermothrix orenii H 168 2 578 146 2698 Clostridium cellulolyticum H10 3 958 683 3978

each genome project, reads were sampled randomly at coverage level 0.1X. The coverage is deﬁned as the average number of times a nucleotide is sampled. This resulted in a total of 35230, 28870 and 35861 reads, respectively. Table 3 shows the names of the organisms and the number of reads generated for the M1 dataset. The reader is referred to [5] for a detailed description of all the datasets. In our experiments we use the NR3 (non-redundant) protein sequence database as reference database for BLASTx. The parameters of the external software we used are set as follows. For BLASTx the default parameters were used. In all experiments we used Evalue cutoﬀ E = 10−6 . Moreover, WSCP was run with pre-processing (−p), number of iterations equal to 1000 (−x1000), one tenth of the best actual cover used as starting partial solution (−a0.1), and 150 columns to be selected for building the initial partial cover at the ﬁrst iteration (−b150). For lack of space, we refer to [12] for a detailed description of the WSCP program. 3.1

Results

We extract taxonomic information from each metagenome dataset as follows. For LWproxy and GWproxy each cluster of reads is represented by one protein. The taxID of such protein is used as taxonomic information of that cluster. For EGWproxy the list of proteins associated to each cluster is transformed into one taxID as described in Section 2.4. In this way, the metagenomic data is transformed into a set of taxID’s of proteins, one for each cluster of reads. Taxonomic information is then retrieved from the NCBI taxonomy (see http://www.ncbi.nlm.nih.gov/Taxonomy/). The NCBI Taxonomy database is a curated set of taxonomic classiﬁcations for all the organisms that are represented in GenBank. Each taxon in the database is associated with a numerical unique identiﬁer called taxID. In the present analysis, the taxonomic information of these known proxygenes is used to determine the taxonomic content of the metagenomic data. 3

Publicly available at ftp://ftp.ncbi.nlm.nih.gov/blast/db

110

G. Folino et al.

We visualize the resulting taxonomic information in two ways. – Graph representation of taxonomic distribution of reads, as done e.g. in [14]. Here analysis at the taxonomic level of phylum and class is performed, where resulting taxa containing less than 100 reads are discarded. – Histogram of phylogenetic identities, as done e.g. in [13]. Shown are the percentages of the total of identiﬁable hits assigned to the phylogenetic groups obtained by means of the taxID of the proxygenes. Here analysis at the class taxonomic level is performed. We apply the above techniques to the proxygenes and taxid obtained from the considered algorithms, as well as to the known taxid’s of the original metagenome data sets, provided by the producers of the benchmark data [5]. We use these latter results as ”golden truth” (GT in short) to evaluate the methods. For lack of space, we show graphs of the taxonomic distribution at phylum and class level only for M1 in Figure 1. Results indicate satisfactory consensus

root

Cyanobacteria

Bacilli

(815)

(3907)

Firmicutes

Clostridia

(12919)

(9011)

Chloroflexi

(4178)

Chloroflexi (class)

(4178)

root

Cyanobacteria

Bacilli

(832)

(3827)

Firmicutes

Clostridia

(12863)

(9036)

Chloroflexi

(4206)

Chloroflexi (class)

(4206)

root

Cyanobacteria

Bacilli

(833)

(3852)

Firmicutes

Clostridia

(12842)

(8989)

Chloroflexi

(4188)

Chloroflexi (class)

(4187)

root

Cyanobacteria

Bacilli

(1868)

(6529)

Firmicutes

Clostridia

(21031)

(14502)

Chloroflexi

(9635)

Chloroflexi (class)

(9635)

Fig. 1. Phylogenetic graph for M1. From top to bottom: LWproxy, GWproxy, EGWproxy and ”Golden Truth”.

Evidence-Based Clustering of Reads and Taxonomic Analysis

111

Fig. 2. Taxonomic distribution at taxonomic class level of the three datasets. From left to right: M1, M2 and M3. From top to bottom: ”golden truth”, GWproxy, EGWproxy and LWproxy.

among the three methods, yielding similar type of graphs. The methods give graphs with the same structure of the ”golden truth” for M1 and M3 datasets. For M2, instead, GWproxy and EGWproxy give graphs with a phylum subtree (Streptophyta) that does not appear in the ”golden truth”; anyway they are made by just 1% and 2% of the covered reads, respectively. Histograms of phylogenetic identities at the class level for the three data sets are shown in Figure 2. The results achieved by the three methods are almost identical.

4

Conclusion and Future Work

In this paper we compared three methods for clustering reads and their application to taxonomic analysis of metagenome data. We discuss advantages and drawbacks of the methods and applied them to perform taxonomic analysis of three real-life metagenome datasets with known taxonomic content. Results of such analysis indicate satisfactory consensus of all the three methods, and very good performance with respect to taxonomic distribution and phylogenetic content. Unfortunately, a drawback of LWproxy is that results can be aﬀected by the choice of the read used in the ﬁrst step of the algorithm, as shown in Example 3. While in the experiments here conducted this drawback

112

G. Folino et al.

does not seem to aﬀect the results, it remains to be investigated whether this drawback does not aﬀect results in general. The algorithms EGWproxy can potentially reﬁne the proxygenes selection made by GWproxy despite we found no evidence of this fact in our analysis yet. In future we intend to introduce a statistical test for measuring signiﬁcance of taxonomic assignment, in order to discard assignments possibly due to the composition of the reference proteome database used when applying BLASTx. Such a test will consider not only the number of reads assigned to a taxa, but also the divergence of their proxygenes as well as the nucleotide composition of the reads. Acknowledgements. We would like to thank Mavrommatis Konstantinos for providing the datasets in [5] as well as useful information about such data.

References 1. Yooseph, S., et al.: The Sorcerer II global ocean sampling expedition: Expanding the universe of protein families. PLoS Biol. 5(3), e16 (2007) 2. McHardy, A., Rigoutsos, I.: What’s in the mix: phylogenetic classiﬁcation of metagenome sequence samples. Current Opinion in Microbiology 10, 499–503 (2007) 3. Raes, J., Foerstner, K., Bork, P.: Get the most out of your metagenome: computational analysis of environmental sequence data. Current Opinion in Microbiology 10, 490–498 (2007) 4. Li, W., Wooley, J., Godzik, A.: Probing metagenomics by rapid cluster analysis of very large datasets. PLoS One 3(10) (2008) 5. Dalevi, D., Ivanova, N., Mavromatis, K., Hooper, S., Szeto, E., Hugenholtz, P., Kyrpides, N., Markowitz, V.: Annotation of metagenome short reads using proxygenes. Bioinformatics 24(16) (2008) 6. Pop, M., Phillippy, A., Delcher, A., Salzberg, S.: Comparative genome assembly. Brieﬁngs in Bioinformatics 5(3), 237–248 (2004) 7. Chan, C., Hsu, A., Tang, S., Halgamuge, S.: Using growing self-organising maps to improve the binning process in environmental whole-genome shotgun sequencing. Journal of Biomedicine and Biotechnology (2008) 8. Folino, G., Gori, F., Jetten, M.S.M., Marchiori, E.: Clustering metagenome short reads using weighted proteins. In: EvoBIO 2009. LNCS, vol. 5483, pp. 152–163. Springer, Heidelberg (2009) 9. Korf, I., Yandell, M., Bedell, J.: BLAST. O’Reilly & Associates, Inc., Sebastopol (2003) 10. Madden, T.: 16. In: The BLAST Sequence Analysis Tool, Bethesda, MD (2002) 11. Altschul, S., Gish, W., Miller, W., Myers, E., Lipman, D.: Basic local alignment search tool. J. Molecular Biology 215(3), 403–410 (1990) 12. Marchiori, E., Steenbeek, A.: An evolutionary algorithm for large scale set covering problems with application to airline crew scheduling. In: Oates, M.J., et al. (eds.) EvoIASP 2000, EvoWorkshops 2000, EvoFlight 2000, EvoSCONDI 2000, EvoSTIM 2000, EvoTEL 2000, and EvoROB/EvoRobot 2000. LNCS, vol. 1803, pp. 367–381. Springer, Heidelberg (2000) 13. Biddle, J.F., et al.: Metagenomic signatures of the Peru margin subseaﬂoor biosphere show a genetically distinct environment. PNAS (105), 10583–10588 (2008) 14. Venter, J., et al.: Environmental genome shotgun sequencing of the sargasso sea. Science (304), 66–74 (2004)

Avoiding Spurious Feedback Loops in the Reconstruction of Gene Regulatory Networks with Dynamic Bayesian Networks Marco Grzegorczyk1 and Dirk Husmeier2 1 2

Department of Statistics, TU Dortmund University, 44221 Dortmund, Germany [email protected] Biomathematics and Statistics Scotland, JCMB, KB, Edinburgh EH9 3JZ, UK [email protected]

Abstract. Feedback loops and recurrent structures are essential to the regulation and stable control of complex biological systems. The application of dynamic as opposed to static Bayesian networks is promising in that, in principle, these feedback loops can be learned. However, we show that the widely applied BGe score is susceptible to learning spurious feedback loops, which are a consequence of non-linear regulation and autocorrelation in the data. We propose a non-linear generalisation of the BGe model, based on a mixture model, and demonstrate that this approach successfully represses spurious feedback loops.

1

Introduction

In systems biology, there has been increased interest in learning regulatory networks and signalling pathways from postgenomic data. Following up on the seminal paper by Friedman et al. [1], Bayesian networks have been widely applied to this end. Their popularity partially stems from the tractability of the marginal likelihood of the network structure, which is the consistent scoring scheme for model selection in the Bayesian context. The practical computation requires the integration of the likelihood over the entire parameter space, though. To obtain a closed-form expression, two probabilistic models with their respective conjugate prior distributions have been employed in the past: the multinomial distribution with the Dirichlet prior, leading to the so-called BDe score [2], and the linear Gaussian distribution with the normal-Wishart prior, leading to the BGe score [3]. These approaches are restricted in that they either require the data to be discretised (BDe) or can only capture linear regulatory relationships (BGe). A non-linear non-discretised model based on heteroscedastic regression has been proposed by Imoto et al. [4]. However, this approach no longer allows the marginal likelihood to be obtained in closed-form and requires a restrictive approximation (the Laplace approximation) to be adopted. Another non-linear model based on node-speciﬁc Gaussian mixture models has been proposed [5]. Here, Ko et al. resort to the Bayesian information criterion BIC of Schwarz [6] for model selection, which is only a good approximation to the marginal likelihood in the limit of very large data sets. Recently, we proposed a generalisation V. Kadirkamanathan et al. (Eds.): PRIB 2009, LNBI 5780, pp. 113–124, 2009. c Springer-Verlag Berlin Heidelberg 2009

114

M. Grzegorczyk and D. Husmeier

of the BGe score [8] based on a combination of a mixture model with the allocation sampler proposed in [7]. In this approach the latent variable allocation is common to the whole network, though, which results in a heterogeneous linear rather than a proper non-linear model. Our work aims to generalise our earlier work [8] by the introduction of node-speciﬁc change-points. This model is similar to the model by Ko et al. [5], with the diﬀerence that the marginal likelihood is computed properly, rather than approximated by BIC.

2

Problems of the BGe Score

When the objective is to infer regulatory networks from time series, as is typically the case in systems biology, the restriction of the model to linear processes can result in the prediction of spurious feedback loops. Consider the simple example shown in Figure 1. The graph shows two interacting nodes. Node X is a regulator of node Y , and it also has a regulatory feedback loop acting back on itself. Node Y is regulated by node X, but does not contain a feedback loop. The ﬁgure shows both the state space representation, i.e. the recurrent graph, and the corresponding dynamic Bayesian network. Note that the latter is a valid DAG obtained by the standard procedure of unfolding the state space graph in time. First assume that the data generation processes are consistent with the BGe model assumption, e.g.: X(t + 1) = X(t) + c + σx · φX (t) and Y (t + 1) = w · X(t) + m + σy · φY (t) where w, m, c, σx , σy are constants, and φ. (.) are iid normally distributed random variables. Under fairly general regularity conditions, the marginal likelihood and, hence, the BGe score is a consistent estimator. This implies that the correct model structure will be learned as m → ∞, where m is the data set size. Next, consider the scenario of a non-linear regulatory inﬂuence that X exerts on Y : X(t + 1) = X(t) + c + σx · φX (t),

Y (t + 1) = f (X(t)) + σy · φY (t)

(1)

for some non-linear function f (.). This non-linear function cannot be modelled with a linear Bayesian network based on the BGe model. Consequently, the prediction of Y (t + 1) from X(t) will tend to be poor. Note that for suﬃciently small noise levels, the Y (t)’s will exhibit a strong autocorrelation, by virtue of the autocorrelation of the X(t)’s, and the regulatory inﬂuence of X(t) on Y (t + 1). If

Fig. 1. State space graph and corresponding dynamic Bayesian network.Left: Recurrent state space graph containing two nodes. Node X has a recurrent feedback loop and acts as a regulator of node Y . Right: The same graph unfolded in time.

Avoiding Spurious Feedback Loops

115

the latter regulatory inﬂuence cannot be learned owing to the linear restriction of our model, the next best explanation is a direct modelling of the autocorrelation between the Y (t)’s themselves. This autocorrelation corresponds to a feedback loop of Y acting back on itself in the state-space graph, or, equivalently, an edge from Y (t) to Y (t + 1) in the dynamic Bayesian network. We would therefore conjecture that the linear restriction of the Bayesian network model may result in the prediction of spurious feedback loops and, hence, to the reconstruction of wrong network structures. Ruling out feedback loops altogether, as we did in [8], will not provide a suﬃcient remedy for this problem, as some nodes – X in the example above – will exhibit regulatory feedback loops (e.g. in molecular biology: transcription factors regulating their own transcription), and it is generally not known in advance where these nodes are.

3

Methodology

3.1

The Dynamical BGe Network

Dynamical Bayesian networks (DBNs) are ﬂexible models for representing probabilistic relationships between interacting variables X1 , . . . , XN . The graph G of a DBN describes the relationships between the variables, which have been measured at equidistant time points t = 1, . . . , m, in the form of conditional probability distributions. An edge pointing from Xi to Xj means that the realisation of Xj at time point t, symbolically: Xj (t), is inﬂuenced by the realisation of Xi at time point t − 1, symbolically: Xi (t − 1). πn = πn (G) denotes the parent node set of node Xn in G, i.e. the set of all nodes from which an edge points to node Xn in G. Given a data set D, where Dn,t and D(πn ,t) are the tth realisations Xn (t) and πn (t) of Xn and πn , respectively, DBNs are based on the following homogeneous Markov chain expansion: P (D|G, θ) =

N m

P (Xn (t) = Dn,t |πn (t − 1) = D(πn ,t−1) , θn )

(2)

n=1 t=2

where θ is the total parameter vector, composed of subvectors θn , which specify the local conditional distributions in the factorisation. The BGe model [3] speciﬁes the distributional form P (D|G, θ = {μ, Σ}) as multivariate Gaussian distribution with expectation vector μ and covariance matrix Σ, and assumes a normal-Wishart distribution as prior distribution P ({μ, Σ−1 }). The local probability distributions P (Xn |πn , θn ) are then given by conditional Gaussian distributions so that only linear relationships between Xn and its parent nodes πn can be modelled. Under fairly weak conditions imposed on the parameters θ (prior independence and modularity) and prior distribution P (θ) (conjugacy), the parameters can be integrated out analytically, as shown by Geiger and Heckerman [3], and the marginal likelihood satisﬁes the same expansion rule as the Bayesian network with ﬁxed parameters: P (D|G) =

N m n=1 t=2

P (Xn (t) = Dn,t |πn (t − 1) = D(πn ,t−1) ) =

N n=1

Ψ (Dnπn )

(3)

116

M. Grzegorczyk and D. Husmeier

where Dnπn := {(Dn,t , Dπn ,t−1 ) : 2 ≤ t ≤ m} is the subset of the data pertaining to node Xn and parent set πn . For the BGe model the factors Ψ (Dnπn ) can be computed according to Eqn. (15) and (24) in [3]. 3.2

The New Mixture/Change-Point BGe Model

To obtain a more ﬂexible model for DBNs we generalise Eq. (2) with a nodespeciﬁc mixture model: P (D|G, V, K, θ) =

N m Kn

P (Xn (t) = Dn,t |πn (t − 1) = D(πn ,t−1) , θkn )δVn (t),k

n=1 t=2 k=1

(4) where δVn (t),k is the Kronecker delta, V is a matrix of latent variables Vn (t), Vn (t) = k indicates that the tth realisation of node Xn , symbolically Xn (t), has been generated by the kth component of a mixture with Kn components, and K = (K1 , . . . , Kn ). Note that the matrix V divides the data into several disjoined subsets, each of which can be regarded as pertaining to a separate BGe model with parameters θkn . The probability model deﬁned in Eq.(4) is eﬀectively a mixture model with local probability distributions P (Xn |πn , θn ) and it can hence, in principle, approximate any probability distribution arbitrarily closely. The vectors Vn are node-speciﬁc, i.e. diﬀerent nodes can have diﬀerent break-points, so that the proposed model has a higher ﬂexibility in modelling non-linear relationships than the BGM model proposed in [8]. Diﬀerent from the free allocation of latent variables in [8], we change the assignment of data points to mixture components via a change-point process. This eﬀectively reduces the complexity of the latent variable space and incorporates our prior belief that, in a time series, adjacent time points are likely to be assigned to the same component. Conditional on the latent variables V and under the assumption that the regularity conditions deﬁned in Geiger and Heckerman [3] are satisﬁed, the marginal likelihood can be computed in closed form: P (D|G, V, K) =

P (D|G, V, K, θ)P (θ)dθ =

Kn N

Ψ (Dnπn [k, Vn ])

(5)

n=1 k=1

where Dnπn [k, Vn ] := {(Dn,t , Dπn ,t−1 ) : Vn (t) = k, 2 ≤ t ≤ m} is the subset of the data pertaining to node Xn and its parents πn that has been assigned to the kth mixture component, symbolically: Vn (t) = k. In the absence of genuine prior knowledge about the regulatory network structure, we assume for P (G) a uniform distribution on graphs, subject to a fan-in restriction of |πn | ≤ 3. As prior probability distribution on the number of mixture components Kn , P (Kn ), we take an iid truncated Poisson distribution with shape parameter λ = 1 restricted to 1 ≤ Kn ≤ 10. We assume that the prior distributions P (Vn |Kn ) of the latent variable vectors Vn conditional on Kn are iid, and we identify Kn with Kn − 1 break-points: bn = {bn,1 , . . . , bn,Kn−1 } on the continuous interval [2, m]. For notational convenience we introduce the pseudo-break-points bn,0 = 2

Avoiding Spurious Feedback Loops

117

and bn,Kn = m. For node Xn the observation at time point t is assigned to the kth component, symbolically Vn (t) = k, if bn,k−1 ≤ t < bn,k . Following [10] we assume that the break-points are distributed as the even-numbered order statistics of L := 2(Kn − 1) + 1 points u1 , . . . , uL uniformly and independently distributed on the interval [2, m]. The motivation for this prior, instead of taking Kn uniformly distributed points, is to encourage a priori an equal spacing between the break-points, i.e. to discourage mixture components that contain only a few observations. The joint probability distribution of the proposed mixture BGe model is given by: P (G, V, K, D) = P (D|G, V, K) · P (G) · P (V|K) · P (K) Kn N πn = P (G) · {P (Vn |Kn ) · P (Kn ) · Ψ (Dn [k, Vn ]) (6) n=1

k=1

Here P (G) is the graph prior, and P (Kn ) the Poisson prior on the number of mixture components for the nth node. The local marginal likelihood terms Ψ (Dnπn [k, Vn ]), which result from Eq.(5), can be computed independently for each k using Eqn. (15) and (24) in [3]. Note that each vector Vn acts as a ﬁlter which divides the data of Xn into Kn diﬀerent compartments, for which separate independent BGe scores can be computed in closed-form. When a mixture component is empty, then Ψ (Dnπn [k, Vn ]) = 1. The term P (V|K) = N n=1 {P (Vn |Kn ) is the prior distribution on the node-speciﬁc allocation vectors Vn , which is induced by the even-numbered order statistics prior on the breakpoint locations bn . Deriving a closed-form expression is involved. However, the MCMC scheme we discuss in the next section does not sample Vn directly, but is based on local modiﬁcations of Vn based on birth, death and reallocation moves. All that is required for the acceptance probabilities of these moves are P (Vn |Kn ) ratios, which are straightforward to compute. 3.3

MCMC Inference

We now describe an MCMC algorithm to obtain a sample {G i , Vi , Ki }i=1,...,I from the posterior distribution P (G, V, K|D) ∝ P (G, V, K, D) of Eq. (6). We combine the structure MCMC algorithm for Bayesian networks of Madigan and York [9] with the change-point model of Green [10], and draw on the fact that conditional on the allocation vectors V, the model parameters can be integrated out to obtain the marginal likelihood terms Ψ (Dnπn [k, Vn ]). Note that this approach is equivalent to the idea underlying the allocation sampler proposed in [7]. The resulting algorithm is eﬀectively an RJMCMC scheme [10] in the discrete space of network structures and latent allocation vectors, where the Jacobian in the acceptance criterion is always 1 and can be omitted. With probability pG = 0.5 we perform a structure MCMC move on the current graph G i and leave the latent variable matrix and the numbers of mixture components unchanged, symbolically: Vi+1 = Vi and Ki+1 = Ki . A new candidate graph G i+1 is randomly drawn out of the set of graphs N (G i ) that can be reached from the

118

M. Grzegorczyk and D. Husmeier

current graph G i by deletion or addition of a single edge. The proposed graph G i+1 is accepted with probability: P (D|G i+1 , Vi , Ki ) P (G i+1 ) |N (G i )| A(G i+1 |G i ) = min 1, · · (7) P (D|G i , Vi , Ki ) P (G i ) |N (G i+1 )| where |.| is the cardinality and the marginal likelihood terms have been speciﬁed in Eq. (5). The graph is left unchanged, symbolically G i+1 := G i , if the move is not accepted. We note that the subsequent analysis will be based on the marginal posterior probabilities of individual edges, which can be estimated for each edge from the MCMC sample G 1 , ..., G I by the fraction of graphs in the sample that contain the edge of interest. With the complementary probability 1−pG we leave the graph G i unchanged, and perform a move on (Vi , Ki ), where Vni is the latent i variables vector of Xn in Vi , and Ki = (K1i , . . . , KN ). We randomly select a node Xn and change its current number of components Kni via a break-point birth or death move, or its latent variable vector Vni by a break-point re-allocation move. The break-point birth (death) move increases (decreases) Kni by 1 and may also have an eﬀect on Vni . The break-point reallocation move leaves Kni unchanged and may have an eﬀect on Vni . If with probability (1−pG )/N a breakpoint move on (Kni , Vni ) is performed, we randomly draw the move type. Under fairly mild regularity conditions (ergodicity, detailed balance), discussed in [10], the MCMC sampling scheme converges to the desired posterior distribution. To ensure detailed balance, the acceptance probabilities for the three break-point moves (Kni , Vni ) → (Kni+1 , Vni+1 ) are chosen of the form min(1, R), see [10], with Ki+1 n

k=1 R= i Kn

Ψ (Dnπn [k, Vni+1 ])

k=1

Ψ (Dnπn [k, Vni ])

×A×B

(8)

where A = P (Vni+1 |Kni+1 )P (Kni+1 )/P (Vni |Kni )P (Kni ) is the prior probability ratio, and B is the inverse proposal probability ratio. The exact form of these factors depends on the move type. (i) For a break-point reallocation (r) we randomly select one of the existing break-points bn,j ∈ {bn,1 , . . . , bn,Kn −1 }, and the replacement value b†n,j is drawn from a uniform distribution on [bn,j−1 , bn,j+1 ] where bn,0 = 2 and bn,Kn = m. Hence, the proposal probability ratio is one, the prior probabilities P (Kni+1 ) = P (Kni ) cancel out, and the remaining prior probability ratio P (Vni+1 |Kni+1 )/P (Vni |Kni ) can be obtained from p.720 in [10]: (bn,j+1 − b†n,j )(b†n,j − bn,j−1 ) Ar = , (bn,j+1 − bn,j )(bn,j − bn,j−1 )

Br = 1

(9)

If there is no break-point (Kni = 1) the move is rejected and the Markov chain is left unchanged. (ii) If a break-point birth move (b) on Kni is proposed, the location of the new break-point b† is randomly drawn from a uniform distribution on the interval [2, m]; the proposal probability for this move is bKin /(m − 2), where bKin is the (Kni -dependent) probability of selecting a birth move. The reverse death move, which is selected with probability d(Kin +1) , consists in discarding

Avoiding Spurious Feedback Loops

119

randomly one of the Kni − 1 + 1 = Kni change-points. The inverse proposal probability ratio is thus given by B = d(Kin +1) (m − 2)/bKin Kni . The prior probability ratio is given by the expression at the bottom of p.720 in [10] (slightly modiﬁed to allow for the fact that Kn components correspond to Kn − 1 break-points), and we get: Ab =

d(Kin +1) (m − 2) P (Kni + 1) 2Kni (2Kni + 1) (bn,j+1 − b† )(b† − bn,j ) , Bb = (10) i 2 P (Kn ) (m − 2) (bn,j+1 − bn,j ) bKin Kni

For Kni = KMAX the birth of a new break-point is invalid and the Markov chain is left unchanged. Note that the ratio of the proposal probabilities for birth versus death moves d(Kin +1) /bKin can be chosen such that it cancels out against the prior ratio P (Kni + 1)/P (Kni ), and the expression simpliﬁes: Ab Bb =

2(2Kni + 1) (bn,j+1 − b† )(b† − bn,j ) (m − 2) (bn,j+1 − bn,j )

(11)

(iii) A break-point death move (d) is the reverse of the birth move, and we get: Ad Bd =

4

(m − 2) (bn,j+1 − bn,j ) 2(2Kni − 1) (bn,j+1 − b† )(b† − bn,j )

(12)

Data

We have evaluated our method, which we henceforth refer to as the Mix-BGe model, on various synthetic data sets. For illustration purposes we present the results obtained for two studies with small networks. The ﬁrst network consists of two domain nodes X and Y . The true network structure is shown in Figure 1. The dynamics of the system are given by the non-linear state-space equations (see Eq. (1)), with the non-linear function f (.) = sin(.). We generated 40 observations by applying Eq. (1), setting the drift term c = 2π/41 to ensure that the complete period [0, 2π] of the sinusoid is involved. The second network is a generalisation of this two node domain where three nodes Y1 , Y2 , and Y3 are regulated by X. The regulatory relationships are again realised by sinusoids, whereby we shift the periods. More precisely, we set: Yi (t+ 1) = sin(X(t)+ τi ·π)+ σy ·φy,i (t) where τ1 = 0, τ2 = 2/3, and τ3 = 4/3. Again we set c = 2π/41, initialised the variables at t = 1 randomly, and generated 40 further observations. We have also applied our method to three gene expression time series of the Interferon regulatory factors Irf1, Irf2 and Irf3 from bone marrow derived macrophages, which we analysed in our earlier work [8]. Data of Irf1, Irf2 and Irf3 were collected at 25 × 30 minute time intervals under three external conditions: (1) infection with Cytomegalovirus (CMV), (2) treatment with Interferon Gamma (IFNγ), and (3) infection with Cytomegalovirus after pre-treatment with IFNγ (CMV+IFNγ). Finally we have applied our method to two gene expression time series from Arabidopsis thaliana cells, which were sampled at 13 × 2 hour time intervals. As in [8] we focus on 9 genes: LHY, CCA1, TOC1, ELF4, ELF3, GI, PRR9, PRR5, and

120

M. Grzegorczyk and D. Husmeier

PRR3, which are known to be involved in circadian regulation [8]. The expressions were measured independently in 2 plants under experimentally generated constant light condition but with diﬀerent pre-histories. In the experiment T20 (T28 ), the plant was entrained in a 10h:10h (14h:14h) light/dark-cycle.

5

Simulations

In all our simulations, data were standardised to zero mean and marginal variance of 1 for all dimensions. The hyperparameters of the normal-Wishart prior were chosen as uninformative as possible subject to certain regulatory conditions discussed in [3]. For the synthetic (real) data we set both the burn-in and the sampling-phase lengths of our MCMC simulations to 50,000 (500,000) each and sampled every 1,000 iterations during the sampling-phase1 . For the real data we started 5 independent MCMC simulations from diﬀerent initialisations on each data set, and we computed the potential scale reduction factor (PSRF) based on the marginal edge posterior probabilities to monitor convergence. As we observed a suﬃcient degree of convergence for all these data sets (P SRF < 1.2), we report only the results of the empty-seeded MCMC runs. For the evaluation of the results, we proceeded in diﬀerent ways. For the synthetic study based on the two-node network of Figure 1, we computed the marginal posterior probabilities for the individual edges. Our main interest was to test our conjecture from Sect. 2 that the BGe score is susceptible to inferring a spurious self loop for node Y . We wanted to test whether this susceptibility could be reduced by the proposed Mix-BGe model. For the second synthetic simulation study, we assessed the network reconstruction accuracy via the area under the ROC (receiver operator characteristic) curve: AUC; this is a standard criterion that has been applied in numerous related articles. In this study we also compare the Mix-BGe model with our BGM model [8], whereby we exchanged the random allocation of the latent variables of our original BGM model by a change-point process. This slight modiﬁcation ensures a fair comparison, and actually improved the performance of the BGM model on the synthetic data sets. Finally, for the real data, we focused on the self-loops again. Since we do not know the true network for the Arabidopsis thaliana data, computing AUC scores is impossible. However, as discussed in Sect. 2, we conjecture that in the presence of temporal autocorrelation in the signals of the (unknown) regulators, many downstream nodes will show spurious self-loops when the network is reconstructed with the BGe model, whereas this susceptibility should be reduced with the proposed Mix-BGe model. We therefore take as an alternative ﬁgure of merit the average marginal posterior probability of a self-loop compared to the average marginal posterior probability of a non-self-loop: ξ =

Nsl Nnl 1 1 P (esl |D) − P (enl |D) Nsl Nnl sl=1

1

(13)

nl=1

We note that even for the largest network with N = 9 nodes each single simulation was c code on a SunFire accomplished within a few hours using non-optimised Matlab X4100M2 machine with AMD Opteron 2224 SE dual-core processor.

Avoiding Spurious Feedback Loops

121

where esl is an edge corresponding to a self-loop, enl is an edge corresponding to a non-self-loop, Nsl is the total number of self loops, and Nnl is the total number of non-self-loops. Lower values of ξ are taken as an indication of a better performance.

6

Results

Panels (a) and (b) of Figure 2 show the AUC scores for the synthetic data. Both ﬁgures are laid out as matrices, in which the rows and columns correspond to diﬀerent standard deviations of the noise in X (rows) and in Y (columns). Note that an increase of the noise in X reduces the autocorrelation of X, while increasing the noise in Y blurs the functional dependence of Y (t + 1) on X(t). The autocorrelation of Y is jointly inﬂuenced by both noise levels. The proposed Mix-BGe model consistently outperforms both the linear BGe model, as well as the common change-point BGM model of Grzegorczyk et al. [8]. Only when the noise in the signal of the hub node X is large (right-most columns) does the proposed Mix-BGe model fail to achieve an improvement. As discussed above, this is a consequence of an increased mis-classiﬁcation of latent variables, owing to the nature of the change-point process. This could in principle be addressed by combining the node-speciﬁc change-points of the present paper with the allocation sampler used in [8] - albeit at the cost of a greatly inﬂated conﬁguration space in latent space. Panels (c) and (d) in Figure 2 show the marginal posterior probabilities of the four possible edges in the two-node network of Figure 1 and the non-linear state space process of Eq. (1). The results in panel (c) of Figure 2 were obtained with the linear BGe model and show a clear propensity for inferring the spurious self-loop Y → Y , in conﬁrmation of our earlier conjecture (see Sect. 2). Compare this with the results for the proposed Mix-BGe model, shown in panel (d) of Figure 2. Here, the spurious self-loop Y → Y is suppressed in favour of the correct edge X → Y . There are two noise regimes in which the spurious self-loop Y → Y has a marginal posterior probability that is higher than or equal to that of the correct edge X → Y . One noise regime is where both noise levels in X and in Y are low (top left corner in panels (c) and (d) of Figure 2). Here, the autocorrelation of Y is so high that the spurious selfloop Y → Y is still favoured over the true edge X → Y ; this is a consequence of the fact that the functional dependence of Y (t + 1) on X(t) is only learned approximately (namely approximated by a mixture model). The second regime is where both noise levels are high (top right corners in the panels (c) and (d) of Figure 2). High noise in Y blurs the functional dependence of Y (t + 1) on X(t), while high noise in X leads to a high mis-classiﬁcation of latent variables and, consequently, a deterioration of the model accuracy; this is a consequence of the fact that latent variables are not allocated individually, as in [8], but according to a change-point process. However, in the majority of noise scenarios, the marginal posterior probability of the correct edge X → Y is signiﬁcantly higher than that of the self-loop Y → Y . This suggests that the proposed MixBGe model is successful at suppressing spurious feedback loops. Finally, Figure 3 shows the results for the real data. It is seen that for four out of ﬁve data sets,

122

M. Grzegorczyk and D. Husmeier σY=0.25

σY=0.5

σy=1

1

1

0

0

0

0

0

1

1

1

1

1

1

0

0

0

0

0

0

0

1

1

1

1

1

1

1

1

0

0

0

0

0

0

0

0

1

1

1

1

1

1

1

1

0

0

0

0

0

0

0

1

1

0

σX=0.25

0

σX=0.5

0

σx=0.1

1

1

0

(a) AUC for N=2 σY=0.5

σY=1

1

1

1

1

0

0

0

0

1

1

1

1

0

0

0

1

1

1

0

0

0

1

1

1

0

0

0

0

σX=0.1

σX=0.25

σX=0.5

σX=1

1

1

1

0

0

0

0

1

1

1

1

0

0

0

0

0

1

1

1

1

1

0

0

0

0

0

1

1

1

1

1

0

0

0

0

σY=0.5 σY=1

σX=1

σX=0.5

1

σY=0.1

σY=0.25

(b) AUC for N=4

σY=0.25

σY=0.1 σX=0.1

σy=0.5

1

1

0

σX=0.25

σy=0.25

σy=0.1

σY=1

1

σX=1

σX=1

σX=0.5

σX=0.25

σX=0.1

σY=0.1

1

(c) BGe edge posteriors (N=2)

(d) MIX-BGe edge posteriors (N=2)

Fig. 2. AUC scores and marginal edge posterior probabilities for the synthetic data. All four panels are laid out as matrices, whose cells correspond to standard deviations σX and σY of the noise in X and Y (or Yi ). All histograms show averages (means/std. deviations) from 20 independent data instantiations. Panels (a) and (b) were obtained for data generated from the 4-node-network described in Section 4 and show average AUC score histograms for BGe (left grey bar) and MIX-BGe (right white bar). The centre black bar shows the AUC score for a mixture model in which each change-point applies to all the variables, as proposed in [8]. In (c) and (d) the histograms show the posterior probabilities of the edges in the 2-node network, as obtained with BGe (c) and the MIX-BGe model (d). Each histogram contains 4 bars, which represent the average posterior probabilities of the 4 possible edges: Left: self-loop X → X (true); centre left: X → Y (true); centre right: self-loop Y → Y (false); right: Y → X (false). It is seen that BGe has a high propensity for learning the spurious feedback loop Y → Y , while MIX-BGe tends to learn an increased probability of the correct edge X → Y (centre left bars).

Avoiding Spurious Feedback Loops

123

0.15 0.15

0.5

0.5

0.5 0.1

0.25

0

0.25

BGE

MIX

0

(a) CMV

0.25

BGE

MIX

(b) IFNγ

0

0.1 0.05

0.05

BGE

MIX

(c) CMV+IFNγ

0

BGE

MIX

(d) T20

0

BGE

MIX

(e) T28

Fig. 3. Results on the Interferon and Arabidopsis thaliana gene expression time series. The histograms shows the self-loop score ξ of Eq.(13) for the BGe model (dark bar) and the proposed MIX-BGe model (light bar). Lower values are taken as an indication of a better performance. The histograms in (a)-(c) were obtained from the Interferon regulatory factor gene expression time series: (a) infection with CMV, (b) pre-treatment with IFNγ, and (c) infection and pre-treatment. The histograms in (d)-(e) were obtained from the Arabidopsis thaliana gene expression time series: (d) 10h:10h light/dark entrainment, (e) 14h:14h light/dark entrainment.

employing the proposed Mix-BGe model leads to a signiﬁcant suppression of the marginal posterior probabilities of potentially spurious self loops. We note that the chosen criterion is not based on a proper gold-standard, as we do not know the true number of genuine feedback loops. The diﬀerence between the results shown in panels (d) and (e) of Figure 3 may appear as surprising. It cannot be ruled out, though, that the entrainment with diﬀerent light-dark cycles may indeed lead to the activation of diﬀerent recurrent pathways, especially given that the interactions between day and evening genes in Arabidopsis thaliana is intrinsically of a recurrent nature [12]. Overall, our ﬁndings summarised in Figure 3 point to the general reduction of the posterior probability of feedback loops inferred with Mix-BGe as compared with BGe. Given that BGe is intrinsically susceptible to inferring spurious feedback loops, as discussed in Section 2, this points to a potential improvement in the network reconstruction accuracy.

7

Discussion

We have demonstrated that when learning dynamic Bayesian networks from time series data, the presence of temporal autocorrelations in the signals of the regulating nodes renders an approach based on the linear BGe score susceptible to spurious feedback loops. We have proposed a non-linear generalisation of the BGe score based on a mixture model and node-speciﬁc change-point processes; this is also a generalisation of the BGM model, where the allocation of data points to mixture components was not node-speciﬁc, but aﬀected all nodes simultaneously [8]. Our simulations have shown that the network reconstruction accuracy is improved, and that spurious feedback loops are avoided. We note that there is a close similarity between our model and the one proposed in [13]. The essential diﬀerence is that the model in [13] learns separate network structures for diﬀerent time series segments. This assumption is reasonable for some scenarios, like morphogenesis. However, for most cellular processes on a shorter time scale, it is questionable whether it is the structure rather than

124

M. Grzegorczyk and D. Husmeier

just the strength of the regulatory interactions that changes with time. The practical problem is potential model over-ﬂexibility. Owing to the high costs of postgenomic high-throughput experiments, time series in systems biology are typically rather short. Modelling short time series segments with separate network structures will almost inevitably lead to inﬂated inference uncertainty. For that reason we have constrained the network structure to remain invariant and only allow the interaction parameters to change. As a direction for future research one might consider the implementation of a hybrid scheme with a soft rather than hard constraint on the network structures, based on the hierarchical Bayesian model proposed in [14].

References 1. Friedman, N., Linial, M., Nachman, I., Pe’er, D.: Using Bayesian networks to analyze expression data. Journal of Computational Biology 7, 601–620 (2000) 2. Cooper, G.F., Herskovits, E.: A Bayesian method for the induction of probabilistic networks from data. Machine Learning 9, 309–347 (1992) 3. Geiger, D., Heckerman, D.: Learning Gaussian networks. In: Proceedings of the Tenth Conference on Uncertainty in Artiﬁcial Intelligence, pp. 235–243 (1995) 4. Imoto, S., Kim, S., Goto, T., Aburatani, S., Tashiro, K., Kuhara, S., Miyano, S.: Bayesian network and nonparametric heteroscedastic regression for nonlinear modeling of genetic network. Journal of Bioinformatics and Computational Biology 1(2), 231–251 (2003) 5. Ko, Y., Zhai, C., Rodriguez-Zas, S.L.: Inference of gene pathways using Gaussian mixture models. In: BIBM International Conference on Bioinformatics and Biomedicine, pp. 362–367. Fremont, CA (2007) 6. Schwarz, G.: Estimating the dimension of a model. Annals of Statistics 6(2), 461– 464 (1978) 7. Nobile, A., Fearnside, A.T.: Bayesian ﬁnite mixtures with an unknown number of components: The allocation sampler. Statistics and Computing 17(2), 147–162 (2007) 8. Grzegorczyk, M., Husmeier, D., Edwards, K.D., Ghazal, P., Millar, A.J.: Modelling non-stationary gene regulatory processes with a non-homogeneous Bayesian network and the allocation sampler. Bioinformatics 24, 2071–2078 (2008) 9. Madigan, D., York, J.: Bayesian graphical models for discrete data. International Statistical Review 63, 215–232 (1995) 10. Green, P.J.: Reversible jump Markov chain Monte Carlo computation and Bayesian model determination. Biometrika 82(4), 711–732 (1995) 11. Gelman, A., Rubin, D.B.: Inference from iterative simulation using multiple sequences. Statistical Science 7(4), 457–472 (1992) 12. Salome, P., McClung, C.: The Arabidopsis thaliana clock. Journal of Biological Rhythms 19, 425–435 (2004) 13. L`ebre, S.: Analyse de processus stochastiques pour la g´enomique: ´etude du mod`ele ´ MTD et inf´erence de r´eseaux bay´esiens dynamiques. PhD thesis. Evry-Val-d’Essone (2008) 14. Werhli, A.V., Husmeier, D.: Gene regulatory network reconstruction by Bayesian integration of prior knowledge and/or diﬀerent experimental conditions. Journal of Bioinformatics and Computational Biology 6(3), 543–572 (2008)

Ligand Electron Density Shape Recognition Using 3D Zernike Descriptors Prasad Gunasekaran1, , Scott Grandison1, , Kevin Cowtan2 , Lora Mak3 , David M. Lawson4, and Richard J. Morris1 1

3 4

Department of Computational & Systems Biology, John Innes Centre, Norwich Research Park, Colney Lane, NR4 7UH Norwich, UK 2 Structural Biology Laboratory, Department of Chemistry, University of York, Heslington, York, UK YO10 5DD Theoretical Systems Biology, Institute of Food Research, Norwich Research Park, Colney Lane, NR4 7UA Norwich, UK Department of Biological Chemistry, John Innes Centre, Norwich Research Park, Colney Lane, NR4 7UH Norwich, UK [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]

Abstract. We present a novel approach to crystallographic ligand density interpretation based on Zernike shape descriptors. Electron density for a bound ligand is expanded in an orthogonal polynomial series (3D Zernike polynomials) and the coeﬃcients from this expansion are employed to construct rotation-invariant descriptors. These descriptors can be compared highly eﬃciently against large databases of descriptors computed from other molecules. In this manuscript we describe this process and show initial results from an electron density interpretation study on a dataset containing over a hundred OMIT maps. We could identify the correct ligand as the ﬁrst hit in about 30 % of the cases, within the top ﬁve in a further 30 % of the cases, and giving rise to an 80 % probability of getting the correct ligand within the top ten matches. In all but a few examples, the top hit was highly similar to the correct ligand in both shape and chemistry. Further extensions and intrinsic limitations of the method are discussed. Keywords: pattern recognition, structural bioinformatics, electron density, protein crystallography, 3D Zernike moments.

1

Introduction

With the success of structural genomics worldwide and the recognised importance of 3D information in unravelling protein function and enzymatic mechanism, computational methods are increasingly serving a vital role in structural biology. From target selection (using sequence analysis, phlyogeny, fold recognition and homology modeling), registration, laboratory information management

Equal contributing ﬁrst author.

V. Kadirkamanathan et al. (Eds.): PRIB 2009, LNBI 5780, pp. 125–136, 2009. c Springer-Verlag Berlin Heidelberg 2009

126

P. Gunasekaran et al.

systems, crystal optimisation, data collection and analysis, the solution of the phase problem (often using molecular replacement techniques with databases of fragments or homology models), to complete and validated models and the prediction of protein function, computational biology has a major role to play. Automated model building into crystallographic electron density has come a long way in the last decade. Although many highly sophisticated algorithms and graphical approaches existed to aid the model building process, prior to the release of packages such as ARP/wARP [1], RESOLVE [2], PHENIX [3], and BUCCANEER [4], the construction of a macromolecular model into electron density often required many months of expert crystallographer time. Electron density interpretation has always been driven by pattern recognition [5]. Initially this process was carried out by the human brain based on structural insights and experience, but alternative approaches based on algorithmic pattern recognition developments such as skeletonizaton [6] quickly reduced the demand on human eﬀort. Automated methods [7,4,8,9,10,11,12,13,14] contributed greatly to and beneﬁtted hugely from structural genomics initiatives and recent developments [15,16,17,18,4] have reached an impressive level of sophistication, providing an intelligent expert system approach to model building for protein structures. Methods for determining structures from small molecule crystallographic data are completely automated and very well advanced. The success of these techniques relies strongly on direct methods, Patterson approaches and suﬃcient data to warrant atomicity [19,20], thus allowing identiﬁed atoms to be connected and the molecule to be built in a rather straightforward manner. These methods break down with decreasing resolution. Whereas proteins have regular repeating elements that can be distinguished already at lower resolution, ligands exhibit less pronounced and regular patterns, making their interpretation problematic. Thus, automated building of protein ligands into macromolecular structures is by comparison less well developed than for proteins, and DNA/RNA molecules seem to have either been ignored or successfully resisted automation. However, a number of original, diverse, and interesting approaches have been developed for the identiﬁcation and modeling of bound ligands. Zwart et al. [21] describe methods for automated identiﬁcation and building of protein-bound ligands in electron-density maps. Their approach relies on a graph-based search for geometrical features which match stereochemical expectations and an atomic labeling algorithm which explores the combinatorics of the problem. This approach has recently been assessed [22] and has shown to be most successful for high resolution data. Based on his powerful methods for density modiﬁcation using structural motifs [23], Terwilliger successfully enhanced and extended the approach to ligand ﬁtting [24]. This technique relies on the placement of core fragments into electron density and then uses a novel procedure to extend and build the remainder of the ligand. The authors performed extensive testing over a large dataset and found their methods functioned well, placing the ligand within 2 ˚ A r.m.s.d of the coordinates in the original structure. Recently, a new development [25] has been described which allows for the identiﬁcation of ligands bound in crystal structures of

Ligand Electron Density Shape Recognition

127

macromolecules. This method uses density and ﬁngerprint correlations. The density correlation is computed from the density after optimization of each entry of a test set of ligands and the ligand density. The ﬁngerprint correlations are lists of computed model density features between the ligands. An interesting development geared towards structure-based drug design is presented in [26]. Their approach aims to automate protein-ligand crystallography for drug design. Ligands are positioned by docking directly into electron density, whilst taking care of protein-ligand interactions. In addition to the identiﬁcation of full ligands, the authors show the potential for analysing fragment-library screening experiments. A force-ﬁeld approach can be included to provide a better ﬁt whilst maintaining good geometry [27]. An alternative approach captures the central axis of the electron density isosurface with a graph. This graph is matched against a graph of the molecular model [28]. This approach was reported to work as well as density peak picking methods and showed promise for the extension to lower resolution [28]. In a novel development using rotation-invariant features derived from moments of various orders, Lamzin and co-workers [29] developed a powerful method for accurately locating planar objects in electron density. This approach was tested on protein and DNA/RNA crystal structures and was able to place the plane centres within 0.5 ˚ A of the correct position. Although not speciﬁcally designed for ligand density interpretation, this approach seems a promising strategy and has much in common with our methodology. In this manuscript we present a method for ligand identiﬁcation based on global pattern recognition using 3D Zernike descriptors. Zernike polynomial techniques have recently been applied with success in the biomolecular sciences [30,31,32]. The advantages of this approach include accurate feature representation and speed of comparison, meaning that large libraries can be scanned eﬃciently. Although for the current application the top hits already identiﬁed the correct ligand in many cases, a more promising set up might be to combine this technique with other methods such as those mentioned above to build a hierarchical search system which ﬁlters quickly through large databases and progressively invests more time only in relatively few promising candidates.

2

Methodology

An overview of the ligand density identiﬁcation method described in this manuscript is shown in Figure 1. The individual steps are explained in detail below. To evaluate the success of this approach, we chose datasets of experimental crystallographic density which we attempted to interpret. The success was judged by how many times we predicted the correct ligand and, in cases where we got the wrong ligand, how distant our prediction was from the correct solution. 2.1

Density Extraction

Clipper library routines [33] were used to 1) compute crystallographic OMIT maps (OMIT maps show the diﬀerence between the density computed from the

128

P. Gunasekaran et al.

Fig. 1. Flowchart of the 3D Zernike descriptor method for ligand density interpretation. Crystallographic diﬀerence density was computed using Clipper routines and passed through our region growing algorithm. The resulting density is projected into Zernike space (the individual colours in the multi-colour sphere representation correspond to diﬀerent Zernike polynomials). The Zernike descriptors are scanned against a database of pre-computed descriptors from existing ligands and the best hit is returned as the most probable ligand.

observed crystallographic structure factors and the theoretical density from a given arrangement of atoms and can be used to spot those parts of the density that have not been modeled or contain errors) by leaving out all the ligands from the model; 2) compute a crystallographic R-factor to check the density was reasonable; 3) to extract an orthogonal grid of diﬀerence density around each ligand. Such electron density maps are shown in the left column in Figures 2 and 3. Using OMIT maps based on models that have ’seen’ the ligand during

Ligand Electron Density Shape Recognition

129

Fig. 2. Density, segmentation and ligands. A selection of density maps for which the 3D Zernike descriptor ligand identiﬁcation gave the correct answer.

reﬁnement introduces a bias towards improved ligand density and yet these maps are by nature often noisy and contain errors from missing data and/or poorly modelled parts of the structure. To address this, we implemented a regiongrowing algorithm following the approach described in [34]. This method allows for the automated segmentation of 3D images. The algorithm starts from a seed point and progresses to visit neighbouring pixels, marks them, and either adds them to the current region list, or not, depending on the deﬁned growth criteria. As the seed point we chose the highest point in the OMIT map and as growth criteria we deﬁned that at least two neighbouring points to be above a given threshold. The choice of threshold was map dependent and our selected values ranged from 1.2 to 2.0 σ (standard deviation of the electron density values). The threshold was varied to produce the most ligand-like shape as judged by the human eye. The average threshold value was about 1.4 σ, so this value was employed for automated density extraction. Images of automatically extracted diﬀerence density are shown in the second column of Figure 2 and 3. 2.2

Zernike Moments

The success of 3D Zernike moments for object recognition in computer science has been documented in the number of publications, outperforming many other well-established techniques in the area. Following developments by [35] and [36] for 3D shape retrieval, we have employed 3D Zernike polynomials as a basis set for molecular shape comparison [30]. This approach was initially used to describe and compare binary objects, but the underlying mathematics are suﬃciently general to allow any function in 3D to be described within the same framework.

130

P. Gunasekaran et al.

Fig. 3. Crystallographic diﬀerence densities, extracted ligand densities and the correct ligands. A selection of density maps for which the correct answer was not obtained. For 1JKX another conformation was ranked top, for 1RY8 the density segmentation picked up only half the ligand density because of a density break, and for 1E2K the region growing algorithm missed the actual ligand density.

As demonstrated in [31] and [37], the Zernike descriptor approach is well-suited for molecular comparisons and can be adapted to a range of physicochemical properties. We recently demonstrated the application of 3D Zernike moments to capture model uncertainty and protein ﬂexibility [32]. A detailed description of the method for matching shapes using Zernike moments has been presented in [35,36,30]. We summarise some of the main points below. Any square-integrable function on the unit ball, f , can be represented as f (r) =

∞ n l

cnlm Znlm (r),

(1)

n=0 l=0 m=−l

in which Znlm (r) are the 3D Zernike polynomials given here as functions of the spherical coordinates, r = (r, θ, φ). The 3D Zernike polynomials are basis functions on the unit ball consisting of a radial term, Rnl (r), and an angular term, Ylm (θ, φ). The angular functions, Ylm (r), are known as spherical harmonics and can themselves be used for molecular shape recognition purposes [38,39]. The function indices are denoted by nlm, where n ranges from 0 to the maximum expansion order, l ≤ n and −l ≤ m ≤ l. The radial part of the Zernike polynomials is given by (n−l)/2 Rnl (r) = Nnlk rn−2k (2) k=0

Ligand Electron Density Shape Recognition

131

for n − l even, otherwise zero. Nnlk is a normalisation constant. See [40,41] for further details and cartesian transformations. The expansion coeﬃcients, cnlm , are called 3D Zernike moments. The determination of the 3D Zernike moments requires the integration of the function of interest, f (r), multiplied by ∗ the complex conjugate Zernike polynomials, Znlm (r), over the unit ball, 1 2π cnlm = 0

0

0

π

∗ Znlm (r)f (r)r2 sin θ dθ dφ dr.

(3)

The symmetry properties of the spherical harmonics induce a symmetry in the coeﬃcients between positive and negative m indices, cnl,−m = (−1)m c∗nlm . The symbol ∗ denotes the complex conjugate. Rotationally invariant descriptors may be constructed from the moments by deﬁning (2l + 1)-dimensional vectors for each index l with all the m indexed coeﬃcients, cnlm , making up each subspace, cnl,−l cnl,−l+1 Fnl = (4) . .. . cnl,l These rotationally invariant coeﬃcients, Fnl , provide an eﬃcient way of comparing shapes, a and b, using a standard L2 metric, nmax n a − F b )2 , d= (Fnl (5) nl n=0 l=0

for a maximum expansion order equal to nmax . Thus, in terms of ligand identiﬁcation the Zernike descriptors can be employed to ﬁnd promising molecules of similar shape to the crystallographic density without the need for translational or rotational searches and optimisation. Once potential candidates have been identiﬁed a more sensitive method could be used, such as [8], and coupled with reﬁnement.

3 3.1

Datasets Electron Density Maps

We downloaded a selection of PDB structures with their crystallographic structure factors. In total we selected 110 structures in a resolution range of 1.5 ˚ A to 2.8 ˚ A, which gave rise to 586 ligand density maps. Only maps corresponding to ligands with more than 10 non-hydrogen atoms were selected, leaving us with 164 maps in total. The size of these ligands varies from 10 to 111 atoms, with the average being 22.7 non-hydrogen atoms.

132

3.2

P. Gunasekaran et al.

Ligand Database

We downloaded the latest version, release 12.1, of the HIC-Up database [42]. This version contains 7870 small molecules extracted from the PDB. The 3D Zernike moments were computed following the procedure outlined in Grandison et al. [32]. In short, each molecule was projected onto the unit ball such that the largest distance from the centre of geometry was scaled to 0.6. The atoms were projected onto a 643 grid using a Gaussian function with a variance deﬁned by the van der Waals radius. For each voxel the geometric moments were computed from which the Zernike moments were determined. The Zernike moments were used to compute rotationally-invariant descriptors, following Equation 4.

4

Results

For each density map we applied the noise reduction algorithm described in the methods section to choose a continuous high density (> 1.5σ) region. This density was expanded in 3D Zernike polynomials to an order or 30 and the rotationally invariant shape descriptors were computed. These shape descriptors were scanned against a database consisting of 7870 pre-computed small molecule shape descriptors and the 164 ligand shape descriptors from the PDB ﬁles used for the density maps. Surprisingly, the density shape was not able to recognise its own ligand in a large number of cases. There are a number of reasons for this. First, although the region-growing algorithm performed well overall, it failed to extract the correct region in almost 30 % of the cases, see Figure 3, due to poor density. There are numerous ways to address this problem but for the

Fig. 4. This plot shows the frequency of correctly identiﬁed ligands (y-axis) as a function of the number of selected molecules (x-axis) over the entire data set. A perfect method would give 1.0 already for the number of molecules equal to one (this is the case for our approach using computed ligand densities from the ligand model). Poor density and insuﬃciently powerful segmentation and image enhancement algorithms limit the performance in our test cases. For real applications the conformational ﬂexibility of ligands will reduce the performance further.

Ligand Electron Density Shape Recognition

133

present analysis we simply excluded those cases in order to evaluate the shape matching procedure rather than the region growing algorithm. Second, in cases where the region-growing procedure performed well, the density was still poor in places and resulted in parts being missed (and other ligands matching better) or moved (resulting in correct ligands in diﬀerent conformations being the best hit). Another surprising ﬁnding is that a higher expansion order often decreases the performance. In [38] we found that for shape matching purposes using spherical harmonics a maximum expansion order of about 6 was suﬃcient and the performance increase for higher order was negligible. Similarly for Zernike descriptors [30], we found an order of about 10 to be suﬃcient for molecule classiﬁcation and that the performance increased only marginally overall with higher orders, dropping slightly in only a few cases for protein comparisons. Here, the higher orders are capturing more noise in the density, whereas the lower orders are making good approximations to the overall shape and thereby performing noise reduction. We found the ligand interpretation performance peaked at a maximum expansion order of 8.

5

Discussion

We have described a novel method for the interpretation of electron density which can be used for the identiﬁcation of ligands in crystallographic electronic diﬀerence density. We have performed an initial analysis on a realistic test dataset which covers a range of ligands from experimental crystallographic data of varying resolution and map quality. We have developed methods to automatically extract the diﬀerence density maps, place them on an orthogonal grid, expand them in 3D Zernike polynomials, and to eﬃciently compare their rotationally invariant moment against other shape databases. Overall, the performance was fair but not outstanding. We could identify the correct ligand in 30 % cases as the top hit and in a further 30 % cases within the top ﬁve matches. Despite encouraging results, several points became apparent during this analysis that require further investigation before the method can be properly automated. The ﬁrst point is the density extraction itself. Advanced image processing tools, more speciﬁcally better techniques for image segmentation such a sophisticated watershed transform, may provide improve this step. Another option would be to allow the user to manually select and modify the density (via the placement of atoms) in existing model building graphics software. However, even with the perfect ligand density this method has limitations in that it performs static global shape matching, although multi-conformer searching presents a tractable solution given the speed of comparison. With the current test set, the conformational diversity was limited and the full eﬀect of this limitation is therefore reduced. This conformational sampling issue could represent a problem, although initial analyses suggest that the number of bound conformations of ﬂexible ligands is rather small [43]. We stress the point that the use of OMIT maps based on models that have ’seen’ the ligand during reﬁnement implies that the current results will be better than in a more realistic scenario with poorer ligand density.

134

P. Gunasekaran et al.

Overall, the performance in selecting the correct ligand as the top hit did not meet our expectations. For this dataset the correct match was usually within the top twenty and with 80 % frequency within the top ten hits. In 30 % of the cases, we could identity the correct ligand with the ﬁrst hit. However, this approach does act as an impressive ﬁlter and delivers a small set of potential candidates. More elaborate methods could then be employed to select the most likely ligand from this set. We therefore suggest that this approach would be best-suited as an eﬃcient pre-screening method, prior to launching more computationally costly techniques.

Acknowledgments The idea to this piece of work is based on discussions with Dr Victor S Lamzin (EMBL Hamburg) about using spherical harmonics to interpret ligand density shapes. We thank Abdullah Kahraman for discussion. This project was funded in part by a BBSRC Tools and Resources grant - grant number CA340H10B.

References 1. Perrakis, A., Morris, R., Lamzin, V.S.: Automated protein model building combined with iterative structure reﬁnement. Nat. Struct. Biol. 6(5), 458–463 (1999) 2. Terwilliger, T.C.: Solve and resolve: automated structure solution and density modiﬁcation. Methods Enzymol. 374, 22–37 (2003) 3. Adams, P.D., Grosse-Kunstleve, R.W., Hung, L.W., Ioerger, T.R., McCoy, A.J., Moriarty, N.W., Read, R.J., Sacchettini, J.C., Sauter, N.K., Terwilliger, T.C.: Phenix: building new software for automated crystallographic structure determination. Acta Crystallogr. D Biol. Crystallogr. 58(Pt 11), 1948–1954 (2002) 4. Cowtan, K.: The buccaneer software for automated model building. 1. tracing protein chains. Acta Crystallogr. D Biol. Crystallogr. 62(Pt 9), 1002–1011 (2006) 5. Morris, R.J.: Statistical pattern recognition for macromolecular crystallographers. Acta Crystallogr. D Biol. Crystallogr. 60(Pt 12 Pt 1), 2133–2143 (2004) 6. Greer, J.: Computer skeletonization and automatic electron density map analysis. Methods Enzymol. 115, 206–224 (1985) 7. Cowtan, K.: Fast fourier feature recognition. Acta Crystallogr. D Biol. Crystallogr. 57(Pt 10), 1435–1444 (2001) 8. Cowtan, K.: Fitting molecular fragments into electron density. Acta Crystallogr. D Biol. Crystallogr. 64(Pt 1), 83–89 (2008) 9. Morris, R.J., Perrakis, A., Lamzin, V.S.: ARP/wARP model-building algorithms. i. the main chain. Acta Crystallogr. D Biol. Crystallogr. 58(Pt 6 Pt 2), 968–975 (2002) 10. Morris, R.J., Perrakis, A., Lamzin, V.S.: ARP/wARP and automatic interpretation of protein electron density maps. Methods Enzymol. 374, 229–244 (2003) 11. Morris, R.J., Zwart, P.H., Cohen, S., Fernandez, F.J., Kakaris, M., Kirillova, O., Vonrhein, C., Perrakis, A., Lamzin, V.S.: Breaking good resolutions with ARP/wARP. J. Synchrotron. Radiat. 11(Pt 1), 56–59 (2004)

Ligand Electron Density Shape Recognition

135

12. Adams, P.D., Gopal, K., Grosse-Kunstleve, R.W., Hung, L.W., Ioerger, T.R., McCoy, A.J., Moriarty, N.W., Pai, R.K., Read, R.J., Romo, T.D., Sacchettini, J.C., Sauter, N.K., Storoni, L.C., Terwilliger, T.C.: Recent developments in the phenix software for automated crystallographic structure determination. J. Synchrotron. Radiat. 11(Pt 1), 53–55 (2004) 13. Terwilliger, T.C.: Automated structure solution, density modiﬁcation and model building. Acta Crystallogr. D Biol. Crystallogr. 58(Pt 11), 1937–1940 (2002) 14. Terwilliger, T.: Solve and resolve: automated structure solution, density modiﬁcation and model building. J. Synchrotron. Radiat. 11(Pt 1), 49–52 (2004) 15. Cohen, S.X., Morris, R.J., Fernandez, F.J., Ben Jelloul, M., Kakaris, M., Parthasarathy, V., Lamzin, V.S., Kleywegt, G.J., Perrakis, A.: Towards complete validated models in the next generation of ARP/wARP. Acta Crystallogr. D Biol. Crystallogr. 60(Pt 12 Pt 1), 2222–2229 (2004) 16. Terwilliger, T.C., Grosse-Kunstleve, R.W., Afonine, P.V., Moriarty, N.W., Zwart, P.H., Hung, L.W., Read, R.J., Adams, P.D.: Iterative model building, structure reﬁnement and density modiﬁcation with the phenix autobuild wizard. Acta Crystallogr. D Biol. Crystallogr. 64(Pt 1), 61–69 (2008) 17. Joosten, K., Cohen, S.X., Emsley, P., Mooij, W., Lamzin, V.S., Perrakis, A.: A knowledge-driven approach for crystallographic protein model completion. Acta Crystallogr. D Biol. Crystallogr. 64(Pt 4), 416–424 (2008) 18. Terwilliger, T.C., Grosse-Kunstleve, R.W., Afonine, P.V., Moriarty, N.W., Adams, P.D., Read, R.J., Zwart, P.H., Hung, L.W.: Iterative-build omit maps: map improvement by iterative model building and reﬁnement without model bias. Acta Crystallogr. D Biol. Crystallogr. 64(Pt 5), 515–524 (2008) 19. Morris, R.J., Bricogne, G.: Sheldrick’s 1.2 ˚ A rule and beyond. Acta Crystallogr. D Biol. Crystallogr. 59(Pt 3), 615–617 (2003) 20. Morris, R.J., Blanc, E., Bricogne, G.: On the interpretation and use of < |E|2 > (d∗) proﬁles. Acta Crystallogr. D Biol. Crystallogr. 60(Pt 2), 227–240 (2004) 21. Zwart, P.H., Langer, G.G., Lamzin, V.S.: Modelling bound ligands in protein crystal structures. Acta Crystallogr. D Biol. Crystallogr. 60(Pt 12 Pt 1), 2230–2239 (2004) 22. Evrard, G.X., Langer, G.G., Perrakis, A., Lamzin, V.S.: Assessment of automatic ligand building in arp/warp. Acta Crystallogr. D Biol. Crystallogr. 63(Pt 1), 108– 117 (2007) 23. Terwilliger, T.C.: Maximum-likelihood density modiﬁcation using pattern recognition of structural motifs. Acta Crystallogr. D Biol. Crystallogr. 57(Pt 12), 1755– 1762 (2001) 24. Terwilliger, T.C., Klei, H., Adams, P.D., Moriarty, N.W., Cohn, J.D.: Automated ligand ﬁtting by core-fragment ﬁtting and extension into density. Acta Crystallogr. D Biol. Crystallogr. 62(Pt 8), 915–922 (2006) 25. Terwilliger, T.C., Adams, P.D., Moriarty, N.W., Cohn, J.D.: Ligand identiﬁcation using electron-density map correlations. Acta Crystallogr. D Biol. Crystallogr. 63(Pt 1), 101–107 (2007) 26. Mooij, W.T.M., Hartshorn, M.J., Tickle, I.J., Sharﬀ, A.J., Verdonk, M.L., Jhoti, H.: Automated protein-ligand crystallography for structure-based drug design. Chem. Med. Chem. 1(8), 827–838 (2006) 27. Wlodek, S., Skillman, A.G., Nicholls, A.: Automated ligand placement and reﬁnement with a combined force ﬁeld and shape potential. Acta Crystallogr. D Biol. Crystallogr. 62(Pt 7), 741–749 (2006)

136

P. Gunasekaran et al.

28. Aishima, J., Russel, D.S., Guibas, L.J., Adams, P.D., Brunger, A.T.: Automated crystallographic ligand building using the medial axis transform of an electrondensity isosurface. Acta Crystallogr. D Biol. Crystallogr. 61(Pt 10), 1354–1363 (2005) 29. Hattne, J., Lamzin, V.S.: Pattern-recognition-based detection of planar objects in three-dimensional electron-density maps. Acta Crystallogr. D Biol. Crystallogr. D64(Pt 8), 834–842 (2008) 30. Mak, L., Grandison, S., Morris, R.J.: An extension of spherical harmonics to regionbased rotationally invariant descriptors for molecular shape description and comparison. J. Mol. Graph. Model 26(7), 1035–1045 (2008) 31. Sael, L., Li, B., La, D., Fang, Y., Ramani, K., Rustamov, R., Kihara, D.: Fast protein tertiary structure retrieval based on global surface shape similarity. Proteins (2008) 32. Grandison, S., Roberts, C., Morris, R.J.: The application of 3d zernike moments for the description of “model-free” molecular structure, functional motion, and structural reliability. J. Comput. Biol. 16(3), 487–500 (2009) 33. Cowtan, K.: The clipper C++ libraries for x-ray crystallography. IUCr Computing Commission Newsletter 2, 4–9 (2003) 34. Revol-Muller, C., Peyrin, F., Carrillon, Y., Odet, C.: Automated 3d region growing algorithm based on an assessment function. Pattern Recogn. Lett. 23(1-3), 137–150 (2002) 35. Canterakis, N.: 3D zernike moments and zernike aﬃne invariants for 3D image analysis and recognition. In: Scandinavian Conference on Image Analysis (1999) 36. Novotni, M., Klein, R.: Shape retrieval using 3d zernike descriptors. ComputerAided Design 36(11), 1047–1062 (2004) 37. Sael, L., La, D., Li, B., Rustamov, R., Kihara, D.: Rapid comparison of properties on protein surface. Proteins 73(1), 1–10 (2008) 38. Morris, R.J., Najmanovich, R.J., Kahraman, A., Thornton, J.M.: Real spherical harmonic expansion coeﬃcients as 3d shape descriptors for protein binding pocket and ligand comparisons. Bioinformatics 21(10), 2347–2355 (2005) 39. Morris, R.J.: An evaluation of spherical designs for molecular-like surfaces. J. Mol. Graph. Model 24(5), 356–361 (2006) 40. Mathar, R.J.: Third order newton’s method for zernike polynomial zeros, arXiv: math.NA/0705.1329 (2007) 41. Mathar, R.J.: Zernike basis to cartesian transformations, arXiv: 0809.2368 math-ph (2008) 42. Kleywegt, G.J.: Crystallographic reﬁnement of ligand complexes. Acta Crystallogr. D Biol. Crystallogr. 63(Pt 1), 94–100 (2007) 43. Funkhouser, T., Glaser, F., Laskowski, R.A., Morris, R.J., Najmanovich, R., Stockwell, G., Thornton, J.M.: Shape-based classiﬁcation of bound ligands. In: Barber, S., Baxter, P.D., Mardia, K.V., Wells, R.E. (eds.) Quantitative Biology Shape Analysis and Wavelets, pp. 39–42 (2005)

Definition of Valid Proteomic Biomarkers: A Bayesian Solution Keith Harris1 , Mark Girolami1 , and Harald Mischak2 1

Inference Group, Department of Computing Science, University of Glasgow, UK {keithh,girolami}@dcs.gla.ac.uk http://www.dcs.gla.ac.uk/inference 2 Mosaiques Diagnostics and Therapeutics AG, Hannover, Germany

Abstract. Clinical proteomics is suﬀering from high hopes generated by reports on apparent biomarkers, most of which could not be later substantiated via validation. This has brought into focus the need for improved methods of ﬁnding a panel of clearly deﬁned biomarkers. To examine this problem, urinary proteome data was collected from healthy adult males and females, and analysed to ﬁnd biomarkers that diﬀerentiated between genders. We believe that models that incorporate sparsity in terms of variables are desirable for biomarker selection, as proteomics data typically contains a huge number of variables (peptides) and few samples making the selection process potentially unstable. This suggests the application of a two-level hierarchical Bayesian probit regression model for variable selection which assumes a prior that favours sparseness. The classiﬁcation performance of this method is shown to improve that of the Probabilistic K-Nearest Neighbour model. Keywords: Proteomic biomarkers, classiﬁcation, sparsity, feature selection, Bayesian inference.

1

Introduction

Proteins and peptides in body ﬂuids hold considerable information on the physiology of an organism and thus can serve as biomarkers for disease. However, the ﬁelds of biomarker discovery and clinical proteomics are suﬀering from high hopes generated by reports on potential biomarkers, most of which subsequently could not be substantiated via validation [1]. This development has resulted in much scepticism from both clinicians and regulatory agencies, which will make the application of valid biomarkers even more of a challenge. This vicious circle has to be broken by pinpointing the major errors made in earlier research and highlighting good practice that will enable the deﬁnition of valid biomarkers with a much higher probability than currently observed. While some of the initial issues have already been dealt with satisfactorily, others are still unresolved. For example, it is now generally accepted that single biomarkers should not be applied, as the complexity of a disease is unlikely to be thoroughly displayed by just one marker, and that a panel of such biomarkers should be employed V. Kadirkamanathan et al. (Eds.): PRIB 2009, LNBI 5780, pp. 137–149, 2009. c Springer-Verlag Berlin Heidelberg 2009

138

K. Harris, M. Girolami, and H. Mischak

instead [1,2]. However, it is equally evident that such a panel must consist of clearly deﬁned biomarkers, and not of an ill-deﬁned signature, as reported in several of the original manuscripts, almost exclusively based on the SurfaceEnhanced Laser Desorption/Ionization (SELDI) technology, that subsequently could not be validated [3,4]. This brings the issue of deﬁnition of a valid biomarker into focus. The fundamental question that should be asked is whether the change observed in the disease (frequency or abundance) of a certain molecule, based on data from a proteomics study, is in fact a result of the disease, or does it merely reﬂect an artefact due to technical variability in the pre-analytical steps, or in the analysis. Other likely suspects for suggesting an apparent but erroneous association with disease are biological variability or bias introduced in the study (for example, due to lifestyle, age and gender). In fact, these two problems are likely responsible for the majority of erroneous biomarkers. The most appropriate answer to this challenge appears to be the application of stringent statistical analysis. Not only does good statistical practice need to be highlighted, but also more sophisticated multivariate selection methods need to be developed, so that valid biomarkers will be deﬁned with a much higher probability than currently observed. To this end, we adopt a Bayesian approach to classiﬁcation and feature selection, as this approach oﬀers formal and well-calibrated probabilities for class prediction which is useful for medical decision making. We compare the Probabilistic K-Nearest Neighbour and hierarchic linear probit regression classiﬁers. Feature selection was incorporated in the latter method by assuming priors that favoured sparse solutions. It should be noted that other classiﬁcation methods like support vector machines with recursive feature elimination, adaptive boosting and random forests could have been used, but that in this paper we decided to focus solely on highlighting possible Bayesian approaches for proteomic biomarker selection. The rest of this paper is organised as follows: in Sect. 2 we discuss the illustrative experiment to ﬁnd biomarkers that diﬀerentiate between males and females. Section 3 describes the classiﬁcation and feature selection methods used in this paper in more detail. Section 4 presents the results of our experiments comparing the classiﬁcation performance of the two methods and the feature selection performance of the three priors used to induce sparsity in the probit regression model. Finally, Sect. 5 discusses the conclusions that can be drawn from our experimental results.

2

Application

To avoid any uncertainty in the assignment of a physiological condition, we choose as an illustrative example deﬁning proteomic diﬀerences between apparently healthy adult males and females. While clinical diagnosis or pathophysiological conditions are in general associated with a certain degree of uncertainty, gender can be assessed with almost 100% conﬁdence. Furthermore, the diﬀerences between male and female, while quite obvious at ﬁrst sight, are likely to be rather subtle at the proteomic level.

Deﬁnition of Valid Proteomic Biomarkers: A Bayesian Solution

139

We chose urine to be the body ﬂuid of interest, since urine has been found to be of much higher stability than blood-derived samples (serum or plasma), hence reducing pre-analytical variability [2,5]. Capillary electrophoresis-mass spectrometry (CE-MS) was used to analyse the urine samples, as this technology allows the routine analysis of a large number of samples and has been thoroughly validated as a platform technology [6,7]. The second urine of the morning was collected from a group of apparently healthy male and female volunteers (aged 21-40) during a routine medical checkup before recruitment at the Hannover Medical School. All samples were prepared and analysed using CE-MS as described in [6,7]. The goal of the analysis was to deﬁne biomarkers that would enable diﬀerentiation between male and female samples (based on the hypothesis that such biomarkers must exist).

3 3.1

Methods Probabilistic K-Nearest Neighbour Classification

The Probabilistic K-Nearest Neighbour (PKNN) classiﬁcation method (see [8] for an empirical analysis) adopts a fully Bayesian approach to obtaining posterior probabilities over the scaling parameter and the number of nearest neighbours to be employed. Markov chain Monte Carlo using the Metropolis-Hastings algorithm is employed to perform posterior sampling and Monte Carlo averaging provide the predicitive probabilities of class labels. Some more detail is provided below. Consider a ﬁnite data sample {(t1 , x1 ), · · · , (tN , xN )} where each tn ∈ {1, · · · , C} denotes the class label associated with the D-dimensional feature vector xn ∈ IRD and the feature space IRD has an associated metric with parameters θ denoted as Mθ . To deﬁne a probabilistic representation of the KNN method an approximate conditional joint likelihood is deﬁned in [9] such that Mθ β exp k δtn tj N j∼n|k p(t|X, β, k, θ, M) ≈ (1) Mθ C n=1 β exp k δctn c=1

j∼n|k

where we deﬁne the N × 1-dimensional vector t as [t1 , · · · tN ]T and the N × Ddimensional matrix X = [x1 , · · · , xN ]T , M denotes the metric employed in the feature space and θ are the associated parameters. The number of nearest neighbours is k and β deﬁnes a scaling variable. The expression Mθ

δtn tj

(2)

j∼n|k

denotes the number of the nearest k neighbours of xn , as measured under the metric Mθ within N − 1 samples from X remaining when xn is removed which

140

K. Harris, M. Girolami, and H. Mischak

we denote as X −n , and have the class label value of tn , whilst each of the terms in the summation of the denominator provides a count of the number of the k neighbours of xn which have class label equaling c. Full posterior inference will follow by obtaining the parameter posterior distribution p(β, k, θ|t, X, M) and subsequent predictions of the target class label t∗ of a new datum x∗ are made by posterior averaging such that p(t∗ |x∗ , t, X, M) = p(t∗ |x∗ , t, X, β, k, θ, M)p(β, k, θ|t, X, M)dβdθ. (3) k

As the required posterior takes an intractable form an MCMC procedure is proposed in [9] and extended in [10] to enable metric inference so that the following Monte-Carlo estimate is employed Ns 1 pˆ(t∗ |x∗ , t, X, M) = p(t∗ |x∗ , t, X, β (s) , k (s) , θ(s) , M) Ns s=1

(4)

where each β (s) , k (s) , θ(s) are samples obtained from the full parameter posterior p(β, k, θ|t, X, M) using a Metropolis style sampler. As the standard KNN method has no straightforward way to learn the metric we restrict this study to posterior inference over k and β and ﬁx the metric to the standard Euclidean metric. We therefore adopt the Metropolis scheme detailed in [9] and obtain samples from the posterior p(β, k, |t, X, M) and employ MonteN s Carlo estimates pˆ(t∗ |x∗ , t, X, M) = N1s s=1 p(t∗ |x∗ , t, X, β (s) , k (s) , M) in the following experimental section. 3.2

Hierarchic Linear Probit Regression Models

The fundamental problem of biomarker selection via CE-MS data is to identify which peptides best discriminate between diﬀerent types of protein samples, in this case between male and female samples. CE-MS data contains a large number of variables (peptides) and the sample size tends to be relatively small so the selection process can be unstable. Hence, models which incorporate sparsity in terms of variables are desirable for this kind of problem. Bae and Mallick [11] proposed a two-level hierarchical Bayesian probit regression model for variable selection which used three diﬀerent priors to incorporate diﬀerent levels of sparsity in the model. Details of this model, the sparsity inducing priors and the Gibbs sampler used to perform posterior sampling are given below. This method is preferable to using support vector machines for performing variable selection as we can obtain predictive probabilities of the class labels for new observations by Monte Carlo averaging, similar to the Probabilistic K-Nearest Neighbour method mentioned earlier. Model. Consider a ﬁnite data sample {(t1 , x1 ), · · · , (tN , xN )} where each tn ∈ {1, 2} denotes the class label associated with the D-dimensional feature vector xn ∈ IRD . Deﬁne the binary regression model as pi = P (ti = 2) = Φ(xTi β),

Deﬁnition of Valid Proteomic Biomarkers: A Bayesian Solution

141

i = 1, . . . n, where β is the D × 1-dimensional vector of unknown regression parameters and Φ is the standard normal cumulative density function linking the probability pi with the linear structure xTi β. Albert and Chib [12] introduced n independent latent variables z = [z1 , . . . , zn ]T into the problem, where zi ∼ N (xTi β, 1) and deﬁne ti = 2 if zi > 0 and ti = 1 if zi ≤ 0. This approach connects the probit binary regression model for ti to a normal linear regression model for the latent variable zi . Bae and Mallick [11] considered diﬀerent sparsity inducing priors for β in a two-level hierarchical Bayesian model. They placed a zero-mean Gaussian prior on β with unknown variances and assigned three diﬀerent priors for the variances under the assumption that they were independent, i.e., β|Λ ∼ N (0, Λ), where 0 = [0, . . . , 0]T , Λ = diag(λ1 , . . . , λD ) and λi is the variance of βi . Prior distributions for Λ. Model I - conjugate Inverse Gamma priors for each λi , i.e., (a/2)+1 D D a 2 1 b Λ∼ IG , ∝ exp − . (5) 2 b λi 2λi i=1 i=1 Model II - exponential priors for each λi , i.e., γλi Λ∼ Exponential(γ) ∝ exp − . 2 i=1 i=1 D

D

(6)

Model III - non-informative Jeﬀreys priors for each λi , i.e., Λ∼

D 1 . λ i=1 i

(7)

Note that Model III is the special case of Model I in which the hyperparameters a and b are both set to 0. Gibbs sampler. 1. Sample zi , for i = 1, . . . , n, from its full conditional distribution

N (xTi β, 1) truncated at the left by 0 if ti = 2, zi |β, ti ∝ (8) N (xTi β, 1) truncated at the right by 0 if ti = 1. 2. Sample β from its full conditional distribution p(β|z, t, Λ) ∝ N (ΣX T z, Σ), where Σ = (X T X + Λ−1 )−1 . 3. Sample Λ from its full conditional distribution. The full conditional distributions for Models I, II and III, respectively, are: p(Λ

−1

|z, t, β) ∝

D i=1

p(Λ−1 |z, t, β) ∝

D i=1

Gamma

a+1 2 , 2 b + βi2 √

InverseGaussian

γ ,γ |βi |

,

(9)

(10)

142

K. Harris, M. Girolami, and H. Mischak

and p(Λ

−1

|z, t, β) ∝

D i=1

Gamma

1 2 , 2 βi2

.

(11)

Predictive classification. The predictive classiﬁcation of the target class label t∗ of a new datum x∗ is given by the following Monte-Carlo estimate: Ns Ns 1 1 Pˆ (t∗ = 2|x∗ ) = p(t∗ = 2|x∗ , β (s) , z (s) , Λ(s) ) = Φ(xT∗ β (s) ), (12) Ns s=1 Ns s=1

where β (s) , z (s) and Λ(s) are the MCMC samples from the posterior distribution.

4 4.1

Experimental Results PKNN Classifiers

To maintain consistency of data representation with other studies on this data the same arbitrary threshold for data sparsity (80%) was employed to reduce the number of covariates to 1524. However, instead of normalising samples with their sum of intensity values, we normalised features with their sum of intensity values, so as to not distort the original feature space before applying our models. A Wilcoxon Rank-Sum non-parametric test was used to provide a ranking of individual covariates based on p-value, and from this it is clear that a very small percentage of the 1524 peptides have any statistical evidence supporting their discrimination ability. Setting a p-value threshold of 2% the number of peptides was reduced further to 229 and these were used in devising a series of PKNN classiﬁers. Starting with the full set of 229 peptides a PKNN classiﬁer was devised by using Metropolis sampling with a burn-in of 5000 samples and a further 45000 post-burn-in samples retained for Monte-Carlo averaging. The proposal distribution was tuned to achieve acceptance rates between 35% to 50%. A randomised ten-fold cross validation (10-CV) was used to obtain estimates of predictive performance and only 0-1 error loss is reported here, however predictive probabilities are obtained from PKNN. These probabilities are used to make decisions based on the cost and threshold selected which in this case as the classes are balanced was set at 0.5. The 10-CV score and associated standard error is reported when 229 peptides are used, then 228 are used where the peptide with the highest p-value is removed. This is done for fourteen peptides after which groups of 10 peptides were removed each time and the 10-CV score was measured. These results are shown in Fig. 1. A minimum mean 10-CV error of 8.68% is achieved, however this is rather meaningless taken on its own without considering the standard error, which would increase if multiple randomisations were employed. At around 220 to 210 peptides the range of error is minimal and this increases as the number of low p-value peptides are removed. It is conjectured that due to the relatively

Deﬁnition of Valid Proteomic Biomarkers: A Bayesian Solution

143

22

Percent Error (10−CV)

20 18 16 14 12 10 8 6

220

200

180

160

140

120

100

80

60

40

Ranked Peptides by p−value

Fig. 1. This graph shows the estimated 10-CV prediction error (mean ± one standard error)

high levels of sparsity in the data which remains a large number of peptides are required to make reasonable predictions, this will be discussed further on in the paper. 4.2

Hierarchic Linear Probit Regression Models

As in Sect. 4.1, we remove all peptides that have more than 80% zero intensity values and normalise each feature with their sum of intensity values. We again use the ranking of covariates provided by the p-values of the Wilcoxon test to further reduce the number of peptides, but this time choose a p-value threshold of roughly 5% to reduce the number of peptides to 350. These peptides were then used to build the three classiﬁers discussed in Sect. 3.2. Like Bae and Mallick [11], we ﬁxed the hyperparameters for Models I and II so that E(λi ) = 10 and Var(λi ) = 100. We ran the Gibbs sampler of Sect. 3.2 for 50,000 iterations and discarded the ﬁrst 20,000 iterations as burn-in. As in Sect. 4.1, a randomised 10-CV was used to assess the predictive performance of the three models. Both Models I and II gave an average test error of 8.2%±2.1%, while Model III gave an average test error of 11.2%±2.0%. It should be noted that tuning the hyperparameters of Models I and II could potentially lead to improved performance. It is not surprising that Models I and II performed similarly, as although the form of the prior distribution for the variance of the regression coeﬃcients was diﬀerent, the mean and variance was set to be the same, and this result is consistent with the ﬁndings of Bae and Mallick [11]. We believe that the poorer performance of Model III is due to the Jeﬀreys prior inducing too much sparsity in the model, similar to the worsening performance of the PKNN classiﬁer seen in Fig. 1 after the number of peptides in the model is reduced below 210. We select potential biomarkers using the posterior variance of β with the idea being that the peptides with larger variance are more important in discriminating between the diﬀerent types of protein samples than those with smaller variance. Figures 2, 3 and 4 show the variance of βi for Models I, II and III, respectively.

144

K. Harris, M. Girolami, and H. Mischak

17 16 mean(lambda)

15 14 13 12 11 10 9 0

0.5

1 Peptide ID

1.5

2 5

x 10

Fig. 2. Plot of the variance of βi versus the peptide ID (Model I)

16

mean(lambda)

15 14 13 12 11 10 9 0

0.5

1 Peptide ID

1.5

2 5

x 10

Fig. 3. Plot of the variance of βi versus the peptide ID (Model II) 7

7

x 10

median(lambda)

6 5 4 3 2 1 0 0

0.5

1 Peptide ID

1.5

2 5

x 10

Fig. 4. Plot of the variance of βi versus the peptide ID (Model III)

We see that Models I and II give very similar results and there are roughly 20 peptides that have signiﬁcantly larger variance than the others. Comparing the top 20 peptides for Models I and II we see that Models I and II give very consistent selections, as 18 peptides are in both top 20s. In particular, both models rank peptides 186673 and 187114 as the two most important peptides by a comfortable margin. Of these 18 peptides, all but three of them had pvalues less than 0.005 in the original Wilcoxon tests and of the top two peptides both had p-values less than 3 × 10−6 . We also checked the sensitivity of the peptide rankings with the 10-CV mentioned earlier and found that the rankings

Deﬁnition of Valid Proteomic Biomarkers: A Bayesian Solution

145

of Models I and II were broadly consistent between folds with the same peptides being ranked highly in the majority of folds. We see from Fig. 4 that Model III induces sparseness much more strongly than Models I and II, as there are only 8 peptides that have signiﬁcantly larger variance than the others. Only two of these peptides were selected by Models I and II, but both models ranked them outside the top 10. Unlike Models I and II, when we checked the sensitivity of the peptide rankings with the results for the 10-CV we found that the selected peptides were rarely consistent between folds. This suggests that the Jeﬀreys prior over-prunes the model and puts very little weight on many peptides useful for classifying the binary response, leading to its worse performance in terms of the average test error found earlier. We thus conclude that the peptides suggested by Models I and II are more likely to make a good set of biomarkers for this problem than those suggested by Model III. 4.3

Classification for the Blinded Test Data

Figures 5, 6, 7 and 8 show the posterior predictive probabilities obtained for each test sample from the PKNN classiﬁer of Sect. 4.1 and Bae and Mallick’s sparse probit regression Models I, II and III of Sect. 4.2, respectively. We see that the PKNN classiﬁer and Bae and Mallick’s Model III tend to give the most conﬁdent predictions, while the posterior predictive probabilities are very similar for Bae and Mallick’s Models I and II and tend to give the least conﬁdent predictions. Note that it is easy to compare the predictive performance of two competing classiﬁers graphically by plotting the predictive probabilities of one method against the other. We also see that the four classiﬁers tend to allocate the test samples to the same class. In fact, all four classiﬁers are in agreement for 71 of the 92 test samples. The test samples where there is disagreement in the predictions of the four classiﬁers tend to happen when at least one classiﬁer gives an unconﬁdent prediction, that is, a posterior predictive probability close to 0.5. Even with the PKNN classiﬁer there are four samples that have a posterior predictive probability of between 0.4 and 0.6. In such cases we would advocate

Predictive Class Probability

1 0.8 0.6

Decision Threshold P > 0.5

0.4 0.2 0

10

20

30

40 50 60 Blinded Test Sample

70

80

90

Fig. 5. Plot of the posterior predictive probabilities from the PKNN classiﬁer

146

K. Harris, M. Girolami, and H. Mischak

Predictive Class Probability

1 0.8 0.6

Decision Threshold P > 0.5

0.4 0.2 0

10

20

30 40 50 60 Blinded Test Sample

70

80

90

Fig. 6. Plot of the posterior predictive probabilities from Bae and Mallick Model I

Predictive Class Probability

1 0.8 0.6

Decision Threshold P > 0.5

0.4 0.2 0

10

20

30 40 50 60 Blinded Test Sample

70

80

90

Fig. 7. Plot of the posterior predictive probabilities from Bae and Mallick Model II

not allocating the test sample to either class, as there is great uncertainty over the true class of the test sample. This transparency in the conﬁdence of our class predictions is a huge advantage of the Bayesian approach over the more commonly used SVM techniques, which cannot provide such a formal and wellcalibrated measure of the conﬁdence of a class prediction. The performance of the four classiﬁers on the blinded test set turned out to be very similar, as Bae and Mallick’s Model II misclassiﬁed 14 out of the 92 samples, while both their Models I and III made 15 misclassiﬁcations, and the PKNN classiﬁer performing slightly worse with 17 misclassiﬁcations. As we would expect, the test samples that were misclassiﬁed tended to have posterior predictive probabilities between 0.3 and 0.7, and thus had class predictions that were not very conﬁdent.

Deﬁnition of Valid Proteomic Biomarkers: A Bayesian Solution

147

Predictive Class Probability

1 0.8 0.6

Decision Threshold P > 0.5

0.4 0.2 0

10

20

30 40 50 60 Blinded Test Sample

70

80

90

Fig. 8. Plot of the posterior predictive probabilities from Bae and Mallick Model III

We then trained Bae and Mallick’s three models on three smaller data sets with 33, 20 and 7 cases and controls, in order to assess how model performance was aﬀected by smaller training set sizes. We discovered that the conﬁdence in our predictions declines signiﬁcantly as the number of training samples decreases. Indeed, when the number of training samples is only 14, almost all the predictive probabilities are between 0.3 and 0.7, which suggests that our predictive performance may be little better than guessing and that the biomarkers suggested by such a small data set would not be substantiated in practice. This suggestion of deteriorating predictive performance as the number of training samples is reduced was conﬁrmed when we unblinded the test samples (see Table 1). Table 1. Test error for diﬀerent training set sizes Training set size Model I 14 28.3% 40 27.2% 66 21.7% 134 16.3%

5

Model II Model III 27.2% 25% 27.2% 23.9% 21.7% 25% 15.2% 16.3%

Conclusions

Sparse models enable us to identify a small number of peptides having the greatest discriminating power, thereby allowing researchers to quickly focus on the most promising candidates for diagnostics and prognostics. The Bayesian approach yields a coherent way to assign new samples to particular classes. Rather than hard rules of assignment, we can evaluate the probability that the new sample will be of a certain type which is more helpful for medical decision making.

148

K. Harris, M. Girolami, and H. Mischak

Meaningful results will only be obtained if the number of training samples collected is suﬃcient to allow the deﬁnition of statistically valid biomarkers. Acknowledgements. K. Harris & M. Girolami are supported by an Engineering and Physical Sciences Research Council (EPSRC) grant EP/F009429/ 1 - Advancing Machine Learning Methodology for New Classes of Prediction Problems. M. Girolami is funded by an EPSRC Advanced Research Fellowship EP/E052029/1. H. Mischak is supported by EU funding through the InGenious HyperCare consortium, grant LSHM-CT-2006-037093.

References 1. Mischak, H., Apweiler, R., Banks, R.E., Conaway, M., Coon, J., Dominiczak, A., Ehrich, J.H.H., Fliser, D., Girolami, M., Hermjakob, H., Hochstrasser, D., Jankowski, J., Julian, B.A., Kolch, W., Massy, Z.A., Neusuess, C., Novak, J., Peter, K., Rossing, K., Schanstra, J., Semmes, O.J., Theodorescu, D., Thongboonkerd, V., Weissinger, E.M., Van Eyk, J.E., Yamamoto, T.: Clinical proteomics: A need to deﬁne the ﬁeld and to begin to set adequate standards. Proteomics - Clinical Applications 1(2), 148–156 (2007) 2. Decramer, S., de Peredo, A.G., Breuil, B., Mischak, H., Monsarrat, B., Bascands, J.L., Schanstra, J.P.: Urine in clinical proteomics. Molecular and Cellular Proteomics 7(10), 1850–1862 (2008) 3. Petricoin, E.F., Ardekani, A.M., Hitt, B.A., Levine, P.J., Fusaro, V.A., Steinberg, S.M., Mills, G.B., Simone, C., Fishman, D.A., Kohn, E.C., Liotta, L.A.: Use of proteomic patterns in serum to identify ovarian cancer. The Lancet 359(9306), 572–577 (2002) 4. Check, E.: Proteomics and cancer - running before we can walk? Nature 429(6991), 496–497 (2004) 5. Mischak, H., Coon, J.J., Novak, J., Weissinger, E.M., Schanstra, J.P., Dominiczak, A.F.: Capillary electrophoresis-mass spectrometry as a powerful tool in biomarker discovery and clinical diagnosis: An update of recent developments. Mass Spectrometry Reviews (October 2008) (in press) 6. Coon, J.J., Z¨ urbig, P., Dakna, M., Dominiczak, A.F., Decramer, S., Fliser, D., Frommberger, M., Golovko, I., Good, D.M., Herget-Rosenthal, S., Jankowski, J., Julian, B.A., Kellmann, M., Kolch, W., Massy, Z., Novak, J., Rossing, K., Schanstra, J.P., Schiﬀer, E., Theodorescu, D., Vanholder, R., Weissinger, E.M., Mischak, H., Schmitt-Kopplin, P.: CE-MS analysis of the human urinary proteome for biomarker discovery and disease diagnostics. Proteomics - Clinical Applications 2(7-8), 964–973 (2008) 7. Jantos-Siwy, J., Schiﬀer, E., Brand, K., Schumann, G., Rossing, K., Delles, C., Mischak, H., Metzger, J.: Quantitative urinary proteome analysis for biomarker evaluation in chronic kidney disease. Journal of Proteome Research 8(1), 268–281 (2009) 8. Manocha, S., Girolami, M.: An empirical analysis of the probabilistic k-nearest neighbour classiﬁer. Pattern Recognition Letters 28(13), 1818–1824 (2007) 9. Holmes, C.C., Adams, N.M.: A probabilistic nearest neighbour method for statistical pattern recognition. J. R. Statist. Soc. B 64(2), 295–306 (2002)

Deﬁnition of Valid Proteomic Biomarkers: A Bayesian Solution

149

10. Everson, R.M., Fieldsend, J.E.: A variable metric probabilistic k-nearestneighbours classiﬁer. In: Yang, Z.R., Yin, H., Everson, R.M. (eds.) IDEAL 2004. LNCS, vol. 3177, pp. 654–659. Springer, Heidelberg (2004) 11. Bae, K., Mallick, B.K.: Gene selection using a two-level hierarchical Bayesian model. Bioinformatics 20(18), 3423–3430 (2004) 12. Albert, J., Chib, S.: Bayesian analysis of binary and polychotomous response data. Journal of the American Statistical Association 88, 669–679 (1993)

Inferring Meta-covariates in Classification Keith Harris, Lisa McMillan, and Mark Girolami Inference Group, Department of Computing Science, University of Glasgow, UK {keithh,lisa,girolami}@dcs.gla.ac.uk http://www.dcs.gla.ac.uk/inference

Abstract. This paper develops an alternative method for gene selection that combines model based clustering and binary classification. By averaging the covariates within the clusters obtained from model based clustering, we define “meta-covariates” and use them to build a probit regression model, thereby selecting clusters of similarly behaving genes, aiding interpretation. This simultaneous learning task is accomplished by an EM algorithm that optimises a single likelihood function which rewards good performance at both classification and clustering. We explore the performance of our methodology on a well known leukaemia dataset and use the Gene Ontology to interpret our results. Keywords: Gene selection, clustering, classification, EM algorithm, Gene Ontology.

1

Introduction

In this paper, we develop a procedure for potentially improving the classiﬁcation of gene expression proﬁles through coupling with the method of model based clustering. Such DNA microarray data typically consists of several thousands of genes (covariates) and a much smaller number of samples. Analysing this data is statistically challenging, as the covariates are highly correlated, which results in unstable parameter estimates and inaccurate prediction. To alleviate this problem, we use the averages of covariate clusters, rather than all the original covariates, to classify DNA samples. The advantage of this approach over using a sparse classiﬁcation model [1,2] is that we can extract a much larger subset of genes with essential predictive power and partition this subset into groups, within which the genes are similar. An overview of our procedure that combines model based clustering and binary classiﬁcation is as follows. By averaging the features within the clusters obtained from a Gaussian mixture model [3,4], we deﬁne “superfeatures” or “meta-covariates” and use them in a probit regression model, thereby attaining concise interpretation and accuracy. Similar ideas, from a non-Bayesian two-step perspective, have been looked at by Hanczar et al. [5] and Park et al. [6]. With our simultaneous procedure, the clusters are formed considering the correlation of the predictors with the response in addition to the correlations among the predictors. The proposed methodology should have wide applicability in areas such as gene selection and proteomic biomarker selection. V. Kadirkamanathan et al. (Eds.): PRIB 2009, LNBI 5780, pp. 150–161, 2009. c Springer-Verlag Berlin Heidelberg 2009

Inferring Meta-covariates in Classification

151

The rest of this paper is organized as follows: in Sect. 2 we introduce our meta-covariate classication model and provide an EM algorithm for learning the parameters of our model from data. In Sect. 3 we illustrate our method with a DNA microarray data example and use the Gene Ontology (GO) to interpret our results. Section 4 discusses the conclusions we draw from our experimental results. Finally, Appendix A gives the full details of our model and shows the derivation of our EM algorithm.

2 2.1

Methodology Model

In the following discussion, we will denote the N × D design matrix as X = [x1 , . . . , xN ]T and the N × 1 vector of associated response values as t where each element tn ∈ {−1, 1}. The K × N matrix of clustering mean parameters θkn is denoted by θ. We represent the K × 1-dimensional columns of θ by θ n and the corresponding N × 1-dimensional rows of θ by θk . The D × K matrix of clustering latent variables zdk is represented as Z. The K × 1 vector of regression coeﬃcients is denoted by w. Finally, we denote the N × 1 vector of classiﬁcation auxiliary variables by y. The graphical representation of the conditional dependency structure in the meta-covariate classiﬁcation model is shown in Fig. 1. From Fig. 1 we see that the joint distribution of the meta-covariate classiﬁcation model is given by p(t, y, X, θ, w) = p(t, y|θ, w)p(X|θ)p(θ)p(w).

(1)

The distribution p(X|θ) is the likelihood contribution from our clustering model, which we chose to be a normal mixture model with equal weights and identity covariance matrices. Similarly, p(t, y|θ, w) is the likelihood contribution from our classiﬁcation model, which we chose to be a probit regression model whose covariates are the means of each cluster, that is, θk , k = 1, . . . , K. Finally, the model was completed by specifying vague normal priors for θ and w. Full details of our model along with the derivation of the following EM algorithm that we used for inference is given in Appendix A. 2.2

Summary of the EM Algorithm

Given the number of clusters K, the goal is to maximise the joint distribution with respect to the parameters (comprising the means of the clusters and the regression coeﬃcients). 1. Initialise θ, w, the responsibilities γ(zdk ) and E(y), and evaluate the initial value of the log likelihood. 2. E-step. Evaluate: exp − 12 xd − θk 2 γ(zdk ) = K (2) 1 2 j=1 exp − 2 xd − θ j

152

K. Harris, L. McMillan, and M. Girolami

X

θ

y

t

w

Fig. 1. Graphical representation of the conditional dependencies within the metacovariate classification model

and E(yn ) =

w T θn + w θn − T

φ(−wT θ n ) 1−Φ(−wT θ n ) φ(−w T θn ) Φ(−wT θ n )

if tn = 1 otherwise.

(3)

3. M-step. Evaluate: E(y) − θT w−k wk + Xγ k + h1 θ0 θk = D wk2 + d=1 γ(zdk ) + h1 and w=

−1 1 θθT + I θE(y). l

(4)

(5)

After updating w in this manner, set the ﬁrst component of the vector to 1, so that the model is identiﬁable. 4. Evaluate the log likelihood and check for convergence. If the convergence criterion is not satisﬁed return to step 2.

3

Experimental Results - Acute Leukemia Data

3.1

Data Description

A typical application where clustering and classiﬁcation have become common tasks is the analysis of DNA microarray data, where thousands of gene expression levels are monitored on a few samples of diﬀerent types. We thus decided to illustrate our proposed methodology for inferring meta-covariates in classiﬁcation with the widely analysed leukaemia microarray dataset of Golub et al. [7], which was downloaded from the Broad Institute Website1 . Bone marrow or peripheral blood samples were taken from 72 patients with either acute myeloid leukaemia (AML) or acute lymphoblastic leukaemia (ALL). Gene expression levels were measured using Aﬀymetrix high-density oligonucleotide arrays containing 7129 probes for 6817 human genes. Following the experimental setup of the original 1

http://www.broad.mit.edu/cgi-bin/cancer/publications/ pub_paper.cgi?mode=view&paper_id=43

Inferring Meta-covariates in Classification

153

paper, the dataset was split into a training set of 38 samples of which 27 are ALL and 11 are AML, and a test set of 34 samples, 20 ALL and 14 AML. The data was preprocessed as recommended in [8]: (a) thresholding, ﬂoor of 100 and ceiling of 16000; (b) ﬁltering, exclusion of probes with max/min ≤ 5 and (max − min) ≤ 500; (c) base 10 logarithmic transformation; and (d) standardising, so that each sample has mean 0 and variance 1. This left us with 3571 probes for analysis. Finally, GO annotations for the appropriate gene chip (Hu6800) were obtained via the Aﬀymetrix NetAﬀx analysis centre2 . 3.2

Results and Discussion

EM algorithm results. Figure 2 shows the minimum and mean test error from 200 runs of our EM algorithm for diﬀerent values of the number of clusters K. It should be noted that we used the K-means clustering algorithm to initialise the matrix of clustering mean parameters θ, while the other parameters were initialised randomly. We see from Fig. 2 that on average the algorithm performs best for around 15 to 25 clusters, with the best case yielding an average test error rate of 9.93% for K = 21 clusters. We also see that for K = 21 clusters, the run that achieved the highest likelihood value also achieved the minimum test error of 2.94%, that is, just one misclassiﬁcation in the test set. The predictions from the highest likelihood model with K = 21 clusters thus appear to improve predictions made by Golub et al. [7], who made ﬁve misclassiﬁcations on the test set, and is competitive with the methods of Lee et al. [1] and Bae and Mallick [2], who misclassiﬁed one and two test samples, respectively. We will now use the Gene Ontology to interpret the results from this model.

0.5

Test error

0.4 0.3 0.2 0.1 0 0

Run with max likelihood value.

50

100

150

200

250

K Fig. 2. Minimum and mean test error after 200 runs of the EM algorithm 2

http://www.affymetrix.com/analysis/index.affx

154

K. Harris, L. McMillan, and M. Girolami Table 1. The best clusters (K = 21) Cluster Probes Controls w rank(abs(w)) Genes 1 20 0 1.00 10 16 2 486 4 0.22 19 412 3 20 0 -1.22 8 20 4 253 0 -1.88 7 230 5 182 0 0.55 15 173 6 240 1 -3.08 4 199 7 110 2 -0.37 18 99 8 60 4 0.16 20 50 9 4 4 -0.15 21 1 10 230 0 -2.66 5 214 11 189 1 -1.10 9 166 12 210 1 0.88 12 183 13 228 0 0.79 13 200 14 230 0 0.55 16 187 15 61 0 3.87 1 56 16 240 0 3.21 3 204 17 213 0 -0.50 17 205 18 17 0 -0.95 11 16 19 267 1 0.75 14 235 20 101 1 -3.79 2 85 21 210 3 2.46 6 175

GO analysis. Table 1 describes each of the 21 probe clusters, with respect to the number of probes allocated to the cluster; the number of control probes allocated to the cluster; its regression coeﬃcient (w); its rank by descending absolute regression coeﬃcient; and the number of genes represented by the probe set. The number of unique Entrez Gene IDs (as obtained from NetAﬀx) was used to count the number of unique genes. 22 of the 59 controls on the microarray survive the initial ﬁltering process (all 22 of these are endogenous controls). Control probes, by design, should not be functionally signiﬁcant. It is therefore encouraging that most (63.64%) of the control probes belong to the four least inﬂuential clusters (with respect to abs(w)): clusters 9 (w = −0.15, ranked 21st), 8 (w = 0.16, ranked 20th), 2 (w = 0.22, ranked 19th) and 7 (w = −0.37, ranked 18th). Furthermore, cluster 8 – the cluster with the lowest absolute regression coeﬃcient – contains only four probes, all of which are control probes. It should be noted that six control probes do occur in the ten ‘signiﬁcant’ clusters; the extent to which these probes are appropriate controls should be investigated further. The clusters are reasonably well balanced, with most clusters containing approximately 200 genes. The largest and smallest clusters (numbers 2 and 9 respectively) have small regression coeﬃcients, indicating that they have limited inﬂuence on the classiﬁer. Using w = 1 as a baseline, ten clusters (numbers 15, 20, 16, 6, 10, 21, 4, 3, 11, 1) are suﬃciently weighted to be of interest (these ten clusters will be

Inferring Meta-covariates in Classification

155

described as the ‘signiﬁcant’ clusters). The aim of this work is to assess whether there is any biological signiﬁcance in the clustering of the probes (or genes): the expectation is that genes clustered together will be carrying out a similar function or functions. As such, GO annotations from the molecular function aspect of the GO were used. The total number of occurrences for each GO term across all genes in a cluster was calculated. By comparing this to the occurrences for each GO term across the entire chip and using the hypergeometric distribution, we can calculate the probability that the terms were encountered by chance. By comparing the occurrence of the GO term in the cluster and the entire chip, we can describe it as over- or under-represented in the gene cluster. Cluster 15, w = 3.87. Most noticeably, metal ion (and speciﬁcally zinc ion) annotations are under-represented in this gene cluster. Further, nucleotide and nucleic acid binding are seen less often than would be expected. Several very speciﬁc terms are found enriched in this gene cluster; of particular interest is a cluster of three enzyme inhibitor activity subterms. Cluster 20, w = −3.79. There is a concentration of very speciﬁc transmembrane transporter activities and oxidoreductase terms. Unlike the previous cluster, protein kinase activity is under-represented; nucleic acid binding is over-represented and receptor activity is under-represented in this cluster. Cluster 16, w = 3.21. In this cluster, zinc ion binding is over-represented, unlike in clusters 15 and 20 (where the term was under-represented and not signiﬁcant respectively). Also interesting is the overrepresentation of the “damaged DNA binding” term - particularly relevant in the context of cancer. Like cluster 15, several general receptor binding terms are over-represented. A small cluster of pyrophosphatase subterms are also over-represented. Cluster 6, w = −3.08. Several metal ion binding terms are over-represented here, including calcium and zinc, and most interestingly – particularly in the context of leukaemia, cancer of the blood – heme binding. Again, several receptor binding and activity terms are over-represented. Cluster 10, w = −2.66. Most noticeably, a small cluster of under-represented terms describe signal transducer activity and several kinds of receptor activities. This is an area of the Gene Ontology that was enriched in clusters 15, 16 and 6 and under-represented in cluster 20. There is signiﬁcant enrichment of DNA binding terms (speciﬁcally DNA topoisomerases). Cluster 21, w = 2.46. Cluster 21 has the most extensive coverage and deepest annotation of the ten signiﬁcant clusters, despite being of comparable size to many others (e.g., 16, 6, 10, 4 and 11). In addition, none of the signiﬁcant annotations are seen less than would be expected: they are all enriched in this cluster. Multiple metal ion binding terms are enriched here as are DNA binding, receptor activity and kinase activity.

156

K. Harris, L. McMillan, and M. Girolami

Cluster 4, w = −1.88. Cluster 4 is enriched for several transcription regulation terms, kinase activities, and DNA and nucleotide binding. Here, enzyme regulator activities are under-represented. Cluster 3, w = −1.22. The genes in cluster 3 are enriched for receptor activity and a speciﬁc receptor activity: ﬁbroblast growth factor receptor activity. Again, receptor binding and activity terms are over-represented and metal ion terms are under-represented. There is enrichment of a speciﬁc enzyme activator activity, apoptotic protease activator activity, of particular interest in the context of cancer. Cluster 11, w = −1.10. A cluster of signal transducer activity/receptor activities are under-represented here; similar to patterns observed in clusters 20, 4 and 10. There are fewer metal (iron, calcium and zinc) ion binding terms and protein kinase annotations than would be expected by chance. Cluster 1, w = 1.00. Cluster 1 deﬁnes the ‘baseline’ for regression model coefﬁcients. This cluster is enriched for ion binding (including iron, ferrous, haem and haemoglobin), ferrochelatase and oxygen transporter activity, signiﬁcant in the context of leukaemia. Table 2 describes each of the ten signiﬁcant clusters with respect to an annotation proﬁle, which considers over-representation and under-representation of metal ion binding terms; DNA or RNA binding terms; receptor activity terms; enzyme regulation terms; receptor binding terms; kinase activity terms; transmembrane transport terms and transcriptional regulation terms. It is clear that none of the clusters are identical with respect to this proﬁle. Receptor activity terms and metal ion binding terms are more often overrepresentated in the gene clusters with positive regression coeﬃcients, and more often under-represented in the gene clusters with negative regression coeﬃcients. Comparison to other methods. In their original paper, Golub et al. [7] identiﬁed 50 genes that were highly correlated with the AML/ALL class distinction. 68% of these genes are assigned to a cluster with an absolute regression coeﬃcient of ≥ 1. Cluster 15, the top ranking cluster with respect to absolute regression coeﬃcient, contains six of these genes and cluster 20, the next most inﬂuential cluster, contains four of these genes. Surprisingly, eight genes are found in cluster 5, which has a low regression coeﬃcient (w = 0.55). More recently, Lee et al. [1] identiﬁed 27 genes as informative, using a Bayesian method for variable selection. In this more reﬁned set, eight (29.63%) of the genes belong to the most inﬂuential cluster (15). In a follow up study where sparsity was imposed on the priors, Bae and Mallick [2] identiﬁed 10 genes using various models. Here, three genes are found in cluster 15 and two genes are found in cluster 20, and only two genes are mapped to clusters with an absolute regression coeﬃcient < 1. Three genes are identiﬁed by all three methods [1,2,7]: Cystatin C, Zyxin and CF3 (transcription factor 3). CF3 is assigned to cluster 5, a comparatively weakly informative cluster; however, both Zyxin and Cystatin C are assigned to cluster 15, the most inﬂuential cluster in the regression model.

Inferring Meta-covariates in Classification

157

Table 2. Summary of cluster annotations Cluster 15 16 21 1 11 3 4 10 6 20

w MIB D/RB RA 3.87 n n y 3.21 y y y 2.46 y y y 1.00 y y -1.10 n n -1.22 n y -1.88 y n -2.66 n y n -3.08 y y -3.79 y n

ER RB KA TMT TRR y y y y y y y ∼ y y y y

y y

y

y y n

y y y y y

y

n

y

MIB = metal ion binding; D/RB = DNA or RNA binding; RA = receptor activity; ER = enzyme regulation; RB = receptor binding; KA = kinase activity; TMT= transmembrane transport; TRR = transcription regulation. y indicates over-representation; n indicates under-representation; ∼ indicates conflicting results.

4

Conclusions

The method is successful in assigning limited inﬂuence to control probes. The clustering of probes reﬂects functional diﬀerences between the genes that they represent. Furthermore, enrichment of metal ion binding and receptor activity annotations appear to correspond with the sign of the regression coeﬃcients; that is, clusters with positive regression coeﬃcients are more often enriched for such annotations, while clusters with negative regression coeﬃcients are often under-represented by such annotations. In a comparison with methods of variable selection in the same dataset, genes important in the discrimination between AML and ALL tend to belong to clusters with high absolute regression coeﬃcients in the model; this is particularly true as the variable selection methods become more sophisticated and fewer genes are found to be signiﬁcant. Of the three genes that are common in three diﬀerent analyses of these data, two (Zyxin and Cystatin C) are assigned to the most inﬂuential cluster in our model. Our experimental results thus indicate that our EM algorithm approach of inferring meta-covariates in classiﬁcation is a promising new methodology with wide applicability. Moreover, the approach can be naturally extended to multiclass classiﬁcation and to incorporate sparsity by employing an Inverse Gamma prior on the variance of the regression coeﬃcients. Future research will focus on developing a Bayesian sampler for the “meta-covariate” classiﬁcation model, possibly using reversible jump Markov chain Monte Carlo or an inﬁnite mixture model to infer directly from the data the optimal number of clusters. Acknowledgements. K. Harris & M. Girolami are supported by the Engineering and Physical Sciences Research Council (EPSRC) grant EP/F009429/

158

K. Harris, L. McMillan, and M. Girolami

1 - Advancing Machine Learning Methodology for New Classes of Prediction Problems. M. Girolami is funded by an EPSRC Advanced Research Fellowship EP/E052029/1. L. McMillan is funded by a grant from SHEFC SRDG.

References 1. Lee, K.E., Sha, N., Dougherty, E.R., Vannucci, M., Mallick, B.K.: Gene selection: a Bayesian variable selection approach. Bioinformatics 19(1), 90–97 (2003) 2. Bae, K., Mallick, B.K.: Gene selection using a two-level hierarchical Bayesian model. Bioinformatics 20(18), 3423–3430 (2004) 3. Bishop, C.M.: Pattern Recognition and Machine Learning. Springer, Heidelberg (2006) 4. Fraley, C., Raftery, A.E.: Bayesian regularization for normal mixture estimation and model-based clustering. Journal of Classification 24(2), 155–181 (2007) 5. Hanczar, B., Courtine, M., Benis, A., Henegar, C., Cl´ement, K., Zucker, J.D.: Improving classification of microarray data using prototype-based feature selection. SIGKDD Explorations 5(2), 23–30 (2003) 6. Park, M.Y., Hastie, T., Tibshirani, R.: Averaged gene expressions for regression. Biostatistics 8(2), 212–227 (2007) 7. Golub, T.R., Slonim, D.K., Tamayo, P., Huard, C., Gaasenbeek, M., Mesirov, J.P., Coller, H., Loh, M.L., Downing, J.R., Caligiuri, M.A., Bloomfield, C.D., Lander, E.S.: Molecular classification of cancer: class discovery and class prediction by gene expression monitoring. Science 286(5439), 531–537 (1999) 8. Dudoit, S., Fridlyand, J., Speed, T.P.: Comparison of discrimination methods for the classification of tumors using gene expression data. Journal of the American Statistical Association 97(457), 77–87 (2002)

A A.1

Derivation of the EM Algorithm Regression

Modelling. In the following subsection, y denotes an N ×1 continuous response vector. Joint distribution. p(y, X, θ, w) = p(y|θ, w)p(X|θ)p(θ)p(w).

(6)

yn = wT θ n + n where n ∼ N (0, 1).

(7)

Regression model.

⇒ p(y|θ, w) =

N

p(yn |θn , w) =

n=1

⇒ log p(y|θ, w) = −

N

T 2 1 1 √ e− 2 (yn −w θn ) . 2π n=1

N 1 N (yn − wT θn )2 − log(2π). 2 n=1 2

(8)

(9)

Inferring Meta-covariates in Classification

159

Clustering model. Normal mixture model with equal weights and identity covariance matrices. K 1 ⇒ p(x) = N (x|θk , I). (10) K k=1

From [3] we have that:

log p(X|θ) ≥ L(q, θ) =

q(Z) log

Z

p(X, Z|θ) q(Z)

,

(11)

where Z is a D × K matrix of latent variables with rows z Td such that z d is a K-dimensional binary random variable having a 1 of K representation in which a particular element zk is equal to 1 and all other elements are equal to 0, and q(Z) is a distribution deﬁned over the latent variables. ⇒ log p(X|θ) ≥

p(Z|X, θold ) log p(X, Z|θ) −

Z

p(Z|X, θold ) log p(Z|X, θold )

Z

(12) = Q(θ, θold ) + const.

p(X, Z|θ) =

(13)

z D

K

1 dk N (xd |θ k , I)zdk . K

(14)

d=1 k=1

⇒ EZ [log p(X, Z|θ)] ≥ −

D K N 1 E(zdk ) (xnd − θkn )2 + const. 2 n=1

(15)

d=1 k=1

Prior distributions. p(θ) =

K

N (θk |θ 0 , hI),

(16)

k=1

where each element of θ0 is set to the corresponding covariate interval midpoint and h is chosen arbitrarily large in order to prevent the speciﬁcation of priors that don’t overlap with the likelihood and allow for mixtures with widely diﬀerent component means. p(w) = N (w|0, lI). (17) E-step. 1 z zdk K N (xd |θk , I) dk zdj E(zdk ) = γ(zdk ) = 1 zdj K N (xd |θ j , I) exp − 21 xd − θk 2 = K 1 . 2 j=1 exp − 2 xd − θ j

zdk

(18) (19)

160

K. Harris, L. McMillan, and M. Girolami

M-step 2 N K D K N 1 1 log p(y, X, θ, w) ≥ − yn − wk θk n − γ(zdk ) (xnd − θkn )2 2 n=1 2 d=1 k=1 n=1 k =1

−

∂ log p(y, X, θ, w) = ∂θkn

yn −

K

K N K 1 1 2 (θkn − θ0n )2 − wk + const. 2h k=1 n=1 2l k=1

wk θ k n

k =1

wk +

D d=1

(20)

1 γ(zdk )(xnd −θkn )− (θkn −θ0n ) = 0. h (21)

y − θT w−k wk + Xγ k + h1 θ 0 ⇒ θk = , 1 wk2 + D d=1 γ(zdk ) + h

(22)

where w −k is w with the k th element set to 0 and γ k is the D × 1-dimensional column of the D × K matrix of responsibilities [γ(zdk )]. N K ∂ log p(y, X, θ, w) 1 = yn − wk θk n θkn − wk = 0. (23) ∂wk l n=1 k =1 −1 1 ⇒ w = θθT + I θy. (24) l A.2

Extension to Binary Classification

Modelling Joint distribution. The joint distribution now becomes p(t, y, X, θ, w) = p(t, y|θ, w)p(X|θ)p(θ)p(w). Classification model.

tn =

1 if yn > 0 −1 otherwise.

yn = wT θ n + n where n ∼ N (0, 1). ⇒ p(t, y|θ, w) =

N

(25)

(26) (27)

p(tn , yn |θ n , w)

(28)

p(tn |yn )p(yn |θn , w)

(29)

p(tn |yn )N yn |wT θn , 1 ,

(30)

n=1

=

N

n=1

=

N

n=1

Inferring Meta-covariates in Classification

where

δ(yn > 0) if tn = 1 p(tn |yn ) = δ(yn ≤ 0) otherwise.

161

(31)

E-step. Then, by taking logarithms and applying Jensen’s inequality, we obtain the following result: Ey [log p(t, y|θ, w)] ≥

N

log p(tn |E(yn ))N E(yn )|w T θn , 1 .

(32)

n=1

δ(yn > 0)N yn |wT θ n , 1 if tn = 1 yn |tn , θ, w ∝ δ(yn ≤ 0)N yn |wT θ n , 1 otherwise. φ(−wT θ n ) wT θn + 1−Φ(−w if tn = 1 Tθ ) n ⇒ E(yn ) = T φ(−w θ ) wT θn − Φ(−wT θnn ) otherwise.

(33)

(34)

We now see that p(tn |E(yn )) = 1 and equation (32) simpliﬁes to Ey [log p(t, y|θ, w)] ≥

N

log N E(yn )|w T θn , 1

(35)

n=1

=−

N 2 N 1 E(yn ) − w T θn − log(2π). 2 n=1 2

(36)

We thus see that the only diﬀerence between equations (9) and (36) is that yn is replaced by E(yn ). Hence, the E-step now involves evaluating E(yn ) using equation (34), in addition to evaluating the responsibilities γ(zdk ) using equation (19). M-step. As the clustering model and the prior distributions are left unchanged, the M-step also remains unchanged except for y being replaced by E(y) in equations (22) and (24).

A Multiobjective Evolutionary Algorithm for Numerical Parameter Space Characterization of Reaction Diﬀusion Systems Tim Hohm and Eckart Zitzler Computer Engineering and Networks Laboratory ETH Zurich 8092 Zurich, Switzerland {tim.hohm,eckart.zitzler}@tik.ee.ethz.ch http://www.tik.ee.ethz.ch/sop/

Abstract. Mathematical modeling is used to assist in studying complex biological systems. Still, setting up and characterizing models pose challenges of its own: identifying suitable model parameters, even when highresolution time course data concerning the system behavior is available, is a diﬃcult task. This task is further complicated when this high-resolution data remains unavailable like for the tissue level systems considered in developmental biology—a type of systems we focus on in the present study. In addition, costly simulations for tissue level systems prohibit excessive simulations during the parameter estimation phase. Here, we propose an approach that is dedicated to assist in the task of parameter space characterization for reaction diﬀusion models—a common type of models in developmental biology. We investigate a method to numerically identify boundaries that partition the parameter space of a given model in regions that result in qualitatively diﬀerent system behavior. Using an Evolutionary Algorithm (EA) combined with an Artiﬁcial Neural Network (ANN), we try to identify a representative set of parameter settings minimizing the distance to such boundaries. In detail we train the ANN on numerical data annotated using analytical results to learn the mapping between parameter space and distances to boundaries, thereby guiding the optimization process of the EA to identify such a set of parameter settings. The approach is tested with respect to its boundary identiﬁcation and generalization capabilities on three diﬀerent reaction diﬀusion systems—for all three we are capable of reliably identifying boundaries using the proposed approach.

1

Introduction

Mathematical modeling is a powerful tool to help understanding processes in complex biological systems [14,20,22]. Especially in the ﬁeld of developmental biology a certain type of models, so called reaction diﬀusion systems, are among the most cited approaches [20]. Dating back to Turing [21] diﬀerent reaction diﬀusion systems are used to explain a range of pattern formation in diﬀerent V. Kadirkamanathan et al. (Eds.): PRIB 2009, LNBI 5780, pp. 162–174, 2009. c Springer-Verlag Berlin Heidelberg 2009

A Multiobjective Evolutionary Algorithm

163

biological systems [5,11,12,13,16,23]. Although mathematical modeling thereby has proven its value in studying biological systems, the task of setting up suitable models poses some challenges of its own: after translating a set of initial hypothesis in a model structure, the usually parameterized models need to be tuned, i.e., the model parameters need to be optimized in order to minimize the deviation between known experimental data and simulation output. This task is complicated especially for tissue level systems considered in developmental biology for which in many cases only scarcely high-resolution quantitative timecourse data is available and due to the fact that the interactions between model components tend to be non-linear [8,17]. In addition, simulations for tissue level simulations are computationally expensive which prohibits excessive simulations during the parameter estimation phase. In the literature, there are mainly three diﬀerent approaches to tackle the afore mentioned task of parameter estimation for multi cell systems where mostly qualitative data is used: (i) tuning model parameters by hand [11,23], (ii) exploiting analytical results concerning the system to guide parameter choices [12], (iii) employing optimization techniques to minimize deviations between experimental data and simulation output [8,17]. All three techniques suﬀer from some limitations: tuning by hand and analytical characterizations of target systems are feasible only for small systems due to the increasingly cumbersome process of the analytical characterization for growing numbers of involved independent parameters and due to the fact that the size of respective parameter spaces grows exponentially with the system size and single simulations are computationally costly. This prohibits the necessary number of tuning steps by hand. Optimization techniques in turn are capable of handling up to mid-sized models but with further growing systems as well suﬀer from the exponential explosion of the parameter space, complexly structured parameter spaces due to non-linear dependencies between model components, and costly simulations. To address the main problem of exponential explosion of these complexly structured parameter spaces we propose a method where we couple an analytical approach with an optimization method. By exploiting analytical results to guide the optimization process, our approach is supposed to numerically reveal the structure of the parameter space comparable to what an analytical characterization would yield. Thereby, we could learn about for instance qualitatively diﬀerent system behaviors a considered model is capable of showing and since the target behavior is described in rather qualitative terms, a parameter setting contained in a matching region of the parameter space should show good agreement with a sought target behavior. In addition, further ﬁne tuning should be possible either by hand or by using one of the afore mentioned optimization techniques where the searched parameter space is restricted to parameter space partitions identiﬁed by our approach. Focusing on reaction diﬀusion systems, in detail we use analytical results gained for a simple system (a simpliﬁed variant of the activator inhibitor system [12]) and machine learning techniques (ANNs [3]) in order to train a predictor to estimate the distance of a given parameter setting from a boundary in parameter

164

T. Hohm and E. Zitzler

space that discriminates between qualitatively diﬀerent system behaviors. This predictor ANN is then used in a stochastic optimization technique (EAs [1,4]) to guide the search process to identify a well distributed set of parameter settings constituting boundaries in parameter space between qualitatively diﬀerent behaving regions of the parameter space. Due to the fact that the general principles inducing diﬀerent system behaviors are similar for all reaction diﬀusion systems, namely the concept of local self-activation and long range inhibition [5,13], the predictor ANN trained on data for the simple reaction diﬀusion system is supposed to generalize well for other systems. After testing the ANN and EA on the activator inhibitor system used for training the ANN, we provide a proof of principles concerning the generalization capabilities of our approach by applying it to two further reaction diﬀusions systems: an activator substrate system [13] and the Brusselator system [16]. In the following, we will ﬁrst introduce the concept underlying our approach and give a detailed description of the approach itself (Sec. 2). We then show results gained for the three considered test systems, activator inhibitor system, activator substrate system, and Brusselator (Sec. 3), and conclude the paper with some ﬁnal remarks (Sec. 4).

2

Approach

Brieﬂy summarizing the concept underlying our approach, we propose to use an EA in order to identify parameter settings for reaction diﬀusion systems that delimit regions in parameter space resulting in qualitatively diﬀerent system behavior. To guide the EA we employ an ANN that was trained using analytical characterization data for a simple reaction diﬀusion system in order to predict for a given parameter setting its distance to such a boundary. In detail we use analytical information concerning a simpliﬁed version of the activator inhibitor system as found in the appendix of [12]. Numerically simulating a grid of parameter settings covering the interesting part of the parameter space, we generate time course data that shows the typical behaviors this system is capable of generating (cf. Fig. 1). Since due to peculiarities of the used integrators the empirical boundaries a slightly shifted with respect to the analytically determined boundaries, we manually adjust the theoretical boundaries to the numerical data. We then compute the shortest distance of each parameter set to the empirical boundaries. In addition we process the numerical data in order to reduce it to some meta characteristics that capture important features to determine to which qualitative region a given parameter setting belongs—a necessary step to allow that the considered characteristics become invariant to the exact speciﬁcation of the considered reaction diﬀusion system and therefore allow for generalization. Using these meta characteristics as inputs and the calculated distances as outputs, an ANN is employed to learn the mapping from input characteristics to distance from parameter setting to a boundary in parameter space. This predictor is then used to guide an EA in order to identify boundaries delimiting regions in parameter space resulting in qualitatively diﬀerent behavior of a given system. In this

A Multiobjective Evolutionary Algorithm

165

Fig. 1. Time courses representing the three qualitatively diﬀerent behaviors of the one dimensional activator inhibitor system. The y-axis of each plot shows the reached concentrations while the x-axis represents time and each of the 100 curves per plot represents the time behavior of the activator of a single cell of the system: (a) a typical oscillating system, (b) a spatially heterogeneous pattern, and (c) a spatially homogeneous behavior. Phase 1: Learning

Phase 2: Parameter Space Exploration Optimization Loop

Training Process

ANN

Numerical Data

Coverage

ANN

EA

Objective Value

Training Data

Objective 2

Simulation

Parameter Setting

Expert Processing

Objective 1

Objective Value

Analytical Data

Putative Boundary Points

Fig. 2. Sketch of the training phase of the ANN as a predictor for the distance between a parameter setting and the closest boundary in parameter space delimiting regions resulting in qualitatively diﬀerent behavior (left) and the EA that identiﬁes points on such a boundary for an unknown reaction diﬀusion system, building on the ANN (right)

context, the EA generates a parameter setting which is then simulated. Using the simulated time course, the inputs for the ANN are determined and building on the prediction of the ANN with respect to the distance to a boundary, the EA than reﬁnes the proposed parameter setting in order to better match a supposed boundary. Both, the training process of the ANN and the EA, are sketched in Fig. 2 and further details are given in the following. 2.1

Training Data Generation

To generate the training data for the ANN we use an already analytical characterized simpliﬁed variant of the activator inhibitor system [12] given by the following equations: ∂a a2 = DΔa + ξ − a + σ ∂t h ∂h = Δh + ξμ(a2 − h) ∂t

(1) (2)

166

T. Hohm and E. Zitzler

This system consists of two interlinked species, an activator a and an inhibitor h. Their respective time behavior is described by partial diﬀerential equations that in addition to time depend on spatial information: a diﬀusion term represented by the Laplace operator Δ and a diﬀusion constant D. Both species encompass a reaction term, perturbed by a uniformly random value ξ ∈ [0.95, 1.05], and a decay term. The reaction- and decay term of the inhibitor are quantiﬁed with a constant μ. In addition, the activator contains the term σ that represents basal expression. This system depends on three constants: D, σ, and μ. To generate numerical data for this system, we consider an implementation of this system in a one-dimensional spatial domain consisting of 100 cells with periodic boundary conditions. We sample the parameter space using an equidistant grid of 5000 parameter settings on the parameter sub-space spanned by (D, μ) and ﬁx σ = 0.001 as well as the initial conditions (ai , hi ) = (0.01, 0.01) of all cells i for both species a and h. The grid spans (D, μ) = [0.006, 0.3] × [0.04, 4] with respective steps of (0.006, 0.04). For numerical integration we consider the interval [0, 1000] of dimensionless time and use an implicit explicit scheme consisting of a modiﬁed Crank-Nicolson integrator and an Adams-Bashford integrator [18]. For time discretization we use a time step of δt = 0.125 and for space discretization we apply a spatial grid in cellular resolution. After identifying the analytically determined boundaries in (D, μ) parameter space in the numerical data, we compute the shortest Euclidean distance for each simulated parameter setting to these boundaries after normalizing the (D, μ) = [0, 0.3] × [0, 4] parameter space to [0, 1] × [0, 1]. The resulting distances are shown in Fig. 3. After thereby generating the outputs used for training the ANN, in a last step we need to reduce the integration data (per parameter setting an n × m matrix X with n being the number of cells and m being the number of considered time points) to a set of meta characteristics that capture system invariant features that allow the ANN to learn the mapping between parameter setting (represented by the features) and the shortest distance to a boundary in parameter space. Analyzing the available time course data we found out that in principle two characteristics should be suﬃcient to characterize the diﬀerent parameter settings: (i) the spatial diﬀerence occurring between all cells during a late

(b) 0.4

0.3 0.2

0.35 0.3 0.25 0.2 0.15 0.1 D 0.05

0.1 0 -0.1 4

3.5

3

2.5

m

2

1.5

1

0.5

0

0

Distance

Distance

(a)

0.3

0.35 0.3 0.25 0.2 0.15 0.1 D 0.05

0.2 0.1 0 4

3.5

3

2.5

m

2

1.5

1

0.5

0

0

Fig. 3. Normalized distances of each parameter setting in the (D, μ) = [0, 0.3] × [0, 4] parameter space as determined from the simulation data (a) and predicted by the ANN (b)

A Multiobjective Evolutionary Algorithm (a)

Concentration

10

10

10

10

10

(b)

6

6 10

4

4 10

Concentration

10

167

2

0

-2

2 10 0 10 -2 10

-4 0

200

400

600

800

-4 10

1000

0

200

(c)

400

600

800

1000

Simulated Time

Simulated Time

(d)

Period 3

1

Discretization Level

2.5

Occurrences

Window 2 1.5 1 0.5 0 0 0

200

400

600

Simulated Time

800

1000

0

5

10

15

20

25

30

35

40

45

Periods

Fig. 4. A sketch of the process used to estimate the dominant oscillation period in time course data: (a) time course data of the activator for a 100 cell activator inhibitor system, (b) time courses are reduced to a single time course that represents the maximum for each time point over the 100 cells (solid curve) and the threshold δthresh used for discretization (dashed curve), (c) discretized time course data for which the periods between ‘1’ peaks are computed, and (d) histogram of the calculated periods and the sliding window used to determine the dominant period in terms of occurrences

integration step and (ii) the dominating oscillation period estimated from the data (for non-oscillating time courses it can be set to a very small positive value, here 10e−14 ). These two characteristics have the advantage that they are invariant with respect two variations of the simulated domain, both in numbers of cells and changes of dimensionality. Still, these two characteristics allow to capture features discriminating between oscillatory and stable system behavior and spatially homogeneous or heterogeneous states. When in addition considering these two characteristics only for the activator, we gain further invariance with respect to possible other realizations of an inhibition, e.g., instead of a direct inhibition by an inhibitor, inhibition could be realized by depleting a substrate. While the computation of the spatial diﬀerence is a straight forward procedure, we brieﬂy explain how we estimate the dominant oscillation period. In a ﬁrst step, we reduce the existing time course data X to a consensus time course Xmax by taking the maximum over all cells for each time point. This has the advantage of generating a more regular signal since due to the stochastic ξ terms the considered time course might show some irregularities in single cells. In a second step we discretize the data using a threshold δthresh = 12 mean (Xmax ). For all time points where Xmax ≥ δthresh , the discretized time course data Xdisc is set to 1 and 0 otherwise. Then, the periods between ‘1’ peaks are determined and gathered in a histogram with buckets encompassing 5 time steps. Using a sliding window covering 5 consecutive buckets, the period with the most occurrences is

168

T. Hohm and E. Zitzler

determined where in case of equal occurrences we have a preference for shorter periods. The process of determining the dominant period is sketched in Fig. 4. 2.2

Artificial Neural Networks

To learn the mapping between inputs extracted from the time course data of the numerical integration to the distance of the respective parameter setting to a boundary in parameter space delimiting partitions of qualitatively diﬀerent system behavior, we chose to use ANNs [3]. Since learning the described mapping is a regression problem, we decided to choose feed forward multi-layer perceptrons with two layers of adaptive weights and in total three layers: an input layer with two neurons, a hidden layer of 50 neurons with hyperbolic tangent functions as activation functions, and an output layer with a single neuron and a linear activation function. To further enhance the predictive capabilities of the ANN, instead of a single ANN we decided to train an ensemble of ANNs [19]. In detail we use the W-SECA procedure proposed by Grannito et al. [6] to construct the ANN ensemble where the ensemble prediction is the weighted mean of all ANNs in the ensemble, using the following weighting function wi for each ensemble member i [6]: e−2 wi = i −2 (3) j ej Here, e is the prediction error of an ensemble member with respect to the data set containing all 5000 data points and j iterates over all ensemble members. Since the single input values can become rather large, to facilitate training we transform the inputs by taking their logarithm. Each ensemble member is trained using the scaled conjugate gradients algorithm [3]. For training, the available data is divided in a training set and a validation set using bootstrapping: the training set consists of 5000 bootstrap samples while the not-sampled points are used as validation set. The ANNs are trained using the training set until the prediction error for the validation set in successive training epochs gets worse. Ensemble predictions of the ensemble used in the following for all parameter settings used during training are shown in Fig. 3. 2.3

Evolutionary Algorithm

Aiming at the identiﬁcation of a representative set of parameter settings delimiting regions in parameter space resulting in qualitatively diﬀerent system behavior, we complement the optimization criterion of minimizing the distance to a boundary by a second objective: the coverage of the parameter space by identiﬁed parameter settings. In order to identify trade-oﬀs between these two objectives we apply the Multiobjective Covariance Matrix Adaption Evolution Strategy1 (MO-CMA-ES) [10], belonging to a class of EAs designed to identify 1

We slightly vary the original MO-CMA-ES: instead of the exact hypervolume we use a Monte Carlo sampling method for hypervolume estimation [2] during environmental selection [9].

A Multiobjective Evolutionary Algorithm

169

compromises between conﬂicting objectives like distance to a boundary and coverage of parameter space; the MO-CMA-ES already showed good results in a comparable situation where on top of the core optimization criterion the coverage of the parameter space had to be considered [9]. The coverage of the parameter space is assessed using a criterion proposed in [9]: the parameter settings xi in a population G of an EA are ranked using their distance to uniformly random parameter settings xj ∈ S (see [9] for a detailed description). In total we draw |S| = 29958 random parameter settings, following Hoeﬀding’s inequality [7] resulting in a probability of 0.95 of resulting in an error in coverage computation err ≤ 0.01 for the considered two dimensional normed parameter space [0, 1]2 .

3

Simulations and Results

In the following we present results of our approach on three test systems: as a proof of principle we use the activator inhibitor system that was used for training data generation; to test the generalizability we use two further, conceptually diﬀerent systems. Before we present the obtained results, we brieﬂy describe the used experimental setup as well as the means of validation for the found settings. 3.1

Methodology

For the optimization process, on each system we used the same ANN ensemble and ran the EA 10 times. For each EA run we allowed 2500 function evaluations using a population size of 50. For the HypE function [2] employed during environmental selection we used 10000 samples and the reference point (1, 100) for the two objectives (i) distance to boundary and (ii) coverage of the parameter space. Each of the EA runs took approximately 2 days on a two chip dual core AMD R Opteron 2.6GHz 64-bit machine with 8GB RAM using MATLAB 7.6 (R2008a) and the NETLAB [15] implementation for ANNs and related algorithms. For the evaluation of the EA runs we considered two diﬀerent factors: (i) the reproducibility of the identiﬁed sets of parameter settings over all EA runs for each system, and (ii) the goodness of the identiﬁed boundaries. Although the reproducibility of the found sets of parameter settings is diﬃcult to asses quantitatively, nevertheless, visual inspection of the sets clearly showed that certain sub spaces contained no identiﬁed parameter settings while others were well-populated for all runs, we deem this visual inspection suﬃcient to document the reproducibility. In order to validate the identiﬁed boundaries we used two diﬀerent approaches: since for the activator inhibitor system the boundaries are known, we visually compared the identiﬁed parameter settings to the known boundaries. For the remaining two systems we validated the putative boundaries inferred from the identiﬁed parameter settings by probing the behavior around the putative boundaries: we simulate parameter settings residing on vectors orthogonal to the assumed boundaries in order to test if a qualitative change of system behavior occurs in the vicinity of the putative boundaries. In addition, using the same probing technique we test if parameter settings located in regions for which no boundary is detected exhibit qualitatively similar behavior.

170

3.2

T. Hohm and E. Zitzler

Proof of Principle

We used the proposed method to identify boundaries partitioning the parameter space in regions resulting in qualitative diﬀerent system behavior for the activator inhibitor system (Eqs. 1–2) that was used for training the ANN ensemble. In a ﬁrst go we observed that the coverage of the parameter space became worse during the optimizations process corresponding with a reduction in number of distinct parameter settings constituting the estimated boundaries. Eventually, the algorithm converged ending up with only one or two parameter settings. When analyzing the landscape of distances predicted by the ANN ensemble, we found out that although the general distance landscape is in good agreement with the calculated distances (see Fig 3) not all boundary constituting parameter settings are mapped to the same globally optimal value: for example in the the region with small D-values and large μ-values the predicted distances become negative and in terms of minimization better than those for other boundary points. Thereby, our approach traded oﬀ coverage for concentrating on the regions containing negative values. In order to prevent these false global optima from dominating the optimization process we decided to cut-oﬀ the predicted distance values on the level of 0. Thereby we achieve that most boundary constituting parameter settings are mapped to the globally optimal value of 0 but at the same time introduce some false positive boundary points, e.g., again in the region with small D-values and large μ-values. Using this modiﬁcation the boundary determined by our approach is in good agreement with the known boundary (see Fig. 5). Still, the (D, μ) ∈ [0, 0.3] × [0, 1.56] regime corresponding to oscillating system behavior contains a considerable number of false positive settings. When again checking the predicted distances (see Fig. 3a) it can be seen that these false positive settings correspond to narrow spikes in the predicted landscape—a fact that could be addressed either by considering the robustness of the predicted distance to a boundary with respect to some sort of neighborhood around the considered parameter setting or by further reﬁning the training process of the ANNs, e.g., by including regularization terms to smoothen ANN outputs by preventing possible over ﬁtting. Nevertheless, although a number of parameter settings corresponds to false positive boundary points, the approach in its current form already clearly shows that large parts of the parameter space belong to qualitatively similar regions and therefore can be neglected. 3.3

Test of Generalizability

After this proof of principle, we tested the generalization capabilities of our approach by running it on the remaining two test systems. When checking the data for the activator substrate system (Eqs. 4–5), the identiﬁed parameter settings clearly outline a boundary from small D-values and large μ-values towards large D-values and small μ-values. To validate if these settings constitute a true boundary between qualitatively diﬀerently behaving parameter space regions we probed the behavior in a neighborhood around the putative boundary using the vectors shown in Fig. 5b. Evaluating the corresponding simulations we could

A Multiobjective Evolutionary Algorithm (a)

(b)

4

4

3.5

3.5

3

3

2.5

2.5

m

m

2 1.5

3

2

b2

1.5

1

1 0.5

0

(c)

4

0.5 0

0.05

0.1

0.15

0.2

0.25

D

0.3

0

171

1 0 0

1

2

a 0

0.05

0.1

0.15

0.2

0.25

3

0 0.1 0.2 0.3 0.4 D 0.5 4

0.3

D

Fig. 5. Plots showing the probable boundary delimiting parameter settings identiﬁed for the three test systems: (a) identiﬁed parameter settings (circles) and analytically determined boundary points (squares) for the activator inhibitor system, (b) identiﬁed parameter settings for the activator substrate system and the used probing vectors, and (c) identiﬁed parameter settings for the Brusselator as well as the used probing vectors and the hyperplane outlining an assumed boundary

conﬁrm that on the lower boarder of the identiﬁed boundary the system shows a change in behavior from a spatially heterogeneous pattern (lower region in Fig. 5b) to a spatial homogeneous pattern (upper region in Fig. 5b). In addition, along the probing vectors located in regions for which no boundary was predicted, indeed no qualitative change in system behavior could be observed. When looking at the putative boundary constituting parameter settings identiﬁed for the Brusselator (see Fig. 5c), one recognizes that identifying boundaries becomes increasingly more diﬃcult when dealing with higher dimensional search spaces especially when the boundaries stem from non-linear relations between parameters. Still, we have been able to identify a hyperplane outlined by found parameter settings. Using the same probing approach (see Fig. 5c for exact location of hyper plane and probing vector) to validate this putative boundary, we observed a change from spatially homogeneous timely stable solutions to timely oscillations when following the probing vector in direction of increasing b. Again, probing regions that according to our approach were not supposed to contain boundaries showed no qualitative change in system behavior.

4

Conclusions

In this study we investigated the proposed approach to exploit analytical information in order to numerically characterize reaction diﬀusion systems. Using an EA, we tried to identify parameter settings that constitute boundaries that partition the parameter space in regions showing qualitatively diﬀerent system behavior. To guide the search process of the EA we employed an ANN ensemble which was trained using numerical data generated for a simple reaction diﬀusion system and annotated with analytical results. We tested our approach on three diﬀerent reaction diﬀusion systems, the activator inhibitor system that was used

172

T. Hohm and E. Zitzler

for training data generation, and two conceptually diﬀerent reaction diﬀusion systems: an activator substrate system and the Brusselator. With the presented results we documented the reliable identiﬁcation of parameter settings residing on boundaries in parameter space as well as the generalizability of our approach for diﬀerent reaction diﬀusion systems. In order to further test out approach we plan to apply it to new and larger systems—although the results obtained for the Brusselator indicate that it might be necessary to generate exponentially growing numbers of parameter settings to reliably outline boundaries in high-dimensional parameter spaces as well as it could become diﬃcult to infer the putative boundaries outlined by the identiﬁed parameter setting with growing dimensionality. Addressing these concerns it could be interesting to slightly alter the scope of our approach: although knowing the complete structure of the parameter space provides valuable information concerning the characterization of a system, in many situations it is suﬃcient to identify a region in parameter space showing a certain qualitatively behavior. Therefore it should be possible to train an ANN ensemble, instead for boundary identiﬁcation, for identiﬁcation of a region in parameter space showing the target behavior. In turn, a small number of parameter settings is suﬃcient to, e.g., indicate the centroid of such a region, as well as it solves the problem of having to derive the exact location of a putative boundary from a set of parameter settings.

References 1. B¨ ack, T., Fogel, D.B., Michalewicz, Z. (eds.): Handbook of Evolutionary Computation. IOP Publishing and Oxford University Press (1997) 2. Bader, J., Zitzler, E.: HypE: An Algorithm for Fast Hypervolume-Based ManyObjective Optimization. TIK Report 286, Computer Engineering and Networks Laboratory (TIK), ETH Zurich (November 2008) 3. Bishop, C.M.: Neural Networks for Pattern Recognition. Oxford University Press, Oxford (1995) 4. Foster, J.A.: Evolutionary Computation. Nat. Rev. Genet. 2(6), 428–436 (2001) 5. Gierer, A., Meinhardt, H.: A theory of biological pattern formation. Kybernetik 12, 30–39 (1972) 6. Granitto, P.M., Verdes, P.F., Cecatto, H.A.: Neural network ensembles: evaluation of aggregation algorithms. Artif. Intell. 163, 139–162 (2005) 7. Hoeﬀding, W.: Probability Inequalities for Sums of Bounded Random Variables. Journal of the American Statistical Association 58(301), 13–30 (1963) 8. Hohm, T., Zitzler, E.: Modeling the Shoot Apical Meristem in A. thaliana: Parameter Estimation for Spatial Pattern Formation. In: Marchiori, E., Moore, J.H., Rajapakse, J.C. (eds.) EvoBIO 2007. LNCS, vol. 4447, pp. 102–113. Springer, Heidelberg (2007) 9. Hohm, T., Zitzler, E.: Multiobjectivization for Parameter Estimation: a Case-Study on the Segment Polarity Network of Drosophila. In: Raidl, G., et al. (eds.) Genetic and Evolutionary Computation Conference (GECCO 2009). ACM, New York (to appear, 2009)

A Multiobjective Evolutionary Algorithm

173

10. Igel, C., Hansen, N., Roth, S.: The Multi-objective Variable Metric Evolution Strategy, Part I. Technical Report IRINI 2005-04, Institut f¨ ur Neuroinformatik, RuhrUniversit¨ at Bochum, 44780 Bochum (2005) 11. J¨ onsson, H., Heisler, M., Reddy, G.V., Agrawal, V., Gor, V., Shapiro, B.E., Mjolsness, E., Meyerowitz, E.M.: Modeling the organization of the WUSCHEL expression domain in the shoot apical meristem. Bioinformatics 21, i232–i240 (2005) 12. Koch, A.J., Meinhardt, H.: Biological pattern formation: from basic mechanisms to complex structures. Rev. Mod. Phys. 66(4), 1481–1510 (1994) 13. Meinhardt, H.: Models of Biological Pattern Formation. Academic Press, London (1982) 14. Murray, J.D.: Mathematical Biology. Springer, New York (2003) 15. Nabney, I.T.: NETLAB: Algorithms for Pattern Recognition. In: Advances in Pattern Recognition, 2nd edn. Springer, Oxford (2003) 16. Prigogine, I., Lefever, R.: Symmetry Breaking Instabilities in Dissipative Systems. J. Chem. Phys. 48, 1695–1700 (1968) 17. Raﬀard, R., Amonlirdviman, K., Axelrod, J.D., Tomlin, C.J.: Automatic parameter identiﬁcation via the adjoint method, with application to understanding planar cell polarity. In: IEEE Conference on Decision and Control, Piscataway, NJ, USA, pp. 13–18. IEEE Press, Los Alamitos (2006) 18. Ruuth, S.J.: Implicit-explicit methods for reaction-diﬀusion problems in pattern formation. J. Math. Biol. 34(2), 148–176 (1995) 19. Sharkey, A.J.C. (ed.): Combining Artiﬁcial Neural Nets: Ensemble and Modular Multi-Net Systems. Springer, London (1999) 20. Tomlin, C.J., Axelrod, D.: Biology by numbers: mathematical modelling in developmental biology. Nat. Rev. Genet. 8, 331–340 (2007) 21. Turing, A.: The chemical basis for morphogenesis. Philos. Trans. R Soc. Lond, B 237, 37–72 (1952) 22. Voit, E.O.: Computational Analysis of Biochemical Systems. Cambridge University Press, Cambridge (2000) 23. Yamaguchi, M., Yoshimoto, E., Kondo, S.: Pattern regulation in the stripe of zebraﬁsh suggests an underlying dynamic and autonomous mechanism. Proc. Natl. Acad. Sci. USA 104(12), 4790–4793 (2007)

A

Further Test Systems

In the following we would like to brieﬂy introduce the remaining two test systems that were used in this study. To be able to test the generalization capabilities of the proposed approach we chose to use conceptually diﬀerent reaction diﬀusion systems compared to the activator inhibitor system used for training purposes; both realize the long-range inhibition by some sort of depleting substrate. Equations 4–5 thereby constitute the activator substrate system [5,12,13]. Alike to the activator inhibitor system used for training, we ﬁx σ = 0.001 and explore the thereby reduced (D, μ) parameter space. The remaining two Eqs. 6–7 form the Brusselator [16] for which we consider the three-dimensional (D, a, b) parameter space. In all four equations ξ represents a random perturbation uniformly drawn from the interval [0.95, 1.05].

174

T. Hohm and E. Zitzler

∂a ∂t ∂s ∂t ∂x ∂t ∂y ∂t

= DΔa + ξa2 s − a + σ

(4)

= Δs + ξμ(1 − sa2 )

(5)

= DΔx + a − (b + 1)x + ξx2 y + σ

(6)

= Δy + bx − ξx2 y

(7)

Knowledge-Guided Docking of WW Domain Proteins and Flexible Ligands Haiyun Lu1 , Hao Li1 , Shamima Banu Bte Sm Rashid1 , Wee Kheng Leow1 , and Yih-Cherng Liou2 1

2

Dept. of Computer Science, School of Computing, National University of Singapore, Singapore 117590 {luhaiyun,lihao,shamima,leowwk}@comp.nus.edu.sg Dept. of Biological Sciences, Faculty of Science, National University of Singapore, Singapore 117543 [email protected]

Abstract. Studies of interactions between protein domains and ligands are important in many aspects such as cellular signaling. We present a knowledge-guided approach for docking protein domains and ﬂexible ligands. The approach is applied to the WW domain, a small protein module mediating signaling complexes which have been implicated in diseases such as muscular dystrophy and Liddle’s syndrome. The ﬁrst stage of the approach employs a substring search for two binding grooves of WW domains and possible binding motifs of peptide ligands based on known features. The second stage aligns the ligand’s peptide backbone to the two binding grooves using a quasi-Newton constrained optimization algorithm. The backbone-aligned ligands produced serve as good starting points to the third stage which uses any ﬂexible docking algorithm to perform the docking. The experimental results demonstrate that the backbone alignment method in the second stage performs better than conventional rigid superposition given two binding constraints. It is also shown that using the backbone-aligned ligands as initial conﬁgurations improves the ﬂexible docking in the third stage. The presented approach can also be applied to other protein domains that involve binding of ﬂexible ligand to two or more binding sites.

1

Introduction

Protein domains are the fundamental units of tertiary structure of many proteins. One of the most important functions of protein domains is to bind speciﬁc ligands to assemble intracellular signaling networks to perform distinct biological functions. The number of deﬁned protein domains has expanded considerably in recent years. Studies of interactions between protein domains and their ligands are crucial for deeper insight of the binding aﬃnities involved. With this vital understanding target prediction of novel domain-binding ligands would be possible, allowing

This research is supported by NUS R-252-000-293-112.

V. Kadirkamanathan et al. (Eds.): PRIB 2009, LNBI 5780, pp. 175–186, 2009. c Springer-Verlag Berlin Heidelberg 2009

176

H. Lu et al.

for subsequent cloning and expression. Determination of possible target ligands by laboratory experimental techniques alone is a known bottleneck requiring intensive consumption of time and resources. Therefore computational techniques are needed to eﬀectively simulate domain bindings. Many protein docking algorithms have been developed to solve the problem. Two broad classiﬁcations [10] of docking algorithms are rigid docking and ﬂexible docking. Rigid docking algorithms solve a simpler version of the protein docking problem termed bound docking by reconstruction of a protein complex from the bound structures of the two proteins that constitute the complex [5,8,14,19,23,27,28]. Docking is framed as a rigid alignment problem of two rigid objects with complementary shapes. Flexible docking algorithms solve the general protein docking problem termed unbound or predictive docking by prediction of binding of two proteins in their free or unbound states [7,9,12,16,18,20,22,26]. This problem regards one or both proteins as ﬂexible objects to account for signiﬁcant conformational shape changes which occur during protein interactions. A ﬂexible molecule often presents a very large number of degrees of freedom posing great diﬃculty for the ﬂexible docking problem. Flexible docking should be used to predict possible binding and potential novel targets for protein domains as the correct conformations of novel targets are usually unknown. Generally, this remains a very diﬃcult and challenging task. Fortunately, known binding site characteristics of protein domains can be used to help solve the diﬃcult docking problem. The knowledge of binding sites of protein domains is very useful for predicting possible ligand bindings. General ﬂexible docking algorithms often make use of the binding site information. However, what information to use and how to use it for eﬀective and accurate ﬂexible docking is a challenge. For example, AutoDock [20] requires the user to specify a bounding box around the protein binding site in which an optimal ligand conformation is searched for. The amount of binding site information used in AutoDock is minimal and thus not very eﬀectively used. This paper presents an approach for docking protein domains and ﬂexible ligands using known binding site information as the constraints. Our approach uses known binding site knowledge to ﬁrst search for the protein domains and the ligand residues recognized by the domains. Then the ligand’s peptide backbone is aligned to the domains based on the binding constraints. Finally, existing algorithms can be used to perform ﬂexible docking, with the backbone-aligned ligands as the initial conﬁguration. To be speciﬁc, we apply the approach to WW domains as an application example in this paper.

2

WW Domains

WW domains are present in signaling proteins found in all living things. They have been implicated in signal mediation of human diseases such as muscular dystrophy, Alzheimer’s disease, Huntington’s disease, hypertension (Liddle’s syndrome) and cancer [3,11,24,25]. WW domains are distinguished by the characteristic presence of two signature tryptophan residues (W) that are spaced 20– 22 amino acids apart (Table 1). They are known to recognize proline-containing

Knowledge-Guided Docking of WW Domain Proteins and Flexible Ligands

177

Table 1. Residue sequences of sample WW domains and ligands Group I II/III IV

PDB WW domain sequence Ligand sequence 1EG4 HFLSTSVQGPWERAISPNKVPYYINHETQTTCWDHPKMTELYQ KNMTPYRSPPPYVPP 2DYF GSWTEHKSPDGRTYYYNTETKQSTWEKPDD GSTAPPLPR 1PIN KLPPGWEKRMSRSSGRVYYFNHITNASQWERPSGNSSSG

peptide ligands and they share similarities with other proline recognition domains such as SH3 and EVH1 domains [17,29]. WW domains are classiﬁed into four groups [11] based on ligand speciﬁcity. Group I binds to ligands containing Proline-Proline-‘Any amino acid’-Tyrosine (PPxY) motif (Table 1). Group II binds to ligands containing Proline-ProlineLeucine-Proline (PPLP) motif (Table 1). Group III recognizes Proline-rich segments interspersed with Arginine (R) residues. Group IV binds to short amino acid sequences containing phosphorylated Serine or Threonine followed by proline. Recent studies show that Group II and III WW domains have very similar or almost indistinguishable ligand preferences, suggesting that they should be classiﬁed into a single group [15]. Our study focuses on the ﬁrst three groups of WW domains, as fewer Group IV samples are available in RCSB Protein Data Bank (PDB) [2]. Examples of WW domains and their corresponding ligand amino acid sequences are presented in Table 1. Group I and II/III WW domains have two binding grooves that recognize ligands [24]. Group I WW domains contain the Tyrosine groove, Group II/III WW domains contain the XP2 groove and both groups contain the XP groove. A Tyrosine groove is formed by three residues, Ile/Leu/Val, His, and Lys/Arg/Gln, and it recognizes Tyrosine (Y) residue of the ligand. An XP groove is formed by Tyr/Phe and Trp residues whereas an XP2 groove is formed by Tyr and Tyr/Trp residues. Both recognize Xaa-Pro (P), including Pro-Pro, segments of the ligand. It is to be noted that the grooves are formed by non consecutive residues in the amino acid sequence because the WW domain protein folds in 3-D to give rise to the grooves (Fig. 1).

(a)

(b)

Fig. 1. WW domain proteins and ligands. (a) 1EG4: Group I WW domain complexed with β-dystroglycan peptide. (b) 2DYF: Group II WW domain complexed with a peptide ligand with PPLP motif. Gray: proteins containing WW domains, blue: Tyrosine groove, green: XP/XP2 grooves, red: ligands, yellow: groove-binding residues of ligands.

178

3

H. Lu et al.

Related Work

Flexible docking algorithms can be classiﬁed into three categories. Rigid docking with refinement methods perform rigid docking of the proteins followed by reﬁnement of their side chains [7,9,12,16,26]. By applying side chain reﬁnement, side chain ﬂexibility can be accounted for to improve docking results. The method of [9] performs optimization of both backbone displacement and side chain conformations based on simulated annealing Monte Carlo. The methods of [7,26] apply biased probability Monte Carlo minimization of the ligand-interacting side chains while [16] uses energy minimization. The algorithm in [12] uses side chain rotamers and rigid body minimization to relax the interfaces of docking results. These methods handle side chain ﬂexibility but not backbone conformational changes. To handle backbone ﬂexibility, HADDOCK [6] performs rigid-body docking followed by reﬁnement of both the backbone and the side chains using simulated annealing based on molecular dynamics (MD). Biochemical and biophysical interaction data such as chemical shift perturbation data resulting from NMR titration experiments are used so it is not a general docking algorithm. Incremental construction algorithms place ligand fragments one at a time at the binding sites of the binding protein [18,22]. They require the knowledge of binding sites to place ligand fragments at the sites. Their computation speed while satisfactory for smaller ligands remains unsuitable for large ligands. Energy minimization methods apply optimization algorithms to search for the ligand conformation with minimum binding energy [13,20]. The objective is to determine the ligand conformation with minimal binding energy. Various optimization algorithms can be applied such as simulated annealing, Monte Carlo and genetic algorithms. In particular AutoDock [20] uses a hybrid Lamarckian genetic algorithm to optimize an empirical energy function that includes van der Waals potential, hydrogen bonding, Coulombic electrostatics and desolvation. The computational cost of such an energy function is very high. So the number of degrees of freedom is often limited to reduce the search space in practice.

4

Knowledge-Guided Protein Docking

The 3 stages in our approach for docking WW domains and ﬂexible ligands are binding groove and motif search, backbone alignment and ﬂexible docking. 4.1

Binding Groove and Binding Motif Search

Given a WW domain protein with known group classiﬁcation (Group I or Group II/III). The two types of binding grooves presented in the WW domain are also known. Each binding groove is formed by residues in a special pattern (Table 2). Residues which form the binding grooves are determined by applying a substring search on the WW domain’s amino acid sequence. From the binding grooves, the corresponding ligand motifs (PPxY or PPLP) are also known. Ligand residues forming possible motifs recognized by the binding grooves on WW domains are determined by a substring search applied on the ligand’s amino acid sequence.

Knowledge-Guided Docking of WW Domain Proteins and Flexible Ligands

179

Table 2. Residue patterns of WW domain binding grooves Binding groove Pattern Tyrosine groove I H K L R V Q XP groove Y F XP2 groove Y Y W

4.2

Example 1EG4: ...WERAISPNKVPYYINHETQTTCW...

W 1EG4: ...WERAISPNKVPYYINHETQTTCW... 2DYF: ...WTEHKSPDGRTYYYNTETKQSTW...

Backbone Alignment

Our backbone alignment method performs ﬂexible alignment of a ligand’s backbone to binding sites given the two binding grooves of a WW domain protein and possible binding motifs of the ligand. A ligand’s residue sequence is divided into 3 segments according to the binding motifs. As an example the ligand sequence of 1EG4 complex (KNMTPYRSPPPYVPP) is divided into KNMTPYRS, PPPY and VPP. The middle segment, an instance of the PPxY motif, is ﬂanked by two residues recognized by two binding grooves of the WW domain. The backbone alignment method aligns the backbone of the middle segment to the WW domain such that the ﬂanking residues ﬁt the grooves. The knowledge of relative positions and orientations of the two ﬂanking residues with respect to the (grooves of) WW domain serve as binding constraints. The other two segments and ligand side chain atoms are added after backbone alignment. The bond angle and bond length between two neighboring atoms are assumed to be ﬁxed, but the torsion angle of a rotatable bond can change to give rise to various conformations of a protein molecule (Fig. 2(a)). Similar assumptions are made in existing ﬂexible docking algorithms. Let ai , i ∈ {1, 2, ..., n}, denote the positions of the n atoms in the middle segment of the ligand backbone (Figure 2(b)). The middle segment has n/3 residues because for each residue 3 backbone atoms N, Cα, and C are considered. The

a2

a i −1

a i +1

a n −1

ei bi a1 (a)

a3

ai (b)

li a n −2

an

Fig. 2. Backbone model. (a) The bond angle bi and bond length li between atoms ai and ai+1 are ﬁxed. However torsion of the bond (indicated by arrow) can rotate atom ai+2 to a new position. (b) Model of atoms in the backbone.

180

H. Lu et al.

two binding constraints specify the atom positions a0 , a1 , a2 and an−2 , an−1 , an , which represent the two ﬂanking residues. We denote the target positions of these constrained atoms as {a∗0 , a∗1 , a∗2 } and {a∗n−2 , a∗n−1 , a∗n }. To satisfy the constraint on the ﬁrst ﬂanking residue, rigid transformation is applied on the backbone to align {a0 , a1 , a2 } to {a∗0 , a∗1 , a∗2 }. To satisfy the constraint on the second ﬂanking residue, we introduce the cost function 1 an−j − a∗n−j 2 2 j=0 2

Cs =

(1)

Minimizing Cs minimizes the distances between the atoms in the last residue and their target positions. Since the backbone can twist but not bend or stretch, constraints on the bond angles and bond lengths should be incorporated in order to correctly deform the backbone to satisfy the constraints on the residues. To represent the stretching and bending constraints we introduce the bond direction ei and bond length li as illustrated in Figure 2(b). Since the ﬁrst three atoms are ﬁxed by constraints after their rigid transformation we have, for i ≥ 3, ai+1 = ai + li ei

(2)

Thus, given {a0 , a1 , a2 }, li and ei determine the positions of the other atoms. So, Cs can also be expressed in terms of li and ei . Since a bond cannot stretch li is ﬁxed to ensure this condition. Also, ei · ei+1 corresponds to the bond angle bi . So the bending constraint is encoded by the cost function Cb =

n−2 1 ei · ei+1 − e0i · e0i+1 2 2 i=1

(3)

where e0i · e0i+1 corresponds to the initial value of bond angle bi . Minimizing Cb minimizes the change of the bond angles while keeping the bond lengths ﬁxed. The peptide bond nearly always has trans conﬁguration since it is more energetically favorable than cis. The backbone omega torsion angles are limited to values of 180◦ ± 5◦ , except for proline residues. There is no limitation on the omega torsion angle for proline because it can be in either trans or cis conﬁguration. Additionally, WW domains often bind to proline-rich ligands and the average distribution of phi, psi torsion angles for polyproline stretches (4 or more consecutive prolines) is (−75◦ , +145◦) ± 10◦ [1]. Let ti denote the torsion angles formed by atoms ai−1 , ai , ai+1 , ai+2 . The torsional constraint is represented by the cost function Ct = ti − t0i 2 (4) ti is limited

where ti = −atan2(ei ei−1 · (ei × ei+1 ), (ei−1 × ei ) · (ei × ei+1 )) and t0i denote the preferred value of ti . Minimizing Ct minimizes the diﬀerence between torsion angles and their preferred values.

Knowledge-Guided Docking of WW Domain Proteins and Flexible Ligands

181

The total cost function for backbone alignment is then C = kb Cb + ks Cs + kt Ct

(5)

where kb , ks and kt are weighting factors. In Eq. 5, the independent variables are the ei ’s. Varying ei changes the torsion angles but not the bond angles because of Cb . Quasi-Newton algorithm [21] is applied to compute the optimal ei that minimizes C to yield the aligned conﬁguration of the ligand’s backbone. 4.3

Flexible Docking

In the second stage knowledge of WW domain binding speciﬁcity is eﬀectively used to obtain backbone-aligned ligands. In the third stage, such ligands are used as starting conﬁgurations for ﬂexible docking. Any existing ﬂexible docking algorithm can be employed in our approach.

5

Experiments

Known WW domain binding targets are used in experiments to test the performance of our approach. 14 WW domain proteins complexed with prolinecontaining ligands were collected from RCSB Protein Data Bank (PDB) [2]. Complexes 1EG4, 1K9R, 1K5R, 1JMQ, 1I5H, 2JO9, 2DJY form WW domain Group I test cases and 2HO2, 2OEI, 2DYF, 1YWI, 2JUP, 2RLY, 2RM0 form Group II/III test cases. The WW domain proteins were separated from their ligands. Molecular Dynamics (MD) simulations were run using the AMBER program [4] to simulate possible unbound ligand conformations. Firstly, the backbone alignment algorithm was tested against rigid superposition given binding site knowledge. Backbone alignment was performed 30 times for each test run. The results are ranked according to the cost computed by Eq. 5 and only the top ranked alignments are recorded. Rigid superposition was performed for each test case based on known binding placements of the two ligand residues which bind to two binding grooves of WW domain. Least squares ﬁt is used to compute the rigid transformation of ligand to minimize the distance between the two binding residues and their ideal positions in binding sites. Results evaluation was performed by comparing the backbone atoms N, Cα and C between two binding constraints of the ligand with those in the bound structure and computing the root mean square deviation (RMSD)(Table 3). The average RMSD of results produced by backbone alignment method is 0.30˚ A with a standard deviation of 0.21˚ A and the average RMSD results produced by rigid superposition is 1.54˚ A with a standard deviation of 0.72˚ A. It is obvious that our backbone alignment method produced better results than rigid superposition. Figure 3 visualizes the results of backbone alignment compared with rigid superposition for 14 test cases. Ligand backbone conformations between two binding constraints resulting from our method are very close to bound structures with RMSD smaller than 0.5˚ A in all cases except 1JMQ. However the placements

182

H. Lu et al.

Table 3. RMSD(˚ A) of backbone alignment results and rigid superposition results Test Case Backbone Alignment Rigid Superposition 1EG4 0.30 1.38 1K9R 0.13 1.77 1K5R 0.28 1.50 1JMQ 0.94 1.71 1I5H 0.42 1.23 2JO9 0.13 0.59 2DJY 0.30 1.02 2HO2 0.27 1.39 2OEI 0.34 1.13 2DYF 0.48 1.99 1YWI 0.19 3.41 2JUP 0.18 0.54 2RLY 0.17 2.19 2RM0 0.12 1.67

produced by rigid superposition are far from optimal, producing good results for only two test cases 2JO9 and 2JUP where input ligand shapes are similar to the bound structures. When input ligand shapes diﬀer greatly from native complexes such as in test cases 1K9R, 1YWI, 2RLY and 2RM0 rigid superposition fails to create good ligand placements satisfying the two binding constraints. To further test our approach’s performance one of the most widely used ﬂexible docking programs, AutoDock was employed at the third stage of our scheme in three experimental settings. In the ﬁrst setting top-ranked ligand backbone alignments generated by the second stage were used as initial conﬁgurations for AutoDock. In the second setting ligand placements resulting from rigid superposition in the previous experiment were used as initial conﬁgurations. In the third setting, AutoDock was run using random initial ligand placements. The conﬁguration ﬁles for AutoDock were prepared using AutoDockTools. The grid parameter ﬁles specifying the 3D search space were manually deﬁned to surround the two WW domain binding sites. The WW domain protein (receptor) was held rigid and the ligand ﬂexible. As AutoDock limits the number of torsional degrees of freedom to 32 AutoDockTools were used to select up to 32 bonds whose torsional rotations moved the most number of atoms. Several parameters were adjusted for all settings in the same way. For example maximum number of energy evaluations was set to 25,000,000 and maximum number of generations to 27,000. 50 AutoDock runs were performed for each test case in each experimental setting and solutions were ranked according to their scores evaluated by AutoDock. Usually a solution with RMSD smaller than 2˚ A is classiﬁed as successful docking and is considered a very good result. A solution with RMSD smaller than 3˚ A is classiﬁed as partially successful. The docking is considered a success if the top-scored solution is successful for each test case. We relaxed the criteria in our evaluation, because most of our test cases possess a large number of degrees

Knowledge-Guided Docking of WW Domain Proteins and Flexible Ligands

1EG4

2HO2

1K9R

2OEI

1K5R

2DYF

1JMQ

1YWI

1I5H

2JUP

2JO9

2RLY

2DJY

2RM0 (a)

183

(b)

Fig. 3. The backbone alignment results (green) are closer than rigid superposition results (red) to the bound structure (blue). (a) WW domain Group I test cases. (b) Group II/III test cases.

of freedom which make the docking problem extremely diﬃcult. We considered the docking successful if one of the top 10 score-ranked solutions has RMSD smaller than 3˚ A. Table 4 lists RMSD results for all three experimental settings. Only 3 test cases (1K9R, 2OEI and 1YWI) are successful in the second and third settings. They all have short ligands comprising of at most 9 amino acids and 12 torsional degrees of freedom. The diﬃculty of docking small ﬂexible ligands is relatively low and AutoDock is able to produce good docking results. However for more diﬃcult test cases AutoDock failed to give successful docking solutions using initial ligand structures produced by rigid superposition or using random initial ligand placement. Unsurprisingly AutoDock results in the second and third settings are fairly close to each other. The reason is that rigid superposition is unable to produce optimal ligand conformations that satisfy the two binding constraints and thus initial ligand structures used by AutoDock in the second

184

H. Lu et al.

Table 4. RMSD analysis of AutoDock results in three experimental settings. The minimum, maximum and average RMSD (˚ A) of the top 10 ranked solutions are listed. Test Torsion Backbone Alignment Rigid Superposition Cases DoF min max avg min max avg 1EG4 41 4.05 8.73 6.79 4.36 11.56 7.22 1K9R 12 2.80 5.07 3.71 2.86 5.93 4.71 1K5R 24 3.33 6.20 4.43 3.73 7.60 5.69 1JMQ 24 3.46 6.08 4.84 3.80 5.71 4.78 1I5H 50 4.68 6.71 5.62 4.76 8.92 7.42 2JO9 28 3.52 6.86 5.08 4.30 7.62 5.92 2DJY 65 4.51 7.87 6.47 5.41 8.24 7.26 2HO2 13 3.73 6.45 5.15 3.03 5.75 4.57 2OEI 12 2.93 5.00 4.16 2.18 4.35 3.27 2DYF 25 2.68 6.72 4.41 3.74 7.85 5.43 1YWI 9 1.90 4.94 3.86 2.48 4.34 3.15 2JUP 16 2.59 5.93 4.39 4.10 6.06 5.36 2RLY 14 2.57 5.86 4.31 3.06 4.13 3.58 2RM0 15 3.43 5.49 4.51 3.19 5.55 4.31

Random Placement min max avg 4.62 9.66 6.52 2.85 6.38 4.65 3.81 7.30 5.29 3.14 8.00 4.91 4.96 10.89 7.73 4.16 10.53 7.06 5.35 9.79 7.20 3.74 7.70 5.38 2.93 5.48 4.16 3.14 5.32 4.04 2.82 4.26 3.38 3.41 6.87 4.88 3.18 5.25 4.15 3.06 5.53 4.09

setting are no better than the random ligand conformations used in the third setting. In the ﬁrst setting 6 test cases (1K9R, 2OEI, 2DYF, 1YWI, 2JUP and 2RLY) are successful. Besides the three simple test cases AutoDock in our approach succeeded in three more cases with larger numbers of torsional degrees of freedom. In particular 2DYF has 25 torsional degrees of freedom which is diﬃcult for ﬂexible docking. Among the