Visual Information and Information Systems: Third International Conference, VISUAL'99, Amsterdam, The Netherlands, June 2-4, 1999, Proceedings

Lecture Notes in Computer Science

Edited by G. Goos, J. Hartmanis and J. van Leeuwen

1614

3 Berlin Heidelberg New York Barcelona Hong Kong London Milan Paris Singapore Tokyo

Dionysius P. Huijsmans Arnold W.M. Smeulders (Eds.)

Visual Information and Information Systems Third International Conference, VISUAL’99 Amsterdam, The Netherlands, June 2-4, 1999 Proceedings

13

Series Editors Gerhard Goos, Karlsruhe University, Germany Juris Hartmanis, Cornell University, NY, USA Jan van Leeuwen, Utrecht University, The Netherlands

Volume Editors Dionysius P. Huijsmans Leiden University, Computer Science Department Niels Bohrweg 1, 2333 CA Leiden, The Netherlands E-mail: [email protected] Arnold W.M. Smeulders University of Amsterdam, Research Institute Computer Science Kruislaan 403, 1098 SJ Amsterdam, The Netherlands E-mail: [email protected]

Cataloging-in-Publication data applied for Die Deutsche Bibliothek - CIP-Einheitsaufnahme Visual information and information systems : third international conference ; proceedings / VISUAL ’99, Amsterdam, The Netherlands, June 2 - 4, 1999. D. P. Huijsmans ; Arnold W.M. Smeulders (ed.). - Berlin ; Heidelberg ; New York ; Barcelona ; Hong Kong ; London ; Milan ; Paris ; Singapore ; Tokyo : Springer, 1999 (Lecture notes in computer science ; Vol. 1614) ISBN 3-540-66079-8

CR Subject Classification (1998): H.3, H.5, H.2, I.4, I.5, I.7, I.3 ISSN 0302-9743 ISBN 3-540-66079-8 Springer-Verlag Berlin Heidelberg New York This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer-Verlag. Violations are liable for prosecution under the German Copyright Law. c Springer-Verlag Berlin Heidelberg 1999
Printed in Germany

Typesetting: Camera-ready by author SPIN 10705199 06/3142 – 5 4 3 2 1 0

Printed on acid-free paper

Preface Visual Information at the Turn of the Millenium Visual information dominates the senses we have been given to observe the world around us. We tend to believe information most when it is in visual form. Television and Internet have accelerated the perfusion of visual information to unprecedented heights. Now that all sensors are turning digital, and personal computers and the Net are powerful enough to process visual information, a new era is being born: the age of multimedia information. The dominant component of multimedia information is visual. Hence the conclusion, we are on the threshold of the age of visual information. The approach of the new millenium provokes these sweeping thoughts. Five hundred years after the invention of printed books, visual information has returned to the forefront of information dissemination, on equal par with textual and numerical information. The practice of designing visual information systems is far removed from such grandiose thoughts. Visual information systems are radically different from conventional information systems. Many novel issues need to be addressed. A visual information system should be capable of providing access to the content of pictures and video. Where symbolic and numerical information are identical in content and form, pictures require a delicate treatment to approach their content. To search and retrieve items on the basis of their pictorial content requires a new, visual or textual way of specifying the query, new indices to order the data, and new ways to establish similarity between the query and the target. A novel element, still lacking research, is the display of the information space of all visual items in the system. Derived from the Third International Conference on Visual Information Systems, held in Amsterdam, this issue of Springer’s Lecture Notes in Computer Science provides a state-of-the-art view on visual information systems. Among the building blocks of visual information systems, the computation of features is currently attracting the most attention. Good features are instrumental in reducing the abundance of information in the picture or in the video to the essence. Ideally speaking the feature is insensitive to irrelevant variations in the data, and sensitive to variations in semantic differences in the data. In the proceedings you will find features of various kinds, where invariance is of specific importance to features for image databases. For browsing and searching for unspecified items in the information space of all items in the system, visual interaction on the ensemble of all items can provide an overview to the surfing user. In the proceedings you will find contributions on query by iterative optimization of the target, displaying the information space, and other ways to trace semantically similar items or documents. It is expected that the topic will attract more attention, more completely fulfilling the name: visual information systems. An important issue of visual search is the similarity measure. It is not easy to decide what makes two objects, example and target, experienced as equal. Simi-

VI

Preface

larity is currently approached as either an exact correspondence (as in standard databases), as a statistical problem (as in object classification), or as a metrical problem (in feature space). It is quite likely that similarity search as a cognitive problem will gain in weight where human-perceived similarity will be core. Similarity search for all practical purposes is proximity search: the subject and the target match by proximity. In the proceedings you will find many different implementations of the notion of proximity. Underlying any information system, there should be a database proper with data structures, query specification, and indexing schemes for efficient search. Where the main emphasis of the techniques embodied here is on processing visual information, the connection to databases, and the database parlance is still underrated. In the proceedings you will find contributions on extensions of the database tradition towards unstructured multimedia items, on data structures especially suited for spatial data, and on new ways to access spatial data. An essential part of visual information processing is the success of capturing the information in the image. Where the biggest problem in computer vision is a successful segmentation step, in image databases several authors find their way around this step. In the proceedings you will find contributions based on characterizing internally similar partitions in the image, salient details, or total image profiles. Contributions on all these and many more aspects of many more topics can be absorbed from the proceedings. Their combination in one LNCS Volume gives an up-to-date overview of all components of visual information systems. All the contributions in this book have been reviewed thoroughly. The editors of this book wish to thank the members of the program committee and the additional reviewers for their effort. Their work has enhanced the final submission to this book. You will find their names on a separate sheet. We thank them cordially. With this book we hope that the conference series on visual information systems will continue on to a long-lived future. The conference chair would like to seize the opportunity to thank the members of the local committee and the conference bureau for making the conference happen. Finally, the support of the members of the visual information systems steering committee has been much appreciated.

March 1999 Arnold W.M. Smeulders Nies Huijsmans

Visual99 Conference Organization

Conference Chair Arnold W.M. Smeulders

University of Amsterdam, NL

The Visual Information Systems Steering Committee S.K. Chang Ramesh Jain Tosiyasu Kunii Clement Leung Arnold W.M. Smeulders

University of Pittsburgh, USA University of California, USA The University of Aizu, J Victoria University of Technology, AU University of Amsterdam, NL

Program Chairs Ruud M. Bolle Alberto Del Bimbo Clement Leung

IBM Watson, USA University of Florence, I Victoria University of Technology, AU

Program Committee Jan Biemond Josef Bigun S.K. Chang David Forsyth Theo Gevers Luc van Gool William Grosky Glenn Healey Nies Huijsmans Yannis Ioanidis Horace Ip Ramesj Jain Rangachar Kasturi Martin Kersten Inald Lagendijk Robert Laurini Carlo Meghini

Technical University Delft, NL Halmstad University, S Pittsburgh, USA Berkeley, USA University of Amsterdam, NL Catholic University, Leuven, B Wayne State University, USA University of California, Irvine, USA Leiden University, NL University of Athens, G City University of Hong Kong, HK University of California, San Diego, USA Penn State University, USA CWI, Amsterdam, NL Technical University Delft, NL Universite C. Bernard Lyon, F IEI CNR, Pisa, I

VIII

Conference Organization

Erich Neuhold Eric Pauwels Fernando Pereira Dragutin Petkovic Hanan Samet Simone Santini Stan Sclaroff Raimondo Schettini Stephen Smoliar Aya Soffer Michael Swain Hemant Tagare George Thoma Remco Veltkamp Jian Kang Wu

University of Darmstadt, D Catholic University, Leuven, B Instituto Superior Tcnico, Lisbon, P IBM, Almaden, USA University of Maryland, USA University of California, San Diego, USA Boston University, USA ITIM CNR, Milan, I Xerox, Palo Alto, USA Technion, Haifa, IL DEC, USA Yale University, USA National Library of Medicine, USA Utrecht University, NL National University of Singapore, SP

Additional Reviewers Giuseppe Amato Sameer Antani Frantisek Brabec Andr Everts Ullas Gargi Sennay Ghebreab Henk Heijmans Gisli R. Hjaltason Bertin Klein Thomas Klement Martin Leissler Michael Lew Ingo Macherius Giuseppe De Marco Vladimir Y. Mariano TatHieu Nguyen S.D.Olabarriaga Patrizia Palamidese Fabio Patern P. Savino Geert Streekstra V.S.Subrahmanian Ulrich Thiel Jeroen Vendrig Marcel Worring

IEI CNR, Pisa, I Penn State University, USA University of Maryland, USA University of Darmstadt, D Penn State University, USA University of Amsterdam, NL CWI, Amsterdam, NL University of Maryland, USA University of Darmstadt, D University of Darmstadt, D University of Darmstadt, D Leiden University, NL University of Darmstadt, D IEI CNR, Pisa, I Penn State University, USA University of Amsterdam, NL University of Amsterdam, NL IEI CNR, Pisa, I IEI CNR, Pisa, I IEI CNR, Pisa, I University of Amsterdam, NL University of Maryland, USA University of Darmstadt, D University of Amsterdam, NL University of Amsterdam, NL

Conference Organization

IX

Local Organizing Committee Theo Gevers Nies Huijsmans Dennis Koelma Carel van den Berg Remco Veltkamp Marcel Worring

University of Amsterdam, Leiden University, University of Amsterdam, PGS, Amsterdam, Utrecht University, University of Amsterdam,

Sponsors Shell Nederland B.V Netherlands Computers Science Research Foundation Advanced School for Computing and Imaging University of Amsterdam Royal Academy of Arts and Sciences

NL NL NL NL NL NL

Table of Contents

Visual Information Systems Supporting Image-Retrieval by Database Driven Interactive 3D InformationVisualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . M. Leissler, M. Hemmje, E.J. Neuhold

1

Video Libraries: From Ingest to Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 R.M. Bolle, A. Hampapur Querying Multimedia Data Sources and Databases . . . . . . . . . . . . . . . . . . . . . 19 S.-K. Chang, G. Costagliola, E. Jungert General Image Database Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 P.L. Stanchev System for Medical Image Retrieval: The MIMS Model . . . . . . . . . . . . . . . . . 37 R. Chbeir, Y. Amghar, A. Flory An Agent-Based Visualisation Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 J. Meddes, E. McKenzie Error-Tolerant Database for Structured Images . . . . . . . . . . . . . . . . . . . . . . . . 51 A. Ferro, G. Gallo, R. Giugno

Interactive Visual Query Query Processing and Optimization for Pictorial Query Trees . . . . . . . . . . . . 60 A. Soffer, H. Samet Similarity Search Using Multiple Examples in MARS . . . . . . . . . . . . . . . . . . . 68 K. Porkaew, S. Mehrotra, M. Ortega, K. Chakrabarti Excluding Specified Colors from Image Queries Using a Multidimensional Query Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 D. Androutsos, K.N. Plataniotis, A.N. Venetsanopoulos Generic Viewer Interaction Semantics for Dynamic Virtual Video Synthesis 83 C.A. Lindley, A.-M. Vercoustre Category Oriented Analysis for Visual Data Mining . . . . . . . . . . . . . . . . . . . . 91 H. Shiohara, Y. Iizuka, T. Maruyama, S. Isobe User Interaction in Region-Based Color Image Segmentation . . . . . . . . . . . . . 99 N. Ikonomakis, K.N. Plataniotis, A.N. Venetsanopoulos

XII

Table of Contents

Using a Relevance Feedback Mechanism to Improve Content-Based Image Retrieval . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 G. Ciocca, R. Schettini Region Queries without Segmentation for Image Retrieval by Content . . . . 115 J. Malki, N. Boujemaa, C. Nastar, A. Winter Content-Based Image Retrieval over the Web Using Query by Sketch and Relevance Feedback . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 E. Di Sciascio, G. Mingolla, M. Mongiello Visual Learning of Simple Semantics in ImageScape . . . . . . . . . . . . . . . . . . . . 131 J.M.Buijs, M.S. Lew

Browsing Information Space Task Analysis for Information Visualization . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 S.L. Hibino Filter Image Browsing: Exploiting Interaction in Image Retrieval . . . . . . . . . 147 J. Vendrig, M. Worring, A.W.M. Smeulders Visualization of Information Spaces to Retrieve and Browse Image Data . . 155 A. Hiroike, Y. Musha, A. Sugimoto, Y. Mori Mandala: An Architecture for Using Images to Access and Organize Web Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163 J.I. Helfman A Compact and Retrieval-Oriented Video Representation Using Mosaics . . 171 G. Baldi, C. Colombo, A. Del Bimbo

Internet Search Engines Crawling, Indexing and Retrieval of Three-Dimensional Data on the Web in the Framework of MPEG-7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179 E. Paquet, M. Rioux A Visual Search Engine for Distributed Image and Video Database Retrieval Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187 J.-R. Ohm, F. Bunjamin, W. Liebsch, B. Makai, K. Mueller, B. Saberdest, D. Zier Indexing Multimedia for the Internet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195 B. Eberman, B. Fidler, R. Iannucci, C. Joerg, L. Kontothanassis, D.E. Kovalcin, P. Moreno, M.J. Swain, J.-M. Van Thong Crawling for Images on the WWW . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203 J. Cho, S. Mukherjea

Table of Contents

XIII

A Dynamic JAVA-Based Intelligent Interface for Online Image Database Searches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211 V. Konstantinou, A. Psarrou

Video Parsing Motion-Based Feature Extraction and Ascendant Hierarchical Classification for Video Indexing and Retrieval . . . . . . . . . . . . . . . . . . . . . . . . 221 R. Fablet, P. Bouthemy Automatically Segmenting Movies into Logical Story Units . . . . . . . . . . . . . . 229 A. Hanjalic, R.L. Lagendijk, J. Biemond Local Color Analysis for Scene Break Detection Applied to TV Commercials Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237 J.M. S´ anchez, X. Binefa, J. Vitri` a, P. Radeva Scene Segmentation and Image Feature Extraction for Video Indexing and Retrieval . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245 P. Bouthemy, C. Garcia, R. Ronfard, G. Tziritas, E. Venau, D. Zugaj Automatic Recognition of Camera Zooms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253 S. Fischer, I. Rimac, R. Steinmetz A Region Tracking Method with Failure Detection for an Interactive Video Indexing Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261 M. Gelgon, P. Bouthemy, T. Dubois Integrated Parsing of Compressed Video . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269 S.M. Bhandarkar, Y.S. Warke, A.A. Khombhadia Improvement of Shot Detection Using Illumination Invariant Metric and Dynamic Threshold Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277 W. Kong, X. Ding, H. Lu, S. Ma Temporal Segmentation of MPEG Video Sequences . . . . . . . . . . . . . . . . . . . . 283 E. Ardizzone, C. Lodato, S. Lopes Detecting Abrupt Scene Change Using Neural Network . . . . . . . . . . . . . . . . . 291 H.B. Lu, Y.J. Zhang Multi-Modal Feature-Map: An Approach to Represent Digital Video Sequences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299 U. Srinivasan, C. Lindley Robust Tracking of Video Objects through Topological Constraint on Homogeneous Motion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 307 M. Liao, Y. Li, S. Ma, H. Lu

XIV

Table of Contents

Spatial Data The Spatial Spreadsheet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317 G.S. Iwerks, H. Samet A High Level Visual Language for Spatial Data Management . . . . . . . . . . . . 325 M.-A. Aufure-Portier, C. Bonhomme A Global Graph Model of Image Registration . . . . . . . . . . . . . . . . . . . . . . . . . . 333 S.G. Nikolov, D.R. Bull, C.N. Canagarajah A Graph-Theoretic Approach to Image Database Retrieval . . . . . . . . . . . . . . 341 S. Aksoy, R.M. Haralick Motion Capture of Arm from a Monocular Image Sequence . . . . . . . . . . . . . . 349 C. Pan, S. Ma

Visual Languages Comparing Dictionaries for the Automatic Generation of Hypertextual Links: A Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 358 I. Gagliardi, B. Zonta Categorizing Visual Contents by Matching Visual ”Keywords” . . . . . . . . . . . 367 J.-H. Lim Design of the Presentation Language for Distributed Hypermedia System . 375 M. Katsumoto, S.-i. Iisaku A Generic Annotation Model for Video Databases . . . . . . . . . . . . . . . . . . . . . . 383 H. Rehatschek, H. Mueller Design and Implementation of COIRS(a COncept-Based Image Retrieval System) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 391 H. Yang, H. Kim, J. Yang Automatic Index Expansion for Concept-Based Image Query . . . . . . . . . . . . 399 D. Sutanto, C.H.C. Leung

Features and Indexes for Image Retrieval Structured High-Level Indexing of Visual Data Content . . . . . . . . . . . . . . . . . 409 A.M. Tam, C.H.C. Leung Feature Extraction: Issues, New Features, and Symbolic Representation . . . 418 M. Palhang, A. Sowmya Detection of Interest Points for Image Indexation . . . . . . . . . . . . . . . . . . . . . . 427 S. Bres, J.-M. Jolion

Table of Contents

XV

Highly Discriminative Invariant Features for Image Matching . . . . . . . . . . . . 435 R. Alferez, Y.-F. Wang Image Retrieval Using Schwarz Representation of One-Dimensional Feature 443 X. Ding, W. Kong, C. Hu, S. Ma Invariant Image Retrieval Using Wavelet Maxima Moment . . . . . . . . . . . . . . 451 M. Do, S. Ayer, M. Vetterli Detecting Regular Structures for Invariant Retrieval . . . . . . . . . . . . . . . . . . . . 459 D. Chetverikov Color Image Texture Indexing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 467 N. Nes, M.C. d’Ornellas Improving Image Classification Using Extended Run Length Features . . . . . 475 S.M. Rahman, G.C. Karmaker, R.J. Bignall Feature Extraction Using Fractal Codes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 483 B.A.M. Schouten, P.M. de Zeeuw

Object Retrieval Content-Based Image Retrieval Based on Local Affinely Invariant Regions . 493 T. Tuytelaars, L. Van Gool A Framework for Object-Based Image Retrieval at the Semantic Level . . . . 501 L. Jia, L. Kitchen Blobworld: A System for Region-Based Image Indexing and Retrieval . . . . . 509 C. Carson, M. Thomas, S. Belongie, J.M. Hellerstein, J. Malik A Physics-Based Approach to Interactive Segmentation . . . . . . . . . . . . . . . . . 517 B.A. Maxwell

Ranking and Performance Assessment of Effectiveness of Content Based Image Retrieval Systems . . . 525 A. Dimai Adapting k-d Trees to Visual Retrieval . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 533 R. Egas, D.P. Huijsmans, M. Lew, N. Sebe Content-Based Image Retrieval Using Self-Organizing Maps . . . . . . . . . . . . . 541 J. Laaksonen, M. Koskela, E. Oja Relevance Feedback and Term Weighting Schemes for Content-Based Image Retrieval . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 549 D. Squire, W. Mueller, H. Mueller

XVI

Table of Contents

Genetic Algorithm for Weights Assignment in Dissimilarity Function for Trademark Retrieval . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 557 D. Y.-M. Chan, I. King

Shape Retrieval Retrieval of Similar Shapes under Affine Transform . . . . . . . . . . . . . . . . . . . . . 566 F. Mokhtarian, S. Abbasi Efficient Image Retrieval through Vantage Objects . . . . . . . . . . . . . . . . . . . . . 575 J. Vleugels, R. Veltkamp Using Pen-Based Outlines for Object-Based Annotation and Image-Based Queries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 585 L. Schomaker, E. de Leau, L. Vuurpijl Interactive Query Formulation for Object Search . . . . . . . . . . . . . . . . . . . . . . . 593 T. Gevers, A.W.M. Smeulders Automatic Deformable Shape Segmentation for Image Database Search Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 601 L. Liu, S. Sclaroff A Multiscale Turning Angle Representation of Object Shapes for Image Retrieval . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 609 G. Iannizzotto, L. Vita Contour-Based Shape Similarity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 617 L.J. Latecki, R. Lakaemper Computing Dissimilarity Between Hand Drawn-Sketches and Digitized Images . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 625 F. Banfi, R. Ingold

Retrieval Systems Document Generation and Picture Retrieval . . . . . . . . . . . . . . . . . . . . . . . . . . . 632 K. van Deemter FLORES: A JAVA Based Image Database for Ornamentals . . . . . . . . . . . . . 641 G. van der Heijden, G. Polder, J.W. van Eck Pictorial Portrait Indexing Using View-Based Eigen-Eyes . . . . . . . . . . . . . . . 649 C. Saraceno, M. Reiter, P. Kammerer, E. Zolda, W. Kropatsch Image Retrieval Using Fuzzy Triples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 657 S.H. Jeong, J.D. Yang, H.J. Yang, J.H. Choi

Table of Contents

XVII

Image Compression Variable-Bit-Length Coding: An Effective Coding Method . . . . . . . . . . . . . . . 665 S. Sahni, B.C. Vemuri, F. Chen, C. Kapoor Block-Constrained Fractal Coding Scheme for Image Retrieval . . . . . . . . . . . 673 Z. Wang, Z. Chi, D. Deng, Y. Yu Efficient Algorithms for Lossless Compression of 2D/3D Images . . . . . . . . . . 681 F. Chen, S. Sahni, B.C. Vemuri

Virtual Environments Lucent VisionT M : A System for Enhanced Sports Viewing . . . . . . . . . . . . . . 689 G.S. Pingali, Y. Jean, I. Carlbom Building 3D Models of Vehicles for Computer Vision . . . . . . . . . . . . . . . . . . . 697 R. Fraile, S.J. Maybank Integrating Applications into Interactive Virtual Environments . . . . . . . . . . 703 A. Biancardi, V. Moccia

Recognition Systems Structural Sensitivity for Large-Scale Line-Pattern Recognition . . . . . . . . . . 711 B. Huet, E.R. Hancock Complex Visual Activity Recognition Using a Temporally Ordered Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 719 S. Bhonsle, A. Gupta, S. Santini, M. Worring, R. Jain Image Database Assisted Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 727 S. Santini, M. Worring, E. Hunter, V. Kouznetsova, M. Goldbaum, A. Hoover Visual Processing System for Facial Prediction . . . . . . . . . . . . . . . . . . . . . . . . . 735 C. Xu, J. Wu, S. Ma Semi-interactive Structure and Fault Analysis of (111)7x7 Silicon Micrographs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 745 P. Androutsos, H.E. Ruda, A.N. Venetsanopoulos Using Wavelet Transforms to Match Photographs of Individual Sperm Whales Identified by Contour of the Trailing Edge of the Fluke . . . . . . . . . . 753 R. Huele, J.N. Ciano From Gaze to Focus of Attention . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 761 R. Stiefelhagen, M. Finke, J. Yang, A. Waibel

XVIII Table of Contents

Automatic Interpretation Based on Robust Segmentation and Shape-Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 769 G. Frederix, E.J. Pauwels A Pre-filter Enabling Fast Frontal Face Detection . . . . . . . . . . . . . . . . . . . . . . 777 S.C.Y. Chan, P.H. Lewis

Visualization Systems A Technique for Generating Graphical Abstractions of Program Data Structures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 785 C. Demetrescu, I. Finocchi Visual Presentations in Multimedia Learning: Conditions that Overload Visual Working Memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 793 R. Moreno, R.E. Mayer Visualization of Spatial Neuroanatomical Data . . . . . . . . . . . . . . . . . . . . . . . . . 801 C. Shahabi, A.E. Dashti, G. Burns, S. Ghandeharizadeh, N. Jiang, L.W. Swanson Visualization of the Cortical Potential Field by Medical Imaging Data Fusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 809 M.C. Erie, C.H. Chu, R.D. Sidman Applying Visualization Research Towards Design . . . . . . . . . . . . . . . . . . . . . . 817 P. Janecek

Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 825

Supporting Image-Retrieval by Database Driven Interactive 3D Information-Visualization Martin Leissler, Matthias Hemmje, Erich J. Neuhold GMD – German National Research Center for Information Technology IPSI – Integrated Publication and Information Systems Institute Dolivostr. 15, 64293 Darmstadt, Germany [leissler, hemmje, neuhold]@darmstadt.gmd.de Abstract. Supporting image-retrieval dialogues between naive users and information systems is a non-trivial task. Although a wide variety of experimental and prototypical image retrieval engines is available, most of them lack appropriate support for end-user oriented front ends. We have decided to illustrate the possible advantages of a tight coupling between interactive 3D information visualization systems and image retrieval systems based on database management systems by deriving requirements from a characteristic application scenario. By means of an “interactive 3D gallery” scenario, the paper provides an overview of the requirements, components, and architecture of a general database-driven 3D information visualization system on the basis of an RDBMS and VRML. The given approach supports loading time as well as runtime database access in various forms. It reflects the overall conceptual framework of our activities in this highly dynamic area of research and forms a basis for many other applications where information objects have to be visualized for interacting users or user groups.

1. Introduction Supporting image retrieval dialogues between naive users and information systems is a non-trivial task. Although supporting the basic “pattern matching” process within the image retrieval mechanism has been tackled by various research activities (e.g. [Pentland et al. 95] [Picard et al. 93] [Wang et al. 97] [Müller & Everts 97]) during the last years, supporting the user interface front end in an image retrieval dialogue in an appropriate way has been neglected to some extent. Most of the work conducted in this area (e.g. [VVB], [Chang et al 97a], [Chang et al 97b], [Chang et al 96a], [Christel et al 96]) applies user-interface paradigms implemented on the basis of 2D interface-toolkits. In contrast to these works we want to outline in this paper how image retrieval user interfaces in distributed front-end scenarios can be supported by means of interactive 3D information visualization technologies. The work presented in this paper is based on concepts, experiments, experiences and insights gained from our work aiming at supporting, e.g., full-text-retrieval and multimedia retrieval dialogues in a similar way. The paper introduces the available base technologies. Furthermore, it outlines an overall architectural system model based on a requirement

Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL'99, LNCS 1614, pp. 1-14, 1999.  Springer-Verlag Berlin Heidelberg 1999

2

Martin Leissler et al.

analysis derived from an example scenario. An overview of implementation aspects of the proposed architectural framework and an outlook on potential future works conclude the paper.

2. Base Technologies Looking at the development of non-immersive interactive information visualization applications over the past few years, the Virtual Reality Modeling Language (VRML) has clearly become the de facto standard for representing interactive 3D worlds on the web or in offline applications. Furthermore, in 1998, VRML97 (the actual specification, [VRML97]) has made its way to become an ISO standard. If we examine the historical development of VRML in detail, it has to be recognized that the first version of the language standard (VRML 1.0) was directly derived from the file format which the OpenInventor toolkit [Wernecke 94] from Silicon Graphics Inc. had defined to exchange 3D scenes between different applications. This version of VRML was completely static, i.e., no user interactivity and even no animation was supported in a VRML1.0 scene. Soon after this first version came the second version (VRML2.0) which is today mostly identical with the actual ISO standard. It incorporates complex interactive behavior, advanced animation features and custom extensibility. However, VRML 97 is still a closed file format which uses its own internal and proprietary event model to describe behavior and user interaction. All information about a scene is contained exclusively within the VRML code. In consequence, there is no “natural” way for a VRML scene to communicate and integrate with other external applications or software components. The task of identifying concepts to solve this problem has mainly been tackled through single working groups within the VRML consortium [Web3D]. One working group, for example, has defined the so called External Authoring Interface (EAI) [EAI] to handle biderectional communication between a Java applet and a VRML97 scene coexisting on a web page. If VRML is, based on its promising starting point, ever to become a seriously used interactive information visualization medium and development platform for all kinds of information system applications, a flexible and efficient integration of the language with existing technological standards such as interfaces to database management systems or application servers has to be achieved. This means built-in standard mechanisms for communication between VRML and external systems have to be derived from the existing standard. Clearly, this problem has to be tackeled from both sides, the VRML language side and the side of external standard application programming interface (API). Situated in this interactive information visualization working context, our research concentrates on VRML and database integration which, in our opinion, is the most urgent problem at hand. By working on the problem of how VRML could communicate with existing database management systems (DBMS), we can also learn a lot about supporting more complex information visualization scenarios, e.g., persistent multi-user scenarios. Until today, all applications that used VRML in

Supporting Image-Retrieval

3

connection with a DBMS had to rely on using custom-coded database access with proprietary APIs such as the Java Database Connectivity (JDBC) [JDBC] or the Microsoft Open Database Connectivity (ODBC) [ODBC]. This is highly unsatisfactory, because people work on the implementation of the same interfacing problems over and over again. A standardization of the VRML side of the DBMS integration has been partially proposed by the database working group of the Web3D consortium [Web3D]. However, we believe that, although the overall observations of the working group are correct and sound, some more work will have to be done in the detailed specification of the features of the database integration – on API side as well as on VRML side. Some examples of necessary features will be provided in later sections of this paper. To derive the neccesary extended requirements, this paper first takes a look at a complex example scenario and later describes how this scenario could possibly be conceptually solved and supported by corresponding architectural system models and implementations.

3. Example Scenario Given that, for example in an electronic commerce application, users want to browse through a visually rich 3D virtual gallery environment (as described in [Müller et al. 99]) filled with, from their subjective point of view, more or less interesting paintings of various artists. They want to have the option to buy a painting or just enjoy reviewing works of their favorite artists, for a while. A soon as they have entered the system, a so called “query interface” is presented in which they can enter a description of their interest. After choosing a few of their favorite artists and selecting some paint styles they like, it is time to submit the query containing the so called “search criteria” to the system. Now, a 3D visually interactive presentation of an art gallery is generated in an area of the screen. The selection of the works-of-art is based on the users’ search criteria. Somewhere else on the screen, an interactive 2D overview visualization which explains the overall architectural structure of the gallery building is visualized. The displayed architectural topology of the gallery building is structured hierarchically and is therefore easy to navigate by selecting areas related to the different artists, painting techniques, and styles. As the users move around in this virtual gallery environment, they orient themselves with the help of so called “landmarks” and “signposts” inside the 3D environment as well as in the 2D overview visualization of the gallery. After a short spatial navigation, a room entrance is reached. A label describes that the works of art displayed in this room match a couple of search criteria defined earlier in the form based query construction dialogue by the user (e.g., a certain artist and his self-portrait style paintings). After navigating through the entrance, a room shows up which contains only the paintings expected in this section of the gallery. As the users navigate into the room, they take their time to study the paintings hanging on the walls of the room. By clicking on one of the paintings in the 3D environment, all information about the painting stored in the database is displayed on a separate area of the screen together

4

Martin Leissler et al.

with a more detailed high-quality image of the painting and further meta-information like, e.g., pricing and sales status. While striving around in some room, it can be recognized that one of the paintings is suddenly marked with a small “not-available” banner. By some coincidence, it must have been removed from the gallery. Either it has been sold to someone else and is therefore no longer available in the 3D gallery or someone has, e.g., not paid his bill to the gallery service provider for renting the space in the gallery. The application aspects and the users’ experience described above demand a set of different interactive information visualization functions to be supported by an overall technical architecture. The concepts which can define a formal basis for the implementation of such an architectural framework and its components are described below. The complete conceptual interactive information visualization model will be described in more detail in a different paper.

4. Other Work Before we describe the detailed conceptual system model and implementational architecture of our approach, we take a short look at existing systems which support scenarios that are similar but not identical to the one described above. 4.1. Virgilio The Virgilio system as described in [Massari et al. 98] and [Constabile et al. 98] is a software system architecture which allows the user to submit a generic query on a database which typically contains multimedia content. As a result, the system constructs a 3D metaphorical visual representation of the hierarchical query result structure through which the user can browse the query result set interactively. One of the main features of Virgilio is that the querys, the visual representation, and the mapping between the query and the visualization are all stored in persistent repositorys and can therefore be easily exchanged. The mapping between visual objects and query structures is called a “metaphor” in Virgilio. This metaphor can be completly user-defined, which means that the appearance of the visual environment can be dynamically adjusted. On the basis of a set of properties of the visual objects (e.g. a room object can contain other objects), the metaphor is intelligently applied to the given query result. A prototypical implementation of the Virgilio architecture exists on the basis of a custom application using a proprietary API (OpenInventor). After choosing a different query, the system has to completely reconstruct the 3D environment. If the underlying data changes during the runtime navigation, this will have no immediate effect on the scene. Furthermore, the querys in the query repository are fixed. The user cannot query the system freely. 4.2. TVIG The TVIG system (The Virtual Internet Gallery) [Müller et al. 99] implements a scenario very similar to the 3D gallery scenario described above. Users can use a

Supporting Image-Retrieval

5

standard web browser to query a relational database for information about artworks (e.g., their favorite artist by name) and retrieve a dynamically constructed 3D gallery visualized in the same web page. While browsing through the gallery, users can click on door handles to enter the rooms containing the paintings. Rooms and paintings are queried and constructed on demand, at runtime, to keep the system load as low as possible. The mapping between the gallery building structure and the search results are user definable from within the system. The implementation of TVIG uses some standards such as HTML, VRML, Java and JDBC, but is mainly based on custom-written code. As in Virgilio, the visual environment does not react immediatly to changes in the underlying database. The visualization is reconstructed if a new search query is submitted by the user. In both systems there are no generic standard mechanisms to communicate between the database and the 3D visualization.

5. Requirements Any scenario similar to the one described above should demand for the following general requirements: First of all, we definitetly need an appropriate information distribution mechanism for the given scenario. In today’s world of globally networked computer systems, it is obvious that an application like a virtual gallery should be a completely web based application in order to be able to reach as many users as possible. The numerous versions of web browsers with all kinds of mutimedia plugins available on client machines makes the www an ideal application platform for our scenario. In addition to browser technology defining the web-based application front end, a generally VR-capable application server is required in the back end of the application solution. Next, we need a standard way of representing and rendering a real time interactive 3D environment for the virtual gallery. This technology should be able to run on as many client platforms as possible and has to support user interaction, scene object behavior and scene animation. Furthermore, the visualization environment has to support interactive navigation in a straightforward way and should seamlessly scale with the available client resources. Since the data for the information objects of interest (in this case paintings) should be available to all users of the application at any time, a peristent storage database is required. This mechanism should also be able to serve a broad bandwith of different client platforms and a large number of users, in parallel. Because the visual environment is based on the highly dynamic content of the persistent storage mechanism as well as on the highly dynamic interests of the users there has to be an integrated mechanism to parametrically construct, reconstruct and adapt the whole visual environment in a very flexible fashion.

6

Martin Leissler et al.

Any user interaction has to be mapped to operations on the persistent storage which are in turn reflected in the 3D environment. This muhas to be enabled while the application is running. Therefore, we need a bidirectional communication mechanism between the running 3D environment and the underlying persistent storage mechanism [Hemmje 99]. Changes in the storage should be reflected in the 3D environment immediatly. Therefore we need a mechanism to automatically notify the 3D scene about changes occuring in the underlying database managed by the persistent storage mechanism.

6. Architectural Model for a Database Driven 3D Visualization Environment Figure 1 displays the general architectural model of a an application scenario like the virtual gallery. It supports the requirements derived in Section 3. The architectural model consists of a browsing client, an application server, and a persistent storage system supporting the information object database displayed in figure 1. Since the communication between the components (depicted by arrows) can potentially take place over a networked connection, the single components can be arbitrarily assigned to hardware platforms, e.g., from all three components on one platform to all on different platforms. The left side of the diagram displays the users’ VR-capable browsing client. After entering the application, the VR client displays the user interface with all its components like, for example, a form-based user interface component which can be used to enter search criteria describing the users’ interest in the customized gallery experience. runtime access

VR Client

results & notification query

request 3D scene

Application Server

DB

result

VR Extension Fig. 1. The general architectural model of a data driven 3D visualization application

If a query (i.e., an information request) containing the search criteria is submitted to the application server, the server recognizes an appropriate 3D scene has to be delivered back to the client. Since the server has to produce the actual data that matches the users’ request, it has to translate user requests into a query which can be sent to the database. The database system processes the query and sends the result back to the application server. Now, the application server can use the retrieved data

Supporting Image-Retrieval

7

to construct the 3D scene with the help of some special server extension. The 3D scene is then sent to the users VR browsing client and displayed properly. As the dynamically constructed 3D scene is interactively browsed, users can interact with certain objects of the surrounding environment, which, in turn, may lead to the necessity to get additional data from the database storage mechanism. If, for example, the user enters an area in which the images for the paintings on the walls have not been retrieved from the database, they have to be retrieved, downloaded, and integrated into the scene during application runtime. Furthermore, if the user clicks on the image of a painting to retrieve information about the artist, price etc. the same holds true. In these cases, the client runtime access mechanism has to be used to query the database and retrieve the results. These have to be directly integrated into the scene, at runtime, too. Furthermore, as soon as any data manipulation action is performed on the database storage mechanism which effects the data visualized in the running 3D scene - be it from external manipulation or from the scene itself – an immediate notification of the client has to be performed to which the clients 3D environment can, in turn, react. By now, we have described the application interaction cycle completely. Note that even though the arrows in the above figure are drawn straight from the client to the database storage mechanism, the runtime communication and notification mechanisms do not necessarily have to go directly from the client system to the storage system. It is possible (and in many cases appropriate) to handle the communication through an additional middleware layer. Taking the described scenario, the derived reqirements, and the proposed general architectural system model as a basis for our implementation approach, we can now look at the technical details which are needed to implement such a system.

7. Implementation Aspects in the Architectural Model Assuming that we use a commonly available web browser as the platform for our application front end, we have furthermore decided to store the information about the artists, paintings, styles, techniques, prices, etc. in the tables of a conventional relational database management system (DBMS) as the platform for the persistent storage mechanism, i.e., the back end of our application. Next we assume that VRML97 is used to present the 3D visualization of the gallery. As stated before, VRML97 is the most advanced and widely-accepted standard available on the market. Therefore, it is most likely to be installed on a large number of client web browsers. However, as stated in the first section, VRML97 has a few limitations. Since it is a closed file format with its own event model, all interactions which could influence external systems have to be translated from and to the VRML event model. To date, there is no predefined way to communicate between a VRML scene and a database system. This limitation applies to both, VRML scenes lying on a web server (loading time database access) and VRML scenes currently running in a client browser (runtime database access).

8

Martin Leissler et al.

If a scenario as the described virtual gallery has to be implemented with reusable components, we have to define and implement standardized and general ways of database communication. 7.1. Loading Time Database Access in VRML Since the 3D gallery is dynamically constructed on the basis of results corresponding to the users search criteria, we cannot use a static VRML scene to represent the visual environment. Therefore, the scene has to be somehow constructed dynamically and more or less on the fly after the user has submitted the search criteria to the system. Because all data for the paintings is stored in a relational database, the application has to translate the users’ search criteria into some query language statements (proposedly SQL-statements) which are then sent to the DBMS to be executed. Furthermore, the results of the query have to be integrated into the process of dynamically constructing the 3D gallery scene. In a typical implementation this would work by letting the custom coded application (typically a Java applet or an ActiveX control) send an SQL statement to the DBMS via some API. Then, while iterating over the query statements’ result set, the VRML scene code is generated by some code segment, deep inside the application, and sent to the VRML browser component. Because of the broad variety of possible visualizations of the 3D content, the probability for being able to reuse the VRML scene constructor code in a different but to some extent similar application scenario is typically very low. Furthermore, even if this implementation technique may work well, what we really want is a flexible and efficient standard way to integrate database results into a dynamically-built VRML scene. This can be achieved by integrating a loading time database access capability into VRML with the help of a server side include mechanism as presented in [Risse et al. 98] and [Müller et al. 98]. This mechanism allows the scene designer to define unfinished VRML templates which contain SQL statements. At loading time of the scene the responsible server on which the VRML scene templates are stored (i.e., either a web server or a DBMS with a custom extension) fills the templates with actual result data from the SQL query results. The client browser retrieves a perfectly VRML compliant scene description which visually represents the actual data from the DBMS. Figure 2 displays a possible implementation approach of the server extension module during the dynamic information visualization process. On the upper left of Figure 2 the VRML scenegraph containing a server side include node and a template subgraph is displayed. Before the server returns the VRML scene, the SQL statement is executed. Then the template subgraph is instanced for each returned row and filled with the query results. The so constructed VRML-compliant scene is returned to the client browser. 7.2. Runtime Database Access in VRML For HTML pages which download once from a web server and are only statically viewed by the user, a server-side include mechanism may be enough database

Supporting Image-Retrieval

9

interaction. But the main difference between a static HTML page and a VRML scene is that, after download, the VRML scene starts to run, i.e., it has a runtime. The user interacts with scene elements, and predefined animation sequences may be running, triggered by scene events. Mapping this to the given scenario means that the user, for example, interactively opens a door of the gallery by clicking on the handle. If such an interaction should trigger a database access (read or write) to dynamically construct the room behind the door, we clearly need some runtime interaction component in VRML capable of sending SQL statements to a database and retrieving the results. Furthermore, it should be possible to distribute the result data into the running VRML scene. Generally, this means a mapping from VRML runtime user interaction to database interaction. In this case, the events occur in the VRML scene and are mapped onto DBMS manipulation sequences. SSI-Node

VRML extension call SELECT s1,s2,... FROM tabelle WHERE C

sqlStatement "SELECT s1,s2,... FROM tabelle INTO f1, f2,... WHERE C“

Template

f1

f2

f3

s1

s2

s3

s4

...

V11

V12

V13

V14

...

V21

V22

V23

V24

...

VN4

...

f4

... VN1

VN2

VN3

Instancing process

... V11

V12

V13

V14

V21

V22

V23

V24

VN1

VN2

VN3

VN4

Fig. 2. Loading time database access with a VRML server side include mechanism.

Again this functionality can be achieved by using proprietary Java code inside the VRML scripting nodes. In a typical VRML scene which uses such a database access mechanism, the custom Java code inside a scripting node reacts to some scene event (representing a user interaction) by accessing a DBMS via a defined API (e.g., JDBC). The results are then collected in a VRML multiple value type field and further distributed in the scene by the VRML event mechanisms. Once again, we have custom code which has to be rewritten from scratch or modified each time the application requirements change. This of course, is costly and a never ending source of bugs which are potentially hard to trace. Moreover, this solution is highly dependent on the VRML browser component which typically leads to unpredictable behavior in cross browser application scenarios.

10

Martin Leissler et al.

What would make the VRML application developer’s life a lot easyer is a standard mechanism for runtime SQL database access out of a VRML scene. The approach has to be general enough to cover all possible cases within a runtime database access scenario. This mechanism is provided by a VRML extension node (prototype) which allows to define arbitrary SQL statements and the distribution of the possible results in the scene graph, as shown in Figure 3. Web Browser VRML Scene

SQL node

(3)

SQL Statement results

(1) (2) RDBMS

Fig. 3. The VRML runtime SQL-node. First, an SQL statement is sent to the database (1). Next, the results are returned into the VRML scene graph (2). Finally, the results are distributed to the defined positions in the graph (3).

An extension node with a similar functionality has been proposed by the database working group (now enterprise working group) of the VRML consortium in their “Recommended practices for SQL database access” [Lipkin 98]. However, while the working groups proposal covers the steps (1) and (2) of Figure 3, our solution also lets the user of such an extension node directly define exactly how the result data is to be distributed across the nodes of the VRML scene graph. Thereby, the developer is spared the burden of writing a couple of custom script nodes for every different scene graph, just in order to distribute data. This leads to less, and more efficient, VRML code. Communication between the SQL node and the DBMS can be implemented through a direct database connection. Because in many application scenarios DBMS connections are expensive, it is also possible to connect all clients to some middleware component via lightweight communication protocols and let this component handle all requests through one keep-alive DBMS connection. The implementation specific details of this approach will be presented in a separate paper. 7.3. Automatic Database Event Triggering with VRML The last requirement mentioned in our analysis is derived from the fact that, although the 3D scene in some way visualizes the actual database content, it does not automatically react to changes in the underlying data. If, for example, one of the paintings in our example scenario is marked as “not-available” in the database, for

Supporting Image-Retrieval

11

some reason (e.g., because the painting has been sold), the VRML scene should react instantly by visualizing this event, e.g., by visualizing a banner on top of the painting. Therefore, we need a mechanism which enables the database to automatically notify the running VRML scene about occuring events. More general, this means a mapping from database events to VRML scene events is required. Typically, in existing applications, this has to be done by reloading and reconstructing the whole scene based on the changed database content, which is unflexible and time consuming. Another possibility is to query the database in regular intervals from within the VRML scene to detect possible changes in the underlying data. This approach could be implemented with the help of the above mentioned runtime query component. However, this unneccesarily consumes network bandwidth and runtime resources even if the database content remains unchanged. An elegant solution to this problem is an active database trigger mechanism which enables the database to contact the VRML scene if some given part of the database has changed in a predefined way. Such a technology needs to define standards how the database should invoke a notification mechanism, how the database events are translated to VRML events and sent to the running scene and, finally, how the VRML scene can handle such events in order to delegate them to the relevant parts in the scenegraph. VRML Client

DB Client

VRML Scene notification

Trigger

Trigger Node

DBMS

access

DB Client DB Client

SQL Node

Fig. 4. Architecture for automatic notification through database triggers

Many different clients of the database system (including the VRML scene itself !) can access the database and change its internal data. Database triggers can be assigned to arbitrary parts of the data (i.e., tables in an RDBMS) and fire a predefined action sequence. In this case the the trigger action launches a notification mechanism which contacts a defined node in the VRML client scene. After distributing the event in the scene a new query can be sent to retrieve the updated data from the database. Note that this last aspect of the presented scenario already supports a shared multi client architecture. As soon as we have multiple clients connected to the system which display different parts of the scene (or even different scenes) based on the data in the storage, the trigger notification mechanism could also be used in shared virtual environment applications which have to synchronize a global application state across multiple connected clients. Again, this is best done by using a middleware component to handle the communication between the database trigger mechanism and the client

12

Martin Leissler et al.

machine. The middleware component can distribute the incoming notification events from the database to the appropriate clients and, at the same, time merge the expensive database connections. The details of this approach are presented in a separate paper. 7.4. Overall Architectural Framework After describing all crucial system components, we can now define a generic overall architectural framework which matches our given requirements and is able to run database driven interactive 3D information visualization applications similar to the described gallery scenario. (Figure 5) Runtime trigger Runtime Loading time

VRML Scene

Application Server

Trigger Node notification SQL Node results

. . .

Trigger notification server

SQL statement

JDBC server

Trigger SQL statement results

VRML Scene Search request

Web server Web server

SQL statement Result rows

Trigger Node VRML scene

VR Extension

DB Client

notification

RDBMS

DB Client . . . DB Client

SQL Node

Fig. 5. Technical architecture of a database driven interactive 3D information visualizatiom system

The above figure displays the interaction of all components. Before VRML clients go into runtime interaction mode, they log into the system and request the 3D environment from the web server based on custom search criteria. The web server, as all other intermediary sever components, is combined under the concept of an application server. Note again that all components in the diagram, including the application server components, can be arbitrarily assigned to physical machines. During the loading-time process (dotted arrows), the web server querys the DBMS, fetches the result data matching the users search criteria, and, finally, returns the customized VRML templates to the client via the VR extension module. As a result of certain interactions (e.g., opening a door in the gallery), an SQL node in the running VRML scene querys the database via a middleware database driver (typically JDBC type 3) and distributes the query results to the proper positions in the scene graph (e.g., puts the pictures on the gallery walls).

Supporting Image-Retrieval

13

If the underliying data for a VRML scene is affected now, either by a VRML client (through an SQL node) or an external DBMS client, the trigger notification mechanism may be launched (normal arrows). The notification event is distributed to the clients via a middleware trigger server. This component notifies exclusively those clients which are affected by the current change in the underlying data (e.g., clients displaying the same scene), and thereby optimizes the network load. Notificatications are distributed in the running VRML scene as events which, in turn, may launch a query from an SQL node to retrieve the most recent data. Note how the trigger mechanism can be used to propagate user interactions (affecting the global database state) on one client across multiple other clients connected to the system.

8. Conclusions and Outlook In this paper, we have presented an architectural framework consisting mainly of three components (VRML server side includes, SQL runtime node and active database triggers) which is capable of running highly dynamic database driven interactive 3D information visualization applications. We have outlined how all architectural components can work together in a complex visualization scenario such as the virtual gallery. However, the presented architectural components can also be used as completely independent stand-alone components in applications with different requirements. Indeed, in most application scenarios not all of the three components need to be used. Our experience has taught us that a combination of only some of them is in most cases already sufficient.

References [Chang et al 97a] Chang, S.-F., Chen, W., Meng, H., Sundaram, H., Zhong, D. (1998). Videoq: An automated content-based video search system using visual cues. In Proceedings of ACM Multimedia 1997. [Chang et al 97b] Chang, S.-F., Smith, J., Meng, H., Wang, H., Zhong, D. (1998). Finding images/video in large archives.‘ In: D-Lib Magazine, February 1997. [Chang et al 96] Chang, Y.-L., Zeng, W., Kamel, I., Alonso, R. (1996). Integrated image and speech analysis for content-based video indexing. In: Proceedings of ACM MM 1996 [Christel et al 97] Christel, M., Winkler, D., Taylor, C. (1997). Multimedia abstraction for a digital video library. In: Proceedings of ACM Digital Libraries '97, pages 21--29, Philadelphia, PA [Costabile et al. 98] Costabile, M. F., Malerba, D., Hemmje, M., Paradiso, A. (1998) Building Metaphors for Supporting User Interaction with Multimedia Databases In: Proceedings of 4th IFIP 2.6 Working Conference on Visual DataBase Systems - VDB 4, L'Aqulia, Italy, May 27-29, p. 47-66, Chapman & Hall 1998 [DBWork] Enterprise Technology Working Group of the Web3D consortium http://www.vrml.org/WorkingGroups/dbwork/ [EAI] Information technology -- Computer graphics and image processing -- The Virtual Reality Modeling Language (VRML) -- Part 2: External authoring interface Committee Draft ISO/IEC 14772-2:xxxx http://www.web3d.org/WorkingGroups/vrml-eai/Specification/

14

Martin Leissler et al.

[Hemmje 99] Hemmje, M. (1999). Supporting Information System Dialogues with Interactive Information Visualization.. To appear in: Dissertation Thesis, Technical University of Darmstadt, 1999 [JDBC] Sun Microsystems, Inc. The JDBCTM Data Access API http://java.sun.com/products/jdbc/index.html [Lipkin 98] Lipkin, D. (1998). Recommended Practices for SQL Database Access http://www.web3d.org/Recommended/vrml-sql/ [Massari et al. 98] Massari, A., Saladini, L., Sisinni, F., Napolitano, W., Hemmje, M., Paradiso, A., Leissler, M. (1998). Virtual Reality Systems For Browsing Multimedia. In: Furth, B. (ed.): Handbook of Multimedia Computing [Müller et al. 98] Müller, U., Leissler, M., Hemmje, M. (1998). Entwurf und Implementierung eines generischen Mechanismus zur dynamischen Einbettung multimedialer Daten in VRMLSzenen auf der Basis eines objektrelationalen DBMS. GMD Research Series, No, 23/1998, GMD – Forschungszentrum Informationstechnik, St. Augustin 1998 [Müller et al. 99] Computing and Systems (ICMCS'99) Mueller A., Leissler M., Hemmje, M., Neuhold E.(1999). Towards the Virtual Internet Gallery. To appear in: Proceedings of IEEE International Conference on Multimedia [Müller & Everts 97] Müller, A.,Everts, A. (1997). Interactive image retrieval by means of abductive inference. In RIAO 97 Conference Proceedings -- Computer-Assisted Information Searching on Internet, pages 450--466, June 1997. [ODBC] Microsoft Press (1997). Microsoft ODBC 3.0 software development kit and programmer’s reference. Microsoft Press, Redmond Washington [Picard et al. 93] Picard, R. W., Kabir, T. (1993). Finding Similar Patterns in Large Image Databases. In: IEEE ICASSP, Minneapolis, Vol. V, pp. 161-164, 1993 [Pentland et al. 95] Pentland, A., Picard, R. W., Sclaroff, S. (1995). Photobook: Contentbased Manipulation of Image Databases. In: SPIE Storage and Retrieval Image and Video Databases II, San Jose, CA, 1995 [Risse et al. 98] Risse, T., Leissler, M. , Hemmje M., Aberer, K. (1998). Supporting Dynamic Information Visualization With VRML and Databases. In: CIKM '98, Workshop on New Paradigms in Information Visualization and Manipulation, Bethesda, November 1998 [VRML97] International Standard ISO/IEC 14772-1:1997 http://www.web3d.org/Specifications/VRML97/ Information technology -- Computer graphics and image processing -- The Virtual Reality Modeling Language (VRML) -- Part 1: Functional specification and UTF-encoding. [VVB] The virtual video browser http://hulk.bu.edu/projects/vvb_demo.html [Wang et al. 97] Wang, J. Z., Wiederholg, G., Firschein, O., Wie, S.X. (1997). Content-based image indexing and searching using Daubechies’ wavelets.In: International Journal on Digital Libraries, Vol.1, Number 4, December 1997, Springer Verlag, pp.311-328 [Web3D] Web3D Consortium (formerly: VRML consortium) http://www.web3d.org Home of the Web3D consortium (formerly http://www.vrml.org) [Wernecke 94] Wernecke, Josie (1994) The Inventor Mentor, Programming Object-Oriented 3D-Graphics with Open Inventor Release 2. Open Inventor Architecture Group; AddisonWesley Publishing Company, Inc. 1994

Video Libraries: From Ingest to Distribution




Ruud M. Bolle and Arun Hampapur IBM T. J. Watson Research Center Yorktown Heights, NY 10598 {bolle,arunh}@us.ibm.com

Abstract. Production, transmission and storage of video will eventually all be in digital form. Additionally, there is a need to organize video efficiently in databases so that videos are easily ingested, retrieved, viewed and distributed. We address and discuss many of the issues associated with video database management.

1

Introduction

The digital form of video will allow us to do many things – some of these things can be envisioned today, others will be discovered during the years to come. The digital form permits computationally extracting video content descriptors. Ideally, video is completely annotated through machine interpretation of the semantic content of the video. In practice, given the state-of-the-art in computer vision, such sophisticated data annotations may not be feasible. Much of the focus in multimedia library efforts has been on text and image databases [1], not on video libraries. We touch upon techniques for processing video as an image of space-time and we argue that the type of video analysis that has to be performed should be geared toward the specific video genre or category – e.g., sports versus soap operas. Processing video to derive annotations is one thing. Another thing, as we discuss first, is the infrastructural demands for putting such video management systems together.

2

Base Video Management System

Figure 1 shows the functional components of a base video management system and their relationship to each other. Ingest and annotation: Video data can be analog or digital. Ingest deals with video digitization in the case of analog video, and parsing a wide variety of digital video formats. Associating annotations with video segments is another part of the ingest process. Annotation consists of 1. Real time logging: Extracting a first level of time dependent indices from a video stream. This may include keyframe extraction and basic audio index information.


The work reported in the paper has been funded in part by NIST/ATP under Contract Number 70NANB5H1174.

Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL’99, LNCS 1614, pp. 15–18, 1999. c Springer-Verlag Berlin Heidelberg 1999


16

Ruud M. Bolle and Arun Hampapur

Meta Database

Retrieval and Browsing

Video Input Ingest and Annotation

Media Database

Media Distribution

Fig. 1. Base video management system 2. Manual annotation and cataloging: In many applications, manually associating information with different time segments of video and linking a unit of video to other media is essential. 3. Automatic offline annotation: Providing content based access to video requires content analysis and annotation. Most of these processes run offline. The data generated by the ingest and annotation process is stored in two locations, namely: Meta database: This can be a traditional relational database system like DB2. A data model for video data management [2] includes a time independent part like, title, producer, directors, length, etc. And, more interestingly, a time dependent part which uses a relational table structure. Media database: These databases handle both the storage and distribution aspects of managing the actual digital video data. They are file servers which are designed to handle streaming media like video. Finally, there is the issue of component inter-operability. Each of the functional blocks shown in Figure 1 is a complex subsystem. This gives rise to issues of inter-operation between the components using a standardized command and control protocol. A research effort that addresses this issue can be found in [3].

3

Video Annotation

Annotation is ideally achieved in a completely automatic fashion [4]. Video is a concatenation of shots. As described in [5], the analysis of video should not depend too much on the reliability of the shot detection algorithm that is used. Moreover, the analysis of video should go beyond computations on pixels just within shots, i.e., between-shot processing is important. Between-shot processing has as goal to derive high-level structure for automatic annotation of, possibly long, video segments. The scene structure of, for example, sitcoms can be rediscovered using clustering of shots [5]. In [6] the concept of motion picture grammars is introduced. The thesis is that video data can be represented by grammars (e.g., [7]). The grammars need to be stochastic [8];

Video Libraries: From Ingest to Distribution

17

stochastic, context-free grammars and hidden Markov models [9] are closely related. Hidden Markov models are used in [10] to detect commercials.

4

Video Retrieval and Distribution

Retrieving video through the formulation of a query is inherently more complicated than retrieving text documents. In addition to text, there is visual and audio information; moreover, there is temporal visual dynamics. Very much like text query formulation, a video query is a sequence of steps. Each step is an active filtering to reduce the number of relevant candidates. Each step allows interactive query formulation, and each gives a more refined query to the next step. Video query (see [5]) can be broken down as: query on the category of video (navigating), query on the text, and/or audio and visual feature descriptions (searching), query on the semantic summary of visual content (browsing) and query on the full-motion audio-visual content (viewing).

5

Specialized Video Management System

Base video systems are currently available as products. Such systems provide most of the infrastructural requirements for managing video. However, effective video management requires the ability to retrieve video based on much higherlevel semantic concepts. This demands the development of specialized video data management systems which are tailored to different domains. Each new application domain will require several additional functionalities which include specialized indexing algorithms, user interfaces, and data models: Indexing algorithms: Depending on the application domain, new indexing strategies are needed. For example, for sports, new event indexing algorithms need to be developed. Say, for basketball, algorithms for detecting events like scoring become critical. User interfaces: The browsing and viewing patterns for video will differ significantly across domains. For example, in a news video management system, the searching will be based on the content of the speech. For sports it will be based more on visual content such as different play-patterns. Thus both the query interface and the video control interface need to suit the domain. Data models: Certain domains may require that the management system be capable of managing several types of media and associations between them. This implies that data models for such systems have to be augmented beyond the simple models used in base video management systems.

6

Discussion

We have described many of the aspects of video database management systems. Video indexing is but one of the components of such systems; video ingest is another important and often neglected component. Finally, complicated!l infrastructures are needed for complete end-to-end systems.

18

Ruud M. Bolle and Arun Hampapur

References 1. A. Gupta and R. Jain, “Visual information retrieval,” Comm. ACM, vol. 40, pp. 70– 79, May 1997. 15 2. A. Coden, N. Haas, and R. Mack, “A system for representing and searching video segments defined by video content annotation methods,” tech. rep., IBM T.J. Watson Research Center, 1998. 16 3. N. Haas, PROPOSED SMPTE STANDARD for television Digital Studion Command and Control (DS-CC) Media and Metadata Location. NIST/ATP HD Studio Joint Venture, 1998. 16 4. A. Nagasaka and Y. Tanaka, “Automatic video indexing and full-motion search for object appearances,” in Proc. IFIP TC2/WG2.6 2nd Working Conf. on Visual Database Systems, pp. 113–127, Sep.-Oct. 1991. 16 5. R. M. Bolle, B.-L. Yeo, and M. M. Yeung, “Video query: Research directions,” IBM J. of R & D, vol. 42, pp. 233–252, March 1998. 16, 17 6. R. Bolle, Y. Aloimonos, and C. Fermuller, “Toward motion picture grammars,” in Proc. IEEE 3rd ACCV, pp. 283–290, Jan. 1998. 16 7. K. S. Fu, Syntactic Pattern Recognition and Applications. Englewood Cliffs, NJ: Prentice Hall, 1982. 16 8. E. Charniak, Statistical Language Learning. MIT Press, 1993. 16 9. X. D. Huang, Y. Ariki, and M. A. Jack, Hidden Markov Models for Speech Recognition. Edinburgh University Press, 1990. 17 10. Y.-P. Tan and R. Bolle, “Binary video classification,” Tech. Rep. RC-21165, IBM T.J. Watson Research Center, 1998. 17

Querying Multimedia Data Sources and Databases1 Shi-Kuo Chang1, Gennaro Costagliola2, and Erland Jungert3 1

Department of Computer Science University of Pittsburgh [email protected] 2 Dipartimento di Matematica ed Informatica Università di Salerno [email protected] 3 Swedish Defense Research Institute (FOA) [email protected]

Abstract. To support the retrieval and fusion of multimedia information from multiple sources and databases, a spatial/temporal query language called ΣQL is proposed. ΣQL is based upon the σ−operator sequence and in practice expressible in SQL-like syntax. ΣQL allows a user to specify powerful spatial/temporal queries for both multimedia data sources and multimedia databases, eliminating the need to write different queries for each. A ΣQL query can be processed in the most effective manner by first selecting the suitable transformations of multimedia data to derive the multimedia static schema, and then processing the query with respect to this multimedia static schema.

1 Introduction The retrieval and fusion of spatial/temporal multimedia information from diversified sources calls for the design of spatial/temporal query languages capable of dealing with both multiple data sources and databases in a heterogeneous information system environment. With the rapid expansion of the wired and wireless networks, a large number of soft real-time, hard real-time and non-real-time sources of information need to be processed, checked for consistency, structured and distributed to the various agencies and people involved in an application [12]. In addition to multimedia databases, it is also anticipated that numerous web sites on the World Wide Web will become rich sources of spatial/temporal multimedia information. Powerful query languages for multiple data sources and databases are needed in applications such as emergency management (fire, flood, earthquake, etc.), telemedicine, digital library, community network (crime prevention, child care, senior citizens care, etc.), military reconnaissance and scientific exploration (field computing). These applications share the common characteristics that information from multiple 1

This research was co-funded by the National Science Foundation, USA, the Swedish National Defence Institute and the Italian National Council of Research (CNR).

Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL'99, LNCS 1614, pp.19 -28, 1999.  Springer-Verlag Berlin Heidelberg 1999

20

Shi-Kuo Chang et al.

sources and databases must be integrated. A typical scenario for information fusion in emergency management may involve live report from a human observer, data collected by a heat sensor, video signal from a camera mounted on a helicopter, etc. Current systems often have preprogrammed, fixed scenarios. In order to enable the end user to effectively retrieve spatial/temporal multimedia information and to discover relevant associations among media objects, a flexible spatial/temporal multimedia query language for multiple data sources and databases should be provided. To support the retrieval and fusion of multimedia information from multiple sources and databases, a spatial/temporal query language called ΣQL is proposed. ΣQL is based upon the σ−operator sequence and in practice expressible in an SQL-like syntax. The natural extension of SQL to ΣQL allows a user to specify powerful spatial/temporal queries for both multimedia data sources and multimedia databases, eliminating the need to write different queries for each. Query language for heterogeneous multimedia databases is a new and growing research area [9, 13]. There has been substantial research on query languages for images and spatial objects [2], and a survey can be found in [5, 6]. Of these query languages, many are based upon extension of SQL [14], such as PSQL [15] and Spatial SQL [8]. Next come video query languages where the focus is shifted to temporal constraints [1] and content based retrieval [3]. While the above described approaches each address some important issues, there is a lack of unified treatment of queries that can deal with both spatial and temporal constraints from both live data sources and stored databases. The proposed approach differs from the above in the introduction of a general powerful operator called the σ−operator, so that the corresponding query language can be based upon σ−operator sequences. The paper is organized as follows. The basic concepts of the σ−query is explained in Section 2. Section 3 introduces elements of Symbolic Projection Theory and the general σ−operator, and Secction 4 describes the SQL query language. An illustration of data fusion using the σ−query is presented in Section 5. Section 6 formalizes the representation for multimedia sources and then gives a query processing example. In Section 7 we discuss further research topics.

2 Basic Concepts of the σ−Query As mentioned in Section 1, the σ−query language is a spatial/temporal query language for information retrieval from multiple sources and databases. Its strength is its simplicity: the query language is based upon a single operator - the σ−operator. Yet the concept is natural and can easily be mapped into an SQL-like query language. The σ−query language is useful in theoretical investigation, while the SQL-like query language is easy to implement and is a step towards a user-friendly visual query language. An example is illustrated in Figure 1. The source R, also called a universe, consists of time slices of 2D frames. To extract three pre-determined time slices from the source R, the query in mathematical notation is: σt (t1 , t2 , t3 ) R. The meaning of the σ−operator in the above query is SELECT, i.e. we want to select the time axis and three slices along this axis. The subscript t in σt indicates the selection of the time axis. In the SQL-like language a ΣQL query is expressed as:

Querying Multimedia Data Sources and Databases

21

SELECT t CLUSTER t1, t2, t3 FROM R

Fig. 1. Example of extracting three time slices (frames) from a video source

A new keyword "CLUSTER" is introduced, so that the parameters for the σ−operator can be listed, such as t1, t2, t3. The word "CLUSTER" indicates that objects belonging to the same cluster must share some common characteristics (such as having the same x coordinate value). A cluster may have a sub-structure specified in another (recursive) query. Clustering is a natural concept when dealing with spatial/temporal objects. The mechanism for clustering will be discussed further in Section 3. The result of a ΣQL query is a string that describes the relationships among the clusters. This string is called a cluster-string, which will also be discussed further in Section 3. A cluster is a collection of objects sharing some common characteristics. The SELECT- CLUSTER pair of keywords in ΣQL is a natural extension of the SELECT keyword in SQL. In fact, in SQL implicitly each attribute is considered as a different axis. The selection of the attributes’ axes defines the default clusters as those sharing common attribute values. As an example, the following ΣQL query is equivalent to an SQL query to select attributes’ axes "sname" and "status" from the suppliers in Paris. SELECT sname, status CLUSTER * FROM supplier WHERE city = "Paris" In the above ΣQL query, the * indicates any possible values for the dimensions sname and status. Since no clustering mechanism is indicated after the CLUSTER keyword the default clustering is assumed. Thus by adding the "CLUSTER *" clause, every SQL query can be expressed as a ΣQL query. Each cluster can be open (with objects inside visible) or closed (with objects inside not visible). The notation is t2o for an open cluster and t2c or simply no superscript for a closed cluster. In the ΣQL language the keyword "OPEN" is used: SELECT t CLUSTER t1 , OPEN t2 , t3 FROM R

22

Shi-Kuo Chang et al.

With the notation described above, it is quite easy to express a complex, recursive query. For example, to find the spatial/temporal relationship between objects having the same x coordinate values x1 or x2 from the three time slices of a source R, as illustrated in Figure 1, the ΣQL query in mathematical notation is: σx (x1 , x2)( σt (t1o, t2o, t3o ) R)

(1)

The query result is a cluster-string describing the spatial/temporal relationship between the objects 'a' and 'b'. How to express this spatial/temporal relationship depends upon the (spatial) data structure used. In the next section we explain Symbolic Projection as a means to express spatial/temporal relationships.

3 A General σ−Operator for σ−Queries As mentioned above, the ΣQL query language is based upon a single operator - the σ−operator - which utilizes Symbolic Projection to express the spatial/temporal relationships in query processing. In the following, Symbolic Projection, the cutting mechanism and the general σ−operator are explained, which together constitute the theoretical underpinnings of ΣQL. Symbolic Projection [7, 11] is a formalism where space is represented as a set of strings. Each string is a formal description of space or time, including all existing objects and their relative positions viewed along the corresponding coordinate axis of the string. This representation is qualitative because it mainly describes sequences of projected objects and their relative positions. We can use Symbolic Projection as a means for expressing the spatial/temporal relationships extracted by a spatial/temporal query. Continuing the example illustrated by Figure 1, for time slice Ct1 its x-projection using the Fundamental Symbolic Projection is: σx (x1 , x2 ) Ct1 = (u: Cx1,t1 < Cx2,t1)

(2)

and its y-projection is: σy(y1 , y2 ) Ct1 = (v: Cy1,t1 < Cy2,t1)

(3)

In the above example, a time slice is represented by a cluster Ct1 containing objects with the same time attribute value t1. A cluster-string is a string composed from cluster identifiers and relational operators. The single cluster Ct1 is considered a degenerated cluster-string. After the σy operator is applied, the resulting cluster Cy1,t1 contains objects with the same time and space attribute values. In the above example, the cluster-string (v: Cy1,t1 < Cy2,t1) has the optional parentheses and projection variable “v” to emphasize the direction of projection. The query σt(t1 , t2 , t3 ) R yields the following cluster-string α: α = (t: Ct1 < Ct2 < Ct3 )

(4)

When another operator is applied, it is applied to the clusters in a cluster-string. Thus the query σx (x1, x2) σt(t1o, t2o, t3o)R yields the following cluster-string β:

Querying Multimedia Data Sources and Databases

β = (t: (u: Cx1,t1 < Cx2,t1) < (u: Cx1,t2 < Cx2,t2) < (u: Cx1,t3 < Cx2,t3))

23

(5)

The above cluster-string β needs to be transformed so that the relationships among the objects become directly visible. This calls for the use of a materialization function MAT to map clusters to objects. Since Cx1,t1 = Cx1,t2 = Cx1,t3 = {a} and Cx2,t1 = Cx2,t2 = Cx2,t3 = {b}, the materialization MAT(β) of the above cluster-string yields: MAT(β) = (t: (u: a < b) < (u: a < b) < (u: a < b))

(6)

The query result in general depends upon the clustering that in turn depends upon the cutting mechanism. The cutting is an important part of Symbolic Projection because a cutting determines both how to project and also the relationships among the objects or partial objects in either side of the cutting line. Usually the cuttings are ordered lists that are made in accordance with the Fundamental Symbolic Projection. The cutting type, κ-type, determines which particular cutting mechanism should be applied in processing a particular σ−query. The general σ−operator is defined by the following expression where, in order to make different cutting mechanisms available, the cutting mechanism κ−type is explicitly included: σaxes, k-type σ-type (clusters)ϕ = stype :

(7)

The general σ−operator is of the type σ−type and selects an axis or multiple axes, followed by a cutting mechanism of the type κ−type on (clusters)ϕ where ϕ is a predicate that objects in the clusters must satisfy. The σ−operator operates on a clusterstring that either describes a data source (e.g. data from a specified sensor) or is the result of another σ−operator. The result of the σ−operator is another cluster-string of type stype. Since the result of the σ−operator is always a cluster-string, a materialization operator MAT is needed to transform the cluster-string into real-world objects and their relationships for presentation to the user.

4 The ΣQL Query Language ΣQL is an extension of SQL to the case of multimedia sources. In fact, it is able to query seamlessly traditional relational databases and multimedia sources and their combination. The ΣQL query language operates on the extended multimedia static structure MSS which will be described in Section 6. A template of an ΣQL query is given below: SELECT dimension_list CLUSTER [cluster_type] [OPEN] cluster_val1, .., [OPEN] cluster_valn FROM source WHERE conditions PRESENT presentation_description which can be translated as follows: "Given a source (FROM source) and a list of dimensions (SELECT dimensions), select clusters (CLUSTER) corresponding to a list of

24

Shi-Kuo Chang et al.

projection values or variables ([OPEN] cluster_val1, ..) on the dimension axes using the default or a particular clustering mechanism ([cluster_type]). The clusters must satisfy a set of conditions (WHERE conditions) on the existing projection variables and/or on cluster contents if these are open ([OPEN]). The final result is presented according to a set of presentation specifications (PRESENT presentation_description)." Each σ−query can be expressed as an ΣQL query. For example, the σ−query σs,κ(s1, s2o, s3, .., sn)φ R can be translated as follows: SELECT s CLUSTER κ s1, OPEN s2, s3, .., sn FROM R WHERE φ

5 An Exmaple of Multi-Sensor Data Fusion In this section, ΣQL will be illustrated with a query that uses heterogeneous data from two different sensors -- a laser radar and a video. An example of a laser radar image is given in Figure 2. This image shows a parking lot with a large number of cars, which look like rectangles when viewed from the top. The only moving car in the image has a north-south orientation while all others have an east-west orientation. Laser radar images are characterized by being three-dimensional and having geometric properties, that is, each image point is represented by x-, y- and z-coordinate values. The particular laser radar used here is a product by SAAB Dynamics of Sweden, which is helicopter born and generates image elements from a laser beam that is split into short pulses by a rotating mirror. The laser pulses are transmitted to the ground, in a scanning movement, and when reflected back to the platform a receiver collects the returning pulses that are stored and analyzed. The result of the analysis is a sequence of points with a resolution of about 0.3 m. The video camera is carried by the helicopter as well and the two sensors are observing the same area. This means that most cars in the parking lot can be seen from both sensors. The moving car shown in two video frames in Figure 3 is encircled. Figure 4 shows two symbolic images corresponding to the two video frames in Figure 3. Almost identical projection strings can be generated from the laser radar image. Basically the query can be formulated as follows. Suppose we are interested in finding moving objects along a flight path. This can be done by analyzing only the video frames, but that may require too much computation time and the problem cannot be solved in real time. Laser radar images can, however, be used to recognize vehicles in real time, which has been shown by Jungert et al. in [9, 10]. However, it cannot be determined from the laser radar images whether the vehicles are moving. The solution is to analyze the laser radar image to first find existing vehicles, determine their positions in a second step, and then verify whether they are moving from a small number of video frames. Finally, in the fusion process, it can be determined which of the vehicles are moving. Subquery1: Are there any moving objects in the video sequence in [t1, t2]? Q1 = σmotion(moving)σtype(vehicle) σxy,interval_cutting(*) σt(To)T mod 10 = 0 and T>t1 and T
Querying Multimedia Data Sources and Databases

25

Subquery2: Are there any vehicles in the laser radar image in [t1, t2]? Q2 = σtype (vehicle) σxyz,interval_cutting(*)σt(To) T>t1 and T
Fig. 2. A laser radar image of a parking lot with a moving car (encircled)

Fig. 3. Two video frames showing a moving white vehicle (encircled) in the parking lot

Fig. 4. Symbolic images of two video frames of moving car and its close neighbors The subquery Q1 selects the video source and opens the video frames. The selection of video frames includes some conditions to specify which frames to accept and in what time interval. In this case we have chosen every tenth video frame within the interval [t1,t2]. In the next selection the σxy-operator is applied to the video frames using the interval cutting mechanism [5, 6]. This operator generates the (u,v)-strings from which the object types are determined by the σtype-operator in the selected frames. Finally the vehicles in motion are determined by the application of the motion operator. The motion string (m) is generated from the time projection string: t: (u: a0s < a 1s a 2s < a 0e < a 1e a 2e < a 3s a 4s a 5s < a 3e a 4e a 5e , v: a0s < a 0e < a 1s a 3s < a 1e a 3e < a 4s < a 4e < a 2s a 5s < a 2e a 5e) < (u: ... , v: ...) < … (u: a1s a 2s < a 1e a 2e < a 0s < a 0e < a 3s a 4s a 5s < a 3e a 4e a 5e , v: a0s < a 1s a 3s < a 1e a 3e < a 0e < a 4s < a 4e < a 2s a 5s < a 2e a 5e) < (u: ... , v: ...) < ...

26

Shi-Kuo Chang et al.

From this string the motion string is generated by the application of a σ-operator which generates a string similar to an implicit merge_or-operation, i.e.: m: t: (u: a0s < a 0e < a ´0s < a ´0e , v: a 0s < a 0e < a ´0s < a ´0e) The subquery Q2 returns first the (u,v)-strings for the time interval [t1,t2]. An intermediate result of the subquery thus becomes: u: a1s a 2s < a 1e a 2e < a 0s < a 0e < a 3s a 4s a 5s < a 3e a 4e a 5e , v: a1s a 3s < a 0s < a 1e a 3e < a 4s < a 4e < a 2s a 5s < a 0e < a 2e a 5e After applying the σxyz-operator, existing vehicles are detected by applying the σtypeoperator. The subqueries can now be fused. A fusion operator φ, called merge_and, is designed, which performs fusion of vehicle information with respect to equality of vehicle type and position (x,y) in time t. Its input comes from multiple data sources of equal type. All object types are consequently equal in both subqueries. Fusion is thus applied such that only those objects selected from the two subqueries that can be associated to each other, will remain in the motion object string that gives only a0 in response. The complete query now looks like: φxytmerge-and(*)(σmotion(moving)σtype(vehicle) σxy,interval_cutting(*)σt(To)T mod 10 = 0 and T>t1 and T
σmedia_sources (videoo)media_sources, σtype (vehicle) σxyz,interval_cutting(*)σt(To) T>t1 and T t1 AND T < t2, SELECT motion CLUSTER moving FROM SELECT type CLUSTER vehicle FROM SELECT x,y CLUSTER interval * FROM SELECT t CLUSTER OPEN (* ALIAS T) FROM SELECT media_sources CLUSTER OPEN video FROM media_sources WHERE T mod 10 = 0 AND T>t1 AND T
6 Multimedia Source Representation and ΣQL Query Execution In the previous sections we have described a data source as a simple projection string. However, in general, in order to describe data sources we need a more complex data structure. In this section we describe an extension of the MSS model proposed in [4] for the description of multimedia data. A multimedia source description is composed of a hierarchy of entities. Each entity has the format where the name is the entity identifier, the type is the entity type and each description is a triple ((d1..dm):{e1, e2, .., en}: rel_expr) with m ≥ 1 and n ≥ 0 containing (a) a list of dimensions di according to which the entity is being clustered; (b) a set of component entity identifiers resulting

Querying Multimedia Data Sources and Databases

27

from the clustering; and (c) a relational expression where relations (depending on the dimensions) are used to relate the component entities A description is legal if n = 0 or a clustering mechanism able to derive the description from the source is available. In the case of n=0, the entity is an atom with respect to the description dimension and the relational expression reduces to a simple value. Depending on the chosen description type (and consequently on the associated clustering mechanism) a source can be seen as a temporal sequence of entities, or a spatial disposition of entities, or as a set of attribute-value pairs, etc. As an example, let us consider a video clip segment showing two trees and two walking persons. The video clip is clustered according to the time dimension in three consecutive frames as shown in Figure 7. The video clip represents Cathy (c) and Bill (b) moving east, and two trees (a). The following entities constitute the MSS representation of the video clip: [R, video, (τ: {R1, R2, R3}: (t: R1 < R2 < R3))] [ R1, frame, (τ: {}: t1),((x, y): {a, b, c}: ((u: a=b< c < a), (v: a < b < a= c))) ] [ R2, frame, (τ: {}: t2),((x, y): {a, b, c}:((u: a < b < a=c),(v: a < b < a= c)] [ R3, frame, (τ: {}: t3), ((x, y): {a, b}: ((u: a < < a=b), (v: a < b < a ))) ] [ a, plant, (object: {}: {(name: tree)})] [ b, person, (object: {}: {(name: Bill)})] [ c, person, (object: {}: {(name: Cathy)})] t2

t1 a

R1 =

c

R2 =

b a

t3

aa

c

b

b

a

c

aa

c

t

a

R3 =

b a

Fig. 5. A video clip segment of three frames R1, R2, R3

The phases needed to process a ΣQL query are as follows: 1. Lexical analysis and syntactic analysis. 2. Transformational analysis to validate semantic correctness, where the following actions are taken: a) comfirm the compatibility between the dimensions and the sources in each sub-query; b) check if the overall query is consistent -- i.e., there exist representations for the intermediate results that make the query execution feasible -- and if so, keep track of the representations (intermediate result type inference); c) for each intermediate result, select a feasible representation (probably with the help from the user); d) if the main source is not structured, then build the MSS schema, else check if there exists a query-engine that supports the querying of the structured source. 3. Query optimization and query execution: If the main source is not structured then populate the MSS structure according to the schema defined during the semantic analysis by extracting the appropriate information from the source. (The semantic validation implies that the algorithms to extract the information from the source are available); execute the query against the MSS; present the results; else send the query to the database query-engine and present the results.

28

Shi-Kuo Chang et al.

7 Discussion As explained in previous sections, ΣQL can express both spatial and temporal constraints individually using the SELECT/CLUSTER construct and nested subqueries. Its limitation seems to be that constraints simultaneously involving space and time cannot be easily expressed, unless embedded in the WHERE clause. Although such constraints may be rare in practical applications, further investigation is needed in order to deal with such complex constraints. ΣQL can be applied directly to a video sequence such as the one shown in Figure 1, by visually selecting and clustering (decomposing) the 2D (or 3D, 4D, etc.) space. Applications to distance learning, remote sensing, etc. can be explored. Another important application of ΣQL, as suggested in Section 6, is to facilitate user-system interaction in selecting feasible representations in the transformational analysis of a σ−query for visual reasoning. References 1. Ahanger, G., Benson, D. and Little, T.D., "Video Query Formulation", Proc.s of Storage and Retrieval for Images and Video Databases II, San Jose, February 1995, SPIE, pp. 280-291. 2. Chan, E. P. F. and Zhu, R., "QL/G - A query language for geometric data bases", Proc. of the 1st Int. Conf. on GIS in Urban Regional and Env. Planning, Samos, Greece, April. 3. Chang, S. F., Chen, W., Meng, H. J., Sundaram, H., and Zhong, D., "VideoQ: An automated Content Based Video Search System using Visual Cues", Proceedings of the Fifth ACM International Multimedia Conference, November 1997. 4. Chang, S. K., "Content-Based Access to Multimedia Information", Proceedings of Aizu International Student Forum-Contest on Multimedia, (N. Mirenkov and A. Vazhenin. eds.), The University of Aizu, Aizu, Japan, Jul 20-24, 1998, pp. 2-41. (Available at www.cs.pitt.edu/Óchang/365/cbam7.html) 5. Chang, S.-K. and Jungert, E., Symbolic Projection for Image Information Retrieval and Spatial Reasoning, Academic Press, 1996. 6. Chang, S.-K. and Jungert E. Pictorial data management based upon the theory of Symbolic Projection, Journal of Visual Languages and Computing, vol. 2, no 3, 1990, pp 195-215. 7. Chang, S.-K., Shi, O. Y. and Yan, C. W., Iconic indexing by 2D strings, IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), Vol. 9, No. 3, pp. 413-428, 1987. 8. Egenhofer, M., "Spatial SQL: A Query and Presentation Language", IEEE Transactions on Knowledge and Data Engineering, Vol. 5, No. 2, 1991, pp. 161-174. 9. Holden, R., "Digital's DB Integrator: a commercial multi-database management system", Proc. of 3rd Int.l Conf.e on Parallel and Distributed Information Systems, Austin, TX, USA, Sept. 28-30 1994, IEEE Comput. Soc. Press, Los Alamitos, CA, USA, pp. 267-268. 10. Jungert, E., Carlsson, C., and Leuhusen, C.,"A Qualitative Matching Technique for Handling Uncertainties in Laser Radar Images", Proceeding of the SPIE conference on Automatic Target Recognition VIII, Orlando, Florida, April 13-17, 1998, pp 62-71 11. Lee, S.-Y. and Hsu, F.-S., Spatial reasoning and similarity retrieval of images using 2D Cstring knowledge representation, Pattern Recognition, vol. 25, 1992, pp 305-318. 12. Lin, C. C., Chang, S. K. and Xiang, J. X., Transformation and Exchange of Multimedia Objects in Distributed Multimedia Systems, ACM Journal of Multimedia Systems, Springer-Verlag, Vol. 4, Issue 1, 1996, 12-29. 13. Li, J. Z., Ozsu, M. T., Szafron, D. and Oria, V., "MOQL: A multimedia Object Query Language", Proc. 3rd Int. Worksh on Multim Info Systems, Como, Italy, Sept -97, pp 19-28. 14. Oomoto E. and Tanaka, K., Video Database Systems - Recent Trends in Research and Development Activities, in Handbook of Multimedia Information Management, (Grosky, W. I., Jain, R. and Mehrotra, R., eds.), Prentice Hall, 1997, pp 405-448. 15. Roussopoulos, N., Faloutsos, C. and Sellis, T., "An Efficient Pictorial Database System for PSQL", IEEE Trans. on Software Engineering, Vol. 14, No. 5, May 1988, pp. 639-650. 16. E. Waltz and J. Llinas, "Multisensor data fusion", Artect House, Boston, 1990.

General Image Database Model Peter L. Stanchev Institute of Mathematics and Computer Science, Bulgarian Academy of Sciences Acad. G. Bonchev St. 8, 1113 Sofia, Bulgaria, [email protected]

Abstract. In this paper we propose a new General Image DataBase

(GIDB) model. The model establishes taxonomy based on the systematisation of existing approaches. The GIDB model is based on the General Image Data model [1] and General Image Retrieval model [2]. The GIDB model uses the powerful features offered by object-oriented modelling, the elegance of the relational databases, the state of art of computer vision and the current methods for knowledge representation and management to achieve effective image retrieval. The developed language for the model is a hybrid between interactive and descriptive query languages. The ideas of the model can be used in the design of image retrieval libraries for an object-oriented database. As an illustration the results of applying the GIDB model to a plant database in the Sofia Image Database Management System are presented.

1 Introduction The image databases are becoming an important element of the emerging information technologies. They have been used in an a wide variety of applications such as: geographical information systems, computer-aided design and manufacturing systems, multimedia libraries, medical image management systems, automated catalogues in museums, biology, geology, mineralogy, astronomy, botany, house furnishing design, anatomy, criminal identification, etc. As well they are becoming an essential part of most multimedia databases. The first survey for image databases appeared in the early 1980 by Tamura and Yokoya [3]. They classify the image database systems into three categories: convention databases, conventional databases with extended function for image processing, and specialised systems designed for a particular application domain. On the other hand Grosky and Mehrota [4] classify the image databases into three categories: systems using relational databases, system based on object-oriented model and systems for image interpretation. Grosky elaborated the ideas of image databases to multimedia databases [5]. There are mainly five approaches towards image database system architecture:

Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL'99, LNCS 1614, pp. 29-36, 1999.  Springer-Verlag Berlin Heidelberg 1999

30

Peter L. Stanchev

(1) Conventional database system as an image database system. The use of a conventional database system as an image database system is based mainly on relational data models and rarely on hierarchical. The images are indexed as a set of attributes. At the time of the query, instead of retrieving by asking for information straight from the images, the information is extracted from previously calculated image attributes. Languages such as Structured Query Language (SQL) and Query By Example (QBE) with modifications such as Query by Pictorial Example (QPE) are common for such systems. This type of retrieval is referred as attribute based image retrieval. A representative prototype system from this class of systems is the system GRIM_DBMS [6]. (2) Image processing/graphical systems with database functionality. In these systems topological, vector and graphical representations of the images are stored in the database. The query is usually based on a command-based language. A representative of this model is the research system SAND [7]. (3) Extended/extensible conventional database system to an image database system. The systems in this class are extensions over the relational data model to overcome the imposed limitations, by the flat tabular structure of the relational databases. The retrieval strategy is the same as in the conventional database system. One of the research systems in this direction is the system GIS [8]. (4) Adaptive image database system. The framework of such a system is a flexible query specification interface to account for the different interpretations of images. An attempt for defining such kind of systems is made in [9]. (5) Miscellaneous systems/approaches. Various other approaches are used for building image databases such as: grammar based, 2-D string based, entity-attributerelationship semantic network approach, matching algorithms, etc. In this paper a new General Image DataBase (GIDB) model is presented. It includes descriptions of: (1) an image database system; (2) generic image database architecture; (3) image definition, storage and manipulation languages.

2 The GIDB Model Description This section we start with some definitions. Definition 1. A Data Model is a type of data abstraction that is used to provide the data conceptual representation. It is a set of concepts that can be used to describe the structure of a database. By the structure of a database we mean the data types, relationships, and constraints that should hold on the data. It can also include a set of operations for database retrieval and update [10]. Definition 2. An Image Database (IDB) is a logically coherent collection of images with some inherent meaning. The images usually belong to a specific application domain. An IDB is designed, built, and populated with images for specifics purpose and represents some aspects of the real world. Definition 3. An Image Database Management System (IDBMS) is a collection of programs that enable the user to define, construct and manipulate an IDB for various applications. An image definition involves specifying the characteristics of

General Image Database Model

31

the application domain, the image indexing mechanism, the image-object recognition mechanism, and the information about the images that will be extracted and stored together with the images. An image database constructing is the process of storing the images themselves on some storage media, together with the logical image description. An image database manipulation includes functions such as querying the database for a retrieval of a specific image and updating the image database to reflect changes of the images in the real word. As well the user could create his own set of programs and bind them into an image database system. Definition 4. An Image Database System (IDBS) is constituted from IDB and IDBMS. The main differences from a conventional database system environment are: (1) the existence of tools for image databases definition, including tools for image indexing and image-object recognition and (2) existence of image processing procedures. Definition 5. The way that the users think about data is called external view level, the way that the data are recognised by the database system is called internal or physical level, and the middle layer is called conceptual level. 2.1 The Generic Architecture of an Image Database The architecture of a generic image database system is given in Fig. 1. Three phases for interactions with the system are provided: domain definition, image entering and image retrieval. In order to introduce new application areas for the system the administrator uses the domain definition phase. At the second phase the images are entered into the system. The third phase is image retrieval. In it the end-users use the system for posing queries and viewing the image features. PHASE

INPUT

PROCESS

RESULT

1. Domain definition a. logical description

Logical Image Definition Language (LIDL)

LIDL processor

Procedure for image indexing

b. physical description

Physical Image Definition Language (PIDL)

PIDL processor

Procedure for physical image storage

2. Image entering a. input the image and image information

images

Image Storage Language (ISL) processor

Logical and physical IDB

b. image updating

ISL updating tools

ISL processor

Logical and physical IDB

c. image deletion

ISL deletion tools

ISL processor

Logical and physical IDB

3. Image retrieval a. image display

Image Manipulation Language-IML

b. logical image display

IML

Query processor

Query processor & Statistical processor

Fig. 1. Generic architecture of an IDBS

Images Semantic data Statistical data

32

Peter L. Stanchev

2.2 Image Data Model Description The proposed Image Data model establishes taxonomy based on the systematisation of the existing approaches. The proposed approach for the image modelling includes: • using language approach, where language structures are used for physical and logical image content description; • using object oriented approach, where the image and the image objects are treated as objects containing appropriate functions calculating its functions. The data model is object oriented. The image itself together with its semantic descriptions is treated as an object in terms of the object oriented approach. The image is presented in two layouts (classes) - logical and physical. A semantic schema of the proposed model is shown in Fig. 2. Image R Logical view Global view Meta attributes

Content-based view

Semantic attributes Objects

Physical view R R

Modelbased view = = Relations

Image header

Image matrix

General purpose view Colour

Texture

Topological Vector Metric

Spatial

Colour Texture Shape Logical Semantic attributes attributes Legend: is-an-abstraction-of (multi-valued) is-an-abstraction-of (one-to-one) = domain dependant R - required

Fig. 2. Semantic schema of the GID model

2.3 Image Retrieval The retrieval model is unique in the sense of its comprehensive coverage of the image features. The main characteristics of the proposed model could be summarised as follows: a.) The images are searched by their general image description model representation [1];

General Image Database Model

33

b.) The model is based on similarity retrieval. Let a query be converted through the general image data model in an image description Q(q1, q2, …, qn) and an image in the image database has the description I(x1, x2, …, xn). Then the retrieval value (RV) between Q and I is defined as: RVQ(I) = Σi = 1, …,n (wi * sim(qi, xi)), where wi (i = 1,2, …, n) is the weight specifying the importance of the ith parameter in the image description and sim(qi, xi) is the similarity between the ith parameter of the query image and database image and is calculated in different way according to the qi, and xi values. They can be: symbol, numerical or linguistic values, histograms, attribute relational graphs, pictures or spatial representations characters. 2.4 IDBS Languages We try to develop the IDBS languages for the GIDB model following an analogy with the conventional database systems languages. Those languages can be divided into three classes: image definition, image storage and image retrieval languages. 2.4.1 Image Definition Language The image definition language consists of two parts: the Logical Image Definition Language (LIDL) and the Physical Image Definition Language (PIDL). The Logical Image Definition Language. One physical image has different logical interpretations. The Global Image Data Model is used to create a logical representation of a physical image. The process of the logical representation is described in Fig. 3. Functions for similarity calculations have been included in the chosen methods. STEP

FORMAT

PROCESS

1.Entering schema

(IDB name, entering media, file format)

reading from outside source into the memory

2. Editing schema

(general parameters, method1, method2, ..., methode)

manipulation, transform, spatial filter, histograms, and morphological filter

3. Global view obtaining

(meta attributes = name1: type1, name2: type2, …, namema: typema; semantic attributes = name1: type1, name2: type2, …, namesa: typesa)

procedure for meta and semantic attribute definition

4. General purpose view obtaining

(colour = method1, method2, ..., methodk1, texture = method1, method2, ..., methodk2,)

procedure for colour and texture definition

5. Segmentation & object definition

(method1, method2, ..., methods)

procedure for image segmentation and object definition

6. Relation definition schema

(method1, method2, ..., methodr)

procedure for relation definition

Fig. 3. The steps in the Logical Image Definition Language

34

Peter L. Stanchev

Physical Image Definition Language. The functions for physical image storage are given in Fig. 4. STEP

FORMAT

PROCESS

1. Physical image storage schema

(method1, method2, ..., method n1)

procedure for physical image storage

2. Logical image storage schema

(method1, method2, ..., method n2)

procedure for logical image storage

3. Indexing mechanism schema

(method1, method2, ..., method n3)

procedure for indexing creation

4. „Thumbnail“ image storage schema

(method1, method2, ..., method n4)

procedure for „thumbnail“ image storage

Fig. 4. The steps in the Physical Image Definition Language

2.4.2 Image Storage Language The ISL contains three parts: (1) image entering language, (2) image updating language and (3) image deletion language. Image updating and deletion are seldom used and are not typical for image databases. For all these operations a specific interactive environment has to be created. Image processing and measurement functions are available in this language to assist the user. 2.4.3 Image Manipulation Language The Image Manipulation Language includes retrieval by attribute value, shape, colour, texture, example image or spatial constrain. The query is translated to a GID model representation and then the GIR model is used to retrieve the desired images. The retrieval method is described in more details in [2].

3 An Example for Applying the GIDB Model The GIDB model manipulation capabilities are illustrated on drawings and pictures of plan image database realised in the Sofia Image Database Management System. 3.1 Image Definition Language The first level of interaction is the domain definition. The definition of the image application area is given in Fig. 5.

General Image Database Model

35

Fig 5. An example of the Logical Image Definition Language

3.2 Image Storage Language and Image Manipulation Language Let a plant image be entered in the image database. The GID description and the image itself are stored in the image database. The Image Manipulation language is described in [2]. An example for a query result is given in Fig. 6.

Fig 6. An example for a query result

36

Peter L. Stanchev

4 Conclusions The main advantages of the proposed model could be summarised as follows: (1) Its generality. The image representation is done trough the general image data model. The image retrieval is based on the general image retrieval model. Therefore the model is applicable to a wide variety of image collections. (2) Its practical applicability. There are numerous methods for the decomposition of the image into objects and for image indexing. According to the application domain the appropriate method could be used. (3) Its flexibility. The model could be customised when used with a specific application. The presented GIDBS model could be extended for distributed IDBS and multimedia database containing text, video and speech signals. At present software realisation of the model for Windows NT is considered in the Sofia Image Database Management System.

Acknowledgement This work is partially supported by a project VRP I 1/99 of the National Foundation for Science Research of Bulgaria and European Commission INCO Copernicus project INTELLECT (PL961099).

References 1.

Stanchev, P.: General Image Data Model. In: 22 intentional conference: Information Technologies and Programming, Sofia (1997) 130-140 2. Stanchev P.: General Image Retrieval Model, In: Proceeding of the Conference of the Union of Bulgarian Math, Pleven (1998) 63-71 3. Tamura, H. Yokoya N.: Image Database Systems: a Survey. Pattern Recognition, Vol. 17, No. 1 (1984) 29-43 4. Grosky W., Mehrotra R.: Image Database Management. IEEE Computer, 22 (1989) 7-8 5. Grosky W.: Multimedia Information Systems. IEEE MultiMedia, Spring (1994) 12-23 6. Stanchev P., Rabitti F.: GRIM_DBMS: a GRaphical IMage DataBase Management System. In: Kunii, T.: Visual Database Systems, North-Holland (1989) 415-430 7. Aref, W., Samet H.: Extending a DBMS with Spatial Operations. In: Second Symposium on Large Spatial Databases, Zurich, Switzerland, ETH Zurich (1991) 299-318 8. Stanchev P., Smeulders A., Groen F.: Retrieval from a Geographical Information System. In: Computing Science in the Netherlands, Amsterdam, The Netherlands, Stichting Mathematish Centrum (1991) 528-539 9. Gudivada V., Raghavan V.: Picture Retrieval Systems: A Unified Perspective and Research Issues. Tech Report, CS-95-03 (1995) 10. Elmasri R., Navathe S.: Fundamentals of Database Systems. The Benjamin/Cummings Publishing Company (1994)

System for Medical Image Retrieval The MIMS Model Richard Chbeir, Youssef Amghar, and Andre Flory LISI - INSA 20 Avenue A. Einstein, F-69621 Villeurbanne - France Tel: (+33) 4 72438595, Fax: (+33) 4 72438597 {rchbeir,amghar,flory}@lisiflory.insa-lyon.fr

Abstract. The multifaceted description of image data presents a couple of problems for traditional information systems designed for textual data. Image databases require efficient and direct spatial search based on image objects and their relationships instead of some cumbersome alphanumeric encodings of the images. Current systems generally approach the problem from different views. Thus, they describe one image facet dependently on application domain. This paper presents a model for Medical Image Management System (MIMS) that allows physicians to retrieve images and required information by combining the strengths of several approaches. Via a convivial iconic interface, our system assign to each image object a graphical representation to describe image objects and their attributes. It allows also to automatically calculate their interrelations. Key-Words : Database management, image data, image indexing and interrogation, multimedia, hypermedia, thesaurus, spatial relation.

1

Introduction

The use of multimedia information in the areas of defense and civilian satellites, spatial data, CAD data, home entertainment and medical systems has witnessed its utility. Manipulating these medias such as text, graphics, image and voice requires an adequate and special management in terms of storage, transmission and manipulation. Traditional systems have been conceived to deal with textual documents; they are not adapted for handling complex objects such as ones of multimedia applications. Current efforts have been focused on retrieving medias using only traditional artificial keys, such as identification number [6]. This obviously limits the consulting power for any domain research. Several approaches have been proposed to solve the induced problem but each one of them tries to manipulate one or more media aspects such as graphical ones ([4]) or semantic one ([2,1,7]). This paper presents a model for medical information system, using medical image as media, whose main objective has been to conceive a general approach and optimize the retrieval process.

Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL’99, LNCS 1614, pp. 37–42, 1999. c Springer-Verlag Berlin Heidelberg 1999


38

Richard Chbeir et al.

In the following section, the description of our data model follows a more detailed description for image representation problem and the medical information problem. Section 3 concludes our actual work and drafts our future one.

2

Medical Image Description

Image description is so person-depended. It differs from person to another and is related to context placed in. Several parameters are taken into consideration: image objects, image context, person culture, application domain, etc. It partially reflects the difficulties encountered in each of indexing and/or retrieving process. For our concern, medical domain is particular and requires various parameters. An example of medical searches would make easier our clarification in this section: ”Select all images for patients older than 50 years, having tumor on lungs”. It’s obvious that one image facet description is so restrictive and unable to satisfy physicians’ needs. In fact, search criterias in medical information system lay not only on image context (patients older than 50 years), but also on semantic content (tumor on lungs). So, it’s primordial to mix approaches to have hybrid system in order to satisfy medical desires. Figure 1 describes the general conceptual schema explained in next sessions. 2.1

Image Context

The medical history of each patient is represented by an Electronic Medical Folder (EMF) [5]. To verify his diagnostic, the physician needs to retrieve all images of a specific patient. ”Select all images of Mr.Robin” could be an obvious example. Thus, each medical image is related to a specific domain (radiology, dermatology, etc.). In addition, we take into consideration several image attributes such as: incidence (axial, coronal, saggital, etc.), type (JPEG, GIF, etc.), device (scanner, MRI, radio, etc.), date. 2.2

Image Description

To specify the medical content of the image, three kinds of semantic objects must be described: – Anatomic Organ (AO 1 ): corresponds to the medical organ that the image presents such as brain, hand, etc. – Pathologic Sign (PS) : identified and detected by physician, it represents a medical anomaly. For instance tumor, fracture, etc. – Anatomic Region (AR): presents the internal structure of AO. For example, the left ventricle, the right lobe, etc. Each medical image corresponds to an AO in which a set of anomalies are positioned on anatomic regions. Each anatomic region and anomaly are represented by a set of descriptors in order to outline their states. A descriptor may be a text, numeric or date. A dangerous tumor, a pumped up lung and a dehydrated lobe could be some examples of what it’s called medical states. 1

These abbreviations will be used later.

System for Medical Image Retrieval: The MIMS Model

39

Fig. 1. The general conceptual schema 2.3 Incertitude Description Incertitude is inherent to image because of description process, which might be automatic or manual. It can be situated at all description levels (AO, AR and PS). Because semantic aspect is so important in medical information system, it’s important to take into consideration such possibility. It’s the case, for example, when the physician is not sure of his diagnostic and hesitates between two anomalies. Our approach provides the possibility of describing each noncertain anomaly by one or several PSs associated to an incertitude degree for each. The incertitude degree is a decimal number majored by the interval [0,1]. Whenever the physician is sure of anomaly, the incertitude degree is evaluated as 1. 2.4

The Thesaurus

This component supports all domain-specific information that must be maintained for each specific application. It represents a dictionary in which terms

40

Richard Chbeir et al.

are interrelated. As we mentioned before, the medical image represents a set of Pathologic Signs (PS) in Anatomic Regions (AR) of the concerned Anatomic Organ (AO). We propose a hierarchical thesaurus 2 built upon these concepts (PS, AR and AO) to normalize indexing process and make more coherent, easier and efficiency both indexing and querying processes. The use of the thesaurus offers the opportunity to combine the two traditional methods: the keywords and the legend methods. 1. The thesaurus relations : The thesaurus terms, in MIMS, are interrelated by a couple of semantic relations: – the synonymy relation: relates a non-descriptor term 3 to a descriptor one. The medical vocabulary is so complex that some equivalent concepts may be different from domain to another. For instance, a tumor is a synonym to a cancer. – The hierarchy relation: defines partial order relation between descriptors in order to classify medical terms and extend queries. It groups generalization / specialization relation (anomaly and tumor) and/or composition relation (head and brain). – The compatibility relation: connects AR to PS. This relation is used essentially to sort PS with regard to compatible AR. For instance, at the left ventricle, only tumor could be found. 2. Thesaurus graphical representation: As it’s mentioned in the first section, it’s so important, when dealing with non-computer specialist, to use convivial interfaces. A graphical iconic interface is proposed in this framework to index and query the medical image database, and to manipulate thesaurus components (Figure 2). – Each AO is attached to a pattern called Organ that physician has to choose in the beginning to present the analyzed image. – The internal structure of the selected organ presenting anatomic regions is presented by a set of polygons. – Each anomaly (or PS) is attached to an icon that could be placed by physician to indicate the anomaly location. 2.5

Spatial Relations

To precise the position of anomaly in anatomic region, the notion of spatial relations is utilized. Euclidean space is used to calculate different spatial relations in both directional and topologic space. We exclude metric space for two reasons: it has to be done manually, and risks to be inappropriate in certain medical domains where measuring variability becomes important [3]. Only validated relations are stored in database and not icon coordinates. This means that scale modification can be done easily because it affects only coordinates and not relations. Two kinds of spatial relations are identified: spatial relations between AR and its PS (AR/PS), and spatial relations between PS of one AR (PS/PS): 2 3

The thesaurus is a knowledge base used to help physicians to formulate their queries. A term semantically near to another one.

System for Medical Image Retrieval: The MIMS Model

41

Fig. 2. The MIMS Indexing Interface

1. AR/PS relations : Concerning the topologic space, only inclusion relation (”IN”) is considered. The creation of such relation is implicitly attached to the creation of AR/PS descriptor whenever the user puts an icon inside the AR. On the other hand, it’s also significant to precise the anomaly direction inside the damaged region in order to locate it. Via barycenters comparison of both damaged region and anomaly, we can locate the direction of the last one. Several eventualities are then possible: left, right, low-right, high, left-high, etc. For that reason, only four relations have been taken into account: ”high, low, left, right” but the possibility that each anomaly may have two directional relations with its damaged region is always maintained. 2. PS/PS relations : Four topologic relations are taken into consideration in our approach: touch, cover, disjoint, mix. To calculate these topologic relations, the rectangular form of each icon associated to PS is utilized. Concerning the directional relations between anomalies, there are only two: High and Left. In fact, equivalence between directional relations allow us to neglect the implicit ones, i.e. if X is higher than Y, then Y is lower than X and vice versa. The main element to achieve this task is always the barycenter.

42

3

Richard Chbeir et al.

Conclusion

This paper describes the current status of our project on medical image management system. A new data model has been introduced, able to use image features and represents all medical image objects aspects. Another contribution exists, it’s the independence aspect between model components that allows us to use our approach in several domains only by changing the knowledge module. Our actual occupation is the amelioration of our model that integrates the knowledge base through an hypermedia thesaurus, a very convivial interface where the user is able to formulate his query in a very flexible manner and access image databases via the Web. The image content evolution seems to be an attractive major. In fact, the patient images describe the medical curriculum evolution . To carry out his diagnostic, the physician needs sometimes to know the chronological sequence of images. Maturity development, disease progression, medical therapeutic, traumatic event and other examples need to integrate the image content evolution in database. Our future work will be devoted to this light in order to provide the possibility in consulting database using medical image content evolution. Also, we aim to construct an online system able to guide the physician during his insertion and interrogation processes. In fact, the construction of another intelligent database based on images description will be explored. The implementation of our approach in several other domains looks like obvious. To prove our believes, we envisage our approach integration in pictorial databases in order to demonstrate the independence aspect of such approach.

References 1. Abad-Mota S., Kulikowski C., Semantic Queries on Image Databases : The IMTKAS model, Proc. Of the Basque international Workshop on Information, BIWIT 95, IEEE, Computer Society Press, P. 20-28. 37 2. Ashley J., Flickner M., Hafner J., Lee D., Niblack W., Petkovic D, The Query By Image Content System. Proceedings Of the 1995 ACM SIGMOD International Conference on management of data, SIGMOD 95, P. 475. 37 3. Clementini E., Di Felice P., Van Oostrom P., A small set of formal topological relationships suitable for end-user interaction, Lecture notes in computer science, 692, Advances in Spatial Databases. Proceedings Of the 3rd international symposium, SSD93, Singapore, Juin 1993, P. 277-295. 40 4. Jeffrey R. Bach, Santanu P., Ramesh J., A Visual Information Management System for the Interactive Retrieval of Faces, IEEE transactions on Knowledge and data engineering, Vol. 5, N 4, August 1993. 37 5. Laforest L., Frnot S., Flory A., A new approach for Hypermedia Medical Records Management, 13th International Congress Medical Informatics Europe (MIE’96), Copenhagen, 1996. 38 6. Mascarini Ch., Ratib O., Trayser G., Ligier Y., Appel R.D. In-house access to PACS images and related data through World Wide Web, European Journal of Radiology, Vol. 22, 1996, P. 218-220. 37 7. Mourad Mechkour EMIR2. An Extended Model for Image Representation and Retrieval., in DEXA’95. Database and EXpert system Applications, London, September, pp395-404, 1995. ISBN 3-540-60303-4. 37

An Agent-Based Visualisation Architecture Jonathan Meddes and Eric McKenzie School of Computer Science, Division of Informatics The University of Edinburgh, James Clerk Maxwell Building The King’s Buildings, Mayfield Road, Edinburgh EH9 3JZ Tel: +44 131 650 5129 Fax: +44 131 667 7209 {jmx,ram}@dcs.ed.ac.uk

Abstract. Advances in storage technology have led to a dramatic increase in the volume of data stored on computer systems. Using a generic visualisation framework, we have developed a data classification scheme that creates an object-orientated data model. This data model is used in an agent-based visualisation architecture to address the problems of data management and visual representations. As part of an ongoing research project we have implemented DIME (Distributed Information in a Multi-agent Environment), a visualisation system based on this approach.


1

Introduction

Computer science is catching up with visualisation, and there is a substantial motivation for computer science to make the effort. We strive to make data more interpretable via a visual representation but visualisation did not start with computers. Bertin’s seminal work with graphical representations [1] is still a widely acknowledged guide to good visualisation practice and predates the widespread introduction of bitmapped displays. The increased storage capacity of computer systems has allowed application developers to retain more detailed data. Managers have realised that they can use this data to have more control of their organisation. From a managers perspective, this has the potential to be an information revolution, and can satisfy their need for enhanced control. Unfortunately, this revolution cannot take place until the information is readily accessible. An effective visualisation uses computer graphics that allows the user to understand data. This paper presents an architecture for a novel approach to visualisation and provides an extensible visualisation system with which to develop our ideas. We have based our research around a generic visualisation framework in which we describe a data classification scheme used to create a data model that accurately classifies the data. The organisation of the data model directly influences the internal operation of our visualisation architecture which is developed using a technique allowing distributed communication orientated visualisation. Finally, we describe the implementation of the visualisation system.


This research is supported by BT Laboratories.

Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL’99, LNCS 1614, pp. 43–50, 1999. c Springer-Verlag Berlin Heidelberg 1999


44

2

Jonathan Meddes and Eric McKenzie

Data Classification

Data classification is the process of identifying data items, structures and relationships within a data series. The result is a data model that can be used by the remaining stages of the visualisation pipeline. Automated systems including SAGE [7] and ANDD [6] emphasised that automated visualisation can only be successful if the system understands the data. The accurate classification of data and relationships using a suitably expressive scheme is crucial to a visualisation system capable of creating effective visualisations. Anomalies introduced at this stage are propagated down the visualisation pipeline; this has the potential to introduce spurious visual cues into a visual representation. Before we describe the main features of our data classification scheme, we shall identify the desirable properties it should exhibit. The data classification scheme attempts to create a model that captures the syntax and semantics of the data. The syntax of data is its raw features that define its structural properties, whereas the semantics of data is the implied meaning projected by the organisation of individual data items. The data classification scheme has two goals: (1) consistently encode the data using a set of well-defined data types so that subsequent stages of the pipeline receive the data model in a familiar format; and (2) capture relationships between data items and allow easy extensions to the model by introducing additional data items and relationships. For illustration purposes, we present a subset of our full data classification scheme that is specifically intended for the classification of geographic and topological based data. An application that generates this type of data is the analysis of wide-area-network traffic; cities have a geographic location, network links have a topological structure and traffic can be measured using a quantitative value. The data model created by our scheme uses an object-orientated paradigm; this is a convenient method of organising the data that simplifies extending the data classes. Every data item is represented in our data model by a data object that is an instantiation of a data class. The basic data classes that are available to the data model are nominal, quantitative, spatial and relational. In a nominal data class, each member is a data object that represents a textual item, e.g., name. A quantitative class contains members representing numerical values, e.g., age. Constrained subclasses provide support for more specialist values such as currency and percentage. The spatial data class has members that represent a location in space. In its basic form the spatial data class is abstract and cannot be instantiated, but the creation of subclasses provides support for two-dimensional, three-dimensional and longitudinal/latitudinal co-ordinates. The relational data class provides the data model with a versatile method of representing relationships. Instantiation of this class creates a data object containing references to its associated objects; the data class also contains references to all the classes of objects which are represented in its relations. Subclasses such as the Complex Data Class (CDC) provide increased specialisation by restricting the types of relations which can be made. The CDC defines a signature consisting of a collection of data classes. Each instantiation creates a Complex

An Agent-Based Visualisation Architecture

45

CDC Domain

Classes Objects

Data item

CDO CDO Tuple CDO

CDO

(a)

(b)

Fig. 1. (a) relational table and (b) its data model Data Object (CDO) and must have exactly the same combination of data classes defining their relationships. Using these basic data classes, a data model that accurately represents the input data is constructed using the data classification scheme. Figure 1 shows an external relational database table (a) translated into the classes and objects of the internal data model (b). In the table, every cell contains a data item, every item from a column is from a similar domain and the data items from a row constitute a tuple. In the data model, a data class captures the domain which is instantiated to describe its data items. A CDC is defined to represent the association between columns in the table. Instantiations create a CDO that captures the association between the individual data items of a tuple. The data contained in the data model is merely a collection of data values and relations. In the absence of further detail describing the contents of the data model, it is meaningless as a self-contained information source. Data is transferred into information by the introduction of context. Typically, context is introduced by the user when they extract data from a model by introducing environmental influences from their domain knowledge. To move the data model from a simple collection of data values and relationships we introduce domain functions to provide context and replicate the environmental influences that dictate how a user changes data to information. In the data model, a domain function provides a basic service that can manipulate the data by mirroring the simple processing and reasoning a user performs when they interpret some data. For example, if we consider the use and capacity of a network link, a typical user of that data can deduce that the average load of that link is represented by use/capacity; a domain function would introduce new load objects to the data model.

3

Visualisation Architecture

The visualisation framework allows us to develop visualisation ideas in an environment where we are not constrained by a specific visualisation architecture.

46

Jonathan Meddes and Eric McKenzie

If we want to create tangible visual representations, we require a visualisation architecture to realise the visualisation system. In the remainder of this paper, we describe a visualisation architecture that demonstrates a novel approach to implementing the visualisation pipeline. A visualisation architecture must address (1) how data is stored or represented within the system, (2) how visual representations of the data are created, and (3) how the operational requirements of (1) and (2) are co-ordinated. The visualisation architecture we have adopted is an agent based-system inspired by behaviours observed in biological systems. 3.1

Principles of an Agent Based Architecture

The simplest way of illustrating the underlying principles behind an agent-based system is to use a biological example. If we consider a room with no windows or doors, the room is essentially a box. Although the room is sparse, it does contain some objects, a telephone, a chair, a blackboard and a stick of chalk. The room is populated by five people; each with a personality that characterises their behaviour. For example, one could posses leadership qualities that leads them to command others, or have organisational skills that makes them request actions from others, or might pace around the room. The person exhibiting the latter behaviour is physically constrained by the boundaries of the room. If they do not rely on the walls to restrict their movement, a cognitive process must be present that not only makes them move but also controls their movement. The inhabitants of the room can communicate using a commonly understood language. The person making the utterance could decide to communicate with another individual, a group of people (e.g., all males or some other commonly accepted group), or broadcast their utterance to everyone. This communication could transfer information, issue instructions (or orders) or ask and respond to questions. It would be fair to assume the people in the room would observe Grice’s maxims of communication [4]. The adoption of these rules means that the people in the room communicate effectively and will not attempt to mislead one another, maliciously or otherwise. A person’s environment is defined by the room and the objects within it. The room is a static boundary that restricts the movement of people and objects. The chair is a passive object that may be used by the people in the room but offers the bear minimum of interaction. Similarly, the blackboard provides a service to the people in the room, but could also be used for storage or as a device to communicate with other people. The telephone is an interesting object; it provides a service allowing people to communicate outside their immediate environment and can affect the people within the environment when it receives an incoming call. In this case the telephone is no longer a passive object that provides a service, it can cause the people in the room to change their behaviour. A simple example will provide an illustration of the types of communication that could take place in the room. It demonstrates the principle of co-operation that is essential for social groups to achieve a common goal. In this example, we want the names of the people in the room to be written on the blackboard. The

An Agent-Based Visualisation Architecture

47

task is initiated by a telephone call that is answered by a person who knows how to use a receiver; the caller issues an unambiguous instruction to “write every persons name on the blackboard”. The person who answered the call is now aware of the task that must be achieved and they take a controlling position in the environment. Luckily, the person who answers the call also has the ability to write on the blackboard and broadcasts a message to all the other people in the room asking them to reply with their name. One by one, the other people in the room reply to the question and as the answers are received they are written on the blackboard. Finally, the task is completed by adding their own name to the bottom of the list. This example demonstrates that some tasks can only be achieved by interaction and co-operation. This distributed computation, co-operation and communication which is evident in the biological example above demonstrates exactly the principles which underpin an agent-based system. In such a system, every person and object is represented as an agent. Agents in the environment have a similar existence to the people in the room; they are autonomous entities but are capable of communication. Through communication, agents can arrange co-operation to achieve common goals. Agents can co-exist in an environment that has a similar constraining effect to the room. The environment also provides a transport mechanism for inter-agent communication; this is analogous to the longitudinal sound waves that are transmitted through the air of the room. We have used this distributed model of computation to create DIME (Distributed Information in a Multi-agent Environment), a visualisation system that moves away from a heavyweight constraint based optimisation algorithm towards a lightweight distributed system that empowers individual data items. The agent environment is populated directly from the data model created by the data classification scheme. The object-orientated nature of the data model easily translates into the agent environment by allowing every data object and class to be represented as an agent. Using this approach provides a convenient method of organising the data within the visualisation system; it also allows the responsibility for creating a visual representation of the data to be devolved to the individual data items and their associates.

vote(property)

P1

P2

Pn action()

Property communication

Agent communication interface

Property communication Information

Instruction

Enivornment communication system

Question Answer

Fig. 2. Architecture of an individual DIME agent with properties (P1 to Pn ) and the communication interface with the agent environment.

48

3.2

Jonathan Meddes and Eric McKenzie

DIME Agent Environment

Using an agent-based philosophy, DIME supports agents operating within an agent environment. We first describe the construction of an agent before describing the operation of multiple agents within the environment; the discussion of our system is presented with a minimum of implementation detail. DIME agent: Figure 2 shows the structure of an agent which is characterised by the properties it holds (P1 to Pn ). Typically an agent is assigned numerous properties and the interaction between the properties dictates its overall behaviour. Properties can be broadly cast into two types, (1) a value property stores details about the agent, and (2) a behaviour property directly describes how the agent behaves within the environment. The internal operation of an agent is controlled by its property communication interface. This is responsible for controlling the individual properties of an agent, organising the updates of each property via a voting mechanism, inter-property communication, and providing access to the agent communication interface for inter-agent communication. When an agent is introduced into the environment, it is activated to perform its tasks. Once activated, an agent uses a roundrobin algorithm allowing each property to optionally perform some action. The combined effect of these actions allows agents to complete their goals. The action of each property is defined within its action method which is invoked each time it is the property’s turn to do some work. This method allows the property to make changes to the character (or state) of the agent by updating or adding properties. Properties in control of the agent are co-operative and operate in a fair and reasonable manner; this is particularly important when a property needs to update the values of other properties in the agent. As a safeguard to an agent being dominated by a minority of properties, we use a voting mechanism in an attempt to reach a consensus on the evolution of the agent. The voting mechanism allows all other properties to cast a vote on their support for the agent to adopt the new property value. The support a property can express for new values ranges from definitely reject to definitely accept, with a neutral position being taken by a property with no interest in the vote. If the outcome of the vote is marginal, a random function makes the casting vote. The operation of an agent is clearly illustrated by a simple example; we limit our consideration to a single agent defined to randomly move around a limited area and only consider its directly relevant properties. This behaviour is expressed using three properties, (1) the position property stores the agents current position in its environment; (2) the boundary property represents the limits of the agents movement; and (3) the random movement property defines the agents behaviour in its environment. Only the random movement property has an action method, the other two characterise the agent but have a passive role in its behaviour. When the random movement property wishes to move the agent, it uses the inter-property communication interface to request the current position of the agent; this is the value of the position property. It can then make a change to the position to represent a random movement in the environment. Before this change can be adopted, a vote must take place involving the position

An Agent-Based Visualisation Architecture

49

and the boundary properties. The position property has no preference in the outcome of the vote and returns a neutral verdict. The boundary property must investigate the proposed change in detail. If the proposed position is outside the boundary, it rejects the change; otherwise, it votes to accept the change and the property communication interface will permit the new value to be adopted. DIME environment: Agents do not directly have access to other agents internal properties and all contact must be channelled through the agent communication interface. This interface provides the necessary functionality to direct messages to individual agents, groups of agents (e.g., all the agents in a class or associated by a relationship agent) or all agents within the environment. Using this communication interface, agents can pass properties to one another using the following styles of communication, (1) instruction messages pass instructions to an agent which must be followed; (2) information messages inform other agents of a property value; and (3) question messages request information from an agent which are responded to using an answer message. Messages are routed by the environment communication system and stored until required by the agents. The agent environment supports the introduction of domain knowledge via domain functions. These specialist agents have two roles, (1) to search for combinations of data agents that can be used to create new agents; and (2) to co-ordinate groups of agents. In the latter role, domain function agents act as a proxy for other agents in the environment. In this guise, the agents for whom it is acting as a proxy are assigned a proxy property; this refers all action and voting to the proxy agent. This mechanism is available to an agent that co-ordinates groups of agents when a substantial influence on their conduct is required. The structure of the data model provides a natural organisation of the data within the visualisation system. Agents have a more complete understanding of the data than any other entity and can create visual representations of themselves using a special agent providing an agent display window. Agents that wish to be represented must posses the visible property; this is responsible for communicating visual information about the agent to the agent display window and encapsulates all the visual information about how to render the agent. This information is received from other properties and can be influenced directly or indirectly by other agents in the environment. Specialist properties can provide the agent with additional knowledge of a suitable visual representation for the data it represents. For example, a Gestalt property gives an agent knowledge about Gestalt principles [5] of appropriate organisation. Another property could provide the agent with knowledge of colour perception [2]. Using such properties, the agents have an improved knowledge of a suitable representation for the data. Agents can collaborate to represent themselves as one visual item in the agent display window. The data model provides several inherent organisational features that are retained within the agent environment. The class agents and their associated data members provide two extreme levels of abstraction. At the most detailed level, agents can provide a visual representation in the agent display window or alternatively the data agents can provide a high level abstraction of the data. A CDO provides an orthogonal dimension of abstraction where

50

Jonathan Meddes and Eric McKenzie

related data items provide an associated concept. In a data model created by the data classification scheme the highest level of abstraction is a complex data class. A CDC agent is an abstract view derived from a table representing all the class agents, data agents and CDO’s. The autonomous but collaborative nature of agents allows them to negotiate suitable visual representations for the natural structures present in the data they represent.

4

Current Work and Future Plans

The agent visualisation architecture has formed the basis for DIME, an ongoing research project implemented using Java and the Java 3D API which has allowed the rapid development of prototype agents and provides a stable platform to introduce additional functionality. DIME provides an interface to the agent environment allowing a user to browse the agent class hierarchy and expand individual classes to study its data agent instantiations. Properties of agents can be investigated and where necessary the characteristics of an agent can be changed by removing, adding or changing its properties. Visible agents are rendered in the agent environment window by the agent rendering system. At present visualisations in DIME use graphical style sheets (GSS) [3]. A GSS specifies the properties that should be assigned to the agents for an effective visual representation of the data they represent. A future enhancement will introduce dynamic property allocation by introducing user profile agents that characterise a user. Such agents are assigned the task of identifying data that is relevant to the user; the agents representing this data are given greater prominence in the visual representation. A long term goal for the development of DIME is to create a system capable of acting as a data repository. Our ambition is for the data management to be controlled by agents encouraging a long-term data management strategy rather than a system for short-term visualisation.

References 1. Bertin, J.: Graphics and graphic information processing. Walter de Gruyter, Berlin, 1981. 43 2. Healey, C.: Choosing effective colours for data visualisation. IEEE Visualisation 1996. 49 3. Felciano, R., Altman, R.: Graphical Style Sheets: Towards Reusable Representations of Biomedical Graphics. Conference on Human Factors in Computer Systems 1998 (to appear). 50 4. Grice, H.: Logic and Conversation. In Cole P and Morgan J eds. Syntax and Semantics: Speech Acts. 3 (1975) 41-58. New York: Academic Press. 46 5. Koffka, K.: Principles of Gestalt Psychology. London New York: Kegan Paul, Trench, Trubner, : Harcourt, Brace. 1935. 49 6. Marks, J.: A Formal Specification Scheme for Network Diagrams That facilitates Automated Design. Journal of Visual Languages and Computing. 2 (1991) 395-414 44 7. Roth, S., Mattis, J.: Data Characterization For Intelligent Graphics Presentation. CHI’90 Proceedings. 44

Error-Tolerant Database for Structured Images

A. Ferro, G. Gallo, and R. Giugno Dipartimento di Matematica Università di Catania, Viale A. Doria 6, 95125, Catania, Italy Phone +39095733051, Fax +39095330091 {ferro, gallo, giugno}@dipmat.unict.it

Abstract. This paper reports the application of an error-tolerant retrieval technique introduced in 1997 by Oflazer [IEEE Transactions on P.A.M.I., vol. 19, No. 12, December 1997] for databases of trees coding pictures of similar objects. The technique is of great interest for the classification and maintenance of historical and archeological pictorial data. We demonstrate the approach on a small postal stamp collection and describe the same application on a large set of pictures.

1

Introduction

Some classes of images can be considered as "statements" in a complex, non-linear, language. Such is the case of icons, coin images, coats and arms, postal stamps etc. All these objects have their own structure and a precise syntax. Although not fully general [3] the "syntactic" approach to image understanding can be usefully applied to machine recognition. The database structure proposed in this paper is one of such applications. A high level approach to classify pictures according to their basic features can be produce great advantages. These features can be the responses to linear and/or nonlinear filters applied to the images or can be features with a precise semantic for the human observer. Features can be recovered and analyzed using object recognition algorithms in a completely automated approach or, in a semiautomatic approach require the intervention of a human operator. Unfortunately there is no simple way, be it automated or dependent on human intervention, of organizing the elements of a generic picture in an unambiguous structure. "Content based" image databases ([1,4,6,7]) are still in an experimental stage although the initial efforts are very promising. Better results are possible when the database includes only very specific kind of images: in particular we propose a semantic approach to build data bases for homogeneous highly structured images. More precisely, a collection of homogeneous and highly structured images is a set of pictures where all elements share the same "layout" but differs for details. Furthermore we require that details range over a finite set of possibilities and that they can be unambiguously classified. Examples of such collections are sets of images of similar objects like those that frequently occur in the cataloging of cultural or artistic artifacts: database of coins, data base of postal stamps, etc. For demonstration sake in the rest of this paper we will describe our Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL'99, LNCS 1614, pp. 51-59, 1999.  Springer-Verlag Berlin Heidelberg 1999

52

A. Ferro et al.

innovative approach referring to a collection of postal stamps. The good results obtained in our toy example has been easily replicated on larger collections (about 100 items) and there is no real obstacle to exclude the same approach on much larger collections (more than 1000 items). We report also experiments with a database built in a complete automated way using the relative distribution of pixels in an image. Although this last approach focuses on issues that are not paramount for human perception promising performance has been obtained also in this case.

(a)

(d)

(g)

(b)

(e)

(h)

(c)

(f)

(i)

Fig. 1. The collection of the stamps used to illustrate our proposal referred in the text as “toy example”. The following ids have been used to refer to each stamp: (a) Italia650; (b) Usa60; (c) Berlin20; (d) Deutschland100; (e) Deutschland80; (f) Italia750; (g) Helvetia90; (h) Helvetia180; (i) Republique Francaise.

This paper is organized as follows: Section 2 describes how to code a picture into a tree, both using an interactive procedure based on the semantic of the image and using a totally automated procedure based on the spatial distribution of the pixels. Section 3 introduces a “metric” for the distance between the coding trees and shows how to organize a collection of trees into a compact, space efficient trie. Section 4 reports the results obtained in the experiments and section 5 draws some conclusions and indicates further research directions.

2

Building Tree from Images

This Section describes how to code an image drawn from a collection of structured and homogeneous images into a tree. The basic idea is to treat a structured image as a sentence in a natural language in order to be able to apply the technique described in [5]. Sentences in a natural language are structured in a general pattern. For example statements expressing the action of a subject on an object can be divided into subparts: subject, verb, object. Each subpart can be further decomposed in noun, article, attribute and so on. This, naturally, suggests describing and storing such sentences with a hierarchical data structure like a tree. The choice of the features to organize in a tree for each

Error-Tolerant Database for Structured Images

53

image/object and the structure of the tree is perhaps the most sensitive parameter of the proposed approach. We experimented with two strategies: one semantically based and one based on the distribution of the pixels. 2.1

Semantically Based Tree Construction

The strategy based on the semantic of images depends on the knowledge of an expert: an accurate analysis has to determine which details and features are helpful in assessing the identity of each data base item. It is important to avoid features that introduce only wasteful noise. To illustrate in a concrete way how the image coding is realized we report one possible scheme for some of the features of postal stamps. A typical "stamp tree" can be seen in Fig. 2 and in Fig. 3. Observe that not all of the many possible details and features have been introduced in our scheme. This has been an intentional choice to reduce the complexity of our example and grants a greater clarity in exposition. As a counter effect if the details mentioned in Fig. 2 are the only ones to be considered, even different stamps (for example (g) and (i) in our collection, shown in Fig. 1) have very similar trees. LAYOUT

COLOR

SUBJECT

BACK GROUND

SOUTH

N

T

EAST

N

WEST

NORTH

T

N

T

N

T

Fig. 2. General tree of a stamp, when the semantic approach is adopted. LAYOUT ∈ {Horizontal, Vertical}; COLOR ∈ {1, 2, 3, More}; SUBJECT ∈ {Monument, Face, Drawing}; BACKGROUND ∈ {Empty, Full}; N ≡ Number; T ≡ Text. V

MORE

M

F

S

T

N

N

T

Fig. 3. The tree of the stamp Italia650.

2.2

Pixels Based Tree Construction

The semantic tree construction gives impressive results but it is, in some way, inadequate because it requires the intervention of a human operator/expert. As an

54

A. Ferro et al.

alternative we tested also a completely automated coding of a picture into a tree based on the pixels distribution. The idea is to decompose a picture A recursively as follows: Step1 Divide A with median horizontal cut into two sub regions: Ah1 and Ah2. Step2 Check if the two resulting sub pictures are close under some prescribed similarity measure (for example they have close mean values for the RGB channels). If Ah1 and Ah2 are close go to Step3 else repeat the subdivision recursively on Ah1 and Ah2. Step3 Divide A with a median vertical cut into two sub regions Av1 and Av2. Step4 Check if the two resulting sub pictures are close under some prescribed similarity measure. If Av1 and Av2 are not close repeat the subdivision recursively on Av1 and Av2 else STOP subdivision.

The decomposition obtained following the above algorithm can be stored in a tree whose leaves bear a label with mean RGB values of the non-decomposable regions. For an example Fig. 4 shows the tree associated to stamp “Italia650”. H

H H

H H

V

V V

H V

V

H V

V

V q

a

b

c d e f g

h

i

l m n

o

p

Fig. 4. The decomposition tree of the stamp Italia650, when the pixels approach is adopted. The label H means "horizontal cut" and V means "vertical cut". The values of the leaves are the mean RGB and in this case they are: a = (227, 225, 192); b = (237, 233, 202); c = (192, 129, 92); d = (182, 128, 96); e = (231, 174, 115); f = (184, 153, 112); g = (154, 147, 119); h = (161, 158, 132); i = (166, 169, 146); l = (145, 160, 141); m = (117, 119, 98); n = (153, 155, 135); o = (163, 87, 68); p = (121, 109, 89); q = (233, 202, 178).

3

Finding Trees in a Forest

The constructions in section 2 show how to turn a collection of pictures into a forest of trees. Oflazer [5] has recently reported efficient ways to retrieve in this forest all the trees "similar" up to some degree to a given query. In particular, given an input

Error-Tolerant Database for Structured Images

55

tree and a similarity threshold, an algorithm is known that efficiently retrieves all the trees in the database whose similarity with the input exceeds the threshold. 3.1

Similarity Measures between Trees

A similarity measure is essential if a flexible, error tolerant, retrieval of database items has to be performed. Following [5] the distance between two trees is defined taking into account structural differences and label differences. Two trees can be different because there are two different labels in corresponding nodes or because some branch in one has no correspondence in the other. Let C be the cost in case of different labels and S the cost for a structural difference. The distance between the trees, is the minimum cost of leaves or branches insertions, deletions or leaf labels changes necessary to change one tree into the other. To be more precise, we adopt the data structure “vertex list sequence” to represent a tree T. This structure is a sequence of lists. There are as many lists in this sequence than leaves in the tree. Each list contains the ordered sequence of vertices in the unique path from the root to the corresponding leaf. For example the tree in the Fig. 3 is represented as follows: ((V, MORE), (V, M), (V, F), (V, S, T), (V, N, N), (V, N, T)) Using this formalism we can introduce the following definition. Definition. Let Z=Z1,…, Zp, denote a generic vertex list sequence of p vertex lists. Z[j] denotes the initial subsequence of Z up to and including the j-th vertex list. We will use X (of length m) to denote the query vertex list sequence, and Y (of length j) to denote the sequence that is a (possibly partial) candidate vertex list sequence (from the database of trees). i=0 ìdist ( X [0], Y [ j ]) = j * S ï j=0 ïdist ( X [i], Y [0]) = i * S ïdist ( X [i − 1], Y [ j − 1]) if X i = Y j i.e., last vertex lists are same ï ïmin(dist ( X [i − 1], Y [ j − 1]) + C , dist ( X [i], Y [ j ]) = í dist ( X [i − 1, Y [ j ]) + S , ï ï dist ( X [i], Y [ j − 1]) + S ) if X i and Y j differ only at the leaf label ï ïmin(dist ( X [i − 1], Y [ j ]), ï dist ( X [i], Y [ j − 1])) + S otherwise î

If the semantic tree coding is adopted Table 1 reports the "similarity values" between all the stamps in our toy collection assuming that C=1 and S=2. This is also the choice we made in the experiments reported in Section 4. If the pixels based tree coding is adopted very different results are obtained. Table 2 reports the "similarity values" between all the stamps in our toy collection assuming that C=1 and S=2 in this case. Fig. 5 is a graphical representation of the distances between the stamp Italia650 and the other stamps stored in our collection obtained following both the semantic and the pixel approach.

56

A. Ferro et al. DIST.

A

B

C

D

E

F

G

H

I

A B C D E F G H I

0 5 11 12 4 3 22 24 22

5 0 11 10 7 8 22 24 22

11 11 0 4 7 9 22 24 22

12 10 4 0 8 13 22 24 22

4 7 7 8 0 7 22 24 22

3 8 9 13 7 0 20 22 20

22 22 22 22 22 20 0 3 2

24 24 24 24 24 22 3 0 4

22 22 22 22 22 20 2 4 0

Table 1. Distances between the stamps in the database with C =1 and S = 2 when the semantic tree coding is adopted. A=Italia650, B=Usa60; C=Berlin20; D=Deutschland100; E=Deutschland80; F=Italia750; G=Helvetia90; H=Helvetia180; I=Republique Francaise. DIST.

A

B

C

D

E

F

G

H

I

A B C D E F G H I

0 20 22 20 114 146 89 90 149

20 0 24 16 116 148 91 92 151

22 24 0 24 112 144 87 88 147

20 16 24 0 116 148 91 92 151

114 116 112 116 0 122 98 96 206

146 148 144 148 122 0 178 176 238

89 91 87 91 98 178 0 50 181

90 92 88 92 96 176 50 0 182

149 151 147 151 206 238 181 182 0

Table 2. Distances between the stamps in the database with C=1 and S=2 when the pixel based tree coding is adopted. A=Italia650, B=Usa60; C=Berlin20; D=Deutschland100; E=Deutschland80; F=Italia750; G=Helvetia90; H=Helvetia180; I=Republique Francaise.

144

22

12

11

22

5

3

20

4

89 24

22

(a)

20

146

90 114

(b)

Fig. 5. The similarity between the stamp Italia650 and the other stamps when (a) the semantic tree coding, (b) the pixels based tree is adopted.

Error-Tolerant Database for Structured Images

3.2

57

Organizing a Forest in a Tree

The trees associated to the elements in our collection are stored in the database with their vertex list sequence representation. The set of vertex list sequences is, in turn, converted into a trie structure [2]. The tree will compress any possible redundancies in the prefixes of the vertex list sequences to achieve a compact data structure. For instance, the trees associated to the stamps of the collection in Fig. 1, when coded account the semantic approach, can be represented as a trie as shown in Fig. 6.

(V, MORE) (H, 1)

(H, D)

(H, MORE)

(V, M)

(H, D)

(H, F)

(V, 2)

(V, 3)

(H, F)

(V, F)

(V, E)

(V, M)

(V, F)

(V, F) (V, F)

(V, F)

(V, F)

(V, E, T) (H, S, T)

(H, S, N)

(V, N, N)

(V, S, T)

(H, S, T) (H, N, T)

(i)

(H, N, N) (H, N, N)

(g)

(V, S, N)

(V, S, N)

(O, N, T)

(h)

(V, N, T)

(f)

(V, E, T)

(V, N, N)

(V, W, T)

(V, W, T)

(d)

(c)

(V, N, N)

(V, S, T)

(V, N, T)

(b)

(V, E, T) ( V, N, N)

(e)

(V, N, T)

(a)

Fig. 6. The tree associated to the trees of stamps of our database. 3.3

Error Tolerant Retrieval of Trees

Our goal is the retrieval of trees that match a query up to some degree of approximation. Standard searching within a trie corresponds to traversing a path starting from the start node, to one of the leaves, so that the concatenation of the labels on the arcs along this path matches the input vertex list sequence. For error-tolerant searching, one has to find all paths from the start node to leaves, such that, the corresponding vertex list sequences are within a given distance threshold t of the query vertex list sequence. To perform efficiently this search, paths in the trie that lead to no solutions have to be pruned early so that the search is bound to a very small portion of data structure. The detailed description of on efficient search strategy is outside the scope of this paper; here we report a sketch of the idea developed in [5]. To perform an efficient depth first probing of the trie one has to compute the similarity distance between subsequences of the query and sequences obtained chaining together the labels of the visited nodes of the trie. This can be done recursively, in a dynamic programming fashion, maintaining a suitable array H of distances where the entry H(i, j) represents the distance between the subsequence of the first i vertex lists in the query and the

58

A. Ferro et al.

sequence of j vertex list obtained visiting a node of depth j of the trie. The array H(i, j) is locally updated during the visit of the trie and during the possible backtracking steps. A backtracking step occurs whenever a similarity threshold t (also known as cutoff distance) is exceeded. The leaves that have been reached during the search, within the cutoff distance bounds are the output of the error tolerant matching procedure. Oflazer has shown that this search can be realized in O(L2log L k1/lceiling(t/S)) where L is the number of leaves in each tree, k is number of the tree in the forest, t is the cutoff distance and S is the cost of adding or deleting a leaf in a certain tree. 4

Results

In this section we report some of the results obtained using our algorithm. A first experiment has been to query extensively the database of the stamps in Fig. 1. We have performed searches of queries already in the database and of queries new to the database. If the query is already in the database we have always obtained an exact match. In Fig. 7 the portion of the trie visited with threshold t = 2 with a query outside of stored collection is represented. The output is the stamp “Deutschland100” that belongs to the very some emission of the query and hence is very similar to it. If the threshold t is set to 4 more outputs are found. In particular the algorithm retrieves Deutschland100 and in addition Berlin20. Although Berlin20 may appear very different from Deutschland300 it shares with it the vertical layout, the woman portrait and few other details that in our coding scheme are important. This shows that it is important to have a coding scheme that closely maps the perceptively and semantically important information of each item in the database.

(i)

(g)

(f) (h)

(d)

(c)

(b)

(e)

(a)

Fig. 7. Search resulting from query the stamp “Deutschland300” and threshold t = 2 as a query. The solid branches are the only branches visited from the search.

The good results obtained in these first simple experiments have been replicated on a larger database regarding 100 stamps coded using a semantic approach. We ran the following quantitative test: a human expert have been asked to retrieve form the collection a set of 4 stamps that are, according to his judgment, close to a "query stamps". We compared the expert selections with the output obtained from the same query using our algorithm. In all cases, both expert and algorithm retrieved the very same copy of the query whenever present. The set of the other two stamps retrieved

Error-Tolerant Database for Structured Images

59

from the expert included some of the output obtained from our algorithm in 42% cases with similarity threshold t = 2. This percentage goes to 84% with similarity threshold t = 4. Similar results have been obtained when the query does not have an exact match in the database. As a final experiment we run the same test on the same 100 stamps database using the alternative coding described in section 2.2. Although the performance decreased because the pixel based coding gives more relevance to features that are clearly not taken into account from a human expert, we believe that some combination of the two coding could produce very reliable results. 5

Conclusion and Research Perspectives

In this paper we have demonstrated an application of the error tolerantly retrieval technique proposed in [5] according two coding strategies. We have conducted experiments using a database of . images with similar layout and different details from a finite set of possibilities. The database has been queried and the answers have been statistically compared with the findings of an expert. The observed performance proves the suitability of the method for database of visual information. Future work following this line of research concerns both applications and more refined theoretical analysis of the problem. It is particularly interesting to characterize the class of images that are best for our technique in more precise terms using the syntactic tools provided from visual languages. For the applications we have undertook the construction of a database for a collection of ancient Greek coins in order to validate the proposed method on a real life situation of moderate size. We intend also to prove the applicability of the proposed technique to Web search, and as a support for electronic commerce. References 1. J. R. Back, C. Fuller, A. Gupta, A. Hampapur, B. Horowitz, R. Humphrey, and R. Jain, “The Virage Image Search Engine: An Open framework for Image Management”, Proc. SPIE, Storage and Retrieval for Still Image and Video Databases, vol. 2670, pp. 76-87, 1996. 2. T. H. Cormen, C. E. Leiserson, R. L. Rivest, “Introduction to Algorithms”, MIT press 1990. 3. D. Marr, "Vision", Freeman and Co., New York, 1982. 4. W. Niblack, R. Barber, W. Equitz, M. Flickner, E. Glasman, D. Petkovic, P. Yanker, C. Faloutsos, G. Taubin, “The QBIC Project: Querying Images by Content Using Color Texture, and Shape”, Proc. SPIE, vol. 1908, Storage Retrieval for Image and Video Databases, 1993, pp. 173-187. 5. K. Oflazer, “Error-Tolerant Retrieval of Trees”, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 19, No. 12, December 1997. 6. A. Pentland, R. W. Picard, S. Sclaroff, “Photobook: Tools for Content-Based Manipulation of Image Database”, Proc. SPIE, vol. 2185, Storage and Retrieval for Image and Video Databases, Feb. 1994, pp. 34-47. 7. J. R. Smith and S. F. Chang, “VisualSeek: A fully automated content-based image query system”, Proc. International Conference on Image Processing, Lausanne, Switzerland, 1996.

Query Processing and Optimization for Pictorial Query Trees Aya Soffer1,2 and Hanan Samet2 1

2

Computer Science Department, Technion City, Haifa 32000, Israel [email protected] Computer Science Department, Institute for Advanced Computer Science University of Maryland at College Park, College Park, Maryland 20742 [email protected]

Abstract. Methods for processing of pictorial queries specified by pictorial query trees are presented. Leaves of a pictorial query tree correspond to individual pictorial queries while internal nodes correspond to logical operations on them. Algorithms for processing individual pictorial queries and for parsing and computing the overall result of a pictorial query tree are presented. Issues involved in optimizing query processing of pictorial query trees are outlined and some initial solutions are suggested. Keywords: image databases, query specification, query optimization, retrieval by content, spatial databases, image indexing

1

Introduction

Image databases must be capable of being queried pictorially. The most common method of doing this is via an example image. The problem with this method is that in an image database we are usually not looking for an exact match. Instead, we want to find images similar to a given query image. One of the main issues is how to determine if two images are similar and whether the similarity criteria that are used by the database system match the user’s notion of similarity. Another difficulty with pictorial queries is that they are usually not very expressive in terms of specifying combinations of conditions and negative conditions. A good pictorial query specification method should leverage on the expressiveness of pictorial queries in terms of describing what objects the target images should contain and their desired spatial configuration, resolve the ambiguities inherent to pictorial queries, and enable specifying combinations of conditions. In our previous work [8], we devised a pictorial query specification technique for formulating queries that specify which objects should appear in a target image




The support of the Lady Davis Foundation and the National Science Foundation under Grant CDA-950-3994 is gratefully acknowledged. The support of the National Science Foundation under Grant IRI-97-12715 is gratefully acknowledged.

Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL’99, LNCS 1614, pp. 60–68, 1999. c Springer-Verlag Berlin Heidelberg 1999


Query Processing and Optimization for Pictorial Query Trees

61

as well as how many occurrences of each object are required. Moreover, the minimum matching certainty between query-image and database-image objects can be specified, and spatial constraints that specify bounds on the distance between objects and the relative direction between objects can be imposed. Expressive power is achieved by allowing a pictorial query specification to be composed of one or more query images and by allowing a query image to be negated. In [9] we extended the pictorial query specification technique to allow the formulation of complex pictorial queries via pictorial query trees where leaves correspond to individual pictorial queries while internal nodes represent logical operations on the set of pictorial queries (or subtrees) represented by its children. In this paper we describe in detail the algorithm for processing individual pictorial queries. It can handle multiple instances of each symbol in the query image as well as in the database image and can also handle wild cards. In addition, we present an algorithm for parsing and computing the overall result of a pictorial query tree. We also discuss some issues that are involved in query optimization of pictorial query trees (i.e., their simplification).

2

Related Work

Most image database research deals either with global image matching based on color and texture features [4,5,10] or with the ambiguity associated with matching one query-image object to another [3]. These methods do not address the case of images that are composed of several objects and their desired spatial configuration. There has been some work on the specification of topological and directional relations among query objects [1,2]. These studies only deal with tagged images. Furthermore, it is always assumed that the goal is to match as many query-image objects to database-image objects as possible. A limited form of spatial ambiguity is allowed in pictorial queries based on the 2D-string and its variants [2]. The spatial logic described in [1] also allows specification of query images in terms of spatial relations between objects and permits users to select the level of spatial similarity. In some cases (e.g., [6]), the images are segmented into regions either automatically or semi-automatically and some queries involving spatial constraints specifying the desired arrangement of these regions can be performed. However, the issue of the distance between objects is not addressed by these or any other method. Furthermore, none of these methods provide Boolean combinations or negations of query images.

3

Building Pictorial Query Trees

An individual pictorial query is specified by selecting the required query objects and positioning them according to the desired spatial configuration. Next, the similarity level in terms of three parameters is specified. The matching similarity level msl is a number between 0 and 1 that specifies a lower bound on the certainty that two symbols are from the same class and thus match. Contextual similarity specifies how well the content of database image DI matches that

62

Aya Soffer and Hanan Samet

Fig. 1. Contextual similarity levels (csl).

Fig. 2. Spatial similarity levels (ssl).

of query image QI (e.g., do all of the symbols in QI appear in DI ?). We use four levels of contextual similarity (see Figure 1). Spatial similarity specifies how good a match is required in terms of the relative locations and orientation of the matching symbols between the query and database image. We use five levels of spatial similarity (see Figure 2). For more details and examples, see [8]. Complex pictorial queries are composed of combinations of individual pictorial queries and are specified via pictorial query trees. Leaves of a pictorial query tree correspond to individual pictorial queries. A negated leaf node (NOT) yields the set of all images that do not satisfy the pictorial query. Internal nodes in the tree represent logical operations (AND, OR, XOR) and their negations on the set of images that satisfy the pictorial query (or query subtree) represented by its children.

AND

AND

2

OR 7

OR

csl = 2 ssl = 2

csl = 2 ssl = 4

(a)

csl = 2 ssl = 4

10

5

10

5

csl = 2 ssl = 4

csl = 2 ssl = 4

csl = 2 ssl = 4

(b)

Fig. 3. Images with a (a) camping site within 5 miles of a fishing site OR a hotel within 10 miles of a fishing site AND an airport northeast of and within 7 miles of the fishing site . (b) camping site within 5 miles OR with a hotel within 10 miles of a fishing site of a fishing site AND with no airport within 2 miles of the fishing site (the line above the query denotes negation). The root of the tree is either a pictorial query or a logical operator, while an internal node corresponds to a logical operator and can have one or more

Query Processing and Optimization for Pictorial Query Trees

63

children. For a conjunction of query images where the same symbol appears in both query images, the user may specify whether the two query-symbols must match the same instance of the symbol in the database image, or whether two different instances are allowed. This is termed object binding.

AND

OR

NOR

10

5

csl = 2 ssl = 4

csl = 2 ssl = 4

csl = 2 ssl = 5

csl = 2 ssl = 5

Fig. 4. Images with a camping site within 5 miles OR with a hotel within 10 miles of a fishing site AND with neither a restaurant nor a cafe . Figures 3 and 4 are examples of pictorial query trees. Figure 3a specifies more than one acceptable spatial constraint (i.e., a camping site within 5 of a or a hotel within 10 of a fishing site ) with an OR. We also fishing site specify that there is an airport northeast of and within 7 miles of the fishing site with an AND. Figure 3b shows negation of a pictorial query specifying within 2 miles of the fishing site . Figure 4 shows that there is no airport negation of logical operators in internal nodes. Here we are seeking images with a camping site within 5 miles of a fishing site OR a hotel within 10 but without either a restaurant or a cafe . miles of a fishing site

4

Pictorial Query Processing

In this section, we present an algorithm for retrieving all database images that conform to a given pictorial query tree specification. 4.1

Processing individual pictorial queries

The first step in the algorithm processes each pictorial query image (i.e., each leaf) individually using function GetSimilarImagesM that takes as input a query image (QI ), the matching similarity level (msl ), the contextual similarity level (csl ), and the spatial similarity level (ssl ) associated with QI . It returns the set of database images RI such that each image DI ∈ RI satisfies the pictorial query. Figure 5 summarizes this algorithm. GetSimilarImagesM handles wildcards as well as multiple instances of each class in the query image and in the database image.

64

Aya Soffer and Hanan Samet

GetSimilarImagesM(logical image QI, msl, csl, ssl)

m←0 /* check matching similarity */ foreach el ∈ QI if (el = wildcard) then rm ← set of all images stored in the database else rm ← set of all images containing C(el) with certainty ≥ msl (use index on class) m←m+1 /* check contextual similarity */ if (csl = 1) ∨ (csl = 2) then
n−1 RI ← i=0 ri elseif (csl = 3) ∨ (csl = 4) n−1 RI ← i=0 ri if (csl = 1) ∨ (csl = 3) then RI ← RI − {I s.t. I includes symbols not in QI} (use index on image id) /* check multiple symbol instance conditions */ if (csl = 1) ∨ (csl = 2) then I RI ← RI − {I s.t. ∃k : nQI k > nk }, i.e. some (k-th) symbol of QI is underrepresented in I /* check spatial similarity */ foreach I ∈ RI for every possible matching of symbols between QI and I check feasibility of this matching w.r.t. spatial constraints if all matchings are infeasible RI ← RI − I return RI ordered by average certainties

Fig. 5. Algorithm to retrieve all database images similar to a query image (QI) conforming to constraints dictated by msl, csl, and ssl. nIk denotes the number of occurrences of the k th symbol in image I

First, for each symbol in the query image it finds all database images, DI, that contain this symbol with certainty ≥ msl. Next, it handles sthe contextual constraints. If csl is 1 or 2 (images should contain all symbols in QI), then it intersects the set of result images from the first step. If csl is 3 or 4 (any one symbol from QI is enough), then it takes the union of the result images. If the contextual similarity level is 1 or 3, then it avoids including images containing symbols that are not present in QI . Next, it checks the case of multiple instances of query symbols in the query image. If csl is 1 or 2, then for every instance of each symbol in QI, it checks whether there exists an instance of the symbol in DI.

Query Processing and Optimization for Pictorial Query Trees

65

Finally, it checks whether the spatial constraints are satisfied for each candidate image I in the candidate image list RI. Since multiple instances of symbols are allowed in QI and in I, this step needs to check many possible matchings. It can be that some mappings between QI symbols and I symbols create feasible configurations while others do not. For each QI symbol create a set of possible matches in I. Selecting one element from each of these sets generates one possible matching. If none of the possible matchings pass the spatial constraints test, then remove the image from the candidate result set. The spatial similarity between any two matchings is calculated using algorithm CheckSsl [8] which determines whether the spatial constraints dictated by a query image QI and spatial similarity level ssl hold in a logical image DI . Images that pass all of the tests are ordered by the average matching certainty of all matching symbols and returned as the result of the query. 4.2

Parsing and evaluating pictorial query trees

ProcessQueryTree(query tree node: N )

S ← set of all images in the database (global variable) if (isLeaf (N )) N R ← GetSimilarImagesM (QI(N ), msl(N ), csl(N ), ssl(N )) if (hasN egationF lag(N )) NR ← S − NR else n←0 foreach M ∈ sons(N ) rn ← P rocessQueryT ree(M ) n ← n+1 
 , , or , possibly inverted) N R ← OP (N )n−1 i=0 ri (OP (N ) can be return N R

Fig. 6. Algorithm to retrieve all images satisfying the query represented by node N of a pictorial query tree. Procedure P rocessQueryT ree parses and evaluates the result of a pictorial query tree. Figure 6 summarizes the algorithm. ProcessQueryTree takes as input a node N in the query tree, and returns the set of images that satisfy the query tree rooted at N . If N is a leaf node, then it checks whether the results of this query are cached from earlier invocations. If they are not, then algorithm GetSimilarImagesM is invoked. If the leaf node is negated in the tree, then the complement of the result images set returned by GetSimilarImagesM is taken. The final result image set is returned. If N is an internal node in the query

66

Aya Soffer and Hanan Samet

tree, then ProcessQueryTree is called recursively on each child of N , followed by applying the appropriate logical operation on the results of these calls. The whole query tree is evaluated in this recursive manner by invoking algorithm ProcessQueryTree with the root of the query tree as an argument. Recall, that users can specify object binding. That is, whether the same instance of an object is to be used when it appears in more than one of the pictorial query images that make up the pictorial query tree. The following is an outline of the additions to our algorithms that are necessary for handling object binding. Algorithm ProcessQueryTree receives as additional input a global set of constraints that stipulates the bindings that were specified as part of the query. This set consists of groups of symbols, where all of the symbols in the same group should be matched to the same symbol instance in the database image. To filter out database images that are incompatible with respect to the binding conditions, we combine these binding constraints with information that is provided by the algorithm GetSimilarImagesM, which is augmented to return for each database image that was found similar to the query image, the mapping between query symbols and matched database symbols.

5

Query Optimization Issues

Several optimization techniques can be applied to improve the efficiency of processing pictorial query trees. These include methods designed for optimization of individual pictorial query processing and optimization of query tree processing. Individual pictorial query processing may be made more efficient by handling spatial and contextual constraints simultaneously rather than one followed by the other as we do now. We addressed this issue in [7]. Two optimizations are possible for computing the result of the pictorial query tree. The first optimization is to change the order of processing individual query images in order to execute the parts that are more selective (i.e., result in fewer images) first. The selectivity of a pictorial query is based on three factors. Matching selectivity estimates how many images satisfy the matching constraint as specified by msl. Contextual selectivity estimates how many images satisfy the contextual constraint as specified by the query image and csl. Spatial selectivity estimates how many images satisfy the spatial constraint as specified by ssl. Depending on ssl, either distance, direction, both, or neither are constrained. Matching and contextual selectivity factors are computed based on statistics stored as histograms in the database which indicate the distribution of classifications and certainty levels in the images. These histograms are constructed when populating the database. Computing spatial selectivity is much more complex. One approach to measuring the distance aspect of the spatial selectivity calculates some approximation of the area spanned by the symbols in the query image. This can be estimated, for example. using an approximation of the convex hull of the symbols in the query image. Details of this method are beyond the scope of this paper. Selectivity of an individual pictorial query (leaf) is computed by combining these three selectivity factors.

Query Processing and Optimization for Pictorial Query Trees

67

The query tree selectivity is computed using a recursive algorithm similar to the one executing the query. If an individual pictorial query is negated in the tree, the selectivity is 1 - the selectivity of the query. The selectivity of a subtree is as follows. For OR or XOR, take the sum of the selectivities of the subtrees minus the probability that a combination of cases occured. For AND, take the product of the selectivities of the subtrees. To illustrate the general use of this optimization method, consider the query trees in Figure 3. In both queries the left side of the tree requests images with within 5 miles of a fishing site OR a hotel within 10 a camping site miles of a fishing site . In query (a), we add the constraint that there exists an airport northeast of and within 7 of fishing site . In our database, we have very few airfields and thus the right side is more selective and it will be processed first. On the other hand in query (b), we add the constraint that within 2 miles of the fishing site . Clearly, in most there is no airport cases there will be no such airport , and thus in this case the right side is not selective and the left side should be processed first. The second form of optimization is to combine individual query images and to process them together. To see its usefulness, we study how the query in Figure 4 is processed using the current algorithm. First, find {CF} all images with a within 5 of a fishing site . Next, find {HF} all images with a camping site hotel within 10 of a fishing site . Then, take the union of these two sets: and {LS} = {CF } ∪ {HF }. Now, find the set {R}: images with a restaurant the set {C}: images with a cafe and compute the set RS = I − (R ∪ C). The final result is the intersection of the two sets: LS ∩ RS. A more sensible way , find the nearest to compute this query is as follows. For each fishing site neighbors up to distance 5 in incremental order. If the next nearest neighbor is a camping site or a hotel , then add this image to the candidate list. Continue retrieving nearest neighbors in incremental order up to distance 10. If , then add this image to the candidate the next nearest neighbor is a hotel list. For each image I in the candidate list, examine all of the objects in I. If or a cafe in I, then remove I from the candidate list. there is a restaurant

References 1. A. Del Bimbo, E. Vicario, and D. Zingoni. A spatial logic for symbolic description of image contents. Jour. of Vis. Lang. and Comp., 5(3):267–286, Sept. 1994. 61 2. S. K. Chang, Q. Y. Shi, and C. Y. Yan. Iconic indexing by 2-D strings. IEEE Trans. on Patt. Anal. and Mach. Intel., 9(3):413–428, May 1987. 61 3. W. I. Grosky, P. Neo, and R. Mehrotra. A pictorial index mechanism for modelbased matching. Data & Know. Engin., 8(4):309–327, Sept. 1992. 61 4. W. Niblack, R. Barber, W. Equitz, M. Flickner, E. Glasman, D. Petkovic, and P. Yanker. The QBIC project: Querying images by content using color, texture, and shape. In Proc. of the SPIE, Storage and Retrieval of Image and Video Databases, vol. 1908, pp. 173–187, San Jose, CA, Feb. 1993. 61 5. A. Pentland, R. W. Picard, and S. Sclaroff. Photobook: Content-based manipulation of image databases. In Proc. of the SPIE, Storage and Retrieval of Image and Video Databases II, vol. 2185, pp. 34–47, San Jose, CA, Feb. 1994. 61

68

Aya Soffer and Hanan Samet

6. J. R. Smith and S.-F. Chang. VisualSEEk: a fully automated content-based image query system. In ACM Int. Conf. on Multimedia, pp. 87–98, Boston, Nov. 1996. 61 7. A. Soffer and H. Samet. Pictorial queries by image similarity. In 13th Int. Conf. on Patt. Recog., vol. III, pp. 114–119, Vienna, Austria, Aug. 1996. 66 8. A. Soffer and H. Samet. Pictorial query specification for browsing through spatially-referenced image databases. Jour. of Vis. Lang. and Comp., 9(6):567– 596, Dec. 1998. 60, 62, 65 9. A. Soffer, H. Samet, and D. Zotkin. Pictorial query trees for query specification in image databases. In 14th Int. Conf. on Patt. Recog., vol. I, pp. 919–921, Brisbane, Australia, Aug 1998. 61 10. M. Swain. Interactive indexing into image databases. In Proc. of the SPIE, Storage and Retrieval for Image and Video Databases, vol. 1908, pp. 95–103, San Jose, CA, Feb. 1993. 61

Similarity Search Using Multiple Examples in MARS
Kriengkrai Porkaew1, Sharad Mehrotra2 , Michael Ortega1 , and Kaushik Chakrabarti1 1

Department of Computer Science University of Illinois at Urbana-Champaign, Urbana, IL 61801, USA 2 Department of Information and Computer Science University of Cailifornia at Irvine, Irvine, CA 92697, USA {nid,sharad,miki,kaushik}@ics.uci.edu

Abstract. Unlike traditional database management systems, in multimedia databases that support content-based retrieval over multimedia objects, it is difficult for users to express their exact information need directly in the form of a precise query. A typical interface supported by content-based retrieval systems allows users to express their query in the form of examples of objects similar to the ones they wish to retrieve. Such a user interface, however, requires mechanisms to learn the query representation from the examples provided by the user. In our previous work, we proposed a query refinement mechanism in which a query representation is modified by adding new relevant examples based on user feedback. In this paper, we describe query processing mechanisms that can efficiently support query expansion using multidimensional index structures.

1

Introduction

In a content-based multimedia retrieval system, it is difficult for users to specify their information need in a query over the feature sets used to represent the multimedia objects [10, 7, 12]. Motivated by this, recently, many content-based multimedia retrieval systems have explored a query by example (QBE) framework for formulating similarity queries over multimedia objects (e.g., QBIC [4], VIRAGE [1], Photobook [9], MARS [6]). In QBE, a user formulates a query by providing examples of objects similar to the one s/he wishes to retrieve. The system converts this into an internal representation based on the features extracted from the input images. However, a user may not initially be able to provide the system with “good” examples of objects that exactly capture their information needs. Furthermore, a user may also not be able to exactly specify the relative


This work was supported by NSF awards IIS-9734300, and CDA-9624396; in part by the Army Research Laboratory under Cooperative Agreement No. DAAL01-962-0003. Michael Ortega is supported in part by CONACYT grant 89061 and MAVIS fellowship.

Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL’99, LNCS 1614, pp. 68–75, 1999. c Springer-Verlag Berlin Heidelberg 1999


Similarity Search Using Multiple Examples in MARS

69

importance of the different features used to represent the multimedia objects to the query. To overcome the above limitations, in the Multimedia Analysis and Retrieval (MARS) project, we explored techniques that allow users to refine the initial query during the retrieval process using relevance feedback [10]. Given an initial query, the system retrieves objects that are most similar to the query. The feedback from the user about the relevance of the retrieved objects is then used to adjust the query representation. Relevance feedback in MARS serves two purposes as follows. Query Reweighting adjusts the relative importance of the different components to the query. It allows the system to learn the user’s interpretation of similarity between objects. Query Modification changes the underlying representation of the query to incorporate new relevant information from the user’s feedback. It overcomes the deficiency of having started from examples that only partially capture the user’s information need. In [11, 12, 7, 10], various models for query reweighting and query modification were explored and compared over diverse multimedia collections. Specifically, two different strategies for query modification have emerged. The first, referred to as query point movement (QPM) [7, 11], attempts to move the query representation in the direction where relevant objects are located. At any instance, a query is represented using a single point in each of the feature spaces associated with the multimedia object. In contrast to QPM, in [10] we proposed a query expansion model (QEM) in which the query representation is changed by selectively adding new relevant objects (as well as deleting old and less relevant objects). In QEM, the query may consist of multiple points in each feature space. Our experiments over large image collections illustrated that QEM outperforms QPM in retrieval effectiveness (based on precision/recall measures) [10]. However, in QEM, its potential drawback is that the cost of evaluating the query grows linearly with the number of objects in the query if done naively. In this paper, we explore efficient strategies to implement QEM that overcome the above overhead. The key is to traverse a multidimensional index structure (e.g., X-tree [2], hybridtree [3], SS-tree [15], etc.) such that best N objects are retrieved from the data collection without having to explicitly execute N nearest neighbor queries for each object in the query representation. We conduct an experimental evaluation of our developed strategies over a large image collection. Our results show that the developed algorithms make QEM an attractive strategy for query modification in content-based multimedia retrieval since it provides better retrieval effectiveness without extensive overhead. The rest of the paper is developed as follows, Sect. 2 describes the contentbased retrieval in MARS. Section 3 describes the proposed approaches to implementing QEM. Section 4 compares the approaches and shows experimental results. Conclusions are given in Sect. 5.

70

2

Kriengkrai Porkaew et al.

Content-Based Retrieval in MARS

This section briefly describes the content-based retrieval mechanism supported in MARS which is characterized by the following models: Multimedia Object Model: a multimedia object is a collection of features and the functions used to compute the similarity between two objects for each of those features. Query Model: A query is also a collection of features. In QEM, a query may be represented by more than one instance (point) in each feature space. Furthermore, weights are associated with each feature, as well as, with each instance in the feature representation. These weights signify the relative importance of the component to the query. Figure 1 illustrates the query structure which consists of multiple features fi and each feature consists of multiple feature instances rij .

w1 w11

F1

R 11

Query

w2

w12

w21

R 12

R 21

F2

w22 R 22

F i = Feature i w i = Importance of Feature i with respect to the other features w ij = Importance of Feature i of Object j with respect to Feature i of the other objects R ij= Representation of Feature i of Object j

Fig. 1. Query Model

Retrieval model: The retrieval model defines how similarity Sim between a query Q and an object O is computed.
n Similarity is computed
n hierarchically over the query tree. That is Sim = i=1 wi Simi , where i=1 wi = 1, n is the number of features used in the queries, and Simi is the similarity between the object and the query
mbased on feature i which is computed as: Simi =
m j=1 wij Simij , where j=1 wij = 1, m is the number of feature instances in the feature i in the query, and Simij is the similarity between instance j and the object based on feature i. Simij is computed using the similarity function determined by the object model. The retrieval process begins with some initial weights associated with nodes at each level of the query tree. For simplicity, initially weights associated with nodes of the same parent are equal. Refinement Model: The refinement model adjusts the query tree and the similarity functions used at different levels of the tree based on the user’s feedback. As discussed in the introduction, the refinement process consists of query reweighting and query modification using query expansion model. The details of the reweighting models, and the query modification models are not critical for the discussion of implementation techniques in this paper and hence omitted due to space restrictions. Details can be found in [10].

Similarity Search Using Multiple Examples in MARS

3

71

Query Processing

At each iteration of query refinement, the system returns to the user N objects from the database that have the highest similarity to the current query representation. Instead of ranking each object in the database and then selecting the best N answers, the query is evaluated in a hierarchical bottom up fashion. First, the best few objects based on each feature individually are retrieved. The similarity values of these objects on individual features are then combined (using the weighted summation model) to generate a ranked list of objects based on the entire query. The process continues until the best N matching objects have been retrieved. We next discuss how feature nodes of the query are evaluated, and the answers are combined to obtain the best N answers for the query. 3.1

Evaluating Feature Nodes

In a query tree, let f be a feature node and r1 , . . . , rm be the instances (points) under the feature space F . The objective of evaluating the feature node is to retrieve N objects from the database that best match f . We will use the notion of distance instead of similarity since the evaluation of the feature node will use multidimensional indexing mechanisms that are organized based on distances. a point x in F and Let drj ,x be the distance between rj and
m
m Df,x be the distance between f and x in F where Df,x = j=1 wj drj ,x and j=1 wj = 1. Thus, the best N matches to f correspond to objects which are closest to f based on the above definition of distance. In the following two subsections, we describe two different strategies of evaluating the best N objects for a given feature node. Both strategies assume that the feature space is indexed using a multidimensional data structure that supports range and k-nearest neighbor queries. Centroid Expansion Search (CES): The idea is to iteratively retrieve next nearest neighbors of some point c (close to r1 , . . . , rm ) in the feature space F using the feature index until the N best matches to f are found. Let x and y be two objects in the feature space F . x is a better match to f compared to y if and only if Df,x ≤ Df,y , or equivalently m 

wj drj ,x ≤

j=1

m 

wi drj ,y

(1)

j=1

Since distance functions are metric, the triangle inequality dictates that drj ,x ≤ dc,x + dc,rj and drj ,y ≥ |dc,y − dc,rj |. Substituting drj ,x , drj ,y in (1): m 

m    wj dc,x + dc,rj ≤ wj |dc,y − dc,rj |

j=1

Since

m  j=1

wj = 1, we get: dc,x +

j=1 m 

m 

j=1

j=1

wj dc,rj ≤

wj |dc,y − dc,rj |

(2)

(3)

72

Kriengkrai Porkaew et al.

Thus, if (3) holds, then (1) also holds. To remove the absolute value from (3), let R = { r1 , . . . , rm }, R1 = { rj ∈ R | dc,rj ≤ dc,y }, and R2 = R − R1 = { rj ∈ R | dc,rj > dc,y }. Replace R1 and R2 in (3), dc,x + ≤

 rj ∈R1



wj dc,rj +

rj ∈R1

dc,x ≤ dc,y − 2 

wj dc,rj

rj ∈R2

wj (dc,y − dc,rj ) + 





wj (dc,rj − dc,y )

rj ∈R2



wj dc,y +

rj ∈R2

dc,x ≤ dc,y − 2

m 



(4)

 wj dc,rj 

(5)

rj ∈R1

wj min(dc,y , dc,rj )

(6)

j=1

Equation (6) provides the strategy to retrieve the best N answers based on the match to f . The strategy works as follows. We find the nearest neighbors to c incrementally. Let x1 , . . . , xP be the objects seen
mso far. We determine the target M, 1 ≤ M ≤ P such that Dc,xM ≤ Dc,xP − 2 j=1 wj min(dc,xP , dc,rj ). By (6), Df,xM ≤ Df,xP +k , k = 1, 2, . . .. Let α = max{Df,xi |i = 1, . . . , M }. We then determine the set {xi |i = 1, . . . , P ∧ Df,xi ≤ α}. All such xi are better matches to f than any object xP +k , k = 1, 2, . . . and are hence returned. If N objects have not yet been returned, the process continues iteratively by retrieving the next closest object to c (i.e., xP +1 ) and repeating the above algorithm.
mNotice that c can be any point. However, the optimal choice of c minimizes j=1 wj dc,rj ; i.e. c should be the weighted centroid of r1 , . . . , rm . This approach does not require any change to the incremental nearest neighbor search algorithm associated with the original multidimensional data structure. However, it does not perform well when query changes dramatically due to the relevance feedback process since the starting centroid is optimal for the original query. Multiple Expansion Search (MES): In this approach, N nearest neighbor for a feature node f is determined by iteratively retrieving next nearest neighbors for each instance r1 , . . . , rm associated with f . Let Rj be the set of ranked results for the instance rj , j = 1, . . . , m. That is, for all x ∈ Rj and y ∈ Rj , drj ,x ≤ drj ,y . Furthermore, let αj be the maximum distance between rj and any object in Rj in the feature space; that is, αj = max{drj ,x |x ∈ Rj }. nRj contains all objects that are in the range of αj from rj . Note that if y ∈ j=1 Rj , then drj ,y > αj
m
m
m for all j. So j=1 wj drj ,y > j=1 wj αj , that is, Df,y > j=1 wj αj . As a result, m
m y ∈ j=1 Rj if Df,y ≤ j=1 wj αj . m Note that if j=1 Rj contains at least N objects x1 , . . . , xN such that for
m all xk , Df,xk ≤ j=1 wi αj , then it is guaranteed that N best matches to the feature node f are contained in m j=1 Rj . Thus, in order to evaluate the best

Similarity Search Using Multiple Examples in MARS

73

N matches to f , MES incrementally evaluates the nearest neighbor for each of the instances r1 , . . . , rn thereby increasing the value of at least m one αj in each step, j = 1, . . . , m until there are at least N objects within j=1 Rj for which
m Df,xk ≤ j=1 wi αj . Many different strategies can be used to expand αj s. The optimal strategy n determines αj that minimize i=1 Ri since then the least number of objects are explored to retrieve the best N objects based on the match to the feature. We try different strategies for determining αj s and compare them in Sect. 4. 3.2

Evaluating the Query Node

Given the best matching answers for each of the feature nodes f1 , . . . , fn , the objective in evaluating the query node is to combine the results to determine the best N objects to the overall query. That is, we need to determine the N objects with the least distance to the
query, where the distance
n between object n and the query is defined as DQ,x = i=1 wi Df,x where i=1 wi = 1. MES discussed for the feature node evaluation can also be used for this purpose and is hence not discussed any further.

4

Experiments

To explore the effectiveness of the algorithms, we performed experiments over a large image dataset (65,000 images) obtained from the Corel collection. Images features used to test the query processing are color histogram [14], color histogram layout [8], color moments [13], and co-occurrence texture [5]. Manhattan distance is used for the first two features and Euclidean distance is used for the last two features [8]. The purposes of this experiment are to compare various approaches we proposed, and to show that QEM can be implemented efficiently. The effectiveness is measured by the number of objects seen before the best N answers are found. A good approach should not need to explore so many objects to guarantee the best N answers and it should not degrade significantly when multiple objects are added to the query. We performed experiments on CES and MES with various parameters. Specifically, CES searches from the centroid of the query point set. In MES, we explored 4 expansion options as follows. Single Search searches only in one of the query points. Balanced Search searches on all query points with equal ranges. Weighted Search searches on all query points with the ranges proportional to the weights of the query points. Inverse Weighted Search searches on all query points with the ranges proportional to the inverse of the weights of the query points. In the experiments, we do not use any index structure in order to avoid hidden effects caused by the specific index structure. Instead, we simulate a k-nearest neighbor search by scanning the dataset and ranking the answers.

74

Kriengkrai Porkaew et al.

The experimental result shows that single search performs the worst. Intuitively, one may expect the weighted search to perform the best among the four approaches. However, surprisingly, even though the weights are not balanced, the balanced search performed better than any search techniques including the centroid expansion search.

2000

10 Centroid Balance Weighted 1/Weight Single

Centroid Balance Weighted 1/Weight 8 Number of objects seen (x Top N)

top N

1500

1000

6

4

500 2

0

0 0

500

1000 Seen

1500

(a) Seen vs best N

2000

1

2

3

4 5 Number of points in a query

6

7

(b) Query size vs Seen/best N ratio

Fig. 2. Experimental Result

Figure 2 compares the different approaches and shows that the number of objects in the query representation has very little impact on the balanced search and the weighted search which are the best searches. The reason is simply because the feature space is sparse and the multiple query points are close together due to the query expansion model which selectively adds relevant query points and removes less relevant ones. Other approaches do not perform well since they may have seen best answers but they cannot guarantee that those answers are among the best ones unless they explore further.

5

Conclusions

Content-based multimedia retrieval and multidimensional indexing are among the most active research areas in the past few years. The two research areas are closely related. The supporting index structure has a big impact on the efficiency of the retrieval. In this paper, we proposed algorithms to extend index structures to support complex queries efficiently in the MARS weighted summation retrieval model. We focussed on an efficient implementation to support QEM proposed in [10]. QEM modifies the query by selectively adding new relevant objects to the query (as well as deleting old and less relevant objects). In contrast, QPM modifies the query by moving the query point in the direction of the relevant objects.

Similarity Search Using Multiple Examples in MARS

75

Our previous work showed that QEM outperforms QPM in retrieval effectiveness. This paper further illustrates that QEM can be efficiently implemented using multidimensional index structures. As a result, we believe that QEM is a viable approach for query refinement in multimedia content based retrieval.

References [1] Jeffrey R. Bach, Charles Fuller, Amarnath Gupta, Arun Hampapur, Bradley Horowitz, Rich Humphrey, Ramesh Jain, and Chiao fe Shu. The Virage image search engine: An open framework for image management. In SPIE Conf. on Vis. Commun. and Image Proc., 1996. 68 [2] S. Berchtold, D. A. Keim, and H. P. Kriegel. The x-tree: An index structure for high-dimensional data. In VLDB, 1996. 69 [3] Kaushik Chakrabarti and Sharad Mehrotra. High dimensional feature indexing using hybrid trees. In ICDE, 1999. 69 [4] M. Flickner, Harpreet Sawhney, Wayne Niblack, Jonathan Ashley, Q. Huang, Byron Dom, Monika Gorkani, Jim Hafine, Denis Lee, Dragutin Petkovic, David Steele, and Peter Yanker. Query by image and video content: The QBIC system. IEEE Computer, Sep 1995. 68 [5] Robert M. Haralick, K. Shanmugam, and Its’hak Dinstein. Texture features for image classification. IEEE Trans. on Sys., Man, and Cyb., SMC-3(6), 1973. 73 [6] Thomas S. Huang, Sharad Mehrotra, and Kannan Ramchandran. Multimedia analysis and retrieval system (MARS) project. In Annual Clinic on Library Application of Data Processing - Digital Image Access and Retrieval, 1996. 68 [7] Yoshiharu Ishikawa, Ravishankar Subramanya, and Christos Faloutsos. Mindreader: Querying databases through multiple examples. In VLDB, 1998. 68, 69 [8] Michael Ortega, Yong Rui, Kaushik Chakrabarti, Sharad Mehrotra, and Thomas S. Huang. Supporting similarity queries in MARS. In ACM Multimedia, 1997. 73 [9] A. Pentland, R.W. Picard, and S. Sclaroff. Photobook: Content-based manipulation of image databases. Int’l Journal of Computer Vision, 18(3), 1996. 68 [10] Kriengkrai Porkaew, Sharad Mehrotra, and Michael Ortega. Query reformulation for content based multimedia retrieval in MARS. In IEEE Int’l Conf. on Multimedia Computing and Systems, 1999. 68, 69, 70, 74 [11] Yong Rui, Thomas S. Huang, and Sharad Mehrotra. Content-based image retrieval with relevance feedback in MARS. In IEEE Int’l Conf. on Image Proc., 1997. 69 [12] Yong Rui, Thomas S. Huang, Michael Ortega, and Sharad Mehrotra. Relevance feedback: A power tool for interactive content-based image retrieval. IEEE Trans. on Circuits and Video Technology, Sep 1998. 68, 69 [13] Markus Stricker and Markus Orengo. Similarity of color images. In SPIE Conf. on Vis. Commun. and Image Proc., 1995. 73 [14] Michael Swain and Dana Ballard. Color indexing. Int’l Journal of Computer Vision, 7(1), 1991. 73 [15] D. White and R. Jain. Similarity indexing with the ss-tree. In ICDE, 1995. 69

Excluding Specified Colors from Image Queries Using a Multidimensional Query Space Dimitrios Androutsos1 , Kostas N. Plataniotis2, and Anastasios N. Venetsanopoulos1 1

University of Toronto Department of Electrical & Computer Engineering Digital Signal & Image Processing Lab 10 King’s College Road, Toronto, Ontario, M5S 3G4, CANADA {zeus,anv}@dsp.toronto.edu WWW:http://www.dsp.toronto.edu 2 Ryerson Polytechnic University Department of Math, Physics & Computer Science 350 Victoria Street, Toronto, Ontario, M5B 2K3, CANADA [email protected]

Abstract. Retrieving images in a database based on user specified colors is a popular low-level retrieval technique. However, the available systems today do not easily allow for a user or a specified query to tag certain colors as unwanted in the query result to ultimately be excluded in the query. Specifically, color histogram techniques do not allow for a direct approach to excluding colors and would require a separate query stage to filter out images containing unwanted colors. In this paper we present our vector-based scheme to image retrieval using a multidimensional query space which naturally accepts the exclusion of specified colors in the overall similarity measure.

1

Introduction

Color image retrieval has received increasing attention lately as the field of image database retrieval grows. It’s importance stems from the fact that color is a lowlevel image feature which is essential to the early stages of human vision. Color is easily recalled and identified and a natural attribute for describing objects and scenes. For these reasons, image retrieval researchers have been trying to find efficient and effective ways to retrieve color images from large databases using color in the query definition [1]. To this end, color indices are created using color histograms to capture the color representation of all the database images [2,3]. Using these indices, a user can retrieve images from the database by building a query by specifying certain colors which they want the retrieved images to contain or by specifying an example image which the retrieved images should match. There are a number of image retrieval systems which employ these techniques and there is much ongoing research in the area [4,5]. Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL’99, LNCS 1614, pp. 76–82, 1999. c Springer-Verlag Berlin Heidelberg 1999


Excluding Specified Colors from Image Queries

77

However, these systems do not address a very important issue in color retrieval namely, color exclusion. It is important for users to be able to tag a certain color or group of colors as unwanted so that they do not appear in the retrieval results. With present systems, specifying which colors to exclude would require an additional stage to filter retrieved images and modify their ranking according to whether or not an exclusion color is present. In this paper we describe how our system addresses this issue by virtue of using a Multidimensional Query Space which incorporates the exclusion of any unwanted colors directly into the image similarity measure, without requiring an extra processing stage.

2

System Description

Utilizing color histograms for indexing and retrieval has gained much popularity. However, there are inherent problems with this technique, which reduce the flexibility and accuracy of the query process and results. In particular, color histograms capture global color activity. Attempts to include spatial information by image partitioning has had some success but storage and computational requirements increase accordingly. In addition, the similarity metrics which are commonly accepted and utilized allow little flexibility and have no valid perceptual basis. We have developed a system which is color vector-based. We do not use histograms to build indices. Instead, we store representative RGB color vectors from extracted color regions, along with spatial color information to build an index of smaller dimension and with more information than a simple color histogram. In this section, we give a brief overview of our system, specifically how the feature extraction is done via segmentation and we also present the distance measure which we use to perform similarity matching. 2.1

Feature Extraction & Indexing

Our feature extraction is based on unsupervised recursive color segmentation. Specifically, we perform HSV-space segmentation while taking into consideration certain perceptual attributes of human color perception and recall. The HSVspace classifies similar colors under similar hue orientations and thus provides a more natural grouping. In addition, it allows for automated segmentation since it allows for fast and efficient automation. It is not dependent on variables, such as seed pixels or number of extracted colors, such as in clustering techniques. The details of our segmentation technique can be found in [8]. However, it is important to note that we: – extract bright colors first – extract and classify white and black regions – treat the saturation histogram as multi-modal instead of bi-modal

78

Dimitrios Androutsos et al.

For each image we extract c colors, which is an image dependent quantity. We calculate the average color of each of the c colors and use that RGB value as each region’s representative vector. These c colors, along with spatial information such as size and location of each region, are used to build each image index. 2.2

Similarity Measure

Since our color indices are actual 3-dimensional color vectors which span the RGB space, a number of vector distance measures can be implemented for retrieval. However, we implement a measure which is based on the angle of a color vector. Angular measures are chromaticity-based, which means that they operate primarily on the orientation of the color vector in the RGB space and therefore are more resistant to intensity changes and it has been found that they provide much more accurate retrieval results than other measures [7]. Specifically, our similarity measure is a perceptually-tuned combination of the angle between two vectors and a magnitude difference part, defined as [9]:       · x − x | x |x 2 i j i j ) 1− √ ) , (1) β(xi , xj ) = exp −α(1 − 1 − cos−1 ( π |xi ||xj | 3 · 2552       angle magnitude where xi and xj are 3-dimensional color vectors, α is a design parameter and √ and 3 · 2552 are normalization factors.

3

2 π

Image Query

During the query process, for each user-specified query color, a similarity measure is calculated using (1), to each representative color vector in a given database index. For each query color, the minimum distance is kept and a multidimensional measure is created which consists of the minimum distances of the query colors to the indexed representative vectors in the given index:

D(d , . . . , d ) = I − (min(β(q , i ), . . . , β(q , i min(β(q , i ), . . . , β(q , i ))), 1

n

1

n

1

1

1

n

m

m )), . . . ,

(2)

where I is a vector of size n with all entries of value 1, q n are the n query colors and im are the m indexed representative color vectors for a given image. 3.1

Multidimensional Query Space

The vector D in (3) exists in a vector space defined by the similarity measure of the specified query colors to the indexed colors. The dimensionality of this space changes and is dependent on the number of query colors. We refer to this space as the multidimensional query space.

Excluding Specified Colors from Image Queries

79

The database image that is the closest match to all the given query colors q1 , q2 , . . . , qn is the one which is closest to the origin of the multidimensional query space. Within this query space, there is a line on which all components of D are equal. We refer to this line as the equidistant line. A distance vector D that is most centrally located, i.e, is collinear with the equidistant line and at the same time has the smallest magnitude, corresponds to the image which contains the best match to all the query colors, as depicted in Figure 1(a). For each query, each database image exists at a point in this multidimensional query space. It’s location and relation to the origin and equidistant line determines it’s retrieval ranking which we quantify by taking a weighted sum of the magnitude of D and the angle,  D to the equidistant line: R = w1 |D| + w2  D,

(3)

where lower rank values R imply images with a closer match to all the query colors. The weights w1 and w2 can be adjusted to control which of the two parameters, i.e., magnitude or angle, are to dominate. We have found that values of w1 = 0.8 and w2 = 0.2 give the most robust results. This is to be expected since collinearity with the equidistant line does not necessarily imply a match with any query color. It implies that each query color is equally close (or far) to the indexed colors. However, as |D| → 0, implies closer matches to one or more colors. Thus, a greater emphasis must be placed on the magnitude component.

q1

equidistant line q1

equidistant line

∆

D

Ξ D

x1

q

2 q2

(a)

(b)

Fig. 1. (a) Vector representation of 2 query colors q1 &q2 , their multidimensional distance vector D and the corresponding equidistant line. (b) the same 2 query colors,1 exclusion color, x1 and the resulting multidimensional distance vector ∆.

80

4

Dimitrios Androutsos et al.

Color Exclusion

Our proposed vector approach provides a framework which easily accepts exclusion in the query process. It allows for image queries containing any number of colors to be excluded in addition to including colors in the retrieval results. From the discussion in Section 3.1 above, we are interested in distance vectors D which are collinear with the equidistant line and which have small magnitude. The exclusion of a certain color should thus affect D accordingly and it’s relation to the equidistant line and the origin. For example, if it is found that an image contains an indexed color which is close to an exclusion color, the distance between the two can be used to either pull or push D closer or further to the ideal and accordingly affect the retrieval ranking of the given image, as shown in Figure 1(b). To this end, we determine the minimum distances of each exclusion color with the indexed representative colors, using (1), to quantify how close the indexed colors are to the exclusion colors:

X (x , . . . , x ) = (min(β( , i ), . . . , δ( , i 1

n

1

1

1

 i

 i

m )), . . . min(β( n , 1 ), . . . , δ( n , m )))

(4)

where ξn are the n exclusion colors and im are the m indexed representative colors of each database image. Equation (4) quantifies how similar any indexed colors are to the exclusion colors. To quantify dissimilarity, a transformation of each vector component of X is required, and then this is merged with D to give a new overall multidimensional distance vector:


= [D I − X ],

(5)

where I is a vector of size n with all entries of value 1. The dimensionality of ∆ is equal to the number of query colors + number of exclusion colors. The final retrieval rankings are then determined from |∆| and the angle which D in (5) makes with the equidistant line of the query color space (i.e., the space without excluded colors). We performed an example query from our database of 1850 natural images, both with exclusion and without. Figure 2(a) depicts the query result when R,G,B colors (26, 153, 33) (green) and (200, 7, 25) (red) were desired and the color (255, 240, 20) (yellow) was excluded. It can be seen that images which contained colors close to yellow were removed from the top ranking results, as compared to Figure 2(b), where yellow was not excluded. We further investigated these exclusion results by determining by how much the retrieval ranking of the images which contained yellow changed. A trained user was asked to look at the top 40 retrieval results for the query of red and green, and determine which of these images contained yellow. This first step resulted with a set of 25 images, which we refer to as X , that contained the exclusion color. The retrieval ranking of each of the images in X was then calculated when the same query also excluded yellow. It was found that none of the images in X remained among the top 40 retrieval results. Furthermore, their ranking decreased significantly and all 25 images were now ranked among the bottom 27% of the entire 1850 image database, i.e., among the 500 least similar images.

Excluding Specified Colors from Image Queries

81

(a)

(b)

Fig. 2. Query result for images with (a) red & green, excluding yellow and (b) red & green, not excluding yellow.

8 images remained in the top 40 retrieval results, which contained red and green and 7 images had their ranking slightly decreased for containing colors that were perceptually close to yellow. The flexibility of this technique allows any number of colors to be excluded in a given color query and can also be incorporated in query-by-example, where a seed image is fed as a query. Furthermore, the amount by which X of (4) affects D can be varied by a simple weighting to tune the strictness of the exclusion.

5

Conclusions

We have shown how easily and effectively our system addresses the concept of color exclusion in a color image query. It is incorporated into the overall similarity calculation of each candidate image in a given query and does not require a post-processing stage to filter out images which contain a color to be

82

Dimitrios Androutsos et al.

excluded. This is accomplished by virtue of the multidimensional query space which the distance measures of the query vectors span and their relation to the equidistant line. The similarity of specified exclusion colors to indexed database colors affects the overall ranking by effectively lowering the rank of a given image which contains color that should be excluded. In our system, any number of colors can be excluded in a given query to provide greater flexibility in how a user query is defined, to ultimately retrieve more valid images from a given database.

References 1. V. N. Gudivada and V. V. Raghavan, “Content-based image retrieval systems,” Computer 28, September 1995. 76 2. M. J. Swain and D. H. Ballard, “Color indexing,” International Journal of Computer Vision 7(1), 1991. 76 3. M. Stricker and M. Orengo, “Similarity of color images,” in Storage and Retrieval for Image and Video Databases III, Proc. SPIE 2420, pp. 381–392, 1995. 76 4. W. Niblack, R. Barber, W. Equitz, M. Flickner, Glasman, D. Petkovic, P. Yanker, C. Faloutsos, and G. Taubin, “The qbic project: Querying images by content using color, texture and shape,” in Storage and Retrieval for Image and Video Databases, M. H. Loew, ed., Proc. SPIE 1908, 1993. 77 5. J. R. Smith and S. F. Chang, “Visualseek: a fully automated content-based image query system,” in ACM Multimedia Conference, November 1996. 77 6. X. Wan and C.-C. J. Kuo, “Color distribution analysis and quantization for image retrieval,” in Storage and Retrieval for Image and Video Databases IV, Proc. SPIE 2670, pp. 8–16, 1995. 7. D. Androutsos, K.N. Plataniotis and A.N. Venetsanopoulos, “Distance Measures for Color Image Retrieval,” International Conference on Image Processing ’98, Chicago, USA, October 1998. 78 8. D. Androutsos, K.N. Plataniotis, A.N. Venetsanopoulos, “A Vector Angular Distance Measure for Indexing and Retrieval of Color,” Storage & Retrieval for Image and Video Databases VII, San Jose, USA, January 26-29, 1998. 77 9. D. Androutsos, K.N. Plataniotis, A.N. Venetsanopoulos, “A Perceptually Motivated Method for Indexing and Retrieval of Color Images,” International Conference on Multimedia Computing Systems 1999, Florence, Italy, June 7-11, 1999. Submitted. 78

Generic Viewer Interaction Semantics for Dynamic Virtual Video Synthesis Craig A. Lindley1 and Anne-Marie Vercoustre2 1

CSIRO Mathematical and Information Sciences Locked Bag 17, North Ryde NSW 2113, Australia Phone: +61-2-9325-3150, Fax: +61-2-9325-3101 [email protected] 2

INRIA-Rocquencourt, France [email protected]

Abstract. The FRAMES project is developing a system for video database search, content-based retrieval, and virtual video program synthesis. For dynamic synthesis applications, a video program is specified at a high level using a virtual video prescription. The prescription is a document specifying the video structure, including specifications for generating associative chains of video components. Association specifications are sent to an association engine during video synthesis. User selection of a virtual video prescription together with the default behavior of the prescription interpreter and the association engine define a tree structured search of specifications, queries, and video data components. This tree structure supports generic user interaction functions that either modify the traversal path across this tree structure, or modify the actual tree structure dynamically during video synthesis.

Introduction

The FRAMES project is developing a system for video database search, content-based retrieval, and virtual video program synthesis. The FRAMES project has been carried out within the Cooperative Research Centre for Advanced Computational Systems established under the Australian Government's Cooperative Research Centres Program. Video components within the FRAMES database are described in terms of a multi-layered model of film semantics, derived from film semiotics. For dynamic video program synthesis applications, a program is specified at a high level using a virtual video prescription (Lindley and Vercoustre, 1998a). Coherent sequences of video are required, rather than just lists of material satisfying a common description. To meet this requirement, the FRAMES system uses an engine for generating associative chains of video sequences, initiated by an initial specification embedded within a virtual video prescription. Once a virtual video prescription has been Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL'99, LNCS 1614, pp.83-90, 1999.  Springer-Verlag Berlin Heidelberg 1999

84

Craig A. Lindley and Anne-Marie Vercoustre

selected, the prescription interpreter and associated instruction processing functions can be allowed to generate a virtual video with no further interaction from the viewer. In this case the resulting presentation has the form of a traditional linear film or video. However, depending upon the viewer’s overall purpose, it may be desirable to steer the ongoing presentation in various ways. For example, the user may wish to steer the presentation towards subjects of interest and away from those of less interest, gain an overview of the area, go into detail, or follow a particular mood or emotion. This paper defines generic user interaction semantics for dynamic virtual video synthesis based upon the data structures and sequencing functions of the FRAMES system. The semantics provide run-time interactions for the viewers of a virtual video; the interactions do not result in any permanent changes to the data structures involved, but affect the way those data structures are used to generate a particular video presentation. We begin with a summary of FRAMES system users and user tasks, provide an overview of the FRAMES system, and summarise the processes that are used to select video components during the generation of a synthesised video sequence. The high level algorithm used within the FRAMES association engine is described, and is seen to define a tree-structured search through the available video components. User interaction semantics are then analysed in terms of generic user interaction strategies, the default data structure that models the selection action of the synthesis engine, and generic interaction operations that can be defined in terms of their effect upon the implied data structure.

FRAMES System Users and User Tasks The FRAMES video synthesis process implies four different author/system user roles that may be involved in the production and use of a virtual video. Within the FRAMES system, video data is a primitive (atomic) data input, organised as a set of discrete video sequences. The video maker may use a variety of software tools and products to create these digital video clips. Interactive video systems that support interaction within a complete video program represent a new medium requiring customised development of video data. The FRAMES video synthesis engine operates upon descriptions associated with raw video data. Hence once the video data is available, a description author must develop a descriptor set and associate descriptors with appropriate video data sequences. The FRAMES environment includes data modeling interfaces to support this authoring process. The interfaces and underlying database are based upon the semiotic model described by Lindley and Srinivasan (1998). Once the descriptions have been created, they are stored in the FRAMES database for use by the video synthesis engine.

Generic Viewer Interaction Semantics for Dynamic Virtual Video Synthesis

85

The FRAMES system can be used with these semantic descriptions to provide basic semantic search and retrieval services, where a user can directly interrogate the database using relational parametric queries, or interrogate the database via the FRAMES association engine either to conduct fuzzy parametric searches, or to generate an associative chain of video components. However, for many users and applications a specific high level program structure may be required. Such a structure can be defined using a virtual video prescription. A prescription, defined by a virtual video prescription author, contains a sequence of embedded queries for generating the low level video content, where the particular order, form, and content of the queries implements a specific type, genre and style of video production. The final end user/viewer community is the audience for whom the virtual video production is created. Such a user will typically select a virtual video prescription according to their current tasks and needs, and use the FRAMES virtual video synthesis engine to generate a virtual video presentation. For dynamic virtual video synthesis, there are a number of ways and points in the process where viewer interaction is meaningful. All viewer interaction functions may be available to the authors of the interaction system, to provide feedback to authors about the appropriateness and effectiveness of descriptions and prescriptions as they are being developed. The authoring process for interactive virtual videos is highly complex, and requires careful coordination between the video makers, description authors, and prescription authors to ensure that these three levels of content are compatible and function correctly to produce coherent viewer sequences. Understanding the principles for doing this effectively is an important topic of ongoing research.

The FRAMES Video Synthesis System

The FRAMES system consists of three primary elements: a virtual video prescription interpreter, a database containing semantic descriptions of individual video components, and the instruction engines for generating sequences of video data. A virtual video prescription represents a high level structure of, or template for, a video program of a particular type, containing a list of instructions for generating a virtual video production (Lindley and Vercoustre, 1998a). The virtual video interpreter reads virtual video prescriptions. A user may select a prescription, which may have values assigned to various embedded parameters to reflect the particular requirements and interests of that user before being forwarded to the interpreter. The interpreter reads the instructions within a prescription sequentially, routing each instruction in turn to an appropriate processor. Three types of instructions may occur within a prescription: direct references to explicitly identified video components, parametric database queries, and specifications for generating an associative chain of video components (Lindley, 1998). Access by direct reference uses an explicit, hard-coded reference to a video data file plus start and end offsets of the required segment (eg. using the referencing syntax of SMIL, Hoschka 1998). Parametric database queries may

86

Craig A. Lindley and Anne-Marie Vercoustre

include complex logical conditions or descriptor patterns. In parametric search, the initial query may form a hard constraint upon the material that is returned, such that all of its conditions must be satisfied. Alternatively, a ranked parametric search can return a list of items ranked in decreasing order of match to the initial query, down to some specified threshold. Access by associative chaining is a less constrained way of accessing video data, where material may be incorporated on the basis of its degree of match to an initial search specification, and then incrementally to successive component descriptions in the associative chain. Associative chaining starts with specific parameters that are progressively substituted as the chain develops. At each step of associative chaining, the video component selected for presentation at the next step is the component having descriptors that most match the association specification when parameterised using values from the descriptors attached to the video segment presented at the current step. The high-level algorithm for associative chaining is: 1. initialise the current state description according to the associative chaining specification. The current state description includes: • the specification of object, attribute, and entity types that will be matched in the chaining process, • current values for those types (including NULL values when initial values are not explicitly given or components of the next instantiation are NULL), • conditions and constraints upon the types and values of a condition, and • weights indicating the significance of particular statements in a specification 2. Generate a ranked list of video sequences matching the current state description. 3. Replace the current state description with the most highly ranked matching description: this becomes the new current state description. 4. Output the associated video sequence identification for the new current state description to the media server. 5. If further matches can be made and the termination condition (specified as a play length, number of items, or associative weight threshold) is not yet satisfied, go back to step 2. 6. End. Since association is conducted progressively against descriptors associated with each successive video component, paths may evolve significantly away from the content descriptions that match the initial specification. This algorithm (described in detail in Lindley and Vercoustre, 1998b) has been implemented in the current FRAMES demonstrator. Specific filmic structures and forms can be generated in FRAMES by using particular description structures, association criteria and constraints. In this way the sequencing mechanisms remain generic, with emphasis shifting to the authoring of metamodels, interpretations, and specifications for the creation of specific types of dynamic virtual video productions.

Generic Viewer Interaction Semantics for Dynamic Virtual Video Synthesis

87

Generic Interaction Strategies User interaction in the context of dynamic virtual video synthesis can take place at several levels, and in relation to several broad types of user task. Canter et al (described in McAleese, 1989) distinguish five discernible strategies that users may use in moving through an information space: 1. 2. 3. 4. 5.

scanning: covering a large area without depth browsing: following a path until a goal is achieved searching: striving to find an explicit goal exploring: finding out the extent of the information given wandering: purposeless and unstructured globetrotting

These strategies are all relevant to interaction with dynamic virtual video synthesis, and the interactive presentation system for virtual videos should support each strategy. To these five strategies we can also add: 6. viewing: allowing the algorithm to generate a video sequence without further direction from a user (ie. the viewer is passively watching a video) Dynamic virtual video syntheses in the FRAMES project uses the viewing model as the default behavior of the system. That is, once a virtual video prescription has been selected, the synthesiser generates the video display based upon that prescription and the semantics defined by the underlying algorithms. The virtual video prescription may define a video program amounting to a scan, browse, search, exploration of, or wander through the underlying video database, depending upon the applicationspecific purpose of the prescription. To provide interactive viewing functions, suitable interfaces must be provided allowing viewers to modify the behavior of the video synthesis engine away from this default behavior within the form defined by the original virtual video prescription.

User Interaction Semantics A prescription can be customised for a particular user by setting its parameter values. Parametric search may be an exact search mechanism (eg. if a traditional relational database is used), or may involve a fuzzy search process that returns identifiers of video component having descriptors that approximately match the search query, ranked in decreasing order of match to the query. A video synthesis system incorporating ranked search can include interfaces allowing users to select from the ranked list of returned results. Associative chaining can be modified in several ways by user interactions, by using user interactions to effectively modify the chaining specification dynamically as chaining proceeds. Users can modify the entity types used to associate the current component with the next component, modify the current

88

Craig A. Lindley and Anne-Marie Vercoustre

entity values, set or reset constraints upon entity values, or modify the weightings upon entity types. Users can also interrupt the default selection of the most highly associated video component by selecting another ranked element as the current element, which will re-parameterise the associative chaining specification at the current point in the chain.

S1

Prescription P1 instr 1 instr 2 S2 . . . Prescription

Fig. 1.

Associative Chain Components: C1 C2 . . . Cp

Ranked components not selected:

S3

C1,1 C1,2 . . . C1,q

S4

Prescription

The semantics of these user interactions can be modeled by regarding the operation of the association engine as a tree search behaviour, as shown on Figure 1. In considering choices that can be made by users, it is useful to regard the starting point as the decision about which virtual prescription to execute, this being the root node of the search tree. Each prescription contains a list of instructions that constitute its child nodes. The algorithm that interprets prescriptions will execute each instructions in sequential order. An instruction (specifically, an instruction that is an association specification) generates a series of video components that are its child nodes in turn, each component being presented for display in the sequential order in which it is returned. Finally, for each selected video component in a series, there is a list of other candidate components that have not been selected, ranked in decreasing order of associative strength (to the previous component in the selected list); this ranked list may be considered to be a set of child nodes for a selected component. Hence the video synthesis process defines an ordered, depth-first traversal of the system data structures and the dynamically generated association structure of video components. The default behavior of the synthesis engine without user interaction satisfies the user interaction strategy identified above as viewing. However, to support scanning, browsing, searching, exploring, and wandering strategies, specific and generic interaction functions can be provided. These are divided into two classes. The first class of interaction functions are those that determine the path taken by the user in traversing the default synthesis tree amount to functions that interrupt or modify the default depth-first traversal behavior of the algorithm. These functions include: • control of whether the process should stop, loop back to some point (eg. as identified on a history list), or proceed to the next default item

Generic Viewer Interaction Semantics for Dynamic Virtual Video Synthesis

89

• jump to a position on the tree other than the next position defined by the depth-first algorithm • display a set of video components in parallel The second class of interaction functions are those that dynamically alter the structure of the default tree during video synthesis are functions that effectively produce an alteration in the specification that is driving the generation of a virtual video production. This can include: • functions that dynamically modify virtual video prescriptions (eg. changing the values of variables used within a prescription during execution) • functions that dynamically modify queries prior to their execution, or as they are being executed. Examples include adding or removing descriptor types that associative matching is taking place against, and modifying the weightings attached to descriptor types.

Related Work Interactive video covers a broad range of technologies and interests, including interactive video editing systems, model-based video image generation, and interactive search and browsing of video data in archives or databases. The FRAMES project is addressing the interactive use of predefined video sequences. Dynamic access to predefined video using content-based retrieval techniques has generally been based upon an information retrieval model in which data is generated in response to a single query (eg. the IBM QBIC system, http:// wwwqbic.almaden.ibm.com/ stage/ index.html); sequencing from this perspective is a contextual task within which content-based retrieval may take place. The MOVI project has incorporated some automated video analysis techniques into an interactive video environment that then uses hard-coded links between video elements (see http:// www.inrialpes.fr/ movi/ Demos/ DemoPascal/ videoclic.html). Unlike these approaches, FRAMES generates links between video sequences dynamically using an associative chaining approach similar to that of the Automatist storytelling system developed at MIT (Davenport and Murtaugh, 1995, and Murtaugh, 1996). The Automatist system uses simple keyword descriptors specified by authors and associated with relatively self-contained video segments. In Automatist, users can interact with the associative chaining process either by explicitly modifying the influence of specific keyword descriptors arranged around the periphery of the interface, or by selecting a less strongly associated video component to become the current displayed component determining the ongoing associative chain. The FRAMES system extends this associative chaining approach by using a highly structured semantic model (described in Lindley and Srinivasan, 1998), which allows greater discrimination on descriptor types, and more types of relationship between sequenced video components. Flexible and modifiable association specifications in FRAMES and the incorporation of direct references and parametric queries in high

90

Craig A. Lindley and Anne-Marie Vercoustre

level prescriptions create opportunities for interaction beyond the simple selection of keywords and ranked components.

Conclusion This paper has presented an analysis of the underlying semantics of user interaction in the context of the FRAMES dynamic virtual video sequence synthesis algorithms. Ongoing research is addressing the presentation of interaction options to users, and the problem of disorientation within the unfolding interactive video.

References Aigrain P., Zhang H., and Petkovic D. 1996 “Content-Based Representation and Retrieval of Visual Media: A State-of-the-Art Review”, Multimedia Tools and Applications 3, 179-202, Klewer Academic Publishers, The Netherlands. Davenport G. and Murtaugh M. 1995 “ConText: Towards the Evolving Documentary” Proceedings, ACM Multimedia, San Francisco, California, Nov. 5-11. Hoschka P.(ed) 1998, “Synchronised Multimedia Integration Language (SMIL) 1.0 Specification” W3C Recommendation 15 June 1998. Lindley C. A. 1998 “The FRAMES Processing Model for the Synthesis of Dynamic Virtual Video Sequences”, Second International Workshop on Query Processing in Multimedia Information Systems (QPMIDS) August 26-27th 1998 in conjunction with 9th International Conference DEXA98 Vienna, Austria. Lindley C. A. and Srinivasan U. 1998 “Query Semantics for Content-Based Retrieval of Video Data: An Empirical Investigation”, Storage and Retrieval Issues in Image- and Multimedia Databases, August 24-28, in conjunction with 9th International Conference DEXA98 Vienna, Austria. Lindley C. A. & Vercoustre A. M. 1998a “Intelligent Video Synthesis Using Virtual Video Prescriptions”, Proceedings, International Conference on Computational Intelligence and Multimedia Applications, Churchill, Victoria, 9-11 Feb., 661-666. Lindley C. A. & Vercoustre A. M. 1998b “A Specification Language for Dynamic Virtual Video Sequence Generation”, International Symposium on Audio, Video, Image Processing and Intelligent Applications, 17-21 August, Baden-Baden, Germany. McAleese R. 1989 “Navigation and Browsing in Hypertext” in Hypertext theory into practice, R. McAleese ed., Ablex Publishing Corp., 6-44. Murtaugh M. 1996 The Automatist Storytelling System, Masters Thesis, MIT Media Lab, Massachusetts Institute of Technology.

Category Oriented Analysis for Visual Data Mining H. Shiohara, Y. Iizuka, T. Maruyama, and S. Isobe NTT, Cyber Solutions Laboratories 1-1 Hikarinooka, Yokosuka-shi, Kanagawa, 239 JAPAN TEL: +81 468 59 3701, FAX: +81 468 59 2332 {shiohara,iizuka,maruyama,isobe}@dq.isl.ntt.co.jp

Abstract. Enterprises are now storing large amount of a data and data warehousing and data mining are gaining a great deal of attention for identifying effective business strategies. Data mining extracts effective patterns and rules from data warehouses automatically. Although various approaches have been attempted, we focus on visual data mining support to harness the perceptual and cognitive capabilities of the human user. The proposed visual data mining support system visualizes data using the rules or information induced by data mining algorithms. It helps users to acquire information. Although existing systems can extract data characteristics only from the complete data set, this paper proposes a category oriented analysis approach that can detect the features of the data of associated with one or more particular categories.

1

Introduction

The great evolution in computing power has enabled businesses to collect and store copious amounts of data. As competition between enterprises intensifies, it is more important for the business strategy to be based on real data. Data mining has thus attracted attention as a way of obtaining such knowledge. Data mining can extract rules from copious amounts of data or classify data by using algorithms established in the field of artificial intelligence. Although it’s suitable for handling copious amounts of data, its algorithms are very difficult to use if the user is not familiar with data analysis. We developed a visual data mining support system that combines data mining algorithms with visualization for better usability. Because effective visualization is needed to help users to discover rules/patterns, the selection of the attribute(s) to be the visualization target is very important. In our system, attribute selection is performed automatically by utilizing data mining methods. The 3 main functions of our system are as follows. 1. extract data characteristics by applying data mining 2. select the effective attributes to be visualized based on the extracted characteristics Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL’99, LNCS 1614, pp. 91–98, 1999. c Springer-Verlag Berlin Heidelberg 1999


92

H. Shiohara et al.

3. create visual representations effectively by mapping the selected attributes to parameters of a plot profile The resulting visualization makes it easier for the user to put forward hypotheses. Existing systems apply data mining only to all data which can lead to the significant characteristics of partial data sets being overlooked. Accordingly, we add category analysis to the existing method to create a more powerful visual data mining support system. This article overviews the visual data mining support method in section 2, and section 3 introduces the new attribute selection method, the category oriented selection method. The remaining sections describe effective visualization offered by this method and some application examples.

2

Visual Data Mining Support Method

The visual data mining support method should help the user in discovering rules/patterns in voluminous data sets easily by combining powerful data mining algorithms with user-friendly visualization techniques. Human beings can well grasp data trends if they are well visualized. Furthermore, when they have background knowledge, they can understand what the trends mean in the real world. If the data set is too large, it is not easy even to guess which attributes are correlated which degrades analysis performance. Therefore, we focused on the selection of the attributes to be visualized and adapted data mining algorithm to support visual analysis. 2.1

Multi-dimensional Visualization System

As the visualization environment, we developed INFOVISER, a multi-dimensional visual analysis tool. INFOVISER transfers character-based data into graphical information and visualizes one record of data as one element such as a node or line. Each attribute of data is mapped to one parameter of a plot profile such as axes, size, color, shape, and so on. This yields multi-dimensional and powerful visualization that is impossible with ordinary bar or circular charts. The degree of multi-dimensionality that can be visualized at once is about 10, and business data have many more attributes. This makes it difficult to manually find the attributes that are correlated. 2.2

Visualized Attribute Selection Method

As described before, the key to visual data mining support is how to find the effective target to be visualized. Selection of a group of attributes that are strongly correlated can lead to the discovery of new rules/patterns. For this purpose, the support system extracts characteristics from data, and selects target attributes based on these characteristics. The user evaluates the visualization results and finds a clue to the next analysis step or, may change

Category Oriented Analysis for Visual Data Mining

93

the extraction process to generate another visualization result. This integration of machine and human analysis is the key to our proposal. Existing attribute selection methods use decision trees or correlation coefficients for extracting the data characteristics of all data. The decision tree method is effective especially when there is an analysis target attribute. The attributes, listed in the tree created by the decision tree algorithm, are selected as visualization targets in order of hierarchy from the root node. When there is no analysis target or no clue, a correlation coefficient matrix is effective. The pairs of attributes having higher correlation coefficients are selected as visualization targets. These methods can overcome the difficulty of setting hypotheses due to excessive attribute number. The system configuration is depicted in Fig.1

Fig. 1. System Configuration of Visual Data Mining Support System

3

Category Oriented Attribute Selection Method

3.1

Requirements

The following are examples of visualization results manually generated using INFOVISER ’s GUI. sumo wrestler a high rank wrestler have good balance of height and weight.(Fig.2) medical checkup 1 a heavy drinker has high γGTP value and high blood sugar level.(Fig.3)

Fig.2. sumo wrestler

Fig.3. medical checkup 1

Fig.4. medical checkup 2

94

H. Shiohara et al.

medical checkup 2 in the case of a light drinker, the person’s obesity is inversely proportional to how much he smokes.(Fig.4) In these examples, the attributes that seem to influence the attribute that attracted the user’s interest like rank of wrestler and obesity, are visualized using particular patterns. However, these attributes were not selected by the existing selection method. Fig.5 depicts the decision tree for medical checkup data whose target is “Drinking”. Because attributes like γGTP or blood sugar or smoke or obesity don’t have high rank, they aren’t visualized. Even the correlation coefficient method didn’t select these attributes because of their small absolute values. ( Table 1) The reason seems to be that these characteristics are seen only Fig. 5. Result of Decision Tree as part of the data and are lost when extraction is applied uniformly to the whole set. Human beings can detect distinguishing patterns in the visualized figures by their pattern recognition ability and induce rules from their background knowledge. For selecting attributes like above examples automatically, it is necessary to detect data subset including prominent patterns. In order achieve this, we propose to combine data categorization with the attribute selection process. That is, grouping data according to some guideline, characterizing each data group, and extracting attributes that have remarkable patterns compare to the complete data set. This should satisfy the following requirements. TREE Root |--TREE NODE 1 | if Sex in { 2 } | |--IF Systolic Pressure <= 157 | | THEN Drinking = 2.375 | +--IF Systolic Pressure > 157 | THEN Drinking = 1 +--if Sex not in { 2 } |--if Cholesterol <= 222.5 | |--if Diastolic Pressure <= 68 | | |--IF Age <= 45 | | | THEN Drinking = 3 | | +--IF Age > 45 | | THEN Drinking = 1 | +--IF Diastolic Pressure > 68 | THEN Drinking = 3.04762 +-IF Cholesterol > 222.5 THEN Drinking = 3.95238

1. able to reflect user’s interest in data categorization 2. able to evaluate partial characteristics for attribute selection 3.2

Data Categorization

The direct way to reflect the user’s interest is to make the analysis target attribute a guideline for categorization. The attributes that have special values within in a certain category, can be considered as those that are correlated to the user’s interest. As for the user’s interest, there are several cases. · there is only one target attribute Table 1. Correlation Coefficient Attribute Correlation Coefficient

Age Height Weight Systolic pressure 0.47 0.32 0.07

-1.37

Diastolic Choles pressure -terol 0.18 0.16

Blood sugar 0.13

COT GPT γGTP Obesity degree Smoke 0.21 0.20 0.24 0.20 0.47

Category Oriented Analysis for Visual Data Mining

95

· there are multiple target attributes · there is no target attribute ( target attribute is not clear) With just one analysis target attribute, categorization is achieved by dividing the data into groups by setting discrimination levels or simply into an equal number. When there are multiple target attributes, such a naive method is not desirable. Categorization that takes account of the correlation of the attributes like multi-dimensional clustering is more suitable. How then can we categorize data if the user does not select any target attributes ? In this case, we use factor analysis ( a statistical technique ) to divide the data attributes into target attributes and explanation attributes (or dependent variables and independent variables) at first. 3.3

Attribute Selection

This section discusses how to evaluate, for attribute selection, the characteristics extracted from each category. As for the correlation coefficient method, a pair of attributes that shows a different trend from the remaining data is taken as the characteristic, rather than just considering high values within one category. That is, a pair of attributes that have low correlation value in the whole data set may show high correlation in a certain category, or show an inverse correlation. We formalize these behaviors into the following expression to evaluate how characteristic the category is. 1 f1 (rA , rp , np ) = (1 − √ )rp (rp − rA ) np np is the number of partial data, rA is the correlation of all data, rp is the correlation of partial data set. In the same way, data distribution can be used for characterizing categories. That is, locating attributes whose distribution is much different from those of the whole data set using basic statistical values such as average and standard deviation. We use the following expression to compare and evaluate the strength of the characteristics created by the categories and attributes. 1 mp − mA f2 (np , mp , sp , mA ) = (1 − √ ) np sp np is the number of partial data, mp is the average of partial data set, mA is average of all data, sp is standard deviation of partial data. Attributes are selected in order of the evaluation of characteristics.

4 4.1

Visualization Method Scatter Chart Representation

In this category oriented analysis method, the visualization target attributes are classified into two groups that are mapped as follows.

96

H. Shiohara et al.

Fig.6 Visualization Result 1

Fig.7 Visualization Result 2

· attributes that categorize data: targeted attributes → color, shape · attributes that characterize data: explanative attributes → X, Y-axis In this method, from which category the characteristics are extracted is significant information, as is the attributes themselves, so we represent the categorizing attributes by color, category by shape, and the extracted attributes as the X-axis and Y-axis. 4.2

Aggregation Representation

When a large amount of data is visualized, figures overlap because the display area is not infinite. This overlapping causes not only visual confusion but also a loss of information. In order to avoid this problem, the number of figures visualized on screen is reduced. One way is to thin the number of data records, but it is possible the remaining data doesn’t retain the original trend. Another way is to summarize neighboring data and this is more desirable from the viewpoint of analysis. It is common to combine figures that are close in ordinary scatter charts. In INFOVISER, however, profiles such as color, shape, size, have meaning. So, if these profiles are ignored and only position is considered when figures are summarized, the data profile information is lost and can’t be evaluated. Therefore, we suppose a virtual space where in all profiles are treated equally, and summarize by distance in this space. This enables visualization by fewer figures without losing the significance of the trends of all data.

5

Application Examples

This section shows results of applying the category oriented methods and the visualization method to test data consisting of medical checkup data. (3600 records with 24 attributes). In this article, only cases of one target attribute and no attribute are shown.

Category Oriented Analysis for Visual Data Mining

97

Case of one target attribute We selected “smoking” as the target attribute. The result of categorization by equal division of value into 6 groups was evaluated using equation f1 and f2 . The following features were extracted. Correlation coefficient matrix: results · as smoking rate increases, the correlation of drinking and uric acid becomes stronger. · as smoking rate increases, the inverse correlation of obesity and HDL cholesterol(good cholesterol) increases. Basic statistics: results · heavy smokers have lower HDL cholesterol and more neutral fat. · light smokers have lower γGTP value ( means healthy hepatic function). Medically it is said that HDL cholesterol reduces as the smoking rate increases, or uric acid increases as the rate of drinking increases. In this result, when the degree of smoking or obesity is large, this phenomenon is seen strongly. Fig.6 and 7 show visualization results. ( Count as size, more smoke as more dark color, and category as shape) Case of no target attribute The top 4 results of factor analysis of the test data are shown in Table 2. By performing categorization using multi-dimensional clustering, and using attributes highly correlated with the first factor, the following features were extracted. Correlation coefficient matrix: results · in the highest factor scoring group (cluster 6), the correlation of total cholesterol and GPT(hepatic index) is high and meal frequency is inversely proportional to the drinking rate. · in the high factor scoring group (cluster 1), the rate of eating before sleep is inversely correlated to meal frequency. Basic statistics: results · in the highest factor scoring group(cluster 6) heavy drinking is common, smoking rate is high, and exercise and sleeping hours are small. · in a high factor scoring group (cluster 1), smoking rate is high. In this case, the height of factor scoring is interpreted as an index of poor healthy.

6

Discussion

We ascertained the effective attributes and visualization results were obtained by applying the proposed method to test data with enough records. A shortcoming is that the visualization result may not be so comprehensible even if the numerical value is significant. That is due to the relative low value of the correlation coefficient or a small difference in distribution. The existing method can generate very plain visualizations. One of the examples generated by the

98

H. Shiohara et al.

existing method indicates that systolic pressure and diastolic pressure are almost proportional and both are highly correlated with obesity as shown in Fig8. However, the proposed method can identify conspicuous characteristics.

7

Conclusion

This article proposed a category oriented analysis method that can detect the strength of the characteristics of different categories, and confirmed that it effectively supports visual data mining. In the future, we will examine a user interpretation support function and other characterization methods.

References 1. K.Kurokawa, S.Isobe, H.Shiohara, “Information Visualization Environment for Character-based Database Systems” VISUAL ’96, pages 38-47, Feb. 1996. 2. Y.Iizuka, et al., “Automatic Visualization Method for Visual Data Mining”, Lecture Notes in Artificial Intelligence Vol.1394, PAKDD-98, pp.174-185, Apr. 1998. 3. B.H.MacCormik, T. A. DeFanti and M.D.Brown, eds., “Visualization in Scientific Computing,” Computer Graphics, Vol.21, No.6, ACM Siggraph, Nov. 1987. 4. A.S.Jacobson, A.L.Berkin and M.N.Orton, “Linkwinds: Interactive Scientific Data Analysis and Visualization”, Communications of the ACM, Vol.37, No.4, Apr.1994. 5. U. M. Fayyad and E. Simoudis, “Knowledge Discovery in Databases”, Tutorial Notes, 14th International Joint Conference on Artificial Intelligence (IJCAI-95), 1995. 6. U. M. Fayyad, G. Piatetsky-Shapiro, P. Smyth, and R. Uthurusamy, “Advances in Knowledge Discovery and Data Mining”, AAAI/MIT Press, 1995. 7. D. A. Keim, “Database and Visualization”, Tutorial Notes, ACM-SIGMOD’96, 1996.

User Interaction in Region-Based Color Image Segmentation Nicolaos Ikonomakis1, Kostas N. Plataniotis2 , and Anastasios N. Venetsanopoulos1 1

Department of Electrical & Computer Engineering Digital Signal & Image Processing Lab University of Toronto 10 King’s College Road, Toronto, Ontario, M5S 3G4, Canada {minoas,anv}@dsp.toronto.edu WWW:http://www.dsp.toronto.edu 2 School of Computer Science Ryerson Polytechnic University 350 Victoria Street, Toronto, Ontario, M5B 2K3, Canada [email protected]

Abstract. An interactive color image segmentation technique is presented for use in applications where the segmented regions correspond to meaningful objects, such as for image retrieval. The proposed technique utilizes the perceptual HSI (hue,saturation,intensity) color space. The scheme incorporates user interaction so that the best possible results can be achieved. Interaction with the user allows the segmentation algorithm to start efficiently and to refine the results. Interaction is performed for the key images (usually the first image or those where new ojects enter the scene) of the video-telephony sequence. The user is allowed to identify on the screen the relevant regions by marking their seeds. The user guidance can be given by the sender, by the receiver, or by both. The effectiveness of the algorithm is found to be much improved over techniques used in the past.

1

Introduction

Image segmentation refers to partitioning an image into different regions that are homogeneous or ”similar” in some image characteristic. It is an important first task of any image analysis process, because all subsequent tasks, such as feature extraction and object recognition, rely heavily on the quality of the segmentation. Image segmentation has taken a central place in numerous applications, including, but not limited to, multimedia databases, color image and video transmission over the Internet, digital broadcasting, interactive TV, video-ondemand, computer-based training, distance education, video-conferencing and

Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL’99, LNCS 1614, pp. 99–106, 1999. c Springer-Verlag Berlin Heidelberg 1999


100

Nicolaos Ikonomakis et al.

tele-medicine. For some specific applications, it is possible to find an automated analysis process that performs segmentation as desired. This may be the case of a surveillance application, where the automated analysis system provides the video encoder detailed information about the object of importance, allowing the selective coding of the scene. A simple, real-time, and automated analysis process based on the detection of moving objects may be used. However, for many applications (multimedia databases, web-based search engines, video-conferencing, tele-medicine, etc.), fully automated analysis schemes provide only part of the desired analysis results [1,2]. The performance of an automatic segmentation scheme on complex video scenes (i.e. lots of background objects) may not produce the desired results. For these applications, user interaction is imperative so that the achieved results can have a meaningful and powerful semantic value. For this reason, more recent research is given to interactive and “human in the loop” systems [1,2,3,4,5]. The QBIC (Query By Image Content) team [5] uses interactive region segmentation for image retrieval purposes. In this paper, an interactive color image segmentation scheme is proposed that employs the perceptual HSI (Hue, Saturation, Intensity) color space to segment color images. More specifically, the proposed scheme is developed for implementation in applications where the segmented regions should correspond to meaningful objects, such as image retrieval or video-telephony type sequences. The region-based segmentation scheme first employs an initial user interaction seed determination technique to find seed pixels to be used in a region growing algorithm. Initial user interaction also includes selecting several values for threshold parameters used in the region growing algorithm. Following the automatic growing of the regions a supervised region merging algorithm is employed to refine the results of the segmentation. The next section explains the segmentation scheme. This is followed by the results and conclusions.

2

Color Image Segmentation

The segmentation scheme presented utilizes the HSI color space, and thus, the color values of the pixel are first converted from the standard RGB (red, green, blue) color values to the HSI color values using well known transformation formulas [6]. The scheme can be split into four general steps: 1. The pixels in the image are classified as chromatic or achromatic pixels by examining their HSI color values. 2. User classifies seed pixels in image. 3. The region growing algorithm is employed on the chromatic and achromatic pixels separately starting from the seed pixels. 4. Regions are merged through user interaction. The region growing algorithm has been presented in the past [7], but with arbitrary unsupervised seed determination. The automatic method gave good

User Interaction in Region-Based Color Image Segmentation

101

results but still needed improvement. Automatic seed determination is one of the most difficult problems in color image segmentation [6]. A good seed pixel is the pixel with the most dominant color and is usually the center pixel of the region. Thus, to determine them, an initial segmentation of the image is needed to find the regions. The new seed determination method presented in this paper constitutes an initial user interaction process. Because the human visual system can segment an image automatically with little or no hesitation [8], the user can achieve the initial segmentation and, thus, determine the best starting pixels. Each step in the above generalization is explained in the following sections. Due to limitation in space, experimental results for the Claire video-telephony type image will only be discussed. 2.1

Chromatic/Achromatic Separation of Pixels

The HSI color model corresponds closely to the human perception of color [6]. The hue value of the pixel has the greatest discrimination power among the three values because it is independent of any intensity attribute. Even though hue is the most useful attribute, there are two problems in using this color value: hue is meaningless when the intensity is very low or very high; and hue is unstable when the saturation is very low [6]. Because of these attributes, in the proposed scheme, the image is first divided into chromatic and achromatic regions by defining effective ranges of hue, saturation, and intensity values. Since the hue value of a pixel is meaningless when the intensity is very low or very high the achromatic pixels in the image are defined as the pixels that have low or high intensity values. Pixels can also be categorized as achromatic if their saturation value is very low, since hue is unstable for low saturation values. From the concepts discussed above, the achromatic pixels in the HSI color space are defined as follows: achromatic pixels:(intensity>90) or (intensity<10) or (saturation<10),

(1)

where the saturation and intensity values are normalized from 0 to 100. Only the intensity values of the achromatic pixels are considered when segmenting the achromatic pixels into regions. Pixels that do not fall into this category are categorized as chromatic pixels. For chromatic pixels all three HSI color values are considered in the algorithm. 2.2

Seed Determination

The region growing algorithm starts with a set of seed pixels and from these grows regions by appending to each seed pixel those neighboring pixels that satisfy a certain homogeneity criterion, which will be described later. An interactive algorithm is used to find the ’best’ seed pixels in the image. The ’best’ seed pixels are defined as the pixels that are in the center of their respective object. A benefit of having an initial user interaction is that different levels of

102

Nicolaos Ikonomakis et al.

segmentation can be employed. More specifically, the user can choose two different sets of seed pixels. One for regions that require a low degree of segmentation (i.e. facial areas), and one for regions that require a high degree of segmentation (i.e. background, clothing, etc.). This is beneficial, because areas where some detail is important can be segmented separately from areas where detail is not essential. The user is first asked to select low level seed pixels from the original image. After this, the user is asked to select the high level seeds. It was found that having only two levels makes it quicker and easier for the user to choose the seed pixels. For classical unsupervised seed determination algorithms, this two level seed determination option is not an easy benefit to implement. 2.3

Region Growing

The region growing algorithm starts with the set of seed pixels determined from the user and from these grows regions by appending to each seed pixel those neighboring pixels that satisfy a certain homogeneity criterion. The algorithm is summarized in the Figure 1. The first seed pixel is compared to its 8-connected neighbors: eight neighbors of the seed pixel. Any of the neighboring pixels that satisfy a homogeneity criterion are assigned to the first region. This neighbor comparison step is repeated for every new pixel assigned to the first region until the region is completely bounded by the edge of the image or by pixels that do not satisfy the criterion. The color of each pixel in the first region is changed to the average color of the all the pixels in the region. The process is repeated for the next and each of the remaining seed pixels. If all of the seed pixels are grown and there are still unassigned pixels in the image further region growing segmentation is employed by considering random seed pixels amongst the unassigned pixels. For the chromatic pixels, the homogeneity criterion used for comparing the seed pixel and the unassigned pixel is that if the value of the distance metric used to compare the unassigned pixel and the seed pixel is less than a threshold value Tchrom than the pixel is assigned to the region. Varying the value of Tchrom controls the degree of segmentation with a low value resorting to an over-segmented image and a high value to an under-segmented image. With the two different levels of seeds found, two different values of Tchrom are used. The seeds that require a low degree of segmentation have a threshold lower than the seeds that require a high degree of segmentation. Besides setting the seed pixels, the user initially sets the values of the low and high level thresholds. If the region growing segmentation results with these thresholds are not perfect the user can refine the threshold values, during the segmentation process, until optimal results are obtained. The distance measure used for the chromatic pixels will be referred to as the cylindrical metric. It computes the distance between the projections of the pixel points on a chromatic plane. It is defined as follows[9]: 1
(2) dcyl (s, i) = (dI )2 + (dC )2 2 , with dI = |Is − Ii |

(3)

User Interaction in Region-Based Color Image Segmentation

103

Select next seed pixel

neighbors are 8 neighbors of seed pixel

compare neighbors to the seed pixel with homogeneity criterion

if any neighbor pixels compared satisfy the homogeneity condition

satisfying pixels are assigned to the region and are the new neighbors

TRUE

FALSE

Fig. 1. The region growing algorithm. and


1 dC = (Ss )2 + (Si )2 − 2Ss Si cos θ 2 ,

where

 θ=

|Hs − Hi | if 360◦ − |Hs − Hi | if

|Hs − Hi | < 180◦ , |Hs − Hi | > 180◦ .

(4)

(5)

The value of dC is the distance between the 2-dimensional (hue and saturation) vectors, on the chromatic plane, of the seed pixel and the pixel under consideration, as shown in Figure 2. Henceforth, dC combines both the hue and saturation (chromatic) components of the color. An examination of the metric equation (Equation (2)) shows that it can be considered as a form of the popular Euclidean distance (L2 norm) metric. A pixel is assigned to a region if the value of the metric dcyl is less than a threshold Tchrom . In the case of the achromatic pixels, the homogeneity criterion used is that if the difference in the intensity values between an unassigned pixel and the seed pixel is less than a threshold value Tachrom than the pixel is assigned to the seed pixel’s region. That is, if (6) |Is − Ii | < Tachrom, then pixel i would be assign to the region of seed pixel s. 2.4

Region Merging

Following the region growing algorithm for image segmentation a user refinement process is employed. The user supervises the results of the region growing

104

Nicolaos Ikonomakis et al.

R

Si

G

pixel under consideration θ Ss

seed pixel

B

Fig. 2. The chromatic plane of the HSI color model. algorithm, correcting the undesired deviations when needed. The user does this by telling the algorithm which regions in the segmented image to merge into one. This further segments the image into less regions. The user specifies how many regions to be merged and then specifies the regions. The color of the regions that are merged is then changed to the average color of the regions. This is defined as: r k=1 ck Nk (7) average color =  r k=1 Nk where r, ck , and Nk are the number of regions being merged, the color vector of region k, and the number of pixels in region k, respectively. This ensures that if a small region is merged with a larger one the average color will be heavily weighted by the larger region.

3

Results and Conclusions

The performance of the interactive color image segmentation scheme was tested with the video-telephony type image Claire. For this type of image, segmentation can be used for the coding and/or compressions of the sequences before transmission. The original Claire image is displayed in Figure 3(a). Interaction with the user allows the automatic analysis process to start efficiently, and to refine the produced results. Interaction is performed for the key images (usually the first image or those where new ojects enter the scene) of the video-telephony sequence. The user is allowed to identify on the screen the relevant regions by marking their seeds. The user guidance can be given by the sender, by the receiver, or by both. Considering the achromatic pixels, it was found that the best results were obtained with a Tachrom threshold value of 15. This is 15% of the maximum achromatic distance. This was the the value used throughout the experimental analysis. Figure 3(b) shows an example of the seed pixels a user may choose for the Claire image. The white pixels refer to the high threshold level seed pixels and the black pixels refer to the low threshold level seed pixels.

User Interaction in Region-Based Color Image Segmentation

(a)

(b)

(c)

(d)

105

(e)

Fig. 3. Claire Image. (a) Original (b) Image showing seeds (white:high level, black:low level) (c) Segmented image before interactive merging (d) Classical segmented image with random seed determination (e) Segmented image after interactive merging.

106

Nicolaos Ikonomakis et al.

Figure 3(c) shows the segmented images, before merging, of Claire with Tchromlow = 10 (5 % of maximum distance) and Tchromhigh = 30 (15% of maximum distance). It was found that these threshold values give the best results for a varied set of images. As can be seen, the facial areas which incorporate a lot of detail are grown through several low level seeds. The background, the jacket, and the hair in the Claire image are segmented using the high level seeds. User refinement is incorporated to merge together similar regions. Figure 3(e) shows the segmented images after merging. As can be seen, the images have less regions. Figure 3(d) shows the segmented images of Claire after segmenting them with the classical method of segmentation with random arbitrary seed determination and no merging. These results are clearly not as good as the one segmented with the interactive segmentation scheme. A color image segmentation scheme using the HSI color space was used to segment video-telephony type images. It incorporated an interactive seed determination algorithm The technique was found to be robust and relatively computationally inexpensive. The effectiveness of the algorithm was found to be much improved over the technique used in the past [7]. Other than video-telephony type images, the proposed interactive scheme can be employed in other applications. These include: web-based search engines, multimedia databases, digital broadcasting, interactive TV, video-on-demand, computer-based training, distance education, and tele-medicine.

References 1. Soltanian, Z.H., Windham, J.P., Robbins, L.,: Semi-supervised segmentation of MRI stroke studies. Proc. of the SPIE vol. 3034 (1997) 437–448. 100 2. Olabarriaga, S.D., Smeulders, W.M.,: Setting the Mind for Intelligent Interactive Segmentation: Overview, Requirements, and Framework. Proc. of the 15th Int. Conf. on Information Processing in Medical Imaging (1997) 417–422. 100 3. Castagno, R., Ebrahimi, T., Kunt, M.,: Video Segmentation Based on Multiple Features for Interactive Multimedia Applications. IEEE Transactions on Circuits and Systems for Video Technology (Sept 1998) vol. 8, no. 5 562–571. 100 4. Correia, P., Pereira, F.,: User Interaction in Content-based Video Coding and Indexing. EUSIPCO III (1998) 2293–2296 . 100 5. Niblack, W., Barber, R., and et al.: The QBIC project: Querying images by content using color, texture and shape. Proc. SPIE Storage and Retrieval for Image and Video Databases (1996). 100 6. Gonzales, R.C., Wood, R.E.,: Digital Image Processing. Addison-Wesley, Massachusetts (1992). 100, 101 7. Ikonomakis, N., Plataniotis, K.N., Venetsanopoulos, A.N.,: A Region-based Color Image Segmentation Scheme. Proc. of the SPIE: Visual Communications and Image Processing vol. 3653 (1999) 1202–1209. 100, 105 8. Marr, D.,: Vision. Freeman, San Fransisco, California (1982). 101 9. Tseng, D.C., Chang, C.H.,: Color Segmentation Using Perceptual Attributes. Proc. 11th Int. Conf. on Pattern Recognition III:Conf c. (1992) 228–231. 102

Using a Relevance Feedback Mechanism to Improve Content-Based Image Retrieval G. Ciocca and R. Schettini Istituto Tecnologie Informatiche Multimediali Consiglio Nazionale delle Ricerche Via Ampere 56, 20131 Milano, Italy {ciocca,centaura}@itim.mi.cnr.it

Abstract. the paper describes a new relevance feedback mechanism that evaluates the distribution of the features of images judged relevant or not relevant by the user, and dynamically updates both the similarity measure and query in order to accurately represent the user's particular information needs. Experimental results are reported to demonstrate the effectiveness of this mechanism.

1. Introduction Content-based retrieval systems operate on image databases to extract relevant images in response to a visual query. Unfortunately, the concept of "relevance" is frequently associated with the semantics of the image, and the encoding and exploitation of semantic information in a general purpose retrieval system is still an unsolved issue. However in practice low-level visual features are often correlated with the semantic contents [1, 8], and image retrieval techniques exploiting these have become a promising research issue [3, 9, 13]. General purpose systems of this type are already available [6, 16]. These systems usually make it possible to extract image descriptions in terms of color, texture, shape and layout features from the images and define the relative search/matching functions that can be used to retrieve those of interest. The performance of an image retrieval system is, consequently, closely related to the nature and quality of the features used to represent image content, but it is not limited to this. Another important issue is that the measure adopted to quantify image similarity is user- and task- dependent [3, 10], and this dependence is not in general understood well enough to permit careful, a-priori selection of the optimal measure. In this paper we present a mechanism that allows the user to query the database and progressively refine the system’s response by indicating the relevance, or irrelevance of the retrieved items. Our approach differs from other recently presented studies [11, 15, 18] in both the strategy for learning the query representing the information needs and the strategy for learning the similarity function. The relevance feedback mechanism presented is a part of a Visual Information Retrieval system currently under development. A full description of all the functionalities of this system is beyond the scope of the paper. We should like to provide here only Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL'99, LNCS 1614, pp.107 -114, 1999.  Springer-Verlag Berlin Heidelberg 1999

108

G. Ciocca and R. Schettini

some experimental evidence that our retrieval mechanism makes it possible to approximate the user's information needs in a great variety of applications.

2. Image Indexing The absolute performance of an image retrieval system is closely linked with to the nature and the quality of the features used to represent the image content. The features used to index the images in the experiments reported here were calculated on the global image and on 5 sub-images and are: • A Color Coherence Vector (CCV) in the CIELAB color space quantized in 64 colors (pixels in regions whose size exceeds 1% of the image size are counted as coherent pixels) [13]; • A histogram of the transition in colors (CIELAB color space quantized in 11 colors) [7]; • Moments of inertia of the distribution of colors in the unquantized CIELAB color space [17]; • A histogram of contour directions opportunely filtered (only high gradient pixels are considered) and using box widths of 15 [4, 9]; • The mean and variance of the absolute values of the coefficients of the subimages of the first three levels of the multi-resolution wavelet transform of the luminance image [4]; • The neighborhood Gray-Tone Difference Matrix (NGTDM), i.e. coarseness, contrast, busyness, complexity, and strength, as proposed by Amadasum et al. [2]; • The spatial composition of the color regions identified by the process of quantization in 11 colors [4].

3. Relevance Feedback The sub-vectors of features are indicated by Xihs, where i is the vector index, h is the index of the feature, and s is the index of the sub-image to which that feature refers. We indicate with Dhs the distance associated with the feature h-th of region s. In our experiment all the features are compared with the L1 distance measure, as it is statistically more robust than the L2 distance measure [14]. The global metric used to evaluate the similarity between two images of the database is, in general, a linear combination of the distances between the individual features: q

p

i j Sim(X i , X j ) = ∑∑ w hs D hs ( X hs , X hs )

(1)

s =1 h =1

in which the whs are weights. There are two problems in this formulation of image similarity. First, since the single distances may be defined on intervals of widely varying values, they must be normalized to a common interval to place equal

Using a Relevance Feedback Mechanism

109

emphasis on every feature score. Second, the weights must often be heuristically set by the user, and this may be rather difficult to do, as there may be no clear relationship between the features used to index the image database and those evaluated by the user in a subjective image similarity evaluation. To cope with the first problem we use Guassian normalization as follows [12, 15]: i j  D ( X i , X j ) − µ11 D hs ( X hs D ( X i , X j ) − µ pq  , X hs ) − µ hs D ( X , X ) =  11 11 11 , ,..., pq pq pq  K σ 11 K σ hs K σ pq   i

T

j

(2)

Assuming that the features' distance distributions have a Gaussian distribution, it can be shown that there is a 68% probability that the feature values will lie within the range of [-1, 1] if K=1, and a 99% probability if K=3 [12]. As we can not assume apriori that the distances have will have Gaussian distributions, the following general relationship holds:   D − µ hs 1 ≤ 1 ≥ 1 − 2 P − 1 ≤ hs K σ K hs  

(3)

According to this relationship the probability that the distance will lie within the range of [-1,1] is of 89% if we set K at 3, and 94% if k is set at 4 [13]. The latter is the default value used in all our experiments. A simple additional shift moves the distances into the [0,1] range. Out-of-range values are mapped to the extreme values, so that they do not bias further processing. At this point our similarity function has the following form: q

p

Sim(X i , X j ) = ∑∑ w hs s =1 h =1

i Dhs ( X hs , X hsj ) + 1 q p j = ∑∑ w hs d hs ( X ihs , X hs ) 2 s =1 h =1

(4)

The algorithm must now determine the weights for the individual distances by a statistical analysis of the distance feature values of the images of the relevance-set, and these weights are used to accentuate, or diminish, the influence of a given feature in the overall evaluation of similarity [18]. In our experience in content-based retrieval relevant images are sometimes selected because they resemble the query image in just some pictorial features. Consequently, after an initial query, one retrieved image may be considered relevant because it has the same color as the query, and another be selected for its similarity in shape, although the two are actually quite different from each other. Letting the relevant set R+, d+hs be the set of normalized distances among the elements of R+, and σ hs+ be the corresponding variance, we similarly, let R- be the set of non-relevant images, and d-hs , the set of normalized distances among the elements of R-. We when consider the union between R+ and R-, and compute the corresponding distance sets d+-hs, letting d*hs be d+-hs \d+hs, * , the corresponding variance. The weight terms are defined as: and σ hs

110

G. Ciocca and R. Schettini

1  ε w +hs =   1 +  ε + σ hs

if R + < 3

(5)

otherwise

0  w = 1  *  ε + σ hs

if R + + R − < 3 or R − = 0

0 w hs =  +  w hs − w *hs

+ < w *hs if w hs

* hs

(6)

otherwise

(7)

otherwise

where ε is a positive constant (set at 0.01 in our experiments). When at least three examples are given, we can take into account negative examples in tuning the similarity weights. For any given feature the first term of whs is high when there is some form of agreement among the feature values of the relevant (positive) set, while the second term is high when there is a similarity between positive and negative examples.

4. Query Processing Query processing consists in modifying the feature vector of the query by taking into account the feature vectors of the images judged relevant by the user. Our approach is to let R+ be the set of relevant images the user has selected (including the original query), Q the average query and σ the corresponding standard deviation, we proceed as follows:

{

Yhs (j) = X ihs (j)

}

X ihs (j) − Qhs (j) ≤ 3σ hs (j)

~ Q hs (j) =

1 Yhs (j)

∑X

i hs Xihs (j)∈Yhs (j)

(j)

∀ h, s, i, and j

(8) (9)

~ that can better represent the The query processing formulates a new query Q hs images of interest to the user, taking into account the features of the relevant images, without allowing one different feature value to bias query computation. The query process could similarly be applied to compute a query representing nonrelevant examples. But this seems of little practical interest: non-relevant examples are usually not similar to each other, and are, consequently, scattered throughout the feature space.

Using a Relevance Feedback Mechanism

111

5. Query Classification At the first iteration, when the user has selected just one image to be searched, all the weights in the similarity function (4) are set at the equal value of 1/ε. For faster tuning of the similarity function we can exploit previous query sessions performed by the user on the same database. To this end the user is allowed to have satisfactory queries memorized together with the corresponding weights in the similarity measure. When the user has already formulated a query "similar" to the new one, the algorithm sets the initial weights of the similarity function at the value of the former query reducing the time and effort needed to adapt ~the similarity measure by means of the k k relevance feedback algorithm. We let Qˆ = Q , w hs be the set of queries and the

{

}

corresponding weights. When a new query Qn is submitted, the system first evaluates its similarity with respect previous queries, using the corresponding weights in the similarity function; it then selects the closest one as follows: ~ Qk , w khs = argmin Q~ k ,w k

hs

q

ˆ ∈Q

p

∑∑ w

k hs

~ d hs (Qn , Qk )

(10)

s=1 h =1

The initial weights selected in Equation (17) are now set as follows:

w k(initial) hs

q p  ~ w khs d hs (Q n , Q k ) ∑∑  k s=1 h =1 w hs if ≤T  1 = pq ε  1 otherwise  ε

0 < T ≤1

(11)

Parameter T allows the user to tune the sensitivity. In our implementation the default is set at 0.1, that is, the new query is considered similar to an old one when the two differ less than the 10% of 1 ε ( pq) , the maximum distance value allowed.

6. Experimental Results and Discussion Users do not find it difficult to provide interactively examples of similar and dissimilar images. However if the image database queried is large and heterogeneous, or the retrieval task particularly complex, the user may find not enough examples of images that are actually very similar to the query in the first screens, and, to avoid the time-consuming visual browsing of the database, may mark as relevant images that are only partially similar. The user's information needs may also be rather vague, such as: find all the images containing people. In both cases images judged relevant may differ widely. Treating all these images in the same way - for example, averaging the features of the relevant images to compute a new query vector or updating the similarity measure [18] - may consequently produce very poor results, while

112

G. Ciocca and R. Schettini

processing all the relevant images as single queries and then combining the retrieval outputs may create an unacceptable computational burden when the database is large. Last but not least, relevant images may have some features, color for example, that are only casually similar. If the system is not able to identify these features and treat them differently, subsequent retrieval iterations will be biased. The key concept of the relevance feedback we have proposed is the statistical analysis of the feature distributions of the images the user has judged relevant, or not relevant, in order to understand what features have been taken into account (and to what extent) by the user in formulating this judgment, so that we can then accentuate the influence of these features in the overall evaluation of image similarity as well as in the formulation of a new query. The archives on which the algorithms were tested contain some 5,000 images: photographs of real landscapes, people and animals; reproductions of antique textiles of the Poldi Pezzoli Museum of Milan; a collection of paintings of the Accademia Carrara of Bergamo; and a collection of ceramics belonging to the Museo Internazionale della Ceramica of Faenza. To evaluate the effectiveness of our retrieval method we applied a measure proposed by Methre et al.[5]. We let S be the number of relevant items the user wanted to retrieve when posing a query; R Iq , the set of relevant images, and R qE , the set of relevant images retrieved in the short list. The effectiveness measure was defined as:  R qI ∩ R qE  if R qI ≤ S I (12)  R q ηS =  I E  Rq ∩ Rq if R qI > S  E  R q If R qI ≤S, the effectiveness was reduced to the traditional recall measure, while if

R Iq >S, the effectiveness corresponded to precision (In our implementation S was set at 24). The effectiveness of the algorithms was tested on the individual databases, and on a combination of them, to evaluate the system's capacity for adaptation. In Table I we have summarized the experimental results for twenty queries for each of the different databases considered. The queries have not been classified: the first iteration corresponds to a similarity measure in which all the features have the same importance. Each bin corresponds to a different database, and reports the average effectiveness value at each of the first three retrieval iterations. The results show that relevance feedback improves the effectiveness of the retrieval considerably for all the databases, and, in general, the second iteration (that is the first relevance feedback iteration) corresponds to the largest single improvement. There is, instead, little benefit in repeating the relevance feedback more than five or six times. It can reasonably be argued that this is due to the limited capability of the low-level features used to exhaustively describe image content, and not to the relevance feedback itself. The integration of other features and the exploitation of unsupervised image segmentation to focus automatically on the significant parts of the database images to

Using a Relevance Feedback Mechanism

113

be indexed should improve the results. This integration will be straightforward for us, as the structure of the relevance feedback mechanism is actually descriptionindependent, that is, the index can be modified, or extended to include other features, without requiring any change in the algorithm. However, as already pointed out by several authors, perceptually similar images are not necessarily similar in terms of low-level features. Therefore, we believe that only the integration of text-based image annotation will make it possible to further increase retrieval effectiveness in a significant way. Figure 1 represents an example of the system's application. Table I. Retrieval effectiveness 1 0,9 0,8 0,7 0,6 0,5 0,4 0,3 0,2 0,1 0

First iterarion Second iteration T hird iteration

Photos (1745 images)

Paintings Ceramic and (1768 images) ancient textiles (942 images)

Figure 1. The retrieval results after the second iteration of relevance feedback.

114

G. Ciocca and R. Schettini

Interested readers may find additional examples at the following address: http://www.test.itim.mi.cnr.it/sitoitim/schettini/relfeme.htm.

References 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18.

Aigrain O., Zhang H., Petkovic D.: Content-Based Representation and Retrieval of Visual Media: A State-of-the-Art Review. Multimedia Tools and Applications, 3 (1996) 179-182 Amadasun M., King R.: Textural features corresponding to textural properties, IEEE Transaction on System, Man and Cybernetics, 19 (1989) 1264-1274 Binaghi E., Gagliardi I., Schettini R.: Image retrieval using fuzzy evaluation of color similarity. International Journal of Pattern Recognition and Artificial Intelligence, 8 (1994) 945-968 Ciocca G., Gagliardi I., Schettini R.: Retrieving color images by content. In: Del Bimbo A., Schettini R. (eds.) Proc. of the Image and Video Content-Based Retrieval Workshop (1998) Desai Narasimhalu A., Kankanhalli M.S.: Benchmarking multimedia databases. Multimedia Tools and Applications, 4 (1997) 333-356. Faloutsos C., Barber R., Flickner M., Hafner J., Niblack W., Petrovic D. Efficient and effective querying by image content. Journal of Intelligent Systems, 3 (1994) 231-262 Gagliardi I., Schettini R.: A method for the automatic indexing of color images for effective image retrieval. The New Review of Hypermedia and Multimedia, 3 (1997) 201224 Gudivada V.N, Rahavan V.V.: Modeling and retrieving images by content. Information Processing and Management, 33 (1997) 427-452 Jain A.K., Vailaya A.: Image retrieval using color and shape. Pattern Recognition, 29 (1996) 1233-1244. Minka T., Picard R.W.: Interactive learning with a “Society of Models”. Pattern Recognition, 30 (1997) 565-581. Mitra M., Huang J., Kumar S.R.: Combining Supervised Learning with Color Correlograms for Content-Based Image Retrieval, Proc. of the Fifth ACM Multimedia 97 Conference (1997). Mood A.M, Graybill F.A., & Boes D.C: Introduzione alla statistica. McGraw-Hill (1988) Pass G., Zabih R., Miller J.: Comparing Images Using Color Coherence Vectors. Proc. Fourth ACM Multimedia 96 Conference (1996) P.J. Rousseeuw P.J., Leroy A.M.: Robust regression and outlier detection, John Wiley & Sons (1987). Rui Y, Huang T.S., Mehrotra S., Ortega M.: A Relevance feedback architecture in content-based multimedia information retrieval systems. Proc. of the IEEE Workshop on Content-based Access of Image and Video Libraries (1997) Smith J.R., Chang S-F.: VisualSEEK: a fully automated content-based image query. Proc. of the Fourth ACM Multimedia 96 Conference (1996) Stricker M., Orengo M.: Similarity of Color Images. Proc. of the SPIE Storage and Retrieval for Image and Video Databases III Conference (1995) Taycher L., La Cascia M., Sclaroff S.: Image Digestion and Relevance Feedback in the ImageRover WWW Search Engine, Proc. of the Visual1997 Conference (1997)

Region Queries without Segmentation for Image Retrieval by Content Jamal Malki, Nozha Boujemaa, Chahab Nastar, and Alexandre Winter INRIA Rocquencourt, BP 105, 78153 Le Chesnay, France [email protected]

Abstract. Content-based image retrieval is today ubiquitous in computer vision. Most systems use the query-by-example approach, performing queries such as "show me more images that look like this one''. Most often, the user is more specifically interested in specifying an object (or region) and in retrieving more images with similar objects (or regions), as opposed to similar images as a whole. This paper deals with that problem, called region querying. We suggest a method that uses a multiresolution quadtree representation of the images and thus avoids the hard problem of region segmentation. Several experimental results are presented in real-world databases.

1

Introduction

Surfimage is a Content-Based Image Retrieval (CBIR) system developed at INRIA since 1996. Its specificity is the capacity to deal with both categories of image databases: • For image databases with ground truth, the system should be as efficient as possible on the specific application. Examples include face recognition or medical image retrieval. A quantitative evaluation of the system can then be reported in terms of recognition rate, precision/recall graph, etc. • For image databases where no ground truth is available, the system should be flexible, since the notion of perceptual similarity is subjective and contextdependent. Smart browsing, query refinement, multiple queries, and partial search on user-defined regions are among the desirable features of the system. Applications include stock photography and the World Wide Web. Surfimage uses the query-by-example approach for retrieving images and integrates advanced features such as image signature combination, multiple queries, query refinement, and partial queries [1,2]. We focus on the latter problem in this paper. Indeed, the user is most often interested in performing a query on an object (or region), rather than on the whole image. The goal of the system is then to retrieve those images in the database that contain similar objects. This observation motivates recent research on spatially-localized features and region matching. Methods range Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL'99, LNCS 1614, pp.115 -122, 1999.  Springer-Verlag Berlin Heidelberg 1999

116

Jamal Malki et al.

from histogram computation without segmentation [3-5], approximate region segmentation [6-9], graphical and spatial structure of the image [10-12]. In this paper, we suggest a multiresolution quadtree approach for performing region queries. It represents a simple and efficient way to select image parts for retreival. Endeed, when manual object segmentation through all database images is too hard or when images are too complex for performing automatic segmentation, quadtrees image subdivision is an appropriate solution for partial queries. Image signatures are then computed systematically on each subimage. A dedicated similarity measure is computed, allowing to retrieve images with similar regions. The similarity measure defines the invariance properties of the query (e.g. "find similar objects anywhere in the image" vs. "find similar objects in the same image location"). Section 2 details feature computation in the multiresolution quadtrees. Section 3 discusses user queries and the associated inducted invariance. In section 4 we present comparative experimental results. We draw the conclusions in section 5.

2. Multiresolution Quadtrees Several problems occur with region (or partial) matching. The first one is segmentation: accurate segmentation of an image into regions in the general case is quite difficult. Another problem is invariance: is the user interested in finding other occurrences the object (or region) with the same position, orientation, and scale, or do they require invariance against these transformations? In order to deal with partial queries, we use a multiresolution quadtree representation (similar to [13]) that localizes image features in structured regions (fig. 1). This approach avoids image segmentation and offers effective alternatives for the invariance problem. A1 A2 A3 Fig. 1. Computation of image signatures on the multiresolution quadtree representation of the original image for 3 levels

Let A the original image and p the number of levels: For a given level i ( 1 ≤ i ≤ p ) A is divided into ni parts such that, for all levels, A is represented by a total number of N subimages:

( )

A = A ni i

ni

p

,

∑n i =1

i

= N,

n i = 4 i −1 , 1 ≤ i ≤ p

(1)

Region Queries without Segmentation for Image Retrieval by Content

117

2.1. Feature Computation

We use a large selection of features offered by the Surfimage system [1]. Examples include color, shape and texture, mostly captured via histograms. The presentation of these commonly used features is not the scope of this paper -for further details, see [14, 15] or [1]. The features are computed on each subimage of the quad tree representation, yielding higher dimensional feature vectors. If we consider the p-level representation, the following feature vector is computed: (2) h = hi , 1 ≤ i ≤ p , n = 4i −1

( ) ni

features

i

( )is defined as the feature h computed on the sub-image A . The set of (h )for a fixed i corresponds to the features of the i-th level of the

where each h

i ni

i ni

i ni

representation. For instance, if the original single feature h is a 64-bin histogram, the dimensionality of the 3-level quad-tree feature is (1+4+16)*64 = 1344. Eventually, note that, when using histograms, we normalize the feature vectors so that they sum up to 1 -they thus represent distributions of that feature in the corresponding sub-image. 2.2. Feature Combination

Combination of different features has been a recent focus of image retrieval [2]. The main problem is how do we combine ``apples and oranges'', i.e. features that have different number of components, different scales etc.? We have experimented with several combination methods: voting, gymnastics-rule, etc… [16]. Among these methods, one seems to be the most appropriate and is described in the following. Under Gaussian assumption, the normalized linear combination method uses the estimated mean µi and standard deviation σi of the distance measure d for each feature i, providing the normalized distance: d ' ( x (i ) , y (i) ) =

d ( x ( i ) , y ( i ) ) − ( µ i − 3σ i ) 6σ i

(3)

Where x(i) and y(i) are the vector signatures of images X and Y within feature i. The new distance measure d' will essentially have its values in [0...1]. The combined distance between X and Y is then: D (Q , X ) =

∑ ρ (d ' ( x i =1

where α = µ i − 3σ i 6σ i

3.

i

(i )

, y ( i ) )) −

∑ ρ ( −α

i

)

(4)

i

, ρ is an increasing function (e.g. ρ (x)=x, ρ (x)=x3 etc.).

User Queries

3.1. Performing a Partial Query

For performing a region query, the user has to specify the regions of interest (RoI) at each level. Note that the RoIs do not have to be connex (fig. 2). For a given level,

118

Jamal Malki et al.

we build the corresponding bounding box, defined as the smallest subimage containing the RoIs (see fig.2). This bounding box (Bb) is able to catch the relative geometric positions of the different RoIs, and will be used for retrieving images . Partial queries composed of RoIs Bounding box Image query Fig. 2. Bounding box of partial query at the third level In connection to the RoIs, the user has to specify which metric (fixed, location invariant) they are using. Note that most of the image signatures that we use are histograms, yielding most often rotation, translation and scale invariance within a subimage. The main invariances that the user can specify are thus: no location invariance (e.g. a face on top left, a hand on bottom left), no relative location encoding (a face and a hand anywhere in the image), location invariance (a face and a hand with specified relative positions). 3.2. Similarity Metric and Invariance Issues

The previous invariances have to be translated into similarity metrics (or equivalently, distances). We have dealt with the ”no invariance'' case in our previous work [16]. The ”loose relative location” case is an alternative which can be easily derived. We describe the location invariance hereafter; the method encodes the relative locations of the RoIs and is translation invariant (fig. 3).

Fig. 3. Authorized translations of partial queries bounding box at level 3 Location invariance is built in as follows. Let RQ the set of Nr partial queries selected

by the user at the level l: R Q =

Nr − 1

UQ

β =0

 Nr −1

Let M the Bb of   



U {Q } , i

i=0

lβ

{

= Q l 0 , K Q l NR − 1

}

Nt is the number of authorized translations. The



similarity measure between a query image Q and any image X in the database is given by:

d (Q , X ) =

Nt − 1

∑ d (M (Q ), M j=0

M j (X ) =

j

( X ))

Nr −1

UQ α

α j =0

l

is the sub-image obtained after a translation j. Note that: j

(5)

Region Queries without Segmentation for Image Retrieval by Content

d (M ( Q ), M j ( X ) ) =

Nr −1

Tm

∑ α∑ δ α β β =0

j

=0

j

(

d Q l β , X lα j

)

119

(6)

Where δ is the Kronecker symbol. 3.3. Redundancy

We note that the quadtree representation of image features is redundant, especially with respect to histogram computation. More precisely, a histogram feature computed at any node of the tree is equal to the sum of the histogram features of its 4 sons (regardless of the normalization procedure). The quadtree representation is thus redundant. However, experimentally, this redundancy is effective and useful: indeed, the lower levels tend to specify the holistic features of the image, thus allowing to restrict the search (e.g. I am looking for red roses, but in landscape scenes only). Note also that the contribution of each subimage to the global metric being equivalent, the lower levels participate less to the overall metric. Examples are shown in section 4. The user selects the first level to focus on an image category (e.g. a landscape), and specifies the query regions in a higher level.

4. Retrieval Results We present results on our homebrew bigdatabase (Fig.4), which was built by merging the MIT Vistex database of textures, the BTphoto database of city and country scenes, a homebrew paintings database, and the homeface database of people in the lab. The total number of images in bigdatabase is 3670. We use a kd-tree structure to optimize the search process. Note that when spatial relations between partial queries are not preserved, retrieval results are different from those displayed in figure 5, 6 and 7.

Fig4. A sample of bigdatabase consisting in 3670 heterogeneous images

120

Jamal Malki et al.

Fig.5 Retrieval of the top left image. Retrieved images are from top left to bottom right in order of best match. Query image at level 1 and partial queries at level 3 are also presented. We search for selections every where in the database images

Fig6: Second example of partial queries on bigdatabase

Region Queries without Segmentation for Image Retrieval by Content

121

Fig 7 Retrieved face images from bigdatabase with both level 1 and 3

Fig 8 Retrieved face images from bigdatabase with only level 1. System returns just face images among texture, landscape images and painting

5.

Conclusion

To address the issue of region matching, we avoided the difficult problem of image segmentation using a multiresolution quadtree representation of the image. Our experience shows that we need two levels: level 1 to pick image category and level 3

122

Jamal Malki et al.

to specify more precise details in a same category. This approach provides flexibility for general and heterogeneous image databases retrieval without using specialized features such for face retrieval as shown in fig.7 and fig.8. Various general features are computed and combined together on each subimage. Dedicated similarity metrics allow for inferring invariance properties -in particular location invariance. Experimental results show that the method is effective for local queries in image databases. They also emphasize the importance of preserving spatial organization of local features to achieve good retrieval results.

6.

References

1. Nastar C., Mitschke M., Meilhac C. and Boujemaa N.: Surfimage: a Flexible Content-Based Image Retrieval System", ACM Multimedia'98", Bristol, September, 1998 2. Nastar C., Mitschke M., Meilhac C.: Efficient Query Refinement for Image Retrieval, Computer Vision and Pattern Recognition (CVPR'98), Santa Barbara, June, 1998 3. Michaeal J. Swain and D. H. Ballard: Color indexing, IJCV,vol.7, n°1, pp.11-32, 1991 4. John R. Smith and Shih-Fu Chang,Tools and Techniques for Color Image Retrieval,Proceedings of Storage & Retrieval for Image and VideoDatabases I, San Jose, CA, USA, February, pp. 426-437, 1996, 5. Jing Huang and Ravi Kumar and Mandar Mitra and Wei-Jing Zhuand Ramin Zabih,Image Indexing Using Color Correlograms,CVPR, Puerto Rico, June, pp.762-768, 1997 6. Sergio D. Servetto and Yong Rui and Kannan Ramchandran and Thomas S. Huang,A: Region -Based Representation of Images in MARS, Special Issue on Multimedia Signal Processing, Journal on VLSI Signal Processing, Oct, 1998 7. Belongie S. and Carson C. and Greenspan H. and Malik J.: Color- and Texture-Based Image Segmentation Using EM and Its Application to Content-Based Image Retrieval", Proceedings of the Sixth International Conference on Computer Vision(ICCV'98), Bombay, January,1998 8.Cinque L., Lecca F., Levialdi S., and Tanimoto S. : Retrieval of images using rich image descriptions, Proceedings of the ICPR, 1998 9. Neill W. Campbell, William P. J. Mackeown and Barry T. Thomas and Tom Troscianko: Interpreting image databases by region classification,Pattern Recognition (Special Edition on Image Databases), vol. 30,n° 4, Apr,pp. 555--563, 1997 10. Simone Santini and Ramesh Jain: The Graphical Specification of Similarity Queries,Journal of Visual Languages and Computing,1997 11. Del Bimbo and Vicario E. : Using Weighted Spatial Relationships in Retrieval by Visual Contents, IEEE workshop on Image and Video Libraries",Santa Barbara,June,1998 12. Soffer A., Samet H. and Zotkin D. : Pictorial Query trees for query specification in image databases,Proceedings of the ICPR,1998 13. Vellaikal A. and Kuo C.: Joint Spatial-Spectral Indexing of JPEG Compressed Data for Image Retrieval, Int'l Conf. on Image Proc.", Lausanne, 1996 14. Pentland A., Picard R. and Sclaroff S. : Photobook: Tools for Content-Based Manipulation of Image Databases", Storage and Retrieval of Image and Video Databases II,vol. 2185, San Jose, 1994 15. Jain A. and Vailaya A. : Image Retrieval Using Color and Shape: Pattern Recognition,vol.29,n°8, 1996 16. Nastar C., Mitschke M., Meilhac C., Boujemaa N., Bernard H. and Mautref M.: Retrieving Images by Content: the Surfimage System, Multimedia Informations Systems'98,Istanbul, September,1998

Content-Based Image Retrieval over the Web Using Query by Sketch and Relevance Feedback E. Di Sciascio, G. Mingolla, and M. Mongiello Dip. Elettrotecnica ed Elettronica, Politecnico di Bari, Via E. Orabona 4, I-70125, Bari Italy Tel.+39(0)805460641 Fax +39(0)805460410 [email protected]

Abstract. This paper investigates the combined use of query by sketch and relevance feedback as techniques to ease user interaction and improve retrieval effectiveness in content-based image retrieval over the World Wide Web. To substantiate our ideas we implemented DrawSearch, a prototype image retrieval by content system that uses color, shape and texture to index and retrieve images. The system avails of Java applets for query by sketch and uses relevance feedback to allow users dynamically refine queries.

1 Introduction Content-based image retrieval (CBIR) [1-3] information systems use information extracted from the content of images for retrieval, and help the user retrieve images relevant to the contents of the query. A number of methodologies, techniques and tools, related to image content processing, have been studied for identification and comparison of image features in order to develop classification and retrieval systems based on (almost) automatic interpretation of image content. Complete image classification, indexing and retrieval based on the content interpretation require semantic interpretation and cannot be afforded with current technology. A surrogate of semantic interpretation is the computation of visual features that can be used as quantitative parameters for the identification of similar images. Thus, the problem of retrieving images with homogeneous content is substituted with the problem of retrieving images visually close to a target one. Several systems have been proposed in recent years in the framework of content-based retrieval [5-20]. With particular reference to papers related to query by sketch, the interested reader is also referred to QBIC [10,11], QVE [18] and a recent work on the use of elastic deformation of user sketches [19]. It must anyway be noticed that these works emphasize more the pattern matching problem than the retrieval by similarity one. Also noteworthy is the work in [20], which uses wavelet-based indexing and query by sketch for color images retrieval. Here the emphasis is in avoiding any user specification but the submitted query sketch. Approaches proposed in [5-9] introduce relevance feedback [4] as a distinguished aspect that can allow improving retrieval results using feedback provided by the user. Porting CBIR systems to the WWW is not a straightforward task. Largest part of current systems basically allow query by example letting the user Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL'99, LNCS 1614, pp. 123-130, 1999.  Springer-Verlag Berlin Heidelberg 1999

124

E. Di Sciascio et al.

browse into the collection until he/she finds an image visually similar to searched ones Once such an image has been found the user can pose a query and retrieve similar images. This process is cumbersome. As image collections grow in size this process may take a lot of time, and eventually reduce the query-retrieval process to trivial browsing. Furthermore the well-experienced problem of current Internet low speed makes the all process much tedious. We have investigated the possibility to increase user's interaction with image retrieval systems over the Internet using query by sketch and giving the user the ability to interact easily with the system by browsing the retrieved images and tuning the response through relevance feedback analysis. As a matter of fact, rather than putting much processing effort into answering a query in the sharpest way, in an image retrieval system it is reasonable to provide a fast reply with a rough retrieval method (yet offering non-trivial discrimination performance). Differently from text-based documents, browsing of a small retrieved set of images is fast and easy since the relevance of the content can be stated at a glance. To address these and other issues related to image information systems, we have started the DrawSearch [5] project with our group at Politecnico di Bari.

query by texture interface (Java applet)

user

query by shape and color interface (Java applet)

feature data archive

WWW

query server

feature extraction

retrieved set viewer (html) image archive

DrawSearch Client side

DrawSearch Server

side

Figure 1. DrawSearch system overview

2 System Overview In order to substantiate our ideas on the use of query by sketch and relevance feedback to increase user's interactivity and retrieval effectiveness in a web based environment, we designed and implemented a prototype system, which is accessible over the Internet at: http://deecom03.poliba.it/DrawSearch/DrawSearch-home.html. Figure 1 shows its main components. Our system extracts as relevant features, as other systems do, color, shape and texture. Two separate user interfaces are provided. The first one allows posing queries by sketch combining shape and color distribution; the second one allows queries by texture content. Reasons for having separate interfaces are various, most relevant for us was to avoid increasing the computational burden on the client side. We are

Content-Based Image Retrieval over the Web

125

anyway integrating relevant texture information in the retrieval stage of the color and shape system. Features extraction is a two-fold problem in our approach. Basically the features described here are extracted off-line for images during the database population stage. Hence, though always relevant, time performance is not critical when referred to this stage. On the other hand, when the features have to be extracted from the user's sketch, time, and hence algorithmic complexity, becomes a primary issue. Furthermore, while features computation on the server side can be done with efficient standard compiled languages, the client side has to rely on the much less efficient Java language. All extracted features share a representation based on the vector space model [4]. This model is based on the association of term vectors to documents, each vector representing a specific document by holding information about the index terms or keywords associated to it. Such information may appear simply as a set of present/not present flags, but more often it is a measure (weight) of the ability of each index term to discriminate the document within the collection. In our image retrieval model, feature vectors play almost the same role that term vectors play in text retrieval, holding normalized values of the image features as indexing information. However, differently from the vector space model, we do not weight the features against the whole image collection. The reason for this difference comes from the different user perception of image similarity with respect to text similarity, and to the different level at which the retrieved items are evaluated: visual for the images, semantic for the text. Image similarity is evaluated on visual properties, and may be verified at a glance. It is therefore independent from the image collection size and variety. The features are used to find similarities rather than to discriminate differences. On the other hand, text reading to verify the adequacy of the retrieved documents is a long process, therefore a text retrieval system must be provided with good discrimination capabilities, relying on the use of words to describe the document meaning and not its appearance. We believe that this model has an intrinsic strength due to its widespread use and its reliability. It also allows an extremely simple integration with text-based information systems. Hence we considered of primary importance feature evaluation schemes that allow a straightforward implementation in this model. Color feature is extracted computing the average values within 16 predefined blocks (in a 4 by 4 arrangement) the image is divided into. Computing the color feature in this way is obviously not very precise; but it allows, with limited computational effort, to represent the color distribution of the most immediately visible components like large objects or background, hence also providing a limited degree of spatial information. The adopted color model is RGB (Red, Green, Blue). Resulting data are normalized to a sum of 1 and arranged in a vector of 48 components. We are currently changing our color model to the HSB one. Anyway results reported in this paper still refer to color distribution computed using RGB. Shape feature extraction in real images requires image segmentation. Although a number of segmentation algorithms have been proposed in the literature, this issue is far from being solved. Most problems come from the ill-posed nature of the edge detection problem. The problem becomes harder when we need segmentation into semantically coherent regions, i.e. objects. We adopted a contour-based segmentation algorithm, which uses as ancillary information color and texture.

126

E. Di Sciascio et al.

a)

b)

c) Figure 2. Query and retrieval results for a by color and shape query. a) sketch drawn on the canvas; b) retrieval results; c) retrieval results after two rounds of relevance feedback.

Shape characteristics extraction has been performed using a modified version of the Fourier descriptors approach proposed in [21]. Our shapes are hence assumed simply connected with borders represented by a closed curve. We currently use the real part

Content-Based Image Retrieval over the Web

127

of the lower 100 Fourier coefficients for shape description, which are arranged in a single vector. As we discard the continuous component, we obtain scale, rotation and traslation invariant coefficients. Retrieval based on texture analysis in real images also requires image segmentation. Our approach uses Gaussian Markovian Random Fields (GMRF) for texture-based segmentation and texture extraction [8]. Two vectors storing mean values and variance values characterize each segmented area, obtained applying the GMRF algorithm.

3 User Interface and Query Processing The basic prerequisite a user interface for visual, content-based, image retrieval system should have is to let the user simply express "what" is his/her information need. Figure 2.a shows the color and shape query interface with a query sketched on the canvas. Here the user simply has to roughly draw a sketch, in terms of color and shape, of the images he/she would like to retrieve. A message window is provided to inform the user of the actions. A control panel shows the values computed when features are extracted. It is worth noticing that separate layers exist for shape and color distribution. A further "border" layer is provided to let the user check if the shape he/she has drawn is closed. This has drawbacks, as it may require drawing twice an object within the shape. The first one on the color layer and the second on the shape layer. This strategy was determined by the need to avoid the further computational burden of extracting the shape information from a color image. Also to reduce the computational burden on the client side, the retrieval interface uses a simple html page to show results. The user interface for texture processing is also a Java applet. Figure 3.a shows a query obtained selecting a user defined area within a database image. Differently from most systems, which basically allow posing a query using pre-selected texture images, our system allows the user to dynamically select a texture area within images in the database in order to increase user's interactivity with the system. Also this interface is endowed of a text area for messages from the system and a control panel to allow inspection of extracted parameters. Retrieval is performed by measuring the distance, in the n-dimensional space defined by the index terms, between the term vector of the query and the term vectors of the documents. In the retrieval by shape and color sub-system, two similarity functions are computed: simC(R,Q) and simS(R,Q), respectively accounting for color and shape. Each function simX(R,Q) representing the similarity between a database image feature, defined by the tuple R = (r0, r1, ..., rn), and the query image feature, also defined by a tuple Q = (q0, q1, ..., qn) is computed using the cosine similarity coefficient, defined as:

sim( R , Q) =

∑ ri q i ∑ ri2 × ∑ q i2

(1)

The resulting coefficients are merged to form the final similarity function as a linear combination:

128

E. Di Sciascio et al.

sim(R,Q) = α × simC(R,Q) + β × simS(R,Q)

(2)

where α and β are weighting coefficients. The weights defined by the coefficients obviously lead to increase or decrease the contribution of a feature with respect to the others. To better characterize the user's information need, these coefficients are dynamically modified during the relevance feedback stage. Figure 2.b shows an example of retrieval results.

a)

b) Figure 3. Query and retrieval results for a query by texture. a) selected query area; b) retrieval results after relevance feedback. Relevant texture areas are the highlighted ones.

In the retrieval by texture sub-system, a single similarity function is computed between the selected query area, represented by its feature vector Q, and textured areas of images in the database, represented by their feature vectors Ri is performed computing the Euclidean distance between vectors storing associated mean values. Such distance is normalised with associated variance values providing the following expression: simT (Q, Ri ) =

4

∑ j =1

(f

(Q ) − f j ( Ri ) )

2

j

σ 2j ( Ri )

(3)

Content-Based Image Retrieval over the Web

129

Resulting retrieved images are indexed according to a growing distance score. Uncertainty is obviously present in image retrieval, due to the weak correspondence between the computed features and the image content perceived by the user. In other words, the system may not match the user perception of similarity. For this reason there is a growing interest in the image database community towards methodologies and techniques for relevance feedback. In our model the user can improve retrieval results by selecting, among the topmost ranked retrieved images, the ones he/she considers relevant. This is known as "positive feedback". Based on experiments, this type of feedback is used in the retrieval by texture subsystem. The other sub-system also implements negative feedback for images considered not relevant. Leaving an image unselected marks it as "don't care" and it does not contribute to the relevance feedback process. A new query is computed by combining the feature vectors of the original query with the ones of the relevant and not relevant images. In practice, the modified query is computed by adding to the feature vector Q associated to the query image the feature vectors X of relevant images and subtracting the not relevant ones Y, respectively weighted with suitable δ and ε coefficients:

Q ( k +1) = Q ( k ) + δ

N rel

∑

i =1

Xi − ε

M notrel

∑

Yi

(4)

i =1

with δ=0 in the retrieval by texture sub-system. The modification is performed separately on the features. The modified query is then used in a new retrieval step. It is worth noticing that the shape and color retrieval utilizes a 3-layer relevance feedback strategy. The first one has been just described and it operates at the feature level. The other two operate on the combined features and on the images domain, i.e. if the user tends to concentrate on a single feature, the systems strengthens the contribution of that feature in the similarity measure. Furthermore if the user concentrates his/her interest on images of a single domain, i.e. all relevant images belong to a single category, the system retrieves in the following relevance feedback stages only images belonging to that domain. Figures 2.c and 3.b show results after relevance feedback processing. Table I shows quantitative results in terms of precision and recall for a first set of tests performed. Please notice mp1=mp2 as in our test we always retrieved the same number of images regardless of any threshold.

4 Conclusions Content-based access appears to be a promising direction to increase the efficiency and accuracy of unstructured data retrieval, yet strategies for specifically web-based CBIR systems have to be devised. We have investigated techniques to increase user's interactivity in such a scenario, proposing query by sketch as an effective way to start a significant query, and relying on relevance feedback to refine retrieval results. We are still integrating and refining our model; first experiments carried out with the aid of some volunteers have shown anyway fair precision and recall scores, and noteworthy improvements obtained by relevance feedback.

130

E. Di Sciascio et al.

TableI. Precision and recall scores Queries results without relevance feedback results with single layer relevance feedback results with two-layer relevance feedback

mp1≡mp2 0,495 0,593 0,637

mr1 0,649 0,778 0,836

mr2 0,659 0,787 0,842

References [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] [20] [21]

Communications of the ACM, 40, 12, 1997. ACM Multimedia Systems, special issue on Multimedia Databases, 3, 5/6, 1995. IEEE Computer, special issue on Content Based Image Retrieval, 28, 9, 1995. G. Salton, Automatic Text Processing, Addison Wesley, 1989. E. Di Sciascio, M. Mongiello, DrawSearch: a Tool for Interactive Content-Based Image Retrieval over the Net, in Proc. of SPIE vol. 3656,.561-572, 1999. Y. Rui, T.S.Huang, S. Mehrotra, Content based image retrieval with relevance feedback in MARS", Proc. of IEEE ICIP'97, 1997. A. Celentano, E. Di Sciascio, Features Integration and Relevance Feedback Analysis in Image Similarity Evaluation, Journal of Electronic Imaging, 7, 2, 1998. E. Di Sciascio, G. Piscitelli, A. Celentano, Textural Features and Relevance Feedback in Image Retrieval, in Visual Database Systems 4, Chapman and Hall, 1998. H.G. Stark, On image retrieval with wavelets, Journal of Imaging Systems and Technology, 7, 200-210, 1996. W. Niblak et al., The QBIC project: Querying images by content using color, texture, and shape, in Proc. of SPIE, vol. 1908, 173-182, 1993. M. Flickner et al., Query by Image and Video Content: The QBIC System, IEEE Computer, 28, 9, 23-31, 1995. P. M. Kelly, T. M. Cannon, D. R. Rush, Query by image example: the CANDID approach, in Proc. of SPIE, vol. 2420, 238-248, 1995. V. E. Ogle, M. Stonebrakes, Chabot: retrieval from a relational database of images, IEEE Computer, 28, 9, 40-56, 1995 R. Bach et al., The Virage Image Search Engine: An open framework for image management, in Proc. of SPIE, vol. 2670, 76-87, 1996. J.R. Smith, S.F. Chang, VisualSEEK: a fully automated content-based image query system, Proc. of ACM Multimedia'96, 1996. W.Y. Ma, B.S. Manjunath, NETRA: A toolbox for navigating large image databases, Proc. IEEE ICIP '97, 1997. R.W. Picard, T.Kabir, Finding similar patterns in large image databases, Proc. ICASSP, 1993. K. Hirata, T. Kato, Query by visual example, content based image retrieval, Lecture Notes in Computer Science, vol. 580, 1992. A Del Bimbo, P. Pala, Visual image retrieval by elastic matching of user sketches, IEEE Trans. PAMI, 19, 2, 1997. C. E. Jacobs, A. Finkelstein, D. H. Salesin. Fast Multiresolution Image Querying. Proc. of SIGGAPH 95, 1995. Y. Rui, A.C. She, T.S. Huang, Modified Fourier descriptors for shape representation-a practical approach, Proc. of 1st workshop on image databases and multimedia search, 1996.

Visual Learning of Simple Semantics in ImageScape Jean Marie Buijs and Michael S. Lew Leiden Institute for Advanced Computer Science Leiden University, Postbus 9512, 2300 RA Leiden, The Netherlands {buijs,mlew}@cs.leidenuniv.nl Abstract Learning visual concepts is an important tool for automatic annotation and visual querying of networked multimedia databases. It allows the user to express queries in his own vocabulary instead of the computer's vocabulary. This paper gives an overview of our current research directions in learning visual concepts for use in our online visual webcrawler, ImageScape. We discuss using the Kullback relative information for finding the most informative features in the case of human faces and generalize the method to other objects/concepts.

1 Introduction In many content based retrieval systems, the user is asked to understand how the computer sees the world. An emerging trend is to try to have the computer understand how people see the world. However, understanding the world is a fundamental computer vision problem which has withstood decades of research. The critical aspect to these emerging methods is that they have modest ambitions. Petkovic[1997] has called this finding "simple semantics." From recent literature, this generally means finding computable image features which are correlated with visual concepts. The key distinction is that we are not trying to fully understand how human intelligence works. This would imply creating a general model for understanding all visual concepts. Instead, we are satisfied to find features which describe some small, but useful domains of visual concepts.

1.1 Visual Search Paradigms Content based search researchers are constantly looking for methods which are usable by nonexperts. The typical method for this intuitive search is by finding a similar image. In this paradigm, the user clicks on an image, and then the search engine ranks the database images by similarity with respect to color, texture, and shape. In sketch search methods, the user draws a rough sketch of the goal image. The assumption is that the sketch corresponds to the object edges and contours. The database images which have the most similar shapes to the user sketch are returned. Sketches represent Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL'99, LNCS 1614, pp. 131-138, 1999.  Springer-Verlag Berlin Heidelberg 1999

132

Jean Marie Buijs and Michael S. Lew

an abstract level of reasoning of the image. Another abstract method uses icons to represent objects and/or concepts in the image. The user places the icons on a canvas in the position where they should appear in the goal image. In this context, the database images must have been preprocessed for the locations of the available objects/concepts. The database images which are most similar by the content of objects/concepts to the iconic user query are returned. For an overview, see Gudivada and Raghavan[1995] and especially, Flickner, et al. [1995]. We designed a system for searching networked multimedia databases called ImageScape. One of the principal query types in ImageScape is semantic icons. Semantic icons are essentially drag-and-drop icons which represent concepts from the user's vocabulary. These can be objects such as human faces, textures such as sand or wood, or even colors. The importance of the method is it does not require the user to learn how the computer understands the image. Instead, the computer learns how humans perceive images.

1.2 Imagescape System Overview When an image is brought to the server, it is analyzed for features pertaining to the semantic icons (i.e. faces, sand, water, etc.) and for extraction of the computer sketches. Then a thumbnail of the image and the features are stored in a compressed database. When a user sends an image query from a WWW Java browser/client program, the query is sent to the server, and matched against the database. The user drawn sketch is compared to the computer sketches and the semantic icons are compared to the automatically extracted features. The best matches are then sent back to the WWW browser/client program to be displayed to the user. In summary, the ImageScape system consists of the following modules: • • • • • •

collection of text, images, audio, and video from the WWW compression of the image database semantic object detection in images computer sketching of images matching between the icons/sketches with the database images Java client connecting to host server for the visual query input and processing

There are other interesting WWW image search engines which have been described in the research literature. The WebSeek[Smith and Chang 1997] system from Columbia University finds similar images and performs automatic text based category classification. The Webseer[Frankel, Swain, and Athitsos 1996] system from the University of Chicago lets users search by the number of faces and by text queries. Taylor, Cascia, and Sclaroff[1997] designed the ImageRover system to primarily use

Visual Learning of Simple Semantics in ImageScape

133

relevance feedback for the search process, and in the PicToSeek system, Gevers and Smeulders[1997] search through the images using similar images and image features. In a previous paper [Lew, et al. 1997], we introduced an early version of this system. In this work, our focus is entirely on the object/concept detection.

2 Learning Simple Semantics In this paper we discuss learning simple semantics, or in another way, visual learning of concepts. This brings into mind the question raised by readers and referees, which is “What is visual learning?” Such a general term could refer to anything having to do with artificial or human intelligence in all its sophistication and complexity. Rather than a vague description, we seek to define it clearly at least within the boundaries of this paper as either (1) feature tuning; (2) feature selection; or (3) feature construction. Feature tuning refers to determining the parameters which optimize the use of the feature. This is often called parameter estimation. Feature selection means choosing one or more features from a given initial set of features. The chosen features typically optimize the discriminatory power regarding the ground truth which consists of positive and negative examples. Feature construction is defined as creating new features from a base set of atomic features and integration rules. In this paper, we focus on feature selection and in the section on future work, we reveal preliminary results toward feature construction. What is an object/concept? For our purposes, an object/concept is anything which we can apply a label or recognize visually. These could be clearly defined objects like faces or more difficult concepts such as textures. Most textures do not have corresponding labels in common language. Object/Concept detection is essential to the usage of the WWW image search engine because it gives the computer the ability to understand our notion of an object or concept. Instead of requiring all users to understand low level feature queries, we are asking the computer to understand the high level queries posed by humans. For instance, if we want to find an image with a beach under a blue sky, most systems require the user to translate the concept of beach to a particular color and texture. In our system, the user can pose the query visually as a beach under a blue sky using icons to represent beach and blue sky, respectively. Giving a complete discussion of visual concept learning would not fit within the scope of a conference paper. In fact, it would require several books to do it justice. Furthermore, we suggest that what is necessary in the field now is a thorough survey on visual concept learning. For the scope of this paper, we give a brief overview of recent visual learning techniques in the research literature. We turn to an example of feature selection in the domain of human face detection, and then observe that it is straightforward to generalize the face detection method to other objects.

2.1 Background Picard[1996] reported promising results in classifying blocks in an image into "at a glance" categories. What this means is that she investigated multiple model methods

134

Jean Marie Buijs and Michael S. Lew

to classify an NxN block into categories which humans could classify without logically analyzing the content. Forsyth, et al. [1996] found objects from feature blobs. More recently, Vailaya, Jain, and Zhang [1998] have reported success in classifying images as city vs. landscape. They found that the edge direction features have sufficient discriminatory power for accurate classification of their test set. Buijs[1998] reported promising results in learning primary colors and textures using the Kullback relative information. The commonality between these methods was using multiple features for object/concept detection. Regarding object detection, the recent surge of research toward face recognition has motivated robust methods for face detection in complex scenery. Representative results have been reported by Sung and Poggio[1998], Rowley and Kanade[1998], Lew and Huijsmans [1996], and Lew and Huang[1996].

2.2 Feature Selection We begin by describing a method of finding human faces in grayscale images with complex backgrounds and then show that the method is easily extensible to other objects/concepts. The Kullback relative information[1959] is generally regarded as one of the canonical methods of measuring discriminatory power. Specifically, we formulated the problem as discriminating between the classes of face and nonface, and used the Kullback relative information as a measurement of the class separation, which is the distance between the classes in feature space. As the class separation increases, the overlap between the classes decreases making the confidence in the class decision increase. The detection algorithm can be stated concisely as follows: (1) Create a ground truth set of positive (face) and negative(nonface) examples (2) Compute the histograms from the ground truth set for the classes face and nonface. (3) Find the N most informative features by maximizing the Kullback relative information combined with a Markov random field. (4) Arrange the N most informative features in a vector, and apply a minimum distance classifier and is shown in Figure 2.1.

Visual Learning of Simple Semantics in ImageScape Eyes

Eyes/nose Templates

,

135

, ...

Kullback Relative Information

NonFace Templates Nose Area ,

, ...

Kullback Relative Information Image

(a) (b) Figure 2.1. (a) Result of using Kullback relative information on face and nonface examples; and (b) the 256 pixels which have the greatest discriminatory power.

2.3 Generalizing to Multiple Models In the previous explanation covering face detection, we used the 256 pixels which had the greatest discriminatory power. For lack of a better word, we define this set of features as a discriminatory model. This discriminatory model has the advantage that for N features in the model, it minimizes the misdetection rate. The question arises then of how to generalize the method to more features such as color, texture, and shape. For each object we wish to detect, a large set of positive and negative examples is collected. We measured a variety of texture, color, and shape features, and for each one we calculate the Kullback discriminant. The candidate features for our system included the color, gradient, Laplacian, and texture information from every pixel as shown in Figure 2.2. For the texture models, we used Trigrams[Huijsmans, Poles, and Lew 1996], LBP [Wang and He 1990], LBP/C, Grad. Mag., XDiff, YDiff, [Ojala, Pietikainen, and Harwood 1996]. For shape comparison, we used the features derived from elastic contours[Del Bimbo and Pala 1997], invariant moments[Hu 1962], and Fourier Descriptors [ Gonzalez and Woods 1993]. Best Model

Positive Examples

Texture Distributions and Projections Trigrams, LBP/C, Grad. Mag, Xdiff, Ydiff

Measure Discriminatory Power

Color RGB, HSV

Negative Example

Shape Invariant Moments, Snakes, Fourier Desc.

Figure 2.2. Selecting the best discriminatory model of N features from texture, color, and shape features.

136

Jean Marie Buijs and Michael S. Lew

3 Feature Construction In the previous discussion, we proposed using the Kullback relative information for feature selection. The next logical step was to consider methods for feature construction. In recent work[Buijs 1998], we presented a rule based method for combining the atomic features of color, texture, and shape toward representing more sophisticated visual concepts. Recall that in feature construction there are atomic features and rules for integrating them. Our atomic features were instances of color, texture, and shape. The atomic colors were red, yellow, purple, green, blue, brown, orange, white, gray, and black. From the Kullback relative information, the color model with the greatest discriminatory power was the HSV. The atomic textural features were coarse, semi-coarse, semi fine, fine, nonlinear, semi-linear, linear, and texture features based on examples: marble, wood, water, herringbone, etc. LBP/C had the greatest discriminatory power regarding the Kullback relative information. For the atomic shape features, we created a basic set of geometric primitives: circular, elliptical square, triangular, rectangular, and pentagonal. We also tuned a variety of shape examples. Shape features were detected using template matching and active contour energy [del Bimbo and Pala 1997]. Simple concepts were represented as AND conjoined boolean expressions: If (color is orange) AND (texture is coarse) AND (shape is circular) Then object is an orange More complex concepts were represented using AND/OR expressions such as: If ((color is yellow OR color is white) AND (texture is fine) AND (texture is nonlinear)) Then object is sand Rules were automatically generated from positive and negative example training sets using decision trees. We ranked the features used in the decision trees by the Kullback relative information, and created the tree using the features with greater discriminatory power first. Results for five outdoor categories are shown in Table 1. Table 1. Probability Of Misdetection Misdetection

forest 0.23

mountain 0.14

sand 0.27

sky 0.09

water 0.15

Visual Learning of Simple Semantics in ImageScape

137

4 Conclusions Visual concept learning has the potential of bringing intuitive searching for visual media to the general public. Regarding the World Wide Web, we can bring image and video search to anyone with a Web browser if these visual learning technologies mature. At Leiden University, we are currently creating a large library of visual feature training databases and detectors. In this paper, we gave an overview of the visual concept learning methods being used for the ImageScape project. From the perspective of visual learning as feature tuning, feature selection, and/or feature construction, we have shown a progression of techniques for learning simple concept domains. Regarding future work, we think that the methods for combining the results of multiple classifiers[Kittler 1998] have the most versatility and potential for improvement of simple semantic detection.

Acknowledgements This research was funded by the Dutch National Science Foundation and the Leiden Institute for Advanced Computer Science

References Buijs, J. M., “Toward Semantic Based Multimedia Search,” Master’s Thesis, , Leiden Institute for Advanced Computer Science, August 13, 1998. Del Bimbo, A., and P. Pala, “Visual Image Retrieval by Elastic Matching of User Sketches,” IEEE Trans. Pattern Analysis and Machine Intelligence, February, pp. 121-132, 1997. Flickner, M., H. Sawhney, W. Niblack, J. Ashley, Q. Huang, B. Dom, M. Gorkani, J. Hafner, D. Lee, D. Petkovic, D. Steele, and P. Yanker, “Query by Image and Video Content: The QBIC System,” Computer, IEEE Computer Society, pp. 23-32, Sept. 1995. Forsyth, D., J. Malik, M. Fleck, T. Leung, C. Bregler, C. Carson, and H. Greenspan, "Finding Pictures of Objects in Large Collections of Images," Proceedings, International Workshop on Object Recognition, Cambridge, April 1996 Frankel, C., M. Swain and V. Athitsos, “WebSeer: An Image Search Engine for the World Wide Web,” Technical Report 96-14, University of Chicago, August 1996. Gevers, T. and A. Smeulders, “PicToSeek: A Content-Based Image Search System for the World Wide Web,” VISUAL’97, San Diego, December, pp. 93-100. Gonzalez, R. and R. E. Woods, “Digital Image Processing”, Addison Wesley, 1993. Gudivada, V. N., and V. V. Raghavan, “Finding the Right Image, Content-Based Image Retrieval Systems,” Computer, IEEE Computer Society, pp. 18-62, Sept. 1995. Hu, M., “Visual Pattern Recognition by Moment Invariants”, IRA Trans. on Information Theory, vol. 17-8, no. 2, pp. 179-187, Feb. 1962.

138

Jean Marie Buijs and Michael S. Lew

Huijsmans, D. P., M. Lew, and D. Denteneer, “Quality Measures for Interactive Image Retrieval with a Performance Evaluation of Two 3x3 Texel-based Methods,” International Conference on Image Analysis and Processing, Florence, Italy, September, 1997. Kittler, J., M. Hatef, R. Duin, and J. Matas, “On Combining Classifiers,” IEEE Trans. Patt. Anal. and Mach. Intel., vol. 20, no. 3, March 1998. Kullback, S. “Information Theory and Statistics,” Wiley, New York, 1959. Lew, M., K. Lempinen, and N. Huijsmans, "Webcrawling Using Sketches," VISUAL'97, San Diego, December, 1997, pp. 77-84. Lew, M. and N. Huijsmans, “Information Theory and Face Detection,” Proceedings of the International Conference on Pattern Recognition, Vienna, Austria, August 25-30, 1996, pp.601-605. Lew, M. and T. Huang, “Optimal Supports for Image Matching,” Proc. of the IEEE Digital Signal Processing Workshop, Loen, Norway, Sept. 1-4, 1996, pp. 251254. Ojala, T., M. Pietikainen and D. Harwood, “A Comparative Study of Texture Measures with Classification Based on Feature Distributions,” vol. 29, no. 1, pp. 51-59, 1996. Petkovic, D., "Challenges and Opportunities for Pattern Recognition and Computer Vision Research in Year 2000 and Beyond, "Proc. of the Int. Conf. on Image Analysis and Processing, September, Florence, vol. 2, pp. 1-5, 1997. Picard, R.. "A Society of Models for Video and Image Libraries." IBM Systems Journal. 1996. Rowley, H, and T. Kanade, Neural Network Based Face Detection, IEEE Trans. Patt. Anal. and Mach. Intell., vol. 20, no. 1, pp. 23-38, 1998. Smith, J. R. and S.F. Chang, “Visually Searching the Web for Content,” IEEE Multimedia, 1997, pg. 12-20. Sung, K. K., and T. Poggio, Example-Based Learning for View-Based Human Face Detection, IEEE Trans. on Patt. Anal. and Mach. Intell, vol. 20, no. 1, pp. 39-51, 1998. Taycher, L., M Cascia, and S. Sclaroff, “Image Digestion and Relevance Feedback in the ImageRover WWW Search Engine,” VISUAL’97, December, San Diego, pp. 85-91. Tekalp, A. M., Digital Video Processing, Prentice Hall, New Jersey, 1995. Vailaya, A., A. Jain and H. Zhang, "On Image Classification: City vs. Landscape," IEEE Workshop on Content-Based Access of Image and Video Libraries, Santa Barbara, June 21, 1998. Wang, L. and D. C. He, “Texture Classification Using Texture Spectrum,” Pattern Recognition 23, pp. 905-910, 1990.

Task Analysis for Information Visualization Stacie L. Hibino Bell Labs, Lucent Technologies 263 Shuman Boulevard, Naperville, IL 60566 USA [email protected] http://www.bell-labs.com/~hibino/

Abstract. Previous research in information visualization has primarily focused on providing novel views and frameworks to aid users in exploring or accessing data; very little work has been done to support users through the full analysis processfrom the raw data to the final results. But what tasks do users perform when analyzing data using an information visualization (infoVis) environment? A task analysis of experts’ use of an existing infoVis system was conducted to examine this question. Results indicate that users work on various tasks outside of data explorationtasks such as conditioning and preparing data, collecting results, and gathering evidence for a presentation. This pilot study identifies key data analysis tasks that expert users perform when using an infoVis environment to analyze some real-life data.

1 Introduction Recent advances in information visualization (infoVis) have led to novel visualizations and new paradigms to support users in accessing and exploring a wide variety of data. General infoVis frameworks [1,13] and applications to fields such as temporal [8], software [9] and medical [10] data are just a few examples of the growing work in infoVis. Most of this work, however, has focused on enhancing data access and exploration; little, if any work has examined the larger problem of supporting users’ data analysis processthe processes they use in transforming raw data into a presentation of results based on using infoVis as their key analysis tool. The continual introduction of more powerful computers and the recent explosion in large data repositories offers new opportunities while posing new challenges to the infoVis community. In our research lab, where we have already developed a suite of infoVis tools capable of handling moderately sized data sets (easily accommodating data sets containing 100,000 data records), we are beginning to see evidence of these challenges as users complain about the effort required to prepare data for exploration or abstract key results from an infoVis analysis of a complex data set. Previous work in information workspaces [3,7] offers glimpses of an organizing framework, but much of this work focuses on organizing data and objects (e.g., documents) rather than a user’s process.

Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL'99, LNCS 1614, pp. 139-146, 1999.  Springer-Verlag Berlin Heidelberg 1999

140

Stacie L. Hibino

In order to understand how to better support users in the infoVis analysis process, we set out to identify the types of tasks they conduct during data analysis with an infoVis tool. In this paper, we describe the task analysis conducted to accomplish this goal. We had three driving questions in this endeavor: • Do users conduct other tasks besides data exploration during data analysis? • If so, what are these tasks? • How important are these tasks to the data analysis process? The task analysis was conducted on five infoVis experts using an existing infoVis environment (EDV: the Exploratory Data Visualizer [13]). Users were asked to analyze a disease data set used as part of an American Statistical Association (ASA) Data Exposition. Their goal at the end of their analysis sessions was to present key results of their analysis in their typical target presentation format. Overview. This paper is divided into four additional sections. In the next section, I describe the experimental method. I then present and discuss the results, summarize some related work and finally, I provide conclusions and describe some future work.

2 Experimental Method The goal of the task analysis was to study expert users analyzing real-life data with an existing infoVis system (EDV [13]) as their primary analysis toolwhere users started with the raw data, analyzed it, and worked towards a presentation of results. In order to examine how users cope with a complex data analysis problem requiring multiple sessions, the users were asked to accomplish this over two separate sessions. Participants. Five users (all male) participated in the study. They were invited to participate based on their expertise in using EDV as well as their experience in using information visualization (infoVis) environments for data analysis. Two users have a statistics background, while the other three hold advanced computer science degrees. EDV. The Exploratory Data Visualizer (EDV) is an information visualization framework which provides linked data views [13]. EDV allows users to create and interact with any number of dynamically linked, user-specified data views including univariate (e.g., bar charts and histograms), bivariate (e.g., scatterplots and table views), multivariate, network, and text-based views (e.g., similar to a spreadsheet). Views are dynamically linked to one another so that a selection made in one view results in automatic highlighting of the corresponding selections in all other views. Data Set. Users analyzed real-life tuberculosis (TB) disease data from the 1991 American Statistical Association Data Exposition [5]. The TB data included one primary file of over 11,000 records and five other supporting data files on information such as census data. Procedure. The users’ goal was to analyze the TB data and create a presentation of key results. They were given two 1-hour sessions, separated in time by at least one week, to accomplish this goal. They used EDV as their primary tool for analyzing the data, and were told that they could use any other tools that they typically use in conjunction with EDV to accomplish the analysis task. In the first session, users read a sheet describing this task analysis study and their goals, and a hard copy of the readme

Task Analysis for Information Visualization

141

file included with the TB data set. They were asked to provide think-aloud verbal protocols of what they were doing as they were doing it, and then proceeded with the analysis task at hand. At the end of the second session, if extra time was available, a short informal post-interview was conducted. Information about the post-questionnaire was sent to users after they had completed their second session. Data Collected. The following data was collected: observational data, files generated during the users’ sessions (e.g., scripts written to transform the data, new files of transformed data, etc.), post-interview notes, and results of the post questionnaire. In addition, output of the users’ screen was captured directly onto video, along with audio of their verbal protocols.

3 Results and Discussion While users did take different approaches to their analyses, their core set of tasks and findings were very similar. However, none of them created an actual presentation of their results. Although they analyzed a real-life data set, they were not motivated to create a presentation and in most cases did not feel that they had enough time to do so. Instead, they either captured their key findings on paper or articulated them aloud during their analysis. Information about presentation-related tasks was gathered through informal post-interviews and through the post-questionnaire. 3.1 High-Level Analysis Tasks During the task analysis, we observed several low-level user tasks. Through grouping similar tasks together, we identified seven categories of high-level tasks: • prepare: data background and preparation tasks, • plan: analysis planning and/or strategizing tasks, • explore: data exploration tasks, • present: presentation-related tasks, • overlay: overlay and assessment tasks, • re-orient: re-orientation tasks (when analysis requires more than one session), and • other: statistics-based tasks. The low-level tasks for each of the task categories are described in Section 3.2. When asked, users did not present an alternative list or any additional categories of highlevel tasks. Users rated the importance of each of the low-level tasks to the analysis process based on a scale of 1 to 5. Figure 1 shows the average importance ratings for each of the high-level task categories, based on an average of all of the low-level task ratings in each category. Overall, the ratings are fairly high, with averages ranging from 3.4 to 4.5 on a 5-point scale. However, taking individual users into account, an analysis of variance indicates that the differences in importance ratings between categories is significant, leading to the following order of importance based on user ratings: plan > explore > prepare > present > statistics > overlay > re-orient tasks.

142

Stacie L. Hibino

Fig. 1. Average importance ratings for task categories (1=unimportant, 5=very important; error bars indicate standard deviation)

3.2 Low-Level Analysis Tasks The low-level tasks were identified through observational data, previous infoVis taxonomies [12], informal interviews and through the post-questionnaire. Tables 1 to 6 summarize the low-level tasks by category and include user examples. While there is some overlap between tasks and task categories, these summaries provide a working taxonomy of data analysis through information visualization. Table 1. Prepare: data background and preparation tasks Task Description Gather background information about data set at hand Understand data sources Get clarification on data ambiguities Collect additional data from other external sources Reformat data for suitable input Check data for potential data errors Check for missing data Transform the data

Example Review TB readme file Note TB data file names, sizes, formats How is “race” different from “ethnicity?” Can I get my almanac? Add header information to a data file for importing into EDV Spot check raw data file Is there any missing data? Split variables, rollup/aggregate data

Table 2. Plan: analysis planning and strategizing tasks Task Description Hypothesize Make a strategy or plan for all or a part of your analysis Identify data formats and variables required for desired views

Example There was a hypothesis that TB incidence increased in HIV infected groups Decide what, how, and how much to investigate or explore We need sums of census data by state…

Task Analysis for Information Visualization

143

Table 3. Explore: data exploration tasks (incorporating tasks from [12]) Task Description Get an overview of the data Investigate data to test hypotheses (top down approach) Explore data in search of trends or exceptions (bottom-up approach) “Query” or filter the database Identify curiosities to investigate further Zoom in on items of interest Remove uninteresting items Identify data clusters Identify relationships between variables Explain view/visualization Identify a trend or exception Verify a trend or exception Drill-down for more details

Example I always like to use something to get some idea of the whole data set… There was a hypothesis that…. We can look at that actually Now let’s look at [the] race [variable] We go to race=2 [African American], we see that they get TB around … So that’s sort of interesting… I wonder, let’s… Let’s look at [just] that peak of youngsters I’m just going to eliminate those early years; concentrate on data where there at least seems to be stable reporting going on Alright, let’s try clustering…cluster view So in terms of age, whites seem to get TB more when they’re older in comparison to the other races… There are two possible answers [explanations] here. One is that… An interesting gap in the [age] data here… There’s a gap in around 12 year olds. Examine alternative view to verify a trend [looking at text records] You can actually count the number of 15 year olds...

Table 4. Present: presentation-related tasks Task Description Gather evidence to answer a hypothesis or driving question Record or keep track of trends and results tested and found Articulate importance of a result (rank it or identify it as “interesting”) Articulate/summarize all and/or key results Decide what to include in presentation of results Create presentation of results Give presentation of results

Example Well, in terms of the urban area hypothesis, it looks like it might be reasonably, likely; District of Columbia, which is an urban area … No significant effect per time-of-year How does this result rank in comparison to the others? I’m going to present a summary of my results in written form What are the top 2-3 interesting results I want to show? Paste screen dumps into an electronic presentation and annotate Communicate results to others

144

Stacie L. Hibino

Table 5. Overlay: overlay and assessment tasks Task Description Take notes Window management Assess your strategy Assess your observations Assess your assumptions about data formats Assess your progress Estimate cost-benefit ratio of additional data collection or conditioning

Example Write down code info: 1=male; 2=female Move and resize windows Is this the right strategy to take? Does this observation make sense? Is my data in the right format to accomplish this part of the analysis? I’m wondering if there’s anything else I haven’t considered which I should look at I should really recode those… but I couldn’t be bothered with that

Table 6. Re-Orient: re-orientation tasks Task Description Review goal(s), Review data and formats, Review notes Review progress Identify starting point for current session

Example Review TB readme file head tb.txt Flip through written notes What I remember doing last time was… So what I wanted to try doing today was…

Due to space limitations, we cannot discuss each of the low-level tasks in detail, but we do highlight some of the more interesting observations here. A couple of interesting data exploration tasks include explaining a visualization and verifying a trend or exception. These tasks, which have not previously been reported in other infoVis taxonomies, indicate that expert users do not just stop at the identification of a trend or exception, and that they consider their analysis to be as much of an investigation as an exploration. Different users have different types of target presentations. Users in this study listed a variety of typical target presentations ranging from static screen dumps to interactive web pages [6] and live EDV demonstrations. For small data sets and in situations where only a few results are identified, presentations are much easier to create and require fewer tasks. Large complex data sets, however, require additional work to keep track of, rank, and decide on presentation contents of results. This is especially the case when users may be sorting through a series of 20 to 30 results. The take notes and window management tasks are overlay tasks that cut across the other task categories. For example, users could take notes while preparing data, planning their analysis, or exploring the data. The assessment tasks identify the metacognitive activity exhibited by users during their analysis sessions. That is, users asked themselves the types of sample questions listed with each task in Table°5. Statistics-Based Tasks. Although users did not conduct any statistical tests during their analysis of the TB data, several users mentioned situations where they either would follow-up with a statistical test or where they thought it would be nice to have system support to conduct a particular statistical test.

Task Analysis for Information Visualization

145

Additional Tasks. Three users listed five tasks between them that they felt were not included in the task list presented in the post-questionnaire. Three of the five tasks listed were very similar to existing tasks while the other two included: • given a relationship or feature of interest, explore what other factors may contribute to it, and • sort and order results so that related results and their impact on each other can easily be accomplished.

4 Related Work Several infoVis taxonomies have been proposed (e.g., [12, 2]), but these typically focus on categorizing aspects of infoVis limited to accessing and exploring dataaspects such as data types, visualization types and exploration tasks. The results of the task analysis presented in this paper indicates that such taxonomies only address a part of the problem, especially when considering the use of infoVis for data analysis rather than only data access. While no infoVis environment currently addresses all of the types of tasks identified through this task analysis, some work has touched on some of the issues identified here. For example, the SAGE system [11] is a knowledge-based presentation system that potentially reduces the users’ task load on analysis planning as well as presentation-related tasks. A second example is an infoVis spreadsheet [4] that provides a framework for organizing data conditioning as well as data exploration based on graphical transformations. Information workspaces (e.g., [3, 7]) have typically focused on organizing data rather than processes. However, one can imagine using a rooms [7] or book metaphor [3] for organizing an infoVis analysis. For example, rooms or books could be used as logical separators for the different types of tasks (e.g., data preparation, analysis planning, data exploration, etc.) or they could be used to separate the analysis along themes or threads (e.g., separate rooms could be dedicated to investigations of different hypotheses). The challenge in using either of these metaphors, however, is in understanding how, if, and when process support can be bridged across rooms.

5 Conclusion and Future Work Users do perform many other tasks beyond data exploration when using an infoVis environment for analyzing data. In this pilot study, I identified six other categories of analysis tasks besides data exploration: data background and preparation, analysis planning and strategizing, presentation-related, overlay and assessment, re-orientation, and statistics-based tasks. Moreover, not only do users conduct these other tasks, they also rate them highly in terms of their importance to the analysis process. In particular, they rate planning and strategizing tasks significantly higher than exploration tasks. We are currently in the process of performing a detailed analysis of the video data and verbal protocols to identify how often users performed the various tasks as well as how much time they spent on each of them. In the mean time, we note that users

146

Stacie L. Hibino

indicated on the post-questionnaire that they typically spend, on average, about 25% of their analysis time on data exploration and at most 40% of their time; thereby spending over half of their analysis time on tasks other than data exploration. In the future, we plan to investigate and prioritize the importance of system support of each of the tasks. Our long-term goal is to work towards a more integrated infoVis framework, one that provides better support to users through the full data analysis process.

Acknowledgments Special thanks to expert users who participated in the study, to Graham Wills for EDV, and to Beki Grinter and Ken Cox for reviewing earlier drafts of this paper.

References 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13.

Ahlberg, C., & Shneiderman, B. (1994). Visual Information Seeking: Tight Coupling of Dynamic Query Filters with Starfield Displays. CHI'94 Conf. Proc. ACM Press, 313-317. Card, S. and J. Mackinlay. (1997). The Structure of the Information Visualization Design Space. IEEE Proceedings of Information Visualization’97, 92-99. Card, S., Robertson, G., and W. York. (1996). The WebBook and the Web Forager: an information workspace for the World-Wide Web. CHI '96. Conf. Proceedings, 111-119. Chi, E.H., Riedl, J., Barry, P., and J. Konstan. (1998). Principles for Information Visualization Spreadsheets. IEEE Computer Graphics & Applications, 18(4), 30-38. 1991 ASA Data Exposition, Disease Data. Available at: http://www.stat.cmu.edu/disease/. Eick, S., Mockus, A., Graves, T. and Karr, A. (1998). A Web Laboratory for Software Data Analysis. World Wide Web Journal, 12, 55-60. Henderson, J. & S. Card. (1986). Rooms: The use of multiple virtual workspaces to reduce space contention in window-based graphical user interfaces. ACM Transactions on Graphics, 5(3), 211-241. Hibino, S. and Rundensteiner, E. (1996). MMVIS: Design and Implementation of a Multimedia Visual Information Seeking Environment. ACM Multimedia’96 Conf. Proc. NY:ACM Press, 75-86. Jerding, D.F., Stasko, J.T. and Ball, T. (1997). Visualizing interactions in program executions. ICSE’97 Conference Proceedings. NY:ACM Press, 360-370. North, C., Shneiderman, B. and Plaisant, C. (1997). Visual Information Seeking in Digital Image Libraries: The Visible Human Explorer. Information in Images (G. Becker, Ed.), Thomson Technology Labs (http://www.thomtech.com/mmedia/tmr97/chap4.htm). Roth, S., Kolojejchick, Mattis, J. and J. Goldstein. Interactive graphic design using automatic presentation knowledge. CHI’94 Conference Proceedings, 318-322. Shneiderman, B. (1996). The eyes have it: A task by data type taxonomy for information visualizations. IEEE Proceedings of Visual Languages 1996, 336-343. Wills, G. (1995). Visual Exploration of Large Structured Datasets. New Techniques and Trends in Statistics. IOS Press, 237-246.

Filter Image Browsing Exploiting Interaction in Image Retrieval J. Vendrig, M. Worring, and A.W.M. Smeulders Intelligent Sensory Information Systems, Department of Computer Science University of Amsterdam Kruislaan 403, 1098 SJ Amsterdam, The Netherlands Tel/fax: ++31-20-525.7463/7490 {vendrig, worring, smeulders}@wins.uva.nl

Abstract. In current image retrieval systems the user refines his query by selecting example images from a relevance ranking. Since the top ranked images are all similar, user feedback often results in rearrangement of the presented images only. The Filter Image Browsing method provides better incorporation of user interaction in the retrieval process, because it is based on differences between images rather than similarities. Filter Image Browsing presents overviews of the database to users and lets them iteratively zoom in on parts of the image collection. In contrast to many papers where a new system is just introduced, we performed an extensive evaluation of the methods presented using a user simulation. Results for a database containing 10,000 images show that Filter Image Browsing requires less effort from the user. The implementation of Filter Image Browsing in the ImageRETRO system is accessible via the Web.

1

Introduction

An important future research direction in image retrieval is the introduction of the ”human in the loop” [4]. User interaction can be helpful to reduce large diverse domains to small specific domains, exploiting human knowledge about the query and context. In our opinion, there are only three image retrieval methods that are truly interactive, i.e. the input and output of the methods accommodate further interaction with the user. The first two methods are subtypes of Query by Navigation, meaning the user input consists of a choice of one of the navigation controls provided by the system. The subtypes are distinguished based on the relation of the navigation controls to the content of the image database: – Visual Inspection. Controls are content-independent, e.g. ”next” buttons. – Query by Association. Controls are content-dependent, e.g. hyperlinks. The third interactive image retrieval method is a special case of the wellknown Query by Pictorial Example method, viz. Query by internal Pictorial Example (QiPE). In a QiPE system user input consists of one or more images Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL’99, LNCS 1614, pp. 147–155, 1999. c Springer-Verlag Berlin Heidelberg 1999


148

J. Vendrig et al.

being selected from the database. The system returns a relevance ranking, i.e. a list containing the distances from the user input images to images in the database. In practice, systems make use of more than one retrieval method. Visual Inspection is always involved in showing the output images of all other methods. Although trivial, the use of Visual Inspection alone is too time consuming. Therefore we concentrate on the characteristics of the other two interactive methods, viz. Query by Association and QiPE. The advantage of Query by Association is the use of structure, which can easily be visualized. However, it requires construction of the relations between the items in the database, which is usually done manually. QiPE is very flexible and dynamic. The search path is determined run-time by computing relevance rankings. However, there are two important drawbacks to the QiPE method. Firstly, users can get stuck at a local optimum. Secondly, the selection of the initial set of images to show to the user is non-trivial. It is generally neglected in literature though. While traditional systems do not help users after answering a query in further specification of their information need, interactive systems do provide the user more overviews than just an initial one. Recent research does take advantage of interaction by providing relevance feedback, e.g. by extracting knowledge from the browse path [5] or giving feedback by visualizing distances between images [3,7]. The Filter Image Browsing method we developed combines the powerful concepts for interaction found in Query by internal Pictorial Example and Query by Association. Furthermore, our method makes use of the browse path of a user when interacting. Filter Image Browsing is described in section 2. In section 3 evaluation criteria and the simulation environment are described. The results of experiments are presented in section 4. Conclusions are given in section 5.

2

Filter Image Browsing

In Filter Image Browsing (FilIB) a user recursively selects dynamically generated clusters of images. By choosing the cluster most similar to the information need, the user zooms in on a small collection of relevant images. The scatter/gather method [2] uses a comparable approach for the retrieval of textual documents, but focuses on the visualization of features of document clusters in the form of words.. The goal of FilIB is to assist in the quick retrieval of images by facilitating interaction between system and user with database overviews. FilIB can be seen as a structuring overlay over QiPE. The structuring overlay handles the lack of overview in traditional QiPE systems. Alternatively, FilIB can be viewed as the addition of a dynamic zoom function for image databases to Query by Association. In this section, the method and its consequences are discussed. Furthermore, a detailed description of the basics of Filter Image Browsing is given.

Filter Image Browsing

149

Fig. 1. Filter Image Browsing retrieval process.

2.1

The Retrieval Process

A Filter Image Browsing retrieval session (Fig. 1) first presents an initial overview of the content of the database in the form of images. The user inspects the images shown and selects the one most similar to the images he is looking for. Then the system performs a similarity ranking for the selected image. Next, the filter step, which characterizes this method, is performed. Only images most alike the query image are used in the remainder of the retrieval session. The three steps overview, selection and reduction are repeated until the set of remaining images is small enough to switch to Visual Inspection. Each overview is based on the subset of images in that state of the retrieval process so that the user zooms in on the image database. However, the reduction filter potentially results in the loss of relevant images during the retrieval process. Once an image is excluded, it cannot be retrieved anymore during that particular session. The amount of reduction and the number of required selections should be balanced to minimize loss of relevant images and time spent searching. For the sake of brevity, we do not consider navigational options such as “back (to previous state)” here. More formally explained, let Is be the active set with Ns images in state s of the retrieval process. In the initial state s=0, I0 is the entire image database of size N0 . In state s=0, the system presents images in the overview Is ⊆Is . Then the user selects seed images i∈ Is . For the sake of simplicity we limit i to one seed image. The system performs a filter operation ϕ resulting in a new active set of images: Is+1 =ϕ(Is ,i). The user ends the session when the desired images are found or when he thinks the images cannot be found. The latter case indicates that either the interactive system failed or the desired images are not present in the database. In the following paragraphs the overview and filter operations are explored in more detail. As shown later, the overview operation is influenced by the filter operation. Therefore, the filter operation is described first. Filter The appliance of a filter results in a change of scope of the active image set. Since the purpose of FilIB is to zoom in on a set of images that suits the user’s information need, reduction is the only filter operation considered here. The goal of reduction is getting a smaller set of images still containing the images desired. Similarity ranking of the active set according to user selected

150

J. Vendrig et al.

images is an appropriate technique to select images for the new (reduced) set. Is+1 then contains the images from Is that are most similar to query images i. A reduction factor ρ is used. E.g., if ρ=0.25, Is+1 is 4 times as small as Is . Thus the reduction operation targets at a fixed size for the new active set so that the outcome is predictable. When all parameters are known, it is easy to compute the number of steps necessary to reduce I0 to an image set suitable for Visual Inspection. Since Ns =ρ·N0 , s can be computed for a fixed end size Nend : send =

log

Nend N0

log ρ

.

(1)

Overview The goal of the overview function is to present to the user a number of images representative for the active image set. The number of presented images must be small, since the user looks at the images by way of Visual Inspection. The images in Is are the only access to other images in Is . Combined with the reduction operation this results in orphan images in Is , i.e. images i for which it is not possible to choose an image from Is resulting in an Is+1 that contains i. Orphan images are always lost in the next state. To guarantee Is is fully covered by Is , we introduce the cover constraint, which says every image in Is must be part of at least one of the possible new active sets Is+1 . Thus the overview function must produce a presentation set Is that complies to the following constraint:
ϕ(i, Is ) = Is . (2) i∈Is

The selection of a presentation set that complies to the cover constraint consists of three stages: Is = postSelection(coverConstraint(preSelection)). In stage 1 a preselection is made, i.e. a set of input images are determined, e.g. randomly or by a user. In stage 2 the preselection is extended so that the total set of images complies to the cover constraint. If the number of images required for compliance to the cover constraint exceeds the maximum number of images shown to the user, the reduction factor should be adapted by user or system. In the optional stage 3 a postselection is made to extend the set to a predefined size. Postselection ensures a predictable amount of output to the user. We have constructed a brute force algorithm that guarantees compliance to the cover constraint for a presentation set Is . The preselection is one seed image, either randomly chosen from I0 , or selected by the user from Is−1 . The image is the first member of Is . Then it is used to perform a reduction on a copy of Is . Images in the (virtually) reduced set are checked as being a child of a member of Is . The algorithm then uses the image least similar to the last seed image to find the new seed image. Again, it is added to Is . The process is repeated until all images in Is are known to be a child of at least one of the images in Is . In a best case scenario, 1/ρ seed images are necessary. In practice the brute force approach resulted in an average of 12 images necessary to cover our data sets for ρ=0.25. In our opinion Is typically should contain about 20 images. The

Filter Image Browsing

151

brute force algorithm does not exceed this maximum. We have implemented a scenario for the postselection of images that consists of the random selection of images from Is . The results for this scenario are discussed in section 4.

3

Evaluation

In order to evaluate the Filter Image Browsing concept an experiment on the ImageRETRO system1 was set up. In the following sections we describe the evaluation criteria and the experimental environment. 3.1

Criteria

For evaluation of the effectiveness of Filter Image Browsing, separate criteria are used for each of the two functions, viz. reduction and overview. It is assumed that the information need of the user, expressed by the set of target images T⊂I0 , is static during the entire session. Reduction Evaluation To measure the effect of discarding images during the filter process, the Recall in each state of the retrieval session is computed:  |Is T | . (3) Recalls = |T | In the best case, the reduction operation discards irrelevant images only, and Recall is 1 (maximum) in every state. The reduction criterion measures to what degree the best case scenario is approached in practice. By definition, Recall is 1 in initial state 0. To prevent forced loss of relevant images, in every state of the retrieval session Ns has to be equal to or greater than the size of T. Overview Evaluation For comparison of presentation set generators, Sought Recall (SR) [6] in each state of the retrieval session is measured. SRs expresses the amount of relevant images the user has actually seen on screen during the current and previous states.        (Ij T )   j=0..s . (4) SRs = |T | The values for SR of FilIB can be predicted by using Recall values, assuming all images in Is have an equal chance of being selected for Is : SRs ≈ 1

|Is | · Recalls . Ns

http://carol.wins.uva.nl/˜vendrig/imageretro/

(5)

152

J. Vendrig et al.

Since SR is known to the user, the prediction can be used to derive Recall. Subsequently conclusions about continuing or restarting the session can be made. In our opinion the use of representative images in FilIB should lead to better results than in the case of the presentation of a relevance ranking, albeit a local optimum. In order to test this hypothesis, SR of both methods is compared. FilIB has to outperform QiPE (relevance ranking system) within a reasonable amount of user interactions. 3.2

Experiment

Domain A large and diverse domain was chosen to populate the image database, viz. the World Wide Web. The collection of 10,000 images2 is representative for the domain. For every image, index values were computed for 9 simple image features [8], primarily based on the distribution of hue, saturation and intensity. Ranking The relevance of an image is computed by averaging similarity scores of all individual features. Since features cannot be expected to have similar distributions of their values, scores cannot be averaged to a meaningful overall score directly. Therefore a normalization function is added to the feature scoring function. For each feature a priori a histogram containing the frequency of a similarity score is made. When the feature scoring function is invoked run-time, it looks up the similarity score in the histogram and returns the percentile, e.g. stating that the similarity between the two objects is in the top 5%. The percentiles of the various features can be compared and averaged with one another because they are independent of the distribution and similarity metric used. The use of the normalization via frequencies does not only allow the use of different types of image features and similarity metrics, e.g. histograms and metric values, but also the use of features of other media than images. Simulation In the evaluation experiment users were simulated by a user model as introduced in [1]. In the user model it is assumed that all users are the same and that the decisions they make are based on image features, so that the modeled users are consistent and unbiased. Target sets defining the simulated information need comply with 3 conditions: – Small distance in feature space, so that clustering the target images is possible. – Same style (visual similarity). Style is defined by objective meta-data (common original site) and subjective evaluation of the images. – Size bandwidth. There is a minimum and maximum for the number of images to focus on finding medium sized groups of images. For the experiment seven target sets3 were selected from the image collection. 2 3

publicly available at http://carol.wins.uva.nl/˜vendrig/icons/ http://carol.wins.uva.nl/˜vendrig/imageretro/target/

Filter Image Browsing

1

153

FilIB (0.2) FilIB (0.3) FilIB (0.4) FilIB (0.5) QiPE

Recall

0.9

0.8

0.7 10000

Sought Recall

1

0.75

0.5

0.25

0

1000

100

Size (logarithmic scale)

10

1

2

3

4

5

6

7

8

# interactions

9

10

11

Fig. 2. Evaluation results. Effect of reduction for various reduction factors (left) and retrieval performance of FilIB and QiPE (right).

The algorithms used for simulation of FilIB and QiPE make use of the predefined target sets and a given presentation size for Is (the desired number of pictures to be shown) fixed on 20 in our experiments. The reduction factor ρ is given as a constant for each retrieval session. Both simulations use the choose seed function, which computes which image in the presentation set is most similar to the entire target set. If one of the images in the presentation set is a member of the target set, it is chosen by default. For QiPE I0 is given, i.c. I0 =overview(I0). The presentation set function returns overview (Is ,i) for FilIB, and the top ranked images from Is for QiPE. The convergence criterion is “Ns equals presentation size” for Filter Image Browsing, and “high similarity of Is and Is+1 (>80% overlap)” for QiPE. Finally, the PresentationSetsimilarity ranking function results in ϕ(Is ,i) for Filter Image Browsing, and a plain similarity ranking relative to seed image i for QiPE.

4

Results & Discussion

Simulations were run for the 7 target sets, with 5 different randomized seeds. In Filter Image Browsing simulations, for each of the reduction factors used, the average of 5 slightly varying reduction factors was used. The reduction factors mentioned are the medians. The results for each Filter Image Browsing simulation are based on 175 runs in total. In Fig. 2 the graphs for the evaluation of the reduction operation are shown. Since the loss of relevant images depends on the reduction factor and the resulting size Ns , Recall is expressed as a function of Ns in the reduction effect graph. The reduction evaluation shows that even though Filter Image Browsing does cause the loss of desired images, far more irrelevant than relevant images are discarded. Low reduction factors of about 0.3, as well as high reduction factors of about 0.5 result in good performance. For comparison of Filter Image Browsing to a QiPE system a cluster based overview was evaluated for both systems. In the case of QiPE this means the initial presentation set only is constructed by

154

J. Vendrig et al.

way of clustering techniques. For Filter Image Browsing four different reduction factors were used. The graphs for Filter Image Browsing simulations stop when converged. The number of states necessary to converge can be computed from equation 1. The maximum number of interactions for both FilIB and QiPE is 11. The retrieval performance graph shows that even though both FilIB and QiPE find approximately the same amount of relevant images, the former method requires less user interactions to reach that result.

5

Conclusions

The concept of Filter Image Browsing shows that incorporating user interaction in an image retrieval method pays off. Subsequent cycles of database overview and reduction lead the user in few steps to a small collection of similar images. The consequences of the inherent ambiguity in the selection of representative images based on a combination of feature similarities is left for future research. The simulations used to evaluate the performance of Filter Image Browsing show satisfying results for all determined criteria. We conclude that more elaborate use of user interaction does result in quicker retrieval of images. Furthermore, the results of Filter Image Browsing are more predictable, as the number of user interactions can be computed a priori. This indicates that the method is helpful as well when desired images are not present in the image collection, since a user does not have to search indefinitely. Thus the combination of Query by internal Pictorial Example and Query by Association into Filter Image Browsing results in a powerful method for browsing through image databases.

References 1. I. Cox, M. Miller, S. Omohundro, and P. Yianilos. Target testing and the PicHunter bayesian multimedia retrieval system. In Proceedings of the Advanced Digital Libraries (ADL’96) Forum, pages 66–75, Washington D.C., 1996. 152 2. D. Cutting, D. Karger, J. Pedersen, and J. Tukey. Scatter/gather: A cluster-based approach to browsing large document collections. In Proceedings of SIGIR’92, Copenhagen, Denmark, 1992. 148 3. Y. Rubner, C. Tomasi, and L. Guibas. Adaptive color-image embeddings for database navigation. In Proceedings of ACCV, pages 104–111, Hongkong, 1998. 148 4. Y. Rui, T. Huang, and S.-F. Chang. Image retrieval: Current techniques, promising directions and open issues. Journal of Visual Communication and Image Representation, 10:1–23, 1999. 147 5. Y. Rui, T. Huang, M. Ortega, and S. Mehrotra. Relevance feedback: A power tool in interactive content-based image retrieval. IEEE Trans on Circuits and Systems for Video Technology, 8(5):644–655, 1998. 148 6. G. Salton and M. McGill. Introduction to Modern Information Retrieval. McGrawHill, New York, 1983. 151

Filter Image Browsing

155

7. S. Santini and R. Jain. Beyond query by example. In Proceedings of the Sixth ACM International Multimedia Conference, pages 345–350, Bristol, England, 1998. 148 8. J. Vendrig, M. Worring, and A. Smeulders. Filter image browsing. Technical Report 5, Intelligent Sensory Information Systems, Faculty WINS, Universiteit van Amsterdam, 1998. 152

Visualization of Information Spaces to Retrieve and Browse Image Data Atsushi Hiroike1 , Yoshinori Musha1 , Akihiro Sugimoto1 , and Yasuhide Mori2 1

2

Information-Base Functions Hitachi Laboratory, RWCP c/o Advanced Research Laboratory, Hitachi Ltd., Hatoyama, Saitama 350-03, JAPAN. Tel. +81-492-96-6111, Fax. +81-492-96-6006 {he,sha,sugimoto}@harl.hitachi.co.jp Information-Base Functions Tsukuba Laboratory, RWCP [email protected]

Abstract. We have developed a user interface for similarity-based image retrieval, where the distribution of retrieved data in a high-dimensional feature space is represented as a dynamical scatter diagram of thumbnail images in a 3-dimensional visualization space and similarities between data are represented as sizes in the 3-dimensional space. Coordinate systems in the visualization space are obtained by statistical calculations on the distribution of feature vectors of retrieved images. Our system provides some different transformations from a high-dimensional feature space to a 3-dimensional space that give different coordinate systems to the visualization space. By changing the coordinates automatically at some intervals, a spatial-temporal pattern of the distribution of images is generated. Furthermore a hierarchical coordinate system that consists of some local coordinate systems based on key images can be defined in the visualization space. These methods can represent a large number of retrieved results in a way that users can grasp intuitively.

1

Introduction

In recent years, image retrieval systems based on similarity have been reported by many researchers and companies [1], [2], [3]. We have been studying this issue from two points of view, i.e. “metrization” and “visualization.” For metrization, we are developing pattern recognition technologies applicable to a large-scale database containing various types of images. For visualization, our purpose is to find a suitable user interface for image retrieval systems, which is the main topic in this report. The importance of visualization in similarity-based image retrieval systems has been discussed by several researchers [4]. We will report our prototype system that uses the dynamical 3-dimensional representation.

2 2.1

Basic Concepts Retrieval System as an Extension of Human Perception

In conventional retrieval systems the conditions of queries are well-defined in an algorithm, e.g. “documents including a keyword”, “data created in 1998”, . . . . Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL’99, LNCS 1614, pp. 155–163, 1999. c Springer-Verlag Berlin Heidelberg 1999


156

Atsushi Hiroike et al.

In similarity-based systems, the definitions of conditions are not clear, because similarity is a subjective or psychological criterion. It changes among individuals and can change within a single individual. A user may not have a definite criterion; that is, the condition of a query may be obscure. Unfortunately, we have not yet developed enough technology to give a general definition of the similarity applicable to an arbitrary image. Therefore it is not guaranteed that a user will be satisfied with a decision of the retrieval system. Furthermore, it is not certain that a user would be satisfied even if the system were as intelligent as a human being, since he or she might not want to accept the decision of another person. Our main objective is to develop a similarity-based retrieval system which allows users to encounter as many data as possible. This is in contrast to the concept of conventional retrieval systems where it is important to reduce the number of retrieved results appropriately. A computer system is inferior to a human being at pattern recognition, but superior in processing a large amount of data. Our system should present an understandable representation of a mass of data to users. It will enable a user to grasp a large information space that cannot be directly perceived. That is, the system will extend the user’s perception system. The system should assist users to make their queries clear and also be adaptable to changes of their intentions. It should not discourage users but should enhance their imagination. 2.2

Representation of Metric Information

In our system, as in other similarity-based retrieval systems, each image is represented as a vector or a set of vectors, where each vector is a feature characterizing a specific property of the image. In many cases, the dimensionality of the vector is high. Our basic idea for the user interface is to use visualization of the metric space that consists of feature vectors. The 2- or 3-dimensional representation of a data distribution in a highdimensional space is not a new topic, but a classical issue in multivariate analysis. However, whereas the conventional usage applies multivariate analysis to represent the static structure of data, we use it to represent the status of the system changed by a user’s operation. In our system, the resultant representation is interactive and dynamic. A user can give a requirement to the system through the visualization space; the requirement induces a change in the internal state of the system; and this change is reported to the user in the visualization space. The internal state of the system consists of abstract quantities, e.g. similarities defined in a high-dimensional space, which are difficult for users to understand. The visualization transforms these quantities into visible ones that can be perceived directly by users. In the desktop-metaphor where logical or symbolic relationships among objects in file systems are visualized, the intuitive operations of files and directories are available. In our system where metric relationships among pattern information are visualized, the intuitive operations in retrieving and browsing image data will be available.

Visualization of Information Spaces to Retrieve and Browse Image Data

157

Most similarity-based retrieval systems present retrieved images as a list sorted according to the similarity. This is quasi-1-dimensional visualization, which represents the order of the similarity between the key image and the results. In the previous report, we proposed a system using 2-dimensional visualization [5], [6]. In that system, retrieved results are displayed as a scatter diagram of thumbnail images in a 2-dimensional space constructed from features used in the similarity calculation. The coordinates of the space are eigenvectors given by applying principal component analysis (PCA) to the feature vectors of the retrieved results. Therefore the space is optimal in the second order to represent a distribution of retrieved data. List-representation may gather similar images in the higher ranks. However, it loses the impression of orderliness in the lower ranks because adjacency among the retrieved images has no meaning. So a user will be unhappy if given a long list. In the 2-dimensional representation, the configuration of data represents the similarity among retrieved data, and gives a well-ordered impression to users. Consequently the system can report more than 100 data without boring a user.

3 3.1

Model of the Metrization Features

Color-based feature A histogram of color distribution in an image is calculated by dividing the color space into Nr × Ng × Nb boxes. If compositional information is needed, an image is divided into Nx ×Ny rectangular areas and multiple histograms are calculated. In this case, the number of dimension is Nr × Ng × Nb × Nx × Ny . Gradient-based feature This feature is based on a direction-distribution of gradient vectors in a gray-scale image. Suppose that the directions of vectors with a range −π/2 ≤ θ < π/2 are quantized into Nθ levels. Let v = {vk |k = 0, . . . Nθ −1} be a vector in which accumulation results are stored and (fx , fy ) be a gradient vector at a pixel. vk is updated as vk → vk + fx2 + fy2 , where k is determined by
the direction of (fx , fy ). Finally, each element of v is normalized as vk → vk /S, where S is the number of pixels used in the updating. In the same way as for the color-based feature, features can be extracted in Nx × Ny separated areas of an image. Furthermore it is well known that the resolution of an image is important in this type of feature. If Nl -level pyramidal data consisting of gray-scale images with different resolutions are given, the number of dimensions of the feature vector is Nθ × Nx × Ny × Nl . Usually, we reduce the dimensions of these features by using PCA on the whole of database. For example, a 1024-dimensional color-feature (Nr = Ng = Nb = Nx = Ny = 4) and a 512-dimensional gradient-feature (Nθ = 8,Nx = Ny = 4, Nl = 4) are transformed into 100-dimensional feature vectors respectively. This reduction of dimensions is necessary to perform statistical calculations depending on the retrieval.

158

3.2

Atsushi Hiroike et al.

Similarity

The definition of similarity between two images is based on the squared distance. Let Nf be the number of feature types, and {xi } and {yi } (i = 0, . . . , Nf − 1) be sets of feature vectors of an image X and an image Y respectively. The similarity between X and Y is defined as    2 wi xi − yi  , (1) s(X, Y ) = exp − i

where each squared distance is assumed to be normalized appropriately,  xi − yi 2 = 1 for all i, and w = {wi } is a non-negative weight vector. Our system allows a user to retrieve images using multiple keys which we call “reference images.” A user can also arbitrarily divide reference images into some groups, each of which we call a “cluster.” The similarity between an image X and a cluster C is defined as s(X, C) = min s(X, Yi ) . Yi ∈C

(2)

A weight vector w is assigned to a cluster. Increasing the total value of w decreases the similarities within a cluster. This is counterintuitive for users. Therefore we transform a weight vector W specified on the user level into w so that an increase in the mean value of W increases the similarities in the cluster. The −2 definition of w is wi = Nf W Wi , where W = i Wi . In addition, each cluster has an attribute of “positive” or “negative.” While in the normal case clusters are positive, an image in a negative cluster is what a user does not want included in retrieved data. Let the similarity of an image X given by the system be s(X), and let C be the nearest cluster to X. s(X) = −s(X, C) if C is negative; otherwise, s(X) = s(X, C). The retrieved results that the system finally reports are limited by the threshold of s(X) and the maximum number of retrieved data. 3.3

Statistical Calculation

To obtain appropriate visualization, we perform statistical calculations depending on the retrieved results. The modified statistics based on retrieved data are calculated within each cluster. Let {Xi } be a set of retrieved images whose nearest cluster is C, and let {xij } (j = 0, . . . , Nf ) be feature vectors of Xi , which are not necessarily identical to the features used in eq. 1. The statistics of the j-th feature in C are defined as  p s(Xi , C) (3) N = i

µj = N −1



p

s(Xi , C) xij

(4)

s(Xi , C)p (xij − µj ) (xij − µj )t .

(5)

i

Σj = N −1

 i

For p = 0, this is identical to the statistics of retrieved data used in our previous model. For p → ∞, this is the statistics of the reference data.

Visualization of Information Spaces to Retrieve and Browse Image Data

159

a. View of the user interface.

b. Spatial-temporal pattern of a distribution of images.

c Fig. 1. The user interface of our system (sample images 1996 PhotoDisc, Inc).

4

Main features of the user interface

In the prototype system, a user selects key images from previously retrieved results (the system usually provides some sample keys at the first step). Fig. 1-a shows a view of the user interface. The background is the visualization space, where the dynamical 3-dimensional scatter diagram of thumbnail images is displayed. In this case, up to 1,000 images are retrieved from among about 10,000. There are some panels at the front. The top one is a “collection panel” where a user keeps his favorite images, which has no effect on the retrieval. The panel displayed at the bottom is a “control panel” equipped with GUI components for controlling the view of the visualization space. The other panels are cluster panels. If a user selects a reference image by clicking the mouse,

160

Atsushi Hiroike et al.

the image is displayed on a cluster panel specified by the user. In this example, the front one is a positive cluster panel, and the back one is a negative cluster panel. If the user considers a retrieved image to be an error or noise, he can add it to the negative cluster panel. The sliders on the cluster panels are for setting the weights of features. The main features of the user interface are as follows. Representation as a spatial-temporal pattern Many transformations from a high-dimensional feature space to a 2- or 3-dimensional visualization space are possible. In the previous system using 2-dimensional representation, a user selects a combination of coordinates by himself. In the present system, the coordinate system changes automatically at some intervals (Fig. 1-b). The system provides a series of 3-dimensional coordinate systems. An element of the series determines the position of each image at a time step. The positions between two adjacent steps are defined by linear interpolation between the positions in the two different coordinate systems. This makes the independent motion of each image and a 3-dimensional spatial-temporal pattern of an image distribution. The buttons in the top left-hand corner of the “control panel” show the current step, and clicking them causes the view to jump to an arbitrary step. The upper one of the two sliders below the buttons controls of the speed of motion, and the lower one is for setting the view at an arbitrary point between two adjacent steps while pausing the motion. Representation of similarity The similarity of each retrieved image is represented as the size of the image in the 3-dimensional space. The function of the transformation from a similarity to a size has two parameters S = βsα , where α determines the intensity of the effect of the similarity and β is a general scaling parameter. The GUI component on the left side of the “control panel,” which is the rectangular area with the small black circle, is the controller for this function. The x-y coordinates of the area correspond to the parameters of α and β. Local coordinate systems and hierarchical representation Fig. 2 illustrates the local coordinate systems based on reference images. Reference images are located on a global coordinate system, and each of the retrieved results is located on a local coordinate system around the most similar reference. In the case of Fig. 1-a, the statistics of retrieved data were calculated (eq. 5), and the eigenspaces of color-based and gradient-based features were used as local and global coordinates respectively. The local coordinates enable intuitive representation of the relationship between a key image and each retrieved image, and more structured representation in displaying a large number of images. The coordinates used in visualizing are defined as a cluster-attribute. When some clusters are defined, the center of each cluster (the mean vector defined in eq. 4) is located on the root coordinate system, which is also constructed from the feature vectors. Walk-through functions The arrow-shaped buttons on the right of the control panel are GUI components for the walk-through: x or y-translations,

Visualization of Information Spaces to Retrieve and Browse Image Data

161

z-translations, and rotations. Our system provides two types of the view rotations: a viewer-centered rotation and an object-centered rotation. The latter allows a user to rotate around a selected image. The changing of similarities is caused by adding or removing reference images or by changing the weights of features. For example, if one selects an image belonging to an island of images as a new key, the image becomes larger and separates from the the island. Similar images also become larger (some of them gradually appear in the visualization space). They crowd around the key image and make a new island of images. If one removes a key image, the island around it disappears. The similar images become smaller and some of them which are not similar to all key images disappear from the space.

Fig. 2. Local coordinate systems.

5

Outline of the System Architecture

The software of the system consists of two server programs and client programs. An M-server (“metrization server”) performs various kinds of calculations based on features. A V-server (“visualization server”) displays a virtual 3-dimensional space. The client programs have usual GUI components and accept a user’s action except for some events which occur in the visualization space. The Mserver keeps the features of the whole images as compressed data on its memory. In response to a request from a client program, the M-server performs the image retrieval, generates the motion routes of retrieved images with the statistical calculations, and sends the series of routes to the V-server. The V-server receives data from the M-server in parallel with updating the view. A V-server needs a large texture mapping memory to create the real-time motions of more than 1,000 images. So we used an Onyx2 (Silicon Graphics Inc.) with 64 MB of texture mapping memory. The most serious problem is that the V-server has to update a lot of objects in the visualization space every time the retrieval condition changes. This is different from the conventional visualization systems or from the user interface models using virtual reality whose contents

162

Atsushi Hiroike et al.

usually remain unchanged. In the worst case, the V-server has to load image files of all the retrieved data. For example, it takes about 8 sec. on our system to read 1,000 thumbnail (64 × 64) images (encoded as JPEG files), which degrades the real-time response to the user’s action. If all images are cached in memory, a similar problem occurs because it is necessary to register the data in the texture memory. To minimize the degradation of the real-time response, the Mserver sorts the data to be sent with respect to its importance defined by the combination of the present similarities and the change in similarities from the last retrieval. If a new request for updating contents is made while the route data is being received, the M-server discards the remainder.

6

Discussion and Conclusions

In this paper, we reported our image retrieval system that uses the dynamical 3-dimensional visualization, which allows users to access a large information space. Compared with the 2-dimensional visualization system in our previous report, the present system is superior in showing a lot of retrieved results to users without boring them, and in its intuitive representation of the internal state of the system. Cost is a clear and simple problem in this system, which needs a high-end graphics workstation, whereas our 2-dimensional version can run on a PC. The more critical problem is in the controllability of the virtual 3-dimensional space. A 3-dimensional representation is useful in browsing, but sometimes a user feels it is inconvenient for operations such as selecting data. A user has to operate a mouse, which is a 2-dimensional pointing device, in order to select an object in the virtual 3-dimensional space. Furthermore in the present system, many functions of the user interface are supplied by conventional 2-dimensional GUI components. A user has to frequently move the cursor from the visualization space to the GUI panels, and sometimes he feels frustration. So we have to develop new methods of data selection and data input to make a user interface with 3-dimensional representation that is easy to operate.

References 1. Flickner, M., et.al., “Query by image and video content: the QBIC system,” IEEE Computer, 28-9, 23–32, 1995. 155 2. Faulus, D.S., Ng, R.T., “An expressive language and interface for image querying,” Machine Vision and Applications, 10, 74–85, 1997. 155 3. Stricker, M., Dimai, A., “Spectral covariance and fuzzy regions for image indexing,” Machine Vision and Applications, 10, 66–73, 1997. 155 4. Gupta, A., Santini, S., Jain R., “In search of information in visual media.” Communication of the ACM, 40-12, 35–42, 1997. 155 5. Musha, Y., Mori, Y., Hiroike, A.: Visualizing Feature Space for Image Retrieval (in Japanese), Procs. of the 3rd Symposium on Intelligent Information Media, 301–308, 1997. 157

Visualization of Information Spaces to Retrieve and Browse Image Data

163

6. Musha, Y., Mori, Y., Hiroike, A., Sugimoto, A.: “An Interface for Visualizing Feature Space in Image Retrieval,” Machine Vision and Applications, 447–450, 1998. 157

Mandala: An Architecture for Using Images to Access and Organize Web Information Jonathan I. Helfman 1

AT&T Labs - Research, Shannon Laboratory Human Computer Interaction Department Room B255, 180 Park Avenue, Florham Park, NJ 07932-0971, USA 2 University of New Mexico, Computer Science Department Room FEC 313, Albuquerque, NM 87131, USA [email protected] http://www.cs.unm.edu/∼jon/

Abstract. Mandala is a system for using images to represent, access, and organize web information. Images from a web page represent the content of the page. Double-clicking on an image signals a web browser to display the associated page. People identify groups of images visually and share them with Mandala by dragging them between windows. Groups of image representations are stored as imagemaps, making it easy to save visual bookmarks, site indexes, and session histories. Image representations afford organizations that scale better than textual displays while revealing a wealth of additional information. People can easily group related images, identify relevant images, and use images as mnemonics. Hypermedia systems that use image representations seem less susceptible to classic hypertext problems. When image representations are derived from a proxy server cache, the resulting visualizations increase cache hitrates, access to relevant resources, and resource sharing, while revealing the dynamic access patterns of a community.

1

Using Images to Represent Web Information

Most web software represents information textually. We use textual bookmarks and history lists in browsers, textual indexes and similarity metrics in search engines, and textual concept hierarchies in link taxonomies. Textual representations and organizations shape our experience of interacting with information [13]. On the web, a selectable image is a link to another resource. Imagemaps link to multiple resources. In general, a selectable image contains more data than a typical string of selectable text and has the potential to provide more of an indication if the link is relevant and worth following. In many cases, images from a web page represent the content of the page. Even the images used for decoration, navigation, and advertisement often provide additional characterizations of a web site (although these can usually be recognized and suppressed). Web images may represent their context for many reasons. There is a rich history of using images to illustrate and illuminate Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL’99, LNCS 1614, pp. 163–171, 1999. c Springer-Verlag Berlin Heidelberg 1999


164

Jonathan I. Helfman

Fig. 1. Mandala client displaying three groups of representations.

manuscripts. Most technical documents contain highly descriptive illustrations, charts, or diagrams. On the web, image formats were standardized before audio or text stylesheets. As a result images are not only used for illustration, but they have become a common strategy for site differentiation. Mandala lets people visualize large groups of web pages by displaying selectable thumbnails of the pages’ images. Groups may be determined in many ways: the URLs in a bookmark file, the history of a browsing session, the results of a query, etc. Mandala’s displays function as visual interactive indexes; they provide an overview of large amounts of information without sacrificing access. Mandala automatically builds groups of representations and provides a user interface for viewing and editing groups of representations and saving them as imagemaps. A snapshot of a Mandala client is shown in Fig. 1. Creating a system that uses images to represent web information requires solving several technical problems. While digital text enjoys an efficient representation, digital images require more memory and bandwidth, as well as support for compression, decompression, and scaling. Web-based systems also require support for HTTP monitoring and HTML parsing. Additional requirements include the ability for people to identify image groups visually and share them with the system easily. These technical challenges have been addressed by Mandala’s modular architecture, which provides a flexible and general platform for visual information research on the web. Mandala’s architecture allows it to function in multiple ways (e.g. as a GUI for organizing and maintaining information represented by images, a visual bookmark facility, an imagemap editor, a cache visualization tool, and a visual look-ahead cache).

Mandala: An Architecture for Using Images

165

Mandala has been fully implemented and is almost fully operational. Mandala’s image server (see Sec. 2.1) is used to generate imagemaps for CoSpace, an experimental VRML system at AT&T Labs. Mandala’s proxy server (see Sec. 2.2) is used to monitor web browsing sessions for PadPrints, a web navigation tool, which builds multi-scale maps of a user’s browsing history[9]. The remainder of this paper describes Mandala’s architecture, as well as some preliminary observations about using image representations.

2

Architecture and Implementation

Figure 2 illustrates Mandala’s component structure. White boxes represent the Mandala components. Solid grey shapes represent standard components that are used without modification (e.g. web servers and browsers). Light grey boundaries indicate runtime constraints. Arrows indicate the exchange of HTTP-like messages consisting of ASCII headers and optional ASCII or binary data.

Fig. 2. Mandala’s Component Structure. Mandala is less concerned with supporting visual queries through feature extraction and indexing than previous Visual Information Systems[11]. Mandala’s architecture does not preclude a component for image analysis and feature extraction, but web images are surrounded by a rich context of meta-information, which seems to provide an ample feature set for indexing[19]. 2.1

Imago: Mandala’s Image Server

Mandala’s image server, called Imago, has been developed to support fast image data compression and decompression, image shrinking, imagemap creation, and

166

Jonathan I. Helfman

image meta-data extraction (i.e. information about the image that is normally hidden in the image’s header, such as its dimensions or total number of colors). While the other Mandala components are written in Java, Imago is written in C, both for speed and reliability. Publicly available C code to read and write GIF and JPG images is relatively robust[3,10] compared to the Java image decompression classes, which frequently throw undocumented exceptions when trying to decode GIF or JPG variations that they don’t fully support. Imago creates thumbnails and imagemaps according to client specifications. A minimal thumbnail specification is a URL for the input image, which causes Imago to use default scale factors and filtering functions. Clients can specify the maximum thumbnail dimension, filter function, return style (whether to return the URL or the data), and replacement style (whether to overwrite an exiting thumbnail of the same name or generate a new name). Additional options ignore input images that don’t meet minimal requirements for size or number of colors. Imago’s default filter function uses a form of hierarchical discrete correlation (HDC) that scales the input by one half[5]. To scale images by arbitrary amounts, Imago uses successive invocations of HDC and a single invocation of more general image rescaling algorithm that creates and scales filter kernels based on the scale factor[20]. The possible slowness of the final pass is minimized because scale factors are guaranteed to be less than one half. A minimal imagemap specification is a stub name for the output and a list of URL pairs for web images and associated resources. The imagemap command supports each of the thumbnail options described above, as well as layout style (grid, tile, random, user-set) and background style (solid color, tiled image). Imago also rates imagemaps according to a heuristic that attempts to identify images that are used for decoration, navigation, or advertisement. The rating scheme is based on image meta-information as opposed to image content. It ranks large, square images with many colors higher than small, wide or tall images with few colors. The rate of an imagemap is the average rate of the imagemap’s thumbnails. Ratings allow clients to identify imagemaps that are most likely to contain useful representations. 2.2

Mirage: Mandala’s Proxy Server

Mandala’s proxy server, called Mirage, has been developed to support local caching of images, transparent web browser monitoring, and HTML parsing. A proxy server is a program that sits between web servers and web clients, such as browsers (see [1] and [12] for good introductions to proxy servers). Requests are copied from clients to servers. Responses, if any, are copied back to clients and possibly cached, in case they are requested again. Proxy servers provide several benefits, such as reduced web latency and increased effective bandwidth[12], increased web access (when the destination server is down, but the resource is cached), and savings on long-distance connection charges[15]. The two main problems for a proxy server implementation are how to determine if a cached file is fresh and which files to remove when the cache gets full. Freshness is difficult to determine because few servers use the Expires HTTP

Mandala: An Architecture for Using Images

167

Fig. 3. Snapshot of a Dynamic Cache Visualization. header and there is no way for a destination server to inform a proxy server when a resource has been updated [14]. When the Expires header is not used, Mirage uses a typical hueristic, which estimates freshness time as a factor of the last-modified time. If the server does not transmit a last-modified time, Mirage assigns the resource a maximum age. When the cache is full, Mirage uses a Least-Recently Used (LRU) algorithm for removing old files from the cache [18]. Researchers disagree about the performance of LRU [1,17,18], however LRU has a more efficient implementation than algorithms that need to compute scores and insert into sorted lists [8]. Mirage improves Mandala’s performance by caching thumbnails and imagemaps generated by Imago and installed on a web server. Mandala also benefits from Mirage’s extensions. Proxy server extensions are a recommended strategy for creating utilities and applications that enhance the experience of web browsing [2,4]. Mirage has been extended to allow registered applications to monitor web browsing sessions[9]. Monitoring with a proxy server is transparent to users who need only configure their browser to use the proxy server via a commandline option or preference setting. Mirage has also been extended to describe its cache and parse HTML, capabilities that Mandala uses to identify references to links and images in the cached pages. A snapshot of a cache visualization is shown in Fig. 3. Displaying selectable images from a proxy server cache has several surprising implications. Cache visualizations improve the cache hit-rate (i.e. the percentage of requests that can be satisfied out of the cache) more than algorithmic approaches, which rely on the chance that people will unknowingly request a page that someone else has requested[1]. Cache visualizations increase access to relevant information. When people can see the contents of the cache, and select an image to access an associated page, they are more likely to access information that is relevant to

168

Jonathan I. Helfman

their needs or interests. When displaying images as they are cached, the cache visualizations reveal dynamic usage patterns of entire communities and promote unprecedented sharing of resources. Streams of similar images represent active hypertext trails of anonymous community members (e.g. the cars in Fig. 3). If any trail is of interest, selecting an image in the trail allows an observer to become a participant immediately, blazing a new trail, which is soon displayed as an additional stream of images in the communal montage. 2.3

Mandala Server

The Mandala Server automatically groups images based on origin (e.g. images from the same web server or from the same browsing session), but could be extended to use similarity of contextual information[19] or extracted features[11]. The Mandala Server also groups images from pages that are reachable from the page most recently requested by the user’s web browser, a capability that allows a Mandala client to function as a visual look-ahead cache. Web searches initiated with any search engine define groups of images from pages that match the query. The Mandala Server communicates with multiple Mandala clients by posting messages to a bulletin-board and sending the clients brief messages that they should check the bulletin-board when they have a chance. This protocol prevents the server from swamping the clients with a flood of messages and image data. Clients are free to read updates when user-activity decreases. 2.4

Mandala Client

Each Mandala client supports automatic layout and animation of image representations. User interactions include image selection, image positioning, and animation control. Clients may be full-fledged applications or Java applets (with slightly reduced functionality). Mandala clients communicate only with a single Mandala server. Double-clicking on an image causes the client to signal the Mandala server, which signals a web browser (via its client API) to display the associated page. Mandala clients use separate windows for each group of representations (see Fig. 1). People edit imagemaps by repositioning thumbnails. People edit groups by dragging and dropping thumbnails between windows. People define new groups by dragging thumbnails into an empty window.

3

Preliminary Observations about Image Representations

With the increased access afforded by viewing large groups of image representations comes associated copyright and privacy concerns. Taking images off web pages may seem to be a copyright violation, but if they are used as an interactive index that promotes access to the information, then they are no more of an infringement than the textual indexes built by search engines. In both cases, the intent is to promote access to original information and encourage the livelihood

Mandala: An Architecture for Using Images

169

of the purveyors of the original information. Privacy becomes a concern when people do not wish to share information about their browsing behavior with a community. In this case, they may choose not to use a proxy server, but then they will not benefit from the communal cache. Image representations seem to diminish the effect of several classic hypertext problems. For example, spatial disorientation is caused by unfamiliarity with possibly complex hypertext structures[6], while cognitive overhead is caused by the general level of complexity associated with multiple choices[7]. Because image representations deemphasize hypertext structure, people navigate through clusters of similar images instead of structures of hypertext links. Images may provide better navigational cues than textually-labeled links because images have been shown to improve human memory for associated information[16]. Other hypertext problems include lack of closure, the inability to determine which pages have been visited or if any nearby unvisited pages are relevant, and embedded digression, the inability to manage multiple, nested digressions[7]. Because people have a remarkable memory for images, they can distinguish quickly between familiar and unfamiliar images[21]. A system that uses image representations may therefore help people identify new information when seeing an unfamiliar image and find previously accessed information by remembering and locating a familiar image. The hypertext problem of trail-blazing, the inability to determine if a link is worth following, may be diminished when using image representations because images have the potential to provide a better indication if a link is worth following than brief textual labels. The hypertext problem of session summarization, the inability to save the state of a browsing session, is alleviated by storing the image representations associated with a browsing session as an imagemap, which provides a visual session summary that preserves the interactive nature of the browsing experience.

4

Conclusions

This paper describes Mandala, a system that provides visual interactive overviews of large amounts of information by using images from web pages to represent those pages (Sec. 1). Mandala’s architecture is discussed and distinguished from earlier Visual Information Systems that support image indexing and visual querying (Sec. 2). The use of a proxy server for transparent monitoring of web browsers is described, and cache visualizations are reported to improve cache hitrates and reveal dynamic communal access patterns (Sec. 2.2). In addition, image representations are shown to increase concerns over copyright and privacy while diminishing classic problems associated with hypertext (Sec. 3).

References 1. Marc Abrams, Charles R. Standridge, Ghaleb Abdulla, Stephen Williams, and Edward A. Fox. Caching proxies: Limitations and potentials. In Proceed-

170

2.

3. 4.

5. 6. 7. 8.

9.

10. 11.

12.

13. 14.

15.

16. 17. 18.

19.

Jonathan I. Helfman ings of the Fourth International World Wide Web Conference, December 1995. http://www.w3.org/pub/Conferences/WWW4/Papers/155/ . 166, 167 Rob Barrett and Paul Maglio. Intermediaries: New places for producing and manipulating web content. In Proceedings of the Seventh International World Wide Web Conference, 1998. http://wwwcssrv.almaden.ibm.com/wbi/www7/306.html. 167 T. Boutell. http://www.boutell.com/gd/. 166 Charles Brooks, Murray S. Mazer, Scott Meeks, and Jim Miller. Applicationspecific proxy servers as http stream transducers. In Proceedings of the Fourth International World Wide Web Conference, December 1995. http://www.w3.org/pub/Conferences/WWW4/Papers/56/. 167 Peter J. Burt. Fast filter transforms for image processing. Computer Graphics and Image Processing, 16(1):20–51, 1981. 166 Jeff Conklin. A survey of hypertext. Technical Report STP-356-86, MCC, February 1987. 169 Carolyn Foss. Tools for reading and browsing hypertext. Information Processing and Management, 25(4):407–418, 1989. 169 Jonathan Helfman. Insights and surprises using image representations to access and organize web information. AT&T Web Implementor’s Symposium, July 1998. http://www.cs.unm.edu/ jon/mandala/wis98/wis98.html. 167 Ron Hightower, Laura Ring, Jonathan Helfman, Ben Bederson, and Jim Hollan. Graphical multiscale web histories: A study of PadPrints. In Hypertext ’98 Proceedings, pages 58–65, 1998. 165, 167 Independent JPEG Group. ftp://ftp.uu.net/graphics/jpeg/. 166 Clement H. C. Leung and W. W. S. So. Characteristics and architectural components of visual information systems. In Clement Leung, editor, Visual Information Systems, Lecture Notes in Computer Science 1306. Springer, 1997. 165, 168 Ari Loutonen and Kevin Altis. World-wide web proxies. In Proceedings of the First International Conference on the World-Wide Web, WWW ‘94, 1994. http://www1.cern.ch/PapersWWW94/luotonen.ps. 166 Marshall McLuhan. The Gutenberg Galaxy: the Making of Typographic Man. University of Toronto Press, 1962. 163 J. C. Mogul. Forcing http/1.1 proxies to revalidate responses, May 1997. http://www.es.net/pub/internet-drafts/ draft-mogul-http-revalidate-01.txt. 167 Donald Neal. The harvest object cache in New Zealand. In Proceedings of the Fifth International World Wide Web Conference, 1996. http://www5conf.inria.fr/fich html/papers/P46/Overview.html. 166 Allan Paivio. Imagery and Verbal Processes. Holt, Rinehart, & Winston, 1971. 169 Tomas Partl and Adam Dingle. A comparison of www caching algorithm efficiency. http://webcache.ms.mff.cuni.cz:8080/paper/paper.html. 167 James E. Pitkow and Margaret M Recker. A simple yet robust caching algorithm based on dynamic access patterns. In Proceedings of the Second World Wide Web Conference, 1994. http://www.ncsa.uiuc.edu/SGD/IT94/Proceedings/DDay/pitkow/ caching.html. 167 Neil C. Rowe and Brian Frew. Finding photograph captions multimodally on the world wide web. In AAAI-97 Spring Symposium Series, Intelligent Integration and Use of Text, Image, Video, and Audio Corpora, pages 45–51, March 1997. 165, 168

Mandala: An Architecture for Using Images

171

20. Dale Schumacher. General filtered image rescaling. In David Kirk, editor, Graphics Gems III, pages 8–16. Academic Press, 1994. 166 21. Lionel Standing, Jerry Conezio, and Ralph Norman Haber. Perception and memory for pictures: Single-trial learning of 2500 visual stimuli. Psychonomic Science, 19(10):73–74, 1970. 169

A Compact and Retrieval-Oriented Video Representation Using Mosaics Gabriele Baldi, Carlo Colombo, and Alberto Del Bimbo Universit` a di Firenze Via Santa Marta 3, I-50139 Firenze, Italy

Abstract. Compact yet intuitive representations of digital videos are required to combine high quality storage with interactive video indexing and retrieval capabilities. The advent of video mosaicing has provided a natural way to obtain content-based video representations which are both retrieval-oriented and compression-efficient. In this paper, an algorithm for extracting a robust mosaic representation of video content from sparse interest image points is described. The representation, which is obtained via visual motion clustering and segmentation, features the geometric and kinematic description of all salient objects in the scene, being thus well suited for video browsing, indexing and retrieval by visual content. Results of experiments on several TV sequences provide an insight into the main characteristics of the approach.

1

Introduction

Quite recently, the rapid expansion of multimedia applications has encouraged research efforts in the direction of obtaining compact representations of digital videos. On the one hand, a compact video encoding is required for high quality video storage; on the other hand, an ad hoc video representation needs to be devised at archival time in order to ease video browsing and content-based retrieval. Past approaches to video compression (see e.g., the standards MPEG 1 and 2) have privileged image processing techniques which, taking into account only the signal-level aspects of visual content, emphasize size reduction over retrieval efficiency. More recently, they have been presented browsing-oriented computer vision techniques to represent videos by reconstructing the very process of film making [4]. These techniques are capable to segment the video into a number of “shots,” each delimited by film editing effects such as cuts, dissolves, fades, etc. The description of shot content relies then on the extraction of salient “keyframes.” However, the above techniques have the limitation of providing only a partial information of video content, being it impossible to reconstruct a video only from its keyframes. The advent of mosaicing techniques [7,8] paved the way to content-based video representations which are both retrieval- and compression-efficient. Such techniques reduce data redundancy by representing each video shot through a single patchwork image composed using all of its frames. Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL’99, LNCS 1614, pp. 171–178, 1999. c Springer-Verlag Berlin Heidelberg 1999


172

Gabriele Baldi et al.

In this paper, a method to represent video content through image mosaics is described. Mosaics are extracted from video data through corner-based tracking and a 2D affine motion model. An original motion clustering algorithm, called DETSAC, is proposed. The obtained video representation features the image and motion description of all salient objects in the scene, and is well suited to both video browsing and retrieval by visual content.

2

Video Segmentation

The primary task of video analysis is video editing segmentation, i.e. the identification of the start and end points of each shot. Such a task implies solving two problems: i) avoiding incorrect identification of shot changes due to rapid motion or sudden lighting change in the scene; ii) detect sharp (cuts) as well as gradual transitions (dissolves). To avoid false shot change detection, a correlation metric based on HSI color histograms is used, which is highly insensitive even to rapid continuous light variations while maintaining reliable to detect cuts. To detect dissolves, a novel algorithm based on corner statistics is used, based on monitoring the minima in the number of salient points detected. During a dissolve, while the previous shot gradually fades out and its associated corners disappear, the new one fades in, its corners being still under the saliency threshold [3].

3

Shot Analysis

Once a video is segmented into shots, each shot is processed so as to extract its 2D dynamic content and allow its mosaic representation. Image motion is processed between successive frames of the shot via a three-step analysis: (1) corner detection; (2) corner tracking; (3) motion clustering and segmentation.

Fig. 1. Corner detection. The pixel locations corresponding to extracted corners are shown in white. Corner Detection. An image location is defined as a corner if the intensity gradient in a patch around it is not isotropic, i.e. it is distributed along two preferred directions. Corner detection is based on the algorithm originally presented by Harris and Stephens in [6], classifying as corners image points with

A Compact and Retrieval-Oriented Video Representation Using Mosaics

173

large and distinct values of the eigenvalues of the gradient auto-correlation matrix. Fig. 1(right) shows the corners extracted for the indoor frame of Fig. 1(left). Corner Tracking. To perform intra-shot motion parameters estimation, corners are tracked from frame to frame, according to an algorithm originally proposed by Shapiro et al. in [9] and modified by the authors to enhance tracking robustness. The algorithm optimizes performance according to three distinct criteria, namely: Frame similarity: The image content in the neighborhood of a corner is virtually unchanged in two successive frames; hence, the matching score between image points can be measured via a local correlation operator. Proximity of Correspondence: As frames go by, corner points follow smooth trajectories in the image plane, thus allowing to reduce the search space for each corner in a small neighborhood of its expected location, as inferred based on previous tracking results. Corner Uniqueness: Corner trajectories cannot overlap, i.e. it is not possible that at a same time two corners share the same image location. Should this happen, only the corner point with higher correlation would be maintained, while the other would be discarded.

Fig. 2. Corner tracking examples. 1st row: translation induced by camera panning; 2nd row: divergence induced by camera zooming; 3rd row: curl induced by camera cyclotorsion.

Since the corner extraction process is heavily affected by image noise (the number and individual location of corners varies significantly in successive frames; also,

174

Gabriele Baldi et al.

a corner extracted in one frame, albeit still visible, could be ignored in the next one), the modified algorithm implements three different corner matching strategies, ensuring that the above tracking criteria are fulfilled: – strong match, taking place between pairs of locations classified as corners in two consecutive frames; – forced match, image correlation within the current frame, in the neighborhood of a previously extracted corner; – backward match, image correlation within the previous frame, in the neighborhood of a currently extracted corner. These matching strategies ensure that a corner trajectory continues to be traced even if, in some instants, the corresponding corner fails to be detected. Fig. 2 shows corner tracking examples from three different commercial videos, and featuring diverse kinds of 2D motions induced by specific camera operations. Each row in the figure shows two successive frames of a shot, followed by the traced corners pattern. Motion clustering and segmentation. After corner correspondences have been established, an original motion clustering technique is used to obtain the most relevant motions present in the current frame. Each individual 2D motion of the scene is detected and described by means of the affine motion model 

 

 x a0 a1 a2 x −x (1) = + a3 a4 a5 y − y y characterizing image points displacements – (x, y) and (x , y  ) denote the coordinates of a same point in the previous and current frames, respectively. Motion clustering takes place starting from the set of corner correspondences found for each frame. A robust estimation method is adopted, guaranteeing on the one hand an effective motion clustering, and on the other a good rejection of false matches (clustering outliers). The clustering technique, called DETSAC (“DETerministic SAmple Consensus”), is an adaptation of the “RANdom SAmple Consensus” (RANSAC) algorithm ([5], see also [2]) to the problem of motion clustering. DETSAC operates as follows. For each trajectory obtained by corner tracking, the two closest corners trajectories are used to compute the affine transformation (i.e., the 6 degrees of freedom a0 , . . . , a5 ) which best fits the trajectory triplet (each corner trajectory provides two constraints for eq. (1), hence three non collinear trajectories are sufficient to solve for the six unknown parameters). The number of trajectories “voting” for each obtained transformation candidate determine the consensus for that candidate. Iterating the candidate search and consensus computation for all possible corner triplets, the dominant motion with maximum consensus is obtained. All secondary motions are iteratively computed exactly in the same way, after the elimination of all the corner points with dominant motion. The RANSAC algorithm is conceived to reject well outliers in a set of data characterized by a unimodal population. Yet, in image motion segmentation, it is highly probable that two or more data populations are presented at

A Compact and Retrieval-Oriented Video Representation Using Mosaics

175

a given time instant, corresponding to independently moving objects. In such cases, RANSAC is likely to produce grossly incorrect motion clusters. As an example, Fig. 3 shows that, when attempting to cluster data from two oppositely translating objects, RANSAC wrongly interprets the two motions as a single rotating motion.

Fig. 3. Motion clusters for two translating objects. Left: Ground truth solution. Right: RANSAC solution.

Although DETSAC is conceived to solve the multimodal distribution problem, however, it achieves this at the cost of a diminished parameter estimation accuracy (nearby trajectories tend to amplify ill-conditioning). Therefore, DETSAC is only meant to provide a rough motion estimate, to be refined later via an iterative weighted least squares strategy, where the higher is the departure of each individual observation from the current estimate, the lower is its associated weight. In such a way, the robustness to outliers of DETSAC is efficiently coupled with the estimation accuracy of least squares. Besides, the above clustering algorithm would exhibit a somewhat tendency to fragment each cluster into subclusters. To avoid that, a further cluster merging step is performed, to ensure that each new cluster is distant enough from previous clusters in the space of affine transformations [10]; if this does not happen, clusters below a minimum threshold are merged together. The distance between two affine transformations A = (a0 , . . . , a5 ) and B = (a0 , . . . , a5 ) is defined as 
2 l 2 2 2 2 d = (p0 + p1 + p3 + p4 ) + p22 + p25 , (2) 2 where pi = |ai − ai | for i = 0, . . . , 5, and l = (w + h)/2 is the average frame size. Qualitatively, eq. (2) expresses the displacement (in pixels) produced in the frame’s periphery as the effect of the difference between the motions A and B. Indeed, each addend under the square root expresses the contribution of each individual parameter to the overall displacement. Another important feature of the clustering algorithm is temporal subsampling. In fact, by limiting initially the motion analysis to every 16 or 32 frames,

176

Gabriele Baldi et al.

slow motions or motions very similar to each other can be succesfully detected and differentiated. Only in a second phase, the motion analysis is refined by iteratively halving the frame interval, until all the frames of a sequence are processed. The motion clusters obtained at higher subsampling levels are used as constraints for clustering refinement, so as to avoid that previously formed clusters are incorrectly merged.

Fig. 4. Residual errors and motion segmentation. The two motion clusters obtained are shown using white and black arrows, respectively. Figure 4 provides a clustering example obtained from a news video, including a man moving in an exterior. As the camera pans rightwards to track the man, the image background moves leftwards; hence, the shot features two main image motions (man, background). The upper row of Fig. 4 reports two successive frames, and the result of corner tracking; in the lower row, they are shown the residual segmentation errors relative to the background (dominant motion) and the man (secondary motion), respectively, together with the final clustering result. The actual motion-based segmentation is performed by introducing spatial constraints to the classes obtained via the previous motion clustering phase. Compact image regions featuring homogenous motion parameters – thus corresponding to single, independently moving objects – are extracted by region growing [1]. The motion segmentation algorithm is based on the computation of an a posteriori error obtained by plain pixel differences between pairs of frames realigned according to the extracted affine transformations.

4

Representation Model

The mosaic-based representation includes all the information required to reconstruct the video sequence, namely, – location of each shot in the overall sequence; – type of editing effect; – mosaic image of the background;

A Compact and Retrieval-Oriented Video Representation Using Mosaics

177

Fig. 5. Four frames from a commercial video, and the associated mosaic.

Fig. 6. Four frames from a news video, the associated mosaic and the foreground moving object.

Fig. 7. Four frames from a commercial video, the associated mosaic and the foreground moving object.

178

Gabriele Baldi et al.

– 2D motion due to the camera (for each frame); – 2D motion and visual appearance of each segmented region (also for each frame). Figure 5 shows some frames of a video sequence featuring composite horizontal and vertical pan camera movements, and the mosaic obtained. The mosaic image captures all the background details which are present at least in one frame of the sequence. Also, the mosaic composes individual frame details into a global description of the background (notice, e.g., that the babies are never visible all together in just one frame). Figures 6 and 7 show two more complicated examples, featuring camera panning and zooming and the presence of an independently moving objects. Notice, again, that both the car in Fig. 6 and the farm in Fig. 7 is almost integrally visible in the mosaic image, although it is not so in each individual frame. The mosaic representation also allows to “erase electronically” an object from a video (e.g., in Fig. 6, the man is segmented out from the background mosaic image).

References 1. D.H. Ballard and C.M. Brown. Computer Vision. Prentice-Hall, 1982. 176 2. T.-J. Cham, and R. Cipolla. A statistical framework for long-range feature matching in uncalibrated image mosaicing. In Proc. Int’l Conf. on Computer Vision and Pattern Recognition CVPR’98, pages 442–447, 1998. 174 3. C. Colombo, A. Del Bimbo, and P. Pala. Retrieval of commercials by video semantics. In Proc. Int’l Conf. on Computer Vision and Pattern Recognition CVPR’98, pages 572–577, 1998. 172 4. J.M. Corridoni and A. Del Bimbo. Structured digital video indexing. In Proc. Int’l Conf. on Pattern Recognition ICPR’96, pages (III):125–129, 1996. 171 5. M.A. Fischer and R.C. Bolles. Random Sample Consensus: A paradigm for model fitting with applications to image analysis and automated cartography. In Communications of the ACM, 24:381-395, 1981. 174 6. C.G. Harris and M. Stephens. A combined corner and edge detector. In Proc. 4th Alvey Vision Conference, pages 147–151, 1988. 172 7. M. Irani, P. Anandan and S. Hsu. Mosaic based representation of video sequences and their applications. In Proc. Int’l Conference on Computer Vision ICCV’95, pages 605–611, 1995. 171 8. H.S. Sawhney and S. Ayer. Compact representations of videos though dominant and multiple motion estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 18:814–830, 1996. 171 9. L.S. Shapiro, H. Wang and J.M. Brady. A matching and tracking strategy for independently moving, non-rigid object. In Proc. British Machine Vision Conference, pages 306–315. 173 10. J.Y.A. Wang and E.H. Adelson. Representing moving images with layers. IEEE Transaction on Image Processing, 3(5):625–638, 1994. 175

Crawling, Indexing and Retrieval of Three-Dimensional Data on the Web in the Framework of MPEG-7 Eric Paquet and Marc Rioux National Research Council Building M-50, Montreal Road Ottawa (Ontario) K1A 0R6 Canada [email protected], [email protected]

Abstract. A system for crawling, indexing and searching three-dimensional data on the Web in the framework of MPEG-7 is presented. After a brief presentation of the Web context, considerations on three-dimensional data are introduced followed by an overview of the MPEG-7 standard. A strategy to efficiently crawl three-dimensional data is presented. Methods designed to automatically index three-dimensional objects are introduced as well as the corresponding search engine. The implications of MPEG-7 for such a system are then analyzed from both the normative and non-normative point of view.

1. Introduction Few among us thought that the Web would encompass so many fields and applications and that there would be such an amount and diversity of information available. In order to be useful that information has to be crawled, indexed and retrieved. If it is not the case the information is lost. The information available on the Web presents two very important characteristics. The first one is the multimedia nature of the content, which can be made of texts, audio, pictures, video, threedimensional data, or any combination of them. The second characteristic is the hypermedia nature. It means that a given document can point to another and hopefully related document. So starting with a few key documents it is possible to cover gradually the network by linking from one document to the next. Crawling and indexing of pictures on the Web has received a lot of attention [1-4]. Threedimensional data have received some attention from the indexing point of view [5] but little if not from the crawling point of view. This paper intends to focus on the crawling, indexing and retrieval of three-dimensional objects.

2. Three-Dimensional Data on the Web The nature of three-dimensional date on the Web is manifold. A three-dimensional datum can be an object or a scene. They can be themselves represented in many formats. Some of them are based on triangular meshes while others are based on parametric representations like the non-uniform rational b-spline or NURBS. In some Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL'99, LNCS 1614, pp. 179-186, 1999.  Springer-Verlag Berlin Heidelberg 1999

180

Eric Paquet and Marc Rioux

cases a volumetric representation is used: the most common is the voxel representation. Some formats have only one resolution or level of detail while others have many. Some formats allow compression but if it is not the case it is always possible to compress the files. Finally they support ASCII, binary or even both representations. Currently there are more than 40 different formats on the Web. Some of them like ACIS are used in specialized application like CAD and most of them count for a small proportion of the total three-dimensional population of the Web. The most important format by far is VRML: shorthand for Virtual Reality Modeling Language. There are many reasons for that: the first one being that retrospectively VRML was probably the first three-dimensional non-CAD standard used by a wide range of companies and applications. VRML is currently available under three versions: VRML 1, 2 and 97. The latest is the ISO version of VRML 2. So not only VRML is a de facto standard but it is also a rightful international standard. For all those reasons we have decided to use VRML as our working format and internal representation in our system. In addition to geometry, three-dimensional files can describe the color distribution. The color can be represented as a set of vertices or as a set of texture maps. For vertices the RGB system is used most of the time. For texture maps it depends on the type of coding used for the maps. In the case of JPEG, a luminance-chrominance representation is used. The color information can be part of the file structure like in the Metastream format or it can be saved externally like in VRML. It has to be pointed out that VRML also support internally defined texture maps.

3. An Overview of MPEG-7 MPEG-7 [6] is formally known as the Multimedia Content Description Interface. MPEG-7 is not to be confused with his predecessors MPEG-1, MPEG-2 and MPEG°4. MPEG-7 describes the content of multimedia objects while MPEG-1, 2 and 7 code the content. MPEG-7 is actually under development and the international standard is expected for 2001. MPEG-7 is made of a normative and non-normative part. The normative part is what is standardized by MPEG while the non-normative part is what is out of the scope of the standard. The normative part of MPEG-7 is the description. The non-normative cover features extraction, indexing and search engine. The reason why MPEG-7 does not standardize those parameters is because they are not necessary for interoperability and because MPEG wants to keep the door wide open for future improvements MPEG-7 is made out of three main elements: the descriptors D, the descriptor schemes DS and the description definition language or DDL [6]. The descriptor is the representation of a feature: a feature being a distinctive or characteristic part of the data. The descriptor may be composite. The description scheme is formed by the combination of a plurality of descriptors and description schemes. The description

Crawling, Indexing and Retrieval of Three-Dimensional Data on the Web

181

scheme specified the structure and the relation between the descriptors. Finally the DDL is the language used to specify the description scheme. The reason why a standard is needed in the field of content-based access is very simple: interoperability. A search engine can efficiently utilize the descriptors only if their structure and meaning is known. If it is not the case the descriptors are in the worst case useless or in the best case misused.

4. The Crawler Before three-dimensional data can be described they need to be located. This process is usually known as crawling. In order to crawl the documents we use a commercial crawler, Web robot or spider made by Excalibur Corporation running on an NT workstation. The spider consists of 4 server programs and one client program running together in client-server architecture. Two of the server programs provide database functionality for the crawler. The other two server programs are the load balancer and the transfer processor. They provide the network transfer and data processing functionality for the crawler. The client program is where the user defines the domain of action of the spider. This is done through a series of configuration files. The O2 database is a commercial object oriented database used to store metadata for crawled documents. It is works in conjunction with the URL server to provide the database component of the spider. The transfer-processor program does the actual download and processing of web documents. This program can transfer and process data in parallel. The load balancer divides the work among the transfer-processors mentioned above. The spider program acts as the manager of a particular crawl job. It decides where the crawl should start from, where it should go, and when it should end. The user controls these behaviors through a set of highly configurable filters that constrains the crawling pattern. These filters act like intelligent decision point for choosing which forward links to follow and which not to follow. The Excalibur spider has not been designed to retrieve three-dimensional data. As a matter of fact this medium has not been taken into consideration in the design of the Web robot. The spider initiates the crawl from a set of locations specified in the configuration files. These URL are chosen for their richness in three-dimensional content and related hyperlinks. The spider does not have any filtering capability for three-dimensional files. Consequently the crawling process has to be divided in two steps. In the first step the location of all documents is saved but not content because the proportion of threedimensional files is very small compare to other types like HTML. Saving them would only overload the system disks. Once the spider has fished to cover the Web domain specified in the configuration files the locations are filtered. The filter keeps only the URL corresponding to three-dimensional files by looking at the file extensions. For example the extension wrl corresponds to the VRML format version

182

Eric Paquet and Marc Rioux

1, 2 and 97. The filter takes into account the fact that some files can be under a compress format and can end up with a zip, gzip or Z extension. Once the location list has been filtered a new list containing only URL of threedimensional files is generated. The spider retrieves those files by performing a second crawl. Files that are not in the VRML format are converted to this working format by a converter. We have two converters: the first one can handle more than 40 different commercial formats while the second one works on most CAD formats. Once the files have been converted to the working format, a set of descriptors is generated for each one of them by the analyzer. Some formats like VRML can link to external texture files. Because the spider was not design for three-dimensional models an additional process is needed to retrieve those files. The URL corresponding to the texture files is determined by analyzing the content of the three-dimensional file and then the corresponding texture file can be retrieved.

5. The Analyzer The analyzer parses each file, extracts the geometry, the color distribution and analyzes the content in order to generate the descriptors and DS. It also creates a small picture of the model corresponding to the file as well as an hyperlink to the file. In order to describe the geometrical shape we introduce the concept of a cord. A cord is a vector that goes from the center of mass of the object to the center of mass of a given triangle belonging to the surface of the object. We benefit from the fact that our working format VRML is based on a triangular mesh representation. In order to define the orientation of a cord we use a reference frame that does not depend on the particular orientation of the object. The reference frame is defined as the eigen vectors of the tensor of inertia of the object. Each axis is identified by the corresponding eigen value. The axes are labeled one, two and three by descending order of their eigen values. The cord orientation is completely determined by the two angles between the cord and the first two axis and the cords are uniquely defined by their modules or norms and those angles. We are interested in the distribution of those cords so we define the cord distribution as a set of three histograms: the first histogram represents the distribution of the first angle, the second histogram the distribution of the second angle and the third histogram the distribution of the norms. The histograms are normalized in order to make their comparison easier. Each histogram has a header made out of a single number representing the number of channels. The size of the histogram is also dictated by the precision of the representation and by the discrimination capability that is needed. The behavior of a cord can be better understood by considering a regular pyramid and a step pyramid. Most people agree that they belong to the same category. If normal vectors would be used to represent the pyramids, five directions

Crawling, Indexing and Retrieval of Three-Dimensional Data on the Web

183

would characterize the regular pyramid while six directions would characterize the step pyramid. So they would be classified as two distinct objects. If a cord representation is used the histograms corresponding to the regular pyramid and to the step pyramid are much more similar. Consequently a cord can be viewed as a slow varying normal vector. In addition to the cord, we propose another description based on a wavelet representation. This is because in addition to be bounded by a surface a threedimensional object is also a volume and should be analyzed as such. In order to analyzed the volume we use a three-dimensional wavelet transform. Let us review the procedure. For our purpose we use DAU4 wavelets which have two vanishing moments. The N×N matrix corresponding to that transform is c 0 c  3    W =     c 2 c  1

c1 − c2

c2 c1

c3 − c0

c0 c3

c1 − c2

c2 c1

c3 − c0 O c0

c1

c2

c3

− c2

c3

c1 c0

− c0

c3

        c3  − c0   c1  − c 2 

(1)

The wavelet coefficients are obtained by applying the matrix W on the three axes defined by the tensor of inertia. We use those axes because the wavelet transform is neither translation nor rotation invariant. In order to apply the transform to the object the latter has to be binarized by using a voxel representation. The set of wavelet coefficients represents a tremendous amount of information that is reduced by computing a set of statistical moments for each level of resolution of the transform. For each moment order, a histogram of the distribution of the moment is built: the channels correspond to the level of detail while the amplitude corresponds to the moment values. The last geometrical descriptor is based on three-dimensional statistical moments. The statistical moments are not rotation invariant. In order to solve that problem they are computed in the same reference frame used for the wavelet and cord descriptors. The order of the moment is related to the level of detail. The colour distribution is simply handle by a set of three histograms corresponding to the red, green and blue channels. The scale or physical dimensions of the model has its own descriptor that corresponds to the dimensions of the smallest bounding box that can contain the model. Whatever the descriptor, histograms are compared by means of the Hamming distance. All those descriptors can be weighted and combine in order to handle a particular query. Because of the non-linear and random behaviour of the error the compared descriptors are weighted according to their rank and not according to the corresponding error [5].

184

Eric Paquet and Marc Rioux

The search engine uses the concepts of direct and indirect query by example. In the case of the direct method, a three-dimensional model is used to specify the query. From the results of the first query one can refine the process by performing additional queries. In the case of the indirect methods key words are used. Those key words correspond to a dictionary of three-dimensional models. One word can correspond to one or more models. A query is performed for each model and the best results for each search are combined to form a unique answer. This process is completely transparent to the user. In addition the later can specify the descriptors and the weight attributed to the geometrical shape, color distribution and scale. An example of a query involving a sword is shown below.

Fig. 1. Some results for a query by shape for a sword using the cord description in a database of more than 1000 objects.

6. Implications of MPEG-7 Let us start with the normative part of MPEG-7. As we saw earlier, the normative part is concerned with the description, which, in our case correspond to the descriptors and DS we have presented in the previous section. Each histogram corresponds to a descriptor. The DS for the distribution of the cords is made out of the histogram of the first angle, of the second angle and of the norms. There are also a DS associated with the histograms associated with the wavelet and moment description. We do not propose a DDL because it is not critical in our case. In order to be suitable for three-dimensional data the DDL should support inheritance, possibly multiple inheritance and, in the case of three-dimensional scenes and object’s parts it should be able to describe the relation between the components. Those relations can be geometrical like the relative positions and orientations but they can also be functional.

Crawling, Indexing and Retrieval of Three-Dimensional Data on the Web

185

In order to be acceptable for MPEG-7, the descriptors and DS should fulfill a set of requirements: among them they should be scalable and they should handle multi-level representations [6]. Our D and DS are scalable at both the object and database level. At the database level the complexity of the search is a linear function of the number of models. At the object level the size of the cord, moment and color distribution does not depend on the number of triangles representing the model. In the case of the wavelet description, the number of level of details has a logarithmic complexity, which is acceptable in most applications. Our descriptors and DS can also handle multi-level representations. This is already the case for the wavelet representation that provides by nature a coarse to fine representation. The same can be said about the moments: the order of the moment is related to the level of detail. In the case of the cord distribution a multi-level DS can be obtained by combining many cord DS and varying the number of channels. This is also a characteristic that MPEG-7 is looking for: forming easily D and DS from existing ones [6]. From an MPEG-7 perspective our D and DS are also useful because they cover a wide variety of applications ranging from CAD, catalogues, medical applications and virtual reality and as far as our experiments have proved, they tend to describe adequately most three-dimensional models. The non-normative part is not part of the standard but nevertheless it can have some indirect implications. Let us consider the extraction of D and DS. The extraction process should not depend on the implementation otherwise the description obtained would not be consistent from one implementation to the next. Let us review our indexing procedure to see if it is implementation dependent. We first convert each model to the VRML format. In almost all cases it is possible to reconvert the models back to their original formats without lost of information. That shows that conversion does not modify the content of the file. Cords extraction is clearly defined. The extraction of the eigen vectors and values is a standard procedure and the determination of the orientation of the cord leads to a unique solution. Some care must be taken with the tensor of inertia. When computing the tensor of inertia the weight of each triangle must be taken into account: this weight corresponds to the area of the surface of the corresponding triangles. This is because low curvature surfaces tend to be represented by only a small number of triangles while high curvature surfaces are represented with many triangles even if their area is comparatively smaller. The same remarks can be made for the moment and wavelet descriptions.

7. Conclusions A system that can crawl, index and retrieve three-dimensional objects from the Web has been presented. The system can automatically retrieve three-dimensional models, describe them and search the Web for similar models. The description is based on the scale, three-dimensional shape and color distribution. It has been shown that the

186

Eric Paquet and Marc Rioux

proposed system could be integrated in the framework of MPEG-7. Our system runs on an NT workstation with two processors. It takes typically 2-5 seconds to compute the descriptors for a given three-dimensional object. The search engine is very fast. With the C++ implementation it takes less then one second to search a database of 2000 objects. The implementation based on Java servlets and Oracle 8 takes about two seconds but is much more flexible. The spider takes full advantage of the multiprocessor architecture and requires 512 Mbytes of memory in order to operate smoothly. In order to process faster the data, the spider requires two physical disks: one for the O2 database and one for the retrieved URL. In its present state the system can only crawl small domain of the Web: typically 20 hosts plus all the hosts to which they link. This limitation could be easily overcome by using more computers. The proportion of three-dimensional links is usually less then 0.5 %. A demo of the search engine is available at http://cook.iitsg.nrc.ca:8800/Nefertiti/Nefertiti.html. This demo shows the potential as well as the performances of the search engine A standard for the description of multimedia content is becoming an important issue: even the most sophisticated search engines retrieve a relatively small fraction of what is available. If a standard would be available, material providers would be able to provide such a description. Commercial interests would motivate them because the market would rapidly be divided between those that can be located because of the description and those that cannot. It is also important to develop descriptors that can be automatically extracted. When introducing MPEG-7 we will have to deal with the legacy of the past: millions of multimedia objects without any description. Only automatically generated descriptors could provide an adequate description of those multimedia objects within a reasonable amount of time

References 1 2 3 4 5 6

M La Cascia et al., “Combining Textual and Visual Cues for Content-based Image Retrieval on the World Wide Web”, Proc. IEEE Workshop on Content-based Access of Image and Video Libraries, pp. 24-28 (1998). W. Bosques et al., “A Spatial Retrieval and Image Processing Expert System for the World Wide Web”, Computers ind. Engng 33, pp. 433-436 (1997). C.-L. Huang and D.-H. Huang, “A content-based image retrieval system”, Image and Vision Computing 16, pp. 149-163 (1998). S.-F. Chang et al., “Exploring Image Functionalities in WWW ApplicationsDevelopment of Image/Video Search and Editing Engines”, Proc. International Conf. on Image Processing, pp. 1-4 (1997). E. Paquet and M. Rioux, "Content-based Access of VRML Libraries", IAPR International Workshop on Multimedia Information Analysis and Retrieval, Lecture Notes in Computer Sciences-Springer 1464, pp. 20-32 (1998). MPEG-7: Evaluation Process Document, ISO/IEC JTC1/SC29/WG11 N2463, Atlantic City (USA), October (1998).

A Visual Search Engine for Distributed Image and Video Database Retrieval Applications Jens-Rainer Ohm, F. Bunjamin, W. Liebsch, B. Makai, K. Müller, B. Saberdest, and D. Zier Heinrich Hertz Institute for Communications Technology, Image Processing Department, Einsteinufer 37, D-10587 Berlin, Germany Phone +49-30-31002-617, Fax +49-30-392-7200 [email protected]

Abstract. This paper reports about an implementation of a search engine for visual information content, which has been developed in the context of the forthcoming MPEG-7 standard. The system supports similarity-based retrieval of visual (image and video) data along feature axes like color, texture, shape and geometry. The descriptors for these features have been developed in a way such that invariance against common transformations of visual material, e.g. filtering, contrast/color manipulation, resizing etc. is achieved, and that they are fitted to human perception properties. Furthermore, descriptors have been designed that allow a fast, hierarchical search procedure, where the inherent search mechanisms of database systems can be employed. This is important for client-server applications, where pre-selection should be performed at the database side. Database interfaces have been implemented in a platformindependent way based on SQL. The results show that efficient search and retrieval in distributed visual database systems is possible based on a normative feature description such as MPEG-7.

1 Introduction Visual-feature based search and retrieval of images and videos in databases is a technique which has attracted considerable research interest recently [1][2][3]. Feature descriptions used to characterize the visual data and retrieval algorithms to search the databases are closely related in these implementations in order to obtain optimum results. The consequence is, that the database provider usually also provides a specific search engine, which can only marginally be adapted to a user's needs. Moreover, it is impossible to perform a net-wide search, retrieving visual data which meet some predefined features from different database sources. With multimedia content emerging over the worldwide computer networks, use of distributed systems becomes necessary. If retrieval of data is supposed to be a de-coupled process, a normative feature description is required. Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL'99, LNCS 1614, pp. 187-194, 1999.  Springer-Verlag Berlin Heidelberg 1999

188

Jens-Rainer Ohm et al.

To meet this challenge, ISO's Moving Pictures Experts Group (MPEG) has started a standardization activity for a "Multimedia Content Description Interface", called MPEG-7, which shall provide a standardized feature description for audiovisual data[4]. The meaning of "features" in this sense is very widespread, and can consist of elements for – high-level description (e.g. authoring information, scripting information, narrative relationships between scenes) ; – mid-level description (e.g. semantic categories of objects or subjects present within a scene) ; – low-level description (e.g. signal-based features like color, texture, geometry, motion of scene or camera). Low-level description categories can be transformed into higher-level ones by setting specific rules, and qualitative separation of these categories is not always straightforward. The work reported in this contribution concentrates on the low-level description ; in this case, automatic extraction of features from the data is usually possible, and definition of matching criteria for the similarity-based retrieval using a specific feature type is more or less unique. Even though, it is not the intention of MPEG-7 to standardize the feature extraction, nor the search/retrieval algorithms, which may be differently optimized for specific applications. Nevertheless, the structure of a normative feature description has a high impact on simplicity of adaptation of a non-normative search algorithm, which uses this description for retrieval purposes. MPEG-7 concepts for description schemes and descriptors are shortly reviewed in Section 2. Section 3 describes feature descriptor examples, how they are combined in a flexible way in the framework of a description scheme, and how the ranking in the search/retrieval is performed. Furthermore, in a distributed retrieval application, the interrelationship between the database (at the server side) and the search engine (at the client side) is of major importance. This aspect is discussed in Section 4. Section 5 describes the implementation, and in Section 6, conclusions are drawn.

2

MPEG-7 Description Concept

The MPEG-7 description will consist of description schemes (DS) and descriptors°(D), which are instantiated as descriptor values. Furthermore, it is planned to create a description definition language (DDL), which will allow to define new description schemes and descriptors for specific applications [5]. The whole description will be encoded, such that efficient storage and transmission are enabled. In this paper, we concentrate on the optimization of description scheme structures, such that they can be used in an efficient way for distributed applications. A description scheme is generally a combination of one or more sub-ordinate descriptor(s) and/or description scheme(s). An example is illustrated in Fig.1, where the DS “A” at the top level is a combination of the Ds “A” and “B”, and DS “B”, which again is a combination of Ds “C” and “D”. Multiple nesting of DSs shall be possible. Each descriptor usually characterizes one single feature of the content. The associated descriptor value can be a scalar or a vector value, depending on the nature of the descriptor.

A Visual Search Engine

189

DS "A"

D "A"

D "B"

DS "B"

D "C"

D "D"

Fig.1. Example of a simple MPEG-7 description scheme structure.

3

Visual Feature Description and Content Retrieval

A generic description of visual content for similarity-based retrieval should be based on a flexible combination of different feature descriptors. Even though there are some examples, where one single descriptor would be sufficient to find similar images (e.g. color feature for sunset scenes), in most cases a more distinguishable specification of the query will be necessary. Combination of descriptors during the search can be achieved in two different ways : 1. Parallel approach. A similarity-cost function is associated with each descriptor, and final ranking is performed by weighted summation of all descriptors’ similarity results. This leads to an exhaustive search, where all features available for all items have to be compared during the similarity determination. 2. Coarse-to-fine approach. If specific features can be classified as dominant, a coarse pre-selection is possible based on these, which separates those items with low similarity. Additional features are employed in subsequent finer phases of the search, only. A proper implementation will yield good results with a much shorter retrieval time than the parallel approach. A search algorithm can also contain elements of the parallel approach (to combine different features) at each level of the coarse-to-fine approach. Descriptor organizations which support both of these approaches are described in the following subsections 3.1-3.3 for the examples of color, texture and contour features. 3.1

Color Feature

The use of color histograms is very efficient to describe the color feature of visual items. For this purpose, we are using a transformation of the color into the HSV (Hue, Saturation, Value) space. It is known that differences within HSV space approximately coincide with the human perception of color differences. The HSV space is quantized into 166 bins in our implementation, and within each bin, the frequency of occurrence of the specific color is calculated. This technique was adopted from [6]. Comparison of the histograms of two visual items is performed by weighted sum-of-squareddifferences calculation. This descriptor is capable to find images of similar color with

190

Jens-Rainer Ohm et al.

a high accuracy. If a re-ordering of the histogram is applied such that dominant colors are compared first, it is possible to implement a coarse-to-fine search strategy. 3.2

Texture Feature

The color feature can be calculated based on the pixel statistics, while the texture of a visual item characterizes the interrelationship between adjacent pixels. For this purpose, we are using a frequency description in the wavelet space, where the image is decomposed into 9 wavelet channels (see Fig.2a). Two image signals have a high similarity, if they are scaled or rotated versions of each other. Scaling or rotation effects a shift in the frequency distribution. To overcome this problem, we are using the values of energies calculated locally over the following frequency channel groups (ref. to Fig. 2b): a) Channel “S”, the “scaled” image ; b) Sums over channels 1+2+3, 4+5+6, 7+8+9, which are rotation-invariant criteria ; c) Sums over channels 1+4+7, 2+5+8, 3+6+9, which are scale-invariant criteria. Horizontal Frequency

Horizontal Frequency

S

1

3

2

Vertical Frequency

Vertical Frequency

5

9

a)

1

3

2 7

7 6

S

4

4

6

5

9

8

8

b)

Fig.2. a Structure of channels in the wavelet decomposition. b Sums over channel groups for rotation-invariant (⋅⋅⋅) and scale-invariant () texture criteria. The energy samples from each of the 6 wavelet combination channels (from b/c) undergo a local threshold operation. Based on the output of this binary operation, one of 64 possible frequency distribution patterns results. The frequency of occurrence (histogram) for each of these patterns is stored as a texture descriptor. In addition, a histogram of mean, energy and variance calculated locally over 64 blocks of the image is extracted from the “S” channel after global mean extraction, and used as similarity criterion at the first stage of a hierarchical (coarse-to-fine) search procedure. 3.3

Contour Feature

If a segmented image or video scene is available, the description can be related to arbitrary-shaped objects instead of rectangular scenes. Basically, all the descriptors introduced above can likewise be applied to segments within an image. In addition,

A Visual Search Engine

191

the geometry or contour of the segment can be used as an additional feature. Geometry or contour features can also be used as standalone criteria, e.g. for certain classes of graphics objects. We are using wavelet descriptors, which characterize the 2D contours either in Cartesian- or polar-coordinate representation [7]. If each frequency band is used as a separate descriptor, it is possible to perform a coarse-to-fine search with a raw contour matching (lowest frequency coefficients) first, followed by more exact comparison at the later stages. The wavelet descriptors allow contours of unequal size and orientation to be compared by their shape similarity. A minimum number of 16 scaling coefficients is required to allow reliable decision at the first stage. ...

DS : Visual Scene

DS : Scene Structure

DS : Visual Object

DS : Visual Object

...

...

DS : Geometry

DS : Color

Contour, Position, Moments, ... DS : Color Transformation

D : Transformation Function

D: Reference Color Space

Descriptor Values

Descriptor Values

DS : Frequency Transformation

D: Analysis Function

DS : Color Histogram

Descriptor Values

D : Scaling Factor

D : Number of Cells

D : Data of Histogram

Descriptor Values

Descriptor Values

Descriptor Values

DS : Motion

DS : Texture

Object Motion, Camera Motion, Parametric Motion Model ...

DS : Wavelet Frequency Pattern Occurence

D: Number of Channels Descriptor Values

D: Threshold Function Descriptor Values

D : Energy, Mean, Var. Histogram Descriptor Values

D : Data of Histogram Descriptor Values

Fig.3. A visual object DS based on color, texture and geometry descriptors. 3.4

Combination of Descriptors and Retrieval Operation

Fig.3 shows the structure of a description scheme characterizing the visual features of an "image object", which may either be a still image, a keyframe from a video, a single segment of one of those, or any other (rectangular or arbitrary-shaped) visualcontent item. This DS can again be a sub-description of a higher-level DS in MPEG°7, e.g. for the purpose of shot description within a video scene. The full resolution is given in the figure only for the color and texture branches. Each descriptor (D) is instantiated by one or more descriptor values (DV). Remark that two different types of DVs are present in our figure – one to characterize the “structural” parameters of the descriptor (quantization functions, number of histogram bins), and another

192

Jens-Rainer Ohm et al.

one giving the feature values of the content (data of histogram). The former type can be shared by all content items which are represented with the same description, while the latter one must be made available “individually” for each item. Our retrieval algorithm designed for image database search supports the following properties, such that the database search is fast, reliable and efficient : – Flexible combination of different features, according to the requirements of a query formulation. This means, that from all features contained in a description, only a subset may be used for a specific query or search step. – Application of a weighting function wi for each specific feature i in a feature combination. This weighting function will usually not be normative in the MPEG7 context, but depends on the query formulation. Since the similarity metric values resulting from the different feature descriptors vary by a high extent, we perform a normalization. The final metric calculated from normalized metrics si becomes s=w1⋅s1+ w2⋅s2+..., with sum of the weights equal to 1.

4

Retrieval in a Client/Server Environment

For audiovisual data search and retrieval, a search engine needs flexible access to the feature description resources. This means, that for specific search tasks only particular subsets of the feature representation data are needed. This can easily be achieved, if not only the visual content items, but also the MPEG-7 description data are organized as items in the database. In this case, only the structure of organization (e.g., in which field of a data table which descriptor values can be found) must be made available in an initial descriptor table which is an entry point to the database. The AV data (to which the description is related) need not necessarily to be stored in the same database, it is sufficient that the description holds a link to the real location. The search engine situated at the client can then access any descriptor values associated with any AV object in any set of data via a database interfaces. Since the Structured Query Language (SQL) [8] is a very common interface supported by most database systems, we have used it for this purpose. The configuration is illustrated in fig.4. Standard SQL mechanisms are employed via the interface to formulate the query in the remote pre-selection.

5

Implementation

The concepts elaborated in this paper have been implemented in a visual-data search engine developed at HHI. For platform independence, the system's core parts – presentation, user interactivity and database interfaces – were realized in JAVA. For database interfacing, we have used the JAVA Database Connectivity (JDBC) Tools, which allow the implementation of a database-independent SQL interface. The user interface is also of high importance in the realization. The search engine's basic visible desktop only includes the most relevant setting capabilities, like data management and selection of basic features for a specific query. Users with more skills can also use fine-tune

A Visual Search Engine

193

settings for optimum search results, e.g. adjust specific weighting between the different texture descriptors available. Fig.5a shows the desktop of the search engine, where the 10 images shown at the right side are the result of the query in ranked order, originating from the image in the left box as a reference. Fig.5b illustrates additional query examples. The search can be performed both on still images and on key frames from video sequences. The presentation interface also contains a video player.

MPEG-7 decoder

feature selection

Interface

Search Engine

Interface

Initial Descriptor Table

1011 1011 1011

text text text

link link link

MPEG-7 stream MPEG-7 database

AV data decoder

AV stream

Interface

Presentation Engine

Interface

AV content query result (link address)

AV content

AV content

client

server

Fig. 4. Configuration of a client-server architecture in visual data retrieval.

a)

b)

Fig.5. a Desktop of the HHI search engine (Windows platform) b More query results (first 10 most similar displayed in each case) generated by the HHI search engine.

194

6

Jens-Rainer Ohm et al.

Conclusions

For distributed visual data retrieval applications, a normative description of visual features as defined by MPEG-7 is necessary, such that interoperability between the search engine at the client side and the database positioned at the server side is enabled. Basically, a distributed configuration imposes additional requirements to the content-based search, especially with respect to data organization and structure of feature description. We have tested different descriptors that can be used in a coarseto-fine search strategy, which can partially be applied at the server end, in order to speed up the query and avoid unnecessary transmission of feature data. We have found that the communication between the search engine and the database can be organized in a very efficient way, using existing interconnection standards such as SQL. For multiple-tier systems, e.g. simultaneous linking with several databases or connection with intelligent agents, an object-oriented approach like CORBA would be more convenient.

7 Acknowledgements This work was supported by the German Federal ministry of education, research, science and technology under grant 01 BN 702.

References 1. 2. 3. 4. 5. 6. 7. 8.

J.R. Smith, S.-F. Chang : “VisualSeek : A fully automated content-based image query system”, Proc. Int. Conf. on Image Proc. (ICIP), Lausanne, 1996 J.R. Bach et al. : “The Virage image search engine : An open framework for image management”, Proc. Storage and Retrieval for Image and Video Databases, SPIE vol. 2670, pp. 76-87, 1996 W. Niblack et al. : “The QBIC project : Querying images by content using colour, texture and shape”, Proc. Storage and Retrieval for Image and Video Databases, SPIE vol. 1908, pp. 173-187, 1993 ISO/IEC/JTC1/SC29/WG11 : “MPEG-7 context and objectives”, document no. N2460, Atlantic City, Oct. 1998 ISO/IEC/JTC1/SC29/WG11 : “MPEG-7 requirements document”, document no. N2461, Atlantic City, Oct. 1998 J.R. Smith : “Integrated spatial and feature image systems : Retrieval, analysis and compression”, PhD thesis, Columbia University, 1997 G. C.-H. Chuang and C.-C. J. Kuo : “Wavelet descriptor of planar curves : Theory and applications”, IEEE Trans. Image Proc. 5 (1996), pp. 56-70 ISO/IEC 9075:1992, "Information Technology --- Database Languages --- SQL"

Indexing Multimedia for the Internet Brian Eberman, Blair Fidler, Robert Iannucci, Chris Joerg, Leonidas Kontothanassis, David E. Kovalcin, Pedro Moreno, Michael J. Swain, and Jean-Manuel Van Thong Cambridge Research Laboratory Compaq Computer Corporation, Cambridge, MA 02139, USA Tel: +1 617 692-7627, Fax: +1 617 692-7650 [email protected]

Abstract. We have developed a system that allows us to index and deliver audio and video over the Internet. The system has been in continuous operation since March 1998 within the company. The design of our system differs from previous systems because 1) the indexing can be based on an annotation stream generated by robust transcript alignment, as well as closed captions, and 2) it is a distributed system that is designed for scalable, high performance, universal access through the World Wide Web. Extensive tests of the system show that it achieves a performance level required for Internet-wide delivery. This paper discusses our approach to the problem, the design requirements, the system architecture, and performance figures. It concludes by showing how the next generation of annotations from speech recognition and computer vision can be incorporated into the system.

1

Introduction

Indexing of Web-based multimedia content will become an important challenge as the amount of streamed digital video and audio served continues to grow exponentially. Inexpensive storage makes it possible to store multimedia documents in digital formats with costs comparable to that of analog formats, and inexpensive, higher-bandwidth networks allow the transmission of multimedia documents to clients in corporate intranets and through the public Internet. The very rapid growth in the use of streaming media players and browser plug-ins demonstrates that this is a compelling medium for users, and the ease of use of products such as Microsoft’s NetshowServerTM and RealNetworks’ RealServerTM will make it possible for even small organizations to distribute their content over the Internet or Intranets. We built the CRL Media Search system (termed Media Search from now on) to investigate mechanisms for indexing video and audio content distributed via the Web. The Media Search service bears significant resemblance to existing search engines on the World Wide Web. Like search engines it allows the users to perform a text search against a database of multimedia documents and return a list of relevant documents. Unlike standard search engines, it also locates the Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL’99, LNCS 1614, pp. 195–203, 1999. c Springer-Verlag Berlin Heidelberg 1999


196

Brian Eberman et al.

matches within the documents – search engines are able to leave this task up to the browser. We completed initial implementation of the system in early March, 1998 and have been running it nearly continuously over Compaq’s corporate intranet, with updates, since then. A broad range of content sources has been added to the system. The content includes broadcast material from the Web, technical talks, produced management conferences, and captured broadcast video content. The audio quality of these assets varies widely and has provided a very useful set of tests. Statistical analysis of the system usage has enabled the development of a user model for extensive scalability testing of the system. This paper summarizes the system design issues and contrasts our solutions with some that have appeared previously in the literature. We then present the basic system architecture and how the system functions, and conclude with discussion of our plans for incorporating additional annotation types into the system.

2

Design Issues

During the past few years a number of video indexing systems have been built. These systems generally take in a video feed from an analog source, digitize the source feed and then perform indexing on the content. Both research and commercial systems have been built. The News-on-Demand project, part of Informedia, at CMU [4] is a good example of this type of system. In this project CNN and other live feeds were digitized, then image analysis was performed to cut the video into shots. Shots are then grouped into scenes using multiple cues. The speech in text format for each scene is then indexed using a standard text indexing system. Users can then send queries to retrieve scenes. Work since [4] has focused on video summarization, indexing with speech recognition, and learning the names of faces. A similar example is a commercial product from VirageTM . This product can provide a system to do real-time digitization, shot cut detection, and closedcaption extraction. Off-line, the system provides a keyframe summary of the video over a Web browser; by selecting time periods represented by pairs of keyframes, the user can add annotations to the index. Finally, Maybury’s [5] system at MITRE focuses on adding advanced summarization and browsing capabilities to a system that is very similar to Informedia. These previous works, and others, employ the shot – an uninterrupted sequence of video from a single camera view – as the basic atomic unit for all additional processing. While shots are clearly important, this type of structuring of the information is not always appropriate. Davenport et al [2] introduced the idea of a stream-based representation of video from which multiple segmentation can be generated. In a stream-based representation, the video is left intact, and multi-layered annotations with precise beginning and ending times are stored as associated metadata with the video. Annotations can be a textual representation

Indexing Multimedia for the Internet

197

of the speech, the name of a person, objects in the scene, statistical summaries of a sequence of video, or any other type of data. A stream-based annotation system provides a more flexible framework and can always be reduced to a shot/scene representation by projecting the time intervals of the other annotations against the shot annotations. In our work, the stream-based approach can be used to produce a text oriented display. For example, if the system has primarily indexed conversations or speech, as ours has, then what is of interest to the user is the structure of the textual representation of the speech. A single keyframe per paragraph could be more appropriate than one determined from image analysis. A second example is content reuse and repurposing. Content companies are very interested in reusing old content for new productions. In this case, the semantic content of the story is not of interest. Instead the basic objects, people, and settings are of value. Annotations should mark their appearance and disappearance from the video. As a final case, consider indexing a symphony video based on instruments and scores. In this case a visually-based temporal segmentation is not appropriate, but one based on the musical structure is. Our system, paired with a Web-based annotation tool we built, can support all these differing styles of annotation. Another difference in our system is that we treat the Web as not just a delivery mechanism, but as the basic infrastructure on which to build. We believe that the ease with which video and audio content can be placed on the web will soon cause a wide assortment of groups to distribute their multi-media content as ubiquitously as their text documents. We have experience with large corporate and institutional archives, and news and other content-production companies. All of these types of organizations are actively considering or have started to make this move. When this happens, video indexing systems will be needed not to index analog feeds, but to index video that has been placed on multi-media content servers. To investigate this issue, we designed our system so that HTTPbased content servers, which we call Media Servers, could be distributed across the organization, or anywhere on the Internet, and then be indexed from one central location. As a proof of the concept, we indexed audio content on NPR’s web site in our system. Users could search this content using our system; when they played one of these clips it was delivered directly from the NPR site. Since our system is designed to be used by a wide variety of users, we built an HTML-based user interface that would work on all major web browsers. To provide access across a world-wide corporate intranet, the content delivery was based on low-bitrate streaming video. The system follows the standard search engine user interaction model. This model consists of two steps: First, users search and get pointers to documents, and then they go to the documents and use the browser’s find command to search for their particular query. Since find is not a browser supported function for video, we had to create a way of supporting find through HTML pages that was consistent with this model. Since we worked with a very broad range of content sources, the audio quality of these materials varied broadly and followed many different production

198

Brian Eberman et al.

formats. Thus our system could not use knowledge of the format for browsing. We further had to develop very robust methods for aligning the available, occasionally inaccurate, transcripts to very long audio segments (often greater than 1 hour in length) containing speech, music, speech over music, speech with background noise, etc. A paper by Moreno et al [6] reports on this algorithm. Finally, this area is rapidly evolving new techniques for automatically annotating content. This, plus our first experience with trying to build a content processing system led us to develop a distributed periodic processing model with a central database as a workflow controller. This approach is discussed more throughly in DeVries [3].

3

System Description

The Media Search system, as shown in Figure 1, is broken into six components: 1) one or more Media Servers, 2) a metadatabase that is a built on a standard relational database, 3) a collection of autonomous periodic processing engines or daemons managed by the metadatabase, 4) an Indexing System which is a modified version of the NI2 index used in the AltaVista engine, 5) a Feeder to synchronize information in the database with the NI2 index, and 6) a Presentation Server that communicates with the other subsystems to construct an HTML response to a user query.

Content

Fig. 1. CRL Media Search Architecture

Media Server A Media Server stores and serves up the actual content and provides a uniform interface to all the media stored in the system. In addition, it handles storage and access control, if required. We use it to store video (MPEG,

Indexing Multimedia for the Internet

199

RealVideoTM ), audio(RealAudioTM ), and images (JPEG). We also implemented on-demand conversion functions to convert from MPEG to other formats. For large-scale Internet applications of the system, on-demand conversion is only used to produce formats accessed by our internal processing daemons. All formats accessed by user queries are pre-produced in order to improve performance. Both the stored files, and the “virtual” files which are computed on-demand are accessed through a uniform URL interface. The URL-based interface to Media Servers provides a layer of abstraction allowing storage versus computation trade-offs, and makes it possible to locate Media Servers anywhere within an intranet or across the Internet. Meta-Database and Daemons The metadatabase, which is built on a relational database, performs three functions. Firstly, it keeps track of the location of all stored formats, or representations, of each multimedia document. Secondly, the metadatabase acts as a central workflow control system for daemon processing. A daemon is a process that performs “work”, typically taking one or more representations and computing an output representation. For example, the alignment daemon takes a transcript and an audio file as input and computes an aligned transcript as output. Central workflow processing is enabled by having each daemon type register the format of its required inputs when that type is first installed. Then each daemon instance can request work from the metadatabase and is given the input URL handles that need to be processed. When a daemon completes work on its input representations, it stores the output representation in a Media Server and registers the availability of the new representation with the metadatabase. This simple model leads to a robust, distributed processing method which scales to the large processing systems needed for the Web. The third role of the metadatabase is to store all the stream-annotation information. Although we physically store the annotations in a way optimized for the presentation user interface, the format of the annotation tables can be thought of as tables giving the type, start time, end time, and value. The system is structured so that it can store arbitrary annotation types in this way. For example, using speech recognition technology we are able to align transcripts to the audio component of the video [6]. The annotation stream then consists of the start and end time of each word in the transcript, and the word itself. The system is sufficiently flexible that we can store a large variety of different forms of annotations, although currently we only store aligned transcripts and images, called keyframes, which represent a video shot. Index and Feeder One of the most important problems in video and audio indexing is not only to index the collection of words that were spoken during the video, but also to be able to determine where a particular word, phrase, or combination occurred. It was our aim to support this capability while still indexing full documents. We modified the NI2 index used by AltaVista [1] to accomplish this task. While the original NI2 index returns the ID of a document containing the words of the user query, our modified version will return multiple hits per document; one hit for every location in the document that matches the query.

200

Brian Eberman et al.

In order to provide within-document indexing, we first had to define what were the within-document match locations to a given query. A match location provides an entry point into the video; the user is then free to play the video for as long as desired. Therefore, a match location naturally defines a subdocument, starting at the match location and ending at the end of the video. The match locations were then defined to be all the locations where terms from the query matched the document, and which defined a subdocument that matched the complete query. We also extended the model to include rank queries, that is, how to rank certain subdocuments matches more highly than others. A standard term frequency / inverse document frequency (tf.idf) metric [7] is used, with each term match multiplied by a position-dependent factor. To enhance the rank of subdocuments where there were many term matches appearing soon after the match location, the position-dependent factor takes the form of an exponential decay with its peak at the beginning of the subdocument. At this point we have not done sufficient information retrieval (IR) testing of this metric to report on the IR performance of within-document ranking.

Fig. 2. CRL Media Search Query Response Page

Presentation Server The Presentation Server communicates with the various services offered by the metadatabase, the Index Server, and the Media Server to allow the user to perform searches, see responses, and view parts of the video stream. A screen shot of the initial response produced by the Presentation Server after a query is shown in figure 2. From this page the user may decide to play the video from the first match within a document, or to go to another page to see all the matches within the video. From this document-specific match page, the user may play the video from any one of the matches within the document, search within the specified document, or progress to yet another page. This third style of page allows the user to browse the video starting at the location of one of the matches.

Indexing Multimedia for the Internet

201

A typical response to a user query is computed as follows: The presentation system first sends a message to the Index Server; then, using a handle returned by the index, finds information in the metadatabase. In this way the presentation system uses the two systems together to compute responses to queries, in effect making the NI2 index an extension of the relational database. Although this type of extension can in principle be achieved with object relational databases or in extended relational databases by putting the index within the database, our specialized external text index that is synchronized with the database offers a higher performance solution.

4

System Performance

The system was tested running on two Digital AlphaServer 4100’s. One machine had four Alpha 21164 processors running at 466 MHz, 2 gigabytes (GB) of memory, and a disk array with 600 GB of data storage. This machine ran the database, index, and Media Server components of Media Search. The second machine had four 400MHz Alpha 21164 processors and 1.5 GB of memory and ran the presentation server. The test harness was a set of client processes probabilistically emulating the user model derived from the system deployed on Compaq’s internal network. The server side used Oracle 8.0.5 as the relational database with a custom developed interface between the presentation layer and the Oracle engine, and the NI2 index engine with the extensions described in section 3. The presentation server used the Apache web server with the presentation CGI scripts developed in Perl. To avoid the high startup overhead of Perl for every invocation of a CGI script we used the FastCGI extensions to CGI which allow Perl processes to become servers and converts CGI calls to socket communication between the Web server and the resident Perl processes. Our system has a throughput of 12.5 user sessions per second (as defined by the user model), resulting in 36 pages served per second, or approximately 3.2 million pages over a 24-hour period. We also achieved an average latency of less than 0.5 seconds per page served. The Presentation System, written in Perl, was the bottleneck in these tests, even with the FastCGI optimizations. Performance improvements to the presentation system could be obtained, for example, by rewriting it in C/C++ instead of Perl. We have also conducted performance tests of the metadatabase as a standalone component since it is the only component of our system whose performance can not be improved simply by replicating the data and using more machines to service requests. All other components can be distributed/replicated among multiple machines fairly easily. The test of the metadatabase component alone running on one AlphaServer 4100 with four Alpha 21164 processors at 466MHz measured a throughput of 29 user sessions per second, equivalent to 88 pages per second or 7.5 million pages over a 24-hour period.

202

5

Brian Eberman et al.

Conclusions and Future Work

We implemented the CRL Media Search system to explore the issues that arise when building a distributed system for indexing multimedia content for distribution over the World Wide Web. The system uses the Internet as the basic platform for both organizing the computation and distributing content to users. We have tested the performance of the system and found that it scales well and can provide an indexing service at a cost comparable to indexing text (HTML) documents. We are investigating adding other meta types of annotation information to the system. For instance, the meta-information extracted by a face detector/recognizer and speaker spotter can be placed directly into the metadatabase. This information can then either be indexed in the current NI2 index or in a second NI2 index. In addition, we are researching indexing crawled multimedia from the World Wide Web, where we do not have transcripts or closed captions. Sound classification and speech recognition are key technologies for this area of research.

Acknowledgements The Media Search project has benefited from the insights, support, and work of many people, including: Mike Sokolov, Dick Greeley, Chris Weikart, Gabe Mahoney, Bob Supnik, Greg McCane, Katrina Maffey, Peter Dettori, Matthew Moores, Andrew Shepherd, Alan Nemeth, Suresh Masand, Andre Bellotti, S. R. Bangad, Yong Cho, Pat Hickey, Ebrahim Younies, Mike Burrows, Arjen De Vries, and Salim Yusufali. Thanks to Beth Logan for comments on drafts of this paper.

References 1. Burrows, M. Method for indexing information of a database. U.S. Patent 5745899, (April 1998). 199 2. Davenport, G., Aguierre-Smith, T. G., and Pincever, N. Cinematic primitives for multimedia. IEEE Computer Graphics and Appplications 11, 4 (July 1991), 67–75. 196 3. de Vries, A. P., Eberman, B., and Kovalcin, D. E. The design and implementation of an infrastructure for multimedia digital libraries. In International Database Engineering Applications Symposium (July 1998). 198 4. Hauptmann, A., Witbrock, M., and Christel, M. News-on-demand - an application of informedia technology. In D-LIB Magazine (Sept. 1995). 196 5. Mani, I., House, D., and Maybury, M. Intelligent Multimedia Information Retrieval. MIT Press, 1997, ch. 12: Towards Content-Based Browsing of Broadcast News Video. 196

Indexing Multimedia for the Internet

203

6. Moreno, P. J., Joerg, C., Van Thong, J.-M., and Glickman, O. A recursive algorithm for the forced alignment of very long audio segments. In International Conference on Spoken Language Processing (1998). 198, 199 7. Salton, G., and McGill, M. J. Introduction to Modern Information Retrieval McGraw-Hill, 1983. 200

Crawling for Images on the WWW Junghoo Cho1 and Sougata Mukherjea2 1

Department of Computer Science Stanford University, Palo Alto, Ca 94305, USA chocs.stanford.edu 2 C&C Research Lab, NEC USA 110 Rio Robles, San Jose, CA 95134, USA sougataccrl.sj.nec.com

Abstract. Search engines are useful because they allow the user to find information of interest from the World-Wide Web. These engines use a crawler to gather information from Web sites. However, with the explosive growth of the World-Wide Web it is not possible for any crawler to gather all the information available. Therefore, an efficient crawler tries to only gather important and popular information. In this paper we discuss a crawler that uses various heuristics to find sections of the WWW that are rich sources of images. This crawler is designed for AMORE, a Web search engine that allows the user to retrieve images from the Web by specifying relevant keywords or a similar image. Keywords: World-Wide Web, Crawling, Site-based Sampling, Non-icon detection.

1

Introduction

Search engines are some of the most popular sites on the World-Wide Web. However, most of the search engines today are textual; given one or more keywords they can retrieve Web documents that have those keywords. Since many Web pages have images, effective image search engines for the Web are required. There are two major ways to search for an image. The user can specify an image and the search engine can retrieve images similar to it. The user can also specify keywords and all images relevant to the user specified keywords can be retrieved. Over the last two years we have developed an image search engine called the Advanced Multimedia Oriented Retrieval Engine (AMORE) [5] (http://www.ccrl.com/amore) that allows the retrieval of WWW images using both the techniques. The user can specify keywords to retrieve relevant images or can specify an image to retrieve similar images. Like any search engine we need to crawl the WWW and gather images. With the explosive growth of the Web it is obviously not possible to gather all the WWW images. The crawlers run on machines that have limited storage capacity, and may be unable to index all the gathered data. Currently, the Web contains more than 1.5 TB, and is growing rapidly, so it is reasonable to expect that most Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL’99, LNCS 1614, pp. 203–211, 1999. c Springer-Verlag Berlin Heidelberg 1999


204

Junghoo Cho and Sougata Mukherjea

machines cannot cope with all the data. In fact a recent study has shown that the major text search engines cover only a small section of the Web [3]. The problem is magnified in an image search engine since image indexing takes more time and storage. Therefore the crawler should be “intelligent” and only crawl sections of the WWW that are rich sources of images. In this paper we present the AMORE crawler and explain several heuristics that can be used to determine WWW sections containing images of interest. The next section cites related work. Section 3 gives an overview of the AMORE system. Section 4 explains the crawler architecture. Section 5 discusses the heuristics used by the crawler. Finally section 6 concludes the paper with suggestions of future work.

2

Related Work

Crawlers are widely used today. Crawlers for the major search engines, for example, Alta Vista (http://www.altavista.com) and Excite (http://www.excite.com) attempt to visit most text pages, in order to build content indexes. At the other end of the spectrum, we have personal crawlers that scan for pages of interest to a particular user, in order to build a fast access cache (e.g. NetAttache http://www.tympani.com/products/NAPro/NAPro.html). Roughly, a crawler starts off with the URL for an initial page. It retrieves the page, extracts any URLs in it, and adds them to a queue of URLs to be scanned. Then the crawler gets URLs from the queue (in some order), and repeats the process [6]. [1] looks at the problem of how the crawler should select URLs to scan from its queue of known URLs. To ensure that the crawler selects important pages first, the paper suggests metrics like backlink count and page rank to determine the importance of a WWW page. Instead of finding the overall importance of a page, in this paper we are interested in the importance of a page with respect to images. Another research area relevant to this paper is the development of customizable crawlers. An example is SPHINX [4], a Java toolkit and interactive development environment for Web crawlers which allows site-specific crawling rules to be encapsulated.

3

AMORE Overview

During indexing the AMORE crawler, discussed in the next section, gathers “interesting” Web pages. The images contained and referred to in these pages are downloaded and the Content-Oriented Image Retrieval (COIR) library [2] is used to index these images using image processing techniques. We also use various heuristics, after parsing the HTML pages, to assign relevant keywords to the images and create keyword indices.

Crawling for Images on the WWW

(a) Semantic Similarity Search with a picture of Egypt

205

(b) Integrated Search with the keyword ship and the picture of a ship

Fig. 1. Examples of different kinds of AMORE searches During searching, AMORE allows the user to retrieve images using various techniques. Figure 1 shows some retrieval scenarios. The user can specify keywords to retrieve relevant images. The user can also click on a picture and retrieve similar images. The user has the option of specifying whether the similarity is semantic or visual. For semantic similarity, the keywords assigned to the images are used. If two images have many common keywords assigned, they are considered to be similar. Thus in Figure 1(a) images of Egypt are retrieved even though they are not visually similar. For visual similarity, the COIR library is used. It looks at features of the images like color, shape and texture to determine similarity using the image indices. AMORE also allows the integration of keyword search and similarity search. Thus Figure 1(b) shows images visually similar to the picture of a ship that are also relevant to the keyword ship.

4

AMORE Crawler

The design of AMORE image crawler embodies two goals that we pursue. First, the crawler should crawl the web as widely as possible. More precisely, we want the crawler to visit a significant number of the existing Web sites. If the crawling is performed only to a small set of sites, the scope and the number of images crawled may be limited and biased. Second, the crawler should not waste much of its resource examining “uninteresting” parts of the Web. For now, the information on the Web is mostly textual, and only a small portion of the Web contains images worthy of being indexed. The crawler should not waste its resource trying to crawl mostly textual parts of the Web.

206

Junghoo Cho and Sougata Mukherjea

Note that these two goals are conflicting. On one hand, we want to gather images from as many sites as possible, which means that the crawler should visit a significant portion of the web. On the other hand, we want to limit the scope of the crawler only to the “interesting” sections. We tried to achieve these two conflicting goals by site-based sampling approach, which will be discussed next. 4.1

Architecture of the Crawler

The crawler of AMORE consists of two different sub crawlers: Explorer and Analyzer. Informally, Explorer discovers “interesting sites” and Analyzer filters out “uninteresting” sections from the identified sites. Figure 2 represents the data flow between these two crawlers.

Crawl

WWW

Explorer

Interesting Pages

Interesting Sites

Crawl

Analyser

AMORE indexer

Interesting pages

Fig. 2. The architecture of the AMORE crawler.

– Explorer Explorer is the big scale crawler whose main job is to discover “interesting” sites from the web. It is optimized to find as many interesting sites as possible, and therefore it tries to visit the web widely but shallowly. More precisely, it differs from most web crawlers in that it only crawls k sample pages for each and every site it found. After sampling k pages from a site, it checks the sample pages to see how many non-icon images the pages contains or refers. (The criteria for icon detection is described in section 5.1 in detail). If more than r% of pages have more than one non-icon image, then the site is considered “interesting”. The Analyzer works on these interesting sites that the Explorer found. Note that even if a site is not found to be interesting, the interesting pages in the site are sent to the AMORE indexer. This allows AMORE to index images from a large number of Web sites.

Crawling for Images on the WWW

207

– Analyzer Analyzer is the small scale crawler whose main job is to identify “interesting” sections from a web site. The input to Analyzer are the “interesting” sites that Explorer found. For each input site, the Analyzer performs more crawling to gather m (>> k) sample pages. These sampled pages are then analyzed to evaluate the directories in the site. For each directory, we calculate its importance as discussed in in section 5.2. Then the Analyzer crawls the directories in the order of their importance. The Analyzer examines all directories whose importance is greater than a threshold. Note that our two step crawling approach is conceptually similar to iterative deepening [7]. Informally, we expand all high level nodes (crawl root level pages of each web site), and we go into deeper (perform more crawling) for the interesting nodes expanded. Also note that there are various parameters in the crawling process like the number of pages to be sampled by the Explorer and the threshold value for importance of the directories in the Analyzer. The AMORE administrator can set these values based on the resource constraints.

5

Heuristics

Fig. 3. Comparing the reasons why images referred to in HTML files were not indexed by AMORE.

5.1

Removing Icon Images

The Web is well-known for its heterogeneity of information. The heterogeneity is also true for images, and different types of images coexist on the Web. At one

208

Junghoo Cho and Sougata Mukherjea

extreme, a small icon is used as the bullet of a bulleted list and at the other extreme, a page embeds a 1024x768 gif image of Gogh’s painting. We believe the images on the Web can be classified into two categories: icons and authentic images. Icons are the images whose main function is enhance the ”look” of a web page. They can be substituted by a symbol (e.g. bullets) or by text (e.g. advertizing banners), but they are used to make the page more presentable. In contrast to icons, authentic images are the images that cannot be replaced by non-images. We cannot substitute the image of Gogh’s painting or the picture of Michael Jordan with text without losing information that we want to deliver. An usability study of AMORE has also shown that people were not interested in the icons when they are using a WWW image retrieval engine. It is generally difficult to identify icons without analyzing the semantic meaning of an image. However, our experiments show that the following heuristics work reasonably well for icon detection: – Size: We remove very small images such as dots which are generally used for HTML page beautification. We only extract images that are more than a certain size (generally > 2000) and have a certain width and height. – Ratio: We don’t extract images if their width is much greater or smaller (> 3 or < 1/3) than their height. This filters out the headings and banners that appear at the top and the sides of many Web pages. – Color: We also remove color images if they have very few colors (<5). This removes uninteresting computer generated logos, etc. – Animated: Surprisingly, most users were also not interested in animated gifs! So they are also removed. – Transparent: We also remove transparent images since they are generally used as headings and logos. About 10% of the images gathered by AMORE were not indexed because they were uninteresting (by the above criteria) or they could not be downloaded to our site (for example, some images were missing even though they were referenced in HTML pages) or they were not the right format (like most browsers AMORE supports only gif and jpeg). Figure 3 is a chart showing the various reasons for which an image was not indexed. 5.2

Finding Interesting Directories

An analysis of the major WWW sites have shown that they are well organized; all pages dealing with the same subject are generally organized in the same directory. It is also true that if a sample of the pages in a directory have images, a majority of the pages are “interesting” from our point of view. These observations are utilized by the Analyzer when it tries to find sections of interest in large Web sites. We use various heuristics to find interesting directories:

Crawling for Images on the WWW

209

– In many Web sites images are kept in directories with relevant names. For example, directories like http://www.nba.com/finals97/gallery/, http://www.si.edu/ natzoo/photos/ and http://www.indiabollywood.com/ gallery/ contain good images. Individuals also organize their site so that the images are kept in directories with meaningful names. For example, we found good images in http://fermat.stmarys-ca.edu/∼jpolos/photos and http://www.mindspring.com/ ∼zoonet/galleries. Therefore, we have a list of keywords (like gallery, photo, images) and if a directory has any of these words, they are considered interesting and crawled in detail. – However, not all interesting directories may have meaningful names. For example, http://cbs.sportsline.com/b/allsport/ has many images. Therefore, for most directories more analysis is necessary. We calculate the importance of a page as i + 1/w where i is the number of non-icon images in the page and w is the number of words (excluding HTML tags). This makes the importance of a page with a lot of text and one image less important than a page with one image and fewer text. The overall importance of a directory is the average of the importance of the sampled pages of the directory. Since the analyzer crawls the directories in order of their importance, image intensive pages will be gathered first and if there are resource constraints, the AMORE administrator can stop the crawling of a site after sometimes. At present, directories whose importance is greater than a pre-defined threshold are considered interesting and only these directories are crawled in detail.

Fig. 4. An evaluation of our directory importance heuristic for http://cnnsi.com. The interesting directories are shown in italics.

210

Junghoo Cho and Sougata Mukherjea

To determine if our heuristics are correct, we have built a visual evaluation interface. Figure 4 shows the interface. Here we are evaluating our heuristics for the http://cnnsi.com Web site. The site is represented as a tree and the interesting directories are shown in italics and red color. It is seen that directories like almanac (URL: http://cnnsi.com/almanac) and features are found interesting while directories like jobs and help are found uninteresting. For a large directory, even if the whole directory is uninteresting, several subdirectories may be found to be interesting. For example, on exploring the hockey directory using our evaluation interface, we find that the directories events and players are found to be interesting while directories like scoreboards and stats are not. On examining the Web site, we found that the directory importance heuristics performed upto our expectation.

6

Conclusion

In this paper we presented the crawler for the AMORE WWW Image Search Engine. The crawler uses several heuristics to crawl at least some pages of all Web sites as well as the “interesting” sections of “interesting” Web sites. This allows the AMORE crawler to achieve the conflicting goals of gathering as many “interesting” images as possible by visiting as few sites as possible. In the future we are planning to do an extensive evaluation of the crawler and extend the technique to other media like video and audio. Our ultimate objective is to develop an effective Multimedia WWW Search engine.

References 1. J. Cho, H. Garcia-Molina, and L. Page. Efficient Crawling through URL ordering. Computer Networks and ISDN Systems. Special Issue on the Seventh International World-Wide Web Conference, Brisbane, Australia, 30(1-7):161–172, April 1998. 204 2. K. Hirata, Y. Hara, N. Shibata, and F. Hirabayashi. Media-based Navigation for Hypermedia Systems. In Proceedings of ACM Hypertext ’93 Conference, pages 159– 173, Seattle, WA, November 1993. 204 3. S. Lawrence and C. Giles. Searching the World-Wide Web. Science, 280(5360):98, 1998. 204 4. R. Miller and K. Bharat. SPHINX: a framework for creating personal, site-specific Web crawlers . Computer Networks and ISDN Systems. Special Issue on the Seventh International World-Wide Web Conference, Brisbane, Australia, 30(1-7):119–130, April 1998. 204 5. S. Mukherjea, K. Hirata, and Y. Hara. Towards a Multimedia World-Wide Web Information Retrieval Engine. In Proceedings of the Sixth International World-Wide Web Conference, pages 177–188, Santa Clara, CA, April 1997. 203

Crawling for Images on the WWW

211

6. B. Pinkerton. Finding what People Want: Experiences with the WebCrawler. In Proceedings of the First International World-Wide Web Conference, Geneva, Switzerland, May 1994. 204 7. S. Russell and P. Norvig. Artificial Intelligence: A Morden Approach. Prentice Hall, 1995. 207

A Dynamic JAVA-Based Intelligent Interface for Online Image Database Searches V. Konstantinou and A. Psarrou School of Computer Science University of Westminster Harrow Campus, Watford Road, Harrow HA1 3TP, London, U.K. [email protected]

Abstract. This paper describes the use of a JAVA-based interface to enable remote image-based searches. The database used for our target system, resides at the Marcianna Library in Venice, Italy. It consists of a number of medieval manuscripts scanned at a variety of resolutions and indexed by both textual and image based information. The system described in this paper is a JAVA based client-server which can be used to interrogate a remote hybrid database and initiate searches using both the textual and image links of this database. Image queries can be based on data supplied by the user, or extracts of the original remote database. In the paper we also discuss the indexing and pattern recognition algorithms which are used on the Server side. This work is part of the HISTORIA project funded under the European Union Libraries Initiative (Telematics for Libraries - Project No:3117). Keywords: Digital libraries, Content-based indexing/retrieval, Java Interfaces, Distributed Databases, Multimedia.

1

Introduction

Scholars and museum curators in the areas of history of art, miniatures, seals, codicology and archivism need an easy and rapid consultation and comparison of manuscript compilations that contain codices of heraldic arms, in order to date objects of art or collect information on family histories or historical events. The great interest of the specialists throughout the world, in the compilations of such manuscripts is witnessed by the large number of consultation requests of the codices to museums and libraries throughout the world. The processing of such requests are currently faced with three main obstacles: – Information is available only locally: At present is not possible to answer all of these requests accurately and there are long waiting times due to the time taken to search through the various sources of information. Such information is usually available in the form of the original manuscripts which are often fragile and can only be used by trained curators. Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL’99, LNCS 1614, pp. 211–220, 1999. c Springer-Verlag Berlin Heidelberg 1999


212

V. Konstantinou and A. Psarrou

– Indexing of codices of arms is subjective: Codices of arms are often described by keywords that characterise objects or shapes depicted on them and their spatial arrangement. However, text classification is not unique because the keywords used in the indexing of codices are subjective to the opinion of each researcher and in most cases are fairly general (i.e. the word ’flower’ can be used to describe a number of real-life flowers). In addition, keywords used in heraldry vary broadly across languages and cultures. – Indexing of codices of arms is not governed by heraldic rules: The most challenging obstacles are met from requests that are concerned with the determination of the origin of codices that don’t have a record in the current database. In this case a search is requested to provide images that closely match the enquired codice. However, Venetian heraldry is not governed by any standard set of heraldic rules and researchers rely more on plain pictorial information. The aim of the HISTORIA project is to overcome the problems associated with handling the original and usually very fragile manuscripts and provide a digital library which is incorporated with search and query tools tailored for (a) extracting information from reproductions of medieval manuscripts (b) meeting the needs of user groups such as historians and curators. We have addressed these issues by designing a digital library that has the following modules and functionality: – Information Server: A special Server (The Historia Server) module that accommodates and controls all of the system modules. The Historia Server is also linked to a Web server through which it can be accessed by both Intranet and Internet users. – User Interface: To enable both local and remote access of the system a JAVA based client-server interface was designed. The interface is loaded whenever a user visits a special Web page linked to the Historia server. – Image base: A specially-designed digital image library is used to store and retrieve low and high definition reproductions of images and text that represent (a) pages of various sources/books that describe the genealogy of Venetian families and (b) Venetian heraldry associated with these families. – Index base: During the acquisition/information collection phase the images are indexed according to family names, colour and pictorial information. – Hyper-imaging: A dedicated hybrid (text and/or image) search engine which finds images that contain similar shapes and colours is used for image or text based searches. – Drawing tool: A drawing tool which allows the user to draw and colour representations that resemble the image that would like to retrieve. The facility can also be used to change and manipulate images already in the database. – Network links: Objects in the digital library can be linked with two types of links: (a) hard coded links that associate pages of medieval books that describe the genealogy of noble Venetian families to images of Venetian heraldry and (b) user defined links that hold the user notes.

A Dynamic JAVA-Based Intelligent Interface

213

– Hyper-text: The user can also create links to individual text notes which can in turn be automatically linked to other text or image related information. For this purpose we have employed a special automatic Hypertext engine developed at the University of Westminster.

2

Background

Research in digital image libraries is widely known as content-based retrieval where search is not based on the indexing and retrieval of information using keywords but on the appearance of whole or parts of the image. The approaches used in many digital libraries are classified according to the type of generic query formats used in retrieving information, for example browsing, colour, texture, sketch, shape, motion and others [6]. In addition, search and retrieval approaches are divided to image domain dependent or independent. Many times, however, the notion of domain independent approaches is associated with techniques that do not preserve any semantic information of the content of the images. Such approaches, even though they might be regarded as generic, fail in the long term to produce satisfactory results for two main reasons: (a) the search is based on some mathematical formula that does not provide unique representation of shapes and (b) does not cater for information that the user may be interested to include. In contrast direct meaningful search on images is only possible by use of semantic-preserving image compression [9]. Such compact representation may or may not be dependent on priory knowledge of the information that appears in the image but preserve essential image similarities. The need for the development of image indexing and retrieval algorithms tailored to specific domains has been raised by a number of curators and art researchers. In particular it was shown that many of the commercial and research image databases available have not been successful in retrieving similar images from an art domain. For example, QBIC [4,3] had only a 10% success rate when used to relate similar landscape paintings of an artist [5]. The main reason for this is that many standard image processing algorithms assume that images are the perspective or parallel projection of a three-dimensional world. However, artists have struggled to represent the world filtered through their imagination and it is rare, for example, to find paintings that, despite superficial appearances, present even a perspective view from a single fixed viewpoint [7]. In our work we are looking at the development of a digital library to host medieval manuscripts that contain hand drawings of (a) rectangular shapes or stripes and (b) irregular shapes of animals or objects. The Algorithms developed to index and search this library have been based on the needs of the users [10]. From the algorithmic point of view, they are based on the analysis of colour histograms and their application on region growing algorithms.

214

3

V. Konstantinou and A. Psarrou

System Overview

To achieve the functionality of the image base described in section 1, the operation of the HISTORIA system can be divided into two main cycles (phases): 1. The acquisition phase: During this phase the image base, the index base and the hard coded network links are created. During the first stage the scanned images are automatically processed to extract features that describe their colour and image regions or the outline of shapes in the image and their spatial organisation. The extracted features are stored in a SQL-formed database and augmented with text information. 2. The query phase: During this phase the user can query the image base through the hyper-imaging and hyper-text links and create his/her own user defined network links. During the query phase the user composes queries using one of the following means (a) ’family name’ or other text associated with the heraldic images (b) appearance information that is related to the pictorial or colour information of the image. During this phase a process similar to the database acquisition is performed to the input image. The features extracted from the search image (if the search is based on an image as mentioned in b above) are stored again in a similar SQL format and matched against the stored database information. The user can select to use more than one of the matching mechanisms available (see Hyper-imaging below). The resulting set of images from the database is based on similarity. The similarity is calculated with the aid of distance functions between the features.

4

Hyper-Imaging

Representation and association of images in HISTORIA is based on two types of image analysis [10]. Colour histogram analysis and regional analysis. 4.1

Colour Histogram Analysis

Colour queries let users find images that have similar colour distribution. This is preferred because the required heraldic features some times relate to the colour used in the coats of arms (emblems). The search algorithm used here is based on colour histogram analysis. Although the RGB basis is good for the acquisition or display of colour information, it is not particularly good basis to explain the perception of colours. Following Ballard and Brown [1] we transform the RGB values to opponent colours defined as follows: rg = r − g by = 2 ∗ g − r − g wb = r + g + b Based on opponent colours histograms we create colour models for the emblems which we then match using a Histogram Intersection technique [11].

A Dynamic JAVA-Based Intelligent Interface

4.2

215

Region Analysis

The main problem in the retrieval of images based on pictorial information is the segmentation of the images into meaningful parts. In our case the segmentation of images require the partition of the images into homogeneous regions that represent composite shapes found in heraldry. Grey level thresholding is a tool widely used in image segmentation, where a single or band of threshold values is used to discriminate objects from background. The most important problem is selecting the grey-level values that constitute the right thresholds for segmenting the image [12]. We can select appropriate threshold values for image segmentation based on information regarding the number of neighbouring pixels that have the same gray level. Such information can be derived from local spatial analysis of the grey level values of the images and can be represented in a co-occurrence matrix. This is a second-order statistics measure of the image gray levels occurring in specified relative positions and reveals spatial variation of gray levels (i.e. homogeneity) in an image. It thus allows homogeneous regions to be identified and separated from cluttered background. In particular, the diagonal of the co-occurrence matrix represents the number of neighbouring pixels that have the same grey level. Consequently, clusters near the diagonal of the co-occurrence matrix provide an indication of the grey level of the homogeneous regions in an image. Using such representation we can select threshold values that minimise the amount of busyness, noise or roughness present in the image in the thresholded image. If we assume that the ideal objects and background have simple compact shapes then noise points and holes in the object and background of the thresholded image and strong irregularities in the object background border are undesirable as we normally want thresholded images to look smooth rather than busy. Busyness measures using co-occurrence matrices Following Chanda and Majumber [2] and Lie [8], we define g(x, y) to be an L-level monochrome image of size M × M . We are interested in considering the co-occurrence of gray levels in pairs of pixels where one is lying at a distance of d in the direction θ with respect to the other. Here we take the value d = 1 and θ as an integer multiple of 12 π. The element Cθ,m,n gives the frequency that a pixel having gray-level n occurs adjacent to a pixel having gray-level m in direction θ. Considering all 4-neighbourhood directions the co-occurrence matrix C is obtained as: C = Cm,n = C0 + Cπ/2 + Cπ + C3π/2 where Cn represents the co-occurrence statistics of pairs of pixels adjacent in the θ direction. Let t be the threshold for image binirisation that maps all gray levels greater than t into the object and all other levels into the background. This mapping partitions the co-occurrence matrix into four distinct areas [12]. 1. matrix elements representing co-occurrences of gray levels in the object. i.e. those C(i, j) such that i > t and j > t (shaded area B3 )

216

V. Konstantinou and A. Psarrou

Fig. 1. The threshold value t divides the co-occurrence matrix into four non-

overlapping blocks. The values on each diagonal element Ci,i of the co-occurrence matrix represent the ith entry of the grey level histogram of the image.

2. matrix elements representing co-occurrences of gray levels in the background. i.e. those C(i, j) such that i ≤ t and j ≤ t (shaded area B4 ) 3. matrix elements representing co-occurrences of object gray levels with background gray levels. i.e. those C(i, j) such that i ≥ t and j < t (shaded areas of B1 ) or i > t and j ≤ t(shaded area and B2 ) Given a threshold t of an image, the measure of busyness C(t) that was used throughout this work is computed by summing those entries of the co-occurrence matrix representing the percentage of object-background adjacencies. (i.e. the entries in the B1 and B2 areas). If C(t) is relatively high for a given threshold we would expect the thresholded image to contain a large number of noise points and/or jagged edges. Conversely a relatively low C(t) would indicate that the threshold chosen results in a smooth picture. C(t) will be zero if all grey levels are mapped into the same output level. To avoid this we require that the threshold lie between the object and background means. Once the co-occurrence matrix C has been computed, the busyness measure C(t) can be calculated for all thresholds using the recurrence relationship t−1 n C(i, t) + Σj=t+1 C(t, j) C(t) = C(t − 1) − Σi=1

where n is the number of gray levels in the image and the dimension of M . The methods of threshold selection using the co-occurrence matrix looks for the threshold for which the number of pairs of border pixels i.e. sum of Cm,n over the blocks B1 and B2 is minimal. In other words, it searches for a threshold which segments the image into the largest homogeneous regions possible.

A Dynamic JAVA-Based Intelligent Interface

4.3

217

Region Labelling

Only under very unusual circumstances can thresholding be successful using a single threshold for the whole image since even in very simple images there are likely to be grey level variations in objects and background. Better segmentation results can be achieved using variable thresholds in which the threshold value varies over the image as a function of local image characteristics. To find such thresholds we represent the busyness measure C(t) obtained from the co-occurrence matrix as a function of threshold t It should be noted that the busyness curve, as a function of threshold must should have the same general shape as the grey level histogram. This is because when we threshold at a point on a histogram peak, i.e. within the object or background gray level range, we may expect high degree of busyness in the thresholded image; whereas when we threshold in the valley between the object and background, the busyness should be relatively low. Therefore, C(t) is represented in a histogram which is searched for local draughts. The local draughts in the histogram correspond to the threshold values that are used to define the boundaries of homogeneous regions in an image. The thresholds found with this process are listed in an ascended order and represent pairwise the boundary values of regions. The labelling process starts separates the images in regions by using stacks and performing a recursive search in neighbouring pixels. The region description of the images is performed using shape metrics that describe: (a) area as the number of pixels occupied by the region, (b) elongation as the ratio of the length and width of the smallest bounding rectangle encompassing the region and (c) compactness as the ratio of the square of the length of the region boundary and the region area size. The description of image regions using simple metric values has the following advantages that they describe a region independent of its colour content and require minimal storage which is held in SQL tables and be queried. Figure 2 shows a search result based on combined histogram and region analysis on one emblem of the Basegnio family.

Fig. 2. Search results on one of the emblems of the Basegnio family.

218

5

V. Konstantinou and A. Psarrou

The HISTORIA JAVA Interface

The HISTORIA end-user interface has been augmented by incorporating a frontend shown in Figure 3 that allows the user to access the database though the WWW and MsqlJava classes.

Fig. 3. The Java interface of HISTORIA

The screen in the Java interface is essentially divided up into four main parts: 1. The top-left area id headed Families and contains a scrolling list of the names of the families for which there are emblems in the database. 2. The top-right area is headed Image Editor. This provides a canvas and a set of drawing tools and options to allow the user to create small images to use as search items. 3. The middle panel is titled Search By and its main purpose is to keep a visual record of the search term that produced the results shown. 4. The bottom panel, titled Search Results, displays the search results for the search term shown in the middle panel. Each retrieved image can the be used as retrieval key as it is or changed using the image editor panel. The functionality of the Java interface is shown in Figure 4. A member of the Badoeri family is selected in the Search By panel and similar images retrieved. The image is then copied to the Image Editor panel where it can be modified and then used as a key retrieval image. The interface is compatible with the JAVA implementation found in both Netscape (v3 or greater) and Internet Explorer (v4 or greater).

A Dynamic JAVA-Based Intelligent Interface

219

Fig. 4. The Search and Editing facilities on the Java Interface The ’canvas’ provided with the interface enables the user to sketch and even colour parts of an emblem that interests him/her. As the Server incorporates both object and colour-based search algorithms the ’sketches’ can be just rough outlines of objects or major coloured regions appearing the emblem in question.

6

HISTORIA Intranet and Internet Access

The use of a Java i/f enables the library which uses the HISTORIA system to make it available to all of the interested researchers without the need for upgrading any of the equipment. The Marcianna library for which the system was developed, currently uses a NOVELL/UNIX network but its users have a variety of machines ranging from 486 Pcs running Windows v3 to more modern Pentium II machines. The system runs on a Silicon Graphics O2 running IRIX version 6.3. The Interface has been designed in such a way as to auto detect local (Intranet) users and enable extended search, printing and display features (high resolution images). This way the Library can maintain its copyright and control the distribution of the high quality images.

7

Experimental Search Results

The prototype image-base currently hosted by Marcianna library, includes approximately 900 images, and the success rates based on two available matching mechanisms are as follows: Image Type Histogram Region Analysis Combined Known 100 100 100 Related 60 70 90 All numbers shown above are success percentages. ’Known’ indicates that the image is already stored in the image base. Related are the relevant images that contain similar items or colours to the ’searched’ image. The success rates

220

V. Konstantinou and A. Psarrou

for the related images were derived by comparing the HISTORIA results and the sets derived by human scholars given the same sources.

8

Conclusion

One benefit of the HISTORIA system is that it can improve access to the information contained within the manuscripts and coats of arms held in database, allowing individual researchers ”hands on” access to the database. With this in mind the documentation system has been designed such that it can be extended to provide individual researchers with a personalized research tool allowing them to record research paths and the results of previous searches and to define their own links between families and images with associated notes. The JAVA interface apart from offering world-wide access, it also enables the operators to protect the copyright of their documents by auto-detecting local users and enabling different features as appropriate. The Interface is currently updated so that it can interrogate a number of Historia servers and in that way enable true distributed image searches.

References 1. D. Ballard and C. Brown. Computer Vision. Prentice Hall, 1982. 214 2. B. Chandra and D. Dutta Majumder. A note on the use of the graylevel cooccurrence matrix in threshold selection. Signal Processing, 15(2):149–167, 1988. 215 3. Myron Flickner et al. Query by image and video content: The QBIC system. IEEE Computer, 28(9):23–32, September 1995. 213 4. W. Niblack et al. The QBIC project: Querying images by content using colour, texture and shape. In Storage and Retrieval for Image and Video Databases I, Proc. SPIE 1908, pages 173–187, 1993. 213 5. Catherine Grout. From ‘virtual curator’ to ‘virtual librarian’: What is the potential for the emergent image recognition technologies in the art-historical domain? In Electronic Imaging and the Visual Arts, London, 1996. 213 6. V. Gudivada and V. Raghavan. Content-based image retrieval systems. IEEE Computer, 28(9):18–22, September 1995. 213 7. J. Lansdown. Some trnds in computer graphic art. In S. Mealing, editor, Computers & Art. intellect, 1997. 213 8. W.-N. Lie. An efficient threshold-evaluation algorithm for image segmentation based on spataial graylevel co-occurrences. Signal Processing, 33(1):121–126, July 1993. 215 9. A. Pentland, R.W. Picard, and S. Sclaroff. Photoshop: Tools for content-based manipulation of image databases. In Storage and Retrieval for Image and Video Databases I, Proc. SPIE 2185, pages 34–47, 1993. 213 10. A. Psarrou, S. Courtenage, V. Konstantinou, P. Morse, and P. O’Reilly. Historia: Final report. Telematics for Libraries, 3117. 213, 214 11. M. Swain and D. Ballard. Color indexing. International Journal of Computer Vision, 7(1):11–32, 1991. 214 12. J. S. Weszka and A. Rosenfeld. Threshold evaluation techniques. IEEE Transactions on Systems, Man, and Cybernetics, 8(8):622–629, 1978. 215

Motion-Based Feature Extraction and Ascendant Hierarchical Classification for Video Indexing and Retrieval Ronan Fable1 and Patrick Bouthemy2 1

IRISA / CNRS IRISA / INRIA Campus universitaire de Beaulieu, 35042 Rennes Cedex, France Tel: (33) 2.99.84.25.23, Fax: (33) 2.99.84.71.71 {rfablet,bouthemy}@irisa.fr 2

Abstract. This paper describes an original approach for motion characterization with a view to content-based video indexing and retrieval. A statistical analysis of temporal cooccurrence distributions of relevant local motion-based measures is exploited to compute global motion descriptors, which allows to handle diverse motion situations. These features are used in an ascendant hierarchical classification procedure to supply a meaningful hierarchy from a set of sequences. Results of classification and retrieval on a database of video sequences are reported.

1

Introduction

Image databases are at the core of various application fields, either concerned with professional use (remote sensing and meteorology from satellite images, road traffic surveillance from video sequences, medical imaging, . . . ) or targeted at a more general public (television archives including movies, documentaries, news, . . . ; multimedia publishing,. . . ). Reliable and convenient access to visual information is of major interest for an efficient use of these databases. Thus, it exists a real need for indexing and retrieving visual documents by their content. A large research amount is currently devoted to image and video database management, [1,7,16]. Nevertheless, due to the complexity of image interpretation and dynamic scene analysis, it remains hard to easily identify relevant information with regards to a given query. As far as image sequences are concerned, content-based video indexing, browsing, editing, or retrieval, primarily require to recover the elementary shots of the video and to recognize typical forms of video shooting such as static shot, traveling, zooming and panning [1,3,15,16]. These issues also motivate studies concentrating on image mosaicing [9], on object motion characterization in case of a static camera [4], or on segmentation and tracking of moving elements, [6]. These methods generally exploit motion segmentation relying either on 2d parametric motion models or on dense optical flow field estimation. They aim at Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL’99, LNCS 1614, pp. 221–229, 1999. c Springer-Verlag Berlin Heidelberg 1999


222

Ronan Fable and Patrick Bouthemy

determining a partition of a given scene into regions attached to different types of motions with a view to extracting relevant moving objects. Nevertheless, they turn out to be unadapted to certain classes of sequences, particularly in the case of unstructured motions of rivers, flames, foliages in the wind, or crowds, . . . , (see Figure 1). Moreover, providing a global interpretation of motion along a sequence, without any prior motion segmentation or without any complete motion estimation in terms of parametric models or optical flow fields, seems in the context of video indexing attractive and achievable to discriminate general types of motion situations. These remarks emphasize the need of designing new low-level approaches in order to supply a direct global motion description, [2,11,13,14]. We propose an original approach to video indexing and retrieval according to the motion content. It relies on the global motion-based features presented in our previous work [2]. They are extracted using a statistical analysis of temporal cooccurrences of local non-parametric motion-related information. These motion indexes are introduced in a flexible ascendant hierarchical classification scheme to determine a meaningful hierarchy from a large video sequence set. It expresses similarities based on some metrics in the feature space. We can easily exploit the computed hierarchy for efficient retrieval with query by example. This paper is organized as follows. In Section 2, we outline the general ideas leading to our work. Section 3 briefly describes the motion-based feature extraction. In Section 4, we introduce the indexing structure and the retrieval procedure. Section 5 contains classification results and retrieval examples, obtained on a large set of video sequences, and Section 6 contains concluding remarks.

2

Problem Statement and Related Work

Video sequences are first processed to extract elementary shots with the technique presented in [3] (note that in the following we may use the term of sequence to deal with an elementary shot). Then, for each previously extracted shot, we intend to characterize the whole spatio-temporal motion distribution in order to build a motion-based indexing and retrieval system. Let us note that, in the same manner, texture analysis methods study the spatial grey-level distribution. In particular, cooccurrence measurements provide efficient tools for texture description in terms of homogeneity, contrast or coarseness [8]. Therefore, we aim at adapting cooccurrence-based features in the context of motion analysis. Preliminary research in that direction was developed by Polana and Nelson for activity recognition [11]. As part of their work, they introduce the notion of temporal texture, opposed to periodic activities or rigid motions, and associated to fluid motions. Indeed, motions of rivers, foliages, flames, or crowds, . . . , can be regarded as temporal textures (see Figure 1). In [14], temporal texture synthesis examples close to the original sequences are reported. However, this work is devoted to these particular cases of dynamic scenes, and cannot be extended to rigid motions or periodic activities. In [13], temporal texture features are extracted based on the description of spatio-temporal trajectories. However, it relies on detection of moving contours

Motion-Based Feature Extraction

a)

223

b)

Fig. 1. Examples of temporal textures : a) foliage b) fire (by courtesy of mit). by a simple thresholding of the pixel-based frame differences, which are known to be noisy. In the subsequent, maps of local motion measures along the image sequence are required as input of cooccurrence measurements. As dense optical flow field estimation is time-consuming and unreliable in case of complex dynamic scenes, we prefer to consider local motion-related information, easily computed from the spatio-temporal derivatives of the intensity. Rather than the normal velocity used in [11], a more reliable information is exploited as explained in the next section. Besides, we intend to design a new video indexing and retrieval approach using the global motion-based features extracted from the temporal cooccurrences statistics. Thus, we first need to determine a meaningful indexing structure on a large dataset. Among all the clustering methods, we focus on ascendant hierarchical classification (AHC), [5,10]. It exploits a Euclidean norm on the motion-based feature space and aims at minimizing the within-class variances. The obtained hierarchical representation is directly exploited for efficient retrieval with query by example.

3 3.1

Extraction of Global Motion-Based Features Local Motion-Related Measures

By assuming intensity constancy along 2d motion trajectories, the well-known image motion constraint relates the 2d apparent motion and the spatio-temporal derivatives of the intensity function, and the normal velocity vn at a point p is −It (p) given by : vn (p) = ∇I(p) where I(p) is the intensity function, ∇I = (Ix , Iy ) the intensity spatial gradient, and It (p) the intensity partial temporal derivative. If the motion direction is orthogonal to the spatial intensity gradient, this quantity vn can in fact be null whatever the motion magnitude. vn is also very sensitive to noise attached to the computation of the intensity derivatives. Nevertheless, an appropriately weighted average of vn in a given neighborhood forms a more relevant motion-related quantity as shown in [12] :
2 s∈F (p) ∇I(s) · |vn (s)|
(1) vobs (p) = max(η 2 , s∈F (p) ∇I(s)2 ) where F (p) is a 3 × 3 window centered on p. η 2 is a predetermined constant, related to the noise level in uniform areas, which prevents from dividing by zero

224

Ronan Fable and Patrick Bouthemy

or by a very low value. Thus, vobs provides us with a local motion measure, easily computed and reliably exploitable. The loss of the information relative to motion direction is not a real shortcoming, since we are interested in interpreting the general type of dynamic situations observed in a given video shot. The computation of cooccurrence matrices can not be achieved on a set of continuous variables. Due to the spreading out of the measures vobs , a simple linear quantization within the interval [inf p vobs (p); supp vobs (p)] is not pertinent. Since it is generally assessed in motion analysis that large displacements can not be handled through a single resolution analysis, we set a limit beyond which measures are no more regarded as reliable. Thus, in practice, we quantize linearly the motion quantities within [0, 4] on 16 levels. 3.2

Global Motion Features

In [11], spatial cooccurrence distributions are evaluated on normal flow fields to classify processed examples in pure motion (rotational, divergent) or in temporal texture (river, foliage). In that case, since studied interactions are spatial, only motions which are stationary along the time axis can be characterized. Moreover, to recover the spatial structure of motion, several configurations corresponding to different spatial interactions have to be computed, which is highly time-consuming. Consequently, we focus on temporal cooccurrences defined for a pair of quantized motion quantities (i, j) at the temporal distance dt by :    (r, s) ∈ Cdt /obs(r) = i, obs(s) = j Pdt (i, j) = (2) |Cdt |  where obs holds for the quantized version of vobs , and Cdt = (r, s) at the  same spatial position in the image grid /∃t, r ∈ image(t) and s ∈ image(t− dt ) . From these cooccurrence matrices, global motion features similar as those defined in [8] are extracted  1:
f = P (i, j) log(Pdt (i, j))   2
(i,j) dt  2    f =
(i,j) Pdt (i, j)/[1 + (i − j) ]  3 2 f = (i,j) (i − j) Pdt (i, j) 

(3)  4 4 2  = i P (i, j) / i P (i, j) − 3 f d d  t   (i,j) (i,j) 
t    f5 =
4 2 (i − j) P (i, j) / dt (i,j) (i,j) (i − j) Pdt (i, j) − 3 where f 1 is the entropy, f 2 the inverse difference moment, f 3 the acceleration, f 4 the kurtosis and f 5 the difference kurtosis. This set of global motion features is in this work computed over all the image grid. In order to cope with non-stationarity in the spatial domain, we can easily obtain a region-based characterization of motion. Indeed, the extraction of the motion descriptors can also be achieved either on predefined blocks or on extracted regions resulting from a spatial segmentation, since we focus only on temporal interactions. In that case, the retrieval process will consist in determining regions of sequences of the database similar in terms of motion properties to those characterized for the processed query.

Motion-Based Feature Extraction

4 4.1

225

Motion-Based Indexing and Retrieval Motion-Based Indexing

Since we plan to design an efficient indexing and retrieval scheme based on the global motion features presented above, we are required to build an appropriate representation of the database. This will allow us to recover easily sequences similar in terms of motion properties to a given video query. Thus, we have to make use of a classification method in order to cluster video sequences into meaningful groups. Among the numerous clustering algorithms, we have selected an iterative process called ascendant hierarchical classification (AHC) [5]. Due to its simplicity of computation and its hierarchical nature, it reveals efficient for image and video database management as shown in [10]. It comes to compute a binary decision tree expressing the hierarchy of similarities between image sequences according to some metrics. Let us consider a set of motion-related feature vectors, fn = (fn1 , . . . , fn5 ) where n refers to a sequence in the database. The AHC algorithm proceeds incrementally as follows. At a given level of the hierarchy, pairs are formed by merging the closest clusters in the feature space in order to minimize the withinclass variance and maximize the between-class centered second-order moment. We will use the Euclidean norm. Moreover, if an element n represented by a feature vector fn is too far from all the others one i.e. minm fn − fm 2 > Vmax , where Vmax is a predefined constant, it forms also a new cluster. This procedure is iterated from the lowest level to the upper one in the hierarchy. To initialize the algorithm at the lowest level, each cluster corresponds to a unique sequence. In our experiments, we have extracted the motion-based descriptors presented in section 3.2 with a temporal distance dt = 1. Nevertheless, we cannot directly use the Euclidean norm with such features of different nature. In order to exploit this norm to compare feature vectors, we compute for the feature f 3 its square root and we raise the features f 4 and f 5 to the one fourth power. 4.2

Retrieval with Query by Example

We are interested in retrieving sequences of the database the most similar to a given video query. More particularly, we focus on matching sequences according to global motion properties. Indeed, the index structure described above provides us with such an efficient hierarchical motion-based retrieving tool. We compute first the hierarchical index structure over the video database. Second, to handle the submitted query, the proposed sequence is processed to extract the meaningful motion-based features. In the same manner as previously, we compute the square root of the feature f 3 and the power one fourth of features f 4 and f 5 in order to use the Euclidean norm as cost function. Then, we explore the hierarchy of sequences as follows. At its upper level, the retrieval algorithm selects the closest cluster, according to the Euclidean distance to the center of gravity of the considered cluster in the

226

Ronan Fable and Patrick Bouthemy

feature space. Then, for each of the children nodes, the distance from the feature vector of the query video to the center of gravity of each cluster is computed, and the cluster with the shortest distance is selected. This procedure is iterated through the index structure until a given number of answers or a given similarity accuracy is reached.

5

Results and Concluding Remarks

We make use of the approach described above to process a database of image sequences. We have paid a particular attention to choose video representative of various motion situations. Indeed, the database includes temporal textures such as fire or moving crowds, examples with an important motion activity such as sport video (basket, horse riding,...), rigid motion situations (cars, train, ...), and sequences with a low motion activity. Finally, we consider a database of 25 video sequences (typically, each sequence is composed of 10 images). First, AHC is applied to the database in the space (f 1 , f 2 , f 3 , f 4 , f 5 ). In Figure 2, the representation of the database in the feature space, restricted to (f 3 , f 4 , f 5 ) space for visualization convenience, is reported. The four sequence classes of level 4 in the hierarchy are really related to different types of motion situations : the class “o” involves temporal textures, the class“x” includes sport video motions, elements of the class “+” are related to rigid motion situations and the class “.” is composed of low motion activity examples.

9 8 7 6 5 4

class o

class x

class o

class * Figure 2.b

3 2 1 3 2.5

6 5

2 4

1.5

3 2

1

1 0.5

0

Figure 2.a

Fig. 2. Representation of the video database obtained with the AHC : a) Spreading of the sequences in the restricted feature space (f 3 , f 4 , f 5 ). Symbol (+,o,.,*) are indexes of classes at the level 4 in the AHC hierarchy. b) Examples representative of the extracted classes. We display the first image of the sequence which is the closest from the center of gravity of its class. Now, we deal with motion-based retrieval for query by example. Fig. 3 shows results obtained with two video queries. The maximum number of answers to

Motion-Based Feature Extraction

query 1 : high activity

answer 1

answer 2

answer 3

query 2 : low activity

answer 1

answer 2

answer 3

227

Fig. 3. Results of motion-based retrieval operations with query by example for a maximum of three answers. We display for each selected sequence its first image. a given query is fixed to 3. The first example is a horse riding sequence. The retrieval process supplies accurate answers of sport shots which appear similar to the query in terms of global motion properties. The second video query is a static shot of a meeting. It is matched with other low motion activity sequences. Let us proceed to a more quantitative evaluation of our approach. Since it seems difficult to directly analyze the accuracy of the classification scheme, we use the following procedure. First, we define a priori sequence classes among the dataset according to visual perception. Then, we analyze the three retrieved answers when considering each element of the base as a query. To evaluate the accuracy of our retrieval scheme, we consider two measures. We count the number of times that the query shot appears as the best answer, and, on the other hand, if the second retrieved sequence belongs to the same a priori class, we consider the retrieval process as correct. In practice, we have determined four a priori sequence classes : the first one with low motion activity, the second with rigid motions, important motion activity examples forms the third one, and temporal textures the fourth one. Even if this evaluation procedure remains somewhat subjective, it delivers a convincing validation of the indexing and retrieval process. Obtained results for the whole database are rather promising : similar query and first retrieved answer (%) correct classification rate according to a priori class (%)

80 75

Table 1. Evaluation of the motion-based indexing and retrieval process

6

Conclusion

We have described an original method to extract global motion-related features and its application to video indexing and retrieval. Motion indexes rely

228

Ronan Fable and Patrick Bouthemy

on a second-order statistical analysis of temporal distributions of relevant local motion-related quantities. We exploit a hierarchical ascendant classification to infer a binary tree over the video database. Examples of retrieval using query by example have shown good results. In future work, we should determine optimal sets of global features adapted to different types of content in the video database, and evaluation over a still larger database should be performed.

Acknowledgments: This work is funded in part by AFIRST (Association Franco-Israelienne pour la Recherche Scientifique).

References 1. P. Aigrain, H.J. Zhang, and D. Petkovic. Content-based representation and retrieval of visual media : a state-of-the-art review. Multimedia Tools and Applications, 3(3):179–202, November 1996. 221 2. P. Bouthemy and R. Fablet. Motion characterization from temporal cooccurrences of local motion-based measures for video indexing. In Proc. Int. Conf. on Pattern Recognition, ICPR’98, Brisbane, August 1998. 222 3. P. Bouthemy and F. Ganansia. Video partioning and camera motion characterization for content-based video indexing. In Proc. 3rd IEEE Int. Conf. on Image Processing, ICIP’96, Lausanne, September 1996. 221, 222 4. J.D. Courtney. Automatic video indexing via object motion analysis. Pattern Recognition, 30(4):607–625, April 1997. 221 5. E. Diday, G. Govaert, Y. Lechevallier, and J. Sidi. Clustering in pattern recognition. In Digital Image Processing, pages 19–58. J.-C. Simon, R. Haralick, eds, Kluwer edition, 1981. 223, 225 6. M. Gelgon and P. Bouthemy. Determining a structured spatio-temporal representation of video content for efficient visualization and indexing. In Proc. 5th European Conf. on Computer Vision, ECCV’98, Freiburg, June 1998. 221 7. B. Gunsel, A. Murat Tekalp, and P.J.L. van Beek. Content-based access to video objects : temporal segmentation, visual summarization and feature extraction. Signal Processing, 66:261–280, 1998. 221 8. R.M. Haralick, K. Shanmugan, and I. Dinstein. Textural features for image classification. IEEE Trans. on Systems, Man and Cybernetics, 3(6):610–621, Nov. 1973. 222, 224 9. M. Irani and P. Anandan. Video indexing based on mosaic representation. IEEE Trans. on PAMI, 86(5):905–921, May 1998. 221 10. R. Milanese, D. Squire, and T. Pun. Correspondence analysis and hierarchical indexing for content-based image retrieval. In Proc. 3rd IEEE Int. Conf. on Image Processing, ICIP’96, Lausanne, September 1996. 223, 225 11. R. Nelson and R. Polana. Qualitative recognition of motion using temporal texture. CVGIP : Image Understanding, 56(1):78–99, July 1992. 222, 223, 224 12. J.M. Odobez and P. Bouthemy. Separation of moving regions from background in an image sequence acquired with a mobile camera. In Video Data Compression for Multimedia Computing, chapter 8, pages 295–311. H. H. Li, S. Sun, and H. Derin, eds, Kluwer, 1997. 223

Motion-Based Feature Extraction

229

13. K. Otsuka, T. Horikoshi, S. Suzuki, and M. Fujii. Feature extraction of temporal texture based on spatiotemporal motion trajectory. In Proc. Int. Conf. on Pattern Recognition, ICPR’98, Brisbane, August 1998. 222 14. M. Szummer and R.W. Picard. Temporal texture modeling. In Proc. 3rd IEEE Int. Conf. on Image Processing, ICIP’96, Lausanne, September 1996. 222 15. W. Xiong and J. C.H. Lee. Efficient scene change detection and camera motion annotation for video classification. Computer Vision and Image Understanding, 71(2):166–181, August 1998. 221 16. H.J. Zhang, J. Wu, D. Zhong, and S. Smolier. An integrated system for contentbased video retrieval and browsing. Pattern Recognition, 30(4), April 1997. 221

Automatically Segmenting Movies into Logical Story Units Alan Hanjalic, Reginald L. Lagendijk, and Jan Biemond Faculty of Information Technology and Systems Information and Communication Theory Group Delft University of Technology P.O.Box 5031, 2600 GA Delft, The Netherlands {alan,inald,biemond}@it.et.tudelft.nl Abstract. We present a newly developed strategy for automatically segmenting movies into logical story units. A logical story unit can be understood as an approximation of a movie episode and as the base for building an eventoriented movie organization structure. The automation aspect is becoming increasingly important with the rising amount of information in emerging digital libraries. The segmentation process is designed to work on MPEG-DC sequences and can be performed in a single pass through a video sequence.

1 Introduction For an easy user interaction with large volumes of video material in emerging digital libraries efficient organization of the stored information is required. In this paper we concentrate on movies as a particularly important class of video programs and emphasize the need for an event-oriented movie organization scheme. Humans tend to remember different events after watching a movie and think in terms of events during the video retrieval process. Such an event can be a dialog, action scene or, generally, any series of shots “unified by location or dramatic incident”[5]. Therefore, an event as a whole should be treated as an elementary retrieval unit in advanced movie retrieval systems. We propose a novel method for automatically segmenting movies into logical story units. Each of these units is characterized by one or several temporally interrelated events, which implies that the segmentation result can provide a concise and comprehensive top level of an event-oriented movie organization scheme. The proposed high-level segmentation method can be carried out in a single pass through a video sequence.

2 From Episodes to Logical Story Units Each shot [1] within a movie program belongs to a certain global context built up around one movie event or several of them taking place in parallel. Thereby, a shot can either be a part of an event or serve for its “description”, by e.g. showing the scenery where the coming or the current event takes place, showing a “story telling” narrator in typical retrospective movies, etc. In view of such a distinction, we will further refer to shots of a movie as either event shots or descriptive shots.

Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL'99, LNCS 1614, pp.229 -236, 1999.  Springer-Verlag Berlin Heidelberg 1999

230

Alan Hanjalic et al.

We can now realistically assume that a standard movie is produced as a series of meaningful segments corresponding to event-oriented global contexts described above, which we will call episodes. An episode is generally a combination of the event shots and descriptive shots related to the event(s) of the episode. It can be simple, if it concentrates on one event only. However, a more complex episode structure is possible as well. This is the case where several events, taking place in parallel, are presented as a series of their interchanging fragments. We denote the fragment i of the j

event j by Ti and introduce a model for the movie structure as shown in Fig. 1, built up by concatenating episodes of different complexity. Descriptive shots

T11

T11

T12

T13

T22

T23

T14

Event-shots

Episode 1

Episode 2

Episode 3

Fig. 1. A sample movie sequence consisting of three episodes. Descriptive shots are described as boxes with lined patterns.

In view of the event-based structure of an episode and the assumed limited number of episodes in a typical movie, a movie segmentation into episodes can provide a highly suitable top level for a compact and comprehensive event-oriented movie organization scheme. However, such a segmentation can be performed precisely, only if the movie script is available. This is not the case in automated sequence analysis systems, especially those operating at the user side [3] of a video transmission network. In such systems, all movie content analysis, segmentation and organization processes are done based on movie’s audiovisual characteristics and their temporal variations, measured and captured by standard audio, image and video processing tools. In this paper, we perform the movie segmentation using visual features only. As a result, the approximates of the actual movie episodes are obtained, which we will call logical story units (LSUs). Various applications in digital video libraries can benefit from an LSU-based movie organization scheme. For example, an overview of a movie can be obtained immediately if one looks at the obtained set of LSUs. Fig. 2 illustrates how a movie can be broken up into LSUs and how existing content-based clustering algorithms can be applied to all shots of an LSU. The shots of each LSU that are most representative can be glued together and be played as movie highlights. One can also use key frames to browse through each individual LSU, which is an especially important feature for LSUs having a complicated structure (e.g. containing several temporally interrelated events). The user only browses through relevant shots, e.g. those relating to the selected LSU (for instance, when searching for a particular movie character in the context of a certain event), and is not burdened with (the many) other shots of a sequence. For each granularity (cluster) level, a key-frame set is available providing video representations through pictorial summaries having different amounts of detail.

Au tomatically Segmen ting Movies into Lo gical Story Units

LSU 1

LSU 2

231

LSU 3 t

Characteristic video shots of an LSU on different granularity levels (e.g. content-based clustering is used)

Key frames of characteristic shots (one key-frame set for each granularity level)

Fig. 2. Possible scheme for movie representation based on LSUs

Few methods dealing with high-level movie segments can be found in literature. In°[2] characteristic temporal events like dialogs, high-motion and high-contrast segments are extracted for the purpose of making a movie trailer, but no attempt is made to capture the entire movie material. In [5] an approach is presented based on time-constrained clustering and label assignments to all shots within a sequence. Predefined models are used to analyze the resulting label sequence and recognize patterns corresponding to dialogs, action segments and arbitrary story units. The effectiveness of this method, especially for segmenting movies into story units, depends however on the applicability of the model used for a story unit. We foresee here several practical problems such as the choice of the interval for time-constrained clustering, which puts an artificial limit on the duration of an episode. Another problem is that characterizing shots by distinct labels simplifies the real interrelation among neighboring shots far too much.

3 Concept of Logical Story Units The concept of an LSU is based on the global temporal consistency of its visual content. Such a consistency is highly probable in view of the realistic assumption that an event is related to a specific location (scenery) and certain characters. It can be expected that within an event every now and then similar visual content elements (scenery, background, people, faces, dresses, specific patterns, etc.) appear and some of them even repeat. Such content matches clearly may not happen immediately in successive video shots, however, most probably within a certain time interval. We first assume that visual content elements from the current shot k1 reappear (approximately) in shot k1 + p1 . Then, shots k1 and k1 + p1 form a linked pair. Since shots k1 and k1 + p1 belong to the same LSU(m), consequently all intermediate shots also belong to LSU(m):

232

Alan Hanjalic et al.

[k1 , k1 + p1 ] ∈ LSU ( m) if p1 ⇐ min A(k1 , k1 + l ) < M ( k1 ). l =1,K,c

(1)

Here, A(k,k+l) is the dissimilarity measure between the shots k and k+l, while c is the number of subsequent shots the current shot is compared with to check the visual dissimilarity. The threshold function M(k) specifies the maximum dissimilarity allowed within a single LSU. Since the visual content is usually time-variant, the function M(k) also varies with the shot under consideration. If there are no subsequent shots with sufficient similarity to the current shot k2 , i.e. the inequality in equation (1) is not satisfied, there is the possibility that one or more shots preceding shot k2 link with shot(s) following shot k2 . Then, the current shot is enclosed by a shot pair that belongs to LSU(m), i.e.

[ k2 − t, k 2 + p2 ] ∈ LSU ( m ) min A( k 2 − i, k 2 + l ) < M ( k 2 ). if (t, p2 > 0 ) ⇐ min i =1,K,r l =− i +1,K,c

(2)

Here r is the number of video shots to be considered preceding the current shot k2 . If for the current shot k3 neither (1) nor (2) is fulfilled, but if shot k3 links with one of the previous shots, then shot k3 is the last shot of LSU(m). The objective is now to detect the boundaries between LSUs, given the described procedure for linking shots. In principle one can check equations (1) and (2) for all shots in the video sequence. This, however, is rather computationally intensive and also unnecessary. According to (1), if the current shot k is linked to shot k+p, all intermediate shot automatically belong to the same LSU and do not have to be checked anymore. Only if no link can be found for shot k, it is necessary to check whether at least one of r shots preceding the current shot k can be linked with shot k+p (for p>0, as stated in (2)). If such a link is found, the procedure can continue at shot k+p, otherwise shot k is at the boundary of LSU(m). The procedure then continues with shot k+1 for LSU(m+1). The LSU boundary detection procedure is illustrated in Fig. 3. (e) (a)

(b)

(d)

LSU(m)

t

(c)

LSU(m+1)

Fig. 3. Illustration of the LSU boundary detection procedure. The shots indicated by (a) and (b) can be linked and are by definition part of LSU(m). Shot (c) is implicitly declared part of LSU(m) since the shot (d) preceding (c) is linked to a future shot (e). Shot (e) is at the boundary of LSU(m) since it cannot be linked to future shots, nor can any of its r predecessors.

Au tomatically Segmen ting Movies into Lo gical Story Units

233

To determine if a link can be established between two shots, we need the threshold function M(k). We compute this threshold recursively from already detected shots that belong to the current LSU. If the minimum of A(k,n) found in equation (1) (or equation (2) if (1) does not hold) denotes the content inconsistency value C(k), then the threshold function M(k) that we propose is:

M (k ) = αC (k , N k )

(3)

Here α is a fixed parameter whose value is not critical between 1.3 and 2.0, and C (k , N k ) is computed as C (k, N k ) =

 1  Nk  ∑ C(k − i) + C0  Nk + 1  i =1 

(4)

The parameter Nk denotes the number of links in the current LSU that have lead to the current shot k, while the summation in (4) comprises the shots defining these links. Essentially the threshold M(k) adapts itself to the content inconsistencies found so far in the LSU. It also uses as a bias the last content inconsistency value C0 of the previous LSU for which (1) or (2) is valid. We now proceed to define the content-based dissimilarity function A(k,n), and assume that the video sequence is segmented into shots, using any of the methods found in literature (e.g. [1]). Each detected shot is represented by one or multiple key frames so that its visual information is captured in the best possible way (e.g. by using°[1]). All key frames belonging to a shot are merged together in one large variable-size image, called the shot image, which is then divided into blocks of HxW pixels. Each block is now a simple representation of one visual-content element of the shot. Since we cannot expect an exact shot-to-shot match in most cases, and because the influence of those shot-content details which are not interesting for an LSU as a whole should be as small as possible, we choose to use only those features that describe the HxW elements globally. In this paper we use only the average color in the L*u*v* uniform color space as a block feature. For each pair of shots (k,n), with k
min

∑ d (bk , bn )

all possible block combinations all blocks b

(5)

234

Alan Hanjalic et al.

where

d (bk , bn ) =

( L (b ) − L (b )) + (u (b ) − u (b )) + (v (b ) − v (b )) *

2

*

k

n

*

2

*

k

n

*

*

k

n

2

(6)

and where all possible block combinations are given by the first item. Unfortunately this is a problem of high combinatorial complexity. We therefore use a suboptimal approach to optimize (5). The blocks bk from a key frame of shot k are matched in the unconstrained way in shot image n starting with the top-left block in that key frame, and subsequently scanning in the line-fashioned way to its bottomright block. If a block bn has been assigned to a block bk , it is no longer available for assignment until the end of the scanning path. For each block bk the obtained match yields a minimal distance value d1(bk). This procedure is repeated for the same key frame in the opposite scanning fashion, i.e. from bottom-right to top-left, yielding a difference mapping for the blocks bk and a new minimal distance value for each block, denoted by d2(bk). On the basis of these two different mappings for a key frame of shot k and corresponding minimal distance values d1(bk) and d2(bk) per block, the final correspondence and actual minimal distance dm(bk) per block is constructed as follows: • dm(bk) = d1(bk) , if d1(bk)= d2(bk) (7a) • dm(bk) = d1(bk) , if d1(bk)< d2(bk) and d1(bk) is the lowest distance value measured on the assigned block in the shot image n (one block in shot image n can be assigned to two different blocks in a key frame from k: one time in each scanning direction) (7b) dm(bk) = ∞ , otherwise. (7c) • dm(bk) = d2(bk) , if d2(bk)< d1(bk) and d2(bk) is the lowest distance value measured on the assigned block in the shot image n (7d) dm(bk) = ∞ , otherwise. (7e) where ∞ stands for a fairly large value, indicating that no objective best match for a block bk could be found. The entire described procedure is repeated for all key frames of shot k, leading to one value dm(bk) for each block of shot image k. Finally the average of the distances dm(bk) of the B best-matching blocks in the shot image k is computed as the final inter-shot dissimilarity value: A(k , n) =

1 ∑ d m (bk ) B B best matching

(8)

blocks

The reason for taking only the B best-matching blocks is that two shots should be compared only on a global level. In this way, we allow for inevitable changes within the LSU, which, however, do not degrade the global continuity of its visual content. After the LSU boundary detection procedure has been explained, the characteristics of the obtained LSUs in view of the actual episodes will now be discussed. For this purpose, we investigate a series of shots a to j, as illustrated in Fig.°4. According to the movie script, the boundary between episodes m and m+1 lies

Au tomatically Segmen ting Movies into Lo gical Story Units

235

between shots e and f. We now assume that the shot e, although belonging to the episode m, has a different visual content than the rest of the shots in that episode. This can be the case if, e.g., e is a descriptive shot, which generally differs from event shots. Consequently, the content consistency could be followed by overlapping links in the LSU(m) up to shot d, so that the LSU boundary is found between shots d and e. If the shot e contains enough visual elements also appearing in the episode m+1, so that a link can be established, e is assumed to be the first shot of the LSU(m+1) instead of shot f. This results in a displaced episode boundary, as shown in Fig. 4. However, if no content-consistency link can be established between shot e and any of the shots from the episode m+1, another LSU boundary is found between shots e and f. Suppose that f is a descriptive shot of the episode m+1, containing a different visual content than the rest of the shots in that episode, so again no content-consistency link can be established. Another LSU boundary is found between shots f and g. If the linking procedure can now be started from shot g, it is considered to be the first shot of the new LSU(m+1). In this case, not a precise LSU boundary is found but one that is spread around the actual episode boundary, taking into consideration all places where the actual episode boundary can be defined. Consequently, the shots e and f are not included into LSUs, as shown in Fig. 4. Such scenarios occur quite often and show that by investigating the temporal consistency of the visual content, only an approximation for the actual episode should be expected. Episode boundary

a b c d e f g h i j LSU(m)

LSU(m+1)

Spread LSU boundary

Fig. 4. LSU versus episode boundary. Note that LSUs are, in general, different than the corresponding episodes since not all episode shots are included into the LSUs.

4 Experimental Validation To test the proposed LSU boundary detection approach, we used two full-length movies. Both of them were available as DC sequences [4], obtained from MPEGstreams with (slightly modified) frame sizes 88x72 and 80x64 respectively. We detected the shots using the method from [1] and represented each shot by two key frames taken from the beginning and end of a shot, in order to capture most of its important visual content elements. To get an idea about the positions of the actual episode boundaries we asked unbiased test subjects to manually segment both movies and took into account only those boundaries registered by all test subjects. These boundaries we call probable. Then, we had our algorithm perform the automatic segmentation of the movies for different values of parameters B and α and compared the automatically obtained boundaries with the probable ones. Thereby, an automatically detected boundary, not registered by any of the test users, was considered as false.

236

Alan Hanjalic et al.

Best performance for both movie sequences was obtained if 50% of blocks were considered in (8) and for the threshold multiplication factor α of 1.4. Thereby, both block dimensions H and W used to segment the shot images were chosen as 8. For these parameter values, 69% of probable boundaries were detected in average for both movie sequences with only 6% of false detections. Low number of false detections is fully compliant with our requirements for conciseness and comprehensiveness of the movie retrieval interface at its top level. These characteristics are not guaranteed if a high percentage of “false episodes” are present, making the first interaction level overloaded and the interaction inefficient. On the other hand, after investigating the missed 31% of probable boundaries, we found out that most of episodes, which could not be distinguished from each other, belong to the same global context (e.g. a series of episodes including a wedding ceremony, a reception and a wedding party). Therefore, the comprehensiveness of the LSU set obtained for B=50% and α =1.4 was not strongly degraded by missed boundaries.

5 Conclusions The high-level movie segmentation approach presented in this paper is based on investigating the visual information of a video sequence and their temporal variations, as well as on the assumption about the global temporal consistency of the visual content within a movie episode. Our work shows that using only visual features of a movie sequence can provide satisfactory segmentation results, although the LSU boundaries only approximate the actual episode boundaries in some cases. The obtained LSUs can be used for developing an efficient event-oriented movie retrieval scheme. As the proposed technique computes the detection threshold recursively, and looks only a limited number of shots ahead during the LSU boundary detection, the entire process including shot-change detection, key-frame extraction, and LSU boundary detection, can be carried out in a single pass through a sequence.

References 1. 2. 3. 4. 5.

Hanjalic A., Ceccarelli M., Lagendijk R.L., Biemond J.: Automation of systems enabling search on stored video data, Proceedings of IS&T/SPIE Storage and Retrieval for Image and Video Databases V, Vol. 3022 (1997) Pfeiffer S., Lienhart R., Fischer S., Effelsberg W.: Abstracting Digital Movies Automatically, Journal of Visual Communication and Image Representation, Vol. 7, No. 4 (1996), pp. 345-353 The SMASH project home page: http://www-it.et.tudelft.nl/pda/smash. Yeo B.-L., Liu B.: On the Extraction of DC Sequence from MPEG Compressed Video, Proceedings of IEEE ICIP (1996) Yeung M., Yeo B.-L.: Video Content Characterization and Compaction for Digital Library Applications, Proceedings of IS&T/SPIE Storage and Retrieval for Image and Video Databases V, Vol. 3022 (1997)

Local Color Analysis for Scene Break Detection Applied to TV Commercials Recognition
Juan Mar´ıa S´ anchez, Xavier Binefa, Jordi Vitri`a, and Petia Radeva Computer Vision Center, Departament d’Inform` atica Universitat Aut` onoma de Barcelona, 08193 Bellaterra, Spain {juanma,xavierb,jordi,petia}@cvc.uab.es

Abstract. TV commercials recognition is a need for advertisers in order to check the fulfillment of their contracts with TV stations. In this paper we present an approach to this problem based on compacting a representative frame of each shot by a PCA of its color histogram. We also present a new algorithm for scene break detection based on the analysis of local color variations in consecutive frames of some specific regions of the image.

1

Introduction

The recognition of known patterns in a given digital video is a task with lots of applications. Among these we have developed a system that recognizes the broadcast of TV commercials on-line and stores their broadcast time, as well as their length. This data can be used by advertisers to check the fulfillment of their contracts with TV stations, as well as to know the impact of their publicity on the viewers, together with audience statistics. Some previous work has been done about the analysis of TV commercials. The aim of Lienhart et al. in [3] is the isolation and extraction of commercial blocks from TV broadcasts by detecting a set of features common to every commercial block. In the same paper, a recognition-based approach is also implemented to detect single commercials out of commercial blocks, using an approximate substring matching algorithm. Colombo et al. [1] obtain some semantic measures of video sequences from a set of perceptual features in order to characterize every commercial and perform a content-based retrieval on a video database. We have not found any previous work about auditing commercials broadcast. The problem we are dealing with consists on creating a suitable and compact representation to store a digital video sequence pattern, so it can be recognized in a given video. We assume that every video sequence is constituted by a set of shots, which can be represented by a single frame. This assumption forces us to implement an algorithm to correctly detect every transition between shots.


This work was supported by the projects TAP97-0463 and TAP-0631 of Spanish Industry Ministry.

Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL’99, LNCS 1614, pp. 237–244, 1999. c Springer-Verlag Berlin Heidelberg 1999


238

Juan Mar´ıa S´ anchez et al.

Different techniques have been used for this purpose, most of them based on color. The simplest one is the color histogram difference between consecutive frames [9]. Some improvements have been proposed by considering the spatial distribution of pixels in the image like the color coherence vectors (CCV) [6] that divide the pixels in each bin into coherent and non coherent, or the joint histograms [7] that build a multidimensional histogram considering a set of local features for each pixel. These algorithms detect almost every sharp transition, but gradual transitions, i.e. dissolves and fades, are not correctly detected due to the low global variation of color from one frame to the next. A completely different alternative is proposed by Zabih et al. in [12], where an algorithm is presented which is based on computing the number of edge pixels that appear and disappear in a given frame as a feature called edge change ratio (ECR). The main problem of this algorithm is dealing with affine transformations of the contents of the image at a low computational cost. The algorithm presented in this paper treats this problem by locally considering some specific regions of the image. This scene break detection algorithm is introduced in section 2 and our TV commercial recognition system is presented in section 3, showing the way we use principal component analysis (PCA) to reduce the size of the representation of every shot and obtain a real time recognition system.

2

Local Color Analysis for Scene Break Detection

We introduce a sharp and gradual transition detection algorithm based on the analysis of color contents of specific regions in consecutive frames in order to get a measure of the global variation between them as a combination of local differences. Let Rji be region j of frame i of a video sequence. A measure of the variation of each region colors from frame i to frame i + 1 can be computed as a difference ∆hij of their associated color histograms hij and hi+1 j . A global difference measure between frames i and i + 1 can then be obtained as a combination of the local differences. The possible transformations of the contents of each region from one frame to the next must be considered in the way we define the pairs of regions (Rji , Rji+1 ) and how their color histograms are built and compared. 2.1

Regions Around Color Transitions

Following our approach, regions Rji are defined from the most significative color transitions of the image, detected as high values of its multispectral gradient. The main interest of these regions is that they can be divided into two subregions by the gradient creases. We consider that one of them (Oji ) belongs to an object in the scene and the other one (Bji ) belongs to a different object or to the background (or both). Each region Rji is defined as follows: 1. The multispectral gradient of the image is obtained using Sobel’s approximation.

Local Color Analysis for Scene Break Detection

239

2. Only significant color transitions are considered applying a threshold, obtained by Otsu’s algorithm [5], to the gradient image and removing its smaller connected regions using a queue based algorithm [11]. 3. A region Rji is defined from each connected region of the gradient image by an adaptative window that contains all of its pixels. If the content of at least one of these sub-regions does not significantly change from one frame to the next, we can say that the main content of the Rji is the are built same in Rji+1 . This situation can be easily detected if hij and hi+1 j in such a way that they contain the same number of pixels from Oji and Bji . Using a queue based algorithm we know which pixels are at a distance less or equal to d from the border between both sub-regions. With this algorithm the color histogram is exactly the same in every orientation of the contents of the region. Suppose that we have a region in frames i and i + 1 which at least one of its sub-regions does not change. Therefore, the values of its corresponding are very close. If we sort their bins to get two vectors colors in hij and hi+1 j i+1 i+1 i i hj (n) and hj (n), n = 0..N − 1, such that |hij (r) − hi+1 j (r)| < |hj (s) − hj (s)|, ∀r < s, we will find the corresponding
K−1to the unchanged region in their K
bins K−1 first positions, where k=0 hij (k) ≈ k=0 hi+1 j (k) ≈ 0.5, and the sum of the differences between these K elements of the vectors will be close to zero. Thus, we will know if a region has not continuity in any of its sub-regions when this sum of differences is high. These changed regions will be named Qij . 2.2

Tolerance to Transformations of Regions

Before applying this difference measure between regions, we must find the pairs of regions (Rji , Rji+1 ) that will be compared, considering the possible transformations of their contents, mainly translations, scalings due to camera zooms and rotations. The window that defines Rji on the gradient image is used to find the displacement of this region correlating with its neighbourhood in the gradient image of frame i + 1. Changes in the scale of Rji contents can mislead the correlation, as shown in figure 1 where it is mistaken as a displacement and the content of the window that defines Rji+1 differs from Rji ’s. To solve this situation, each connected region of the gradient image of frame i + 1 is divided into smaller pieces with a maximum of p pixels and a region Rji is defined for each piece, so all of them can be correctly located in frame i + 1 as shown in Fig. 1. Although rotations are mainly considered in the way we compute the color histograms of the regions as we have previously said, when the orientation of the contents of Rji changes, the shape of the window that defines it may change as well. Figure 2 shows this situation. We assume that between two consecutive frames, the rotation angle of an element of the scene is small, and so it is the change in the shape of the window that defines Rji . This change can be overcome by increasing the window size. If an Rji+1 can not be found for a given Rji , i.e. the maximum of the correlation with its surroundings is close to zero, we know that the contents of Rji have

240

Juan Mar´ıa S´ anchez et al.

Fig. 1. Tolerance to zooms dividing the connected regions of the gradient image into smaller pieces.

Fig. 2. Change in shape of the window that defines an Rji due to a rotation. completely changed from one frame to the next, with no need of computing the i difference between hij and hi+1 j . These regions will be named Pj . 2.3

Detecting and Classifying Scene Breaks

Once we know which regions Rji do not have continuity in frame i+1, we compute a measure of the global variation of the scene between two consecutive frames. The variation of a large region is more significant than a smaller one, so we weigh up each region contribution to the global measure by the number of pixels it contains. Then, this measure is computed as: L−1


V (i, i + 1) =

j=0

|Pji | + N
−1 j=0

M−1
j=0

|Qij | (1)

|Rji |

where |Rji | is the number of pixels in Rji . This measure lets us detect and classify different kinds of scene breaks. Cuts are characterized by high values of V (i, i+1) caused by the sudden change of the scene content. Dissolves are mainly due to the Pji ’s because new edges gradually appear in frame i + 1 far from the location of the ones in the previous frames. In a fade-out, every Rji turns out to be a Pji because all the gradient creases disappear, but a fade-in can not be detected using V (i, i + 1). As gradient creases gradually appear, V (i + 1, i) must be used instead. As far as we are forced to compute V (i, i + 1) and V (i + 1, i) to detect both kinds of fades, we can also improve the detection of cuts and dissolves, because both of them are high in these transitions, so their sum is a more robust measure than only one of them. Figure 3 shows their values corresponding to each different kind of scene break.

Local Color Analysis for Scene Break Detection

241

Fig. 3. Values of V (i, i+ 1) () and V (i + 1, i) () during a cut (left), a dissolve (center) and both kinds of fades (right).

2.4

Experimental Results

We have tested our algorithm on a video sequence of Spanish TV commercials acquired at 25 frames per second with a 160 × 120 pixels resolution. This sequence contained several scene breaks including 65 cuts, 13 dissolves, 3 fades-out and 2 fades-in. All of them were correctly detected by our algorithm. 15 false positives were reported during this sequence mainly caused by frame sequences where there are no significant regions in the gradient image. In this situation, regions Rji can not be defined and the algorithm behaviour is unexpected. Only 4 of them were due to fast motion of large objects in the scene. For our application to commercial recognition, false positives turn out to be redundancy in their representation, which may be good for the recognition process, while the detection of every scene break is a must.

3

Recognition of TV Commercials

In this section, we introduce our approach to the recognition of previously stored patterns of commercials in a given TV broadcast. The main drawback of commercial recognition is the change of their length in different broadcasts in order to reduce their cost for advertisers. This reduction is achieved removing shots and/or shortening their length. Our representation of a commercial assumes that every video sequence can be divided into shots. The information contained in each shot can be represented by a single image. For efficiency, we have chosen the first frame of a shot as its representative. Then, a commercial is stored as the sequence of representative frames of its shots. The recognition process is achieved detecting the transitions between shots, getting their first frames and searching the stored database for the matching ones. The database is looked over in such a way that the elements with the highest matching probability are first compared. Let’s suppose that we have previously detected and identified a shot of the input video sequence, so we know which commercial is being broadcasted. When we detect the next shot, it is compared with every shot representative of the current commercial to know if it is still being broadcasted. If it finished, the new shot can very probably be the first one of another commercial, so its representative frame is compared with

242

Juan Mar´ıa S´ anchez et al.

every first shot representative of the commercials in the database. Even if none of them matched it, it still must be compared with the rest of shot representatives of every stored commercial because the first shot of a commercial can be removed. The comparison of shot representatives is based on their color histograms in order to achieve some invariance to small changes in the representative frame of the shot, as can be caused by removing its first few frames. Color histograms are affected by color intensity variations in different TV stations broadcasts, which can be modeled as a shift in each color channel histogram. Every representative frame color channel is normalized depending on its average intensity value before searching the database. 3.1

Dimensional Reduction Using PCA

If we want to work with a large commercial database in real time, the size of the elements to compare should be reduced to very few dimensions. This is achieved applying PCA to the color histograms of shot representatives. This statistical technique has been used extensively for object recognition tasks (Turk and Pentland [10], Nayar et al. [4], Huttenlocher et al. [2], Seales et al. [8]). Let h be the color histogram of a frame represented as a column vector. Given a set of learning histograms hm , m = 1..M , we define the matrix X = [h1 − c, . . . , hM − c], where c is the average of the hm ’s. The average histogram is subtracted from each hm so that the predominant eigenvectors of XX T will capture the maximal variation of the original set of histograms. The eigenvectors of XX T are an orthogonal basis in terms of which the hm ’s can be rewritten. Let ei , i = 1..N , denote each of its eigenvectors and let E be the matrix [e1 , . . . , eN ]. Each hm is rewritten in terms of the orthogonal basis defined by the eigenvectors of XX T as gm = E T (hm − c). As distances are preserved under an orthonormal change of basis, it can be shown that ||hm − hn ||2 = ||gm − gn ||2 [8]. The most important fact is that hm can be approximated using just those eigenvectors corresponding to the k largest eigenvalues, rather than all N eigenvectors, where k << N . This low-dimensional representation is intended to capture the important characteristics of the set of learning images. Let fm be the vector of coefficients of gm corresponding to the k largest eigenvalues. Then, the sum of squared differences, ||hm − hn ||2 , is approximated as ||fm − fn ||2 . This size reduction of the elements to compare allows us to speed up the recognition process by computing difference measures of the elements in the kdimensional subspace instead of in the original N -dimensional space, while the maximal variation of the original set is kept. 3.2

Real Application Results

We have tested our system on real prime time Spanish TV broadcasts, using a PC and acquiring the video sequence directly with a Matrox Rainbow Runner video capture board. Although our scene break detection algorithm could not be fully used in real time because it has not been optimized yet, we have implemented it as a helper to a simple color histogram difference algorithm. When this measure

Local Color Analysis for Scene Break Detection

243

(a) Representative frames of the 10 first shots of a “Little Maidness” commercial.

(b) Representative frames of the 5 shots of a different “Little Maidness” commercial.

Fig. 4. Two different commercials of the same product.

gets over a low threshold (which reports many false positives), our new algorithm is asked to confirm the detection of the transition. We have learned 23 commercials in our system, with a total of 399 shot representatives. Some of these commercials were broadcasted more than once during the test sequence. The learned commercials appeared a total of 81 times during the 9 hours the system was running. All of them were recognized and their lengths were exactly reported, including those broadcasted by a different TV station of the learning one. The test sequence also contained different commercials of the same product with some similar shots like the ones shown in Fig. 4. Some cases were reported correctly, but some were not because the learned key frames did not correspond to the ones found during the broadcast. Although some of them are similar, the projections of their color histograms do not match. A better key frame selection would let us detect these commercials. However, as far as they are actually different commercials, both of them should be learned if they must be recognized by the system. Only 4 shots were reported as wrong commercials during this test. They can easily be detected by the user if one frame is stored for every detected commercial.

4

Conclusions

This paper has introduced two contributions to digital video analysis. First of all, we have developed an application of pattern recognition in digital video

244

Juan Mar´ıa S´ anchez et al.

sequences based on their segmentation into shots, and on the dimensional reduction of their representative frames, using PCA of their color histograms, to speed up the recognition process and achieve the goal of commercial recognition during TV broadcasts in real time. Our system has shown to be very accurate detecting the broadcast of commercials as well as their length. On the other hand, this approach forces us to correctly detect transitions between the shots of the sequence, including cuts, dissolves and the two kinds of fades. We have introduced a new scene break detection algorithm based on the local analysis of color variation between consecutive frames in some specific regions of the image. We have chosen the regions around color transitions due to its high perceptual content that lets us interpret what is happening in each one of them. The experimental results have been very satisfacting. Future work has to be done optimizing the algorithm in order to fully use it in real time applications.

References 1. C. Colombo, A. Del Bimbo, and P. Pala. Retrieval of commercials by video semantics. In Proc. Computer Vision and Pattern Recognition, pages 572–577, 1998. 237 2. D. P. Huttenlocher, R. H. Lilien, and C. F. Olson. Object recognition using subspace methods. In Proc. of the 4th European Conference on Computer Vision, volume I, pages 536–545, April 1996. 242 3. R. Lienhart, C. Kuhm¨ unch, and W. Effelsberg. On the detection and recognition of television commercials. In Proc. IEEE Conf. on Multimedia Computing and Systems, pages 509–516, Ottawa, Canada, June 1997. 237 4. S. K. Nayar, H. Murase, and S. A. Nene. Parametric appearance representation. In Early Visual Learning, pages 131–160. Oxford University Press, 1996. 242 5. N. Otsu. A threshold selection method from gray-level histograms. IEEE Transactions on Systems, Man and Cybernetics, SMC, 9(1):62–66, January 1979. 239 6. G. Pass and R. Zabih. Histogram refinement for content-based image retrieval. In Proc. of the 3rd Workshop on Applications of Computer Vision, Sarasota, Florida, December 1996. 238 7. G. Pass and R. Zabih. Comparing images using joint histograms. ACM Journal of Multimedia Systems, 1998. (to appear). 238 8. W. B. Seales, C. J. Yuan, W. Hu, and M. D. Cuts. Object recognition in compressed imagery. Image and Vision Computing, 16:337–352, 1998. 242 9. M. J. Swain and D. H. Ballard. Color indexing. International Journal of Computer Vision, 7(1):11–32, 1991. 238 10. M. Turk and A. Pentland. Eigenfaces for recognition. Journal of Cognitive Neuroscience, 3(1):71–86, 1991. 242 11. L. Vincent. Algorithmes morphologiques ` a base de files d’attente et de lacets. Extension aux graphes, Ph. D. dissertation, Ecole Nationale Sup´erieure des Mines de Paris, France, 1990. 239 12. R. Zabih, J. Miller, and K. Mai. A feature-based algorithm for detecting and classifying scene breaks. In ACM Conference on Multimedia, San Francisco, California, November 1995. 238

Scene Segmentation and Image Feature Extraction for Video Indexing and Retrieval Patrick Bouthemy1 , Christophe Garcia2 , R´emi Ronfard3 , George Tziritas2 , Emmanuel Veneau3 , and Didier Zugaj1 1

INRIA–IRISA, Campus Universitaire de Beaulieu, 35042 Rennes Cedex, France {bouthemy,dzugaj}@irisa.fr 2 ICS–FORTH, P.O.Box 1385, GR 711 10 Heraklion, Crete, Greece {cgarcia,tziritas}@csi.forth.gr 3 INA, 4 avenue de l’Europe, 94366, Bry-sur-Marne, France {rronfard,eveneau}@ina.fr

Abstract. We present a video analysis and indexing engine, that can perform fully automatic scene segmentation and feature extraction, in the context of a television archive, based on a library of image analysis functions and templates.

1

Introduction

The ESPRIT project DiVAN aims at building a distributed architecture for the management of - and access to - large television archives. An important part of the system is devoted to automatic segmentation of the video content, in order to assist the indexing and retrieval of programs at several levels of details from individual frames to scenes and stories, using a combination of annotated semantic information and automatically extracted content-based signatures. In Section 2, the template-based architecture of the system is described. The tools developed in this architecture are presented in Section 3. In Section 4, we briefly sketch how those tools can be integrated into complete analysis graphs, driven by the internal structure of television programs.

Fig. 1. Architecture of the video indexing system in DiVAN: Analysis is controlled by scripts stored in the template library, and launches tools from the tools library. Both libraries are accessed via the CORBA ORB. Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL’99, LNCS 1614, pp. 245–253, 1999. c Springer-Verlag Berlin Heidelberg 1999


246

2

Patrick Bouthemy et al.

Template-Based Indexing and Retrieval

In DiVAN, a video indexing session is distributed between several machines, using connections to two remote CORBA servers - a video analysis server and a template library server (Figure 1). Each program template contains the taxonomy of event categories (such as shots, scenes and sequences) meaningful to a collection of programs (such as news, drama, etc.), along with their temporal and compositional structure. In addition, the template contains algorithmic rules used to recover the structure of each individual program, based on observable events from the tools server (such as dissolves, wipes, camera motion types, extracted faces and logos). Section 3 reviews the tools available in the DiVAN system, and Section 4 presents examples of template-based indexing using those tools.

3

Shot Segmentation and Feature Extraction

The first step in the analysis of a video is the partitioning into its elementary subparts, called shots, according to significant scene changes (cuts or progressive transitions). Features such as keyframes, dominant colors, faces and logos can also be extracted to represent each shot. 3.1

Cut Detection

Our algorithm involves the frames that are included in the interval between two successive I-frames. It is assumed that this interval is short enough to include only one cut point. Given this assumption, the algorithm proceeds as follows. Step1: Perform cut detection considering the next I-frame pair and using one of the histogram metrics and a predefined cut detection threshold. Step2: Confirm the detected cut in the next subinterval between two consecutive I-, P-frames, included in the interval of I-frames. Step3: Perform cut detection between two successive I-, P-frames using histogram or MPEG macro block motion information. If the cut is not confirmed in this subinterval, go to step 2. Step4: Determine the exact cut point in the subinterval, by applying a histogram or MPEG motion based cut detection method on each frame interval-pair. If the cut is not confirmed, go to step 2, otherwise, go to step 1. Key-frame extraction is performed during shot detection, and is based upon a simple sequential clustering algorithm, using a single cluster C of the candidate key-frames, which is successively updated by including the I-frame that follows the last element in display order. C is a set of K consecutive I-frames and Cm is an ”ideal” frame with the property that its features match the mean vector (”Mean Frame”) of the corresponding characteristics of frames Ck (k = 1..K). Assuming that the only feature used for classification is the colour/intensity histogram Hk of frames Ck , the ”mean frame” histogram Hm is computed. Then, in order to measure the distance δ(Ck , Cm ), an appropriate histogram metric is used. Based on this distance, the key-frame CK (the representative frame of C) is determined, according to a threshold.

Scene Segmentation and Image Feature Extraction for Video Indexing

3.2

247

Progressive Transitions and Wipes

The method we propose for detecting progressive transitions relies on the property of temporal coherence of the global dominant image motion between successive image pairs within a shot. To this end, we exploi a robust estimation criterion and a multiresolution scheme to estimate a 2D parametric motion model accounting for the dominant image motion, in presence of secondary motions [7]. The minimization problem is solved by using an IRLS (Iteratively Re-weighted Least Squares) technique. The set of pixels conforming with the estimated dominant motion forms its estimation support. The analysis of the temporal evolution of the support size supplies the partitioning of the video into shots [2]. A cumulative sum test (Hinkleys test) is used to detect significant jumps of this variable, accounting for the presence of shot changes. It can accurately determine the beginning and the end time instants of the progressive transitions. This technique, due to the use of the robust motion estimation method, is fairly resilient to the presence of mobile objects or to global illumination changes. More details can be found in [3]. It is particularily efficient to detect dissolve transitions. To handle wipe transitions between two static shots, we have developed a specific extension of this method. In the case of wipes, we conversely pay attention to the outliers. Indeed, as illustrated in Figure 2, the geometry of the main outlier region reflects the nature of the wipe with the presence of a rectangular stripe. To exploit in a simple and efficient way this property, we project the outlier binary pixels onto the vertical and horizontal image axes, which supplies characterizable peaks. Then for detecting wipes, we compute the temporal correlation of these successive projections. The location and value of the correlation extremum allow us to characterize wipes in the video. One representative result is reported in Figure 2.

3.3

Dominant Color Extraction

We characterize the color content of keyframes by automatically selecting a small set of colours, called dominant colours. Because, in most images, a small number of colors ranges capture the majority of pixels, these colours can be used efficiently to characterize the color content. These dominant colors can be easily determined from the keyframe color histograms. Getting a smaller set of colours enhances the performance of image matching, because colour variations due to noise are removed. The description of the image, using these dominant colours, is simply the percentages of those colors found in the image. We compute the set of dominant colors using the k-means clustering algorithm. This iterative algorithm determines a set of clusters of nearly homogeneous colours, each cluster being represented by its mean value. At each iteration, two steps are performed: 1) using the nearest-neighbour rule, pixels are grouped to clusters, and 2) each new cluster is represented by its mean color vector.

248

Patrick Bouthemy et al.

Image 358

Image 359

Outlier image 358 weight=0

50

50

50

100

100

100

150

150

150

200

200

200

250

250

a)

50

100

150

200

250

300

350

b)

250

50

Horizontal projection image 358

100

150

200

250

300

350

c)

50

100

Vertical projection image 358

150

200

250

300

350

Projection Correlations

1

1

9

0.9

0.9

8

0.8

0.8

0.7

0.7

0.6

0.6

0.5

0.5

0.4

0.4

0.3

0.3

0.2

0.2

0.1

0.1

7 6 5 4 3

d)

0

50

100

150

200

250

300

350

e)

0

2 1

50

100

150

200

250

f)

0

0

100

200

300

400

500

600

700

Fig. 2. Detection of wipes: (a,b) Images within a wipe effect, (c) Image of outliers in black, (d,e) projections on horizontal and vertical axes, (f ) plot of the correlation criterion. 3.4

Logo Detection

We introduce a simple but efficient method to perform the detection of static logos, which is the most frequent case in TV programs. To speed up the detection, we specify the expected screen location of the logo by defining its associated bounding box B (see Figure 3). The method proposed is based on the two following steps : 1) determination of the logo binary mask, 2) evaluation of a matching criterion combining luminance and chrominance attributes. If the logo to be detected is not rectangular, the first step consists to build the exact logo mask for detection. That is, we have to determine which pixels belong to the logo in the bounding box. From at least two logos from images with different background, a discriminant analysis technique is used to cluster pixels in two classes : those belonging to the logo and the others. The clustering step is performed by considering image difference values for the luminance attribute {a1 }, and, directly values of chrominance images for chrominance attributes {a2 , a3 } in the Y, U, V space. The two classes will be discriminated by considering a threshold T for 2 (T ) for each attribute {a1 , a2 , a3 }. We consider the inter-class variance σInterC ∗ each attribute {ak }. The optimal threshold T then corresponds to an extremum 2 (T ). of the inter-class variance σInterC We display in Figure 3 an example of histogram of the image difference pixels 2 (T ) supplying within the bounding box, a plot of the inter-class variance σInterC the dissimilarity measure between the two classes, and the determined binary mask M. The matching simply consists in comparing each pixel that belongs to the logo mask in the current color image of the sequence with the value in the reference logo image. The criterion is given by the sum of the square differences between these values over the logo mask.

Scene Segmentation and Image Feature Extraction for Video Indexing

image with logo

image with logo

50

50

100

100

150

150

200

200

250

250

Bounding box image 1.L

Bounding box image 2.L

5

5

10

10

15

15

20

20

25

25

30 50

100

150

200

250

300

350

50

100

Image of Chrominance 1.U

150

200

250

Attribute distribution a

2

300

30

350

5

10

15

20

25

30

5

10

15

0.9

2

20

25

30

Binary mask M 2

Inter−class variance

1.U

0.06

249

0.8

2

0.7

4

0.05

4

6

0.6

0.04

6

0.5

8

8

0.03

0.4 10

10

0.3

0.02

12

12

0.2 0.01

14

14

0.1

a)

16 2

4

6

8

10

12

14

16

b)

0 100

120

140

160

180

200

220

c)

0 100

120

140

160

180

200

220

d)

16 2

4

6

8

10

12

14

16

Fig. 3. Top row: Images with the same logo and associated logo bounding boxes. Middle row: Determination of the binary masks (d) from image of chrominance U (a). Plot of the distribution of attribute {a2 } (b), and of the inter-class variance 2 (T ) (c). values σInterC 3.5

Face Detection

We have developed a fast and complete scheme for face detection in color images where the number of faces, their location, their orientation and their size are unknown, under non-constrained conditions such as complex background [5]. Our scheme starts by performing a chrominance-based segmentation using the YCbCr color model which is closely related to the MPEG coding scheme. According to a precise approximation of the skin color sub-spaces in the YCbCr color model, each macro-block average color value is classified. A binary mask is computed in which a ”one” corresponds to a skin color macro-block, and a “zero” corresponds to a non-skin color macro-block. Then, scanning through the binary mask, continuous “one” candidate face regions are extracted. In order to respect the speed requirements, we use a simple method of integral projection like in [9]. Each macro-block mask image is segmented into non-overlapping rectangular regions that contain a connected “one” region. We search for candidate face areas in these resulting areas. Given that we do not know a priori the size of the faces regions contained in the frame, we first look for the largest possible ones. Then, we iteratively reduce their size and run over the possible aspect ratios and the possible positions in each binary mask segment area. Constraints related to size range, shape and homogeneity are then applied to detect these candidate face regions and possible overlaps are resolved. The main purpose of the last stage of our algorithm is to verify the face detection results obtained after candidate face areas segmentation and remove false alarms caused by objects with color similar to skin colors and similar aspects ratios such as exposed parts of the body or parts of the background. For this purpose, a wavelet packet analysis scheme is applied using a similar method to the one we described in [3,4] in the case of face recognition. For a candidate face region, we search for a face in every position inside the region and for a possible number of dimensions of a bounding rectangle. This search is based on a feature vector that is extracted from a set

250

Patrick Bouthemy et al.

of wavelet packet coefficients related to the corresponding region. According to a metric derived from the Bhattacharyya distance, a subregion, defined by its position and size in candidate faces area is classified as a face or non-face area, using the prototype face area vectors acquired in a previous training step.

4

Scene Segmentation

Shot segmentation and feature extraction alone cannot meet the needs of professional television archives, where the meaningful units are higher-level program segments such as stories and scenes. This section focuses on the segmentation of programs into scenes and stories, as described by some generic templates (special-purpose templates can also be created for specific program collections, although they are not described in this paper for lack of space). 4.1

Basic Program Template

Our basic program template implements a scene segmentation algorithm introduced by Yeo et al. [10]. A scene is defined as a grouping of contiguous shots, based on image similarity. This algorithm uses three parameters - an image similarity function, an average scene duration ∆T and a cluster distance threshold. The similarity function uses a combination of extracted features - color histograms, dominant colors, camera motion types, number, size and positions of faces, etc. Figure 4 shows the analysis graph for scene analysis using dominant colors. The result of the indexing session consists of a hierarchically organized scene track. Independently, a soundtrack is also extracted, but it is not taken into account in the building of the scene graph.

Fig. 4. Analysis graph for the basic program template. The details of the scene segmentation algorithm follows the original algorithm by Yeo et al. A binary tree is constructed in order to represent the similarity between keyframes. A temporal constraint is introduced in order to reduce computation time. To calculate the N (N − 1) distances describing the similarity between clusters, we start with the pairwise distances between keyframes separated by less than a temporal window, as done in [2]. Distances are accessed via

Scene Segmentation and Image Feature Extraction for Video Indexing

251

a priority stack for efficiency reasons. At each step, the smaller distance among the N(N-1) distance is selected, the two corresponding clusters are merged, and all distances are updated through the Lance-Williams formula. A binary tree is built by successive merging of the nearest clusters, until the threshold value is reached. The scene segmentation is computed according to methods described in [2]. We build an oriented graph where the nodes are the previously selected clusters and the edges are temporal succession links. The oriented graph is segmented into its strongly connected components, which are assumed to correspond to scenes. Finally, we complete the scene track by adding all progressive transitions at their appropriate level (between shots or scenes). We obtained good scene segmentation results using regional color histograms or dominant colors in programs with long sequences, each corresponding to a different setting (as in drama or fiction). In many other cases, this approach to scene segmentation suffers some limitations, due to its restrictive definition of a scene. For instance, a short story item in a news broadcast will be incorporated into a larger scene, because of the recurrent anchorperson images before and after the story.

4.2

Composite Program Template

The DiVAN composite program template is used to introduce a broader definition for program segments such as stories, using cues such as logo or face detection. Figure 5 shows the complete analysis graph for a composite program. The program is segmented into stories, based on detected logos, wipes, dissolves and soundtrack changes, and stories are segmented into scenes using the previously described algorithm. In [8], we state the story segmentation problem as a CSP (constraint satisfaction problem), in the framework of an extended description logics, whose solutions are all the segmentations of a composite program, compatible with the given template.

Fig. 5. Analysis graph for a composite program template.

252

5

Patrick Bouthemy et al.

Conclusion

This paper describes the first prototype of the DiVAN indexing system, with a focus on the image processing library and the general system architecture. Template-based video analysis systems have been proposed in academic research [6,1], but no such system is available commercially. The Video Analysis Engine by Excalibur introduces a choice of parameters for shot segmentation but the list of choices is closed. The DiVAN prototype gives the users control of many efficient video analysis tools, though the edition of their own templates, and will be thoroughly tested by documentalists at INA, RAI and ERT in Spring of 1999. The second DIVAN prototype will provide many more functions, such as audio analysis, face recognition, query-by-example, moving objects analysis and camera motion analysis.

Acknowledgments This work was funded in part under the ESPRIT EP24956 project DiVAN. We would like to thank the Innovation Department at Institut National de l’Audiovisuel (INA) for providing the video test material reproduced in this communication.

References 1. P. Aigrain, P. Joly, and V. Longueville. Medium knowledge-based macrosegmentation of video into sequences, in Intelligent Multimedia Information Retrieval, M.T. Maybury, Editor. 1997, MIT Press. 252 2. P. Bouthemy and F. Ganansia. Video partioning and camera motion characterization for content-based video indexing. In Proc. 3rd IEEE Int. Conf. on Image Processing, ICIP’96, Lausane, September 1996. 247 3. C. Garcia, G. Zikos, G. Tziritas. A Wavelet-based framework for face recognition. In Int. Workshop on Advances in Facial Image Analysis and Recognition Technology, 5th European Conference on Computer Vision, June 2 – 6, 1998, Freiburg, Germany. 249 4. C. Garcia, G. Zikos, G. Tziritas. Wavelet Packet analysis for face recognition. To appear in Image and Vision Computing (1999). 249 5. C. Garcia, G. Zikos, G. Tziritas. Face detection in color images using wavelet packet analysis IEEE Multimedia Systems’99 (1999) 249 6. B. Merialdo and F. Dubois, An agent-based architecture for content-based multimedia browsing, in Intelligent Multimedia Information Retrieval, M.T. Maybury, Editor. 1997, MIT Press. 252 7. J.M. Odobez and P. Bouthemy. Robust multiresolution estimation and objectoriented segmentation of parametric motion models. Jal of Visual Communication and Image Representation, 6(4), 1995. 247 8. J. Carrive, F. Pachet., R. Ronfard, Using description logics for indexing audiovisual documents. International Workshop on Description Logics, Trente, Italy, 1998. 251

Scene Segmentation and Image Feature Extraction for Video Indexing

253

9. H. Wang and S.-F. Chang. A highly efficient system for automatic face region detection in MPEG video. IEEE Transactions on Circuits and Systems For Video Technology, 7(4) (1997) 615-628. 249 10. M. Yeung, B.-L. Yeo & B. Liu. Extracting story units from long programs for video browsing and navigation. International Conference on Multimedia Computing and Systems, June 1996. 250

Automatic Recognition of Camera Zooms* Stephan Fischer, Ivica Rimac, and Ralf Steinmetz Industrial Process and System Communications Department of Electrical Engineering and Information Technology Technical University of Darmstadt Merckstr. 25 • D-64283 Darmstadt • Germany

Abstract. In this paper we explain how camera zooms in digital video can be recognized automatically. To achieve this goal we propose a new algorithm to detect zooms automatically and present experimental data on its performance.

1

Introduction

In this paper we describe a new algorithm to recognize zooms automatically. Besides only recognizing that a zoom has taken place it is often very useful to compute an exact scaling factor of a frame. We developed a new algorithm which both recognizes zooms and computes a scaling rate. The algorithm can be used to derive semantic features of digital film, such as automatic scene groupings or trailers [Fis97], [BR96]. The paper is structured as follows: Following a review of related work in section 2. In section 3 we propose a new algorithm to recognize camera zooms. In section 4 we present experimental results we obtained using our algorithm. Section 5 concludes the paper and gives an outlook.

2

Related Work

The recognition of zooms as well as the calculation of zoom factors has been investigated in the context of self-calibration of cameras in the past. [PKG98] describe a versatile approach which can be used to self-calibrate a camera. The authors specialize the general case towards scenarios where the focal length varies. The difference to the work described in this paper is that our algorithm is much simpler and hence faster to compute. The same statement is true for the work described by [HA97] and [FLM92] which. Tse and Bakler describe a global zoom/pan estimation algorithm in [TB91], which requires analysis of a high-density field of motion vectors. Since determining an entire frame of motion vectors with high spatial resolution is a very time consuming process, Zhang et al. [ZGST94] propose the block-matching algorithm for computing the motion vector field. But this method is inherently inaccurate when there is multiple *

This work is supported in part by Volkswagen-Stiftung, Hannover, Gemany.

Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL'99, LNCS 1614, pp. 253-260, 1999.  Springer-Verlag Berlin Heidelberg 1999

254

Stephan Fischer et al.

motion inside a block. Furthermore, block-matching is particularly problematic during a zoom sequence because the area covered by the camera angle expands or contracts during a zoom. Finally, the motion of a large object in a camera shot proved to be a major source of error for camera pan detection.

3

Zoom Detection

A zoom operation is defined by an enlargement of the content of video frames towards the center of the zoom (zoom-in) or by a reduction (zoom-out). The overall size of the frames remains constant during this operation. A zoom-in is shown in Figure 1. The arrows specify that the image is the result of a zoom-in from an earlier image.

Fig. 1. Example of a zoom operation.

• The zoom factor is the factor by which an image has to be scaled to obtain the next image of a video.

3.1

Zoom Recognition Based on Spline Interpolation

The approach we propose uses splines to recognize the fact that a zoom took place as well as to calculate a zoom factor. It is not our goal to calculate the self-calibration of a camera, however, we merely compute the zoom effect. The principal idea is that in any zoom-in or zoom-out sequence either an image A can be found as part of an image B or an image B can be found in an image A, if A is preceding B in time. The task should therefore be to use a mathematical approach to find an image in another. This can be achieved using the spline interpolation of images. Spline interpolation serves to interpolate a set of discrete values by a curve or by a surface in order to model a continuous function. To interpolate images a variety of different approaches to spline interpolation are well known, for example cubic polynomial splines, non rational uniform B-splines (NURBS) or thin-plate splines. In our approach we used natural cubic splines. Two important advantages of the spline approach can be identified: First, no computation of motion vectors is needed. Motion

Automatic Recognition of Camera Zooms

255

vectors themselves already contain a significant amount of errors caused by camera, digitalization and calculation artifacts also influencing the following zoom recognition process. Secondly a precise zoom factor indicating the amount of enlargement or reduction in size of an image can be computed. The approach we propose is to transform two images A and B of a video sequence to a spline representation and then to stretch the spline representing image A in order to locate the spline representing image B and vice versa. Both directions have to be applied as it is unknown in advance if a zoom-in or a zoom-out is to be found and as it is not sure if an image is contained totally in another image. By an adaptation of a variable scaling factor the real zoom factor can be computed. As the exact mathematical formulation of splines is beyond the scope of this paper we start to explain our method being applied for one-dimensional images thus simplifying the understanding of the underlying processes. We then extend the algorithm to twodimensional images. 3.1.1

Zoom Recognition in One-Dimensional Images

In the case of one-dimensional images two arrays A and B can be used containing values representing the brightness of a specific pixel of an image. Let B be an image which is the result of a zoom operation and thus some sort of transformation of image A (see Figure 2).

Brightness

Brightness

Zoom-out operation

Zoom-in operation

Image A

Time

Time

Time

Time

Zoom operations

Image B

Fig.2. Zoom operations. Representing images A and B as splines a spline B can be located in a spline A if image B is a video frame to be presented later in time than frame A (zoom-in operation). In the case of a zoom-out operation a search of spline A in spline B will be successful. A zoom-out can thus be recognized searching for image A in image B. A great advantage of this representation is that the search can be performed very efficiently as only a limited set of spline values have to be compared and as the comparison is not bound to values lying on the pixel grid. The algorithm to compute the zoom factor of a pair of images A and B is shown in Figure 3.

256

Stephan Fischer et al.

Calculate splines representing images A and B. Search pattern of spline B in spline A. If pattern can be found then result := zoom-in. Else search pattern of spline A in spline B. If pattern can be found then result := zoom-out. Else result := no zoom. Fig. 3. Zoom recognition in 1D-images.

In a first step the splines representing images A and B have to be calculated. A very efficient way to compare splines is to use the second derivative of the images which can be computed applying a Laplace-operation. The use of the second derivative guarantees that uniform areas of the images as well as regions of a constant gradient disappear speeding up the calculation of the splines as well as the matching process between the images. As the Laplace-operation calculates the second derivative the zero values correspond to the turning points in the images if and only if the first derivative is nonlinear. Searching for turning points to match A and B the following two conditions are possible: image A is contained in image B entirely in a smaller form, or image A is only partially contained in image B due to fast camera panning. In the first case a potential zoom factor lambda can be estimated using the following formula: TP 1, 2 TP1, 1 λ = ----------------------------TP 2, 2 TP2, 1 where the numerator denominates the turning points (TP) in image A. After estimating lambda the validity has to be checked by picking x-values randomly and by verifying spline A( x ) = d + splineB( λx )

where d is the distance by which spline A has been translated in spline B. If the verification fails the next pair of turning points TP2,2 and TP2,1 is used to compare the images. This step is repeated until A has been located in B. It should be noted that the matching process is very fast as only those values have to be compared where both splines are equal. As a result a zoom factor greater than 1 denotes a zoom-out while a value less than 1 denotes a zoom-out. In the second case a verification can fail if the selected turning point of image A is not contained in image B anymore due to fast camera panning. To solve this problem a new pair of turning points has to be chosen in spline A similar to the choice of new points in spline B. This has to be repeated until the spline has been located. In the case where no zoom has been recognized we assume that no zoom is contained in the video.

Automatic Recognition of Camera Zooms

257

Calculate splines representing images A and B. SplineMatch := FALSE, result := undefined select first turning point (TP) in A while ((SplineMatch == FALSE) AND (turning points in A unvisited)) select second turning point in A while ((SplineMatch == FALSE) AND (turning points in B unvisited)) find turning point (TP) in B using zerocrossings (second derivative) calculate zoom factor using second TP and evaluate result by testing spline equality with x- and y-direction if zoom factor can be calculated SplineMatch := TRUE result := zoom else select new base TP in B select new base TP in A if (result == undefined) result := no zoom. Fig. 4. Zoom recognition in 2D-images.

3.1.2

Zoom Recognition in Two-dimensional Images

The spline comparison is much more complex in the two-dimensional case. The search for a spline equality based on rows or columns is hereby a problem as rows or columns of image A are not only shrunk but also displaced towards the zoom center in image B in case of a zoom-in. Also the target coordinates of the displacement don’t have to be integers. To solve this problem we propose the following algorithm which is only applied to one direction (either x or y) while we use the complementary axis to validate the search results (see Figure 4). The search for turning points by localizing the zeros of the Laplace image cannot be performed by checking the zeros at the whole-numbered values of the spline solely as zeros between those could be missed. To avoid this problem zero crossings have to be examined. In the case of a change of the sign between two values of the second derivative a zero must have occurred between these. Applying this technique also turning points can be localized which are not located on the pixel grid. The algorithm first tries to match the images using turning points on the horizontal axis. In case of a failure the y-direction is examined. To speed up the algorithm we start the localization process in image B with x- or y-values corresponding to those of the turning point in image A and extend the search radius as the algorithm continues its execution.

258

Stephan Fischer et al.

4

Experimental Results

We tested the spline algorithm with synthetical as well as with real image sequences. The performance has been defined as the amount of recognized zooms in relation to the length of the sequence. Our experimental data contained sequences without camera panning and zooms but with object movement (control instance), sequences containing zooms solely, sequences with zoom and camera panning and sequences with zooms, camera panning and object movement. We also tested the robustness of the algorithm by creating an artificial noise of variable strength in the source images. Video images very often include a specific amount of noise caused by camera and compression errors. In a first set of experiments we examined the performance of the algorithm applied to one-dimensional synthetic image sequences. The results are listed in Table 1. Tab. 1. 1D zoom detection

Experiment

Recognition performance

Sequences without camera operations Sequences with zoom only Sequences with zoom and camera panning Sequences with zoom, camera panning and strong object motion 100

Zoom recognized [%] 0 100 100 84.29

False positive [%] 0 0 0 11.93

1D-Rate 2D-Rate

80 60

Scale up by zoom factor

40 Image i

20 00

Image (i+1) Video frames

10

20

30 40 Noise in percent

50

60

Fig. 5. 1D-zoom recognition in the presence Fig. 6. Generation of synthetical images of noise

It is not surprising that a zoom cannot be recognized in the presence of strong object movement anymore. Experiments have shown that moving objects which are larger than 23 percent of the size of the total image cause a distortion which makes a zoom

Automatic Recognition of Camera Zooms

259

recognition impossible. This result was obtained generating synthetical image sequences containing moving objects and increasing their size step by step. Surprisingly, the size of the zoom factor does not influence the recognition rate at all. We identified slow zooms as well as fast zooms with an equal success. Figure 5 shows the impact of noise with respect to the experimental results. These experiments were conducted using sequences containing zooms and camera panning. It is not surprising that a significant amount of noise lowers the performance of the algorithm. A similar behavior can be observed applying the algorithm to two-dimensional images. The results are slightly worse and can be explained with the efficiency of the localization which only approximates the exact location of the spline. The results for the zoom detection using 2D-images are shown in Table 2. Tab. 2. Results of 2D-zoom recognition

Experiment

Zoom recognized False positive [%] [%] Sequences without camera operations with object 2.19 1.1 movement Sequences with zoom only 98.03 0 Sequences with zoom and camera panning 94.77 3.11 Sequences with zoom, camera panning and 82.97 15.33 strong object motion We tested the performance of the spline algorithm with 50 videos consisting of 1000 frames each. Concerning the synthetical videos we adjusted the value of the zoom factor according to the function resulting in zoom factors traversing the interval [0;2a] periodically. Adjusting a and t we were able to create different zoom variations within the synthetical sequence. For a zoom-in we constructed the synthetical images in such a way that we scaled up an image i by the zoom factor and cut out the edge regions in order to obtain the new image (i+1) equal in size to image i (see Figure 6). As all of our experimental sequences started with a zoom-in we buffered the images of the zoom-in and played them in reverse order to obtain the respective zoom-out. This way we were able to produce sequences consisting of periodic zoom-ins and zoomouts. The zoom factor can be adjusted varying the argument of the sine function thus creating slower or faster zooms. The scaling of the synthetical zooms can furthermore be adjusted using the parameter a. The mean experimental values we measured with the synthetical clips and with the real-world clips are shown in Table 2. A result of the experiments is that errors occurring during the zoom recognition process are mainly due to noise included in the images. This can in particular be shown applying some sort of smoothing, for example a Gaussian filter before calculating the splines. The efficiency can then be further improved. It turns out that a zoom recognition using splines outperforms the analysis of motion vector fields (both optical flow and MPEG vectors). Errors caused by noise in the original images and by object motion lower the performance of the algorithm. Considering object motion however it should be possible to find regions in an image where the distortion caused by the moving object is

260

Stephan Fischer et al.

rather small. These areas can then be used to increase the performance of the algorithm.

5

Conclusions and Outlook

In this paper we propose a new method to recognize zooms automatically which also calculates a precise scaling factor between successive frames of a video sequence. In our experimental section we have shown that the algorithm is has a reliability greater than 80 percent. Although the localization algorithm is quite robust we currently think about an integration of our algorithm with algorithms to compute the camera calibration. Another problem any algorithm to recognize zooms faces is object movement. In another set of experiments we currently examine if the spline localization can be performed on selected parts of an image where no object movement takes place.

References [BBBB93] R.H. Bartels, J.C. Beatty, K.S. Booth, E.G. Bosch, and P. Jolicoeur. Experimental comparison of splines using the shape-matching paradigm. ACM Trans. Graph. 12(3), pp. 179-208, 1993. [BR96] J. S. Boreczky and L. A. Rowe. A comparison of video shot boundary detection techniques. Journal of Electronic Imaging, 5(2):pp. 122-128, 1996.

[BX94] C. Bajat and G. Xu. NURBS approximation of surface / surface intersection curves. Adv. Comput. Math.2(1), pp. 1-21, 1994. [BFB84] J.L. Barron, D.J. Fleet, and S.S. Beauchemin. Performance of Optical Flow Techniques. International Journal of Computer Vision, 12(1), 1984. [Fi97] S. Fischer. Feature combination for content-based analysis of digital film. PhD thesis, University of Mannheim, 1997. [FLM92] O. Faugeras, Q.-T. Luong, and S. Maybank: Camera self-calibration: Theory and experiments. Proc. ECCV, 1992. [HA97] A. Heyden, K. Aström: Euclidean Reconstruction from Image Sequences with Varying and Unknown Focal Length and Principal Point, Proc. CVPR, 1997. [Hoe89] M. Hötter. Differential Estimation of the Global Motion Parameters Zoom and Pan. Signal Processing, 16, pp. 249-265, 1989. [ISM93]K. Illgner, C. Stiller, and F. Müller. A Robust Zoom and Pan Estimation Technique. Proceedings of the International Picture Coding Symposium PCS’93, Lausanne, Switzerland, 1993. [LK81] B. Lucas and T. Kanade. An iterative image registration technique with an application to stereo vision. DARPA IU Workshop, pp. 121-130, 1981. [MKG98] M. Pollefeys, R. Koch, and L. Van Gool: Self-Calibration and Metric Reconstruction in spite of Varying and Unknown Internal Camera Parameters, Proc. Of ICCV, 1998. [TB91] Y.T. Tse and R.L. Bakler. Global zoom pan estimation and compensation for video compression. Proceedings ICASSP, pp. 2725-2728, 1991. [ZGST94]H. Zhang, Y. Gong, S.W. Smoliar, and S.Y. Tan. Automatic Parsing of News Video. Proceedings of IEEE Conf. on Multimedia Computing and Systems, 1994.

A Region Tracking Method with Failure Detection for an Interactive Video Indexing Environment Marc Gelgon, Patrick Bouthemy, and Thierry Dubois IRISA/INRIA Campus universitaire de Beaulieu 35042 Rennes cedex, France [email protected]

Abstract. In this paper, we address the problem of structuring video documents into meaningful spatio-temporal entities, involving the tracking of spatio-temporal zones of interest. We present a new technique for tracking over time a region in the image which is initially user-delimited. The tracking method is based on the robust estimation of a 2D affine motion. In order to facilitate the task of the user, the scheme includes the original idea of an automatic tracking failure detection procedure which exploits at low cost information related to the estimated motion model. Experimental results are provided on excerpts of a real movie.

1

Introduction

In this paper, we address the problem of structuring video documents into meaningful spatio-temporal entities, and focus on the issue of tracking given objects of interest within shots. These entities are the shots and spatio-temporal zones of interest within each shot. Such an analysis of a video document is an essential step for accessing (video browsing and indexing) and for manipulating (video editing) the video content. A survey of such applications of video analysis is presented in [5]. The topics revolving around extraction of zones of interest for enhanced browsing and exploitation of video documents, include the generation of video summaries [8,11], and so-called “hypervideo links” that can be established between different instances of the same object in one or several documents [2]. From the spatial and temporal inter-relations between extracted objects, scenario recognition can be achieved [6] with a view to indexing. Partitioning the video into shots is generally the first analysis step. This task has received wide attention. Exploitable results can be obtained in a reasonable amount of time, and sometimes, directly from the MPEG bit stream [18]. However, as reported in a comparative study [7], effective detection of progressive transitions is a stumbling stone of many approaches, and hence remains the subject of current and recent work [4,17]. Structuring a shot into meaningful spatio-temporal zones (also called Video Objects in MPEG-4 terminology) is Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL’99, LNCS 1614, pp. 261–269, 1999. c Springer-Verlag Berlin Heidelberg 1999


262

Marc Gelgon et al.

also a much studied issue, both by the video analysis and video coding communities. The use of motion, possibly with the assistance of intensity or color, as an objective criterion for extracting “zones of interest” in an automatic manner is widespread. It is indeed meaningful in many situations, notably with the “surveillance scene” type involving moving people or vehicles. Recent advances in this field can be for example found in [10,15]. Still, there are limitations to the use of motion as a zone-of-interest delimiter. First, by its principle, motion can naturally not catch all “interesting zones” (e.g. faces), or induces undesirable results (a moving vehicle will be extracted with his moving shadow). Then, in practice, even if motion is a relevant criterion, the task at hand is often complex with general video contents. One has to perform either motion detection or segmentation, to model the apparent motion of these objects that may be articulated or that may change their attitude, one has to cope with the so-called aperture problem and parallax-related issues, and so forth. Automatic motionbased segmentation remains a computationally difficult and expensive process, even if used for some off-line applications related to video indexing as in [8]. Truly, there is also a need, in the context of interactive general-purpose video indexing system, for assistance from the user to achieve an interactive segmentation, and conversely, a real need from video users for assistance from the system in tracking spatio-temporal zones of interest along a shot. Indeed, we focus in this paper on temporal tracking of a given zone. This zone can be delimited by the user in a first frame. It can also be extracted in an automatic way, then manually pointed in the obtained segmentation map. The primitive actually tracked could be an intensity contour delimiting this zone [12], a few points of interest [16] or the whole information contained in the region. Tracking an initially user-defined zone with deformable meshes has been proposed in [9], and through the intensity-homogeneous or color-homogeneous zones which compose it in [13]. Besides, tracking techniques are bound to fail occasionally in difficult situations. Hence, it is crucial and helpful, in interactive applications, that the zone tracker could automatically warn the user of a failure in tracking, in order to give him the hand to re-define the zone in the frame to be tracked. The automatic tracking can then restart from this re-initialization. Such an alarm feature can improve user comfort and work efficiency. In this paper, we propose, after a first step for automatically partitioning a video into shots, a method for tracking in nearly real-time an initially userdefined or user-selected region, including a tracking failure detection technique. These steps exploit information related to the apparent motion measured in the image. More precisely, we make use of a robust multiresolution estimation technique of 2D affine motion model, and of the associated estimation support. The remainder of this paper is organized as follows. Section 2 first overviews the dominant affine motion model estimator technique employed, and how it is applied to automatic partitioning of the video into shots. Section 3 deals with the tracking of zones. We illustrate these techniques through experimental results and provide concluding remarks (Section 4).

A Region Tracking Method with Failure Detection

2

263

Dominant Motion Estimation

The tracking scheme described in this paper exploits motion and motion-related information. For use in the successive video structuring steps, we are interested in measuring apparent (2D) motion between two successive images, either globally in the image, for partitioning the video into shots, or on a region within the image, in order to track this region. In either case, we represent the apparent motion field at hand by a 2D parametric (affine) model. Since several motions may be present in the estimation zone S, we only seek for the estimation of the dominant one. To estimate the dominant motion without prior motion segmentation, we employ a technique based on robust statistics. We outline here the method employed for estimating a 2D parametric motion model, further detailed in [14]. This method takes advantage of a multiresolution framework and an incremental scheme. It minimizes a M-estimator criterion to ensure the goal of robustness to outliers formed by the points corresponding to secondary motions or to areas where the image motion equation is not valid. We have chosen the affine motion model wΘ defined at point p = (x, y), considering a reference point
 (xg , yg ) by: a1 + a2 (x − xg ) + a3 (y − yg ) (1) w Θ (p) = a4 + a5 (x − xg ) + a6 (y − yg ) This motion model is a good trade-off between complexity and representativity. In practice, the image centre is taken as reference point in the case of global motion estimation in the image, or the region centre is selected in the case of region-based motion estimation. The parameter vector Θ = (a1 , a2 , a3 , a4 , a5 , a6 ) is estimated between images I(t) and I(t + 1) in an incremental way. The method for detecting shot changes uses affine motion models accounting for the global dominant image motion between successive images [3]. The evolution of the size of the associated estimation support enables the detection of both cuts and progressive transitions, with the same framework, and with a single parametrization. This technique is also fairly resilient to difficult situations such as the presence of large mobile objects and global illumination changes. We refer the reader to [3,4] for further detail on the technique.

3 3.1

Zone Tracking with Automatic Failure Detection Tracking

Given the video automatically divided into shots, the user can now define objects he wishes to track. Let a user-defined polygon delimitate at time t the zone to be tracked. An affine motion model is estimated over this region using the multiresolution robust estimator outlined in section 2. (Other 2D parametric motion models could also be handled as well.) We then apply to each of the vertices of the polygon the motion vector provided at this vertex by the estimated affine motion model. The set of projected vertices form the estimate for the tracked region at the next time instant. The use of a robust motion estimator enables the proposed technique to cope with a variety of challenging situations.

264

Marc Gelgon et al.

These include possible shadow effects or changes in illumination, partial occlusion and the fact that an affine model may not reflect well enough the real 2D geometric transform. First, the robustness of the estimator makes the tracking resilient to such difficulties. Then, should a slight error however occur, motion model estimation is quite robust to pixels unduly included in the tracked zone. In other words, correct track of the region can be kept despite degration in the accuracy of region boundary. This processing is repeated over successive pairs of images within the shot. Besides, let us point out that this tracking module can be run from any image in the shot either in a forward or in a backward mode (if possible). A classical alternative way of tracking would be to perform intensity correlation. To assess the comparison with our technique, let us point out that our method estimates with a high accuracy a full affine motion model. Such a motion model is quite important to account sufficiently for region boundary deformation in the image due to 3D motion of the object, hence to perform effective tracking. We think this is quite difficult to carry this out with a correlation matching technique as robustly and efficiently as it is done here. Moreover, the robust motion estimation step described above can be performed in nearly real-time. 3.2

Tracking Failure Detection

We describe here how a severe unaccuracy of the region boundary estimate can be detected in most cases. This is particularly helpful to ensure satisfactory extraction of the zone of interest over the whole shot. Motion model  is in fact estimated iteratively as the cumulated value of parameter vector Θ    n = arg min increments ∆Θ n : ∆Θ ρ (ri ) (2) ∆Θn pi ∈I with ri = I(pi + wΘbn (pi ), t + 1) − I(pi , t) + ∇I(pi + wΘbn (pi ), t + 1).w∆Θn (pi ) ρ(x) is a hard-redescending M-estimator. Solving for these increments consists in the following minimization problem  1 : ψ(ri ) n = arg min ∆Θ wi ri2 with wi = . (3) ∆Θn 2 ri p i

where ψ is the derivative of the ρ function, corresponding to the M-estimator. As explained in [14], the minimization problem defined in relation 2 is solved using a irls (Iteratively Re-weighted Least Squares) technique. Once the dominant motion estimation step is completed, the final value of the quantity wi indicates if a point pi is likely or not to belong to the part of the image undergoing this dominant motion. In the former case, wi is close or equal to 1, in the later case, wi is equal or close to 0 (outliers). We define the support of the dominant motion as the set of points pi satisfying wi ≥ ν, where ν is a predefined threshold, (typically 0.2). The pertinent information for the following is in fact the size, denoted ζt , of this support (in fact, we use a normalized version of this size within [0,1]). The principle of the proposed technique is that, as long as the object tracking is correct, most pixels within the estimated region (polygon transformed over time) should be found conform to the estimated dominant motion model,

A Region Tracking Method with Failure Detection

265

and thus the value of ζt should be close to 1. Should the region boundary become suddenly significantly erroneously estimated, it will generally happen that more pixels attached to a motion different to that of the tracked region (e.g. background motion) will be included in the estimation support, and rejected as outliers by the motion estimator. This would induce a strong decrease in ζt . The estimated boundary of the tracked region often drifts only slowly over time. In such a case, a slow decrease of ζt occurs. The use of Hinkley’s test [1] enables the detection of downward jumps of variable ζt . Since this test is cumulative and accounts for all past values of the studied variable, sudden or more progressive decreases can be detected with the same criterion and the same threshold parameter values. Besides, in the case of a progressive jump, the beginning of this jump can be accurately located in time. Implementation of this test is simple and its computational cost is very low. Let us denote m0 the mean of ζt before a jump. This mean value is estimated on-line and re-estimated after each jump detection. δmin denotes the minimal magnitude of jump we wish to detect. The following computation and test are carried out over time :  k
 δmin Tk = ζt − m0 + (k ≥ 0) 2 t=0 Mk = max Ti ; jump detection if Mk − Tk > α 0≤i≤k

α is a user-set threshold. We locate the beginning of the jump at the time following the last instant for which Mk −Tk = 0, i.e. the beginning of a significant downward evolution of the studied variable. When a failure is detected, the beginning of failure is automatically located in time and the user is requested to redefine the region in this frame. Motion-related information is an effective means of coping with the detection of progressive drifts. Indeed, the content of the region in terms of intensity pattern must be allowed to slowly evolve over time, to tolerate changes in illumination and object attitude. Although the tracking technique works for tracking zones that belong to the background scene, tracking failure detection does not work in such case, since the motion within and surrounding the region is the same.

4

Experimental Results and Concluding Remarks

The proposed scheme has been applied to a part of an Avengers TV movie. The following results illustrate region tracking applied to zones of interest over a whole shot. They all come from the Avengers excerpt but the first. In the “Irisa” sequence (Fig. 2a), a zone corresponding to a building is selected. Tracking performs well throughout a zoom effect and despite strong occlusion. In the “Cliff Road” sequence (Fig. 2b), a car is tracked. Camera and car motion are complex, but the region boundary is rather well maintained over this long sequence, in spite of the change in attitude, camera-to-objet distance and strong

266

Marc Gelgon et al.

illumination changes. The head of the person is selected in the “Emma” sequence (Fig. 2c). The 2D dynamic content of this shot is complex and an interactive approach is inescapable to extract and track such a subjective element as the head. Though the inter-frame motion is strong, satisfactory results are obtained. The last shot (Fig. 2d) shows a car entering a pond. The complex forward motion of this car is quite well accounted for, and the water splashes do not severely affect motion measurement. 1

0.9

0.8

0.7

0.6

ζt

0.5

0.4

0.3

0.2

0.1

0

0

10

20

30

40

50

60

frame number

a

b

c

Fig. 1. Temporal evolution of ζt is plotted above, the three pictures below it are zooms on the tracked object at frames (a) 1, (b) 55 (c) 56

Fig. 1 shows are example of failure detection, applied to the “Cliff Road” sequence, where this time the white car is tracked. The temporal evolution of ζt provides an example of tracking failure detection (strong decrease of ζt ) corresponding to incorrect motion estimation between frames 55 and 56. Tracking failure detection has shown to perform well for most objects we have attempted to track in a variety of sequences, with few false alarms. Let us point out that it is not necessary to be able to track over very long image sequences, but only over the length of a shot. Parameter α is tuned so as to favour false alarms w.r.t. failure mis-detection. With the current version, tracking time is about 0.5 sec. per frame, but it should decrease to approximately 0.2 sec. in some further implementation. We have presented a new technique for tracking in nearly real-time an image region within a shot. It relies on robust motion estimation, and involves the original idea of including a tracking failure detection method which exploits, at low cost, information related to the estimated motion model. Both tracking and failure detection have shown effective on many real sequences including movies and adverts.

A Region Tracking Method with Failure Detection

a1

a2

a3

b1

b2

b3

b4

b5

b6

c1

c2

c3

d1

d2

d3

267

Fig. 2. Region tracking on several sequences : the first image of the series shows the manually defined boundary : (a) “Irisa”sequence at time 1,12 and 116; (b) “Cliff Road”sequence at time 1,15,30,45,75,95; (c) “Emma” sequence at time 1,20,44; (d) “Car in pond” sequence at time 1,10,17. Sequences (b,c,d) are shots of the Avengers TV movie.

268

Marc Gelgon et al.

Acknowledgments: This work is funded in part by Alcatel CRC, Marcoussis, France, and by AFIRST (Association Franco-Israelienne pour la Recherche Scientifique). The authors would like to thank INA (Institut National de l’Audiovisuel, D´epartement Innovation) for providing the MPEG-1 Avengers sequence.

References 1. M. Basseville. – Detecting changes in signals and systems - a survey. – Automatica, 24(3):309–326, 1988. 265 2. S. Benayoun, H. Bernard, P. Bertolino, P. Bouthemy, M. Gelgon, R. Mohr, C. Schmid and F. Spindler. – Structuring video documents for advanced interfaces. – ACM Multimedia Conference, Demo session, Bristol, Sept. 1998. 261 3. P. Bouthemy and F. Ganansia. – Video partitioning and camera motion characterization for content-based video indexing. – In Proc. of 3rd IEEE Int. Conf. on Image Processing, volume I, pages 905–909, Lausanne, September 1996. 263 4. P. Bouthemy, M. Gelgon, and F. Ganansia. – A unified approach to shot change detection and camera motion characterization. – INRIA Report 3304, Nov. 1997. 261, 263 5. P. Correia and F. Pereira. – The role of analysis in content-based video coding and indexing. – Signal Processing, (66):125–142, 1998. 261 6. J.D. Courtney. – Automatic video indexing via object motion analysis. – Pattern Recognition, 30(4):607–625, April 1997. 261 7. U. Gargi, R. Kasturi, and S. Antani. – Performance characterization and comparison of video indexing algorithms. – In Conf. on Computer Vision and Pattern Recognition (CVPR’98), pages 559–565, Santa Barbara, June 1998. 261 8. M. Gelgon and P. Bouthemy. – Determining a structured spatio-temporal representation of video content for efficient visualisation and indexing. – In 5th European Conference on Computer Vision (ECCV’98), Freiburg, June 1998. 261, 262 9. B. G¨ unsel, A.M. Tekalp, and P. van Beek. – Content-based access to video : temporal segmentation, visual summarization and feature extraction. – Signal Processing, (66):261–280, 1998. 262 10. M. Irani and P. Anandan. – A unified approach to moving object detection in 2D and 3D scenes. – IEEE Trans. on Pattern Analysis and Machine Intelligence, 20(6):577–589, June 1998. 262 11. M. Irani and P. Anandan. – Video indexing based on mosaic representations. – Proc. of IEEE, 86(5):905–921, May 1998. 261 12. M. Isard and A. Blake. – Contour tracking for stochastic propagation of conditional density. – In 4th European Conf. on Computer Vision, Cambridge UK, April 1996. – LNCS 1065, Springer Verlag. 262 13. F. Marques and C. Molina. – Object tracking for content-based functionalities. – In SPIE Visual Communication and Image Processing (VCIP-97), volume 3024, pages 190–198, San Jose, 1997. 262 14. J-M. Odobez and P. Bouthemy. – Robust multiresolution estimation of parametric motion models. – Jal of Visual Communication and Image Representation, 6(4):348–365, December 1995. 263, 264

A Region Tracking Method with Failure Detection

269

15. J.M. Odobez and P. Bouthemy. – Direct incremental model-based image motion segmentation for video analysis. – Signal Processing, 66(3):143–156, mai 1998. 262 16. Y. Rosenberg and M. Werman. – Representing local motion as a probability distribution matrix and object tracking. – In Proc. of Conf. on Computer Vision and Pattern Recognition, Puerto-Rico, June 1997. 262 17. W. Xiong and J.C.M. Lee. – Efficient scene change detection and camera motion annotation for video classification. – Computer Vision and Image Understanding, 71(2):166–181, August 1998. 261 18. B.L Yeo and B. Liu. – Rapid scene analysis on compressed video. – IEEE Trans. on Circuits and Systems for Video Technology, 6(5):533–544, December 1995. 261

Integrated Parsing of Compressed Video Suchendra M. Bhandarkar1, Yashodhan S. Warke1 , and Aparna A. Khombhadia2 1

Department of Computer Science The University of Georgia, Athens, Georgia 30602–7404, USA 2 Netscape Communications Corporation 501 E. Middlefield Rd, Bldg 14, MV-093 Mountain View, California 94043–4042, USA

Abstract. A technique for detecting scene changes in compressed video streams is proposed which combines multiple modes of information. The proposed technique directly exploits and combines the luminance, chrominance, motion compensation information and the prediction error signal in an MPEG1-coded video stream. By performing minimal decoding of the compressed video stream, the proposed technique results in significant savings in terms of execution time and memory usage. The technique is capable of detecting abrupt scene changes (cuts), gradual scene changes (dissolves), and dominant camera motion in the form of pans and zooms in an MPEG1-coded video stream. Experimental results show that combining multiple modes of information is more effective in detecting cuts and dissolves. The proposed technique is capable of processing video frames in real time and could be used for the rapid generation of key frames for real-time browsing of video streams and for indexing to support content-based access to video libraries.

1

Introduction

Video parsing or scene change detection is typically used to extract key frames from a video stream. These key frames can then used for rapid video browsing and automatic annotation and indexing of video streams to support contentbased query access to a video database. The video parsing operation is primarily domain-independent and therefore is a crucial first step that precedes domaindependent analysis of the video [9]. Video parsing techniques that operate upon compressed video directly have a considerable advantage in terms of execution time and memory usage. Techniques that directly parse MPEG1-coded video typically use the histograms of the DC images derived from the DCT coefficients of the I frames [6], the variance of the DC coefficients in the I and P frames [2] or the proportion of macroblocks with valid motion vectors in the P and B frames [4,9] to detect abrupt scene changes. Yeo and Liu [8] present algorithms for detecting abrupt and gradual scene changes, intrashot variations and flashlight scenes in MPEG1coded and MJPEG-coded video data using DC images. Bhandarkar and Khombadia [1] present a technique for the detection of abrupt and gradual scene Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL’99, LNCS 1614, pp. 269–277, 1999. c Springer-Verlag Berlin Heidelberg 1999


270

Suchendra M. Bhandarkar et al.

changes that exploits motion compensation information and the prediction error in the MPEG1-coded video stream. However, video parsing techniques that rely on only one source or mode of information are seen to suffer from certain significant shortcomings. For example, techniques that rely only on chrominance and/or luminance values (or their average values in the DC images), are prone to misses when there is little change in background color or luminance between successive video shots and to false positives when there is a change in background color or background luminance due to change in ambient lighting within a single shot. The relative motion of objects between successive frames is more effective in detecting such scene changes. Moreover, certain types of scene changes such as pans, zooms etc. cannot be detected reliably with luminance and/or chrominance information alone. Using motion information in isolation also has its limitations. In MPEG1-coded video, I frames do not contain motion compensation information. Consequently, scene changes occurring at I frames cannot be detected if the video parsing technique relies solely on motion compensation information. The same is true of MJPEG video which consists solely of I frames. In this paper, we present a novel and efficient approach to video parsing that integrates luminance, chrominance and motion information, all of which are computed directly from an MPEG1-coded (compressed) video stream. It is shown that by appropriately combining luminance, chrominance and motion information, one can design a more accurate and robust video parsing technique. Since the proposed technique entails minimal decompression of the compressed video, it is capable of parsing video in real-time (i.e., at rates in excess of 30 frames/sec).

2

Description of MPEG1 Video

MPEG1 video compression [3] relies on two basic techniques: block-based motion compensation for reduction of temporal redundancy and Discrete Cosine Transform (DCT)-based compression for the reduction of spatial redundancy. The motion information is computed using 16 × 16 pixel blocks (called macroblocks) and is transmitted with the spatial information. The MPEG1 video stream consists of three types of frames: intra-coded (I) frames, predictive-coded (P) frames and bidirectionally-coded (B) frames. I frames use only DCT-based compression with no motion compensation. P frames use previous I frames for motion encoding whereas B frames may use both previous and future I or P frames for motion encoding. An I frame in MPEG1-coded video is decomposed into 8 × 8 pixel blocks and the DCT computed for each
block.
The DC term of the block DCT, termed as DCT (0, 0) is given by 18 7i=0 7j=0 IB (i, j) where IB is the image block. Operations of a global nature performed on the original image can be performed on the DC image [6,8,9] without significant deterioration in the final results. For P and B frames in MPEG1-coded video, motion vectors (MVs) are defined for each 16 × 16 pixel region of the image, called a macroblock. P frames have macroblocks that are motion compensated with respect to an I frame in the

Integrated Parsing of Compressed Video

271

immediate past. These macroblocks are deemed to have a forward predicted MV (FPMV). B frames have macroblocks that are motion compensated with respect to, either a reference frame in the immediate past, immediate future or both. Such macroblocks are said to have an FPMV, backward predicted MV (BPMV) or both, respectively. The prediction error signal is compressed using the DCT and transmitted in the form of 16 × 16 pixel macroblocks.

3

Scene Change Detection in MPEG1-Coded Video

Our technique is currently designed to detect two important types of scene changes in MPEG1-coded video streams: cuts and dissolves. A cut in a video stream is characterized by an abrupt scene change. A dissolve in a video stream is characterized by a gradual scene transition in which the present (outgoing) scene gradually fades out while the next (incoming) scene gradually fades in. Our technique is also capable of detecting dominant camera motion during zooms and pans and classifying the corresponding scenes as such. 3.1

Detection of Cuts Using DC Images

Let Xi , i = 1, 2, . . . , N be a sequence of DC images. The difference sequence Di , i = 1, 2, . . . , N − 1, is generated where Di = d(Xi , Xi+1 ). Each element of Di is the difference of the cumulative DC values of two successive images. A scene cut is deemed to occur between frames Xl and Xl+1 if and only if (1) The difference Dl is the maximum within a symmetric sliding window of size 2m − 1, i.e. Dl ≥ Dj , j = l − m + 1, . . . , l − 1, l + 1, . . . , l + m − 1, and (2) Dl is at least n times the magnitude of the second largest maximum in the sliding window. The parameter m is set to be smaller than the minimum expected duration between the scene changes [8]. Since the DC images have a luminance component and two chrominance components the difference image is the weighted sum of the absolute values of the individual component differences. 3.2

Detection of Cuts Using Motion Information

Since a cut in a video stream is characterized by an abrupt scene change, a typical motion estimator will need to intracode almost all the macroblocks at a cut. Thus, by computing the extent of motion or motion distance (MD) within each macroblock in the current frame relative to the corresponding macroblock in the previous frame and setting it to a very high value if such correspondence cannot be determined, scene cuts can be detected. The computation of MD is complicated by the fact that the corresponding macroblocks of successive frames may have MVs based on different reference frames. As an example, consider Fig. 1. Let the macroblock under consideration be (i, j) for frames #1 and #2. Also let M V1 = (u1 , v1 ) be the FPMV of the (i, j)th macroblock of frame #1, and M V2 = (u2 , v2 ) be the BPMV of

272

Suchendra M. Bhandarkar et al.

MV1

I frame (# 0)

MV1

MV2

(i, j)

(i, j)

B frame (# 1)

MV2

B frame (# 2)

P frame (# 3)

Fig. 1. Motion Distance Computation: Possible scenario I

MV1

MV4

MV1

MV2

(i, j)

(i, j)

MV2 II

IV MV3 MV3

III

I frame (# 0)

B frame (# 1)

B frame (# 2)

P frame (# 3)

Fig. 2. Motion Distance Computation: Two-level indirection

the (i, j)th macroblock of frame #2. Here the (i, j)th macroblock of frame #2 has a BPMV based on the (i, j)th macroblock of frame #3. However, the corresponding macroblock in frame #1 has a FPMV based on the (i, j)th macroblock of the reference frame #0. In order to calculate the MD or the motion in frame #2 relative to frame #1, we need to find the MVs for the corresponding macroblocks of successive frames based on the same reference frame. Thus, we require a two-level indirection for the (i, j)th macroblock of frame #2 to obtain its MV based on frame #0. This two-level indirection is shown using dotted arrows in Fig. 2. The order of computation is indicated by the number on the arrows. This two-level indirection yields M V4 = (u4 , v4 ), which is the MV of the (i, j)th macroblock of frame #2 based on the  reference frame #0. In the above case, the MD can be computed as MD = (u4 − u1 )2 + (v4 − v1 )2 . Since the computation of M D involves floating point arithmetic, we approximated it by MDa the computation of which entails only integer arithmetic: MDa = |u4 − u1 | + |v4 − v1 |. In general, for two consecutive frames Prev and Curr the corresponding macroblocks may have an FPMV, a BPMV, both or none. Thus, depending on the type of MVs the macroblocks possess, sixteen cases need to be dealt with. We refer the interested reader to [1] for a detailed explanation of how MD is computed in each of these cases. In order to detect cuts, we calculate the motion edge magnitude for each frame which is defined as the sum of MD’s (or MDa ’s) of all the macroblocks in the frame. We detect scene cuts by thresholding the peaks within a sliding window of length 2m + 1 frames in the plot of the motion edge magnitude versus frame number using the same criteria discussed in Section 3.1.

Integrated Parsing of Compressed Video

3.3

273

Detection of Gradual Scene Changes Using DC Images

c2

| c2-c1 |

c1

a1

a2

a1-k

a2-k

a1

a2

Fig. 3. Gradual Transition and Difference Sequence

A gradual scene transition is modeled as a linear transition from c1 to c2 , in the time interval [a1 , a2 ] (Fig. 3) and modeled as [8]:  n < a1 ,  c1 , 1 gn = ac22 −c (1) −a1 (n − a2 ) + c2 , a1 ≤ n < a2 ,  n ≥ a2 c2 , Assuming that k > a2 − a1 , the difference sequence Dik (gn ) = d(Xi , Xi+k ) is given by [8]:  0, n < a1 − k,    |c2 −c1 |   [n − (a − k)], a1 − k ≤ n < a2 − k,  a2 −a1 1 k a2 − k ≤ n < a1 , Di (gn ) = |c2 − c1 |, (2)  |c2 −c1 |   − (n − a ), a ≤ n < a , 2 1 2    0, a2 −a1 n ≥ a2 The plots of gn and Dik (gn ) are shown in Fig. 3. The plateau between a2 − k and a1 has a maximum constant height of |c2 − c1 | if k > a2 − a1 . In order to detect the plateau width, it is required that for fixed k: (1) |Dik − Djk | < , where j = i − s, . . . i − 1, i + 1, . . . , i + s, and k k (2) Dik ≥ l × Di−k/2−1 or Dik ≥ l × Di+k/2+1 , for some large value of l. Since the width of the plateau is k − (a2 − a1 ) + 1, the value of k should be chosen to be ≈ 2(a2 −a1 ) where a2 −a1 is the (expected) length of the transition. 3.4

Detection of Gradual Scene Changes Using Motion Information

During a dissolve, the motion estimation algorithm typically finds the best matching blocks in the reference frame(s) for blocks in the current frame but at the cost of higher prediction error. Also, in a typical dissolve, the error is uniformly distributed in space and value over all of the macroblocks. These observations are encapsulated in the following metrics (i) The average error

274

Suchendra M. Bhandarkar et al.


M
N

E(i, j) over the M · N macroblocks should be high,
M
N 1 2 = MN [E − E(i, j)]2 should be high (ii) The error variance σE i=1 j=1
M avg
N ijE(i,j) j=1 2 and (iii) The error cross covariance σij =
i=1 − iavg javg , where M
N E(i,j)
M
N
M
N i=1 j=1 iE(i,j) jE(i,j)
j=1
j=1 and javg =
i=1 , should be low. iavg =
i=1 M N M N Eavg =

1 MN

i=1

4

i=1

j=1

j=1

E(i,j)

i=1

j=1

E(i,j)

Detection of Scenes with Camera Motion

Scenes containing dominant camera motion such as pans and zooms can be detected and classified based on the underlying pattern of the MVs. In a pan, most of the MVs are aligned in a particular direction. For a zoom-in most of the MVs point radially inwards towards the focus-of-expansion (FOE) whereas for a zoom-out they point radially outwards from the FOE. Let θij be the direction of the MV associated with the ijth macroblock. 1
M
N Let θavg = MN i=1 j=1 θij be the average of the MV directions in the frame. For a frame to qualify as a member of a pan shot, the variance σθ2 =
M
N 1 2 i=1 j=1 [θij − θavg ] should be less than a predefined threshold. MN For zoom-ins and zoom-outs we analyze the MVs of frame in each of the four quadrants. Let M Vx and M Vy denote the MV components along the horizontal and vertical directions respectively. We compute the total number of macroblocks in each quadrant satisfying the following criteria: Zoom-In: (i) First Quadrant: M Vx < 0 and M Vy < 0; (ii) Second Quadrant: M Vx > 0 and M Vy < 0; (iii) Third Quadrant: M Vx > 0 and M Vy > 0; and (iv) Fourth Quadrant: M Vx < 0 and M Vy > 0. Zoom-Out: (i) First Quadrant: M Vx > 0 and M Vy > 0; (ii) Second Quadrant: M Vx < 0 and M Vy > 0; (iii) Third Quadrant: M Vx < 0 and M Vy < 0; and (iv) Fourth Quadrant: M Vx > 0 and M Vy < 0. If the total number of macroblocks in the frame satisfying the aforementioned quadrant-specific conditions exceeds a specified threshold then the frame is classified as belonging to a zoom-in or zoom-out as the case may be.

5

Integrated Parsing

The current integration of luminance-based, chrominance-based and motionbased approaches is based on computing a joint decision function which is the weighted sum of the detection criteria of the individual approaches. At present the weights are selected by the user. Automatic weight selection is a topic we wish to pursue in the future. At this time the integrated approach applies only to scene change detection i.e., detection of abrupt and gradual scene changes. Detection of pans and zooms is done based on motion information alone. The GUI for the system has been developed in Java using JDK 1.1.5. The GUI allows the user to select the MPEG video, decide which parsing approach to use, decide

Integrated Parsing of Compressed Video

275

various threshold values, decide the relative weights of the individual approaches in the integrated approach, and view the plots of the various detection criteria.

6

Experimental Results

The MPEG1 video clips, used for testing the proposed technique were either generated in our laboratory or downloaded over the Web [7]. The Tennis sequence contains cuts at frames 90 and 150. It was observed that the scene changes were successfully detected only by the MV approach (Fig. 4), with the DC difference approach displaying very small peaks in the luminance and chrominance domain (Fig. 5). The integration of the two approaches indicates clear peaks at both scene cut points (Fig. 6) thus showing it to be more reliable. The Spacewalk1 sequence contains 3 different dissolve sequences: between frames 74 and 85, between frames 155 and 166, and between frames 229 and 242. Both the approaches detected the 3 dissolves accurately. However, it was found that the DC k-difference approach overestimated the span of the dissolve (Fig. 7). The results of the integrated approach (Fig. 9) were found to be more accurate than those of the MV approach (Fig. 8) and the DC k-difference approach (Fig. 7) individually. The videos used for testing the zoom and pan detection were created in our laboratory using a video camera. The results for one of the zoom sequences are depicted in Fig. 10. This clip has 267 frames with 2 zoom-outs, between frames 77 and 90, and frames 145 and 167. The algorithm successfully detected these zoom-out sequences (Fig. 10). The results for one of the pan sequences are shown in Fig. 11. The clip has a pan sequence starting from frame 90 and ending at frame 153. The algorithm successfully detected this pan sequence, by displaying a low value for the variance of the MV angle (Fig. 11).

7

Conclusions and Future Work

The integrated approach presented in this paper, combines information from multiple sources such as luminance, chrominance and motion and is capable of more reliable detection of cuts and dissolves, and also pans and zooms in MPEG1 video. Since the MPEG1 video is minimally decompressed during parsing, the approach results in great savings in memory usage and processing bandwidth. Future work will investigate detection of other scene parameters such as wipes, morphs and object motion, the automatic selection of the various threshold values and also learning these threshold values from examples.

References 1. S.M. Bhandarkar and A.A. Khombadia, Motion–based Parsing of Compressed Video, Proc. IEEE IWMMDBMS, Aug. 1998, pp. 80–87. 269, 272 2. H. Ching, H. Liu, and G. Zick, Scene decomposition of MPEG compressed video, Proc. SPIE Conf. Dig. Video Comp., Vol. 2419 Feb. 1995, pp. 26-37. 269

276

Suchendra M. Bhandarkar et al.

Fig. 4. Tennis: Motion Fig. 5. Tennis: DC differ- Fig. 6. Tennis: Inteedge plot ence plot grated approach plot

Fig. 7. Spacewalk1: DC k- Fig. 8. Spacewalk1: Error Fig. 9. Spacewalk1: Intevariance plot difference plot grated approach plot

Fig. 10. Pan7: Plot of % of pixels satisfying zoom criteria

Fig. 11. Pan5: Plot of MV angle variance

3. D.L. Gall, MPEG: A video compression standard for multimedia applications, Comm. ACM, Vol. 34(4), 1991, pp. 46-58. 270 4. V. Kobla, D. Doermann, K.I. Lin and C. Faloutsos, Compressed domain video indexing techniques using DCT and motion vector information in MPEG video, Proc. SPIE Conf. Stor. Retr. Img. Vid. Dbase., Vol. 3022, Feb. 1997, pp. 200-211. 269 5. J. Meng and S.F. Chang, CVEPS - A Compressed Video Editing and Parsing System. Proc. ACM Conf. Multimedia, Nov. 1996, pp. 43-53. 6. K. Shen and E. Delp, A fast algorithm for video parsing using MPEG compressed sequence, Proc. IEEE Intl. Conf. Image Process., Oct. 1995, pp. 252-255. 269, 270 7. University Of Illinois, Urbana-Champaign. ACM Multimedia Lab for sources of MPEG video data, URL : http://www.acm.uiuc.edu/rml/Mpeg/ 275 8. B.L. Yeo and B. Liu, Rapid scene analysis on compressed video, IEEE Trans. Cir. and Sys. for Video Tech., Vol. 5(6), 1995, pp. 533-544. 269, 270, 271, 273

Integrated Parsing of Compressed Video

277

9. H.J. Zhang, C.Y. Low, and S.W. Smoliar, Video parsing and browsing using compressed data, Jour. Multimedia Tools Appl., Vol. 1(1), 1995, pp. 89-111. 269, 270

Improvement of Shot Detection Using Illumination Invariant Metric and Dynamic Threshold Selection Weixin Kong, Xianfeng Ding, Hanqing Lu, and Songde Ma National Laboratory of Pattern Recognition Institute of Auotomation, Chinese Academy of Sciences P.O.Box 2728, Beijing 100080, P.R.China [email protected]

Abstract. Automatic shot detection is the first step and also an important step for content-based parsing and indexing of video data. Many methods have been introduced to address this problem, e.g. pixel-by-pixel comparisons and histogram comparisons. But gray or color histograms used in most existing methods ignore the problem of illumination variation inherent in the video production process. So they often fail when the incident illumination varies. And because shot change is basically a local process of a video, it is difficult to find an appropriate global threshold for absolute difference measure. In this paper, new techniques for shot detection are proposed. We use color ratio histograms as frame content measure, because it is robust to illumination changes. A local adaptive threshold technique is adopted to utilize the local characteristic of shot change. The effectiveness of our methods is validated by experiments on some real-world video sequences. Some experimental results are also discussed in this paper.

1. Introduction and Related Work Archiving and accessing multimedia information have become important tasks in several important application fields. Areas that will benefit from advances on this subject include VOD (video on demand), DLI (digital library), etc. Of all the media types, video is the most challenging one, because it combines all the other media information into a single bit stream. Because of its length and unstructured format, it is hard to efficiently browse and retrieve large video files. The most popular used approach consists in assigning key words to each stored video file, and doing retrieval only on these key words. But these key words often can’t capture the rich content of the videos. So the method to browse and retrieve video sequences directly by their content is becoming urgently needed. As far as video browsing and retrieval are concerned, the primary requirements are the construction of video structure. For a typical video film, it has an obvious structure hierarchy, i.e. video, scene, shot and frame. A shot is a sequence of video frames generated during a continuous camera operation. It can serve as the smallest indexing Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL'99, LNCS 1614, pp. 277-282, 1999.  Springer-Verlag Berlin Heidelberg 1999

278

Weixin Kong et al.

unit. So shot detection is an important first step in content-based video browsing and retrieval. In recent years, substantial efforts have been devoted to shot-based video analysis. A survey on this topic can be found in [1]. An assumption that is often made is that the content should not change greatly from one frame to the next within one camera shot. So, In general, shot boundaries can be detected by employing a difference metric to measure the change between two consecutive frames. A shot boundary is declared if the difference between the two frames exceeds a certain threshold. Therefore, the key issues in detecting shot boundaries are selecting suitable difference metrics and appropriate thresholds. Different solutions have been designed using pixel- or block- based temporal image difference [2,3], or difference of gray and color histograms [4,5]. Histograms are robust to object motion, i.e. two frames having an unchanged background and objects will show little difference in their overall gray or color distribution. And they are simple to compute. So they have been widely used in shot-based video analysis. And several authors claim that this measure can achieve good trade-off between accuracy and speed. In this paper, we also address the problem of segmenting the film into a sequence of shots based on difference of histograms. The rest of the paper is organized as follows. In section 2, we discuss our illumination invariant frame content metric: color ratio histogram. In section 3, we propose our shot detection algorithm. We use color ratio histograms as frame content measure, because it is robust enough against all noncontent-based changes (some small movements, changes of illumination condition etc.). To utilize the local characteristic of shot change, a local adaptive threshold technique is adopted. The effectiveness of our methods is validated by experiments on some real-world video sequences. Some experimental results are also discussed in this section. Conclusion and future work are in section 4.

2.Illumination Invariant Frame Content Metric Gray or color histograms used in most existing methods ignore the problem of illumination variation inherent in the video production process. So they often fail when the incident illumination varies. Even simple lighting changes will result in abrupt changes in histograms. This limitation might be overcome by preprocessing with a color constancy algorithm. In [6], Wei Jie etc. proposed a color-channelnormalization method, then they reduced the three-dimension color to two-dimension chromatic and defined a two-dimension chromatic histogram. Their method can discount simple spectral changes of illumination. But it is computationally expensive and can’t do with spatial changes of illumination. In [7], Funt etc. showed that neighbourhood based color ratios are invariant under various changes both in spectrum and spatiality. They studied the use of color ratio histograms as features for indexing image database. Now we adopt color ratio histograms in shot detection. Color ratio histogram can be formulated as following:

Improvement of Shot Detection

279

H (i , j , k ) = å z( x , y ), x,y

ì1, if d R ( x , y ) = i d G ( x , y ) = j d B ( x , y ) = k z( x, y) = í 0, else î 2 d k ( x , y ) = ∇ i k ( x , y ), k = R, G , B

(1)

i k ( x , y ) = log f k ( x , y ), k = R, G , B where f ( x, y ) is the RGB color values at position ( x , y ) , i ( x, y ) is their logarithms and d ( x, y ) is the Laplacian difference of i ( x, y ) . Color ratio histogram is the histogram of d ( x, y ) .Obviously, it can be seen as the histogram of color ratios between a small image region. Since the ratios of color RGB triples from neighboring locations are relatively insensitive to changes in the incident illumination, this circumvents the need for color constancy preprocessing. Logarithm can be calculated by table lookup and Laplacian transform can be implemented by simple mask convolution operation. So its computational cost is very low.

3. Shot-Based Video Analysis Shot-based video analysis involves segmenting the video clip into a sequence of camera shots by detecting shot boundaries. There are two types of shot boundary: camera breaks and gradual transitions. The later kind of boundaries is obviously more difficult to detect. 3.1 Camera Break Detection Figure 1 shows the difference of traditional color histograms versus time in a sequence of 300 frames. There are two camera breaks at frame 11 and 238. But in the shot between these two frames there are many illumination changes due to firework. As a result, many peaks of difference value occur. For example, at frame°226, a very high pulse exists. But actually there is only a simple lighting change, the contents of two consecutive frames are basically same. While in figure°2, which illustrates the difference of color ratio histograms, these pulses have been effectively smoothed. And we can easily detect the two breaks. Another improvement of our algorithm is that we introduce a relative difference measure between two consecutive frames. Because the shot change is basically a local characteristic of a video, It is difficult to find an appropriate global threshold for absolute difference measure. For a highly active video sequence, there will be a lot of frames where absolute histogram differences are large and using only absolute measures will bring many false alarms. The increment ratios of the histogram differences instead emphasis on this locality. It can be described as: CD = D ( H n +1 , H n ) / D ( H n − H n −1 )

Here D ( H

n +1,

H

n

(2)

) is the absolute difference measure of two frames.

280

Weixin Kong et al.

Figure 3 shows this relative measure. It can be seen that this measure is more reasonable in detecting shot changes.

Fig. 1.

Fig. 2.

Fig. 3.

3.2 Gradual Change Detection Unlike camera breaks, a gradual shot change does not usually causes sharp peaks in the inter-frame difference of a video sequence, and can be easily confused with object or camera motion. It often lasts more than ten or more frames instead of only one frame. So gradual changes are usually determined by observing the inter-frame differences over a period of time. In [4], Zhang etc. proposed a method called twin comparison technique. Two thresholds Tb , Ts , Ts < Tb are set for camera breaks and gradual changes respectively.

Improvement of Shot Detection

281

If the histogram difference between two consecutive frames satisfies Ts < SD(i, i + 1) < Tb , the ith frame is marked as the potential start frame of a gradual change. For every potential frame detected, it is compared to subsequent frames. This is called an accumulated comparison Ac . The comparison is computed till Ac > Tb and SD < Ts . The end of the gradual change is declared when this condition is satisfied. But the thresholds in their method are difficult to set. These thresholds should also vary in a long video sequence. Based the twin comparison method, we develop a new local adaptive threshold technique. We first calculate the average a and standard deviation σ of the histogram differences of the frames within a temporal window preceding the current frame. Then the histogram difference of the current frame is compared with this average value. We use a + ( 2 ~ 3)σ as the threshold to detect the start frame of a gradual change and use a + (5 ~ 6)σ to detect the end frame. 3.3 Shot Detection Experimental Results Our approach has been validated by experiments with several kinds of video sequences. These sequences contain usual features related to film producing and editing, including lighting condition, object and camera motion, and editing frequency. Table 1 and 2 refer to performance results obtained in our experiments on two sequences, where the error rate reported on last column is computed as the ratio: (False+Missed)/Real. Shot Type

False

Missed

Real

Error Rate

Camera Break

2

1

96

3.1%

Gradual Change

1

0

10

10%

Table 1. Experimental results on sequence 1(5975 frames) Shot Type

False

Missed

Real

Error Rate

Camera Break

3

1

113

3.5%

Gradual Change

1

1

13

15%

Table 2. Experimental results on sequence 2(9136 frames)

282

Weixin Kong et al.

4 Conclusions and Future Work In this paper, new techniques for shot detection are proposed. We use color ratio histograms as frame content measure. It is robust to illumination changes. A local adaptive threshold technique is adopted to utilize the local characteristic of shot change. The effectiveness of our methods is validated by experiments on some realworld video sequences. Experimental results show that our method is effective in detecting both camera breaks and gradual changes. From these experiments, we also find that the amount of shots in a typical film is very large (one shot per 3 seconds). So only shot level video structure can’t guarantee efficient browsing and retrieval. Higher semantic level analysis of the video content and the construction of scene structure are very important. A scene is defined as a sequence of shots related by semantic features. It is the scene that constitutes the semantic atom upon which a film is based. Obviously the construction of scene structure is a far more difficult research task when compared with shot detection. And little work has been done to this problem. Based on our accurate shot detection algorithm, we will study the problem of scene structuring in the future.

References 1. J.S. Boreczky and L.A. Rowe, Comparison of video shot boundary detection techniques, Proc. SPIE Conf. Storage and Retrieval for Image and Video Databases IV, Vol. 2670, pp170--179, 1996. 2. K.Otsuji, Y.Tonomura and Y.Ohba, Video browsing using brightness data, Proc. SPIE Conf. Visual Communications and Image Processing, pp.980-989, November 1991. 3. A.Nagasaka and Y.Tanaka, Automatic video indexing and full-video search for object appearances, Proc. 2nd Visual Database Systems, pp119-133, October 1991 4. H.Zhang, A.Kankanhalli, and S.Smoliar, Automatic partitioning of full-motion video , Multimedia Systems, vol. 1, pp10-28, 1993 5. I.K.Sethi, N.Patel, A Statistical Approch to Scene Change Detection, Proc. SPIE Conf. Storage and Retrieval for Image and Video Databases, vol.2420, pp329-338, 1995 6. J.Wei, M.S.Drew, and Z.-N.Li, Illumination-invariant video segmentation by hierarchical robust thresholding, Proc. SPIE Conf. Storage and Retrieval for Image and Video Databases, vol.3312, pp.188-201,1998 7. B.V.Funt and G.D.Finlayson, Color Constant color indexing, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.17, pp522-529, 1995

Temporal Segmentation of MPEG Video Sequences Edoardo Ardizzone, Carmelo Lodato, and Salvatore Lopes CNR-CERE Centro Studi sulle reti di Elaboratori Viale delle Scienze, 90128 Palermo, Italy [email protected] {ino,toty}@cere.pa.cnr.it

Abstract. The video segmentation is a fundamental tool for the video semantic content evaluation. In multimedia application, videos are often in MPEG-1 format. In this paper, an algorithm for the automatic shot segmentation of MPEG-1 sequences is presented. The adopted method is based on heuristic considerations concerning the characteristics of MPEG-1 video streams. In particular, the pattern structure and the I-, B- and P-frame sizes are taken in account. The proposed algorithm has been applied to MPEG-1 sequences and some results are reported.

1

Introduction

The effective use of video databases requires videos be indexed not only by textual data, but also (and mainly) by the visual features they contain. This kind of databases are conventionally called “content-based video databases” (CBVD) and systems that allow to retrieve images by their visual content are often referred to as “content-based retrieval system” (CBRS).1 The visual content is described by features related to color, texture, object structure, etc. Features are normally extracted from images in a manual, or semi-automatic, or automatic way during the phase of DB population, and stored in a feature database. During the query phase, the feature DB is searched for features most similar to those one provided by the user, and the related images, sorted in order of similarity, are shown to the user. As far as videos are concerned, motion features, e.g. related to the objects’ motion or to camera movements, are also important. Anyway, the first step of feature extraction is normally a temporal segmentation process. The objective of this process is the detection of scene cuts, in order to reduce the video to a sequence of short dynamic scenes2, generally characterized by a set of homogeneous features. Each scene may be therefore characterized by the features of one or more representative frames, i.e. still images3. The operation and the characteristics of several systems of this kind may be found in [2,3,5]. For example, the methodology and the algorithms used by JACOB, a general purpose system particularly suited for storage and retrieval of TV sequences, have been described in [2]. Some examples of “content-based retrieval systems” are described in [4,6,9,10,11] Often referred to as shots in the literature. 3 Often referred to as r-frames in the literature. 1 2

Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL'99, LNCS 1614, pp.283 -290, 1999.  Springer-Verlag Berlin Heidelberg 1999

284

Edoardo Ardizzone et al.

Most of these systems, JACOB included, operate on uncompressed video sequences. Nevertheless, video sequences are often compressed for efficient transmission or storage. Therefore, compressed videos have to be decompressed before indexing algorithms may be applied, thus requiring the application of computationally intensive processing steps. More recently, pre-processing of compressed videos directly on compressed domain has been proposed by several authors, mainly for MPEG-1 bitstreams [12-14]. Algorithms for scene change detection in MPEG-1 compressed video sequences have been proposed in [15]. Some video indexing methods based both on motion features (mainly camera movements and operations such as zooming and panning) and on motion-based spatial segmentation.of single frames have been presented in [16]. In this paper we propose a method for scene cut detection which does not need any decompression. The method is based on the external analysis of the characteristics of the MPEG-1 bitstream. In particular, the frame pattern of MPEG-1 coding, the size and the size changes of I, P and B frames are used to decide where a scene cut is more probable to be. The decision is based on heuristics. Since no decompression process is necessary, and moreover the analysis is based on very simple computation, the algorithm is very speedy. Moreover, as shown later, it is accurate enough to be used as a tool for prreliminary segmentation step. The rest of the paper is organized as follows. In section 2, MPEG-1 characteristics are reviewed. Section 3 describes the proposed algorithm, and section 4 reports the first experimental results.

2

MPEG-1 Coding Characteristics

The MPEG-1 standard concerns the audio and video digital signal compression [1]. The video compression is achieved removing the redundant information in a sequence of pictures. The compression ratio can be chosen in such a way that the compression process does not alter the quality of the compressed sequence. The standard concerns also the multiplexing of audio and video signals, but in this paper only video streams are treated. The video compression process exploits the spatial redundancy in a single picture and the temporal redundancy between pictures that are close to each other. The spatial redundancy is reduced using a block coding technique. The temporal redundancy is reduced using a motion estimation technique. During the compression process, the MPEG-1 encoder decides whether the arriving picture should be compressed using the block or the motion estimation technique. The frames compressed using only the block coding are called intra-frames or I-frames. Frames with motion estimation coded respect to the previous one are called predicted frames or P-frames. Frames coded respect to the previous and following frames are called bidirectionally predicted frames or B-frames. The P frames are always coded respect to the closest previous I or P frames. B frames are always coded respect to the previous and following I or P frames. An MPEG-1 video will be characterised by a recurrent sequence of I-, P- and B- frames, always starting with an I-frame, commonly called pattern. Generally, the pattern structure depends from the frame rate, because it is necessary to code at last two I frames every second for reasons related to random access and error propagation. The pattern structure is decided at the coding time.

Temporal Segmentation of MPEG Video Sequences

3

285

The Algorithm

The proposed algorithm for the automatic segmentation into shots of MPEG-1 sequences is essentially based on the search of “points” potentially representing scene cuts. These “points” or “events” are detected analyzing the external characteristics of the MPEG-1 videos. In this study, the characteristics taken in account are the pattern structure and the sizes of all frame types. Internal characteristics as intensity, chrominance or motion vectors are not considered. This choice agrees with the simplicity and speed requirements expressed in the introduction. The algorithm scans a whole MPEG-1 trace searching any anomalies with respect to pattern structures or to frame sizes. As already said, an MPEG-1 sequence consists in a recurrent structure of I-, P- and B- frames called pattern. The pattern length and configuration depends from the video frame rate. For example, the coded pattern IBBPBBPBBPBB is 12 frames long for a 24 frame per second video. The pattern IBBPBBPBBPBBPBB is 15 frames long when the video is coded at 30 frame per second. Normally, the pattern remains unchanged for the whole duration of the sequence. On the other hand, some MPEG-1 encoders can modify the coding sequence to the aim of improving on the perceptual quality of compressed videos. Such a modification consists, for instance, in coding a P frame instead of a B frame, or in the truncation of the current pattern before its completion. Generally, pattern changes happen very seldom and subsequently a succession of normal pattern restarts. Pattern changes can be necessary, for example, to code fast evolving scenes. Digital videoediting tools that operate directly on compressed sequences could also introduce pattern changes or truncations. IBBPBBPBBPBB-IBBPBBPBBPBBPBB-IBBPBBPBBPBB

pattern change

Frame I n 837

Frame I n 852

Fig. 1. Correspondence between the pattern and semantic content change

From an analysis of MPEG-1 videos including localized alterations of the normal or prevalent pattern, a strict correlation between changes of pattern structure and changes in the semantic content of the pictures in the frames before and after the pattern modifications has been observed. It follows that the frames corresponding to the pattern changes could be regarded as scene cuts between contiguous distinct shots. Thus, the complete patterns that precede and follow the modified one could be considered belonging to different shots. In Fig. 1, a typical observed case is reported.

286

Edoardo Ardizzone et al.

The figure shows the pattern change and the corresponding scene change between the I frames preceding and following it. The other methods adopted for detecting potential scene cuts are all derived from an analysis of frame sizes. I-frames, as already said, include all the necessary information for their decoding. For this reason, their sizes are in some way correlated to the picture complexities. I-frames of similar size will represent pictures of the same complexity but not necessarily equals. That is, it is not possible to find a useful relationship between the semantic content of a picture and the size of the frame coding it. Nevertheless, I-frames following each other in the same video in a very short time interval do not show significant variation on their size. On the other hand, a significant size variation between consecutive Iframe pairs can probably indicate a semantic content variation in the interval between them. Although this consideration can seem rather obvious from the qualitative point of view, the quantitative determination of the threshold to be overcome by a size variation to be significant is not so easy. This conclusion derives from the analysis of several video sequences of different subjects and compressed with different MPEG-1 encoders. Threshold values vary strongly from a sequence to another one and many trials have been carried out in order to find a method suitable for sequences of any type. An easy to implement procedure for threshold determination is explained in the following. Firstly, frame sizes have been normalised using the following relationship:

I *j =

I j − I min

(1)

I max − I min

where I*j, Ij are respectively the normalised and the original size of j-th frame, Imax and Imin the sizes of the greatest and of the smallest I-frame. Then, the statistical distribution of the differences ∆I* between the normalised size of the consecutive I frame pairs have been evaluated. Fig. 2 shows the statistical distribution of ∆I* for an examined sequence in the (0, 1) interval. 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0%

minimum percentage

threshold value 0

0,1

0,2

0,3

0,4

∆ I*

0,5

0,6

0,7

0,8

0,9

1

Fig. 2. Threshold value determination for ∆I*c

The curve plotted in Fig. 2 expresses the cumulative frequency of ∆I* values. The threshold value determination is achieved searching the first point of that curve above a fixed value of the cumulative frequency (90% in the plot), with a null value of the tangent. Such a criterium, chosen to discern ∆I* values that are statistically anomalous can be easily applied to all types of sequences. A value of ∆I* over the thresholds allows to detect a potential scene cut happening in a two pattern long interval. If the

Temporal Segmentation of MPEG Video Sequences

287

transition between two contiguous shots is very smooth, with a very slow fade-in fade out effect, the correspondent ∆I* values could be not large enough to signal the change. In such a situation, a useful consideration could be done on the normalised frame sizes I*. As matter of fact, although the scene evolution is such as that the values of ∆I* stay below the threshold value, the corresponding I* can still show an anomalous behaviour. That is, the value I* exceeding a proper threshold, can mark anyway a potential scene cut that otherwise would not be detected. The threshold value for I* is evaluated with the same procedure presented above. P-frames are coded using both block and motion estimation technique, that is, reducing the temporal redundancy with respect to the nearest previous I or P frames, and also the spatial redundancy of coded pictures. For this reason, the P-frame sizes represent themselves the scene evolution with respect to the reference frame. A succession of consecutive P-frames of large size will represent a scene that evolves fast. It is reasonable to suppose that a P-frame of a very large size is located at the beginning of a new shot. Just as for the previous case, the P-frame sizes have been normalised using the same relationship (1), substituting I-frame size with P-frame size and, from the statistical distribution, is possible to find the first point with a null tangent value above a fixed value of the cumulative frequency curve. The difference ∆P* of normalised sizes between consecutive P-frame pairs can also be used to give an evaluation of the variation rate in time of the corresponding pictures. In analogy with already discussed cases, P-frames with the corresponding ∆P* greater than a threshold value are searched by the algorithm. This search applies to scenes where there is a relative motion between the camera and the subject. In this case, there could be, for example, a succession of P-frames with P* all below the threshold value, but that differ each other significantly. In such a situation, also the ∆P* could reveal a shot transition undetectable instead. The same consideration for P-frames can be applied to B-frames too. A further search can be done considering the sum of frame sizes belonging to the same pattern, that is the GOP (Group of Pictures) size. The results of all the searches (I* , ∆I* , P* ,∆P* , B* , ∆B* , GOP*, ∆GOP*) except for pattern change depend strongly from the threshold values chosen for each search. These values are derived from a statistical analysis fixing a maximum percentage value of items that probably will exceed the threshold value. For instance, a 95% value implicates that no more than 5% of the items will overcome the fixed threshold. This parameter depends on the number of the items in the sample and on the desired result. As matter of fact, increasing this parameter, the number of items exceeding the threshold decreases and the probability that the detected items correspond to effective scene changes increases. Conversely, decreasing this value in order to detect more events could result in an excessive oversampling. From a qualitative study on MPEG 1 videos, it has been observed that the trends of frame size of each type vary considerably during the whole sequence. This suggests that the choice of a unique threshold value for a long sequence could not produce good results. For this reason, the results can be improved on by applying the above search procedures on portions of the original video streams to the aim of finding suitable threshold values for each subinterval. The stream partitioning in subintervals can be performed on the basis of the local video characteristics or from statistical considerations. Naturally, the partitioning can not be too fine for the sake of simplicity and for reasons correlated to the significance of the statistical sample. In fact, with too few items the statistical

288

Edoardo Ardizzone et al.

analysis can not be applied. Once all the searches have been performed, the subsequent task of the algorithm consists in an analysis of all detected events in order to eliminate the oversampling. Because each search produces events that are signalled from frames of different type, a single event corresponding to a scene cut could be marked several times on frames that are close to each other. A filtering process is then required in order to avoid an incorrect evaluation of multiple close events as distinct events, substituting a set of close signals with a single event. Close events can be considered coincident if they are in correspondence with frames belonging to the same pattern, or to two adjacent patterns or to an interval of fixed length. So, setting up a minimal resolution of the algorithm, the segmentation in shots shorter than a fixed interval is avoided. In this situation we assumed that the last I-frame before the fixed interval and the last I-frame within the interval, belong to different shots. A resolution of two patterns would result in neglecting scene shorter than 1 second. From a practical perspective, in an application for storing and indexing the segmented shots in a content-based database, it could be less significant to detect shots of short length.

4

Experimental Results

The presented algorithm has been tested and tuned on a set of MPEG-1 sequences of different subject, length and quality. In this section, the reported results regard three MPEG-1 movies, chosen for the good video quality, for their length and for the number of scene cuts included. The transitions between adjacent shots are characterised by both abrupt and slow changes. The used movies are available as demo in the software set of a commercial graphic video card. The relevant video characteristics are summarised in table 1. Table 1. Relevant characteristics of the MPEG-1 video sequences used in the trials

Name Ontario Toronto History

N of frames 7008 4302 5808

Length (s) 292 179 242

N of shots 136 119 47

The algorithm requires the knowledge of the type and the size of each frame. The collection of all the necessary information from an MPEG-1 movie 5000 frame long requires about 20 s of processing time on a pc pentium 300MHz. The required processing time for the automatic shot segmentation requires about 0.15 s. Two series of trials has been performed. In the first set, the parameters have been fixed in order to achieve the most successful events minimising the oversampling. To this end, a procedure for automatic determination of the relevant parameters (number of subintervals, minimal percentage for threshold value evaluation) has been developed. The length of each subinterval has been taken in such a way to obtain a statistical sample ranging from 250 up to 350 items. In the second set of trials, a minimal percentage value of 80% for the threshold evaluations has been fixed in order to maximise the number of successful events with no care for the oversampling. The trials of the second set have been carried out in the perspective of using the proposed

Temporal Segmentation of MPEG Video Sequences

289

algorithm as pre-processing phase in the framework of a more sophisticated automatic shot segmentation system. As matter of fact, more complex systems need to decode partially or completely the video streams, thus requiring a considerable longer processing time. A pre-processing phase carried out with the proposed algorithm could reduce greatly the items to be processed obtaining a relevant gain in the global processing time. In both trials, the algorithm resolution has been set to 20, thus avoiding that two consecutive signals are not less than 20 frame far from each other. This value derives from the actual implementation of the filtering phase that is not accurate enough for lesser values of the resolution. The resolution can be improved on up to the intrinsic precision of each search using more sophisticated filtering techniques. The adopted resolution value does not necessary implicate that the algorithm can not detect shots shorter than 20 frame. In fact, in the current implementation, each shot is represented by the frame extracted from the middle of the interval limited by two consecutive events. Thus, the adopted value does not preclude a priori the correct determination of a shorter shot. Table 2 reports all the relevant parameters for the two series of trials. Table 2. Parameter values for the two trial sets Sequence Ontario - I Toronto - I History - I Ontario - II Toronto - II History - II

N of intervals I P B 2 7 18 1 4 11 1 5 15 2 7 18 1 4 11 1 5 15

Minimum percentage I* ∆I* P* ∆P* B* ∆B* 0.92 0.98 0.99 0.92 0.98 0.99 0.92 0.98 0.99 0.80 0.80 0.80 0.80 0.80 0.80 0.80 0.80 0.80

100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0%

Resolution (n frames) 20 20 20 20 20 20

undetected detected

Ontario-I

Toronto-I

History-I

Ontario-II

Toronto-II

History-II

Fig. 3. Summary of trial results

The results of all the trials are reported in Fig 3. In more details, the percentage values of scene cuts detected and undetected for each sequence and for each trial are reported. As can be seen in the histogram of Fig. 3, the successful percentage for the sequence History reaches the 100% in the second trial. For the sequence Ontario and Toronto the number of the detected shots increases, but it does not reach the 100%. Both Ontario and Toronto sequences are characterised by a relevant number of shots shorter than 20 frames. Not all these shots can be detected with the current implementation of the filtering process. But, if the shots longer than 2 patterns are

290

Edoardo Ardizzone et al.

only considered, for both sequence the successful percentage will be 100% and 97% for Ontario and Toronto respectively. The oversampling, as a percentage of real number of scene cuts, is below 30 % in the first set of trials and increases in the second one up to 27%, 86% and 350% for Toronto, Ontario and History respectively. Adopting a more accurate filtering process in order to increase the algorithm resolution, the results in terms of successful percentage and reduced oversampling would be improved on.

References 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16.

D. Le Gall, “MPEG-1: a Video Compression Standard for Multimedia Applications”, Comm.of the ACM, April 1991, Vol. 34, No. 4. M. La Cascia, E. Ardizzone, “JACOB: Just a Content-Based Query System for Video Databases”, Proc. ICASSP-96, May 7-10, Atlanta, GA. E. Ardizzone, M. La Cascia, “Automatic Video Database Indexing and Retrieval”, Multimedia Tools and Applications, 4, pp. 29-56, Kluwer, 1997.. V. N. Guditava and V.V. Raghavan, “Content-Based Image Retrieval Systems”, IEEE Comp., Sept. 1995. M. Flickner, H. Sawhney, W. Niblack, J. Ashley, Q. Huang, B. Dom, M. Gorkani, D. Lee, D. Petkovic, D. Steele, P. Yanker, “Query by Image and Video Content: The QBIC System”, IEEE Comp. Sept. 1995. D. Lee, R. Barber, W. Niblack, M. Flickner, J. Hafner, D. Petkovic, “Query by Image Content Using Multiple Objects and Multiple Feature: User Interfaces Issues”, Proc. of ICIP 1994. D. Lee, R. Barber, W. Niblack, M. Flickner, J. Hafner, D. Petkovic, “Indexing for Complex Queries on a Query-By-Content Image Database”, International Conference on Pattern Recognition 1994, volume 1, pages 142-146. A. Nagasaka and Y. Tanaka, “Automatic Video Indexing and Full-motion Search for Object Appearence”, in Proc. IFIP TC2/WG2.6 Second Working Conference on Visual Database Systems, Sept. 30-Oct. 3, 1991, pp. 113-127. V.E. Ogle and M. Stonebraker, “Chabot: Retrieval from a Relational Database of Images”, IEEE Comp. Sept. 1995. P. M. Kelly, M. Cannon, D. R. Hush, “Query by Image Example: The CANDID Approach”, Proc. of SPIE – Storage and Retrieval for Image and Video Databaase III, 1995. A.Pentland, R. W. Picard, S. Sclaroff, “Photobook: Content-Based Manipulation of Image Databases”, SPIE Storage and Retrieval Image and Video Databases II, No. 2185, Feb 610, 1994, San Jose. A. L. Yeo and B.Liu, “Rapid Scene Analysis on Compressed Video”, IEEE Transaction on Circuits and Systems for Video Technology, vol. 5, no. 6, Dec. 1995. Boon-Lock Yeo, Bede Liu, “On The Extraction of DC Sequences from MPEG-1 Compressed Video”, Proc. of International Conference on Image Processing, October 1995. J. Meng and S.-F Chang, “Tools for Compressed-Domain Video Indexing and Editing”, SPIE Conference on Storage and Retrieval for Image and Video Database, Vol. 2670, San Jose, CA, Feb. 1996. J. Meng, Y. Juan and S. F. Chang, “Scene Change Detection in a MPEG-1 Compressed Video Sequence”, Digital Video Compression: Algorithms and Technol., vol. SPIE -2419, pp. 14-25, Feb. 1995. E. Ardizzone, M. La Cascia, A. Avanzato and A. Bruna, “Video Indexing Using MPEG-1 Motion Compensation Vectors”, submitted to IEEE ICMCS99.

Detecting Abrupt Scene Change Using Neural Network∗ H.B. Lu and Y.J. Zhang Department of Electronic Engineering Tsinghua University, Beijing 100084, China

Abstract: A real-time algorithm is proposed for the detection of abrupt scene changes, which makes use of a dual (one big and one small) window and single-side checking to avoid the false detection and miss detection caused by the violent motion of camera and/or large objects. In addition, a multi-layer perceptron is used to solve the problem of parameter determination in the proposed algorithm. The performance of our algorithm has been experimentally compared with that of some typical methods by using real video sequences. The recall rate is greatly improved while keeping high precision rate.

1.

Introduction

Digital video is a significant component of multimedia information systems, and the most demanding in terms of storage and transmission requirements. Content-based temporal sampling of video sequence is an efficient method for representing the visual information by using only a small subset of the video frames. These frames are obtained by so called video segmentation techniques. Through temporal segmentation, input video streams are decomposed into their fundamental units— shots and then representative frames called key frames can be extracted. Here we define shot as a continuous sequence of frames from one camera operation. Each shot usually contains closely related visual contents. There are many special video effects that have been used in video productions, some frequently used ones are cut, i.e., abrupt scene change, as well as gradual scene change, such as fade and dissolve. In this paper, we discuss cut detection with high recall and precision rate. Here the definitions of recall rate and precision rate are: correct detection recall rate = correct detection + missed detection

correct detection correct detection + false detection The paper is organised as follows: section 2 discusses some current methods for cut detection. Section 3 presents an efficient method for cut detection, which is based on a novel dual window concept and is implemented by using a multi-layer neural network. Some detection experiments with real video films are presented in section 4 and the result discussions are given in section 5. precision rate =

∗

This work has been supported by NNSF (69672029) and HTP (863-317-9604-05).

Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL'99, LNCS 1614, pp. 291-298, 1999.  Springer-Verlag Berlin Heidelberg 1999

292

2.

H.B. Lu et al.

Related Previous Work

The existing procedures for cut detection often consist of two steps: measuring the disparity according to some metrics and comparing the disparity with a predetermined threshold. Many metrics have been reported, and most of them fall into three categories: metrics based on histogram comparison, metrics based on first and second order intensity statistics, and metrics based on pixel difference in which the structure of images is considered. Some quantitative comparisons of these metrics can be found in [1,2]. In this section we first present our own evaluation of several related metrics, then make some discussions on threshold selection. 2.1

Previous Metrics for Cut Detection

Scene change detection has been carried out both in compressed and uncompressed domains. The ideas are very similar, i.e. sequentially comparing each video frame with its adjacent ones and marking the point at which a large disparity is detected. How to define metrics to measure the disparity between two frames? In compressed domain, the popular methods first compare consecutive "I" frames to find out possible ranges of scene change, and then use "B" and "P" frames between two consecutive "I" frames to locate the accurate position of scene change [3]. This method does not need to decode compressed stream, so it is very fast. But it is hard to detect gradual scene change by only using the coded information. In uncompressed domain, the major techniques used are based on pixel difference, histogram comparison, edge difference, and motion vector. The simplest method is based on histogram comparison. The observation is that if two frames have similar backgrounds and common objects their histograms will show little difference. Since the grey level histogram represents some global information of frames, the comparison based on histogram is insensitive to small motions and noises. But as the histogram discards the space distribution of grey level, some abrupt scene changes would be missed as two frames with different objects may have similar histograms. Another popular method for scene change detection in uncompressed domain is based on cumulating pixel difference, i.e. comparing each pixel in one frame with its corresponding pixel in the next frame, and summing up the difference over the whole frame. If the total difference value is bigger than a pre-determined threshold, then an abrupt scene change is declared. The major problem with this method is that it is very sensitive to camera and object motion. One possible solution is to smooth the frame first, so that each pixel has, for instance, the mean value of its 8 nearest neighbours. This approach would also filter out some noise in the frame, but it could only compensate for minor camera and/or object motion. 2.2

Threshold Selection

To determine a cut position, a disparity threshold should be set. In [4], cuts are identified when the histogram differences are beyond 5 or 6 standard deviations from the mean value. However, when some violent motions occur inside a shot, many consecutive frames will be identified as cuts. In [5], a technique called "sliding window method (SWM)" is proposed. Let Xi, i = 1, 2, …, N be a sequence of DC images, the difference sequence Di, i = 1, 2, …, N–1 are formed using the formulation below:

Detecting Abrupt Scene Change Using Neural Network

Di = D (i, i + 1) =

å | I i ( x, y ) − I i +1 ( x, y ) |

293

(1 )

x, y

where Ii(x, y) is the intensity of pixel (x, y) in frame-i. To detect scene change from the difference sequence Di, a sliding window of size°2m–1 is defined, and the frame to be detected is placed in the middle of this window. A scene change from Xl to Xl+1 is declared if the following two criteria are fulfilled: (1) Dl ≥ Dj, j = l-m+1, …, l–1, l+1, …, l+m–1; (2) Dl ≥ t × Dk where Dk is the second largest maximum inside the sliding window, t is a pre-defined parameter. With the help of local information, the precision and recall rates of this method are thus improved. But we find that three problems are still existing. First, violent intra-shot motions of object or camera around shot boundary can cause a sequence of high peaks near the shot boundary and the criterion (2) can not be satisfied in such a case. For example, a segment of inter-frame difference from a test sequence (see section 4) is shown in Fig.1. There exists a shot between peak a and b, but in practice the cut position b has been missed because the motion. The second problem is related to false declaration. Fig 2 shows another segment of inter-frame difference from a test sequence. The peak a and peak b indicate two cut positions, while c and d are small peaks due to adding and removing movie subtitle. Because there is practically no any motion between peak a and peak b, so peak c and peak d become two significant local peaks which would be identified as cut positions.

Fig.1

Example of missed detection

Fig.2

Example of false detection

The selection of parameter t is the third a problem. A proper parameter is very important for cut detection. In [5], the sensitivity of the parameter t is investigated. The parameters should be selected with a better trade-off between decreasing the missed detection and decreasing the false detection. It went without saying that some cut positions will be missed where the value of maximum over the value of second maximum is just less than the selected t, so criterion (2) can not be satisfied.

294

3.

H.B. Lu et al.

A New Approach for Cut Detection

To overcome the weakness of previous work, a new cut detection approach is designed. Fig. 3 depicts the block diagram. Input video sources can be compressed video, such as MPEG streams, or uncompressed video sequences, such as image sequence captured by image grabber. In order to get actual image sequence for detection, we partially decode MPEG stream to get its DC sequence, while for uncompressed video an 8×8 average is carried out. Both cut and flashlight positions are detected from this sequence. Since flashlights can not be considered as cuts either from the definition of cut or from the content of video [5], we identify flashlights as intra-shot events and discard them to get the final abrupt scene change positions. Compressed Video Stream

Extract DC Image

Uncompressed Video Stream

8×8 Average

Video Sequence

Detect Flashlight

Detect Cut

Fig. 3

Shots

Diagram for cut detection

In the following, we first describe our cut detection approach, which is based on dual-window and single-side checking, and then we use multi-layer perceptron to improve its performance. 3.1

Dual-Window Method for Cut Detection

For judging the performance of video segmentation, recall rate and precision rate are often used. However, there are some conflicts between recall rate and precision rate. Since our objective of video segmentation is for further video browsing and retrieving, we hope that the detection method should first provide high recall rate, and then high precision rate is considered. To this purpose, the pixel difference comparison is selected as one of the important metrics. A dual (one big and one small) window approach is used. The big window is used for selecting the probable cut positions, and the small window that centred at the probable position selected is used for determining the real cut positions. The decision made in the small window is helped by a single-side checking technique in contract to double-side checking technique as described in [5]. To avoid false detection from the single-side checking, we divide the image to four blocks, then compare histogram of the corresponding block in consecutive frames. The whole algorithm can be described by the following steps: (1) Define a disparity metric between frame-j and frame-k: D ( j, k ) =

å f (| I j ( x, y ) − I k ( x, y ) |)

x, y

N

(2 )

where N is the number of pixels in one frame. The function f (.) is defined as follows:

Detecting Abrupt Scene Change Using Neural Network

ìï1 x > T f ( x) = í ïî0 otherwise

295

(3 )

It helps to make a selective statistics to exempt small disparity values. (2) Define a big window of size WB. Let the current detecting frame-l locate in the window. The mean inter-frame difference from this window is calculated. (3) Define a small window of size WS = 2m–1, and let frame-l locate at the centre of this window (4) Let Dl = D(l–1, l). If both of the following criteria (single-side criteria) are satisfied (t1 and t2 are predefined constants): (a) Dl ≥ t1 × mean (b) Dl ≥ t2 × Dleft .OR. Dl ≥ t2 × Dright where Dleft = max(Dj), j = l-m+1, …, l–1; Dright = max(Dj), j = l+1, …, l+m–1, then we consider frame-l as a possible cut position (otherwise return to (2) and consider the next frame). (5) For further affirmation, another metric (where hj, hk are the histograms of frame j, k, respectively. The denominator is a normalizing factor) is defined: M

δ ( j, k ) =

å | h j (i ) − hk (i ) |

i =1 M

å{h j (i ) + hk (i )}

(4 )

i =1

If δ (l–1, l) ≥ t3 (t3 is a predefined constant) is also satisfied, frame-l is identified as a cut position. Return to (2). Three points are to be noted here: (1) The big window is used to avoid false detection. Using mean of interframe differences can avoid the false detection caused by SWM as pointed out in section 2, at the same time reduce the number of searching positions greatly. (2) The single-side criteria are used in step (4) to avoid the miss detection caused by violent camera and object motions. Since single window may introduce false positions, we add a criterion δ (l–1, l) ≥ t3. (3) The spatial distribution information is lost in global histogram, so we divide image into n×n blocks, and compute δi, i = 1, 2, …, n×n from the corresponding blocks between consecutive frames, and then use the average δ (l − 1, l ) = å δ i n × n . The problem that different scenes have similar histograms can be amended. 3.2

Thresholding Using Multi-Layer Perceptron (MLP)

The above algorithm solves the problems of global threshold selection and two weakness of SWM. But the parameters t1, t2, t3 are still difficult to select. A possible solution is to observe several video clips, then select a good value of these parameters according to experiments.

296

H.B. Lu et al.

We consider artificial neural network is a suitable way to fit the requirement. The goal of neural-network was to solve problem without explicit programming. The neurons and networks were supposed to learn from examples and store the obtained knowledge in a distributed way among the connection weights. Neural networks are on-line learning systems, intrinsically non-parametric and model-free. Since neuralnetwork classifiers are often able to achieve better reliability than classical statistical or knowledge-based structural recognition methods through their adaptive capability [6], we use multi-layer perceptron (MLP) here. The practical MLP we used has three full-connected layers, as a classifier to identify whether the current position is a cut position or not. The structure of MLP is shown in Fig.4. Our focus here is how to extract features to form input vector, since efficient feature extraction is crucial for reliable classification. According to the analysis in section 3.1, four features are extracted: inter-frame difference on frame-l using metric (2); the ratio of Dl over Dleft; the ratio of Dl over Dright; and δ (l–1, l), i.e. input layer has four neurons. We define the input vector I = [I1, I2, I3, I4] as follows: I1 = Dl , I 2 = Dl / Dleft , I 3 = Dl / Dright , I 4 = δ (l − 1, l ) =

1 n×n åδi n × n i =0

(5 )

This input vector forms a 4-D space. The values of Ii are all much bigger on cut positions than those on non-cut positions, so it is easy to achieve robust classification by using MLP in this 4-D space. In order to classify the input frame-l represented by the input vector into two classes, we define an output vector O = [O1, O2]. If O1 < O2, then frame-l is declared as a cut position, otherwise a non-cut position. We calculate the number of neurons in hidden layer using formulation: n h = ni + no + k , where k is a constant between 1~10. As here ni = 4, n0 = 2, we

obtain nh = 4 ~ 13. We choose nh = 10 for more robust system. O1

Output layer

Hidden layer

1

Input layer

2

I1

Fig. 4

3

O2

..........

4

I2

I3

nh

I4

Structure of MLP

In our MLP, the input neurons use the linear functions, and the neurons in hidden and output layer use the sigmoid functions. Modified back-propagation training algorithm is applied here [7]. Instead of minimising the squares of the differences between the actual and target values summed over the output units and all cases, the following error function is to be minimised:

Detecting Abrupt Scene Change Using Neural Network

M

E = −å

N

å [Tmn ln On + (1 − Tmn ) ln(1 − Omn )]

297

(6 )

m =1 n =1

where m runs over cases, M is the total number of cases, N is the total number of output units, Omn is the actual value (between 0 and 1) of output unit n, and Tmn is target value of output unit n. The coupling strengths wji are updated according to the following rule: M

∆w ji ( s + 1) = −η å ( m =1

∂E ) + α∆w ji ( s ) ∂w ji

(7 )

where s represents the sweep number, m runs over cases, M is the total number of cases, η is the learning rate, α is momentum factor. In order to assure that all coupling strengths are changed by the iterative learning procedure, they have to be initialised with small values (random numbers ranging from –0.03 to +0.03 in our case). In real implementation, since t1 and t2 in step (4) of section 3.1 are larger than 1, so those positions that don't satisfy the criterion "Dl > mean .AND. (Dl > Dleft .OR. Dl > Dright)" are not cut positions, and can be first filtered out to reduce the searching time.

4.

Experiment Results

In order to compare our proposed algorithms with SWM, four representative video clips have been chosen for test. One is from "Four Wedding One Funeral", there exists static shot, flashlight, and many camera operations; one is about air battles from "Top Gun", in which both camera and object motions are very violent; one is from "Fifth Element", there are a number of big object motions; and one is selected from cartoon to test the algorithm performance under fast scene change. First, we compare DWM with SWM. We choose parameters as: WB = 500, m = 12, t1 = 1.2, t2 = 2, t3 = 0.3 for DWM, m = 10, t = 2 for SWM. The results for the four test video clips are listed in Table 1 and Table 2, respectively. From these tables, it is clear that the recall rate of DWM is much higher than that of SWM. Mainly benefited from dual windows, DWM avoids many miss detections caused by the violent motions, The precision rate is slightly improved because small local maximums can be discarded. But as SWM, DWM also has the problem of selecting proper parameters. As we can see in video clip from "Fifth Element", SWM missed 18 positions while DWM still misses 13 cut positions. Then we test DWM-MLP method. Here WB = 500, m = 12 are chosen for feature extraction. Another video clip from "Airforce No.1" is used to train the MLP, since some representative types of cuts exist in this clip. The train algorithm has converged after ten epochs. The above mentioned four video clips are used to test the MLP and the results are listed in Table.3. As we expected, MLP here is very robust, 98.4% recall rate and 96.6% precision rate are achieved for these test clips. It is worthy to mention that by using DWM-MLP, only 1 cut position is missed in video clip of "Fifth Element".

298

H.B. Lu et al. Table 1.

Test results obtained by using SWM (m = 10, t = 2)

Video Clip Frames Total Cuts Correct Missed False Recall Precision Four Wedding One Funeral 3434 18 16 2 2 89% 89% Top Gun 1602 36 28 8 3 78% 90% Fifth Element 2674 61 43 18 0 70% 100% Cartoon 1402 12 10 2 0 83% 100% Totals 9112 127 97 30 5 76.4% 95% Table 2.

Test results obtained by using DWM (WB = 500, m = 12, t1 = 1.2, t2 = 2, t3 = 0.3)

Video Clip Frames Total Cuts Correct Missed False Recall Precision Four Wedding One Funeral 3434 18 18 0 0 100% 100% Top Gun 1602 36 36 0 3 100% 92% Fifth Element 2674 61 48 13 0 79% 100% Cartoon 1402 12 12 0 1 100% 92% Totals 9112 127 114 13 4 89.8% 96.6% Table 3.

Test results obtained by using DWM-MLP (WB = 500, m = 12)

Video Clip Frames Total Cuts Correct Missed False Recall Precision Four Wedding One Funeral 3434 18 18 0 0 100% 100% Top Gun 1602 36 36 0 3 100% 92% Fifth Element 2674 61 60 1 0 98% 100% Cartoon 1402 12 11 1 1 100% 92% Totals 9112 127 125 2 4 98.4% 96.6%

5.

Discussion

A real-time method for robust cut detection in uncompressed as well as compressed video is proposed. The main features of this approach are dual-window and single side checking which are used to select probable positions of cut. To achieve a robust classification, a multi-layer perceptron is applied. Our algorithms can effectively avoid the false detection and miss detection caused by the violent motion of camera and/or large objects.

References 1. 2. 3. 4. 5. 6. 7.

J.S. Boreczky and L.A.Rowe, SPIE, V.2664: 170-179, 1996. R.M. Ford et al., Proceedings of IEEE ICMCS, 610-611, 1997. V.Kobla and D. Doermann, SPIE, V.3022: 200-211, 1997 H.J.Zhang et al., Multimedia Systems, V.1: 10-28, 1993. B.L. Yeo and B. Liu, IEEE Trans. CSVT-5: 533-544, 1995. K.Fukushima, Neural Network, V.1: 119-130, 1988. A.Van Ooyen and B.Nienhuis, Neural Network, V.5: 465-471, 1992

G

Multi-modal Feature-Map: An Approach to Represent Digital Video Sequences1 Uma Srinivasan and Craig Lindley CSIRO Mathematical and Information Sciences Locked Bag 17, North Ryde NSW 1670, Australia Building E6B, Macqaurie University Campus, North Ryde NSW Phone: 61 2 9325 3148, Fax: 61 2 93253200 {Uma.Srinivasan,Craig.Lindley}@cmis.csiro.au

Abstract. Video sequences retrieved from a database need to be presented in a compact, meaningful way in order to enable users to understand and visualise the contents presented. In this paper we propose a visual representation that exploits the multi-modal content of video sequences by representing retrieved video sequences with a set of multi-modal feature-maps arranged in a temporal order. The feature-map is a ‘collage’ represented as a visual icon that shows: the perceptual content such as a key-frame image, the cinematic content such as the type of camera work, some auditory content that represents the type of auditory information present in the sequence, temporal information that shows the duration of the sequence and its offset within the video.

1

Introduction

Video sequences retrieved from a database need to be presented in a compact, meaningful way in order to enable users to understand and visualise the contents presented. Currently most approaches to visualisation [6,10,13,14] and presentation of video sequences deal with one modality at a time [2,3]. While the above approaches offer different ways of presenting visual summaries of videos, the basic information presented represents only the visual content in videos. One piece of work that uses both audio and visual content is described in [7]. In this paper we propose a visual representation where video sequences are represented with a set of multi-modal feature-maps arranged in a temporal order. The feature-map is a ‘collage’ represented as a visual icon that shows the following: (i) perceptual content such as a key-frame image, (ii) cinematic content such as the type of camera work, (iii) some auditory content that represents the type of auditory information present in the sequence, (iv) temporal information that shows the duration of the sequence and its offset within the 1

The authors wish to acknowledge that this work was carried out within the Cooperative Research Centre for Advanced Computational Systems established under the Australian Government's Cooperative Research Centres Program.

Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL'99, LNCS 1614, pp. 299-306, 1999.  Springer-Verlag Berlin Heidelberg 1999

300

Uma Srinivasan and Craig Lindley

video, and (v) some indication of sematic associations represented in the video sequence. As each feature-map represents one video sequence, it also serves as a visual index to the retrieved video sequences. The multi-modal feature-maps arranged in a temporal order serve as a browsing tool that can conserve and postpone the utilisation of the entire bandwidth of limited network resources.

2

What is a Feature-Map?

A feature-map is a visual representation of audio and visual contents of a video sequence and its function is to facilitate the visualisation of digital content at a semantic level. Information represented in a feature-map may be available as annotated descriptions of audio and visual content, which are generated either manually or semi-automatically. While the idea of feature-map is independent of the level of automation, in this paper we focus on representing those features that can be automatically detected, annotated and stored in a video database. Model of Digital Video Features Most models of video are based on a temporal organisation where a video is described at various levels such as frame, shot, and scene. We use such a model to get the keyframe and temporal component of the feature-map. However, to incorporate the feature-based components within the feature-map, we have taken a different approach. We have developed a feature model that is based on the general research direction and is shown in Figure 1. This model forms the basis of identifying features that can be represented in the feature-map. Video features

Auditory

Visual

Speech

Camera-operation Object-based based

Silence

MonologueDialogue

Pan

Zoom

Zoom-in Pan-left

Pan-right

Tilt

MovingStationary object object Male voice Female voice

Music

Instrumental

Other sounds

vocal

Bird sounds

ExplosionCrowd Noise

Zoom-out foreground background

Fig. 1. Classification of Audio and Video features in Digital Videos

Multi-modal Feature-Map

301

Feature extraction has been studied from different perspectives. On the visual side, some groups have studied camera-motion detection [1], while others have studied object-motion-based features [5]. The audio analysis groups have largely focussed on analysing audio content in the digital videos [9],[10]. Our focus here is to use existing feature extraction techniques and also allow the model to evolve, as more featureextraction research results become available2. In order to represent the features shown in the model, we need an unambiguous way to represent and combine features that can occur at multiple levels. For example, ‘silence’ can occur at any of the levels; as silence may occur within a piece of music or during a conversation. The visual representation has to address this issue of granularity in a meaningful way. Visual Representation of Features As the feature-map is envisaged as a visual representation of features present in a video sequence, we propose associating each feature with an icon that is a symbolic representation of that feature. The symbols should also be chosen such that they are unique and do not interfere with the actual key-frame image of an object. This calls for a visual lexicon and a visual grammar that offers a systematic approach to presenting these visual icons such that they convey objective information. Visual Lexicon The visual lexicon has to represent multiple modalities such as audio and visual features in an unambiguous way. Initially we restrict the lexicon to only those features represented in the feature model shown in Figure1. We expect the model and the lexicon to grow as it becomes possible to detect more features in digital videos. A feature-map should meet the following requirements: (i) it should summarise the automatically detected features (ii) it should be easy to develop /learn, (iii) it should minimise redundancy, (iv) it should be easier and quicker to understand than textual annotation. An important criterion used while designing the icons has been that it should be possible to combine them in such a way, that the resulting symbols unambiguously convey the meaning of multiple features available in the video sequence. In summary, the philosophy here is that ‘a picture is worth a thousand words. (As no industry standard icons were available to us, we have come up with the visual symbols shown in Tables 1 and 2. The feature-map idea can be used by substituting these icons with industry-specific symbols, should they become available.) Table 1 shows some audio features represented using visual icons. The shaded symbols indicate the presence of sound. The unshaded symbols represent silence within that feature. For example, under music, we could have some silence within a musical event. Table 2A shows the visual representation of features that represent camera-motion, and Table 2B shows a visual representation of features that represent object motion. 2

With digital cameras and MPEG4 and MPEG7 compression schemes the list of detectable features is likely to change rapidly.

302

Uma Srinivasan and Craig Lindley

Contrasting colours shows foreground and background objects. These feature-based icons form part of a feature map to indicate the nature of contents present in the retrieved video sequences. Table 1. Representation of Audio Events

Audio Feature

Visual Representation

Audio Feature

Dialogue

Music

Monologue

Music - Male

Explosion

Music-female

Table 2A. Camera-motion features Camera Visual Operation Representation Pan – left

Visual Representation

Table 2B. Object-motion features Object-based Visual Features Representation Moving object

Pan-right

Foreground object

Zoom-in

Background object

Zoom-out Tilt-up Tilt-down

When designing iconic systems, it is desirable to have some rules to construct the image so that it conveys the same meaning to all users. This calls for a visual grammar to specify some simple composition rules. Visual Grammar Sentences in visual languages are assemblies of pictorial objects (or icons) with spatial relationships. For a continuous media such as video, the temporal aspects also need to be represented. For the purposes of constructing a feature-map, the visual grammar has to specify rules to address the following: (i) features that need to be represented in a feature-map, and what constitutes a valid feature-map, (ii)

Multi-modal Feature-Map

303

composition rules for the layout of feature-based icons, key-frame images and temporal information associated with the features that need to be represented In order to address the above issues, we have organised the grammar rules into 2 groups: feature selection rules and composition rules. At this stage we have just enumerated the criteria for these rules. Developing a more rigorous syntax will be part of our on-going work. Feature selection criteria (i) Auditory and visual features specified in the query will be represented in the feature-map, through associated feature-icons. (ii) Features available in the retrieved video sequences (ie, pre-determined, annotated and stored in the database) will also be represented. (iii) In case of overlapping features, only the most dominant feature will be displayed in the feature-map. (This is also due to the fact, it becomes increasingly difficult to detect overlapping features and usually only the dominant one can be detected easily.) Composition criteria Feature-maps have both temporal and spatial components that need to be represented. In addition, the composition rules have to accommodate multiple modalities available in the video sequences. (i) Each retrieved video sequence will have a representative key-frame. In the most simplistic case this could be the first frame of the sequence. (ii) The spatial arrangement of features (specified and/or extracted) will be based on their temporal ordering. (That is if there are multiple features within a video sequence, they would be presented in the order of their appearance.) (iii) Temporal information such as start-offset and duration of the sequence will be represented. (iv) The above 3 components will form part of the feature-map that represents the returned video sequence. (v) The feature-maps will be placed in a temporal order along the basic time-line of the video User Interaction User interaction involves query formulation and display of returned sequences. The user interface should support query formulation at three levels. (i) At a purely semantic level; which from our discussions with people managing archives, is often the preferred level for a general user of a digital video library. (ii) A combination of semantics and feature levels; which may be preferred by more informed users such as television archivists and librarians, (iii) At the level of features; which could be useful for advanced users such as sound editors, film directors, etc. This requirement calls for an appropriate mapping between the semantic and feature levels. Figure 2 shows the proposed mapping scheme. In order to formulate a query, the general user, ie, the first category shown above, would use information related to the video domain and the application domain, (ie the two left ovals). The second category of user would use a combination of concepts and features (middle and right oval), and the third category of user would use the features directly.

304

Uma Srinivasan and Craig Lindley Application Domain Concepts

Video Domain

Content-based Features

game Sports

Object motion Camera panning

Player movements

Commentator’s views Conversation

Documentary

Dialogue

Commentator’s views News

Monologue

Music

Main news

Loud sounds

Commercial l breaks

Fig. 2. Mapping Semantics to Content-based Features Allowing users to map out the concepts-features relationship provides us with some rich semantics that are often difficult to capture and model. As the feature-map represents all the features that are related to a concept as perceived by a user, it provides a good visualisation of video content that is unique to the user specifying the query. Figure 3 shows a set of feature-maps returned from a query to retrieve sequences about the Australian election campaign. The key-frames shown in this example are the first frames of returned sequences. The first image shows that the sequence represented by this image has a camera tilt operation, followed by a zoom-in, followed by some speech. The key frame shows the Australian symbol. The start time and the duration of the sequence is shown as part of the image. The second image shows that there is music followed by a zoom-in followed by a speech. The image (insides of a piano) in the key-frame indicates that the associated music. The third image shows there is a zoom-in operation followed by some speech, which is followed by a crowd cheer and a moving object.

03:20 05:10

00.25 02:30

00:50

06:30 08:15

03:30

)6:30 Video time line

Fig. 3. Mapping Semantics to Content-based Features

Multi-modal Feature-Map

305

The picture of John Howard in th e key frame combined with the features gives a reasonable understanding of the contents of that sequence. Feature-Map Construction The framework we have developed as part of the FRAMES [13] project provides a supporting environment to conduct some experiments in generating feature-maps as described in this paper. Figure 4 shows the query processor component of that framework. Display

Client Front-end application

Query

Display M anager

Feature Map Builder Key frame images Auditory features

Concept-feature associations Query Processor SQL queries

Visual features Temporal offset

Video stream

Video Server

video sequence reference

Application schema Database Server

Application Model

Fig. 4 . Architrecture to generate multi-modal feature-map Information about specific audio and visual events of interest are stored in the database. We have developed our own algorithms to detect audio [10] and video events [12]. The Query Processor translates the queries specified at a semantic level into SQL qureies that relate to video objects and associated features characterised by their temporal and feature-based attributes. The Feature-map Builder collates the auditory and visual features by associating the query and the results returned with appropriate visual images to generate a set of multi-modal feature-maps arranged in a temporal order. The Display Manager links each feature-map with the appropriate video sequence delivered by the video server, and presents a user interface with featuremaps arranged in a temporal order to facilitate browsing through a bunch of video sequences that have been retrieved based on a query condition. An interesting extension to visualising video content would be to enable the play back of only that mode which is chosen from the feature-map. What this means is if the auditory symbol in the feature-map is clicked, the audio content should be played back. This aspect needs further investigation and will form part of our on-going research activity.

306

3

Uma Srinivasan and Craig Lindley

Conclusion

The feature-map presented in this paper enables us to represent some important audio and visual information available in video sequences. The feature-maps provide a form of temporal compression where the loss of information involved is affordable with respect to the function of the feature-map. In the context of presenting information from a digital library of videos, such a representation offers a compact pictorial summary at a semantic level rather than at a purely perceptual level such as displaying colour histograms or audio wave patterns.

References 1. Aigrain, P.; Zhang, H., and Petkovic, D. Content-Based Representation and Retrieval of Visual Media. MULTIMEDIA TOOLS AND APPLICATIONS. 1996a; 3179-202. 2. Arman; Depommier; Hsu, and Chiu. Content-based Browsing of Video Sequences, Proceeedings of ACM international Conference on Multimedia `94; 1994; California. ACM; 1994. 3. Bolle, R; Yeo, B., and Yeung, M. Video Query, Beyond the keywords. IBM Research Report. 1996 Oct. 4. Bolle, Rudd M.; Yeo, Boon-Lock, and Yeung, Minerva M. Video Query and Retrieval1997; 13-23. 5. Chang, S. F.; Chen, W.; Meng, H. J.; Sundaram, H., and Zhong, D. A fully automated content-based video search engine supporting spatiotemporal queries. IEEE transactions on circuits and systems for video technology. 1998 Sep; 8(5):602-615. 6. Jain, R. editor. Communications of the ACM. ACM. Vol. 40, 1997. 7. Lienhart, Rainer; Pfeiffer, Silvia, and Effelsberg Wolfgang. Video Abstracting. Communications of the ACM. 1997 Dec; 40(12). 8. PfeiffeR, S.; Fischer, S., and Effelsberg , W. Automatic Audio Content Analysis . Proceedings of ACM Multimedia, 94; 1994; Boston. 1996. 9. Samouelian; A,.Robert-Ribes, J.and Plumpe, M. Speech, Silence, Music and Noise Classification of TV Broadcast Material. Proc. 5th International Conference on Spoken Language Processing; 1998 Dec; Sydney. 1998 Dec. 10. Smoliar, S. W. and Zhang, H. J. Content-based video indexing and retrieval. IEEE Multimedia. 1994 Summer; 343-350. 11. Srinivasan, U.; Gu, L.; Tsui, K., and Simpson-young, W. G. A Data Model to support Content-based Search on Digital Video Libraries. Australian Computer Journal. 1997 Nov; 29(4):141-147. 12. Srinivasan, U; Lindley, C., and Simpson-Young, W. G. A Multi-Model Framework for Video Information Systems. Semantic Issues in Multimedia Systems, Kluwer Academic Publishers, 85-107. 13. Taniguchi, Y.; Akutsu, A., and Tonomura, Y. PanoramaExcerpts: Extracting and Packaging PAnoramas for Video Browsing. Proc. ACM Multimedia 97; Seattle. 1997 Nov. 14. Yeung, M. M. and Yeo, B. L. Video visualization for compact presentation and fast browsing of pictorial content. IEEE transactions on circuits and systems for video technology. 1997 Oct; 7(5):771-785.

Robust Tracking of Video Objects through Topological Constraint on Homogeneous Motion Ming Liao, Yi Li, Songde Ma, and Hanqing Lu National Laboratory of Pattern Recognition Institute of Auotomation, Chinese Academy of Sciences P.O.Box 2728, Beijing 100080, P.R.China Tel. 86-10-62542971, Fax 86-10-62551993 [email protected]

Abstract. Considering the currently available methods for the motion analysis of video objects, we notice that the topological constraint on homogeneous motion is usually ignored in piecewise methods, or improperly imposed by blocks that do not have physical correspondence. In this paper we address the idea of area-based parametric motion estimation with spatial constraint involved, in order that the semantic segmentation and tracking of non-rigid object can be undertaken in interactive environment, which is the center demand of applications such as MPEG-4/7 or content-based video retrieval. The estimation of global motion and occlusion can also be computed through the tracking of background areas. Besides, based on the proposed hierarchical robust framework, the accurate motion parameters between correspondent areas can be obtained and the computational efficiency is improved remarkably.

1. Introduction The semantic description of the object motion in video sequence has been a continuous research topic of motion analysis, and is becoming a hot point of current research[1]. This is mainly because of its potential application background such as MEPG-4/7 or content-based video retrieval, which will bring considerable market profits. Nevertheless, until now, the desired target is still beyond technical ability, for the various appearances of the objects and the environment seems difficult to be unambiguously described by a formal method, as a result the totally automatic segmentation is impossible. Practically, interaction is inevitable and motion segmentation and tracking of objects becomes the center problem. Current research efforts on this topic [2,6] are generally based on the assumption that, homogeneous motion exists respectively in areas of object appearance, so that local parametric models such as affine model or perspective model can be applied, and the problem is converted to the spatial clustering of parametric motion field. Since the original motion field is basically derived from intensity-based optical flow field [7] or patch based homogeneous motion analysis [6], some robust estimation methods [8] are implemented to resolve the fundamental problems of optical flow computation such as the boundary problem. Conclusively, by trying to describe the object motion from Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL'99, LNCS 1614, pp. 307-316, 1999.  Springer-Verlag Berlin Heidelberg 1999

308

Ming Liao et al.

local details, most of these approaches work in a bottom up style, which has the following shortcomings: 1) The computation of homogeneous motion patches is very sensitive to the parametric motion clustering on optical flow field, as a result trivial patches may be derived, and the semantic description of objects motion is impossible. The main reason of this problem is that, not only data conservation but also topological homogeneity such as connectivity, continuity and neighborhood is maintained in the motion of physical objects, but no spatial topological constraints are imposed on the clustering in general approaches. This is also the case of some top down methods [9]. 2) Although robust methods can be applied, intensity-based optical flow computation is still unstable to noise, boundary, large displacement, etc. 3) Global motion, i.e., the motion of the camera, if not considered, will affect the local motion and violate the assumption of homogeneous motion. On the contrary, intensity-based estimation of global motion is also unstable and sensitive to local motion. 4) Efficiency of piecewise computation to motion field is another problem. To resolve it, block based methods [6] are proposed. However, these blocks don't correspond to any physical areas in image surface, therefor is harmful to the precision. To deal with these problems, we propose a hierarchical robust framework in which a perspective motion based multi-scale splitting, merging and compensation of spatial areas are applied to impose a topological constraint in the computation. The reason of selecting perspective model is that, for top down analysis, large areas may be considered, while the depth invariance assumption of affine model can not be satisfied. This is not the case of bottom up analysis. The center idea of our method is that, motion of object, esp. its parametric representation, is smooth up to a precision threshold and sampling frequency, therefor it is possible to predict and track the integral motion of objects if comparatively accurate description is obtained by interaction at first step. Violation to the prediction can be re-computed from top down through motion based multi-scale area splitting, merging and compensation, so that multi-motion and occlusion can also be coped with, at the same time both precision and efficiency are ensured. After prediction, component of large motion is obtained in optical flow computation and only residual needs to be calculated, therefor the small motion requirement of parametric optimization is satisfied. After all, the estimation of global motion is undertaken by background areas tracking and motion clustering. This paper is organized as follows: section 2 discusses the area based robust estimation of perspective motion and its Kalman prediction. Section 3 discusses the region growing based watershed transformation on multiple features, as well as the established scale space. Section 4 presents our hierarchical segmentation framework for motion based area splitting, merging and compensation. Section 5 shows some experimental results and section 6 concludes the paper.

Robust Tracking of Video Objects through Topological Constraint

309

2. Area-Based Perspective Motion: Robust Estimation and Prediction For a rigid area A moving along a plane far from the camera, its motion can be represented by the perspective model with θ as the motion parameter, i.e.;

1 x y 0 0 0 x 2 u( p( x , y ) ∈ A, θ ) =&  0 0 0 1 x y xy

xy  θ = Mθ 2 y 

(1 )

It can be proved that, after this transformation, the topological attributes of areas such as continuity, connectivity and neighborhood can be reserved. The motion parameter θ of area A between image pair I t and I t +1 can be estimated by

arg min θ

∑ E ( I t ( p ) − I t +1 ( u( p, θ )))

p ( x , y ) ∈A u( p ,θ ) ∈I t + 1

(2 )

where E is an energy function. For the classical LST energy function, it is based on the normal distribution assumption of data points, which is not the general case of motion estimation. Although the problem can be resolved efficiently by continuation methods such as SOR (Simultaneous Over-Relaxation) or Newton-Gaussian method because of the convexity of LST, it is very sensitive to the outliers, i.e., data points that should not be considered. This is because, data points far from the true solution, as we can't classified them as outliers in the beginning of computation, contribute much more than correct data points to the energy. To resolve this problem, the robust estimator such as the truncated quadratic, Geman & Mclure, Lorentzian, etc., is proposed [8]. However, as these estimators are not convex, GNC (deterministic graduated non-convexity) or stochastic optimization such as simulated annealing or genetic algorithm have to be applied, which lost the efficiency, and very sensitive to the initial solution. So a convex robust estimator is important. In our method the convex estimator is improved from [10] originally defined as (see Fig. 1(a))

 η 2 σ 2 η ≤σ  η σ − 1 otherwise

ρ σ (η ) = 

(3)

When implemented technically, there are a second zero point of energy in (3). This zero point, although doesn't exist theoretically, does exist in computation, which will bring much trouble. The infinite energy of the outlier at infinite distance is another problem. To overcome them, at the same time reserve the convexity, we modify it as (See Fig, 1(b))

 η2 σ 2 η ≤σ ρσ (η) =  n−1 n 2 − 1 2 + ( η − n σ ) ( σ ) nσ < η ≤ (n+1)σ, n ∈Z + 2 

(4)

310

Ming Liao et al.

Fig. 1 (a). Robust Estimator (3) and its

Fig. 1(b). Robust Estimator (4) and its

Gradient.

Gradient.

Since estimator (4) converges to 2 at infinite point and keeps convex, (2) can be resolved by continuation method, nevertheless (2) is still not a convex problem. This is because, the accumulation area, i.e., A(θ ) , is θ -related. For an infinite perspective plane, A(θ ) is the whole plane and (2.1) is convex. But for a bounded area [M, N], its convexity is determined by the status of the two intensity surfaces. This is also the case of the standard optical flow equation. As a result, simple SOR can only obtain the local optimal solution nearest to the zero point. This is why the assumption of small and local homogeneous motion must be imposed, and the global optimal solution can be found only if the initial solution of the iteration is properly given. This initial solution can be predicted by traditional estimation method such as the Kalman motion filter. Generally, variance of zooming does not occur usually, and as the planar assumption requires, the distance of the object from the camera should be very large, so variance in depth can also be ignored. Therefor only three components need to be estimated, i.e., rotation angle α and 2 translations dx, dy along axis, which can be assumed independent to each other. For area At ∈ I t with motion parameter

θ = ( a1 , a 2 , a 3 , a 4 , a 5 , a 6 , a 7 , a 8 ) T , we have  α   arctg ( a 2 2 a 1 − a 4 2 a 5 )      a3   dx  =      a6   dy  

(5)

After applying a linear estimation filter Γ such as the α − β − λ filter [11], suppose their predicted value is (α ′ , dx ′ , dy ′ ) T , the predicted θ ′ between I t and I t +1 can be easily calculated. Using θ ′ as the initial solution to resolve (2), the SOR algorithm generally converges to a global or rational solution.

Robust Tracking of Video Objects through Topological Constraint

311

3. Region Growing Based Watershed Transformation on Multiple Features Intensity-based watershed transformation [12] is extensively implemented for image segmentation as an area feature extractor, and fast algorithm is also proposed. To improve the correspondence between the watershed areas and physical structure of image, other quantifiable features such as gradient, texture, etc., are introduced [4,13]. Furthermore, when considering motion analysis, motion parameters can also be involved[14]. At this time, the region growing process of catchment basin computation is based on the area similarity, which linearly synthesizes all these features, i.e., N

Sim( A1 , A2 ) = ∑ α i ⋅ dist i ( Γ i ( A1 ), Γ i ( A2 ))

(6)

i =1

where Γi is an feature filter; dist i is the distance operator with respective dimension and α i is the correspondent weighting coefficients. Conjugate areas A1 and A2 will be merged if their similarity is above a threshold T. For computational purpose, a multiscale framework is desirable. To achieve this, the merging threshold T can be designed up to a scale factor, i.e., T is determined by the so-called scale of region growing and large scale will derive a large merging threshold. In this way less areas will survive in larger scales and a strict scale space without any boundary deviation is established. In our experiments, primary area partition is based on morphological gradient. In the following region growing steps, besides the mean and variance of intensity, position of mass center of conjugate primary areas are taken into account. When doing motion based area merging as mentioned in next section, distance between perspective motion parameters of conjugate areas is also considered. As we have explained previously, totally automatic segmentation of objects from image is beyond currently available technology. As a result interactively marking of objects is necessary. Here we apply the top down marking from the segmentation of the multiple features based watershed transformation. Objects are marked firstly in large scales, and the rest parts are marked in smaller scale, until mask image with enough precision is arrived. After marking, a hybrid scale representation of objects is obtained, which is the basis of the objects tracking. The first prediction to the motion of the object mask is assigned to be zero for the α − β − λ filter.

4. Hierarchical Segmentation of Areas with Homogeneous Motion Our framework of hierarchical segmentation of areas with homogeneous motion is composed of three stages, i.e., splitting, merging and compensation of uncertain areas. Fig 2 gives an illustration to the framework. Following we will discuss them sequentially.

312

Ming Liao et al. Image1

Image2

....

Image(n)

Multiple Features Based Region Growing Areas in Scale1

Areas in Scale2

......

Areas in Scale(n)

Areas in Scale1

Areas in Scale2

......

Areas in Scale(n)

Interactive Marking

Objects Mask(1)

Prediction

Homogeneous Motion Estimation

Threshold1

Global Motion

Splitting

No

Validate

for next image Threshold2

Threshold3

Merging

for next image

for next Image

Compensation

Objects Mask(2)

Fig. 2. Illustration of Our Hierarchical Framework For a connected object mask, or a connected background area, if homogeneous motion is assigned and possible motion is predicted, the perspective parameters can be calculated robustly according to section 2. As the homogeneous motion assumption may not be correct, large energy higher than a threshold will derive in the computation. This time the splitting of the area should be undertaken. Since these connected areas are composed of catchment basins in hybrid scales as addressed in section 3, the splitting can be accomplished by reducing the maximum scale of the component subareas. This splitting process is continued until the required energy threshold is satisfied. After the splitting stage, the initial motion of each object area, as well as the global motion which is the dominant motion of unmarked background areas, is obtained, and the motion field of the whole image is estimated. Notice that discontinuity motion at boundary is implicitly resolved, which is difficult to piecewise methods. At this time the area merging can be processed by the multiple features based region growing with the motion involved in the calculation of area similarity as addressed in section 3, and larger areas with re-computed homogeneous motion are derived. The similarity threshold is fixed to a global value, and motion of multiple objects as well as the global motion is determined in I t . Since conjugated areas may have different motion, the warped image of I t using the obtained motion field may not cover the whole area of I t +1 . Those uncovered areas are named as uncertain areas. Their motion is determined by the followed compensation stage. Simply said, for those uncertain areas, we try to combine them to one of their conjugated certain areas. The combination is determined by the drawback analysis, i.e. if an uncertain area is assigned to the motion of one of its

Robust Tracking of Video Objects through Topological Constraint

313

conjugated certain area, its correspondent area in I t can be determined by inverse warping of the assigned motion, and the similarity can be calculated. The motion, which produces maximum similarity above a specified threshold, is finally assigned to the concerned uncertain area. After an iterative compensation and combination, every certain area will grow to a limitation. For those uncertain areas whose maximum similarity is still below the threshold, it is classified as a newly appeared area, which means it is occluded in I t . In this way the occlusion is resolved. Their initial motion is assigned to the global motion. This idea is firstly proposed in [14]. In our paper, it works on areas but not points. After these three stages, the final motion of background and each object is obtained and inputted into the α − β − λ filter to predict their motion in I t + 2 . Because our framework totally works on connected and conjugated areas, and in a top down manner, the topological constraint is inclusively imposed, and violation to topological integrity is avoided, as a result the efficiency and the reliability are remarkably improved.

(a)

(d)

(b)

(e)

(c)

(f)

(g)

Fig. 3. Sequence taken from lab: (a) Previous frame. (b) Current frame. (c) next frame. (d) Spatial-temporally segmented and tracked result of Previous frame, different gray lever denotes regions with different motion.(e) spatially segmented result of Current frame, due to assumption of unified motion in arm region is violated, the arm region is spitted to three regions in finer scale.(f) Spatial-temporal segmented result of Current frame after region merging. (g) Spatial-temporally segmented and tracked result of next frame.

314

Ming Liao et al.

5. Experimental Results In this section, experimental result of two sequences is given. One is a sequence taken from lab, another is the foreman sequence. Fig. 3 show a example of region spitting and merging. We can see that from previous frame to current frame ,the arm region undergo a unified motion and can be tracked together. But from current one to next frame, the arm regions undergo two different motion, so our assumption of unified motion in arm region is violated and we have to spit it as seen in (e).In ( f ) we give the result after region merging. As for Foreman sequence, we give our tracking result of the frame 181,184,187. Please note that there exist large cameral motion, in our method, the region tracking and cameral motion estimation are under the unified scheme. In spite of large displacement between frames, our method still can track the foreman pretty well.

(a)

(e)

(b)

(f)

(c)

(g)

Fig. 4. Foreman sequence (a) Frame 181. (b) Frame 184 (c) Frame 187 (e) Tracked result of Frame 181. (f) Tracked result of Frame 184. (g) Tracked result of Frame 187.

Robust Tracking of Video Objects through Topological Constraint

315

6. Conclusion Through the proposed hierarchical robust framework, homogeneous motion field is analyzed in the unit of area from large scales to small, so that topological constraints is implicitly involved in the motion based region growing. A splitting stage, followed by a merging and the compensation stage, is applied in the computation process in order that top down analysis is fulfilled. Compared with the general piecewise or patch based bottom up methods, our methods is of more reliability and efficiency. Since the performance of this framework heavily relies on the quality of area partition, the linear combination style of area similarity calculation, i.e., equation (6), is not satisfactory enough. More rational measurement and new features such as the color information can also be introduced. Optimal region growing as a symbolic problem also needs more consideration. These are our future works.

References [1] Paulo Correia, Fernando Pereira, "The role of analysis in content-based video coding

and indexing" Signal Processing special issue on video sequence segmentation for content-based processing and manipulation, Volume 66, No.2,April 1998. [2] F. Marqués and Cristina Molina." An object tracking technique for content-based functionalities", SPIE Visual Communication and Image Processing (VCIP-97) , volume 3024 pp. 190-198, San Jose, USA,1997. [3] F. Marqués, B. Marcotegui and F. Meyer. "Tracking areas of interest for content-based functionalities in segmentation-based coding schemes". Proc.ICASSP'96, volume II, pages 1224-1227, Atlanta (GA), USA, May 1996. [4] F. Marqués. Temporal stability in sequence segmentation using the wathershed algorithm. In P. Maragos, R. Schafer and M. Butt, editors, Mathematical Morphology and its Applications to Image and Signal Processing, pages 321-328, Atlanta (GA), USA, May 1996. Kluwer Academic Press. [5] D. Zhong and S.-F. Chang, "Spatio-Temporal Video Search Using the Object Based Video Representation," IEEE. Intern. Conf. on Image Processing, invited talk, special session on video technology, Santa Barbara, Oct. 1997. [6] D. Zhong and S.-F. Chang, "Video Object Model and Segmentation for Content-Based Video Indexing," IEEE Intern. Conf. on Circuits and Systems, June, 1997, Hong Kong. (special session on Networked Multimedia Technology & Application) [7] Lothar bergen and Fernand Meyer "Motion Segmentation and Depth ordering Based on Morphological Segmentation" Proc.ECCV, 531-547,1998 [8] M. J. Black and P. Anandan, "The Robust Estimation of Multiple Motions: Parametric and Piecewise-Smooth Flow Fields, Computer Vision and Image Understanding", 63(1), 75-103, 1996 [9] J.R.Bergen,P.J.Burt, R.Hingorani,and S.peleg. "Computing two motions from three frames".Proc.ICCV,pages27-32,December 1990

[10] P. Huber, Robust Statistics, Wiley 1981 [11] Y.Bar-Shalom and T.E. Fortmann. Tracking and Data Association. Academic Press,Inc.1988.

[12] L. Vincent and P. Soille, "Watersheds in Digital Space: An Efficient Algorithm Based on Immersion Simulation", IEEE Transaction on Pattern Analysis and Machine Intelligence, 13(6), 583-598, 1991

316

Ming Liao et al.

[13] M.Pardas and P.Salembier. "3D morphological segmentation and motion estimation for image sequence" EURASIP Signal Processing,38(1):31-43,1994. [14] Jae Gark Choi, Si-Woong Lee and Seong-Dae Kim "Video Segmentation Based on Spatial and Temporal Information"Proc.ICASSP'97,2661-2664,1997.

The Spatial Spreadsheet Glenn S. Iwerks1 and Hanan Samet2
1

2

Computer Science Department, University of Maryland, College Park, Maryland 20742 [email protected] Computer Science Department, Institute for Advanced Computer Studies University of Maryland, College Park, Maryland 20742 [email protected]

Abstract. The power of the spreadsheet can be combined with that of the spatial database to provide a system that is flexible, powerful and easy to use. In this paper we propose the Spatial Spreadsheet as a means to organize large amounts of spatial data, to quickly formulate queries on that data, and to propagate changes in the source data to query results on a large scale. Such a system can be used to organize related queries that not only convey the results of individual queries but also serve as a means of visual comparison of query results. Keywords: spreadsheets, spatial databases, visualization

1

Introduction

In this paper we introduce the Spatial Spreadsheet. The purpose of the Spatial Spreadsheet is to combine the power of a spatial database with that of the spreadsheet. The advantages of a spreadsheet is the ability to organize data, to formulate operations on that data quickly through the use of row and column operations, and to propagate changes in the data through the system. The Spatial Spreadsheet consists of a 2D array of cells containing data. Updates can propagate through the array via cell operations. Operations can be single cell operations, row operations, or column operations. Column operations iterate over rows in a column and row operations iterate over columns in a row. Cell values can be instantiated by the user or can be a result of operations performed on other cells. In the classic spreadsheet paradigm, cell values are primitive data types such as numbers and strings whereas in the Spatial Spreadsheet, cells access database relations. The relation is part of a spatial relational database. A relation is a table of related attributes. A tuple in a relation is one instance of these related items. Each table is made up of a set of tuples [9]. Attributes in a spatial database relation can be primitive types such as numbers and strings or spatial data types such as points, lines and polygons.


The support of the National Science Foundation under Grant IRI-97-12715 is gratefully acknowledged.

Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL’99, LNCS 1614, pp. 317–324, 1999. c Springer-Verlag Berlin Heidelberg 1999


318

Glenn S. Iwerks and Hanan Samet

Spatial attributes stored in the relations associated with each cell can be displayed graphically for visualization of query results. This allows the effects of updates on the base input relations to be observed through the graphical display when changes occur. The rest of this paper is organized as follows. Section 2 gives some background on spreadsheets and spatial databases. Section 3 describes the Spatial Spreadsheet. Section 4 provides some implementation details. Section 5 draws some concluding remarks as well as gives some directions for future research.

2 2.1

Background The Classic Spreadsheet

The classic spreadsheet was designed as an accounting tool. It permitted the user to quickly formulate calculations on the data through column and row operations. It also allowed the user to easily observe how changes in the input data affected a whole series of calculations. The original spreadsheet was laid out in a two-dimensional array of cells in rows and columns. Users could populate the rows and columns with numeric data. They could then perform operations on entire columns (or rows) and populate additional columns with the results. 2.2

Spreadsheet for Images

Spreadsheets for Images (SI) is an application of the concept of a spreadsheet to the image processing domain [6]. In this case, the concept of a spreadsheet is used as a means of data visualization. Each cell in the spreadsheet contains graphical objects such as images and movies. Formulas for processing data can be assigned to cells. These formulas can use the contents of other cells as inputs. This ties the processing of data in the cells together. When a cell is modified, other cells that use it as input are updated. A somewhat related capability is provided by the CANTATA programming language to be used with the KHOROS system [8]. 2.3

SAND Browser

The SAND Browser is a front end for the SAND [2] spatial relational database. The user need only to point and click on a map image to input spatial data used in the processing of query primitives. The results of the queries are then displayed graphically. This gives the user an intuitive interface to the database to help the visualization of the data and the derivation of additional information from it. However, such a system does have limitations. In the SAND Browser one primitive operation is processed at a time. When the user wants to make a new query, the results of the previous operation are lost unless they are saved explicitly in a new relation. As a result, there is no simple and implicit way to generate more complicated queries from the primitives. In presenting the Spatial Spreadsheet we will propose some possible solutions to these limitations of the SAND Browser while still maintaining its ease of use and intuitive nature.

The Spatial Spreadsheet

319

Figure 1: Example query results in top-level window

3

The Spatial Spreadsheet

The Spatial Spreadsheet is a front end to a spatial database. A spatial database is a database in which spatial attributes can be stored. Attributes of a spatial relational database may correspond to spatial and non-spatial data. For example, spatial data types may consist of points, lines, and polygons. Numbers and character strings are examples of non-spatial data. By mapping the world coordinates of the spatial data to a bitmap it may be converted to an image for visualization of the data. The Spatial Spreadsheet provides a means to organize the relational data and query results in a manner that is intuitively meaningful to the user. One may apply meaning to a column, a row, or an entire set of columns or rows to organize data. For example, spatio-temporal data may be organized so that each row corresponds to a different time period and each column corresponds to a different region in the world.

320

Glenn S. Iwerks and Hanan Samet

The Spatial Spreadsheet is made up of a 2D array of cells. Each cell in the spreadsheet can be referenced by the cell’s location (row, column). In the Spatial Spreadsheet, each cell represents a relation. A cell can contain two types of relations: a persistent relation or a query result. A persistent relation is a relation that exists in a permanent state. This is not to say that the data in the relation does not change but rather that the relation existed before the spreadsheet was invoked and will continue to exist after the spreadsheet exits unless explicitly deleted by the user. The second type of a relation contains the result of a query posed by the user. The user decides if a query result will persist or not. The user can pose simple queries. Simple queries are primitive operations. Some examples of a primitive operation are selection, projection, join, spatial join [5], window [1], nearest neighbor [4], etc. Primitive operations are composed to create complex queries. 3.1 Example Let us consider a simple example (see Figure 1). Suppose that we are concerned about flooding in 3 different regions of the world: A, B and C. Roads close to rivers may get washed out when the rivers flood. We want to know what roads in these regions are close to a river at or near flood stage. For each of these regions we have a relation containing all the rivers at or near flood stage. We open these river relations in the first column of our spreadsheet (i.e., column 0). We let row 0 correspond to region A, row 1 to region B, and row 2 to region C. We open relations in column 1 that store position information for roads in each region. Our column operation is to find all the roads in cells in column 1 that are within 500 meters of a river in the cell in column 0 of the same row and store the result in column 2. In a modified version of SQL [9] the query might look as follows. SELECT * FROM Cell(X,1), Cell(X,2), distance(Cell(X,1).river, Cell(X,2).road) d WHERE d < 500 The modification to SQL1 introduced here is the Cell() function. Instead of giving an explicit relation name in the FROM clause, we introduce the Cell() function that takes a row and a column value and returns a relation. The presence of the variable X for the row parameter tells the system to iterate over all open relations in the given columns. The operation producing the result in column 3 is an example of a column operation. Similarly, one can iterate over all the columns in a row using a row operation. One can also perform single cell operations. 3.2

Design

The design of the Spatial Spreadsheet is object-oriented. Figure 2 shows the basic object model of the Spatial Spreadsheet in UML notation [3]. The figure shows 1

SQL is not actually used in the Spatial Spreadsheet system. It is only used here for example purposes.

The Spatial Spreadsheet

321

six class objects: Spreadsheet, Cell, Display, Relation, Query and Processor. It is important to note the distinction between a Cell object and what has been previously referred to as a cell. A cell is an element in the spreadsheet array. A Cell object is a class object named “Cell” used in the design and underlying implementation of the spreadsheet. Likewise, a Relation object is the class object named “Relation” not to be confused with a relation in the relational database. In the remainder of this paper we will distinguish object names by using the italic font. When the Spatial Spreadsheet is started, an instance of the Spreadsheet object is created. This is the top-level object and acts as the root aggregator to all other objects. The primary responsibility of the Spreadsheet object is to keep track of Cell objects, global states, and the organization of cells in the top-level window of the graphical user interface. A Spreadsheet object can have one or more Cell objects. Query objects and Relation objects are Cell objects — that is, they are derived from Cell objects. An instance of a Cell object is created when a persistent relation is opened or a cell is needed to process and store a primitive operation. Cell objects have member data items to keep track and manipulate their own relation. Cell objects can be associated with other Cell objects. Query objects derived from Cell objects use these associations to keep track of which other Cell objects it uses as input. All Cell objects use these associations to keep track of which Query objects use them as input. This becomes important in update propagation. Each Cell object has a Display object. The Display object’s role is to display data from the relation for the user. Display objects can display information for the user in several ways including a meta data display, tuple-by-tuple display of raw data, and a graphical display for spatial data types. In the graphical display spatial attributes are rendered by projecting their coordinates onto a 2D bitmap as a means of data visualization. Each Query object also has a Processor object. Processor objects are responsible for processing primitive operations.

Figure 2: Spatial Spreadsheet Object Model: boxes indicate class objects, diamonds indicate aggregate or “has a” relationships, and triangles indicate inheritance.

322

3.3

Glenn S. Iwerks and Hanan Samet

Update Propagation

There are two ways the data stored in a relation open in the spreadsheet can be changed. The first way is by an outside source. In particular, another process that accesses the underlying database can make changes to the data. The second way is by the actions of the spreadsheet itself. If a persistent relation is updated by an outside source, the effects of those changes need to be propagated to all the other cells that directly or indirectly use that relation as input. Consider the river and road example. Suppose it has been raining a lot in region B and the relation containing the information on rivers at or near flood stage is updated by inserting more rivers. In this case, the Cell object holding the result in column 2 for region B would need to be updated after the change occurred in column 0. The propagation process works as follows. A relation corresponding to a Relation object is updated. The Relation object is notified and it marks itself as “dirty”. When a Relation object or a Query object becomes dirty it then informs all Cell objects depending on it for input that they are now dirty too. It may be useful to think of the Cell objects in the spreadsheet as nodes in a directed graph. Edges directed into a node indicate Cell object inputs. Nodes in the graph having no incoming edges are Relation objects. All the other nodes are Query objects. We will refer to Query objects that have no outgoing edges as terminals. The manner in which queries are created ensures that there are no cycles in this directed graph. Therefore, we do not have to check for cycles while passing messages. Eventually, these messages are passed through all possible paths from the initial dirty Relation object to all terminals reachable from the initial Relation object. Since there are no cycles, message passing will cease. After all Cell objects are marked dirty that can be marked dirty, the initial dirty Relation object marks itself as “clean”. The PropagateClean() method is invoked for each Cell object that uses the Relation object as direct input. The PropagateClean() method propagates the update. PropagateClean() { If all my inputs are clean and I am active then { Mark myself clean and recalculate primitive operation For each Cell object J that uses me as input do Call J’s PropagateClean() method } } It is necessary to propagate all the “dirty” messages all the way through the graph of Cell objects before recalculating any primitive operations associated with a Cell object otherwise some Cell objects might recalculate their operations more than once. For example, suppose that Cell object X recalculates its operation as soon as one of its inputs, say Cell object Y, indicates that a change has occurred. If Cell object Y is also input to Cell object Z which in turn is input to Cell object X, then Cell object X would have to update itself again after it is informed that Cell object Z has been updated. If this situation was not prevented, then there could be as many as O{n2 } updates. This situation

The Spatial Spreadsheet

323

is prevented by informing each Cell object of all imminent updates before any updates are actually performed. This essures O{n} updates. Note that individual Cell objects may be set “active” or “inactive” by the user. An inactive Cell object blocks the update of itself and blocks the propagation of updates to it’s dependents. This avoids spending time updating Cell objects in the spreadsheet in which the user is not currently interested. Updates may propagate automatically whenever a change occurs or only as desired by the user. At the top level, the Spreadsheet object has an UpdateSpreadsheet() method. This is called to initiate update propagation. UpdateSpreadsheet() { For each Cell object K in the spreadsheet do If K is dirty then Call K’s PropagateClean() method. } 3.4

Graphical User Interface

Rather than expressing operations on cells with a query language such as SQL, the simple operations associated with cells are created through the use of the “wizard”. The wizard consists of one or more popup windows that guide the user through the steps of instantiating a cell. To start the wizard, the user clicks on an empty cell. At each step, the wizard offers the possible choices to the user and the user selects the desired choice with the mouse. In some cases, the user may still have to type something. In particular, this is the case when an expression is required for a selection or join operation. At present, the user is required to type the entire expression. As in the SI system [6], we chose to use Tcl [7] for expressions. This requires the user to be knowledgeable of the expression syntax. This error-prone aspect detracts from the GUI’s ease of use. We intend to replace this with a more intuitive system in the future. The main window consists of an array of cells (see Figure 1). Cells can be expanded or contracted by sliding the row and column boundaries back and forth. Theoretically, the spreadsheet could hold an unlimited number of rows and columns but to simplify the implementation we limit the number of rows and columns. We can still start the system with a large number of cells and hide those that are not being used by moving the sliders. Display of spatial attributes is not limited to the graphical display in a single cell. Each graphical display can display spatial attributes from any relation associated with any cell in the spreadsheet. This allows the user to make visual comparisons by overlaying different layers in the display. The Spatial Spreadsheet also provides a global graphical display in a separate top-level window.

4

Implementation

The Spatial Spreadsheet is an interface used to interact with a spatial relational database. The spatial relational database we use is SAND [2]. SAND provides the database engine that underlies the system. It contains facilities to create,

324

Glenn S. Iwerks and Hanan Samet

update and delete relations. It provides access methods and primitive operations on spatial and non-spatial data. The Spatial Spreadsheet extends the basic set of primitive queries to include the classic selection, projection and nested loop join operations. The implementation of the Spatial Spreadsheet is object-oriented and was written entirely in incremental Tcl (iTcl) and incremental Tk (iTk). It runs on Sun Sparc and Linux systems.

5

Concluding Remarks

We have described how the power of the spreadsheet can be combined with a spatial database. The Spatial Spreadsheet provides a framework in which to organize data and build queries. Row and column operations provide a mechanism for rapid query creation on large amounts of related data. The systematic tabulation of the data as found in the two-dimensional array of the Spatial Spreadsheet enables the user to visually compare spatial components and pick out patterns. The user can also see how query results change as updates occur. An important issue for future work that was not addressed here is update propagation optimization. In particular, the output of any given Query object may be the result of many steps along the way between it and initial Relation objects. Currently the method of computation is determined in a procedural manner by the user. In the future we will focus on converting this to a declarative form and using query optimization techniques to improve refresh efficiency when updates occur.

References 1. W. G. Aref and H. Samet. Efficient window block retrieval in quadtree-based spatial databases. GeoInformatica, 1(1):59–91, April 1997. 320 2. C. Esperan¸ca and H. Samet. Spatial database programming using SAND. In M. J. Kraak and M. Molenaar, editors, Proceedings of the Seventh International Symposium on Spatial Data Handling, volume 2, pages A29–A42, Delft, The Netherlands, August 1996. 318, 323 3. M. Fowler and K. Scott. UML Distilled, Applying the Standard Object Modeling Lanuage. Addison-Wesley, Reading, MA, 1997. 320 4. G. R. Hjaltason and H. Samet. Distance browsing in spatial databases. Computer Science Department TR-3919, University of Maryland, College Park, MD, July 1998. (To appear in ACM Transactions on Database Systems). 320 5. G. R. Hjaltason and H. Samet. Incremental distance join algorithms for spatial databases. In Proceedings of the ACM SIGMOD Conference, pages 237–248, Seattle, WA, June 1998. 320 6. M. Levoy. Spreadsheets for images. In Proceedings of the SIGGRAPH’94 Conference, pages 139–146, Los Angeles, 1994. 318, 323 7. J. K. Ousterhout. Tcl and the Tk Toolkit. Addison-Wesley, April 1994. 323 8. J. Rasure and C. Williams. An integrated visual language and software development environment. Journal of Visual Languages and Computing, 2(3):217–246, September 1991. 318 9. A. Silberschatz, H. F. Korth, and S. Sudarshan. Database System Concepts. McGraw-Hill, New York, third edition, 1996. 317, 320

A High Level Visual Language for Spatial Data Management Marie-Aude Aufaure-Portier and Christine Bonhomme Laboratoire d’Ingénierie des Systèmes d’Information INSA & UCBL Lyon F-69 621 Villeurbanne [email protected] [email protected]

Abstract. In this paper, we present a visual language dedicated to spatial data called Lvis. This language has been defined as an extension of the Cigales visual language based on the Query-By-Example principle. The language is based on predefined icons modelling spatial objects and operators that are used to build a visual query. The visual query is then translated into the host language of Geographic Information Systems (GIS). A major problem of such a language is that visual queries are generally ambiguous because of multiple interpretation of the visual representation. We first present a brief state of the art of languages dedicated to GIS and then formally define our visual language. The global architecture of the system is described. We then focus on visual ambiguities and propose a model of detection and resolution of these ambiguities.

1 Introduction Many recent research have been recently done in the field of Geographic Information Systems (GIS) especially for data storage, new indexing methods, query optimization, etc. [1]. A main characteristic of GIS is to manage complex and large amount of data. A fundamental research area concerns the definition of high level user interface because GIS users are generally non-computer scientists. Many applications are concerned by spatial data: urban applications, geomarketing, vehicle guidance and navigation, tourism and so on. Human actors implied in these applications are architects, engineers, urban planners, etc. GIS applications have recently migrated towards citizen-oriented applications. This makes crucial the definition of simple and user-friendly interfaces. Cartographic information can be graphically visualized (maps, pictograms, etc.) using marketed GIS, but, in most cases the languages developed for queries and updates are very poor and dedicated to only one specific system. The consequence is that end-users applications cannot be supported by other systems. Another drawback is the complexity for non-computer specialists to design and develop applications. However, the main characteristic of spatial information is to be graphical. This implies that graphical or visual languages are well suited for spatial applications. Graphical languages are based on the use of symbols representing the data model concepts. These symbols are only pure graphical conventions, without any Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL'99, LNCS 1614, pp. 325-332, 1999.  Springer-Verlag Berlin Heidelberg 1999

326

Marie-Aude Aufaure-Portier and Christine Bonhomme

metaphorical power, and consequently need to be explained and memorized. Visual languages use metaphors to show the concepts. Metaphors take the mental model of the end-user into account. We propose a visual language defined as an extension of the Cigales language [2]. This language, Lvis, is based on the use of icons. These icons represent the spatial objects stored into the database and the spatial operators used to build a visual query. The extension concerns: (1) the definition of new operators such as logical operators; (2) the detection and resolution of visual ambiguities due to the principle of query construction; (3) the integration of this language into a customizable visual environment [3] devoted to the design and development of spatial applications. We firstly present a brief state of the art of visual languages for spatial applications. Then, we define our visual language. We then present how to deal with visual ambiguities and propose a detection and resolution model of a particular class of visual ambiguities. A prototype is now available and briefly described in this paper. We then conclude on our future work and perspectives.

2 State of the Art Many propositions have been made the last decade in the field of spatial information retrieval. We can distinguish between the query language approach and the hypermedia approach. We can classify the propositions of query languages into two kinds: (1) textual approaches (natural language and extensions of SQL), (2) nontextual approaches (tabular, graphical or visual languages). Natural language [4] approach seems the most suitable for the end-user. But, a serious difficulty of this approach is that many ambiguities must be solved. Another troublesome issue is that query formulation can be verbose and difficult (generally, a drawing is better than a long sentence). This approach can be seen as a good complement for graphical and visual approaches. Many extensions of the SQL language have been proposed [5,6]. These extensions are necessary in order to allow Data Base Management Systems to store and retrieve spatial information. However, this class of languages are not suited to end-users because of the difficulty to express spatial relations in natural language and the lack of conviviality for technical languages such as extensions of SQL. Tabular approaches [7,8] are defined as extensions of QBE (Query-By-Example)°[9]. The main difficulty is to express joins. Graphical languages make a better use of visual medium but the underlying concepts are not perceived in a metaphorical way. Considering that spatial information is visual, visual languages [2,10,11,12] have been proposed. Some works have also been done to design new metaphors [13,14]. Visual Languages use icons and metaphors to model spatial objects, spatial relations between objects and queries. The user’s mental model is taken into account°[15]. A metaphor can be seen as a mapping between a domain with a high level of abstraction and another domain with a low level of abstraction. An icon can be viewed as a visual representation of a concept. This approach has been expanded very rapidly because of the evolution of the applications towards citizens and the requirements of the end-users of conviviality and ease of use of the interface. Visual languages offer an intuitive and incremental view of spatial queries but lack from a poor expressive power, execution inefficiency and multiple interpretations for a

A High Level Visual Language for Spatial Data Management

327

query. Two main approaches have been developed to design visual languages: (1) the end user draws a pattern using a set of icons, and (2) the end-user makes a drawing directly on the screen using the blackboard metaphor. The first approach is illustrated by the Cigales language [2] and the second one by the Sketch! [10] and SpatialQuery-By-Sketch [12] languages. The reader can refer to [16] to have more details about query languages for GIS. The main advantage of these two approaches comes from the fact that the user does not have any constraint to express a query and no new language to learn. The main limitation is that a query can lead to multiple interpretations. The user’s drawing may not represent the real world (error due to the mental representation of the user) and may lead to a wrong interpretation or may not represent the user’s viewpoint. These languages can be seen as precursors for visual querying in the GIS domain application and provide two different approaches. The main contribution is that users having a low level in computers can express queries with an intuitive manner. These languages also permit the visual definition of spatial views. Nevertheless, many limitations still remain. The main limitation comes from the ambiguities of visual languages. This problem is the object of section 4. Another limitation is that alphanumerical and spatial data are not uniformly supported. A few operators have no graphic equivalence like operators used for reasoning and deduction

3 Definition of the Visual Language Lvis This section describes the general architecture of our project, then defines the syntax and semantics of the Lvis language. A prototype is already available on the marketed GIS MapInfo and is described in section 3.1. 3.1 Architecture of the Project Lvis is integrated into a customizable design and development environment [3]. The end-user interface is based upon icons and pop-up menus. A visual query can be seen with a visual representation (visual metaphors), a technical representation (spatial objects and operators involved in the query) and a textual representation (extended SQL). A query is first expressed using the visual language: incremental composition of icons and operators. This query is then translated into an intermediate language in order to be independent from the GIS. This intermediate language is based on the functionalities proposed in SQL3-MM [17]. The query is then translated, using a specific driver, towards the host language of the GIS plat-form. A prototype is under current development and already available for simple queries, i.e. queries with only one operator. The visual query is then translated into a structural representation (binary tree). A textual representation is then extracted from this structural representation. This textual representation is then transformed into the GIS host query language (MapInfo host language). The current work concerning our prototype is the integration of complex queries and the graphical display of query results. The graphical interface is given in Figure 1.

328

Marie-Aude Aufaure-Portier and Christine Bonhomme

Fig. 1. Graphical interface of the Lvis language

3.2 Definition of the Language This section describes the syntax and semantics of Lvis. The alphabet of the language is divided into two sub-sets: the spatial object types set (polygons and lines) and the operators set (figure 2). Operators Set theory Logical

Spatial Topological Intersection Inclusion Adjacency Disjunction Equality

Metrical Point selection Ray selection

Intersection Union Identity Difference Exclusive conjunction

And Or Not

Interactive Structural selection Point Creation Radius Modification Rectangle Deletion Any Area Buffer Zone

Fig. 2. Operators set

The two spatial object types to be handled are polygonal and linear objects. We assume that a punctual object would be represented by a polygonal object the area of which is null. This set ST of spatial object types is defined by: ST: STN x STI et STN = {Polygonal, Linear]}, STI={ , ∀ st ∈ ST , st = (name st , icon st ) ∧ name st ∈ STN ∧ icon iconts =

iff namets = "Polygonal",

st

} ∈ STI ∧

iff namets = "Linear"

Another set of object types is the set of data types. We only consider in this paper spatial data, i.e. objects of the database that own a spatial type. The icons that represent these object types generally use visual metaphors and aim at being as closely as possible to the mental models of the users.

A High Level Visual Language for Spatial Data Management

329

DT: DTN x DTI x ST et DTN is the set of names of object types stored in the database, DTI is the set of icons of object types stored in the database, ST is the set of spatial object types previously defined. ∀dt ∈ DT , dt = (namedt , icondt, typeSst ) ∧ namedt ∈ DTN ∧ icondt ∈ DTI ∧ icondt = ficondt (namedt ) ∧ typeSdt ∈ ST ficondt, is a mapping function that associates an icon to a name of data types. The set of operators contains spatial, set theory's, logical, interactive selection and structural operators. Spatial operators are composed of topological and metrical operators (figure3). The choice of the topological operators has been made in accordance to those that are supported by the normalized spatial SQL [17]. All of these operators are either binary or unary operators.

4 How to Deal with Ambiguities? Visual ambiguities can occur at two different levels. The first level concerns the visual representation of the query by the system and the second level is how the visual query is interpreted by end-users. On one hand ambiguities appear when several visual representations are suitable for a given query. The system must decide which one of these visual representations will be displayed to the user. On the other hand ambiguities are generated when a visual representation of a given query is interpreted in different ways. This second case of ambiguities, called interpretation ambiguities, is minimized thanks to the syntax and semantics of our visual language. For example, colours are used to associate the icon of an object with its shape and the symbol ‘?’ indicates the target object of the query. Moreover, the technical representation of a query reminds the steps of its formulation. Thus, we have focused our work on the first case of ambiguities, called representation ambiguities. Firstly a classification of (visual) ambiguities types have been defined (figure 3). Four main types of ambiguities have been distinguished: visual ambiguities tied to the (1) topological relations between the objects of a query; (2) location of objects expressed in Cartesian or cardinal coordinates; (3) geometry of objects; (4) number of occurrences for a given spatial relation between objects. The two first classes are subdivided into three subclasses: simple ambiguities between simple objects, grouping ambiguities between groups of objects and intersection ambiguities between intersections of objects. Figure 4 shows an example of ambiguous visual representation for each one of these classes. Topology

Location

AMBIGUITY TYPE Simple Grouping Relations with objects’ intersections Simple Grouping Relations with objects’ intersections

Geometry Number of relations between two objects Fig. 3. Taxonomy of visual ambiguities

C11 C12 C13 C21 C22 C23 C3 C4

330

Marie-Aude Aufaure-Portier and Christine Bonhomme

C11 Some of spatial relations may be not explicitly specified by the user (e.g.: spatial relation between A and C)

C12 The object A is disjoint from a group of objects. Must A be located inside/outside the grouping objects?

C13 Does the system allow the user to specify spatial relations between the intersections of objects (spatial relation between A ∩ B and C)?

C21 Does the system allow the user to specify the coordinates of the objects?

C22 Does the system allow the user to specify the coordinates of grouping objects?

C23 Does the system allow the user to specify distances between the intersections of objects?

C3 Does the system allow the user to exactly specify the shape of the objects?

C4 Does the system allow the user to specify: the number of occurrences of a same spatial relation between two objects (left figure); several different types of spatial relations (right figure)?

Fig. 4. Example of ambiguous visual representations

Our study is concentrated to the handling of topological ambiguities that are concerned with the intersections of objects and especially to the problem of the "don't care" relations. We illustrate the problem of "don't care" relations taking an example of a spatial query. Consider the query “Which towns are crossed by a river and have a forestry zone?” This query is expressed with our language in two steps: at first, specification of an intersection relation between an object type “Town” and an object type “River”; then, formulation of another intersection relation between the same object type “Town” and a new object type “Forest”. But the user did not specify the spatial relation between the objects “River” and “Forest” just because he doesn't care. What must decide the system? Which visual representation must be chosen for these two objects and for the whole query? (figure 5). Which towns are crossed by a river and have a forestry zone? The spatial relation between the

objects towns and forest doesn’t care ⇒ A few visual representations exist!

Fig. 5. Example of a “don’t care” relation

A High Level Visual Language for Spatial Data Management

331

To solve this problem, a model has been proposed for the detection of visual ambiguities. It determines the set of possible visual representations for a given query. This model is based on the intersection levels between objects involved in queries and is defined as a graph-type model including nodes and edges (figure 6 (a)). Nodes and edges can be either enabled or disabled according to spatial criteria of the query. The main advantage of this model is that the model for queries composed of a given number of objects is built only once and is updated according to the spatial criteria of the other queries. This model is conceivable for queries with up to four objects (figure 6 (b)). For a query with n objects, the model contains 2n-1 nodes and Card(2n-1,2) = 2n-1!/(2!*(2n-1-1)!) relations that can be specified between the objects. We assume that most of spatial queries contain less than four objects. So this model can be integrated to our language. Objects # Nodes # 1 2 3 4 5

1 3 7 15 31

Possible representations # 2 8 128 32 768 2 147 483 648

Fig. 6. Model of detection of visual ambiguities. (a) The graph structure of the model; (b) The complexity of the model

When the user submits a visual query, the system searches for the set of possible visual representations all over the graph of the query. If more than one possible representation exists the system decides which one will be the less ambiguous for the user. To do that we think that it is necessary to allow interactions between the system and the user in order to build a user profile. For example, keeping the user's preferences concerning visual representations of queries types that are often formulate could be a good and efficient strategy. The system becomes so a personalized system, indeed even a self-adapted system. We think too that it could be interesting to let the user modifying itself the visual representation of its query. This can be realized by mean of dynamic alterable visual representation of the queries. Spatial criteria of queries (and so spatial relations between the objects of the queries) still remain to be true whatever changes may be done on the visual representation.

5 Conclusion and future work This paper presents Lvis, an extension of the visual language Cigales devoted to spatial information systems. This language is based upon a query-by-example philosophy. We then focus on how to detect and solve visual and representation ambiguities. We have defined a resolution model for “don’t care” relationship between spatial objects. This model is realistic for queries containing less than four objects and will be integrated to our prototype. We must now study the others cases of

332

Marie-Aude Aufaure-Portier and Christine Bonhomme

ambiguities. The prototype has been developed according to the architecture of the project described in this paper. We must now validate it in collaboration with potential end-users. A first set of cognitive tests have already been realized. Some conclusions about these tests have been extracted and must be confirmed.

References 1. Laurini, R., Thompson D.: Fundamentals of Spatial Information Systems, The APIC series, Academic Press (1992) 2. Aufaure-Portier, M-A.: A High-Level Interface Language for GIS, Journal of Visual Languages and Computing, Vol. 6 (2), Academic Press (1995) 167-182 3. Lbath, A., Aufaure-Portier, M-A., Laurini, R.: Using a Visual Language for the Design and Query in GIS Customization, 2nd International Conference on visual information systems (VISUAL97), San Diego, (1997) 197-204 4. Bell, J.E.: The experiences of New Users of a Natural Language Interface to a Relational Database in a Controlled Setting, First Int. Workshop on Interfaces to Database Systems, Ed. R. Cooper, Springer-Verlag (1992) 433-454 5. Costagliola, G., and al.: GISQL - A Query Language Interpreter for Geographical Information Systems, IFIP Third Working Conference on Visual Database Systems (1995) 247-258 6. Egenhofer, M.: Spatial SQL : A Query and Presentation Language, IEEE Transactions on Knowledge and Data Engineering (1994),Vol. 6 (1) 86-95 7. Staes, F., and al.: A Graphical Query Language for Object Oriented Databases, IEEE Workshop on Visual Languages (1991) 205-210 8. Vadaparty, K., and al.: Towards a Unified Visual Database Access, SIGMOD Record (1993) Vol. (22) 357-366 9. Zloof, M.M.: Query-by-Example : A Database Language, IBM Systems Journal (1977) Vol. 16 (4) 324-343 10. Meyer, B.: Beyond Icons : Towards New Metaphors for Visual Query Languages for Spatial Information Systems, Proceedings of the first International Workshop on Interfaces to Database Systems (R. Cooper ed.), Springer-Verlag (1993) 113-135 11. Benzy, F., and al.: VISIONARY: a Visual Query Language Based on the User Viewpoint Approach, Third International Workshop on User-Interfaces to Database Systems (1996) 12. Egenhofer, M.J.: Query Processing in Spatial-Query-by-Sketch, Journal of Visual Languages and Computing (1997) Vol. 8 (4) 403-424 13. Egenhofer, M.J., Bruns, H.T.: Visual Map Algebra : A Direct-Manipulation User Interface for GIS, Third Working Conference. on Visual Database Systems (IFIP 2.6), (1995) 211226 14. Kuhn, W.: 7±2 Questions ans Answers about Metaphors for GIS User Interfaces, Cognitive Aspects of Human-Computer Interaction for Geographic Information Systems (T. Nyerges, D. Mark, R. Laurini & M. Egenhofer ed.) (1993) 113-122 15. Downs, R.M., Stea, D.: Maps in Minds, Reflections on cognitive mapping, Harper and Row Series in Geography (1977) 16. Aufaure-Portier, M.A., Trepied, C.: A Survey of Query Languages for Geographic Information Systems, Proceedings of IDS-3 (3rd International Workshop on Interface to Database), published in Springer Verlag's Electronic Workshops in Computer Series (1996) 14p (www.springer.co.uk/eWiC/Worshops/IDS3.html) 17. ISO/IEC JTC1/SC21/WG3 DBL-SEL3b (1990)

A Global Graph Model of Image Registration S. G. Nikolov, D. R. Bull, and C. N. Canagarajah Image Communications Group, Centre for Communications Research University of Bristol, Merchant Venturers Building Woodland Road, Bristol BS8 1UB, UK Tel: (+ 44 117) 9545193, fax: (+ 44 117) 9545206 {Stavri.Nikolov,Dave.Bull,Nishan.Canagarajah}@bristol.ac.uk

Abstract. The global graph model of image registration is a new visual framework for understanding the relationships and merits between the wide variety of existing image registration methods. It is a global, dynamically updateable model of the state-of-the-art in image registration, which is designed to assist researchers in the selection of the optimal technique for a specific problem under investigation. Two-dimensional and three-dimensional graph display techniques are used in this paper to visualise the new model. The Virtual Reality Modeling Language (VRML) was found to provide a very suitable representation of such a 3-D graph model.

1

Introduction

Image registration is a common problem in many diverse areas of science including computer vision, remote sensing, medical imaging, and microscopy imaging. Image registration can be defined as the process which determines the optimal correspondence between two or more images. Such images may be acquired from one and the same object: (a) at different times; (b) under different conditions; (c) from different viewpoints; (d) from various sensors. One of the images I1 is taken to be the reference image, and all other images I2 , I3 , . . . , In , called input images, are matched to the reference image. To register the images, a transformation must be found, which will map each point of an input image to a point in the reference image. The mapping has to be optimal in a way that depends on what needs to be matched in the images. Over the years, a great variety of image registration techniques have been developed for various types of data and problems. These techniques have been independently proposed and studied by researchers from different areas, often under different names, resulting in a vast collection of diverse papers on image registration. Research areas which have contributed significantly to the development of image registration techniques comprise computer vision and pattern recognition, medical image analysis, remotely sensed image processing, 3-D microscopy, astronomy, computer aided design (CAD), and automatic inspection. Each of these areas has developed its own specialised registration methods. The need to compare the different approaches to image registration has recently led to the publication of Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL’99, LNCS 1614, pp. 333–340, 1999. c Springer-Verlag Berlin Heidelberg 1999


334

S. G. Nikolov et al.

several review papers [1,18,10]. Most of these review papers try to classify image registration methods according to some classification scheme, e.g. the primitives used to match the images, the type and complexity of the transform utilised to align the images, etc. The great majority of such schemes are very much domain specific. Reviews of image registration methods for alignment of medical images, for example, can be found in [15,11,18,10]. The similarity of some registration methods, applied to images from different research areas, however, suggests the usefulness of a global taxonomy of image registration techniques, where such techniques are compared not only on similar images from one single area (e.g. medical images, microscopic images), but also across scientific areas and across different scales (i.e. macroscopic versus microscopic images). The only comprehensive review paper on image registration methods spanning images and methods from different research areas, is the paper published by Brown [1]. All image registration methods in [1] are described according to a four-component classification scheme. In this paper, we propose a new model of the image registration process. This model, which we call the global graph model of image registration, is an attempt to put together results from many diverse areas into a single representation, where the similarities and differences between the image registration methods and their components may be clearly seen. The global graph model of image registration is much like a taxonomy of image registration methods, although we would prefer to view it as a dynamically updateable, multi-component, graphical representation of the image registration process. The model has been derived from the model proposed by Brown, while several extensions have been added. The aim of this paper is to present the new graph model, rather than to review the existing techniques in image registration. Hence, only a few example papers from several research areas are used to build a nucleus of the graph model.

2

Brown’s Image Registration Model

In her review of image registration techniques, Brown [1] considers image registration as a combination of four key components: (a) feature space (FS) - the set of image features which are extracted from the reference image and from the input images, and are used to perform the matching; (b) search space (SSp) - the class of potential transformations that establish the correspondence between the input images and the reference image; (c) search strategy (SSt) - the method used to choose which transformations have to be computed and evaluated; (d) similarity metric (SM) - which provides a quantitative measure of the match between the reference image and the transformed input images, for a given transformation chosen in the search space, using the search strategy. Brown has reviewed numerous articles on image registration and has classified all image registration methods into several tables [1], corresponding to the four components of her model. However, in Brown’s paper, it is very difficult to see

A Global Graph Model of Image Registration

335

the relations between the tables, and furthermore, on the basis of these tables only, it is impossible to track down how the choices of each component are put together in each paper to form a complete image registration method. Finally, Brown’s model is static and cannot be updated on-line, thus being an excellent snapshot of the state-of-the-art in image registration at the time of its publication. Our global graph model is an attempt to overcome these disadvantages.

3 3.1

A Global Graph Model of Image Registration Extensions to Brown’s Image Registration Model

We propose to add the following new components to Brown’s model (Fig. 1 (left)): (e) image space (IS) - this is the space of images to be registered, grouped into classes on the basis of the area of research (e.g. medical images, remote sensing images, etc.); (f) dimension - the dimension of the images, which may be 2-D, 3-D, 4-D; (g) paper - the publication which describes a new image registration technique or a new application of a known registration algorithm. The last two additional components may be regarded as meta-components, because they specify some characteristics of instances from the other major components of the model. More meta-components can be added to the model, but here we want to keep the image registration model as simple and as general as possible.

3.2

A 2-D Global Graph Model of Image Registration

An example of the 2-D global graph model of image registration is given in Fig. 1 (left). The basic components of the model in Fig. 1 (left) define several layers in the global graph. Each new paper is a subgraph of the global graph. The root of this subgraph is the reference to the paper while its nodes are instances from the successive layers of the global graph. Generally, several kinds of subgraphs of the global graph can be distinguished: (a) paper graph - a graph which presents image registration results published in a specific paper. Three example paper graphs (dashed, solid, and bold edges) are included in Fig. 1 (left); (b) layer graph - a graph connecting all the nodes in one layer of the global graph model (e.g. all the image classes in the IS, as illustrated in Fig. 1 (right)). Fig. 1 (right) shows only some example image classes and modalities. More areas or new modalities can be added to the IS layer; (c) comparison graph - a graph which compares several different alternatives of some steps of the image registration process. A comparison graph may show the difference between two complete image registration algorithms, or it may compare only some steps of these algorithms; (d) area graph - an area graph is a generalisation of a comparison graph, where all the images from the IS are from one research area. Thus, the similarities and differences between image registration methods applied to images from a certain area (e.g. medical images, remote sensing images) can be observed at a glance.

336

S. G. Nikolov et al.

paper

Le Moigne [12]

Nikolov [14]

Studholme [17]

IS

remote sensing

microscopy

medical

dimension

2-D

3-D

FS

WT maxima

intensity

SSp

piece-wise polynomial

SSt

SM

affine

rigid

Hierarchical Techniques

Normalized crosscorrelation function

Correlation coefficient

Relative Entropy

Fig. 1. The 2-D global graph model of image registration (left). Example papers included: Le Moigne [12] (dashed edges), Studholme [17] (bold edges), and Nikolov [14] (solid edges). The different nodes of the IS layer (right). All abbreviations are given in [13]. A double-ended arrow between any two modalities shows that registration of images from these modalities has been studied in a specific paper (paper graph). 3.3

A 3-D Global Graph Model of Image Registration

There is growing evidence that the human brain can comprehend increasingly complex structures if these structures are displayed as objects in 3-D space [20,19]. If the layers of the global graph are displayed as parallel planes in 3-D space, a 3-D global graph model of image registration (Fig. 2) can be built. The use of multiple abstraction levels is a common approach to visualisation of very large graphs. Several techniques have been proposed in the past for constructing 3-D visualisations of directed and undirected graphs [19,2,7], multi-level clustered graphs [3,6] and hierarchical information structures [16]. Three-dimensional graphs have also been successfully used as graphic representation of knowledge bases [5]. The main advantage of 3-D multi-level graph display over 2-D graph display, especially when it comes to very large graphs, is that the additional degree of freedom allows the numerous graph nodes to be spread across several levels, making the overall graph structure much more conceivable. The nodes in one layer can be positioned according to some kind of closeness measure, which is specific for this layer (e.g. Fig. 1 (right)), and thus can be grouped into meaningful clusters. Subgraphs, e.g. paper graphs and layer graphs, may be regarded as cross-sections of the global graph and can be plotted as 2-D graphs for easier interpretation. Large graphs can be displayed using a variety of methods such as: (a) all the information associated with the

A Global Graph Model of Image Registration

337

Fig. 2. A 3-D display of part of the global graph model.

nodes and edges of the graph is displayed; (b) several views or zoom-in maps are plotted; (c) distorting views such as fish-eye lens are utilised; (d) stereopsis; (e) animation; (f) virtual reality. While most of these approaches generate one or several static views of the graph structure and display them to the observer, virtual reality allows the viewer to interactively examine the graph structure, or some of its details, by navigating around it (i.e. by rotation, zoom and translation of the whole structure). Hence, we have decided to use a 3-D virtual reality representation of the global graph model of image registration. One question of paramount importance is how to update the global graph model so that it stays up-to-date with the state-of-the-art in image registration. New image registration methods and new results should be easily incorporated in the global graph model. Modifications of the structure and relations in the model, in view of new developments, will also inevitably become necessary. Therefore, a dynamical 3-D representation of the global graph model is needed, which will be available to researchers from different scientific areas, who may submit new components and new methods and thus update the model.

338

3.4

S. G. Nikolov et al.

A VRML Representation of the 3-D Global Graph Model

The Virtual Reality Modeling Language (VRML) is a file format for describing interactive 3-D objects and scenes to be experienced on the World Wide Web (WWW). With the introduction of VRML 2.0 (Moving Worlds) which was replaced by VRML97 in December 1997, VRML is considered to be the de facto standard for describing and sharing 3-D interactive worlds over the WWW. We have decided to use VRML as a mean to visualise the global graph model of image registration because of the following reasons: (a) VRML files can be displayed on virtually any computer (multi-platform support); (b) VRML provides fast and high-quality rendering; (c) it comprises a rich set geometrical primitives which can be used to construct various graph displays; (d) VRML is becoming more and more popular for scientific data visualisation and exploration. So far, there have been only a few attempts to use VRML to describe and display graphs [8]. The additional degree of freedom, compared to 2-D graphs, and the fact that the viewer can navigate around the graph structure and look at it from any position and angle, create numerous difficulties, which have to be taken into account when constructing 3-D graphs using VRML. In the case of 3-D layered graphs, some of the problems that have to be solved are: (a) how to position the nodes in each layer (what kind of closeness measure to use); (b) how to add new nodes and edges to the graph so that it stays balanced and aesthetically pleasing. The global graph model will evolve in time, which means that the spatial arrangement of its nodes and edges will also change frequently; (c) how to display node labels in 3-D. Several possibilities exist: the VRML text node can be used, or alternatively, text as texture can be mapped to some geometrical primitives (e.g. spheres, cylinder, cones, etc.) which represent graph nodes (see Fig. 3); (d) what kind of orientation to use for the text labels so that the text is always readable. One simple solution is to present alternative views of horizontal or vertical text labels, depending on the position of viewer. A more sophisticated way is to track down the position of the viewer and to keep the text labels always parallel to his ’eyes’. (e) each node which belongs to the paper layer can be implemented as a link to the original paper. If a paper is available on-line, the link points to the file with the paper. Thus, paper nodes in the graph are anchors to on-line publications. VRML anchors are specified by the VRML anchor node and a URL indicating the WWW address of the destination resource. A problem which remains open is how to keep all such paper links up-to-date, having in mind that some of the on-line resources will change their addresses in time. An initial 3-D VRML graph is displayed in Fig. 3. The optimal VRML representation of the global graph model of image registration is still under investigation. Since the goal of the global graph model is to be really global, and thus accepted by most of the members of the image registration research community, the optimal VRML representation will be searched for by means of constructing several different VRML graphs and collecting feedback about the usefulness and aesthetic merits of each one of them.

A Global Graph Model of Image Registration

339

Fig. 3. A VRML representation of the graph model shown in Fig. 1 (left). This VRML 2.0 file was generated with the new version of the GraphViz program [4]. A short description of the global graph model of image registration, including the VRML 2.0 representation, can be found at http://www.fen.bris.ac.uk/elec/research/ccr/imgcomm/fusion.html

4

Conclusion and Acknowledgements

In this paper we have presented a new graph model of the image registration process. This new model is an extension to Brown’s four-component model. The new global graph model has several advantages over other image registration models, i.e. it is domain independent, dynamically updateable, and it visually displays the similarities and differences between various image registration methods and their components. A VRML representation of the 3-D global graph model is presented and several problems connected with its construction and display are discussed in the paper. Similar graph models can also be used in other image related research areas, e.g. to characterise content-based retrieval systems, where the IS will comprise the different media (text, audio, image, video), and the FS may consist of texture, colour, motion, etc. This work was funded by UK EPSRC Grant #GR/L53373. We are grateful to Dr. Stephen North from AT&T Bell Laboratories for providing the dot [9] and GraphViz programs [4].

340

S. G. Nikolov et al.

References 1. L. G. Brown. A survey of image registration techniques. ACM Computing Surveys, 24(4):325–376, 1992. 334 2. R. F. Cohen, P. Eades, T. Lin, and F. Ruskey. Three-dimensional graph drawing. In R. Tamassia and I. G. Tollis, editors, Graph Drawing (Proc. GD ’94), volume 894 of Lecture Notes in Computer Science, pages 1–11. Springer-Verlag, 1995. 336 3. P. Eades and Q. Feng. Multilevel visualization of clustered graphs. In Graph Drawing ’96 Proceedings. Springer-Verlag, 1996. 336 4. J. Ellson, E. Gansner, E. Koutsofios, and S. North. GraphViz: tools for viewing and interacting with graph diagrams. The GraphViz program is available at http://www.research.att.com/sw/tools/graphviz. 339 5. K. M. Fairchild, S. T. Poltrock, and F. W. Furnas. SemNet: Three-Dimensional Graphic Representations of Large Knowledge Bases. Lawrence Erlbaum, 1988. 336 6. Qingwen Feng. Algorithms for Drawing Clustered Graphs. PhD thesis, University of Newcastle, Australia, April 1997. 336 7. A. Garg and R. Tamassia. GIOTTO3D: a system for visualizing hierarchical structures in 3D. In Graph Drawing ’96 Proceedings. Springer-Verlag, 1996. 336 8. Cristian Ghezzi. A geometric approach to three-dimensional graph drawing. Technical report, Computation Dept, UMIST, Manchester, UK, 1997. 338 9. E. Koutsofios and S. C. North. Drawing graphs with dot. Technical report, AT&T Bell Laboratories, Murray Hill, NJ, USA, 1992. 339 10. J. B. A. Maintz and M. A. Viergever. A survey of medical image registration. Medical Image Analysis, 2(1):1–36, March 1998. 334 11. C. R. Maurer and J. M. Fitzpatrick. A review of medical image registration. In R. J. Maciunas, editor, Interactive Image-Guided Neurosurgery, pages 17–44. American Assoc of Neurological Surgeons, 1993. 334 12. J. Le Moigne and R. F. Cromp. The use of wavelets for remote sensing image registration and fusion. Technical Report TR-96-171, NASA Goddard Space Flight Center, 1996. 336 13. S. G. Nikolov. A Global Graph Model of Image Registration. Technical Report UoB-SYNERGY-TR01, Image Communications Group, Centre for Communications Research, University of Bristol, May 1998. 336 14. S. G. Nikolov, M. Wolkenstein, H. Hutter, and M. Grasserbauer. EPMA and SIMS image registration based on their wavelet transform maxima. Technical Report TR-97, Vienna Univesity of Technology, Austria, 1997. 336 15. C. A. Pelizzari, D. N. Levin, G. T. Y. Chen, and C. T. Chen. Image registration based on anatomical surface matching. In Interactive Image-Guided Neurosurgery, pages 47–62. American Assoc of Neurological Surgeons, 1993. 334 16. G. G. Robertson, J. D. Mackinlay, and S. Card. Cone trees: Animated 3-D visualization of hierarchical information. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems. ACM, 1991. 336 17. C. Studholme, D. L. G. Hill, and D. J. Hawkes. Automated 3D registration of truncated MR and CT images of the head. In David Pycock, editor, Proc. of BMVA, pages 27–36, 1995. 336 18. P. A. van den Elsen, E. Pol, and M. Viergever. Medical image matching - a review with classification. Eng. Med. Biol., 12(1):26–39, March 1993. 334 19. C. Ware, D. Hui, and G. Franck. Visualizing object oriented software in three dimensions. In CASCON 1993 Proceedings, 1993. 336 20. Y. Xiao and Milgram. Visualization of large networks in 3-D space: Issues in implementation and experimental evaluation. In CAS 1992 Proc., 1992. 336

A Graph–Theoretic Approach to Image Database Retrieval Selim Aksoy and Robert M. Haralick Intelligent Systems Laboratory Department of Electrical Engineering University of Washington, Seattle, WA 98195-2500 U.S.A. {aksoy,haralick}@isl.ee.washington.edu http://isl.ee.washington.edu

Abstract. Feature vectors that are used to represent images exist in a very high dimensional space. Usually, a parametric characterization of the distribution of this space is impossible. It is generally assumed that the features are able to locate visually similar images close in the feature space so that non-parametric approaches, like the k-nearest neighbor search, can be used for retrieval. This paper introduces a graph–theoretic approach to image retrieval by formulating the database search as a graph clustering problem to increase the chances of retrieving similar images by not only ensuring that the retrieved images are close to the query image, but also adding another constraint that they should be close to each other in the feature space. Retrieval precision with and without clustering are compared for performance characterization. The average precision after clustering was 0.78, an improvement of 6.85% over the average precision before clustering.

1

Motivation

Like in many computer vision and pattern recognition applications, algorithms for image database retrieval have an intermediate step of computing feature vectors from the images in the database. Usually these feature vectors exist in a very high dimensional space where a parametric characterization of the distribution is impossible. In an image database retrieval application we expect to have visually similar images close to each other in the feature space. Due to the high dimensionality, this problem is usually not studied and the features are assumed to be able to locate visually similar images close enough so that non-parametric approaches, like the k-nearest neighbor search, can be used for retrieval. Unfortunately, none of the existing feature extraction algorithms can always map visually similar images to nearby locations in the feature space and it is not uncommon to retrieve images that are quite irrelevant simply because they are close to the query image. We believe that a retrieval algorithm should be able to retrieve images that are not only close (similar) to the query image but also close (similar) to each other. Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL’99, LNCS 1614, pp. 341–348, 1999. c Springer-Verlag Berlin Heidelberg 1999


342

Selim Aksoy and Robert M. Haralick

In this work, we introduce a graph–theoretic approach for image retrieval by formulating the database search as a graph clustering problem. Graph–theoretic approaches have been a popular tool in the computer vision literature, especially in object matching. Recently, graphs were used in image segmentation [8,7,4] by treating the image as a graph and defining some criteria to partition the graph. Graphs did not receive significant attention in image retrieval algorithms mainly due to the computational complexity of graph-related operations. Huet and Hancock [5] used attributed graphs to represent line patterns in images and used these graphs for image matching and retrieval. Clustering the feature space and visually examining the results to check whether visually similar images are actually close to each other is an important step in understanding the behavior of the features. This can help us determine the effectiveness of both the features and the distance measures in establishing similarity between images. In their Blobworld system, Carson et al. [3] used an expectation-maximization based clustering algorithm to find canonical blobs to mimic human queries. In our work we also use the idea that clusters contain visually similar images but we use them in a post-processing step instead of forming the initial queries. The paper is organized as follows. First, the features used are discussed in Section 2. Then, a new algorithm for image retrieval is introduced in Section 3, which is followed by the summary of a graph–theoretic clustering algorithm in Section 4. Experiments and results are presented in Section 5. Finally, conclusions are given in Section 6.

2

Feature Extraction

The textural features that are used were described in [1,2]. The feature vector consists of two sets of features which intend to perform a multi-scale texture analysis which is crucial for a compact representation in large databases containing diverse sets of images. The first set of features are computed from the line-angleratio statistics which is a texture histogram of the angles between intersecting line pairs and the ratio of the mean gray levels inside and outside the regions spanned by those angles. The second set of features are the variances of gray level spatial dependencies and are computed from the co-occurrence matrices for different spatial relationships. Each component in the 28-dimensional feature vector is normalized to the [0, 1] interval by an equal probability quantization.

3

Image Retrieval

After computing the feature vectors for all images in the database, given a query image, we have to decide which images in the database are relevant to it. In most of the retrieval algorithms, a distance measure is used to rank the database images in ascending order of their distances to the query image, which is assumed to correspond to a descending order of similarity. In our previous work [1,2] we defined a likelihood ratio to measure the relevancy of two images, one being

A Graph–Theoretic Approach to Image Database Retrieval

343

the query image and one being a database image, so that image pairs which had a high likelihood value were classified as “relevant” and the ones which had a lower likelihood value were classified as “irrelevant”. The distributions for the relevance and irrelevance classes were estimated from training sets and the likelihood values were used to rank the database images. We believe that a retrieval algorithm should be able to retrieve images that are not only similar to the query image but also similar to each other, and formulate a new retrieval algorithm as follows. Assume we query the database and get back the best N matches. Then, for each of these N matches we can do a query and get back the best N matches again. Define S as the set containing the query image and at most N 2 + N images that are retrieved as the results of the original query and N additional queries. Then, we can construct a graph with the images in S as the nodes and can draw edges between each query image and each image in the retrieval set of that query image. We call these edges the set R where R = {(i, j) ∈ S × S | image j is in the retrieval set when image i is the query}. The distances between images which correspond to two nodes that an edge connects can also be assigned as a weight to that edge. We want to find the connected clusters of this graph (S, R) because they correspond to similar images. The clusters of interest are the ones that include the original query image. The ideal problem now becomes finding the maximal P , where P ⊆ S such that P × P ⊆ R. This is called a clique of the graph. The images that correspond to the nodes in P can then be retrieved as the results of the query. An additional thing to consider is that the graph (S, R) can have multiple clusters. In order to select the cluster that will be returned as the result of the query, additional measures are required. In the next section we define the term “compactness” for a set of nodes. The cluster with the maximum compactness can then be retrieved as the final result. If more than one such cluster exist, we can select the one with the largest number of nodes or can compute the sum of the weights of the edges in each of the clusters and select the one that has the minimum total weight. This method increases the chance of retrieving similar images by not only ensuring that the retrieved images are close to the query image, but also adding another constraint that they should be close to each other in the feature space. In the next section we describe a graph–theoretic clustering algorithm which is used to find the clusters. Section 5 presents experimental results.

4

Graph–Theoretic Clustering

In the previous section, we proposed that cliques of the graph correspond to similar images. Since finding the cliques is computationally too expensive, we use the algorithm by Shapiro and Haralick [6] that finds “near-cliques” as dense regions instead of the maximally connected ones. Another consideration for speed is to compute the N -nearest neighbor searches offline for all the images in the database so that only one N -nearest neighbor search is required for a new query, which is the same amount of computation for the classical search methods.

344

Selim Aksoy and Robert M. Haralick

In the following sections, first we give some definitions, then we describe the algorithm for finding dense regions, and finally we present the algorithm for graph–theoretic clustering. The goal of this algorithm is to find regions in a graph, i.e. sets of nodes, which are not as dense as major cliques but are compact enough within some user specified thresholds. 4.1

Definitions

– (S, R) represents a graph where S is the set of nodes and R ⊆ S × S is the set of edges. – (X, Y ) ∈ R means Y is a neighbor of X. The set of all nodes Y such that Y is a neighbor of X is called the neighborhood of X and is denoted by Neighborhood(X). – Conditional density D(Y |X) is the number of nodes in the neighborhood of X which have Y as a neighbor; D(Y |X) = #{N ∈ S | (N, Y ) ∈ R and (X, N ) ∈ R}. – Given an integer K, a dense region Z around a node X ∈ S is defined as Z(X, K) = {Y ∈ S | D(Y |X) ≥ K}. Z(X) = Z(X, J) is a dense region candidate around X where J = max{K | #Z(X, K) ≥ K}. – Association of a node X to a subset B of S is defined as A(X|B) =

#{Neighborhood(X) ∩ B} , #B

0 ≤ A(X|B) ≤ 1.

(1)

– Compactness of a subset B of S is defined as C(B) =

1
A(X|B) , #B

0 ≤ C(B) ≤ 1.

(2)

X∈B

4.2

Algorithm for Finding Dense Regions

To determine the dense region around a node X, 1. Compute D(Y |X) for every other node Y in S. 2. Use the densities to determine a dense–region candidate set for node X by finding the largest positive integer K such that #{Y | D(Y |X) ≥ K} ≥ K. 3. Remove the nodes with a low association (determined by the threshold MINASSOCIATION) from the candidate set. Iterate until all of the nodes have high enough association. 4. Check whether the remaining nodes have high enough average association (determined by the threshold MINCOMPACTNESS). 5. Check the size of the candidate set (determined by the threshold MINSIZE). When MINASSOCIATION and MINCOMPACTNESS are both 1, the resulting regions correspond to the cliques of the graph.

A Graph–Theoretic Approach to Image Database Retrieval

4.3

345

Algorithm for Graph Theoretic Clustering

Given dense regions, to find the clusters of the graph, 1. Merge the regions that have enough overlap, determined by the threshold MINOVERLAP, if all of the nodes in the set resulting after merging have high enough associations. 2. Iterate until no regions can be merged. The result is a collection of clusters in the graph. Note that a node can be a member of multiple clusters because of the overlap allowed between them.

5

Experiments and Results

The test database consists of 340 images which were randomly selected from a database of approximately 10,000 aerial and remote sensing images. The images were grouped into 7 categories; parking lots, roads, residential areas, landscapes, LANDSAT USA, DMSP North Pole and LANDSAT Chernobyl, to form the groundtruth. 5.1

Clustering Experiments

The first step of testing the proposed retrieval algorithm is to check whether the clusters formed by the graph–theoretic clustering algorithm are visually consistent or not. First, each image was used as a query to search the database, and for each search, N top-ranked images were retrieved. Then, a graph was formed with all images as nodes and for each node N edges correspond to its N topranked images. Finally, the graph was clustered by varying the parameters like N , MINASSOCIATION and MINCOMPACTNESS. In order to reduce the possible number of parameters, MINSIZE and MINOVERLAP were fixed as 12 and 0.75 respectively. The resulting clusters can overlap. This is a desired property because image content is too complex to be grouped into distinct categories. Hence, an image can be consistent with multiple groups of images. To evaluate the consistency of a cluster, we define the following measures. Given a cluster of K images, CorrectAssociationk =

#{i | GT(i) = GT(k), i = 1, . . . , K} K

(3)

gives the percentage of the cluster that image k is correctly associated with, where GT(i) is the groundtruth group that image i belongs to. Then, consistency is defined as Consistency =

K 1
CorrectAssociationk . K

(4)

k=1

To select the best set of parameters, we define a cost function Cost = 0.7(1 − Consistency) + 0.3(Percentage of unclustered images)

(5)

346

Selim Aksoy and Robert M. Haralick

1

0.9

0.8

0.7

Consistency

0.6

0.5

0.4

0.3

0.2

0.1

0

0

0.1

0.2

0.3

0.4 0.5 0.6 0.7 Percentage of Unclustered Images

0.8

0.9

1

Fig. 1. Consistency vs. Percentage of unclustered images for N ∈ {10, . . . , 70}, MINCOMPACTNESS ∈ {0.3, . . . , 1.0}, MINASSOCIATION ∈ {0, . . . , MINCOMPACTNESS}, MINSIZE = 12, MINOVERLAP = 0.75. Dashed lines correspond to the minimum cost. and select the parameter set that minimizes it. Here Consistency is averaged over all resulting clusters. Among all possible combinations of the parameters given in Figure 1, the best parameter set was found as {N, MINCOMPACTNESS, MINASSOCIATION} = {15, 0.6, 0.4}, corresponding to an average Consistency of 0.75 with 6% of the images unclustered. Example clusters using these parameters are given in Figure 2. We observed that decreasing N or increasing MINCOMPACTNESS or MINASSOCIATION increases both Consistency and Percentage of unclustered images. 5.2

Retrieval Experiments

We also performed experiments using all of the 340 groundtruthed images in the database as queries and, using the parameter set selected above, retrieved images in the clusters with the maximum compactness for each query. For comparison, we also retrieved only 12 top–ranked images (no clustering) for each query. Example queries without and with clustering are shown in Figures 3 and 4. We can observe that some images that are visually irrelevant to the query image can be eliminated after the graph–theoretic clustering. An average precision of 0.78 (compared to 0.73 when only 12 top-ranked images are retrieved) for the whole database showed that approximately 9 of the 12 retrieved images belong to the same groundtruth group, i.e. are visually similar to the query image. We also observed that, in order to get an improvement by clustering, the initial precision before clustering should be large enough so that the graph is not dominated by images that are visually irrelevant to the query image. In our experiments, when the initial precision was less than 0.5, the average precision after clustering was 0.19. For images with an initial precision greater than 0.5, the average precision after clustering was 0.93. The better the features are, the larger the improvement after clustering becomes.

A Graph–Theoretic Approach to Image Database Retrieval

(a) Consistency = 1

(b) Consistency = 1

(a) Using only 12 top–ranked images.

(b) Using graph–theoretic clustering.

347

Fig. 2. Example clusters for N =15, MINCOMPACTNESS=0.6, MINASSOCIATION=0.4, MINSIZE=12, MINOVERLAP=0.75.

Fig. 3. Example query 1. Upper left image is the query. Among the retrieved images, first three rows show the 12 most relevant images in descending order of similarity and the last row shows the 4 most irrelevant images in descending order of dissimilarity. When clustering is used, only 12 images that have the smallest distance to the original query image are displayed if the cluster size is greater than 12.

348

Selim Aksoy and Robert M. Haralick

(a) Using only 12 top–ranked images.

(b) Using graph–theoretic clustering.

Fig. 4. Example query 2.

6

Conclusions

This paper addressed the problem of retrieving images that are quite irrelevant to the query image, which is caused by the assumption that the features are always able to locate visually similar images close enough in the feature space. We introduced a graph–theoretic approach for image retrieval by formulating the database search as a problem of finding the cliques of a graph. Experiments showed that some images that are visually irrelevant to the query image can be eliminated after the graph–theoretic clustering. Average precision for the whole database showed that approximately 9 of the 12 retrieved images belong to the same groundtruth group, i.e. are visually similar to the query image.

References 1. S. Aksoy and R. M. Haralick. Textural features for image database retrieval. In Proc. of IEEE Workshop on CBAIVL, in CVPR’98, pages 45–49, June 1998. 342 2. S. Aksoy, “Textural features for content-based image database retrieval,” Master’s thesis, University of Washington, Seattle, WA, June 1998. 342 3. C. Carson et al.. Color- and texture-based image segmentation using EM and its application to image querying and classification. submitted to PAMI. 342 4. P. Felzenszwalb and D. Huttenlocher. Image segmentation using local variation. In Proc. of CVPR, pages 98–104, June 1998. 342 5. B. Huet and E. Hancock. Fuzzy relational distance for large-scale object recognition. In Proc. of CVPR, pages 138–143, June 1998. 342 6. L. G. Shapiro and R. M. Haralick. Decomposition of two-dimensional shapes by graph-theoretic clustering. IEEE PAMI, 1(1):10–20, January 1979. 343 7. J. Shi and J. Malik. Normalized cuts and image segmentation. In Proc. of CVPR, pages 731–737, June 1997. 342 8. Zhenyu Wu and Richard Leahy. An optimal graph theoretic approach to clustering: Theory and its application to image segmentation. IEEE PAMI, 15(11):1101–1113, November 1993. 342

Motion Capture of Arm from a Monocular Image Sequence Chunhong PAN and Songde MA Sino-French Laboratory in Computer Science, Automation and Applied Mathematics National Laboratory Of Pattern Recognition Institute of Automation, Chinese Academy Sciences Sciences [email protected]

Abstract. The paper develops a new motion capture method from a monocular sequence of 2D perspective images. Our starting point is arm motion. We first extract and track the feature points from image sequence based on watershed seqmentation and Voronoi diagram, then by rigidity constraint and motion modelling constraint we use motion analysis method to yield 3D information of feature points. Finally the obtained data is attuned to simulate motion of model. A experiment with real images are included to demonstrate the validity of the theoretic results.

1

Introduction

Recently, human motion analysis has increased in importance for visual communications, virtual reality, animation and biomechanics [5,1,2]. While generating appealing human motion is the central problem in virtual reality and computer animation. A wide variety of techniques are presented in the process of creating a complex animation. Generally speaking, these techniques can be grouped into three main classes: keyframing [10], procedure [8], and motion capture [9]. The generated animations by these techniques are so-called keyframe animation, procedural animation, and motion capture animation. Up to now motion capture is the only effective method to generate the human arbitrary motion. Motion capture employs special special sensors or markers to record the motion of a human performer by multiple cameras from different directions. The recorded data is then used to generate the motion for an animation. The system is able to estimate position with an accuracy of 0.1 diameter. However, to achieve such accuracy, it is necessary to have a complicated system composed of many special markers, and 4-8 cameras that need to be accurately calibrated. Furthermore, many sensors have to be worn by a person all the time, it is stressful and hard to handle in many applications which limits the use of the system. The determination of 3D motion by analysis of two or more frames captured at different instants is a great research topic in computer vision [12]. Generally there are two distinct ways. The first approach is based on optic flow [3]. By computing optic flow of images the 3D motion of a rigid body can be determined. The second method depends on correspondence of features [7,6]. By extracting Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL’99, LNCS 1614, pp. 349–357, 1999. c Springer-Verlag Berlin Heidelberg 1999


350

Chunhong Pan and Songde Ma

a small number of features in images corresponding to 3D object features and then using correspondence of these features in successive images the parameters of motion can be obtained in theory. In this paper we study the unconstrainted problem of estimating the 3D motion of the human arm based on the determination of the 3D motion. The obtained information are attuned to generate animation of arm. We first extract and track joints in the images, and determine the 3D motion of the joints from these 2D correspondences. It is known that extracting and tracking feature points in the grayscale sequence images or the segmented images are very difficult. While based on the 1D stick figure we can track the joint points conveniently in sequence images. In order to obtain good correspondences of feature points in sequence images, we made pre-processing on the grayscale sequence images. First using the gradient watershed segmentation we get binary edge images, then based on Voronoi Diagram we skeletonize the binary edge images to obtain a 1D figure stick. Using the 1D stick figure we can obtain a good correspondences of joint points over image sequence manually.

2 2.1

Pre-Processing of Image Sequence Motion Segmentation Based on Gradient Watershed

As a flexible, robust and efficient tool, watershed transformation has been widely applied on applications such as grayscale image segmentation, coding, etc. Here in order to obtain the robust area extraction , we use the gradient watershed segmentation under geodesic reconstruction and the stratification transformation [14]. As a non-linear tool for image processing, gradient watershed segmentation has shown its special ability to extract areas with well correspondence to objects in images. Efficient algorithm are also proposed in the literature [13]. Nevertheless, simply applying the watershed transformation on image will definitely result in over-segmentation. To eliminate it, the scale space of gradient watershed is necessary. In order to establish this scale space, two methods were proposed, i.e. the multiscale geodesic reconstruction filter, and stratification transformation. The watershed transformation W S on intensity image I(M, N ) can be denoted on an 8-connectivity digital grid as W S(I) = {CB, W A}, where W A is a local-connected subset of I that composes the watershed arc and CB is the set of catchment basin. Without loss of generality processing, we allow catchment basin to include the conjoint watershed arcs and simplify the definition of watershed transformation as: W S(I) = {CBi | i = 1, 2....p},

I = ∪CBi ;

(1)

Suppose Rmin (I) = ∪1<=i<=p Ni (gi(0) ) , in which Ni is the individual regional minima with intensity gi(0) , we have: CBi = CBi (gi(0) ) = Γ ZI (Ni );

(2)

Motion Capture of Arm from a Monocular Image Sequence

351

where Γ ZI is the transformation of geodesic influence zone on intensity surface I. So for CBi , Ni is its only regional minima. When discussing the scale space of gradient watershed, the multiscale filter Φn we use is the combination of opening by reconstruction of erosion and closing by reconstruction by dilation, i.e., Φn (I) = γnrec (ϕrec i (I));

(3)

The feature(the catchment basin of watershed transformation) extractor of scale n is ∇ ∇ W Sn∇ (I) = W S(∇m (Φn (I))) = {CBi(n) | 1 ≤ i ≤ pn }; (4) here ∇m is the morphological gradient operator defined as: ∇m (f ) = δ1 (f ) − 1 (f ).

(5)

According to Eq.(2), suppose the gradient watershed partition of an intensity images I is ∇ W S0∇ (I) = {CBi(0) (gi(0) ) | 1 ≤ i ≤ p0 )}; (6) for a point I[x, y], its stratification transformation ψs (I[x, y]) is defined as ∇ (gi(0) ); 1 ≤ i0 ≤ p0 , ψs (I[x, y]) = gi(0) , if [x, y] ⊂ CBi(0)

(7)

From Eq.(4) and Eq.(5), the segmentation operator we use is defined as W Sn∇ (ψ(I)) = W S(∇m (φn (ψ(I)))); 2.2

(8)

Skeletonization Based on Voronoi Diagram

Skeletonization in the plane denotes a process which transforms a 2D object into 1D line representation, comparable to a stick figure. The skeletonization algorithm we use is first based on the Voronoi diagram(VD) [11]. The Voronoi diagram is a well-known tool in Computational Geometry. Let pi and pj denotes two elements from a set Ω of N points in the plane. The locus of all points which are closer to pi than to pj is a half-plane H(pi , pj ) containing pi which is bounded by the perpendicular bisector of pi pj . Consequently, the locus Vi (pj ) of points closer to pi than to any other point pj amounts to a convex Voronoi polygon and can be computed by intersecting at most N-1 half planes H(pi , pj ); First-order Voronoi diagram can be expressed as V1 (pi ) =




H(pi , pj );

(9)

i =j

then the first-order VD of Ω is the collection of all Vi (pi ):  V1 (pi ); V or1 (Ω) = pi ⊆Ω

(10)

352

Chunhong Pan and Songde Ma

The Voronoi diagram V is made of all the edges of these areas [4]. By construction, all the points P in V are located at the same distance of 2 nearest sites A and B on the boundary of P. Theoretically, the skeleton S of P can be also defined as the median line between opposite boundaryes of P, thus S is a subset B between the nearest sites A and B of ν. Here by calculating the angle AV used for defining the Voronoi diagram and V, a point of the boundary between two voronoi region owned by A and B, we can decide if it belongs or not to S. When A and B are located on the same sides of the boundary of the pore space, B will be small, and When A and B are located on the opposite the angle AV sides, the angle will be large. A point V belongs to the median line if the angle B is large enough, which can be represented in the following form: AV B > K} S = {V /V ∈ ν, AV

(11)

where K is an empirical tuning constant which is chosen to avoid disconnected areas in the median line as well as ”barbules”, and A, B are the nearest sites of V in ν .

3

Performance Analysis with Constraint Fusion

Articulated model of arm: We proposed the stick model for human arm, which it consists of 3 joint points and the links between adjacent joint points are rigid. In the articulated model, the points represent model features and segments represent rigidity constraints. The model has 7 degrees of freedoms(DOF). Here in order to keep the model as simple as possible and to have as few degrees of freedom as possible, we do not model the shape of the hand and simply assume it to extend along the axis of the forearm. In our approach, we first use rigid constraints between the feature points of the arm model. it means that the distance between certain feature points of the model should remain constant during the whole tracking process. Furthermore, from anthropometric data we know that lengths of links are fractions of body height. For example, the relative lengths of the forearm and upperarm are 0.206 and 0.22 of body height respectively, So the constraint about length relationship between the forearm and upperarm can also be added to our algorithm. Motion analysis and rigidity constraint: The image geometry of the multiple-frame motion analysis problem is shown in Figure 2. A pinhole camera model is used. Images are taken of a moving object at evenly-spaced times. (j) The feature point Pi denotes the ith point at time tj . Its 3D space co(j) (j) (j) (j) (j) ordinates will be given by Pi (Xi , Yi , Zi ). The projection of Pi onto (j) (j) the image plane will be denoted by P = (xi , yi ). We use subscript i to denote the ith point, the superscript (j) denotes ”primes” and refers to points at




(2) (3) tj (zi = zi , zi = zi ). When a pinhole camera is being used, the object and image point coordinate are related by (for j = 0, 1, 2, ... ): (j)

(j)

xi

=

f Xi

(j)

Zi

(j)

;

(j)

yi

=

f Yi

(j)

Zi

;

(12)

Motion Capture of Arm from a Monocular Image Sequence 1 0 0 1

shoulder-3DOF

11 00 00 11 00 11 00 11 00 11

353

11 00 00 11

elbow_1DOF wrist_2DOF

0 1 0 1 1 0 0 111 0 1 1 000 1 0 00 1 11 1 00 0 1 00 11 00 11 0 1 00 11 0 1 0 1 11 00 00 11 00 11

1 0 0 1 0 1

Fig. 1. An articulated model consisting of arm, the points represent model features (j)

(j)

Here f is the focal length of the camera. Images coordinate (xi , yi ) can be known from image plane when frames are sampled at different time, and correspondences of points over all the views are easily achieved due to checking out manually based on 1D figure stick. Thus the problem can be stated as: P0 y Z P1

P0’

P1’ o

Y

O

x

X

Fig. 2. Basic perspective imaging geometry for motion analysis Given correspondences of points throughout all the frames, how to (j) (j) (j) (j) find Pi (Xi , Yi , Zi ). It is well-known that from a monocular image sequence, the 3D depth of an isolated point is impossible to recover only by uncalibrated camera in rigidity constraints, In our analysis, we attempt to track only the arm, for simplicity, we keep the 3D position of the shoulder fixed in space and first calculate the relative 3D coordinates of another endpoints. In this case, P0 is stationary throughout (j) all the views. Equivalently, Z0 is equal for all j. Here we use an approach of decomposition. The arm under analysis is decomposed into two parts, each containing a single link: forearm and upperarm. Since we have assumed that the shoulder is fixed the upperarm is simpler part than forearm. We first analyze the motion of the upperarm and then propagate the analysis to forearm. By using the approach we can obtain 3D motion information of arm. First we have rigidity constraints over frames: (j)

P0 − P1 = P0

(j)

− P1 = l;

f or j = 0, 1, 2...

354

Chunhong Pan and Songde Ma (j)

(j)

l denotes the length of the link, and P0 , P1 are two endpoints of any link. For upperarm due to position of shoulder fixed, usimg Eqs.(12) this can be written as: (x21 + y12 + f 2 )Z12 − 2(x0 x1 + y0 y1 + f 2 )Z0 Z1 = (j)2 (j)2 (j)2 (j) (j) (13) (x1 + y1 + f 2 )Z1 − 2(x0 x1 + y0 y1 + f 2 ) (j) Z0 Z1 = l (j)

In above equations the depths Z0 and Z1 are unknown variables for j = 0, 1, 2, .... In fact if the shoulder is fixed the depth Z0 is only the scale. Our aim (j) is to calculate the relative depths Z1 over views. Motion modelling and optimal constraint: To obtain the unique 3D infomation of feature points over frames. A motion model is needed, here we assume that the arm moves with smooth motion during a short period of time. We know that when the shoulder fixed, the feature point of the elbow always moves in surface of sphere with radius of length of upperarm. Let dj be the (j) (j+1) distance between P1 and P1 when the feature point of the elbow moves (j)

from P1

(j+1)

to P1

(j)

(j+1)

, and θj be the angle between the line P1 P1

(j+1) (j+2) P1 P1 .

and the

(j) P1

line Here denotes point trajectory of the elbow over views. The smooth motion means that the variances of the distance and the angle above have the minimal values. This can be represented as follow: J1 = min

N −1 

|dj − d|;

j=1

J2 = min

N −2 

|θj − θ|;

j=1

where: d=

N −1 N −2   1 1 dj and θ = θj (N − 1) j=1 (N − 2) j=1

It is an optimization problem with rigidity constraint which can be solved by dynamic programming. Since the shoulder is fixed, the its depth Z0 can be seen as a scale, while other endpoint of upperarm satisfies the rigidity constraint which means it moves on the surface of the sphere. By adding motion modelling to this approach we obtain relative 3D coordinates of feature point of elbow by programming. When the relative 3D information of two endpoints of the upperarm is determined by the above method the 3D coordinates of one endpoint of the adjacent forearm become known, and the length of the upperarm up to scale become also known. From above rigidity constraint equation this yield a two-degree polynomial equation with one unknown variable for each frame. Generally there are two solutions to a two-degree polynomial equation. We lay above optimal constraints on the unknown endpoint of the forearm, furthermore the depths Z (j) must be physically realizable solutions: Z (j) > 0, so we obtain a unique solution for Z (j)

Motion Capture of Arm from a Monocular Image Sequence

4

355

Experiment

The human model which consists of the arm was constructed from rigid links connected by rotary joints with one, two and three degrees of freedoms. The dynamic model shown in Figure 3 has 15 body segments which were represented by some rigid generalized cylinders. The points of contact between the rigid links were skinned automatically.

Fig. 3. Model used to perform experiment In order to acquire to real data to test our algorithms, we used a video camera to record scenes of arm moving. We videotaped the motion in a unconstrainted scene, and generated a image sequence. The figure 5 give some sampled frames from a database sequence. Then we pre-processed the image sequence by using segmentation based on watershed and skeletonization based on Voronoi Diagram discussed above. The figure 6 and 7 show the binary edge images and skeleton images respectively. From the skeleton image sequence, we obtained a set of 2D trajectories of feature points by manually tracking the joint points such as elbow and wrist. The (x,y) coordinates of each dot of all the sample’s frames is used to analyze motion. The figure 4 shows 3D trajectories of elbow and wrist up to scale. Obviously when the length of the upperarm or forearm is determined one can get a real 3D coordinates of elbow and wrist. While once the body height is known the lengths of arm can be easily calculated. Finally the scale 3D coordinates of elbow and wrist obtained from above approach are used to generate motion of the arm modelled by the rigidity generalized cylinders. Figure 8 shows the simulated motion of the arm. Due to noise, correspondences with error, and approximate assumption on articulated model, in fact it is impossible to satisfy the rigidity constraints exactly. But our motion model is based on the smooth movement, and we obtain the 3D data of the joints by the optimal numberical search, so when the movement is small the method is effective.

5

Conclusion

A new motion capture method based on feature point correspondence over frames is proposed. We first pre-process the image sequence, and obtain 1D skeleton images. By rigidity constraint and motion modeling we yield the 3D information of feature points. Experimental results show that the method is efficient.

356

Chunhong Pan and Songde Ma

Fig. 4. Motion trajectories of elbow and wrist

Fig. 5. A sampled image sequence with arm moving

Fig. 6. The segmented binary edge images

Fig. 7. The skeleton of binary edge image

Motion Capture of Arm from a Monocular Image Sequence

357

Fig. 8. Simulated motion of a human arm

References 1. Devi L. Azoz Y. and Sharma R. Tracking hand dynamics in unconstrainted environments. In Proceedings of IEEE International Conference on Computer Vision, pages 274–280, 1998. 349 2. Barsky B. Badler N. and Zeltzer D. Making Them Move. Morgan Kaufmann, 1991. 349 3. Horn B.K.P. and Schunk B.G. Determining optical flow. Artificial Intelligence, 17:185–203, 1981. 349 4. Yu Z.Y. Delerue J.F., Perrier E. and Velde B. New algorthms in 3d image analysis and their application to the measurement of a spatialized pore size distribution in soils. to appear in Journal of Physics and Chemistry of the Earth, 1998. 352 5. Ureslla E. Goncalves L., Bernardo E.D. and Perona P. Monocular tracking of the human arm in 3d. In Proceedings of IEEE International Conference on Computer Vision, pages 764–770, 1995. 349 6. Netravali A.N. Holt R.J., Huang T.S. and Gian R.J. Determining articulated motion from perspective views: A decomposition approach. Pattern Recognition, 30:1435–1449, 1997. 349 7. Robert J.H. and Netravali A.N. Number of solutions for motion and structure from multiple frame correspondence. Intel. J. of Computer Vision, 23:5–15, 1997. 349 8. Hodgins J.K. Biped gait transition. In Proceedings of the IEEE International Conference on Robotics and Automation, pages 2092–2097, 1991. 349 9. David J.S. Computer puppetry. IEEE Computer Graphics and Applications, 18:38– 45, 1998. 349 10. Shoemaker K. Animation rotation with quaternion curves. In Proceedings of SIGGRAPH’85, pages 245–254, 1985. 349 11. Ogniewicz R.L. and Kubler O. Hierarchic voronoi skeletons. Pattern Recognition, 28:343–359, 1995. 351 12. Huang T.S. and Netravali A.N. Motion and structure from feature correspondence: A review. In proc. IEEE, volume 88, pages 252–258, 1994. 349 13. L. Vincent. Morphological grayscale reconstruction in image analysis: Applications and efficient algorithms. IEEE Transaction on Image Processing, 2:176–201, 1993. 350 14. Songde Ma Yi Li, Ming Liao and Hangqing Lu. Scale space of gradient watershed. to appear in Journal of Image and Graphics, 1998. 350

Comparing Dictionaries for the Automatic Generation of Hypertextual Links: A Case Study Isabella Gagliardi and Bruna Zonta CNR-ITIM Via Ampere 56, 20131 Milano Tel +39 02 7064 3270 / 53, Fax +39 02 7064 3292 {isabella, bruna}@itim.mi.cnr.it

Abstract. There is a great need for tools that can build hypertexts from "flat" texts in an automatic mode, assigning links. This paper addresses the problem of the automatic generation of similarity links between texts that are relatively homogeneous in form and content, such as the cards of an art catalogue. The experimentation it describes has compared the results obtained using weighted and unweighted supervised dictionaries with those produced using weighted and unweighted automatic dictionaries.

Introduction There is a great need for tools that can build hypertexts from "flat" texts in an automatic, or at least partially automatic mode, especially when the hypertexts concerned have a very high number of nodes and links. The work needs to be done more rapidly, and it must be sytematized, in part to avoid involuntarily following different philosophies of thought in assigning links. A study of the state of the art, through a bibliographical search facilitated by the great number of articles investigating the problem that can be found in journals [11] and proceedings, as well as on Internet, shows that the effectiveness of the algorithm depends greatly on the characteristics of the texts to which it is applied. The problem and the solutions proposed in the literature are present in two extremes: strongly structured documents, that is, equipped with indexes, chapters, subdivisions, crossreferences, etc. on the one hand; and linear, unstructured documents on the other. In this paper we address the problem of the automatic generation of associative links between texts that are relatively homogeneous in form and content, such as the cards of a catalogue describing works of art. The basic idea is that the presence in two cards of a certain number of common terms in proportion to the sum of their terms indicates that these cards can be linked to each other for their "conceptual similarity", and that the corresponding objects can as a consequence also be compared to each other, for their "perceptual similarity" [16]. In this process dictionaries, that is, the set of terms used to establish whether the two cards can be linked, and in what measure, play an important role. We have prepared a series of dictionaries, differing in the way they are built and in their Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL'99, LNCS 1614, pp. 358-366, 1999.  Springer-Verlag Berlin Heidelberg 1999

Comparing Dictionaries for the Automatic Generation of Hypertextual Links

359

semantic content - supervised, supervised and weighted, automatic, automatic and weighted - and these dictionaries have been tested with an algorithm that calculates the similarity. The algorithm, the dictionaries, and the results obtained from the application of each are presented in this paper. In particular, the first section discusses related works, together with concepts of help in understanding our work; the second describes our approach, while the third contains details of the experiments performed, and preliminary results. The section Conclusion and Future Developments points to more, in-depth research on the correspondence between the conceptual similarities of the cards and the perceptual similarities of the objects they describe. An advance in this direction would make it possible, starting from texts, to automatically introduce links of comparison between similar images in multimedia art catalogues. This study is part of the project of the Italian National Research Council (CNR) on "Beni culturali: metodi e strumenti per la creazione di archivi multimediali nel settore della ceramica" (Cultural Resources: methods and tools for the creation of multimedia archives in the ceramic sector) developed at ITIM in Milan. 1. Related works The increasing availability of collections of on-line textual documents too large to allow the manual authoring and construction of a hypertext, is the main reason for the current interest in the study and implementation of fully or partially automated techniques. A pioneering few [9,12,13] began research in this field before hypermedia applications became as widespread as they are today. In 1995 a workshop on "IR and the Automatic Construction of Hypermedia" was held during the ACM SIGIR conference, and in 1997 the authoritative journal IP&M published a monographic issue on the subject [11]. In [3], Agosti supplied the key notions involved in the automatic construction of hypertexts, together with a brief selection of experiments conducted in this field. Salton et al. [14] proposed a technique that can be used to create links between text segments and practically construct a hypertext at retrieval time. They had to deal with the problem of identifying internally consistent fragments from available texts, and used a graph representation to show the results. More recently, Allan [4] has addressed in particular the problem of managing the different types of links. The technique he proposes provides a wholly automatic method for gathering documents for a hypertext, associating the set, after identifying the type of link, with its description. Document linking is based upon IR similarity measures with adjustable levels of strictness. Agosti & Crestani [2] have proposed a design methodology for the automatic generation of an IR hypertext, starting from a collection of multimedia documents and using well established IR techniques. Tudhope [17] has designed a semantic hypermedia architecture, in which the semantic similarity of information units forms the basis for the automatic construction of links integrated into hypermedia navigation. This architecture has been implemented in a prototype application: "A museum of social history".

360

Isabella Gagliardi and Bruna Zonta

In 1995 the authors designed and implemented a hypermedia Information Retrieval application on CD-ROM: “Sixteenth Century Genoese Textiles”[6,7]. Identification of the hypertextual links is based upon a pattern matching method, in two different contexts: • given a glossary, the catalogue cards are automatically connected to the items in the glossary, forming referential links. The results have been most satisfactory; • the texts of the cards indicate cross-references and comparisons, such as "different techniques, but similar type of decoration: Savona, Sanctuary of Our Lady of Charity, Museo del Tesoro, no. 183 [card in the same catalogue]; Mayer van Den Bergh Museum, inventory no. 1655 [referring to a different catalogue]". In this case the program links the card contained in the same catalogue, but ignores the second. More generally, the program's task is to associate only card within the archives, ignoring cross-references to other catalogues. This algorithm has been moderately successful. 2. Our Approach Here The present procedure has been designed to automatically define the links among textual cards in multimedia art catalogues, where every card corresponds to an image of an object. Most of the cards contain partially structured texts, some of which in a predefined format (name of the object, shape, dimensions, date, brief technical notes, etc.), and the rest in free text describing the subject represented in the object and how the object is decorated. A card of this type is rarely more than a page long, and usually of the length and style common to the catalogue. The free text tends to be concise, with few, and at any rate not significant repetitions. From the point of view of automatic management, the parts in fixed text are easily subdivided into fields, while the uniform length and style of the parts in free text constitute a good premise for similarity comparison and any assigning of links. The fact that there are few repetitions eliminates the problem of the frequency of terms in the text of each card. Unfortunately, these cards are in general compiled by various people, with different degrees of expertise, over a period of time which may be very long. Consequently, the terminology is usually not homogeneous, and the texts would have to be normalized, or classified, to be considered a satisfactory basis for effective automation. The procedure described here has been designed to calculate the similarity between cards that have already been normalized: when this similarity is found, the two texts are connected by a link of the associative type. The similarity model used is the conceptual "contrast" type, which considers similarity an intersecting of features [16], in this case, of terms. The basic idea is that the presence of terms common to two different cards indicates that these can be considered similar to each other. The possible links thus identified are a function of the number of terms present in the two cards, and have a "strength" of (0,1). The similarity between two texts is defined by the number of terms in common in proportion to the total number of terms of the two cards.

Comparing Dictionaries for the Automatic Generation of Hypertextual Links

361

The model used has clearly suggested the choice of the well known formula proposed by Salton [15]: 2( wi termi ∩ w j term j ) simi , j = wi termi ∪ w j term j

to which weights could be assigned. wi is the weight associated with the termi throughout the catalogue, as we shall explain below. The results depend, obviously, upon the terms chosen for comparison. This choice can be made in two extreme manners: automatically, with the use of lists of stopwords, or manually by experts in the domain who indicate the more significant terms according to certain criteria. Each method produces a different dictionary, the automatic is the richer one, the supervised, the more specific. We have compared the results obtained using a supervised dictionary and a weighted supervised dictionary, with those obtained using an automatic dictionary. 3. The Experiment

The experiment is part of a CNR project for the preservation of "Cultural Resources": the objective has been to compare the effectiveness of an automatic dictionary with that of supervised dictionaries, to see whether and how much the results improved in the latter case. The art catalogue employed in the experiment was The Cora Donation: Medieval and Renaissance Ceramics of the International Museum of Ceramics of Faenza [1], containing over 800 cards describing as many objects. In the catalogue each ceramic object is represented by an image, in color or black and white, and described by a card in text. Figure 1 shows a typical catalogue card and the corresponding image. The image elements and the textual description of subject-decoration do not always correspond exactly. This is due primarily to the fact that the image is bi-dimensional, while the description refers to a three-dimensional object. The free text, that is, text describing the subject and/or the decoration of the object, was used to assign the links. Explicit references present in the field "Analogies" were ignored, since they had already been experimented on the Genoese Textiles Catalogue [6,7] We did use these references later to verify the quality of the links assigned with our procedure. For the same purpose, when the cross-reference was of the type "Decoration like that of the preceding piece", the original description was repeated. Various trials were run in the course of the experimentation, each with a different dictionary, and the results were then compared. The Supervised Dictionary (SD) was created by: • Extraction of the descriptors: over 1000 terms considered significant for the description of the subject and decoration were extracted manually. • Creation of the lexicon: these descriptors were reduced to about 700 by unifying the variations, which were essentially three in kind: − graphic (with/without quotation marks, upper/lower case, ...); − morphological- derived (singular/plural, name/adjective, name/diminutive, ...); − lexical ("composition/decoration", "writing/inscription/caption", "woman/female").

362

Isabella Gagliardi and Bruna Zonta

A taxonomy was also created, grouping the descriptors in categories, and these in turn in higher level categories, for a total of three levels. These categories were used in the experimentation with weights. The Weighted Supervised Dictionary (WSD) contains the same terms as the SD, but weights have been applied to these. The weight can be assigned automatically, on the basis of the number of times the term occurs in the entire collection, following well-established procedures of Information Retrieval (IR), or manually, considering the importance of the term in the domain, or in the collection, regardless of its frequency. The former procedure was used here. Originally the adopted weights were: 1 for frequencies from 1 to 25, 0.75 for frequencies from26 to 50, 0.50 for frequencies from 51 to 75 0.25 for frequencies from 76 to 100 and 0.10 for frequencies over 100. To be able to assign an additional value to strongly characterizing term ("lion") as compared with those designating abstract decorative borders ("braid"), the above values have been diminuished. After many tests, the values have been set at 0.70, 0.55, 0.40, 0.25 and 0.10 respectively., so that adding the value of 0.30 to terms such as "lion" moves them two class higher. A procedure was also set up that allowed the user to assign a greater/lesser weight to some terms (or categories of terms) at query time, in order to express any specific interest. The Automatic Dictionary (AD) contains the words extracted by the ISIS Information Retrieval System. Consequently it is composed of all the terms present in the fields of the subject and decoration, except for the terms present in a stoplist. No stemming procedure has been applied, as no satisfactory algorithm is available for the Italian language. Since adding weights to this dictionary did not produce substantially different results, those results are not shown here. 4. Preliminary Results

Both the supervised and automatic dictionaries always assigned a similarity value of 1 when the texts were identical, and similarity values that varied, but were always among the highest, when the "Analogies" field contained cross-references. As for the differences among the various dictionaries, we saw that the SD generally assigned higher values than the SWD, but on the whole in accord with it. Because of the way it is structured, the AD considers a larger number of terms, but does not recognize equivalent terms, and consequently assigns lower and less differentiated values, without excluding, as the other dictionaries did, terms designating colors. This means that color plays a determinant role, not desired in this case, in the identification of links. The table below summarizes the number of terms in each dictionary, the number of times these appear on the cards, and the average number of terms per card, together with the number of cards with non-zero similarity. The SD gave no results for three of the cards for extraneous reasons, such as the fact that the description of the subject had not been entered in the proper place.

Comparing Dictionaries for the Automatic Generation of Hypertextual Links

No.of terms in the dictionary No. of times terms appear on the cards Average no. of dictionary terms per card No. of cards with non-zero similarity

AD 2040 17266 18 918

SD 690 5869 6 915

363

SWD 690 5869 6 915

We ran the program on all the cards in the catalogue. The following table lists, for each of the first seven cards, the card linked to it with the highest similarity value, and the value computed by each of the three dictionaries. CardId 001 002 003 004 005 006 007

AD linked card 775 246 754 697 747 696 847

sim. 0,285 0,451 0,562 0,571 0,382 0,450 0,297

SD linked card 632 751 192 692 654 689 282

sim. 0,454 0,500 0,444 0,500 0,428 0,461 0,444

WSD linked card 622 582 186 359 654 446 282

sim. 0,452 0,496 0,360 0,473 0,490 0,326 0,438

The following table summarizes the above results, where by minimum and maximum values is always meant the value of the card linked with the highest value (the first card in a decreasing order). Minimum similarity value Maximum value (excluding 1) Average similarity value Absolute interval of variation Interval of variation (excluding 1)

AD 0,142 0,965 0,609 0,858 0,823

SD 0,153 0,923 0,632 0,847 0,770

WSD 0,111 0,984 0,615 0,889 0,873

The eight observers who participated in the evaluation of the results obtained by the three dictionaries were shown the image of the object corresponding to the query card, and the images of the objects corresponding to the cards the different dictionaries linked with it. To facilitate their task and allow them to repeat it a sufficiently large number of times, only the card linked with the highest value by each dictionary was considered. Their task consisted in ranking by decreasing similarity values the images compared with the query. All eight found it difficult at first to restrict their attention rigorously to the task of evaluating only the similarity of subjects and decorations, ignoring shape, use, color, epoch, and style. However, with some experience the task was more readily performed. The images taken for comparison were 50, and the images compared with these 150, of which, however, only 97 were different, since the same image could be selected more than once, either in different contexts, or because the dictionaries agreed in the same context. The observers, who did not know which dictionaries had assigned which links, found the following similarities: order I II III

AD 8 15 27

SD 19 18 13

WSD 23 17 10

364

Isabella Gagliardi and Bruna Zonta

In the course of the experiment the observers found that some of the images called forth by supervised dictionaries did not at all resemble the query image. Analysis of the cards and the relative descriptors identified at least two reasons for these incongruities: either the descriptors referred to parts of the three-dimensional ceramic object that were not visible in the two-dimensional image, or the texts of the cards were not sufficiently appropriate and specific. The first drawback would be easily eliminated by using a series of images or films that show the object from various points of view. The second could be remedied, at least in part, by establishing norms for the compilation of the cards, together with a Thesaurus of the domain. The program was written in Microsoft Visual Basic 4.0(TM) with a control for management of the HTML pages. Microsoft Access 97(TM) was used for the database; the images of the objects in the catalogue, originally in TIF format, were converted to GIF format, and in general reprocessed by a Microsoft Photo Editor (TM). 5. Conclusions and Future Developments

This paper has presented a procedure for the automatic generation of hypertextual links among texts (cards) in art catalogues. We have used the very simple formula defined by G. Salton et al. (enriched with weights) to thoroughly examine the role of dictionaries in the successful realization of the links. In our experiment four different dictionaries were created and tested, and the results of three of these, the SD, WSD and AD (the WAD registered results very similar to those of the AD), were evaluated, on the basis of the corresponding images, by eight observers. As anticipated, better results were obtained with the supervised dictionaries than with the automatic dictionary. To effectively automate the entire procedure would take universal taxonomies such as ICONCLASS, or at least domain Thesauruses that could serve as guidelines for the drafter of the cards, and as filters for the automatic compilation of the dictionary. Integrating the algorithm presented here in a system that allows the automatic linking of images on the basis of their perceptual similarities may further improve results. Automatically classifying the pictorial content of the images to create the textimage links, or evaluating the "semantic" similarity of the images on the basis of lowlevel features alone (with a corresponding evaluation of their perceptual similarity), is generally an arduous task. But it will be possible in this application, despite the complexity of the images, because of the homogeneity of the database. In any case this will allow us to investigate any correlations between an image and the textual description of the represented object, or between the textual description of an object and the features that represent the corresponding image. A prototype system for the automatic creation of image-to-image and text-to-image links is now in an advanced state of construction. To create the text-to-image link we plan to apply the CART classification strategy. For the creation of image-to-image links, the set of features and measure for the perceptual-semantic similarity of the images will be selected by means of a relevance feedback mechanism which we are now in the process of developing.[8,10]

Comparing Dictionaries for the Automatic Generation of Hypertextual Links

365

ID code no.: 487 Object: Albarello Heights: 20 cm; diameter of base 8,4 cm Material: Majolica Origin: Montelupo Period: XVI century Subject: In the central area, there are two circular medallions, framed by festoons, containing the S. Bernardino IHS monogram; between the two medallions, a decoration with Persian palmettos. Glaze: orange, green, yellow, blue, red.

ID code no.: 488 Object: Mug Heights: 20,5 cm; diameter of base 8,5 cm Material: Majolica Origin: Montelupo Subject: Front: a circular cartouche with an ondulate border and containing the S. Bernardino I.H.S. monogram framed by a festoon. Sides: vertical bands with Persian palmettos. Under the handle the initial P. Glaze: brown, blue, orange and green.

Fig. 1. Id. Card no. 487

Fig. 3. Id card no. 488 linked to card 487, using the WSD, with the value of 0.862

D code no: 402 Object: Mug Material: Majolica Origin: Cafaggiolo Period: 1520 ca. Subject: front: a large circular medallion with a festoon and the S. Bernardino I.H.S monogram; the remaining surface is decorated with grotesques on a blue background; back: under the handle, a graffito monogram SP. Glaze: orange, gray, yellow, blue, green and brown. Analogies: Preservation: Good Fig. 2. Id card no. 402 linked to card 487, using the SD, with the value of 0.777

Code no.: 490 Object: Globular vase with two handles Subject: On both the faces circular medallions containing shield with palms and framed by festoons; surrounded by Persian palmettos. Under the handle the initial P. Glaze: blue, green., orange, yellow and red. Fig. 4. Id card no. 490 linked to card 487, using the AD, with the value of 0.666

366

Isabella Gagliardi and Bruna Zonta

References 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16.

17.

La donazione Galeazzo Cora: ceramiche dal medioevo al XIX secolo, Museo Internazionale delle Ceramiche in Faenza, Gruppo Editoriale Fabbri, 1985, Milano. Agosti M, F. Crestani, M. Melucci, “Design and implementation of a tool for the automatic construction of hypertexts for information retrieval”, Information Processing & Management, Vol. 32(4), pp. 459-476, 1996, Elsevier Science Ltd. Agosti M., F. Crestani, M. Melucci, “On the use of information retrieval techniques for the automatic construction of hypertext”, Information Processing & Management, Vol. 33(2), pp. 133-144, 1997, Elsevier Science Ltd. Allan J., “Building hypertext using information retrieval”, Information Processing & Management, Vol. 33(2), pp. 145-159, 1997, Elsevier Science Ltd. Carrara P., Della Ventura A., Gagliardi I., “Designing hypermedia information retrieval systems for multimedia art catalogues”, The New Review of Hypermedia and Multimedia, vol. 2, pp. 175-195, 1996. Carrara P., Gagliardi I., "A collection of antique Genoese textiles: an example of hypermedia Information Retrieval", poster session HIM 95, Konstanz (Germany), 57/4/95. Carrara P., Gagliardi I., Della Ventura, A. CD-ROM Tessuti Genovesi del Seicento, new version , 1996. Ciocca G., Schettini R., “Using a Relevance Feedback Mechanism to Improve Contentbased Image Retrieval”, Third International Conference on Visual Information Systems, Amsterdam, 2-4 June 1999 (submitted). Frisse M. E., “Searching for information in a hypertext medical handbook”, Communications of the ACM, Vol. 31(7), 1988. Gagliardi I., R. Schettini, G. Ciocca, “Retrieving Color Images by Content”, in Image And Video Content-Based Retrieval, February, 23rd 1998, CNR, Milano. Information Processing & Management, Vol. 33(2), 1997, Elsevier Science Ltd. Pollard R., “A hypertext-based thesaurus as a subject browsing aid for bibliographic databases”, Information Processing & Management, Vol. 29(3), pp. 345-357, 1993, Pergamon Press Ltd. Rada R., “Converting a Textbook to Hypertext”, ACM Trans. On Inf. Sys Vol. 10(3), pp. 294-315, July 1992. Salton G., A. Singhal, M. Mitra, C. Buckley, “Automatic text structuring and summarization”, Information Processing & Management, Vol. 33(2), pp. 193-207, 1997, Elsevier Science Ltd. Salton G., Automatic text processing, Addison-Wesley, 1989, New York. Similarity in language, thought and perception, edited by Cristina Cacciari, Brepols, 1995. Tudhope D., Taylor, “Navigation via similarity: automatic linking based on semantic closeness”, Information Processing & Management, Vol. 33(2), 1997, Elsevier Science Ltd.

Categorizing Visual Contents by Matching Visual “Keywords” Joo-Hwee Lim RWCP
, Information-Base Functions KRDL Lab

21 Heng Mui Kent Terrace, S(119613), Singapore Tel: +65 874-6671, Fax: +65 774-4990 [email protected]

Abstract. In this paper, we propose a three-layer visual information processing architecture for extracting concise non-textual descriptions from visual contents. These coded descriptions capture both local saliencies and spatial configurations present in visual contents via prototypical visual tokens called visual “keywords”. Categorization of images and video shots represented by keyframes can be performed by comparing their coded descriptions. We demonstrate our proposed architecture in natural scene image categorization that outperforms methods which use aggregate measures of low-level features.

1

Introduction

Automatic categorization of text documents has received much attention in the information retrieval and filtering community (e.g. [7,8]). Visual content categorization is relatively less explored in multimedia database and retrieval research, though pattern classification and object recognition are well studied fields. This is because in general visual contents (images, videos etc) are complex and illdefined. Most often than not, visual content categorization involves human visual perception. The latter is difficult due to two problems. First, interpreting visual data is underconstrained. A visual content can be associated with multiple consistent interpretations of the world. Second, semantically similar contents can be manifestated in many instances with variations in illumination, translation, scale etc. Many existing visual information systems (e.g. [15]) extract and annotate the data objects in the visual content manually, often with some assistance of user interfaces. It is assumed that once keywords are associated with the visual content, text retrieval techniques can be deployed easily. Although text descriptions are certainly important to reflect the (largely conceptual) semantics of multimedia data, they may result in combinatoric explosion of keywords in the attempt of annotation due to the ambiguous and variational nature of multimedia data.




Real World Computing Partnership Kent Ridge Digital Labs

Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL’99, LNCS 1614, pp. 367–374, 1999. c Springer-Verlag Berlin Heidelberg 1999


368

Joo-Hwee Lim

Also there is a limit to how much semantic information the textual attributes can provide [3]. Visual content-based retrieval systems (e.g. [11,13,2]) have mainly focused on using primitive features such as color, texture, shape etc for describing and comparing visual contents. Very often, aggregate measures of an image’s color and texture are employed as a signature for image similarity comparison. This will often produce results incongruent with human expectations [9]. For example, images sharing similar overall color distribution can differ greatly in semantic content. We argue this point further with the following scenario analysis. Suppose a coast/seaside image I0 (left half of Figure 1)is scrambled into I1 (right half of Figure 1). Based on distributions of color or other low level features solely, I0 and I1 will be considered similar though they are perceptually dissimilar. Scrambling I0 in different ways can easily produce perceptually incoherent images I2 , I3 · · · to fool a search engine that relies only on distribution of low level features and make its performance looks bad for comparison.

Fig. 1. An example image and its scrambled version When these feature-based techniques are applied to individual objects, an object is often the focus for retrieval and not much consideration has been given to the interrelationship among the objects. In a different approach that advocates the use of global configuration, the work reported in [14] developed a method for extracting relational templates that capture the color, luminance and spatial properties of classes of natural scene images from a small set of examples. The templates are then used for scene classification. Although the method improves over previous effort [9] that hand-crafted the templates, scene representation and similarity matching are computed through the relationships between adjacent small local regions which seem rather complex for comprehension. In this paper, we propose a three-layer visual information processing architecture for extracting concise non-textual descriptions from visual contents. Starting from the pixel-feature layer, the architecture progressively extracts locally salient visual information and spatially distributed configuration information present in the visual contents at the next two higher layers respectively. In a nutshell, visual contents are described in terms of prototypical visual tokens called visual “keywords”. The resulting descriptions are coded via singular value decomposition

Categorizing Visual Contents by Matching Visual “Keywords”

369

for dimensionality and noise reduction. To demonstrate our novel architecture, we employ these coded descriptions for content comparison in a scene categorization task. When compared with the popular methods that rely on distribution of low-level features, our method has shown superior classification performance.

2 2.1

Content Description & Comparison Visual “Keywords” Extraction

A key to alleviate the problems of ambiguity and variations in visual content for visual information processing task such as categorization is to exploit its inherent statistical structure. There are prototypical visual entities present in the contents of a given distribution of visual documents (e.g. digital images, video shot keyframes). Using statistical learning methods, these visual “keywords” can be derived from a sufficiently large sample of visual tokens of a visual content domain. A visual token is a coherent unit (e.g. region of pixels) in a visual document. A visual content can then be spatially described in terms of the extracted visual “keywords”. For supervised learning, detectors for salient objects such as human faces, pedestrians, foliage, clouds etc can be induced from a training set of positive and negative examples of visual tokens collected from visual documents of a given visual content domain (e.g. [12]). Detectors may be further specialized for different views (e.g. faces of frontal and side views, skies of cloudy and clear days etc) to improve their detection accuracy. Alternatively unsupervised methods such as self-organizing maps, fuzzy cmeans algorithm, and the EM algorithm can be used to discover regularities in the visual tokens in visual documents. Clusters that represent prototypical visual tokens are formed from a training set of visual tokens sampled from visual documents of a given visual content domain. 2.2

Architecture

The proposed architecture has three layers (Figure 2). The lowest layer is a collection of low-level feature planes at pixel level (pixel-feature layer). For example, the color feature of an image can have three R,G,B planes of the same resolution. The middle layer, Type Registration Map (TRM), is an abstraction of the lowest layer. More precisely, given an image I with resolution M × N , its TRM G has a lower resolution of P ×Q, P ≤ M, Q ≤ N . Each pixel or node (p, q) of G has a receptive field R [1] that specifies a two-dimensional region of size rx × ry in I which can influence the node’s value. That is, R = {(x, y) ∈ I|xp ≤ x ≤ xp , yq ≤ y ≤ yq } where rx = xp − xp + 1, ry = yq − yq + 1, and (xp , yq ) and (xp , yq ) are the starting and ending pixels of the receptive field in I respectively. We further allow tessellation displacements dx , dy > 0 in X, Y directions respectively such that adjacent pixels in G along X direction (along Y direction) have receptive fields in I which are displaced by dx pixels along X direction (dy pixels along Y

370

Joo-Hwee Lim

B spatial histogram layer

A Q

type registration layer

sx

P sy N ry

pixel-feature layer rx

M dy

dx

Fig. 2. Three-layer content description architecture direction) in I. That is, two adjacent G pixels share pixels in their receptive fields unless dx ≥ rx (or similarly dy ≥ ry ). For simplicity, we fix the size of receptive field (rx , ry ) and the displacements (dx , dy ) for all pixels in G and assume that (M − rx ) is divisible by dx ((N − ry ) is divisible by dy ). A visual token tj is a receptive field in I. It can be characterized by different perceptual features such as color, texture, shape, and motion etc. The number of visual tokens in a visual document D can be quantified by the spatial dimensions of its TRM G. Every pixel or node (p, q) in a TRM G registers the set/class membership of a visual token governed by its receptive field against T numbers of visual “keywords” which have been extracted. In short, a TRM is a threedimensional map, G = P × Q × T , that registers local type information. Likewise, the highest layer, Spatial Histogram Map (SHM), is a summary of TRM. A receptive field S of size sx ×sy and a displacement size cx , cy are used to tessellate the spatial extent (P, Q) of TRM with A × B, A ≤ P, B ≤ Q receptive fields. The memberships G(p, q, t) (∈ [0, 1]) of visual “keywords” t at TRM pixel (p, q) that falls within the receptive field of SHM pixel (a, b) are histogrammed into frequencies of different visual “keywords”, H(a, b, t) as
G(p, q, t). (1) H(a, b, t) = (p,q)∈S(a,b)

where S(a, b) denotes the receptive field of (a, b). 2.3

Singular Value Decomposition

We apply Singular Value Decomposition (SVD) to SHMs extracted from visual contents analogous to Latent Semantic Analysis (LSA) [5]. We form the fre-

Categorizing Visual Contents by Matching Visual “Keywords”

371

quency matrix X that associates visual “keywords” and visual documents as follows. Each column denotes a visual document in the form of H(a, b, t). Each row is about a visual term t in the receptive field of pixel (a, b). Thus each entry of X takes the value of H(a, b, t). SVD is carried out on X [5], X = U ΣV T

(2)

where U, V are the matrices of left and right singular vectors, and Σ is the diagonal matrix of singular values. A coded description Ω of a visual document D (a query example or a database document) is computed as Ω = DT Uk Σk−1 (3) where Uk , Σk are approximated (truncated) versions of U, Σ respectively. Using this coded description, similarity between two images x and y can be compared using appropriate similarity measures between their corresponding Ωx and Ωy .

3

Experimental Results

Natural scene images from prepackaged PhotoCD collections from Corel [4,9,14] are used as test data in our experiments. We preclassify 347 images into the following non-overlapping classes (of sizes): coasts/seasides (59), fields (95), forests/trees (72), snowy mountains (85), and streams/waterfalls (36). Figure 3 shows three samples (rows) from each class (columns), in the left-toright order as given in the previous sentence. Given an image (a query sample or a visual document), normalized to resolution 256 × 384, we extract color and orientation features based on YIQ color model and Haar wavelet coefficients respectively. The RGB channels of a natural scene image are transformed into their equivalent values in the YIQ color space. An one-level Haar wavelets decomposition is applied to the Y channel to obtain the horizontal (H), vertical (V), and diagonal (D) details. Haar wavelets are chosen because they are fastest to compute and have been used with success [6,12]. As a result of preprocessing, an image is transformed into 3 YIQ planes of size 256 × 384 and 3 HVD planes of size 128 × 192. To extract visual tokens, a 32 × 32 receptive field and a 8 × 8 displacement size are used for TRM on each YIQ plane. Equivalently, a 16 × 16 receptive field and a 8 × 8 displacement size are used for the HVD planes. A receptive field extracted from each of the YIQ planes is histogrammed into 100 bins in [0, 1] and the mode is taken as the feature value for the receptive field. For the HVD planes, only the 50 largest-magnitude coefficients for each plane are retained [6]. The feature value for a 16× 16 receptive field is the frequency of these prominent coefficients. In short, a visual token is represented by a 6-dimension feature vector summarizing its dominant color and orientation components. One third of the visual tokens extracted from all images (i.e. 13 of 452, 835) are subjected to fuzzy c-means clustering. The resulting T cluster centers are the visual “keywords”. A TRM G is therefore a 29 × 45 matrix of T -element vector.

372

Joo-Hwee Lim

Fig. 3. Sample images from five classes (columns) A 15 × 18 receptive field and a 7 × 9 displacement size are applied to G, resulting in a SHM H of size 3 × 4. Since each histogram covers T types of visual “keywords”, the term vector has 12×T elements. After SVD, the k largest factors are retained to form Ω. The similarity measure used is cosine. The leave-one-out method and K-nearest-neighbour (K-NN) classifier are adopted. Each of the 347 images is used as an unknown input to the K-NN classifier using the rest of 346 images as training set. The classification rate is averaged over all 347 images. For K-NN, the number of nearest neighbours was ranged over K = 1, 3, 5, · · · , 19 and the best result is selected. Voting is done by summing up the similarity scores of the votes (up to K) from each class, which works better than sums of counts of votes in our empirical study. Table 1 summarizes the result for different methods compared. The label “ColorHist” denotes the method that uses YIQ color histograms for comparing natural scene images. To maintain compatibility, 100 bins are also used for each of the 3 YIQ histograms, resulting in a 300-dimension vector for each image. Likewise, the result of label “Wavelets” is produced by comparing visual contents based on the 50 largest-magnitude wavelet coefficients in each of the 128 × 192 HVD planes. The coefficients are quantized into {−1, 0, 1} depending on the signs of the truncated coefficients [6]. The label “CH+W” represents method that combines those of “ColorHist” and “Wavelets” with equal weights. The label “200-vk” corresponds to the result of using the output

Categorizing Visual Contents by Matching Visual “Keywords”

373

of our proposed architecture, H(a, b, t), with 200 visual “keywords” (vk), which peaks among the number of vk = 20, 30, 40, 50, 60, 80, 100, 120, 200 attempted. Based on 200-vk, SVD was carried out with the number of factors retained, k = 10, 20, 30, 50, 70, 90. The label “200-vk,k=50” shows the best result among the values of k.

Table 1. Comparison of different methods Methods

Classif. %

ColorHist Wavelets CH+W 200-vk 200-vk,k=50

57.1 38.0 59.1 62.5 66.9

From Table 1, we see that our proposed visual “keywords” and coded descriptions better describe and discrimine visual contents in our experiments than the popular methods that rely on aggregate measures of low-level features. Table 2 shows a breakdown of classification rates for each of the five classes. It is interesting to note that the performance of our proposed method is roughly proportional to the size of training set in the classes. The streams/waterfalls class seems to be the toughest class for all methods compared due to its small sample size and variations in our data. Comparing with histogram-based methods, our method scores better in the classes: fields, forests/trees, snowy mountains. The seas and mountains appear in varying spatial layouts in the coasts/seaside images. Thus they tend to favor global aggregate measures more than a regular tessellation of our method used in this paper. With a context-sensitive spatial layout [10], the result will be improved.

Table 2. Class breakdown of classification rates Class (size) coasts/seaside (59) fields (95) forests/trees (72) snowy mountains (85) streams/waterfalls (36)

ColorHist CH+W 200-vk,k=50 71.2 43.2 62.5 65.9 38.9

79.7 46.3 58.3 68.2 38.9

50.8 75.8 69.4 77.6 38.9

374

4

Joo-Hwee Lim

Conclusions

In this paper, we have described a novel visual content description generation architecture. Low-level features of a visual content are progressively abstracted into spatial histograms of visual “keywords” and coded by SVD for effective and efficient similarity matching. Encouraging experimental results on image categorization of natural scenes have been obtained when compared to popular methods that use aggregate measures of low-level features. We will consider supervised learning [12] and other coding scheme in further experimentation.

References 1. Arbib, M.A. (Ed.): The Handbook of Brain Theory and Neural Networks. The MIT Press (1995). 369 2. Bach, J.R. et al.: Virage image search engine: an open framework for image management. In Storage and Retrieval for Image and Video Databases IV, Proc. SPIE 2670 (1996) 76–87. 368 3. Bolle, R.M., Yeo, B.L., Yeung, M.M.: Video query: research directions. IBM Journal of Research and Development 42(2) (1998) 233–252. 368 4. Corel (1998). http://www.corel.com. 371 5. Deerwester. S. et al.: Indexing by latent semantic analysis. J. of the Am. Soc. for Information Science, 41 (1990) 391–407. 370, 371 6. Jacobs, C.E., Finkelstein, A., Salesin, D.H.: Fast multiresolution image querying. In Proc. SIGGRAPH’95 (1995). 371, 372 7. Larkey, L.S., Croft, W.B.: Combining classifiers in text categorization. In Proc. of SIGIR’96 (1996) 289-297. 367 8. Lewis, D.D., Ringuette, M.: A comparison of two learning algorithms for text categorization. In Proc. of SIGIR’94 (1994) 81-93. 367 9. Lipson, P., Grimson, E., Sinha, P.: Configuration based scene classification and image indexing. In Proc. of CVPR’97 (1997) 1007–1013. 368, 371 10. Lim, J.H. (1999). Learnable Visual Keywords for Image Classification. (in preparation). 373 11. Niblack, W. et al.: The QBIC project: querying images by content using color, textures and shapes. Storage and Retrieval for Image and Video Databases, Proc. SPIE 1908 (1993) 13–25. 368 12. Papageorgiou, P.C., Oren, M., Poggio, T.: A general framework for object detection. In Proc. ICCV (1998). 369, 371, 374 13. Pentland, A., Picard, R.W., Sclaroff, S.: Photobook: content-based manipulation of image databases. Intl. J. of Computer Vision, 18(3) (1995) 233–254. 368 14. Ratan, A.L. Grimson, W.E.L.: Training templates for scene classification using a few examples. In Proc. IEEE Workshop on Content-Based Analysis of Images and Video Libraries (1997) 90–97. 368, 371 15. Rowe, L.A. Boreczky, J.S., Eads, C.A.: Indices for user access to large video database. Storage and Retrieval for Image and Video Databases II. Proc. SPIE 2185 (1994) 150–161. 367

Design of the Presentation Language for Distributed Hypermedia System Michiaki Katsumoto and Shun-ichi Iisaku Communications Research Laboratory of the Ministry of Posts and Telecommunications 4-2-1 Nukui-Kitamachi, Koganei City, Tokyo 184-8795 Japan Tel: +81-42-327-6425 Fax: +81-42-327-7129 [email protected]

Abstract. We describe a new control language for our Dynamic Hypermedia system, HMML, which controls multimedia presentations by extending HTML. HTML is a language for used displaying information on the browser. This language displays text, images, movies, etc. on a window. If Java or Dynamic HTML is used, then viewing moving objects is al so possible. However, these languages are not necessarily capable of scene synchronization and lip synchronization. Moreover, although SMIL provides simple scene synchronization it does not guarantee QoS requirements. Therefore, a language is needed for providing lip synchronization and complicated scene synchronization which guarantees QoS requirements.

1

Introduction

We have designed new presentation models for a next-generation hypermedia system with a sophisticated hypermedia structure. Hardman et al[4,5]. organizes hypertext presentations by nodes and links, multimedia presentations by a combination of continuous and discrete media, and hypermedia presentations by an extended hypertext presentation model in which each node organizes one multimedia presentation. However, this definition of a multimedia presentation is inadequate because it does not clearly define the temporal synchronization between continuous media, such as audio and video, and between continuous media and discrete media, such as images, graphics and text, for the presentation scenario. It also does not consider the transmission of scenario depended media over a network, while maintaining the temporal relation between spacial relation. Consequently, we defined a hypermedia presentation model as one consisting of several multimedia presentations[1]. In a previous paper, we designed a Hypermedia-on-Demand system (HOD) [2] based on client-agent-server architecture to provide hypermedia presentations. In addition to this, we provide the control functions for hypermedia presentations[3]. Multimedia information on the Internet can be accessed by using the World Wide Web browser. The HTML (HyperText Markup Language) is used for displaying information on the browser[6]. This language can display text, images, movies, etc. on a window. If Java [7] or Dynamic HTML [8] is used, then it is also possible to view moving objects. However, these languages are not necessarily capable of scene and lip synchronization of movies and audio. Moreover, although Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL'99, LNCS 1614, pp. 375-382, 1999.  Springer-Verlag Berlin Heidelberg 1999

376

Michiaki Katsumoto and Shun-ichi Iisaku

SMIL (Synchronized Multimedia Integration Language) provides simple scene synchronization, it does not guarantee QoS requirements[9]. Therefore, a language and it s functions are needed for providing lip and complicated scene synchronization which guarantees QoS requirements. In this paper we describe HMML (Hypermedia Markup Language) which is used for controlling hypermedia presentations, and discuss the control functions for it.

2

Dynamic Hypermedia System

2.1

The Architecture

The Dynamic Hypermedia System (DHS) is a network-oriented platform for multimedia information networks to provide multimedia information based on hypermedia presentation. Its architecture is made up of three components; client agents, a knowledge agent, and multimedia databases, as shown in Fig. 1. The client agents are located at user stations and provide the users with multimedia presentation capabilities. The knowledge agent manages the links to information units through dynamic linking methods[10] and generates multimedia objects. The multimedia databases (MDB) manage multiple media objects, such as text, image, and video and audio data. Client Agents Media Media Presentation Event ManagerPresentation Manager Manager Hypermedia Hypermedia Manager Network Network Interface

Knowledge Agent Query Hypermedia presentation

Link Manager

Med ia d ata

...

Network

Query

MDB

G

MDB

Knowledgebase

Object Manager

Network Interface

MDB Media Server Network Interface

. ..

MDB

Netw ork Knowledge Agent

MDB MDB:Multimedia database G: Gateway

Fig. 1. Architecture of Dynamic Hypermedia System

2.2

Presentation Models

We have proposed three presentation models for an advanced information infrastructure: the dynamic hypertext, multimedia and dynamic hypermedia models. These models constitute the next-generation hypermedia information system. They are more sophisticated than the Amsterdam Hypermedia Model [4,5] because they include dynamically linking mechanisms and QoS guarantee functions for making hypermedia information access more flexible. Dynamic Hypertext Model: One fundamental difference between this model and the conventional hypertext model is that this model supports a dynamical link to the next node during user interaction. The next node is linked by dynamic link methods which search for media data to match the user s intellectual background or level of interest in the information. Multimedia Model: The multimedia model is fairly self-explanatory. Several media sources are integrated temporally and spatially to create presentations.

Design of the Presentation Language for Distributed Hypermedia System

377

Dynamic Hypermedia Model: This model, shown in Fig. 2, integrates the dynamic hypertext model with the multimedia model. In other words, a node of the dynamic hypermedia model is constituted by the multimedia model. MMS

RTP

CRTP

Temporal Navigation

RTP

RTP

TEXT

DL CS

VIDEO AUDIO

MMS

Temporal Navigation

RTP

CRTP

RTP

VIDEO

IMAGE

AUDIO Scenario Time

Temporal Navigation

Scenario Time

MMS RTP DL CS

CRTP

DL CS

Temporal Navigation

RTP

RTP

TEXT VIDEO AUDIO

MMS: Multimedia Structure RTP: Reference Time Point CRTP: Current Reference Time Point DL: Dynamic Linking CS: Context Switching

IMAG E

Scenario Time

Temporal Navigation

Fig. 2. Illustration of the hypermedia model.

3

Presentation Control Language

The presentation control language is a language which describes the control structure to present hypermedia presentations with scenarios in the dynamic hypermedia system. The presentations are provided by based on the scenario which is interpreted and performed by this presentation language the on the multimedia browser in the client agent. For a multimedia scenario, the structure, time control, and navigation of this language is also considered. Moreover, the source of this language is written by text which is extended HTML, describes the scenario with consideration of the capability of the components so that the scenario can be read easily. Details of the functions are described below. 3.1

Structure

A multimedia scenario described by the HMML has a hierarchic structure that consists of two or more scenarios, and these scenarios serve as components of the high-order layer scenario . The concept of the hierarchy of a scenario can be extended in a high-order layer. Although, in the HMML the hierarchy of a scenario can be extended to the number of layers. The scenario in the DHS defines four layers: media layer (the 1st layer), scene layer (the 2nd layer), story layer (the 3rd layer), and title layer (the 4th layer). The media layer: In this layer, behavior of a single media is described, such as animation objects, images, video, text, and button. The scene layer: In this layer, the behavior of media scenarios are described. The head of the scenario of this scene layer is described as the Reference Time Point (RTP) and is used for carrying out navigation. The story layer: In this layer, the behavior of scene scenarios is described. This scenario serves as the description unit in presentation control language.

378

Michiaki Katsumoto and Shun-ichi Iisaku

The title layer: In this layer, the behavior of two or more stories is described. 3.2

Control of Time

In this section we describe the control of presentation time of a multimedia scenario. The multimedia scenario has a hierarchic structure and two or more scenarios exist in each hierarchy. Presentation time control exists for each scene and for the overall the scenario. The presentation time control for each scene is described as local time information enclosed in the scenario. That is, the single scenario, which is a component of a multimedia scenario, describes the starting time information of the scene as an origin of local time of the scenario. Suppose that scenario A of the 2nd hierarchy contains scene 1 and 2 of the 1st hierarchy. The start time of scene 2 is described as T2 in scenario A. When the time of the events of scene 2 is described as t1, the time T of the events in scenario A can be expressed as T = T2+t1. 3.3

Navigation

The HMML can also perform navigation according to the scenario described. Navigation points out the time and spatial moves between presentations. Two kinds of navigation are specified, temporal navigation and dynamic linking methods. Temporal navigation: This navigation event moves to the head of the scene or RTP within the scenario by the event of navigation during scene reproduction and starts the presentation at the scene or RTP. Dynamic Linking: This navigation event moves to the head of another story by the event of navigation during the multimedia presentation.

4

Language Specification

4.1

Definition of a Window

The rectangular viewing area on the display is called a window. There are five kinds of logical windows and four of actual windows, for example the general window is actually specified by the presentation control language. The display window shows the viewing area of the display itself and can show the size of the viewing area required by the scenario needs. The general window specifies the height and width by the number of dots. The story window, scene window, and media window specify the position (X, Y). The position of the origin is expressed with the offset from the upper layer window. The size of the window expresses the height and width by the number of dots. 4.2

Architecture

The HMML describes the information element by using the structured notation technique and structured tag. The top layer is only allowed the statement: ... .

Design of the Presentation Language for Distributed Hypermedia System

379

The statement of a tag is permitted by the 2nd layer: , <MEDIA>, <STORY>, and , <EVENT_ACTION>. The tag is reflected in the general window. The tag shown by which is inside of the general window and the tag shown by <MEDIA> which is outside the story window are described by the 2nd layer. That is information of the media object which is displayed on the story window. The 3rd or less layers are the same. The skeleton model shown in Fig. 3 is an example of a description of a scenario. In this example, the story consists of one scene containing one media scenario. <MEDIA ID=1 ATTRIBUTE="JPEG"> <MEDIA_INFO> <STORY> <STORY_INFO> <SCENE ID=1> <SCENE_INFO> <MEDIA ID=2> <MEDIA_INFO> <EVENT ID=7> <EVENT_ACTION>

Fig. 3. Example of structuring description.

4.3

Events and the Actions

Two kinds of events can be described in the HMML, user event and timer event. This event handling becomes indispensable when performing navigation. A timer event is set up beforehand, when it is described in the scenario and the scenario is performed. For example, after the end of a story, making an automatically navigation without user interactions. An action describes the operation started when an event occurs. The relationship of an event and an action is determined when a scenario is constituted. That is, a different scenario can be constituted if the association of an event and an action differs, even when combining the same scenes in the same sequence. Therefore, the description of an action and the statement of a corresponding relationship of an event and an action are located outside of the <STORY>.

380

Michiaki Katsumoto and Shun-ichi Iisaku

5

Presentation Management Module

5.1

Multimedia Controller

Coupled multimedia controllers, one in a client agent and one in knowledge agent, work together in one presentation. The multimedia controllers execute the synchronization control based on the scenario. The multimedia controller in a client agent manages and controls all media objects that constitute a presentation, and controls scene synchronization by the message passing method. Moreover, if the scenario time which shows the progress grade of a presentation is managed and the reference time move event from a user is detected, a control message will be transmitted to all media objects, and the move of a present reference time of day will be performed, as shown in Fig. 4. The modification of the QoS parameter of the whole scenario, in which a stream management and control module supervises the load effect within a system and the status of a network, negotiates with other modules according to the status of the shift. Notification of the modification is sent to the multimedia controller. The multimedia controller which receives the notification then notifies each media object about the parameter which maintains QoS priority in a scenario. A presentation is provided in each media object, maintaining QoS assurance with the new QoS parameter by the QoS maintenance module. In the knowledge agent a transmission schedule is changed if needed. Client Agent Clock

Multimedia Controller

pause

Messages/ Event

Knowledge Agent

Multimedia Controller

Messages

message

message

start

...

...

Network

Media Object 1 Media Object 2 Media Object n TEXT

VIDEO

AUDIO IMAGE

MDB Media Presentaion Manager

MDB

...

MDB

MDB: Multimedia Database

Fig. 4. Multimedia Controller.

5.2

Hypermedia Controller

The hypermedia controller, within the client agent and the knowledge agent, controls the time of starts and terminates the multimedia controller by the message passing method; and controls the context switching based on the description of author described in each presentation scenario, as shown in Fig. 5. Moreover, the client agent receives a status message from the multimedia controller in the client agent which makes possible for the context control to perform the synchronization.

Design of the Presentation Language for Distributed Hypermedia System Client Agent

Knowledge Agent

Databases Network

Network

Message

Event Manager Hypermedia Manager

Message

Link Manager

KB

Objec t Manager

Message

Hypermedia Controller

Hypermedia Controller

Message

Message

...

...

M .

MOs

M .

MMC . .

.

MOs

. .

MMC .

.

MOs

381

.

MRPC MDB

MC .

. .

MOs MOs

KB : Knowledge-base MRPC: Multicast RPC

MC .

. .

Media data .

MDB

.

MOs

Media port A Media port B

MDB

Media port C MMC:Multimedia controller

MO: Media object

MDB: Multimedia Database

Fig. 5. Hypermedia Controller

5.3

User Events

The event which a user uses to control multimedia presentations in the multimedia presentation and in the hypermedia presentation are as follows; start: for starting a new multimedia presentation; pause: for pausing a multimedia presentation; resume: for releasing a multimedia presentation;, jump: for referencing a time point to temporal navigation; quit: for ending a multimedia presentation; select: for navigation by dynamic linking. These events are in the control messages of HMML. For example, in select , a user clicks the media data (objects, such as a button, are also included) of the multimedia presentation directly in the hypermedia presentation. This action triggers and performs navigation from the embedded support dynamically to the RTP or the new multimedia presentation.

6

Conclusion

This paper described a language for controlling the multimedia presentation and the hypermedia presentation by the presentation control module. The functional validation of HMML using the original browser is completed now. We will make a general purpose browser.

References [1] M. Katsumoto, N. Seta, and Y. Shibata, “A Unified Media Synchronization Method for Dynamic Hypermedia System,” Journal of IPSJ, Vol. 37, No. 5, pp. 711-720, May 1996. [2] M. Katsumoto and S. Iisaku, “Design of Distributed Hypermedia System Based on Hypermedia-on-Demand Architecture,” Journal of IPSJ, Vol. 39, No.2, Feb. 1998.

382

Michiaki Katsumoto and Shun-ichi Iisaku

[3] M. Katsumoto and S. Iisaku, “Design of the Presentation Controller Functions for Distributed Hypermedia System,” Proc. of ICOIN-12, pp.206-211, Jan. 1998. [4] L. Hardman, D.C.A. Bulterman, and G. Rossum, “Links in Hypermedia: the Requirement for Context,” ACM Hypertext ‘93, pp.183-191, Nov. 1993. [5] L. Hardman, D.C.A. Bulterman, and G. Rossum, “The AMSTERDAM Hypermedia Model: Adding Time and Context to the Dexter Model,” Comm. ACM, Vol. 37, No. 2, pp. 5062,1994. [6] T. Berners Lee and D. Connolly, “Hypertext Markup Language - 2.0,” IETF RFC 1866, Nov. 1995. [7] http://java.sun.com/ [8] D. Gulbansen and K. Rawlings, “Special Edition Using Dynamic HTML,” QUE corporation, 1997. [9] http://www.w3.org/TR/1998/REC-smil-19980615/ [10] M. Katsumoto, M. Fukuda, and T. Shibata, “Kansei Link Method based on User Model,” Proc. of ICOIN-10, pp. 382-389, 1995.

A Generic Annotation Model for Video Databases Herwig Rehatschek1 and Heimo Müller2 1

Institute of Information Systems, JOANNEUM RESEARCH Steyrergasse 17, A-8010 Graz, Austria 2 Faculty of Arts, Word & Image studies Vrije Universiteit Amsterdam de Boelelaan 1105 1081 HV Amsterdam, Netherlands

Abstract: The change from analogue broadcasting to digital MPEG-2 channels among the satellite programs resulted in new demands on video databases and archives. Digital archives offer on the one hand a reduction of storage costs, and enable on the other hand easy reuse of already existing material. However, searching for appropriate film material in large archives is still a tedious problem. This paper describes a generic annotation model for MPEG movies which enables the user to structure the film in as many hierarchical levels as needed and to annotate any physical or logical part of the film with generic definable attributes. The model was implemented in prototype system which additionally offers a query and ordering facility per web browser and Internet.

1

Introduction

An increasing number of satellites offering digital MPEG-2 channels (e.g. DF 1, Astra Service, Premiere Digital, RAI, Intelsat, ...) mark the start of a new age in the distribution of films and videos. This results in an increasing demand on content annotation in order to reuse already existing archive material for cost effective productions. However, searching for appropriate film material in a large film archive is still a tedious task. Parts of films can just be searched and retrieved if annotations are available. In praxis there are many different ways of annotation depending on the overall approach (annotation based on a thesaurus, keywords or only free text) and the application domain (broadcast archive, industrial archive, cultural archive). An additional problem occurs by using different annotation languages and country specific character sets. When film archives are opened for commercialization or for the public the awkward handling of analogue film material becomes a problem. Digitization offers a number of advantages including reduction of storage costs, no progressive decay, fast availability in different qualities (MPEG-1/for previewing purposes, MPEG-2/for sending, ...), reusing and copying of material without loss of quality and fast access for internal personnel (Intranet) and customers (Internet). Within our implemented prototype system some of these problems are addressed and solved. A major focus was given on the interoperability across different application Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL'99, LNCS 1614, pp. 383-390, 1999.  Springer-Verlag Berlin Heidelberg 1999

384

Herwig Rehatschek and Heimo Müller

domains and the problem of import/conversion of existing annotation data. The cross platform exchange of annotation records was studied in detail. The system offers tthree annotation possibilities, a thesaurus based, one with generic keywords in combination with free text and an automatic annotation facility.

2

Related Work

Several efforts are undertaken in order to define appropriate data models for storing multimedia data. One model for storing a physical, time based representation of digital video and audio was introduced by [1]. General concepts for the physical modeling of digital video and audio data are discussed and a specific model for storing Quicktime movies is introduced. The application of the general concepts allows the specific physical modeling of any other video format. The Layered Multimedia Data Model (LMDM) developed by [8] emphasizes the sharing of data components by dividing the process of multimedia application development into smaller pieces. LMDM claims for the separation of data, manipulation and presentation. Both modeling approaches do not concentrate an the topic of generic film annotation using user definable attributes and values which can be attached to any physical or logical unit (e.g. an act, scene, shot) of a film. A lot of research has been done on the development of digital video databases and archives. Siemens has implemented the CARAT-ARC system [2], which is an open system for storing, indexing and searching multimedia data. Annotation of data is supported by either using a thesaurus or free text. However, the system is not designed for supporting off-line units, e.g. outsourcing of annotation and/or encoding to geographically dispersed locations. The VideoSTAR experimental database system [5], which was developed by the Norwegian Institute of Technology, supports storage of media files, virtual documents, video structures and video annotations in four repositories. Content based querying and retrieval of film parts is achieved by annotation of logical parts of a film (sequence, scene, shot, compound units). Despite the relational data model of VideoSTAR offers annotation it is not generic in the sense that users can define new categories but limited to four categories, which can hold free text. There exist several sites offering search for film meta-information and download of movie clips on the internet. Some just offer an alphabetically ordered list of films with previews, others offer a database system with access to stored film meta-information [7], [4].

3

The Prototype System

This section gives an overview of our prototype system architecture and its high level building blocks. The system is a very large digital video database holding all films in MPEG-2 format. Sources remain stored on Digital Betacam in order to fulfill any special format wishes of customers (e.g. S-VHS). Each film has annotations attached

A Generic Annotation Model for Video Databases

385

which allows the search of specific parts or objects (e.g.: acts, scenes, shots, actors, ...) in a film. Basically the system consists of four units (see Figure 1): compression street(s), annotation site(s), central digital video database and the web interface for online search and ordering. MPEG-II Compression LowRes Compression

Video

Tape Storage Shelf (10-15 tapes, HiRes + LoRes)

Disk Array Caching

Compression Street

Annotation Site

Carrier

SGML-Based Interface

Annotations Film Annotation

Annotation Software SGMLBased Interface

SGMLBased Interface Central DFC database

Central database

Delivery of customer ordered film parts in the specified format

Tape Archive (robot)

HTTP

Media Production on Demand, e.g. by the Annotation Process or a customer request

Internet Web Interface

Customer with Standard WWW Browser

Figure 1: High level building blocks of the Digital Film Center

According to Figure 1 the filling process of the database can be described as follows: incoming videos are first encoded at the compression sites in two formats: MPEG-2 for storage a the central video tape archive and resell and MPEG-1 for low resolution previews and annotation purposes. The encoded material is then sent together with some film metainformation to a central video database on DLT tapes. The metainformation is stored in SGML [6] format in order to make the system as open as possible. The metainformation of the film is imported, the MPEG data remains on the DLT tape. The database stores a reference to the tape and the location for later access. Now the film is ready for annotation and can be checked out by an annotation site. For this purpose the MPEG-1 representation together with the already existing film metainformation is sent to an annotation site using again SGML as an exchange format. Since the compression streets and the annotation sites have a special SGML

386

Herwig Rehatschek and Heimo Müller

based off-line interface for importing / exporting information and data to the central video database these units can be built at geographically dispersed locations all over the world. At the annotation site the film is being annotated using a special annotation software. The annotation is sent back in the same SGML based format to a central database. Now information about the film and all parts of the film is contained in the video database. These information can be searched by customers via a web interface. Because of the attached annotations the search of parts of the film or specific objects within the film become possible. Any parts of the films can later be ordered on-line vie the web interface.

4

The Generic Annotation Model

The most important and central component of the prototype system is the digital video database. It holds all information related to films and parts of films. Within the database there exist two main views on films: the logical and the physical view. The starting point is the logical film. It has several physical representations, and is the target of annotations. This is different to current systems, where in most cases a physical representation of a film is annotated. Both views are modeled in the data scheme of the prototype system. One physical representation is the reference source. When describing differences in terms of annotation of different representations (e.g. different language versions, or evening versus late night versions) all annotations are made relative (or in reference) to the time codes of the reference version. We want to stress the fact, that when annotating a film there exist basic semantics: the temporal static structure of the film (referred to as static annotation) and annotations, which vary in their semantics (referred to as dynamic annotation). E.g., when annotating a video database of a hospital, person annotation will describe patients, when describing news material, we annotate real world persons and their historic actions, and in the annotation of a movie, actors/characters of the movie are described. The annotation model of the database, therefore defines a model to describe the basic semantics (temporal and logical structure of video) and provides a method to describe the dynamic part of the annotation data. The temporal model allows to construct any given structure of a film in as many levels as needed. Subdivisions of films into several units are supported (e.g. a film may be divided into several acts; an act may consist of several scenes and a scene may be divided into several shots where each shot has a special frame of interest). The data model consists of the units parts, sequences and groups. Parts are the smallest units. They address a number of frames (or even just one) and are defined by a start and end time code. Sequences can be defined recursive and can therefore again contain sequences. This allows a modeling of as many levels as needed. Besides sequences groups can be formed which represent any combination of parts, sequences and also again groups.

A Generic Annotation Model for Video Databases

387

35 mm

VHS

copy

Digi Beta copy

representation

MPEG-II

AVI

y co p

py co

MPEG-1

cop

y

Quicktime

8 mm

ANNOTATION -> characteristics of film material (e.g: defects (scratches, color degradation, dust), quality, ...)

film

version

start time code: 00:21:23:14 end time code: 00:21:47:00 quality: poor

Figure 2: Physical representation of an example film

Groups do not have the requirement to contain continuous sequences of time codes and are therefore a good instrument to structure a film according to key scenes (e.g. for a trailer or advertisement production). Since a film can have more than one version these structures can exist for each version and can be actually different. The video database supports versions and can hold a structure for each of them. This is indicated by the version level at the bottom of Figure 3. An example for the temporal structuring of a movie is given in Figure 3. The film "Opernball" is structured in acts, scenes and shots using our hierarchical film data model. In this example parts represent shots, sequences of first order scenes and sequences of second order acts. All entities of the logical and physical structure of a film can be annotated and therefore also be searched. The semantics of such an annotation is defined in the so called "annotation style file". An annotation style file holds a number of annotation attributes. Annotation attributes can be defined generic, in the sense that the user can define the attribute's name and its type. One annotation style concentrates on one special kind of movie, e.g. a medical film or a documentary film, and has therefore special annotation attributes. E.g. for a documentary film some attributes could be "geography / city" (type text), "animal / mammal / body temperature" (type number). Different styles can be created and stored in the database. Annotation styles are defined by using SGML [6] in combination with natural language description. The set of all annotation styles used is called video object foundation class.

Herwig Rehatschek and Heimo Müller

parts

part 1

part 2

part 3

sequence 1 (level 0)

part 4

part 5

part n-1

part n

sequence 2 (level 0)

sequences sequence 1 (level 1)

group 3

group 1 groups group 2

version

film

ANNOTATION -> characteristics of film units (e.g: persons, objects, description,...)

388

shot: Philippe skates with the floor brush start time code: 00:21:23:14 end time code: 00:21:47:00 persons: Phillipe location: living room

scene: skating and singing persons: Phillipe, Hanni location: living room

act: the day before persons: location: -

group: Hanni and Phillip persons: Hanni, Phillip location: living room

title: Opernball

Figure 3: Logical structure of an example film

The generic implementation within the RDBMS was implemented by defining attributes by a name, a detailed description, a data type and a possible default value. Such defined attributes can now be assigned to any logical or physical film entities (e.g. parts, sequences, physical representations, etc.). Next to the generic annotation the system supports a thesaurus based keyword annotation. The thesaurus can be defined by the user with a special tool and stored within the database. The database supports different thesauri according to different kind of movies. All annotation and encoded data is stored in the central database which is accessible over the world-wide web to customers for searching and ordering. The web interface provides access to the database material for customers. By filling an electronic shopping cart authorized customers can order the desired film material - which actually can be parts of a film - in the desired quality. The querying possibilities offered support the search for generic annotation attributes as well as free text search. The result represents parts of the film which have been returned e.g. on the query "return all parts which contain a table ". Next to the detailed description of the part including start and end time code and all the generic annotation attributes authorized customers are able to preview material by clicking on a link within the result. Previews are stored on hard disks for fast access and not on mass storage devices, where the high-quality material of the archive is kept.

A Generic Annotation Model for Video Databases

5

389

Results and Conclusions

This paper addressed a digital video database system which allows (1) storage of digital videos and corresponding metainformation (2) generic annotation of user defined film structures (3) search access on annotation data via a web interface and a standard WWW browser. The video database of the prototype system is designed as a large geographically dispersed system. Many encoding suites produce MPEG-2 videos, one central video database holds metainformation, annotations and references to the MPEG-2 files stored in a tape robot, many annotation sites add film annotations to the stored movies. The central database can be accessed via a web interface by a standard WWW browser all over the world. The generic film data model of the system allows the hierarchical structuring of a film in as many levels as needed. This can be done on the one hand for the logical structure (e.g. acts, scenes, shots and frames) and on the other hand for the physical representation of a film. To each of these logical and physical entities annotations can be attached. The generic annotation model is the most remarkable part of the video database. The generic annotation model allows the free definition of annotation attributes with any user defined name and type. These annotation attributes can be structured in so called "annotation styles". Different annotation styles can be stored in the video database. One style refers to one specific annotation topic (e.g. medical films, action films, ...). The generic annotation is done by a special annotation software which supports the annotator with a graphical user interface and a MPEG-I preview. A second annotation possibility is thesaurus-keyword based, where the thesaurus can be dynamically created and exchanged. A web interface was developed in order to search the database and download previews. The web interface offers registered users the search for entire films (e.g. title search) and parts of a film. Search results can be collected in a shopping cart and on-line ordering can take place. The quality of the ordered film material can be chosen by the customer. The prototype does not use a proprietary exchange format among the distributed units. All interfaces between the central video database, the annotation software and the encoding suites are SGML-based which makes the prototype an open system. Imports from and exports to other video database systems, e.g. Media Vault, become possible.

6

Outlook

Currently annotation styles are defined with SGML and natural language description. In the future formal specification methods could be used for describing the semantics of the annotation fields, and their relations. The development of Video Object Foundation Classes will be stressed in the future, which describe a framework of basic objects semantics, e.g. persons, settings, speech,

390

Herwig Rehatschek and Heimo Müller

movement patterns, and methods of specializing these objects for a specific annotation style. The new member of the MPEG family, called "Multimedia Content Description Interface" (in short ‘MPEG-7’), will extend the limited capabilities of proprietary solutions in identifying existing content notably by specifying a standard set of descriptors that can be used to describe various types of multimedia information. Developments on this standard will be closely monitored and checked for integration into the prototype system.

7

Acknowledgments

This project was partially funded by the European Union (“Digital Film Center” ESPRIT project Nr. 25075, “VICAR” ESPRIT project Nr. 24916). Specific thanks go to our colleagues Bernd Jandl and Harald Mayer and to our Greek partners within the project which helped to realize this system.

8

References

[1]Ch. Breiteneder, S. Gibbs, D. Tsichritzis. "Modelling of Audio/Video Data", pp. 322-339, Karlsruhe, Germany, 1992 [2]R. Depommier, N. Fan, K. Gunaseelan, R. Hjelsvold. "CARAT-ARC: A scalable and Reliable Digital Media Archiving System", IEEE Int. Conf. on Image Processing, 1997 [3]P. England, R. Allen, et. al. "The Bellcore Video Library Toolkit. Storage and Retrieval for Image and Video Databases“, pp. 254-264, San Diego/La Jolla, CA, USA 1996 [4]Film.com Inc., January 1998. [5]R. Hjelsvold, R. Midtstraum. "Databases for Video Information Sharing", Proc. of the IS&T/SPIE, San Jose, C.A., Feb. 1995. [6]International Organization for Standardization, 1986. Information processing---Text and office systems---Standard Generalized Markup Language (SGML), Geneva ISO, 1986. [7]The Internet Movie Database, January 1998. [8]Schloss, Wynblatt. “Providing Definition and Temporal Structure for MM data“, Proc. of the Second ACM Int. Conf. on MM, CA., ACM Press, ISBN 0-89791-686-7, S. Francisco, 1994

Design and Implementation of COIRS (A COncept-Based Image Retrieval System)
Hyungjeong Yang, Hoyoung Kim, and Jaedong Yang

Dept. of Computer Science Chonbuk National University, Chonju Chonbuk, 561-756, South Korea Tel: (+82)-652-270-3388, Fax: (+82)-652-270-3403 {hjyang,hykim,jdyang}@jiri.chonbuk.ac.kr

Abstract. In this paper, we design and implement COIRS (COnceptbased Image Retrieval System). It is a content-based image retrieval system to search for images as well as indexing them based on concepts. The concepts are detected by a thesaurus called triple thesaurus. The triple thesaurus consists of a series of rules defining the concepts. COIRS adopts an image descriptor called triple to specify the spatial relationships between objects in an image. An image is indexed by a set of triples - each of them is enrolled into an inverted file, pointing to the image. We also develop a query processor to retrieve relevant images by evaluating a user query. The query formulated in terms of triples is evaluated by matching its triples with those of the inverted file.

1

Introduction

For the last decade, a large volume of image collections from various sources such as medical diagnosis, the military, the fashion industry, and broadcasting has brought forth a variety of image retrieval techniques. One simple technique is to search images based on descriptions manually produced. However, the number of images is not amenable to such a method. Moreover, describing even one image is not a trivial work since much knowledge encoded in it is equivalent to thousands of words in general [7]. Contents-based retrieval techniques are therefore necessitated to analyze images based on the characteristic of their content. Some generic attributes used for indexing and searching images are color, texture, shape and spatial relationship. QBIC [1], Stars [6], and Photobook [7] are attempts to index images based on the attributes. However, these systems alone may not satisfy user queries if retrieves images turn out to be relevant only when they are conceptually related with the queries. For example, most of the conventional image retrieval systems fail to retrieve kitchen pictures since they can’t deal with the concept,
kitchen
. To retrieve such images, the systems may ask users to explicitly list the components such as a dining table, a cupboard, and a dishwasher that the kitchen should include




This work was supported by the KOSEF no. 97-0100-1010-3 To whom correspondence should be addressed: [email protected]

Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL’99, LNCS 1614, pp. 391–399, 1999. c Springer-Verlag Berlin Heidelberg 1999 

392

Hyungjeong Yang et al.

together with the explicit specification of their possible spatial configuration. Obviously, it may be a tiresome work to specify the concept in such a way. Concept-based image retrieval techniques appear to be a unique solution for providing users with a higher-level query interface. In this paper, we describe the design and implementation of COIRS(COnceptbased Image Retrieval System). It differs from extant content-based image retrieval systems in that it enables users to query based on concepts, that is, high level objects identified from a spatial configuration of primitive objects in an image. COIRS adopts an image descriptor called triple to specify the spatial relationships between objects. All images are therefore indexed by a set of associated triples. A triple thesaurus defines concepts by the logical connective of triples. The triples are used for formulating queries as well as indexing images. We also develop a query processor to evaluate a query by matching its triples with those of the inverted file.

2

Image Indexing by Triples

An image is represented by a set of ordered triples in COIRS. A triple specifies a spatial relationship between two objects in an image [2,3]. For example, if an object b is located at the north side of another object a, the triple would be < a, b, north >. Fig. 1 shows a symbolized image p1 , where a vase(b) with flowers(c) is located on a table(a) with an apple(d) and a bunch of grapes(e) at its east side. The image p1 is now represented by Tp1 = {< a, b, north >, < a, c, north >, < a, d, north >, < a, e, north >, < b, c, north >, < b, d, east >, < b, e, east >, < c, d, southeast >, < c, e, southeast >, < d, e, east >}. We assume one symbol is assigned for one object regardless of the size of the object. We also restrict the spatial relationship into eight directions such as north, southeast, west, etc.

Fig. 1. Symbolized Image p1 Any known technique is unavailable to automatically performing object recognition since objects to be recognized originate from disparate domains and images contain considerable noise [6].COIRS provides a visual image indexer to facilitate manual object labeling in an image and the specification of their relative position. It is an assistant tool designed to minimize manual work when

COIRS – A COncept -Based Image Retrieval System

393

indexing images. To label an object in an image, dragging a Minimum Bounded Rectangle (MBR)and then entering its name are requested(Fig. 2(a)). As each object is labeled, the generated triples are displayed through the triple viewer. While such a manual labeling may incur a considerable overhead, it has some advantages over automatic one. One is that even scenes extremely difficult to analyze can be indexed. For example, natural scenes containing mountains and rivers can be indexed in terms of triples.

Fig. 2. Visual Image Indexer and Inverted File

The triples produced by the visual image indexer are inserted into an inverted file(Fig. 2(b)). The inverted file consists of triples and the link of the images indexed by the triples. We constructed this file based on the technique presented in [4].

3 3.1

Recognition of Concept Objects Triple Thesaurus

A concept is a composite object into which more than one object is aggregated according to their spatial relationships. Objects except the concepts are primitive objects. A triple thesaurus captures the concepts from the logical connective of triples. Let T and O be a set of all triples and objects respectively. Then a triple thesaurus C to detect concepts is defined as the following function [8]. C : 2T → O, i.e., C({t}) = c f or t ∈ T and c ∈ O. For example, the two primitive objects, b and c in p1 can be combined into a flowervase f , which is a concept: C({< b, c, north >}) = f . Such a thesaurus may be implemented by CFG(Context Free Grammar), PDA(pushdown automata), or rules, which turn out to be the same mechanism. In COIRS, a set of production rules in CFG is used to detect concepts by YACC (Yet Another Compiler-Compiler).

394

Hyungjeong Yang et al.

Yacc Code of Triple Thesaurus %token NORTH,SOUTH,WEST,EAST %token NORTHEAST,NORTHWEST,SOUTHEAST,SOUTHWEST %token TABLE,VASE,FLOWER,APPLE,GRAPE,ORANGE %token AND,OR %% Concept_Object : Flower_vase { add_cobject("flowervase"); } | Fruits { add_cobject("fruits"); } | Still_life { add_cobject("still_life"); } ; Flower_vase : ’<’ VASE ’,’ FLOWER ’,’ NORTH ’>’ ; Fruits : ’<’ Fruits ’,’ Fruits ’,’ Location ’>’ | Fruit ; Still_life : ’<’ Flower_vase ’,’ Fruits ’,’ Side_below ’>’ AND ’<’ TABLE ’,’ Fruits ’,’ NORTH ’>’ AND ’<’ TABLE ’,’ Flower_vase ’,’ NORTH ’>’ ; Fruit : APPLE | GRAPE | ORANGE ; Location : NORTH | SOUTH | WEST | EAST | NORTHWEST | NORTHEAST | SOUTHWEST | SOUTHEAST ; Side_below : WEST | EAST | SOUTHWEST | SOUTHEAST | SOUTH ; Now constituents of the concept c, comp(c) is defined as {o1, o2 | C({< o1 , o2 , r >})}. For example, comp(f ) = {b, c}. 3.2

Triple Generation for Concepts

To determine a spatial relationship between a concept and other objects, we now define ordering of directions. Let a concept c ∈ O, o, o
∈ comp(c) and / comp(c)). For r, r
∈ D, < o, oj , r >, < o
, oj , r
>∈ Tp for all oj (oj ∈ Op ∧ oj ∈
if r is above r’ or r = r in (Fig. 3(a)), then we say r subsumes r’. It is denoted by r ≥ r
[8].

Fig. 3. Ordering between Directions and 4 Bits Representation

A function GenConceptTriple() generates concept triples to specify spatial relationships between a newly generated concept and other objects in the image.

COIRS – A COncept -Based Image Retrieval System

395

The spatial relationship between the objects is defined by GLB(Greatest Lower Bound)/LUB(Lowest Upper Bound). Concept-direction set named R is a set of spatial relationships between all constituents of a concept c and an object o in the image. R is obtained by Dir Comp(). Function to generate concept triples GenConceptTriple() Input : A Concept Object and other objects in an image Output : Triples Involving c Begin R = Dir_Comp(c) if GLB(R) exists then return (c,o,GLB(R)) else if LUB(R) exists then return (c,o,LUB(R)) return NULL End. A spatial relationship r is represented by four bits(Fig. 3(b)). ’AND’ and ’OR’ bit operators are used to calculate GLB and LUB respectively. We first perform pairwisely GLB(R) = AN D(r1 , r2 ) where r1 , r2 ∈ R. If GLB(R) = 0, it is the target direction, but if not, LU B(R) = OR(r1 , r2 ) is calculated as an alternative direction. When neither GLB(R) = 0 nor LU B(R) = 0, NULL is returned. This means that any representative direction can not be obtained. Nonexistence of GLB or LUB between a concept object and any other objects entails that their spatial relationship cannot be defined in terms of eight directions. Such cases are not generated as triples in COIRS. Since refined frameworks supporting ’same’, ’surround’ and ’overlap’ may require considerably extensive work, we leave it to further research. In p1 , to determine a direction between a concept
f lowervase(f )
and an object
apple(d)
, let’s obtain concept-direction set R = {east, southeast} between comp(f ) = {b, c} and ’d ’ from Dir Comp(). Since southeast = 0101 and east = 0100 in (Fig. 3(b)), GLB(southeast, east) = AN D(0101, 0100) = 0100, i. e., east. In other words, since the spatial relationship between f and d may be viewed as east and southeast simultaneously, r = GLB(R) = east is fixed as a representative relationship satisfying both of two. < f, d, east > is hence added to the triple set. Similarly, {< a, f, north >, < f, d, east >, < f, e, east >} are generated and then added to the triple set. Another advantage of our system is that it provides a level of abstraction in capturing concepts. For example, in (Fig. 1), suppose ’g’ is defined as a concept ’fruits’ by C({< d, e, east >}) = g. The spatial relationship between ’f ’and ’g ’is then r = GLB(R) = east. Furthermore, we can extract the whole semantics of p1 , i.e., a ’still-life((h)’ image, if we define C({< a, f, north >, < a, g, north >, < f, g, east >}) = h. It describes that an image where a flower vase(f) and fruits(g) are at the north of a table(a), and g is to the east of f may be viewed as a still-life image.

396

4

Hyungjeong Yang et al.

Concept-Based Query Evaluation

A query is evaluated by matching the query triples with those of the inverted file. A query may be given by two ways; object icons and triples. Like other Graphical User Interface (GUI), Icon-based query interface is intuitive and easy to use(Fig. 4(a)). The interface is divided into four parts: a sketchpad, an icon set, a query triple viewer and a result viewer. The query is given by locating icons at the sketchpad from the icon set provided by COIRS. For user convenience, a user is also allowed to directly input an object name by pointing a place on the sketchpad.

Fig. 4. User Interface

A query may be also issued through a triple-based interface by inputting objects and the spatial relationship between them(Fig. 4(b)). It is composed of three parts: a query triple editor, a query triple viewer, and a result viewer. The query triple editor allows users to construct a simple query or compound query by using logical connectives such as ’AND’ and ’OR’. Once objects and a spatial relationship enter, the corresponding query triples are displayed through the triple viewer. If the spatial relationship is one of right, left, above, below or side, it is transformed into several triples. For example, if the spatial relationship of a triple is ’below’, it is translated into three triples whose spatial relations are southwest, south, and southeast respectively. It is also possible that a query can be formulated in terms of objects alone without their spatial relationships. Then the query is converted to eight or-relationship triples having eight directions. The result viewer now shows the result of the query from the following function to retrieve images.

COIRS – A COncept -Based Image Retrieval System

397

Retrieve_Images() input: query triple output: image ids Begin while(query_triple != EOF) Begin token=Get_Token(query_triple) if monotriple = = token then set_of_ids = InvertedFileLookup (token) else logical_con_st = token End return (CompoundQueryOp(set_of_ids, logical_con_st)) End Since COIRS can even extract the semantics of a whole image, it is possible that COIRS retrieves an image by a concept which covers it. For example, to retrieve still-life images, input would be simply ’still-life’ into ’concept of image’ field in the query editor.

5

System Implementation

COIRS was fully implemented with Motif and C++ on top of Sun Solaris OS. It consists of four modules: a visual image indexer, a triple thesaurus, an inverted file and a query processor. The visual image indexer facilitates object labeling and the specification of relative position of objects. The thesaurus captures the concepts by analyzing triples, thereby extracting image semantics. A query is evaluated by matching the triples of the query with an inverted file. Shown in Fig. 5 is the whole configuration of COIRS incorporating the four modules.

Fig. 5. System Configuration of COIRS There are two approaches in implementing the query processor of COIRS: top down evaluation and bottom up evaluation. In top down evaluation, references to the triple thesaurus are confined to concepts appearing in the triples of user

398

Hyungjeong Yang et al.

queries. Any other reference for trying to detect concepts in images is not made. The inverted file, therefore, does not include any triple containing concepts. When evaluating a query, every triple containing concepts is translated into more than one concept-free triple yet semantically equivalent to. Target images may be retrieved by searching the concept-free triples in the inverted file. On the contrary, in bottom up evaluation, every concept is detected and then the generated triples involving concepts are inserted into the inverted file prior to query evaluations. The triples in a user query may hence match its direct counterpart in the inverted file. Currently our query processor in COIRS adopts the bottom up evaluation for not compromising the user response time, avoiding query processing delay due to query reformulation. However, the bottom up evaluation also has the drawback that concept detection be time consuming when images contains too many objects. Judgement on which one is better may depend on the characteristic of application domains.

6

Conclusions and Further Work

In this paper, we developed COIRS as an advanced content-based image retrieval system. The main advantages of COIRS are that 1) it is a higher level image retrieval system in comparison with other systems that retrieve images only relying on syntactical information such as colors, shape or texture, and 2) it provides an integrated framework into which extant content-based technologies can be uniformly incorporated. As further researches, complementary works for our framework may be needed. First, we should solve the problem of determining ambiguous spatial relationships between objects which can not be specified in terms of only eight directions. For example, we should remove a difficulty in specifying a direction which may be either east or southeast, but more likely, southeast. Introducing fuzzified spatial relationship may be an alternative for that. Second, the thesaurus introduced in this paper should be developed in greater detail, since it is a core component for capturing image semantics. Rule-based languages such as prolog or CLIPS may be exploited to construct the thesaurus.

References 1. Ashley, J., et al. : Automatic and Semiautomatic Methods for Image Annotation and Retrieval in QBIC. In: Proceeding of Storage and Retrieval for Image and Video Databases III, Vol. 2, 420, SPIE (1995) 24-25. 391 2. Chang, C. C. and Lee, S. Y.: Retrieval of Similar Pictures on Pictorial Databases. In: Pattern Recognition, (1991) 675-680. 392 3. Chang, C. C.: Spatial Match Retrieval of Symbolic Pictures. In: Journal of Information Science and Engineering, Vol. 7, (1991) 405-422. 392 4. Cook, C. R. and Oldehoeft, R. :A letter-oriented minimal perfect hashing function. In: ACM SIGplan Notices 17 (1982) 18-27. 393 5. Han, J. J., Choi, J. H., Park, J. J. and Yang, J. D. :An Object-based Information Retrieval Model : Toward the Structural Construction of Thesauri, In :Proceeding of International Conference ADL98, (1998) 117-125.

COIRS – A COncept- Based Image Retrieval System

399

6. Li, John Z. and Ozsu, M. Tamer : STARS: A spatial Attributes Retrieval System for Images and Videos. In : Proceedings of the 4th International Conference on Multimedia Modeling(MMM’97), Singapore (1997). 391, 392 7. Pentland, A., Picard, R. W., Scaroff, S.: Photobook: Tools for Content-based Manipulation of Image Databases. In: International Journal of Computer Vision (1996). 391 8. Yang, J. D. and Yang, H. J.: A Formal Framework for Image Indexing with Triples: Toward a Concept-based Image Retrieval, In: International Journal of Intelligent System.: submitted (1998). 393, 394

Automatic Index Expansion for Concept-Based Image Query Dwi Sutanto and C. H. C. Leung Communications & Informatics Victoria University of Technology Ballarat Road, Melbourne 3001, Victoria, Australia {dwi,clement}@matilda.vut.edu.au

Abstract. Search effectiveness in an image database is always a trade-off between the indexing cost and semantic richness. A solution that provides a significant degree of semantic richness that simultaneously limits the indexing cost is presented. The query schemes are able to enhance the query speed by adopting a semantically rich structured form for high-level image content information, as well as exploiting the efficiency of conventional database search. Through the use of rules, which can be either pre-defined or dynamically incorporated, a new level of semantic richness can be established, which will eliminate the costly detailed indexing of individual concepts. The query algorithm incorporates rule-based conceptual navigation, customized weighting, incremental indexing and relevance feedback mechanisms to enhance retrieval effectiveness.

1 Introduction With rapid advances in powerful multimedia computers and the Internet, pictorial query algorithm has attracted significant research attention. There are many different methods and techniques that have been proposed. They largely fall into two categories: concept-based [1,7,8,9,11,14,15,16,19] and contentbased [2,3,4,5,6,10,12,13,17]. Concept-based methods are mainly text-based approaches which allow users to post their query either simply using keywords or using a form of natural language. Content based methods, on the other hand, are pixelbased which allow users to post their query by an example or by image contents (color, shape, texture, etc.). Each type of method has its advantages and disadvantages. Query by example (QBE), for example, is suitable if a user has a similar image at hand, and a query will recall entities having similar image signature. However, it would not perform well if the image is taken from a different angle, having a different scale, or placed in a different setting. It is often difficult to query images by their contents where users have to tell/select a color composition, outline the drawing, select a texture etc. Because content-based methods are not rich in image semantics, it is difficult to use them to query high level visual concepts like an image Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL'99, LNCS 1614, pp. 399-408, 1999.  Springer-Verlag Berlin Heidelberg 1999

400

Dwi Sutanto and C.H.C. Leung

of a Melbourne Spring Festival. Multiple domain recall is another disadvantage of approaches like QBIC [3,4,5]; e.g. a query for 70% of blue color and 30% of white color may return an image of a white car parked behind a blue wall, a white book on a blue table, an image of a beach with white sand, etc. For these reasons, it is obvious that text-based queries are preferable to QBE. Text-based queries are also very much faster than QBE and QBIC because text processing only takes a small fraction of time compared to image processing. Another advantage of text-based queries is the ease with which a user could prepare the query, because they use human language for the queries. In this paper, we develop a text-based query system based on the Ternary Fact Model (TFM) database [1,7,14]. Unlike conventional text-based system which relies on keywords for database index and query, TFM has an underlying visual entityrelationship index representations, rule-based conceptual hierarchy, and other features to support semi-automatic indexing and to enhance query performance using thesaurus system, relevance feedback mechanism, and user-tailored weighting components.

2 Image Indexing Paradigm Our approach to image indexing is developed from the basic Ternary Fact Model°[1,7,14], which is based on a textual descriptive approach to represent image contents. The TFM modeling approach has shown to give competent performance in terms of recall and precision in the retrieval of images. The representation consists of five discrete fact types: elementary facts, modified facts, outline facts, binary facts and ternary facts. Elementary facts are atomic objects in the image that are meaningful to the human users, such as apple, book, chair, etc. Modified facts are elementary facts augmented with descriptive properties, such as red apple, thick book, old chair, etc. Outline facts are abstract concepts derived from the image, such as war, party, celebration, etc. Binary facts are relationships linking two elementary or modified facts, such as a boy eats an apple, a book is on the chair, etc. Finally ternary facts are relationships linking three elementary or modified facts, such as a boy peels a green apple with a knife, a man puts a book on the chair, etc. Table 1 illustrates the main features of the model. It is quite possible to extend the model to link more than three facts; however, it was found that three-fact relationships are sufficient to provide a simple yet adequate representation in most situations [7]. Despite the richness of its representations, TFM still relies on a significant amount of manual work to interpret and index images (due to limitations in current image recognition algorithms), in comparison with pixel-based systems. In this paper present a mechanism which aims to eliminate part of this manual work by having the computer to semi-automatically build high level indices. We shall achieve this goal by employing a knowledge based system (KBS) of rules which will automate the process of generating high level indices (outline facts) from low level indices (elementary facts). This KBS will also be able to expand and restructure elementary facts and outline facts into a hierarchy. This expansion is depicted in Figure 1.

Automatic Index Expansion for Concept-Based Image Query

401

In Figure 1, elementary facts are labeled as low level objects, while outline facts are labeled as medium or high level concepts. Basically, apart from the left most set, which contains all the elementary facts, the remaining sets contain outline facts. In Figure 1, a dot represents a fact and a line to the left of the fact relates that particular fact to its components. An outline fact is composed of two or more elementary facts or other outline facts drawn from a lower level. Although Figure 1 only shows one medium level concept, in practice we can have several. From this representation, it is clear that the higher the level of the concept, the less the number of the facts in the set. This is an important characteristic that will be exploited in database search, which will be discussed later. low level object

med. level object/concept

high level concept

Figure 1. Index Expansion

The index is entered manually/extracted automatically from the image by human/computer in term of atomic objects. Atomic objects are defined as the smallest entity that cannot be decomposed further into components. For example if a table is defined as an atomic component, then in the future we would not be able to recognize a table leg or a table top from the database. Therefore, depending on the application, an indexer will have to predetermine to what extent he/she wants to decompose an image object into atomic indices. From these atomic indices, the machine will develop higher level indices using its knowledge-based system. It is the task of the rules to perform the creation of the higher level indices. A rule consists of an IF part, which lists all the conditions that must be satisfied, and a THEN part, which concludes the rule given that the conditions are met. A rule is created by assigning lower level (atomic) objects/indices in the IF part and higher level object/index in the THEN part. Boolean AND or OR can used to define the relationship among the atomic objects in the condition of the rule. The AND part is the necessary condition, and the OR part is the optional condition. By validating the rule conditions with existing indices (which could be elementary facts or outline facts) and obtaining higher level indices from the hypothesis of the rule, we can create a new index entry automatically. In other words, we build a higher level index from lower level (atomic) indices that might be directly recognizable from the image. This indexing mechanism will avoid inconsistency in human perception of the image concept when the process is performed manually. For retrieval, users can take the advantage of these high level indices to speed up the searching time and narrow down

402

Dwi Sutanto and C.H.C. Leung

the search space. We illustrate below how to construct a typical rule to define a high level object 'computer'. IF

THEN

there exists a monitor a CPU a keyboard a mouse the OBJECT is a computer

AND AND AND

In turn, we can treat the object 'computer' as an intermediate level index, and then use it as a condition for a higher level object description as in the following example. IF

THEN

there exists a desk a computer a telephone the OBJECT is an office

AND AND AND

In this way, we structure the index representations into a hierarchy. Therefore, several atomic indices will be able to define intermediate indices and several intermediate indices will be able to define higher level indices and so on. The benefit of this method is the reusability of the rules. Once a rule that defines an object is created, it can be used for different images in the database as well as new images.

3 Index Organization Indices created in the previous Section have to be organized in such a way that will facilitate fast and efficient retrieval for a query. We shall explain this organization through the following example. Suppose that we have four pictures in the database with the following contents: Table 1. Elementary Indices Image #1 ribbon balloon light cake candle people

Image #2 ribbon balloon light Xmas tree present Santa Claus

Image #3 ribbon flower car

Image #4 tree flower lawn

In the database, these objects will be treated as elementary facts (elementary indices) of the pictures which will be stored in an elementary index table. One or more picture numbers are related to each index entry indicating to which images an index entry belongs. For example, images #1, #2, and #3 are related to index ribbon, as ribbon is present in these images.

Automatic Index Expansion for Concept-Based Image Query

403

Suppose that we have created a knowledge based system using the following rules: Rule 1 IF there exists a ribbon AND a balloon AND a light THEN the OBJECT is a decoration Rule 3 IF there exists a tree AND a flower AND lawn THEN the OBJECT is a garden Rule 5 IF there exists a decoration AND a Christmas tree AND a present AND a 'Santa Claus' THEN the OBJECT is a Christmas event

Images [1,2,3] [1,2] [1,2] [1,2] Images [4] [3,4] [4] [4] Images

Rule 2 IF there exists a ribbon AND a flower AND a car THEN the OBJECT is a wedding party Rule 4 IF there exists a decoration AND a cake AND a candle AND people THEN the OBJECT is a birthday party

Images [1,2,3] [3,4] [3] [3] Images [1,2] [1] [1] [1] [1]

[1,2] [2] [2] [2] [2]

Upon the execution of these rules, we will generate new indices that represent intermediate or high level indices. These indices will be stored in different tables corresponding to the level of abstraction or the stage of the creation of the index. Table 2. Index Table

Atomic Index Table ribbon 1,2,3

Intermediate Index Table decoration 1,2

balloon candle cake people Xmas tree present Santa Claus light car flower tree lawn

Wedding P. garden

1,2 1 1 1 2 2 2 1,2 3 3,4 4 4

3 4

High Level Index Table Birthday 1 party Xmas event 2

404

Dwi Sutanto and C.H.C. Leung

In our algorithm we have to include picture numbers as extended conditions of the rule, because indices are bound to picture numbers. To evaluate the rule, the inference engine requires that all of the conditions are satisfied. It is possible that one rule will satisfy more than one picture, however it is required that all of the rule conditions have to satisfy each picture. Table 2 illustrates the index tables corresponding to the above rules.

4 Query Processing Processing a query for such a database consists of searching and matching between query components and index components. For elementary facts, the program will compare elementary facts from the query and those from the database index. For modified facts, matching has also to be performed on the modifier and elementary fact. For binary and ternary facts, the algorithm have to also verify any existing relationship among the facts. These are the basic operation of the query processor; however, other factors need to be considered so as to achieve a more efficient and effective search. We examine below this processing mechanism in more detail. Weights or significance measures provide users with a means to give priorities to selected query components in order to optimize the results of the query. Applying higher fact weight values will increase the value of the fact recall rate (since these would be more significantly figured in the retrieval ranking), while higher modifier and link weight values will increase the value of the precision rate. Depending on the query outcome, user may want to further tailor the query results by widening the scope of the result or narrowing the result. Weights are also given for other extended features such as background/foreground image, image category, color, etc. Processing the query commences from user input. The query may be entered using a natural language, providing it has a structure similar to the indexing language in the TFM. The query is processed to remove all unwanted components. Because TFM only covers a subset of the natural language, full natural language processing is not needed. A query sentence is broken down into several query clauses. Each query clause is processed separately. The results of each of these query clauses will be merged at a later stage to obtain the final result of the query. Each clause phrase is then transformed into the Ternary Fact Model structure, to obtain query components. A summary of the components extraction procedure is sketched below. 1. The first noun identified from a clause is considered as an elementary subject fact candidate. 2. If the elementary subject fact candidate is directly followed by one or more nouns, then the last noun is considered as the actual elementary subject fact. 3. Words (nouns or adjectives) before the elementary subject fact is considered as modifiers. 4. The first noun extracted from the clause after the elementary subject fact will be treated as an elementary object fact candidate. The actual elementary object fact is determined as in step 2.

Automatic Index Expansion for Concept-Based Image Query

405

5. Words between the elementary subject fact phrase and elementary object fact phrase are considered as a fact linker. 6. The same procedure applies for finding the ternary fact components. Once we get all of the facts, the search engine is ready to perform a search on the database. The actual search that needs to be conducted is in the elementary/outline fact table, because all the elementary/outline facts are stored in this table, which means that the search result is sufficient to determine whether or not any further search is needed. Thus, if this search does not result in any match, then any further search in other tables will not yield anything. Indices in this table are organized as a one to one relationship between elementary facts and image reference numbers indicating to which images the facts belong. For every match, the search engine will record the reference number which will be used as a pointer to data in other tables. Basically the search engine will only need to verify which index modifiers among the set pointed to by the reference numbers from the main search match the query modifiers. The result from these verifications should return an even smaller set of reference numbers than the previous one. Similarly the reference number from this modified fact table will be used to verify data in binary facts table, which in turn will be used to verify data in the ternary facts table using the same technique. This algorithm eliminates searching time needed at higher level tables (modified, binary, and ternary facts tables) using the first level search. We diagrammatically depict the whole process in Figure 2. ref. fact v ref.

mod. fact v ref.

fact v

link fact v

img10 dog 0.6 img10 brown dog 0.6 img10 dog 0.7 eat bone 0.1

elementary facts modified facts binary facts

Figure 2. Query Processing

In Figure 2, there is included a field v associated with each record. This is a value field obtained from the image source which represents the significance measure for each associated fact within the image. This value is also returned by a function within the search engine for every match found. Figure 3 illustrates the query calculations.

P( Q| I n ) = ∑ Wi P( Fi | I n ) i

(1)

406

Dwi Sutanto and C.H.C. Leung

P(Q|In) = siginificance measure for image In P(Fi|In) = significance value of feature i in image In Wi = user-specified query weight for feature i Fi = query features i = number of features

RESULTS QUERY Q Wi Fi P(Fi|In) In

Figure 3. Query Formula

Upon any match the query processor will assign hit values for all individual facts and then calculate total hit values for a query clause using Equation 1 [18] for each image recalled. Equation 1 basically adds all weighted features being queried in a given image. The value returned by the formula indicates how close a recalled image matches to a given query.

5 Relevance Feedback Relevance feedback mechanism allows users to fine tune their queries by indicating to the system which query results are relevant. Usually this is achieved by clicking on a number of relevant images from the query results. The system will further search the database to find more images which are similar to users’ feedback. To process this feedback, a feedback processor will obtain index data from the selected images to be used as a new query. To maintain the track of the query, the new query will have to include the initial query as well. This makes possible similarity retrieval based on high-level objects rather than low-level features as is the case for most other systems. ref. fact Rb Rf v 5 a 5 b 5 c img10 p 5 7 0.6 7 q r 7

R u le 5 : if a , b , c th e n p R u le 7 : if p , q th e n r a p b c q ru le 5 ru le 7

r

Figure 4. Feedback Mechanism

The above feedback is beneficial provided that the first query result successfully recalls a number of relevant images. However, sometimes it is necessary to widen or to narrow the query concepts, if there is no relevant images being recalled. This can be done by following the rule network. To support this feature, we include a field to store the rule number for every outline facts deduced during the indexing process. Figure 4 details this mechanism, which illustrates two joining rules, i.e. rules 5 and 7. Suppose

Automatic Index Expansion for Concept-Based Image Query

407

that the query processor recalls entity p. Using this rule network as an example, we could widen the query by following (move forward) the path of rule 7, or narrow the query by following (move backward) the path of rule 5.

6 Concluding Remarks Search effectiveness on an image database is always a trade-off between the indexing cost and semantic richness. To support a high degree of semantic richness, the cost of indexing tends to be high as significant human involvement is necessary. Some degree of human indexing involvement is inevitable as automatic recognition algorithms are still unable to provide, even in a limited way, the semantic level required of many applications. Fully automatic indexing of semantically rich contents is unlikely to be achievable in the foreseeable future. We have presented a solution that provides a significant degree of semantic richness but at the same time limiting the indexing cost. Ternary Fact Model query schemes are able to enhance the query speed by adopting a semantically rich canonical form for high-level image content information, and exploiting the structural efficiency of conventional database search. Through the use of rules, which can be either pre-defined or dynamically incorporated, a new level of semantic richness can be established, which will eliminate the costly indexing of individual concepts. Necessarily, this kind of semi-automatic, high-level indexing entails some degree of uncertainty which would affect adversely the retrieval precision. This requires the use of weights to indicate the degree of relevance of query outcomes which are often linked to the measure of reliability of the deductive rules for generating an index entry. However, compared with the cost of manually entering every index entry, such a degree of uncertainty appears to be quite workable and acceptable.

References 1. Sutanto, D. and Leung, C.H.C., “Automatic Image Database Indexing”, 2. 3. 4. 5. 6.

Proceedings of the Multimedia and Visual Information Systems Technology Workshop, pp. 15-19, October 1997. Gudivada, Venkat N. and Raghavan, Vijay V., “Content-Based Image Retrieval Systems”, IEEE Computer, pp. 18-31, 1995. Barber, R. et.all., “ULTIMEDIA MANAGER: Query By Image Content and its Applications”, IEEE Comput. Soc. Press: Digest of Papers, Spring Compcon ‘94, pp. 424-429, 1994. Barber, R. et.all., “A Guided Tour of Multimedia Systems and Applications: Query by Content for Large On-Line Image Collections”, IEEE Computer Society Press, pp. 357-378, 1995. Flickner, Myron et.all., “Query by Image and Video Content: The QBIC System”, IEEE Computer, Vol. 28 Issue 9, pp. 23-32, September 1995. Campanai, M., Del Bimbo, A., and Nesi, P., “Using 3D Spatial Relationships for Image Retrieval by Contents”, Proc. IEEE Workshop on Visual Languages, 1992.

408

Dwi Sutanto and C.H.C. Leung

7. Leung, C.H.C. and Zheng, Z.J., “Image Data Modelling for Efficient Content

Indexing”, Proc IEEE International Workshop on Multi-Media Database Management Systems, pp. 143-150, 1995. 8. Yang, Li and Wu, Jiankang, “Towards a Semantic Image Database System,” Data & Knowledge Engineering, Vol. 22, No. 3, pp. 207-227, May 1997. 9. Li, Wen-Syan et al., ”Hierarchical Image Modeling for Object-Based Media Retrieval”, Data & Knowledge Engineering, Vol. 27, No. 2, pp. 139-176, September 1998. 10. Jorgensen, Corinne, “Attributes of Images in Describing Tasks”, Information Processing & Management, Vol. 34, No. 2/3, pp. 161-174, March/May 1998. 11. Shakir, Hussain Sabri, “Context-Sensitive Processing of Semantic Queries in an Image Database System”, Information Processing & Management, Vol. 32, No. 5, pp. 573-600, 1996. 12. Gudivada, Venkat N., “Modeling and Retrieving Images by Content”, Information Processing & Management, Vol. 33, No. 4, pp. 427-452, 1997. 13. Eakins, John P. et al., “Similarity Retrieval of Trademark Images”, IEEE Multimedia, April-June 1998. 14. Leung, C. H. C. and Sutanto, D. “Multimedia Data Modeling and Management for Semantic Content Retrieval”, in Handbook of Multimedia Computing, Furht, B. (Ed.), CRC Press (To Appear). 15. Chua, Tat-Seng et al., “A Concept-Based Image Retrieval System”, Proceedings of the Twenty Seventh Hawaii International Conference on System Sciences, Vol. 3, pp. 590-598, Jan 1994. 16. Chang, S. F. et. al., “Visual Information Retrieval from Large Distributed Online Repositories”, Comm. ACM, Vol. 40, Dec 1997, pp. 63-71. 17. Srihari, Rohini K., “Automatic Indexing and Content-Based Retrieval of Captioned Images”, IEEE Computer, pp. 49-56, September 1995. 18. Hou, Tai Yuan, et al, “Medical Image Retrieval by Spatial Features”, 1992 IEEE International Conference on Systems, Man, and Cybernetics, Vol. 2, pp. 1364-9, October 1992. 19. Grosky, W. I., “Managing Multimedia Information in Database Systems”, Comm. ACM, Vol. 40, Dec 1997, pp. 72-80.

Structured High-Level Indexing of Visual Data Content Audrey M. Tam and Clement H.C. Leung Communications & Informatics Victoria University of Technology, Australia Footscray Campus (FO119), P.O. Box 14428 Melbourne CMC, VIC 8001, Australia Fax:+61 3 9688 4050 {amt,clement}@matilda.vut.edu.au

Abstract. Unstructured manual high-level indexing is too open-ended to be useful. Domain-based classification schemes reduce the variability of index captions and increase the efficiency of manual indexing by limiting the indexer's options. In this paper, we incorporate classification hierarchies into an indexing superstructure of metadata, context and content, incorporating high-level content descriptions based on the ternary fact model. An extended illustration is given to show how metadata can be automatically extracted and can subsequently help to further limit the indexer's options for context and content. Thus, this structure facilitates the indexing of high-level contents and allows semantically rich concepts to be efficiently incorporated. We also propose a form of data mining on this index to determine rules that can be used to semi-automatically (re)classify images.

1

Classification Schemes for High-Level Indexing

Multimedia databases must store a vast amount of information about their data objects – data about the semantic content and data about the data [4,5,6,8]. Both indexing and querying of visual databases can be done on the basis of low-level data content – texture, colour, shape etc. – and high-level semantic content – what the image “means” to humans. Low-level content can be automatically extracted for indexing but high-level content must be indexed manually or semi-automatically. Unstructured manual high-level indexing is too open-ended to be useful: like a Rorschach test, the results say more about the human indexer's viewpoint than about the images [7]. Domain-based classification schemes can reduce the variability of index captions and increase the efficiency of manual indexing by limiting the indexer's options. A classification scheme may resemble the conventional library classification system or it may be structured to accommodate a particular domain. It might include (Is-a) hierarchies of subclasses and (Has-a) hierarchies of components; hence, it need not be a tree structure. For example, in the transport domain, a bike lane is a subclass of onDionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL'99, LNCS 1614, pp. 409-416, 1999.  Springer-Verlag Berlin Heidelberg 1999

410

Audrey M. Tam and Clement H.C. Leung

road facility and also a component of a road. Classification schemes are able to remove large portions of the search space from consideration. With multiple classification schemes, it would be possible to reduce the set of candidate images to manageable proportions (Figure 1). The leaf nodes in the search trees in Figure 1 would correspond to data object id, with the links either implemented as pointers or logical references, in which case set operations (e.g., intersection) would need to be incorporated. Classification Scheme 1

Classification Scheme 2

Candidate Set Figure 1. Search tree pruning using multiple classification

Although classification schemes will help to limit the search, they are seldom sufficiently exhaustive to allow the target images to be pinpointed exactly, in which case the underlying classification scheme can be supplemented by the ternary fact model proposed in [10]. This data model, which forms the basis for the systematic representation of image content, has as its basic building blocks facts – nodes in the underlying classification scheme. These may be modified and linked together in pairs or triplets to express the subject matter of an image. Examples of facts in the transport domain are given below; elementary and outline facts are in upper case, modifiers are in lower case and links are in title case: • • • • •

Elementary Facts: TRAM, BICYCLE Outline Fact: TRAFFIC JAM, CRITICAL MASS RIDE Modified Facts: articulated TRAM, folding BICYCLE Binary Fact: tall MAN Riding folding BICYCLE Ternary Fact: yellow TAXI Hitting red CAR Towing TRAILER

We can incorporate classification hierarchies and high-level content descriptions based on the ternary fact model into an indexing superstructure based on the following idea. Within a particular domain – tourism, sports, transport – the attributes of data objects can be divided into three categories:

Structured High-Level Indexing of Visual Data Content

•

• •

411

Metadata or data about the data object or its contents, usually not derivable from viewing the data object itself, e.g., materials used in constructing a racing bicycle or the location of a bike path. The media type of the data object is also a kind of metadata. A standard for describing multimedia data is the goal of MPEG-7 (Multimedia Content Description Interface) [11]. In many cases, metadata treat a data object or its contents as entities within the EntityRelationship framework of database design and store relevant attributes of the entities as structured data fields in either a relational database or an object database. Context or the data object's significance in this domain, e.g., a racing bicycle in sports or a bicycle lane in transport. Content or the data object itself and the information contained in the data object that is meaningful for this domain. The content of the data object may be distinguished into high-level and low-level contents. In this paper, we concentrate on high-level content, which can be difficult to extract automatically. Additional structure can be imposed on content, e.g., image objects may be judged to be in the foreground, background or middleground.

The distinction between context and content may be expressed by analogy with books. Context is the subject classification of a book and content is the information in a book, which is usually structured into chapters or sections. An image of Montmartre contains artists and easels; a book about Richard Feynman contains an explanation of quantum electrodynamics. Other examples are shown in Figure 2, which also demonstrates how the same image can be indexed differently in two different domains, with different entities linked to the metadata database. The sports domain metadata database contains tables for athletes and sporting equipment while the transport domain metadata database contains tables for bicycling facilities. Sports: Racing bicyclist Kathy Watt

Transport: Bike lane Metadata

Image of Kathy Watt on

riding on

St. Kilda Road

monocoque Metadata

bike lane

Female cyclist riding on St. Kilda Rd bike lane

Figure 2. Context, content and metadata of same image in different domains

412

Audrey M. Tam and Clement H.C. Leung

We shall describe a data model that forms the basis for the systematic representation of image content, using domain-based classification schemes to create a structured indexing syntax. This structure facilitates manual indexing of high-level content and, through the use of data mining rules, allows semantically rich concepts to be efficiently incorporated.

2

Example Application

An example application will illustrate the level of detail that may be appropriate for each set of attributes. Tourism is an application domain that relies heavily on images to attract customers. A tourism image database would store photographs of tourist destinations and activities. Retrieved images could be used in the production of brochures for general distribution or tailored to the interests of specific customers. 2.1

Metadata

The metadata of a tourism image includes: ! ! ! ! !

Administrative data to control access and payment: identifiers of creators and editors, creation and modification data, usage fee codes Data about the image structure for structure-dependent image-analysis operations and for composition into a presentation document: image format, resolution, bits per pixel, compression data Geographical location: region, country, state, city, locality or GPS coordinates. Calendar time of the event represented in the image (usually the same as the creation time): month, day, hour, minute, second to the appropriate level of precision. Can be recorded by the camera and may even be part of the image. Other domain-dependent data such as hotel tariffs, height of monuments etc. The indexer would not be required to enter most of this data as it would be archived and updated as part of a conventional database.

The first four types of metadata would be needed for any photographic image. Note that most of these metadata can be (semi-)automatically derived from the image or require the indexer to specify or select only an identifier that links into a conventional database. Linking other existing conventional databases as metadata can extend the utility of the image database. For example, linking to a conservation database can generate environmentally aware captions such as the following: Alpine butterfly on an orange everlasting, Mt Buffalo National Park, Victoria. The ecology of alpine areas could be affected by rising temperatures associated with the greenhouse effect. [2]

Structured High-Level Indexing of Visual Data Content

2.2

413

Context

The context of an image can be selected from the application domain’s classification hierarchies. In our tourism example, the scene might represent a particular celebration, landmark, tourist activity, accommodation etc. Location and time metadata can be used here to prune the classification trees presented to the indexer.

2.3

High-Level Content

The high-level content of interest to the user is entirely domain-dependent. The domain determines the objects and relationships that users are likely to search for and the vocabulary that users are likely to employ when searching for semantic content. Because the context-level classifications remove ambiguities, captions at this level can convey more detail about the significant objects in an image. Because the metadata include geographical and temporal data as well as metadata of significant objects, captions do not need to repeat these data. A significant entity needs only an identifier to link it into the metadata database. We suggest the following objects1 and modifiers for a tourism image database: ! !

!

! !

Person := NATIVE|VISITO