Lecture Notes in Computer Science Commenced Publication in 1973 Founding and Former Series Editors: Gerhard Goos, Juris Hartmanis, and Jan van Leeuwen
Editorial Board David Hutchison Lancaster University, UK Takeo Kanade Carnegie Mellon University, Pittsburgh, PA, USA Josef Kittler University of Surrey, Guildford, UK Jon M. Kleinberg Cornell University, Ithaca, NY, USA Alfred Kobsa University of California, Irvine, CA, USA Friedemann Mattern ETH Zurich, Switzerland John C. Mitchell Stanford University, CA, USA Moni Naor Weizmann Institute of Science, Rehovot, Israel Oscar Nierstrasz University of Bern, Switzerland C. Pandu Rangan Indian Institute of Technology, Madras, India Bernhard Steffen TU Dortmund University, Germany Madhu Sudan Microsoft Research, Cambridge, MA, USA Demetri Terzopoulos University of California, Los Angeles, CA, USA Doug Tygar University of California, Berkeley, CA, USA Gerhard Weikum Max Planck Institute for Informatics, Saarbruecken, Germany
6524
Kuo-Tien Lee Wen-Hsiang Tsai Hong-Yuan Mark Liao Tsuhan Chen Jun-Wei Hsieh Chien-Cheng Tseng (Eds.)
Advances in Multimedia Modeling 17th International Multimedia Modeling Conference, MMM 2011 Taipei, Taiwan, January 5-7, 2011 Proceedings, Part II
13
Volume Editors Kuo-Tien Lee Jun-Wei Hsieh National Taiwan Ocean University Keelung, Taiwan E-mail: {po,shieh}@mail.ntou.edu.tw Wen-Hsiang Tsai National Chiao Tung University Hsinchu, Taiwan E-mail:
[email protected] Hong-Yuan Mark Liao Academia Sinica Taipei, Taiwan E-mail:
[email protected] Tsuhan Chen Cornell University Ithaca, NY, USA E-mail:
[email protected] Chien-Cheng Tseng National Kaohsiung First University of Science and Technology Kaohsiung, Taiwan E-mail:
[email protected]
Library of Congress Control Number: 2010940989 CR Subject Classification (1998): H.5.1, I.5, H.3, H.4, I.4, H.2.8 LNCS Sublibrary: SL 3 – Information Systems and Application, incl. Internet/Web and HCI ISSN ISBN-10 ISBN-13
0302-9743 3-642-17828-6 Springer Berlin Heidelberg New York 978-3-642-17828-3 Springer Berlin Heidelberg New York
This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. springer.com © Springer-Verlag Berlin Heidelberg 2011 Printed in Germany Typesetting: Camera-ready by author, data conversion by Scientific Publishing Services, Chennai, India Printed on acid-free paper 06/3180
Preface
Welcome to the proceedings of the 17th Multimedia Modeling Conference (MMM 2011) held in Taipei, Taiwan, during January 5–7, 2011. Following the success of the 16 preceding conferences, the 17th MMM brought together researchers, developers, practitioners, and educators in the field of multimedia. Both practical systems and theories were presented in this conference, thanks to the support of Microsoft Research Asia, Industrial Technology Research Institute, Institute for Information Industry, National Museum of Natural Science, and the Image Processing and Pattern Recognition Society of Taiwan. MMM 2011 featured a comprehensive program including keynote speeches, regular paper presentations, posters, and special sessions. We received 450 papers in total. Among these submissions, we accepted 75 oral presentations and 21 poster presentations. Six special sessions were organized by world-leading researchers. We sincerely acknowledge the Program Committee members who have contributed much of their precious time during the paper reviewing process. We would like to sincerely thank the support of our strong Organizing Committee and Advisory Committee. Special thanks go to Jun-Wei Hsieh, Tun-Wen Pai, Shyi-Chyi Cheng, Hui-Huang Hsu, Tao Mei, Meng Wang, Chih-Min Chao, Chun-Chao Yeh, Shu-Hsin Liao, and Chin-Chun Chang. This conference would never have happened without their help.
January 2011
Wen-Hsiang Tsai Mark Liao Kuo-Tien Lee
Organization
MMM 2011 was hosted and organized by the Department of Computer Science and Engineering, National Taiwan Ocean University, Taiwan. The conference was held at the National Taiwan Science Education Center, Taipei, during January 5–7, 2011.
Conference Committee Steering Committee
Conference Co-chairs
Program Co-chairs
Special Session Co-chairs Demo Co-chair Local Organizing Co-chairs
Publication Chair Publicity Chair
Yi-Ping Phoebe Chen (La Trobe University, Australia) Tat-Seng Chua (National University of Singapore, Singapore) Tosiyasu L. Kunii (University of Tokyo, Japan) Wei-Ying Ma (Microsoft Research Asia, China) Nadia Magnenat-Thalmann (University of Geneva, Switzerland) Patrick Senac (ENSICA, France) Kuo-Tien Lee (National Taiwan Ocean University, Taiwan) Wen-Hsiang Tsai (National Chiao Tung University, Taiwan) Hong-Yuan Mark Liao (Academia Sinica, Taiwan) Tsuhan Chen (Cornell University, USA) Jun-Wei Hsieh (National Taiwan Ocean University, Taiwan) Chien-Cheng Tseng (National Kaohsiung First University of Science and Technology, Taiwan) Hui-Huang Hsu (Tamkang University, Taiwan) Tao Mei (Microsoft Research Asia, China) Meng Wang (Microsoft Research Asia, China) Tun-Wen Pai (National Taiwan Ocean University, Taiwan) Shyi-Chyi Cheng (National Taiwan Ocean University, Taiwan) Chih-Min Chao (National Taiwan Ocean University, Taiwan) Shu-Hsin Liao (National Taiwan Ocean University, Taiwan)
VIII
Organization
US Liaison Asian Liaison European Liaison Webmaster
Qi Tian (University of Texas at San Antonio, USA) Tat-Seng Chua (National University of Singapore, Singapore) Susanne Boll (University of Oldenburg, Germany) Chun-Chao Yeh (National Taiwan Ocean University, Taiwan)
Program Committee Allan Hanbury Andreas Henrich Bernard Merialdo Brigitte Kerherve Cathal Gurrin Cees Snoek Cha Zhang Chabane Djeraba Changhu Wang Changsheng Xu Chia-Wen Lin Chong-Wah Ngo Christian Timmerer Colum Foley Daniel Thalmann David Vallet Duy-Dinh Le Fernando Pereira Francisco Jose Silva Mata Georg Thallinger Guntur Ravindra Guo-Jun Qi Harald Kosch Hui-Huang Hsu Jen-Chin Jiang Jia-hung Ye Jianmin Li Jianping Fan Jiebo Luo Jing-Ming Guo Jinhui Tang
Vienna University of Technology, Austria University of Bamberg, Germany EURECOM, France University of Quebec, Canada Dublin City University, Ireland University of Amsterdam, The Netherlands Microsoft Research University of Sciences and Technologies of Lille, France University of Science and Technology of China NLPR, Chinese Academy of Science, China National Tsing Hua University, Taiwan City University of Hong Kong, Hong Kong University of Klagenfurt, Austria Dublin City University, Ireland EPFL, Swiss Universidad Aut´ onoma de Madrid, Spain National Institute of Informatics, Japan Technical University of Lisbon, Portugal Centro de Aplicaciones de Tecnologias de Avanzada, Cuba Joanneum Research, Austria Applied Research & Technology Center, Motorola, Bangalore University of Science and Technology of China Passau University, Germany Tamkang University, Taiwan National Dong Hwa University, Taiwan National Sun Yat-sen University, Taiwan Tsinghua University, China University of North Carolina, USA Kodak Research, USA National Taiwan University of Science and Technology, Taiwan University of Science and Technology of China
Organization
Jinjun Wang Jiro Katto Joemon Jose Jonathon Hare Joo Hwee Lim Jose Martinez Keiji Yanai Koichi Shinoda Lap-Pui Chau Laura Hollink Laurent Amsaleg Lekha Chaisorn Liang-Tien Chia Marcel Worring Marco Bertini Marco Paleari Markus Koskela Masashi Inoue Matthew Cooper Matthias Rauterberg Michael Lew Michel Crucianu Michel Kieffer Ming-Huei Jin Mohan Kankanhalli Neil O’Hare Nicholas Evans Noel O’Connor Nouha Bouteldja Ola Stockfelt Paul Ferguson Qi Tian Raphael Troncy Roger Zimmermann Selim Balcisoy Sengamedu Srinivasan Seon Ho Kim Shen-wen Shr Shingo Uchihashi Shin’ichi Satoh
IX
NEC Laboratories America, Inc., USA Waseda University, Japan University of Glasgow, UK University of Southampton, UK Institute for Infocomm Research, Singapore UAM, Spain University of Electro-Communications, Japan Tokyo Institute of Technology, Japan Nanyang Technological University, Singapore Vrije Universiteit Amsterdam, The Netherlands CNRS-IRISA, France Institute for Infocomm Research, Singapore Nanyang Technological University, Singapore University of Amsterdam, The Netherlands University of Florence, Italy EURECOM, France Helsinki University of Technology, Finland Yamagata University, Japan FX Palo Alto Lab, Inc., Germany Technical University Eindhoven, The Netherlands Leiden University, The Netherlands Conservatoire National des Arts et M´etiers, France Laboratoire des Signaux et Syst`emes, CNRS-Sup´elec, France Institute for Information Industry, Taiwan National University of Singapore Dublin City University, Ireland EURECOM, France Dublin City University, Ireland Conservatoire National des Arts et M´etiers, France Gothenburg University, Sweden Dublin City University, Ireland University of Texas at San Antonio, USA CWI, The Netherlands University of Southern California, USA Sabanci University, Turkey Yahoo! India University of Denver, USA National Chi Nan University, Taiwan Fuji Xerox Co., Ltd., Japan National Institute of Informatics, Japan
X
Organization
Shiuan-Ting Jang Shuicheng Yan Shu-Yuan Chen Sid-Ahmed Berrani Stefano Bocconi Susu Yao Suzanne Little Tao Mei Taro Tezuka Tat-Seng Chua Thierry Pun Tong Zhang Valerie Gouet-Brunet Vincent Charvillat Vincent Oria Wai-tian Tan Wei Cheng Weiqi Yan Weisi Lin Wen-Hung Liau Werner Bailer William Grosky Winston Hsu Wolfgang H¨ urst Xin-Jing Wang Yannick Pri´e Yan-Tao Zheng Yea-Shuan Huang Yiannis Kompatsiaris Yijuan Lu Yongwei Zhu Yun Fu Zha Zhengjun Zheng-Jun Zha Zhongfei Zhang Zhu Li
National Yunlin University of Science and Technology, Taiwan National University of Singapore Yuan Ze University, Taiwan Orange Labs - France Telecom Universit`a degli studi di Torino, Italy Institute for Infocomm Research, Singapore Open University, UK Microsoft Research Asia, China Ritsumeikan University, Japan National University of Singapore University of Geneva, Switzerland HP Labs Conservatoire National des Arts et Metiers, France University of Toulouse, France NJIT, USA Hewlett-Packard, USA University of Michigan, USA Queen’s University Belfast, UK Nanyang Technological University, Singapore National Chengchi University, Taiwan Joanneum Research, Austria University of Michigan, USA National Taiwan University, Taiwan Utrecht University, The Netherlands Microsoft Research Asia, China LIRIS, France National University of Singapore, Singapore Chung-Hua University, Taiwan Informatics and Telematics Institute Centre for Research and Technology Hellas, Greece Texas State University, USA Institute for Infocomm Research Asia, Singapore University at Buffalo (SUNY), USA National University of Singapore, Singapore National University of Singapore, Singapore State University of New York at Binghamton, USA Hong Kong Polytechnic University, Hong Kong
Organization
Sponsors Microsoft Research Industrial Technology Research Institute Institute For Information Industry National Taiwan Science Education Center National Taiwan Ocean University Bureau of Foreign Trade National Science Council
XI
Table of Contents – Part II
Special Session Papers Content Analysis for Human-Centered Multimedia Applications Generative Group Activity Analysis with Quaternion Descriptor . . . . . . . Guangyu Zhu, Shuicheng Yan, Tony X. Han, and Changsheng Xu
1
Grid-Based Retargeting with Transformation Consistency Smoothing . . . Bing Li, Ling-Yu Duan, Jinqiao Wang, Jie Chen, Rongrong Ji, and Wen Gao
12
Understanding Video Sequences through Super-Resolution . . . . . . . . . . . . Yu Peng, Jesse S. Jin, Suhuai Luo, and Mira Park
25
Facial Expression Recognition on Hexagonal Structure Using LBP-Based Histogram Variances . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Lin Wang, Xiangjian He, Ruo Du, Wenjing Jia, Qiang Wu, and Wei-chang Yeh
35
Mining Social Relationship from Media Collections Towards More Precise Social Image-Tag Alignment . . . . . . . . . . . . . . . . . . . Ning Zhou, Jinye Peng, Xiaoyi Feng, and Jianping Fan Social Community Detection from Photo Collections Using Bayesian Overlapping Subspace Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Peng Wu, Qiang Fu, and Feng Tang Dynamic Estimation of Family Relations from Photos . . . . . . . . . . . . . . . . Tong Zhang, Hui Chao, and Dan Tretter
46
57
65
Large Scale Rich Media Data Management Semi-automatic Flickr Group Suggestion . . . . . . . . . . . . . . . . . . . . . . . . . . . . Junjie Cai, Zheng-Jun Zha, Qi Tian, and Zengfu Wang A Visualized Communication System Using Cross-Media Semantic Association . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Xinming Zhang, Yang Liu, Chao Liang, and Changsheng Xu
77
88
XIV
Table of Contents – Part II
Effective Large Scale Text Retrieval via Learning Risk-Minimization and Dependency-Embedded Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Sheng Gao and Haizhou Li
99
Efficient Large-Scale Image Data Set Exploration: Visual Concept Network and Image Summarization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chunlei Yang, Xiaoyi Feng, Jinye Peng, and Jianping Fan
111
Multimedia Understanding for Consumer Electronics A Study in User-Centered Design and Evaluation of Mental Tasks for BCI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Danny Plass-Oude Bos, Mannes Poel, and Anton Nijholt Video CooKing: Towards the Synthesis of Multimedia Cooking Recipes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Keisuke Doman, Cheng Ying Kuai, Tomokazu Takahashi, Ichiro Ide, and Hiroshi Murase
122
135
Snap2Read: Automatic Magazine Capturing and Analysis for Adaptive Mobile Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yu-Ming Hsu, Yen-Liang Lin, Winston H. Hsu, and Brian Wang
146
Multimodal Interaction Concepts for Mobile Augmented Reality Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Wolfgang H¨ urst and Casper van Wezel
157
Image Object Recognition and Compression Morphology-Based Shape Adaptive Compression . . . . . . . . . . . . . . . . . . . . . Jian-Jiun Ding, Pao-Yen Lin, Jiun-De Huang, Tzu-Heng Lee, and Hsin-Hui Chen People Tracking in a Building Using Color Histogram Classifiers and Gaussian Weighted Individual Separation Approaches . . . . . . . . . . . . . . . . Che-Hung Lin, Sheng-Luen Chung, and Jing-Ming Guo Human-Centered Fingertip Mandarin Input System Using Single Camera . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chih-Chang Yu, Hsu-Yung Cheng, Bor-Shenn Jeng, Chien-Cheng Lee, and Wei-Tyng Hong Automatic Container Code Recognition Using Compressed Sensing Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chien-Cheng Tseng and Su-Ling Lee
168
177
187
196
Table of Contents – Part II
Combining Histograms of Oriented Gradients with Global Feature for Human Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Shih-Shinh Huang, Hsin-Ming Tsai, Pei-Yung Hsiao, Meng-Qui Tu, and Er-Liang Jian
XV
208
Interactive Image and Video Search Video Browsing Using Object Trajectories . . . . . . . . . . . . . . . . . . . . . . . . . . Felix Lee and Werner Bailer Size Matters! How Thumbnail Number, Size, and Motion Influence Mobile Video Retrieval . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Wolfgang H¨ urst, Cees G.M. Snoek, Willem-Jan Spoel, and Mate Tomin An Information Foraging Theory Based User Study of an Adaptive User Interaction Framework for Content-Based Image Retrieval . . . . . . . . Haiming Liu, Paul Mulholland, Dawei Song, Victoria Uren, and Stefan R¨ uger
219
230
241
Poster Session Papers Generalized Zigzag Scanning Algorithm for Non-square Blocks . . . . . . . . . Jian-Jiun Ding, Pao-Yen Lin, and Hsin-Hui Chen
252
The Interaction Ontology Model: Supporting the Virtual Director Orchestrating Real-Time Group Interaction . . . . . . . . . . . . . . . . . . . . . . . . . Rene Kaiser, Claudia Wagner, Martin Hoeffernig, and Harald Mayer
263
CLUENET: Enabling Automatic Video Aggregation in Social Media Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Zhuhua Liao, Jing Yang, Chuan Fu, and Guoqing Zhang
274
Pedestrian Tracking Based on Hidden-Latent Temporal Markov Chain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Peng Zhang, Sabu Emmanuel, and Mohan Kankanhalli
285
Motion Analysis via Feature Point Tracking Technology . . . . . . . . . . . . . . . Yu-Shin Lin, Shih-Ming Chang, Joseph C. Tsai, Timothy K. Shih, and Hui-Huang Hsu Traffic Monitoring and Event Analysis at Intersection Based on Integrated Multi-video and Petri Net Process . . . . . . . . . . . . . . . . . . . . . . . . Chang-Lung Tsai and Shih-Chao Tai Baseball Event Semantic Exploring System Using HMM . . . . . . . . . . . . . . Wei-Chin Tsai, Hua-Tsung Chen, Hui-Zhen Gu, Suh-Yin Lee, and Jen-Yu Yu
296
304 315
XVI
Table of Contents – Part II
Robust Face Recognition under Different Facial Expressions, Illumination Variations and Partial Occlusions . . . . . . . . . . . . . . . . . . . . . . . Shih-Ming Huang and Jar-Ferr Yang Localization and Recognition of the Scoreboard in Sports Video Based on SIFT Point Matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jinlin Guo, Cathal Gurrin, Songyang Lao, Colum Foley, and Alan F. Smeaton 3D Model Search Using Stochastic Attributed Relational Tree Matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Naoto Nakamura, Shigeru Takano, and Yoshihiro Okada A Novel Horror Scene Detection Scheme on Revised Multiple Instance Learning Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Bin Wu, Xinghao Jiang, Tanfeng Sun, Shanfeng Zhang, Xiqing Chu, Chuxiong Shen, and Jingwen Fan
326
337
348
359
Randomly Projected KD-Trees with Distance Metric Learning for Image Retrieval . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Pengcheng Wu, Steven C.H. Hoi, Duc Dung Nguyen, and Ying He
371
A SAQD-Domain Source Model Unified Rate Control Algorithm for H.264 Video Coding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mingjing Ai and Lili Zhao
383
A Bi-objective Optimization Model for Interactive Face Retrieval . . . . . . Yuchun Fang, Qiyun Cai, Jie Luo, Wang Dai, and Chengsheng Lou
393
Multi-symbology and Multiple 1D/2D Barcodes Extraction Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Daw-Tung Lin and Chin-Lin Lin
401
Wikipedia Based News Video Topic Modeling for Information Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Sujoy Roy, Mun-Thye Mak, and Kong Wah Wan
411
Advertisement Image Recognition for a Location-Based Reminder System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Siying Liu, Yiqun Li, Aiyuan Guo, and Joo Hwee Lim
421
Flow of Qi: System of Real-Time Multimedia Interactive Application of Calligraphy Controlled by Breathing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Kuang-I Chang, Mu-Yu Tsai, Yu-Jen Su, Jyun-Long Chen, and Shu-Min Wu Measuring Bitrate and Quality Trade-Off in a Fast Region-of-Interest Based Video Coding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Salahuddin Azad, Wei Song, and Dian Tjondronegoro
432
442
Table of Contents – Part II
Image Annotation with Concept Level Feature Using PLSA+CCA . . . . . Yu Zheng, Tetsuya Takiguchi, and Yasuo Ariki Multi-actor Emotion Recognition in Movies Using a Bimodal Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ruchir Srivastava, Sujoy Roy, Shuicheng Yan, and Terence Sim
XVII
454
465
Demo Session Papers RoboGene: An Image Retrieval System with Multi-Level Log-Based Relevance Feedback Scheme . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Huanchen Zhang, Haojie Li, Shichao Dong, and Weifeng Sun Query Difficulty Guided Image Retrieval System . . . . . . . . . . . . . . . . . . . . . Yangxi Li, Yong Luo, Dacheng Tao, and Chao Xu HeartPlayer: A Smart Music Player Involving Emotion Recognition, Expression and Recommendation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Songchun Fan, Cheng Tan, Xin Fan, Han Su, and Jinyu Zhang Immersive Video Conferencing Architecture Using Game Engine Technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chris Poppe, Charles-Frederik Hollemeersch, Sarah De Bruyne, Peter Lambert, and Rik Van de Walle Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
476 479
483
486
489
Table of Contents – Part I
Regular Papers Audio, Image, Video Processing, Coding and Compression A Generalized Coding Artifacts and Noise Removal Algorithm for Digitally Compressed Video Signals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ling Shao, Hui Zhang, and Yan Liu
1
Efficient Mode Selection with BMA Based Pre-processing Algorithms for H.264/AVC Fast Intra Mode Decision . . . . . . . . . . . . . . . . . . . . . . . . . . . Chen-Hsien Miao and Chih-Peng Fan
10
Perceptual Motivated Coding Strategy for Quality Consistency . . . . . . . . Like Yu, Feng Dai, Yongdong Zhang, and Shouxun Lin Compressed-Domain Shot Boundary Detection for H.264/AVC Using Intra Partitioning Maps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Sarah De Bruyne, Jan De Cock, Chris Poppe, Charles-Frederik Hollemeersch, Peter Lambert, and Rik Van de Walle
21
29
Adaptive Orthogonal Transform for Motion Compensation Residual in Video Compression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Zhouye Gu, Weisi Lin, Bu-sung Lee, and Chiew Tong Lau
40
Parallel Deblocking Filter for H.264/AVC on the TILERA Many-Core Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chenggang Yan, Feng Dai, and Yongdong Zhang
51
Image Distortion Estimation by Hash Comparison . . . . . . . . . . . . . . . . . . . Li Weng and Bart Preneel
62
Media Content Browsing and Retrieval Sewing Photos: Smooth Transition between Photos . . . . . . . . . . . . . . . . . . . Tzu-Hao Kuo, Chun-Yu Tsai, Kai-Yin Cheng, and Bing-Yu Chen
73
Employing Aesthetic Principles for Automatic Photo Book Layout . . . . . Philipp Sandhaus, Mohammad Rabbath, and Susanne Boll
84
Video Event Retrieval from a Small Number of Examples Using Rough Set Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Kimiaki Shirahama, Yuta Matsuoka, and Kuniaki Uehara
96
XX
Table of Contents – Part I
Community Discovery from Movie and Its Application to Poster Generation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yan Wang, Tao Mei, and Xian-Sheng Hua
107
A BOVW Based Query Generative Model . . . . . . . . . . . . . . . . . . . . . . . . . . . Reede Ren, John Collomosse, and Joemon Jose
118
Video Sequence Identification in TV Broadcasts . . . . . . . . . . . . . . . . . . . . . Klaus Schoeffmann and Laszlo Boeszoermenyi
129
Content-Based Multimedia Retrieval in the Presence of Unknown User Preferences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Christian Beecks, Ira Assent, and Thomas Seidl
140
Multi-Camera, Multi-View, and 3D Systems People Localization in a Camera Network Combining Background Subtraction and Scene-Aware Human Detection . . . . . . . . . . . . . . . . . . . . . Tung-Ying Lee, Tsung-Yu Lin, Szu-Hao Huang, Shang-Hong Lai, and Shang-Chih Hung
151
A Novel Depth-Image Based View Synthesis Scheme for Multiview and 3DTV . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Xun He, Xin Jin, Minghui Wang, and Satoshi Goto
161
Egocentric View Transition for Video Monitoring in a Distributed Camera Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Kuan-Wen Chen, Pei-Jyun Lee, and Yi-Ping Hung
171
A Multiple Camera System with Real-Time Volume Reconstruction for Articulated Skeleton Pose Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Zheng Zhang, Hock Soon Seah, Chee Kwang Quah, Alex Ong, and Khalid Jabbar A New Two-Omni-Camera System with a Console Table for Versatile 3D Vision Applications and Its Automatic Adaptation to Imprecise Camera Setups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Shen-En Shih and Wen-Hsiang Tsai 3D Face Recognition Based on Local Shape Patterns and Sparse Representation Classifier . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Di Huang, Karima Ouji, Mohsen Ardabilian, Yunhong Wang, and Liming Chen An Effective Approach to Pose Invariant 3D Face Recognition . . . . . . . . . Dayong Wang, Steven C.H. Hoi, and Ying He
182
193
206
217
Table of Contents – Part I
XXI
Multimedia Indexing and Mining Score Following and Retrieval Based on Chroma and Octave Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Wei-Ta Chu and Meng-Luen Li
229
Incremental Multiple Classifier Active Learning for Concept Indexing in Images and Videos . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Bahjat Safadi, Yubing Tong, and Georges Qu´enot
240
A Semantic Higher-Level Visual Representation for Object Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ismail El Sayad, Jean Martinet, Thierry Urruty, and Chabane Dejraba Mining Travel Patterns from GPS-Tagged Photos . . . . . . . . . . . . . . . . . . . . Yan-Tao Zheng, Yiqun Li, Zheng-Jun Zha, and Tat-Seng Chua Augmenting Image Processing with Social Tag Mining for Landmark Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Amogh Mahapatra, Xin Wan, Yonghong Tian, and Jaideep Srivastava News Shot Cloud: Ranking TV News Shots by Cross TV-Channel Filtering for Efficient Browsing of Large-Scale News Video Archives . . . . Norio Katayama, Hiroshi Mo, and Shin’ichi Satoh
251
262
273
284
Multimedia Content Analysis (I) Speaker Change Detection Using Variable Segments for Video Indexing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . King Yiu Tam, Jose Lay, and David Levy Correlated PLSA for Image Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Peng Li, Jian Cheng, Zechao Li, and Hanqing Lu Genre Classification and the Invariance of MFCC Features to Key and Tempo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tom L.H. Li and Antoni B. Chan Combination of Local and Global Features for Near-Duplicate Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yue Wang, ZuJun Hou, Karianto Leman, Nam Trung Pham, TeckWee Chua, and Richard Chang Audio Tag Annotation and Retrieval Using Tag Count Information . . . . . Hung-Yi Lo, Shou-De Lin, and Hsin-Min Wang
296
307
317
328
339
XXII
Table of Contents – Part I
Similarity Measurement for Animation Movies . . . . . . . . . . . . . . . . . . . . . . . Alexandre Benoit, Madalina Ciobotaru, Patrick Lambert, and Bogdan Ionescu
350
Multimedia Content Analysis (II) A Feature Sequence Kernel for Video Concept Classification . . . . . . . . . . . Werner Bailer
359
Bottom-Up Saliency Detection Model Based on Amplitude Spectrum . . . Yuming Fang, Weisi Lin, Bu-Sung Lee, Chiew Tong Lau, and Chia-Wen Lin
370
L2 -Signature Quadratic Form Distance for Efficient Query Processing in Very Large Multimedia Databases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Christian Beecks, Merih Seran Uysal, and Thomas Seidl Generating Representative Views of Landmarks via Scenic Theme Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yi-Liang Zhao, Yan-Tao Zheng, Xiangdong Zhou, and Tat-Seng Chua Regularized Semi-supervised Latent Dirichlet Allocation for Visual Concept Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Liansheng Zhuang, Lanbo She, Jingjing Huang, Jiebo Luo, and Nenghai Yu Boosted Scene Categorization Approach by Adjusting Inner Structures and Outer Weights of Weak Classifiers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Xueming Qian, Zhe Yan, and Kaiyu Hang A User-Centric System for Home Movie Summarisation . . . . . . . . . . . . . . . Saman H. Cooray, Hyowon Lee, and Noel E. O’Connor
381
392
403
413
424
Multimedia Signal Processing and Communications Image Super-Resolution by Vectorizing Edges . . . . . . . . . . . . . . . . . . . . . . . . Chia-Jung Hung, Chun-Kai Huang, and Bing-Yu Chen
435
Vehicle Counting without Background Modeling . . . . . . . . . . . . . . . . . . . . . Cheng-Chang Lien, Ya-Ting Tsai, Ming-Hsiu Tsai, and Lih-Guong Jang
446
Effective Color-Difference-Based Interpolation Algorithm for CFA Image Demosaicking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yea-Shuan Huang and Sheng-Yi Cheng
457
Table of Contents – Part I
Utility Max-Min Fair Rate Allocation for Multiuser Multimedia Communications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Qing Zhang, Guizhong Liu, and Fan Li
XXIII
470
Multimedia Applications Adaptive Model for Robust Pedestrian Counting . . . . . . . . . . . . . . . . . . . . . Jingjing Liu, Jinqiao Wang, and Hanqing Lu
481
Multi Objective Optimization Based Fast Motion Detector . . . . . . . . . . . . Jia Su, Xin Wei, Xiaocong Jin, and Takeshi Ikenaga
492
Narrative Generation by Repurposing Digital Videos . . . . . . . . . . . . . . . . . Nick C. Tang, Hsiao-Rong Tyan, Chiou-Ting Hsu, and Hong-Yuan Mark Liao
503
A Coordinate Transformation System Based on the Human Feature Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Shih-Ming Chang, Joseph Tsai, Timothy K. Shih, and Hui-Huang Hsu An Effective Illumination Compensation Method for Face Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yea-Shuan Huang and Chu-Yung Li Shape Stylized Face Caricatures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Nguyen Kim Hai Le, Yong Peng Why, and Golam Ashraf i-m-Breath: The Effect of Multimedia Biofeedback on Learning Abdominal Breath . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Meng-Chieh Yu, Jin-Shing Chen, King-Jen Chang, Su-Chu Hsu, Ming-Sui Lee, and Yi-Ping Hung Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
514
525 536
548
559
Generative Group Activity Analysis with Quaternion Descriptor Guangyu Zhu1 , Shuicheng Yan1 , Tony X. Han2 , and Changsheng Xu3 1
Electrical and Computer Engineering, National University of Singapore, Singapore 2 Electrical and Computer Engineering, University of Missouri, USA 3 Institute of Automation, Chinese Academy of Sciences, China {elezhug,eleyans}@nus.edu.sg,
[email protected],
[email protected] Abstract. Activity understanding plays an essential role in video content analysis and remains a challenging open problem. Most of previous research is limited due to the use of excessively localized features without sufficiently encapsulating the interaction context or focus on simply discriminative models but totally ignoring the interaction patterns. In this paper, a new approach is proposed to recognize human group activities. Firstly, we design a new quaternion descriptor to describe the interactive insight of activities regarding the appearance, dynamic, causality and feedback, respectively. The designed descriptor is capable of delineating the individual and pairwise interactions in the activities. Secondly, considering both activity category and interaction variety, we propose an extended pLSA (probabilistic Latent Semantic Analysis) model with two hidden variables. This extended probabilistic graphic paradigm constructed on the quaternion descriptors facilitates the effective inference of activity categories as well as the exploration of activity interaction patterns. The experiments on the realistic movie and human activity databases validate that the proposed approach outperforms the state-ofthe-art results. Keywords: Activity analysis, generative modeling, video description.
1
Introduction
Video-based human activity analysis is one of the most promising applications of computer vision and pattern recognition. In [1], Turaga et al. presented a recent survey of the major approaches pursued over the last two decades. Large amount of the existing work on this problem mainly focused on the relatively simple activities of single person [10,9,5,12,4], e.g., sitting, walking and handwaving, which has achieved particular success. In recent years, recognition of group activity with multiple participators (e.g., fighting and gathering) is gaining increasing amount of interests [19,18,15,17]. Upon the definition given by [1], where an activity is referred to a complex sequence of actions performed by several objects who could be interacting with each other, the interactions among the participants reflect the elementary characteristics of different activities. The effective interaction descriptor is therefore essential for developing sophisticated approaches of activity recognition. K.-T. Lee et al. (Eds.): MMM 2011, Part II, LNCS 6524, pp. 1–11, 2011. c Springer-Verlag Berlin Heidelberg 2011
2
G. Zhu et al.
Most previous research stems from the local representation in image processing. Although the widely used local descriptors are demonstrated to allow for the recognition of activities in the scenes with occlusions and dynamic cluttered backgrounds, they are solely representations of appearance and motion patterns. An effective feature descriptor for activity recognition should have the capacity of describing the video in terms of the object appearance, dynamic motion as well as the interactive properties. With the activity descriptor, how to make the decision for the activity category classification using the feature representation accordingly is another key issue for activity recognition. Two types of approach are widely used: approaches based on generative model [5,4] and the ones based on discriminative model [10,9,19,18,15,12,17]. Considering the mechanism of human perception for group activity, the interactions between objects are firstly distinguished and then synthesized as the activity recognition result. Although discriminative models have been extensively employed because they are much easier to build up, the construction of discriminative models essentially focus on the differences among the activity classes yet ignore the interactive properties involved. Therefore, discriminative models cannot facilitate the interaction analysis and discover the insight of the interactive relations in the activities. In this paper, we firstly investigate how to effectively represent video activities in the interaction context. A new feature descriptor, namely quaternion descriptor, consists of four types components in terms of appearance, individual dynamic, pairwise causalities and feedbacks of the video active objects, respectively. The components in the descriptor describe the appearance and motion patterns as well as encode the interaction properties in the activities. Resorting to the bag-of-words method, the video is represented as a compact bag-ofquaternion feature vector. To recognize the activity category and facilitate the interaction pattern exploration, we then propose to model and classify the activities in a generative framework which is based on an extended pLSA model. Interactions are modeled in the generative framework, which is able to explicitly infer the activity patterns.
2
Quaternion Descriptor for Activity Representation
We propose to construct the quaternion descriptor by extracting the trajectory atoms and then modeling the spatio-temporal interaction information within these trajectories. 2.1
Appearance Component of Quaternion
It has been demonstrated that the appearance information encoded in the image frames can provide critical implication on the semantic categories [21,11]. For video activity recognition over frame sequence, this source of information is also very useful in describing the semantics of activities.
Generative Group Activity Analysis with Quaternion Descriptor
3
In recent years, the well known SIFT feature [6] is acknowledged as one of the most powerful appearance descriptors and has achieved overwhelming success in object categorization and recognition. In our approach, the appearance component of quaternion descriptor is measured as the average of all the SIFT features extracted at the salient points residing on the trajectory. For a motion trajectory with temporal length k, the SIFT average descriptor S is computed from all the SIFT descriptors {S1 , S2 , . . . , Sk } along the trajectory. The essential idea of the appearance representation is two-fold. First, the tracking process ensures that the local image patches on the same trajectory are relatively stable, and therefore the resultant SIFT average descriptor provides a robust representation for certain aspect of visual content in the activity footage. Second, the SIFT average descriptor can also encode partially the temporal context information which will contribute the recognition task [13]. 2.2
Dynamic Component of Quaternion
We propose to calculate the Markov stationary distribution [23] as the dynamic representation in the quaternion. The Markov chain is a powerful tool for modeling the dynamic properties of a system as a compact representation. We consider each trajectory as a dynamic system and extract such a compact representation to measure the spatio-temporal interactions in the activities. Fig. 1 shows the extraction procedure of dynamic component.
Fig. 1. The procedure for dynamic component extraction. (a) Displacement vector quantization; (b) State transition diagram; (c) Occurrence matrix; (d) Markov stationary distribution.
The existing work [20] has demonstrated that a trajectory can be encoded by the Markov stationary distribution π if it can be converted into an ergodic finitestate Markov chain. To facilitate this conversion, a finite number of states are chosen for quantization. Given points P and P within two consecutive frames −−→ on the same trajectory, D = P P denotes the displacement vector of two points. To perform a comprehensive quantization on D, both of the magnitude and orientation are considered as shown in Fig. 1(a). We translate the sequential relations between the displacement vectors into a directed graph, which is similar to the state diagram of a Markov chain (Fig. 1(b)). Further, we establish the equivalent matrix presentation of the graph and perform row-normalization on the matrix to obtain a valid transition matrix P for a certain Markov chain (Fig. 1(c)). Finally, we use the iterative algorithm in [20] to compute the Markov stationary distribution π (Fig. 1(d)), which is
4
G. Zhu et al.
1 (I + P + · · · + Pn ) , (1) n+1 where I is an identity matrix and n = 100 in our experiments. To further reduce the approximation error from using a finite n, π is calculated as the column average of An . More details about the extraction of Markov chain distribution can be found in [20]. An =
2.3
Causality and Feedback Components
To describe the causality and feedback properties, we propose a representation scheme based on Granger causality test (GCT) [8] and time-to-frequency tranform [24]. Given a concurrent motion trajectory pair of Ta = [Ta (1), . . . , Ta (n), . . .] and Tb = [Tb (1), . . . , Tb (n), . . .], we assume that the interaction between two trajectories is a stationary process, i.e., the prediction functions P (Ta (n)|Ta (1 : n − l), Tb (1 : n − l)) and P (Tb (n)|Ta (1 : n − l), Tb (1 : n − l)) do not change within a short time period, where Ta(1 : n − l) = [Ta (1), . . . , Ta (n − l)] and the same to Tb (1 : n − l), l is a time lag avoiding the overfitting issue in prediction. To model P (Ta (n)|Ta (1 : n − l), Tb (1 : n − l)), we can use kth order linear predictor: Ta (n) =
k
β(i)Ta (n − i − l) + γ(i)Tb (n − i − l) + a (n) ,
(2)
i=1
where β(i) and γ(i) are the regression coefficients and a (n) is the Gaussian noise with standard deviation σ(Ta (n)|Ta (1 : n − l), Tb (1 : n − l)). We use the linear predictor to model P (Ta (n)|Ta (1 : n−l)) and the standard deviation of the noise signal is denoted as σ(Ta (n)|Ta (1 : n − l)). According to the GCT theory, we can calculate two measurements, namely causality ratio rc as rc =
σ(Ta (n)|Ta (1 : n − l)) , σ(Ta (n)|Ta (1 : n − l), Tb (1 : n − l))
(3)
which measures the relative strength of the causality, and feedback ratio rf as rf =
σ(Tb (n)|Tb (1 : n − l)) , σ(Tb (n)|Ta (1 : n − l), Tb (1 : n − l))
(4)
which measures the relative strength of the feedback. We then calculate the z-transforms for both sides of Eq. (2). Afterwards, the magnitudes and the phases of the z-transform function at a set of evenly sampled frequencies are employed to describe the digital filter for the style of the pairwise causality/feedback. In our approach, we employ the magnitudes of the frequency response at {0, π/4, π/2, 3π/4, π} and the phases of the frequency response at {π/4, π/2, 3π/4} to form the feature vector fba . Similarly, we can define the feature vector fab by considering Ta as the input and Tb as the output of the digital filter, which characterizes how the object with the trajectory Ta affects the motion of the object with the trajectory Tb . The causality ratio and feedback ratio characterize the strength of one object affecting another one, while the extracted frequency response fab and fba convey
Generative Group Activity Analysis with Quaternion Descriptor
5
how one object affects another one. These mutually complementary features are hence combined to form the causality and feedback components of quaternion descriptor in the pairwise interaction context.
3
Generative Activity Analysis
Given a collection of unlabeled video sequences, we would like to discover a set of classes from them. Each of these classes would correspond to an activity category. Additionally, we would like to be able to understand activities that are composed of the mixture of interaction varieties. This resembles the problem of automatic topic discovery which can be figured out by latent topic analysis. In the following, we will introduce a new generative method based on pLSA modeling [22], which is able to both infer the activity categories and discover the interaction patterns. 3.1
Generative Activity Modeling
Fig. 2 shows the extended pLSA graphic model which is employed with the the consideration of both activity category and interaction variety. Compared with the traditional philosophy, the interaction distribution is modeled in our method and integrated into the graphic framework as a new hidden variable.
Fig. 2. The extended pLSA model with two hidden variables. Nodes are random variables. Shaded ones are observed and unshaded ones are unobserved (hidden). The plates indicate repetitions. d represents video sequence, z is the activity category, r is the interaction variety and w is the activity representation bag-of-word. The parameters of this model are learnt in an unsupervised manner using an improved EM algorithm.
Suppose we have a set of M (j = 1, . . . , M ) video sequences containing the bag-of-words of interaction representations quantized from the vocabulary of size V (i = 1, . . . , V ). The corpus of videos is summarized in a V -by-M cooccurrence table M , where m(wi , dj ) is the number of occurrences of a word wi ∈ W = {w1 , . . . , wV } in video dj ∈ D = {d1 , . . . , dM }. In addition, there are two latent topic variables z ∈ Z = {z1 , . . . , zK } and r ∈ R = {r1 , . . . , rR } which represent the activity category and interaction variety resided in a certain activity. The variable rt is sequentially associated with each occurrence of a word wi in video dj . Extending from the traditional pLSA model, the joint probability P (w, d, z, r) which translates the inference process in Fig. 2 is expressed as follows P (d, w) = P (d)P (z|d)P (r|z)P (w|r) . (5) z∈Z r∈R
6
G. Zhu et al.
It is worth noticing that an equivalent symmetric version of the model can be obtained by inverting the conditional probability P (z|d) with the help of Bayes’ rule, which results in P (d, w) = P (z)P (d|z)P (r|z)P (w|r) . (6) z∈Z r∈R
The standard procedure for maximum likelihood estimation in latent variable models is the Expectation Maximization (EM). For the proposed pLSA model in the symmetric parametrization, Bayes’ rule yields the E-step as P (z)P (d|z)p(r|z)P (w|r) P (z, r|d, w) = , P (z )P (d|z )p(r |z )P (w|r ) P (z|d, w) =
z
r
P (z, r|d, w) ,
P (r|d, w) =
r
P (z, r|d, w) .
(7)
(8)
z
By standard calculations, one arrives at the following M-step re-estimation equations: m(d, w)P (z|d, w) m(d, w)P (z|d, w) w d P (d|z) = , P (w|r) = , (9) m(d , w)P (z|d , w) m(d, w )P (z|d, w ) d ,w
P (z) =
d,w
1 m(d, w)P (z|d, w) , R
1 m(d, w)P (z, r|d, w) , (10) R
P (z, r) =
d,w
P (r|z) =
3.2
P (z, r) , P (z)
d,w
R≡
m(d, w) .
(11)
d,w
Generative Activity Recognition
Given that our algorithm has learnt the activity category models using extended pLSA, our goal is to categorize new video sequences. We have obtained the activity-category-specific interaction distribution P (r|z) and the interaction-pattern-specific video-word distribution P (w|r) from a different set of training sequences at learning stage. When given a new video clip, the unseen video is “projected” on the simplex spanned by the learnt P (r|z) and P (w|r). We need to find the mixing coefficients P (zk |dtest ) such that the Kullback-Leibler divergence between the measured empirical distribution P (w|dtest ) and P (w|dtest ) = K k=1 P (zk |dtest )P (r|zk )P (w|r) is minimized [22]. Similar to the learning scenario, we apply the EM algorithm to find the solution. The sole difference between recognition and learning is that the learnt P (r|z) and P (w|r) are never updated during inference. Thus, a categorization decision is made by selecting the activity category that best explains the observation, that is Activity Category = arg max P (zk |dtest ) . (12) k
Generative Group Activity Analysis with Quaternion Descriptor
3.3
7
Interaction Pattern Exploration and Discovery Based on Generative Model
Generative model facilitates the inference of the dependence among the different distributions in the recognition flow. In the extend pLSA paradigm, the distribution of the interaction patterns is modeled as a hidden variable r bridging the category topic z and visual-word observation w explicitly encoded by P (r|z) and P (w|r), respectively. Two tasks can be achieved by investigating one of the distributions P (w|r), namely interaction amount discovery and pattern exploration. The aim of the discovery is to infer the optimal amount of interaction patterns in the activities. The strategy is to transverse the sampled amount of interaction patterns and observe the corresponding recognition performance. We define K = {1, . . . , R} as the candidate set of the amount of interaction patterns. Give an interaction pattern amount k ∈ K, the corresponding extended pLSA model is learnt and the recognition performance is denoted as mk . Therefore, we can obtain the optimal interaction pattern amount OptInterN o in the underlying activities as OptInterN o = arg max{mk } . (13) k
4
Experiments
To demonstrate the effectiveness of our approach, we performed thorough experiments on two realistic human activity databases: the HOHA-2 database of movie videos used in [19] and the HGA database of surveillance videos used in [18]. These two databases are chosen for evaluation because they exhibit the difficulties in recognizing realistic human activities with multiple participants, which is in contrast to the controlled settings in other related databases. The HOHA-2 database is composed of 8 single activities (i.e., AnswerPhone, DriveCar, Eat, GetOutCar, Run, SitDown, SitUp, and StandUp) and 4 group activities (i.e., FightPerson, HandShake, HugPerson, and Kiss), in which 4 group activities are selected as the evaluation set. The HGA database consists of 6 group activities, in which all the samples are employed for evaluation. A brief summary of these two databases used in the experiments is provided in Table 2. More details about the databases can be found in [19,18]. To facilitate efficient processing, we employ the bag-of-words method to describe the quaternion descriptors in one activity video footage as a compact feature vector, namely bag-of-quaternion. We construct a visual vocabulary with 3000 words by K-means method over the sampled quaternion descriptors. Then, each quaternion is assigned to its closest (in the sense of Euclidean distance) visual word. 4.1
Recognition Performance on HOHA Database
In the pre-processing of this evaluation, the trajectory atoms in every shot are firstly generated by salient points matching using SIFT features. It has been
8
G. Zhu et al. Table 1. A summary of the databases for human group activity recognition Database HOHA-2 group subset [19] HGA [18] Data source Movie clips Surveillance recorders # Class category 4 6 # Training sample 823 4 of 5 collected sessions # Testing sample 884 1 of 5 collected sessions Table 2. A summary of the databases for human group activity recognition Database HOHA-2 group subset [19] HGA [18] Data source Movie clips Surveillance recorders # Class category 4 6 # Training sample 823 4 of 5 collected sessions # Testing sample 884 1 of 5 collected sessions
demonstrated that this trajectory generation method is effective for the motion capture in movie footage [20]. In the experiment, the appearance and dynamic representations are extracted from every salient point trajectory. The salient points residing in the same region may share the similar appearance and motion patterns. The extraction of causality and feedback descriptors on the raw trajectory atoms is not necessary and computation-intensive. Efficiently, we perform a spectral clustering based on normalized cut algorithm [3] on the set of raw trajectory atoms in one video shot. The average trajectory atom, which is calculated as the representative for the corresponding cluster, is employed as the input of the extraction of causality and feedback descriptors. To construct the graph for clustering, each node represents a trajectory atom and the similarity matrix W = [ei,j ] is formed with the similarities defined as ei,j = (PiT · Pj ) · (SiT · Sj ) ,
(14)
where i and j represent the indices of trajectory atoms in the video shot, P = {(x1 , y1 ), (x2 , y2 ), . . . , (xn , yn )} is the set of spatial positions of a trajectory atom and S = {s1 , s2 , . . . , sn } is its SIFT descriptor set. To quantitatively evaluate the performance, we calculate the Average Precision (AP) as the evaluation metric. Note that the AP metric is calculated on the whole database for an equitable comparison with the previous work although we only investigate 4 group activities. Fig. 3 shows the recognition results by using different types of interaction features as well as their combination compared with the state-of-the-art performance in [19]. In [19], the SIFT, HoF and HoG descriptors are extracted from the spatio-temporal salient points detected by 2D and 3D Harris detectors. Bag-of-words features are built as the compact representation for the video activities, which are the input of the SVM classifier. From Fig. 3, we can conclude that our quaternion descriptor and generative model yields the highest AP performance than the latest report. More specifically, the Mean AP is improved from the latest reported 37.8% [19] to 44.9%.
Generative Group Activity Analysis with Quaternion Descriptor
9
Fig. 3. Comparison of our approach with the salient point features and discriminative model in [19] on HOHA-2 database
Another observation from the results is that the pairwise causality and feedback features outperform other components in the quaternion descriptor, which demonstrate that the interaction features are indispensable for the task of group activity recognition. 4.2
Recognition Performance on HGA Database
The HGA database is mainly proposed for human trajectory based group activity analysis. The humans in the database are much smaller so that the appearance features do not present any contribution to the recognition task. Consequently, the trajectory atoms of HGA database are generated by blob tracking. Each human in the activity video is considered as a 2D blob, and then the task is to locate the positions in the frame sequence. Our tracking method is based on the CONDENSATION algorithm [7] with manual initializations. About 100 particles were used in the experiments for the tradeoff of accuracy and computational cost. Accordingly, dynamic and causality/feedback components in the proposed quaternion descriptor are employed to describe the activity video footage. Fig. 4 lists the recognition accuracies in terms of confusion matrices by using different types of interaction descriptors as well as their combination. From Fig. 4(a) and Fig. 4(b), we can observe that the causality/feedback component outperforms the dynamic component on HGA database. This is easy to understand that for two different activity categories, e.g., walking-in-group and gathering, the motion trajectory segments of the specific person may be similar while they have different interactive relations, which can be easily differentiated by the pairwise representation. Therefore, when combing two types of interaction descriptors, the recognition performance can be further improved as shown in Fig. 4(c). Compared with the results reported in [18], in which the best performance is 74% for average accuracy of all the activities, the proposed work achieves the better result with 87% average accuracy. Note that the confusions are reasonable in the sense that most of the misclassification occurs between very similar motions, for instance, there is a confusion between run-in-group and walk-in-group.
10
G. Zhu et al.
Fig. 4. The confusion matrices of HGA database recognition results with different interaction representations
4.3
Interaction Pattern Exploration and Discovery
We further evaluate the capacity of the proposed extended pLSA for exploring and discovering the interaction patterns in HGA database. The reason we select HGA database as the evaluation set is that the trajectory atoms in HGA are the human blob loca in spatio-temporal space which bear the intuitive semantics for visualization. Taking the pairwise causality and feedback interactions as the example, we explored different numbers of interaction patterns, varying from 8 to 256. The corresponding pLSA model was learnt against each number on the training sessions and then evaluated on the testing session. Fig. 5 shows the exploration results of the recognition performance against the variant amount of interaction patterns. From Fig. 5, we can observe some insight between the supposed pattern amount and the recognition performance. The performance is significantly improved by 17.2% for average recognition accuracy when increasing the pattern number from 8 to 32 because more and more interactions can be covered by the learnt model. However, the performance is degenerated and dropped down by 10.5% with the increase of the pattern amount to 256. This is due to the
Fig. 5. The exploration results of the recognition performance against the number of pairwise interaction patterns on HGA database
Generative Group Activity Analysis with Quaternion Descriptor
11
fact that the learnt model with larger pattern amount intends to overfit to the training data, resulting to a less generalizable model. Therefore, the amount of interaction patterns in HGA database is inferred as 32.
References 1. Turaga, P., Chellappa, R., Subrahmanian, V.S., Udrea, O.: Machine recognition of human activitites: a survey. T-CSVT (2008) 2. Bobick, A., Davis, J.: The recognition of human movement using temporal templates. T-PAMI (2001) 3. Shi, J., Malik, J.: Normalized cuts and image segmentation. T-PAMI (2000) 4. Wang, Y., Mori, G.: Human action recognition by semilatent topic models. T-PAMI (2009) 5. Niebles, J.C., Wang, H., Li, F.F.: Unsupervised learning of human action categories using spatial-temoral words. IJCV (2008) 6. Lowe, D.: Distincitive image features from scale-invariant keypoints. IJCV (2004) 7. Isard, M., Blake, A.: CONDENSATION - Conditional density propagation for visual tracking. IJCV (1998) 8. Granger, C.W.J.: Investigating causal relations by econometric models and crossspectral methods. Econometrica (1969) 9. Liu, J., Luo, J., Shah, M.: Recognizing realistic actions from videos in the wild. In: CVPR (2009) 10. Laptev, I., Lindeberg, T.: Space-time interest points. In: ICCV (2003) 11. Torralba, A., Murphy, K., Freeman, W., Rubin, M.: Context-based vision system for place and object recognition. In: ICCV (2003) 12. Schuldt, C., Laptev, I., Caputo, B.: Recognizing human actions: a local svm approach. In: ICPR (2004) 13. Liu, Z., Sarkar, S.: Simplest representation yet for gait recognition: averaged silhouette. In: ICPR (2004) 14. Andrade, E., Blunsden, S., Fisher, R.: Modelling crowd scenes for event detection. In: ICPR (2006) 15. Ryoo, M.S., Aggarwal, J.K.: Hierarchical recognition of human activities interacting with objects. In: CVPR (2007) 16. Turaga, P., Veraraghavan, A., Chellappa, R.: From videos to verbs: mining videos for activites using a cascade of dynamical system. In: CVPR (2007) 17. Zhou, Y., Yan, S., Huang, T.: Pair-activity classification by bi-trajectory analysis. In: CVPR (2008) 18. Ni, B., Yan, S., Kassim, A.: Recognizing human group activities with localized causalities. In: CVPR (2009) 19. Marszalek, M., Laptev, I., Schmid, C.: Actions in Context. In: CVPR 2009 (2009) 20. Sun, J., Wu, X., Yan, S., Cheong, L.F., Chua, T.S., Li, J.: Hierarchical spatiotemporal context modeling for action recognition. In: CVPR (2009) 21. Mortensen, E., Deng, H., Shapiro, L.: A SIFT descriptor with global context. In: CVPR (2005) 22. Hofmann, T.: Probabilistic latent semanic indexing. In: ACM SIGIR (1999) 23. Breiman, L.: Probability. Society for Industrial Mathematics (1992) 24. Jury, E.I.: Sampled-data control systems. John Wiley & Sons, Chichester (1958)
Grid-Based Retargeting with Transformation Consistency Smoothing Bing Li1,2 , Ling-Yu Duan2 , Jinqiao Wang3 , Jie Chen2 , Rongrong Ji2,4 , and Wen Gao1,2 1 2
Key Lab of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190 China Institute of Digital Media, School of Electronic Engineering and Computer Science, Peking University, Beijing 100871 China 3 National Lab of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing 100190 China 4 Visual Intelligence Laboratory, Department of Computer Science, Harbin Institute of Technology, Heilongjiang, 150001 China
Abstract. Effective and Efficient retargeting are critical to improve user browsing experiences in mobile devices. One important issue in previous works lies in their semantic gap in modeling user focuses and intensions from low-level features, which results to data noise in their importance map constructions. Towards noise-tolerance learning for effective retargeting, we propose a generalized content aware framework from a supervised learning viewpoint. Our main idea is to revisit the retargeting process as working out an optimal mapping function to approximate the output (desirable pixel-wise or region-wise changes) from the training data. Therefore, we adopt a prediction error decomposition strategy to measure the effectiveness of the previous retargeting methods. In addition, taking into account the data noise in importance maps, we also propose a grid-based retargeting model, which is robust and effective to data noise in real time retargeting function learning. Finally, using different mapping functions, our framework is generalized for explaining previous works, such as seam carving [9,13] and mesh based methods [3,18]. Extensive experimental comparison to state-of-the-art works have shown promising results of the proposed framework.
1
Introduction
More and more consumers prefer to watch images over the versatile mobile devices. As the image resolutions vary much and the aspect ratio of a mobile display differs from each other, properly adapting images to a target display is useful to make wise use of expensive display resources. Image retargeting aims to maximize the viewer experience when the size or aspect ratio of a display is different from the original one. Undoubtedly, users are sensitive to any noticeable distortion of re-targeted pictures. Persevering consistency and continuity of images is important. So we propose a generic approach to effective and efficient image retargeting, which is applicable to mobile devices. K.-T. Lee et al. (Eds.): MMM 2011, Part II, LNCS 6524, pp. 12–24, 2011. c Springer-Verlag Berlin Heidelberg 2011
Grid-Based Retargeting with Transformation Consistency Smoothing
13
Fig. 1. Illustrating image/video retargeting from a supervised learning viewpoint
Many content-aware retargeting methods have been proposed such as cropping[4,5,1], seam craving[13,9,15,11], multi-operator[10], mesh based retargeting[12,18,3,7,17]. Cropping[4,5,1] works on an attention model to detect important regions, and then crops the most important region to display. Seam carving[13,9,15]tries to carve a group of optimal seams iteratively based on an energy map from images/videos. Rubinstein proposes to combine different retargeting methods including scaling, cropping, seam craving in[10]. In addition, mesh based methods[12,18,3,7] partition source images/videos, where more or less deformation is allowed by adjusting the shape of a mesh, while, for important regions, the shapes of relevant meshes are committed to be kept well. Generally speaking, content aware retargeting may be considered as a sort of supervised learning process. Under the supervision from a visual importance map, content aware retargeting aims to figure out a mapping function in charge of removing, shrinking or stretching less important regions, as well as preserving the shape of important regions, as illustrated in Fig 1. Either user study or image similarity measurement applies to evaluate the effectiveness of a retargeting method. On the other hand, the result of content aware methods heavily relies on the quality of an importance map. Most importance maps are generated by low-level visual features such as gradient, color contrast and so on. Due to the lack of high-level features, an importance map cannot recover the meaningful object regions exactly to assign proper values to objects. As the importance map cannot truly represent a user’s attention, content aware retargeting guided by noisy importance map is actually a weakly supervised learning process. From a learning viewpoint, a good model should avoid overfitting, where low variance and high bias is preferred to deal with data noise. However, the seam carving method [9] removes 8-connected seams containing the lowest energy each time. It can be considered as a sort of local approximation to keep the shape of salient regions. As a result, a seam carving method has high variance and low bias. It is very sensitive to noise. For example, when the seams crossing an object produce the lowest energy, removing seams tend to fragment an object in the resulting images. Similarly, by global optimization, mesh based methods have lower variance to reduce the negative influence of noise data similar to filter smoothing. Their resulting images are smoother than pixel-wise methods. Unfortunately, serious shape transformation leads to too complex model involving many degrees of freedom. When an object covers several meshes with each mesh assigned different importance value, the object inconsistency would happen, e.g.
14
B. Li et al.
big head and small body, or a screwed structural object. As an variant of mesh based method, Wang[18] uses vertexes of the mesh to describe quad deformation in an objective function. However, it is not easy to well control the shape transformation of quad grids in optimization; moreover, most grids are irregular quadrilateral in their results. So the resulting grids may fail to preserve the structure of complex backgrounds, although efforts have been made to minimize the bending effects of the grid lines. To summarize, in the cases of certain amount noisy training data, those existing retargeting methods are sensitive due to the models’ higher variance. Undoubtedly, too many freedom degrees of a retargeting model leads to the spatial inconsistency of salient objects and the discontinuity of less important regions. Thus, we propose a grid based optimization approach to retargeting an image. The basic motivation is to reduce the model variance by constraining the gridbased shape transformation over rectangular grids. Then the aspect ratio of a display can be characterized by arranging a set of rectangles, where the change of grids’ aspect ratio is used to measure distortion energy. A nonlinear objective function is employed to allocate unavoidable distortion to unimportant regions so as to reduce the discontinuity within less important regions. In addition, as the nonlinear optimization model to build up is convex programming, a global optimal solution can be obtained by an active-set method. Overall, as our model confines the degrees of freedom, our method is effective to accommodate the weak supervision of noisy importance maps from low-level feature computing. This makes our method more generic in a sense. Our major contributions can be summarized as follows: 1. We propose a generalized retargeting framework from a supervised learning viewpoint, which introduces an optimized retargeting strategy selection approach in term of adapting to the training data quality. by adopting different learning functions, previous retargeting approaches, such as seam carving [9], mesh based approaches [18][3] can be derived from our model. 2. We present a grid-based model to effectively reduce the mapping function complexity, which is robust to the importance map noise (from cluttered background) and inferiority (for delineating the salient objects). By a quadratic programming approximation, our objective function optimization complexity can be linear to training data. 3. Our proposed objective function makes best use of the unimportant regions for optimization consistency between meaningful objects (in important regions) and content continuity in non-important regions. Also, it enables parameter adjusting to favor desirable results with user preferences (shown in Fig 4).
2
Visual Retargeting Framework
In this section, we come up with a general content aware retargeting framework from a supervised learning process point of view. To well keep important regions at the cost of distorting less important ones, retargeting methods are working out an optimal mapping function g that g : IM → SP s.t.
boundary
constraints
(1)
Grid-Based Retargeting with Transformation Consistency Smoothing
15
IM is the importance of pixels/regions, SP denotes the desirable pixel- or regionwise changes such as removing, shrinking, stretching and preserving. 2.1
Retargeting Transformation in Either Local or Global Manner
Our framework aims to abstract any retargeting transformation in a unified way from the local and global points of view. A typical learning problem involves training data (im1 , sp1 ), ..., (imn , spn ) containing an input vector im and an corresponding output sp. The mapping function g(im) is to approximate the output from training data. The approximation can be a local or global one. Local Methods. For the local method, the input/output data come from a local region as {(imk1 ,e1 , spk1 ,e1 ), . . .,(imkn ,en , spkn ,en )}, k1 , e1 . . . kn , en = local region , ki , ei is the position of pixel. The function g is a local approximation of the output, similar to K-Nearest neighbor. In training data, sp may be set to several values for different region operations like removing, shrinking, etc. For example, an image is partitioned into several regions according to importance measurements, where the regions can be determined in different ways, such as detecting objects, locating seams with lowest energy, spotting a window with large importance and so on. As a local approximation, the mapping function leads to a sort of independent retargeting based on each individual region. In other words, the process of keeping the important regions is independent of shrinking/stretching the less important regions. For the sake of simplicity, we set spr = 1 to the region/pixel requiring good local preservation, otherwise spr = −1. The function can be simply defined as: −1 k, e ∈ unimportant region sp ˆ k,e = g(imk,e ) = (2) 1 k, e ∈ important region Global Methods. For a global method, input/output come from a whole image = source image, where the as {(imri , spri ),. . .,(imrn , sprn )}, r1 r2 . . . rn whole image is partitioned into regions or pixels r1 . . . rn = source image. The mapping function is to approximate the output in a global manner. To accomplish a satisfactory global fitting on the training data, the risk Remp(g) is defined as Remp (g) = L(spri , g(imri )) (3) r( i)∈image
L(spri , g(imri ) calculates a weighted discrepancy between an original region and a target region. The g(im) is thus obtained by minimizing Remp(g). As a typical global one, meshes based methods impose mesh-based partition to a source image.g(im) measures each region’s original shape. In meshes based methods, L(spri , g(imri ) is defined as: L(spri , g(imri )) = D(ri ) · w(imri )
(4)
where D(ri ) measures the distortion of the meshes, w(imri ) is a weighting function of the importance of regions ri . Increasing or reducing the distortion of meshes can be controlled by adjusting w(imri ) accordingly.
16
2.2
B. Li et al.
On the Effectiveness of a Retargeting Method
To measure the effectiveness is an important issue to design a good retargeting method. This is closely related to how to measure the performance of a mapping function. The performance of a mapping function strongly depends on correctly choosing important regions in training data. In cases of noisy data, too complex mapping function would lead to overfitting like distortion of an object. To select a good model, a performance measurement of the mapping function should be provided to select the best model based on different types of noisy or clean data. [14] presented the prediction error to measure the effectiveness of a mapping function. A learnt function’s prediction error is related to the sum of the bias and the variance of a learning algorithm [6], which can be formulated as [14]. For the training data Q and any im , the prediction error decomposition is: EQ [(g(im; Q) − E(sp|im))2 ] =(EQ [g(im; Q)] − E[sp|im])2 + EQ [(g(im; Q) − EQ [g(im; Q)])2 ]
(5)
EQ [g(im; Q)] − E[sp|im])2 is bias, EQ [(g(im; Q) − EQ [g(im; Q)])2 ] is variance. To avoid overfiting for the generality of a retargeting function, our goal is to decrease the variance in fitting the particular input im. For a local method, the mapping function variance depends on the pixel number k of each local region. When k is too small, the function has higher variance but lower bias. Such a function exhibits higher complexity that incurs many degrees of freedom. Thus, retargeting is sensitive to noises and may artifact in the objects with rich structures like seam carving[13,9]. When k is large (e.g., all the pixels of a cropping window), this variance is lower. So the impact of noisy data is decreased; however, taking cropping methods [4,5,1] as an example, some objects or parts would be discarded when several important objects are far from each other. For global methods[18,3], the mapping function not only depends on region importance but also their distributions, for which the model would be more complex. Overall, the mapping function is smoother than a local method. 2.3
An Instance of Our Proposed Framework
As discussed above, a good retargeting method has to seek a tradeoff between the bias and the variance based on the quality of training data. In this section, we come up with an instance by taking into account the quality of the importance map. As a visual importance map cannot recover the regions of salient objects exactly, the actual training data is noisy. Therefore, we would like to choose a mapping function with lower variance to reduce the negative influence of noise data. So our instance prefers a global method. Our instance is committed to maintain lower variance. An optimization approach with lower variance is applied to reduce noise data’s influence. We constrain the grid-based shape transformation over rectangular grids. The change of grids aspect ratio is used to measure distortion energy in retargeting. This is advantageous than Wang’s models[18] that too many degrees of freedom often leads
Grid-Based Retargeting with Transformation Consistency Smoothing
17
to deformation of objects. Moreover, we provide a user a few parameters to optimize the use of unimportant regions to keep important regions’ shape. In the cases of noisy data, a lower variance can reduces the influence of noisy data. But the shape transformation of unimportant region is less flexible, which would affect the preservation of important region’s shape. So we introduce user input to alleviate this disadvantage. Through setting a few parameters in our object function, we may amplify the difference between the important and unimportant region’s importance values. The objective function is described as follows: Objective Function. We use the edges of grids rather than the coordinates of vertices to measure the distortion energy of each grid. A nonlinear objective function is employed to reallocate distortion to a large proportion of (all) unimportant regions to avoid discontinuity. To minimize the grid distortion energy, the objective function is defined as: (6) min (yi (t) − ars · xj (t))m · snij m > 2 and is a even number, n > 1. ars is the aspect ratio of the original grid respectively, xi , yj is the width and height of the target grid gij ,respectively. sij is the importance value of grid gij . The weight snij is to control the distortion of grid gij . The more snij is, the more the grid would be preserved. As our approach has high bias and low variance, restricting shape transformation of grids is at the expense of less flexible adjustment of unimportant regions, which would in turn affect the shape preservation of the remaining important region. To optimize the use of unimportant regions’ shape, we introduce user input to alleviate this disadvantage. By adjusting parameters n, m, we may get two types of retargeting effects: 1) the adapted results tend to preserve important regions more; or 2) allowing more smoothness between grids within unimportant regions. More details can be found in the subsequent section.
3
Grid Based Image Retargeting
In this section, we introduce our grid-based image retargeting, involving rectangular grids based shape transformation constraints as well as a nonlinear objective function to reallocate distortion to less important regions. Our method contains three basic stages. Firstly, we calculate gradient map and visual attention map to determine important regions. Secondly,we divide the source image into grids, each grid generating a importance measure. The model of grid optimization is solved at the granularity of grids. The optimal solution is applied to transform source grids to target grids. Finally, the image retargeting is accomplished by a grid based texture mapping algorithm[2]. 3.1
Importance Map
We combine gradient map and visual attention map[16] to generate importance map.The importance map is defined as: IM Sk,e = α · GSk,e + (1 − α) · ASk,e
α ≥ 0;
(7)
18
B. Li et al. Comparation on Power of Importance 4 3.5
n=1 n=2 n=3 n=4
3
Weight
2.5 2 1.5 1 0.5 0
0
0.2
0.4
0.6
0.8
1
1.2
1.4
Importance
Fig. 2. Influence of the importance variant’s power on the resulting weights
GSk,e and ASk,e is important value of the pixel at the position (k, e) in the gradient map and the attention map, respectively. In different types of images such as nature and building, different empirical parameters α can be set to obtain good results.It is noted that, for some homogeneous regions, whose distortions are rarely perceived by humans after transformation; while, for irregular textured regions, i.e., higher values in a gradient map, people can tolerate such distortions. Consequently, the addition of an attention map would reduce the overall importance values in irregular textured regions. 3.2
Grid-Based Resizing Model
The grids are divided as follows. An image is divided into N × N grids, and the grids are denoted by M = (V, E) in which V is the 2D grid coordinate, E are the edges of grids. Each grid is denoted by G = {g11, g12 , . . . , gij . . . gN N } with its location i, j. Owing to the constraint of rectangular grids, all the grids in each row have the same height while the grids in each column have the same width. So the edge is simply denoted by E = {(x1 , y1 ), . . . , (xi , yj ), (xN , yN )}, and xi , yj is the width and the height of the grid gij , respectively. Clearly, our model has fewer parameters to optimize than [18]. Computing the importance of a grid. With the uniform division of N × N grids, the importance of grid gij can be calculated as follows: k,e∈g IM Sk,e sij = × N2 (8) Stotal Stotal is the sum of importance values of all the pixels in the image. A mean importance value of 1 is imposed to isolate important grids from unimportant ones instead of any value between 0 to 1 in [18]. A grid is considered as an unimportant one if its important value is less than 1. From Fig 2, for an important grid gij , the larger n is, the bigger snij is; while, for an unimportant grid gij , the larger n is, the less snij is. Thus increasing n can further improve the important grid’s importance but decrease the unimportant grid’s importance. However, n cannot be too large, otherwise it would break the visual discontinuity in unimportant regions. Empirically, the range of n is [1,7].
Grid-Based Retargeting with Transformation Consistency Smoothing Distortion of Grid:m=2
19
Distortion of Grid:m=4
Trend of Feasible Solution 20
10
10
0
100
50 y
50 x
0 0
100
0
Projecetion of funtion 25 20
m=2 m=4
Equation Constraint:x1+x2=12
100
50 y
50 x
0 0
Projecetion of funtion 25
m=2
20
15
15
10
10
5
5
0
m=4
0 100 50 0 y
50
100 x
100 50 0 y
50
100
100
Value of Objective
20
Variable
x
Fig. 3. The effects of choosing different m on the retargeted image (ars = 2)
Boundary constraints. We introduce the constraints as follows: N yi (t) = HT i=1 i=1 xj (t) = WT
(9)
HT ,WT are the height and width of a target grid, respectively. Note that the minimum height or length of a grid is set to one pixel, as adjacent grids should not overlap each other. The Objective Function. To minimize the sum of grid distortion energy, we employ the objective function 6 as mentioned in section 2. Increasing m improves the continuity of a whole image. With increasing m , these grids with similar weights are subject to similar shape changes the edge disparity between large weighted grids and small weighted grids also becomes smaller. To clarify, we provide a three-dimensional graph of the object function in Fig 3, and take m = 2 and m = 4 respectively. For simplicity, the images are divided into 2 × 2 grids and the height of grids remains unchanged; the 3D graph is projected to a 2D curve to indicate the flat trend of m = 2 and the dramatic trend of m = 4. A two-way arrow and a hollow circle are used to illustrate the solutions’ movements constrained by equation (11) when increasing m. In addition, m cannot be too large; otherwise, the effects is similar to scaling an image. We empirically set m = 2, 4, 6. Global Solution. To get a global solution, we employ an active-set method to solve this optimization problem. This nonlinear program is a convex programming, and any local solution of a convex programming is actually a global solution. (yi (t) − ars · xj (t))m · snij is a convex function, so our objective function (yi (t) − ars · xj (t))m · snij is a convex one. Moreover, the equality constraints are linear functions and the inequality constraints can be seen as a concave function. The solutions satisfying equality and inequality constraints finally form a convex set. When a local solution is resolved, the global solution is yielded.
20
B. Li et al.
For the convex programming, the Hessian matrix of the objective function is positive semidefinite. The complexity is similar to a linear programming that depends on the number of grids instead of real resolution of an image.
4
Experiments
To evaluate the effectiveness and efficiency of our retargeting method, we conducted our experiments on a variety of images. In total 120 images are collected to cover six typical classes: landscape, structure (e.g., indoor/outdoor architecture), computer graphics & painting, daily life (with human/animal as the subject). Daily life are further classified into long shot, medium shot, and close up. To avoid the confusion from picture classification, we apply two rules as below: (1) an image is primarily classified into daily life as long as the image contains human/animal; otherwise, (2) an image containing building is classified into structure, no matter whether the image belongs to computer graphics/painting or not. Those categories may be related to different types of noisy or clean data in a sense. For instance, computer graphics & painting or a long shot is comparatively cleaner (i.e., the importance map tend to exactly delineate the salient object) than other classes. Efficiency. Our algorithm was implemented on a standard laptop with 2.26 GHz duo core CPU, 2GB memory. Our dataset consists of diverse images in sizes and aspect ratios. The largest image size is 1920×1200, while the smallest size is 288×240. Each optimization process costs less than 0.015s for 20×20 grids. As the complexity of our algorithm relies on the grid division instead of the real image resolution, our algorithm is much more efficient than seam carving. Effectiveness. Fig 4 illustrates the effects of parameter m, n on retargeting results. Note that the girl’s head and the volleyball have higher importance values so that they are kept well in Fig 4(c)(d)(e)(f). By comparing different effects from Fig 4(c)(d)(e), we can find that when choosing the same m, increasing n can preserve the shape of the important region (e.g., the girl’s head), while unimportant regions (background) are much distorted; By comparing Fig 4(d)(f), we can find that when keeping the same n , increasing m improves the visual content continuity at unimportant regions. In the subsequent user study, most subjects prefer (f) since its entire consistency are best achieved, even though the grids covering the girl are somehow squeezed. Furthermore, our method are compared with the scaling and other two representative methods [18,9]. Note that when noisy importance maps in source images (especially for structure and close up) cannot delineate important objects from non-important regions, the retargeting problem become more challenging. In our empirical dataset, each type has at least 20 images Fig 5 demonstrate that our method is more efficient in preserving the consistence of objects and the continuity of unimportant regions, even in the case of serious noises (e.g. the sixth row in Fig 5). In contrast, the seam carving method [9] brings about considerable shape artifacts of an object, especially more structured ones, as indicated in the 1, 2, 3, 4, 6, 7 rows in Fig 5), since, at object regions, some seams
Grid-Based Retargeting with Transformation Consistency Smoothing
(a)
(b)
(c)
(d)
(e)
21
(f)
Fig. 4. Results for a daily life picture at medium shot. (a) Original image. (b) Scaling. (c) m = 2,n = 1.(d)m = 2,n = 3. (e)m = 2,n = 5. (f)m = 4,n = 3.
with lower importance are falsely removed. Wang’s method [18] distorts objects, as indicated in the 1, 2, 3 , 4 rows in Fig 5). When an important object covers several grids but the importance of these grids are different from each other, most grids with low importance become irregularly quadrilateral after retargeting, so that Wang’s method may fail to preserve the structure of an object. User Study. A subjective evaluation is further performed by user study. The results of several popular methods are provided to subjects. By means of user preference and scoring evaluation, the effectiveness are measured quantitatively. In total 10 students participated in user study. We showed each participant an original image and a randomly ordered sequence of retargeting results with different methods including scaling, non-homogeneous resizing[8], seam carving[9], Wang’s method[18] and our method. Each participant are required to choose the results most visually similar to the original image. As listed in Tab1, most participants prefer our results. Referring to Fig5, let’s investigate the results. For landscape, our results are comparable to Wang’s method. But seam craving may greatly alter the content of an image, probably without any noticeable distortion sometimes (see the 7th row in Fig 5). For a long shot, except seam carving and scaling, most methods produce similar results. The reason is unimportant region occupies a large portion and has lower values in computed importance map. However, seam carving tends to change the depth of field. See the fifth row of Fig 5. For a close-up, subjects prefer our method and scaling, since the calculated importance maps are often seriously noisy. In such cases, the major part of an image is important, whereas these parts have lower value in importance map. Moreover, the unimportant part is not large enough to adapt it to target size. Consequently, non-homo resizing, seam carving, and Wang’s method distort important regions with lower importance. So the objects are distorted, while our method produces smoother results. It is easy to observe that, for structure, medium shot, and computer graphics & painting, users generally prefer our results, as discussed above. Limitations. Like all content aware retargeting methods, our method are still impacted by the importance map. Our results may be reduced to that of scaling when the most major part of an image is considered as important by visual content computing. Due to the importance map, the retargeting results are not preferred by users actually. If we could lower such value in irregular textured regions, more spaces would be saved for important objects. Ideally, if some
22
B. Li et al.
(a)
(b)
(c)
(d)
(e)
Fig. 5. Comparison Results. Columns from left to right: (a) Source image, (b) scaling, (c) Rubinstein et al’s results[9], (d) Wang et al’s results [18], (e) Our results. Rows from top to bottom: (1) Computer Graphics, (2) Architecture, (3) Indoor, (4) Medium Shot, (4) Long Shot, (5) Close Up, (6) Landscape.
Grid-Based Retargeting with Transformation Consistency Smoothing
23
Table 1. Preference statistics of ten participants in user study (%) Daily
Landscape
Structure
CG& Painting
Long
Medium
Close up
Scaling
2.1
3.5
4.5
5.10
1.30
24.40
Non-homo resizing[8]
8.4
6.2
8.9
22.4
11.8
20.2
Seam Carving[9]
15.2
10 .6
9.4
11.2
12.7
1.4
Wang’s method[18]
30.1
10.7
23.7
25.8
17.3
19.3
Our results
44.2
79.5
53.5
35.6
56.9
34.7
descriptors are available that distinguish irregular textured regions out from objects, our method is able to produce more desirable results.
5
Conclusion and Discussions
We proposed a general content aware retargeting framework from a supervised learning viewpoint. We proposed to measure retargeting performance based on prediction error decomposition [14]. We further propose a grid-based retargeting model to ensures transformation consistency in model learning. This grid-based model is optimized by solving a nonlinear programming problem. There are two merits in the proposed framework: (1) our framework is generalized for previous works, in which by incorporating different learning functions, many state-of-theart retargeting methods can be derived from our framework. (2) based on our grid-based learning structure, our model is suitable for real time applications on mobile devices. Acknowledgements. This work was supported in part by National Basic Research Program of China (973 Program) 2009CB320902, in part by National Natural Science Foundation of China No. 60902057 and No. 60905008, in part by CADAL Project and NEC-PKU Joint Project.
References 1. Santella, A., Agrawala, M., DeCarlo, D., Salesin, D., Cohen, M.: Gaze-based interaction for semi-automatic photo cropping. In: SIGCHI Conference on Human Factors in Computing Systems (2006) 2. Hearn, D., Baker, M.: Computer graphics with OpenGL (2003) 3. Guo, Y., Liu, F., Shi, J., Zhou, Z., Gleicher, M.: Image retargeting using mesh parametrization. IEEE Transactions on Multimedia (2009) 4. Chen, L., Xie, X., Fan, X., Ma, W., Zhang, H., Zhou, H.: A visual attention model for adapting images on small displays. Multimedia Systems (2003) 5. Liu, H., Xie, X., Ma, W., Zhang, H.: Automatic browsing of large pictures on mobile devices. In: ACM Multimedia (2003) 6. James, G.M.: Variance and bias for general loss functions. Mach. Learn. (2003) 7. Shi, L., Wang, J., Duan, L., Lu, H.: Consumer video retargeting: context assisted spatial-temporal grid optimization. In: ACM Multimedia (2009) 8. Wolf, L., Guttmann, M., Cohen-Or, D.: Non-homogeneous content-driven videoretargeting. In: ICCV (2007)
24
B. Li et al.
9. Rubinstein, M., Shamir, A., Avidan, S.: Improved seam carving for video retargeting. ACM Transactions on Graphics (2008) 10. Rubinstein, M., Shamir, A., Avidan, S.: Multi-operator media retargeting. ACM Transactions on Graphics (2009) 11. Matthias, G., Kwatra, V., Han, M., Essa, I.: Discontinuous seam-carving for video retargeting. In: CVPR (2010) 12. Gal, R., Sorkine, O., Cohen-Or, D.: Feature-aware texturing. In: Eurographics Symposium on Rendering (2006) 13. Avidan, S., Shamir, A.: Seam carving for content-aware image resizing. ACM Transactions on Graphics (2007) 14. Geman, S., Bienenstock, E., Doursat, R.: Neural networks and the bias/variance dilemma. Neural. Comput. (1992) 15. Kopf, S., Kiess, J., Lemelson, H., Effelsberg, W.: Fscav: fast seam carving for size adaptation of videos. In: ACM Multimedia (2009) 16. Liu, T., Sun, J., Zheng, N., Tang, X., Shum, H.: Learning to detect a salient object. In: CVPR (2007) 17. Niu, Y., Liu, F., Li, X., Gleicher, M.: Warp propagation for video resizing. In: CVPR (2010) 18. Wang, Y., Tai, C., Sorkine, O., Lee, T.: Optimized scale-and-stretch for image resizing. ACM Transactions on Graphics (2008)
Understanding Video Sequences through Super-Resolution Yu Peng, Jesse S. Jin, Suhuai Luo, and Mira Park The School of Design, Communication and IT, University of Newcastle, Australia {yu.peng,jesse.jin,suhuai.luo,mira.park}@uon.edu.au
Abstract. Human-centred multimedia applications are a set of activities that human directly interact with multimedia, which consists of different forms. Within all multimedia, video is an ultimate resource, by which people could obtain sensory information. Since limitations on the capacity of imaging devices as well as shooting conditions, we cannot usually acquire high quality video records that desired. This problem could be addressed by super-resolution. We propose a novel scheme in the present paper for super-resolution problem, and make three contributions: (1) on the stage of image registration according to previous approaches, the reference image is picked out through observing or randomly. We utilise a simple but efficient method to select the base image; (2) a median-value image, rather than the average image used previously, is adopted as the initialization for estimate of super-resolution; (3) we adapt the traditional Cross Validation (CV) to a weighted version in the process of learning parameters from input observations. Experiments on synthetic and real data are provided to illustrate the effectiveness of our approach. Keywords: Super-resolution, reference image selection, median-value image, Weighted Cross Validation.
1 Introduction With the dramatic development of digital imaging as well as internet, multimedia blending seamlessly text, pictures, animation, and movies becomes one of the most dazzling and fastest growing areas in the fields of information technology [1]. Multimedia is widely used in entertainment, business, education, training, science simulations, digital publications, exhibits and so much more, within all of which human’s interaction and perception is crucial. Low-level features, however, cannot engender accurate understanding of sensory information. As an important media, video providing people visual information usually associates with low quality problems since the limitations on imaging device itself and shooting conditions. For example, people cannot recognize faces or identify car license plates from a video record with low resolution and noises. We usually tackle this botheration through super-resolution, which is a powerful technique of reconstructing a high resolution image from a set of low-resolution observed images, at the same time complementing de-blurring and noise removal. The “super” here means the well characters closing to original scene. K.-T. Lee et al. (Eds.): MMM 2011, Part II, LNCS 6524, pp. 25–34, 2011. © Springer-Verlag Berlin Heidelberg 2011
26
Y. Peng et al.
In order to interpret it in perspective, a brief review of super-resolution is needed. As a well-studied field, comprehensive reviews were carried out by Borman and Stevenson [2] in 1998 and Park [3] in 2003. Although there are lots of approaches to super-resolution, the formulation of this problem usually falls into either frequency or spatial domain. Historically, the earliest super-resolution methods are raised in frequency domain [4, 5, 6] as the underlying mechanism of resolution improvement, that is restoration of frequency components beyond the Nyquist limit of the individual observation image samples [3]. However, the unreality assumption of global translation with these kind methods is too limited to use widely. On the other hand, spatial methods [7, 8, 9, 10] dominated soon because of its better handling of noise, and a more natural treatment of the point spread blur in imaging. Many super-resolution techniques are derived from Maximum Likelihood (ML) methods, which seek a super-resolution that maximizes the probability of the observed low-resolution input images under a given model, or Maximum a Posteriori (MAP) methods, which must use an assumptive prior information of the highresolution image. Recently, Pickup [10] proposed a completely simultaneous algorithm for super-resolution. In this MAP framework based method, parameters of prior distributions are learnt automatically from input data, rather than being set by trial and error. Additionally, superior results are obtained by optimizing over both the registrations and image pixels. In this paper, we take an adaptive method for super-resolution basing on this simultaneous framework. The three improvements are presented: at the beginning of superresolution, image registration is ultimately crucial for accurate reconstructed results. Most of previous approaches ignored the selection of reference image, which is the basic frame for other sensed images in the process of registration. In our method, we introduce an effective evaluation for reference image selection. As the second development, a median-value image is utilized as the initialization in place of average image, the outperformance is demonstrated. Finally, we adopt a Weighted Cross Validation (WCV) to learn prior parameters. As we present, our is more efficient than the traditional Cross Validation. The paper is organized as follows. Notations for reasoning about the superresolution problem are laid out in section 2, which then goes on to present popular methods for handling super-resolution problem. In section 3, we give particular consideration to image registration and prior information, which are ultimately crucial in the whole process. Section 4 introduces the adaptive super-resolution scheme basing on simultaneous model with our three developments. All the experiments and results are given in section 5, while section 6 will concludes the paper and highlights several promising directions for further research.
2 The Notation of Super-Resolution As we know, in the process of recording video, some spatial resolution would be lost unavoidably for a number of reasons. The most popular choice is describing this information lost process through a generative model, which is a parameterized, probabilistic forward process model of observed data generation. Several comprehensive
Understanding Video Sequences through Super-Resolution
27
generative models with subtle differences were proposed [2,10,11,12,13], generally these models described the corruptions with respect to geometrical transformation, illumination variation, blur engendered by limited shutter speed or sensor point spread effect, down-sampling, and also noise that occurs within sensor or during transmission. As [8,14], the warping, blurring and sub-sampling of the original highresolution image is modelled by an sparse matrix , where ,with pixels, of which has pixels. is assumed to generate a set of low-resolution images, the We could use the following equation to express this model: (2.1) Where scalars and characterize the global affine photometric correction results from multiplication and addition across all pixels respectively, and is noise occurring in this process. In terms of generative model, the simple Maximum Likelihood (ML) solution to the super-resolution problem is the super-resolution image which maximizes the , Although ML super-resolution is really fast, it likelihood of the observed data is an ill-conditioned problem, some of whose solutions are subjectively very implausible to the human viewer. The observations are corrupted by noise, the probability of which would be also maximized at the same time when we maximize the probability of observations. The Maximum a Posterior (MAP) solution is introduced to circumvent this deficit. The MAP solution is derived from Bayesian rule with a prior distribution over x to avoid infeasible solutions, it is given as: ,
,
(2.2)
More recently, Pickup [10] proposed a simultaneous method depending on MAP framework by which we find super-resolution image at the same time as optimizing registration, rather than reconstructing super-resolution image after fixing registration results of low-resolution frames.
3 Image Registration and Prior Distribution Almost all multi-frame images super-resolution algorithms need to use the estimate of the motion relating the low-resolution inputs within sub-pixel accuracy in order to set up the constraints on the high-resolution image pixel intensities which are necessary for estimating the super-resolution image. Generally speaking, image registration is the process of overlaying two or more images of the same scene taken under different conditions. It geometrically aligns two images—the reference and sensed image. The majority of the registration methods consist of the following four steps: feature detection, feature matching, transformmodel estimation, and image re-sampling with transformation [15]. In the sense of super-resolution, a typical method for registering image is to find interest points in low-resolution image set, then use robust methods such as RANSAC [16] to estimate the point correspondences and compute homographies between images.
28
Y. Peng et al.
With standard super-resolution methods, super-resolution image is estimated after fixing registration between low-resolution observations. Accurate low-resolution images registration is critical for the success of super-resolution algorithms. Even very small errors in the registration of the low-resolution frames can have negative consequences on the estimation of other super-resolution components. Within many superresolution methods, many authors consider that image registration and super-resolution image estimation as two distinct stages. However, we prefer to consider that image registration should be optimizing at the same time as recovering super-resolution, as these two stages are not truly independent, as the super-resolution problem is akin to a neural network, the components and the parameters of them are cross-connected with each other. Besides registration, prior distribution is another core in super-resolution, we need it to constrain the original high-resolution image in order to steering this inverse problem away from the infeasible solutions, in practice the exact selection of image priors is a tricky job as we usually should guarantee image reconstruction accuracy and the computational cost, since some priors are much more expensive to evaluate than others. In general, authors propose to use a Markov Random Filed (MRF) with smooth function over the original image to enable robust reconstruction. The practicality of MRFs model depends on the fact that through information contained in the local physical structure of images, we could sufficiently obtain a good global image representation [14]. Therefore, MRFs allow representation of image in terms of local characterization of images. A typical MRF models assumes that the image is locally smooth to avoid the excessive noise amplification at high frequency in ML solution, except those natural discontinuities such as boundaries or edges. As well, we aim to make MAP solution a unique optimal super-resolution image. As the basic ML solution is already convex, the convexity-preserving smooth-priors become attractive. Gaussian image priors give a closed-form solution quickly with a least-squares-style penalty term for image gradient values. Natural images contain edges, however, where there are locally high image gradients that are undesirable to smooth out. As a good substitute, Huber function is quadratic for small values of input, but linear for large values, so it benefits from penalizing edges less severely than Gaussian prior, whose potential function is purely quadratic. The smooth functions vary with different parameters. Therefore, appropriate priors parameters would make considerable benefit on the super-resolution results. There are several methods available that allow us to learn values of prior parameters, which we will return later.
4 The Adaptive Simultaneous Approach 4.1 Overview of This Adaptive Simultaneous Method This simultaneous model consists of two distinct stages: as in Fig. 1, the first stage covers a set of initializations of image registration, point spread function, prior parameters, and coarse estimate of super-resolution image. In the second stage, it is a big optimization loop, which incorporates an outer loop that updating prior parameters, super-resolution image and registration parameters, and two inner loops that if the maximum absolute change in any updated parameters in the outer loop beyond preset convergence thresholds, return to the beginning of the second stage.
Understanding Video Sequences through Super-Resolution
29
Fig. 1. Basic structure of the adaptive simultaneous method. The first part is initialization as the start point for super-resolution. The second part is optimizing for accurate solutions.
4.2 Reference Image Selection Image registration, as the beginning step of this simultaneous method, is crucial for the final super-resolution image. As interpreted above, photometric registration depends on geometric registration, which we usually achieve by a standard algorithm— RANSAC [16]. The problem of reference image selection, however, is too often ignored in current super-resolution techniques. In the procedure of geometric registration, other images are aligned according to the reference image. Therefore, inappropriate selection of reference image would bring in poor quality on registration results, then super-resolution image. Within the vast majority of papers, authors either pick it out randomly or omit the step. In our proposed super-resolution scheme, we handle this problem via a simple but effective method, that is Standard Mean Deviation (SMD), with which the information of low-resolution images is estimated as: ∑
∑
,
,
∑
∑
,
,
(4.1)
Where and represent the number of row and column pixels in each input observations respectively, and , is the intensity of the pixel , . This approach for the reference image selection we propose here is practically sensible. In most superresolution cases, reference image is hard to be told via observing the image content, in which there is no huge difference but only subtle shift exists between low-resolution images. Through (4.1), higher SMD of the image means ample intensity changes, which implies that great information this image has. We can keep much detail as possible as the most informative image is chose as reference image, which would provide a fine foundation for the following operations in the whole diagram. 4.3 Median-Value Image In the traditional methods, an average image is always chosen as a starting point for super-resolution estimation. The average image is formed by a simple re-sampling
30
Y. Peng et al.
scheme applied to the registration input images. Every pixel in average image is a weighted combination of pixels in observations, which are generated from original high-resolution image according to the weights in W. Although it is very robust to noise in observations, the average image is too smooth to bring incorrectness into estimation of super-resolution. We overcome this problem via choosing median-value image instead, each pixel in which is obtained by: , ∑
,
(4.2) (4.3)
Where is recovered from low-pixels in observations, each pixel in which are generated from super-pixels contained within the pixel’s Point Spread Function (PSF) according to weights . Therefore, depends on the point spread function. The process of acquiring median image is illustrated in Fig. 2. We obtain each pixel of median-value image via selecting median from recovered from all low-resolution images, as (4.2). In (4.3), the care about the lighting change is taken.
Fig. 2. The obtaining of median-value image. From original high-resolution to low resolution images is imaging process. A pixel in median value image depends on certain pixels in low-resolution image, and is created from original high-resolution according to specific Point spread function.
Fig. 3. Comparing median image with average image. The right is ground truth image. The center and the right are average and median image respectively, which are obtained from 5 synthetic low-resolution images. Median image outperforms average one, which is overly smooth.
Understanding Video Sequences through Super-Resolution
31
Note that the resulting images shown in Fig. 3, average image is overly smooth comparing with the original high-resolution image while the resulting median-value image preserves edge information as well as removing noise. 4.4 Learning Prior Parameters via Weighted Cross-Validation As interpreted in 3.2, an appropriate prior distribution, as a regularizer, is crucial to keep the super-resolution image away from infeasible solutions. Different superresolutions are obtained according to varying prior parameters such that prior strength and function ‘shape’ in Huber-MRF. Rather than choosing prior parameters empirically, learning them from inputs is more sensible. Cross validation is a typical method to estimate these parameters. Recently, Pickup [10] raised a pixel-wise validation approach, which has a clear advantage than cross-validation method based on whole-image –wise. Leave-one-out is the basic philosophy of Cross-Validation (CV), in the case of super-resolution, the idea is that a good selection of prior parameters should predict the miss low-resolution image of the inputs. That is, if an arbitrary observations of all the low-resolution is left out, then the regularized super-resolution image, according to the selected prior parameters, should be able to predict this one fairly well. In previous CV for super-resolution, input observations is split into two sets those are training set and validation set, from the former of which we obtain the super-resolution, via the latter of which the error is found. We would make mistakes, however, when there are validation images misregistered. Instead, in pixel-wise method, validation pixels are selected at random from the collection of all the pixels comprising the overall input observations. The cross-validation error is measured by holding a small percentage of the low-resolution pixels in each image back, and performing Huber-MAP . The obtained super-resolution super-resolution using the remaining pixels image is then projected down into the low-resolution frame ) under corresponding generative model and the mean absolute error is recorded. By this recorded error, we determine whether the estimate of super-resolution gets an improvement and is therefore worthy for consideration for gradient descent, when we optimize the prior parameters. We leave out a small percent of pixels from each low-resolution image and seek the value of prior parameters that minimizes the prediction errors, measured by the CV function: ,
(4.5)
Where is the prior operator chosen as Huber-MRF in our experiment, the prior parepresents regularized by prior rameter is a scalar. . Note that lighting selected, and gives the regularized solution, change is not taken in account as the global photometric variation does not make contribution in gradient direction. Although this developed CV overcomes the misregistration problem, we would lose many valuable pixels in each observation those could be the training set. In this
32
Y. Peng et al.
paper, we utilized Weighted Cross Validation (WCV) [17] method in the process of learning prior parameters, which could avoid misregistration problem as well as keep as many pixels in training set as possible. The WCV method is a “leave-weighted one-out” prediction method, we leave out part pixels of an observation in turn and optimize the value of that minimizes the prediction error. The WCG method in one image is illustrated in Fig. 4, we should notice that the operation should be implemented for every low-resolution image in each iteration. In fact, in leaving out the whole ith observation, the derivation seeks to minimize | | the prediction error , when x is the minimizer of ∑
|
,
|
(4.6)
Fig. 4. Weigthed cross validation for one low-resolution image. For each input observation, we first leave out part pixels from it as the validation set, the remaining pixels with all other images are taken as training set for super-resolution. We then project this obtained superresolution into the low-resolution frame, the error is recorded.
We could define the matrix 1,1,1,1 the jth entry, then the above minimization is equivalent to min
|
|
1,0,1
1 , where 0 is
(4.7)
The weighted cross validation method is derived in a similar manner, but we instead use a weighted “leave-one- out” philosophy. We define a new matrix 1,1,1,1 1, √1 ,1 1 , consider 0 1, where √1 is the jth diagonal entry of . By using the WCV method, we seek a super-resolution estimate to the minimization problem, min
|
|
(4.8)
Understanding Video Sequences through Super-Resolution
33
5 Experiments Results A set of selected frames from a clip of video record are used to test the proposed approach. Fig. 5 shows one of the sequences consisting of 9 frames, and the area of the interest that the logo of Newcastle University, Austrtalia. The logo sequences are unclear as a result of down-sampling, warping, and noise. In our experiment, we compare the result obtained by this raised method with the reconstructions via ML and tradition MAP. We choose Gaussian PSF with std=0.375 with zoom factor of 2 for all three methods, Huber prior with 0.08 and =0.04 for MAP, and Weighted Cross Validation with w=0.2 for learning prior parameters. As the three super-resolution images shown in Fig. 6, this proposed method outperforms the others.
Fig. 5. Input low-resolution frames from a video record. The top is one of 9 frames from a clip. The bottoms are interest areas from all 9 frames.
Fig. 6. Super-resolution obtained through three methods. The left is the ML result, in which we can see the corruptions by amplified noise clearly. The next one is reconstruction from MAP. The final one is super-resolution by this adaptive simultaneous method, the significant development can be seen.
6 Conclusion A novel simultaneous approach to super-resolution is presented in this paper. We make three developments: First, we introduce an effective method for reference image selection in image registration which is always ignored. Next, a median value image is taken in place of average image as the initialization, and we show the advantages. Finally, we utilized Weighted Cross Validation for learning prior-parameters, this method could keep as much pixels in training set as possible while avoiding
34
Y. Peng et al.
misregistration problem. In the end, experimental results show the outperformance of this adaptive method. Further work directions include more robust methods for reference image selection and super-resolution with object tracking.
References 1. Multimedia Design and Entertainment, http://www.clearleadinc.com 2. Borman, S., Stevenson, R.L.: Spatial Resolution Enhancement of Low-resolution Image Sequences. Technical report, Department of Electrical Engineering, University of Notre Dame, Notre Dame, Indiana, USA (1998) 3. Park, S.C., Park, M.K., Kang, M.G.: Super Resolution Image Reconstruction: a technical overview. IEEE Signal Processing Magazine, 1053–5888 (2003) 4. Tsai, R.Y., Huang, T.S.: Uniqueness and Estimation of Three Dimensional Motion Parameters of Rigid Objects with Curved Surfaces. IEEE Transaction on Pattern Analysis and Machine Intelligence 6, 13–27 (1984) 5. Tom, B.C., Katsaggelos, A.K., Galatsanos, N.P.: Reconstruction of A High Resolution Image from Registration and Restoration of Low Resolution Images. In: Proceedings of the IEEE International Conference on Image Processing (ICIP), pp. 553–557 (1994) 6. Nguyen, N., Milanfar, P., Golud, G.: Efficient Generalized Cross Validation with Applications to Parametric Image Restoration and Resolution Enhancement. IEEE Transaction on Inage Proceeding 10(9), 1299–1308 (2001) 7. Peleg, S., Keren, D., Schweitzer, L.: Improving Image Resolution using Subpixel Motion. Pattern Recognition Letters 5(3), 223–226 (1987) 8. Irani, M., Peleg, S.: Motion Analysis for Image Enhancement: Resolution, Occlusion, and Transparency. Journal of Visual Communication and Image Representaion 4, 324–355 (1993) 9. Capel, D.P.: Image Mosaicing and Super-resolution. PhD thesis, University of Oxford (2001) 10. Pickup, L.: Machine Learning in Multi-frame Image Super-resolution. PhD thesis, University of Oxford (2001) 11. Farsiu, S., Robinson, M.D., Elad, M., Milanfar, P.: Fast and Robust Multi-Frame Super Resolution. Image Processing 13(10), 1327–1344 (2004) 12. Irani, M., Peleg, S.: Improving Resolution by Image Registration. Graphical Models and Image Processing 53, 231–239 (1991) 13. Suresh, K.V., Mahesh Kumar, G., Rajagplalan, A.N.: Super-resolution of License Plates in Real Traffic Videos. IEEE Transaction on Intelligent Transportation Systems 8(2) (2007) 14. Zhao, W., Sawhney, H.S.: Is super-resolution with optical flow feasible? In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002. LNCS, vol. 2350, pp. 599–613. Springer, Heidelberg (2002) 15. Zitova, B., Flusser, J.: Image Registration Methods: a survey. Image and Vision Computing 21, 977–1000 (2003) 16. Hartley, R.I., Zisserman, A.: Multiple View Geometry in Computer Vision, 2nd edn. Cambridge University Press, Cambridge (2004) 17. Chung, J., Nagy, J.G., O’leary, D.: Weighted-GCV Method for Lanczos-Hybrid Regularization. Electronic Transactions on Numerical Analysis 28, 149–167 (2008)
Facial Expression Recognition on Hexagonal Structure Using LBP-Based Histogram Variances Lin Wang1 , Xiangjian He1,2 , Ruo Du2 , Wenjing Jia2 , Qiang Wu2 , and Wei-chang Yeh3 1
Video Surveillance Laboratory Guizhou University for Nationalities Guiyang, China
[email protected] 2 Centre for Innovation in IT Services and Applications (iNEXT) University of Technology, Sydney Australia {Xiangjian.He,Wenjing.Jia-1,Qiang.Wu}@uts.edu.au,
[email protected] 3 Department of Industrial Engineering and Engineering Management National Tsing Hua University Taiwan
[email protected]
Abstract. In our earlier work, we have proposed an HVF (Histogram Variance Face) approach and proved its effectiveness for facial expression recognition. In this paper, we extend the HVF approach and present a novel approach for facial expression. We take into account the human perspective and understanding of facial expressions. For the first time, we propose to use the Local Binary Pattern (LBP) defined on the hexagonal structure to extract local, dynamic facial features from facial expression images. The dynamic LBP features are used to construct a static image, namely Hexagonal Histogram Variance Face (HHVF), for the video representing a facial expression. We show that the HHVFs representing the same facial expression (e.g., surprise, happy and sadness etc.) are similar no matter if the performers and frame rates are different. Therefore, the proposed facial recognition approach can be utilised for the dynamic expression recognition. We have tested our approach on the well-known Cohn-Kanade AU-Coded Facial Expression database. We have found the improved accuracy of HHVF-based classification compared with the HVF-based approach. Keywords: Histogram Variance Face, Action Unit, Hexagonal structure, PCA, SVM.
1 Introduction Human facial expression reflects human’s emotions, moods, attitudes and feelings etc.. Recognising expressions can help computer learn more about human’s mental activities and react more sophisticatedly, so it has enormous potentials in the field of humancomputer interaction (HCI). Explicitly, the expressions are some facial muscular movements. Six basic emotions (happiness, sadness, fear, disgust, surprise and anger) were K.-T. Lee et al. (Eds.): MMM 2011, Part II, LNCS 6524, pp. 35–45, 2011. c Springer-Verlag Berlin Heidelberg 2011
36
L. Wang et al.
defined in the Facial Action Coding System (FACS) [4]. FACS consists of 46 action units (AU) which depict basic facial muscular movements. Basically, how to capture the expression features precisely is vital for expression recognition. Getting expression features can be divided into two categories: spatial and spatio-temporal approaches. Because spatial approaches [5][10][24] do not model the dynamics of facial expressions, spatio-temporal approaches have dominated the recent research for ficial expression recognition. In their spatio-temporal approaches, Maja et al. [17][18][23] detected AUs by using individual feature GentleBoost [22] templates built from Gabor wavelet features and tracked temporal AUs based on Particle Filter (PF). Then, the SVMs [2][19][20] was applied for classification. Petar et al. [1] treated the facial expression as a dynamic process and proposed that the performance of an automatic facial expression recognition system could be improved by modeling the reliability of different streams of facial expression information utilising multistream Hidden Markov Models (HMMs) [1]. Although the above spatio-temporal approaches have taken into account modeling dynamic features, the model parameters are often hard to be obtained accurately. In our previous approach [3], the extraction of expression features saved the dynamic features into a Histogram Variance Face(HVF) image by computing the texture histogram variances among the frames of a face video. The common Local Binary Pattern (LBP) [12][13] was employed to extract the face texture. We classified the HVFs using Support Vector Machines (SVMs) [2][19][20] after applying Principal Component Analysis (PCA) [15][16] for dimensionality reduction. The accuracy of HVF classification was very encouraging and highly matched human’s perception on original videos. In this paper, we extend our work shown in [3]. We, for the first time, apply LBPs defined on the hexagonal structure [6] to extract the Hexagonal Histogram Variance Faces (HHVFs). This novel approach not only greatly reduce the computation costs but also improve the accuracy for facial expression recognition. The rest of the paper is organised as follows. Section 2 introduces the LBPs on the hexagonal structure, and describes the approach to generating the HHVF image from an input video based on the LBP operator. Section 3 presents the dimensionality reduction using PCA, and the training and recognition steps using SVMs. Experimental results are demonstrated in Section 4. Section 5 makes the conclusions of this paper.
2 Hexagonal Histogram Variance Face (HHVF) The HVF image is a representation of the dynamic features in a face video [3]. An extension of HVF represented on the hexagonal structure, namely HHVF, is performed in this section. 2.1 Fiducial Point Detection and Face Alignment For different expression videos, normally the scales and locations of human faces in frames are various. To make all the HHVFs have the same scale and location, it is critical to detect the face fiducial points. Bilinear interpolation is used to scale the face images to the same size. To detect the fiducial points, we apply Viola-Jones face detector [21], a real-time face detection scheme based on Haar-like features and AdaBoost learning. We detect and locate the positions of eyes for the iamges on the Cohn-Kanade
Facial Expression Recognition on Hexagonal Structure
37
expression database [9]. Each face image is cut and scaled according to eyes’ positions and the distance between the eyes. 2.2 Conversion from Square Structure to Hexagonal Structure We follow the work shown in [7] to represent images on the hexagonal structure. As shown in Figure 1, the hexagonal pixels appear only on the columns where the square pixels are located. As illustrated in Figure 1, for a given hexagonal pixel (denoted by X but not shown in the figure), there exist two square pixels (denoted by A and B but again not shown in the figure), lying on two consecutive rows and the same column of X, such that point X falls between A and B. Therefore, we can use the linear interpolation algorithm to obtain the light intensity value of X from the intensities of A and B. When we display the image on the hexagonal structure, every two hexagonal rows as shown in Figure 1 is combined into one single square row with their columns unchanged.
Fig. 1. A 9x8 square structure and a constructed 14x8 hexagonal structure [7]
2.3 Preprocessing and LBP Texturising After each input face image is aligned, cut, rescaled and normalized, we replace the values of the pixels outside the ellipse area around the face by 255, and keep the pixel values unchanged in the elliptic face area. To eliminate the illumination interference, we employ an LBP operator [6][12][13] to extract the texture (i.e., LBP) values in each masked face. LBP was originally introduced by Ojala et al. in [11] as texture description and defined on the traditional square image structure. The basic form of an LBP operator labels the pixels of an image by thresholding the 3 × 3 neighborhood of each pixel by the grey value of the pixel (the centre). An illustration of the basic LBP operator is
38
L. Wang et al.
Fig. 2. An example of computing LBP in a 3 × 3 neighborhood on square structure [6]
Fig. 3. An example of computing HLBP in a 7-pixel hexagonal cluster [6]
shown in Figure 2. Similar to the construction of basic LBP on the square structure, the basic LBP on the hexagonal structure, called Hexagonal LBP (HLBP) is constructed as shown in Figure 3 [6] on a cluster of 7 hexagonal pixels. By defining the HLBPs on the hexagonal structure, the number of different patterns has been reduced from 28 = 256 on the square structure to 26 = 64 on the hexagonal structure. More importantly, because all neighboring pixels of a reference pixel have the same distance to it, the grey values of the neighboring pixels have the same contributions to the reference pixel on the hexagonal structure. 2.4 Earth Mover’s Distance (EMD) Earth Mover’s Distance (EMD) [14] is a cross-bin approach and able to address the shift problem caused by noise because slight histogram shifts do not affect the EMD much. EMD is consistent with the human’s vision because that two histograms will have greater EMD value if they look more different in most cases. EMD can be formalized as the following linear programming problem. Let P = {(p1 , wp1 ), . . . , (pm , wpm )}
(1)
be the first signature with m clusters, where pi is the cluster representative and wpi is the weight of the cluster. Let Q = {(q1 , wq1 ), . . . , (qn , wqn )}
(2)
be the second signature with n clusters. Let also D = [dij ] be the ground distance matrix where dij is the ground distance between clusters pi and qj . Then EMD is to find a flow F = [fij ], where fij is the flow between pi and qj , that minimizes the overall cost [14]
Facial Expression Recognition on Hexagonal Structure
W ORK(P, Q, F ) =
m n
dij fij
39
(3)
i=1 j=1
subject to the following constraints [14]: 1. f ij ≥ 0; i ∈ [1, m] , j ∈ [1, n], n 2. fij ≤ wpi ; i ∈ [1, m], j=1 m 3. fij ≤ wqj ; j ∈ [1,n] and i=1 m n m n 4. i=1 j=1 fij = min i=1 wpi , j=1 wqj . EMD is defined as the resulting work normalized by the total flow [14]: m n i=1 j=1 dij fij EM D(P, Q) = m n i=1 j=1 fij
(4)
In general, the ground distance dij can be any distance and will be chosen according to the problem in question. We employ EMD to measure the distance between two histograms when calculating histogram variances in the temporal direction. In our case, pi and qj are the grayscale pixel values, which are in [0, 63]. wpi and wqj are the pixel distributions at pi and qj respectively. The ground distance dij that we choose is the square of Euclidean distance between pi and qj , i.e., dij = (pi − qj )2 . 2.5 Histogram Variances The following steps for computing the histogram variance are similar to the ones shown in [3]. 1. Suppose that a sequence consists of P face texture images. We firstly break down each image evenly into M × N blocks, denoted by Bx,y;k , where x is row index, y is column index and k is the k-th frame in the sequence. Here, the block size, rows and columns are corresponding to those images displayed on the square structure as described in Subsection 2.2. We then calculate every gray-value histogram of Bx,y;k , denoted by Hx,y;k , where x = 0, 1, . . . , M − 1; y = 0, 1, . . . , N − 1; k = 0, 1, . . . , P − 1. 2. Calculate the histogram variance var(x, y): var(x, y) =
P −1 1 EM D(Hx,y;k , μx,y ), P
(5)
k=0
where μx,y is the mean histogram μx,y =
P −1 1 Hx,y;k P
(6)
k=0
and EM D(Hx,y;k , μx,y ) is the Earth Mover’s Distance between Hx,y;k and μx,y .
40
L. Wang et al.
3. Construct an M × N 8-bit grayscale image as our HHVF. Suppose that hhvf (x, y) denotes the pixel value at coordinate (x, y) in an HHVF image. Then, hhvf (x, y) is computed by 255 ∗ var(x, y) hhvf (x, y) = 255 − . (7) max{var(x, y)} Figure 4 shows some HHVF examples extracted from happiness, surprise and sadness videos respectively. To consider if the different block sizes may affect the recognition, we obtain HHVFs with size 3 × 3 in our experiments, then with the sizes of 6 × 6 and 12 × 12.
Fig. 4. Examples of HHVF images
3 Classifying HHVF Images Using PCA+SVMs HHVF records the dynamic features of the expression. As we can see in Figure 4, for the expressions of happiness, surprise and sadness, the homogeneous HHVFs look similar and HHVFs belonging to different expressions have very distinct features. To verify the performance of HHVF image’s features, we utilise the typical facial recognition technologies PCA+SVMs. 3.1 PCA Dimensionality Reduction In our experiments, all pixel values of an HHVF image construct an n × 1 column vector zi ∈ Rn , and an n × l matrix Z = {z1 , z2 , . . . , zl } denotes the training set which consists of l sample HHVF images. The PCA algorithm finds a linear transformation orthonormal matrix Wn×r (n >> r), projecting the original high n-dimensional feature space into a much lower r-dimensional feature subspace. xi denotes the new feature vector: xi = W T · zi (i = 1, 2, . . . , l). (8)
Facial Expression Recognition on Hexagonal Structure
41
The columns of matrix W (i.e., eigenfaces [16]) are the eigenvectors corresponding to the largest r eigenvalues of the scatter matrix S: l S= (zi − μ)(zi − μ)T
(9)
i=1
where μ is the mean image of all HHVF samples and μ =
1 l
1
i=1 zi .
3.2 SVMs Training and Recognition SVM [2][20][19] is an effective supervised classification algorithm and its essence is to find a hyperplane that separates the positive and negative feature points with maximum margin in the feature space. Suppose αi (i = 1, 2, ..., l) denotes the Lagrange parameters that describe the separating hyperplane ω in an SVM. Finding the hyperplane involves getting the nonzero solutions αi (i = 1, 2, ..., l) of a Lagrangian dual problem. Once we have found all αi for given a labeled training set {(xi , yi )|i = 1, 2, ..., l}, the decision function becomes: l f (x) = sgn αi yi K(x, xi ) + b , (10) i=1
where b is the bias of the hyperplane, l is the number of training samples, xi (i = 1, 2, ..., l) is the vector of PCA projection coefficients of HHVFs, yi are the labels of train data xi for (i = 1, 2, ..., l), and K(x, xi ) is the ‘kernel mapping’. In this paper, we use linear SVMs, so K(x, xi ) = x, xi , (11) where x, xi represents the dot product of x and xi . Since the SVM is basically a two-class classification algorithm, here we adopt the pairwise classification (one-versus-one) for multi-class. In the pairwise classification approach, there is a two-class SVM for each pair of classes to separate the members of one class from the members of the others. There are maximum C62 = 15 two-class SVM classifiers for the classification of six classes of expressions. For recognising a new HHVF image, all C62 = 15 two-class classifiers are applied and the winner class is the one that receives the most votes.
4 Experiments 4.1 Dataset Our experiments use the Cohn-Kanade AU-Coded Facial Expression Database [9]. We select 49 subjects randomly from the database. Each subject has up to 6 expressions. The total number of expression images is 241. The image sequences belonging to the same expression have the similar duration but their frame rates are various and are from 15-30 frames per second. For each expression, we use about 80% HHVFs for PCA+SVMs training and use all HHVFs for classification (i.e., testing).
42
L. Wang et al.
4.2 HHVFs Generation The faces of selected subjects are detected and cut. Then these faces are scaled to size of 300 × 300 and aligned. These cut and rescaled images are then converted to the images on the hexagonal structure. The size of the new images are of 259×300. HLBP operator is then applied to the images on the hexagonal structure. The final data dimension is reduced to 220 after the PCA operation. 4.3 Training and Recognition As a supervised learning, the training data (HHVFs) are labeled according to their classes. Since the Cohn-Kanade database contains only the AU-Coded combinations for image sequences instead of expression definitions (i.e. surprise, happy, anger etc.), we need to label each HHVF with an expression definition manually according to FACS before feeding it to SVMs. In terms of human perception, we are quite confident to recognise original image sequences of happiness and surprise. Therefore, the training data for these two classes can be labeled with high accuracy. This implies that these two expressions have evidently unique features. Our experimental results (Table 1) testify this point with high recognition rate when we train and test only these two expressions. In Table 1, FPR stands for false positive rate. Table 1. Recognition rates of happy and surprise HHVFs 3 × 3 blocks 6 × 6 blocks 12 × 12 blocks Recognition rate FPR Recognition rate FPR Recognition rate FPR Happy 100% 2.17% 97.78% 2.17% 97.78% 2.17% Surprise 97.82% 0.00% 97.82% 2.22% 97.82% 2.22%
When we manually label HHVFs of anger, disgust, fear and sadness, nearly half of them are very challenging to be correctly classified. Neither AU-Coded combinations nor human’s perception can satisfactorily classify these facial expression sequences, especially the anger and sadness sequences. Actually, AU-Coded prototypes in FACS (2002 version) [9] have created large overlaps for these expressions. From human’s vision perspective, one expression appearance of a person may reflect several different expressions, so the image sequence of a facial expression can often be interpreted as for different expressions. An investigation [8] about facial expression recognition by human indicates that compared to the expressions of happy and surprise, the expressions of anger, fear, disgust and sadness are much more difficult to be recognised by people. (see Table 2). Despite the above-mentioned difficulties, we have tried our best to manually label the expressions. We have also conducted the following experiments: 1. We test for only anger, disgust, fear and sadness classes. For these four tough expressions, we train a set of classifiers having C42 = 6 two-class classifiers and test the HHVFs based on majority voting. Table 3 shows our results.
Facial Expression Recognition on Hexagonal Structure
43
Table 2. A recent investigation of facial expression recognition by human in [8] Happy Surprise Angry Disgust Fear Sadness Recognition rate(%) 97.8 79.3 55.9 60.2 36.8 46.9
Table 3. Recognition rates of anger, disgust, surprise and sadness HHVFs 3 × 3 blocks 6 × 6 blocks 12 × 12 blocks Recognition rate FPR Recognition rate FPR Recognition rate FPR Angry 74.36% 13.51% 76.92% 13.51% 74.36% 14.41% Disgust 73.68% 10.71% 73.68% 9.82% 71.05% 10.71% Fear 68.57% 8.69% 71.43% 7.83% 68.57% 8.69% Sadness 68.42% 5.36% 68.42% 5.36% 68.42% 5.35%
2. We consider all HHVFs, and train C62 = 15 two-class classifiers. We use this set of classifiers to recognise all HHVFs based on voting. We obtain the experimental results as shown in Table 4. Table 4. Recognition rates of all sorts of HHVFs 3 × 3 blocks 6 × 6 blocks 12 × 12 blocks Recognition rate FPR Recognition rate FPR Recognition rate FPR Happy Surprise Angry Disgust Fear Sadness
93.33% 91.3% 82.05% 78.95% 77.14% 78.95%
0.0% 0.51% 7.43% 4.93% 5.34% 0.49%
95.56% 93.48% 82.05% 81.58% 80.00% 78.95%
0.0% 0.51% 5.94% 4.43% 5.34% 0.49%
95.56% 93.48% 76.92% 81.58% 77.14% 78.95%
0.0% 0.51% 5.94% 4.93% 5.34% 1.48%
4.4 Discussion 1. From Table 1, we can see that both happy and surprise HHVFs have produced very high recognition rates. For example, the happy HHVFs reach amazing 100% recognition rate with only 2.17% false positive rate (FPR). These results coincide with our observation on the original image sequences, as human can also easily identify the original happy and surprise sequences from the Cohn-Kanade database. 2. From Table 3, the recognition rates for anger, fear, disgust and sadness HHVFs are much lower. This reflects the challenges that we have encountered when labeling the training data (that nearly half of the training data for these four expressions are not convincing us to accurately label their classes because of the expression feature entanglement). 3. Table 4 shows the recognition results when all six expressions were fed to SVMs for training, we can see that the happy and surprise HHVFs still stand out and the rest are hampered by the entanglement of features. Taking into account our difficulties for labeling the training data, the recognition rates in Table 4 make sense.
44
L. Wang et al.
4. The size of blocks is not critical to our results, although the segmentation into 6 × 6 blocks has the best performance in our experiments as shown in Table 4.
5 Conclusions Our experiments demonstrate that HHVF is an effective representation of the dynamic and internal features of a face video or an image sequence. They show that the accuracy has been improved when comparing with the results shown in [3]. The accuracy has been increased on average from 83.68% (see Table 5 in [3]) to 85.27% (see Table 4 above) when looking only the block size of 6 × 6. HHVF is able to integrate well the dynamic features of a certain duration of expression into a static image through which the static facial recognition approaches can be utlised to recognise the dynamic expressions. The application of HHVFs fills the gap between the expression recognition and facial recognition.
Acknowledgements This work is supported by the Houniao Program through Guizhou University for Nationalities, China.
References 1. Aleksic, P.S., Katsaggelos, A.K.: Automatic facial expression recognition using facial animation parameters and multistream hmms. In: Information Forensics and Security, vol. 1, pp. 3–11 (2006) 2. Boser, B., Guyon, I., Vapnik, K.: An training algorithm for optimal margin classifiers. In: Fifth Annual Workshop on Computational Learning Theory, pp. 144–152 (1992) 3. Du, R., Wu, Q., He, X., Jia, W., Wei, D.: Local binary patterns for human detection on hexagonal structure. In: IEEE Workshop on Applications of Computer Vision, pp. 341–347 (2009) 4. Ekman, P., Friesen, W.: Facial action coding system. Consulting Psychologists Press, Palo Alto (1978) 5. Feng, X., Pietikinen, M., Hadid, A.: Facial expession recognition with local binary partterns and linear programming. Pattern Recognition and Image Analysis 15(2), 546–548 (2005) 6. He, X., Li, J., Chen, Y., Wu, Q., Jia, W.: Local binary patterns for human detection on hexagonal structure. In: IEEE International Symposium in Multimedia, pp. 65–71 (2007) 7. He, X., Wei, D., Lam, K.-M., Li, J., Wang, L., Jia, W., Wu, Q.: Local binary patterns for human detection on hexagonal structure. In: Advanced Concepts for Intelligent Vision Systems (to appear, 2010) 8. Jinghai, T., Zilu, Y., Youwei, Z.: The contrast analysis of facial expression recognition by human and computer. In: ICSP, pp. 1649–1653 (2006) 9. Kanade, T., Cohn, J.F., Tian, Y.: Comprehensive database for facial expression analysis. In: Proceedings of the Fourth IEEE International Conference on Automatic Face and Gesture Recognition, Grenoble, France, pp. 46–53 (2000) 10. Littlewort, G., Bartlett, M., Fasel, I., Susskind, J., Movellan, J.: Dynamics of facial expression extracted automatically from video. In: CVPR (2004)
Facial Expression Recognition on Hexagonal Structure
45
11. Ojala, T., Pietik¨ainen, M., Harwood, D.: A comparative study of texture measures with classification based on feature distributions. Pattern Recognition 29, 51–59 (1996) 12. Ojala, T., Pietik¨ainen, M., M¨aenp¨aa¨ , T.: Gray scale and rotation invariant texture classification with local binary patterns. In: Vernon, D. (ed.) ECCV 2000. LNCS, vol. 1842, pp. 404–420. Springer, Heidelberg (2000) 13. Ojala, T., Pietik¨ainen, M., M¨aenp¨aa¨ , T.: Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. Pattern Analysis and Machine Intellifence 24, 971–987 (2002) 14. Rubner, Y., Tomasi, C., Guibas, L.J.: The earth mover’s distance as a metric for image retrieval. International Journal of Computer Vision 40(2), 99–121 (2000) 15. Turk, M., Pentland, A.: Eigenfaces for recognition. Journal of Cognitive Neuroscience 3(1), 71–86 (1991) 16. Turk, M., Pentland, A.: Face recognition using eigenfaces. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 586–591 (1991) 17. Valstar, M., Patras, I., Pantic, M.: Facial action unit detection using probabilistic actively learned support vector machines on tracked facial point data. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (2005) 18. Valstar, M., Pantic, M.: Fully automatic facial action unit detection and temporal analysis. In: Computer Vision and Pattern Recognition Workshop, vol. 3, June 17-22, p. 149 (2006) 19. Vapnik, V.N.: Statistical learning theory. Wiley Interscience, Hoboken (1998) 20. Vapnik, V.: The nature of statistical learning theory. Springer, Heidelberg (1995) 21. Viola, P., Jones, M.J.: Robust real-time object detection. In: ICCV (2001) 22. Vukadinovic, D., Pantic, M.: Fully automatic facial feature point detection using gabor feature based boosted classifiers. In: SMC, pp. 1692–1698 (2005) 23. Vukadinovic, D., Pantic, M.: Fully automatic facial feature point detection using gabor feature based boosted classifiers. In: Systems, Man and Cybernetics (ICSMC), vol. 2, pp. 1692–1698 (2005) 24. Ying, Z., Fang, X.: Combining lbp and adaboost for facial expression recognition. In: ICSP, October 26-29, pp. 1461–1464 (2008)
Towards More Precise Social Image-Tag Alignment Ning Zhou1,2 , Jinye Peng1 , Xiaoyi Feng1 , and Jianping Fan1,2 1
School of Electronics and Information, Northwestern Polytechnical University, Xi’an, P.R. China {jinyepeng,fengxiao}@nwpu.edu.cn 2 Dept. of Computer Science, UNC-Charlotte, Charlotte, NC 28223, USA {nzhou,jfan}@uncc.edu
Abstract. Large-scale user contributed images with tags are increasingly available on the Internet. However, the uncertainty of the relatedness between the images and the tags prohibit them from being precisely accessible to the public and being leveraged for computer vision tasks. In this paper, a novel algorithm is proposed to better align the images with the social tags. First, image clustering is performed to group the images into a set of image clusters based on their visual similarity contexts. By clustering images into different groups, the uncertainty of the relatedness between images and tags can be significantly reduced. Second, random walk is adopted to re-rank the tags based on a cross-modal tag correlation network which harnesses both image visual similarity contexts and tag co-occurrences. We have evaluated the proposed algorithm on a large-scale Flickr data set and achieved very positive results. Keywords: Image-tag alignment, relevance re-ranking, tag correlation network.
1
Introduction
Recently, as online photo sharing platforms (e.g. Flickr [7], Photostuff [9], etc) are becoming increasingly popular, massive images have been uploaded onto the Internet and have been collaboratively tagged by a large population of real world users. User contributed tags provide semantically meaningful descriptors of the images which are essential for large-scale tag-based retrieval systems to work in practice [14]. In such a collaborative image tagging system, users have been tagging the images according to their own social or cultural backgrounds, personal knowledge and interpretation. Without controlling the tagging vocabulary and behavior, many tags in the user provided tag list might be synonyms or polysemes or even spams. Also, many tags are personal tags which are weakly related or even irrelevant to the image content. A recent work reveals that many Flickr tags are imprecise and only around 50% tags are actually related to the image contents [11]. Therefore, the collaboratively tagged images are weakly-tagged images because the social tags may not have exact correspondences with the underlying K.-T. Lee et al. (Eds.): MMM 2011, Part II, LNCS 6524, pp. 46–56, 2011. c Springer-Verlag Berlin Heidelberg 2011
Towards More Precise Social Image-Tag Alignment
47
image semantics. The synonyms, polysemes, spams and content irrelevant tags in these weakly-tagged images may either return incomplete sets of the relevant images or result in large amounts of ambiguous images or even junk images [4,6,5]. It is important to better align the images with the social tags to make them more searchable and to leverage these large-scale weakly-tagged images for computer vision tasks. By achieving more accurate alignment between the weakly-tagged images and their tags, we envision many potential applications: 1. Enabling Social Images to be More Accessible: The success of Flickr proves that users are willing to manually annotate their images with the motivation to make them more accessible to the general public [1]. However, the tags are provided in a free way without a controlled vocabulary based on their own personal perceptions. Lots of social tags are synonyms or polysemes or enven spams, which pose a big limitation for users to locate their interested images accurately and efficiently. To enable more effective social image organization and retrieval, it is very attractive to develop new algorithms to achieve more precise alignment between the images and their social tags. 2. Creating Labeled Images for Classifier Training: For many computer vision tasks, e.g. object detection and scene recognition, machine learning models are often used to learn the classifiers from a set of labeled training images [2]. To achieve satisfied performance, large-scale of training examples are required, due to (a) the number of object classes and scenes of interest could be very large; (b) the learning complexity for some object classes and scenes could be very high because of visual ambiguity; and (c) a small number of labeled training images are incomplete or insufficient to interpret the diverse visual properties of large amounts of unseen test images. Unfortunately, hiring professionals to label large amounts of training images is cost-sensitive which is a key limitation for the practical use of some advanced computer vision techniques. On the other hand, the increasing availability of large-scale user contributed digital images with tags on the Internet has provided new opportunities to harvest large-scale images for computer vision tasks. In this paper, we proposed a clustering based framework to align images and social tags. By crawling large number of weakly-tagged images from Flickr, we first cluster them into different groups based on visual similarities. For each group, we aggregate all the tags associated to the images within this cluster to form a larger set of tags. A random walk process is then applied onto this tag set to further improve the alignment between images and tags based on a tag correlation network which is constructed by combining both image visual similarity contexts and tag co-occurrences. We evaluated our proposed algorithm on a Flickr database with 260,000 images and achieved very positive results.
2
Image Similarity Characterization
To achieve more effective image content representation, four grid resolutions (whole image, 64 × 64, 32 × 32 and 16 × 16 patches) are used for image partition
48
N. Zhou et al.
and feature extraction. To sufficiently characterize various visual properties of the diverse image contents, we have extracted three types of visual features: (a) grid-based color histograms; (b) Gabor texture features; and (c) SIFT (scale invariant feature transform) features. For the color features, one color histogram 3 is extracted for each image grid, thus there are r=0 2r ×2r = 85 grid-based color histograms. Each grid-based color histogram consists of 36 RGB bins to represent the color distributions in the corresponding image grid. To extract Gabor texture features, a Gabor filter bank, which contains twelve 21 × 21 Gabor filters in 3 scales and 4 orientations, is used. The Gabor filter is generated by using a Gabor function class. To apply Gabor filters on an image, we need to calculate the convolutions of the filters and the image. We transform both the filters and the image into the frequency domain to get the products and then transform them back to the space domain. This process can calculate Gabor filtered image more effectively. Finally, the mean values and standard deviations are calculated from 12 filtered images, making up to 24-dimensional Gabor texture features. By using multiple types of visual features for image content representation, it is able for us to characterize various visual properties of the images more sufficiently. Because each type of visual features is used to characterize one certain type of visual properties of the images, the visual similarity contexts between the images are more homogeneous and can be approximated more precisely by using one particular type of base kernel. Thus one specific base kernel is constructed for each type of visual features. For two images u and v, their color similarity relationship can be defined as: κc (u, v) =
R−1 1 1 D (u, v) + Dr (u, v) 0 R R−r+1 2 2 r=0
(1)
where R = 4 is the total number of grid resolutions for image partition, D0 (u, v) is the color similarity relationship according to their full-resolution (image-based) color histograms, Dr (u, v) is the grid-based color similarity at the rth resolution. Dr (u, v) =
36
D(Hir (u), Hir (u))
(2)
i=1
where Hir (u) and Hir (v) are the ith component of the grid-based color histograms at the rth image partition resolution. Their local similarity relationship can be defined as: κs (u, v) = e−ds (u,v)/σs (3) ds (u, v) =
i
j
ωi (u)ωj (v)d1 (si (u), sj (v)) i j ωi (u)ωj (v)
(4)
where σs is the mean value of ds (u, v) in our images, ωi and ωj are the Hessian values of the ith and jth interesting points. Their textural similarity relationship can be defined as: κt (u, v) = e−dt(u,v)/σt , dt (u, v) = d1 (gi (u), gj (v))
(5)
Towards More Precise Social Image-Tag Alignment
49
where σt is the mean value of dt (u, v) in our images, d1 (gi (u), gj (v)) is the L1 norm distance between two Gabor textural descriptors. The diverse visual similarity contexts between the online images can be characterized more precisely by using a mixture of these three base image kernels (i.e., mixture-of-kernels) [3]. κ(u, v) =
3
βi κi (u, v),
i=1
3
βi = 1
(6)
i=1
where u and v are two images, βi ≥ 0 is the importance factor for the ith base kernel κi (u, v). Combining multiple base kernels can allow us to achieve more precise characterization of the diverse visual similarity contexts between the two images.
3
Image Clustering
To achieve more effective image clustering, a graph is first constructed for organizing the social images according to their visual similarity contexts [8], where each node on the graph is one particular image and an edge between two nodes is used to characterize the visual similarity context between two images, κ(·, ·). By taking such image graph as the input measures of the pairwise image similarity, automatic image clustering is achieved by passing messages between the nodes of the image graph through affinity propagation [8]. After the images are partitioned into a set of image clusters according to their visual similarity contexts, the cumulative inter-cluster visual similarity context s(Gi , Gj ) between two image clusters Gi and Gj is defined as: s(Gi , Gj ) = κ(u, v) (7) u∈Gi v∈Gj
The cumulative intra-cluster visual similarity contexts s(Gi , Gi ) and s(Gj , Gj ) are defined as: s(Gi , Gi ) = κ(u, v), s(Gj , Gj ) = κ(u, v) (8) u∈Gi v∈Gi
u∈Gj v∈Gj
By using the cumulative intra-cluster visual similarity context to normalize the cumulative inter-cluster visual similarity context, the inter-cluster correlation c(Gi , Gj ) between the image clusters Gi and Gj is defined as: c(Gi , Gj ) =
2s(Gi , Gj ) s(Gi , Gi ) + s(Gj , Gj )
(9)
Three experimental results for image clustering are given in Fig. 1. From these experimental results, one can observe that visual-based image clustering can provide a good summarization of large amounts of images and discover comprehensive knowledge effectively. The images in the same cluster will share similar visual properties and their semantics can be effectively described by using a same set of tags.
50
N. Zhou et al.
Fig. 1. Image clustering results and their associated tags
4
Image and Tag Alignment
In order to filter out the misspelled and content irrelevant tags, we adopt a knowledge-based method [12] to discern content-related tags from contentunrelated ones and retain only the content-related tags to form our tag vocabulary. In particular, we first choose a set of high level categories, including “organism”, “artifact”, “thing”, “color” and “natural phenomenon”, as a taxonomy of the domain knowledge in computer vision area. The content relatedness of a particular tag is then determined by resorting to the WordNet [15] lexicon which preserves a semantical structure among words. Specifically, for each tag, if it is in the noun set of WordNet, we further traverse along the path which contains of hypernyms of the tag until one of the pre-defined categories is matched. If successfully matched, it is considered as a content-related tag, otherwise it is decided as content-unrelated one. Rather than indexing the visually-similar images in the same cluster by loosely using all these tags, a novel tag ranking algorithm is developed for aligning the the tags with the images according to their relevance scores. The alignment algorithm works as follows: (a) The initial relevance scores for the tags are calculated via tag-image co-occurrence probability estimation; (b) The relevance scores for these tags are then refined according to their inter-tag cross-modal similarity contexts based on tag correlation network; and (c) The most relevant tags (top k tags with largest relevance scores) are selected automatically for image semantics description. Our contributions lie in integrating tag correlation network and random walk for automatic relevance score refinement. 4.1
Tag Correlation Network
When people tag images, they may use multiple different tags with similar meanings to describe the semantics of the relevant images alternatively. On the other hand, some tags have multiple senses under different contexts. Thus the tags are strongly inter-related and such inter-related tags and their relevance scores should be considered jointly. Based on this observation, a tag correlation network is generated automatically for characterizing such inter-tag similarity contexts more precisely and providing a good environment to refine the relevance
Towards More Precise Social Image-Tag Alignment
51
Fig. 2. Different views of our tag correlation network
scores for the inter-related tags automatically. In our tag correlation network, each node denotes a tag, and each edge indicates the pairwise inter-tag correlation. The inter-tag correlations consist of two components: (1) inter-tag cooccurrences; and (2) inter-tag visual similarity contexts between their relevant images. Rather than constructing such tag correlation network manually, an automatic algorithm is developed for this aim. For two given tags (one tag pair) ti and tj , their visual similarity context γ(ti , tj ) is defined as: γ(ti , tj ) =
1 κ(u, v) |Ci ||Cj |
(10)
u∈Ci v∈Cj
where Ci and Cj are the image sets for the tags ti and tj , |Ci | and |Cj | are the numbers of the web images for Ci and Cj , κ(u, v) is the kernel-based visual similarity context between two images u and v within Ci and Cj respectively. The co-occurrence β(ti , tj ) between two tags ti and tj is defined as: β(ti , tj ) = −P (ti , tj )log
P (ti , tj ) P (ti ) + P (tj )
(11)
where P (ti , tj ) is the co-occurrence probability for two tags ti and tj , P (ti ) and P (tj ) are the occurrence probabilities for the tags ti and tj . Finally, we define the cross-modal inter-tag correlation between ti and tj by combining their visual similarity context and co-occurrence, given as φ(ti , tj ) = α · γ(ti , tj ) + (1 − α) · β(ti , tj )
(12)
where α is the weighting factor and it is determined through cross-validation. The combination of such cross-modal inter-tag correlation can provide a powerful framework for re-ranking the relevance scores between the images and their tags.
52
N. Zhou et al.
The tag correlation network for tagged image collections from Flickr is shown in Fig. 2, where each tag is linked with multiple most relevant tags with values of φ(·, ·) over a threshold. By seamlessly integrating the visual similarity contexts between the images and the semantic similarity contexts between the tags for tag correlation network construction, the tag correlation network can provide a good environment for addressing the issues of polysemy and synonyms more effectively and disambiguating the image senses precisely, which may allow us to find more suitable tags for more precise image tag alignment. 4.2
Random Walk for Relevance Refinement
In order to take advantage of the tag correlation network to achieve more precise alignment between the images and their tags, a random walk process is performed for automatic relevance score refinement [13,10,16]. Given the tags correlation network with n most frequent tags, we use ρk (t) to denote the relevance score for the tag t at the kth iteration. The relevance scores for all these tags at the −−→ kth iteration will form a column vector ρ(t) ≡ [ρk (t)]n×1 . We further define Φ as an n × n transition matrix, its element φij is used to define the probability of the transition from the tag i to its inter-related tag j. φij is defined as: φ(i, j) φij = k φ(i, k)
(13)
where φ(i, j) is the pairwise inter-tag cross-modal similarity context between i and j as defined in (12). The random walk process is thus formulated as: ρk (t) = θ ρk−1 (j)φtj + (1 − θ)ρ(C, t) (14) j∈Ωj
where Ωj is the first-order nearest neighbors of the tag j on the tag correlation network, ρ(C, t) is the initial relevance score for the given tag t and θ is a weight parameter. This random walk process can promote the tags which have many nearest neighbors on the tag correlation network, e.g., the tags, which have close visualbased interpretations of their semantics and higher co-occurrence probabilities. On the other hand, this random walk process can also weaken the isolated tags on the tag correlation network, e.g., the tags, which have weak visual correlations with other tags and low co-occurrence probabilities with other tags. This random walk process is terminated when the relevance scores converge. For a given image cluster C, all its tags are re-ranked according to their relevance scores. By performing random walk over the tag correlation network, the refinement algorithm can leverage both the co-occurrence similarity and the visual similarity simultaneously to re-rank the tags more precisely. The top-k tags, which have higher relevance scores with the image semantics, are then selected as the keywords to interpret the images in the given image cluster C. Such image-tag alignment process provides better understanding of the crossmedia information (images and tags) as it couples different sources of information
Towards More Precise Social Image-Tag Alignment
53
(a)
(b)
(c)
Fig. 3. Image-tag alignment: (a) image cluster; (b) ranked tags before performing random walk; (c) re-ranked tags after performing random walk
together and allow us to resolve the ambiguities that may arise from a single media analysis. Some experimental results for re-ranking the tags are given in Fig. 3. From these experimental results, one can observe that our image-tag alignment algorithm can effectively find the most relevant keywords to better align images with corresponding tags.
5
Algorithm Evaluation
All our experiments are conducted on an Flickr data set collected from Flickr by using Flickr API. There are 260,000 weakly tagged images in this set. To assess the effectiveness of our proposed algorithms, our algorithm evaluation work focuses on assessing the effectiveness of image clustering and random walk on our algorithm for image-tag alignment. The accuracy rate is used to assess the effectiveness of the algorithms for image-tag alignment, given as: N δ(Li , Ri )
= i=1 (15) N where N is the total number of images, Li is a set of the most relevant tags for the ith image which are obtained by automatic image-tag alignment algorithms, Ri is a set of the keywords for the ith image which are given by a benchmark image set. δ(x, y) is a delta function, ⎧ ⎨ 1, x = y, δ(x, y) = (16) ⎩ 0, otherwise
bread carriage caribou carrot tiger brush maker cruiser lake bridge cart car river pluto rainbow brooklyn yacht boat mountain park peak machine brick pin mickey yukon wood aircraft fall toyota dish vegetable shirt cupboard sunset carpet galaxy cabin district black flour cabinet oak candy cavern arch rug street tabriz forest furniture spectrum reef broccoli wall vintage glacier island mount harbour cascade
Accuracy bread carriage caribou carrot tiger brush maker cruiser lake bridge cart car river pluto rainbow brooklyn yacht boat mountain park peak machine brick pin mickey yukon wood aircraft fall toyota dish vegetable shirt cupboard sunset carpet galaxy cabin district black flour cabinet oak candy cavern arch rug street tabriz forest furniture spectrum reef broccoli wall vintage glacier island mount harbour cascade
Accuracy
54 N. Zhou et al.
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2 Without Clustering Without Random Walk Integration
0.2
Fig. 4. Image-tag alignment accuracy rate, where top 20 images are evaluated interactively. Average accuracy: Without Clustering = 0.6516, Without Random Walk = 0.7489, Integration = 0.8086.
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
Without Clustering Without Random Walk Integration
0.1
Fig. 5. Image-tag alignment accuracy rate, where top 40 images are evaluated interactively. Average accuracy: Without Clustering = 0.5828, Without Random Walk = 0.6936, Integration = 0.7373.
Towards More Precise Social Image-Tag Alignment
55
It is hard to obtain suitable benchmark image set in large size for our algorithm evaluation task. To avoid this problem, an interactive image navigation system is designed to allow users to provide their assessments of the relevances between the images and the ranked tags. To assess the effectiveness of image clustering and random walk for image-tag alignment, we have compared the accuracy rates for our image-tag alignment algorithm under three different scenarios: (a) image clustering is not performed for reducing the uncertainty between the relatedness between the images and their tags; (b) random walk is not performed for relevance re-ranking; (c) both image clustering and random walk are performed for achieving more precise alignment between the images and their most relevant social tags. As shown in Fig. 4 and Fig. 5, it can be seen that incorporating image clustering for uncertainty reduction and performing random walk for relevance re-ranking can significantly improve the accuracy rates for image-tag alignment.
6
Conclusions
In this paper, we proposed a cluster-based framework to provide a more precise image-tag alignment. By clustering visually similar images into clusters, the uncertainty between images and social tags has been reduced dramatically. In order to further refine the alignment, we have adopted a random walk process to re-rank the tags based on a cross-modal tag correlation network which is generated by using both image visual similarity and tag co-occurrences. Experimental results on real Flickr data set has empirically justified our developed algorithm. The proposed research on image-tag alignment may provide two potential applications: (a) enabling more effective tag-based web social image retrieval with higher precision rates by finding more suitable tags for social image indexing; (b) by harvesting social tagged images from Internet, our proposed research can create more representative image sets for training a large number of object and concept classifiers more accurately, which is a long-term goal of the multimedia research community. Acknowledgments. This research is partly supported by NSFC-61075014 and NSFC-60875016, by the Program for New Century Excellent Talents in University under Grant NCET-07-0693, NCET-08-0458 and NCET-10-0071 and the Research Fund for the Doctoral Program of Higher Education of China (Grant No.20096102110025).
References 1. Ames, M., Naaman, M.: Why we tag: motivations for annotation in mobile and online media. In: CHI, pp. 971–980 (2007) 2. Datta, R., Joshi, D., Li, J., Wang, J.Z.: Image retrieval: Ideas, influences, and trends of the new age. ACM Comput. Surv. 40(2) (2008)
56
N. Zhou et al.
3. Fan, J., Gao, Y., Luo, H.: Integrating concept ontology and multitask learning to achieve more effective classifier training for multilevel image annotation. IEEE Transactions on Image Processing 17(3), 407–426 (2008) 4. Fan, J., Luo, H., Shen, Y., Yang, C.: Integrating visual and semantic contexts for topic network generation and word sense disambiguation. In: CIVR (2009) 5. Fan, J., Shen, Y., Zhou, N., Gao, Y.: Harvesting large-scaleweakly-tagged image databases from the web. In: Proc. of CVPR 2010 (2010) 6. Fan, J., Yang, C., Shen, Y., Babaguchi, N., Luo, H.: Leveraging large-scale weaklytagged images to train inter-related classifiers for multi-label annotation. In: Proceedings of the First ACM Workshop on Large-scale Multimedia Retrieval and Mining, LS-MMRM 2009, pp. 27–34. ACM, New York (2009) 7. Flickr. Yahoo! (2005), http://www.flickr.com 8. Frey, B.J., Dueck, D.: Clustering by passing messages between data points. Science 315, 972–976 (2007) 9. Halaschek-Wiener, C., Golbeck, J., Schain, A., Grove, M., Parsia, B., Hendler, J.: Photostuff-an image annotation tool for the semantic web. In: Gil, Y., Motta, E., Benjamins, V.R., Musen, M.A. (eds.) ISWC 2005. LNCS, vol. 3729. Springer, Heidelberg (2005) 10. Hsu, W.H., Kennedy, L.S., Chang, S.-F.: Video search reranking through random walk over document-level context graph. In: ACM Multimedia, pp. 971–980 (2007) 11. Kennedy, L.S., Chang, S.-F., Kozintsev, I.: To search or to label?: predicting the performance of search-based automatic image classifiers. In: Multimedia Information Retrieval, pp. 249–258 (2006) 12. Liu, D., Hua, X.-S., Wang, M., Zhang, H.-J.: Retagging social images based on visual and semantic consistency. In: WWW, pp. 1149–1150 (2010) 13. Liu, D., Hua, X.-S., Yang, L., Wang, M., Zhang, H.-J.: Tag ranking. In: WWW, pp. 351–360 (2009) 14. Sigurbj¨ ornsson, B., van Zwol, R.: Flickr tag recommendation based on collective knowledge. In: Proceedings of the 17th International Conference on World Wide Web (WWW 2008), Beijing, China, April 21-25, pp. 327–336 (2008) 15. Stark, M.M., Riesenfeld, R.F.: Wordnet: An electronic lexical database. In: Proceedings of 11th Eurographics Workshop on Rendering. MIT Press, Cambridge (1998) 16. Tan, H.-K., Ngo, C.-W., Wu, X.: Modeling video hyperlinks with hypergraph for web video reranking. In: Proceeding of the 16th ACM International Conference on Multimedia, MM 2008, pp. 659–662. ACM, New York (2008)
Social Community Detection from Photo Collections Using Bayesian Overlapping Subspace Clustering Peng Wu1, Qiang Fu2, and Feng Tang1 1
Multimedia Interaction and Understanding Lab, HP Labs 1501 Page Mill Road, Palo Alto, CA, USA {peng.wu,feng.tang}@hp.com 2 Dept. of Computer Science & Engineering University of Minnesota, Twin Cities
[email protected]
Abstract. We investigate the discovery of social clusters from consumer photo collections. People’s participation in various social activities is the base on which social clusters are formed. The photos that record those social activities can reflect the social structure of people to a certain degree, depending on the extent of coverage of the photos on the social activities. In this paper, we propose to use Bayesian Overlapping Subspace Clustering (BOSC) technique to detect such social structure. We first define a social closeness measurement that takes people’s co-appearance in photos, frequency of co-appearances, etc. into account, from which a social distance matrix can be derived. Then the BOSC is applied to this distance matrix for community detection. BOSC possesses two merits fitting well with social community context: One is that it allows overlapping clusters, i.e., one data item can be assigned with multiple memberships. The other is that it can distinguish insignificant individuals and exclude those from the cluster formation. The experiment results demonstrate that compared with partition-based clustering approach, this technique can reveal more sensible community structure. Keywords: Clustering, Social Community.
1 Introduction Social relationship is formed based on people’s social activities. Individual’s daily activities define, evolve and reflect his/her social relationship with the rest of the world. Given the tremendous business value embedded in the knowledge of such social relationship, many industrial and academic efforts have been devoted to reveal the (partial) social relationship through studying certain types of social activities, such as email, online social networking, and instant messaging. In this paper, we present our work on discovering people’s social clusters and relationship closeness from analyzing photo collections. The work presented in [1] focuses on human evaluation of the social relationship in photos but not constructs the relationship through photo analysis. In [5], the social relationship is considered known and used to improve the face recognition performance. K.-T. Lee et al. (Eds.): MMM 2011, Part II, LNCS 6524, pp. 57–64, 2011. © Springer-Verlag Berlin Heidelberg 2011
58
P. Wu, Q. Fu, and F. Tang
Outside the photo media scopes, many works have been reported to discover social relationship in other kinds of social activities, such as the one in [4], which aims to construct the social structure from analyzing emails. In [6], a graph partition based clustering algorithm is proposed to address the social community detection from a collection of photos. Given a collection of photos, the faces identified through tagging or face detection are first grouped into different people identities. These identities form the vertices in the social graph, and the strength of the connection among them is measured by a social distance metric that takes the following factors into account. Given vertices of people and the distance measure of vertices, an undirected graph is constructed. The social community detection is formulated as a graph clustering problem and an eigenvector-based graph clustering approach [3] is adopted to detect the communities. Although using the eigenvector-based graph clustering algorithm, a reasonable set of communities are detected. There are two prominent shortcomings of the graph partition based clustering approach in detecting the social communities. First, one vertex can only belong to one cluster. In social context, that is equivalent to enforce one person play only one social role in the society, which is almost universally untrue in real life; Secondly, every vertex has to belong to a cluster. It is common that the photo collection contains not only the active members of communities, but also some individuals who just happen to present or are passing-through attendees of social events. However, as long as they are captured in the social graph, they will be assigned to a certain community, although they should be indeed treated as the “noise” data from analysis perspective. These shortcomings motivate us to use the Bayesian Overlapping Subspace Clustering technique [7] to address the social community detection challenge, as it has intrinsic treatment of noise data entries and support of overlapping memberships. The rest of the paper is organized as follows: Section 2 introduces a distance measurement of social closeness of people in a photo collection and the resulted distance matrix; In Section 3, we provide an overview of the BOSC framework and in particular, its application on the distance matrix; Experiment results are presented in Section 4. Section 5 concludes a paper with a short discussion.
2 Social Closeness Denote all the photos in the collections as set I = {I1, I 2 ,L, I M }, and all the people that appeared in the photos as set P = {P1, P2 ,L, PN }. We identify people in the photo1 collection through two channels: 1) automatic face detection and clustering; 2) manual face tagging. If a person’s face was manually tagged, the face detection and clustering will be skipped to respect the subjective indication. Otherwise, the algorithms proposed in 0 are applied to produce people identifiers. The end result of this identification process is a list of rectangles {R(Pi, Ij)} for any Pi ∈ P that is found in Ij∈ I. For any two people Pi and Pj , we define a function to indicate the social closeness of the pair that is formed by taking the following heuristics into account: 1) if Pi ’ and Pj ’s face locations are close/closer in photos, they are also close/closer in relationship; 2) the more faces are found in a photo, the less trustworthy the face location
Social Community Detection from Photo Collections Using BOSC
59
distance is; 3) the more co-appearances of Pi and Pj, the more trustworthy the face location distance is. These assumptions are captured in the following formula: d ( Pi , Pj ) = [
1 q ∑ (d Il ( Pi , Pj ) ∗ f Il − 1)] q l =1
(1)
where Il , l = 1,..., q , are all the photos that contain both Pi and Pj . fIl is the number of faces detected in photo Il . dIl ( Pi , Pj ) is the distance of faces of Pi and Pj captured in photo Il .Using the above distance measure, we derive a distance matrix X of dimension N × N, with each entry xuv = d ( Pu , Pv ) = xvu .
3 Community Detection Using BOSC In this Section, we first give an overview of the Bayesian Overlapping Clustering (BOSC) model and algorithm. We then modify the BOSC algorithm to handle distance matrices and detect overlapping communities. 3.1 BOSC Overview Given a data matrix, the BOSC algorithm aims to find potentially overlapping dense sub-blocks and noise entries. Suppose the data matrix X has m rows, n columns and k sub-blocks. The BOSC model assumes that each sub-block is modeled using a parametric distribution p j [ j]1k ( [ j ]1k j 1, ... k ) from any suitable exponential family. The noise entries are modeled using another distribution p( θ j 1 ) from the same family. The main idea behind the BOSC model is as follows: Each row u and each column u v v respectively have k -dimensional latent bit vectors zr and zc which indicate their sub-block memberships. The sub-block membership for any entry xuv in the matrix is obtained by an element-wise (Hadamard) product of the corresponding row and u v column bit vectors, i.e., z zr . zc . Given the sub-block membership z and the sub-block distributions, the actual observation xuv is assumed to be generated by a multiplicative mixture model so that k ⎧ 1 z p j ( xuv θ j ) j if z ≠ 0 ⎪ j =1 p( xuv zru , zcv , θ } = ⎨ c( z ) ⎪ p( x θ ) otherwise uv k +1 ⎩
∏
u
v
(2)
where c( z ) is a normalization factor to guarantee that p(• zr , zc ,θ } is a valid distribuu v tion. If z = zr . zc = 0 , the all zeros vector, then xuv is assumed to be generated from the nose component p(• θ k +1 ) . The Hadamard product ensures that the matrix has uniform/dense sub-blocks with possible overlap while treating certain rows/columns as noise. Since it can be tricky to work directly with the latent bit vectors, the BOSC model places suitable Bayesian priors on the sub-block memberships. In particular, it asj j k sumes that there are k Beta distributions Beta(α r , β r ) , [ j ]1 corresponding to the rows
60
P. Wu, Q. Fu, and F. Tang j
j
k
u, j
and k Beta distributions Beta (α c , βc ) , [ j ]1 corresponding to the columns. Let π r dej j note the Bernoulli parameter sampled from Beta(α r , β r ) for row u and sub-block j m k v, j where [u]1 and [ j ]1 . Similarly, let π c denote the Bernoulli parameter sampled from j j Beta (α c , βc ) for row v and sub-block j where [v]1n and [ j]1k . The Beta-Bernoulli distributions are assumed to be the priors for the latent row and column membership vecu v tors zr and zc . In particular, the generative process is shown in [2]. Let Zr and Zc be m × k and n × k binary matrices that have the latent row and column sub-blocks assignments for each row and column. Given the matrix X , the learning task is to infer the joint posterior distribution of ( Zr , Zc ) and compute the model parameters (α r∗, β r∗ ,α c∗ , βc∗ ,θ ∗ ) that maximize log p( X α r , βr , α c , β c ,θ ) . We can then draw samples from the posterior distribution and compute the dense-block assignment for each entry. The BOSC algorithm is an EM-like algorithm. In E-step, given the model parameters (α r , β r , α c , βc ,θ ), the goal is to estimate the expectation of the loglikelihood E[log p ( X α r , βr ,αc , β c , θ )] where the expectation is with respect to the posterior probability p( Z r , Z c | X ,α r , β r , α c , βc ,θ ) . The BOSC algorithm uses Gibbs sampling to approximate the expectation. Specifically, it computes the conditional probabilities u, j v, j of each row (column) variable zr ( zc ) and constructs a Markov chain based on the conditional probabilities. On convergence, the chain will draw samples from the posterior joint distribution of ( Zr , Zc ) , which in turn can be used to get an approximate estimate of the expected log-likelihood. In M-Step, the BOSC algorithm estimates (α r∗ , β r∗ , α c∗ , βc∗ ,θ ∗ ) that maximizes the expectation.
Fig. 1. BOSC generative process
Social Community Detection from Photo Collections Using BOSC
61
Fig. 2. BOSC generative process for distance matrix
3.2 BOSC for Distance Matrix The BOSC framework deals with general matrices. However, distance matrices are symmetric and the rows and columns represent the same set of users in the social network, which implies the row and column cluster assignments have to be identical, i.e., Zr = Zc . Suppose there are N individuals in the social network and the distance matrix is X . We can slightly modify the OSC generative process to support the symmetry nature of the distance matrix as shown in [2]. Note that because the matrices are symmetric, we only need k Beta distributions Beta(α j , β j ) , [ j ]1k for the individuals in the social network. The learning task now becomes inferring the joint posterior distribution of Z and computing the model parameters (α ∗ , β ∗ ,θ ∗ ) that maximizes log p ( X α , β ,θ ). The EM-like process is similar to the description in Section 0.
4 Experiment Results The image set used in the experiment contains 4 photo collections and each corresponding to a distinct social event. The 4 social events are “Birthday party” (78 photos), “Easter party” (37 photos), “Indiana gathering” (38 photos) and “Tennessee gathering” (106 photos). There are 54 individuals captured in those photos. The core people that establish linkages among these events are a couple (Dan and Oanh) and their son (Blake). The “Birthday party” is held for the son, and attended by his kid friends, their accompanying parents, members from Oanh’s family and the couple. The “Easter party” is attended by members of the wife’s family, the couple and the son. Both the “Indiana gathering” (38 photos) and “Tennessee gathering” are attended by the members of the husband’s family, in addition to the couple and the son. Table 1 lists the number of photos in each collection and number of individuals appeared in each collection.
62
P. Wu, Q. Fu, and F. Tang
The clustering results using the graph partition based method 0 can be found in Table 2. Table 3 shows the clustering results from using BOSC. In BOSC implementation, we set k = 4. Several observations emerge from comparing the results in Table 2 and Table 3: • Handling of the noise date: The most noticeable difference is that Cluster 4 in Table 32 is no more a cluster in Table 3. In fact, the two individuals “adult 2” and “adult 3” do not appear in any of the clusters in Table 3. Examine the photo collection shows that “adult 2” and “adult 3” co-appears in just one photo and only with each other. Apparently, excluding such individuals in the community construction is more sensible than treating them as singular social cluster. In addition, a few other individuals, such as “adult 8”, who appears in just one photo with one active member is the “Birthday party” collection, are also excluded in the community formation in Table 3. This difference is due to BOSC’s capability of treating some data entry as noise data, which allows the community detection less interfered by noise. • Overlapping cluster membership: Another key observation is that the clusters are overlapping. As seen in Table 3, Dan appears in all 4 clusters, Blake in 3 clusters and Oanh in 2 clusters. Such result reflects the pivot role played by the family in associating different communities around them. This insight is revealed by BOSC’s overlapping clustering capability, and not achievable through the graph partition approach. • Modeling capability and limitation: As shown in Table 3, Cluster 1 and 3 both consist of people who attended “Indiana gathering” and “Tennessee gathering”. And Cluster 3 is actually a subset of Cluster 1. Cluster 2 and 4 consist of most of people attended the “Birthday party” and the “Easter party” and Cluster 2 is almost a subset of Cluster 4. One can clearly see two camps of social clusters, one is Dan’s family members and the other is Blake’s friends and Oanh’s family members. Considering the photos of “Birthday party” are twice as much as that of “Easter party” and “Birthday party” is attended by both Blake’s friends and Oanh’s family members, such partition is reasonable. Another factor that may contribute to the less distinction of Blake’s friends from Oanh’s family members is the EM-like algorithm’s convergence to local maximum, which can be remedied algorithmically. Overall, the experiment results validate the merits of the BOSC algorithm in treating the noise data and support overlapping membership. Table 1. Photo collections and people appeared in each collection Collections
#. Photo
Birthday party
78
Easter party Indiana gathering Tennessee gathering
37 38 106
People adult 1, adult 4, adult 5, adult 6, adult 7, adult 8, adult 9, adult 10, Alec, Anh, Blake, Blake’s friend 1, Blake’s friend 2, Blake’s friend 3, Blake’s friend 4, Blake’s friend 5, Blake’s friend 6, Blake’s friend 7, Blake’s friend 8, Blake’s friend 9, Dan, Jo, Landon, Mel, Nicolas, Oanh Alec, Anh, Anthony, Bill, Blake, Calista, Dan, Jo, Landon, Mel, Nicolas, Nini, Oanh, adult 4, adult 2, adult 3 Allie, Amanda, Blake, Dad, Dan, Hannelore, Jennifer, Lauren, Mom, Rachel, Stan, Tom, Tracy Allie, Amanda, Blake, Bret, Cindy, Dad, Dan, Grace, Hannelore, Jennifer, Jillian, Katherine, Kevin, Kurt, Lauren, Mom, Oanh, Tracy, Phil, Rachel, Reid, Rich, Sandra, Stan, Tom
Social Community Detection from Photo Collections Using BOSC
63
Table 2. Detected social clusters Cluster Cluster 1 Cluster 2
People Stan, Lauren, Dad, Reid, Amanda, Bret, Kevin, Hannelore, Cindy, Jennifer, Tom, Rachel, Allie, Sandra, Jillian, Grace, Oanh, Phil, Tracy, Kurt, Dan, Rich, Katharine, Mom adult 10, Blake's friend 2, Blake's friend 1, Blake's friend 5, adult 6, adult 9, Nicolas, Blake's friend 3, Blake's friend 4, Blake's friend 6, Blake's friend 7, Alec, Blake's friend 9, Blake's friend 8, adult 1, adult 5, Blake
Cluster 3
Jo, Landon, Anthony, Anh, Bill, adult 4, adult 7, Nini, Calista, Mel, adult 8
Cluster 4
adult 3, adult 2
Table 3. Detected social clusters using BOSC Cluster Cluster 1 Cluster 2 Cluster 3
Cluster 4
People Lauren, Reid, Amanda, Bret, Kevin, Cindy, Tom, Rachel, Sandra, Oanh, Jillian, Phil, Kurt, Dan, Rich, Blake, Mom, Katherine, Dad, Stan, Hannelore, Jennifer, Allie, Grace, Tracy adult 10, adult 9, Dan, Blake, Blake's friend 2, Blake's friend 1, Blake's friend 4, Blake's friend 6, Blake's friend 8 Lauren, Reid, Amanda, Bret, Cindy, Tom, Rachel, Jillian, Phil, Kurt, Dan, Mom, Dad, Jennifer, Allie, Grace Adult 10, Landon, adult 7, Mel, Oanh, Anh, adult 1, Calista, Dan, Blake, Anthony, Jo, Bill, Nini, Nicolas, Alec, Blake’s friend 1, Blake’s friend 2, Blake’s friend 5, Blake’s friend 6, Blake’s friend 3, Blake’s friend 4, Blake’s friend 9, Blake’s friend 7
5 Conclusions In this work, we first described a metric to measure people’s social distance by examining their co-appearances in photo collections. Then a subspace clustering algorithm is applied to the social distance matrix of people to detect the social communities embedded in the photo collections. The experiment results illustrate meaningful social clusters within photo collections can be revealed by the proposed approach effectively.
References 1. Golder, S.: Measuring social networks with digital photograph collections. In: Proceedings of the Nineteenth ACM Conference on Hypertext and Hypermedia (June 2008) 2. Gu, L., Zhang, T., Ding, X.: Clustering consumer photos based on face recognition. In: Proc. of IEEE International Conference on Multimedia and Expo, Beijing, pp. 1998–2001 (July 2007) 3. Newman, M.E.J.: Finding community structure in networks using the eigenvectors of matrices. Phys. Rev. E 74, 036104 (2006)
64
P. Wu, Q. Fu, and F. Tang
4. Rowe, R., Creamer, G., Hershkop, S., Stolfo, S.J.: Automated social hierarchy detection through email network analysis. In: Proceedings of the 9th WebKDD and 1st SNA-KDD 2007 Workshop on Web Mining and Social Network Analysis (August 2007) 5. Stone, Z., Zickler, T., Darrell, T.: Autotagging Facebook: Social network context improves photo annotation. In: Computer Vision and Pattern Recognition Workshops (June 2008) 6. Wu, P., Tretter, D.: Close & closer: social cluster and closeness from photo collections. In: ACM Multimedia 2009, pp. 709–712 (2009) 7. Fu, Q., Banerjee, A.: Bayesian Overlapping Subspace Clustering. In: ICDM 2009, pp. 776– 781 (2009)
Dynamic Estimation of Family Relations from Photos Tong Zhang, Hui Chao, and Dan Tretter Hewlett-Packard Labs 1501 Page Mill Road, Palo Alto, CA 94304, USA {tong.zhang,hui.chao,dan.tretter}@hp.com
Abstract. In this paper, we present an approach to estimate dynamic relations among major characters in family photo collections. This is a fully automated procedure which first identifies major characters in photo collections through face clustering. Then, based on demographic estimation, facial similarity, coappearance information and people's positions in photos, social relations of people such as husband/wife, parents/kids, siblings, relatives and friends can be derived. A workflow is proposed to integrate the information from different aspects in order to give the optimal result. Especially, based on timestamps of photos, dynamic relation trees are displayed to show the evolution of people's relations over time. Keywords: consumer photo collections, family relation estimation, dynamic relation tree, face analysis, face clustering.
1 Introduction We can often figure out people’s relations after looking at somebody’s family photo collections for a while. Then, what about letting the computer automatically identify important people and their relations, and even form a family relation tree like the one shown in Fig.1, through analyzing photos in a family collection? Such a technology would have a wide range of applications. First of all, major people involved in images and their relations form important semantic information which is undoubtedly useful in browsing and searching images. For example, the information may be used in generating automatic recommendations of pictures for making photo products such as photo albums and calendars. Secondly, it may have other social and business values and broader uses in social networking, personalized advertisement and even homeland security. Moreover, a dynamic relation tree that shows the evolution of family relations over a relatively long period of time (e.g. multiple years) provides one kind of life log which provides perspectives of people, events and activities within glances. Existing work on discovering people’s relations based on image content analysis is quite rare. Golder presented a preliminary investigation on measuring social closeness in consumer photos, where a weighted graph was formed with people co-appearance information [1]. A similar approach was taken in [2], but a graph clustering algorithm was employed to detect social clusters embedded in photo collections. Comparing with such prior work, our proposed approach aims at integrating information from K.-T. Lee et al. (Eds.): MMM 2011, Part II, LNCS 6524, pp. 65–76, 2011. © Springer-Verlag Berlin Heidelberg 2011
66
T. Zhang, H. Chao, and D. Tretter
wife
husband
grandma grandpa
grandma grandpa parents
parents Extended family
Extended family
daughter
Nuclear family
siblings
siblings brother
niece
sister-in-law
sister
niece nephew
brother
sister-in-law
nephew nephew
brother
niece
brother
niece
couple
Relatives
Friends
Fig. 1. One example of family relation tree automatically estimated based on face analysis
multiple aspects of image analyses and revealing more details of social relationships among the people involved in an image collection.
2 Estimation of People’s Relations 2.1 What Can Be Obtained with Image Analysis With results from previously developed image analysis techniques including face recognition and clustering, demographic estimation and face similarity measurement, as well as contextual information such as co-appearance of people in photos, people’s relative positions in photos and photo timestamps, the following clues can be obtained to discover people’s relations. •
Major characters in a family photo collection
Based on state-of-the-art face recognition technology, we developed a face clustering algorithm in earlier work which automatically divides a photo collection into a number of clusters, with each cluster containing photos of one particular person [3]. As the result, major clusters (that is, those having a relatively large number of photos) corresponding to frequently appearing people may be deemed as containing main characters of the collection. Shown in Fig.2 are major clusters obtained from one consumer image collection by applying face clustering. Each cluster is represented by one face image in the cluster, called a face bubble; and the size of the bubble is proportional to the size of the cluster. In such a figure, it is straightforward to identify people who appear frequently in the photo collection.
Dynamic Estimation of Family Relations from Photos
67
Fig. 2. Main characters identified in a photo collection through face clustering
•
Age group and gender of major characters
We applied learning based algorithms developed in prior work to estimate the gender and age of each main character in a photo collection [4] [5]. In age estimation, a person is categorized into one of five groups: baby (0-1), child (1-10), youth (10-18), adult (18-60) and senior (>60). Even though each classifier only has accuracy rate around 90% for an individual face image, it can be much more reliable when applied to a cluster with big enough number of face images. In Table 1, it shows the number of faces classified into each gender or age group within each face cluster in a photo set, where M, F indicate male and female, respectively; B, C, A and S indicate baby, child, adult and senior, respectively. Since the photos span a period of several years, some subjects may belong to multiple age groups such as B/C and A/S. The numbers in red indicate the gender/age group estimated for the cluster from majority vote, and the numbers in orange indicate the second dominant age group if available. As can be seen, while there are mistakes in estimating gender and age for faces in each cluster; cluster wise, all the classification results match with the ground truth correctly. •
Who look similar to each other
Since each face cluster contains images of one person, the similarity measure between two clusters then indicates how similar the two people look like. In our work, each cluster is represented by all of its member faces with which every face in another
68
T. Zhang, H. Chao, and D. Tretter
cluster is compared, and the distance of the two clusters is determined by the average distance of the K nearest neighbors. It has been found out that clusters of adults with blood relations such as parents/children and siblings normally have high similarity measures with each other. Table 1. Estimation of gender and age group of main characters in a photo collection Face Clusters
Ground Truth
Detected Face
Gender Estimation M
No.1 No.2 No.3 No.4 No.5 No.6 No.7 No.8 No.9 No.10 No.11 No.12 No.13 No.14
•
F, B/C F, B/C F, A M, A M, C F, A M, S M, S F, S M, A/S F, A F, S M, A M, C
436 266 247 215 86 72 65 62 57 40 33 30 18 17
178 109 37 171 46 13 49 41 17 22 2 4 16 13
Age Estimation
F
B
C
A
S
258 157 210 44 40 59 16 21 40 18 31 26 2 4
94 88 2 4 12 1 1 2 0 1 0 0 0 1
274 162 30 44 71 1 7 1 6 4 0 4 4 13
61 14 186 115 2 59 11 3 11 8 33 4 12 3
7 2 29 52 1 11 46 56 40 27 0 22 2 0
Who are close with each other (appear in the same photo, and how often)
Whether and how often two people appear together in photos reveal how close they are with each other. A co-appearance matrix can be obtained for major clusters in a photo collection containing the number of co-occurrences between people. It provides compensating information to the similarity matrix of clusters introduced above. As shown in Fig.3, for one person, people who look most like him (e.g. siblings, parents) and people who are most close with him (e.g. wife, kids) are listed separately. •
Who are in the same social circle (appear in the same event)
Applying clustering to the co-appearance matrix and using photo timestamps, we can find out groups of people who appear together in the same event, and thus figure out social circles in a photo dataset. Particularly, people who appear in a series of group photos taken in one event (e.g. family reunion, company outing, alumni reunion) may be recognized as belonging to one circle of relatives, colleagues or classmates. •
Who are intimate with each other
In a family photo collection, couples, sibling kids and nuclear family members often have exclusive photos of themselves. Besides that, people’s positions in photos also
Dynamic Estimation of Family Relations from Photos
69
provide useful cues regarding intimate relations. For example, couples usually stand or sit next to each other in group photos. Parents/grandparents tend to hold the family kids in photos. Touching-faces position (two faces that touch each other) usually only happens between husband and wife, lovers, parents/grandparents and kids, or siblings.
These are people who look like him
These are people who are close with him
Fig. 3. Face clustering result of CMU Gallagher’s dataset [6]. On the left: face clusters in order of cluster size. On the right: first row – clusters most similar to selected cluster in face; second row – clusters having most co-appearances with selected cluster. Lower-right panel: images in selected cluster.
•
Evolution of relations
People’s relations evolve over time. With the help of photo timestamps, for a photo collection spanning a relatively long period of time (e.g. from a few years to dozens of years), the changes in major characters can be detected. Events such as adding or passing away of family members and occasional visits of extended family members, relatives or friends can be discovered. Also, people who appeared at different stages of one’s life may be identified. 2.2 Constructing a Relation Tree We proposed a workflow to identify people’s relations and derive a relation tree from photo collections using the clues introduced above. Members of the nuclear family of the photo owner are first recognized, followed by extended family members, and then other relatives and friends. •
Identifying nuclear family members
From all major clusters, which are defined as clusters containing more than M images (M is empirically determined, e.g. M=4 or 5), the most significant clusters are selected through a GMM procedure in the kid’s clusters (baby, child, junior) and adult clusters
70
T. Zhang, H. Chao, and D. Tretter
Nuclear family
447
274
254
224
92
Candidate nuclear family members Fig. 4. Nuclear family estimation. Left: working sheet (the number under each face is the size of face cluster of that character); Right: identified nuclear family members.
(adult, senior), respectively, in order to find candidates of kids and parents of the nuclear family [7]. Among these candidates, a link is computed between each pair of people based on the number of co-appearances, and with weights added to cases such as exclusive photos, close positions in group photos, face-touching position and babyholding position. An example is shown in Fig.4, where four of the candidates have strong link between each two of them, while the fifth person only has light link with some of the others. Thus, the nuclear family members were identified as in the right. •
Identifying extended family members
Extended family members are defined as parents and siblings of the husband or wife, as well as siblings’ families. These people are identified through facial similarity, coappearances and positions in photos. As shown in Fig.5, major senior characters that have high facial similarity with husband or wife and strong links with nuclear family members were found. Each senior couple can be further recognized as the parents of the husband or the wife according to the link between them and their links with the husband or wife. A sibling of the husband/wife usually has strong link not only with the husband/wife but also with the corresponding parents in terms of facial similarity and co-appearances. He/she also often has co-occurrences with nuclear family kids with intimate positions. A sibling’s family can be identified as those who have strong links with the sibling, as well as co-appearances with nuclear family members and the corresponding parents. •
Identifying relatives and friends
Other major characters are determined as relatives or friends of the family. People are identified as relatives if they co-appear not only with the nuclear family, but also with the extended family members in photos of different events or in a relatively large amount of photos in one event. The rest major characters are determined as friends. In Fig.6, a group of nine people were found to appear together in dozens of photos (including a number of group photos) taken within two events. The nuclear family and the husband’s sister’s family are also in the group photos. Therefore, these people are identified as in one relative circle of the husband’s side.
Dynamic Estimation of Family Relations from Photos
69
58
65
Extended family
30
par
ent s
Nuclear family
par
sib li
ling s ib
74
43
71
ent
s
ng
18
Fig. 5. Extended family estimation. Left upper: identifying parents of husband/wife; Left lower: identifying a sibling of the husband and her family. Right: estimated extended family. Extended family par ent s
Nuclear family ling sib
e n ts par
sib
ling
Relatives or friends
Fig. 6. Identifying circles of relatives and friends
3 Estimation of the Evolution in People’s Relations 3.1 Dynamic Relation Tree – One Kind of Life Log With people’s relations identified in the above described process, a dynamic relation tree is built which places the people in their corresponding positions in the relation circles, and the view of the tree changes when different time periods are selected.
72
T. Zhang, H. Chao, and D. Tretter
Fig. 7. Snapshots of a dynamic relation tree which was constructed from a family photo collection. This collection contains photos spanning seven years. One view of the tree was generated for each of the years from 2001 to 2007.
Dynamic Estimation of Family Relations from Photos
73
That is, while people and their relations are identified by analyzing all the images in the photo collection, only those people who appear in the specified period of time (e.g. a certain year or a certain event) are shown in one view of the dynamic tree. It thus reveals the evolution of relations over time, and provides one kind of life log with glance views that reminds one of events and people involved in his past. Fig.7 contains snapshots of a dynamic relation tree derived from a family photo collection containing photos spanning 7 years. One view is included for each of the years 2001~2007, showing people appearing that year with face images taken during that time. Many stories can be told from viewing these snapshots. For example, it was the end of 2001 that the family had their first digital camera and only a few of them appeared. Grandparents on both sides got together with the nuclear family every year except that the wife’s parents did not make it in 2006. The family went to see greatgrandmother in 2003. In that same year, the husband’s sister came to visit them. We can see the daughter’s favorite teacher in 2004. Two neighbor families appear in almost every year and the kids have grown up together. If we zoom into specific events in certain years, more details may be discovered such as parents of the wife’s side were together with the family during Thanksgiving in 2004, while parents of the husband’s side spent Christmas with them that year. From these snapshots, we can also see how the looks of the kids have changed when they grew from pre-schoolers to early teenagers. Comparing with a static tree showing who are the major characters and how they connect with each other [7], a dynamic tree presents additional details about the activities and changes in the people and their relations.
Fig. 8. Determining a kid’s age based on photo timestamps and age group classification
74
T. Zhang, H. Chao, and D. Tretter
3.2 Discovery of Current Status and Changes in the Family Circle With the help of photo timestamps and estimation of people’s relations, many events in the family circle can be discovered such as the birth of a baby or the passing away of a family member. Here, in particular, we propose a method that predicts the current age of an individual based on images in his/her face cluster over multiple years. One example is shown in Fig.8. The age group classifier is applied to each face image in a person’s cluster. The images are then divided into one-month periods, and within each period, the percentage of images classified to each age group is computed as plotted in the figure. Next, for each age class, a polynomial function is estimated to fit the data. The polynomial curves of the baby class and the child class over 4 years are shown. The crossing points of the polynomial functions for any two neighboring classes are found, and if one crossing point has value larger than a threshold (e.g. 0.35), it is considered to be a significant transitional point that indicates the time the person transits from one age group to the next. In this example, the transitional time from baby to child class is around June 2008; thus the person was estimated to be around one year old at that time and his/her current age can be estimated accordingly.
4 Experiments and Discussions We tested the proposed approach on three typical family photo collections. Each one spans between 7-9 years, and contains photos of a large number of major people, including nuclear family members (husband, wife and kids), extended families on both sides, other relatives and friends. Using our approach described in this paper, the estimated relations match with the ground truth quite accurately except for a few extreme cases. For example, the final Extended family par e
nts
par
s ent
Nuclear family si
Relatives or friends
g blin
sib ling
Relatives or friends
Fig. 9. Relation tree automatically derived from a family photo collection
Dynamic Estimation of Family Relations from Photos
75
relation tree derived from the collection introduced in section 2.2 is shown in Figure 9. In this tree, one person indentified as a “family relative or friend” is actually the portrait of Chairman Mao in Beijing’s TianAnMen Square. This false alarm happened because the family took quite some photos there at different times. Another issue is that the estimated gender is wrong for three people in this collection, showing that our gender estimation may not be reliable for young kids and seniors. Furthermore, the concept of dynamic tree helps to resolve some confusing cases in a static tree. Still in this collection, one senior man is close with two senior women (one is his late wife, another is his current wife), and both women have strong links with the nuclear family, which makes a difficult case in a static tree. However, as the two women’s appearances do not have any overlap in time, it is straightforward to place them in the dynamic tree. There are also cases that need extra rules to guide to the right result. For instance, in one collection, an adult male has a number of co-appearances with the wife’s sister including two exclusive ones; however, he is also in photos with the husband’s family plus the person and the husband are highest on each other’s facial similarity ranking. We had to add the rule that blood relation estimation has higher priority than that of the significant-other relation, and thus assign the person as a sibling of the husband. We believe that with experiments on more family photo collections, there will be new cases appearing all the time that need rules to be added or expanded to accommodate all different variations of relations.
5 Conclusions and Future Work We presented an approach to automatically figure out main characters and their relations in a photo set based on face analysis technologies and image contextual information. On top of this, a dynamic relation tree is built in which the appearing people and their looks change over time to reveal the evolutions and events in the family’s life over multiple years. Experiments have shown that with existing techniques, quite accurate results can be obtained on typical family photo collections. Only preliminary work has been done with this approach. In the following, more family image datasets will be collected and tested on to make the rule-based system more robust. More learning elements will be added into the workflow to replace hard coded rules so that the system can adapt to different relation cases by itself. We will also investigate use cases of the approach and produce more useful relations trees.
References 1. Golder, S.: Measuring Social Networks with Digital Photo-graph Collections. In: 19th ACM Conference on Hypertext and Hypermedia, pp. 43–47 (June 2008) 2. Wu, P., Tretter, D.: Close & Closer: Social Cluster and Closeness from Photo Collections. In: ACM Conf. on Multimedia, Beijing, pp. 709–712 (October 2009) 3. Zhang, T., Xiao, J., Wen, D., Ding, X.: Face Based Image Navigation and Search. In: ACM Conf. on Multimedia, Beijing, pp. 597–600 (October 2009)
76
T. Zhang, H. Chao, and D. Tretter
4. Gao, W., Ai, H.: Face Gender Classification on Consumer Images in a Multiethnic Environment. In: The 3rd IAPR Conf. on Biometrics, Univ. of Sassari, Italy, June 2-5 (2009) 5. Gao, F., Ai, H.: Face Age Classification on Consumer Images with Gabor Feature and Fuzzy LDA Method. In: The 3rd IAPR International Conference on Biometrics, Univ. of Sassari, Italy, June 2-5 (2009) 6. Gallagher, A.C., Chen, T.: Using Context to Recognize People in Consumer Images. IPSJ Trans. on Computer Vision and Applications 1, 115–126 (2009) 7. Zhang, T., Chao, H., et al.: Consumer Image Retrieval by Estimating Relation Tree from Family Photo Collections. In: ACM Conf. on Image and Video Retrieval, Xi’an, China, pp. 143–150 (July 2010)
Semi-automatic Flickr Group Suggestion Junjie Cai1 , Zheng-Jun Zha2 , Qi Tian3 , and Zengfu Wang1 1
University of Science and Technology of China, Hefei, Anhui, 230027, China 2 National University of Singapore, Singapore, 639798 3 University of Texas at San Antonio, USA, TX 78249
[email protected],
[email protected],
[email protected],
[email protected]
Abstract. Flickr groups are self-organized communities to share photos and conversations with common interest and have gained massive popularity. Users in Flickr have to manually assign each image to the appropriated group. Manual assignment requires users to be familiar with existing images in each group and it is intractable and tedious. Therefore it prohibits users from exploiting the relevant groups. For solution to the problem, group suggestion has attracted increasing attention recently, which aims to suggest groups to user for a specific image. Existing works pose group suggestion as the automatic group prediction problem with a purpose of predicting the groups of each image automatically. Despite of dramatic progress in automatic group prediction, the prediction results are still not accurate enough. In this paper, we propose a semiautomatic group suggestion approach with Human-in-the-Loop. Given a user’s image collection, we employ the pre-built group classifiers to predict the group of each image. These predictions are used as the initial group suggestions. We then select a small number of representative images from user’s collection and ask user to assign the groups of them. Once obtaining user’s feedbacks on the representative images, we infer the groups of remaining images through group propagation over multiple sparse graphs among the images. We conduct experiment on 15 Flickr groups with 127,500 images. The experimental results demonstrate the proposed framework is able to provide accurate group suggestions with quite a small amount of user effort. Keywords: Flickr Group, Semi-automatic, Group Suggestion.
1
Introduction
In Web 2.0 era, social networking is a popular way for people to connect, express themselves, and share interests. Popular social networking websites include MySpace1 , Facebook2 , LinkedIn3 , and Orkut4 for finding and organizing 1 2 3 4
http://www.myspace.com/ http://www.facebook.com/ http://www.linkedin.com/ http://www.orkut.com/
K.-T. Lee et al. (Eds.): MMM 2011, Part II, LNCS 6524, pp. 77–87, 2011. c Springer-Verlag Berlin Heidelberg 2011
78
J. Cai et al.
Fig. 1. Home page of the group ”cars”
contacts, LiveJournal5 and BlogSpot6 for sharing blogs, Flickr7 and Youtube8 for sharing images or videos, and so on. One important social connection feature on these Websites is Group, which refers to the self-organized community with declared, common interest. In particular, groups in Flickr are the communities where users gather to share their common interests on certain type of events or topics captured by photos. For example, the Flickr group of “cars” is illustrated in Fig.1. Most of the activities within a Flickr group start from sharing images: user would contribute his/her own images to the related group, comment on other members’ images, and discuss related photographic techniques[1][3]. Flickr now obligates user to manually assign each image to the appropriated group. Manual assignment requires users to be familiar with existing images in each group and match the subject of each image in user’s collection with the topics of various groups. This work is intractable, tedious, and thus prohibits user from exploiting the relevant groups[1][2]. To tackle this problem, group suggestion has been proposed recently, which aims to suggest appropriate groups to user for a specific image. Existing works pose group suggestion as an automatic group prediction problem with a purpose of predicting the groups of each image automatically. For example, Perez and 5 6 7 8
http://www.livejournal.com/ http://googleblog.blogspot.com/ http://www.flickr.com/ http://www.youtube.com/
Semi-automatic Flickr Group Suggestion
79
« ᱑tree᱒ ᱑baby᱒ ᱑animal᱒ groups
animal baby baby
animal baby baby baby
baby baby baby
baby baby baby
baby baby baby
prediction: animal red ᱑tree᱒ «« User Image Collection
baby baby
᱑animal᱒
red user input: baby
«« ᱑baby᱒
᱑sky᱒
Groups Classifiers based on Multiple Kernel Learning
animal baby Group Prediction
««
«« animal baby baby Representative Image Selection and User Labeling
Group Inference
baby baby Group Suggestion
Fig. 2. Flowchart of our approach. The green rectangles indicate final group suggestions corrected by our method.
Negoescu analyzed the relationship between image tags and the tags in groups to automatically suggest groups to users[6]. In addition to tags used in[6], visual content was also utilized to predict the groups of images. Duan et al. presented a framework that integrates a PLSA-based image annotation model with a style model to provide users with groups for their images[4]. Chen et al. developed a system named SheepDog to automatically add images into appropriate groups by matching the semantic of images with the groups, where image semantic is predicted through image classification technique[8]. Recently, as reported in [1][3], Yu et al. converted group suggestion to group classification problem. They integrated both visual content and textual annotations(i.e. tags) to predict the events or topics of the images. Specifically, they firstly trained a SVM classifier to predict the group of each image and then refined the predictions through sparse graph-based group propagation over images within a same user collection. Although encouraging advances have been achieved in automatic group suggestion, the suggestion result is still not accurate enough. Motivated by above observations, in this paper, we propose a semi-automatic group suggestion approach with Human-in-the-Loop. Fig.2 illustrates the flowchart of our approach. Given a user’s image collection, we employ the prebuilt group classifiers to predict the group of each image. These group predictions are used as the initial suggestions for the user. As aforementioned, the automatic group prediction is not accurate enough. We thus introduce user in the loop [15] and conduct the following two steps repeatedly: (a) sample images are selected from user’s collection and the corresponding suggestions from group classifiers are then presented to user to amend the wrong ones, and (b) the groups of remaining images are inferred based on user’s feedbacks and group predictions in last round. From technical perspective, the semi-automatic group suggestion framework contains three components: (a) group classifier building, (b) representative image selection, and (c) group inference. Specifically, we employ Multiple Kernel Learning(MKL) technique [9] to build a SVM classifier for each group with one-vs-all strategy. To select representative images, we incorporate sample uncertainty [12] into Affinity Propagation algorithm and select the images with high uncertainty and representativeness. After obtaining user’s feedbacks
80
J. Cai et al.
on the selected images, we infer the groups of remaining images through group propagation over multiple sparse graphs of the images. We conduct experiment on 15 Flickr groups with 127,500 images. The experimental results demonstrate the proposed framework is able to provide accurate group suggestions with quite a small amount of user effort. The rest of this paper is organized as follows. The proposed semi-automatic group suggestion framework is elaborated in Section 2. The experimental results are reported in Section 3 followed by the conclusions in Section 4.
2 2.1
Approach Image Features
Three popular visual descriptors, i.e., GIST, CEDD and Color Histogram, are extracted to represent image content[1]. GIST. GIST descriptor [5]has recently received increasing attention in the context of scene recognition and classification tasks. To compute the color GIST description, the image is segmented by a 4 by 4 grids for which orientation histograms are extracted. Our implementation takes as input a square image of fixed size and produces a 960-dimensional feature vector. CEDD. Color and Edge Directivity Descriptor(CEDD)[10] is a new low level feature which incorporates color and texture information into a histogram. CEDD size is limited to 54 bytes per image, rendering this descriptor suitable for use in large image databases. A 144-dimensional CEDD feature vector is extracted for each image. Color Histogram. Color histogram is a widely used visual feature. We extract a 512 RGB color histogram with dividing the RGB color space into 8*8*8 bins. Finally, a 512-dimensional color feature vector is extracted for each image. 2.2
Group Classifier Building
The curse of dimensionality has always been a critical problem for many machine learning tasks. Directly concatenating different types of visual features into a long vector may lead to poor statistical modeling and high computational cost. In order to bypass this problem, we employ Multiple Kernel Learning (MKL)method[9] to build a SVM classifier for each group. Denoting the kernel similarity between sample x and y over j-th feature by Kj (x, y), we combine multiple kernels as a convex combination as: K(x, y) =
K j=1
βj Kj (x, y), with
βj ≥ 0,
K
βj = 1.
(1)
j=1
The kernel combination weights βj and the parameters of the SVM can be jointly learned by solving a convex, but non-smooth objective function. We here follow the implementation at http://asi.insa-rouen.fr/enseignants/ arakotom/.
Semi-automatic Flickr Group Suggestion
2.3
81
Representative Image Selection
We use the Affinity Propagation (AP) method to identify small number of images that accurately represent user’s image collection[11][14]. Different from the typical AP, sample uncertainty is incorporated into AP as the prior of images. This modified AP is thus able to select the images with high representativeness as well as high uncertainty. The entropy has been widely used to measure sample uncertainty [12]. The uncertainty in binary classification problem is defined as follows p(y|x) log p(y|x), (2) H(x) = − y∈{0,1}
where p(y|x) represents a distribution of estimated class membership. We simply extend Eq.2 to compute sample uncertainty in multiple group prediction as H(x) = −
k
p(yi |x) log p(yi |x).
(3)
i=1 yi ∈{0,1}
The resulted H(x) is used as the preference of sample x in AP algorithm. AP aims to cluster image set I = {Ii }N i=1 into M (M < N ) clusters based on sample similarity s(Ii , Ij ). Each represented by the most representative image is called “exemplar”. In AP, all the images are considered as potential exemplars. Each of them is regarded as a node in a network. The real-valued message is recursively transmitted via the edges of the network until a good set of exemplars and their corresponding clusters emerge. Let Ie = {Iei }M i=1 denote the final exemplars and e(I) represent the exemplar of image I. In brief, the AP algorithm propagates two kinds of information between images: 1) the “responsibility” r(i, j) transmitted from image i to image j, which measures how well-suited Ij is to serve as the exemplar for Ii by simultaneously considering other potential exemplar for Ii , and 2) the “availability” a(i, j) sent from candidate exemplar Ij to Ii , which reflects how appropriately Ii chooses Ij as exemplar by simultaneously considering other potential images that may choose Ij as their exemplar. This information is iteratively updated by r(i, j) ← s(Ii , Ij ) − max {a(i, j ) + s(Ii , Ij )}, j=j a(i, j) ← min{0, r(j, j)} + max{0, r(i , j)}
(4)
i i,j
The “self-availability” a(j, j) is updated by a(j, j) := i =j max{0, r(i , j)}. The above information is iteratively propagated until convergence. Then, the exemplar e(Ii ) of image Ii is chosen as e(Ii ) = Ij by solving arg maxj {r(i, j)+a(i, j)}. 2.4
Group Inference
After obtaining user’s feedbacks on selected images, our task is to infer the groups of remaining images. It is reasonable to assume that many images in a user image
82
J. Cai et al.
collection are usually similar and the similar images should be assigned to same group. Therefore, the groups of remaining images can be inferred by propagating user’s feedbacks to these images. Let I = {I1 , . . . , Il , Il+1 , . . . , IN } denote images in certain user’s collection containing l labeled samples and N − l unlabeled samples. Xg = {xg,1 , . . . , xg,l , xg,l+1 , . . . , xg,N } denotes the feature vectors on g-th modality, where xg,i ∈ Rdg represents the i-th sample. Here, we infer the groups of {Il+1 , . . . , IN } resorting to group propagation over multiple sparse graphs among images I. Let Gg = {I, Wg } denote the sparse graph on g-th modality. Wg is the affinity matrix, in which Wg,ij indicates the affinity between sample i and j. Wg can be obtained by solving the following optimization problem[13]. Wg = arg min Wg 1 ,
s.t. xg,i = Ag,i Wg ,
(5)
where the matrix Ag,i = [xg,1 , . . . , xg,i−1 , xg,i+1 , . . . , xg,N ]. Afterwards, we conduct group propagation over K sparse graphs {Gg }K g=1 . Wang et al.[7] have proposed an optimized multi-graph-based label propagation algorithm, which is able to integrate multiple complementary graphs into a regularization framework. The typical graph-based propagation framework estimates class labels (i.e.groups in our case) of images over the graphs such that they satisfy two properties: (a) they should be close to the given labels on the labeled samples, and (b) they should be smooth on the whole graphs. Here, we extend Wang et al.’s method to further require the estimated groups should be consistent with the initial predictions from our group classifiers. The group inference problem can then be formulated as the following optimization problem. K fi fj f ∗ = arg min{ αg (Wg,ij | − |2 +μ |fi −yi |2 +ν |fi −fi0 |2 )}, f D D g,ii g,jj g=1 i,j i i (6)
where Dg,ii = j Wij , fi is the to-be-learned confidence score of sample i with respect to certain group, fi0 is the initial prediction from the corresponding group classifier, and yi is the user’s feedback on sample i. α = {α1 , α2 , ..., αK } are the K weights which satisfy αg > 0 and g=1 {αg } = 1. The regularization framework consists of three components: the first term is a loss function that corresponds to the first property to penalize the deviation from user’s feedbacks; the second term is a regularizer to address the label smoothness; and the last term is a regularizer to prefer the consistence between the estimated group assignment and initial prediction. If the weights α are fixed, we can derive the optimal f as f = (I +
K K 1 ν 1 μ αg Lg + )−1 + (I + αg Lg + )−1 f0 μ g=1 μ ν g=1 ν
(7)
where Lg = D−1/2 (D − Wg )D1/2 is the normalized graph Laplacian. g g However, the weights α, which reflects the utilities of different graphs, are crucial to the propagation performance. Thus, α should be also optimized automatically to reflect the utility of the multiple graphs. To achieve this, we make a
Semi-automatic Flickr Group Suggestion
83
relaxation by changing αg to ατg , τ > 1[7]. Note that ατg achieves its minimum when αg = 1/K with the constraint αg = 1. We then solve the joint optimization of f and α by using the alternative optimization technique. We iteratively optimize f with fixed α and then optimize α with fixed f until convergence [7]. The solution of f and α can be obtained as ( fT L
αg = K
1
1 g f +μ|f −Y
|2 +ν|f −f0 |2
) r−1 (8)
1 1 r−1 g=1 ( f T Lg f +μ|f −Y |2 +ν|f −f0 |2 )
K K τ τ 1 g=1 αg Lg ν −1 1 g=1 αg Lg μ f = (I + + ) + (I + + )−1 f0 K K μ μ ν ν ατg ατg g=1
3
(9)
g=1
Experiments
3.1
Data and Methodologies
We collect 127,500 user images from 15 groups automatically via Flickr Group9 API. All of groups are related to popular visual concepts: “baby”, “bird”, “animal”, “architecture”, “car”, “flower”, “green”, “music”, “night”, “tree”, “red”, “wedding”, “sky”, “snow” and “sunset.” Each group contributes 8,500 images to our dataset on average. For each group, we assign the users and their images into testing subset if they contribute more than 100 images to this group. As a result, the testing subset contains 203 users with 24,367 images. The remaining 103,133 images are used as training samples. Some sample images of each group are illustrated in Fig. 3.
animal
flower
red
architecture
green
wedding
baby
bird
car
music
night
tree
sky
snow
sunset
Fig. 3. Sample images from our dataset
In the experiment, we utilize a simple but effective linear kernel K(xi , xj ) = xTi xj for SVM classifier. We employ the modified AP algorithm to select representative images from each user’s collection. The visual similarity between two 9
http://www.flickr.com/groups/
84
J. Cai et al. Table 1. The comparison of average accuracy on our dataset Visual Feature Descriptor Average Accuracy Color Histogram 39.0% GIST 43.7% CEDD 51.2% MKL 54.0%
K images is calculated as g=1 exp(−xg,i − xg,j 2 ), where xg,i is the feature vector of image Ii . For each representative image selected by AP, there is a score reflecting the significance of that image. In each round, we choose five representative images which have the highest scores and query the user for the groups. 3.2
Evaluation of Group Classifiers
We firstly evaluate our group classifiers and compare them against SVMs with the three features in section 2.1 respectively. Table 1 shows the comparison of average prediction accuracy, while Fig. 4 illustrates the comparison of accuracy over each class. We can see that our MKL classifiers achieve the best performance among four methods. It achieves around 15.0%, 10.3% and 2.8% improvements as compared to SVMs with Color Histogram, GIST and CEDD visual feature descriptors, respectively. 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Color Histogram
Gist Descriptor
CEDD Descriptor
MKL
Fig. 4. Classification performance comparison on our dataset
3.3
Evaluation of Group Inference
We evaluate the effectiveness of group inference that is achieved through multisparse-graph group propagation based on initial predictions from group classifiers as well as users’ feedbacks on selected images. The weights in Eq.6 are initialized as α1 =α2 =α3 = 1/3. The parameter μ and ν in Eq.7 are set empirically to 50 and 5, respectively.
Semi-automatic Flickr Group Suggestion
85
Prediction Accuracy
0.85 0.8 0.75 0.7 0.65 0.6 0.55 0.5 0
1
2
3
4 5 Iteration
6
7
8
9
10
Fig. 5. Group suggestion performance comparison over ten iterations 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Initial Label Propagation
Multi-Graph Label Propagation (10 times)
Fig. 6. Prediction accuracy comparison on our dataset
Fig. 5 shows the group suggestion performance over ten iterations. We can see that group inference plays an active role in the loop of group suggestion and effectively improves the group prediction accuracy. As compared to the predictions from group classifiers, group inference improves the prediction accuracy by 8% and 24% in the first and last iteration, respectively. In each iteration, it takes only 0.5 seconds to infer the groups of images and select new representative images for the next iteration. Fig. 6 provides the detailed comparison of prediction accuracy over each group. It shows that the average prediction accuracy can be significantly improved from 0.62 in the first iteration to 0.78 in the last iteration. Fig. 7 illustrates the suggested groups for some sample images within three users’ collections. The first image in each collection is the selected representative image. The arrow indicates the change of group suggestions. The group lying at the left side is the initial suggestion from group classifiers, while the group lying at the right side is the one inputted by users for selected images or the final group suggestion generated by group inference. From the above experimental results, we can see that our semi-automatic group suggestion approach can outperform the automatic approach requiring quite a small amount of user effort.
86
J. Cai et al.
animal baby baby
baby
baby animal baby baby
bird
baby baby red baby
««
flower animal animal
animal
animal baby animal car animal
car
animal animal
««
car animal
sky tree tree
night tree tree
tree
car tree
night tree tree night tree
««
Fig. 7. Final group suggestions for sample images. Initial predication results lie on the left of arrow, while suggestion results are located on the right of arrow.
4
Conclusion
In this paper, we have proposed a semi-automatic group suggestion framework with Human-in-the-Loop. The framework contains three components: group classifier building, representative image selection, and group inference. Specifically, we employ the pre-built group classifiers to predict the group of each image in each user’s collection. After obtaining user’s feedbacks on some selected images, we infer the groups of remaining images through group propagation over multiple sparse graphs of the images. The extensive experiments demonstrate our proposed framework is able to provide accurate group suggestions with minimal user’s effort.
References 1. Yu, J., Jin, X., Han, J., Luo, J.: Mining Personal Image Collection for Social Group Suggestion. In: IEEE International Conference on Data Mining Workshops, Washington, DC, USA, pp. 202–207 (2009) 2. Yu, J., Joshi, D., Luo, J.: Connecting people in photo-sharing sites by photo content and user annotations. In: Proceeding of International Conference on Multimedia and Expo, New York, USA, pp. 1464–1467 (2009) 3. Yu, J., Jin, X., Han, J., Luo, J.: Social Group Suggestion from User Image Collections. In: Proceedings of the 19th International Conference on World Wide Web, Raleigh, North Carolina, USA, pp. 1215–1216 (2010) 4. Duan, M., UIges, A., Breuel, T.M., Wu, X.: Style Modeling for Tagging Personal Photo Collections. In: Proceeding of the International Conference on Image and Video Retrieval, Santorini, Fira, Greece, pp. 1–8 (2009) 5. Douze, M., Jegou, H., Sandhawalia, H., Amsaleg, L., Schmid, C.: Evaluation of GIST descriptors for web-scale image search. In: Proceeding of the International Conference on Image and Video Retrieval, Santorini, Fira, Greece, pp. 1–8 (2009) 6. Negoescu, R.A., Gatica-Perez, D.: Analyzing Flickr Groups. In: Proceeding of the International Conference on Image and Video Retrieval, Niagara Falls, Canada, pp. 417–426 (2008)
Semi-automatic Flickr Group Suggestion
87
7. Wang, M., Hua, X.-S., Yuan, X., Song, Y., Dai, L.-R.: Optimizing Multi-Graph Learning: Towards A Unified Video Annotation Scheme. In: Proceedings of the 15th ACM International Conference on Multimedia, Augsburg, Germany, pp. 862– 871 (2007) 8. Chen, H.-M., Chang, M.-H., Chang, P.-C., Tien, M.-C., Hsu, W., Wu, J.-L.: SheepDog-Group and Tag Recommendation for Flickr Photos by Automatic Search-based Learning. In: Proceeding of the 16th ACM International Conference on Multimedia, Canada, pp. 737–740 (2008) 9. Rakotomamonjy, A., Bach, F., Canu, S., Grandvalet, Y.: SimpleMKL. Journal of Machine Learning Research 9, 2491–2521 (2008) 10. Chatzichristofis, S., Boutalis, Y.: CEDD: Color and Edge Directivity Descriptor. A Compact Descriptor for Image Indexing and Retrieval. In: Computer Vision System, pp. 312–322 (2008) 11. Frey, B., Dueck, D.: Clustering by Passing messages Between Data Points. Science, 319–726 (2007) 12. Lewis, D.D., Gale, W.A.: A Sequential Algorithm for Training Text Classifiers. In: Proceeding of the 17th Annual International ACM-SIGIR Conference on Research and Development in Information Retrieval, pp. 3–12 (1994) 13. Wright, J., Yang, A., Ganesh, A., Sastry, S., Ma, Y.: Robust Face Recognition via Sparse Representation. IEEE Transaction on Pattern Analysis and Machine Intelligence 31(2), 210–227 (2009) 14. Zha, Z.-J., Yang, L., Mei, T., Wang, M., Wang, Z.: Visual Query Suggestion. In: Proceeding of the 17th ACM International Conference on Multimedia, Beijing, China, pp. 15–24 (2009) 15. Liu, D., Wang, M., Hua, X.-S., Zhang, H.-J.: Smart batch Tagging of Photo Albums. In: Proceeding of the 17th ACM International Conference on Multimedia, Beijing, China, pp. 809–812 (2009)
A Visualized Communication System Using Cross-Media Semantic Association Xinming Zhang1,2, Yang Liu1,2, Chao Liang1,2, and Changsheng Xu1,2 1
National Labratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijng, China 2 China-Singapore Institute of Digital Media, Singapore {xmzhang,liuyang,cliang,csxu}@nlpr.ia.ac.cn
Abstract. Can you imagine that two people who have different native languages and cannot understand other’s language are able to communicate with each other without professional interpreter? In this paper, a visualized communication system is designed to facilitate such people chatting with each other via visual information. Differing from the online instant message tools such as MSN, Google talk and ICQ, which are mostly based on textual information, the visualized communication system resorts to the vivid images which are relevant to the conversation context aside from text to jump the language obstacle. The multi-phase visual concept detection strategy is applied to associate the text with the corresponding web images. Then, a re-ranking algorithm attempts to return the most related and highest quality images at top positions. In addition, sentiment analysis is performed to help people understand the emotion of each other to further reduce the language obstacle. A number of daily conversation scenes are implemented in the experiments and the performance is evaluated by user study. The experimental results show that the visualized communication system is able to effectively help people with language obstacle to better understand each other. Keywords: Visualized Communication, Sentiment Analysis, Semantic Concept Detection.
1 Introduction The growing trend toward globalization has brought with a lot of opportunities and favorable conditions for transnational trade and traveling abroad. It is inevitable that people have to communicate with foreigners frequently. Besides face to face chatting, many online instant messaging tools such as MSN, Google talk, ICQ, are designed to help human communicate with each other regardless of wherever and whenever they are. However, it is difficult for such tools to enable people who have different native languages and do not understand other’s language to communicate smoothly. Existing instant messaging systems mentioned above purely transmit textual information, but the language obstacle makes them useless for foreigners’ conversation. At this time, the machine translation techniques can help them to understand each other. But sometimes the translated result of the machine translation may mislead the users because of its inaccuracy. Therefore, it is necessary to provide a solution for jumping the language obstacle. K.-T. Lee et al. (Eds.): MMM 2011, Part II, LNCS 6524, pp. 88–98, 2011. © Springer-Verlag Berlin Heidelberg 2011
A Visualized Communication System Using Cross-Media Semantic Association
89
In daily communication with foreigners, visual information such as hand signs, body gestures is more understandable when people do not know the language with each other. Since such visual information is a kind of common experience for the people coming from all over the world, for example, if you perform the body gesture of “running”, anybody knows that you want to express the “run” meaning. It enlightens us that it is possible to assist people in communication resorting to visual information such as images, graphic signs. To the best of our knowledge, few studies have been investigated on assisting foreigners’ communications. Some research [1][2][3] aims at detecting chatting topic via the chatter textual information, whose performance mainly lies on the quality of the chatting textual information, instead of making conversation understandable and easy. In this paper, we propose a visualized communication system to provide a solution for foreigners’ conversation with multimedia information. Different from traditional chatting system, our system focuses on the intuitive visual information rather than the textual information. A multi-phase semantic concept detection strategy is proposed to associate the textual information with images collected from the web. Then, the image quality assessment is conducted based on a re-ranking strategy to prune the unsatisfied images and make sure the good quality images at the top positions in the retrieval list. In addition, the sentiment analysis is applied to express the chatter’s emotion with corresponding graphic signs to make the conversation much more understandable. Finally, the representative images and sentiment graphic signs are displayed in proper organization to assist the chatters in understanding the conversation. Our contributions can be summarized as follows: 1. Visualization is a novel technique to assist human communication, which is able to easily jump the language obstacle to make human understand each other. 2. Multi-phase semantic concept detectors not only pay attention to the objects in sentence, but also focus on the activities referred in the conversation, which make the visualized system more precise and vivid. 3. Embedding the sentiment analysis and images re-ranking can significantly prune the ambiguity in conversation and express the chatters’ intention. The remaining sections are organized as follows: Section 2 briefly review the related work. Section 3 introduces the framework of our system. The technical details of each component are described in the Section 4. The experimental results are reported in section 5. We conclude the paper with future work in Section 6.
2 Related Work The rapid growth of Internet leads to more and more chatting tools such as MSN and ICQ. These tools are usually hard to help people of different countries to understand each other due to the language barrier. To tackle this problem, various cartoon expressions or animations are applied to vividly convey the speaker’s intention or emotion. However, current image assistant function is rather limited as well as inconvenient. Specifically, the available image resource in current chatting tools is quite limited and the users have to manually specify the proper image. Motivated by the web
90
X. Zhang et al.
images analysis and annotation, we believe a visualized communication system can greatly promote the mutual understanding between people. To the best of our knowledge, little work has been directly conducted on such image facilitated mediation system. In the following, we will briefly review related work on topic detection in users’ chat messages, semantic concept extraction and re-ranking with sentiment analysis. Existing research on instant message mainly concentrates on processing the textual information. The approach in [3] used Extended Vector Space Model to cluster the instant messages based on the users’ chat content for topic detection. The methods in [1] aimed to extract certain topics from users’ talk and both of them were implemented in an offline manner. In contrast, Dong et.al. [2] analyzed the structure of the chat based on online techniques for topic detection. In the conversation among foreigners where talkers do not understand other’s language, these schemes can do little to facilitate people’s mutual understanding. . In the contrary to such limited expression ability of textual language, images are extremely useful in such cross language mediation. As long as semantically related images are presented, people with different native languages can accurately catch the other’s meaning. To implement such visualized communication, an ideal system should recommend some images closely related to users’ conversation. To extract semantics from images/videos, a series famous concept detectors are proposed, such as Columbia374 [8], VIREO-374 [9] and MediaMill-101 [7]. In [7], the proposed challenging problem for automatic video detection is to pay attention to intermediate analysis steps playing an important role in video indexing. Then they proposed a method to generate the concept detectors, namely MediaMill-101. But the main problem in these handy detectors is that they are all the concepts belonging to the noun domain. The noun detectors are indeed helpful to the low-level feature representing the high-level semantic, but they are limited to important nouns pattern ignoring that lots of necessary modes in people’s chatting include concepts companying the action. It is more meaningful to construct concept detector for the verb-noun phrase to detect uses’ real meaning during the dialogue. At the same time, the task in [4] concentrated on the keypoint-based semantic concept detection. In their work, it covered 5 representation choices that can influence the performance of keypoint-based semantic concept detection. These choices consist of the size of visual word vocabulary, weight scheme, stop word removal, feature selection and spatial information. Its experiment showed that if we choose appropriate configuration of the above items, we can derive a good performance in keypoint-based semantic concept detection. The other fact involves the effect in users’ visualization of the images filtered by semantic concept detectors. Re-ranking and sentiment analysis can help to solve this problem. Current re-ranking techniques can be partitioned into two categories. One is primarily depending on visual cues extracted from the images [Zha et al. MM09]. These visual cues in the re-ranking part are usually different from the visual information during the original search results (e.g. [11]). Another re-ranking technique is co-reranking method utilizing jointly the textual and visual information (e.g. [12]), but these work could not prove the quality of the image. Ke et.al. [10] proposed a novel algorithm to assess the quality of a photo. Their work can judge whether a photo is professional or amateur. Finally sentiment analysis can be considered as an additional function for enrich users’ chatting experience.
A Visualized Communication System Using Cross-Media Semantic Association
91
3 Framework The framework of our proposed visualized communication system is shown in Figure 1. The system consists of two parts: offline training phase highlighted by the yellow background and online communication phase highlighted by the dark green background.
NLP
PIC Mood Pictures
Img1 Img2 Img3 Img4
Feature Extraction
Img1
Re-ranking
Img3
Semantic Concept Detector Set
Fig. 1. The framework of our Visualized Communication System
The purpose of offline phase is to associate the proper images with the specific text information by resorting to semantic concept detectors. We collect the image sets for the predefined concepts which are usually appeared in the human daily conversation content. Then, the semantic concept detectors are trained using low-level visual features by SVM classifier. Four modules are contained in the online phase, which are the natural language processing (NLP), semantic concept detection, re-ranking and sentiment analysis. At first, the NLP module extracts the objects and activity keywords referred in the content from the conversation. At the same time, the translated sentence pair will be directly transmitted to the users’ interface. Then, we retrieve the relevant images by querying the keywords on Google image. The concept detection is applied to filter the noisy images. An image quality based re-ranking module is adopted to express the conversation means with high quality images. The sentiment analysis module is to help people to clearly understand the conversation mood of each other. Finally, the recommended images and sentiment graphic sign are organized and displayed to assist the conversation. The details of each part will be described in section 4.
92
X. Zhang et al.
4 Image Recommendation In this section, we introduce the technical details of the four modules in online phase in Figure 1. 4.1 Google Translation and NLP It is intuitive to use translation software or online translation service (e.g. Google Translation) to help people speaking different languages to communicate with each other. However, the state-of-the-art machine translation techniques are far from the real applications in general domains. One of the problems for machine translation is the translation ambiguity which may mismatch users’ original intent. For example, a Chinese student who cannot speak English and is going to visit his old friend living in Nanyang Technological University does not know the meaning of the phrase in a road sign saying “Go straightly by 500m to NTU”, and then he will seek help on the Google translation website. However, the translation result mismatches the users’ original intent “Nanyang Technological University” with “National Taiwan University” in Chinese. Therefore, after reading the translation, the Chinese student may be confused and may think that “Is this direction right?” From this example, the translated result has its ambiguity. To solve this problem, adding some images to associate with the specific word or sentence to be translated is necessary. If the word “NTU” can be sent to Google image search engine, the images both describing the Nanyang Technological University and National Taiwan University are returned. Thus it is easy to understand that NTU in above example means “Nanyang Technological University”. To the extracted keywords which are assigned by the images, we can utilize the NLP tools [13] to analyze the structure of a specific sentence. In our work, NLP tools can help remove stop words at first and Part of Speech (POS) tool will return the part of speech of each word in the sentence. Then we can extract the nouns pattern and the combination of transitive verb and nouns. This combination represents the activity of the object depicted by the sentence. 4.2 Multi-phrase Concept Detector For the visualized communication system, the crucial issue is how to visualize the conversation content. In other word, the system should automatically associate the visual information such as images with textual information at the semantic level. Therefore, the high-level concept detection is applied to tackle this problem. Semantic concept detection is a hot research topic as it provides semantic filters to help analysis and search of multimedia data [4]. It is essentially a classification task that determines whether an image is relevant to a give semantic concept, which also can be defined by text keywords. In our system, the semantic concept detection is applied for associating the conversation content with the images. However, traditional concept detectors usually focus on the noun phrases, which mostly denote the objects in an image. In human daily conversation, many activities will be appeared in the conversation. Therefore, it is necessary to extend current concept detection scope from pure nouns to using the activity concept detectors to clearly express the actions in the conversation. Figure 2 gives an example about the difference between the noun concept and activity concept.
A Visualized Communication System Using Cross-Media Semantic Association
93
Fig. 2. The difference between the noun and activity concepts
From Figure 2, we can see that the images that describing the noun and activity concept are apparently different. Suppose that a man likes car but cannot drive a car. He says to anther foreign person via a chatting system “Do you know how to drive a car?” If a chatting system is mainly based on noun concepts, it will detect the concept “car” and then return the images possibly like the images in the first row in Figure 2. After watching the images, the foreigner still cannot get the meaning of driving a car. However, if we trained not only the noun concept detectors but also more activity concept detectors representing the human’s activity, the images lied in the second row in Figure 2 can be presented to the user. The main difference between the noun concept detector and activity concept detector lies in that the activity concept detector can express the human’s action. In order to implement the function of detecting the action in the users’ conversation, two tasks should be involved. First we pre-define most usual the transitive with noun phrases in our chat and collect the corresponding web images to train for the concept detectors representing the human activity. We follow the approach in [4]. Once these concept detectors are prepared, they can judge whether a new image contains the concept they required. 4.3 Re-ranking Due to the various qualities returned by the search engine, the re-ranking algorithms may be not a good choice if applied in our system. Although maybe these re-ranking methods can correlate the images to the query, the performance of the re-ranked list may be aggravated if these top images are blurry. We follow the three criteria proposed in [10], which is aim to assess the quality of the photos taken by different people, to conduct re-ranking of searched images. This is different from the traditional re-ranking techniques, which rank the image list again based on another visual cue. Now the image set filtered by semantic concept detectors can be derived in section 4.2, but these images only mean they are related to the noun concept or activity concept. Therefore, if utilizing the traditional re-ranking approaches, it does not seem useful for proving the images with high quality ranked at the top positions. Take the noun concept “volleyball” as an example. If one of the speakers wants to learn what the volleyball looks like, the “volleyball” concept detector can return the images after filtering the images excluding the volleyball. Suppose that the images
94
X. Zhang et al.
Fig. 3. The professional and amateurish images
in the first row in Figure 3 are obtained after concept detector filtering, we can see clearly that the first two images are closer to the speaker’s intent, while the third one depicts volleyball in a volleyball match which seems disorganized to the user’s intention. Therefore, if we recommend the third image to the user, the user must be very unsatisfactory. To assess whether an image is professional, there exist three factors. The first one is Simplicity. Compared with snapshots which are often unstructured, busy, and cluttered, it is easy to separate the subject from the background in professional photos. The second one is realism which is another quality to differentiate the snapshots and professional photos. Snapshots look “real” while professional photos look “surreal”. The last one is about photographers’ basic techniques. It is extremely rare for an entire photo taken by a professional to be blurry. 4.4 Sentiment Analysis Sometimes the same sentence attached to different users’ attitudes will generate different meanings. If two or more users cannot know others’ attitude, they may be misunderstood by the text sentence without expressing others’ emotion. Therefore, it is essential to augment the sentimental information into the system.
Fig. 4. The predefined four sentiment sets
A Visualized Communication System Using Cross-Media Semantic Association
95
Here we mainly want to visualize the mood of a user’s conversation via some images. We predefined four sentiment sets representing the approving opinion, the opposing opinion, happy mood and angry mood respectively in Figure 4. These four sets contain most of our daily used words which could express the emotion of a person and some punctuation as well. At the same time, we correlate some emotional pictures to these sets. When we detect these elements belonging to one set or both sets, corresponding pictures can be selected to the users.
5 Experiments In our experiment, we define 15 scenes (namely traveling, asking the way and so on) which involve about total 90 concept detectors consisting of 30 noun semantic concept detectors and 60 activity semantic concept detectors, respectively. The data are collected via the Google image search engine, Flickr and other sources. In order to train the concept detectors, we follow the rule that the combination of local and global features can boost the performance of the semantic concept detectors [4]. Therefore, we utilize three low-level features, bag-of-visual-words (BOW), color moment (CM) and wavelet texture (WT). In BOW, we generate a 500 dimensional vector attached to each image. In CM, the first 3 moments of the 3 channels in HSV color space over 8 × 8 grid partition is used to form a 384 dimensional color moment vector. For WT, we use 3× 3 grids and each grid is represented by the variances in 9 Haar wavelet sub-bands to generate an 81 dimensional wavelet texture vector. The raw outputs from the SVM classifiers and then converted to posterior probabilities using Platt’s method. The probabilities are combined as a score, which indicates the confidence of detecting a concept in an image by average fusion. We divide our experiments into 2 parts. The first part (section 5.1) gives the accuracy of 20 activity concept detectors selected from all the scenes. An interactive user interface of our system is shown in section 5.2. We conduct a user study to evaluate the visualized communication system in section 5.3. 5.1 The Accuracy of the Semantic Activity Concept Detectors
We totally selected 6 semantic concept detectors from the traveling scene. The accuracy of human action detection of the traveling scene is shown in Figure 5. There are 6 groups along the horizontal axis, each consisting of 4 bars representing the accuracy of the CM, WT, BOW and the average fusion of them respectively. In general, the performance of BoW outperforms those of CM and WT due to the superiority of the local feature. The CM achieves the comparable result with the WT. Moreover, the fusion result achieves the best in most cases, which can be attributed to the complementary superiority collaborating both local and global features. In some specified cases, the results of CM are the best, such as diving and go surfing, which is mainly due to the homogeneous background in images related to these concepts. 5.2 User Interface of Visualized Communication System
The user Interface of our system is shown in Figure 6. It can be divided into 2 parts. The first part in the title region lists users’ name and the current system time. Users can select the scene in this region.
96
X. Zhang et al.
&0 :7 %R: )XVLRQ
Climb the mountains
Diving
Drink water Eat biscuits
Go surfing
Ride bikes
Activity Detector
Fig. 5. The accuracy of the semantic concept detectors
Fig. 6. User Interface of Visualized Communication System
The other part consists of two sections. The left section recommends the images related to the users’ conversation. The right section shows the current user list and the conversation. Our system can be used by the users speaking different languages. The translated sentences will be displayed under the original sentence. 5.3 The User Study
We invited 20 foreigners to evaluate our visualized communication system. We predefine five ranking scores for our proposed system. These five rankings stand for the degree of satisfaction, namely “Very satisfied”, “satisfied”, “Just so-so”, “Not satisfied” and “Disappointed”. The evaluating scores are shown in Figure 7.
A Visualized Communication System Using Cross-Media Semantic Association
97
From the evaluating results, we can see that all the users are satisfied in most cases. However, there are still some disappointed votes to our system. We think that the reasons may come from two aspects. One is that deficiency lies in the essential association between the textual information and visual information. It is hard to find proper image to represent every keyword due to semantic gap. The other is that concept detection result will affect the performance of the image recommendation if the concept detection result is poor. However, the result in Figure 7 proves that our system can enhance the quality of the chatting between the users speaking different languages by resorting to the sentiment graph and recommended images.
Fig. 7. The Evaluation Result of User Study
6 Conclusion The visualized communication system incorporating the multimedia items via visual information can really help the users who have an obstacle to communicate with each other. The multi-phase visual concept detection strategy can detect the most actions related to the conversation so that they can provide some assisting and relevant images for the users. The experimental results show that the visualized communication system is able to effectively help people with language obstacle to better understand each other. In the future, we plan to further study the user’s profile information, such as age, gender and education background, to provide rich multimedia images/video to facilitate the mediation process. In addition, user feedback technology will also be utilized to improve the system’s utility with less operation but more suitable accommodation.
References 1. Adams, P.H., Martell, C.H.: Topic Detection and Extraction in Chat. In: 2008 IEEE International Conference on Semantic Computing, pp. 581–588 (2008) 2. Dong, H., Hui, S.C., He, Y.: Structural analysis of chat messages for topic detection. Online Information Review, 496–516 (2006) 3. Wang, L., Jia, Y., Han, W.: Instant message clustering based on extended vector space model. In: Proceedings of the 2nd International Conference on Advances in Computation and Intelligence, pp. 435–443 (2007)
98
X. Zhang et al.
4. Jiang, Y.-G., Yang, J., Ngo, C.-W., Hauptmann, A.G.: Representations of Keypoint-Based Semantic Concept Detection: A Comprehensive Study. IEEE Transitions on Multimedia, 42–53 (2009) 5. Jiang, Y.G., Ngo, C.W., Chang, S.F.: Semantic context transfer across heterogeneous sources for domain adaptive video search. In: Proceedings of the Seventeen ACM International Conference on Multimedia, pp. 155–164 (2009) 6. Snoek, C.G.M., Huurnink, B., Hollink, L., de Rijke, M., Schreiber, G., Worring, M.: Adding semantics to detectors for video retrieval. IEEE Transaction on Multimedia 9(5), 975–986 (2007) 7. Snoek, C.G.M., Worring, M., Van Gemert, J.C., Geusebroek, J.M., Smeulders, A.W.M.: The challenge problem for automated detection of 101 semantic concepts in multimedia. In: Proceedings of the 14th Annual ACM International Conference on Multimedia, p. 430 (2006) 8. Yanagawa, A., Chang, S.-F., Kennedy, L., Hsu, W.: Columbia university’s baseline detectors for 374 lscom semantic visual concepts. In: Columbia University ADVENT Technical Report #222-2006-8 (2007) 9. Jiang, Y.-G., Ngo, C.-W., Yang, J.: Towards optimal bag-of-features for object categorization and semantic video retrieval. In: Proceedings of the 6th ACM International Conference on Image and Video Retrieval, p. 501 (2007) 10. Ke, Y., Tang, X., Jing, F.: The design of high-level features for photo quality assessment. In: CVPR 2006 (2006) 11. Natsev, A., Haubold, A., Tesic, J., Xie, L., Yan, R.: Semantic concept-based query expansion and re-ranking for multimedia retrieval. In: ACM Multimedia, p. 1000 (2007) 12. Yao, T., Mei, T., Ngo, C.W.: Co-reranking by Mutual Reinforcement for Image Search. In: Proceedings of the 6th ACM International Conference on Image and Video Retrieval (2010) 13. http://nlp.stanford.edu/software/tagger.shtml 14. Shih, J.-L., Chen, L.-H.: Color image retrieval based on primitives of color moments. In: IEE Proceedings-Vision, Image, and Signal Processing, p. 370 (2002) 15. Van de Wouwer, G., Scheunders, P., Dyck, D.V.: Statistical Texture Characterization from Discrete Wavelet Representations. IEEE Transactions on Image Processing, 592-598 (1999) 16. Zhang, J., Marszalek, M., Lazebnik, S., Schmid, C.: Local features and kernels for classification of texture and object categories: A comprehensive study. IJCV, 213–238 (2007) 17. Keshtkar, F., Inkpen, D.: Using Sentiment Orientation Features for Mood Classification in Blogs. IEEE, Los Alamitos (2009) 18. Pang, B., Lee, L., Vaithyanathan, S.: Thumbs up?: sentiment classification using machine learning techniques. In: Proceedings of the ACL 2002 Conference on Empirical Methods in Natural Language Processing, vol. 10 (2002) 19. Zha, Z.-J., Yang, L., Mei, T., Wang, M., Wang, Z.: Visual query suggestion. In: Proceedings of ACM International Conference on Multimedia, pp. 15–24 (2009)
Effective Large Scale Text Retrieval via Learning Risk-Minimization and Dependency-Embedded Model Sheng Gao and Haizhou Li Institute for Infocomm Research (I2R), A-Star, Singapore, 138632 {gaosheng,hli}@i2r.a-star.edu.sg
Abstract. In this paper we present a learning algorithm to estimate a risksensitive and document-relation embedded ranking function so that the ranking score can reflect both the query-document relevance degree and the risk of estimating relevance when the document relation is considered. With proper assumptions, an analytic form of the ranking function is attainable with a ranking score being a linear combination among the expectation of relevance score, the variance of relevance estimation and the covariance with the other documents. We provide a systematic framework to study the roles of the relevance, the variance and the covariance in ranking documents and their relations with the different performance metrics. The experiments show that incorporating the variance in ranking score improves both the relevance and diversity. Keywords: Risk minimization; Diversity search; Language Model.
1 Introduction The task in the information retrieval (IR) system is to retrieve documents relevant to the information needs of users and to rank them with respect to their relevance to the query. Thus, the design of ranking function (or ranker) that computes the relevance score has been the central topic in the past decades. One of the popular ranking principles is based on Bayesian theory, where the documents are ranked according to the odds of the relevance and irrelevance probability for a query-document pair (e.g. [6, 7, 9, 10]). Many rankers are thus derived with the different assumptions of query model and document model such as Okapi [12], Kullback-Leibler divergence and query log-likelihood [6, 7, 8], cosine distance with the tfidf features [2]. These functions share three common properties: 1) the uncertainty of parameter estimation on query or document models is ignored; 2) the ranking score thus omits uncertainty of calculation; and 3) the relevance score, often calculated for each query-document pair, excludes the document relationship. For example, the document or query models are point estimation in LM with the ML criterion [6, 7, 8] as well as the ranking score. When we use the point estimation of relevance in ranking, the risk arises due to the uncertainty of estimation. In addition, because of the independency assumption on documents when calculating relevance score, the diversity related to the query topic is deteriorated. The recent research in diversity search tries to address the problem so that the top-N documents deliver as many relevant sub-topics as possible [3, 4, 19, 20, 21]. K.-T. Lee et al. (Eds.): MMM 2011, Part II, LNCS 6524, pp. 99–110, 2011. © Springer-Verlag Berlin Heidelberg 2011
100
S. Gao and H. Li
However, none of the methods can calculate the ranking score in a principle way when the document dependency is embedded, most of them using an ad-hoc method to interpolate the relevance and divergence scores. In the paper we present a principle method to learn a ranking function when the document relation is embedded. The novel ranker is derived by minimizing a risk function, which measures the expected loss between the true ranking score and the predicted one on a set of documents interested (For clarity, we use the ranking score to refer to the final value for ranking documents. It is the sum of relevance score and uncertainty score. The relevance score holds its traditional meaning that refers to query-document similarity while the uncertainty score measure the risk in estimating similarity score and related to the diversity.). With suitable assumptions (See section 2.), an analytic form is resulted in and the ranking score becomes a linear combination among the expectation of the relevance score, the uncertainty (variance) of relevance estimation, and the abstract of the covariance between the document and the others, which measures the document dependency. The evaluation is done on the ImageCLEF 2008 set used for the ad-hoc photo image retrieval for diversity search. We study the roles of variance and covariance in ranking and their effects on mean average precision (MAP, relevance measure) and cluster recall at the top-N (CR@N, diversity measure). Our analysis will show that incorporating the variance improves both the relevance performance and diversity performance, and the addition of the covariance further improves the diversity performance at the cost of relevance performance. In the next Section, the proposed risk-minimization and dependency-embedded ranking function (RDRF) is introduced. Then we report the experiments in Section 3. The related work and our findings are discussed in Sections 4 and 5 respectively.
2 Learning RDRF Ranker In the section, we will elaborate the RDRF ranker using the statistical methodology. Firstly, the ranking function is treated as a random variable rather than a deterministic value; secondly, the objective function is defined, which measures the expected loss between the true ranking scores and the predicted ones over the documents; lastly, a particular RDRF ranker is derived based on the query log-likelihood ranking function. 2.1 Ranking Functions as Random Variables , , | | ( :the iIn the LM based IR, the relevance score between a query, th term frequency in the query. | |: the size of the vocabulary.), and a document, , , | | ( : the i-th term frequency in the document.), can be calculated from the query log-likelihood generated by the document model, , , | | , , , | | , and the document or from the discrepancy between the query LM, LM . Here the query log-likelihood function (see Eq.(1)) is chosen as the ranking function to develop the RDRF ranker, while the principle can be applied to the others. ∑|
|
log
(1)
Effective Large Scale Text Retrieval
101
In the case, the query model is the query term frequency and the document model is unigram LM. Conventionally, LM is computed using the ML criteria before calculating the ranking scores. Obviously, the ranking score is one value of Eq. (1) at the estimated point of document model. To improve robustness of score estimation, we can sample as many points as possible from the distribution, , to collect a large number of scores for each querydocument pair. We then use their average as the ranking score. Such the estimation should be more robust than point estimation. However, it is time consuming. is Fortunately, we will soon find that sampling is not necessary. Because is a random function. For each query-document pair, we have a random, thus to characterize the distribution of query-document relevance random variable in is integrated out with some suitable assumptions. score. 2.2 Risk-Minimization Based RDRF Ranker For each query-document pair, we have to characterize the distribution of relevto ance score. Thus, given a set of documents (or the corpus), we have a set of describe the joint distribution of the set of relevance scores. The RDRF algorithm tries to measure and minimize the loss between the true ranking scores and the predicted ones on the documents in order to find the optimal one. 2.2.1 Loss Function on the Set We first define a few notes used in the discussions. : the corpus with | | documents. : the i-th document in . : the ranking function of (See Eq.(1)). | , , : the distribution of depending on the query , the document excluding . and the set of documents : the predicted ranking score of . : the overall ranking score on the set of documents. The whole corpus is used here for discussion. The findings are applicable in the other cases. | , : the distribution of . Now the overall ranking score is defined as ( : the weight of ), ∑|
|
(2)
We try to seek a set of estimation so that Eq. (2) is maximized. It is noted that is | , connected by Eq. (1). should be known in theory as the function of are estimated from the documents. We denote the predicted overall ranking score as . Its optimal estimation is found by minimizing Eq.(3), ,
,
,
| ,
.
(3)
Here · | · is the expected loss. · is the loss function, which is the LINEX function in Eq.(4) [13, 15, 18] ( is the risk weight.). 1
, And the optimal Bayesian estimation
is
(4)
102
S. Gao and H. Li
1
∑
⁄2
!
⁄6
⁄24.
(5)
Herein ’s are the cumulants. For example, (the mean of ), (the measure of skewness of ), while is the n-th (the variance of ), moment of . Eq. (5) tells that the optimal overall ranking score is only related to the moments of . In the paper, only the mean and variance are used. It is not our target to know . In the next, we will introduce how to connect with the estimation of individual ranking score when the document dependency is embedded. 2.2.2 Document-Dependent Relevance Score are calculated as in Eq.(6) and Eq.(7), From Eq. (2), it is easy to verify that and ∑| ∑|
|
∑|
|
(6)
|
(7)
is the mean of and is the covariance between and . ReHere placing Eq.(6) and Eq.(7) into Eq. (5), we can get the overall ranking score as, ∑|
|
⁄2 ∑|
⁄2
|
(8)
In Eq. (8) there are three components in the rectangle bracket. The first one is the expectation of relevance score for . The second one is its variance, an uncertainty measure coming from the document itself. And the last one is the covariance coming from the other documents, an uncertainty measure due to dependency on the other . Here documents. The last two summarizes the overall uncertainty in estimating the variance and covariance are treated separately because of their different effects on the ranking relevance (See section 3 for experimental studies.) Now we separate the term related to and get a novel ranking function as, ⁄2 ∑|
⁄2 ̃
|
(9)
and . However, it is not easy. In the To calculate ̃ , we need to know following, we discuss two practical ways for computation of Eq.(9). A) Documents are independent of each other when calculating and . Therefore, for the query log-likelihood function, the mean is calculated as, ∑| |
|
∑|
|
log
|
log
(10)
is the posterior distribution. Here it is the Dirichlet distribution, ∏|
|
|
(11)
| |
∏
. Γ · : gamma function.
the document length.
According to the properties of the Dirichlet distribution, log
.
(12)
Effective Large Scale Text Retrieval
· : the digamma function),
Therefore, the mean is calculated as ( ∑| Similarly,
|
(13)
is derived as in Eq. (14) ( ∑|
· : trigamma function),
|
(14)
B) Documents are dependent when calculating covariance To get the covariance, we calculate from Eq.(15). ∑|
103
| ,
log
. log
(15)
In order to get log log , we need to estimate to the Bayes’ rule, it is found that,
,
|
,
. According
,
(16)
is independent of given . Thus, we When we derive Eq. (16), we assume induce an order in the document pair, i.e. Eq.(16) measuring the information flow to . In general, it is not equal to the information flowed from to . We from will soon discuss its effect on the covariance. According to the properties of the Dirichlet distribution, Eq.(17) is derived to calculate
log
log
(
: a pseudo-document with the term frequency
), log
log
log
log
(17)
Now the covariance is computed as, ∑|
|
log
∑|
|
log
log
(18)
In Eq. (18), the first sum is the expectation of relevance score of . The second sum is the difference between the expectation of relevance score for conditioned on , ) and the expectation of (Document model is (Document model is | ). Since , , , Eq. (18) is asymmetric. Strictly saying, it is not a covariance. But here we still use the term to measure the dependency between the documents. 2.2.3 Discussions Substituting Eqs.(13, 14, 18) into Eq.(9), we will get the ranking score for each querydocument pair in case of the document dependency included. It is obvious that the value of the covariance (plus variance) is not comparable in the scale to that of expectation. It is tough to balance them by adjusting the risk weight , because the range of
104
S. Gao and H. Li
the risk weights depend on the size of document set. To normalize such effect, herein we introduce 3 types of methods to calculate the covariance abstract. A) Covariance average The average of covariance (plus variance) is used as the covariance abstract, i.e., ̃
| |
∑|
̃ will have the comparable range with noted that . The prior weights
|
(19)
for the variable size of | |. It is are discarded in the paper.
B) Maximal covariance The average covariance can smooth the effects of the uncertainties coming from different documents. Like in MMR [3] to use the maximal margin as the measure of diversity, we can also use the maximal covariance (plus variance) as the abstract, i.e., ̃
max
,| |
(20)
Thus, if a document has the higher covariance with the others, i.e. more similar in the content with others, it will get larger penalty in calculating ranking score. It means that the ranker will promote the documents which contain novel information. C) Minimal covariance Similarly, the minimal covariance can also be used. ̃
min
,| |
(21)
In the above discussions, we know that we need to find a working set for each document in order to calculate the covariance. For simplicity, the whole corpus is selected in the above. In practice, the working set may depend on the individual document and thus vary according to the applications. For example, if we use the rankers in Eqs. (19-21) to rank all documents in the corpus, the working set is the corpus. But if we want to re-rank the documents in a ranking list in a top-down manner, the working set for a document, saying , may only include the documents that are ranked higher than in the ranking list.
3 Experiments We implement the proposed rankers based on the Lemur toolkit1. Lemur is the representative of the up-to-date technologies developed in IR models. It is used to develop the benchmark system and our ranking systems. 3.1 Evaluation Sets The experiments are carried out on the ImageCLEF 2008 (CLEF08) set, which is officially used for evaluating the task of the ad-hoc image retrieval. It has 20,000 documents with the average length 19.33 words. In the paper we do experiments only on the text modality. The query contains the terms in the field of title with an average 1
http://www.lemurproject.org
Effective Large Scale Text Retrieval
105
length 2.41 terms. Totally 39 queries are designed. Because the queries are designed to evaluate the diversity, they have much ambiguity. 3.2 System Setup and Evaluation Metrics The Jelinek-Mercer LM2 is chosen for document representation. The baseline system is built on the traditional query log-likelihood function (See Eq. (1)). Although the proposed rankers can be applied to retrieve and rank the documents from the whole corpus, considering the computation cost, we run our rankers on a subset which contains the top-1000 documents in the initial ranking list generated by the baseline. The performances of different systems are compared in terms of multiple metrics including mean average precision (MAP) for the relevance performance and cluster recall at the top 5 (CR@5) and 20 (CR@20) for the diversity performance. The cluster recall at the top-N documents measures the accuracy of the sub-topics in the top-N documents for a query. It is calculated by the number of sub-topics covered in the topN documents divided by the total sub-topics of the query. In the ImageCLEF08 set, the sub-topic label is tagged for each document and query in the pooled evaluation set besides the relevance label3 . 3.3 Result Analysis Now we study the behaviors of the RDRF rankers. From the discussion in Section 2, we know that the RDRF based ranking scores contain 3 components: expectation of the relevance score, variance and covariance. In addition, there is a tuning ter . In the following, we will study the performance as a function of the risk weight in the following conditions: 1) variance without document relation (See section 3.3.1), 2) covariance (See section 3.3.2), 3) various covariance abstract methods (See section 3.3.3) and 4) working set selection (See section 3.3.4). In the first two studies, (Eq. (20)) is chosen while the working set is same in the first 3 stuthe ranker dies4. 3.3.1 Effects of Variance Figure 1 depicts the changing performance as a function of the increasing risk weight in the case of only the variance being considered. Obviously, the performances are improving as the weight is increasing. At some points, they reach their maximum and then drops. The maximal MAP, CR@5 and CR@20 are 0.1310 ( 3), 0.1340 ( 1) and 0.2316 ( 4), respectively. In comparison with the baseline performance, which has 0.1143 for MAP, 0.1182 for CR@5 and 0.1990 for CR@20, the significant improvements are observed5. The relatively improvements are 14.6% (MAP), 13.4% (CR@5) and 16.4% (CR@20), respectively. Results are reported here based on the smoothing parameter 0.1. Findings are similar for others. 3 http://imageclef.org/2008/photo 4 The working set for a document contains the documents ranked higher than it in the initial ranking list. For computational consideration, currently only the top-100 documents are reranked using the learned rankers and the other documents in the initial ranking list keep their ranking positions. 5 We test statistical significance using t-test (one-tail critical value at the significance level 0.05). 2
106
S. Gao and H. Li
Therefore, the Bayesian estimation of ranking score gets the better performance than the traditional point estimation based method. The addition of variance further improves the performance. In the experiment, the performances with the zero risk weight are 0.1195 (MAP), 0.1244 (CR@5) and 0.2022 (CR@20) respectively, which are worse than the performances obtained with the optimal risk weights. Our investigations on Eqs.(9, 14, 20) reveal that the variance functions as a penalty added to the relevance estimation. With the same term frequency, the term penalty in a long document is higher than that in a short document. The trigamma function in Eq.(14) also normalizes the effect of the document lengths on the term contribution. Due to the normalization of term and document length, the estimated ranking score become robust and has the positive effect on the performances. MAP
CR@5
CR@20
0.2 0.1 0 0
2
4
6
8
10
Fig. 1. Performance vs. risk weight b (only the variance is considered) VAR MAX_COV
MAX_COV
0.2
0.2
0.15
0.15
0.1
0.1
0.05
0.05
0
0 0
2
4
(a) MAP
6
8 10
VAR MAX_COV
VAR 0.3 0.2 0.1 0 0
2
4
(b) CR@5
6
8
10
0
2
4
6
8
10
(c) CR@20
Fig. 2. Performance vs. risk weight b for the covariance based ranker (MAX_COV) and the variance only ranker (VAR)
3.3.2 Effects of Covariance We now add the covariance into the ranking score. Its effect on the performance is illustrated in Figure 2 and is compared with the performance when the dependency is ignored (i.e. only variance, see section 3.3.1). We have the following findings. 1) The inclusion of covariance improves the diversity performance. In the experiments, adding the dependency obviously improves the diversity performance compared with the case where only the variance is included. We study
Effective Large Scale Text Retrieval
107
maximum of CR@5 and CR@20 for the covariance based ranker (MAX_COV) and the variance based ranker (VAR). The CR@5 has a relative increment 12%, which reaches 0.1501, while CR@20 improves 4.7% which achieves 0.2424. Since the covariance measures the document dependency, it is not surprising to see it improve the diversity performance. 2) Incorporating the covariance decreases MAP, which coincides with the observations in [3, 4, 19, 20]. Since the diversity only concerns the novelty among the documents rather than the similarity, we can understand that in most of times, the diversity algorithms might have negative effects on the relevance metric. 3.3.3 Effects of Covariance Abstract Methods Figure 3 illustrates the performances for the rankers based on 3 covariance abstract methods, i.e. covariance average (AVG_COV), maximal covariance (MAX_COV) and minimal covariance (MIN_COV) (see Sec. 2.2.3 for details). AVG_COV MAX_COV MIN_COV
0.15
AVG_COV MAX_COV MIN_COV
AVG_COV MAX_COV MIN_COV
0.3
0.15
0.05
0.1
0.05 0
2
4
6
8 10
0
(a) MAP
2
4
6
8
0
10
(b) CR@5
2
4
6
8
10
(c) CR@20
Fig. 3. Performance vs. risk weight b for the rankers with covariance average (AVG_COV), maximal covariance (MAX_COV) and minimal covariance (MIN_COV) MAX_COV_N MAX_COV
0.2
MAX_COV_N MAX_COV
MAX_COV_N MAX_COV
0.2
0.2
0.1
0.1 0
0 0
2
4
(a) MAP
6
8
10
0 0
2
4
(b) CR@5
6
8
10
0
2
4
6
8
10
(c) CR@20
Fig. 4. Performance as a function of risk weight b for the rank-listed working set (MAX_COV) and the working set with all documents (MAX_COV_N)
We find that MAX_COV works best. MIN_COV is the worst while AVG_COV is in the middle. When we compare MAX_COV with AVG_COV, the biggest gap is
108
S. Gao and H. Li
found for CR@20 which reaches 20% relative increment. For CR@5, MAX_COV obtained 8.2% increment. 3.3.4 Working Set Selection In the above experiments, we collect all documents that are ranked higher than the document of interest as the working set. Thus, each document has a different working set. We call it the rank-listed working set. But if the ranking order is not available, we can use all documents as the working set. Figure 4 compares the performances between two working set schemes. One is used in the above experiments (MAX_COV) and another is the working set which contains all documents to be ranked (MAX_COV_N). In the experiments, the latter working set includes top-100 documents to be re-ranked for each document. Figure 4 shows that the rank-listed working set works better. But their maximal performances are similar. It is seen that the performance of MAX_COV_N drops quicker than that of MAX_COV off the optimal setting. In other words, MAX_COV is more stable. This is because that the MAX_COV_N includes more documents in its working set than MAX_COV set. Thus, the MAX_COV_N incurs more noises in calculating the document dependency measure.
4 Related Work Lafferty & Zhai [6, 16] presented the risk minimization framework for IR model, where the user browsing documents is treated as the sequential decision. They proved that the commonly used ranking functions are the specials of the framework with the particular choices of loss functions, query and document models and presentation schemes. Although finding the optimal decision is formulated as minimizing the Bayesian loss, in practice they approximated the objective function using the point estimation in their work. In our work the full Bayesian approach is exploited by integrating out the model parameters. Thus, our work results in a novel ranker, which contains: 1) the similarity expectation, a measure of the average relevance and 2) the covariance (plus variance), a measure of the risk of the expectation estimation that is related to diversity performance. The above novelties also make our work different from the risk-aware rankers presented by [13, 18]. In their work, each term is treated as an individual ranker and the uncertainty of the term ranker is estimated independently. Therefore, they need to combine all term rankers to get the query-document ranking score. In comparison, we directly estimate the expectation and the uncertainty. This gives us a single value for each query-document pair to measure the expected relevance score and the uncertainty. Incorporating the covariance into the ranking score is now a natural result in our risk-minimization and document-dependency embedded framework. In contrary, the alternative method is used to calculate the covariance in [13]. When applied to diversity search, our work is quite different from the methods such as [3, 4, 19, 20]. In their works, they developed the methods to estimate the diversity degree of the documents and then linearly combined the diversity scores with the similarity scores in the original ranking list. This means that their similarity scores
Effective Large Scale Text Retrieval
109
and diversity scores are estimated separately which may follow the different criteria. But in our work, they are derived under a unified risk-minimization framework.
5 Conclusion We have presented a learning algorithm to achieve a ranking function where the uncertainty of relevance estimation and document relations are embedded. Learning the ranking function is formulated in the framework of Bayesian risk-minimization. With proper assumptions, an analytic form of the ranking function is attainable. The resulted ranking score becomes a linear combination among the expectation of relevance score, variance of expectation estimation and the covariance with other documents. The presented algorithm provides a systematic way to study the relation among the relevance, diversity and risk. The roles of the variance and covariance in ranking are empirically studied. The inclusion of variance in the ranker improves the performance of relevance as well as diversity. And incorporating covariance can further improve the diversity performance but decrease relevance performance (MAP). The tunable risk weight allows us to balance the relevance and diversity. In future, we will investigate how to embed an adaptive query model in the framework.
References 1. Allan, J., Van Rijsbergen, C.J.: Probabilistic models of information retrieval based on measuring the divergence from randomness. ACM Trans. On Information Systems 20(4), 357–389 (2002) 2. Baeza-Yates, R., Ribeiro-Neto, B.: Modern information retrieval. Addison-Wesley Publisher, Reading (1999) 3. Carbonell, J., Goldstein, J.: The use of MMR, diversity-based reranking for reording documents and producing summaries. In: Proc. of SIGIR 1998 (1998) 4. Chen, H., Karger, D.R.: Less is more: probabilistic models for retrieving fewer relevant documents. In: Proc. of SIGIR 2006 (2006) 5. Jelinek, F., Mercer, R.: Interpolated estimation of markov source parameters from sparse data. Pattern Recognition in Practice, 381–402 (1980) 6. Lafferty, J.D., Zhai, C.: Document language models, query models and risk minimization for information retrieval. In: Proc. of SIGIR 2001 (2001) 7. Lavrenko, V., Croft, W.B.: Relevance-based language models. In: Proc. of SIGIR 2001 (2001) 8. Ponte, J.M., Croft, W.B.: A language modeling approach to information retrieval. In: Proc. of SIGIR 1998 (1998) 9. Robertson, S.E., Sparck Jones, K.: Relevance weighting of search terms. Journal of the American Society for Information Science 27(3), 129–146 (1976) 10. Robertson, S.E., Walker, S.: Some simple effective approximations to the 2-poisson models for probabilistic weighted retrieval. In: Proc. of SIGIR 1994 (1994) 11. Robertson, S.E.: The probability ranking principle in IR. Readings in information Retrieval, 281–286 (1997) 12. Robertson, S.E., Walker, S., Hancock-Beaulieu, M., Gatford, M., Payne, A.: Okapi at TREC-4. In: Proc. of Text Retrieval Conference, TREC (1995)
110
S. Gao and H. Li
13. Wang, J., Zhu, J.H.: Portfolio theory of information retrieval. In: Proc. of SIGIR 2009 (2009) 14. Zaragoza, H., Hiemstra, D., Tipping, M., Robertson, S.E.: Bayesian extension to the language model for ad hoc information retrieval. In: Proc. of SIGIR 2003 (2003) 15. Zellner, A.: Bayesian estimation and prediction using asymmetric loss functions. Journal of the American Statistical Association 81(394), 446–451 (1986) 16. Zhai, C., Lafferty, J.D.: A risk minimization framework for information retrieval. Information Processing and Management 42(1), 31–55 (2006) 17. Zhai, C., Lafferty, J.D.: A study of smoothing methods for language models applied to information retrieval. ACM Trans. on Information Systems 22(2), 179–214 (2004) 18. Zhu, J.H., Wang, J., Cox, I., Taylor, M.: Risk business: modeling and exploiting uncertainty in information retrieval. In: Proc. of SIGIR 2009 (2009) 19. Zhai, C., Cohen, W., Lafferty, J.: Beyond independent relevance: methods and evaluation metrics for subtopic retrieval. In: Proc. of SIGIR 2003 (2003) 20. Gollapudi, S., Sharma, A.: An axiomatic approach for result diversification. In: Proc. of WWW 2009 (2009) 21. Radlinski, F., Kleinberg, R., Joachims, T.: Learning diverse rankings with multi-armed bandits. In: Proc. of ICML 2008 (2008)
Efficient Large-Scale Image Data Set Exploration: Visual Concept Network and Image Summarization Chunlei Yang1,2 , Xiaoyi Feng1 , Jinye Peng1 , and Jianping Fan1,2 1
School of Electronics and Information, Northwestern Polytechnical University, Xian, P.R.C. 2 Dept. of Computer Science, UNC-Charlotte, Charlotte, NC 28223, USA
Abstract. When large-scale online images come into view, it is very important to construct a framework for efficient data exploration. In this paper, we build exploration models based on two considerations: inter-concept visual correlation and intra-concept image summarization. For inter-concept visual correlation, we have developed an automatic algorithm to generate visual concept network which is characterized by the visual correlation between image concept pairs. To incorporate reliable inter-concept correlation contexts, multiple kernels are combined and a kernel canonical correlation analysis algorithm is used to characterize the diverse visual similarity contexts between the image concepts. For intra-concept image summarization, we propose a greedy algorithm to sequentially pick the best representation of the image concept set. The quality score for each candidate summary is computed based on the clustering result, which considers the relevancy, orthogonality and uniformity terms at the same time. Visualization techniques are developed to assist user on assessing the coherence between concept-pairs and investigating the visual properties within the concept. We have conducted experiments and user studies to evaluate both algorithms. We observed very good results and received positive feedback.
1 Introduction With the exponential availability of high-quality digital images, there is an urgent need to develop new frameworks for image summarization and interactive image navigation and category exploration [1-2]. The project of Large-Scale Concept Ontology for Multimedia (LSCOM) is the first such kind of efforts to facilitate more effective end-user access of large-scale image/video collections in a large semantic space [3-4]. There are more than 2000 concepts and 61901 labels for each concept in the LSCOM project, which is still a small subset compared to all the available online resources. Commercial image collection web site, such as Flickr.com, has 2 billion images and are still increasing. Considering the scale of problem we are dealing with, to effectively organize the relationship between large number of concepts and also to better summarize the large number of data within each concept will be the focus of our work. To organize the relationship of inter-concept pairs, concept ontology can be used to navigate and explore large-scale image/video collections at the concept level according to the hierarchical inter-concept relationships such as “IS-A” and “part-of” [4]. However, following issues make most existing techniques for concept ontology construction unable to support effective navigation and exploration of large-scale image collections: K.-T. Lee et al. (Eds.): MMM 2011, Part II, LNCS 6524, pp. 111–121, 2011. c Springer-Verlag Berlin Heidelberg 2011
112
C. Yang et al.
(a) Only the hierarchical inter-concept relationships are exploited for concept ontology construction [5-6]. When large-scale online image collections come into view, the interconcept similarity relationships could be more complex than the hierarchical ones (i.e., concept network) [7]. (b) Only the inter-concept semantic relationships are exploited for concept ontology construction [5-6], thus the concept ontology cannot allow users to navigate large-scale online image collections according to their visual similarity contexts at the semantic level. It is well-accepted that the visual properties of the images are very important for users to search for images [1-4, 7]. Thus it is very attractive to develop new algorithm for visual concept network generation, which is able to exploit more precise inter-concept visual similarity contexts for image summarization and exploration. To reduce the scale of the image set of each concept, while maintaining visual perspectives of this concept as much as possible, image summarization methods can be conducted to select a subset of images which have the best representative visual properties. Jing et al. [8] used the ”local coherence” information to find the most representative image, which has the maximum number of edge connections in the group. Simon et al. [9] proposed a greedy algorithm to find representative images iteratively and also considered likelihood and orthogonality score of the image. Unfortunately, neither algorithms took into consideration the global clustering information, such as the distribution of the clusters, the size, mean value and variance value of each cluster. Based on these observations, this paper will focus on: (a) integrating multiple kernels and incorporating kernel canonical correlation analysis (KCCA) to enable more accurate characterization of inter-concept visual similarity contexts and generate more precise visual concept network; (b) supporting similarity-preserving visual concept network visualization and exploration for assisting users on perceptual coherence assessment; (d) iterative image summarization generation considering the global clustering information, which is specifically characterized by relevancy, orthogonality and uniformity score; and (e) generating visualization of image summarization results within each of the concept. The remainder of this paper is organized as follows. Section 2 introduces our approach for image content representation, similarity determination and kernel design. Section 3 introduces our work on automatic visual concept network generation. Section 4 describes our work on intra-concept image summarization algorithm. We visualize and evaluate our visual concept network and image summarization results in section 5. We conclude this paper at section 6.
2 Data Collection, Feature Extraction and Similarity Measurement The images used in this work are collected from the internet. There are totally 1000 keywords used to construct our data set, with some of the keywords derived directly from Caltech-256 [13] and LSCOM concept list. It is not an easy job to determine the meaningful text terms for crawling images from the internet. Many people use the textterm architecture from WordNet. Unfortunately, most of the text terms from WordNet may not be meaningful for image concept interpretation, especially when you need a
Efficient Large-Scale Image Data Set Exploration
113
Fig. 1. The taxonomy for text term determination for image crawling
Fig. 2. feature extraction frameworks: image-, grid-,segment-framework
large number of keywords; Some other people use the most popular tags from Flickr or some other commercial image-share web sites as the query keywords. The problem, again, is that these tags may not represent a concrete object or something that has a visual form, examples are like “2010”, ”California”, ”Friend”, “Music” etc. Based on the above analysis, we have developed a taxonomy for nature objects and scenes interpretation [10]. Thus we follow this pre-defined taxonomy to determine the meaningful keywords for image crawling as shown in Fig. 1. Because there is no explicit correspondence between the image semantics and the keywords extracted from the associated text documents, images returned are sometimes junk images or weakly-related images. We apply the algorithms introduced in [11] for cleansing the images which are crawled from the internet(i.e., filtering out the junk images and removing the weakly-tagged images). For each of the keywords, or concepts as used in this paper, approximately 1000 images are kept after cleansing. For image classification applications, the underlying framework for image content representation should be able to: (a) characterize the image contents as effectively as possible; (b) reduce the computational cost for feature extraction to a tolerable degree. Based on these observations, we have incorporated three frameworks for image content representation and feature extraction as shown in Fig. 2: (1) image-based framework; (2) segment-based framework and (3) grid-based framework. The segment-based framework has the most distinguish power and the best representation of the image, however, it always suffer from the large computation burden and sometimes may over segment the image. On the other hand, image-based framework is most computationally efficient but is too coarse to model the local information. As a tradeoff, we find
114
C. Yang et al.
the grid-based framework most suitable for our system in terms of both efficiency and effectiveness. Feature extraction is conducted on the grids of the images as described above. The global visual features such as color histograms can provide the global region statistics and the perceptual properties of entire region, but they may not be able to capture the object information, or in other words ,the local information, within the region[12,14]. On the other hand, the local visual features such as SIFT(Scale Invariant Feature Transform) features can allow object-level detail recognition against the cluttered backgrounds [12,14]. In our implementation, we incorporated three types of visual features as color histogram, gabor texture and SIFT, which are described in detail as follows. Color histogram: We performed on the HSV color space to extract the histogram vectors. HSV color space outperforms the RGB color space by its invariance to the change of illuminance. We conducted the histogram extraction on 18 bins of Hue components and 4 bins of Saturation components which yields a 72 dimension feature vector. Gabor texture: We apply gabor filter to different orientation and scale of the image region and the homogenous texture feature of the region is represented by the mean and standard deviation value of the transformed coefficients. Specifically, the extraction is conducted on 3 scales and 4 orientations which yields a 24 dimension feature vector. SIFT: We use the SURF (Speed Up Robust Features)[14]descriptor which is inspired by the SIFT descriptor but has a much faster computation speed in the real implementation. The parameters are configured as blob response threshold equals 0.001 and maximum number of interpolation step equals 5. The similarity measurement for color histogram and gabor texture is defined as follows: Similarity(i, j) = max −||xi − xj ||2 i,j∈X
(1)
where i, j are from the grid set of X, which is composed by 5 regional grids. By calculating the similarity score for each of the grid pairs, the maximum score is taken as the similarity score between the two images. The similarity measurement for SIFT feature is defined as Similarity(i, j) =
total # of matches total # of interesting points
(2)
We have also studied the statistical property of the images under each feature subset as introduced above. The gained knowledge for the statistical property of the images under each feature subset has been used to design the basic image kernel for each feature subset. Because different basic image kernels may play different roles on characterizing the diverse visual similarity relationships between the images, and the optimal kernel for diverse image similarity characterization can be approximated more accurately by using a linear combination of these basic image kernels with different importance. For a given image concept Cj , the diverse visual similarity contexts between its images can be characterized more precisely by using a mixture of these basic image kernels (i.e., mixture-of-kernels) [11, 15-17].
Efficient Large-Scale Image Data Set Exploration
κ(u, v) =
5
αi κi (ui , vi ),
i=1
5
αi = 1
115
(3)
i=1
where u and v are the visual features for two images in the given image concept Cj , ui and vi are their ith feature subset, αi ≥ 0 is the importance factor for the ith basic image kernel κi (ui , vi ).
3 Visual Concept Network Generation We determine the inter-concept visual similarity contexts for automatic visual concept network generation with the image features and kernels introduced above. The interconcept visual similarity context γ(Ci , Cj ) between the image concepts Ci and Cj can be determined by performing kernel canonical correlation analysis (KCCA) [18] on their image sets Si and Sj : γ(Ci , Cj ) =
θT κ(Si )κ(Sj )ϑ max θ, ϑ θT κ2 (Si )θ · ϑT κ2 (Sj )ϑ
(4)
The detailed explanation of the parameters can be found in our previous report of [22]. When large numbers of image concepts and their inter-concepts visual similarity contexts are available, they are used to construct a visual concept network. However, the strength of the inter-concept visual similarity contexts between some image concepts may be very weak, thus it is not necessary for each image concept to be linked with all the other image concepts on the visual concept network. Eliminating the weak interconcept links can increase the visibility of the image concepts of interest dramatically, but also allow our visual concept network to concentrate on the most significant interconcept visual similarity contexts. Based on this understanding, each image concept is automatically linked with the most relevant image concepts with larger values of the inter-concept visual similarity contexts γ(·, ·) (i.e., their values of γ(·, ·) are above a threshold δ = 0.65 in a scale from 0 to 1). Compared with Flickr distance [19], our algorithm for inter-concept visual similarity context determination have several advantages: (a) It can deal with the sparse distribution problem more effectively by using a mixture-of-kernels to achieve more precise characterization of diverse image similarity contexts in the high-dimensional multi-modal feature space; (b) By projecting the image sets for the image concepts into the same kernel space, our KCCA technique can achieve more precise characterization of the inter-concept visual similarity contexts.
4 Image Summarization Algorithm When the user explores inside of each concept, there are still thousands of images need to be displayed. In order to summarize the data set, we need to find the most representative images which is composed by a subset of the original data set. The summarization problem is eventually a subset selection problem and can be interpreted formally with mathematical terms as follows. Given a image data set V of N images, our goal is to
116
C. Yang et al.
find a subset S ⊂ V that can best represent the original data set V. We introduce the quality term Qv for each v ∈ V expressing its capability to represent the entire data set. The vi with the highest value is considered as the best candidate to be a summarization of V and thus can be added into S, we name the set of S as the “summarization pool”. Traditional image summarization model will partition or cluster the data set and find the summarization image within each cluster. We propose to build a model that not only make use of the cluster information but also consider the inter-cluster relationship of the data and then build a global objective function. We apply the affinity propagation algorithm to cluster the data set into several clusters and record the size, mean and variance value of each cluster. Affinity propagation has demonstrated its superiority in automatic number of clusters, faster converge speed and more accurate result compared with other methods, like k-means. The similarity measurement used in affinity propagation is defined in Eqn (1). For our proposed image summarization algorithm, we go through each element in V for the best Qv to be added as a candidate summarization. The representativeness of an image v can be reflected by the following three aspects: 1. Relevancy: the relevance score of v is determined by the size, mean and variance value of the cluster that v belongs to, for which v ∈ c(v). A candidate summarization comes from the cluster with big size, small variance and small distance value d(v, vmean ) to the mean. In other words, it should be most similar to other images. 2. Orthogonality: the orthogonality score penalizes candidates from same cluster, or it penalize candidates which are too similar to each. It is not recommended to select multiple candidates from one cluster, because these candidates tends to bring redundancy to the final summarization. 3. Uniformity: the uniformity score penalizes candidates appears to be an outlier of original data set. Although the outliers always show a unique perspective of the original set, or in other words, a high relevancy score, it should not be considered as a summarization. Base on the above criteria, we formulate the final quality score as a linear combination of the three terms: ˆ Qv = R(v) + αO(v, S) + βU (v, S)
(5)
ˆ represent for the Relevancy, Orthogonality and Uniwhere R(v), O(v, S) and U (v, S) formity score respectively. We further define the formulation of the three terms as follows. For Relevancy score: R(v) =
|c(v)| ∗ Lc(v) σc(v) + d(v, μc(v) ) + 1
(6)
where c(v) denotes the cluster that contains v, or v ∈ c(v). |c(v)| is the number of elements in c(v), μc(v) and σc(v) are the mean and standard deviation of the cluster, Lc(v) is the number of linkage from v. Within each cluster, similar images are linked together. The similarity measurement is defined by SIFT feature as in Eqn (2). The similarity score above a pre-set threshold (0.6) is defined as a match. The matched image pairs are linked together, while un-matched pairs are not linked. Lc(v) can be also seen as the degree of v in the match graph. For Orthogonality score:
Efficient Large-Scale Image Data Set Exploration
O(v, S) =
−
J(v )
0 if J(v ) = ∅ if J(v ) = ∅
1 d(v,v )+2
117
(7)
where J(v ) = {v |c(v ) = c(v), v ∈ S}. J(v ) is empty if none of the elements v in S comes from the same cluster with v, or else, J(v ) is not empty and a penalty term will be applied. For Uniformity score: ˆ =− U (v, S)
1 d(v, μSˆ) + 3
(8)
ˆ where Sˆ = V \ S, μSˆ is the mean value of S. In the above terms, d(, ) is the Euclidean distance, i , i = 1, 2, 3 is a positive number small enough to make sure the denominator in the fraction is non-zero. Applying the formulation of Qv will give the best summarization of the concept set. For a fixed number of summarizations, |S| = k, we need to iterate the calculation of Qv for k times. The process of the proposed algorithm is close to greedy algorithm and can be summarized as follows: Algorithm 1. Cluster-based Image Summarization Algorithm 1: Initialization: S = ∅ 2: For each image v ∈ V, compute ˆ Qv = R(v) + αO(v, S) + βU(v, S) 3: The v with the maximum quality Qv is add into S S =S∪v 4: If stop criteria is satisfied, stop. Otherwise, V = V \v repeat from step 2.
At each iteration, the Qv is calculated for every v in V, and the best v is found and added into S. For fixed size summarization problem, the stop criteria is that S is enlarged into the pre-defined size. For automatic summarization problem, the stop criteria is that Qv is above a pre-defined value, and this value is 0 in our implementation. The parameter of α and β is determined experimentally and here in our implementation we have α = 0.3 and β = 0.4.
5 System Visualization and Evaluation For inter-concept exploration, to allow users to assess the coherence between the visual similarity contexts determined by our algorithm and their perceptions, it is very important to enable graphical representation and visualization of the visual concept network, so that users can obtain a good global overview of the visual similarity contexts between the image concepts at the first glance. It is also very attractive to enable
118
C. Yang et al.
Fig. 3. System User Interface: left: global visual concept network; right: cluster of the selected concept node
interactive visual concept network navigation and exploration according to the inherent inter-concept visual similarity contexts, so that users can easily assess the coherence with their perceptions. Based on these observations, our approach for visual concept network visualization exploited hyperbolic geometry [20]. The essence of our approach is to project the visual concept network onto a hyperbolic plane according to the inter-concept visual similarity contexts, and layout the visual concept network by mapping the relevant image concept nodes onto a circular display region. Thus our visual concept network visualization scheme takes the following steps: (a) The image concept nodes on the visual concept network are projected onto a hyperbolic plane according to their inter-concept visual similarity contexts by performing multi-dimensional scaling (MDS) [21] (b) After such similarity-preserving projection of the image concept nodes is obtained, Poincare disk model [20] is used to map the image concept nodes on the hyperbolic plane onto a 2D display coordinate. Poincare disk model maps the entire hyperbolic space onto an open unit circle, and produces a non-uniform mapping of the image concept nodes to the 2D display coordinate. The visualization results of our visual concept network are shown in Fig. 3, where each image concept is linked with multiple relevant image concepts with larger values of γ(·, ·). By visualizing large numbers of image concepts according to their inter-concept visual similarity contexts, our visual concept network can allow users to navigate large amounts of image concepts interactively according to their visual similarity contexts. For algorithm evaluation, we focus on assessing whether our visual similarity characterization techniques (i.e., mixture-of-kernels and KCCA) have good coherence with human perception. We have conducted both subjective and objective evaluations. For subjective evaluation, users are involved to explore our visual concept network and assess the visual similarity contexts between the concept pairs. In such an interactive visual concept network exploration procedure, users can score the coherence between the inter-topic visual similarity contexts provided by our visual concept network and their perceptions. For the user study listed in Table 2, 10 sample concept pairs are selected equidistantly from the indexed sequence of concept pairs. One can observe that our visual concept network has a good coherence with human perception on the underlying inter-concept visual similarity contexts.
Efficient Large-Scale Image Data Set Exploration
119
For objective evaluation, We find center concepts and their first-order neighbor as the clusters. By clustering the similar image concepts into the same concept cluster, it is able for us to deal with the issue of synonymous concepts effectively, e.g., multiple image concepts may share the same meaning for object and scene interpretation. Because only the inter-concept visual similarity contexts are used for concept clustering, one can observe that some of them may not semantic to human beings, thus it is very attractive to integrate both the inter-concept visual similarity contexts and their inter-concept semantic similarity contexts for concept clustering. As shown in Table 2, we have also compared our KCCA-based approach with Flickr distance approach [19] on inter-concept visual similarity context determination. The normalized distance to human perception is 0,92 and 1.42 respectively in terms of Euclidean distance, which means KCCA-base approach performs 54% better than Flickr distance on the random selected sample data. Table 1. Image concept clustering results group 1 urban-road, street-view, city-building, fire-engine, moped, brandenberg-gate, buildings group 2 knife, humming-bird, cruiser, spaghetti, sushi, grapes, escalator, chimpanzee group 3 electric-guitar, suv-car, fresco, crocodile, horse, billboard, waterfall, golf-cart group 4 bus, earing, t-shirt, school-bus, screwdriver, hammock, abacus, light-bulb, mosquito
Table 2. Evaluation results of perception coherence for inter-concept visual similarity context determination: KCCA and Flickr distances concept pair user score KCCA (γ) Flickr Distance urbanroad-streetview 0.76 0.99 0.0 cat-dog 0.78 0.81 1.0 frisbee-pizza 0.56 0.80 0.26 moped-bus 0.50 0.75 0.37 dolphin-cruiser 0.34 0.73 0.47 habor-outview 0.42 0.71 0.09 monkey-humanface 0.52 0.71 0.32 guitar-violin 0.72 0.71 0.54 lightbulb-firework 0.48 0.69 0.14 mango-broccoli 0.48 0.69 0.34
For intra-concept exploration, we simply display the top 5 image summarization results for each of the concept as shown in Fig.4. The user will have a direct impression about the visual properties of this concept. The accessability to the full concept set is also provided as shown in Fig.3. For image summarization evaluation task, efficiency, effectiveness, and satisfaction are three important metrics. We will design a user study based on these three metrics: 1. A group of 10 users were told to find top 5 summaries of three concepts as “Building”, “Spaghetti” and “Bus”; 2. Their manual selection results were gathered to find the ground truth summarization of the give concept by picking the image views with the highest votes.
120
C. Yang et al.
Fig. 4. Top 5 summarization results for “Building”, “Spaghetti” and “Bus”
3. Users were guided to evaluate our summarization system and give a satisfaction feedback compared with their own understanding of the summarization of this concept. As a result, the user took around 50 seconds in average to find top 5 summarizations from the concept data set. Comparatively, given the affinity propagation clustering results. The calculation of top image summaries can be finished almost at real time, which shows a big advantage for large scale image summarization. Averagely, 2 out of 5 images generated by our algorithm coincided with the ground truth summarizations we derived from the users. We define “coincide” by a strong visual closeness to each other, even if the images may not be identical. Considering the size of the concept set, and the limited number of summarizations we derived, the performance is quite persuasive. After exploring our system, the user also provided satisfactory feedback on the user friendly operation interface, fast responding speed and reasonable return results.
6 Conclusion To deal with large-scale image collection exploration problem, we have proposed novel algorithms for inter-concept visual network generation and intra-concept image summarization. The visual network reflects the diverse inter-concept visual similarity relationships more precisely on a high-dimensional multi-modal feature space by incorporating multiple kernels and kernel canonical correlation analysis. The image summarization iteratively generates most representative images of the set by incorporating the global clustering information while computing the relevancy, orthogonality and uniformity term of the candidate summaries. We designed user interactive system to run the algorithms on a self-collected image data set. The experiment observed very good result and the user study provide positive feedback. This research is partly supported by NSFC-61075014 and NSFC-60875016, by the Program for New Century Excellent Talents in University under Grant NCET-07-0693, NCET-08-0458 and NCET-10-0071 and the Research Fund for the Doctoral Program of Higher Education of China (Grant No.20096102110025).
Efficient Large-Scale Image Data Set Exploration
121
References 1. Smeulders, A.W.M., Worring, M., Santini, S., Gupta, A., Jain, R.: Content-based image retrieval at the end of the early years. IEEE Trans. PAMI 22(12), 1349–1380 (2000) 2. Hauptmann, A., Yan, R., Lin, W.-H., Christel, M., Wactlar, H.: Can high-level concepts fill the semantic gap in video retrieval? A case study with broadcast news. IEEE Trans. on Multimedia 9(5), 958–966 (2007) 3. Benitez, A.B., Smith, J.R., Chang, S.-F.: MediaNet: A multimedia information network for knowledge representation. In: Proc. SPIE, vol. 4210 (2000) 4. Naphade, M., Smith, J.R., Tesic, J., Chang, S.-F., Hsu, W., Kennedy, L., Hauptmann, A., Curtis, J.: Large-scale concept ontology for multimedia. IEEE Multimedia (2006) 5. Cilibrasi, R., Vitanyi, P.: The Google similarity distance. IEEE Trans. Knowledge and Data Engineering 19 (2007) 6. Fellbaum, C.: WordNet: An Electronic Lexical Database. MIT Press, Boston (1998) 7. Wu, L., Hua, X.-S., Yu, N., Ma, W.-Y., Li, S.: Flickr distance. In: ACM Multimedia (2008) 8. Jing, Y., Baluja, S., Rowley, H.: Canonical image selection from the web. In: Proceedings of the 6th ACM International CIVR, Amsterdam, The Netherlands, pp. 280–287 (2007) 9. Simon, I., Snavely, N., Seitz, S.M.: Scene Summarization for online Image Collections. In: ICCV 2007 (2007) 10. Fan, J., Gao, Y., Luo, H.: Hierarchical classification for automatic image annotation. In: ACM SIGIR, Amsterdam, pp. 11–118 (2007) 11. Gao, Y., Peng, J., Luo, H., Keim, D., Fan, J.: An Interactive Approach for Filtering out Junk Images from Keyword-Based Google Search Results. IEEE Trans on Circuits and Systems for Video Technology 19(10) (2009) 12. Lowe, D.: Distinctive image features from scale invariant keypoints. Intl Journal of Computer Vision 60, 91–110 (2004) 13. Grin, G., Holub, A., Perona, P.: Caltech-256 object category dataset, Technical Report 7694, California Institute of Technology (2007) 14. Bay, H., Ess, A., Tuytelaars, T., Van Gool, L.: Speeded-up robust features (surf). Comput. Vis. Image Underst. 110(3), 346–359 (2008) 15. Sonnenburg, S., R¨atsch, G., Sch¨afer, C., Sch¨olkopf, B.: Large scale multiple kernel learning. Journal of Machine Learning Research 7, 1531–1565 (2006) 16. Zhang, J., Marszalek, M., Lazebnik, S., Schmid, C.: Local features and kernels for classification of texture and object categories: A comprehensive study. Intl Journal of Computer Vision 73(2), 213–238 (2007) 17. Torralba, A., Murphy, K.P., Freeman, W.T.: Sharing features: efficient boosting procedures for multiclass object detection. In: IEEE CVPR (2004) 18. Huang, J., Kumar, S.R., Zabih, R.: An automatic hierarchical image classification scheme. In: ACM Multimedia, Bristol, UK (1998) 19. Vasconcelos, N.: “Image indexing with mixture hierarchies. In: IEEE CVPR (2001) 20. Barnard, K., Forsyth, D.: Learning the semantics of words and pictures. In: IEEE ICCV, pp. 408–415 (2001) 21. Naphade, M., Huang, T.S.: A probabilistic framework for semantic video indexing, filterig and retrieval. IEEE Trans. on Multimedia 3(1), 141–151 (2001) 22. Yang, C., Luo, H., Fan, J.: Generating visual concept network from large-scale weaklytagged images. In: Advance in Multimedia Modeling (2010)
A Study in User-Centered Design and Evaluation of Mental Tasks for BCI Danny Plass-Oude Bos, Mannes Poel, and Anton Nijholt University of Twente, Faculty of EEMCS, PO Box 217, 7500 AE Enschede, The Netherlands {d.plass,m.poel,a.nijholt}@ewi.utwente.nl
Abstract. Current brain-computer interfacing (BCI) research focuses on detection performance, speed, and bit rates. However, this is only a small part of what is important to the user. From human-computer interaction (HCI) research, we can apply the paradigms of user-centered design and evaluation, to improve the usability and user experience. Involving the users in the design process may also help in moving beyond the limited mental tasks that are currently common in BCI systems. To illustrate the usefulness of these methods to BCI, we involved potential users in the design process of a BCI system, resulting in three new mental tasks. The experience of using these mental tasks was then evaluated within a prototype BCI system using a commercial online role-playing game. Results indicate that user preference for certain mental tasks is primarily based on the recognition of brain activity by the system, and secondly on the ease of executing the task. Keywords: user-centered design, evaluation, brain-computer interfacing, multimodal interaction, games.
1
Introduction
The research field of brain-computer interfaces (BCI) originates from the wish to provide fully-paralyzed people with a new output channel to enable them to interact with the outside world, despite their handicap. As the technology is getting better, the question arises whether BCI could also be beneficial for healthy users in some way, for example, by improving quality of life or by providing private, handsfree interaction [11,18]. There are still a lot of issues yet to solve, such as delays, bad mental task recognition rates, long training times, and cumbersome hardware [8]. Current BCI research concentrates on improving the recognition accuracy and speed, which are two important factors of how BCI systems are experienced. On the other hand there is a lot of interest for making BCI a more usable technology for healthy users [1,13]. But in order for this technology to be accepted by the general public, other factors of usability and user experience have to be taken into account as well [14,19]. There is some tentative research in this direction, such as Ko et al. who evaluated the convenience, fun and intuitiveness of a BCI game they developed [6], and Friedman K.-T. Lee et al. (Eds.): MMM 2011, Part II, LNCS 6524, pp. 122–134, 2011. c Springer-Verlag Berlin Heidelberg 2011
A Study in User-Centered Design and Evaluation of Mental Tasks for BCI
123
et al. who looked into the level of presence experienced in two different navigation experiments [3]. But a lot of research still needs to be conducted. In this paper we focus on applying HCI principles to BCI design. In Section 2 we apply a user-centered design method to the selection of mental tasks for shapeshifting in the very popular massively-multiplayer online role-playing game R R , developed by Blizzard Entertainment, Inc . Within the World of Warcraft large group of healthy users, gamers are an interesting target group. Fed by a hunger for novelty and challenges, gamers are often early adopters of new paradigms [12]. Besides, it is suggested that users will be able to stay motivated and focused for longer periods if the BCI experiment can be presented in a game format [4]. Afterwards, in Section 3 and 4 the selected mental tasks are evaluated in a user study; in this evaluation the focus was on the user preferences – which among others include recognition, immersion, effort, ease of use – for the designed mental tasks. The main research questions we try to answer are: Which mental tasks do the users prefer, and why? How may this preference be influenced by the detection performance of the system? The results are discussed in Section 5 and conclusions of this user-centered design and evaluation can be found in Section 6.
2
User-Centered Design: What Users Want
One of the problems facing BCI research is the uncovering of usable mental tasks that trigger detectible brain activity. The tasks (by convention indicated by the name of the corresponding brain activity) that are currently most popular are: slow cortical potentials, imaginary movement, P300, and steady-state visuallyevoked potentials [15,17]. Users regularly indicate that these tasks are either too slow, nonintuitive, cumbersome, or just annoying to use for control BCIs [10,12]. Current commercial applications are a lot more complex and offer a lot more interaction possibilities than applications used in BCI research. Whereas current game controllers have over twelve dimensions of input, BCI games are generally limited to one or two-dimensional controls. Also, the mental tasks that are available are limited in their applicability for intuitive interaction. New mental tasks are needed that can be mapped in an intuitive manner with the system action. One way to discover mental tasks that are suitable from a user perspective is to simply ask the user what they would like to do to trigger certain actions. R In World of Warcraft , the user can play an elf druid who can shape-shift into animal forms. As an elf, the player can cast spells to attack or to heal. When in bear form, the player can no longer use most spells, but is stronger and better protected against direct attacks, which is good for close combat. R players of varying In an open interview, we asked four World of Warcraft expertise and ages what mental tasks they would prefer to use to shape-shift from the initial elf form to bear, and back again. The participants were not informed about the limits of current BCI systems, but most people did need an introduction to start thinking about tasks that would have a mental component.
124
D. Plass-Oude Bos, M. Poel, and A. Nijholt
They were asked to think of using the action in the game, and what it means to them, what it means to their character in the game, what they think about when doing it, what they think when they want to use the action, and then to come up with mental tasks that fit naturally with their gameplay. The ideas that the players came up with can be grouped into three categories. For the user evaluation of these mental tasks in Section 3, these three categories were translated into concrete mental tasks, mapped to the in-game action of shapeshifting. Each category consists of a task, and its reverse, to accommodate the shapeshifting action in the directions of both bear and elf form. 1. Inner speech: recite a mental spell to change into one form or the other. The texts of spells subsequently used were derived from expressions already used in the game world. The user had to mentally recite “I call upon the great bear spirit” to change to bear. “Let the balance be restored” was the expression used to change back to elf form. 2. Association: think about or feel like the form you want to become. Concretely, this means the user had to feel like a bear to change into a bear, and to feel like an elf to change into an elf. 3. Mental state: automatically change into a bear when the situation demands it. When you are attacked, the resulting stress could function as a trigger. For the next step of this research, this had to be translated into a task that the users could also perform consciously. To change to bear form the users had to make themselves feel stressed; to shift into elf form, relaxed.
3
User Evaluation Methodology
The goal of the user evaluation was to answer the following questions in this game context: Which mental tasks do the users prefer, and why? How may this preference be influenced by the detection performance of the system? Fourteen healthy participants (average age 27, ranging from 15 to 56; 4 female) participated in the experiment voluntarily. All but one of the participants were right-handed. Highest finished education ranged from elementary school to a R master’s degree. Experience with the application World of Warcraft ranged from “I never play any games” to “I raid daily with my level 80 druid”. Three participants were actively playing on a weekly basis. A written informed consent was obtained from all participants. The general methodology to answer these questions was as follows. In order to measure the influence of the detection performance of the system the participants were divided in two groups, a so-called “real-BCI” and “utopia-BCI” R with “utopia-BCI” decided group. The group that played World of Warcraft for themselves whether they performed the mental task correctly, and pressed the button to shapeshift when they had. In this way a BCI system with 100% detection performance (an utopia) was simulated. The group that played World R of Warcraft with “real-BCI” actually controlled their shapeshifting action with their mental tasks, at least insofar as the system could detect it.
A Study in User-Centered Design and Evaluation of Mental Tasks for BCI
125
The participants came in for experiments once a week for five weeks, in order to track potential changes over time. During an experiment, for each pair (change to bear, to elf) of mental tasks (inner speech, association, and mental state), the participant underwent a training and game session and filled in questionnaires to evaluate the user experience. The following sections explain each part of the methods in more detail. 3.1
Weekly Sessions and Measurements
The participants participated in five experiments, lasting about two hours each, over five weeks. The mental tasks, mentioned above, were evaluated in random order to eliminate any potential order effects, for example, due to fatigue or user learning. For each task pair, a training session was done. The purpose of the training session was manifold: it gathered clean data to evaluate the recognizability of the brain activity related to the mental tasks, the user was trained in performing the mental tasks, the system was trained for those participants who play the game with the real BCI system, and the user experience could be evaluated outside the game context. A training session consisted of two sets of trials with a break in between during which the participant could relax. Each set started with four watch-only trials (two per mental task), followed by 24 do-task trials (twelve per mental task). The trial sequence constituted of five seconds watching the character in their start form, followed by two seconds during which the shape-shifting task was presented. After this the participant had ten seconds to perform the mental task repeatedly until the time was up, or just watch if it was a watch-only trial. At the end of these ten seconds, the participant saw the character transform. See Figure 1 for a visualization of the trial sequence. During the watch-only trials, the participant saw exactly what they would see during the do-task trials, but they were asked only to watch the sequence. This watch-only data was recorded so it could function as a baseline. It is possible that simply watching the sequence already induces certain discriminable brain activity. This data provides the possibility to test this. The character in the videos was viewed from the back, similar to the way the participant would see the avatar in the game. At the end of the training session, the participant was asked to fill in forms to evaluate the user experience. The user experience questionnaire was loosely based on the Game Experience Questionnaire [5]. It contained statements for which the participants had to indicate their amount of agreement on a five-point Likert scale, for example: “I could perform these mental tasks easily”, “It was tiring to use these mental tasks”, and “It was fun to use these mental tasks”. The statements can be categorized into the following groups: whether the task was easy, doable, fun, intuitive, tiring to execute, whether they felt positive or negative about the task, and whether the mapping to the in-game action made sense or not. After the training session, the participant had roughly eight minutes to try the set of mental tasks in the game environment. The first experiment consisted only
126
D. Plass-Oude Bos, M. Poel, and A. Nijholt
Fig. 1. Training session trial sequence: first the character is shown in their start form, then the task is presented, after which there is a period during which this task can be performed. At the end the animation for the shape-shift is shown.
Fig. 2. Orange feedback bar with thresholds. The user has to go below 0.3 to change to elf, and above 0.7 to change to bear. In between no action is performed.
of a training session. For weeks two up to and including four, the participants were split up into the “real-BCI” and “utopia-BCI” group. Groups were fixed for the total experiment. During the last week all participants followed a training and played the game with the real BCI system. The “real-BCI” group received feedback on the recognition of their mental tasks in the form of an orange bar in the game (see Figure 2). The smaller the bar, the more the system had detected the mental task related to elf form. The larger, the more the system had interpreted the brain activity as related to bear form. When the thresholds were crossed the shape-shift action was executed automatically. The “utopia-BCI” group participants had to interact with a BCI system with (a near) 100% performance. Since this is not technical feasible yet, one could rely on a Wizard of Oz technique [16]. The users wear the EEG cap and have to perform the mental tasks when they wanted to shapeshift and the Wizard decides when it was performed correctly. In this case, however, the Wizard would have no way of knowing what the user is doing as there is no external expression of the task. The only option left to simulate a perfect system is to let the participants evaluate themselves whether or not they had performed the task correctly. Then they pressed the shape-shift button in the game manually. At the end of the session, the user experience questionnaire was repeated, to determine potential differences between the training and game sessions. The game session questionnaire contained an extra question to determine the perceived detection performance of the mental tasks. A final form concerning the experiment session as a whole was filled in at the end of the session. The participants were asked to put the mental tasks in order of preference, and to indicate why they choose this particular ordering.
A Study in User-Centered Design and Evaluation of Mental Tasks for BCI
3.2
127
EEG Analysis and Mapping
The EEG analysis pipeline, programmed in Python, was kept very general, as there was no certainty about how to detect these selected mental tasks. Common Average Reference was used as a spatial filter, in order to improve the signalto-noise ratio [9]. The bandpass filter was set to 1–80Hz. The data gathered during the training session was sliced in 10-second windows. These samples were whitened [2], and the variance was computed as an indication of the power in the window. A support vector machines (SVM) classifier trained on the training session data provided different weights for each of the EEG channels. To make the BCI control more robust to artifacts, two methods were applied to the mapping of classification results to in-game actions. A short dwelling was required to trigger the shape-shift, so it would not be activated by quick peaks in power. Secondly, hysteresis was applied: the threshold that needed to be crossed to change into a bear was higher than the threshold required to revert back to elf form. In between these two thresholds was a neutral zone in which no action was performed, see also Figure 2.
4 4.1
Results Which Mental Tasks Do Users Prefer and Why?
In the post-experiment questionnaire, the participants were asked to list the mental tasks in order of preference. The place in this list was used as a preference score, where a value of 1 indicated first choice, and 6 is the least preferable. These values were rescaled to match the user experience questionnaire values: ranging from 1 to 5, with 5 most preferable, and therefore 3.0 indicates a neutral disposition in preference order. Sixty-nine measurements were obtained from 14 participants over five weeks. One week one participant had to leave early and could not fill in his preference questionnaire. The average preference scores show a general preference for the association tasks, and the mental state seems to be disliked the most. But this paints a very simplistic image, as there are large differences between the “real-BCI” and “utopia-BCI” groups. To better understand the effects of the different aspects, Figure 3 shows the preference and user experience scores for each of the three mental task pairs, separate for the two participant groups. Whereas for the “real-BCI” group the mental state tasks are most liked, for the ”Utopia BCI” group, they are most disliked. Similarly, The “utopia-BCI” group most preferred inner speech, which was least preferred by the “real-BCI” group. Because of these large differences, these two groups need to be investigated as two separate conditions. 4.2
What Is the Influence of Recognition Performance on Task Preference?
Although it is not possible to completely separate the influence of the recognition performance and other aspects that differ between the participant groups, based
128
D. Plass-Oude Bos, M. Poel, and A. Nijholt
Fig. 3. User experience, preference, and perceived performance scores for the “utopiaBCI” and “real-BCI” groups, separate for the three mental task pairs, averaged for weeks 2 to 4. The bar plot is annotated with significant differences between task pairs (association, inner speech, mental state; with a line with star above the two pairs) and game conditions (“utopia-BCI”, “real-BCI”; indicated by a star above the bar).
A Study in User-Centered Design and Evaluation of Mental Tasks for BCI
129
on the user experience scores, recognition perception scores, and the words the participants used to describe their reasoning for their preference, it is possible to explain the discrepancy between the two conditions and get an idea of the influence of recognition performance. Inner speech is preferred by the “utopia-BCI” group, mainly because it is considered easy and doable. Although the inner speech tasks were rated highly by both groups, the system recognition had a heavy impact: it is the least preferred task pair for the “real-BCI” participants. The association tasks are valued mostly for their intuitiveness and the mapping with the in-game task, by the “utopia-BCI” group. Where the bad detection of inner speech mainly affected the preference scores, for association the user experience is significantly different on multiple aspects: easy, intuitive, and positive. The opposite happens for mental state. This task pair scores low across the board by both groups, yet it was preferred by the “real-BCI” group. It was also the task that was best recognized by the system, which is reflected in the perceived recognition scores. Based on these results, it seems that the recognition performance has a strong influence on the user preference, which is the most important consideration for the “real-BCI” group. For the “utopia-BCI” group different considerations emerge, where the ease of the task pair seems to play a dominant role, followed by the intuitiveness.
Fig. 4. Counts for the categories of words used to describe the reasoning behind the participant’s preference ranking, total for weeks 2 to 4
This view is confirmed by looking at the reasons the participants describe for their preference ranking, see Figure 4. The words they used were categorized, and the number of occurrences within each category was used as an indication of how important that category was to the reasoning. To reduce the number of categories, words that indicated a direct opposite and words that indicated a similar reason were merged in one category. For example, difficult was recoded to easy, and tiring was recoded to effort. In the words used by the “real-BCI”
130
D. Plass-Oude Bos, M. Poel, and A. Nijholt
group, recognition performance is the most used (n = 15), more than twice as any other word category (n <= 7). The ’Utopia BCI’ group mostly referred to the ease of executing the task (n = 12, where n <= 5 for the other word categories). Other issues that were often mentioned were effort, feels good, and speed. 4.3
Correlations between Preference and User Experience
Given the fact that participants indicated task recognition and ease to be the most important considerations for their preference, do these aspects from the user experience questionnaire also show a correlation to the preference scores? Weeks 1 and 5 were excluded from this analysis as both groups perform the tasks in the same conditions in these weeks (“Utopia” in week 1 and “Real” in week 2). Therefore the number of samples for the correlation tests are 63 (3 weeks, 3 task pairs, 7 participants), except for one case where there are some missing samples due to a participant having to leave early. For the correlation tests with the two conditions (“Real” and “Utopia”) combined, there are twice the number of samples. There are no perceived recognition scores for the “utopiaBCI” group.
Table 1. Pearson correlation coefficients and p-values for the correlation of user experience componenents and perceived recognition rate with the preference scores for the mental task pairs. Significance annotation: p <= 0, 005 in bold. doable easy fun intuitive mapping posit. negat. concentr. tiring recogn. Utopia r -,482 -,542 -,478 -,384 -,395 -,579 ,297 ,273 ,254 p ,000 ,000 ,000 ,002 ,002 ,000 ,019 ,032 ,047 Real r -,218 -,293 -,294 ,057 ,079 -,189 ,201 ,246 ,137 -,316 p ,095 ,023 ,023 ,665 ,550 ,147 ,124 ,058 ,295 ,014 All r -,360 -,415 -,391 -,203 -,190 -,410 ,252 ,259 ,199 -,316 p ,000 ,000 ,000 ,025 ,036 ,000 ,005 ,004 ,028 ,014
For the “utopia-BCI” group, the expected correlations with the ease-related aspects doable and easy were found, as well as correlations with most of the other aspects. After correction for multiple tests, the correlations with concentration and tiring are not significant. There were no significant correlations of preference with any of the user experience aspects for the “real-BCI” group, but the most relevant correlation was with the perceived recognition performance by the system, which is as expected. Looking at the conditions combined, the most significant correlations are for doable, easy, fun, and positive. The relation between preference and recognition performance is not that apparent when investigating this correlation. Yet, the correlations do show the importance of easy, expressed in how easy and how doable it is to perform the task. They also show that other aspects can be important as well, such as fun, how intuitive it is, and the mapping to the in-game action.
A Study in User-Centered Design and Evaluation of Mental Tasks for BCI
5
131
Discussion
The participants were varied in both age and experience with the game. It would be interesting to investigate the influence of these aspects on the preference and user experience. The participants in the “utopia-BCI” group had to evaluate their execution of the mental tasks themselves. This means we had no control over whether they were really performing the tasks seriously. It is unsure how well people can evaluate their own task execution and how this may affect their experience. This is a problem in any methodology that relies on self-evaluation in any part of the experiment. Nonetheless, we assume that participants really did perform the mental tasks: during the game sessions short delays were perceived where the participants seemed to pause, after which the button was pressed. For the “utopia-BCI” participants the brain activity has been recorded as well, which would have been another motivator. The recordings cannot be used to be certain about the task execution as the execution may be different for different participants, and it is unsure whether there are clear differences between the brain signals for the task pairs to begin with. Aspects that were indicated by the participants to influence preference ratings overall, in order of number of mentions from high to low, are: ease, recognition, feels good. Certain terms could be grouped because of similarity, like ease and effort, which would make the results more clear. As expected, the perceived recognition performance of the system was low. This fact could have a big impact on the interpretation of the scores of the “real-BCI” group. It is however interesting to note that for the mental state task, the participants did feel the system responded to their mental actions. Over time, the perceived recognition increased, and the recognition scores for the assocation tasks also rise above the neutral response. Secondly, the participants have a tendency to be optimistic about the system recognition. A score of 1 would indicate really no recognition at all, and even though the recognition for inner speech for example would have been minimal, the average rating is still above 2. This optimism about self-assessment of the level of control you have is know from other studies [7]. This analysis focused on the features derived from explicit subject reporting, to show how valuable this information can be, or at least to indicate the importance of the user in it all. This information should preferably be combined with more objective measures, such as the speed and accuracy of the BCI system mentioned earlier. The goal when evaluating BCI systems should be to obtain a complete image in order to discover where the main problems lie. If the mental task is disliked, then perhaps it might be better to look into other tasks that are suitable for the application instead of trying to gloss it over with recognition performance improvements. It is also important to keep in mind that these findings come from a limited R context: the shapeshifting task in World of Warcraft . The task itself could have had impact on what the participants value most in a mental task. It is not possible to see task preference independent of the in-game action as they are
132
D. Plass-Oude Bos, M. Poel, and A. Nijholt
inherently linked. If the game is fast-paced, tasks that can be executed more quickly will be more suitable, both on a functional and experiential level. If a task matches the in-game situation it has an advantage compared to a task that is similar in all other aspects. If a task will be executed often the effort and time of execution become more important than when it is only to be done every now and then. Finally, the point was made that healthy users will be less accepting than patients of the problems that current BCIs still struggle with. But it is important to consider that patients will also benefit from usability improvements.
6
Conclusions
When evaluating BCIs, current research only pays attention to task recognition performance, speed, and the derivative: bit rates. Human-computer interaction research shows that for a user to accept and value this new means of interaction other aspects may be important as well, generally summarized as usability and user experience. For this research, we involved potential users in the design process of determining which mental tasks to use for certain actions within the application. This resulted in three categories of mental tasks that are not listed among the most frequently used tasks in current BCI applications, and which are appreciated by the users (mental state, inner speech, and association). The three pairs of mental tasks uncovered in the design process were evaluated based on user experience with the help of our prototype system. By asking the participants about their experience, we gained new insights into what potential users like or dislike about these particular mental tasks for this BCI system and why. In the context of this experiment, user preference for mental tasks seems to be based on (in order of preference) accuracy of task recognition by the system, ease of performing the mental task, and lastly by factors such as fun, intuitiveness, and suitability for the task. The fact that the recognition by the system was indicated to be so important to the participants in this experiment could validate the current focus of BCI research on speed and accuracy. To prove such a generalization, further research is required. However, though the speed and accuracy of detection for a given mental task can be improved with better hardware and analysis methods, the mental task itself remains the same. When the performance of BCI systems gets better, other aspects of user experience , such as ease, fun and intuitiveness, will play a more prominent role. When designing and evaluating systems it would, both for patients and healthy users, be beneficial to take the usability and user experience into account.
Acknowledgements We would like to thank Betsy van Dijk for her thorough feedback, and we also gratefully acknowledge the support of the BrainGain Smart Mix Programme of
A Study in User-Centered Design and Evaluation of Mental Tasks for BCI
133
the Netherlands Ministry of Economic Affairs and the Netherlands Ministry of Education, Culture, and Science. This work was supported by the Information and Communication Technologies Coordination and Support Action “Future BNCI” within the FP7 framework, Project Number 248320.
References 1. Blankertz, B., Popescu, F., Krauledat, M., Fazli, S., Tangermann, M., M¨ uller, K.: Challenges for Brain-Computer Interface Research for Human-Computer Interaction Applications. In: ACM CHI Workshop on Brain-Computer Interfaces for HCI and Games (2008) 2. Dornhege, G., Blankertz, B., Krauledat, M., Losch, F., Curio, G., Muller, K.: Combined optimization of spatial and temporal filters for improving brain-computer interfacing. IEEE Transactions on Biomedical Engineering 53(11), 2274–2281 (2006) 3. Friedman, D., Leeb, R., Guger, C., Steed, A., Pfurtscheller, G., Slater, M.: Navigating virtual reality by thought: what is it like? Presence: Teleoperators and Virtual Environments 16(1), 100–110 (2007) 4. Graimann, B., Allison, B., Gr¨ aser, A.: New Applications for Non-invasive BrainComputer Interfaces and the Need for Engaging Training Environments. In: BRAINPLAY 2007 Brain-Computer Interfaces and Games Workshop at ACE (Advances in Computer Entertainment), pp. 25–28 (2007) 5. IJsselsteijn, W., de Kort, Y., Poels, K., Jurgelionis, A., Bellotti, F.: Characterising and measuring user experiences in digital games. In: International Conference on Advances in Computer Entertainment Technology (2007) 6. Ko, M., Bae, K., Oh, G., Ryu, T.: A study on new gameplay based on braincomputer interface. In: Barry, A., Helen, K., Tanya, K. (eds.) Breaking New Ground: Innovation in Games, Play, Practice and Theory: Proceedings of the 2009 Digital Games Research Association Conference, Brunel University (2009) 7. Langer, E.: The illusion of control. Journal of personality and social psychology 32(2), 311–328 (1975) 8. L´ecuyer, A., Lotte, F., Reilly, R.B., Leeb, R., Hirose, M., Slater, M.: BrainComputer Interfaces, Virtual Reality, and Videogames. Computer 41(10), 66–72 (2008) 9. McFarland, D., McCane, L., David, S., Wolpaw, J.: Spatial filter selection for EEG-based communication. Electroencephalography and clinical Neurophysiology 103(3), 386–394 (1997) 10. Molina, G.: Detection of High-Frequency Steady State Visual Evoked Potentials Using Phase Rectified Reconstruction. In: 16th European Signal Processing Conference, EUSIPCO 2008 (2008) 11. Nijholt, A., van Erp, J., Heylen, D.K.J.: BrainGain: BCI for HCI and Games. In: Proceedings AISB Symposium Brain Computer Interfaces and Human Computer Interaction: A Convergence of Ideas, pp. 32–35 (2008) 12. Nijholt, A., Tan, D., Pfurtscheller, G., Brunner, C., Mill´ an, J.d.R., Allison, B., Graimann, B., Popescu, F., Blankertz, B., M¨ uller, K.R.: Brain-computer interfacing for intelligent systems. IEEE Intelligent Systems pp. 76–83 (2008) 13. Nijholt, A., Tan, D., Mill´ an, J.d.R., Graimann, B., Jackson, M.: Brain-computer interfaces for HCI and games. In: Proceedings ACM CHI 2008: Art. Science. Balance, pp. 3925–3928 (2008)
134
D. Plass-Oude Bos, M. Poel, and A. Nijholt
14. Plass-Oude Bos, D., Reuderink, B., van de Laar, B., G¨ urk¨ ok, H., M¨ uhl, C., Poel, M., Nijholt, A., Heylen, D.: Brain-computer interfacing and games. In: Tan, D., Nijholt, A. (eds.) Brain-Computer Interfaces: Applying our Minds to Human-Computer Interaction, ch. 10. Springer, Heidelberg (2010) 15. Reuderink, B.: Games and Brain-Computer Interfaces: The State of the Art. Tech. Rep. TR-CTIT-08-81, Human Media Interaction, Faculty of EEMCS, University of Twente (2008) 16. Salber, D., Coutaz, J.: Applying the wizard of oz technique to the study of multimodal systems. Human-Computer Interaction 753, 219–230 (1993) 17. Tonet, O., Marinelli, M., Citi, L., Rossini, P., Rossini, L., Megali, G., Dario, P.: Defining brain–machine interface applications by matching interface performance with device requirements. Journal of Neuroscience Methods 167(1), 91–104 (2008) 18. Wolpaw, J.R., Loeb, G.E., Allison, B.Z., Donchin, E., Nascimento, O.F., Heetderks, W.J., Nijboer, F., Shain, W.G., Turner, J.N.: BCI meeting 2005 - workshop on signals and recording methods. IEEE Transactions on Neural System and Rehabilitation Engineering 14(2), 138–141 (2006) 19. Wolpaw, J., Birbaumer, N., McFarland, D., Pfurtscheller, G., Vaughan, T.: Brain-computer interfaces for communication and control. Clinical Neurophysiology 113(6), 767–791 (2002)
Video CooKing: Towards the Synthesis of Multimedia Cooking Recipes Keisuke Doman1 , Cheng Ying Kuai1, , Tomokazu Takahashi2 , Ichiro Ide1,3 , and Hiroshi Murase1 1
2
Graduate School of Information Science, Nagoya University Furo-cho, Chikusa-ku, Nagoya 464-8601, Japan {kdoman,kuai,ide,murase}@murase.m.is.nagoya-u.ac.jp Faculty of Economics and Information, Gifu Shotoku Gakuen University 1-38 Naka-Uzura, Gifu 500-8288, Japan
[email protected] 3 Institute for Informatics, University of Amsterdam Science Park 904, 1098 XH Amsterdam, The Netherlands
[email protected]
Abstract. In this paper, we propose the concept of synthesizing a multimedia cooking recipe from a text recipe and a database composed of video clips depicting cooking operations. A multimedia cooking recipe is a cooking recipe where each cooking operation is associated with a corresponding video clip depicting it, aimed to facilitate the understanding of cooking operations. In order to synthesize such a multimedia cooking recipe from an arbitrary text-based cooking recipe, a large number of video clips describing various cooking operations should be prepared in the database. Thus, we propose a method to build a database composed of video clips depicting cooking operations, which detects and classifies cooking operations in the cook shows, and tags them with cooking operations in a corresponding cooking recipe text. We also introduce a prototype multimedia cooking recipe interface named “Video CooKing” to demonstrate our concept. Keywords: cooking recipe, visualization, cooking video, cooking operation, motion analysis, automatic tagging.
1
Introduction
Recently, the number of cooking recipe texts posted on the Web is increasing. For example, “Cookpad”1 is a recipe-based social networking service where users can post original recipes and also report results and comments. It is so popular that it is said that one fourth of Japanese women in their thirties accesses this service. However, most of the cooking recipes on the Web are text-based and do not have enough explanations about cooking terms, especially cooking operations. 1
Currently at Brother Industries, Ltd., Japan. COOKPAD Inc., “COOKPAD,” http://cookpad.com/
K.-T. Lee et al. (Eds.): MMM 2011, Part II, LNCS 6524, pp. 135–145, 2011. c Springer-Verlag Berlin Heidelberg 2011
136
K. Doman et al.
Although some cooking recipes may have image-based explanations, they are not always sufficient for the understanding of some cooking operations. Therefore, in this paper, we propose the concept of synthesizing a multimedia cooking recipe from a text-based recipe, and also introduce a prototype interface, named “Video CooKing”. As shown in Fig. 1, a multimedia cooking recipe is a cooking recipe where each cooking operation is associated with a corresponding video clip depicting it. Compared with an existing text-based cooking recipe, the multimedia cooking recipe makes each cooking operation more understandable with a visual explanation. In fact, a cooking assistance software for a portable game machine “Nintendo DS”, that provides users with multimedia explanations of cooking recipes, is already commercially available2. However, to create such a software, visual explanations should be prepared manually in advance. We consider that it is unrealistic to manually prepare numerous video clips depicting cooking operations on various ingredients. To solve this problem, we are aiming at obtaining such video clips from cook shows automatically. A cook show contains video clips depicting each cooking operation in its corresponding cooking recipe, and is broadcast with closedcaptions. We obtain a set of tagged video clips depicting cooking operations from many cook shows, and consequently build a database with them. Once such a database is built, we can apply the method proposed in this paper to cooking recipes without corresponding cook shows, for example cooking recipes on the Web. Meanwhile, Hamada et al. proposed the “Cooking Navi” system that analyzes the dependency structure in a cooking recipe text [2], aligns each step to a video segment obtained from a corresponding cook show [3], and also presents the steps along the dependency structure with the aligned video while a user cooks [4]. However, this system can generate a multimedia cooking recipe only when the cooking recipe has a corresponding cook show. This paper is organized as follows. The next section illustrates the method for associating a cooking recipe text with its corresponding cook show to build a database composed of video clips depicting cooking operations. Section 3 introduces a prototype multimedia cooking recipe interface named “Video CooKing”. The paper ends with a summary and discussion of future works in Section 4.
2
Composing a Database of Video Clips Depicting Cooking Operations
As shown in Fig. 2, the flow of associating a cooking recipe text with a cook show is composed of mainly three processes: 1) text processing, 2) image processing and 3) integration. The text processing part analyzes the cooking recipe text and the closed-captions (CC) to extract tags (a pair of an “ingredient” and a 2
Nintendo Co., Ltd., “It talks! DS Cooking Navi (in Japanese),” http://www.nintendo.co.jp/ds/a4vj/
Video CooKing: Towards the Synthesis of Multimedia Cooking Recipes
(Title)
137
(Title)
(Ingredients)
(Ingredients)
(Preparation Steps)
(Preparation Steps)
(a) Text-based cooking (b) Video clips of cooking recipe operations
(c) Multimedia cooking recipe
Fig. 1. Synthesis of a multimedia cooking recipe ((a) + (b) → (c))
Cooking recipe text
Text processing (“onion”, “slice”)
Closed-captions
Text processing
Cooking video
Image processing
Integration
Cooking show
Fig. 2. Process flow for tagging a video clip depicting cooking operations from a cook show
“cooking operation”). The image processing part extracts video clips depicting cooking operations from the cook show, and then classifies them. The integration part associates the tags with the video clips depicting cooking operations. Although the following explanation is based on the processing of Japanese cooking recipe text, it should be possible to be applied to other languages by a similar approach. Accordingly, some language-specific details are omitted in the following explanations for simplicity. 2.1
Text Processing: Extracting Cooking Operations and Corresponding Ingredients
In the text processing part, pairs of an “ingredient” and a “cooking operation” are extracted as tags for video clips depicting cooking operations from a cooking recipe text and CCs. The structure of a cooking recipe text and CCs is shown in Fig. 3. Generally, a cooking recipe is composed of “Ingredients” where all ingredients are listed, and “Preparation Steps” where cooking procedures are described as a numbered list. Meanwhile, CCs contain the transcript of the speech and the timing of its utterance in the audio track. The details of the process are described below.
138
K. Doman et al.
Analyzing the cooking recipe text and the CC. First, morphological analysis is applied to each sentence in “Ingredients” and “Preparation Steps” of the input cooking recipe text. Next, nouns (ingredients) that commonly appear in both “Ingredients” and “Preparation Steps”, together with verbs (cooking operations) are extracted from “Preparation Steps”, respectively. Then, considering the grammar, each cooking operation is associated with its target ingredients in “Preparation Steps”. As a result, pairs of an “ingredient” and a “cooking operation” are obtained from the cooking recipe text. Similarly, pairs of an “ingredient” and a “cooking operation” are obtained from the CC, too.
Time
(Title)
Speech transcript
(Ingredients)
(Preparation Steps)
(a) Cooking recipe text
(b) Closed-captions
Fig. 3. Structure of a cooking recipe text and closed-captions
2.2
Image Processing: Classifying the Video Clips Depicting Cooking Operations from the Cook Show
In general, as shown in Fig. 4, a “face shot” and a “hand shot” appear alternately in a cook show. The upper body of a person is captured in the face shot, whereas the hand of a person is zoomed-up in the hand shot. It is generally considered that the hand shot is more important for cooking assistance, since the closeup of the current state or a cooking operation is captured in the shot [6]. For this reason, we focus on the hand shot and classify scenes in them into the following three categories considering motion. – “Repetitious motions”: A scene containing motions that repeat several times. It is further classified into the following two categories: • “Converged”: A scene where the periodic changes of pixel values are observed in a specific area of a frame. (ex. cut) • “Distributed”: A scene where periodic changes of pixel values are observed in a wide area of a frame. (ex. fry, mix)
Video CooKing: Towards the Synthesis of Multimedia Cooking Recipes Cut
Cut Face shot
Cut Hand shot
Cut Face shot
139 Cut
Hand shot
t
Cooking operation scene
Face shot
Current state scene
Fig. 4. General structure of a cook show
– “Current state”: A scene containing no dynamic motion. (ex. stew, boil, “ingredient (noun)”) – “Other motions”: Other than the above. (ex. serve, season) The details of the classification of the scenes are described next. Classifying the hand shots. First, the input cook show is segmented into shots, and then only hand shots are extracted from them by the method proposed by Miura et al. [6]. Next, for each segment composed of several continuous frames in a hand shot, an eigenspace is constructed. Then each frame in the segment is projected onto the eigenspace. As the motion feature for the classification, we focus on the trajectory drawn in each eigenspace (Fig. 5 (center column)), especially that drawn on the first eigenaxis (Fig. 5 (right column)). Then, the trajectory of each segment is classified with the following conditions: ⎧ ⎪ ⎨ “Repetitious motions” “Current state” ⎪ ⎩ “Other motions”
if m ≥ θm if m < θm and Δr ≤ θΔr otherwise
(1)
where m is the number of peaks, Δr is the difference of the minimum and the maximum values of the trajectory, and θm and θΔr are the thresholds of m and Δr, respectively. Here, the peak is defined as a point that meets the following conditions: ⎧ g(t) − g(t + 1) ≥ θ1 , ⎪ ⎪ ⎪ ⎨ g(t) − g(t − 1) ≥ θ , 1 (2) ⎪ g(t + 1) − g(t + 2) ≥ θ2 , ⎪ ⎪ ⎩ g(t − 1) − g(t − 2) ≥ θ2
140
K. Doman et al.
0.1 0
-0.1 0.2
-0.2 -0.2
-0.1
1st eige0 n
0.1 0 xis na ige e 2nd -0.1
0.1
axis
0.2
1st eigen axis
3rd eigen axis
0.2
0.2
0.1
0
-0.1
-0.2
-0.2
0
64
128
191
255
191
255
191
255
Time [frame]
(a) Repetitious motions
0.1 0
-0.1 0.2
-0.2 -0.2
0.1 s axi n e eig 0
-0.1
1st eige0 n
-0.1 0.1
axis
0.2
-0.2
2nd
1st eigen axis
3rd eigen axis
0.2
0.2
0.1
0
-0.1
-0.2
0
64
128
Time [frame]
(b) Current state
0.1 0
-0.1 0.2
-0.2 -0.2
-0.1
1st eige0 n
0.1 0 xis na e g ei
-0.1 0.1
axis
0.2
-0.2
2nd
1st eigen axis
3rd eigen axis
0.2
0.2
0.1
0
-0. -0.1
-0. -0.2
0
64
128
Time [frame]
(c) Other motions Fig. 5. Example of the analysis results for each motion category. In each motion category, the left column is an input video clip, the center column is its trajectory on the eigenspace, the right column is the trajectory projected onto the first eigenaxis.
where g(t) is the value on the first eigenaxis at time t, θ1 and θ2 are the thresholds of the peak strengths. Finally, we regard a series of the segments classified into the same motion category as a scene. Classifying the repetitious motions. The second-stage of the classification is performed only for “repetitious motions”. First, frequency analysis is applied to continuous frames in an input scene, and then, after segmenting the frame into blocks, the temporal change of pixel values is calculated per block. Next, the number of repetitions in each block are counted, where an explicit peak at a certain frequency exists. Example of the results of the repetition count is shown in Fig. 6. Next, regarding each block as a sample point, eigenvalues λ1 , λ2 are calculated by applying PCA (Principal Component Analysis) to the distribution of repetition counts in a frame. As shown in Fig. 7, there is a clear difference between “converged” and “distributed” in the distribution of the repetition counts.
Video CooKing: Towards the Synthesis of Multimedia Cooking Recipes
141
Fig. 6. Example of the frequency analyses. The deeper the color, the larger the count.
(a) Converged
(b) Distributed
Fig. 7. Example of the results of PCA applied to the distribution of repetition counts in a frame. The dashed lines represent the first and the second eigenaxes. There is a clear difference between the distribution of repetition counts in “Converged” and “Distributed”.
Focusing on this, the input scene is classified further into two motion categories by the following condition:
“Converged” “Distributed”
if (λ1 − λ2 ) ≥ θλ otherwise
(3)
where θλ is the threshold of the variance on each axis. 2.3
Integration: Tagging the Video Clips Depicting Cooking Operations
Each video clip depicting a cooking operation classified according to the process described in 2.2 is associated with a tag (pair of an “ingredient” and a “cooking operation”) obtained according to the process described in 2.1 as follows: First,
142
K. Doman et al.
the pairs of an “ingredient” and a “cooking operation” that commonly appear in both cooking recipe text and CC are extracted by matching them. Next, each pair is tagged to a video clip depicting a cooking operation according to the time stamp in the CC. Here, only tags that correspond to the motion category of the video clip depicting a cooking operation are selectively tagged. By applying the method to many cook shows, we will obtain a set of tagged video clips depicting cooking operations, and consequently build a database composed of them. 2.4
Experiment
We evaluated the association performance of the method described above with the following experiment. Experimental conditions. Eight cooking recipes and corresponding cook shows3 (320 × 240 pixels, 30 fps and in total 75 min.) were used. A Japanese morphological analyzer MeCab4 was used to obtain the parts-of-speech of the terms. Cut detection and hand shot extraction from the cook shows were performed manually for this experiment, and the extracted hand shots without corresponging tags were excluded from the classification targets. We manually labeled each scene for the ground-truth. The size of the blocks and the window width for the classifications of “repetitious motions” were set to 16 × 16 pixels and 256 frames (about 8 seconds), respectively according to the result of a preliminary experiment. We evaluated the tagging accuracy based on two counting rules: 1) a tagging is judged as correct if a pair of an “ingredient” and a “cooking operation” is correctly tagged to a scene (pair-count) or 2) if only a “cooking operation” is correctly tagged (solo-count). Experimental results. 129 scenes were obtained as the result of the classification of the hand shots, and 135 pairs of an “ingredient” and a “cooking operation” were tagged to them. The tagging accuracy was 52.6% based on the pair-count rule, 68.1% based on the the solo-count rule. Discussions. In the text processing part, there were many mis-matchings of tags caused by different expressions of a similar cooking operation. We consider that we can cope with this problem by using a thesaurus of cooking terms. In the image processing part, we consider that there are two important points in order to obtain higher classification accuracy: 1) treatment of a scene with a large camerawork which yields a large motion, and 2) choice of the best thresholds for accurate classification of “repetitious motions” and “other motions”. In the 3 4
NHK Educational Corp., “Today’s cooking for everyone (in Japanese),” http://www.kyounoryouri.jp/. Kyoto University, “Japanese morphological analyzer MeCab,” http://mecab.sourceforge.net/
Video CooKing: Towards the Synthesis of Multimedia Cooking Recipes
143
integration part, most of the mis-taggings were caused by the existance of an operation that is not mainly-focused but captured in the frame (Fig. 8(a)), or operations or states that do not involve motion (Fig. 8(b)). As for the former (Fig. 8(a)), although the tagging result was (“water”, “boil”), the camerawork was focusing on dissolving scorch in the boiled water using a spoon. As for the latter (Fig. 8(b)), although the tagging result was (“egg”, “become hard”), we judged the tagging result as incorrect since such a cooking operation could not be observed as a motion. However, in view of the complexity of associating a cooking recipe text with a video clip depicting a cooking operation, we consider that the experimental results are sufficient for our purpose at the current stage, since we consider that the user may select what s/he needs from the video clips presented in an interface.
(a) Tagging result: (“water”, “boil”)
(b) Tagging hard”)
result:
(“egg”,
“become
Fig. 8. Examples of mis-tagging
3
Video CooKing: A Prototype Multimedia Cooking Recipe Interface
We implemented a prototype interface “Video CooKing” as shown in Fig. 9 to demonstrate the concept of the multimedia cooking recipe. In the interface, each cooking operation in “Preparation Steps” in the left column is linked with its one or more corresponding video clips depicting cooking operations. When a linked cooking operation is clicked, a list of ingredients that are the targets of the cooking operation is shown in the right column. Each ingredient is linked to a video clip depicting a cooking operation if there is more than one corresponding video clips that exist in the database. Users can play / stop the video clip themselves. We consider that this interface that enables an user to browse a multimedia cooking recipe, facilitates the understanding of cooking operations.
144
K. Doman et al.
Fig. 9. Video CooKing: a prototype multimedia cooking recipe interface. The original cooking recipe on the Web6 is shown in the left column, and video clips depicting cooking operations corresponding to each cooking operation in the recipe text is shown in the right column when clicked.
Video CooKing: Towards the Synthesis of Multimedia Cooking Recipes
4
145
Conclusion
In this paper, we proposed the concept of synthesizing a multimedia cooking recipe and also introuced a prototype interface. Experimental results showed the effectiveness of our method in tagging a pair of an “ingredient” and a “cooking operation” to a video clip obtained from a cook show, and consequently, the capability of building a database composed of video clips depicting cooking operations. Future work includes the improvement of the interface and the tagging [5]. Acknowledgments. Parts of this work were supported by the Grants-in-Aid for Scientific Research (21013022) from the Japanese Ministry of Education, Culture, Sports, Science and Technology. The “Media Integration Standard Toolkit”5 was used for the implementation of the system, and some thumbnail images from the “Video database for evaluating video processing” [1] was used for explanations in this paper.
References 1. Babaguchi, N., Etoh, M., Satoh, S., Adachi, J., Akutsu, A., Ariki, Y., Echigo, T., Shibata, M., Zen, H., Nakamura, Y., Minoh, M., Matsuyama, T.: Video database for evaluating video processing (in Japanese). Tech. Rep. IEICE (PRMU2002-30) (June 2002) 2. Hamada, R., Ide, I., Sakai, S., Tanaka, H.: Structural analysis of preparation steps on supplementary documents of cultural TV programs. In: Proc. Fourth Int. Workshop on Information Retrieval with Asian Languages (IRAL 1999), Taipei, Taiwan, pp. 43–47 (November 1999) 3. Hamada, R., Miura, K., Ide, I., Satoh, S., Sakai, S., Tanaka, H.: Multimedia integration for cooking video indexing. In: Aizawa, K., Nakamura, Y., Satoh, S. (eds.) PCM 2004. LNCS, vol. 3332, pp. 657–664. Springer, Heidelberg (2004) 4. Hamada, R., Okabe, J., Ide, I., Satoh, S., Sakai, S., Tanaka, H.: Cooking Navi: Assistant for daily cooking in kitchen. In: Proc. Thirteenth ACM Int. Multimedia Conf. (ACMMM 2005), Singapore, pp. 371–374 (November 2005) 5. Ide, I., Shidochi, Y., Nakamura, Y., Deguchi, D., Takahashi, T., Murase, H.: Multimedia supplementation to a cooking recipe text for facilitating its understanding to inexperienced users. In: The Second Workshop on Multimedia for Cooking and Eating Activities (CEA 2010) (December 2010) 6. Miura, K., Hamada, R., Ide, I., Sakai, S., Tanaka, H.: Motion based automatic abstraction of cooking videos. In: Proc. ACM Multimedia 2002 Workshop on Multimedia Information Retrieva, lMIR 2002 (December 2002)
5
Nagoya University, “Media Integration Standard Toolkit: MIST,” http://mist.murase.m.is.nagoya-u.ac.jp/
Snap2Read: Automatic Magazine Capturing and Analysis for Adaptive Mobile Reading Yu-Ming Hsu1, Yen-Liang Lin2, Winston H. Hsu1,2, and Brian Wang3 1
Department of Computer Science and Information Engineering, National Taiwan University 2 Institute of Networking and Multimedia, National Taiwan University 3 Institute for Information Industry, Taiwan {leafwind,yenliang}@cmlab.csie.ntu.edu.tw,
[email protected],
[email protected]
Abstract. The rise of electronic book services and the prevalence of mobile devices reveal the needs for mobile reading and the booming business opportunities in e-book developments. However, incompatible e-book readers, non-uniformed e-book formats, or even limited screen resolution causes the inconvenience of reading documents on handheld devices. Meanwhile, it is difficult to read physical magazines that do not have the corresponding digital copies. Therefore, we propose a system, Snap2Read, that can automatically segment the captured document images (i.e., from the physical magazines) in mobile phones into readable patches (e.g. text, title, image), and then scale them into suitable size so that users can easily browse the digitalized magazine pages via the mobile phone with simple clicks. Keywords: mobile reading, page segmentation.
1 Introduction When it comes to reading on handheld device, the presently-available e-book readers (e.g., “Kindle”) and their related software might come to our mind first. Although the release of Apple iPad heated up the competition fore-book readers, the companies that have survived (e.g., Amazon, B&N) stand firm because of their rich resources of digitalized book content. This shows us how important digital content is. Our work gives another possibility for mobile reading: automatically capture and analyze paper documents from physical magazines that users own and turn them into digitalized pages, adaptively readable in mobile devices (cf. Fig. 1). Users do not have to buy various versions of a certain e-book or their corresponding readers from different companies. As long as it is a paper book (i.e., physical magazine) they own, it can be captured (or scanned), cut into the proper size, and turned into the right format to be read on mobile device. Unlike the traditional process (scanning followed by OCR, i.e., Optical Character Recognition,) which limits reading (i.e., offer text format only, may have error, do not preserve layout, etc.), our approach can preserve the original appearance. Other than panning (i.e., dragging) the whole page while reading, not knowing the present location information and losing track of the reading progress, our system is much more flexible and humanized. K.-T. Lee et al. (Eds.): MMM 2011, Part II, LNCS 6524, pp. 146–156, 2011. © Springer-Verlag Berlin Heidelberg 2011
Snap2Read: Automatic Magazine Capturing and Analysis for Adaptive Mobile Reading
147
Fig. 1. System overview. (A) A user is interested in a physical magazine page and then takes a snapshot at it1. (B) Page Segmentation step decomposes the whole page into homogeneous blocks. (C) Zone Classification step predicts the labels on the segmented blocks. (D) Mobile Adaptation step changes the classified blocks into readable patches and further determines their reading sequence. (E) The user can then be guided to read by simple clicks.
Mobile interaction is a common topic in recent years. Erol et al. [1] tried to link the physical paper to digital documents. They utilized BWC (Brick Wall Coding) features to retrieve digital documents, and their application aimed at different types of annotation on retrieved document. Liao et al. [2] further provided fine-grained user interface for users, and then the retrieved digital content can be selected, copied or queried. However, they did not address the reading process of users on small-size screen devices. Putting an entire large resolution document on mobile to read may be considered annoying, but if the content can be rendered properly. Many researches have addressed this issue which focuses on web page browsing. Xie et al. [3] used both spatial and content features to learn a block importance model so that they can classify the “importance” of each segmented block, and then simply aggregate them to the final output list by ranking their importance. Hattori et al. [4] created an “object list” to link each segmented element and proved to be more efficient than commercial mobile web browsers. The main differences between our work and those works focused on web page reading are as follows. First, segmenting web pages usually takes the “content” into account (i.e., html tags, text information, etc.), while our inputs only have visual features from the snapshots. Second, their output segmented blocks is DOM (Document Object model) based, which can be further decoded as plain texts. Thus adapting the content to 1
We make an assumption that the image is clear and does not have image distortion resulted from shooting angle. To snap and read the paper is possible if the rectification is conducted correctly and through the help of camera sensors through our preliminary developments.
148
Y.-M. Hsu et al.
fit the screen has never been a problem, while for document images, the segmented blocks should be further split and padded for different screen resolutions. Third, they only create a simple viewing list for vertical direction reading and do not preserve the layout. By applying the page segmentation techniques on document image, we segment the document directly on the appearance, so the layout of our output does not change. There are many algorithms can be categorized into several classes in page segmentation researches. (See the reviews in [5].)They usually focus on skew tolerance, time efficiency or certain document types and most of them, unavoidably, need a certain fixed threshold, while our requirement is that the system can be tolerant to different types of layout structure and font size. The key contributions of this paper are: 1. To the best of our knowledge, Snap2read represents one of the first attempts that transform physical papers (especially magazines) into electronic readings, thus presenting another venue for mobile reading services. 2. To make reading experience more comfortable, suitably rendering the segmentation blocks on mobile devices is nontrivial. We employ the classification technique to enhance page adaptation results by knowing the block types. 3. We propose an adaptive morphological approach for page segmentation, which aims to structure captured magazines on mobile devices. 4. Experiments on hundreds of manually collected magazine pages (with segmentation ground truth) show the promising results of our proposed system. The detailed description of our method is stated at section 2. Section 3 shows the experimental results, and section 4 gives a conclusion and possible future works.
2 The Method The purpose of our system is to segment the document image into homogeneous blocks with the maximum size (Section 2.1 - Page Segmentation), to classify them into a set of predefined categories (Section 2.2 - Zone Classification), and finally to render them to fit for different screen resolutions (Section 2.3 - Mobile Adaptation). 2.1 Page Segmentation Previous approaches usually focus on one single language or specific types of documents. However, we cannot assume our input to be a certain type of magazines for reading activities on mobile devices, so we do not give any fixed parameters related to character font size, line spacing or layout structure, which traditional page segmentation methods do. Like other morphological methods, our method is a bottom-up approach [5]. The main idea is to group those small connected components into larger regions by dilation. However, we dilate iteratively and automatically select the appropriate dilation kernel.
Snap2Read: Automatic Magazine Capturing and Analysis for Adaptive Mobile Reading
149
2.1.1 Pre-processing The purpose of the preprocessing step is to filter out noises, dividing lines, and blocks that are possibly images. They tend to be merged with others during dilation, so we must make sure that other main components (mostly texts) will not be affected. The detailed steps: (1) To take efficiency into consideration, resize the image to a proper area measure (i.e., about 900*650). (2) Do global threshold binarization. (3) For those fore ground (i.e., white) pixels, apply connected component analysis. (4) For each component, if its proportion of height to width is significantly high (i.e., 20 times), then remove it from the original image. If the component is big enough (i.e., 1.25%) and its density, the total number of foreground pixels divided by the area size, is high enough (i.e., 0.1), it will be considered to be “possibly image block” and be extracted in advance. The examples are illustrated in Fig. 2.
(a)
(b)
(c)
(d)
Fig. 2. An illustration of the pre-processing steps: (a) Original image. (b) Binarization. (c) Connected component analysis. Components are in different colors. (d) Noise removal and image block pre-extraction.
Note that in the step (2), we have also tried edge detector and local threshold binarization method, which are widely used in document image processing [5]. However, although the former depicts the salient edges, it also turns complex image regions into many edge fragments. The latter is mostly used on scanned document images that only contain texts in order to make them robust to lighting change, but it, too, has the similar effect mentioned above on image regions. As a result, we decide to use global threshold binarization to preserve image blocks. 2.1.2 Adaptive Dilation Threshold In this work, we enlarge small components by dilation operation to group them together, and the square kernel is applied. It enlarges both vertical and horizontal regions of a component, but how to determine the kernel size is vital and nontrivial. A fact is that the main font size may change from one magazine to another, or from one language to another. Thus, using fixed dilation kernel size for segmentation may not work on every case, so we need to determine it dynamically. During the iterative dilation process, we find that the number of components will drop rapidly at a certain kernel size, and the most suitable size is at the turning point
150
Y.-M. Hsu et al.
(e.g., red arrow in Fig. 3).The physical meaning of this phenomenon is that a large number of characters and words are merged into lines or paragraphs at the same time. We then count the number of component versus kernel sizes (e.g. a blue line in figure 3), and the turning point can be found by applying approximate second order differential. To ensure that almost all components are sufficiently merged, we also add a constraint that the number of components should not be less than a certain number (i.e., 30). # of components 500
Kernel size
Fig. 3. The relationship of component numbers to kernel size, this figure includes20 pages (blue lines). The red arrow indicates the turning point which is suitable for the kernel size.
(a)
(b)
(c)
(d)
Fig. 4. Dilation for little blocks mergence and redundant blocks removal. (a) An example intermediate image from preprocessing step, in which title and footnote are split into several fragments. (b) Remove the large components and dilate again. (c) There exists some inside or non-informative block. (d) The final output example which does not contain those redundant blocks.
In the last step, minimum enclosing rectangles are extracted from those components which are large enough (i.e., at least0.3% of whole page area), but some remaining components are not merged well enough (over-segmentation) because the spacing is larger than main texts (e.g., title, footnote, etc.) Therefore, we use a 1.5-time-large kernel to dilate again for the remaining small components (cf. Fig. 4). Inside blocks and small blocks (i.e., less than 0.1% of whole page area) are also removed (cf. Fig. 4).
Snap2Read: Automatic Magazine Capturing and Analysis for Adaptive Mobile Reading
151
2.2 Zone Classification The rectangles (blocks) produced by the segmentation step above will be rearranged into meaningful “patches” in accordance with different screen resolutions of the device during the next adaptation step, while images and text blocks have different adaptations: Images cannot be further segmented, but text blocks can be split if the yare too large to accommodate themselves to the screen. Therefore, it is necessary to classify images and text blocks (cf. Fig. 1). 2.2.1 Features for Classification We use the early fusion scheme – multimodal features concatenated as a long one – to combine features so that we can learn a classifier by SVM (Support Vector Machine) [6] for each label (e.g., text, image). The three features we used are spatial feature, GCM (Grid Color Moment), and PHOG (Pyramid of Histograms of Orientation Gradients.)[7]Their detailed descriptions are as follows. Spatial feature contains the coordinate and size of a specified region (i.e., x, y, width and height). As for color feature (GCM), we adopt the first order moment (mean) and the second order moment (variance) for color feature. The image will be partitioned into several (i.e., 8*8) sub-blocks, and for each block, calculate its mean and variance values. As a result, the GCM feature is avector with 8 * 8 (blocks)* 3 (color planes) * 2 (moments) = 384 dimensions. The shape feature (PHOG descriptor [7]) represents the “local shape,” and the “spatial layout” of the image. To calculate PHOG, first extract edge contours by Canny edge detector, and the image is divided into 4l 44 sub-blocks at level l. The HOG of each grid at each pyramid resolution level is then calculated. In this paper, we set level up to 2 (i.e., l = 0, 1, 2) and 8 bins for HOG. Thus, by concatenating different level of resolutions of HOGs, it can be formulated as a vector representation with (40 + 41 + 42 ) *8 = 168 dimensions. Their dimensions are 4, 384 and 168, respectively. The concatenated feature vector is then measured 556 dimension, and each dimension will be scaled into [-1, +1]. 2.2.2 Model Selection We use the RBF (Radial Basis Function) kernel for classification, so there are two main parameters g and c to be determined (i.e., gamma and cost, respectively). In order to select the suitable model for prediction, we apply 5-fold cross validation on total 1430 page segments and get average accuracy around 0.95. 2.3 Mobile Adaptation Although the segmented blocks are composed with homogeneous components (cf. Fig. 6 and Fig. 7), they cannot be read directly because we extract them as large as possible for each region type, which may not fit the screen resolution, so we must adjust the blocks to readable patches. As mentioned above, only text blocks may need to be further split (e.g., the wide text block in Fig. 1(E).). Generally speaking, the English articles tend to stretch in vertical direction, while the Chinese articles tend to expand horizontally. Thus, we
152
Y.-M. Hsu et al.
adopt a heuristic approach (take English language as an example): For each block, we scale it along its width, and split it into several patches according to its height, and pad those patches whose height are not sufficient to prevent distortion. (cf. Fig. 5) As for the reading sequence, images will have higher priority than texts, and then we rank them from upper left corner to the lower right corner (for Chinese magazines, rank from upper right corner to lower left corner). We also provide a transparent overview window at the upper right corner of the mobile interface to indicate the current location on whole page. Thus users do not have to zoom-in and zoom-out repeatedly to obtain the geometric information.
1
2
1
2 3a
3 4
5 6
(a)
3b
4a
6a
4b 5
6b
4c
6c (b)
Fig. 5. An illustration of adaptation. (a) Original segmented blocks (b) The adapted (scaled, padded) patches. The reading sequence is 1, 2, 3a, 3b, 4a, 4b, etc…, and it is used to guide users so that they can read the page conveniently through clicks without losing track of page context.
3 Experimental Results This section describes our dataset and how we label the ground truth, and it also shows the evaluation of our work. 3.1 Magazine Dataset To the best of our knowledge, there is no public dataset for page segmentation evaluation. Previous page segmentation researches usually tend to use their own private dataset depending either on their target document genre (e.g., newspaper, journal), or on a specific language. Although we know that in recent years, ICDAR Page Segmentation Competition has created their own dataset with rich types of sources, only those who participate in the competition can gain access to the dataset. Further more, they do not provide documents in Asian languages (e.g. Chinese, Japanese, Korean), which do not have clear bounding boxes for each word, while we expect our system can work well on both type of languages. Thus we create a dataset on our own. We selected 4 different popular magazines: for Chinese language, “Common Wealth” and “Business Weekly” are adopted; we also take 2 English magazine named
Snap2Read: Automatic Magazine Capturing and Analysis for Adaptive Mobile Reading
153
“Business Week” and “Science”. For each magazine, we manually filter out advertisement pages and select 30 scanned pages, and the detail is listed in Table 1. To collect groundtruth, we use an editor named GEDI [8], a highly configurable document image annotation tool. It reads an image file, and when annotation is done, it produces a corresponding XML file in GEDI format. Table 1. Magazine dataset for segmentation and classification experiments Magazine Common Wealth Business Weekly Business Week Science
Language Chinese Chinese English English
# of pages 30 30 30 30
Scanned resolution 1184*1573 963*1280 944*1260 944*1203
3.2 Page Segmentation and Zone Classification Performance Because the evaluation metrics of the previous methods are usually computed pixelwise, which is aimed at OCR. However, our output is rectangle-based, which is aimed at locating reading patches. As a result, comparing the two does not make sense. Although we do not compare them directly, we adopt one of the most widely used metrics in ICDAR 2005 [9], and try to illustrate our performance with their intuition.We have annotated three types of entities (i.e., categories): text, image and footnote.For each entity, the EDM (Entity Detection Metric) is calculated. First, evaluate how much they overlapped between a ground truth zone and a result zone by keeping a global matrix MatchScore, which is defined by function ⎧ T (G j ∩ Ri ∩ I ) ⎫ , if ( g j = ri ) ⎪ ⎪ T ( G R I ) ∪ ∩ MatchScore(i, j ) = ⎨ ⎬ j i ⎪ ⎪ 0, otherwise ⎩ ⎭
(1)
Where I denotes all image pixels, Gj: all pixels inside the ground truth j, Ri: all pixels inside the result i, gj: the entity type of the ground truth j, ri: the entity type of the result i, and T(s):a function that counts the elements of sets. Second, three types of matches are defined (i.e., one-to-one, one-to-many and many-to-one) according to their MatchScore: If the MatchScore of ground truth zone j and result zone i is higher than the accept threshold (i.e., 0.6), then it is a one-to-one match. (See figure 8 for more explanation) If there are K ground truth zones jk (k = 1, 2…K) overlapping the same result zone i, and each of their MatchScore is between the accept threshold and the reject threshold (i.e., 0.1<MatchScore(i,jk) < 0.6, k = 1, 2…K), but their summation is higher than the accept threshold, then it is a many-to-one match, and vice versa. For simplicity, the acceptable matched number for each entity is defined as MatchNumber = (w1*one-to-one + w2*one-to-many + w3*many-to-one), where w1 = 1 and w2 = w3 = 0.75 for partial match penalty. Then DetectRatet and RecognAccuracyt for entityt are defined asDetectRatet = MatchNumber/Nt, and RecognAccuracyt = MatchNumbert/Mt. Nt is the number of ground truth regions for t’th entity, and Mt is the number of result regions for t’th entity. The DetectRatet and RecognAccuracyt
154
Y.-M. Hsu et al.
represent the acceptable ratio among all ground truth zones and all result zones for t’th entity, respectively. The Entity Detection Metric score for each entity (text, image, footnote) is then defined as
EDM t =
2* DetectRatet * RecognAccuracyt DetectRatet + RecognAccuracyt
(2)
There overall page segmentation performance is promising. See the breakdowns in Table 2. The page segmentation results are sampled in Fig. 6 and Fig. 7. We also
(a)
(b)
(c)
Fig. 6. Chinese magazine results. (The blue, red and green bounding boxes indicate text, image and footnote, respectively.) (a) A Common Wealth example (b) A Business Weeklyexample (c) Anover-segmentation examplewhich divides a flow chart intotext blocks.
(a)
(b)
(c)
Fig. 7. English magazine results. (a) A Business Week example (b) A Science example (c) An over-segmentation example results from figures with unclear bounding boxes.
Snap2Read: Automatic Magazine Capturing and Analysis for Adaptive Mobile Reading
155
found the results are satisfactory as rendering them in reading patches in two Android phones with different resolutions. The result of Business Weekly seems to have better performance than others (cf. Table 2(a).), because its layout is less complicated than others and its text block size is mostly large and rectangle shaped, while Business Week has lower performance on image category (cf. Table 2(b).) because it has a lot of figures and tables combined with text explanation inside the bounding boxes, and Footnote category usually has lower performance because parts of them are removed during redudant rectangle removal step,but they are thought of as non-informative blocks, our system does not guide users to read them. Thus the lower performance on footnote category does not really matter.
Fig. 8. An illustration of accept threshold selection. The groundtruth blocks are marked as magenta and the result blocks are marked as blue. Low MatchScore mostly comes from those small blocks (about 0.8 for logos and 0.6 for footnotes), because a trifling difference (less than 10 pixels) between the two region boundary can result in a large number of percentage of area measure. Thus the smaller blocks tend to have the lower MatchScore. However, this situation does not impede the reading process of users. As a result, we set the accept threshold at 0.6. Table 2. Page Segmentation results (T: text. I: image, F: footnote) Common Wealth Business Weekly(a) Business Week T
Nt Mt DetectRatet RecognAccuracyt EDMt
I
F
T
I
F
T
I
213 44 55 203 49 52 195 91 199 53 42 209 54 40 251 134 0.80 0.93 0.49 0.91 0.91 0.62 0.89 0.76 0.86 0.77 0.64 0.89 0.83 0.80 0.70 0.51 0.83 0.84 0.56 0.90 0.87 0.70 0.78 0.61(b)
F
84 67 0.50 0.63 0.56
Science T
I
F
245 77 122 250 81 79 0.89 0.81 0.52 0.88 0.77 0.81 0.88 0.78 0.64
156
Y.-M. Hsu et al.
4 Conclusion and Future Works This work demonstrates a possibility that people can turn the physical magazines into mobile e-book automatically and read them everywhere by simply snapping a shot. Compared to the text only e-books, our method can preserve layout appearence and images, free from being restricted by certain formats and hardware. It is also possible to do magazine retrieval if there is a magazine database.Thus, if users see an interestingmagazine by chance, they can retrieve parts of the magazine instead of buying them at full price. Furthermore, if we can apply image rectification techniques, the angle of inclination resulting from taking a snapshot will not be under so many restrictions as before through the help of mobile sensors, and the retrieval performance can also be improved. We are developing the system for leveraging mobile sensors for boosting snapshot and rectification quality. Meanwhile, we are also evaluating the proposed mobile reading system on Android phones for subjective performance.
References 1. Erol, B., Antúnez, E., Hull, J.J.: HOTPAPER: Multimedia Interaction with Paper using Mobile Phones. In: ACM Conference (2008) 2. Liao, C., Liu, Q.: PACER: Toward A Cameraphone-based Paper Interface for Fine-grained and Flexible Interaction with Documents. In: ACM MM (2009) 3. Xie, X., Miao, G., Song, R., Wen, J.-R., Ma, W.-Y.: Efficient Browsing of Web Search Results on Mobile Devices Based on Block Importance Model. In: Proc. Pervasive Computing and Communications. IEEE, Los Alamitos (2005) 4. Hattori, G., Hoashi, K., Matsumoto, K., Sugaya, F.: Robust Web Page Segmentation for Mobile Terminal Using Content-Distances and Page Layout Information. In: ACM WWW (2007) 5. Okun, O., Doermann, D., Pietikäinen, M.: Page Segmentation and Zone Classification: The State of the Art. In: UMD (1999) 6. Chang, C.-C., Lin, C.-J.: LIBSVM: a library for support vector machines (2001), http://www.csie.ntu.edu.tw/~cjlin/libsvm 7. Bosch, A., Zisserman, A., Munoz, X.: Representing shape with a spatial pyramid kernel. In: CIVR (2007) 8. GEDI: Groundtruthing Editor, http://gedigroundtruth.sourceforge.net/ 9. Antonacopoulos, A., Gatos, B., Bridson, D.: ICDAR 2005 Page Segmentation Competition. In: ICDAR (2005)
Multimodal Interaction Concepts for Mobile Augmented Reality Applications Wolfgang Hürst and Casper van Wezel Utrecht University, PO Box 80.089, 3508 TB Utrecht, The Netherlands
[email protected],
[email protected]
Abstract. Augmented reality on mobile phones – i.e. applications where users look at the live image of the device’s video camera and the scene that they see is enriched by 3D virtual objects – provides great potential in areas such as cultural heritage, entertainment, and tourism. However, current interaction concepts are often limited to pure 2D pointing and clicking on the device’s screen. This paper explores different interaction approaches that rely on multimodal sensor input and aim at providing a richer, more complex, and engaging interaction experience. We present a user study that investigates the usefulness of our approaches, verify their usability, and identify limitations as well as possibilities for interaction development for mobile augmented reality. Keywords: Mobile augmented reality, multi-sensor input, interaction design.
1 Introduction In this paper, we explore new interaction metaphors for mobile augmented reality (mobile AR), i.e. applications where users look at the live image of the video camera on their mobile phone and the scene that they see (i.e. the reality) is enriched by integrated 3-dimensional virtual objects (i.e. an augmented reality; cf. Fig. 1). Even if virtual objects are registered in 3D and updated in real-time, current interaction concepts are often limited to pure 2D pointing and clicking via the device’s touch screen. However, in order to explore the tremendous potential of augmented reality on mobile phones for areas such as cultural heritage, entertainment, or tourism, users need to create, access, modify, and annotated virtual objects and their relations to the real world by manipulations in 3D. In addition to common problems with interaction on mobile phones such as small screen estate, interaction design for mobile AR is particularly difficult due to several characteristics. First of all, since we are interacting with the (augmented) reality, interface design is somehow limited. Normally, an appropriate interface design can often deal with mobile interaction problems. For example, one can limit the number of buttons and increase their sizes so they can easily be hit despite small screen sizes. In mobile AR however, placement, style, and size of virtual objects is dictated by the video image and the virtual augmentation, resulting for example in sizes of objects that are hard to select (cf. Fig. 2, top). In addition, interfaces of normal applications can be operated while holding the device in a stable and comfortable position, K.-T. Lee et al. (Eds.): MMM 2011, Part II, LNCS 6524, pp. 157–167, 2011. © Springer-Verlag Berlin Heidelberg 2011
158
W. Hürst and C. van Wezel
Fig. 1. Mobile Augmented Reality enhances reality (represented by the live video stream on your mobile) with 3D virtual objects (superimposed at a position in the video that corresponds to a related spot in the real world). In this example, a virtual flower grows in the real plant pot.
whereas mobile AR applications require users to point their phone to a certain scene, often resulting in uncomfortable and unstable positions of the phone (especially when touching the screen at the same time; cf. Fig. 2, center). Being forced to point the device to a specific position in space also limits the possibility to interact via tilting (which has become one of the most commonly used interaction metaphors for controlling virtual reality data on mobiles e.g. for mobile gaming). Finally, input signals resulting from touch screen interactions are 2-dimensional, whereas augmented reality requires us to manipulate objects in 3D (cf. Fig. 2, bottom). The goal of this paper is to explore alternatives to touch screen based interaction in mobile AR that take advantage of multimodal information delivered by the sensors integrated in modern smart phones, i.e. camera, accelerometer, and compass. In particular, the accelerometer combined with the compass can be used to get the orientation of the phone. In fact, these sensors are used to create the augmented reality in the first place by specifying where and how virtual 3D objects are registered in the scene of the real world. We present an approach that uses this information also to select and manipulate virtual 3D objects. We compare it with touch screen interaction and a third approach that analyzes the camera image. In the latter case, the tip of your finger is tracked when it is moved in front of the camera and gestural input is used for object manipulation. Using gestural input is currently a hot trend in human computer interaction as illustrated by projects such as Microsoft’s Natal (now Kinect) [1] and MIT’s SixthSense [2]. Gesture-based interaction on mobile phones has also been explored by both industry [3] and academia [4] but in relation to different applications than augmented reality and utilizing a user-facing camera. Work in mobile AR that takes advantage of the front facing camera include analysis of hand-drawn shapes [5] and 3D sketches [6] as well as various marker-based approaches such as tangible interfaces [7]. However, most of these have been applied in indoor environments under rather restricted and controlled conditions, and often rely on additional tools such as the utilization of a pen or markers. [8] is an example for work in traditional augmented reality (e.g. using head mounted displays) that utilizes hand gestures. However, we are unaware of any work applying this concept to augmented reality on mobile phones. The purpose of the study presented in this paper is to verify if this concept is also suitable in a mobile context. In the following, we describe the three interaction concepts and tasks we want to evaluate (section 2), present our user study (section 3) and results (section 4), and conclude with a discussion about the consequences and our resulting future work on interaction design for mobile AR (section 5).
Multimodal Interaction Concepts for Mobile Augmented Reality Applications
159
Interface design. A good interface design allows us to deal with many interaction problems, e.g. by making buttons big enough or enlarging them during interaction for better visibility (left). In augmented reality, size and position of the objects is dictated by reality and thus objects might be too small to easily select or manipulate them (right).
Holding the device. Whereas normally one can hold the device in a stable and comfortable position during interaction (left), augmented reality requires us to point the phone to a certain spot in the real world, resulting in an unstable position esp. during interaction (right).
2D vs. 3D. Touch screen based interaction only delivers 2dimensional data, hence making it difficult to control virtual objects in the 3D world (e.g. put a flower into a plant pot; right).
Fig. 2. Potential problems with mobile augmented reality interaction
2 Interaction Concepts and Tasks Our goal is to compare standard touch screen based interaction with two different interaction concepts for mobile AR: one that depends on how the device is moved (utilizing accelerometer and compass sensors) and one that tracks the user’s finger in front of the camera (utilizing the camera sensor). Our overall aim is to target more complex operations with virtual objects than pure clicking. In the ideal case, a system should support all kinds of object manipulations in 3D, i.e. selection, translation, scaling, and rotation. However, for an initial study and in order to be able to better compare it to touch screen based interaction (which per default only delivers 2dimensional data), we restrict ourselves here to the three tasks of selecting virtual objects, selecting entries in context menus, and translation of 3D objects in 2D (i.e. left/right and up/down). In order to better investigate the actual interaction experience and eliminate noise from hand and finger tracking, we use a marker that is attached to the finger that will be tracked (cf. below). For the standard touch screen based interaction, the three tasks have been implemented in the following way: selecting an object is achieved by simply clicking on it on the touch screen (cf. Fig. 3, left). This selection evokes a context menu that was implemented in a pie menu style [9] which has proven to be more effective for pen and finger based interaction (compared to list-like menus as commonly used for mouse or touchpad based interaction; cf. Fig. 3, right). One of these menu entries puts
160
W. Hürst and C. van Wezel
Fig. 3. Touch screen interaction: select an object (left) or an entry from a context menu that pops up around the object after selection (right)
Fig. 4. Device based interaction: selection by pointing a reticule (left) to the target till the related bar is filled (right)
the object in “translation mode” in which a user can move an object around by clicking on it and dragging it across the screen. If the device is moved without dragging the object, it stays at its position with respect to the real world (represented by the live video stream). Leaving translation mode by clicking at a related icon fixes the object at its new final position in the real world. In terms of usability, we expect this approach to be simple and intuitive because it conforms to standard touch screen based interaction. In case of menu selection, we can also expect it to be reliable and accurate because we are in full control of the interface design (e.g. we can make the menu entries large enough and placed as far apart of each other so they can be hit easily with your finger). We expect some accuracy problems though when selecting virtual object, especially if they are rather small, very close to each other, or overlapping because they are placed behind each other in the 3D world. In addition, we expect that users might feel uncomfortable when interacting with the touch screen while they have to hold the device upright and targeted towards a specific position in the real world (cf. Fig. 2, center). This might be particularly true in the translation task for situations where the object has to be moved to a position that is not shown in the original video image (e.g. if users want to place it somewhere behind them). Our second interaction concept uses the position and orientation of the device (defined by the data delivered from the integrated accelerometer and compass) for interaction. In this case, a reticule is visualized in the center of the screen (cf. Fig. 4, left) and used for selection and translation. Holding it over an object for a certain amount of time (1.25 sec in our implementation) selects the object and evokes the pie menu. In order to avoid accidental selection of random objects, a progress bar is shown above the object to illustrate the time till it is selected. Menu selection works in the same way by moving the reticule over one of the entries and holding it till the bar is filled (cf. Fig. 4, right). In translation mode, the object sticks to the reticule in
Multimodal Interaction Concepts for Mobile Augmented Reality Applications
161
Fig. 5. Interaction by using a green marker for tracking that is attached to the tip of the finger
the center of the screen while the device is moved around. It can be placed at a target position by clicking anywhere on the touch screen. This action also forces the system to leave translation mode and go back to normal interaction. Compared to touch screen interaction, we expect this “device based” interaction to take longer when selecting objects and menu entries because users can not directly select them but have to wait till the progress bar is filled. In terms of accuracy, it might allow for a more precise selection because the reticule could be pointed more accurately at a rather small target than your finger. However, holding the device at one position over a longer period of time (even if it’s just 1.25 sec) might prove to be critical especially when the device is hold straight up into the air. Translation with this approach seems intuitive and might be easier to handle because people just have to move the device (in contrast to the touch screen where they have to move the device and drag the object at the same time). However, placing the object at a target position by clicking on the touch screen might introduce some inaccuracy because we can expect that the device shakes a little when being touched while held up straight in the air. This is also true for the touch screen based interaction, but might be more critical here because for touch screen interaction the finger already rests on the screen. Hence, we just have to it release it whereas here we have to explicitly click the icon (i.e. perform a click and release action). Touch screen based interaction seems intuitive because it conforms to regular smart phone interaction with almost all common applications (including most current commercial mobile AR programs). However, it only allows to remotely controlling the 3-dimensional augmented reality via 2D input on the touch screen. If we track the users’ index finger when their hand is moved in front of the device (i.e. when it appears in the live video on the screen), we can realize a finger based interaction where the tip of the finger can be used to directly interact with objects, i.e. select and manipulate them. In the ideal case, we can track the finger in all three dimensions and thus enable full manipulation of objects in 3D. However, since 3D tracking with a single camera is difficult and noisy (especially on a mobile phone with a moving camera and relatively low processing power) we restrict ourselves to 2D interactions in the study presented in this paper. In order to avoid influences of noisy input from the tracking algorithm, we also decided to use a robust marker based tracking approach where users attach a small sticker to the tip of their index finger (cf. Fig. 5). Object selection is done by “touching” an object (i.e. holding the finger at the position in the real world where the virtual object is displayed on the screen) till an associated
162
W. Hürst and C. van Wezel
Fig. 6. Holding the finger too close to the camera makes it impossible to select small objects (left). Moving your hand away from the camera decreases the size of the finger in the image but can result in an uncomfortable position because you have to stretch out your arm (right).
progress bar is filled. Menu entries can be selected in a similar fashion. In translation mode, objects can be moved by “pushing” them. For example, an object is moved to the right by approaching it with the finger from the left side and pushing it rightwards. Clicking anywhere on the touch screen places the object at its final position and leaves translation mode. Gesture based interaction using finger or hand tracking can be a very powerful way for human computer interaction in a lot of situations. However, in relation to mobile AR, there are many shortcomings and potential problems, most importantly due to the limited range covered by the camera. For example, the finger becomes very large in the image when being close to the camera, making it unsuitable for selection of small or overlapping objects. Moving it further away should increase possibilities for interaction, but might result in uncomfortable positions where the arm has to be stretched out quite far in order to create a smaller image of the finger on the screen (cf. Fig. 6). In addition, moving an object by pushing it from the side might turn out to be difficult depending on which hand is used (e.g. pushing it from the right using the left hand) and either result in an awkward hand position or even force people to switch the hand in which they hold the device. We implemented all three interaction concepts on a Motorola Droid/Milestone phone with Android OS version 2.1. In the next chapter we present a user study to verify their intuitive advantages and disadvantages discussed above.
3 User Study We evaluated the interface concepts described in the previous section in a user study with 18 participants (12 male and 6 female, 5 users at ages 15-20 years, 8 at ages 2130, 1 at ages 31-40 and 41-50, and 3 at ages 51-52). For the finger tracking, users were free in where to place the marker on their finger tip. Only one user placed it on his nail. All others used it on the inner side of their finger as shown in Figures 5 and 6. Eleven participants held the device in their right hand and used the left hand for interaction. For the other seven it was just the other way around. No differences could be observed in the evaluation related to marker placement or hand usage.
Multimodal Interaction Concepts for Mobile Augmented Reality Applications
163
Fig. 7. Object selection task: test case (single object, left), easy (multiple non-overlapping objects, center), and hard (multiple overlapping objects, right)
A within-group study was used, i.e. each participant tested each interface (subsequently called touch screen, device, and finger) and task (subsequently called object selection, menu selection, and translation). Interfaces were presented in different order to the participants to exclude potential learning effects. For each user, tasks were done in the following order: object selection, then menu selection, Fig. 8. Translation task: move object to then translation, because this would also target (white square on the right) be the natural order in a real usage case. For each task, there was one introduction test in which the interaction method was explained to the subject, one practice test in which the subject could try it out, and finally three “real” tests (four in case of the translation task) that were used in the evaluation. Subjects were aware that the first two tests were not part of the actual experiment and were told to perform the other tests as fast and accurate as possible. The three tests used in the object selection task can be classified as easy (objects were placed far away from each other), medium (objects were closer together), and hard (objects overlapped; cf. Fig. 7). In the menu selection task, the menu contained three entries and users had to select one of the entries on top, at the bottom, and to the right in each of the three tests (cf. Fig. 3, 4, and 5, right). In the translation task, subjects had to move an object to an indicated target position (cf. Fig. 8). The view in one image covered a range of 72.5°. In two of the four tests, the target position was placed within the same window as the object that had to be moved (at an angle of 35° between target and initial object position to the left and right, respectively). In the other two tests, the target was outside of the initial view but users were told in which direction they had to look to find it. It was placed at an angel of 130° between target and object, to the left in one case, and to the right in the other one. The order of tests was randomized for each participant to avoid any order-related influences on the results. For the evaluation, we logged the time it took to complete the task, success or failure, and all input data (i.e. the sensor data delivered from the accelerometer, compass, and the marker tracker). Since entertainment and leisure applications play an important role in mobile computing, we were not only interested in pure accuracy and performance, but also in issues such as fun, engagement, and individual preference.
164
W. Hürst and C. van Wezel
Hence, users had to fill out a related questionnaire and were interviewed and asked about their feedback and further comments at the end of the evaluation.
4 Results Figure 9 shows how long it took the subjects to complete the tests averaged over all users for each task (Fig. 9, top left) and for individual tests within one task (Fig. 9, top right and bottom). The averages in Figure 9, top left show that touch screen is the fastest approach for both selection tasks. This observation conforms to our expectations mentioned in the previous section. For device and finger, selection times are longer but still seem reasonable. Looking at the individual tests used in each of these tasks, times in the menu selection task seem to be independent of the position of the selected menu entry (cf. Fig. 9, bottom left). Almost all tests were performed correctly: only three mistakes happened over all with the device approach and one with the finger approach. In case of the object selection task (cf. Fig. 9, top right), touch screen interaction again performed fastest and there were no differences for the different levels of difficulty of the tasks. However, there was one mistake among all easy tests and in five of the hard tests the wrong object was selected thus confirming our assumption that interaction via touch screen will be critical in terms of accuracy for small or close objects. Finger interaction worked more accurate with only two mistakes in the hard test. However, this came at the price of a large increase of selection time. Looking into the data we realized that this was mostly due to subjects holding the finger relatively close to the camera resulting in a large marker that made it difficult to select an individual object that was partly overlapped by others. Once the users moved their hand further away from the camera, selection worked well as indicated by the low amounts of errors. For the device approach, there was a relatively large number of errors for the hard test, but looking into the data we realized that this was only due to a mistake that we made in the setup of the experiment: in all six cases, the reticule was already placed over an object when the test started and the subjects did not move the device away from it fast enough to avoid accidental selection. If we eliminate these users from the test set, the time illustrated for the device approach in the hard test illustrated in Figure 9, top right increases from 10,618 msec to 14,741 msec which is still in about the same range as the time used for the tests with easy and medium levels of difficulty. Since all tests in which the initialization problem did not happen have been solved correctly, we can conclude that being forced to point the device to a particular position over a longer period of time did not result in accuracy problems as we suspected. In the translation task, we see an expected increase in time in case of the finger and touch screen approach if the target position is outside of the original viewing window (cf. Fig. 9, bottom right). In contrast to this, there are only small increases in the time it took to solve the tasks when the device approach is used. In order to verify the quality of the results for the translation task, we calculated the difference between the center of the target and the actual placement of the object by the participants in the tests. Figure 10 illustrates these differences averaged over all users. It can be seen that
Multimodal Interaction Concepts for Mobile Augmented Reality Applications
45000
25000
AVERAGE TIME (IN MSEC) OVER ALL USERS & TESTS
165
OBJECT SLECTION TASK (INDIVIDUAL TESTS)
40000 20000
35000 30000
DEVICE
FINGER
DEVICE
FINGER
TOUCH SCREEN
TOUCH SCREEN
15000
25000 20000
10000
15000 10000
5000
5000 0
0
OBJECT SELECTION
14000
MENU SELECTION
MENU SELECTION TASK (INDIVIDUAL TESTS)
12000
TRANSLATION DEVICE FINGER TOUCH SCR.
EASY
70000 60000
10000
50000
8000
40000
6000
30000
4000
20000
2000
10000
MEDIUM
HARD
TRANSLATION TASK (INDIVIDUAL TESTS)
DEVICE FINGER TOUCH SCREEN
0
0 BOTTOM
RIGHT
35° (LEFT)
TOP
35° (RIGHT)
130° (LEFT)
130° (RIGHT)
Fig. 9. Time it took to solve the tasks (in msec, averaged over all users) 0 35° (LEFT) 35° (RIGHT)
100
200
300
400
500
600
700
800
900
1000
AVERAGE DISTANCE FROM TARGET IN TRANSLATION TASK DEVICE
FINGER
TOUCH SCREEN
130° (LEFT) 130° (RIGHT)
Fig. 10. Correctness of translation task (squared Manhattan distance in screen units)
the device approach was not only the fastest but also very accurate, especially in the more difficult cases with an angel of 130° between target and initial object position. Finger based interaction on the other hand seems very inaccurate. A closer look into the data reveals that the reason for the high difference between target position and actual object placement are actually many situations in which the participants accidentally hit the touch screen and thus placed the object at a random position before reaching the actual target. This illustrates a basic problem with finger based interaction, i.e. that users have to concentrate on two issues: moving the device and “pushing” the object with their finger at the same time. In case of device based interaction on the other hand, users could comfortably take the device in both hands while moving around thus resulting in no erroneous input and a high accuracy as well as fast solution time. A closer look into the data also revealed that the lower performance in case of the touch based interaction are mostly due to one outlier who did extremely bad (most likely also due to an accidental input) but otherwise they are at about the same level of accuracy as the device approach (but of course at the price of a worse performance time; cf. Fig. 9).
166
W. Hürst and C. van Wezel
Based on this formal analysis of the data, we can conclude that touch based interaction seems appropriate for selection tasks. For translation tasks, the device based approach seems to be more suitable, whereas finger based interaction seems less useful in general. However, especially in entertainment and leisure applications, speed and accuracy are not the only relevant issues, but fun and engagement can be equally important. In fact, for gaming, mastering an inaccurate interface might actually be the whole purpose of the game (think of balancing a marble through a maze by tilting your phone appropriately – an interaction task that can be fun but is by no means easy to handle). Hence, we asked the participants to rank the interaction concepts based on performance, fun, and both. Results are summarized in Table 1. Rankings for performance clearly reflect the results illustrated in Figures 9 and 10 with touch based interaction ranked highest by eleven participants, and device ranked first by six. Only one user listed finger based interaction as top choice for performance. However, when it comes to fun and engagement, the vast majority of subjects ranked it as their top choice, whereas device and touch were ranked first only four and one time, respectively. Consequently, rankings are more diverse when users were asked to consider both issues. No clear “winner” can be identified here, and in fact, many users commented that it depends on the task: touch and device based interaction are more appropriate for applications requiring accurate placement and control, whereas finger based interaction was considered as very suitable for gaming applications. Typical comments about the finger based approach characterized it as “fun” and “the coolest” of all three. However, its handling was also criticized as summarized by one user who described it as “challenging but fun”. When asked about the usefulness of the three approaches for the three tasks of the evaluation, rankings clearly reflect the objective data discussed above. Table 2 shows that all participants ranked touch based interaction first for both selection tasks, but the vast majority voted for the device approach in case of translation. It is somehow surprising though that despite its low performance, finger based interaction was ranked second by eight, seven, and nine users, respectively, in each of the three tasks – another indication that people enjoyed and liked the approach although it is much harder to handle than the other two. Table 1. Times how often an interface was ranked first, second, and third with respect to performance versus fun and engagement versus both (T = touch screen, D = device, F = finger)
Ranked 1st Ranked 2nd Ranked 3rd
PERFORMANCE T D F 11 6 1 3 6 9 4 6 8
T 1 6 11
FUN D 4 10 4
F 13 2 3
PERF. & FUN T D F 8 3 7 6 8 4 4 7 7
Table 2. Times how often an interface was ranked first, second, and third with respect to the individual tasks (T = touch screen, D = device, F = finger)
Ranked 1st Ranked 2nd Ranked 3rd
OBJECT SELECT. T D F 18 10 8 8 10
MENU SELECT. T D F 18 11 7 7 11
TRANSLATION T D F 3 15 6 3 9 9 9
Multimodal Interaction Concepts for Mobile Augmented Reality Applications
167
5 Conclusion Interacting with augmented realities on mobile devices is a comprehensive experience covering many ranges, characteristics, and situations. Hence, we can not assume to find a “one size fits all” solution for a good interface design, but most likely have to provide different means of interaction depending on the application, the goal of a particular interaction, and the context and preference of the user. Our study suggests, that a combination of touch screen based interaction (which achieved the best results in the shortest time in both selection tasks and was rated highest by users in terms of performance) with device dependent input (which achieved the best results in the shortest time in the translation task and was highly ranked by the users for this purpose) is a promising approach for serious applications that require exact positioning and accurate control over the content. On the other hand, interaction via finger tracking (which had low performance values but was highly rated and appreciated by the users in terms of fun, engagement, and entertainment) seems to be a promising approach for mobile gaming and other leisure applications. Obviously, our study was only intended as a first step in the direction of better, more advanced interfaces for mobile AR. In relation to serious applications, our future work aims at further investigating touch and device based navigation, especially in relation to other tasks, such as rotation and scaling of objects. In relation to leisure games, our goal is to further investigate interaction via finger and hand tracking, especially in relation to “real” 3D interaction where moving the finger in 3D space can be used to manipulate objects in 3D, for example by not only pushing them left/right and up/down but also forward and backward.
References 1. Xbox Kinect (formerly known as Microsoft’s project Natal), http://www.xbox.com/en-US/kinect (last accessed 10/15/10) 2. Mistry, P., Maes, P.: SixthSense: a wearable gestural interface. In: ACM SIGGRAPH ASIA 2009 Sketches (2009) 3. EyeSight’s Touch Free Interface Technology Software, http://www.eyesight-tech.com/technology/ (last accessed 10/15/10) 4. Niikura, T., Hirobe, Y., Cassinelli, A., Watanabe, Y., Komuro, T., Ishikawa, M.: In-air typing interface for mobile devices with vibration feedback. In: ACM SIGGRAPH 2010 Emerging Technologies (2010) 5. Hagbi, N., Bergig, O., El-Sana, J., Billinghurst, M.: Shape recognition and pose estimation for mobile augmented reality. In: Proceedings of the 2009 8th IEEE International Symposium on Mixed and Augmented Reality (2009) 6. Bergig, O., Hagbi, N., El-Sana, J., Billinghurst, M.: In-place 3D sketching for authoring and augmenting mechanical systems. In: Proceedings of the 2009 8th IEEE International Symposium on Mixed and Augmented Reality (2009) 7. Billinghurst, M., Kato, H., Myojin, S.: Advanced Interaction Techniques for Augmented Reality Applications. In: Proceedings of the 3rd international Conference on Virtual and Mixed Realit, Part of HCI International 2009 (2009) 8. Lee, M., Green, R., Billinghurst, M.: 3D natural hand interaction for AR applications. In: Proceedings of Image and Vision Computing, New Zealand (2008) 9. Callahan, J., Hopkins, D., Weiser, M., Shneiderman, B.: An empirical comparison of pie vs. linear menus. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (1988)
Morphology-Based Shape Adaptive Compression Jian-Jiun Ding, Pao-Yen Lin, Jiun-De Huang, Tzu-Heng Lee, and Hsin-Hui Chen Graduate Institute of Communication Engineering, National Taiwan University, 10617, Taipei, Taiwan
[email protected],
[email protected],
[email protected],
[email protected],
[email protected]
Abstract. In this paper, we use morphology to improve the performance of the object oriented coding scheme. The object oriented compression algorithm is popular in recent years. However, due to the high frequency components of the edges, the reconstructed image is always seriously disturbed by the object edges (i.e., the ring effect) when using the algorithm. In this paper, we use the morphological operation to separate an object into the interior and the contour sub-regions before applying the shape-adaptive DCT. Since the contour subregion is separated apart, the high frequency components in it will not disturb the interior region and the reconstructed image will have much better quality. Simulation results show that the proposed algorithm outperforms the conventional object oriented compression algorithm. Keywords: image coding, image processing, shape, object oriented methods, morphological operations.
1 Introduction Image compression using object-oriented segmentation is based on the idea that the color values of the pixels within the same object always have high correlation [1]. The object-based coding scheme is popular in recent years [2]-[11]. It plays an important role on the MPEG-4 standard 3[4][5]. A variety of object-oriented image compression techniques were proposed to realize the functionality of making visual objects available in the compressed form 2, [6][11]. The shape-adaptive transformation is applied on the arbitrarily shaped region to achieve higher compression rate by exploiting high correlation between the pixels in the specific object. The earliest approach is to calculate the customized DCT basis functions for all the image objects using generalized orthogonal transform [2][6]. One of the most popular methods recently is the shape-adaptive DCT (SA-DCT) [7][8][9]. The SA-DCT flushes samples in an arbitrarily shaped object to a specific edge of its bounding box before performing the 1-D DCT in the flushed direction. However, the existence of the high frequency components in boundary regions reduces the coding performance and image quality. In this paper, we propose an improved object-oriented compression scheme which can efficiently represent and compress the image contents since the textures, patterns, and other characteristics in an object usually share the same or similar color values K.-T. Lee et al. (Eds.): MMM 2011, Part II, LNCS 6524, pp. 168–176, 2011. © Springer-Verlag Berlin Heidelberg 2011
Morphology-Based Shape Adaptive Compression
169
and the color intensity variations are low. Higher compression ratio and better image quality of reconstruction image can then be achieved. Moreover, the proposed method is totally backward compatible with the existing coding standards.
2 Proposed Morphological Segmentation Using Erosion For an arbitrarily-shaped object, the pixels within it have similar or identical color values. However, the contour region of an object contains a great portion of the high frequency components because the color values at the edge of an object usually vary significantly. When we use the object-oriented compression technique instead of cutting the entire image into the conventional 8×8 blocks, it is possible that the resulting arbitrary-shape DCT coefficients contain a considerable amount of high frequency components. Since the main cause of the potential distortion and noise artifacts is due to the fact that the high frequency components are truncated during the quantization process, using the object-oriented compression technique could amplify the distortion and noise artifacts. To avoid the distorted artifacts, we could segment the entire image in a more detailed way. However, it would produce more boundary information that needs to be recorded. Since we know that the variations in pixel values occur around the contour region of an arbitrary shape, we can divide an arbitrarily-shaped image object into the contour sub-region and the interior sub-region. In Fig. 1, we show an example of the contour sub-region and the interior sub-region. Then we apply the shape-adaptive transformation and encode them separately. Note that we do not need to record the boundary of the overall interior sub-region because it can be retrieved by the original boundary in the decoding process. In Fig. 2, we depict the overall procedure of our proposed algorithm. The hatched block shows the role of the morphological operation corresponding to other procedure in the whole image compression processing scheme. It should be noted that the morphological operation (erosion) is used to divide the object into two separate parts as shown in Fig. 1. The proposed image segmenting method is completely discussed as below. Contour sub-region The overall object
Interior sub-region Fig. 1. The contour and interior sub-regions of an object
We use a simple test image depicted in Fig. 3, which has six arbitrarily-shaped image objects as an input image to be compressed using our proposed scheme. The way we divide an arbitrarily-shaped image object is by a morphological operation called
170
J.-J. Ding et al.
Fig. 2. Block diagram of our proposed algorithm
Interior sub -regions Shapeadaptive DCT
Contour sub-regions Shapeadaptive DCT
Fig. 3. Divide the image objects by morphological erosion
binary erosion [1]. The overall erosion process is illustrated in Fig. 3. As mentioned earlier, we obtain the shapes of the objects in binary format by filling the pixels inside the boundaries with zeroes. For two binary image set A and B, the erosion of A by B, denoted A B, is defined as
A B = { z | ( B ) z ⊆ A} ,
(1)
where the translation of set B by point z = (z1, z2), denoted (B)z, is defined as [1]
( B) z = {c | c = b + z , for b ∈ B} ,
(2)
Morphology-Based Shape Adaptive Compression
171
By the erosion operation, each of the objects is divided into two parts: the contour sub-region and the interior sub-region. The contour sub-region β (A) obtained by mask B can be expressed as
β ( A) = A − A B .
(3)
We take Fig. 3 for example. We erode the binary arbitrary shapes by a structuring element B of a size of 5×5 pixels as depicted in Fig. 3. The six contour sub-regions of the binary shapes are obtained by subtracting the eroded shapes from the original ones using (3). Finally, we multiply the original arbitrary-shape image objects by both the binary contour shapes and the interior shapes to get the final contour sub-regions and the interior sub-regions of the six objects, respectively. The shape adaptive DCT [7][8][9] are then performed on both contour regions and interior regions, then the resulting DCT coefficients are quantized and encoded separately.
3 Quantization of the Coefficients Because of the high portion of high frequency components in the contour sub-region, actually we can use higher quantization step size for the contour sub-region in order to achieve high compression ratio with better image quality. Basically, most of the shape-adaptive transformation, such as the SA-DCT, is backward compatible. Block-based quantization are used after performing these transformation, whereas the approach using generalized orthogonal transform to generate customized DCT basis functions may not use the original block-based quantizer. So we define an extendable and object-shape-dependent quantization array Q(k) as a linearly increasing line:
Q ( k ) = Qa k + Qc
(4)
for k = 1, 2,…, M. The two parameters Qa and Qc are the slope and the intercept of the line, respectively, and M is the number of the DCT coefficients within the same arbitrarily-shaped object. Each DCT coefficient F(k) is then divided by the corresponding quantization array Q(k) and rounded to the nearest integer:
⎛ F (k ) ⎞ Fq ( k ) = Round ⎜ , ⎜ Q ( k ) ⎟⎟ ⎝ ⎠
(5)
where k = 1, 2,…, M. The parameter Qc in (4) is important because it affects the quantization quantity of the DC term and the low frequency components whose value is usually greater than the high frequency components. The other parameter Qa affects the high frequency portion of the arbitrary-shape DCT coefficients the most. As mentioned in Section I quantization is fundamentally lossy especially at low bitrates compression. This problem is the main motivation of our proposed morphological segmentation algorithm discussed in Section 2.
172
J.-J. Ding et al.
4 Coding Technique of the Image Object After the quantization stage, the quantized arbitrarily-shaped DCT coefficients can be encoded by the same coding technique that is used to the shape adaptive transformation in previous stage. For the arbitrarily-shaped DCT using generalized orthogonal transform method, we actually do not need to record the length of the arbitrarily-shaped DCT coefficients of each object because it can be obtained by the boundary information which is not in the scope of this paper. Therefore, in the decoding process, we normally decode the bit stream of boundaries first. The point number inside a boundary is then counted and it should be exactly the length of DCT coefficients in the corresponding object. Furthermore, the contour sub-region and interior sub-region can be obtained by eroding the mask of the object.
5 Simulation Results and Performance Comparison The comparisons of the conventional and the proposed object-oriented image compression algorithms are made in this section. First, we use the image in Fig. 3 as the testing image. The image with size 256 × 256 is first divided into six small segments. In our algorithm, each small object is performed by the erosion operation before applying SA-DCT. As shown in Fig. 4, it is obvious that our proposed method have 43 42.5
SADCT SADCT with erosion
42
[dB]
41.5 41
PSNR
40.5 40 39.5 39 38.5 38 0.4
0.5
0.6
0.7
0.8 4
Bitrate(×10 )
0.9
1
1.1
[Bits]
Fig. 4. Comparison of the PSNRs of the reconstructed images with test image in Fig. 3. (Blue line): the result of the conventional method. (Red line): the result of the proposed method.
Morphology-Based Shape Adaptive Compression
40.5 40
173
SADCT SADCT with erosion
[dB]
39.5 39 38.5
PSNR
38
Barbara
37.5 37 36.5 36 6.5
7
7.5
8
8.5
Bitrate(×104)
9
9.5
10
10.5
[Bits]
Fig. 5. Comparison of the PSNRs of the reconstructed images with test image Barbara. (Blue line): the result of the conventional method. (Red line): the result of the proposed method. 49 SADCT SADCT with erosion
48
[dB]
47
PSNR
46
Akiyo
45
44
43
42 4
5
6
7
8 4
Bitrate(×10 )
9
10
11
[Bits]
Fig. 6. Comparison of the PSNRs of the reconstructed images with test image Akiyo. (Blue line): the result of the conventional method. (Red line): the result of the proposed method.
174
J.-J. Ding et al.
greater performance than the conventional object-oriented compression algorithm especially when the bit rate is low. It means that the contour sub-region extracted by erosion operation contains large amounts of high frequency components that are later quantized by higher quantization coefficients. Moreover, the proposed method performs better in low bit-rate condition. In Fig. 5 and Fig. 6, we show the comparison of the PSNRs with the Akiyo and Barbara images. We can find that the proposed method outperforms the conventional object-oriented compression algorithm. In Figs. 7, 8 and 9, we show some examples of the reconstructed images. The results in Fig. 7(b)(d), Fig. 8(b)(d), and Fig. 9(b) show that, using the proposed method, the blocky and blurry artifacts are much less than using conventional object-oriented compression algorithm. It can be seen that the blurring phenomena caused by the boundary region are significantly reduced. Therefore, our algorithm can indeed achieve much better reconstructed images and hence improve the performance of shape-adaptive image compression. (a) Conventional method
(b) Proposed method
(c) Conventional method
(d) Proposed method
Fig. 7. The reconstructed images of (Left) the conventional object-oriented compression scheme and (Right) the PROPOSED SCHEME. The results show that the reconstructed images of our proposed method have much better qualities.
Morphology-Based Shape Adaptive Compression (a) Conventional method
(b) Proposed method
(c) Conventional method
(d) Proposed method
SADCT
175
SADCT with erosion operation
Fig. 8. The reconstructed images of (Left) the conventional object-oriented compression scheme and (Right) the PROPOSED SCHEME. The results show that the reconstructed images of our proposed method have much better qualities. (a) Conventional method SADCT
(b) Proposed method SADCT with erosion operation
Fig. 9. The reconstructed images of (Left) the conventional object-oriented compression scheme and (Right) the PROPOSED SCHEME. The results show that the reconstructed images of our proposed method have much better qualities.
176
J.-J. Ding et al.
The original object-oriented compression algorithm suffers from the blurring phenomena because there are many high frequency components in the boundary regions of the objects. Since our algorithm uses morphological erosion to separate an object into interior and contour parts, as in Fig. 3, the high frequency components in the boundaries will not affect the interior regions. It makes reconstructed images can have higher quality.
6 Conclusions As evident from the simulation results, the proposed morphology-based shape adaptive compression makes significant improvements by the separation of contour sub-region which contains high portion of high frequency components. The proposed morphology-based arbitrarily-shaped image compression scheme has high adaptability because it is applicable to different inputs, different shape-adaptive transformation. The complexity of our proposed method is almost the same as that of the original object-oriented compression algorithm. However, from the simulations in Figs. 7-9, the qualities of the reconstructed images are much improved.
References 1. Gonzalez, R.C., Woods, R.E.: Digital Image Processing, 2nd edn. Prentice Hall, New Jersey (2002) 2. Chang, S.F., Messerschmitt, D.: Transform Coding of Arbitrarily Shaped Image Segments. In: Proc. 1st ACM Int. Conf Multimedia, Anaheim, CA, pp. 83–90 (1993) 3. Sikora, T.: MPEG-4 Video Standard Verification Model. IEEE Trans. Circuits Syst. Video Technol. 7, 19–31 (1997) 4. Marpe, D., Schwarz, H., Wiegand, T.: Context-Based Adaptive Binary Arithmetic Coding in the H.264/AVC Video Compression Standard. IEEE Trans. Circuits Syst. Video Technol. 13, 620–636 (2003) 5. Richardson, L.E.G.: H.264 and MPEG-4 Video Compression: Video Coding for NextGeneration Multimedia. John Wiley and Sons, Chichester (2004) 6. Gilge, M., Engelhardt, T., Mehlan, R.: Coding of Arbitrarily Shaped Image Segments Based on a Generalized Orthonormal Transform. Signal Process: Image Commun. 1, 153– 180 (1989) 7. Sikora, T., Makai, B.: Shape-Adaptive DCT for Generic Coding of Video. IEEE Trans. Circuits Syst. Video Technol. 5, 59–62 (1995) 8. Sikora, T.: Low Complexity Shape-Adaptive DCT for Coding of Arbitrarily Shaped Image Segments. Signal Process: Image Commun. 7, 381–395 (1995) 9. Kauff, P., Schüür, K.: Shape-Adaptive DCT with Block-Based DC Separation and a DC Correction. IEEE Trans. Circuits Syst. Video Technol. 8, 237–242 (1998) 10. Li, S., Li, W.: Shape-Adaptive Discrete Wavelet Transforms for Arbitrarily Shaped Visual Object Coding. IEEE Trans. Circuits Syst. Video Technol. 10, 725–743 (2000) 11. Moon, J.H., Park, G.H., Chun, S.M., Choi, S.R.: Shape-Adaptive Region Partitioning Method for Shape-Assisted Block-Based Texture Coding. IEEE Trans. Circuits Syst. Video Technol. 7, 240–246 (1997)
People Tracking in a Building Using Color Histogram Classifiers and Gaussian Weighted Individual Separation Approaches Che-Hung Lin, Sheng-Luen Chung, and Jing-Ming Guo Department of Electrical Engineering National Taiwan University of Science and Technology Taipei, Taiwan {M9507208,slchung}@mail.ntust.edu.tw,
[email protected]
Abstract. This paper focuses on people tracking within a building with the installment of distributed overhead cameras. Our primary concern is to keep tracks of: number of people entrance to a particular area; and the whereabouts trajectory of a particular person within the monitored building. With image taken from ceiling mounted camera, pedestrian’s physiologic contour is analyzed from four different viewing angles to form a person’s identity signature. In doing so, techniques to locate a person’s head, to predict his/her movement direction, to separate overlapped physiological blobs, and to differentiate different people by color histogram classifier have been proposed. Special attention has been paid for system configurability such that the proposed software architecture can be deployed to different floor plans. We have conducted a continuing surveillance monitoring on the third floor of EE in NTUST, and the result shows moderate surveillance performance: 93% accuracy in entrance counting, and 76% accuracy in identification checking. Keywords: people tracking system, surveillance, color histograms classifier, Gaussian weighted individual separation, movement prediction.
1 Introduction Surveillance is always a concerned issue for security reason. The continuing cost reduction in cameras has increased the umber of camera installation. With this trend in deploying of more an more cameras, it is practically impossible to rely on human inspectors to track monitored objects (people) from different camera channels at the same time. In contrast to RFID-based solutions to tracking people, the focus of this work is to investigate solution that does not require people’s conscious participation to identify them. Efficient solutions under of premise of no active people’s involvement are likely to lead to more context awareness applications, where services are provided based on particular person at particular place and moment. Many methods for people tracking have been proposed in the literature: approximate color histogram is used to recognize human positions by their silhouette [1]-[2]; however, the color histogram is easily influenced by the changes of image distance. K.-T. Lee et al. (Eds.): MMM 2011, Part II, LNCS 6524, pp. 177–186, 2011. © Springer-Verlag Berlin Heidelberg 2011
178
C.-H. Lin, S.-L. Chung, and J.-M. Guo
RFID is proposed [3] to detect people’s location; however, this approach requires people wearing a tag to be identified. The scope of the people tracking problem addressed in this paper is outlined as follows: The target of monitored coverage is a floor plan composed of corridors, exit/entrance to a room or the floor, access to elevators and stairways; the floor plan is partitioned into adjacent areas. The concerned issue is to track the whereabouts (a particular area in the floor plan) of a distinguished individual. Instead of distinguishing who is who, which requires a priori information on the names of people, we are interested instead in which is which, which for instance requires the distinction of a person of particular dressing to another. The equipment employed is a group of distributed ceiling mounted cameras, each of which is installed at fixed height and with a fixed coverage at the junction of two areas. To address the issue raised above, a Head Center Determination (HCD) approach is proposed to detect a person’s presence under a camera. The same notion of head center is also used to separate individuals from a overlapped image possibly containing several people using another proposed approach: Gaussian Weighted Individual Separation (GWIS). Since a person appears differently with different viewing angles from the ceiling mounted camera, physiological contours viewed from four different view angles are taken into account. Moreover, a Color Histogram Classifier (CHC) is proposed for both easier matching operation as well as reducing sensitivity for lighting effect. Particular attention is also paid to the status tracking of each entrant and the coordination between different cameras to yield a comprehensive view on people’s whereabouts on the floor plan.
(a)
(b)
Fig. 1. ROI of an IP Cam. (a) Snapshots of PTS. (b) Specification of a ROI.
Fig. 2. Setting of each ROI through areas
People Tracking in a Building Using Color Histogram Classifiers and GWIS Approaches
179
2 People Tracking in a Building Floor Plan Our solution to the PTS problem covering the whole floor plan relies on the efficient approaches of HCD, GWIS and CHC to the processing of the image from each ROI. The three key elements are introduced as below. A. Head Center Determination (HCD) With a fixed camera, a moving object can be obtained by applying background image subtraction. Easy to apply and efficient in delivering results, this method is suited for real-time processing. Background image subtraction is to maintain a background image first and then subtract the background images from subsequent images. After the subtraction, three difference images Pd _ r , Pd _ g , and Pd _ b associated to current R, G, and B color channels can be obtained. Additional Y, Cb, and Cr as well as H, S, and I can be obtained from R, G, and B [5]. Among these, R, G, B, Y, Cb, Cr, and S are employed to separate hair, skin, and the rest foreground from the background. The thresholds are empirical values from numerous test samples. In this work, the head region is composed of hair region and skin region. The blue part in Fig. 3 represents head region and the white part is considered as foreground. The head size should be larger than 180 pixels to claim a potential pedestrian existence according to experimental results; without this threshold, some objects with similar color as human head but are of smaller sizes may be incorrectly inferred as people. The amount of foreground pixels is critical in determining whether to enter background update procedure. In general, when the number of foreground pixels is lower than an empirical threshold of 640, it means the current image has no pedestrian presence. Consequently, the PTS ignores tracking procedure and enters background update phase, which can significantly reduce computational complexity otherwise required.
(a)
(b)
Fig. 3. Image binarization method. (a) Current image. (b) Analyzed image.
The background update procedure is to withstand the light effect in the environment when system is operated in different time frames (morning, afternoon, and night). When the foreground pixels of are lower than that of a predefined threshold for a total of 300 successive frames, PTS will update the current image as the updated background image. To give a rough estimate on how often the background is updated: it takes 16~79 ms to process one image from a camera, and there are a total of eight cameras. Hence, a total of 300 successive images take about 4.8~23 seconds, which is roughly the time used to enable background update procedure in this work.
180
C.-H. Lin, S.-L. Chung, and J.-M. Guo
Pf > TBU ⎧ 0 Cch = ⎨ + C Pf ≤ TBU 1 ⎩ ch
(1)
where the subscript ch denotes the channel index of each camera; Pf denotes the amount of foreground pixels calculated by background image subtraction; TBU denotes the foreground pixels threshold for background update, which is set at 400, for deciding the next object tracking step; Cch denotes the number of times that the foreground pixels of a channel are lower than that of the background updated threshold. To avoid the number of foreground pixels of the last updated background images close to TBU , when Cch is three hundred times, PTS updates images when Pf < 0.8TBU . In the proposed PTS, the image fed from each camera is of size 160x120. Through analyzing more than seven hundred pedestrians passing through the environment, we conclude that the minimum head requires at least 180 pixels. A mask of size 17x17 is used to decide the head region: when a pixel of the image is known as a foreground pixel, the pixel is considered at the center of the mask, and the totality of the pixels of head color inside this mask is calculated. When the totality is greater than 180 pixels, PTS concludes the center point of the mask as a head region. The formula is given as ⎧0 ( Pf < TBU ) OR (Ch,i , j < 180) Ph ,i , j = ⎨ ⎩1 ( Pf > TBU ) AND (Ch,i , j ≥ 180)
(2)
where Ch,i , j denotes the number of head pixels with center position (i,j); Ph,i , j denotes the possible head region when the value is 1. Figure 4 shows a processed example, in which the blue pixels represent the head area, and the red pixels represent the possible head center.
(a)
(b)
Fig. 4. Analyzing position of human head. (a) Current image (b) Analyzed image.
The parameter Ph,i , j is then used to analyze the number of people. First, we denote M s as a 17x17 mask used for scanning possible human head locations, and Pm,i , j is a
160x120 array storing marks for human blobs, which is initially set as 0 for each pixel. The processes are organized as below:
Pm,i , j
⎧ 0, if (f ∀ { x, y ∈ M s } , Pm,i + x, j + y = 0) AND ( Ph,i , j =0); ⎪ = ⎨ Pm ,i , j + 1, if (∀ { x, y ∈ M s } , Pm,i + x, j + y = 0) AND (Ph,i , j = 1); ⎪P , if (∀ { x, y ∈ M s } Pm,i + x , j + y = 1) AND (Ph ,i , j = 1). ⎩ m ,i , j
(3)
People Tracking in a Building Using Color Histogram Classifiers and GWIS Approaches
181
Eq. 3 assigns a number for each head center. A more accurate head center can be obtained by averaging the coordinates of the pixels with the same number in Pm,i , j . B. Gaussian Weighted Individual Separation (GWIS) As the head center is determined as stated in the previous sub-section, the proposed GWIS is applied to extract individual contours form the foreground image. According to experimental results, the maximum width of human contour is around 50 pixels. Thus, a mask of size 51x51 is employed to detect human contour. In essence, those pixels that are close to the head center have higher likelihood belonging to that person’s contour. Hence, the mask of Gaussian shape. The head center is mapped to the center of the Gaussian mask. Notably, the color red is more sensitive to lighting effect than blue and green; as a result, two different types of Gaussian masks are used to account for these two different types of color channels: red vs. blue and green. In this work, TP _ R and TP _ GB denote the adaptive Gaussian thresholds for red and green/blue, respectively, as given in the following general form: x2 y2 ⎤ ⎧ ⎡ −( + ) 2σ x 2 2σ y 2 ⎪⎪C ⎢ 1 ⎥ + TL e ⎥ ⎨ ⎢ 2πσ xσ y ⎦ ⎪ ⎣ TL ⎪⎩
if (x 2 + y 2 )
(4)
if (x + y ) ≥ D 2
2
where (C , σ x , σ y , TL , D) = (2000, 10, 10, 40, 254) and (2500, 13, 13, 30, 342) for TP _ R and TP _ GB , respectively. Figure 5 shows a binarized result using the proposed
GWIS, where the blue frame indicates the head center using the proposed HCD method. After the application of GWIS on human contour detection, overlapped pixels
(a)
(b)
Fig. 5. Binarized result using the proposed GWIS. (a) Test image. (b) Binarized result.
(a)
(b)
(c)
Fig. 6. Overlapped pixels elimination. (a) Current image. (b) Binarized result without overlapped pixels removal. (c) Binarized result with overlapped pixel removal.
182
C.-H. Lin, S.-L. Chung, and J.-M. Guo
may result when two walkers are too close to each other, as shown in Fig. 6(b). Similarity in color structure is to separate individual contours. In doing so, the head pixels are removed from contour pixels, and the average of the rest contour pixels are calcui lated as ABC , where i denotes the ith person. The overlapped pixels are classified by i comparing their values with ABC : they are then classified into the most comparable person’s contour. A classified example is shown in Fig. 6(b). It is clear that the proposed strategy can effectively solve the overlapped pixel problem. Some small broken objects surrounding the main objects are observed from Figs.5-6. These annoying objects can be eliminated using image patch method, where two types of masks of sizes 3x3 and 5x5 are adopted to raster scan the binarized image. Suppose a foreground pixel is detected with the 3x3 mask, the 5x5 mask is then applied to measure if number the foreground pixels more than 15 in this area; if no, the pixel is removed from the foreground pixel candidate. Figure 7 shows a processed example.
(a)
(b)
Fig. 7. Result of image patch method. (a) Original binarized result. (b) Processed result by the image patch method.
C. Color Histogram Classifiers (CHC) This sub-section introduces how to use CHC to differentiate and characterize each individual. The color spaces employed in this work include HSV and YCbCr. The major aim of CHC is to distribute unstable data to different fuzzy sets [6] according to different degree of membership. In this work, multiple fuzzy sets within color range 0~255 are involved. A fuzzy model is illustrated in Fig. 8 (a) in which x denotes grayscale; f ( x) is the degree of membership, and the range between S 0 and S1 is called fuzziness. The relationship between a fuzzy set and degree of membership is given as: ⎧ x-S0 ⎪ S -S , S0 <x<S1 ; ⎪ 1 0 ⎪ 1, S1 ≤ x ≤ S2 ; μ A1 (x)= ⎨ ⎪ S3 -x , S <x<S ; 2 3 ⎪ S3 -S2 ⎪ Otherwise. ⎩ 0,
(5)
where μA (x) denotes the characteristic function of set A1 . According to numerous 1 samples testing, the optimized number of fuzzy sets is eight, and the optimized
People Tracking in a Building Using Color Histogram Classifiers and GWIS Approaches
(a)
183
(b)
Fig. 8. Color histograms classifier. (a) Model of fuzzy set. (b) Classifier.
Fig. 9. Range of four identities
fuzziness range is 30. Figure 8 (b) shows the histogram classifier composed of the eight fuzzy sets in the proposed PTS. Through normalization, three color histograms of the detected contour are enhanced to 0~255 and multiplied by the color histogram classifier as given in Fig. 8 (b) to obtain a 48-byte identification signature composed of Cb, Cr , and S. Every two bytes are used to record an accumulate fuzzy sets’ degree of membership as C A f = C A f + Vx • μ A f ( x ) ,
(6)
where Af denotes the fth of fuzzy set in classifier; C A denotes the accumulated data of f
Af ; Vx denotes the color histograms data of the input parameter 0~255, and the suffix f=0, …, 7. In practice, we observed that the histogram of detected contour is significantly influenced by capturing angle of the ceiling mounted camera. To accommodate these differences, an ROI is divided into four co-centric areas, as shown in Fig. 9. Four identification signatures associated to the contour images taken from the four respective areas are constructed for each individual. Together, these four make an identity signature for the current user ID till he/she leaves from the floor of plan. Throughout the operation, the proposed PTS, composed of several ROIs, keeps the identity signatures for all the active people current in the system. Identification of a person’s identity by checking the current person’s identity signature against all active identity signatures is initiated when ROI fails to track the object, or when the pedestrian walks into the different ROI. In [7], the margin of error Ed is evaluated as Ed = Ed + C Ai f − C A* f
(7)
184
C.-H. Lin, S.-L. Chung, and J.-M. Guo
where C A* and C Ai denote the current identity and the identities in the system’s dataf f base, respectively. A threshold, empirically 800, is used for Ed to determine whether a new user ID is created. When many C Ai ’s margin of error are smaller than the threshold, f
PTS selects the one with the minimum Ed as the detected object’s identity.
3 Experimental Result Figure 10 shows the overall appearance of the GUI, where eight components are involved: 1) Real-time view of images captured from all installed cameras on the floor plan; 2) detected object view of current camera; 3) background update window; 4) message window used to show entrance and departure from a ROI of a person; 5) function buttons used to start facility connection, manually update background, start and stop operation, and delete data; 6) databases query for a people tracking history records on a selected ROI and indicated timing interval; 7) response view for query results, and 8) historical trajectories view which replays the entrance history of a selected object from his/her entrance to the floor plan till departure.
Fig. 10. GUI of the proposed PTS
The proposed PTS has been put to operate continuously on the third floor of NTUST for three hours: 2008-08-25 AM11:00~PM14:00. The Correct Detection Rate (CDR) of the proposed method has been checked against the Ground Truth (GT) in the recorded video. The experimental results are organized with three types of activities: 1.
2.
3.
Checking Leaving Direction (CLD): The number of passing people entering and then leaving into and from the current ROI; this is the number of people passing through the current ROI. Checking User ID (CUI): Number of right match of a user ID when the designated person leaves from an ROI and enters into another; this is the number of people who are being corrected tracked between ROIs. Checking Merges (CM): The individual counts of people when entering an ROI as a group, which is the number of people who are separated from a cluster of people and counted as individual.
People Tracking in a Building Using Color Histogram Classifiers and GWIS Approaches
185
An inspection is provided, which can be summarized to some possible reasons: 1.
2.
3.
Lighting effect: The lighting conditions in Cam 1, Cam 6, Cam 7 and Cam 8 are not equal. When a person walks under the dark area, PTS may mistakenly take his/her shadow as part of foreground image. Unreachable area: The respective width of corridor of Cam5 and Cam8 is larger than that of ROI: the passing of a person may not be detected by the cameras above. Object similarity: This work uses the cameras mounted on the ceiling to overlook the objects. The features which can be utilized are much less than lateral or front view detections. Thus, when individuals with similar contours or outfit colors are difficult to be separated.
Table 1 organizes the overall performances: with 750 people passing through all ROIs during the three-hour period, the CDRs of the CID, CUI, and CM are 90%, 76%, and 70%, respectively. Regarding the processing time, the proposed PTS system can process 84~495 frames/sec with images of size 160x120 using eight cameras. In average, it takes 1.706 ms to process an image when no object enters the system, and 10.8 ms when objects enter. We now compare the features of the proposed PTS with some former related works in the literature. By making the assumption that a walking person is less likely to change his/her clothes, studies of [8] and [1] use color histograms of the foreground object as the signature. If two people keep being overlapped, the signature obtained from color histograms cannot be trusted, and the CDR decreases accordingly. In [4], overhead analysis is also adopted by setting one camera on the ceiling to avoid the overlapping problem. However, the scope is limited to one camera and thus one ROI. In contrast, the coverage of the proposed PTS is a floor plan that is monitored by several cameras; as such, coordination among different ROIs is important. We apply the approach of setting camera on the ceiling to avoid the overlapping problem. Accompanying with this strategy, the CHC is employed to provide a stable identified data under multi-camera, which can avoid identifying color histograms from being influenced by lighting effect. Table 1. Overall Performances CID
CUI
CM
Ground truth
750
750
79
Result
701 93%
572 76%
55 70%
Correct detection rate
4 Conclusions In this work, we investigate the design and implementation of PTS by distributed ceiling mounted cameras. By dividing the monitored floor plan into adjacent areas, the
186
C.-H. Lin, S.-L. Chung, and J.-M. Guo
proposed PTS intends to keep tracking of not only the number of people entrance to each area, but also the presence trajectory of each entrant. Each entrant’s physiological contour in a ROI under a camera is characterized from four different viewing angles. With the proposed HCD and GWIS, the head centers and the associate physiological contours are determined. Surveillance experiments on the third floor of EE department in NTUST have been conducted. With techniques geared to reduced processing time at each ROI and the pedestrian handover between ROIs, the system response time, collected from our experiment, shows satisfactory: With eight cameras installed, the proposed PTS updates images 240 frames/sec for the whole system, or in averagely, 10 frames/second for each camera. The accuracy of the proposed system has been checked with recorded video. The result shows moderate surveillance performance: 93% accuracy in entrance counting, and 76% accuracy in identification checking.
References 1. Bahadori, S., Iocchi, L., Leone, G.R., Nardi, F., Scozzafava, L.: Real-Time Location and Tracking Through Fixed Stereo Vision. Applied Intelligence 26(2), 83–97 (2007) 2. Krumm, J., Harris, S., Meyers, B., Brumitt, B., Hale, M., Shafer, S.: Multi-Camera Multi-Person Tracking for EasyLiving. In: Proceedings of the 2000 Third IEEE International Workshop on Visual Surveillance, pp. 3–10 (July 2000) 3. Nohara, K., Tajika, T., Shiomi, M., Kanda, T., Ishiguro, H., Hagita, N.: Integrating Passive RFID Tag and Person Tracking for Social Interaction in Daily Life. In: Proceedings of the 17th IEEE International Symposium on Robot and Human Interactive Communication, RO-MAN, art. no. 4600723, pp. 545–552 (2008) 4. Velipasalar, S., Tian, Y.L., Hampapur, A.: Automatic counting of interacting people by using a single uncalibrated camera. In: Proc. 2006 IEEE International Conference on Multimedia and Expo., pp. 1265–1268 (2006) 5. Sonka, M., Hlavac, V., Boyle, R.: Image processing, analysis, and machine vision, 2nd edn. PWS Publishing (1998) 6. Negnevitsky, M.: Artificial Intelligence: A Guide to Intelligent Systems. Addison-Wesley, Reading (2002) 7. Piva, S., Zara, M., Gera, G., Regazzoni, C.S.: Color-based video stabilization for real-time on-board object detection on high-speed trains. In: Proceedings of IEEE Conference on Advanced Video and Signal Based Surveillance, pp. 299–304 (July 2003) 8. Krumm, J., Harris, S., Meyers, B., Brumitt, B., Hale, M., Shafer, S.: Multi-camera multi-person tracking for easyLiving. In: Proceedings of the 2000 Third IEEE International Workshop on Visual Surveillance, pp. 3–10 (July 2000)
Human-Centered Fingertip Mandarin Input System Using Single Camera Chih-Chang Yu1, Hsu-Yung Cheng2, Bor-Shenn Jeng3, Chien-Cheng Lee3, and Wei-Tyng Hong3 1
Department of Computer Science and Information Engineering, Vanung University, No.1, Wannung Rd., Jhongli City, Taoyuan, Taiwan 2 Department of Computer Science and Information Engineering, National Central University, No.300 Jhongda Rd., Jhongli City, Taoyuan, Taiwan 3 Department of Communications Engineering / Communication Research Center, Yuan-Ze University, Chung-Li, Taoyuan, Taiwan
Abstract. Designing a user friendly Chinese input system is a challenging task due to the logographic nature of Chinese characters. Using fingertips and cameras to replace pens and touch panels as input devices could reduce the cost and improve ease-of-use and comfort of the computer-human interface. In this work, Chinese character entry is achieved using Mandarin Phonetic Symbol (MPS) recognition via on-line fingertip tracking. In the proposed system, particle filters are applied for robust fingertip tracking. Afterwards, MPS recognition is performed on the tracked fingertip trajectories using Hidden Markov Models. In the proposed system, the challenges of entering, leaving, and virtual strokes caused by video-based fingertip input can be overcome. We conduct experiments to validate that the MPS symbols written by fingertips are successfully and efficiently recognized using the proposed framework. Keywords: Vision-based; Mandarin Phonetic Symbol recognition.
1 Introduction With the prevalence of mobile devices and all-in-one computers, consumers desire a better and easier to use input interface. Pen computing emerged due to the urgent need of convenient input interfaces. However, pens could be easily lost and would cause great inconvenience. Therefore, using fingertips to replace pens is more favorable for users. Special devices have been investigated to provide more convenient input interface, such as the pressure sensitive marking menus described in [1]. Also, touch panels that require no pens have become very popular on mobile devices such as smart phones in recent years. However, touch screens are thicker and heavier compared to normal display devices. Therefore, video-based fingertip input is a possible way of human interface design in small and light weight portable devices since both high-tier and low-tier cell phones are equipped with cameras nowadays. As for laptop or desktop computers, touch panels are criticized to be not conformed to ergonomics under some circumstances. Especially for the emerged all-in-one computers, using K.-T. Lee et al. (Eds.): MMM 2011, Part II, LNCS 6524, pp. 187–195, 2011. © Springer-Verlag Berlin Heidelberg 2011
188
C.-C. Yu et al.
touch panels as input interface is not convenient because the users would need to approach the screen. By incorporating video-based fingertip tracking techniques, the proposed video-based input concept could allow users to control the computer and input data with ease. The concept of the proposed input interface is to allow users to use their fingers to write on any surface. Video-based fingertip input system is an alternative input method with high potential that provides improved ease-of-use and comfort to mobile device and computer users. Chinese character input is challenging because it is not as straightforward as typing languages based on alphabets. Hanyu Pinyin and Mandarin Phonetic Symbol (MPS) are two important methods for Chinese character entry. In Taiwan, the Mandarin Phonetic Symbol set, which includes 37 symbols as shown in Figure 1, is the most prevailing Chinese input method. However, many users have found the design of MPS layout on keyboards or keypads unsatisfactory, especially on all-in-one and mobile devices. In this paper, we propose a Chinese input system via Mandarin Phonetic Symbols using video-based fingertip tracking techniques with single camera. The rest of paper is organized as follows. In the next section, the proposed system is explained in details. In Section 3, we demonstrate the experimental results. In Section 4, conclusions are made and future works are discussed.
Fig. 1. The 37 Mandarin Phonetic Symbols
2 The Proposed System Figure 2 illustrates the framework of the proposed system. In the proposed system, the fingertip is identified and tracked in the input video automatically so that the trajectory of the fingertip can be recorded. Then, the recorded trajectory of the fingertip is recognized as Mandarin Phonetic Symbols. Finally, the Chinese Characters can be determined by the input Mandarin Phonetic Symbols and user selection. The fingertip detection and tracking procedures are described in subsection 2.1. Recognition of Mandarin Phonetic Symbols is elaborated in subsection 2.2. 2.1 Fingertip Detection and Tracking The fingertip is detected by adopting the method described in [2]. First, the hand silhouette is extracted using background subtraction. Then, we employ the inner border tracing algorithm to extract the border of the silhouette. These ordered pixels are called border-pixel-vector (BPV). As shown in Figure 3 (a), we select the middle point Pm of line P1 P2 as the reference point. Then, we calculate the Euclidean
Human-Centered Fingertip Mandarin Input System Using Single Camera
189
distance between Pm and BPV in counterclockwise order. A distance distribution diagram shown in Figure 3 (b) can be obtained and the global maximum corresponds to the location of the fingertip (Figure 3 (c)).
Fig. 2. Proposed System Framework
(a)
(b) (c) Fig. 3. Fingertip Detection
We apply a particle filter to track the target fingertip in this work. Particle filters have abilities to deal with non-linear motions and non-Gaussian noises [3] and therefore are suitable for dealing with targets that are close to the camera and having dramatic motion changes that cannot be easily modeled by Kalman filters. Various designs of particle filters have been applied in different vision based applications and systems [3]-[6]. Formulating the tracking problems as dynamic systems, the distribution of p( x k | z1:k −1 ) is obtained via the Chapman–Kolmogorov equation [4] in the prediction stage under the assumption that the pdf p ( x k −1 | z1:k −1 ) at time instance k-1 is known, as shown in Eq. (1). Here x k denotes the system state at time instance k and z1:k −1 denotes the measurement from time instance 1 to k-1. In the update stage,
190
C.-C. Yu et al.
when the measurement
z k at time instance k is available, it can be used to update the
required pdf p( x k | z1:k ) using Eq. (2). The particle filters aim to approximate the probability distribution of the system state by a sample set xk(i) , i = 1… Ns, each associated with a weight wk (i). In the proposed framework, we use an ellipse to model the region of the target fingertip. We assume that the ratio of the major axis a k to minor axis bk remains constant at every time instance k. Therefore we only need to consider the position, scale, and rotational angle of the ellipse. Thus, the system state x k at time instance k is defined as x k = [u k v k s k θ k ]T , where u k and v k are the coordinates of the target fingertip in the image plane, s k is the scaling factor, and θ k is the rotational angle of the ellipse. p( x k | z1:k −1 ) = ∫ p ( x k | x k −1 ) p( x k −1 | z1:k −1 ) dx k −1
p ( x k | z1:k ) =
p ( z k | x k ) p ( x k | z1:k −1 ) p ( z k | z1:k −1 )
(1) (2)
where p ( z k | z1:k −1 ) = p ( z k | x k ) p( x k | z1:k −1 )dx k . ∫ 2.2 Mandarin Phonetic Symbol Recognition For recognizing symbols and characters, Optical Character Recognition (OCR) and Online Character Recognition (OLCR) have both had significant advances over the past decades and achieved very high recognition rates. However, the characters obtained by the proposed video-based fingertip input system have several key differences from traditional OCR or OLCR characters obtained by scanners, touch panels or pen tablets. First of all, in our applications, the characters or symbols would include the entering stroke and leaving stroke. Secondly, all the virtual strokes are mixed with the actual strokes. Moreover, the trajectories obtained by video object tracking may not be smooth. Therefore, the challenges of the proposed system are greater than those of the traditional OCR and OLCR. As shown in Figure 4, the entering stroke and the leaving stroke are the first and the last virtual strokes, respectively. And the starting stroke and ending stroke are the first and the last real strokes. There could be intermediate virtual strokes between the real strokes (as shown in Figure 4 (b)). We have observed that the intermediate virtual stroke is somewhat stable. However, the entering and leaving strokes are very unstable because the fingertip of the users could enter the scene from anywhere. The unstable entering and leaving strokes would severely interfere with the recognition process. Hence, it is necessary to eliminate the entering and leaving strokes to obtain reasonable recognition results. In order to distinguish the entering and leaving strokes from other strokes, we detect the turning points of the tracked trajectory. First, the trajectory is encoded using an Inter-Frame Directional (IFD) code with eight directions, as shown in Figure 5. The trajectory of a phonetic symbol generates a sequence of code numbers. For example, the symbol could generate the following code sequence: 666660002222666666.
Human-Centered Fingertip Mandarin Input System Using Single Camera
191
Fig. 4. Entering stroke, leaving stroke, starting stroke, and ending stroke 2 3
1
4
0 5
7 6
Fig. 5. The inter-frame directional code with eight directions
To detect the turning point, we first take the difference between the code of each frame ct and the code of its previous frame ct −1 . The difference value dt at frame t is defined in the following equation.
⎧ct − ct −1 − 8 if ct − ct −1 > 4 ⎪ d t = ⎨ct − ct −1 + 8 if ct − ct −1 < −4 ⎪c − c otherwise t −1 ⎩ t
(3)
When dt equals to zero, the direction of fingertip does not change. When dt is positive, the fingertip changes its direction counterclockwise. Conversely, the fingertip changes its direction clockwise when dt is negative. Two criteria are used for turning point detection. More specifically, the position of fingertip (ut,vt) is regarded as a turning point if
192
C.-C. Yu et al.
1. both |dt| and |dt+dt+1| are above the threshold. 2. |dt+dt-1| is above the threshold. In our experiments, the threshold value is 2, which is about 90 degree. The detected turning points of a symbol are shown in red circles in Figure 4 (a). The trajectory between two turning points can be regarded as a stroke. We check all the strokes and eliminate those strokes which are too short. The short strokes are usually noises and provide little contribution for recognition. Hence, we discard these short strokes for better recognition results. After obtaining the turning points, we can define the entering and leaving strokes, which correspond to the trajectories before the first turning point and the last turning point. However, sometimes there is no leaving stroke. This is because the users may leave the scene very quickly, or the leaving stroke may have the same direction of the ending stroke. In such cases, we should not eliminate the last stroke. Thus, we check the length of the last stroke. If the last stroke is long enough, we treat it as an ending stroke and keep it. Otherwise we treat it as the leaving stroke and discard it. Figure 4 (a) and (b) show two recorded trajectories with entering and leaving strokes. Figure 4 (c) and (d) show the remaining strokes for the two trajectories after entering and leaving stroke elimination. In recent years, the Hidden Markov Models (HMMs) [7] have been extensively applied to speech recognition to resolve segmentation, time warping, stochastic randomness, etc. Prominent results have been widely recognized in the open literature. The length of the IFD code sequence varies due to the writing speed, and HMMs are suitable recognition models for variable length data. Therefore, MPS recognition is performed using HMMs. A Hidden Markov Model is a finite set of states, each of which is associated with a probability distribution. In a particular state an observation can be generated, according to the associated probability distribution. The IFD codes for each sequence serve as observations of HMM. Given an HMM λ and a sequence of observations O = {O1, O2,…, Ot}, the probability p(O|λ) which represents how possible O is generated by λ can be computed. Each Mandarin Phonetic Symbol s is trained independently to optimize the model parameter λs by a set of given observation sequence O. The training process can be achieved by the Baum-Welch algorithm and the likelihood is calculated using the forward algorithm [7].
3 Experimental Results In this section, we demonstrate the experimental results that verify the feasibility and effectiveness of the proposed system. First, the intermediate tracking trajectories of the symbol are shown in Figure 6 (a)-(h). We can clearly see the entering and leaving strokes of the tracked trajectory. The effects of unstable entering and leaving strokes are demonstrated in Figure 7. In Figure 8, we show the effect of the entering and leaving stroke elimination procedure. After eliminating the entering and leaving strokes from the original tracked trajectories (Figure 8 (a)), the remaining strokes (Figure 8 (b)) are much more stable and suitable for the recognition purpose.
Human-Centered Fingertip Mandarin Input System Using Single Camera
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
193
Fig. 6. Intermediate tracking trajectories of the symbol
Fig. 7. Unstable entering and leaving strokes
(a)
(b) Fig. 8. Tracked trajectories. Top row: original tracked trajectories. Bottom row: trajectories after removing entering and leaving strokes.
Twenty Mandarin Phonetic Symbol sets are collected from five different people, including one left-hander and four right-handers. We adopt the leave-one-out cross validation rule in order to obtain unbiased results. Due to the effect of confusion symbols, we use the cumulative matching score (CMS) as the recognition results. Chinese input system usually involves user selection and therefore top ranked symbols can be provided to the user for selection. Since the confusion symbols are less than 3, we only list the top three matching result. Moreover, we also examine the effect on recognition rate using different number of hidden states in HMM. Table I lists the recognition rate using different number of hidden states. We can find that the recognition rate (best 3 matches) is 98.38% with standard deviation 2.39 when the number of states in HMM equals to 4. Figure 9 shows a clear comparison of recognition rates using different number of states Q, which verifies the robustness of the proposed method. Notice that the recognition rates in Table I and Figure 9 are before confusion symbol verification. It is noticeable that the symbol and are totally dissimilar in their appearance. However, they have quite similar inter-frame directional code sequences which would confuse the trained HMMs. Overall speaking, the recognition rate using HMM is
194
C.-C. Yu et al.
satisfying at rank 3. We believe that with further enhancement of confusion symbol verification procedure, high recognition rates that support practical usage can be achieved. Table 1. Recognition rate of the 37 phonetic symbols
Number of hidden states
Rank 1 70.41 % ± 10.09 % 81.22 % ± 7.26 % 82.84 % ± 8.56 % 81.35 % ± 6.07 % 81.49 % ± 7.91 %
Q=2 Q=3 Q=4 Q=5 Q=6
Rank 2
Rank 3
88.92 % ± 6.73 % 92.97 % ± 3.44 % 94.59 % ± 4.30 % 93.92 % ± 3.81 % 93.51 % ± 5.5 %
96.49 % ± 3.52 % 96.89 % ± 2.2 % 98.38 % ± 2.39 % 97.57 % ± 3.02 % 97.30 % ± 2.77 %
rec o g n itio n ra te(% )
100 90 80
R an k 1
70
R an k 2 R an k 3
60 50 2
3
4
5
6
numb er o f hid d en state s
Fig. 9. The recognition rate of 37 phonetic symbols using different number of hidden states
4 Conclusions This paper proposes a video based Chinese input system. Video based input system using fingertips is a new research topic that is not seen in the current literature a lot. However, it is certainly a topic with strong potential and high research value. Although this paper focuses on Mandarin Phonetic Symbols, the concept of the proposed video-based fingertip input method can also be applied to other general input systems. Unlike other languages based on alphabets, Chinese character input systems are more challenging because of its logographic nature. In this work, a particle filter is used to track the target fingertip. The tracked trajectories are coded by Inter-Frame Directional coding. After locating the turning points and strokes of the input, the entering and leaving strokes can be eliminated so that the recognition process would not
Human-Centered Fingertip Mandarin Input System Using Single Camera
195
be affected by these unstable virtual strokes. Then, the Inter-Frame Directional code of each trajectory is analyzed by Hidden Markov Models for Mandarin Phonetic Symbol recognition. From the experimental results, we verify the performance of the proposed system. For future work, we would investigate algorithms that could robustly locate and track fingertips in backgrounds of various colors and under different lighting conditions. Also, developing sophisticated confusion symbol verification procedures could greatly improve the accuracy of recognition. We believe that the proposed input interface is with high potential and can be applied to various devices.
Acknowledgments The authors would like to thank National Science Council of Taiwan for supporting this work. This work is supported by National Science Council under project code 99-2628-E-008-098.
References [1] Lai, J.T.C., Li, Y., Anderson, R.: Donuts: Chinese Input with Pressure-Sensitive Marking Menu. In: ACM Symposium on User Interface Software and Technology (October 2005) [2] Han, C.-C., Cheng, H.-L., Lin, C.-L., Fan, K.-C.: Personal Authentication Using Palmprint Features. Pattern Recognition 36(2), 371–381 (2003) [3] Isard, M., Blake, A.: Condensation – Conditional density propagation for visual tracking. International Journal on Computer Vision 1(29), 5–28 (1998) [4] Arulampalam, M.S., Maskell, S., Gordon, N., Clapp, T.: A tutorial on particle filters for online nonlinear/non-Gaussian Bayesian tracking. IEEE Transactions on Signal Processing 50(2), 174–188 (2002) [5] Zheng, W., Bhandarkar, S.M.: Face detection and tracking using a Boosted Adaptive Particle Filter. Journal of Visual Communication and Image Representation 20(1), 9–27 (2009) [6] Gustafsson, F., et al.: Particle filters for positioning, navigation and tracking. IEEE Transactions on Signal Processing 50(2), 425–437 (2002) [7] Rabiner, L.R.: A tutorial on hidden Markov models and selected applications in speech recognition. Proceedings of the IEEE 77(2), 257–286 (1989)
Automatic Container Code Recognition Using Compressed Sensing Method Chien-Cheng Tseng1 and Su-Ling Lee2 1
Department of Computer and Communication Engineering, National Kaohsiung First University of Science and Technology, Kaohsiung, Taiwan
[email protected] 2 Department of Computer Science and Information Engineering, Chung-Jung Christian University, Tainan, Taiwan
[email protected]
Abstract. In this paper, an automatic container code recognition method is presented by using compressed sensing (CS). First, the compressed sensing approach which uses the constrained L1 minimization method is reviewed. Then, a general pattern recognition framework based on CS theory is described. Next, the CS recognition method is applied to construct an automatic container code recognition system. Finally, the real-life images provided by trading port of Kaohsiung are used to evaluate the performance of the proposed method. Keywords: Container code recognition, Compressed sensing, L1 minimization.
1 Introduction For the purpose of efficient transportation, it is important to automatically and intelligently manage the containers in the gates of the trading port. If the gates are managed by human inspection and manual registration to ensure the correct access of containers, it is a very time-consuming task. To increase the competition capability of port, it is necessary to construct an automatic system that records the exit and entry of the containers by using the techniques of pattern recognition, image processing and computer vision. Until now, many academic research papers and commercial products have been reported to achieve this purpose [1]-[7]. These systems are called container number recognition, container code recognition or container identity number recognition etc. Generally speaking, there are four stages in these methods. First, the captured image is preprocessed to enhance the quality of image and eliminate the unwanted distortion or noise. Second, the container code area is extracted to remove the background. Third, the container code is segmented into a character string. Finally, the character recognition engine is applied to recognize each character. The more details of these methods are briefly described below: In [1], a neural network model is applied to container number recognition. This method can attain a high level of accuracy even though the input patterns are highly distorted. In [2], a vehicle and container number recognition system is developed by using techniques of image enhancement, fast focusing, variable thresholding, image K.-T. Lee et al. (Eds.): MMM 2011, Part II, LNCS 6524, pp. 196–207, 2011. © Springer-Verlag Berlin Heidelberg 2011
Automatic Container Code Recognition Using Compressed Sensing Method
197
normalization, neural network and domain knowledge filter. Under outdoor environment test, this method has proved to achieve accuracy higher than 95%. In [3], the segmentation of container code is carried on by applying top-hat morphological operator, thresholding, connected component labeling and size filter. In [4][5], the container image is filtered with both adaptive linear and nonlinear filters to reduce noise. Tests have been made to show character lines can be properly located with a rate above 98%. In [6], the feature-based local intensity gradient and adaptive multithreshold methods are combined to segment the codes from the background with high performance. In [7], a commercial product called SeeGate container code recognition system has been developed. On the other hand, a new signal acquisition method is discovered in recent years, named as compressed sensing (CS) or compressive sampling. This new approach states that a signal can be precisely recovered from only a small set of measurements under certain conditions of sparsity and incoherence [8]. So far, CS theory has been successfully applied to various signal processing areas including image compression, face recognition, speech recognition, radar, astronomy, medical imaging, video coding and DNA microarrays [9]-[17]. Due to the success of compressed sensing, it is interesting to use this method to develop a recognition system that can recognize the container code in various situations. The purpose of this paper is to solve this problem, as shown in Fig.1. This paper is organized as follows: In section 2, the compressed sensing approach which uses constrained L1 minimization method is reviewed. In section 3, a general recognition framework based on CS theory is described. In section 4, the CS recognition method is applied to construct an automatic container code recognition system. In section 5, the experimental test results are shown to demonstrate the effectiveness of the proposed method. Finally, a conclusion remark is made.
Location and Recognition algorithm
GSTU3601680
Fig. 1. The container code recognition problem. The input is the captured image and the output is the container code with 11 characters.
2 Compressed Sensing In this section, the compressed sensing (CS) theory is reviewed briefly. There are two parts in CS framework. One is measurement, the other is reconstruction. Now, these two parts are descried below. Let signal x be x = [ x1 , x 2 , " , x N ] T , where N is the number of signal. To get the information of signal x , we choose M basis φ k = [φ k 1 , φ k 2 , " , φ kN ]T ( k = 1, 2 , " , M ) to sense this signal. The measured signals are given by
198
C.-C. Tseng and S.-L. Lee
y k = x,φ k
k = 1, 2, " , M
(1)
where <, > denotes the inner product operator. Using the matrix representation, (1) can be written as the form
y = Φx
(2)
The above matrix Φ and vector y are
⎡ φ11 ⎢φ Φ = ⎢ 21 ⎢ # ⎢ ⎣φ M 1
φ12 φ 22
" φ1 N " φ2N % # " φ MN
⎤ ⎥ ⎥ ⎥ ⎥ ⎦
(3a)
y = [ y1 , y 2 , " , y M ]T
(3b)
#
φM 2
In many signal processing applications, the signal x is very sparse, that is, x contains only K non-zero valued elements where K is much less than N . Based on this sparseness assumption, we can reconstruct the signal x = [ x1 , x 2 , " , x N ] T from the measurement y = [ y 1 , y 2 , " , y M ] T even though M is much less than N . Thus, y is a sparse representation of the signal x . Next, the details of reconstruction are described. In the literature, there are two reconstruction cases to be studied. One is noiseless case, the other is noisy case. In noiseless case, the signal x can be recovered from measurement y by solving the constrained L1 minimization problem below: Min x
1
subject to y = Φ x
(4)
where L1 norm is defined by
x =
1
= | x1 | + | x 2 | + " | x N | (5)
N
∑ | xi | i =1
The problem in (4) can be solved by using linear programming easily. For noisy case, the measurement is corrupted by noise, that is,
y = Φx + v
(6)
where v is zero-mean white noise vector with size M × 1 . In this case, the L1 minimization with quadratic constraint is used to recover signal x approximately below: Min x
1
subject to
y − Φx
2
≤ε
(7)
where ε bounds the amount of noise in the measurement. This problem can be solved by convex programming.
Automatic Container Code Recognition Using Compressed Sensing Method
199
3 Pattern Recognition Using Compressed Sensing In this section, the conventional pattern recognition method based on similarity matching is first reviewed. Then, a pattern recognition approach using compressed sensing is developed. Given the pre-stored template vectors ψ 1 , ψ 2 ,…, ψ L and the input pattern P , conventional recognition procedure is described below: Step 1: Extract the feature vector Y from the input pattern P . Step 2: Compute the similarity score between feature vector Y and each template vector ψ 1 , ψ 2 ,…, ψ L . One typical method is to compute the normalized correlation below:
si =
Y ,ψ i
i = 1, 2 ,..., L
Y ψi
(8)
Step 3: Perform similarity score sorting and find the highest one to make decision. That is, if
k = arg max s i
(9)
i
then input pattern P belongs to the k th class. Based on the above procedure, it is clear that two main tasks of conventional method are similarity matching and score sorting. Using the sparse representation in compressed sensing, these tasks can be accomplished by solving constrained L1 minimization. The details are described below. When the compressed sensing theory is applied to the pattern recognition problem, the most important thing is to get the sparse representation of the problem. From the conventional recognition method, we see that the feature vector Y must be almost equal to one of the template vector ψ k if the input pattern P is close to one of the pattern used to construct template vector. Based on this fact, feature vector Y can be written as
Y ≈ [ψ
1
ψ
ψ
"
2
k
"
ψ
k
"
L
⎡0 ⎤ ⎢0 ⎥ ⎢ ⎥ ⎢#⎥ ]⎢ ⎥ ⎢1 ⎥ ⎢#⎥ ⎢ ⎥ ⎣⎢ 0 ⎦⎥
Define matrix Φ and vector X below
Φ = [ψ 1
ψ
X = [0
0
2
"
"
ψ 1
"
ψ 0]
T
L
(10)
]
(11a) (11b)
then (10) can be rewritten as
Y ≈ ΦX
(12)
200
C.-C. Tseng and S.-L. Lee
Because there is only one non-zero element in vector X , this vector is sparse. So, (12) is a sparse representation of input feature vector. Let V be the error between Y and Φ X , then (12) can be further written as
Y = ΦX + V
(13)
This means that feature vector Y is a noisy measurement of Φ X . Using CS theory, the sparse vector X can be recovered from the measurement Y , so we can get the location k of nonzero element in the recovered vector X to accomplish the recognition task. Finally, the procedure is summarized below: Step 1: Extract the feature vector Y from the input pattern P . Step 2: Solve the following L1 minimization with quadratic constraint to recover signal X from the feature vector Y : Min X
1
subject to Y − Φ X
2
≤ε
(14)
Let the optimal solution of this problem be denoted by X opt . Step 3: Find the maximum non-zero element of X opt . If the location of this element is k, then input pattern P belongs to the k th class. In next section, the above recognition method will be used to construct an automatic container code recognition system.
4 Container Code Recognition System In this section, we will develop a container code recognition system. The flow chart of the proposed algorithm is shown in Fig.2. The main stages include preprocessing, binarization, morphological filtering, location, segmentation, normalization, CS-based character recognition and post-processing. In the following, the details of each stage will be described. Stage 1. Preprocessing: There are two tasks in preprocessing stage. One is to reduce the image size, the other is to transform color input image to gray level image. Fig.3 shows the preprocessing results including the input image and output image. Stage 2. Binarization and morphological filtering: For binarization, the histogram of gray level image is first computed. Then, we choose a threshold T to let the percentage of pixels whose values are greater than T be equal to 15%. Finally, the binary image is obtained by using the rule: If the pixel value is greater than T, it is set to one. Otherwise, the pixel is set to zero. For morphological filtering, the binary image is first eroded 4 times to remove noise. Then, the eroded image is dilated 4 times to restore the image size. Fig.4 shows the processing results of binarization and morphological filtering. Stage 3. Character location and segmentation: Five steps involved in this stage are described below: First, the 8-adjacent connected component labeling is used to extract all possible objects in the binary image. Second, the objects whose size and aspect ratio are much different from those of characters of container code are removed. Third, we find the most right-side object and use it to determine the candidate area of
Automatic Container Code Recognition Using Compressed Sensing Method
201
Container image
Preprocessing
Image binarization and morphological filtering
Character location, segmentation and normalization
Character template
CS-based recognition
Post-processing
Recognition result
Fig. 2. The flow chart of the proposed container code recognition method based on compressed sensing (CS)
(a)
(b)
Fig. 3. The preprocessing results. (a) Input color container image. (b) The output image.
202
C.-C. Tseng and S.-L. Lee
(a)
(b)
(c)
(d)
Fig. 4. The results of binarization and morphological filtering. (a) Histogram. (b) Binary image. (c) Eroded image. (d) Dilated image.
(a)
(b)
(c)
(d)
Fig. 5. The processing results of location and segmentation. (a) The 24 objects after performing 8-adjacent connected component labeling. (b) The 17 objects after performing the size filtering. (c) The candidate area (white highlighted area) after find the most right-side object "o". (d)) The final 11 extracted objects.
Automatic Container Code Recognition Using Compressed Sensing Method
203
container code. Forth, all objects in the candidate area are extracted and arranged in order from left to right. Fifth, each object is normalized to the prescribed size for detailed matching. Fig.5 shows show the processing results of this stage. Fig.5(a) shows the 24 objects after performing 8-adjacent connected component labeling. Fig.5(b) depicts the 17 objects after performing the size filtering. Fig.5(c) shows the candidate area (the white-highlighted area) after finding the most right-side object "0". Fig.5(d) shows the final 11 extracted objects. Clearly, 11 characters of container codes have been segmented successfully. Stage 4. Character template: In this paper, the multi-template is built for solving the multi-font problem. Fig.6 shows 60 templates in our method. Based on the statistical data, we know that English alphabets J, K, O, P, Q, V, Y, Z, are not used in container code, so these templates are not necessary to be built. Moreover, there exist some distorted characters in container code image, so several distorted templates are also built for the robustness of recognition.
Fig. 6. The multi-font character template of container code whose size is equal to 50x50
Stage 5. CS-based recognition: Because there are eleven segmented characters in container image, the recognition problem is how to obtain the content of these eleven characters which are composed of number digits and English alphabets. The first task of the CS-based method is to construct the matrix Φ from character images in Fig.6. Given all template images with size N r × N c in the database, the procedure to con-
struct the matrix Φ is described below:
Step 1: Construct a random matrix R with size M × N r N c . Each element in matrix R is a zero-mean unit-variance Gaussian random variable. Note that the Gram Schmidt orthogonalization method is also applied to transform matrix R into a unitary matrix. Step 2: Transform two-dimensional (2-D) i-th binary image in the database into a onedimensional (1-D) vector B i by arranging pixel values according to the order from left to right and from top to bottom. The size of vector B i is N r N c × 1 .
204
C.-C. Tseng and S.-L. Lee
Step 3: The template vector ψ i is then given by
ψ i = RB i
(15)
Step 4: Use all template vectors to construct matrix Φ below:
Φ = [ψ 1 ψ 2
" ψk
" ψL]
where L is the number of template images. After template matrix Φ has been constructed, the second task is how to recognize the unknown input segmented character image P by using matrix Φ . The CS-based recognition procedure is described below: Step 1: Transform 2-D input binary image P into a 1-D vector A by arranging pixel values according to the order from left to right and from top to bottom. The size of vector A is N r N c × 1 . Step 2: The feature vector Y is then given by
Y = RA
(16)
Step 3: Solve the following constrained L1 minimization problem to get the vector X from the feature vector Y : Min X
1
subject to Y − Φ X
2
≤ε
Let the optimal solution of this problem be denoted by X opt . Step 4: Find the maximum non-zero element of X opt . If the location of this element is k, then input image P belongs to the k th class. Now, one example are used to illustrate the effectiveness of the above CS-based recognition method. The input image is “D” in Fig.7(a). The parameters are chosen as N r = N c = 50 , L = 60 , M = 100 , and ε = 0 . 001 . The optimal solution X opt of the constrained L1 minimization problem is shown in Fig.7(b). It is clear that there is the largest peak at index k = 5 that corresponds to the English alphabet “D”, as shown in template database of Fig.6. Thus, the recognition result is correct. 150
Xopt(k)
100
50
0
(a)
5
10
15
20
25
30 k
35
40
45
50
55
60
(b)
Fig. 7. A recognition example of the CS-based method. (a) Input character image. (b) The optimal solution X opt of the L1 minimization.
Automatic Container Code Recognition Using Compressed Sensing Method
205
Stage 6. Post-processing: In one of the following three cases, we know that the recognition result must be incorrect. First, if number of character of container code is not equal to eleven, the result is not correct. This wrong result comes from the fail of segmentation stage. In our experience, the alphabet "U" often distorts into "11" such that the 8 adjacent connected component labeling can not group "11" as one object "U". Second, the fourth character of all container images provided by Kaohsiung port is always the alphabet "U". Thus, when the fourth character of recognition result is not "U", the result must be incorrect. Third, the check digit (i.e. the last character) can be used to check whether the recognition result is correct or not even though the number of character is 11 and the fourth character is U. The checking rule can be found in the literature easily. So far, the recognition method has been described. In next section, several experiments are used to evaluate the performance of this method.
5 Experimental Results In this section, several experiments are conducted to test the proposed method. And, some discussions are also made. Experiment 1: In this experiment, successful cases are reported. Fig.8 shows the recognition result. The input container images are white characters with blue or red backgrounds. Because our algorithm does not use color information, the recognition method is robust to color variation. After checking by inspection, it is easy to see that recognition results are correct.
(a)
(b)
Fig. 8. The recognition results in experiment 1. (a) White character with blue background. (b) White character with red background.
Experiment 2: In this experiment, fail cases are reported. Fig.9 shows the recognition result. It is easy to see that there are some severely distorted characters in the input container images which cause the segmented results are incorrect. However, the users know the results are not correct in the post-processing stage because the total number of character is not equal to 11. Thus, some remedy procedure may start for handling
206
C.-C. Tseng and S.-L. Lee
these fail recognition cases. In our experience, most of incorrect results are due to the fail of character location and segmentation. Thus, to enhance the location and segmentation algorithms is an important future work.
(a)
(b)
Fig. 9. The recognition results in experiment 2. Both input images are white character with green background. Many severely distorted characters are in the input images which cause the segmented results are incorrect.
6 Conclusions In this paper, an automatic container code recognition method has been presented by using compressed sensing (CS). First, the compressed sensing approach which uses constrained L1 minimization method is described. Then, a general pattern recognition framework based on CS theory is studied. Next, the CS recognition method is applied to construct an automatic container code recognition system. Finally, the real-life images provided by trading port of Kaohsiung are used to evaluate the performance of the proposed method.
References 1. Lui, H.C., Lee, C.M., Gao, F.: Neural network application to container number recognition. In: Fourteenth Int. Conference on Computer Software and Application, pp. 190–195 (1990) 2. Lee, C.M., Wong, W.K., Fong, H.S.: Automatic character recognition for moving and stationary vehicles and containers in real-life images. In: Int. Joint Conf. on Neural Network, pp. 2824–2828 (1999) 3. Igual, I.S., Garcia, G.A., Jimenez, A.P.: Preprocessing and recognition of characters in container codes. In: The 16-th Int. Conf. on Pattern Recognition, pp. 143–146 (2002) 4. He, Z.W., Liu, J.L., Ma, H.Q., Li, P.H.: A new localization method for container autorecognition system. In: IEEE Int. Conf. on Neural Networks and Signal Processing, pp. 1170–1172 (2003) 5. He, Z.W., Liu, J.L., Ma, H.Q., Li, P.H.: A new automatic extraction method of container identity codes. IEEE Trans. on Intelligent Transportation Systems, 72–78 (2005)
Automatic Container Code Recognition Using Compressed Sensing Method
207
6. Pan, W., Wang, Y.S., Yang, H.: Robust container code recognition system. In: The 5-th World Congress on Intelligent Control and Automation, pp. 4061–4065 (2004) 7. http://www.htsol.com 8. Candes, E.J., Wakin, M.B.: An introduction to compressive sampling. IEEE Signal Processing Magazine, 21–30 (2008) 9. Romberg, J.: Imaging via compressive sampling. IEEE Signal Processing Magazine, 14– 20 (2008) 10. Wright, J., Yang, A.Y., Ganesh, A., Sastry, S.S., Ma, Y.: Robust face recognition via sparse representation. IEEE Trans. on PAMI, 210–227 (2009) 11. Gemmeke, J.F., Cranen, B.: Using sparse representations for missing data imputation in noise robust speech recognition. In: EUSIPCO 2008 (2008) 12. Parvaresh, F., Vikalo, H., Misra, S., Hassibi, B.: Recovering sparse signals using sparse measurement matrices in compressed DNA microarrays. IEEE Journal of Selected Topics in Signal Processing, 275–285 (2008) 13. Donoho, D.L.: Compressed Sensing. IEEE Trans. on Information Theory, 1289–1306 (2006) 14. Bobin, J., Starch, J.L., Ottensamer, R.: Compressed sensing in astronomy. IEEE Journal of Selected Topics in Signal Processing, 718–726 (2008) 15. Ye, J.C.: Compressed sensing shape estimation of star-shaped objects in Fourier imaging. IEEE Signal Processing Letters, 750–753 (2007) 16. Herman, M., Strohmer, T.: High-resolution radar via compressed sensing. IEEE Trans. on Signal Processing, 2275–2284 (2009) 17. Provost, J., Lesage, F.: The application of compressed sensing for photo-acoustic tomography. IEEE Trans. on Medical Imaging, 585–594 (2009)
Combining Histograms of Oriented Gradients with Global Feature for Human Detection Shih-Shinh Huang1, , Hsin-Ming Tsai1 , Pei-Yung Hsiao2 , Meng-Qui Tu2 , and Er-Liang Jian3 1
2
Dept. of Computer and Communication Engineering National Kaohsiung First University of Science and Technology Dept. of Electrical Engineering, National University of Kaohsiung 3 Chung-Shan Institute of Science and Technology
Abstract. In this work, we propose an algorithm of combining Histograms of Oriented Gradients(HOGs) with shape of head for human detection from a non-static camera. We use AdaBoost algorithm to learn local characteristics of human based on HOGs. Since local feature is easily affected by complex backgrounds and noise, the idea of this work is to incorporate the global feature for improving the detection accuracy. Here, we adopt the head contour as the global feature. The score for evaluating the existence of the head contour is through the Chamfer distance. Furthermore, the matching distributions of the head and non-head are modeled by Gaussian and Anova distributions, respectively. The combination of the human detector based on local features and head contour is achieved through the adjustment of the hyperplane of support vector machine. In the experiments, we exhibit that our proposed human detection method not only has higher detection rate but also lower false positive rate in comparison with the state-of-the-art human detector.
1
Introduction
Human detection is an extensive research in the field of computer vision and has many applications, such as surveillance, intelligent user interface, and pedestrian warning system for intelligent vehicle. The work to successfully detect pedestrian is one of the most difficult problems as it challenges the hardness on a broad range of deformable object appearances and poses, various types of human clothes, complex backgrounds, and many varied illumination conditions in the outdoor environments. 1.1
Related Work
To address the aforementioned challenges, many works have been proposed in the literature. In general, the previous studies on human or pedestrian detection can be classified into two classes, motion-based approach and appearance-based approaches.
Corresponding author.
K.-T. Lee et al. (Eds.): MMM 2011, Part II, LNCS 6524, pp. 208–218, 2011. c Springer-Verlag Berlin Heidelberg 2011
Combining HOGs with Global Feature for Human Detection
209
Fig. 1. Overview of the proposed approach for human detection
Motion-Based Approach. The motion-based approach achieves pedestrian detection by observing the motion patterns resulting from human’s movement in the video sequences. The gait of human’s walking is one of an apparent pattern for human detection and has been used in the literature [1]. They observed that the walking figures at a specific row over a period of time are like a braid and thus proposed a braid model for gait detection. For the purpose of describing human gait in more detail, a pendulum model and a twin-pendulum model are respectively proposed in [2] and [3]. Then, some image processing techniques are adopted to find the features for fitting the models. Instead of using an explicit model, low-level features resulting from the human motion are extracted and learned for pedestrian detection. Little and Boyd [4] took the optical flow of two successive images as feature points of the human’s motion and analyzed the motion periodicity to settle the existence of moving human. Recently, Viola et al. [5] computed the different motion directions between two consecutive images as pedestrian features and then the AdaBoost algorithm was applied to obtain a set of discriminative one for further detection. However, this kind of approaches generally has two limitations. Firstly, they fail to detect the pedestrians without motion. Secondly, a period of time is necessary for successful detecting the motion pattern so that they may be not suitable for real-time applications. Appearance-Based Approach. The appearance-based approach uses a set of appearance features of static human image to detect the existence of the pedestrian. Global features, such as symmetry or shape of human are widely used
210
S.-S. Huang et al.
feature for pedestrian detection in the literature. Broggi et al. [6] had applied vertical symmetry properties of human shape to calculate the position and the size of a human. Another typical solution of using global shape is based on template matching, which constructs the human templates from different viewing angles and poses. It detects the humans by comparing the extracted shape with the constructed templates. Gavrila et al. [7] and Liu et al. [8] characterized the human shapes by silhouette or edge image and then transferred them into distance transformed images. However, some aforementioned approaches which purely take the global feature as pedestrian descriptors, such as entire human shape, tend to fail in detecting partially occluded humans. Thus, some researches detect human via parts of the human body and analyze their relations to reconstruct the human shape. Mohan et al. [9], chosen an Adaptive Combination of Classifiers (ACC) to detect different body parts and integrated all part-classifiers to categorize the humans. For more examples using local features for human detection, [10] and [11] are edgelets and shapelets, respectively. The edgelet feature is a set of predefined edgelet templates, which differ from each other in number of edge, orientations, single or pair. Similar to edgelet, shapelet is a piece of shape. Histograms of Oriented Gradients (HOGs) firstly proposed by Dalal et al. [12] is a kind of local features and their success in human detection has been proven in many works [13,14]. Although many sophisticated methods are proposed to improve the representation ability of original HOGs, it suffers from the problem of false detection in complex background due to the lack of introducing the semantic features. Accordingly, we propose a framework that combines the HOGs-based approach with one global feature to alleviate this problem.
1.2
Approach Overview
The idea behind the proposed approach is to combine the method using local feature, HOGs, with global feature, head contour. The introduction of the global feature is to facilitate the human detection in a more semantic manner. Figure 1 shows the overview of our proposed approach. The proposed system mainly consists of two phases: training and detection. Given a set of database including human and non-human samples, the objective of training phase is to find out an effective human detector. The HOGs [12] is used to represent the part of human appearance. Then, a Adaboost algorithm is applied to learn a classifier for human detection. Similarly, the template matching algorithm using Chamfer distance is applied to the database for modeling the distributions, respectively, for head and non-head cases. At the detection stage, the HOGs classifier and head detection algorithm are both used to evaluate the existing confidence of human in the image. If they have non-consistent results, the matching score of head detection is used to adjust the HOGs classifier. The rest of this paper is organized as follows. In Section 2, we give a overview of how to learn the HOGs classifier for human detection. The head matching algorithm using Chamfer distance is described in Section 3. Section 4 describes
Combining HOGs with Global Feature for Human Detection
211
the way to combine the detection results from HOGs classifier and head detector. Finally, we conclude this work in Section 5 with some discussions.
2
HOGs-Based Human Detector
In the pedestrian detection literature, the histogram of oriented gradients (HOGs) [12] are proven for its effectiveness and outperform wavelet approaches. Accordingly, the HOGs is used for describing the human appearance in this work. In this section, we will give an overview of how to use the HOGs for human detection. 2.1
Histogram of Oriented Gradients
We only use the rectangular HOGs for feature description which means that the used block is formed with 4 rectangular cells. Each cell is represented by a 9-D feature vector. An example of rectangular HOGs is shown in Figure 2.
Fig. 2. The example of rectangular HOGs
For each pixel (x, y), the gradient computation is calculated by applying two discrete derivative kernels Gh = [−1, 0, 1] and Gv = [−1, 0, 1]T to respectively obtain the horizontal difference dh (x, y) and vertical difference dv (x, y). The gradient magnitude mag(x, y) and orientation θ(x, y) can be derived as follows: mag(x, y) = dh (x, y)2 + dv (x, y)2 (1) θ(x, y) = tan−1 ddhv (x,y) (x,y) Then, each cell in the feature block is represented by a orientation histogram. For saving computation, we quantize the orientation ranging from 0 to 360 degrees into 9 bins, that is, each of which corresponds to 40 degrees. The summation of mag(x, y) × wG (x, y) is accumulated in a specific bin according to θ(x, y), where wG (x, y) is the weighting mask from a Gaussian distribution. Therefore, four 9-D orientation histograms are used to represent a feature block.
212
2.2
S.-S. Huang et al.
Learning of Human Detector
In this work, the sizes of blocks used are 12, 24, 36, 48, and 60 for the detecting window 64 × 128. The aspect ratio for each block can be one of the following choices: (1 : 1), (1 : 2), and (2 : 1). However, it is time-consuming to use all blocks for pedestrian detection as the work in [12]. For efficiency consideration, Zhu et. al [14] constructed a cascaded human detector using the Adaboost algorithm proposed in [15]. Each feature for representing the block is a 36-D vector. The weak classifiers is a separating hyperplane computed by using a linear support vector machine (SVM).
3
Head Contour Detection
The human detector selected in the aforementioned section is a set of HOGs and the detection is achieved by the combination of these local features. In general, this approach has a tendency toward false detection in case of complex environment. In order to alleviate this problem, we incorporate a more semantic feature called head into the detection framework. Due to the high degree-of-freedom of human body, no global features can be always completely visible and detected in all cases. Among these global features, it is obvious that head contour is a more robust one which has the properties of low variance in appearance and high visibility from different views. Figure 3 shows some examples of head contours in different views. In some works [6], it has been applied to find the candidates of pedestrians in the image.
Fig. 3. Examples of head contour from different view angles
Obviously, the head contour is a Ω-shape and we apply the template matching algorithm using the Chamfer distance to measure the possibility of the head existence. Figure 4 (a) shows the head template Ω for matching the head contour. To facilitate the matching computation, we apply the distance transform to the head template which is defined as: DT (p) = mind(p, q) q∈Ω
(2)
Combining HOGs with Global Feature for Human Detection
213
Fig. 4. The detection of Ω-shape. (a) is the template for matching; (b) shows the detection region; (c) and (d) are the correct detection results and the two examples of mis-detection are shown in (e) and (f).
where d(p, q) is the Euclidean distance between two points p and q. For an observed image, a Sobel edge detector is firstly applied to compute the edge map. Based on the assumption that the head is always located at the top of a human, we only scan the windows inside the detection region R for head matching. The red rectangle in Figure 4 (b) is the detection regions defined in this work. The Chamfer distance used to evaluate the matching score of a window w is expressed as: 1 d(w) = DT (p) (3) N p∈e(w)
where e(w) is the edge map of the window w. In order to eliminate the regions without no edge points, we only take the windows which have edge points larger than a pre-defined threshold into account. Thus, we define the matching score for head existence as: sH = min d(w) (4) w∈R
The detection results shown in Figure 4 (c)(d) are the window with the largest score. It exhibits that the template matching can effectively detect the head in the image. However, there may be some regions have large matching score so that the head detection using Ω-shape may fail as shown in Figure 4 (e)(f). That is the reason why not directly detecting human through head matching.
4
Feature Combination
In this section, we introduce how to combine the matching score from head detector with the HOGs human classifier. 4.1
Head Distribution Modeling
However, the matching score of the head contour is just a relative measure rather than absolute one and is not suitable used for justify the existence of human. Consequently, we model the probability distributions of the matching score with
214
S.-S. Huang et al.
respect to head and non-head cases. In this work, we select 924 human and 829 and 829 non-human