THE SEMANTIC WEB CRAFTING INFRASTRUCTURE FOR AGENCY
Bo Leuf Technology Analyst, Sweden
THE SEMANTIC WEB
THE SEMANTIC WEB CRAFTING INFRASTRUCTURE FOR AGENCY
Bo Leuf Technology Analyst, Sweden
Copyright # 2006
John Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester, West Sussex PO19 8SQ, England Telephone (+44) 1243 779777
Email (for orders and customer service enquiries):
[email protected] Visit our Home Page on www.wiley.com All Rights Reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning or otherwise, except under the terms of the Copyright, Designs and Patents Act 1988 or under the terms of a licence issued by the Copyright Licensing Agency Ltd, 90 Tottenham Court Road, London W1T 4LP, UK, without the permission in writing of the Publisher. Requests to the Publisher should be addressed to the Permissions Department, John Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester, West Sussex PO19 8SQ, England, or emailed to
[email protected], or faxed to (+44) 1243 770620. This publication is designed to provide accurate and authoritative information in regard to the subject matter covered. It is sold on the understanding that the Publisher is not engaged in rendering professional services. If professional advice or other expert assistance is required, the services of a competent professional should be sought. Other Wiley Editorial Offices John Wiley & Sons Inc., 111 River Street, Hoboken, NJ 07030, USA Jossey-Bass, 989 Market Street, San Francisco, CA 94103-1741, USA Wiley-VCH Verlag GmbH, Boschstr. 12, D-69469 Weinheim, Germany John Wiley & Sons Australia Ltd, 42 McDougall Street, Milton, Queensland 4064, Australia John Wiley & Sons (Asia) Pte Ltd, 2 Clementi Loop # 02-01, Jin Xing Distripark, Singapore 129809 John Wiley & Sons Canada Ltd, 22 Worcester Road, Etobicoke, Ontario, Canada M9W 1L1 Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic books. Library of Congress Cataloging-in-Publication Data Leuf, Bo, Technology Analyst, Sweden. The Semantic Web: crafting infrastructure for agency/Bo Leuf. p. cm. Includes bibliographical references and index. ISBN 0-470-01522-5 1. Semantic Web. I. Title. TK5105.88815.L48 2005 025.04–dc22 2005024855 British Library Cataloguing in Publication Data A catalogue record for this book is available from the British Library ISBN-13 978-0-470-01522-3 (HB) ISBN-10 0-470-01522-5 (HB) Typeset in 10/12pt Times Roman by Thomson Press (India) Limited, New Delhi Printed and bound in Great Britain by Antony Rowe, Chippenham, Wilts This book is printed on acid-free paper responsibly manufactured from sustainable forestry in which at least two trees are planted for each one used for paper production.
I dedicate this book to the many visionaries, architects, and programmers, who all made the Internet an interesting place to ‘live’ and work in. And especially to Therese.
Contents Foreword Preface Part I
xiii xv
Content Concepts
1
Enhancing the Web Chapter 1 at a Glance There and Back Again Resource Identifiers Extending Web Functionality From Flat Hyperlink Model To Richer Informational Structures The Collaboration Aspect Extending the Content Model Mapping the Infosphere Well-Defined Semantic Models
3 3 4 5 11 17 22 23 26 27 30
2
Defining the Semantic Web Chapter 2 at a Glance From Model to Reality The Semantic Web Concept Representational Models The Road Map Identity Markup Relationships Reasoning Agency on the Web Semantic P2P The Visual Tour The Map The Architectural Goals The Implementation Levels
31 31 32 33 43 46 46 49 49 51 52 52 53 54 55 57
viii
Contents
3
Web Information Management Chapter 3 at a Glance The Personal View Creating and Using Content Authoring Publishing Exporting Databases Distribution Searching and Sifting the Data Semantic Web Services Security and Trust Issues XML Security Trust A Commercial or Free Web
4
Semantic Web Collaboration and Agency Chapter 4 at a Glance Back to Basics The Return of P2P WebDAV Peer Aspects Peering the Data Peering the Services Edge Computing Automation and Agency Kinds of Agents Multi-Agent Systems
Part II 5
59 59 60 63 64 65 66 66 69 71 81 83 84 91 97 97 98 99 101 103 103 105 106 108 109 111
Current Technology Overview
Languages and Protocols Chapter 5 at a Glance Markup The Bare Bones (HTML) Disentangling the Logic (XHTML and XML) Emerging Special-Purpose Markup Higher-Level Protocols A Messaging Framework (SOAP versus REST) Expressing Assertions (RDF) The Graphical Approach Exchanging Metadata (XMI) Applying Semantic Rules RDF Query Languages Multi-Agent Protocols
117 117 118 119 119 122 124 125 131 141 144 146 148 153
Contents
ix
6
Ontologies and the Semantic Web Chapter 6 at a Glance Ontology Defined Ontologies for the Web Ontology Types Building Ontologies Web Ontology Language, OWL Other Web Ontology Efforts Knowledge Representation Conceptual Graphs Promoting Topic Maps Description Logics
155 155 156 159 160 161 165 169 172 172 173 176
7
Organizations and Projects Chapter 7 at a Glance Major Players W3C Semantic Web Communities Dublin Core Metadata Initiative DARPA EU-Chartered Initiatives AIC at SRI Bioscience Communities Biopathways Consortium
177 177 178 178 182 183 186 187 190 193 193
8
Application and Tools Chapter 8 at a Glance Web Annotation The Annotea Project Evaluating Web Annotation Infrastructure Development Develop and Deploy an RDF Infrastructure Building Ontologies Information Management Haystack Digital Libraries Applying RDF Query Solutions Project Harmony DSpace Simile Web Syndication RSS and Other Content Aggregators Metadata Tools Browsing and Authoring Tools Metadata Gathering Tools
195 195 196 198 205 208 208 209 213 213 214 216 216 219 223 224 225 226 226 226
x
9
Contents
Examples of Deployed Systems Chapter 9 at a Glance Application Examples Retsina Semantic Web Calendar Agent MusicBrainz and Freedb Semantic Portals and Syndication SUMO Ontology Open Directory Project Ontolingua Industry Adoption Adobe XMP Sun Global Knowledge Engineering (GKE) Implemented Web Agents Agent Environments Intelligent Agent Platforms
Part III
229 229 230 232 233 235 242 245 246 251 252 255 256 256 258
Future Potential
10 The Next Steps Chapter 10 at a Glance What Would It Be Like? Success on the Web Medical Monitoring Smart Maintenance And So It Begins Meta-Critique Where Are We Now? Intellectual Property Issues The Road Goes Ever On Reusable Ontologies Device Independence
263 263 264 264 270 272 272 273 283 291 293 294 296
11 Extending the Concept Chapter 11 at a Glance Externalizing from Virtual to Physical The Personal Touch Pervasive Connectivity User Interaction Engineering Automation Adaptability Whither the Web? Evolving Human Knowledge Towards an Intelligent Web The Global Brain Conclusions Standards and Compliance
299 299 300 301 302 306 310 312 313 314 315 316 316
xi
Contents
The Case for the Semantic Web We can Choose the Benefits Part IV
318 320
Appendix Material
Appendix A Technical Terms and References At a Glance Glossary RDF RDF Schema Example Listing RDF How-to
325 325 325 333 333 336
Appendix B Semantic Web Resources At a Glance Further Reading Book Resources Other Publications Web Resources Miscellaneous
339 339 339 339 340 341 343
Appendix C Lists
345
Index
355
Foreword As an individual, as a technologist, as a business person, and as a civic participant, you should be concerned with how the Semantic Web is going to change the way our knowledgebased society functions. An encouraging sign is that one of the first large-scale community-based data pools, the Wikipedia, has grown to well over half a million articles. It is an ominous indicator that one of the first large-scale governmental metadata assignment projects is going on in China for the purpose restricting personal access to political information. The Semantic Web is not a single technology; rather it is a cluster of technologies, techniques, protocols, and processes. As computational power becomes more powerful and more ubiquitous, the amount of control that information technology will hold over people’s lives will become more pervasive, and the individual’s personal control ever less. At the same time, the employment of anonymous intelligent agents may buy individuals a new measure of privacy. The Semantic Web is the arena in which these struggles will be played out. The World Wide Web profoundly transformed the way people gain access to information; the Semantic Web will equally profoundly change the way machines access information. This change will transform yet again our own roles as creators, consumers and manipulators of knowledge. —Mitchel Ahren, Director of Marketing Operations, AdTools | Digital Marketing Concepts, Inc.
Preface This is a book that could fall into several different reader categories – popular, academic, technical – with perhaps delusional ambitions of being both an overview and a detailed study of emerging technology. The idea of writing The Semantic Web grew out of my previous two books, The Wiki Way (Addison-Wesley, 2001) and Peer to Peer (Addison-Wesley, 2002). It seemed a natural progression, going from open co-authoring on the Web to open peersharing and communication, and then onto the next version of the Web, involving peercollaboration between both software agents and human users. Started in 2002, this book had a longer and far more difficult development process than the previous two. Quite honestly, there were moments when I despaired of its completion and publication. The delay of publication until 2005, however, did bring some advantages, mainly in being able to incorporate much revised material that otherwise would have been waiting for a second edition. The broader ‘Semantic Web’ as a subject still remains more of a grand vision than an established reality. Technology developments in the field are both rapid and unpredictable, subject to many whims of fate and fickle budget allocations. The field is also ‘messy’ with many diverging views on what it encompasses. I often felt like the intrepid explorer of previous centuries, swinging a machete to carve a path through thick unruly undergrowth, pencil-marking a rough map of where I thought I was in relation to distant shorelines and major landmarks. My overriding concern when tackling the subject of bleeding-edge technologies can be summed up as providing answers to two fundamental reader questions: What does this mean? Why should I care? To anticipate the detailed exposition of the latter answer, my general answer is that you – we – should care, because these technologies not only can, but most assuredly will, affect us more than we can possibly imagine today.
Purpose and Target The threefold purpose of the book is rather simple, perhaps even simplistic: Introduce an arcane subject comprehensively to the uninitiated. Provide a solid treatment of the current ‘state of the art’ for the technologies involved. Outline the overall ‘vision of the future’ for the new Web.
xvi
Preface
My guiding ambition was to provide a single volume replete with historical background, state of the art, and vision. Intended to be both informative and entertaining, the approach melds practical information and hints with in-depth analysis. The mix includes conceptual overviews, philosophical reflection, and contextual material from professionals in the field – in short, all things interesting. It includes the broad strokes for introducing casual readers to ontologies and automated processing of semantics (not a unique approach, to be sure), but also covers a sampling of the real-world implementations and works-in-progress. However, the subject matter did not easily lend itself to simple outlines or linear progressions, so I fear the result may be perceived as somewhat rambling. Well, that’s part of the journey at this stage. Yet with the help of astute technical reviewers and the extended period of preparation, I was able to refine the map and sharpen the focus considerably. I could thus better triangulate the book’s position on the conceptual maps of both the experts and the interested professionals. The technologies described in this book will define the next generation Internet and Web. They may conceivably define much of your future life and lifestyle as well, just as the present day Web has become central to the daily activities of many – the author included. Therefore, it seems fitting also to contemplate the broader implications of the technologies, both for personal convenience and as instigator or mediator of social change. These technologies can affect us not only by the decisions to implement and deploy them, but sometimes even more in the event of a decision not to use them. Either way, the decision taken must be an informed one, ideally anchored in a broad public awareness and understanding of what the decision is about and with some insight into what the consequences might be. Even a formal go-ahead decision is not sufficient in itself. The end result is shaped significantly by general social acceptance and expectations, and it may even be rejected by the intended users. Some readers might find the outlined prospects more alarming than enticing – that is perhaps as it should be. As with many new technologies, the end result depends significantly on social and political decisions that for perspective require not just a clear vision but perhaps also a glimpse of some dangers lurking along the way. We can distinguish several categories of presumptive readers: The casual reader looking for an introduction and overview, who can glean enough information to set the major technologies in their proper relationships and thus catch a reflection of the vision. The ‘senior management’ types looking for buzzwords and ‘the next big thing’ explained in sufficient detail to grasp, yet not in such unrelenting technical depth as to daze. The industry professional, such as a manager or the person responsible for technology, who needs to get up to speed on what is happening in the field. Typically, the professional wants both general technology overviews and implementation guides in order to make informed decisions. The student in academic settings who studies the design and implementation of the core technologies and the related tools.
Preface
xvii
The overall style and structure of the book is held mainly at a moderate level of technical difficulty. On the other hand, core chapters are technical and detailed enough to be used as a course textbook. All techno-jargon terms are explained early on. In The Semantic Web, therefore, you are invited to a guided journey through the often arcane realms of Web technology. The narrative starts with the big picture, a landscape as if seen from a soaring plane. We then circle areas of specialist study, thermal-generating ‘hotspots’, subjects that until recently were known mainly from articles in technical journals with limited coverage of a much broader field, or from the Web sites of the institutions involved in the research and development. Only in the past few years has the subject received wider public notice with the publication of several overview books, as noted in Appendix B.
Book Structure The book is organized into three fairly independent parts, each approaching the subject from a different direction. There is some overlap, but you should find that each part is complementary. Therefore, linear cover-to-cover reading might not be the optimal approach. For some readers, the visions and critiques in Part III might be a better starting point than the abstract issues of Part I, or the technicalities in Part II. Part I sets the conceptual foundations for the later discussions. These first four chapters present mostly high-level overviews intended to be appropriate for almost anyone. The first chapter starts with the basics and defines what Web technology is all about. It also introduces the issues that led to the formulation of the Semantic Web initiative. Chapter 2 introduces the architectural models relevant to a discussion of Web technologies and defines important terminology. Chapter 3 discusses general issues around creating and managing the content and metadata structures that form the underpinnings of the Semantic Web. Finally, Chapter 4 looks at online collaboration processes, which constitute an important motivating application area for Semantic Web activities. Part II focuses on the technologies behind the Semantic Web initiative. These core chapters also explore representative implementations for chosen implementation areas, providing an in-depth mix of both well-known and lesser-known solutions that illustrate different ways of achieving Semantic Web functionality. The material is detailed enough for Computer Studies courses and as a guide for more technical users actually wanting to implement and augment parts of the Semantic Web. Chapter 5 provides layered analysis of the core protocol technologies that define Web functionality. The main focus is on the structures and metadata assertions used to describe and manage published data. Chapter 6 is an in-depth study of ontologies, the special structures used to represent term definitions and meaningful relationships. Chapter 7 introduces the main organizations active in defining specifications and protocols that are central to the Semantic Web. Chapter 8 examines application areas of the Semantic Web where prototype tools are already implemented and available.
xviii
Preface
Chapter 9 expands on the previous chapters by examining application areas where some aspect of the technology is deployed and usable today. Part III elevates the discussion into the misty realms of analysis and speculation. Chapter 10 provides an ‘insights’ section that considers the immediate future potential for Semantic Web solutions, and the implications for users. Chapter 11 explores some directions in which future Web functionality might develop in the longer term, such as ubiquitous connectivity and the grander theme of managing human knowledge management. Finally, the appendices supplement the main body of the book with a terminological glossary, references, and resources – providing additional detail that while valuable did not easily fit into the flow of the main text.
Navigation This book is undeniably filled with a plethora of facts and explanations, and it is written more in the style of a narrative rather than of reference-volume itemization. Despite the narrative ambitions, texts like this require multiple entry points and quick ways to locate specific details. As a complement to the detailed table of contents and index, each chapter’s ‘at a glance’ page provides a quick overview of the main topics covered in that chapter. Technical terms in bold are often provided with short explanations in the Appendix A glossary. Scattered throughout the text you will find the occasional numbered ‘Bit’ where some special insight or factoid is singled out and highlighted. Calling the element a ‘Bit’ seemed to convey about the right level of unpretentious emphasis – they are often just my two-bits worth of comment. Bits serve the additional purpose of providing visual content cues for the reader and are therefore given their own List of Bits in Appendix C. When referencing Web resources, I use the convention of omitting the ‘http://’ prefix because modern Web browsers accept addresses typed in without it. Although almost ubiquitous, the ‘www.’ prefix is not always required, and in cases where a cited Web address properly lacks it, I tried to catch instances where it might have been added incorrectly in the copyedit process.
The Author As an independent consultant in Sweden for 30 years, I have been responsible for software development, localization projects, and design-team training. A special interest was adapting an immersive teaching methodology developed for human languages to technical-training fields. Extensive experience in technical communication and teaching is coupled with a deep understanding of cross-platform software product design, user interfaces, and usability analysis. All provide a unique perspective to writing technical books and articles.
Preface
xix
Monthly featured contributions to a major Swedish computer magazine make up the bulk of my technical analyst writing at present. Coverage of, and occasional speaking engagements at, select technology conferences have allowed me to meet important developers. I also maintain several professional and recreational Internet Web sites, providing commercial Web hosting and Wiki services for others.
Collaborative Efforts A great many people helped make this book possible by contributing their enthusiasm, time, and effort – all in the spirit of the collaborative peer community that both the Web and book authoring encourage. Knowledgeable professionals and colleagues offered valuable time in several rounds of technical review to help make this book a better one and I express my profound gratitude for their efforts. I hope they enjoy the published version. My special thanks go to the many reviewers who participated in the development work. The Web’s own creator, and now director of the World Wide Web Consortium, Sir Tim Berners-Lee, also honoured me with personal feedback. Thanks are also due to the editors and production staff at John Wiley & Sons, Ltd. Personal thanks go to supportive family members for enduring long months of seemingly endless research and typing, reading, and editing – and for suffering the general mental absentness of the author grappling with obscure issues.
Errata and Omissions Any published book is neither ‘finished’ nor perfect, just hopefully the best that could be done within the constraints at hand. The hardest mistakes to catch are the things we think we know. Some unquestioned truths can simply be wrong, can have changed since we learned them, or may have more complex answers than we at first realized. Swedish has the perceptive word hemmablind, literally blind-at-home, which means that we tend not to see the creeping state of disarray in our immediate surroundings – think of how unnoticed dust ‘bunnies’ can collect in corners and how papers can stack up on all horizontal surfaces. The concept is equally applicable to not always noticing changes to our particular fields of knowledge until someone points them out. Omissions are generally due to the fact that an author must draw the line somewhere in terms of scope and detail. This problem gets worse in ambitious works such as this one that attempt to cover a large topic. I have tried in the text to indicate where this line is drawn and why. Alternatively, I might sometimes make overly simplified statements that someone, somewhere, will be able to point to and say ‘Not so!’. My excuse is that not everything can be fully verified, and sometimes the simple answer is good enough for the focus at hand. A book is also a snapshot. During the course of writing, things changed! Constantly! Rapidly! In the interval between final submission and the printed book, not to mention by the time you read this, they have likely changed even more. Not only does the existing software continue to evolve, or sometimes disappear altogether, but new implementations can suddenly appear from nowhere and change the entire landscape overnight.
xx
Preface
Throughout the development process, therefore, book material was under constant update and revision. A big headache involved online resource links; ‘link-rot’ is deplorable but inevitable. Web sites change, move, or disappear. Some resources mentioned in the text might therefore not be found and others not mentioned might be perceived as better. The bottom line in any computer-related field is that any attempt to make a definitive statement about such a rapidly moving target is doomed to failure. But we have to try.
Book Support and Contacting the Author The Internet has made up-to-date reader support a far easier task than it used to be, and the possibilities continue to amaze and stimulate me. Reader feedback is always appreciated. Your comments and factual corrections will be used to improve future editions of the book, and to update the support Web site. You may e-mail me at
[email protected], but to get past the junk filters, please use a meaningful subject line and clearly reference the book. You may also write to me c/o the publisher. Authors tend to get a lot of correspondence in connection with a published book. Please be patient if you write and do not get an immediate response – it might not be possible. I do try to at least acknowledge received reader mail within a reasonable time. However, I suggest first visiting the collaborative wiki farm (follow links from www.leuf.com/ TheSemanticWeb), where you can meet an entire community of readers, find updates and errata, and participate in discussions about the book. The main attraction of book-related Web resources is the contacts you can form with other readers. Collectively, the readers of such a site always have more answers and wisdom than any number of individual authors. Thank you for joining me in this journey. Bo Leuf Technology Analyst, Sweden (Gothenburg, Sweden, 2003–2005)
Part I Content Concepts
1 Enhancing the Web Although most of this book can be seen as an attempt to navigate through a landscape of potential and opportunity for a future World Wide Web, it is prudent, as in any navigational exercise, to start by determining one’s present location. To this end, the first chapter is a descriptive walkabout in the current technology of the Web – its concepts and protocols. It sets out first principles relevant to the following exploration, and it explains the terms encountered. In addition, a brief Web history is provided, embedded in the technical descriptions. Much more than we think, current and future technology is designed and implemented in ways that critically depend on the technology that came before. A successor technology is usually a reaction, a complement, or an extension to previous technology – rarely a simple plug-in replacement out of the blue. New technologies invariably carry a legacy, sometimes inheriting features and conceptual aspects that are less appropriate in the new setting. Technically savvy readers may recognize much material in this chapter, but I suspect many will still learn some surprising things about how the Web works. It is a measure of the success of Web technology that the average user does not need to know much of anything technical to surf the Web. Most of the technical detail is well-hidden behind the graphical user interfaces – it is essentially click-and-go. It is also a measure of success that fundamental enhancements (that is, to the basic Web protocol, not features that rely on proprietary plug-in components) have already been widely deployed in ways that are essentially transparent to the user, at least if the client software is regularly updated.
Chapter 1 at a Glance Chapter 1 is an overview chapter designed to give a background in broad strokes on Web technology in general, and on the main issues that lead to the formulation of the Semantic Web. A clear explanation of relevant terms and concepts prepare the reader for the more technical material in the rest of the book. There and Back Again sets the theme by suggesting that the chapter is a walkabout in the technology fields relevant to the later discussions that chapter by chapter revisit the main concepts, but in far greater detail.
The Semantic Web: Crafting Infrastructure for Agency Bo Leuf # 2006 John Wiley & Sons, Ltd
The Semantic Web
4
Resource Identifiers defines fundamental identity concepts, protocol basics, and how content can at all be located in the current Web by the user’s client software. Extending Web Functionality examines proposed ways to enhance the basic Web transport protocol, as well as protocol-independent methods. From Flat Hyperlink Model describes the current navigational model of the Web, especially the hyperlink, and highlights the areas where it is lacking. After a wishlist of Web functionality, To Richer Informational Structures explores strategies for extending the hyperlink model with background information about the content. The Collaboration Aspect explores one of the driving forces for a new Web, after which Extending the Content Model shows why a unified way to handle content is important in any extension. Mapping the Infosphere discusses ways that have been tried to map what is on the Web so that users can find what they are looking for. Well-Defined Semantic Models introduces why current lexical mappings are insufficient for the task, especially if searching and processing is to be automated.
There and Back Again The World Wide Web was conceived and designed as an open information space defined by the hyperlink mechanism that linked documents together. The technology enabled anyone to link to any other document from hyperlinks on a published Web page – a page anyone could see, and link to in turn. The whole system could self-grow and self-organize. No permissions were required to set up such links; people just linked to whatever other published resources they found interesting and useful. The only requirements to participate were a simple Internet connection and a place to put a Web page. This open nature is fundamental to many of the early design decisions and protocol implementations, sometimes in ways that were later obscured or marginalized. Bit 1.1
The Web is an open universe of network-accessible information
This definition of the Web, formulated by Tim Berners-Lee, provides in all its simplicity the most fundamental description of the Web’s potential. Open network access enables a potentially infinite resource, for people both to contribute to and use. The explosive growth of the Web and the content it mediates is in many ways a direct consequence of this design. It has given rise to a remarkable plethora of both content and functionality, sometimes unexpected. I am very happy at the incredible richness of material on the Web, and in the diversity of ways in which it is being used. There are many parts of the original dream which are not yet implemented. For example, very few people have an easy, intuitive tool for putting their thoughts into hypertext. And many of the reasons for, and meaning of, links on the web is lost. But these can and I think will change. Tim Berners-Lee (www.w3.org/People/Berners-Lee/FAQ.html), ‘inventor’ of the Web and director of the W3C.
Enhancing the Web
5
In addition, the original design had the goal that it should not only be useful for humanto-human communication but also support rich human–machine and machine–machine interactions. In other words, the intent was that machines would be able to participate fully and help in the access and manipulation of this information space – as automated agents, for example. Bit 1.2 The Web had the twin goals of interactive interoperability and creating an evolvable technology The core values in Web design are expressed in the principle of universality of access – irrespective of hardware or software platform, network infrastructure, language, culture, geographical location, or physical or mental impairment. Before going further into the nature of such interactions and the infrastructure that is to support them, we need to explore the fundamental issues of resource identity, and how naming schemes relate to the protocols used to access the resources.
Resource Identifiers The full interoperability and open-ended nature of the Web was intended to be independent of language, as evident in the way the design specified the universality of referencing resources by identity through the Universal Resource Identifier (or URI). The principle that absolutely anything ‘on the Web’ can be identified distinctly and uniquely by abstract pointers is central to the intended universality. It allows things written in one language to refer to things defined in another language. Properties of naming and addressing schemes are thus defined separately, associated through the dereferencing protocol, allowing many forms of identity, persistence and equivalence to refer to well-defined resources on the Web. When the URI architecture is defined and at least one dereferencing protocol implemented, the minimum requirement for an interoperable global hypertext system is just a common format for the content of a resource (or Web object). Anyone can create a URI to designate a particular Web resource (or anything, actually). This flexibility is at the same time both the system’s greatest strength and a potential problem. Any URI is just an identifier (a ‘name’ often opaque to humans), so simple inspection of it in isolation does not allow one to determine with certainty exactly what it means. In fact, two different URIs might refer to the same resource – something we often also run across in naming schemes in the ‘everyday’ world. The concept of unique identifiers finds expression in many fields, and is crucial to ‘finding’ and handling things in a useful way. Any identifier scheme assuredly defines a useful namespace, but not all schemes provide any useful dereferencing protocol. Some examples from the latter category are the MIME content identifier (cid ) or message identifier (mid ) spaces, the MD5 hash code with verifiable pure identity (often used as secure verification of file identity), and the pseudo-random Universally Unique Identifier (uuid ). They all identify but cannot locate.
6
The Semantic Web
The ability to utilize such namespace schemes in the URI context provides valuable functionality—as is partially illustrated by some peer-to-peer (p2p) networks, such as Freenet (described in the previous book, Peer to Peer). Even a simple persistent identity concept for connection-oriented technologies, for which no other addressable content exists, can prove more useful than might be expected. Witness the ubiquitous mailbox namespace defined by the ‘mailto:’ protocol – unfortunately named, however, since URIs are functionally nouns not verbs. The resulting URIs define connection endpoints in what is arguably the most well-known public URI space, persistent virtual locations for stores of e-mail messages.
Understanding HTTP Space The other most well-known public URI space is the HTTP namespace – commonly called ‘the Web’. It is characterized by a flexible notion of identity and supports a richness of information about resources and relating resources. HTTP was originally designed as a protocol for remote operations on objects, while making the exact physical location of these objects transparent. It has a dereferencing algorithm, currently defined by HTTP 1.1, but augmented by caching, proxying and mirroring schemes. Dereferencing may therefore in practice take place even without HTTP being invoked directly. The HTTP space consists of two parts: Domain Name, a hierarchically delegated component, for which the Domain Name System (DNS) is used. This component is a centrally registered top level domain (TLD): generic (gTLD, such as example.com) or national (ccTLD, example.se). Relative Locator, an opaque string whose significance is locally defined by the authority owning the domain name. This is often but need not (indeed should rarely) be a representation of a local directory tree path (relative to some arbitrary ‘root’ directory) and a file name (example: /some/location/resource). A given HTTP URI (or resource object identity) is commonly written as a URL, a single string representing both identity and a Web location. URL notation concatenates the parts and prefixes the protocol (as http:// ). As a rule, client software transparently maps any ‘illegal’ characters in the URL into protocol-acceptable representations, and may make reasonable assumptions to complete the abbreviated URL entry. In practice, the domain component is augmented by other locally defined (and optional) prefix components. Although still formally DNS arbitrated, the prefix is determined by the local authority. It is resolved by the DNS and Web (or other service) servers in concert. Most common is ‘www.’ but it may also be a server name, a so-called vanity domain, or any other local extension to the domain addressing scheme. A related important feature is to dereference the physically assigned IP number bound to a particular machine. Domain names therefore improve URI persistence, for example when resources might move to other machines, or access providers reallocate. However, persistence of locators in HTTP space is in practice not realized fully on the Web.
Enhancing the Web
Bit 1.3
7
URLs are less persistent overall than might reasonably be expected
Improving persistence involves issues of tool maturity, user education, and maturity of the Web community. At a minimum, administrators must be discouraged from restructuring (or ‘moving’) resources in ways that needlessly change Web addresses. The design of HTTP and the DNS makes addressing more human-readable and enables almost complete decentralization of resources. Governance is freely delegated to local authorities, or to endpoint server machines. Implementing a hierarchical rather than flat namespace for hosts thus minimizes the cost of name allocation and management. Only the domain-name component requires any form of formal centralization and hierarchical structure – or rather, only as currently implemented does it require centralized registrars and domain-name databases for each TLD. The domain name is, strictly speaking, optional in HTTP. It is possible, if not always convenient (especially with the trend to share IP in virtual hosting), to specify HTTP resource addresses using the physical IP number locally assigned by an access provider. Other protocol spaces, such as Freenet, in fact dispense with domains altogether and rely instead on unique key identifiers and node searches. As a feature common and mandatory to the entire HTTP Web, as it is currently used, the DNS root is a critical resource whose impartial and fair administration is essential for the world as a whole. Ownership and governance of the root of the DNS tree and gTLD subtree databases has in recent years been subject to considerable debate. The situation is only partially and nominally resolved under international arbitration by ICANN. The Semantics of Domain Names Another issue concerning gTLD allocation, with relevance to the concept of ‘meaning’ (or semantics) of the URI, is the design intent that the different domain categories say something about the owners. The original international gTLDs and their intended application were clear, for example: ‘.com’ for commercial organizations with a true international presence (those with presence just in a single national region were recommended to use instead countrycode domains). ‘.org’ for non-profit organizations with an international presence. ‘.net’ for international providers of network services, for example Web hosts or service providers. Such a division provides basic meta-information embedded in the URL in a way that is easy to see. However, so many U.S. companies registered under .com that the common perception became that .com designated a U.S. business. This skew was due mainly to the original strict rules concerning the allocation of .us domains by state, rather than allowing a company with a multi-state presence to use a single national domain. The .com domains became in fact the most popular on the Web, a status symbol no matter what the purpose. It also gave rise to the pervasive ‘dotcom’ moniker for any Internet-related
The Semantic Web
8
business venture. In a similar vein, though less so, the popular but mistaken association arose that .org and .net were U.S. domains as well. Administration by a U.S. central registrar only strengthened this false association. These three gTLDs are considered open domains, because there are no formal restrictions on who may register names within them. Anyone, anywhere, can therefore have a dotcom, dotorg, or dotnet domain – business, individual, whatever. Further confusion in the role of these gTLDs arose when domain names became traded commodities. Public perception tends to equate brand name with domain name, regardless of the TLD. Attractive names thus became a limited commodity and the effective namespace smaller. Most brand owners are likely to register all three at once. Bit 1.4
The TLD level of domain names currently lacks consistent application
The TLD system defines the root of the URLs that, as subsets of URIs, are supposed to provide an unambiguous and stable basis for resource mapping by resource owners. Reality is a bit more complex, also due to hidden agendas by the involved registrars. Therefore, over time, the original semantic division was substantially diluted, and essentially it is today lost from public awareness. However, the intent of at least one of the original seven gTLD was preserved: ‘.int’ is used only for registering organizations established by international treaties between governments, or for Internet infrastructure databases. Since the early Internet was mostly implemented in the U.S. and the administration of the gTLD names was then under U.S. governance, the other three original categories were quickly reserved for U.S. bodies: ‘.edu’ for universities and corresponding institutions of higher education that qualified, became in practice the domain for U.S.-based institutions. ‘.gov’ became reserved exclusively for the United States Government and its federal institutions. ‘.mil’ became reserved exclusively for the United States Military. Other countries must use qualified ccTLDs for the same purpose, such as .gov.uk. A further special TLD, ‘.arpa’ is provided for technical infrastructure purposes. In an apparent attempt to reclaim some inherent TLD meaning, new usage-restricted gTLDs were proposed in 2000. First implemented were .biz, .info, .pro, and .name, while .coop, .aero, and .museum are still pending (see FAQ at www.internic.net/faqs/ new-tlds.html ). Since 2004, ICANN has added .asia, .cat, jobs, .mail, .mobi, .post, .tel, .travel, and .xxx to the proposal list. Unfortunately, we already see semantic erosion in some of the newly implemented TLDs. Also, the addition of more TLD namespaces not only risks further confusion in public
Enhancing the Web
9
perception (.biz or .com?), but can be detrimental to the original concept of reducing the cost of namespace management. Brand owners may feel compelled redundantly to add further TLDs to protect against perceived misappropriation. Bit 1.5
Conflicting interests are fragmenting the Web at the TLD-level
Issues of ownership and intent involve complex and changing policy and politics, not easily pinned down in any lasting way, and often at odds with the Web’s underlying concept of universality. The resulting confusion, market pressures, and market conflict subtly distort the way we can usefully map the Web. In 2004, the W3C was, in fact, moved to state that the influx of new TLD subtrees was harmful to the Web infrastructure, or at the very least incurred considerable cost for little benefit. Detailed critique is given for some of these ‘special-interest’ proposals (see reasoning at www.w3.org/DesignIssues/TLD and www.w3.org/2004/03/28-tld ), for example: Implementing .mobi would seem to partition the HTTP information space into parts designed for access from mobile devices and parts not so designed. Such a scheme destroys the essential Web property of device independence. The domain name is perhaps the worst possible way of communicating information about the device. A reasonable requirement for adaptive handling of mobile devices is that it be transparent, by way of stylesheets and content negotiation. Content-filtering through TLD (as in .xxx) is unlikely to be effective, even assuming bestcase consensus, applicability, and binding international agreements on appropriate domain owners and suitable content in the respective TLD. We may also assume that many companies would merely redirect new special-interest domains (such as .travel ) to existing ones (an established .com, for instance). As it happens, even the superficially clear geographic relationship of the ccTLDs to the respective countries has been considerably diluted in recent years. The increased popularity of arbitrary businesses, organizations, or individuals registering attractive small-country domains as alternatives to the traditional dotcom ones causes more TLD confusion. In part, this practice reflects a preference for country codes that impart some ‘useful’ association, which can become popular in the intended contexts – for example, Tongan ‘.to’ as in ‘http://come.to’, or Samoan ‘.ws’ recast as meaning ‘website’ or ‘worldsite’. In part, it is a result of the increasing scarcity of desired or relevant dotcom names. This ‘outsourced’ registration, usually U.S.-based through some licensing agreement, generates significant foreign income (in U.S. dollars) for many small Pacific island nations, but assuredly it confuses the previously clear knowledge of ccTLD ownership. Pervasive Centralization Despite the decentralized properties inherent to the HTTP space design, the subsequent evolution of the Web went for a time in a different direction.
10
The Semantic Web
A needless fragmentation situation arose, with different protocols used to transfer essentially the same kind of text (or message) objects in different contexts – for example, e-mail, Web page content, newsgroup postings, chat and instant messaging. Such fragmentation, which leads to multiple client implementations, is confusing to the user. The confusion is only recently in part resolved by advanced multiprotocol clients. Web implementations also ignored to a great extent the interactive and interoperative design goals – much to the disappointment of the early visionaries who had hoped for something more open and flexible. This deficiency in the Web is part of what prompted the development of other clients in other protocols, to implement functionality that might otherwise have been a natural part of the HTTP Web. The pervasive paradigm of the Web instead became one of centralized content-providing sites designed to serve unilaterally a mass of content-consuming clients. Such sites constrain user interaction to just following the provided navigational links. Web pages thus became increasingly ‘designer’ imprinted, stylistic exercises ‘enhanced’ with attention-grabbing devices. Technology advances unfortunately tended to focus on this eyeball functionality alone. In fact, the main visitor metric, which tellingly was used to motivate Web advertising, became ‘page hits’ or ‘eyeball click-through counts’ rather than any meaningful interaction. Most Web-browser ‘improvements’ since the original Mosaic client (on which MS Internet Explorer, the long dominant browser, is based) have thus dealt more with presentational features than any real navigational or user-interaction aspects. Worse, much of this development tended towards proprietary rather than open standards, needlessly limiting the reach of information formatted using these features.
Revival of Core Concepts Despite the lackluster Web client implementations with respect to user functionality, the potential for interactive management of information on the Web remained an open possibility – and a realm in recent years extended by alternative peer architectures. The way the Internet as a whole functions means that nothing stops anyone from deploying arbitrary new functionality on top of the existing infrastructure. It happens all the time. In fact, Internet evolution is usually a matter of some new open technology being independently adopted by such a broad user base that it becomes a de facto new standard – it becomes ever more widely supported, attracting even more users. Bit 1.6 Internet design philosophy is founded in consensus and independent efforts contributing to a cohesive whole Principles such as simplicity and modularity, basic to good software engineering, are paralleled by decentralization and tolerance – the life and breath of Internet. Several open technologies have in recent years provided a form of revival in the field, with a focus on content collaboration. One such technology, explored in an earlier book, The Wiki Way, is a simple technology that has in only a few years transformed large parts of the visible Web and redefined user interaction with Web-published content.
Enhancing the Web
11
Wiki technology relies on the stock Web browser and server combination to make collaborative co-authoring a largely transparent process. It leverages the existing clientserver architecture into a more peer-based collaboration between users. Prominent examples of large-scale deployment are Wikipedia (www.wikipedia.com) and related Wikimedia projects, and many open-source support sites (sourceforge.net). Other extending solutions tend to be more complex, or be dependent on special extensions to the basic Web protocol and client-server software (such as plug-in components, or special clients). Yet they too have their validity in particular contexts. This tension between peer and hierarchical models was further explored and analyzed in considerable detail in the book Peer to Peer, along with the concept of agency. Although these peer technologies lie outside the immediate scope of the current book, some aspects drawn from these discussions are taken up in relevant contexts in later chapters.
Extending Web Functionality With the enormous growth of the Web and its content since its inception, it is clear that new implementations must build on existing content to gain widespread usage, or at least allow a relatively simple and transparent retrofit. This development can be more or less easy, depending on the intentions and at what level the change comes. So far, we have seen such changes mainly at the application level. Long sought is a change at a more profound level, such as a major extension to the underlying Web protocol, HTTP. Such a change could harmonize many functionality extensions back into a common and uniform framework on which future applications can build. Current HTTP does in fact combine the basic transport protocol with formats for limited varieties of metadata – information about the payload of information. However, because it is descended from the world of e-mail transport (and an old protocol), HTTP metadata support as currently implemented remains a crude architectural feature that should be replaced with something better. Bit 1.7 The Web needs a clearer distinction between basic HTTP functionality and the richer world of metadata functionality A more formalized extension of HTTP, more rigorous in its definitions, can provide such a needed distinction and thus bring the Semantic Web closer to reality. Extending HTTP HTTP was designed as part of a larger effort by a relatively small group of people within the IETF HTTP Working Group, but Henrik Frystyk Nielsen (the specification author) claims that this group did not actually control the protocol. HTTP was instead considered a ‘common good’ technology, openly available to all. In this vein of freedom, HTTP was extended locally as well as globally in ways that few could predict. Current extension efforts span an enormous range of applications, including distributed authoring, collaboration, printing, and remote procedure call mechanisms.
12
The Semantic Web
However, the lack of a standard framework for defining extensions and separating concerns means that protocol extensions have not been coordinated. Extensions are often applied in an ad hoc manner which promotes neither reusability nor interoperability. For example, in the variant HTTPS space, a protocol distinction is made needlessly visible in the URI. Although HTTPS merely implies the use of HTTP through an encrypted Secure Socket Layer (SSL) tunnel, users are forced to treat secure and insecure forms of the same document as completely separate Web objects. These instances should properly be seen as transparent negotiation cases in the process of dereferencing a single URI Therefore, the HTTP Extension Framework was devised as a simple yet powerful mechanism for extending HTTP. It describes which extensions are introduced, information about who the recipient is, and how the recipient should deal with them. The framework allows parameters to be added to method headers in a way that makes them visible to the HTTP protocol handler (unlike CGI parameters, for example, that must remain opaque until dealt with by the handler script on the target server). A specification of the HTTP Extension Framework is found in RFC 2774. (Among other sources, see userfriendly Web site www.freesoft.org/CIE/RFC/r to search and read RFC documents.) Otherwise, the most ubiquitous and transparent functionality change in the Web in recent years was an incremental step in the basic Web protocol from HTTP 1.0 to HTTP 1.1. Few users noticed this upgrade directly, but a number of added benefits quickly became mainstream as new versions of server and client software adapted. Examples of new features include virtual hosting (to conserve the IP-number space), form-based upload (for browser management of Web sites), and MIME extensions (for better multimedia support). Many would instead like to see something more like a step to ‘HTTP-NG’ (Next Generation) that could implement a whole new range of interoperable and interactive features within the base protocol. WebDAV is one such initiative, styled as completing the original intent of the Web as an open collaborative environment, and it is discussed in Chapter 4. Consider, after all, how little HTTP 1.x actually gives in the form of ‘exposed interfaces’, otherwise often termed ‘methods’ and known by their simple names. These methods are in effect the ‘action verbs’ (one of many possible such sets) applied to the ‘identifier nouns’ (URI) of the protocol. The main method is to access resources: GET is the core request for information in the Web. It takes the form of HTTP header specifying the desired information object as either the unique location (URL) or the process (CGI) to produce it. (A variant, HEAD, might support the return of header information but without data body.) GET is actually the only HTTP method that is required always to be supported. It has a special status in HTTP, in that it implements the necessary dereferencing operation on identifiers – it defines the namespace. As such, GET must never be used in contexts that have
Enhancing the Web
13
side-effects. Conversely, no other method should be used to perform only URI dereferencing, which would violate universality by defining a new namespace. Many ad hoc extensions and p2p-application protocols are based solely on GET. The GET-queried resource is expected to return the requested information, a redirection URI, or possibly a status or error message. It should never change state. In the request response, an appropriate representation of the URI-specified object is transferred to the client – not, as is commonly assumed, the stored literal data. Representations, encoding, and languages acceptable may be specified in the GET-header request fields, along with any specific client-side information. These and other factors affect both what is returned and the chosen format or encoding. Client-side input may be handled by two other methods: PUT is the method to store information at a particular Web location specified by a valid URL. It is structurally similar to GET, except that the header is also associated with a body containing the data to be stored. Data in the body comprise opaque bitstreams to HTTP agents. POST is in effect an indirect method to pass an information object to a Web server. It is entirely up to the server to determine both when and how to deal with it (process, store, defer, ignore, or whatever). The header URI specifies the receiving server agent process (such as a named script), possibly including a suggested URI for the object. Other methods sometimes seen are essentially only extensions of these basic three. It is optional for the responding HTTP agent (the Web server) to implement an appropriate process for any of these input methods. The following are some formally registered examples: CHECKOUT and CHECKIN are version-control variants corresponding to GET and PUT, but with the added functionality of locking and unlocking, respectively, the data object for access/change by other users. TEXTSEARCH and SPACEJUMP are extended forms of GET applied to search and map coordinate positioning. LINK and UNLINK are variants of POST to add and remove meta information (object header information) to an object, without touching the object’s content. In fact, it is relatively rare to see PUT used these days. Most servers are configured to deny such direct, external publishing requests, except perhaps in specific, well-authenticated contexts. Instead, POST is used to pass data from client to server in a more open-ended way, by requesting that a server-defined link be created to the passed object. Although, traditionally, POST is used to create, annotate, and extend server-stored information, it was successfully MTME-extended in v1.1 to serve as a generic data upload and download mechanism for modern browsers. The use of POST allows tighter server control of any received information. Perhaps more relevant to security, it gives server control over processing, storage location, and subsequent access. The client may suggest a storage URI, but the server is never obliged to use it. Even if a POST is accepted by the server, the intended effect may be delayed or overruled by subsequent processing, human moderation, or batch processing. In fact,
The Semantic Web
14
the creation of a valid link may not only be significantly delayed, it may never occur at all. Overall, the guiding design thought is that HTTP requests may be cached. From the perspective of the client/user, simple requests must be stateless, repeatable, and free of side-effects. Bit 1.8
HTTP is, like most basic Internet protocols, stateless
State tracking is, however, desired in many situations, but must then be implemented using message-based mechanisms external to the base protocol. Information stored clientside in Web-browser ‘cookies’ are but one example of such workaround measures. It must be noted that POST is strictly speaking neither repeatable nor free from sideeffect, and thus not a proper replacement for PUT (which is both). In fact, POST requests are by definition declared noncachable, even though it might prove possible in some situations to cache them with no side-effects. Modern browsers correctly warn the user if a POST request is about to be reissued. The reason is of course that a POST often alters information state at the server, for example to generate content change, or to commit a unique credit-card transaction. State considerations place serious constraints on possible functionality in the context of the current HTTP Web.
Extending in Other Layers Extending the transport protocol (HTTP) is by no means the only proposed solution to extending Web functionality. The growing adoption of XML, not just in preference to HTML as markup but crucially as a real language for defining other languages, provides an alternative extension framework. XML is more broadly application-managed than HTML, and its functionality definitions are independent of whether the underlying HTTP is extended or not. Implementations include message-based protocols to extend Web functionality by embedding the extension methods inside the message content passed by the base protocol. The use of XML for inter-company remote operations became prevalent in 2001, mainly because of its ability to encapsulate custom functionality and convey specialized content meaning. This kind of functionality extension is what led to the development of Web Services, a term implying standard functionality distributed across the Web. Even the markup change alone has direct potential benefits on existing Web functionality – for example, search. Most search engines read the format languages, usually HTML tags that may often be applied inappropriately from the point of view of logical structure. Consequently, the search results tend to reflect the formatting tags rather than actual page content as expressed in natural language. XML markup can express semantic meaning and thus greatly improve search-result relevancy. There are both pros and cons in the XML approach to extending the Web framework. Some of both aspects hinge on the fact that the implemented message passing often relies on
Enhancing the Web
15
the HTTP POST method. Embedding messages in GET headers imposes unacceptable message constraints. Extended headers, for example, might be passed as POST content and thus be opaque to the HTTP agent. When referring to a particular resource addresses, this embedded reference cannot be cached or stored in the way a GET-request URI can. A typical workaround is a syntax extraction procedure to let the agent recover a workable URI from the POST header response to a GET request. The solution allows HTTP agents not to consider content-embedded extensions when responding to arbitrary requests. The XML extension approach shares header opaqueness, seen from the point of view of HTTP agents (such as servers, proxies, and clients), with more proprietary extensions anchored in specific applications. Nonetheless, XML extension at least has the virtue of being an open standard that can be adopted by any implementation and be freely adapted according to context. It is ongoing work. Functionality extension is not simply about extending the protocol; other issues must also be considered. Content Considerations A major obstacle facing anyone who wishes to extend the usefulness of the Web information space to automated agents is that most information on the Web is designed explicitly for human consumption. Some people question what they see as a ‘geek’ focus on extending and automating the Web with new gee-wizardry. Instead, they suggest that most people seem happy with the Web as designed for human browsing, and do not necessarily have a problem with that. Most people, they say, are finding more or less what they want, and are finding other things of interest as they browse around looking for what they want. Just because current Web content is designed supposedly for humans does not mean that it is a ‘successful’ design, however. For example, published Web content commonly relies on any number of implied visual conventions and assumptions to convey intended structure and meaning. Many of these often complex stylistic elements are neither easily nor unambiguously interpreted by ‘most people’ even with explicit instruction, and much less is it possible to decode them by machines. Furthermore, such style conventions can vary significantly between different authors, sites, and contexts. Embedded metadata describing the intent behind arbitrary visual convention would not just aid machines, but could provide unambiguous cues to the human reader – perhaps as consistent client-side styling according to reader preference, or as cue pop-up. The advantage of such explicit guidance is most obvious in the case of browsers for the visually impaired, a group of human readers who often have significant problems dealing with existing content supposedly designed for them. Even in cases where the information is derived from a database, with reasonably well-defined meanings for at least some terms in its tabled layout, the structure of the presented data is not necessarily evident to a robot browsing the site. Almost any such
16
The Semantic Web
presentation has implicit assumptions about how to interpret and connect the data. Humans readers familiar with the material can deal with such assumptions, but unaided machines usually cannot. For that matter, the material might not even be possible to browse for a machine following embedded hyperlinks. Other, visual navigational conventions might have been implemented (such as image maps, form buttons, downloaded ActiveX controls, scripted mouse-state actions, flash-animated widgets, and so on). Bit 1.9 Machine complexities that depend on human interpretation can signal that the problem statement is incorrectly formulated A fundamental insight for any attempt to solve a technological problem is that tools and technologies should be used in appropriate ways. In cases where it becomes increasingly difficult to apply the technology, then it is likely that significant advances require fundamentally changing the nature of the problem. Instead of tackling the perhaps insurmountable artificial-intelligence problem of training machines to behave like people (‘greater AI’), the Semantic Web approach is to develop languages and protocols for expressing information in a form that can be processed by machines. This ‘lesser AI’ approach is transparent to the human users, yet brings information-space usability considerably closer to the original and inclusive vision. Another content issue is the proliferation of different content formats on the Web, which usually require explicit negotiation by the user. This barrier not only complicates and hides the essential nature of the information on the Web, but also makes the content more difficult to handle by software agents. Granted, the initial design of HTTP did include the capability of negotiating common formats between client and server. Its inclusion was based on the correct assumption that the ideal of a single content format would remain a vision obstructed by the wild proliferation of proprietary data formats. However, this negotiation feature has never really been used to advantage. The rapid adoption of HTML as the common formatting language of the Web did make the need less pressing, but the real reason is that the ever-larger list of possible formats made automatic negotiation impractical. Instead content format became classified (and clarified, compared to traditional extensionbased advertising) by importing the MIME-type concept from the realm of e-mail. Reading the MIME declaration tells the handler/client how to handle an otherwise opaque data stream, usually deferring interpretation to a specified client-side application. Although MIME declarations now formally refer to a central registry kept by IANA, there is no reason why the Web itself cannot be used as a distributed repository for new types. A transition plan would allow such a migration. Unqualified MIME types are in this scheme interpreted as relative URIs within a standard reference URI, in an online MIME registry.
Enhancing the Web
17
Services on the Web The issues of distributed services (DS) and remote procedure calls (RPC) constitute a kind of parallel development to the previously considered Web extensions, and are also considered a future part of the Semantic Web. At present, they coexist somewhat uneasily with the Web, since they operate fundamentally outside Web address space yet fulfill some of the same functionality as the proposed Web Services (WS). DS and RPC applications often use proprietary protocols, are highly platform dependent, and are tied to known endpoints. WS implementations function within the Web address space, using Web protocols and arbitrary endpoints. They also differ from DS and RPC remote operation work in that WS transactions are less frequent, slower, and occur between non-trusted parties. Issues such as ‘proof of delivery’ become important, and various message techniques can become part of the relevant WS protocols. Further discussion of these issues is deferred to later chapters. Instead the next few sections trace the conceptual developments that underpin the Semantic Web.
From Flat Hyperlink Model In the Beginning was the Hyperlink . . . The click-to-browse hyperlink, that core convention of text navigation, and ultimately of Web usability, is in reality a key-data pair interpreted in a particular way by the client software. It reads as a visible anchor associated with a hidden representation of a Web destination (usually the URL form of a URI). The anchor can be any structural element, but is commonly a selection of text or a small graphic. By convention, it is rendered visually as underlined or framed, at least by default, though many other rendering options are possible. Bit 1.10 Good user interface design imbues simple metaphors with the ability to hide complex processing The clicked text hyperlink in its basic underlined style might not have been visually very pleasing, yet it was easily implemented even in feature-poor environments. Therefore, it quickly became the ubiquitous symbol representing ‘more information here’. Functionally, the hyperlink is an embedded pointer that causes the client to request the implied information when it is activated by the appropriate user action. The pragmatics of this mechanism, and the reason it defined the whole experience of the Web, is that it makes Web addressing transparent to the user. A simple sequence of mouse clicks allows the user freely to browse content with little regard for its location. The concept is simple and the implementation ingenious. The consequences were farreaching. Figure 1.1 illustrates this concept. This ‘invention of the Web’ has generally been attributed to Tim Berners-Lee (knighted for his achievements in 2004, thus now Sir Tim), who also founded the World Wide Web Consortium (W3C, www.w3.org) in 1994. A brief retrospective can summarize the early development history.
18
The Semantic Web
Figure 1.1 Conceptual view of hyperlink functionality as it is currently implemented to form the World Wide Web. The interlinking hyperlinks in Web content provide a navigational framework for users browsing it
In the late 1980s, Tim led an effort with Robert Cailliau at the CERN nuclear research center in Switzerland to write the underlying protocols (including HTTP) for what later came to be known as the World Wide Web. The protocols and technologies were disseminated freely with no thought of licensing requirements. The early work on the Web was based on, among other things, earlier work carried out by Ted Nelson, another computer and network visionary who is generally acknowledged to have coined the term ‘hypertext’ in 1963 and used it in his 1965 book, Literary Machines. Hypertext linking subsequently turned up in several contexts, such as in online helpfiles and for CD-content navigation, but really only became a major technology with the growth of the public Web. The matter of public availability deserves further discussion. Although it may seem like a digression, the issues raised here do in fact have profound implications for the Web as it stands now, and even more so for any future Semantic Web. Patents on the Infrastructure Since hyperlink functionality is seen as a technology, like any other innovation, it is subject to potential issues of ownership and control. Even though the W3C group published Web protocols and the hyperlink-in-HTML concept as open and free technology (expressed as ‘for the greater good’), it was only a matter of time before somebody tried to get paid for the use of such technologies when the economic incentive became too tempting. The issue of patent licensing for the use of common media formats, client implementations, and even aspects of Internet/Web infrastructure is problematic. While the legal outcome of individual cases pursued so far can seem arbitrary, taken together they suggest a serious threat to the Web as we know it. Commercialization of basic functionality and infrastructure according to the models proposed by various technology stakeholders would be very restrictive and thus unfortunate for the usability of the Web. Yet such attempts to claim core technologies seem to crop up more often. In some cases, the patent claims may well be legitimate according to current interpretations – for example, proprietary compression or encryption algorithms. But in the context of the Web’s status as a global infrastructure, restrictive licensing claims are usually damaging.
Enhancing the Web
19
Further complications arise from the fact that patents are issued by respective countries, each with its own variant patent laws. Trying to apply diverging country-specific patents to a global network that casually disregards national borders – in fact, appears innately to transcend such artificial boundaries – seems impossible. It beomes especially difficult if one country’s patented technology is another’s publicdomain technology (public-key cryptography was but one early example of this quandary). In practice, U.S. patent and copyright interpretations tend to be enforced on the Web, yet this interim state of affairs is unacceptable in the long term because it suggests that the global Internet is owned by U.S. interests. If a particular technology is critical for a widely deployed functionality on the Web, allowing arbitrary commercial restrictions easily leads to unreasonable consequences. Many corporations, realizing this problem, are in fact open to free public licensing for such use (usually formally with the W3C). Sadly, not all business ventures with a self-perceived stake in Web technology are as amenable. Bit 1.11
Commercialization (thus restriction) of access cripples functionality
This view, while not popular among companies that wish to stake out claims to potential pay-by-use markets, seems valid for information-bearing networks. The natural tendency is for access and transaction costs to rapidly go towards zero. Unrestricted (free, or at least ‘microcost’) access benefits the network in that the increasing number of potential connections and relations exponentially increase its perceived value. Basic p2p network theory states that this value increases as a power relationship of the number of nodes – as 2n. Ownership issues should also be seen in relation to the general assault we can see on ‘free’ content. It is not easy to summarize the overall situation in this arena. Commercialization goals are pursued by some decidedly heavyweight interests, and often even long-established individual rights get trampled in the process. Legislation, interpretations, and application seem to shift almost daily. At least for hyperlink technology, the legal verdict appears to be a recognition of its importance for the common good. For now, therefore, it remains free. On the other hand, a failing perhaps of the current Web implementation is that it has no easy solutions to accommodate commercial interests in a user-friendly and unobtrusive way – the lack of a viable micropayment infrastructure comes to mind. Bit 1.12
Issues of ownership compensation on the Web remain unresolved
Without transparent and ubiquitous mechanisms for tracking and on-demand compensation of content and resource usage, it seems increasingly likely that the Web will end up an impoverished and fragmented backwater of residual information. Related to this issue is the lack of a clear demarcation of what exactly constitutes the ‘public good’ arena, one that despite commercial interests should remain free. Table 1.1
The Semantic Web
20
Table 1.1 The issue of ‘free’ usage and development or creation type Type
Open
Proprietary
Free to use freely Free to use, limited (licensed, non-commercial use) Pay to use (buy or license)
free stuff community rare, low cost
public good promotional usual case
outlines one way of looking at the dimensions of ‘free’ in this context. Not specifically located in it is the ‘free as in beer’ stance – that is, affordable for anyone. Hyperlink Usability What, then, are the characteristics of usability of the hyperlink? Of particular interest to later discussions on extensibility is the question: In what way is the usability lacking in terms of how we would like to use the Web? The current conceptual view of the hyperlink, as it has been implemented, has several implications for usability that are not always evident to the casual user. Current usage is deeply ingrained, and the functionality is usually taken for granted with little awareness of either original design intentions or potential enhancements. The major benefit is of hiding the details of URI and URL addressing from the user, behind a simple user action: clicking on a rendered hyperlink. Another benefit is the concept of ‘bookmarking’ Web resources for later reference, again through a simple user action in the client. These actions quickly become second nature to the user, allowing rapid navigation around the Web. On the other hand, despite the ingenious nature of the hyperlink construct, the deficiencies are numerous. They can be summarized as follows: Unidirectional linkage. Current implementations provide no direct means to ascertain whether hyperlinks elsewhere point to particular content. Even the simple quantitative metric that X sites refer to Web resource Y can be valuable as an approximate measure of authoritativeness. Mapping and presenting parent-child-sibling structures is also valuable. Backlink information of this nature can be provided indirectly by some search index services, such as Google, but this information is both frozen and often out-of-date. Unverified destination. There is no way to determine in advance whether a particular destination URL is valid – for example, if the server domain exists or the information is stored in the address space specified. The address can have been incorrectly entered into the hyperlink, or the server’s content structure reorganized (causing the dreaded ‘linkrot’ effect). Filtered access. Many sites disallow direct or ‘deep’ linking, either by denying such linked access outright or by redirecting to some generic portal page on the site. Content bookmarked in the course of sequential browsing may therefore not be accessible when following such links out of context. Other access controls might also be in effect, requiring login or allowed categories of users – and increasingly, the subscription model of content access.
Enhancing the Web
21
Unverified availability. Short of trying to access the content and receiving either content or error code, a user cannot determine beforehand if the indicated content is available – for example, the server may be offline, incapable of responding, or just applying filtered access. Relevance, reputation and trust issues. The Web is superbly egalitarian; anyone can publish anything, and all content hyperlinks have the same superficial value. In the absence of mechanisms for hyperlink feedback, the user has no direct way of determining the relevance or trustworthiness of the linked content. Visual spoofing of link destination is easy, which can lead users astray to sites they would never wish to visit, something often exploited by malicious interests on Web pages. As can be seen, the browsing user (also agents, and even the servers) could benefit from information about what exactly a hyperlink is pointing to, and from an indication of its status, operational as well as reputational. One-way linkage cannot provide this information. Existing tools are ‘out-of-band’ in the sense that they are not integrated into the browsing experience (or the client-server transactions), but instead rely on external, human-interpreted aids such as search engines or separate applications (say hyperlink checkers). Wouldn ’t it be nice if . . . Some features users might find useful or convenient when browsing the Web can be listed as a sampler of what the current infrastructure and applications either do not do, or do only with difficulty or limitations in special cases/applications. At this point, we are not concerned about the ‘how’ of implementation, only the user experience for some fairly simple and straightforward enhancements. Some of these features are hard to describe adequately in the absence of the ‘back-end’ data and automation, and they will depend greatly on commercial, political, and social decisions along the way as to what is desired or allowed. Context awareness, which would allow ‘intelligent’ client behavior. Ideally perhaps, semantic parsing of document context could reduce the need for user decision and simply present ‘most-probable’ options as distinctive link elements at the rendered location. Alternatively, the client might provide an on-the-fly generated sidebar of ‘related links’, including information about the site owner. Several proprietary variations of the theme are found in adware-related generation of ‘spurious’ hyperlinks in displayed content, which however is a more intrusive and annoying implementation in the absence of other cues. A simple example of a context feature for content is found in newer browser clients, where the user can highlight text and right-click to access Web search or other useful actions, as illustrated in Figure 1.2. Persistent and shareable annotations, where users can record private or public comments about Web content automatically attached to a given document (URI) and ideally internal location, in context. Public comments would then be shareable among an entire community of users – perhaps most usefully in goal-oriented groups.
22
The Semantic Web
Figure 1.2 A simple example of context awareness that allows a browser user to initiate a number of operations (such as Web search, Dictionary or Encyclopaedia lookup, or translation) based on a highlighted block of text and the associated context menu
Persistent and shareable ratings, which is a complement to annotations, providing some compiled rating based on individual user votes. A mouse-over of a link might then pop up a tooltip box showing, for example, that 67% of the voting users found the target document worthwhile. Of course, not everyone finds such ratings useful, yet such ratings (and their quality) might be improved by a consistent infrastructure to support them. More realtime data, which would result from the ability of Web clients and services to compile Web pages of data, or embed extended information, culled from a variety of sources according to the preferences of the user. Persistent and shareable categorizations, which would be a collaborative way to sort Web documents into searchable categories. The common theme in most wish lists and methods of addressing the deficiencies of the current Web model is that the issues are based on adding information about the content. Then the user client can access, process, and display (or use it) in ways transparent but useful to the user – ideally with some measure of user control.
To Richer Informational Structures The further information about the content that the previous shortcomings and wish list highlight include the metadata and relational aspects, some of which might be implemented as distributed services on the Web. One idealized extension of the originally unidirectional information flow could look as in Figure 1.3.
Enhancing the Web
23
Figure 1.3 Conceptual view of hyperlink metadata functionality as it might be implemented to form an enhanced Web. Metadata and relational data are gathered and presented to the user for each embedded link
The point in the enriched hyperlink model is that further information about each hyperlink destination is automatically gathered and processed in the background. This behavior is an extension of how some caching solutions already now parse the currently loaded content and pre-fetch all hyperlink-referenced content into a local cache, ready for the next user selection. In the new model, the client software would also be busy compiling information about the referenced content, ready to present this to the user even before a next destination is selected. Link metadata might be presented to the user as tooltip text, for example, or in ancillary windows on a mouse-over of the link. A very limited and static simulation of this feature can be seen in the use of the ‘title’ attribute in a hyperlink markup (as shown in Figure 1.4) – the difference is that this kind of metainformation indicator must be precoded by the content author as an attribute in the hyperlink tag itself.
The Collaboration Aspect The way that we have viewed the Web has generally been one of computer users browsing a web of content; independent readers of static information, or possibly also sole publishers of personal Web sites. This view was never the whole truth.
Figure 1.4 Static meta-information precoded by content author into the hyperlink tag of the content itself, using the ‘title’ attribute. When rendering the page, the browser pops up a ‘tooltip’ box to show the hidden text when the cursor is placed over the link
24
Bit 1.13
The Semantic Web
In the early days of the Web, considerable collaboration was the rule
What is interesting about the early collaborative effort is that it occurred in a smaller and more open community, with implicit trust relationships and shared values. Among those who maintained Web pages, compiling and publishing resource lists became a major part of the effort. To find information on the Web, a user would visit one or another of these lists of hyperlinks and use it as a launchpad, long before the idea of ‘portal sites’ became popular on the Web. The implicit metadata in such a reference represents the value assessment of the person creating the list. Such personal lists became more specialized and interlinked as their maintainers cross-referenced and recommended each other, but they also became vastly more difficult to maintain as the Web grew larger. Some directory lists, almost obsessively maintained, eventually grew into large indexing projects that attempted to span the entire Web – even then known to be an ultimately hopeless goal in the face of the Web’s relentless and exponential growth. One such set of lists, started as a student hobby by David Filo and Jerry Yang in 1994, quickly became a mainstay for many early Internet users, acquiring the name ‘Yet Another Hierarchical Officious Oracle’ (YAHOO). By 1996, Yahoo! Inc. (www.yahoo.com) was a viable IPO business, rapidly diversifying and attaining global and localized reach. Yahoo! quickly became the leader, in both concept and size, with a large staff to update and expand manually the categories from a mix of user submissions, dedicated browsing, and eventually search-engine database. Other search engines went the other way: first developing the engine, then adding directories to the Web interface. A more recent collaborative venture of this kind is the Open Directory project (www. dmoz. org). An offshoot to cross-linked personal lists and a complement to the ever-larger directories was the ‘WebRing’ (www.webring.org) concept. Sites with a common theme join an indexed list (a ‘ring’) maintained by a theme ringmaster. A master meta-list of categories tracks all member rings. To appreciate the size of this effort, WebRing statistics were in July 2002 given as 62,000 rings and 1.08 million active sites, easily capable of satisfying any user’s special interests indefinitely. However, the WebRing phenomenon appears to have peaked some years ago. presumably shrinking in the face of the ubiquitous search-engine query. The figures show a clear trend: 52,250 rings with 941,000 active sites in August 2003, and 46,100 rings with 527,000 sites in February 2005 The individual user of today tends increasingly to use the more impersonal and dynamic lists generated by search engines. An estimated 75% of user attempts to seek Web information first go through one of the major search engines. Although search-engine automation has reduced the relative importance of individual collaboration in maintaining the older forms of resource lists, it is still not the panacea that most users assume. Issues include incomplete indexing of individual
Enhancing the Web
25
documents, query-relevancy ranking, and handling of non-HTML and proprietary formats. Search engines also face the same problem as other approaches of scaling in the evergrowing Web. They are unable to index more than a small fraction of it. Estimates vary, but the cited percentage has been shrinking steadily – recently 10%, perhaps even less. The bottom line is that centralized indexing in the traditional sense (whether manual or automatic) is increasingly inadequate to map the distributed and exponential growth of the Web – growth in size, variety, updates, etc. Bit 1.14
New requirements and usage patterns change the nature of the Web
The adaptation of any network is driven by requirements, technology adoption, resource allocation, and usage – if only because new and upgraded resources usually target areas of greatest loading in order to maintain acceptable levels of service. Peer to Peer (a previous book) is replete with examples. Returning to collaboration, it has become more important to support new and more flexible forms – between individuals and groups, between people and software, and between different software agents. Loose parallels could be drawn between this adaptive situation for the Web and the changing real-world requirements on telephone networks: As first, subscriber conversation habits changed slowly over the years, and group conferencing became more common for business. Later, cellular usage became ubiquitous, requiring massive capacity to switch calls between line and mobile subscribers from many diverse providers. As dial-up Internet access became more common, individual subscriber lines and exchanges became increasingly tied up for longer intervals. Digitizing the network and applying packet switching to all traffic helped solve some congestion problems by letting separate connections coexist on the same circuit. In all cases, telecom operators had to adapt the network, along with associated technologies, to maintain an acceptable service in the face of shifting requirements. The Web is a virtual network. The underlying Internet infrastructure (and the physical access and transport connectivity under it) must constantly adapt to the overall usage patterns as expressed at each level. New collaboration technology has generally just deployed new application protocols on top of the existing Internet and Web ones, forming new virtual networks between the affected clients. Better collaboration support is achieved by instead enhancing the basic Web layer protocol. Other, proprietary solutions are possible (and some do exist), but making these features a fundamental part of the protocol ensures maximum interoperability. Just as we take for granted that any Web browser can display any standard-compliant Web page, so should clients be able to access collaboration functionality. Providing mechanisms for dealing with content meaning that will automatically provide intelligent mechanisms for promoting collaboration in all its forms is also one of the aims of the Semantic Web.
The Semantic Web
26
Figure 1.5 The traditional distinction between ‘‘document’’ and ‘‘message’’ is illustrated in the context of the client-server model, along with some terms relevant to requesting and viewing Web content
Extending the Content Model Discussions of ‘content’ without qualifications or explanations might be misleading for several reasons, and the term is easily misunderstood; this is why the terms ‘information’ and ‘information object’ crop up in these discussions. It is a common assumption that ‘Web content’ refers to the text or other media objects provided by Web pages and their related links – static information physically stored in welldefined server locations. In addition, historically, there has always been a distinction maintained between ‘messages’ on the one hand, and ‘documents’ (or files, or content) on the other (see Figure 1.5); that protocols are defined on top of messages in order to transport documents, and that further protocols are defined in terms of messages exchanged by previously defined protocols, and so on. Such recursive definitions, while sometimes confusing, occur all the time and can prove very useful. As implied earlier, p2p protocols that define separate virtual networks are defined by the exchange of underlying HTTP messages. However, especially in the context of XML, it is often better to view content as the sideeffect of an ongoing exchange of messages between various endpoints. It matters not whether these endpoints are servers of static Web pages, database servers, Web services, other users, or any form of software agent. Also, it does not matter whether the information is actually stored somewhere, compiled from several sources, or generated totally on the fly. Higher-level protocols are generally implemented as message-based transactions that can encapsulate information and pointers to information in many complex ways. Recall that requesting information on the Web, even in a basic protocol such as HTTP, only returns a representation of the information, encoded in some way acceptable at both endpoints – only a message, in effect, from the process providing it and thus inextricably bound to the protocol. Bit 1.15
Messages pass representations of information
The issue of information representation by proxy and subsequent interpretation is far more fundamental than it might seem. The concept has deep philosophical roots, and farreaching implications about how we know anything at all and communicate.
Enhancing the Web
27
This generalized view, that all content is the same exchange of messages expressed in a single protocol, amounts to a useful simplification. It can result in a merging of the current fragmentation of different protocols for e-mail, file transfer, newsgroup postings, instant messaging, chat, Web pages, and other ‘content’. The same client, based on such a harmonized protocol, could transparently handle arbitrary information objects in a consistent way from the user perspective.
Mapping the Infosphere Navigating the wealth of information on the Web is not a trivial task, even assuming that it is all openly accessible. Various methods have gradually increased or decreased in relative importance for users over the years. In roughly chronological order, these navigational methods are summarized in the following list: Resource lists present hyperlinks that particular users have collected and recommended on their own Web sites. An outgrowth of this resource was the ‘meta list’, or list of hyperlinks to sites that maintained such lists. The quality and comprehensiveness of such lists could vary immensely. Meta directories are large hyperlink lists compiled by dedicated staff into some kind of hierarchical category trees, often with the aim of being a comprehensive guide to all content. Examples are the Yahoo! Web Site Directory (www.yahoo.com), Google directory (directory.google.com) and the Open Directory Project (dmoz.org). A problem with hierarchical listings is classification, which can be arbitrary, and may or may not coincide with the user’s perception of an appropriate category. Search engines provide ad hoc lists of hyperlinks to Web pages (or other Internet resources) culled from indexed databases collected and updated by automated ‘spiders’ that systematically traverse the Web hyperlinks on each Web site they can reach. Search results are often ranked according to some opaque metric of ‘relevancy’, but depend critically on the user’s ability to formulate ‘optimal’ search criteria based on suitable terms, along with the quality of the indexing process. Special interest compilations lie somewhere between idiosyncratic lists compiled by individuals, and the comprehensive Web directories. They aggregate self-nominated, like-themed sites into interlinked Web rings. Each site in a ring can directly or indirectly refer to its neighbors through a ring index, each ring organized in a category hierarchy in some meta ring. Web services can automate and manage resource discovery and content retrieval (including compilation and recasting) on behalf of a user. Such services might incorporate ‘learning routines’ to adapt to each user’s history of requests to determine scope and relevancy criteria, for example. Currently, Web information is largely ‘atomistic’, with relatively few and arbitrary relations defined between the different information objects – see also the previous discussion about hyperlink usability. This deficiency and the constraints of the current hyperlink model clearly have a severe impact on user navigation. The lack also severely hampers the development of automated agents for such purposes.
28
The Semantic Web
Structure and Relationship Traditionally, maps of Internet resources in general, and the Web in particular, have by necessity been very simplistic and one-dimensional: Human-compiled hierarchy directories have the virtue of including some, albeit variable, amount of metadata based on human assessment of the link target and classification into categories. Expressed relational metainformation is an external construct, however, independent of any explicit hyperlinks between sites. Automated link-walk compilations result in maps that are in this case purely structural – a ‘snapshot’ of Web hyperlinks. While subsequent indexing of at least portions of content from each node does allow for term searches to create simple maps that reflect literal term usage (and thereby indicate term-related linking), the content semantics are largely ignored. We can illuminate this distinction and some of the issues involved by looking to some well-established conventions from paper publishing. Paper-published content is inherently linear, because it has its origins in oral narrative. Unlike transient speech, however, published sentences are frozen in a medium that the reader can to some extent perceive outside of time, freely choosing what to read at any time. This characteristic enables the concept of multiple entry points based on some overall mapping. Paper publishing thus developed structural and semantic aids to the reader to provide such entry points into the content: Chapters and sections. As a structural device, the division of a text into chapters and sections imparts a layer of semantics in that the logical division helps order content and implies relationships between the parts. Table of contents. Given an expressed logical structure, it is possible to provide a list of headings and corresponding numbered pages. The reader can use the table of contents to gain an overview of the content structure and quickly find specific content by following the logical divisions. Abstracts and summaries. Older novels commonly provided short abstracts for each chapter, and although this practice was later dropped in fiction, the abstract lives on in academic publications. Summaries (especially executive summaries) are a similar device. Both variants are commonly set apart from the main text by special styling (such as extrawide margins, vertical whitespace, and italic font). A listing of keywords may be considered as an extremely abbreviated summary. Index. From the other end (and it is not coincidence that the index is found at the back of a book), a list of terms and their page-location in the text constitutes an index. Creating a useful index is a highly developed art; it is not simply a compilation of occurrences of all the words. Instead It critically relies on the compiler’s understanding of relevancy and importance – of which words or phrases to include, which occurrences to index in terms of the semantic context, which occurrences to ignore, which relationships to highlight using categories and multi-level listings, just to name a few considerations. Other semantic-mapping contructs also exist in publishing, but those listed here suffice for the purpose of comparison with the Web. For example, comparing with the previous
Enhancing the Web
29
examples of Web maps, we can see a rough correspondence between the human-compiled category list and the table of contents. In passing, we may note that despite well-developed mechanisms for multiple entry points for the reader, not to mention advanced non-linear reading possibilities in electronic form, most written content maintains the fiction of linear narrative, from beginning to end. The search-engine list of links derived from automated indexing would perhaps approximate the index, except for the fact that such a list compilation is currently an index created solely by listing all occurrences of the search terms. It lacks the semantic order and weighting that the expert indexer imparts, and critically, the distinction between different usages and meanings – for example, between the noun ‘book’ and the verb ‘to book’ (as in make a reservation). Natural language is replete with ambiguities of usage that only information about semantic context can even begin to resolve. With clever programming, Web searches can at least be made tolerant of some common typographical mistakes and regional spelling variations, but these searches remain semantically ignorant. Proper logical markup of text structure in Web content can somewhat offset this analytical lack, but at the cost of reliance on external conventions that may or may not be correctly applied in particular instances. Text markup still does not convey inherent semantic information, only overlain syntactical. Although Google is commonly cited as ‘the best’ search engine, mainly due to the way it leverages backlinks to provide relevant results ranking, it is not hard to find consistently ‘bad’ or skewed results in simple searches. Such frustrations are often due to the engine’s strictly lexical basis combined with the skew effect of some kinds of linked content. For search words that also relate to product names, for example, Google ranks online shopping pages higher than any other results, probably because of the wealth of product links both on-site and from affiliate sites. Advanced syntax may therefore be needed to find information sites not devoted to selling products. Links, Relationship, and Trust The fact that object A unidirectionally links to object B says little about any real relationship between them. Any metainformation about this relationship is largely inferred by the user. For example, we are willing to believe in a high degree of relevance of a hyperlink on a page that refers to another page on the same site, even in the absence of any explicit declarations about it. Such trust may prove unfounded, however, especially if the link is ‘broken’ (that is, the URL is temporarily or permanently invalid). But the point is that the hyperlink itself contains no metadata whatsoever about the relationship or target status. At best, the contextual text around the link might explain something about the target page; at worst, it might be totally misleading. The situation is even less reliable when the link targets an object outside the site, because then the page/link author does not even have nominal control over the target object. Target site contact might have changed drastically since the link was created, for example, or the
30
The Semantic Web
target site might employ filtering or redirection that makes the direct reference invalid and the target hard or impossible to track down manually. Trust is also at issue when a link (URL) points to a Web page that defines an association between URL and URI – perhaps the easiest way to create and publish a URI Anyone can do it, and anyone can on the face of it misrepresent an association to a particular URI. Trust issues of this nature are probably best resolved by embedding currently accepted mechanisms based on content hashes and public key signing of the published assertion. The assertion can then be independently verified as unaltered and the publisher identity verified through some established chain of trust – in the worst case, just identified as the same publisher of other similarly-signed material.
Well-Defined Semantic Models When considering strategies to map information on the Web, one can begin by outlining some of the features that the mapped content, commonly expressed as ‘documents’ in these contexts, should exhibit or promote: It must be possible to define precisely a language (as a set of tokens, grammar, and semantics) in which to deal with ‘first class objects’ (which means document objects that are URI-referenced). Such a language is a prerequisite to processing. It must be possible to make documents in a mixture of languages (that is, allow language mixing). It is a fact of life that no single language can or will be used in all contexts. Extensibility is natural and alternate structures must be allowed to coexist. Every document should be self-defining by carrying the URI(s) of the language(s) in which it is written. This mechanism enables both humans and more importantly software to determine the appropriate methods for analyzing and processing the information. It must be possible to process a document while understanding only a subset of the languages (that is, partial understanding). In other words, full functionality must ‘degrade gracefully’ so that simpler processing can safely ignore the full set of features. The requirements in this list form the basis for XML development, for example, and it defines a first step towards enabling the encoding of semantic meta-information. The current Web recommendation is that XML should be used when a new language is defined. A new language (or, for that matter, new features extending an existing language) must then be defined as a new namespace within XML. Although this kind of language defining might be seen as akin to misguided attempts to (re)define all natural languages in terms of some grand metalanguage (as in the tradition of applying Latin grammar), as an artificial construct a priori applied to other artificial constructs, it can be both attainable and practical. Given the language, how do we deal with the meaning of what is expressed in that language? This issue is at the core of the Semantic Web, and a discussion around it forms the bulk of Chapter 2.
2 Defining the Semantic Web When discussing the Semantic Web, it is prudent to note that we are in fact dealing with but one of several interrelated emerging technology domains: Web Services (providing distributed functionality), Grid Services (integrating distributed resources), and the Semantic Web (mapping and managing the resources). This book’s primary focus is on the last of these. In order to map Web resources more precisely, computational agents require machinereadable descriptions of the content and capabilities of Web accessible resources. These descriptions must be in addition to the human-readable versions of that information, complementing but not supplanting. The requirement to be machine-readable is non-negotiable, because in future – in fact, practically already now – almost all software (and devices) will be innately Web-aware. Leveraging autonomic-computing technology enables agents and empowered devices to display independent activity and initiative, thus requiring less hands-on management by users. Already, an important expectation about software is that even ‘dumb’ applications can deal with basic Web connectivity in ways appropriate to their primary function.
Chapter 2 at a Glance This chapter introduces the architectural models and fundamental terms used in the discussions of the Semantic Web. From Model to Reality starts with a summary of the conceptual models for the Semantic Web, before delving further into detailed terms. The Semantic Web Concept section analyzes and motivates this particular extension to the Web. Representational Models reviews document models relevant to later discussions. The Road Map introduces some core generic concepts relevant to the Semantic Web and outlines their relationships. Identity must uniquely and persistently reference Web resources; interim solutions are already deployed.
The Semantic Web: Crafting Infrastructure for Agency Bo Leuf # 2006 John Wiley & Sons, Ltd
The Semantic Web
32
Markup must enable the distinction between the content representation and the metadata that defines how to interpret and process it. Relationships between data objects must be described. Reasoning strategies for inference engines must be defined. Agency on the Web can be implemented to apply reasoning on the other layers. Semantic P2P outlines the role of peer-to-peer technology in this context. The Visual Tour summarizes the high-level view. This section highlights The Map, The Architectural Goals and The Implementation Levels, and discusses the recent W3C architecture recommendations.
From Model to Reality The concept of a ‘semantic Web’ supposes a plan for achieving a set of connected applications for data on the Web in such a way as to form a consistently readable logical Web of data – the actual implementation of the Semantic Web (‘sweb’). The steps needed to get from the model of the Semantic Web to reality are many. With our current understanding of the model, the process will require many iterations of model refinement as we design the systems that will implement any form of Semantic Web. The Web was designed as an information space, with the goal not only that it should be useful for human–human communication, but also that machines would be able to participate and help. One of the fundamental obstacles to this vision has been the fact that most information on the Web is designed for human consumption. Even if derived from a database with well defined meanings (in at least some terms) for its columns, the structure of the data is rarely evident to a robot browsing it. Bit 2.1
When parsing data, humans read, while machines decode
Actually, humans do both: first decoding the syntactical representation, then applying built-in semantic rules and context inferences to interpret the meaning of the data. For machines to ‘read’ in this way, they also need semantic rules and inference logic. Initially leaving aside the ‘greater AI’ problem of training machines to behave like people (though assuredly some research along these lines is always being pursued), the ‘lesser AI’ and sweb immediate approach is instead to develop languages and codified relationships for expressing information in a machine-processable form. The machines must be able to do more than simply decode the data representations on the syntactical level; they must be able to perform logical processing and inference on well-defined meanings of the information tokens parsed from the data. Before going on to discuss possible models for this kind of machine enhancement, we might reasonably ask: What are the requirements on the content representation for meaningful reading to be feasible? The answer depends a fair bit on whom you ask. However, there is some consensus for the following content features: distinguishing markup from data; avoiding ambiguity; discouraging arbitrary variation.
Defining the Semantic Web
33
The linguist might at this point object that natural languages apparently thrive with a remarkably high degree of the latter two, ambiguity and variation – despite the many formal rigidities of syntax, grammar, and definition. Even the far more rigid and literal programming languages have their fair share of both characteristics. As it happens, people not only cope with ambiguity and variation, they actively crave and enjoy it, as evidenced in various forms of literature, poetry, and humour. It would seem that we as humans are predisposed forever to balance on the linguistic cusp between coherency and incoherency, playing constant games with the way we use language. This constant play and variation is one of the hallmarks of a natural, living language. On the other hand, computer languages (and by extension computer-logic implementations) have as a basic assumption the elimination of ambiguity, or at least they implement some formal abstraction so that the programmer need not consider it. Bit 2.2 Current computer logic design is blind to important parts of human communication It seems paradoxical that in this way we deliberately exclude the ability to automate the processing of a significant component of the information we produce. A few computer-languages do exist that are arguably exceptions to this literalistic constraint, inherently allowing some idiomatic variations and implicit referencing in ways that are similar to human languages. However, most programmers find such languages ‘difficult’ or ‘quirky’. While few might appreciate the ability to write poetry in code, such frivolous activities must have at least some bearing on the issue of creating programs that can deal with texts in ways that approximate human semantic processes. One practical field that demonstrates the necessity of semantic processing is that of machine translation of human languages. The limitations of syntactical and lexical parsing, not to mention the profound misconceptions of how humans process language meaning, have for decades confounded the practical realization of automated translation. At best, current implementations can achieve only tolerable results within narrow specialist fields, providing but a first approximation of meaning for a human reader.
The Semantic Web Concept Returning to the ‘plan’ to extend the Web (more of a vision, actually), we can start with a sort of ‘mission statement’ made by three of the top names behind the Semantic Web initiative: The Semantic Web is an extension of the current Web in which information is given well-defined meaning, better enabling computers and people to work in cooperation. Tim Berners-Lee, James Hendler, Ora Lassila
The Semantic Web
34
A reformulated version appears later to guide the formal W3C ‘initiative’: The goal of the Semantic Web initiative is as broad as that of the Web: to create a universal medium for the exchange of data. www.w3.org/2001/sw/Activity
To achieve this goal, existing Web content needs to be augmented with data (for example in the form of semantic markup) to facilitate its handling by computers, and with documents (or more generally, resources) that are intended solely for computers and software agents. It is envisioned that the resulting infrastructure, transforming the Web into the Semantic Web, will spur the development of automated Web services to fulfill a wide variety of purposes. Computers will be able discern the meaning of semantic data by following hyperlinks to term definitions and rule sets. Thus armed, they can reason about content logically and make decisions that are relevant to the context and to user expectations. Bit 2.3
Humans are the current Web’s semantic component
Today, humans are required to process the information culled from Web (Internet) resources to determine ultimately its meaning and relevance (if any) for the task at hand. Sweb technology would move some of that processing into software. Some writers want to (and often do) call such semantic processing simply AI. But after more than half a century of shifting expectations and broad popularization, the generic term ‘AI’ seems too loaded with old baggage and popular misconceptions to use without further qualification. For now, a better approach is to use more neutral and specific terms (such as ‘business logic’ or ‘semantics’) when discussing processing functionality in the context of the Semantic Web. Reasoning aside, we are not as yet dealing with any sort of ‘thinking’ software in the popularized ‘self-aware AI’ sense. Part III revisits the issue, however, and does discuss the broader AI aspects from a speculative perspective. The Near-term Goal More useful for conceptualization might be to see the envisioned change as moving from a Web that now reads like a random spread of open books, to a Web that instead reads like a well-structured database – not a centrally dictated order, though some might advocate such a vision, just a locally applied, machine-decipherable structure. In other words, the near-term goal is for a Web that is better suited to answering queries about the information published there. At its simplest level, such a change would mean that, for example, search engine results could be pre-filtered for some measure of ‘correctness’ in relation to the query’s semantic context. Such filtering is impossible in lexical searches. But perhaps more profoundly, it can mean the contextual structure to make actual user-invoked ‘searching’ less relevant. When asked what the ‘killer application’ of the Semantic Web would be, the clued-in proponents reply: The Semantic Web itself! They justify this reply by noting that the killer application of the current Web is precisely the Web itself – the Semantic Web is in this view
Defining the Semantic Web
35
just another expression of new innate functionality of the same magnitude. The capabilities of the whole are therefore too general to be thought about in terms of solving one particular problem or creating one essential application. The Semantic Web will have many undreamed-of uses and should not be characterized in terms of any single known one. Bit 2.4
The ‘Metadata Web’ must grow as the original Web did
As was the case with the growth of the original hyperlinked Web of content, it can be argued that until anyone can easily create metadata about any page and share that metadata with everyone, no true ‘semantic web’ will arise. The Semantic Web is most likely to develop first as expanding ‘islands’ of new functionality that over time become more generally adopted as various service providers realize the utility (and derived benefits or profits) of doing so. Such ‘semantic intrawebs’ are a reasonable first step to test the new approaches and tools, at a smaller yet fully functional scale, and therefore improve them before they are unleashed into the Web at large. To provide some context for this evolution, consider the following:
The original Internet was designed to pass messages between systems. The Web as we know it is largely about finding information. Web Services implement distributed functionality on the Web. The Grid intends to integrate functionality provided by the parts. The Semantic Web is concerned with describing the available resources, making the data accessible, and providing the agency to manage them.
Although very superficial descriptions, these items do capture something of the essence of the development. The latter three give rise to a qualitatively different dynamic nature in the Web than what we are currently used to.
Addressing some Practical Features Let us put the abstract vision into a few simple possible contexts by recalling some of the deficiencies and desired features discussed in Chapter 1 and seeing what a Semantic Web infrastructure could add in practice to the personal user experience. User-context tracking. Imagine that your Web client could (if you want it to) track your semantic context. For example, it might pre-filter search results to show only those hits with a high correlation to the meaning implied by previous contexts. Imagine further that meaningfully related search terms were automatically generated to catch resources that would otherwise be missed only because a particular search word did not occur or was spelled differently in the target. When browsing, the client might always highlight Web links that lead to resources with a high semantic correlation to the current context, based on background metadata processing. It might even suggest links not mentioned in the displayed content, and provide pop-up comments on demand.
36
The Semantic Web
Context-defined interactivity. Suppose the semantic context could bring to the foreground appropriate media types, or even different interactions with the data browsed. A table could be just a matrix of values, a graph could be just a bitmap picture of relations. Each could be a gateway to further information about each element/relationship. Elements could be manipulated in various ways to investigate other relationships. All would depend on the current or selected context. User-preference tracking. Anyone experiencing the ever-increasing prevalence of e-mail spam, and faced with the imponderable oddities of available junk-mail filters (such as in MS Outlook or in subscribed Web-based services), must wish for some simpler solution. Ideally one would simply tell a filtering agent to ‘ignore further mails of this kind’ and have it identify ‘similar’ content reliably, transparently adapting to a wide range of variation. Actually, what the user might often find more useful is the converse feature of the agent being able to highlight and give priority processing to desired mail that is semantically ‘valid’ and matches a current context or preference setting. The common thread here is an enhanced cooperation between resource server, browsing client, and user – where detailed content analysis and classification relevant to the current user is delegated to software agents, and where even the selection of data can depend on the context. Such enhancement can only work in a decent way if the software is capable of ‘seeing’ the meaning of content in a defined context, not just the lexical words and structural tags. Although this list may give the impression that we assume all such enhancements to be beneficial to and desired by the average user, this conclusion would be unfounded. At a minimum, as implied in the first item, the features should be optional and easily controlled by the user. At worst, otherwise useful features might be hijacked by content-publisher or advertising interests, reducing them to the equivalent of intrusive commercial spots in broadcast media. Bit 2.5
An important issue: Metadata functionality on whose terms?
The ideal vision presupposes that the user is in control and is the benefactor of the outlined enhancements. Although the distributed and peer architecture of the Web is inclined to promote (or at least allow) such a development, other strong interests in government and business often follow different, less benign agendas. Early precursors to context and tracking features have already appeared in limited ways, but they are not as yet semantically very clever. More critically, most of them are in fact not addressing user requirements at all, but are being pushed by business and advertising interests. Thus the user has little control over how these features work, or of when they are active. Often, the data collected is just reported back to the advertiser site, not helping the user in any way. The best that can be said for the context-tailored user profiles we have seen so far is that they may reduce the incidence of totally irrelevant banner ads – such as U.S. mortgage rates displayed to apartment dwellers in London. The user must of course first agree to such external profiling.
Defining the Semantic Web
37
What it Means for the Publisher In the Web of today, publishing content all too often means creating one-off presentations. Data are selected, a layout chosen, and the result stored as a page on a server somewhere. Updating changing data requires new presentations to be created and uploaded to the server. (Initially, I used the term ‘static’ here, but found it misleading on reading, as much such content is distressingly ‘dynamic’ to the user visually, in the form of animated images and intrusive shockwave-flash elements.) Moving to database-driven, dynamic pages (in the intended updated-content sense) is a step forward, in that it ensures presentations of the most current data. However, these pages remain locked to a particular layout. The publisher still needs to determine (or guess) the requirements of the visiting user, and design the database and page format accordingly. Trouble is, there is really no way of knowing what the visitor is looking for, or how the data will be used. Both issues are crucial, however, to a successful presentation design. Bit 2.6
Metadata allows ‘application-neutral’ publishing of data
The ideal for multi-purpose publishing is to provide ‘raw data’ that can be used and reused in a variety of contexts. Data descriptions can decouple the resource itself from issues of presentation, media type, and client behavior – user aspects otherwise to be determined by the collecting agent in the current context of the user. Granted, not everyone wants to publish raw data, and there are many legitimate reasons for retaining strict control of presentation, media type, and purpose. Clearly, the decision must remain with the publisher. However, by restricting access to fixed formats, the publisher also constrains the possible uses of the information – and incidentally risks that another, unsupported use will simply lift a snapshot of the data (soon out of date, perhaps) to embed in another presentation. Many useful applications of data are unforeseen (and unforeseeable) by the original publisher. The realistic choice for control thus lies somewhere between the two following extremes: Constrained access with complete control of presentation (‘link to this page’), yet paradoxically no control at all of ultimate information use and relevancy elsewhere. Unfettered access, with little direct control of how the information is presented, yet with the high and desired probability that users are linking to the resource directly to pick up the most current data on demand, instead of relying on stale static snapshots of earlier data. The latter approach does work in business, and it is gradually being adopted as companies realize the competitive benefits of allowing real-time access to the raw content data. For example, Amazon.com, with a long history of encouraged product linking from affiliate Web pages, has moved in the direction of more service-oriented support, such as for dynamic backlinks to Amazon servers. These define-once links automatically pull and embed up-to-date product images and pricing data when the affiliate page is viewed.
The Semantic Web
38
With similar access to manufacturer or vendor databases, resellers in general can quickly produce online catalogues as living documents, where product information and pricing is sourced and embedded in real time. Service providers can implement real-time inventory status, customer ordering, and support for sold services. All such solutions become easier to implement flexibly when consistent metadata exists alongside the required information. It is a question of business realizing that open-ended formats for published data do in fact benefit the bottom line. Extending in-house information to reuse by intermediaries greatly extends the reach of both the manufacturer and the vendor. Bit 2.7
People do not just want to read about it, they want to interact with it
Metadata descriptions and access to ‘raw’ data brings the virtual world closer to our real world, because we can then do more things online, on our own terms. As an example, imagine the added value of user/agent interaction affecting the presentation of a Web-published cookbook of recipes. The concept as such is fairly simple (and very popular), but a common reader issue is about scaling the size of the recipe to fit the dinner context. A sweb version that ‘understands’ content and context could, for example, recast browsed recipes to your current requirements (such as the specific number of people for dinner), offer substitutions for ingredients that you flag as unavailable (or point you to more suitable recipes), and suggest links to shops near your location that stock what the recipe requires. Smart agents could even warn that a particular recipe will not easily scale to the required size in the context of your kitchen, and the user interaction would start to resemble a conversation as you vary parameters and preconditions to find a better solution. Disentangling the Markup As noted, a core part of any Semantic Web implementation must concern the concept of machine-understandable documents. The ‘easy’ part is the clean separation of actual content (and critically, its meaning) from the presentation aspects – that is, from how the content is to be rendered by any given client in a particular context. Current HTML encoding practices routinely entangle structural and visual markup (see detailed discussion of markup in Chapter 5), despite the long availability of CSS to separate out the latter in a consistent and reusable way. Part of the blame for this state of affairs lies with still-inconsistent support of CSS in authoring and client implementations, and part lies with the original design of tag sets to aid computer typesetting. This legacy concern with typefaces, explicit visual layout, and the need to check the consistency of the document structure lives on in HTML. In addition, HTML is unable to address self-defining semantic markup; it can only deal with a limited and externally predefined syntactic structure. Bit 2.8
Useful semantic markup must be extensible and self-defining
We can never know (and therefore never standardize markup for) all possible contexts in advance. Usage evolves, and local yet portable extensions must be supported.
Defining the Semantic Web
39
Figure 2.1 Making Web content machine-understandable requires disentangling the meaningful content from the markup. Separating structure and presentation is only part of the answer. XML goes further by allowing self-defining tags to set apart meaningful parts of the text
Figure 2.1 depicts something of this relationship, adding a comparison with XML which, with its extensible and self-defining nature, appears to be the markup language of choice. Style sheets can be specified as CSS or XSL. Customized XML tags can set apart bits of text in ways that are semantically meaningful for understanding the context, and so provide a starting point for machine analysis of it. Even in XML, however, systematic reuse and repurposing of content requires that all presentational characteristics be stripped from the dataembedded markup. Although this task would seem straightforward, along the lines of CSS but applied to XSL, it in fact proves very difficult. One surprising result when attempting to formulate simple and consistent XSL rules for existing content is that much erstwhile ‘content XML’ idiom turns out to be mostly presentational in nature. Humans seem predisposed to mix presentation and semantics – it may be just the way we perceive the world. Perhaps our human perspective is so innately skewed to presentation scaffolding that we literally cannot isolate the semantic meaning without using strict tools to resolve it. Bit 2.9
Meaning is in the eye of the beholder
More research is needed into discovering how humans parse and classify meaning from various forms of presentation and communication. The later sections on ontologies, however, do provide some tools to grapple with this situation, and perhaps suggest a way out of the dilemma when we try to devise structures that can be processed by machines. Framework Constraint Any descriptive framework contains both visible and hidden assumptions, and XML is no different. For example, XML (as a derivative of SGML) presumes a tree structure when used to describe data. It expects, or tries to enforce, a universal linear representation of hierarchical structure, with important consequences for what it can and cannot describe easily in its syntax.
40
The Semantic Web
Figure 2.2 An illustration of the problem of excerpting content that is found in the context of linear markup. Missing or out-of-context tags break the syntactical sequence for correctly parsing markup
This artificial constraint can be a problem, especially when it comes to ‘excerpting’ content for use elsewhere, such as is easily envisioned when machines are to process information for remote presentation. In particular, the tree-based syntax makes assumptions about markup scope and span that can be undefined outside the linear context of the original text, as illustrated in Figure 2.2. Tree-based markup under any recognized SGML/XML convention is unable to easily represent all arbitrary structures. It fails to adequately represent overlapping or concurrent structures (as in the illustrated example). This fundamental limitation does not belittle the proven usefulness of the possible representations, but any system based on such a markup convention must work under constraint and therefore will not be able to handle easily other content structures. Another simple example is text citation, where a number of common conventions (such as annotations, quoter’s emphasis, ellipsis, and clarifying insertions) would perhaps translate to multiple layers of overlay markup, complicated by the additional fact that the cited content is no longer within the original markup context. Note that such constraints in practice also apply to much research and many constructs that relate to markup, as is shown in the following sections. The presumption of a tree for syntax (and hence for the structure of texts), however, is not necessary, despite its pervasiveness and seeming logic – it is only an assertion about data, not an inherent property of the data, and it is a partial description at best. Available processing technologies in the XML realm only obscure the issue. They rely on the same tree-based syntax and thus on the consequent presumption about the structure of data. Not all solutions are amenable to such descriptions. Embedding or Not Separating all the markup from the content, not just the presentational parts, has been proposed as a solution to several current markup issues. Relational pointers into the untagged data structure would then associate specified spans with the relevant metadata information. The additional overhead of maintaining parallel data structures and relational pointers between them is offset by the vastly greater ease of processing content in a more consistent and context-independent way. (A reasoned argument on this topic is ‘Embedded Markup Considered Harmful’ published by Ted Nelson in 1997 – see www.xml.com/pub/a/w3j/ s3.nelson.html.)
Defining the Semantic Web
41
Although the approach promises many advantages, it seems unlikely to be widely adopted any time soon. Interest in such a radical change of Web infrastructure has been low among content creators, so markup will continue to be embedded in the foreseeable future. We might see it as yet another example of how new technologies are constrained by previously chosen implementations, where identifiable advantages are offset by the perceived investment of globally adopting a revolutionary change. Parsing, Human versus Machine Even at the syntactical level, humans and machines parse data differently. In this context, we can speak of the ‘readability’ (or conversely, the ‘opacity’) of a document. Machine parsing inherently prefers binary material with no extraneous material or redundancy – a long opaque string of data that humans find hard or impossible to read. Such material is easy to process, using pattern-matching algorithms to extract tagged elements. Human readers prefer visual patterns, such as ample white space to group and separate semantic objects. They also prefer to have visual ‘noise’ (such as markup tags and other human-opaque linear elements) suppressed or transformed into typographical or layout conventions. Finally, humans experience a semantic dimension that enables, among other qualities, the appreciation of well-formed prose as an enhancement to easy reading. What we see in XML (and in fact the entire SGML family), when applied in isolation, is unfortunately a schizophrenic approach that concentrates on machine representation (that is, a syntactical tag system) to solve human-use issues on a case-by-case basis. In particular, the lack of a generic solution approach causes significant problems when trying to create syntax rules for machine processing. Informational Model Syntactic details aside, the Semantic Web approach must tackle the task of how information is to be represented. It is not just about improving the capability to search for information, although this goal is important enough given the pervasive use of search engines as a starting point for browsing. Just how important is search-engine technology? In only a decade the dominating method for information discovery and research in general has become user-initiated search on the Web – which is on the face of it an amazing development. It is often perceived as faster to look something up using a Web browser than to walk to the bookshelf and consult a printed reference work. It is assuredly convenient . . .when it works. Precise statistics are hard to come by (and interpret properly), but yearly surveys (such as reported at www.searchenginewatch.com) suggest that at least a tenth of all traffic to Web sites today is due to users first clicking on links from search-engine results – individual sites may report as many as three of four hits from such sources. Eight of ten first-time visits might arise from search results.
42
The Semantic Web
Users looking for products, for example, are far more likely to just plug the product name into a search engine (a surveyed third do so) and explore the resulting links, than to start by browsing shopping sites or to click on ads (less than a tenth together). On the other hand, many surveyed users follow a direct link to a site (such as a bookmark or typed-in Web address) and thus presumably already know where they are going. How did they learn of this address originally? In some cases, it is easy to surmise that casting a known brand name as a dotcom address might work; in other cases, it is not so evident. Although clearly significant, it remains highly conjectural how search-engine statistics actually correlate with the way people successfully find relevant sites to revisit (and hence generate the non-search related hits). Nevertheless, the assumption is driving a fast-growing, multi-million-dollar advertising industry devoted to favorable placement of ads on search result pages. Also, it motivates considerable efforts by webmasters to ‘optimize’ search engine ranking for their sites. Significant in this context is a 2001 survey that found that 97% of Fortune 100 companies had some type of site architecture problem that might give them problems being found by current search engines. The ‘might’ should be emphasized, however, since the most common problem related to missing or incorrect meta tags (that is, keywords to describe content) – these days the major search engines tend to ignore embedded metadata keywords in Web pages due to earlier misuse. Bit 2.10
Lexical (that is, syntactic) search characterizes the current Web
The pervasiveness and apparent usefulness of this model can easily blind us both to its inherent failings and to the potentials of alternative models. As the Web continues to expand, it becomes increasingly difficult for users to obtain relevant information efficiently. The common search strategies rarely scale well. The best results tend to come from engines that have useful ranking strategies such as backlink algorithms, or from engines that use a mix of sources, both human-classified directories and general indexing databases. All of these methods have known problems, however. The ultimate filter remains human inspection and judgement, and the effects of this extra user effort can be considerable. This last point is why result pages that provide a few lines of hit context are so useful. Yet even the best search-engines can be maddeningly inefficient in some situations and then generate mostly irrelevant results. Reasons for search engine inefficiency are detailed in Chapter 1. When search engines fail to satisfy, however, it is mainly because they are limited to reading the format tags and doing syntax analysis on a selection of published documents, sometimes not in their entirety. Despite indexing several billions of pages, such a database still represents only a small and arbitrary selection of total Web textual content – probably less than a tenth. The search situation for non-text content is far worse, though slowly improving. The greater volume of Web content thus remains inaccessible to automated remote indexing, for one or another reason. It matters little to the frustrated user whether a search is blocked due to restrictive access control, site-local archival solutions, non-standard navigation elements, or application-specific content formats (that lack Web markup).
Defining the Semantic Web
43
A new set of representational tools for metadata is required to make this immense repository of information accessible for a wide range of new applications. New interfaces are envisioned for users looking for information, and for applications that might process and compile the information for other, higher-level purposes. Requested information may not reside in any single Web location, or in a specific format, but semantic logic will enable programs to identify metadata and data elements on various sites, understand or infer relations, and piece them together reliably. Such will be the deployed infrastructure of the Semantic Web. Over time, much of the information published on the Web will tend to be packaged in metadata as a matter of course, and some resources may even lack any immediate ‘page’ location as such. Instead of searching for pages of information, the new model will involve framing queries to submit to information services, locally generating ‘content pages’ on the fly that satisfy the initial search requirements and provide further links to in-depth material. The result may be more like informal briefings or summaries. In fact, most retrieved information might be for agent consumption only and not reach the user directly at all. Bit 2.11
The implied query in searching will shift from ‘Where?’ to ‘What?’
This shift corresponds to a general shift in addressing from URL to URI. Strictly speaking, most users do not care where the information is stored, they just want useful answers and working functionality. The answer may be ‘out there’, but the user wants it ‘here’ and ready to use.
Representational Models In this section, we briefly review the main representational models that have relevance for the later discussion of the Semantic Web and related work. Of necessity, numerous technical terms are introduced here, albeit in very abbreviated contexts. Some of these topics are later discussed in detail in Chapters 5 and 6. Document Object Model The Document Object Model (DOM) is a tree-like model of objects contained in a document or Web page, defining its elements and allowing selective access to them for reading or manipulation. The abstraction of the DOM programming interface (API) provides a standardized, versatile view of a document’s contents that promotes interoperability. The DOM is an established tool for dynamically manipulating document data structures, not a data structure descriptor itself. The description is derived from the documentembedded markup. Like the SGML family of markup languages, the DOM also presupposes (or imposes) a tree-like structure in the data. However, a linear structure might not always be relevant or useful. Complementing the tree-based DOM API is an event-based Simple API for XML (SAX). It reports parsing events (such as the start and end of elements) directly to the application through callbacks, and does not usually build an internal tree at all. The application implements handlers to deal with the different events, much like handling events in a
The Semantic Web
44
graphical user interface. SAX is ideal for small-resource devices that need to locate specific elements in a single pass. As the custodian of DOM, the W3C maintains it as an open API to encourage standards compliance. See www.w3.org/D0M/Activity for an overview of its status. Early on, however, Microsoft added its own proprietary extensions to the DOM in implementations of its Internet Explorer browser. Since IE still remains the dominant desktop Web browser, this divergence creates interoperability problems for anyone creating Web content or programming applications that rely on DOM. Resource Description Framework The Resource Description Framework (RDF) is a general-purpose language for representing information in the Web. It is a common specification framework to express document metadata (that is, the meaning of terms and concepts used to annotate Web resources) in a form that computers can readily process. The purpose of RDF is to provide an encoding and interpretation mechanism so that resources can be described in a way that particular software can understand it. Bit 2.12
RDF is a core technology in the Semantic Web
In addition, most higher-level representations depend on or derive from the RDF model of expressing relationships and defining resources. RDF commonly uses XML for its syntax and URIs to specify entities, concepts, properties, and relations. The model is based upon the idea of making statements about resources in the form of a subject-predicate-object expression (called a triple). Essentially, the subject is the resource being described, the predicate is the resource’s trait or aspect that is being described, and the object is the value of that trait. This terminology is taken from logic and linguistics, where subject-predicate or subjectpredicate-object structures have very similar but clearly distinct meanings. Chapter 5 discusses RDF in greater detail. Machine-usable RDF statements tend to be very verbose by human standards due to the requirement that resource descriptions be unique. By contrast, human-readable statements to convey the same meaning are typically much shorter, but prove useless to a machine trying to parse the meaning. This difference derives from the way human readers intuitively understand and use context and implicit reference. Machine parsing needs all the relations explicitly spelled out. Schema The term schema denotes a map of concepts and their relationships. Invariably, such a map is both incomplete and simplified, often with a tight focus on a particular mapping need. The
Defining the Semantic Web
45
role of a schema as a representational model in the context of Web information is to mediate and adjudicate between human and machine semantics. Schemas based on RDF are described using RDFS, a language for defining RDF vocabularies, which specifies how to handle and label the elements. This schema view is intentionally broader than that of operational semantics. In the latter, the meaning of a token is what we want it to be for processing software to deal appropriately with the data. The former view is more likely to represents a structuralist stance in suggesting a sign relation between the signifier and the signified. As noted earlier, a common presumption is that the data structure is linear and tree-based. This constraint appears again in DTD and XML schema constructs. Nevertheless, XMLbased schemas are successfully used to model data in many database applications, and thus viewed as ‘superior’ for this purpose. XML provides the self-defining relationships; no need then to supply the data in half a dozen different formats, or to create and impose a new ‘standard’ exchange format. Regular tree grammar theory has established itself as the basis for structural specification in XML-based schemas. The other two constituents of a data model are the constraint specification, and the set of operators for accessing and manipulating the data. The field of constraint specification is still being studied, with different approaches such as ‘path-based constraint specification’ and ‘type-based constraint specification’. XML operators are available as part of XPath, XSLT, and XQuery. In Part II we examine much of the mechanics involved, mainly in Chapter 5. An alternative to constructing schemas using regular grammars is the rules-based approach. Its proponents often critique the otherwise uncritical adoption of grammarbased systems to define all schemas, sometimes retrofitting various ‘layer’ structures of pointers to handle the graph-like cases where the inherent tree-structure is inadequate. Some purists might beg to differ with the polarization often seen, especially when ‘rulesbased’ discussion segues into the claim of being a ‘semantic’ approach – to them, rule-sets are just another expression of a grammar, while semantics is about the means to choose between different rules and syntax. The pragmatic view for now is to follow current usage in this respect. However, the contrast between the two views can perhaps be illustrated by the following analogy, which is admittedly simplistic: Grammar-based schemas are similar to road maps; they try to locate precisely all mapped items on the grid. Implicit is the attempt to describe what is true for all of the data, all of the time. Getting from one point to another in such a map involves parsing the map and making decisions based on what one finds in the relationships (for instance, the topology and the available road connections). Rules-based schemas are like route descriptions; they try to express the things that make some particular information at some particular time and context different from other data of the same type. In practical terms, this operative approach is required for the data-entry and debugging phase. To summarize: the first tries to abstract away invariants, the second tries to find abstractions to express variations. In general, XML schemas and DTDs belong to the first category because they are concerned with mapping relationships.
The Semantic Web
46
Bit 2.13
Road-map and route-description schemas are largely complementary
This seeming dichotomy appears in most descriptive contexts, as classic state-oriented versus process-oriented views. Inspired design can leverage optimized use of each. The basic polarity between grammar-based and rules-based might usefully be expanded into a description of four layers to characterize schema languages, going from grammar to semantics. (If the example terms are not familiar at this time, please just ignore them for now.) Structural patterns. Processing is the binary result of matching the source against patterns expressed by a pattern language. Examples are DTD, TREX, RELAX, and the XSchema content model. Datatypes. Processing is the binary result of matching the source against datatypes expressed by a datatype language. Cited examples are XSchema datatypes and UML/XMI. Structural rules. Processing evaluates generalized expressions about structure against the source, and the results trigger actions – for example, to accept or reject the source as valid. Semantics. Processing is based on semantic rules that express meaning rather than formal structure. The rules may be constructed using RDFS, defining ontologies of meaning, such as for ‘business logic’ rules. A developer discussion relating to the last layer was once seen to make the implied criticism that our understanding of the requirements for a comprehensive semantic layer is at present still so rudimentary that a strict formalized description of it is not possible. Such a nihilistic view is contested by those who in fact work at developing semantic rules, but there is truth to the supposition that we still have much to learn.
The Road Map To explore the Semantic Web – design, implementation, and potential – we need to outline many aspects of its probable architecture, in particular what might be termed the essential components for it. In many respects, the outline describes an architectural plan untested by anything except thought experiments, more vision than substance, yet well on the way to becoming a reality. Even though many of the following concepts were introduced in Chapter 1, and are covered in far greater detail in later chapters, it is relevant here to summarize systematically the essentials as they pertain to the Semantic Web. Therefore, we adopt the viewpoint of the sweb developer looking for the general solutions that each concept might offer, rather than considering the deficiencies that are evident in current implementations. The following are then the ‘essential components’ of the Semantic Web.
Identity Handling resources on the Web requires a strong and persistent implementation of unique identity. In local contexts, looser or transient forms of identity might be acceptable, but such constructs will not work in the global Web context. The Uniform Resource Identifier is the generic solution, but it was not directly implemented in the early Web. The URI subset of Uniform Resource Locators (URL)
Defining the Semantic Web
47
was used to base global identity on an abstract ‘location’ instead – as protocol plus domain, plus local path or access method. However, the URL implementation has serious deficiencies in terms of persistency, among other things, in part due to its visible dependency on these arbitrarily and locally defined paths. No longer included in technical specifications, the now-deprecated term URL is only found in ‘informal’ usage and legacy documents. Bit 2.14
Actual location on the Web (URL) is less useful than it may seem
At best, a given URL represents a logical (and reasonably persistent) ‘location’ locally dereferenced by the resource server at the specified Internet domain. It may just reflect an actual but arbitrary (and likely to change) filesystem path for server storage. The user gains little and loses much by having the resource’s actual storage location or access method exposed in the URL. The unpredictable mobility or availability of Internet resources at specified URLs is an inconvenience at best, often resulting in cryptic error messages from the server instead of helpful redirection, but in the expanded context of the Semantic Web, explicitly exposing location becomes a critical flaw. For librarians (to take an example of a group on the front lines of implementing functionality in the direction of a semantic Web), it is a serious problem already. The deficiencies of the URL model compromise library services to patrons and impose unacceptably large burdens on catalog maintenance. Hence the desire for central repositories of uniquely named resource identifiers. The existence of a named identifier must be maintained even when the referenced resource no longer exists or is unavailable. Therefore, another URI subset, Uniform Resource Names (URN), has been under careful development by the IETF for some time, with the intent to provide persistent identities based on unique names within a defined namespace – as urn:namespace-identifier:named-resource. It is no coincidence that national libraries in many countries actively solicit the free registration of URN-URI addresses to online resources. A modification to the current DNS system, for example, would resolve current issues with changing resource URLs by providing dynamic translation from URN pointers to the actual server-relative locations (as expressed by updated URLs). Bit 2.15
Resource reference by name (URN) is better than by location (URL)
Most human-friendly locator schemes since the beginning of recorded history have been based on named references, leveraging the fundamental ability of humans to give and use names in a mapping or descriptive context. Importantly, names are portable. In practice, the URL and URN subsets do overlap somewhat. For example, a mailbox URL address for e-mail is also a URN, in that the domain-qualified target name is sufficient and
48
The Semantic Web
globally unique, and does not visibly depend on any server-local specifics such as path or method. The persistence requirement of URN schemes is more an outcome of the social structures that evolve to meet a common community need than a technological issue. Such evolution can be slow – in the case of librarians, too slow. Therefore, an interim solution was deployed: the Persistent Uniform Resource Locator (PURL) service. PURL can often be seen referenced in XML database-exchange constructions. The OCLC (Online Computer Library Center, Inc., www.oclc.org) maintains PURL Org (www.purl.org), a global resource in the form of a database of registered unique identifiers. OCLC is a non-profit, global library cooperative serving 41,000 libraries in 82 countries and territories around the world. The PURL service is in effect a URL redirector, as shown in Figure 2.3. A PURL URI associates a particular URL and XML term with a defined classification element, such as ‘http://purl.org/dc/elements/1.1/publisher’ (which, by the way, is a semantic definition). Resource references are defined using PURLs and references to the PURL server. The PURL service responds to the client request with the currently valid associated URL in its database, which the client can then use to access directly the resource. PURLs have been assigned to records cataloged in the Internet Cataloging Project, funded by the U.S. Department of Education to advance cataloging practices for Internet resources. This project represents the leading edge of MARC description of Internet resources, a mechanism by which computers exchange, use, and interpret bibliographic information. MARC data elements make up the foundation of most library catalogs used today. Bit 2.16
PURL is a pragmatic hack that implements URN-like functionality
Implemented as a transparent Web service, PURL is an example of a transitional infrastructure enhancement referenced using resource-embedded pointers. The expected syntax of URNs is clear enough to afford confidence that the syntax of transitional PURLs can be inexpensively and mechanically translated to the later URN form that is expected to be integral to the next-generation-internet infrastructure.
Figure 2.3 How PURL provides an intermediate step between the deficiency of impermanent URLs and the permanency of URNs. The PURL server acts as a redirector, dereferencing unique PURL identities into the currently valid URL
Defining the Semantic Web
49
In the meantime, the PURL model lends itself to distribution across the Net, with servers run by organizations with a commitment to maintaining persistent naming schemes (libraries, government organizations, publishers, and others).
Markup The syntactical component of the Semantic Web is the markup language that enables the distinction between the content representation and the metadata that defines how to interpret and process it. For various reasons, the current choice of markup language is XML, because among other things it fulfills the dual requirements of self-defining and extensible document descriptions. XML also provides well-trodden paths for normalizing markup and upgrading existing HTML content, possibly through transitional formats such as XHTML 1.0. Bit 2.17
XML is a fundamental building block for the Semantic Web
XML depends on URI, but in turn it is the foundation for most of the higher layers. The visible part of this component is the markup syntax, expressed as the reserved textpattern tags embedded in the document but invisible in human-use rendering. Primarily, markup language is for describing content structure. In practice, relevant parts of markup are client-interpreted and visually rendered in ways appropriate to accepted typographical and layout conventions. For non-visual rendering or other contexts, other forms of rendering would be more appropriate. When read aloud for the blind, ‘emphasis’, for example, would be a shift in inflection rather than an italic font style, and list or table items would be explicitly declared. XML is well developed, and many sources and books provide both overview and in-depth studies. The principles of XML markup are simple, as are the methods for extending the language with new markup – it is after all called ‘extensible’ for a reason. In the context of this book, it is sufficient to note the salient concepts: Reserved text-pattern ‘tags’ that define ‘containers’ in which content and metadata can be placed in a structured way. URI references to the resources that can provide information about how the XML tags used are to be interpreted. The basic syntax of XML is also found in other constructs, since it lends itself well to embedded definitions of more complex structures. An overview of relevant markup languages and protocols is found in Chapter 5.
Relationships Given identities and markup, a natural step is to attempt to identify and describe relationships between data published on the Web. Much useful and helpful functionality can be achieved by combining data from several independent sources.
50
The Semantic Web
Since most content is published independently in a variety of formats that cannot directly be parsed and ‘understood’ by agents, the sweb solution is to introduce a metadata framework in which such relationships can be expressed, and thus provide a query and conversion mechanism for content elements. Bit 2.18
Relationship expressions form the core of semantic query services
RDF and RDFS are languages well-suited to constructing relationship expressions. The simplest such relationship is the concept of equivalence, so that for example corresponding fields in different databases can be recognized as describing the same property, despite arbitrary field labels. Shared fields provide ways to answer queries that a single database cannot satisfy. Significant to the Semantic Web is the idea that such data relationships can be identified and published independently as metadata by anyone. Metadata resources enable agents to correlate data from multiple databases by identifying common fields and providing conversion rules. Actually, the application is even broader in that it enables the semantic wrapping of legacy resources and software in general. A simple representation of such a model is given in Figure 2.4. Related to this relationship layer, and implied by the concept of conversion, we also require logical rules somewhere. In the Semantic Web, such rules are assumed to be published documents themselves, linked to relevant data documents. Obviously, relationship rules can be added to the document stores of either database, but the openness of the concept frees us from a reliance on specific database owners to maintain such relationships. The relationship rules can be published as independent annotations, leveraging the ability of users to notice such relationships.
Figure 2.4 Relationships between data in independently published databases can be identified and published as a separate Web resource. Agents can thereafter use the published metadata to convert and correlate data from multiple, independent sources
Defining the Semantic Web
51
A logical layer enables a large number of important functions, for example:
Deduction of one type of document from a document of another type. Validation of a document against a set of rules of self-consistency. Authentication for allowed access and use of a document. Inference of valid relationships based on conversion and inference rules. Deduction of implicit knowledge from existing assertions. Conversion of a general query into more specific sub-queries. Resolution of a query by conversion from unknown terms into known ones.
Implementing such functionality is straightforward: a mechanism for quoting, simple predicate logic, and quantification expressions. It is the basis for creating a query language, and subsequently resolving the query into a result. RDF is the language framework chosen for this role in the Semantic Web.
Reasoning Application of logic requires some form of reasoning engine. Implementing high-level reasoning is in effect the holy grail of designing agents. This conceptual level is as yet underdeveloped in the Semantic Web – it is in crucial parts unspecified, unlike the lower protocol, language and logic layers. At this stage in sweb development, the architects are happy to defer the problem of reasoning. Currently, there is no definitive algorithm for answering questions – basically, finding proofs. Implementations rest on pragmatics. Construction of a proof is done in most applications according to some set of constrained rules; validation of a general proof is left to third parties. Thus the ‘proof’ of any query’s answer is given as the chain of assertions and reasoning rules invoked by transactions, with pointers to supporting material in other documents. RDF and derived knowledge structures have recently added well-defined model theories with standard associated proof theories. The design goal here is that practical queries be sufficiently constrained that the answers are computable – or are at least ensured to terminate in finite, reasonable time. The HTTP GET request will contain a proof that the client has a right to the response, while the response will be a proof that the response is what was asked for. A second goal is a form of adaptive forward compatibility. This interesting idea means that a reasoning engine of a given version should be able to reason its way through a descriptive document sufficiently well to be able to deduce how to read a document written in a later version of the language. A related idea is that of adaptive emulation, which means that an engine should be able to read a description of a second independent implementation (for similar functionality) well enough that it can process data produced by it.
Bit 2.19
Reasoning engines are a prerequisite for adaptive Web agents
Independent software agents with initiative need to reason about things and infer appropriate responses from context. They also need to adapt for forward compatibility.
The Semantic Web
52
In this context we become concerned with ontologies, at least in the sense that computer science has redefined the term for reasons of practicality, if only because we become concerned with vocabularies and some form of meaning. In addition, perhaps more significantly, we need a mechanism to ensure a reasonable expectation of repeatable results to the same query. The full scope of investigating ontologies is examined in Chapter 6.
Agency on the Web A diversity of algorithms exist today in search engines on the Web. Taken with the prooffinding algorithms in pre-Web logical systems, this abundance suggests that the Semantic Web will see many forms of agent evolve to provide answers to different forms of query and to perform a wide variety of tasks. In fact, it will probably be wrong to speak of ‘agent’ in the singular, isolated sense that we are used to when thinking of applications. Instead, user queries and delegated tasks seem likely to be handled by collaborative services using distributed instances of functionality all over the Web. A perceived ‘agent’ might therefore just be the local interface instance to transient swarms of collaborating services, each summoned for a particular task. This model is a multidimensional analog to the common Unix practice of ‘piping’ independent programs to process data sequentially. It may appear to be a complete application, but is only a transient combination. It seems probable that a wide range of agents will be tailored to particular contexts, to particular query scope and span constraints, and to various forms of cooperative ‘swarming’ to fulfill user requests. A veritable ecology of agent software, capable of varying degrees of reasoning and initiative, could in time become an ‘intelligence infrastructure’ (please excuse the hype) to overlay the virtual networks of data transfer that exist today. Bit 2.20
Adaptive Web agents enable interactive discovery of Web resources
First, as a rule, people are interested in the answers and results, not the detailed process of obtaining them. Second, people are interested in easy ways to assess and validate the results and the sources, but often it is enough to know an abstract metric of trust without necessarily and immediately seeing a detailed list of the actual sources and their credentials or reputations. In that context, people using the new Web would no longer be concerned about the location or format of information, or precisely which services need to be found and accessed, or for that matter if the requested information is even compiled in the required way. They would delegate the queries to their immediate agent interfaces and simply await ‘digest’ results. Alternatively, they might engage in a ‘conversation’ around the request, interactively refining precision while background processing keeps up to the changing criteria.
Semantic P2P Peer-to-peer technology is invoked implicitly when speaking of distributed collaboration and Web services. Simply providing computers on the Internet with more capabilities of processing and understanding the semantics of information will not be sufficient to bring Semantic Web technology to its full potential. Moreover, the way in which information is
Defining the Semantic Web
53
accessed on the Internet must undergo significant changes, away from the prevailing centralized publish-and-store-pages model. In part, the shift in infrastructure can be seen as a belated response to the shift in computer technology, with ever more powerful desktop PCs deployed and interconnected to provide an alternative to centralized mainframe storage and processing. We are seeing a general move towards a ‘ubiquitous computing’ infrastructure, and in consequence, ever distributed storage and retrieval methods – a complex tapestry of p2p topologies. Centralized storage and processing is unlikely to disappear completely – various contexts will either require or prefer such resources. However, even these traditional server architectures seem likely to connect increasingly to, and utilize, peer-based services to accomplish their tasks. After several pendulum swings between the extremes of centralized and distributed models, designs favoring hybrid solutions appear to be gaining acceptance in many application areas. Bit 2.21
Distributed p2p can make precise publishing location irrelevant
Just as resources will be accessed mainly by location-agnostic name (URN), publishing may often be to a named service rather than any specific server location. The key shift is that sweb resources, both data storage and processing, seem likely to be primarily distributed in adaptive, virtual networks. Who really cares where the data are stored and processor cycles are consumed, as long as it is all secure enough and available on demand? There are many advantages to encrypted and redundant storage across multiple machines on a global network, and such storage could easily become the norm for most cases. Prototype and commercial systems already exist on intranets and in the wild, and more are under development. As an example, the OceanStore Project (at oceanstore.cs.berkeley.edu), is a wellestablished distributed storage solution. It is intended to scale to billions of users around the world using an infrastructure of untrusted server resources. Technically, it is called solving distributed (and guaranteed) object location in a dynamic network. A related project is Tapestry, an overlay location and routing infrastructure that provides location-independent routing of messages directly to the closest copy of an object or service. It uses only point-topoint links, without centralized resources, and is self-organizing, fault-resilient, and loadbalancing. OceanStore looks to be a viable model for providing one aspect of infrastructure for the Semantic Web, that of location-agnostic ubiquitous storage and processing. The interesting part of the prototyping is that anyone can download working software and set up miniclusters on selected machines on the LAN or across the Internet. This process can form the islands that over time can merge into a complete global infrastructure, leveraging existing machine and network resources.
The Visual Tour Putting all the preceding into a coherent picture may not be the easiest task, for we often venture into terra incognito – uncharted regions about which we can only speculate, since neither the finished applications, nor sometimes even the data models they are to process, exist.
54
The Semantic Web
Figure 2.5 A ‘road map’ of sorts, based on a more detailed W3C diagram, which although dated, still reflects the relationships between different Semantic Web components examined in this book. See text for a discussion on how to read the arrows
Still, even the barest outline contours on the map serve a purpose by orienting the viewer in the relationships between prototypes, planned components, and vision. Not everyone is comfortable with simple graphs, but the subsequent discussions through this book might be augmented by the visual relationships they imply.
The Map Figure 2.5 illustrates the lay of the land, as it were, by describing some of the relationships between now-identifiable components in the vision of the Semantic Web. The diagram is based on a larger, more detailed road map prepared by the W3C some years ago, and describes the relationships as Why?/How? links. In other words, following any arrow answers ‘How do we realize the previous component?’ while going in the reverse direction answers ‘Why do we need the preceding component?’. Some of these components have been built (solid outline), at least as functional prototypes, others were planned (dashed outline). Still others remain preliminary or conjectural (no outline) until it is seen how such a system will work in practice. Some components, such as ‘trust’, require not just technological solutions, but also critical political decisions and social acceptance. Although this diagram may seem unusual due to the combined relationships, depending on which direction you read the arrows, and perhaps ‘backwards’ to some in its overall sense, it is still useful in seeing dependency relationships. You can read the ‘flow’ as in the following example, starting in an arbitrarily chosen node: To realize Agents (follow arrows emanating from this node), we need to implement Query Services, Trust mechanisms, and Logic systems. Why we need Agents (in this view, backtrack the arrow leading to this node) is to reach the goal of the Semantic Web. You may wish to refer back to this diagram from later discussions of specific components just to refresh their particular placement in the overall scheme of things. Variations and extensions of this diagram also appear later (in Chapter 5) to place other components in this same, goal-oriented scheme. Figure 2.6 shows an alternative ‘stacking’ view of layer dependencies, one of several variants popularized by Tim Berners-Lee (see www.w3.org/DesignIssues/diagrams/sw-stack-
Defining the Semantic Web
55
Figure 2.6 An alternative view of dependency relationships, as depicted in the ‘stack layer’ model popularized by Tim Berners-Lee
2002.png). It intends to show how certain implementation areas build on the results of others but, like most simple diagrams of this nature, it is only indicative of concept, not formally descriptive.
The Architectural Goals It is useful to summarize the goals introduced in the earlier text:
Identity, by which we understand URIs (not locator URLs). Data and Structuring, as represented by Unicode and XML. Metadata and Relationships, as defined in RDF. Vocabularies, as expressed in RDF Schema. Semantic Structure, as represented in Web Ontologies. Rules, Logics, Inferencing, and Proof, which to a great extent still remain to be designed and implemented to enable agency. Trust, as implemented through digital systems and webs of trust, and also requiring further development and acceptance. Now in the first years of the 21st century, we begin to see a clearer contour of what is to come, the implementation of components that recently were merely dashed outlines of conjectured functionality on the conceptual chart. Mindful of the rapid pace of sweb development in 2003 and 2004, the W3C issued a new recommendation document in late 2004, the first of several planned, that specifies more clearly the prerequisites and directions for continued development and deployment of swebrelated technologies: ‘Architecture of the World Wide Web’ (Vol 1, December 2004, w3.org/ TR/webarch/ ). The recommendation adopts the view that the Web builds on three fundamentals that Web Agents (which includes both human users and delegated software in the form of user agents) must deal with in a compliant way: Identification, which means URI addressing of all resources. Interaction, which means passing messages framed in standard syntax and semantics over standard protocols between different agents.
The Semantic Web
56
Table 2.1. W3C Principles and Good Practice recommendations for the Web Design Aspect
W3C Principle
W3C Good Practices
Constraints
Global Identifiers
Global naming leads to global network effects.
Identify with URIs. Avoid URI aliasing. Consistent URI usage. Reuse URI schemes. Make URIs opaque. Reuse representation formats. (New protocols created for the Web should transmit representations as octet streams typed by Internet media types.) Representation creators should be allowed to control metadata association. (Resource state must be preserved for URIs published as simple hypertext links.) A URI owner should provide consistent representations of the resource it identifies. URI persistence. A data format specification should provide for version information. An XML format specification should include information about change policies for XML namespaces. A specification should provide mechanisms that allow any party to create extensions. Also, specify agent behavior for unrecognized extensions. Separation of content, presentation, interaction. Link identification. Web-wide linking. Generic URIs. Hypertext links where expected. Namespace adoption. Namespace documents. QNames must be mapped to URIs. XML content should not be assigned Internet media type ‘text’, nor specify character encoding.
Assignment: URIs uniquely identify a single resource. (Agents: consistent reference.) (Transparency for agents that do not understand.)
Formats
Metadata
Interaction
Safe retrieval.
Representation
Reference does not imply dereference.
Versioning Namespace
Extensibility
Composition Hypertext data
XML-based data
XML Media type
Specifications Exceptions
Orthogonal specifications Error recovery (Consent may be by policy rules, based on informed not requiring interactive human user consent. interruption for correction.)
Agents must not ignore message metadata without the consent of the user. (Agents must not incur obligations by retrieving a representation.)
(Useful agent directives: ‘must ignore’ and ‘must understand’ unrecognized content.) (Usability issues.)
QNames are indistinguishable from URIs. (Reasons of correct agent interpretation.)
Defining the Semantic Web
57
Formats, which defines standard protocols used for representation retrieval or submittal of data and metadata, and which convey them between agents. The document goes on to highlight a number of Principles and Good Practices, often motivated by experiences gained from problems with previous standards. Table 2.1 summarizes these items. This table may seem terse and not fully populated, but it reflects the fact that W3C specifications are conservative by nature and attempt to regulate as little as possible. Items in parenthesis are expanded interpretations or comments included here for the purpose of this summary only. The recommendation document goes into descriptive detail on most of these issues, including examples explaining correct usage and some common incorrect ones. It motivates clearly why the principles and guidelines were formulated in a way that can benefit even Web content authors and publishers who would not normally read technical specifications. As with this book, the aim is to promote a better overall understanding of core functionality in order that technology implementers and content publishers achieve compliance with both current and future Web standards.
The Implementation Levels Two distinct implementation levels are discernible when examining proposed sweb technology, not necessarily evident from the previous concept maps: ‘Deep Semantic Web’ aims to implement intelligent agents capable of performing inference. It is a long-term goal, and presupposes forms of distributed AI that have not yet been solved. ‘Shallow Semantic Web’ does not aspire as high, instead maintaining focus on the practicalities of using sweb and KR techniques for searching and integrating available data. These more short-term goals are practical with existing and near-term technology. It is mainly in the latter category we see practical work and a certain amount of industry adoption. The following chapters examine the major functionality areas of the Semantic Web.
3 Web Information Management Part of the Semantic Web deals necessarily with strategies for information management on the Web. This management includes both creating appropriate structures of data-metadata and updating the resulting (usually distributed) databases when changes occur. Dr. Karl-Erik Sveiby, to whom the origin of the term ‘Knowledge Management’ is attributable, once likened knowledge databases to wellsprings of water. The visible surface in the metaphor represents the explicit knowledge, the constantly renewing pool beneath is the tacit. The real value of a wellspring lies in its dynamic renewal rate, not in its static reservoir capacity. It is not enough just to set up a database of knowledge to ensure its worth – you must also ensure that the database is continually updated with fresh information, and properly managed. This view of information management is applicable equally to the Web, perhaps the largest ‘database’ of knowledge yet constructed. One of the goals of the Semantic Web is to make the knowledge represented therein to be at least as accessible as a formal database, but more importantly, accessible in a meaningful way to software agents. This accessibility depends critically on the envisioned metadata infrastructure and the associated metadata processing capabilities. With accessibility also comes the issue of readability. The Semantic Web addresses this by promoting shared standards and interoperable mapping so that all manner of readers and applications can make sense of ‘the database’ on the Web. Finally, information management assumes bidirectional flows, blurring the line between server and client. We see instead an emerging importance of ‘negotiation between peers’ where although erstwhile servers may update clients, browsing clients may also update servers – perhaps with new links to moved resources.
Chapter 3 at a Glance This chapter deals with the general issues around a particular implementation area of the Semantic Web, that of creating and managing the content and metadata structures that form
The Semantic Web: Crafting Infrastructure for Agency Bo Leuf # 2006 John Wiley & Sons, Ltd
The Semantic Web
60
its underpinnings. The Personal View examines some of the ways that the Semantic Web might affect how the individual accesses and manages information on the Web. Creating and Using Content examines the way the Semantic Web affects the different functional aspects of creating and publishing on the Web. Authoring outlines the new requirements, for example, on editing tools. Publishing notes that the act of publishing will increasingly be an integral and indistinguishable part of the authoring process. Exporting Databases discusses the complement to authoring of making existing databases accessible online. Distribution considers the shift from clearly localized sources of published data to a distributed and cached model of availability based on what the data are rather than where they are. Searching and Sifting the Data looks at the strategies for finding data and deriving useful compilations and metadata profiles from them. Semantic Web Services examines the various approaches to implementing distributed services on the Web. Security and Trust Issues are directly involved when discussing management, relating to both access and trustworthiness. XML Security outlines the new XML-compliant infrastructure being developed to implement a common framework for Web security. Trust examines in detail the concept of trust and the different authentication models that ultimately define the identity to be trusted and the authority conferred. A Commercial or Free Web notes that although much on the Web is free, and should remain that way, commercial interests must not be neglected.
The Personal View Managing information on the Web will for many increasingly mean managing personal information on the Web, so it seems appropriate in this context to provide some practical examples of what the Semantic Web can do for the individual. Bit 3.1
Revolutionary change can start in the personal details
Early adoption of managing personal information using automation is one sneak preview of what the Semantic Web can mean to the individual. The traditional role of computing and personal information management is often associated with intense frustration, because the potential is evident even to the newcomer. From the professional’s viewpoint, Dan Connolly coined what became known as Connolly’s Bane (in ‘The XML Revolution’, October 1998, in Nature’s Web Matters, see www.nature. com/nature/webmatters/xml/xml.html): The bane of my existence is doing things I know the computer could do for me.
Web Information Management
61
Dan Connolly serves with the W3C on the Technical Architecture Group and the Web Ontology Working Group, and also on Semantic Web Development, so he is clearly not a newcomer – yet even he was often frustrated with the way human–computer interaction works. An expanded version of his lament was formulated in a 2002 talk: The bane of my existence is doing things I know the computer could do for me . . . and getting it wrong!
These commentaries reflect the goal of having computer applications communicate with each other, with minimal or no human intervention. XML was a first step to achieving it at a syntactic level, RDF a first step to achieving it at a semantic level. Dan’s contribution to a clearer perception of Semantic Web potential for personal information management (PIM) has been to demonstrate what the technology can do for him, personally. For example, Dan travels a lot. He received his proposed itineraries in traditional formats: paper or e-mail. Eventually, he just could not bear the thought of yet again manually copying and pasting each field from the itinerary into his PDA calendar. The process simply had to be automated, and the sweb approach to application integration promised a solution. The application-integration approach emphasizes data about real-world things like people, places, and events, rather than just abstract XML-based document structure. This sort of tangible information is precisely what can interest most users on the personal level. Real-world data are increasingly available as (or at least convertible to) XML structures. However, most XML schemas are too constrained syntactically, yet not constrained enough semantically, to accomplish the envisioned integration tasks. Many of the common integration tasks Dan wanted to automate were simple enough in principle, but available PIM tools could not perform them without extensive human intervention – and tedious manual entry. A list of typical tasks in this category follows. For each, Dan developed automated processes using the sweb approach, usually based on existing Web-published XML/RDF structures that can serve the necessary data. The published data are leveraged using envisioned Web-infrastructure technologies and open Web standards, intended to be as accessible as browsing a Web page. Plot an itinerary on a map. Airport latitude and longitude data are published in the Semantic Web, thanks to the DAML project (www.daml.org), and can therefore be accessed with a rules system. Other positioning databases are also available. The issue is mainly one of coding conversions of plain-text itinerary dumps from travel agencies. Applications such as Xplanet (xplanet.sourceforge.net) or OpenMap (openmap.bbn.com) can visualize the generated location and path datasets on a projected globe or map. Google Maps (maps.google.com) is a new (2005) Web solution, also able to serve up recent satellite images of the real-world locations. Import travel itineraries into a desktop PIM (based on iCalendar format). Given structured data, the requirements are a ruleset and iCalendar model expressed in RDF to handle conversion.
The Semantic Web
62
Import travel itineraries into a PDA calendar. Again, it is mainly a question of conversion based on an appropriate data model in RDF. Produce a brief summary of an itinerary suitable for distribution as plain text e-mail. In many cases, distributing (excerpts of ) an itinerary to interested parties is still best done in plain text format, at least until it can be assumed that the recipients also have swebcapable agents. Check proposed work travel itineraries against family constraints. This aspect involves formulating rules that check against both explicit constraints input by the user, and implicit ones based on events entered into the user’s PDA or desktop PIM. Also implicit in PDA/PIM handling is the requirement to coordinate and synchronize across several distributed instances (at home, at work and mobile) for each family member. Notify when the travel schedule might intersect or come close to the current location of a friend or colleague. This aspect involves extended coordination and interfacing with published calendar information for people on the user’s track-location list. An extension might be to propose automatically suitable meeting timeslots. Find conflicts between teleconferences and flights. This aspect is a special case of a generalized constraints analysis. Produce animated views of travel schedule or past trips. Information views from current and previous trips can be pulled from archive to process for display in various ways. A more technical discussion with example code is found at the Semantic Web Applications site (www.w3.org/2000/10/swap/pim/travel). Most people try to juggle the analysis parts of corresponding lists in their heads, with predictably fallible results. The benefits of even modest implementations of partial goals is considerable, and the effort dovetails nicely with other efforts (with more corporate focus) to harmonize methods of managing free-busy scheduling and automatic updating from Web-published events. Bit 3.2
Good tools let the users concentrate on what they do best
Many tasks that are tedious, time-consuming, or overly complex for a human remain too complex or open-ended for machines that cannot reason around the embedded meaning of the information being handled. The availability of this kind of automated tool naturally provides the potential for even more application tasks, formulated as ad hoc rules added to the system. For example: ‘If an event is in this schedule scraped from an HTML page (at this URL), but not in my calendar (at this URI), generate a file (in ics format) for PIM import’. The next step in automatic integration comes when the user does not need to formulate explicitly all the rules; that is, when the system (the agent) observes and learns from previous actions and can take the initiative to collect and generate proposed items for user review based on current interests and itineraries.
Web Information Management
63
For example, the agent could propose suitable restaurants or excursions at suitable locations and times during a trip, or even suggest itineraries based on events published on the Web that the user has not yet observed.
Creating and Using Content In the ‘original release’ version of the Web (‘Web 1.0’), a platform was provided that enabled a convenient means to author, self-publish, and share content online with the world. It empowered the end-user in a very direct way, though it would take a few iterations of the toolsets to make such authoring truly convenient to the casual user. In terms of ease-of-use, we have come a long way since the first version, not least in the system’s global reach, and practically everyone seems to be self-publishing content these days on static sites, forums, blogs, wikis, etc. (call it ‘Web 2.0’). But almost all of this Web page material is ‘lexical’ or ‘visual’ content – that is, machine-opaque text and graphics, for human consumption only. Creating Web content in the context of the Semantic Web (the next version, ‘Web 3.0’) demands more than simply putting up text (or other content) and ensuring that all the hyperlink references are valid. A whole new range of metadata and markup issues come to the fore, which we hope will be adequately dealt with by the new generation of tools that are developed. Bit 3.3
Content must be formally described in the Semantic Web
The degree of possible automation in the process of creating metadata is not yet known. Perhaps intelligent enough software can provide a ‘first draft’ of metadata that only needs to be tweaked, but some user input seems inevitable. Perhaps content authors just need to become more aware of metadata issues. Regardless of the capabilities of the available tools, authors must still inspect and specify metadata when creating or modifying content. However, current experience with metadata contexts suggests that users either forget to enter it (for example, in stand-alone tools), or are unwilling to deal with the level of detail the current tools require (in other words, not enough automation). The problem might be resolved by a combination of changes: better tools and user interfaces for metadata management, and perhaps a new generation of users who are more comfortable with metadata. A possible analogy in the tool environment might be the development of word-processing and paper publishing tools. Unlike the early beginnings when everything was written as plain text, and layout was an entirely different process relegated to the ranks of professional typesetters and specialized tools, we now have integrated authoring-publishing software that supports production of ready-to-print content. Such integration has many benefits to be sure. However, a significant downside is that the content author rarely has the knowledge (or even inclination) to deal adequately with this level of layout and typographical control. On the other hand, the issue of author-added metadata was perhaps of more concern in the early visions. Such content, while pervasive and highly visible on the Web, is only part of the current content published online.
The Semantic Web
64
Many existing applications have quietly become RDF-compliant, and together with online relational databases they might become the largest sources of sweb-compliant data – we are speaking of calendars, event streams, financial and geographical databases, news archives, and so on. Bit 3.4
Raw online data comes increasingly pre-packaged in RDF
The traditional, human-authored Web page as a source of online data (facts as opposed to expressed ‘opinion’) might even become marginalized. Aggregator and harvester tools increasingly rely on online databases and summaries that do not target human readers. Will we get a two-tier Web with diverging mainly-human-readable and mainly-machinereadable content? Possibly. If so, it is likely to be in a similar way to how we now have a two-tier Web in terms of markup: legacy-HTML and XML-compliant. Whether this becomes a problem or not depends – all things considered, we seem to be coping fairly well with the markup divide.
Authoring As with earlier pure-HTML authoring, only a few dedicated enthusiasts would even presume to author anything but the actual visible content without appropriate tool sets to handle the complexities of markup and metadata. Chapter 8 explores the current range of available tools, but it is clear that much development work remains to be done in the area. Minimum support for authoring tools would seem to include the following: XHTML/XML markup support in the basic rendering sense, and also the capability to import and transform existing and legacy document formats. Web annotation support, both privately stored and publicly shared. Metadata creation or extraction, and its management, at a level where the end-user can ‘easily’ do it. Integration of numerous types of information into a consistent view – or rather, into several optional views. Several of these criteria are addressed in at least the proof-of-concept stage by the W3C Amaya combined editor/browser client (see www.w3.org/Amaya/). The ideal Web 3.0 tool thus looks back at the original Web browser design, which was to have been both browsing and publishing tool. It adds, however, further functionality in a sensible way to provide users with the means to manage most of their online information management needs. In short, the user interface must be capable of visualizing a broad spectrum of now disparate local and online information items and views. Bit 3.5
The separation of browsing and authoring tools was unfortunate
In the Semantic Web, we may expect to see a reintegration of these two functions in generalized content-management tools, presenting consistent GUIs when relevant.
Web Information Management
65
The concept of browse or author, yet using the same tool or interface in different modes, should not be that strange. Accustomed as we are to a plethora of distinct dedicated ‘viewers’ and ‘editors’ for each format and media type, we do not see how awkward it really is. Tools can be both multimedia and multimode. This assertion should not be interpreted as sanctioning ‘monolithic bloatware, everything to everyone’ software packages – quite the opposite. In the Semantic Web, any application can be browser or publisher. The point is that the user should not have to deal explicitly with a multitude of different interfaces and separate applications to manage Web information. The functional components to accomplish viewing and editing might be different, but they should appear as seamless mode transitions. Working with Information One of the goals of the Semantic Web is that the user should be able to work with information rather than with programs. Such a consistent view becomes vital in the context of large volumes of complex and abstract metadata. In order to realize the full vision of user convenience and generation of simple views, several now-manual (usually expert-entry) processes must become automatic or semiautomatic: creating semantic annotations; linking Web pages to ontologies; creating, evolving, and interrelating ontologies. In addition, authoring in the Semantic Web context presumes and enables a far greater level of collaboration. Chapter 4 specifically explores collaboration space in this context.
Publishing Some people might debate whether creation and publishing are really very separate processes any more. Even in the paper-publishing world, it is more and more common for the content author also to have significant (or sometimes complete) control of the entire process up to the final ready-to-print files. The creation/publishing roles can easily merge in electronic media, for example, when the author generates the final PDF or Web format files on the distribution server. In the traditional content-creation view for electronic media, any perceived separation of the act of publishing is ultimately derived from the distinction between interim content products on the local file system, accessible only to a relative few, and the finished files on a public server, accessible to all. This distinction fades already in collaborative contexts over the existing Web. Now consider the situation where the hard boundary between user system and the Web vanishes – or at least becomes arbitrary from the perspective of where referenced information resides in a constantly connected, distributed-storage system. Interim, unfinished content may be just as public as any formally ‘published’ document, depending only on the status of applied access control.
The Semantic Web
66
Web logs are just one very common example of this blurring. Open collaboration systems, such as Wiki, also demonstrate the new content view, where there is no longer any particular process, such as a discrete FTP-upload, that defines and distinguishes ‘published’ content from draft copy. Bit 3.6
The base infrastructure of the Semantic Web supports versioning
With versioning support in place, managing interim versions of documents and collaboration processes online becomes fairly straightforward.
Exporting Databases Increasingly important, and a complement to explicit authoring, are technologies for exporting existing database content to the Semantic Web. In fact, a significant frustration in the business and consumer world, and consequent disenchantment with Web technology in general, has been the lack of direct access to business databases. Customers must often manually browse to another company’s site, drill down to the sought-for published document, and in worst case, parse a downloaded and hard-to-read PDF or MS Word document for the required information. A frozen PDF rendition of a database selection of records might in addition be critically out of date. Bit 3.7
Online business requires online database access
It is that simple, and any other offline solution, including Web-published static extracts (HTML tables or PDF) simply will not do. That more companies on the Web do not understand this issue can only be attributed to the ‘printed brochure’ mentality. Data export, or rather its online access, is easiest if the database already is RDF compliant, because then the problem reduces to providing a suitable online interface for the chosen protocol. Then customers can access realtime data straight from the company database.
Distribution In the traditional Web, it would seem that content is published to one specific location: a Web server on a specified machine. This content-storage model, however, is a serious oversimplification. Distribution was an issue even in the early Internet, and for example USENET is based on the idea of massive replication across many connected servers, motivated by the fact that overall bandwidth is saved when users connect to their nearest news server instead of to a single source. For reasons of reliable redundancy and lessened server loading, mirror sites also became common for file repositories and certain kinds of Web content. Such a visible mirror-server model requires a user to choose explicitly between different URLs to reach the same resource. However, transparent solutions for distributed storage and request fulfilment that reference a single URL are much more common than the surfing user might imagine.
Web Information Management
67
Content Caching or Edge Serving In particular, the growth of very high-demand content, such as provided by search engines, large corporations and institutions, and by popular targets such as major news sites, Amazon.com and eBay, all rely on transparent, massively distributed content caching. Behind the scenes, requests from different users for content from the same stem URL are actually fulfilled by distributed server clusters ‘nearest’ their respective locations – an approach often termed ‘edge’ serving. An entire infrastructure fielded by specialized companies grew up to serve the market, such as Akamai (www.akamai.com) that early on made a place for itself on the market by deploying large-scale solutions for aggressively distributed and adaptive storage. Distribution is also about functionality, not just serving content, and about having clusters or swarms cooperatively implement it. In the Semantic Web, we might be hard pressed to separate cleanly what is ‘content’ and what is ‘functionality’. The rendered result of a user request can be a seamless composite constructed by agents from many different kinds of Web resources. Storage can be massively distributed in fragmented and encrypted form as well, so that it is impossible to locate a particular document to any individual physical server. In such a case, user publishing occurs transparently through the client to the network as a whole instead of a given server – or rather to a distributed publishing service that allocates distributed network storage for each document. Retrieval is indirect through locator services, using unique keys generated from the content. Retrieval can then be very efficient in terms of bandwidth usage due to parallel serving of fragments by swarms of redundant hosts.
Syndication Solutions An area related to publishing and distribution is known as syndication, which in one sense, and as commonly implemented, can be seen as a user-peer form of mirroring. Syndication of content, however, differs from mirroring of content in the way the name (borrowed from news media and broadcast television) implies; it distributes only subscriber-selected content from a site, not entire subwebs or repositories. Syndication is currently popular in peer-to-peer journalism (p2pj) and for tracking new content in subscribed Web logs (commonly called ‘blogs’). Web syndication technology is currently based on two parallel standards, both with the same RSS acronym due to historical reasons of fragmented and misconstrued implementations of the original concept: Really Simple Syndication (RSS 0.9x, recently re-engineered as RSS 2.x to provide namespace-based extensions) is based on XML 1.0. It is now used primarily by news sites and Web logs to distribute headline summaries and unabridged raw content to subscribers. This standard and its development are described at Userland (see backend.userland.com/ rssO91 and blogs.law.harvard.edu/tech/rss). RDF Site Summary (known as RSS 1.x) is a module-extensible standard based on RDF, in part intended to recapture the original semantic summary aspects. It includes an RDF Schema, which is interesting for Semantic Web applications. The home of this standard is found at Resource.org (see web.resource.org/rss/1.0/for details). Official modules include Dublin Core, Syndication and Content. Many more are proposed.
68
The Semantic Web
Somewhat conflicting perspectives exist on RSS history (see rsswhys.weblogger.com/ RSSHistory). In any case, we find in all about nine variants of RSS in the field, adhering in greater or lesser degree to one of these two development forks. The name issue is especially confusing because RSS 0.9 was originally called RDF Site Summary when it was deployed in 1999 by Netscape (the company). It then provided the capability for ‘snapshot’ site summaries on the Netscape portal site for members wishing to advertise their Web site. Although conceived with RDF metadata summary in mind, the actual Netscape implementation dropped this aspect. Other developers and RSS portal sites later spread the concept, leveraging the fact that RSS could be used as an XML-based lightweight syndication format for headlines. Some advocates redefine RSS as ‘just a name’ to avoid the contentious acronym interpretation issue altogether. Others note the important distinction that the two parallel standards represent what they would like to call ‘RSS-Classic’ and ‘RSS 1.0 development’ paths, respectively: MCF (Meta Content Framework) ! CDF (Channel Definition Format) ! scriptingNews ! RSS 0.91 ! RSS-Classic. MCF ! XML-MCF ! RSS 0.90 ! RSS 1.0. Be that as it may. The formats can to some extent be converted into each other. In any case, a plethora of terms and new applications sprang from the deployment of RSS: Headline syndication, as mentioned, carries an array of different content types in channels or feeds: news headlines, discussion forums, software announcements, and various bits of proprietary data. Registries provide classified and sorted listings of different RSS channels to make user selection easier. Aggregators decouple individual items from their sites and can present one-source archives of older versions of content from the same site, and in some cases with Scrapers, advanced searching and filtering options. Scrapers are services that collect, ‘clean’, and repackage content from various sources into their own RSS channels. Synthesizers are similar to Scrapers, except that the source channels are just remixed into new ones, perhaps as a user-configured selection. Somewhere along the way, the ‘summary’ aspect was largely lost. Even as RSS, for a while, became known as Rich Site Summary (v0.91), it was morphing into full-content syndication. In this form, it became widely adopted by Web-log creators (and Web-log software programmers). The blog community saw RSS in this non-semantic, non-summary sense as a way for users to subscribe to ‘the most recent’ postings in their entirely, with the added convenience of aggregating new postings from many different sources. However, nobody knows how much of this subscription traffic is in fact consumed actively. Many users with unmetered broadband are in fact ‘passive’ RSS consumers; they turn on many feed channels with frequent updates (default settings) but hardly ever read any of
Web Information Management
69
the syndicated content piped into their clients. It is just too time-consuming and difficult to overview, yet the user is loath to disconnect for fear of missing something. Bit 3.8 Using RSS to syndicate entire content is a needless threat to Internet bandwidth In 2004, RSS-generated traffic was getting out of hand, effectively becoming a form of network denial-of-service load from some sources. In particular, many clients are poorly implemented and configured, hitting servers frequently and needlessly for material that has not been updated. Adoption of the RDF version for syndicating metadata (RSS 1.0), on the other hand, is an important asset for Semantic Web publishing since it easily enables agent processing. Additionally, it requires far less bandwidth. This kind of publishing and distribution of metadata, with pointers to traditionally stored content, is arguably more important and useful than distributing the actual content in its current format. Syndicated metadata might yet become an important resource to find and ‘mine’ published content in the future Web. At present, however, much RSS traffic is less useful.
Searching and Sifting the Data ‘Mining the Web’ for information is by now an established term, although it might mean different things to different actors. However, we are likely to think of simply harvesting ‘published data’ as-is, without really reflecting on how to compile the results. Pattern matching and lexical classification methods similar to current search engines might suffice for simple needs, but the current lack of machine-readable ‘meaning’ severely limits how automated or advanced such literal filtering processes can be. Nevertheless, much current data mining is still based on simple lexical analysis, or possibly search engine output. A simple example is the way e-mail addresses are harvested from Web sites simply by identifying their distinctive syntactical pattern. The ever-mounting deluge of junk e-mail is ample testament to the method’s use – and unfortunate effectiveness. To accomplish more than just sorting key-value associations based on discrete words and phrases requires a different approach, one that the enhanced infrastructure of the Semantic Web would provide. Perhaps then too a modicum of control may limit abusive harvesting. Usage Mining Broader application of Web mining deals not only with Web data as such, nor even with the metadata relationships, but rather with actual usage. The field of Web Usage Mining (WUM) has attracts significant commercial interest. Although commercial ventures probably account for most innovation and actual deployments in the field these days, their proprietary solutions are rarely published openly. Generally speaking, the visible products (deliverables) in WUM are the finished analysis results and the services to produce them – for example, the Nielsen/NetRatings real-time research and analysis products about Internet users (see www.nielsen-netratings.com). Lists
The Semantic Web
70
of available mining tools and resources are otherwise available at the Knowledge Discovery Nuggets Directory at www.kdnuggets.com – a searchable portal site. The WebSIFT project (Web Site Information Filter, formally the WEBMINER project, see www.cs.umn.edu/Research/websift/) defines open algorithms and builds tools for WUM to provide insights into how visitors use Web sites. Other open sources for WUM theory and relevant algorithms are found in the papers presented at the annual IEEE International Conference on Data Mining (ICDM, see www.cs.uvm.edu/~xwu/icdm.html). WUM processes the information provided by server logs, site hyperlink structures, and the content of the Web site pages. Usage results are often used to support site modification decisions concerning content and structure changes. As an overview, WUM efforts relate to one or more of five application areas: Usage characterization has its focus on studying browser usage, interface-design usability, and site navigation strategies. Application might involve developing techniques to predict user behavior in interactions with the Web and thus prepare appropriate responses. Personalization means tracking and applying user-specified preferences (or assumed contexts) to personalize the ‘Web experience’. It is often seen as the ultimate aim of Webbased applications – and used to tailor products and services, to adapt presentations, or to recommend user actions in given contexts. System improvement means to optimize dynamically server (or network) performance or QoS better to meet actual usage requirements. Application includes tweaking policies on caching, network transmission, load balancing, or data distribution. The analysis might also detect intrusion, fraud, and attempted system break-ins. Site modification means to adapt an individual site’s actual structure and behavior to better fit current usage patterns. Application might make popular pages more accessible, highlight interesting links, connect related pages, and cluster similar documents. Business intelligence includes enterprise Web-log analysis of site and sales related customer usage. For example, IBM offers a number of WUM-based suites as services to other corporations. Most WUM applications use server-side data, and thus produce data specific to a particular site. In cases where multi-site data are desired, some form of client or proxy component is required. Such a component can be as simple as a persistent browser cookie with a unique ID, and having participating sites use page elements to reference a common tracking server that can read it. This profiling method is commonly used with banner ads by advertisers. By contrast, browser toolbars can actively collect profiled usage data from multiple sites directly from the client – for example, feedback to a search engine page ranking function. Many Web users are unaware of browser-deployed WUM strategies such as these. Undisclosed profiling is considered privacy invasion, but it is widely practiced. Bit 3.9
WUM technologies have relevance to adaptive sweb agents
The main difference to current WUM is how the data are used. In sweb agents, most profiling data might never perculate beyond a small circle of agents near the user.
Web Information Management
71
Technologies based on usage mining might come into play when agent software in the Semantic Web is to adapt automatically to shifting user and device contexts. In addition to explicit user-preference data, agent-centric WUM components might mine an assortment of published data and metadata resources to determine suitable configurations and to match offered device capabilities to the tasks at hand. It is reasonable to suppose that profile disclosure to third parties be both user configurable and otherwise anonymized at the agent-Web interface.
Semantic Web Services Query services, Web annotation and other online toolset environments provide simple forms of Web Services (WS), a catch-all term that has received extra prominence in the .NET and Sun ONE era of distributed resources. The development of deployable WS is driven largely by commercial interests, with major players including Microsoft, IBM and BEA, in order to implement simple, firewall-friendly, industry-accepted ways to achieve interoperable services across platforms. WS advocacy should be seen in the context of earlier solutions to distributed services, such as CORBA or DCOM, which were neither easy to implement nor gained broad industry support. The perception has also been that commercial WS development existed in opposition to the more academic Semantic Web (SW). WS developers drew criticism for not respecting the Web’s existing architecture, while the SW project was critiqued for its overly visionary approach, not easily adaptable to current application needs. However, contrary to this perception of opposition, WS and SW are inherently complementary efforts with different agendas, and as such can benefit each other: Web Services can be characterized as concerned with program integration, across application and organizational boundaries. Traditionally, WS handles long-running transactions and often inherits earlier RPC design. The Semantic Web is primarily concerned with future data integration, adding semantic structure, shareable and extensible rule sets, and trust in untrusted networks. It builds on URI relationships between objects, as uniform yet evolvable RDF models of the real world. For example, WS discovery mechanisms are ideally placed to be implemented using sweb technology. A WS-implemented sweb protocol (SOAP) may prove ideal for RDF transfer, remote RDF query and update, and interaction between sweb business rules engines. Therefore, to gain focus we can at least identify a joint domain of Semantic Web Services (SWS). The tersest definition of modern WS is probably ‘a method call that uses XML’, and it is not a bad one, as far as it goes. The W3C defines WS as ‘a software application defined by a URI, whose interfaces and bindings are capable of being defined, described and discovered as XML artifacts’ (as illustrated in Figure 3.1). The current state of support suggests that SWS is an excellent applications integration technology (despite the way many weary IT professionals might cringe at the term ‘integration’), but that it may not yet be ready for general use – important layers in the SWS model are not yet standard, nor available as ready-to-use products.
72
The Semantic Web
Figure 3.1 Web services can be seen as a triangular relationship of publishing, finding, and interacting with a remote service on the Web
The W3C Web Services Workshop, led by IBM and Microsoft, agreed that the WS architecture stack consists of three stack components: Wire is based on the basic protocol layers, XML overlaid by SOAP/XML, with extension layers as required. Description is based on XML Schema, and is overlain with the layers Service Description, Service Capabilities Configuration, Message Sequencing, and Business Process Orchestration. Discovery involves the use of the discovery meta-service, allowing businesses and trading partners to find, discover, and inspect one another in Internet directories. The conceptual stack underlying the WS layer description is expressed in two dimensions. The stack layers run from transport specifications, through description aspects, up to discovery. Descriptions include both business-level and service-level agreements. Overlaying this stack are QoS, Security, and Management aspects. Unfortunately, each vendor, standards organization, or business defines Web Services differently – and in the case of Microsoft WS .NET Architecture, even differently over time. Despite repeated calls for broad interoperability and common standards, some vendors continue to use (or return to) proprietary solutions. The state of WS development is indicated by the general list of resources published and sorted by category at the Directory of Web Services and Semantic Web Resources (at www.wsindex.org). Discovery and Description Two main discovery and description mechanisms are used in WS contexts: UDDI (Universal Description, Discovery, and Integration) and WSDL (Web Services Description Language). They operate at different levels. UDDI is a specification that enables businesses to find and transact with one another dynamically. It functions as a registry interface maintaining only business information, each record describing a business and its services. The registry facilitates discovering other businesses that offer desired services, and integrating with them. Typically, UDDI entries point to WSDL documents for actual service descriptions.
Web Information Management
73
WSDL is an industry standard for describing Web-accessible services (that is, their Web interfaces). As an XML-based language, it is machine processable. However, the intended semantics of the interface elements are only accessible to human readers. In WSDL a service is seen as a collection of network endpoints which operates on messages. A WSDL document has two major parts: first, it specifies an abstract interface of the service, then, an implementation part binds the abstract interface to concrete network protocols and message formats (say SOAP, HTTP). Purpose of WS One way to classify Web Services is by their primary purpose. For example, a few broad application categories are: Administrative Metadata, Agents, Architecture, Brokering, Citation, Collection, Description, E-commerce, Education, Government, Libraries, and Registry. Professionals working within these application areas have increasing access to WS technologies, as these are developed and deployed, but only to the degree that they become aware of them. An example might illustrate the issues and directions. Efforts related to the management of metadata currently tend to focus on the standards and tools required by applications within the group’s main constituency. An early primary focus is e-license management for digital libraries. Librarians everywhere have increasingly been struggling with the management of electronic resources, but they have not known where to go for help in an area of the profession that seemed devoid of expert advice. For example, expenditures for electronic resources have grown enormously in the past decade, accounting for upwards of half the library budget in some cases. Yet the ability of library staff to manage this growing collection of resources typically remains unchanged; most of the time is still spent managing print resources. Commonly used integrated library systems do not provide tools to help manage electronic resources. Administrative metadata, elements about licensed resources, do not fit comfortably into most library systems. It is difficult for libraries to know what use restrictions might apply to e-journals and aggregated online texts, and how to triage and track related access problems. Most libraries simply lack the new kinds of tools required by these processes. Only in recent years have Web-based resources became available and been published in any greater way. One such resource, useful for tracking developments in this field, is the Cornell University ‘Web hub for Developing Administrative Metadata for Electronic Resource Management’ (at www.library.cornell.edu/cts/elicensestudy/home.html). The Web Hub is especially useful in highlighting e-resource management solutions implemented at the local level by different libraries. Such systems are typically developed by large research institutions, perhaps affiliated with a university library, but a growing number of vendors seem interested in developing generic products to meet the needs of most academic libraries. Whither WS? The entire field of Web Services remains in a state of flux despite several years of intense development, and not all proposed ‘Web services’ have very much to do with the Semantic
74
The Semantic Web
Web. In part, there has been a trend to call every new incarnation of e-commerce functionality an e-service, or if on the Web, a WS. In part, some confusion has been due to vacillation on the part of Microsoft (MS) in both its definition and deployment of .NET component technologies. Confusion and caution reign. On the one hand, there is the risk that the MS approach would fragment the Web into incompatible WS segments, simply due to the massive dominance by MS in the desktop market. On the other, there has been the fear that the new Web would drift into the embrace-and-extend sphere of proprietary-modified open standards, thus defaulting into a kind of corporate hegemony control of protocol and security by a single company. It is a fact that core MS WS runs on proprietary code only on MS Windows servers. A number of independent projects have been progressing under the guidance of several major actors. We will now look briefly at the major WS efforts, ending with Microsoft’s .NET. A realistic view might be that, ultimately, a successful deployment of SWS as a new Web infrastructure will require that all the actors who are committed and cooperating around open standards should achieve the vision, or at least interoperate with it. At present, however, the major players are in critical areas still (or again) pushing proprietary solutions, which risks locking corporate environments into vendor-specific commitments difficult to migrate from later. Fortunately, the WS paradigm is firmly message based, depending on XML, SOAP (or any other XMLP-compliant protocol), and other platform-independent specifications. This independence gives hope for bridging vendor divides. The Sun Approach Sun Microsystems promotes the J2EE platform for WS (java.sun.com/j2ee/), through the intermediary of the open Java Community Process (JCP, www.jcp.org). Both standards and architecture are open and distributed. The platform is based on Java, in theory independent of operating systems. As Sun invented the Java language and promotes competitor Unix-based operating systems, WS-platform debates often become slanted as a ‘Sun-interests against MSinterests’ confrontation. Such acrimony has often been detrimental to Web-universal solutions. For developers working in Java, the announcement in 2002 of an official suite of API specifications for Web services from Sun was welcome news. Sun had long been criticized for inaction on this front, given the early availability of the MS .NET SDK. However, rather than proposing transitional hack solutions to as-yet uncharted technical areas, Sun’s strategy had been instead to work with open source initiatives on the major pieces of the Web services puzzle. Also announced was the Java Web Services Developer Pack (JWSDP) to provide an integrated development platform. In the JWSDP, Sun packaged some reference technology implementations into a single, free download, adding some infrastructure to ease Web service deployment. Thus, the JWSDP let Java developers create useful Web services with available components, instead of forcing them to either assemble those tools themselves or
Web Information Management
75
use another vendor’s tools. Importantly, it is all open source, not proprietary code, and anchored in a developer community. Other aided tasks include compiling and registering a WSDL file for the implemented Web service, so that others can find it online and use it. Supported in this context is the UDDI v2.0 registry standard. Metering and monitoring capability as of JXTA v2.1 enables automated query and inspection of traffic and memory usage on remote network peers. Using Java makes core WS-implementations platform independent, though at the apparent cost of committing to a single language. In actuality, support for other languages does exist, though Java remains the primary language for implementations. Java is good for building complex systems but has itself been perceived as complex. Recent technology releases have positioned developers in a better position to address the challenges of building real-world Web services. Apache Group The Apache Group collects a number of evolving WS efforts at ws.apache.org. Since the group is historically defined by the web server, it is natural that several WS projects evolved here. Axis (ws.apache.org/axis/ ) is an implementation of SOAP vl.l, the proposed standard fulfilling the W3C XMLP requirements (see the protocol overview in Chapter 5). Essentially, it corresponds to what would have been Apache SOAP v3.0, a previous project, but rewritten from scratch. The project has focus on the basic building blocks for implementing WS primitives. Since Apache is a major server workhorse on the Web, having native SOAP support goes a long way to implementing WS applications on existing Apache servers, perhaps best expressed in the Tomcat and WebSphere application server contexts. Axis builds on the Apache notion of pluggable modules and APIs to extend core server functionality with advanced message processing and request forwarding to defined services. Other projects of interest include: WSIF, a simple Java API (invocation framework) for transparently invoking Web services irrespective of their location. WSIL, a distributed Web service discovery method (inspection language) specifying how to inspect a Web site for available Web services. Several other projects can also be found that are considered more experimental. DARPA DAML WS The DAML Services arm (www.daml.org/services/) of the DAML program (DARPA Agent Markup Language, see the DARPA section in Chapter 7) started by developing a DAMLbased Web Service Ontology (DAML-S), with early releases of specifications in 2000–2003. The DAML focus is mainly on the languages, specifications, and protocols involved in the client-WS communications, defining the common interfaces that should be shared by technologies developed by others.
76
The Semantic Web
In a subtle refinement of focus in November 2003, however, the renamed Semantic Web Services arm released the expected language implementation specifications as OWL-S v1.0 instead. OWL (Web Ontology Language, described in Chapter 6) received official W3C recommendation in August 2003, and this move toward implementation status has naturally affected other projects. The current OWL-S program also works on supporting tools and agent technology to enable automation of services on the Semantic Web. The focus is on descriptions of WS at the application level, describing what a particular WS implementation does, rather than specifying how it does it. Other WS description languages, such as WSDL, are specifications at the communications level of message and protocol, while discovery mechanisms such as UDDI serve mainly as pointers to service addresses and ports. DAML-S/OWL-S complements the lower-level specifications prevalent in these other approaches that are mainly concerned with the core platform architecture. DAML-S/OWL-S deals with the application-level issues of discovery, invocation, interoperation, composition (of new services from existing selections), verification, and execution monitoring. A service implementation must in this context specify two things: A service profile to describe what it requires from agents and what it provides for them. A service model (known as a Process) to describe workflow and possible execution paths. A profile would be used both by the WS implementation to advertise its availability and capabilities, and by requestors to specify their needs and expectations. Brokering services can then match requestors and providers based on their respective profiles. A third subontology may specify how service details are implemented (Grounding). Despite having generated a lot of interest through its promise to add semantics to WS descriptions, only a small number of these WS descriptions appear as yet to exist. Furthermore, most of these do not point to real services, representing instead constructs from the DAML-S community. A possible cause may be the perceived lack of sustained contact with and interest from the main implementers driving deployed WS-technology, Microsoft and Sun, both of whom have vested interests in promoting in-house WS languages and specifications. A 2003 Dutch study of the situation, ‘An experience report on using DAML-S’ (www.iids.org/publications/essw03.pdf, also see www.iids.org/pubs), concluded that DAML-S was superior to existing WS languages as it allowed the use of formally defined domain knowledge. However, it did not yet adequately address real-life situations. With regard to the following years of OWL-S development (v1.1 was released in November 2004), much the same concerns and conclusions appear applicable. Effort is being made to integrate WS made up of many different services into a semantic-interoperable context for practical business use. The DAML site references tools, security issues, example services, and a number of ontology/agent use cases, sorted by architecture and by language. Environment categories studied include B2B, Grid, Ubiquitous, Web, and Agent. The language-oriented categories
Web Information Management
77
include Advertising and Matchmaking, Negotiation and Contracting, Process Modelling, Process Enactment, and a catch-all Unclassified. Recognizing that information security and privacy play an increasingly important role in a world in which the Web is central to business, industry, finance, education, and government, several OWL-S categories have been proposed and developed to resolve such issues: Credential Ontology defines the capability to specify access control restrictions of Web pages or Web services that use authentication as a requirement for authorized access. Security Mechanisms Ontology defines the capability to interface on a high level of abstraction among various security standards and notations. Service Security Extensions Ontology defines the capability to annotate the security properties of SWS. Agent Security Extensions Ontology defines the capability to annotate the security properties of agents. Privacy Ontology defines the capability to express privacy policies to protect information, and a protocol to support matching of privacy policies across different contexts. Cryptographically Annotated Information Object Ontology defines the capability to capture encrypted or signed input or output data of services. One may see these efforts as a measure of technology maturity on the road to implementing practical services to realize the greater sweb vision. .NET Web Services Assessing the .NET (‘dot-net’) way of implementing an ‘intelligent Web’ and its related services proved difficult. Where the previous constellations of effort, even at Sun, had clear focus on technical implementation and open discussion among developers, with Microsoft one entered the realm of corporate marketing strategies and largely proprietary technology. Whatever the expressed (and shifting) aims of the .NET effort, the dominance of MS applications on the desktop and in the corporate intranet (and the way these applications work) also ensures that any Microsoft policy to ‘embrace and extend’ existing open or industry standards inevitably affects deployment of sweb technologies. Current MS overviews (for example at www.microsoft.com/net/ and msdn.microsoft.com/ webservices/) describe mainly XML Web Services. These XML-WS are somewhat loosely defined – in business terms by promised synergies, and in technology terms by selected client-server capabilities: Partner and client connectivity through exposure of useful functionality over an ‘industrystandard’ Web protocol (mainly SOAP over HTTP). Personal, integrated user experiences through smart devices that utilize sufficiently detailed descriptions of WS interfaces to allow client applications to talk to the services (based on metadata published as WSDL). Increasing revenue streams by enabling businesses to create their own WS, advertised through registry discovery so that potential users can find appropriate services easily (using UDDI).
78
The Semantic Web
Security aspects inherent in using open HTTP and site-dependent authentication systems are currently resolved by extending SOAP with security components, following the published WS-Security Specifications. Other, more secure transports are promised as soon as the technology matures and higher-level services are developed. The official view is regularly restated that Web Services ‘have made a lot of progress’ over the last few years, and that work on the platform continues. However, the long status documents from Microsoft (see www.microsoft.com/net/basics/) remain remarkably vague about practical applications, even after several years. The reader may consider it fortunate that although Microsoft has a long-standing bad habit of sowing serious linkrot by often and gratuitously ‘reorganizing’ its extensive Web sites, in recent years it does implement a modicum of redirection from older URLs, at least for a time. The previous URL paths have if fact ‘only’ suffered changes to mixed case. One functional service is Microsoft Alerts (www.microsoft.com/net/services/alerts/), by now fairly established. The original ‘.NET’ moniker in the name was dropped at some point. Alerts is a notification service that delivers personalized news, weather, and information to people on the move – in effect, a streaming-content aggregator service using RSS. It is hard to assess how pervasively used it is, but it has been promoted heavily for several years and integrates with the typical suite of Windows productivity applications. Other staple services offered (such as Passport, Messenger Connect for Enterprises, and MapPoint) are not ‘complete’ but are components requiring integration with larger business processes. The view expressed as early as 2001 remains curiously unfulfilled, despite the constant hype, that by simply subscribing to a rich selection of available ‘plug and play’ WS functions, developers could create a plethora of ‘richer user experiences’ quickly and easily. Larger-scale deployment of .NET services appear mainly as restricted trials within Microsoft itself, as described in a series of whitepaper studies. Development focus at Microsoft has been on ‘management services’ such as download center, content storage and retrieval, and generic Web platforms. Less seriously, Microsoft employees developed an internal service to track left-over snacks, SnackAlerts, which can notify subscribing employees where the snacks are in 79 Microsoft buildings around the world. Microsoft’s WS specifications are a moving target, evolving in many places in parallel – an ongoing process where they are defined, refined, and sometimes deprecated seeming at whim. Neither are we sure of who the major players are, or will be, since it depends on the practical scope of interoperability and chosen authentication mechanisms – how open MS chooses to make them. Therefore, it is impossible to say with any great precision what .NET is, or will be, until the respective applications are ‘fully deployed’ and functional. It is not enough to say it ‘is a set of software technologies for connecting information, people, systems, and devices’, yet many MS explanations say little more. Even the promise of ‘small, reusable applications’ in XML for WS seems to ignore the complications of real-world business applications. Perhaps the reason is that most of the implementations are buried in corporate systems.
Web Information Management
79
Bit 3.10 Visible MS XML-WS appears mainly to be a commoditized side-effect of selling Web-aware solutions to implement core corporate business processes This considered view is based on the fact that MS has long had only two core business models, supporting corporate solutions and selling the ubiquitous Windows desktop, combined with a proven track record of successful salesmanship based on the concept of convenience. Some amusement has been occasioned in the press over the fact that even lead spokespeople for Microsoft have had great difficulty explaining the ‘.NET vision’ intelligently or consistently. Developers in the .NET area might reasonably take issue with the apparent dismissal suggested by this critique, engrossed as they are in specific technical details that appear well-defined from their insider perspective, but many analysts agree that the overall MS strategy is muddled at best. Bit 3.11
Sweb WS is mainly an architecture model rather than a market sector
The Microsoft approach is typically characterized by a product focus on componentapplication-suite. Therefore, management appears genuinely confused in pursuing the ‘Web Services market’ and by how to transition from vision to product. Stated goals slide into obscurity each year as the next great initiative is announced. Is that any way to run a vision? One critical factor is that the design of MS software has often had lock-in effects on ensuing versions of hardware, making alternative solutions difficult. Consider how many companies are locked to the MS-Word document format(s) despite attempts to develop format-compatible third-party software. Many aspects of current .NET WS implementation require proprietary MS software at some level or another, possibly with lock-in features to specific hardware. Of particular concern is the introduction of a proprietary ‘information rights management’ (IRM) mechanism in MS Office 2003 and later applications. This implementation of IRM can block general access to Web-published documents produced using them. At the very least, it may prove legally impossible to use third-party applications to view such documents. Either way, it breaks the established concept of universal access to Web-published content. We cannot deny that there are legitimate reasons for corporate users (and even individuals) to desire effective content-control mechanisms. However, once deployed, it will be awkward and rare to produce any content without inadvertently ‘locking’ it for other applications and users. Bit 3.12
Current IRM/DRM mechanisms are a threat to the Semantic Web vision
When the dust finally settles, ways may have been found to let free exchange of information on the Web coexist with economic interests. Non-intrusive micropayment solutions and less draconian copyright and licensing enforcement appear to be necessary ingredients for such a resolution.
80
The Semantic Web
A mitigating circumstance is the recently adopted policy of the W3C to recommend particular technologies for the Web only if they are freely offered for the public good without restrictive ties. Although this policy ensures open standards in the infrastructure and WS, it does little to stem user adoption of needlessly restrictive technologies when producing and publishing Web content. DotGNU The DotGNU project (www.dotgnu.org) was started in reaction to, and aims to be a complete and open-source replacement for, Microsoft’s .NET initiative. It independently implements the Common Language Infrastructure (CLI) that defines ‘.NET’ functionality. Revisited in 2005, the project has evidently gained considerable adoption, claiming to be the industry leader and provider of Free Software solutions for desktop and server applications. Three major components of this initiative are currently being implemented: DotGNU Portable .NET suite of tools to build and execute .NET-compatible applications on any platform. The cross-platform suite includes, among other things, C/C# compilers, assembler, disassembler, and runtime engine. PhpGroupWare suite to provide multi-user collaborative Web-space, including many useful XML-RPC Web-service components ready to use in custom WS implementations. DotGNU Execution Environment (DGEE) to provide virtual server implementations and resource management support to run the WS implementations. The project goals are to provide a reasonably .NET-compatible system and then improve on what Microsoft is offering – for example, by decentralizing it. Therefore, DotGNU is based on a peer-to-peer system for service discovery, avoiding the use of centralized authentication/authorization portals such as the ‘Passport’ system. The strategy for DotGNU is not only technologically superior, it is claimed, but it also serves to ensure that no single entity would ever have the kind of monopoly power to control (or end) the free exchange of digital information. The latter is openly presented as a strong motivation: . . . to prevent Microsoft from achieving their stated goal ‘that the era of ‘‘open computing’’, the free exchange of digital information that has defined the personal computer industry, is ending’. Read the quotation in context and learn how the Free Software movement works to prevent this disaster. The contribution of the DotGNU project might turn out to be important.
In keeping with the open approach, the developer community runs DotGNU Wiki (wiki. dotgnu.org) as its forum of discussion and general Web resource. Using WS At this point, you might well wonder how to use Web services when the platforms are still evolving, and when there is so much flux, uncertainty, and lurking incompatibility. In fact, today’s Web service tools and frameworks, irrespective of vendor, do provide enough core functionality to build interesting distributed applications. Expect a steep learning
Web Information Management
81
curve, however, and be prepared to develop your own solutions, or to pay for proprietary solutions developed by others. Some higher-level services such as WS-Security are beginning to gel with the advent of support in a number of new tools (such as the Web Services Development Kit). Other services are still in their development stage as specifications are reviewed and early implementations expose areas where the specifications may need solidifying. Until then, you can rely on traditional HTTP-level mechanisms for security and other features. From the business perspective, some make the analogy that Web Services are similar to business services. The comparison is to a great degree a valid one, but at the same time it is constrained by the products and services view. However, just as most companies outsource aspects of their business process that are not their core competencies, such as shipping and materials handling, Web services might correspond to outsourced development of certain services they use all the time but may not wish to implement in-house. Authentication is the one such service implemented and deployed in a major way on the public Web. A substantial number of Web sites today rely on the .NET Passport authentication services, even though it has both suffered numerous embarrassments and received considerable criticism. The service is required when using any other .NET WS implementation. Originally introduced with an absolute dependency on Microsoft central servers, the current .NET approach appears to allow at least a limited delegation of authentication authority to other non-MS authentication networks. In late 2002, this security interoperability was mediated between MS and a large coalition of independents under Liberty Alliance (www.projectliberty.org) by MS-partner and LAfounding member RSA Security (www.rsasecurity.com). RSA, with its critical focus on Secure Web Services, and a deeper understanding of the positions on both sides of the divide, clearly had a pivotal role here.
Security and Trust Issues Reliability, availability, and security of data and services are goals that sometimes seem difficult to attain even under centralized control and governance. How then can one possibly realize them under the distributed and ad hoc management that characterizes in the Semantic Web, with its mix of roaming users and authority-delegated software agents? How indeed? Part of the answer depends critically on the adopted solutions for trust, discussed later. Central to the issue is a pervasive and transparent management of strong public key cryptography, especially in the form of digital signatures, complemented by webs of trust and proofing logic. This same technology can be used, not only to verify reliably unique person identities, even anonymous ones (!), but also to identify uniquely and make tamper-proof particular documents, document versions, and in fact any Web resources. It is for this reason that the visual map of the Semantic Web (Figure 2.5 in Chapter 2) includes encryption, signature, and proof logic as key parallel components.
The Semantic Web
82
Bit 3.13
Authenticated authority must work in a roaming context
As one of the cornerstones in the Semantic Web context is that access should be automatic and transparent regardless of location or used devices, the underlying authentication model must remain valid and secure across many ad hoc platforms. These same infrastructure components can also resolve some of the problems facing attempts to implement reasonable business models in a Web context. Technologies may change, but the essential requirements remain much the same, comprising the key concepts of Authentication, Authorization, Integrity, Signature, Confidentiality, Privacy, and more recently, Digital Rights Management and Information Rights management. Table 3.1 summarizes the meaning of these concepts. Another essential requirement of security standards in the Web context is that they work naturally with content created using XML (or with XML-derived languages and protocols). Transparency is another essential characteristic, in that integrity, confidentiality, and other security benefits should apply to XML resources without preventing further processing by standard XML tools – whether at message endpoints, or in intermediate processing. Although older security technologies provide a core set of security algorithms and technologies that can be used in XML contexts, the actual implementation of these is inappropriate for most XML applications: The proprietary binary formats require specialized software for interpretation and use, even just to extract portions of the security information. Older technologies tend to assume specific components integrated into the endpoint applications, introducing awkward dependencies. Older standards are not designed for use with XML and thus lack support for common XML techniques for managing content (such as URI or XPath pointers). Table 3.1
Summary of essential security concepts
Concept
Question answered
Comments
Authentication Authority Integrity
Who am I? (Verify asserted identity against some trusted authority.) What may I access and do? Is the information intact?
Signature
Is the information certified?
Confidentiality
Is the information safe from unauthorized disclosure?
Privacy
Is individual and sensitive information safe from unauthorized disclosure? How may I use or share this information?
The later Trust section further discusses authentication, identity, and role issues. Individual and role. See above. Prevent accidental or malicious change, or at least detect it. Ties in with identity issues. Might certify an identity or authority. Encryption makes information unreadable even if access controls are circumvented. Governance issues of how to use sensitive information. Consent.
Digital Rights Management
Usually now a combination of access control and embedded enforcement of usage license.
Web Information Management
83
A unified and open framework of new sweb-oriented standards and implementations, however, is evolving to address these issues on the Web. Since public and corporate awareness seems dominated by a focus on Microsoft’s .NET solutions, it is important to explore the subject in general terms. In particular, we need to highlight the alternatives to central authentication by proxy authorities, with their proprietary interpretations of ‘trust’ and ‘security’ as a ‘product’ to sell, in order to assess properly the role of these concepts in the broader Semantic Web context.
XML Security XML Security defines a common framework with processing rules that can be shared across applications using common tools. Therefore, no extensive customizing of applications is needed to add security. While XML Security reuses the concepts, algorithms, and core technologies of the legacy security systems in the interests of interoperability, it also introduces the changes necessary to support extensible integration with XML. The core XML Security standards comprise two Semantic Web cornerstones seen in the map and stack views (Figures 2.5 and 2.6) introduced at the end of Chapter 2: integrity and signatures, as XML Digital Signature (XML DigSig); confidentiality, as XML Encryption (XML Enc). Persistent integrity and security is associated with the content itself, rather than with any single transport, processing, or storage method. The implementation thus provides seamless end-to-end security, irrespective of intermediate processing, and reusing existing technologies when possible. XML Security realizes these goals by the simple expedient of transforming binary representations (such as for digital keys, certificates, or encrypted data) into plain text – or rather to a well-defined subset of text (such as Base64) that is guaranteed invariant in other message processing on any platform. Residual issues do remain in the conflict between exact representation in signed text and digests, for example, and the relative laxness in XML that allows whitespace variations and some token substitutions (such as single and double quotes) in ‘identical’ messages. Such issues are generally resolved by transforming to ‘canonical’ form before applying a signature. Related supporting core standards not explicitly on the simple sweb map are also needed: Key Management, as XML Key Management Specification (XKMS), defines protocols for the public-key management services and identity binding needed to implement data integrity and signatures. Authentication and Authorization Assertions, as Security Assertion Markup Language (SAML), define authentication and authorization assertions and shared framework used to establish identity and authority. Authorization Rules, as XML Access Control Markup Language (XACML), state the rules and policies used to manage access.
The Semantic Web
84
In addition to these standards, we must add major XML Security applications: Web Services Security, as WS-Security specifications that outline how XML DigSig and XML Enc may be used with XMLP (SOAP) messages. Also specified is how security claims (for example, identity credentials) may be included with a message. Privacy Management, as specified in the Platform for Privacy Preferences (P3P), defines the XML vocabulary to manage human-policy issues ensuring that organizations and individuals with legitimate access to information collected about an individual do not misuse it. Digital Rights Management, in the form of extensible Rights Markup Language 2.0 (XrML), provides a framework, a language and processing rules to enable managed interoperability in content workflow with multiple participants and applications, and among different ‘viewers’. We might note in the context of WS-Security that Microsoft and IBM released a Web Services security architecture and roadmap in 2002 that outlined a strategy and series of specifications to bring together different security technologies. Despite the stated aim of providing a unified, flexible, and extensible security framework for WS-Security, the core group of IBM, Microsoft, Sun, and VeriSign subsequently started pushing proprietary solutions instead.
Trust The concept of trust has both formal and informal components, but in both cases ultimately it is based on accepted knowledge, such as an established identity. In the more formal contexts, this knowledge requires some form of explicit verification; in the informal ones, it is implicit (typically in a shared history). Bit 3.14
Both sweb agency and e-commerce require secure authentication
Whether implemented authentication mechanisms become tightly centralized or distributed trust systems remains to be seen. Perhaps a middle ground can be found. In this first discussion of trust, the focus is on the fundamental issue of authenticating identity assertions. Establishing trust rests on some process of authentication, and on some ultimate authority that can validate the assertion about a given identity. Opinion diverges about whether central authorities or distributed ones are most appropriate (or secure) for the Web. Using a central authentication proxy raises many concerns, from scaleability and technical reliability of systems not yet tested on s